I agree about taking a taxi from Penn Station.
A couple of caveats, other than traffic:
Taxi lines are sometimes very long. It's POSSIBLE that you can exit Penn Station on the 7th Avenue side and walk across the street to the Pennsylvania Hotel, where you may be able to get a taxi faster than on the end of a long taxi line. The line, however, does move pretty quickly.
There is also an exit from the station to 8th Ave, which is closer to where you want to go (the NYC cruise ship port is at 55th St and 12th Ave) but there are fewer taxis that come to the 8th Ave side of Penn Station. The escalator up to the 8th Ave exit of the station is actually closer to the train tracks.
There are cruise ports in Brooklyn and in New Jersey, but we're assuming you're talking about the New York City Port Authority terminal in Manhattan.