It depends upon how much luggage you have. And, I suppose, on the weather.
Believe it or not, it will likely be faster to walk from Columbus Circle than to take a taxi. That's because the queue to get up the ramp to the terminal is outrageously long when there are ships in port. Same for your return trip.
If you can roll your luggage without much effort, it will save you both time and $. But, if neither is a consideration, or if you do not wish to transport your luggage on foot, you can cab it from Penn Station. You might also get off at Columbus Circle and flag a taxi there or by one of the nearby hotels (Es*** House on Central Park South is the closest).