Wendy, I cruised from New York to the Caribbean a couple of years ago. If you want to spend time outdoors during the first and last couple of days of your cruise, you'll need warm clothing. Of course, the further south you get, the warmer it will be. On the first and second days out of New York, I wore long pants, a long sleeved shirt and a jacket to sit on a chair on an open deck, and I was not really warm enough. Once doing some brisk walking around the deck, I warmed up. But if you're going to sit outside, do bring something warm to wear, possibly whatever you plan to wear to travel to the ship in Baltimore. You'll have it with you, anyway.
Of course, there will be a ton of things to do inside the ship, so you won't need to step outdoors if you don't want to!
Have a great time.