3 hour drive? Go for it! The Baltimore cruise port is easy to access and parking is plentiful.
Another nice thing is that given the ports visited Customs is more relaxed. They know Bermuda doesn't put up with drugs and other illegal activity, and so it was pretty much just a look at the passports, 'do you have any alcohol or tobacco to declare', and off we went to the shuttle bus.