I meant to reply to this sooner...It looks like you're going in May, so it will still be fairly cool when you're there.
Bermuda has a million things to do and having three days makes it so relaxing! Crystal and fantasy caves are amazing.They are right next to the Swizzle Inn, home of the rum swizzle. Horshoes beach is beautiful as Penny said, even just to look.
Even if they're not beach people, it's really fascinating to see Bermuda by boat. They have many tours that include glass bottom boats and other narrated tours. Bermuda has beautiful coral reefs. I enjoyed just hopping on the bus and riding around the island from end to end.
There are also tons of great restaurants like the Hog Penny pub in Hamilton or the BoneFish or the Frog and Onion right in the dockyard area. Although the food is expensive, the drinks are comparable to the ship and we usually would just order appetizers and save the big eating for the ship.
You will have a wonderful time...just remember to pack some rain gear. Bermuda gets an averae of 180 days of rain a year!