My friend and I went on a 5 day cruise on the Celebration our of Jacksonville in June of this year. We are both 50(ish) and had a wonderful time. I thought previously that anything less than a 7 day was missing something, but it really was fine.
We visited two ports (Nassau and Key West) and had two days at sea, one coming and one going. The difference in a 7 day is one more sea day and one more day in port.
Again, the 4 day will visit one port with one day at sea coming and one going. If it's better on the budget, go for it.
However, that being said, I think I would only go for less than 5 days if it was my only choice. If you have some flexibility in time and price, go for the longer cruise!
Enjoy and hope you have the ball we did!