When I was on the Victory in June, my husband and I both wore shorts to dinner (on the cruise casual nights). My husband wears khaki-type shorts and a shirt with a collar. I wear either capri's or bermuda shorts and a nice blouse. With airlines charging the extra baggage fees, I will pack as little as I can get by with.
FYI, I did see two guys in gym shorts, ratty T-shirts and flip-flops get turned away from the dining room. Though honestly, as long as I don't have to see body parts peeking out, I couldn't care less what other people wear in the dining room.
Speaking of peeking-out parts, some of those "formal" gowns - holy cow!