We were on the Legend this June and thought it was all fantastic as you say. The food, service etc, were second to none and we would not hesitate to sail on her again. We also sailed on the Explorer last year (same size/class of ship as Navigator) and whilst we had a great time - found that the two ships in many ways can not be compared. The Voyager class ships are just huge and spectacular to see, but yes, loads more people! There are many more bars to drink at and speciality restaurants. But we love the dining room food and never used the speciality restaurant anyway! Also, the passengers on the Explorer were in the main a lot younger than on the Legend. I don't know how old you are, but we are both in our early 30's and sometimes found the Explorer a little too frantic with all the constant activities (including lots of pool games and belly flop competitions!) and if you just want to relax and chill out on your sea days, you can't beat Legend!
Another big thing we noticed on Legend, was that all (and I mean EVERY staff member, from the Captain down to the maintenance personnel) would make a point of saying hello when they passed you in the hallways. We loved this and DID NOT get this treatment on Explorer. I don't know if we were unlucky in this, or if it's a big ship/little ship thing....
I don't think you can have a bad cruise which-ever ship you choose - it all depends on what you are looking for from your holiday. This was just our experience!
Have a great cruise!