Thanks for the comments. I called the tour operations. You guys are the experts. How does the plan below sound?
1. Tour starts at 10 AM from Visitor's Center
2. Narrated tour (Sightseeing) 10 - 10:30
3. Stop by the Cliff Walk for 15 mins if weather is good
4. Continue the narrated tour til 11:20
5. Drop off at the mansion of our choice (end of the guided tour)
6. Visit the Breakers for 2 hours
7. Go to the Cliff Walk from the back of the Breakers. Walk north to the beginning of the Cliff Walk
8. Take the yellow trolley back to Visitor's Center (They will us a $5 one way ticket.)
p.s. They said that bring your own water and snacks. Restaurants are only in the Marble House and the Elm.