I have to agree with you. Unless you are used to driving on the left hand side of the road, and are experienced in horrendous traffic, either take a tour or use public transportation.
Just getting to seven mile beach s an ordeal in a taxi. I couldn't imagine driving in that chaos.
If it is your first visit to Grand Cayman, enjoy the town and seven mile beach.
Just my opinion,