We had a short time in Victoria last summer. From my review, which is located here: Westerdam review - Alaska by LisaP
, "Armed with a map, we took a taxi into town (CD$9) and walked along the main street. ... I suggest, if you are not taking an organized tour, to have researched the sites of Victoria, map out the ones of interest, and bring your map with you when you leave the ship. We were here from 6:00 pm until midnight, so we had a 5:45 dinner in the main dining room, left the ship (after the crowds) at 7:00, and were back onboard at 9:30 pm."
We had open seating dining on this cruise, so we were able to dine earlier very easily.