Is Castries a good walk about?
We did the southern caribbean last summer (San Juan, St. Thomas, ST. Maarten, Antigua, St. Lucia and Barbados), and in each port, we walked around the main towns near the port.
Of all of them, we disliked St. Lucia the most. It reminded us of Tijuana, but worse. We took a cab from the port to the center of town, but didn't stay more than 1.5 hours because of the way it was.
My suggestion is to go on a tour of the island. It is the most beautiful place. . . if you forget Castries!!
Just my .02!!