My first question would be, did you ask your travel agent if s/he thinks Silversea is too sedentary for you? I'm asking because s/he knows you and we don't, and should know what you're looking for in a cruise, your previous vacation experiences, what you liked and didn't like about them, your lifestyle, etc.
Having said that, I'll add that I'm sure you know that the more upscale a cruise, and the longer the cruise, the older the demographic of the average passenger will be. I don't think you'll be the only ones in your age bracket on Silversea, but that doesn't answer the question of whether the cruise will not be lively enough for you.
One other option to consider is a larger suite on one of the cruise lines that may serve a younger demographic and where the ships are larger, giving you more optons as to activities and entertainment (but also more of your fellow passengers). Crystal comes to mind, or even a mass market cruise line, again, depending on what you're looking for.
Again, if your travel agent has served you well in the past, I'd suggest discussing your concerns with him or her, to make sure Silversea is the best cruise line to suit your needs.