If they say 4pm that should be the actual time you leave. Embarkation usually starts after 11am. This way they are able to turn around all of the cabins & public areas.
The time for embarkation can vary. My last cruise I had no waiting at all & was on in minutes. The luggage process does take some time to actually arrive to your cabin.
So if you think you want to change clothes or into a bathing suit put it in your carry-on.
As for disemarkation it is usually an early day(sorry)
If the actual process does not start until 11am though, you may be in luck.
Usually you have your bags packed the night before & leave them out. Then in the morning they like you to leave your cabin early so they can clean. Again, the entire process can go fast or slow. Does not sound like Customs will be an issue, so that saves time.
Have a great cruise