Well, you certainly not go from Portoferraio since that is the main city on the island of Elbe (you would have to be quite a swimmer to get to the mainland). From Portofino it is possible, but unless you are on a tour or have your own rental car its a bit of a hassle. You would have to get a bit inland to the nearest train station which will take some time not to mention the waiting time for the next train. If you are interested in doing this let me know here with a post and I will do a little homework on the best train station and connections. I am guessing you do not have a lot of experience in Europe so your best bet might be to take a cruise line tour (I personally hate their tours but its good if you are not adventuress) or hire a private car/driver/guide for the day (best pre-booked on-line). The other option is to rent a car and drive yourselves, but I hestiate to recommend this unless you have some experience driving in Italy (its crazy but fun).