Yes, and it's easy! In the Map Location screen, there is a field called "Tooltip." Edit this field to specify the description of each location. This is what appears when someone hovers over those location pinpoints.
It looks like this in the wp-admin:
And this is how it will appear on your map:
