Responding to your points in the order I prefer...
The thing is, keeping the solar panels on the Moon is a lot easier.
Not at all. The Moon is above an Earthly horizon for half a month and below it for the next half month. So your lunar panels would deliver no power for weeks on end, and do it every month. Completely useless. As I said, a dumb idea.
You could put a giant mirror in Earth orbit to reflect the lunar power beam during the periods when the beam would otherwise be off, but if you did that you may as well have a power satellite in geosync orbit. Not to mention the inverse-square advantages of having an SPS ten times closer than the moon, meaning the SPS would need a hundred times less collecting area than the lunar farm, and the receiving rectenna on Earth would be a hundred times smaller.
You do not need to go to Phobos. There is research on hybrid solid/liquid rockets which use Aluminum as propellant and LOX as oxidizer for example. The Isp is kind of on the low side but it is relatively easy to mine and process using Lunar materials.
When you turn an aluminum rocket off, the fuel has an inconvenient tendency to freeze: imagine a solid plug of metal blocking the paths inside the engine. How do you restart the engine in that case? If you want to slow down at the destination, you have to restart it. I wouldn't hold my breath while waiting for an aluminum-fueled rocket to be practical.
Besides, once the Phobos infrastructure is in place, the water will be almost free, as we can use a portion of the water mined from the Martian moon as propellant to move the rest of the water.
(If it turns out that Phobos has no water, then substitute some other body with abundant water, such as Ceres. Again, the water will be almost free.)
