I finally got around to fixing this. I dissected a few mirror in the junkyard and just replaced the mount since I didn't want to deal with repainting.
And yes the side mirror just pops off. The backside of it looks like this:
Without the mirror, it looks like this (except the upper spring is missing:
You take that apart (now you can see the second spring back there, it's not attached to anything):
You remove the mirror to leave just the mount:
Then you remove the tweeter:
Then you remove the foam piece between the tweeter and the mount:
Then you remove the mount (I pushed the side mirror wire to the outside to avoid having to remove the door panel). You undo those three bolts then push:
Then just bolt on the new mirror mount and reassemble. Pulling the springs back was the only challenge, but I still managed to do it myself.