I tried like crazy to get it off, and didn't succeed. I used the ball joint puller with no luck, and so I grinded it off. Now the fun part is drilling up through that bolt thats left, and popping the new ball joint back in.
Would have been easy drilling from the top of the bolt, but the dang...