I guess it makes more sense than slamming your testicals against the wall.
In a way relating to human anatomy that has caused me to remove this phrase from my usage in recent years (because I worried how others would take it) the balls=testicles actually always made sense to me, but I’m not going to explain it.
However, now that I know what the most literal interpretation of the phrase actually is, I can feel safe using it again!
Do you happen to have a source for that? I’m not challenging you, I want to have a good one handy for other discussion, and don’t.