"testicles," early 14c., from plural of ball (n.1). See also ballocks. Meaning "courage, nerve" is from 1928. Balls to the wall, however, probably is from World War II Air Forces slang, from the ball that topped the aircraft throttle, thrust to the bulkhead of the cockpit to attain full speed.
Ball-busting "difficult" is recorded by 1944; ball-breaker "difficult job or problem" is by 1954. Ball-buster, disparaging for "dominant female, woman who destroys men's self-confidence" is from 1954; ball-breaker in this sense is by 1970 (of Bella Abzug).