Everything we know by heart enriches us and helps us find ourselves. If it should get in the way of finding ourselves, it is because we have no personality.
I've been a woman for a little over 50 years and have gotten over my initial astonishment. As for conducting an orchestra, that's a job where I don't think sex plays much part.