I used to point out how few women were invited to participate in conferences at the Oberwolfach mathematics research center, until someone said to me "If you weren't invited to Oberwolfach, it must be because you're not good enough to be invited."
From this and many similar experiences, I learned that my observations and suggestions are more likely to have a positive effect when I don't stand to benefit. I have more credibility when I'm the only female invited speaker, than when there are no female speakers.
It's more effective when someone else speaks up on your behalf.
But when the speaker list is all male and probably shouldn't be, who should speak up?
In 2006, at my suggestion the Association for Women in Mathematics enacted a policy that anyone can ask the AWM President to send a letter to organizers of a conference reminding them of the benefits of considering potential speakers from a wide pool so that good people are not overlooked (if not for their current conference, then for future ones). Suggestions included having a diverse organizing committee, making an effort to think of potential speakers whose demographics, mathematical interests, or geographical areas aren't represented on the organizing committee, using the Mathscinet searchable database to check that important areas or people aren't overlooked, and contacting colleagues and organizations that could help suggest names or ways to go about finding them. We referred to it as "The Letter" (as in "I wonder if those conference organizers might benefit from being sent The Letter").