While there are countless females who promote feminist values, it is not often that men do the same. So when a male does come forward in defense of women, it’s easy to shower him in praise. But Matt McGorry says he wants us to hold his applause.

According to The Huffington Post, the How to Get Away With Murder star told Cosmopolitan.com in an interview that he thinks it’s easier for men to be feminists, especially considering he “doesn’t have to deal with the ‘level of violence’ and death threats women who speak out against sexism so often experience online.”

But beyond experiencing a less severe reaction, McGorry said that he is often lauded for being a feminist — something he thinks may be frustrating to some people.

“I feel like I might be upset if I was a person of color or a woman,” he told Cosmopolitan.com. “I can imagine that every time I’m getting positive grace for this, it might remind people of the difference in how people of privilege are treated versus people who aren’t [as privileged] are treated.”

For him, being a male and a feminist wouldn’t merit such praise in an ideal world because, if he had it his way, it “wouldn’t be a rare thing.”