Seeing like a survey panel
Q: Did you see more than 90% of LGBTQ adults in the US have had the Covid vaccine?
A: How could you even know that?
Q: From Twitter. And! Yahoo! News!
A: But…
Q: It makes sense, right? LGBTQ+ people are less likely to be on the anti-vaccine side of US culture wars, and there’s community experience with health activism
A: But there are queer and trans people in Alabama, not just in San Francisco. And a significant homeless population
Q: But that’s what the survey says
A: Two words: sampling frame
Q: Ok, what’s a sampling frame?
A: It’s the list you work from when you sample people: a list of phone numbers or houses or email addresses or workplaces or whatever. It defines the population you’re going to end up estimating
Q: So they’d just need a list of all the LGBTQ+ people in the US
A:
Q: Ok, yes, that would be scary. How did they really do it?
A: They had a list of some of the LGBTQ+ people in the US (press release, PDF report)
Q: Where did they get the list?
A: “Research participants were recruited through CMI’s proprietary LGBTQ research panel and through our partnerships with over 100 LGBTQ media, events, and organizations.”
Q: That sounds like it might not be very representative
A: “Because CMI has little control over the sample or response of the widely-distributed LGBTQ Community Survey, we do not profess that the results are representative of the “entire LGBTQ community.””
Q: Exactly. It might be useful for marketing, but it seems like it’s not going to be representative. They’ll miss some big groups of people
A: “Instead, readers of this report should view results as a market study on LGBTQ community members who interact with LGBTQ media and organizations. CMI views these results as most helpful to readers who want to reach the community through LGBTQ advertising, marketing, events, and sponsorship outreach. Results do not reflect community members who are more closeted or do not interact much with LGBTQ community organizations. More than likely, bisexual community members are also underrepresented in the results.”
Q: When you’re talking in italics like that, does it mean you’re quoting the report?
A: It does. Or the press release
Q: Sounds like they have all the right disclaimers
A: The disclaimers fell off on the way to Yahoo! News! and Twitter, though.
Thomas Lumley (@tslumley) is Professor of Biostatistics at the University of Auckland. His research interests include semiparametric models, survey sampling, statistical computing, foundations of statistics, and whatever methodological problems his medical collaborators come up with. He also blogs at Biased and Inefficient See all posts by Thomas Lumley »