A couple of weeks ago I posted about that Williams Institute brief that said that there were nine million LGBT people in the US. Gary Gates averaged results from several problematic studies that asked people what their sexuality was and arrived at 3.5% of the population being LGBT, assuming that averaging several untruths of unknown merit produces the truth.
Unsurprisingly, the Wall Street Journal found that other demographers don't agree with his methods:
One problem they cited with Dr. Gates's findings is that they combine results from surveys with different sample sizes and interview formats. The California Health Interview Survey canvassed about 50,000 Californians in 2009 by phone, finding that 3.2% identified themselves as gay, lesbian or bisexual. In contrast, roughly 5,900 people took Indiana University's online National Survey of Sexual Health and Behavior in 2009, and nearly twice as many-- 5.6%--identified themselves that way.
"I think there are a lot of problems with every one of those data sets," says Randall Sell, associate professor at Drexel University's school of public health. A concern, he says, is that people are more likely to reveal their sexual identity via computer than by phone or in person.
There are problems with every one of those studies that Gates averaged together, said Professor Sell. The Wall Street Journal put together a small graphic to show just how much variance can be caused by methodology:
The lowest result in Gates's average was based on face-to-face interviews. Is anyone surprised that people are much less likely to tell someone they're talking to that they're queer instead of an online poll?
Also, does it make them any less gay or bisexual if they're not willing to say it in a face-to-face interview?
A companion piece in the Journal brings up those Witeck-Combs polls we hear about every now and then, which are also online polls (conducted by Harris Interactive). They produce another number, a number which, for no given reason, wasn't averaged into Gates's final number:
Other estimates are even higher than the top of the range Gates included. Witeck-Combs Communication, a Washington, D.C., marketing and public-relations company, has since 2000 been surveying gays and lesbians in a partnership with Harris Interactive. Based on roughly 150 surveys, Bob Witeck, chief executive and co-founder of Witeck-Combs, estimates that 6.7% of American adults are gay, lesbian, bisexual or transgendered. However, LGBT people also tend to be early adopters of technology, according to the firm's surveys -- which might also make them more comfortable joining an online polling panel such as Harris's. "It's possible" that explains the higher numbers, Witeck said.
"There's some risk and all forms of bias creep into methodologies and sampling," Witeck elaborated in an email. "Nonetheless, I think the Harris model gives us a high degree of confidence that any bias in this direction is helped by their propensity weighting process."
6.7% is a lot higher than 3.5%. Does anyone think that Harris Interactive is deliberately fudging the data? Finding how many people are LGBT is impossible for a variety of reasons I laid out in my previous post, but is there an accusation against Harris Interactive of adding another layer of conscious bias? If there is, I haven't heard about it. And yet their online survey produced different results than the online survey Gates cited.
How does Gates respond to these charges against his brief?
Dr. Gates says without more information about the validity of each survey, averaging the results is the best compromise. "You can make an argument they're all credible," he says.
If you ever wondered why people who study the hard sciences (physics, biology...) scoff at the social sciences and use quotation marks around the "science" part, it's because of statements like the one above. Falsehoods don't compromise to produce the truth. Studies asking different questions don't compromise to answer another question. It's like how teaching both evolution and creationism to students isn't a compromise - they both can't be literally correct at the same time.
Gates cites five surveys on sexuality and health in the US that asked people their sexual orientation. The percent who identified as gay, lesbian, and bisexual (combined) was, from top to bottom, 5.6%, 3.7%, 3.2%, 2.9%, and 1.7%. When it came to just bisexual people, the answers ranged from 0.7% to 3.1%. For just gays and lesbians, it ranged from 1.0% to 2.5%.
With transgender people, Gates cited studies that ranged from 0.1% of the population being transgender to 2.0%, based on what exactly was asked. Trans people agree about the meaning of the word "transgender" even less than gay people agree about the meaning of "homosexual," and lots of people we'd think would be included in a survey of transgender people simply identify as their actual gender while others don't have the language to describe how they're feeling.
If one makes the assumption that the LGB population is stable and clearly-defined enough to be found (I'm not convinced but I'll assume for the sake of argument), then it can't possibly be 5.6% and 1.7% at the same time. It's one or the other or neither, but there's no reason to believe it's the average of the two.
If one doesn't make such an assumption, then they don't put out press releases saying that the LGBT population is nine million.
I also take issue with Gates saying that "they're all credible." One of the studies focused just on California - do we hold presidential elections in California and then extrapolate the rest of the states out under the assumption that they'd all vote Democratic at the same rate? It'd sure be cheaper to run elections that way. Another only asked people ages 18-44; would it be democratic to just let those people vote in elections under the assumption that older people would vote the same way?
I haven't read them so I don't know how credible each is (Gates surely has and could have made a judgment about which was the most credible), but at least two weren't applicable.
The Journal's companion piece also provides another reason producing an number of LGBT people is impossible that I had never heard about before:
Many people's confusion with terms surrounding sexual orientation contributes to the difficulty in producing solid estimates. Randall Sell, associate professor at Drexel University's school of public health and administrator of the informational website GayData.org, has conducted cognitive testing among older Americans who say they are bisexual because they aren't familiar with the term -- "bi means two, and that must mean man and woman," Sell said, describing their reaction.
In our country of 310,000,000 people and thousands of subcultures, there are lots of people who don't know what the word "bisexual" means. You can hand them a paper to fill out and then get a number of how many of them say they're bisexual, but that number isn't going to be worth anything. It'll still be a number, though, and journalists are always willing to print numbers!
In the days since I first posted, lots of news sources have been reprinting the nine million number without questions and with few caveats. It reminds me of this column from the Washington Post's ombudsman about how journalists are notoriously bad at math, which cited the many mathematical errors that appeared in the paper and concluded:
Many newsrooms provide remedial math training, but that's not been done at The Post. It should be considered. And given the increasing usage of numbers in reporting and graphics, The Post should pay heightened attention to math and statistical literacy when evaluating prospective hires.
We read the press corps we have, not the press corps we might want or wish to have at a later time.
But the issue isn't just math, it's credulousness. While we're willing to challenge people on moral or philosophical grounds, journalists take numbers from someone with a big degree without question. Math is seen as a mysterious subject and numbers must all be taken as real because to question one means to question them all, and the world falls apart when the latter happens. (Plus it's hard to make a deadline if you read the fine print.)
The ombudsman also wrote about statistics specifically:
"I think what's going on is that when journalists see a number, they take it at face value and don't question it," Maier said. "With numbers, I think journalists tend to abdicate that scrutiny."
Martell agreed, explaining that those intimidated by math tend to "panic" when forced to deal with numbers.
"You don't really have to know that much about statistics to read a statistical paper critically," she said, adding that reporters often cite numbers and statistics touted in news releases without questioning their accuracy.
Why do journalists "abdicate that scrutiny" when they see a number? It's probably because, as one person interviewed put it: "I think we have a culture where it's okay to say, 'I'm a journalist, which means I'm terrible at math.'" A database manager said that it's "charming" for Post journalists to say they "can't do math" and that math errors are tolerated while spelling errors are not.
That's pretty much what happened regarding this number. The Wall Street Journal was willing to look more closely because they have a column devoted to statistics, but that's about it. Even though it seems on the low end to a lot of people and doesn't take into account all the people who would benefit from LGBT legislation, expect it to become an un-cited truth in future media reports.
img Alex Blaze