A new analysis of anonymous Google search data by data scientist Seth Stephens-Davidowitz has uncovered uncomfortable truths about human desires, prejudices and beliefs. The research suggests that people are far more honest in their online searches than in surveys, providing a unique window into the human psyche.
Stephens-Davidowitz explains that social desirability bias leads people to underreport embarrassing behaviours and thoughts in surveys, even when anonymous. Historical data from 1950 showed that Denver residents exaggerated their voting and charity records. Modern studies confirm little has changed: University of Maryland graduates misreported GPAs and donations to appear more successful.
By contrast, Google searches offer a 'digital truth serum,' says Stephens-Davidowitz. The conditions of online searches—alone, anonymous, no interviewer—encourage honesty. He spent four years studying this data, revealing insights into mental illness, sexuality, religion and health. He calls it 'the most important dataset ever collected on the human psyche.'
The analysis addresses tough questions, such as the true proportion of gay American men, which surveys have struggled to answer accurately. The Google data suggests that people reveal their genuine concerns and behaviours when they think no one is watching.
