X Algorithm & Politics: Bias, Echo Chambers & Impact

A study examining the effects of algorithmic and chronological feeds on X, formerly known as Twitter, found differences in user experience and attitudes. Researchers conducted an experiment with participants recruited by YouGov, a professional online polling firm.

Participants

Study participants self-reported using X at least several times a month. The sample was 78% white, 52% male, and relatively well-educated, with 58% having completed at least four years of university. Political affiliation among participants was 46% Democrat and 21% Republican. Sixty-six percent of participants used X at least once a day, and 94% at least once a week. Twenty-seven percent posted at least once a day, and 53% at least once a week. Detailed summary statistics and a comparison with nationally representative Twitter users are available in Supplementary Information section 1.4.

Experimental Design

Participants were randomly assigned to either an algorithmic or chronological feed in exchange for compensation. Information on the compensation scheme and the experiment’s appearance on X is available in Supplementary Information section 1.2. The randomization process was effective, with no systematic differences in demographic characteristics or social media usage between groups beyond what would be expected by chance. The only notable difference was that 77% of those assigned to the algorithmic group were already using that feed, compared to 75% in the chronological group. All analyses accounted for initial feed settings, with further details in Supplementary Information section 1.7.

Data on Outcomes

Post-treatment survey data was used to assess attitudes. Natural language processing methods, using Llama 3-based classification, were applied to analyze feed content and followed accounts, categorizing posts by political leaning (conservative or liberal) and type (political activists, entertainment, news media). Details are available in Supplementary Information sections 1.5 and 1.6. Validation exercises for the Llama 3 annotations, including comparisons with machine-learning classifiers and human annotators, are found in Supplementary Information sections 1.5 and 1.6.

ITT Effects Estimates

The study used a specific model to estimate the effects of switching feed settings, comparing outcomes between those assigned to algorithmic and chronological feeds, based on their initial settings. The model focused on the effect of switching the algorithm on for users who initially had it off, and vice versa. Unconditional estimates controlled for initial feed setting and pre-treatment values, while estimates controlling for pre-treatment covariates used Generalized Random Forests (GRFs). Baseline ITT estimates are shown in Fig. 2. Covariates used in the GRF approach included initial feed setting, gender, age, X usage indicators, educational attainment, race, frequency of use and posting, political affiliation, device used for the survey, life satisfaction, happiness, and affective polarization. Technical details are in Supplementary Information section 2.2.1.

Throughout the analysis, ITT estimates are presented graphically, with unconditional specifications in blue and those controlling for pre-treatment covariates in orange. Estimates labeled ‘Chrono to Algorithm’ represent the effect of switching to the algorithmic feed, while ‘Algorithm to Chrono’ represents switching to the chronological feed.

Additional analyses in Supplementary Information section 2.4 included controlling for demographic characteristics, pre-treatment X use, and political affiliation. Results were also compared using a specification that included the predicted probability of having an algorithmic initial feed, obtained using LASSO. Controlling for post-treatment X use was also performed to assess robustness.

Tests for symmetry in the effects of switching the algorithm on and off were conducted, examining whether β₁ = − β₂ and whether 7 weeks of algorithm exposure had similar outcomes to those initially on the algorithmic feed. Results are in Supplementary Information section 2.4.

Compliance and LATE Estimates

Due to minimal non-compliance, the study also estimated Local Average Treatment Effects (LATE) for compliers using instrumental variables regressions, with assigned feed setting as the instrument. Actual feed use was determined by treatment assignments and self-reported compliance. The instruments were strong, with F-statistics exceeding 1,000 in all specifications. LATE estimates are reported in Extended Data Fig. 3, and are similar to the ITT estimates given the high compliance rate (85.38%). Additional LATE estimates using a more conservative compliance definition are in Supplementary Information Fig. 2.13. Results remained robust when restricting the analysis to respondents whose compliance was confirmed through observation (Supplementary Fig. 2.12).

Attrition and Lee Bounds Estimates

Significant attrition occurred between pre- and post-treatment surveys, but attrition rates did not differ across treatment groups. Lee bounds were used to assess potential bias from selective attrition, concluding that attrition did not meaningfully affect the findings. Details are in Supplementary Information section 1.8.2, with results in Extended Data Fig. 6.

Ethics Statement

Ethical approval was granted by the Ethics Committee of the University of St. Gallen, Switzerland. Informed consent was obtained from all participants. Participants received compensation in points, with an exchange rate of 1,000 points equaling US$1.00. They received 500 points (US$0.50) for the pre-treatment survey, with an additional 2,500 points (US$2.50) for using the assigned feed and completing the post-treatment survey. Additional compensation was available for using a Chrome extension. The experimental interventions were limited to feed settings, with no manipulation of user content. Additional ethical safeguards are detailed in Supplementary Information section 1.1.

Further research design information is available in the Nature Portfolio Reporting Summary.