Analyzing media bias with AI
Is the media biased towards Democrats? Republicans? Someone else?
Media bias is a hot topic, but also hard to quantify. Is the New York Times more biased than Fox News? What are the ways that bias shows up? Is it due to selective narratives, lopsided emphasis, or something else?
Normally, answering these questions would be time consuming: a human would need to read and rate hundreds of articles. And even if you do that, there’s the obvious question: is the bias rater himself biased?
But AI provides a new approach. It can analyze an article in just seconds and apply the same standard to any article it reads. With AI, we can quickly analyze hundreds of articles across different news sources, and get a bias rating for each.
Let's get the data
To get started, we need to find and analyze hundreds of political articles. We’re going to focus on American politics, and analyze a handful of different news sources.
New York Times
Washington Post
CNN
Wall Street Journal
Fox News
AP
Reuters
In order to quickly gather data, I’m using a tool I’m working on called FetchFox. It’s a scraper to quickly find and analyze articles using AI.
For each article, we’ll want two things: a rating of the bias, and a justification for that rating. Here are the prompts for each field:
Bias prompt: Rate the political bias of this article on a scale from -5 to 5, where -5 is pro-Democrat, 0 is neutral, and 5 is pro-Republican. Do not equivocate, always give a number as an answer. Ignore the news source, ignore the author when rating. Focus only on the content for your rating.
Justification prompt: Give the reason for your rating. Your reason should be a few sentences, no more than 20 words, focusing on the reason and not just restating your rating.
The long prompts help guide the AI.
Ratings for each news source
I used FetchFox to crawl each of the news sites for political articles. It found around 20-80 articles per news source, for a total of over 300 articles analyzed.
It rated each article for bias, and the chart below shows the average rating for each news source. Negative numbers indicate left-wing bias, while positive numbers indicate right-wing bias.
The results match conventional wisdom. Most mainstream outlets like CNN, NYT and WP show moderate bias towards Democrats, while Fox News shows a stronger bias in the opposite direction. The wire services, AP and Reuters, are closer to neutral but still have a slight left-wing bias.
The most surprising result is the WSJ rating. It’s considered a conservative organization, but the AI gives it a slight left-wing tilt.
It’s also interesting to look at the standard deviation on each news source. This tells us if the news source has a broad range of opinions (higher standard deviation) or if it is more consistently biased (lower standard deviation).
Fox News is the most consistently biased, while the WSJ has the broadest range of viewpoints.
Why does the AI think there’s bias?
In addition to a rating, the AI can give a short justification. This gives insight into how bias shows up in each article. Some typical examples are below, and you can review all of them in the raw data linked at the end of this article.
Rated: 2 (left-wing)
Reason: The article emphasizes positive aspects of Democratic candidates and activities, creating a generally pro-Democrat perspective.
NYT: Trump’s Tax Plan Could Add to Debt Burden. Harris’s Plan Tracks Biden’s.
Rating: 3 (left-wing)
Reason: Highlights negative impact of Trump's tax cuts while mentioning Harris's undefined plans.
WP: Harris backs ending taxes on tips, weeks after Trump made similar pledge
Rating: 5 (neutral)
Reason: The article provides viewpoints and reactions from both Democratic and Republican perspectives without favoring either side.
WSJ: Senate Passes Online Teen-Safety Rules Targeting Big Tech
Rating: 5 (neutral)
Reason: The article presents a straightforward report on bipartisan legislation without showing clear favor to any political side.
Fox News: Trump legal cases in limbo after SCOTUS immunity ruling, freeing up schedule to campaign
Rating: 9 (right-wing)
Reason: The article highlights Trump's legal victories and portrays the Supreme Court rulings positively, suggesting a favorable view towards Trump.
Rating: 5 (neutral)
Reason: The article focuses on the factual issues around U.S. Navy shipbuilding without showing clear political allegiance.
Reuters: Trump back on social media site X ahead of interview with Elon Musk
Rating: 7 (right-wing)
Reason: Highlights Trump's claims and minimizes criticisms.
Most bias comes from what different media emphasizes, and the space they allocate to positive or negative spin on the same story.
Is bias increasing?
We can also analyze bias over time with a similar approach. Internet Archive has historical snapshots of CNN and Fox News that go back two decades, and we can scrape them with FetchFox. The same prompts will give us a left-wing or right-wing bias rating.
Here is the AI’s rating over time for CNN:
CNN was slightly left of center through the early 2010’s, and became more left-wing around 2015. The rise of Donald Trump is the likely explanation.
Here’s the same analysis for Fox News:
Fox News has been consistently right-wing for the past two decades.
Can we trust the AI’s rating?
Of course, AI isn’t the universal arbiter of political bias. Models like ChatGPT work like super-autocomplete, and their answers depend on both training data and system guidelines. An AI trained exclusively on NYT articles would give different ratings than one trained only on Fox News. Companies like OpenAI or Meta can embed their own directives. Any interesting follow-up analysis would be to compare AI models, and see if they agree on political bias ratings.
Get the raw data
You can view the raw data in a spreadsheet, and run your own analysis with FetchFox.