Claude beat Gemini on a 150-page document test, but not for the reason you'd think

The context window isn't something you think about when you first start using LLMs like Gemini or Claude. It honestly doesn't affect most of the short interactions you have on these platforms either. However, that changes when you want consistent source fidelity in the answers you get from an AI chatbot.

For example, when you want to analyze a complex essay or a multi-part white paper, the context window in an AI tool like Claude and Gemini becomes particularly useful. For what it's worth, both Claude and Gemini tout their large context windows, but we must see whether these claims are backed by real-world performance.

The question was rather simple: how do the context windows of Claude and Gemini perform when dealing with a large document?

The real reason I decided to run this test

A recurring bottleneck with dense documents and two competing answers

Curiosity aside, there was a personal reason to find out which of the significantly larger context windows for Claude and Gemini was the better option. I regularly use AI tools for a variety of research-oriented tasks. For instance, I may have to summarize a chapter or find intra-textual references.

As much as I love NotebookLM for its source fidelity, I cannot always approach it for my regular queries. So, naturally, Claude and Gemini became my go-to options. The problem is the confusion I face when responses from Claude and Gemini differ significantly, especially when I submit a prompt with a large attachment.

Given that I deal with instances where I cannot afford AI hallucinations, I figured it was best to put the confusion to an end. Essentially, I wanted to determine which of the larger context windows worked when having the whole picture matters.

How I structured the comparison without tilting the odds

The 150-page document, the exact same prompts, and what I was actually measuring

As I said, I wanted to understand how both Claude and Gemini handle a sufficiently large document as context. However, I did not want them to consider a simple narrative either. On the contrary, I wanted both AI tools to handle a document with intra-textual connections and multiple content types, including paragraphs and tables. I decided to go with the detailed syllabus of a master's program that I was already familiar with.

There were also a few more steps to avoid any bias. One: I decided to ask each AI three questions:

1. A summary-level question
2. A retrieval that demands intra-textual reference
3. A synthesis requiring comprehension

These questions were selected because of their varying levels of skill demand. I also made sure to use the same prompts as Gemini and Claude. Two, I was comparing the responses from Claude and Gemini with my understanding of the document in question, rather than doing a face-to-face comparison between the AI tools. This way, I could point out where each AI tool worked well and where it left room for improvement.

Claude retained the full context; Gemini started dropping threads partway through

The behavior difference emerged in specific ways

As you can guess, Claude and Gemini did a decent job of answering all three prompts, and they were confident in their responses as well. However, except for one section, the performance was not particularly comparable. Here are some insights I gained while trying to understand the behavior of Claude and Gemini within a 150-page document.

The responses to the first prompt, which asked both AIs' tools to summarize the document's core argument, were comparable and equally good. Both Claude and Gemini provided detailed summaries of the syllabus, but I found Claude's response slightly more detailed. However, from a fidelity standpoint, both Claude and Gemini were in great places. Things changed soon, though.

The second prompt asked both AI tools to perform intra-text referencing, requiring them to connect two ideas from different parts of the document. Here, however, I noticed how Claude retained the full context, whereas Gemini missed some important points. For instance, both responses listed multiple courses under a single category, but Gemini missed several suitable options from different semesters.

The inference-based prompt also showed similar effects. Sure, both Claude and Gemini did infer from the text, but the breadth of the said inference was a different question. Claude included appropriate evidence and connected these points together before presenting a rather synthesized response. On the other hand, Gemini seemed to reiterate the surface-level meaning alone.

Therefore, if you are concerned about anything other than summaries, you have a choice to make.

What a larger context window actually buys you in practice

Token limits matter less than consistency, and consistency has real limits

In light of what I have learned from Claude and Gemini, I want to resume the discussion of the token window. For reference, the strongest model of Google's Gemini has a token window of 2 million. Claude's models, however, are set at 200K tokens. The 150-page document I used for this task was well within the token window for each tool. So, I wasn't really worried about the request hitting a wall either. But I now have a newfound perspective on how consistency matters more than the context window.

I don't mean to say that the context window doesn't matter, but there is no point in advertising a 1-million- or 2-million-token window if the model loses attention and its response degrades over time. Unfortunately, this happened with Gemini, which is the larger context window between the two. The larger context window doesn't help Gemini when it gradually misses pointers from the reference document.

In comparison, Claude has done way better at intra-document referencing and synthesis. It has been done more because of the model's attention span rather than solely because of the context window.

Gemini's 1M/2M token window still matters

Let's say, let's not believe that the larger context window of Gemini doesn't matter at all. There are times when you have to increase the context by two or tenfold, and this is where those 1 million tokens come to help you.

You may have a number of 200-page documents or a very wide report that needs to be analyzed. This is where Gemini comes to help. Of course, you will get a better response from Gemini when you use it for certain tasks, such as summarization, quick lookups, or finding the relationship between different documents, instead of diving deep into one.