Can AI judge literature – and write it? A 2023 test, repeated with the models of 2026

In short: In 2023, a widely discussed study showed that AI could neither write good stories nor judge literature. We repeated it with the models of 2026, using the same stories, the same tests and the same expert verdicts. The result: AI stories now pass 77–91% of the literary tests instead of 2–13%, and GPT-6.1 Sol is practically level with the originals from the New Yorker. The AI raters now reliably recognise whether a text is literature, but not what a single text is missing. The remaining gap between human and machine is called originality. As of 19 October 2026.

A few days ago we explained here how we choose the writing model for Hermes 3000: with an open benchmark in which three AI raters compare texts blind. One question remained open, and it's an uncomfortable one. If one AI judges the texts of another AI, how do we know its verdict is worth anything?

Literary scholarship gave a clear answer to that, at least in 2023. We wanted to know whether it still holds.

The 2023 study: "Art or Artifice?"

In 2023, a team from Columbia University and Salesforce Research led by Tuhin Chakrabarty examined what distinguishes AI stories from good literature (study, CHI 2024). To do so, the researchers had writers, creative writing instructors and literary agents describe how they recognise good prose. From this they built 14 yes/no tests, modelled on a classic creativity test (the Torrance Test), in four areas:

• Fluency: Is the story coherent and understandable? Is the pacing right? Does it show rather than merely summarise? Is the ending earned?
• Flexibility: Does the text shift perspectives, moods, structures?
• Originality: Is the idea, the form, the theme surprising?
• Elaboration: Are the characters developed, is the world tangible to the senses, is the language layered?

Then GPT-3.5, GPT-4 and Claude 1.3 wrote twelve stories based on the plots of real New Yorker stories, the same length as the originals. Ten experts assessed the 48 stories, each story by three of them, without knowing which was written by whom. The result was clear: the originals passed 85% of the tests, the AI stories 9 to 30%. And when the researchers used AI models themselves as raters, their verdicts were at chance level.

The nice thing about the study: the authors published everything – the tests, the AI stories and all 2,016 individual expert verdicts with their reasoning. They are not allowed to pass on the twelve New Yorker stories themselves; we used them for the analysis through a subscription, and they are not in our repository.

Step 1: Can today's AI raters judge literature?

Before trusting AI raters, we had to test them. We had the three raters from our benchmark – Claude Opus 5.5, GPT-6.1 Sol and Gemini 4 Argon – apply the same 14 tests to the same 48 stories, using the wording of the original prompts, and compared their verdicts with the experts' majority verdict. For the study's old raters, our procedure reproduces exactly the published figures; so we are measuring with the same yardstick.

At first glance the difference is dramatic. Agreement with the experts, measured as κ (0 = chance, 0.4 = moderate, 0.6 = substantial):

That is as much as or more than the experts achieve among themselves. A result that good is suspicious, so we took a closer look.

Most of it is the distinction between human and machine. With the experts, the New Yorker stories pass around 90% of the tests, the 2023 AI stories around 15%. Today's raters hit both levels almost exactly; the old raters, by contrast, waved almost everything through (Gemini Pro said "yes" to 97% of all tests). If you look only within a group – does the rater recognise which test a particular story fails? – the progress is small:

So two things work today, and one doesn't. The raters reliably recognise what level a text is at. And when you rank AI stories by the number of tests passed, they even agree with the expert majority slightly better than a single expert agrees with the other two. What they cannot reliably say is which weakness a single text has. Then again, the experts rarely agree on that for weaker texts either.

Step 2: Do AI models write better stories today?

Now the actual question. We repeated the 2023 experiment: the same twelve plots, the same task ("Write a story in the style of the New Yorker based on this plot"), the same length matching to the original. This time the 2026 models wrote, and all stories – originals, 2023 and 2026 – were assessed by the same three raters with the same 14 tests. A test counts as passed if the majority of raters agree.

Our raters are stricter than the experts back then: they give the 2023 stories only 2–13%, where the experts gave them 9–30%. For the comparison this doesn't matter, because all the verdicts come from the same raters. The leap remains: from a tenth of the tests to three quarters up to almost all of them.

Where the remaining gap lies. In fluency and elaboration, all 2026 models reach over 90%, as much as the originals. The difference lies in originality: the New Yorker stories pass 83% of these tests, Sol 72%, the other models 31–53%. In terms of craft, the machines now write on a par, but they surprise less often.

The writing specialist. Hemmingway-1, a model trained explicitly to "write like a human", comes last in the field with 74% and has the lowest originality score (31%). It also barely kept to the length requirement, even after many rounds of correction.

Did the models simply copy?

That was our first suspicion. The New Yorker stories were published from 2021 to 2023; a 2026 model could have read them during training and, with the plot as a cue, retold them from memory. That would explain a leap without the models writing any better.

So we compared every story word for word with its original:

• Not a single 2026 story shares even one sequence of eight words with its original.
• The longest shared word sequence is five or six words long, typical everyday phrases. The same is true of the 2023 stories.
• Shared four-word sequences make up 0.08–0.22% of a text (2023: 0.04–0.05%).

The models wrote their stories themselves. What this rules out is retelling from memory – not that a model knows the style and construction of the originals.

And the length?

Longer texts tend to pass more tests, because there is more room for characters and world. Gemini 4, Luna and Hemmingway-1 wrote too long despite repeated instructions. So we calculated the scores a second time using only the stories that deviate from the original by at most 200 words. The results change by at most one percentage point, by three for Hemmingway-1. The excess length gives no model an advantage.

An attempt that didn't work

We also wanted to apply the tests to the German texts from our benchmark, and to set a human yardstick next to them: the openings of twelve German novellas by Kafka, Musil, Zweig, Fontane, Thomas Mann and Schnitzler, cut to the same length as the tasks.

The result was surprising and instructive: on 300 words, the classics scored worse than most of the models, even though the raters recognised every single text. The reason lies in the measuring instrument. The models wrote self-contained scenes; the classics provide novel openings that don't have to resolve anything after 300 words. The tests reward what a finished story shows. On excerpts, this instrument does not measure literary quality. Only complete texts are suitable as a yardstick, and that is exactly what we did in the experiment with the New Yorker stories.

What we don't know

One caveat matters to us. We only validated the raters on New Yorker stories and on 2023 AI stories. The 2026 stories are new territory for them, and the raters are themselves 2026 models. It is conceivable that they rate the style of their own generation more highly than human experts would. Whether experts really see the best AI stories today as almost at New Yorker level can only be settled by a human cross-check. We're planning that next in our Text Arena.

What this means for Hermes 3000

• The model choice holds. Our benchmark relies on exactly what AI raters do well today: ranking texts by quality. GPT-6.1 Sol, available in Hermes for paid books, is ahead here too.
• Pointers, not verdicts. When Hermes comments on a passage ("the ending feels abrupt"), that is a pointer, not a finding. According to the study, exactly this kind of individual verdict is unreliable, in humans and machines alike. Things that can be checked, on the other hand – breaks in logic, timeline errors or shifts in perspective – the models detect well.
• Originality stays with you. In terms of craft, models can do an astonishing amount today. What they manage less often is the surprising: your own idea, the unusual form. That is the part of a book that has to come from you.

Frequently asked questions

Can AI models judge literature in 2026?

Partly. They reliably recognise whether a text is at the level of professional literature, and rank texts by quality about as well as a single expert. They cannot reliably name the specific weakness of a single text.

Do AI models write as well as humans today?

According to our tests, the best 2026 models meet almost as many criteria of good short stories as published New Yorker stories; GPT-6.1 Sol 91% against 92%. The gap lies in originality. The rating was done by AI raters; confirmation by human experts is still pending.

Did the models copy the original stories?

No. No 2026 story shares a sequence of eight words with its original; the longest shared word sequence is five to six words long.

What is the Torrance Test of Creative Writing?

A set of 14 yes/no questions on the quality of a short story in the areas of fluency, flexibility, originality and elaboration. It was developed in 2023 by researchers from Columbia University and Salesforce Research together with writers and literary agents.

Can I check the results?

Yes. Code, tests, the 2026 AI stories, all verdicts and the analysis are public on GitHub. Only the New Yorker originals are not included, for copyright reasons.

Try it yourself

In Hermes 3000 you can open the model arena in any book under ⋮ → AI settings, have several models continue the same sentence opening and decide for yourself what you like. Hermes 3000 is free up to 50 standard pages: Try it now.