In short: After the writing models, we tested every other place where an AI works inside Hermes 3000: the consistency check, the chapter summaries, the AI widgets and the book chat. Every task ran with eleven models, using exactly the instructions Hermes uses for your book. Three things change: AI widgets are now built by GPT-6.1 Sol, the consistency check and chapter summaries are handled by Claude Haiku 5.5, and the book chat answers with GPT-6 Luna. You don't need to change any settings. As of 21 October 2026.
A few days ago we explained how we choose the writing model for Hermes 3000. But writing is only one of many places where Hermes asks a language model for help. When you finish a chapter, an AI summarises it. When you run the consistency check, an AI looks for contradictions. When you add a quiz or a timeline to your non-fiction book, an AI writes the little program behind it. And in the book chat, an AI answers your questions about the whole manuscript.
Until now these tasks ran on fast models from Google that were good enough when we introduced them. Models have improved since. So we measured again.
How we tested
For the results to say something about Hermes, every test ran with exactly the instructions Hermes uses in your book, and the answers were processed the way Hermes processes them. A summary that Hermes can't read counts in the test the way it would reach you.
We wrote the texts specifically for the test, and for a reason: with an invented chapter about a ferry cooperative on the Elbe, no model can know anything that isn't in the text. Every number, name and date in an answer can be checked, and whatever isn't in the text was made up.
For the book chat we needed a whole book. We took Theodor Fontane's novella Unterm Birnbaum (Under the Pear Tree), changed every name and place, and deliberately altered three details. A model that remembers the classic instead of reading the book gives itself away by naming the original name or the original date.
Two AI raters judged the results, Claude Opus 5.5 and GPT-6.1 Sol. Where there is a clear answer, such as planted errors, the answer key decides.
Consistency check: twice as many contradictions found
For the consistency check we wrote a short story in four chapters and planted seven contradictions. The heroine's eyes are green in chapter 1 and brown in chapter 3. Her younger sister is suddenly five years older. The glassworks stands on the market square, and later on the edge of town. And the heroine tells her neighbour about the sale of the furnace a day before she learns of it at the notary's.
We added four traps that look like contradictions but aren't, for example a lie that is exposed later in the book. Reporting them is a false alarm.
The previous model found three of the seven contradictions. Claude Haiku 5.5 finds six. The hardest case, a character knowing something she can't know yet, was caught only by the three very strongest models. The new model isn't flawless either: in the test it raised two alarms where none was needed. A finding from the consistency check is still a pointer for you to check, but it now misses far less.
Chapter summaries: your book's memory
When you finish a chapter, Hermes writes a summary of it. You rarely see it, but it matters: it is the memory that later AI help draws on when it needs to know what has already happened in your book.
We measured against ten facts that a good summary of our test chapter has to keep. The previous model kept six and on average added three to four things that aren't in the chapter in that form. Claude Haiku 5.5 keeps all ten and invents next to nothing. Because these summaries are written so often, much of what builds on them gets better too.
AI widgets: less made up, better to use
AI widgets are the small interactive elements Hermes builds into a chapter on request: a quiz, a timeline, flashcards, a chart, an overview of the characters. We gave eight typical requests to eleven models. Every widget ran in the same environment as in your editor, at desktop and phone width, and a test program clicked through all its buttons. The raters saw the screenshots, the code and the chapter.
GPT-6.1 Sol builds the best widgets: 8.8 out of 10 points against 7.7 for the previous model. The biggest difference is in the facts. The previous model put a little over four numbers, names or events per widget that weren't in the chapter; Sol just under two. The gap is widest for relationship maps and settings, where a widget has to bring order to a story.
Book chat: the whole book in view
In the book chat you ask questions about your manuscript, and the AI has the whole book in front of it. We asked fourteen questions the way authors ask them: about a detail, about which chapter a scene is in, about connections across several chapters, about things the book doesn't say at all, and for help phrasing a scene.
The good news first: every model we tested answered from the book, not from memory, all of them found the right chapters, and all of them said honestly when something isn't in the book. The differences show in the harder questions, especially writing help. GPT-6 Luna reached 88 and 93 per cent of the possible points, the previous model 86 to 89 per cent. Luna is also the model that writes your text in Hermes by default, so the book chat now answers with the same model that writes with you.
What stays the same
The writing models don't change: GPT-6 Luna writes by default, and for paid books you can choose GPT-6.1 Sol or Claude Opus 5.5. Proofreading and tightening text stay as they are; the previous model already did well there. If one of the new services is ever unavailable, Gemini automatically steps in again, so your work isn't interrupted.
How solid this is
Each task ran once, not a hundred times. Large gaps, such as three versus six contradictions found or six versus ten facts kept, are clear. A one-point difference on a ten-point scale would not be, and we based no decision on differences like that. Twice along the way we found and fixed mistakes in our own measurement before deciding anything. New models keep arriving almost every week. The tests are now ready, and we'll repeat them whenever a new model looks likely to be better.
Frequently asked questions
Do I need to change any settings to use the new models?
No. The consistency check, chapter summaries, AI widgets and the book chat use the new models automatically.
Can I choose the model for the book chat myself?
No, we set the book chat centrally. You can still choose the model Hermes writes your text with in your book's AI settings.
Will my existing chapter summaries be rewritten?
Not automatically. New summaries are written with the new model as soon as you finish a chapter or have a summary created again.
Will you test the models again?
Yes. The tests are built so we can repeat them with every new model, and that's exactly what we do whenever a switch might be worth it.