AI Fiction Has a Shape, Not Just a Style

In Who Wrote It Is the Wrong Question I argued two things. First, that the most valuable part of a text (the idea, the angle, the construction) is not legible in the finished product. Second, that automated AI detection is the wrong instrument for every concern people actually have about machine-written text.

A new study forces me to qualify the first claim and, read carefully, strengthens the second. It is worth going through slowly, because it will be cited in the coming months as proof that "AI can be detected after all", and that is not quite what it shows.

What the study did

StoryScope (Jenna Russell and colleagues, University of Maryland and Google DeepMind; COLM 2026) asks whether AI-written fiction can be told apart from human fiction without looking at style at all: no word choice, no sentence rhythm, no em-dashes. Only the narrative decisions: who drives the plot, how time is arranged, when information is revealed, how the story ends.

The design is clever. The authors took 10,272 published short stories, had a model reverse-engineer a writing prompt from each, and gave that prompt to five current models: Claude Sonnet 4.6, GPT-5.4, Gemini 3 Flash, DeepSeek V3.2 and Kimi K2.5. That yields six versions of the same premise, about 61,600 stories averaging 4,750 words.

A pipeline of language models then turned each story into a structured outline, compared the six versions of each premise against one another, and distilled the differences into 304 closed questions: How explicitly does the story state its theme? Is the resolution driven by the protagonist's choice or by outside events? How often does the narrative jump in time? Another model answered those questions for every story, and a standard classifier was trained on the answers.

What machines do differently

With narrative features alone, the classifier separates human from AI stories at 93.2 % macro-F1. Assigning a story to one of the six sources still works at 68.4 %, against a chance level of 16.7 %. The differences behind those numbers are more interesting than the numbers themselves:

• AI explains its themes. In 77 % of AI stories the narrator spells out what the story means; in human stories, 52 %. A grieving character's arc ends with the lesson stated.
• AI tells tidy, single-track stories. 79 % of AI stories have no subplot at all (human: 57 %). Resolutions come from the protagonist's own choice or from inner acceptance; loose ends are rare.
• AI shows emotion through the body. 81 % of AI stories render feelings as tight throats and cold sweat; only 8 % simply name the emotion. Human authors write "she was afraid" 29 % of the time.
• Humans break the frame. They jump in time, open at the funeral and work backwards, let a revelation force a re-reading of earlier scenes, name the books and authors they are in conversation with, and occasionally address the reader directly.
• Humans leave moral judgement open. Protagonists are framed as morally ambivalent in 59 % of human stories and 38 % of AI stories.

The finding I consider most important is not about detection at all. In the feature space, the five models sit close together, and the human stories sit apart from them. The distance between the human and AI centres is 1.6 times the distance between the models. Models built by five different organisations, trained on different data, converge on one way of telling a story. Human stories are also more dispersed: for any given premise, the human version is the most unusual of the six in 57.8 % of cases.

Does it hold up?

The authors released their code, the generated stories and the extracted features. When I reran the classifiers on the released data, the full feature set matched the paper almost exactly (96.1 % against 96.0 %). The narrative-only model came out 1.2 points higher than reported (94.4 % against 93.2 %), most likely because the released code excludes only the 39 explicit style features, while the paper also removes eight further borderline features. The variation across data splits is small (94.3–95.0 %).

This is a partial reproduction, and it is worth being precise about its reach. It confirms that the published feature labels separate human from AI stories as reported. It does not repeat the expensive part: generating the prompts, inducing the features and having a language model annotate 61,000 stories. Whether those labels are themselves valid is a separate question that a rerun of the classifier cannot answer.

Not everything is in the repository either: the code for selecting the 30 "core" features, the baselines and the robustness tests is missing, and the paper gives two different sizes for its own test set (1,377 and 1,384 prompts). These are blemishes, not fatal flaws.

Where I would be careful

The human stories are not a neutral sample. They come from anthologies in the Books3 collection, which includes Poe, Dickens, Wilde, Conan Doyle and The Yellow Wallpaper. Addressing the reader directly, breaking the fourth wall, and naming other works are conventions of the nineteenth century as much as marks of a human hand. The study does not control for publication date, so part of what reads as "human versus machine" may be "old versus new".

The comparison is asymmetric. The human story is the original; the models receive a summary of it. Those summaries often already contain the theme. The example prompt in the paper ends with the protagonist being "forever altering her perspective". A model that receives the moral in the prompt is likely to state it in the story.

The line between narrative and style is soft. Which features count as "style" was decided by a language model. Conveying emotion through bodily sensation, density of sensory description, a fondness for smells: these are classified as narrative, but they sit very close to prose texture. The claim "even without style" is weaker than it sounds.

One model family appears at almost every stage. Gemini generated the prompts, wrote one of the five AI corpora, answered all 304 questions for every story, and performed the stylistic rewriting in the robustness test. Self-preference is not examined.

Rarity is not originality. That thousands of different human authors produce more varied stories than five models is close to inevitable. The finding is real, but it says more about the models than about the humans.

What this changes about my earlier argument

My claim that originality of thought is not legible in the finished text needs a qualification, though not a retraction. At the level of a corpus, patterns of narrative construction are visible. Over thousands of stories, you can see that models over-determine meaning, avoid structural risk and converge on the same narrative defaults. That is a finding about how stories are built, not about their surface, and it is more durable than counting em-dashes. What remains invisible is the process: who conceived a given choice, who selected it, and who developed it, whether a person, a model or both working together.

At the level of a single text, the gain is smaller than it looks. Within its controlled setting, the study does discriminate well on individual stories. What it does not provide is proof of authorship for a text arriving from outside that setting. Across all five data splits, the narrative-only classifier labelled 10.6 % of the human-written stories as AI-generated (1,083 of 10,216), while letting through only 1.5 % of the AI stories (739 of 51,080). That is under ideal conditions: long texts, a closed set of five known models, a classifier trained on exactly this kind of material, and human stories deliberately weighted up in training.

Suppose, purely for illustration, that these error rates carried over unchanged to a setting where one submission in ten was machine-written. Of 1,000 submissions, the classifier would correctly flag about 99 of the 100 AI texts and wrongly flag about 95 of the 900 human ones. Roughly every second flag would land on a human author. That is the base-rate problem Tsigaris and Teixeira da Silva describe for academic detectors, reappearing with a far more sophisticated instrument. Outside the laboratory the error rates would not stay the same, and there is little reason to expect them to improve.

Text-based baselines in the same study reach 99.9 %, which looks like the opposite conclusion. It is not. Those classifiers were trained and tested on the same five models, at the same lengths, on the same premises. That is the controlled laboratory in which detectors have always looked good. Weber-Wulff and colleagues showed what happens outside it.

So StoryScope moves the debate, but not in the direction its headlines will suggest. It does not supply a verdict on individual authors, and statistical patterns settle none of the questions my earlier piece was about: who conceived the work, who answers for it, and what has to be disclosed. What it supplies is something more useful: a vocabulary for the defaults machine-written fiction falls back on. Explicit morals, single-track plots, resolution by acceptance, emotions performed through the body, an unwillingness to leave the reader uncertain.

None of these is a flaw in itself. A fable states its moral; plenty of fine stories run in a straight line. They become failures of craft when they replace a deliberate choice with a formula, and that is a judgement an editor can make about the text in front of them, in its genre and on its own terms, without accusing anyone of anything. That is the fourth recommendation from my earlier piece (assess quality, not provenance), now with considerably better evidence behind it.

Sources

• Russell, J., Rajendhran, R., Iyyer, M. & Wieting, J. F. (2026): StoryScope: Investigating idiosyncrasies in AI fiction. Conference on Language Modeling (COLM 2026), pre-proceedings. Preprint: arXiv:2604.03136, version 6, which also lists Chau Minh Pham as an author. Code and data: github.com/jenna-russell/storyscope.
• Hamilton, S., Wilkens, M. & Piper, A. (2025): NarraBench: A comprehensive framework for narrative benchmarking. arXiv:2510.09869. The taxonomy of narrative dimensions StoryScope builds on.
• Tsigaris, P. & Teixeira da Silva, J. (2026): AI detecting AI in academic writing: why most AI detector findings are false. Next Research. doi:10.1016/j.nexres.2026.101396.
• Weber-Wulff, D. et al. (2023): Testing of detection tools for AI-generated text. International Journal for Educational Integrity 19, article 26. doi:10.1007/s40979-023-00146-z.

The reproduction figures are my own and cover only the classification step: the released XGBoost pipeline (commit b82d935) run on the published feature labels, not a repetition of prompt generation, feature induction or annotation. The 94.4 % and 96.1 % come from the repository's default run, which evaluates the first of five folds grouped by story title. For the fold range and the error rates I departed from the repository: stories grouped by prompt ID (61,296 stories with all six versions present), all five folds, test predictions pooled, unchanged hyperparameters, default decision threshold of 0.5. A different threshold trades one error for the other; it does not remove the base-rate problem.

Topics: AI · Authorship · Research Integrity

New posts by email: subscribe to the newsletter. One email per post, no tracking.