Training a model to identify AI-generated web content from structure alone
Recorded: Sept. 22, 2026, 6 p.m.
| Original | Summarized |
[2609.15369] SlopShape: Identifying AI-Generated Commercial Web Content
Skip to main content Search Log in Search arXiv Press Enter to search · Advanced search Computer Science > Computation and Language arXiv:2609.15369 (cs) [Submitted on 14 Sep 2026 (v1), last revised 17 Sep 2026 (this version, v2)] Abstract:Word-level detectors identify unedited AI-generated text almost perfectly, but the literature documents their brittleness under rewording, and a word-level score neither characterizes a text nor identifies which AI model wrote it. We ask whether AI-generated text can be identified one level deeper, from structural signatures: how information is presented, in what order, with what evidence, and in what voice. We replicate StoryScope (Russell et al., 2026), which showed such patterns for AI-generated fiction, on commercial content: 2,250 pre-ChatGPT human blog posts from 268 company domains against 11,250 AI mirrors from five frontier models. A 214-feature instrument, applied by an LLM and validated in a human gold-annotation session (human-human kappa 0.928, human-model 0.946), detects AI posts from its 187 structural features alone at 98.0 macro-F1 on held-out companies, unchanged (98.1) when every AI post is reworded by its own model. The signal characterizes and attributes: AI posts share a tidy, self-announcing shape, 79.3% are attributed to the correct source against a 16.7% chance rate, and human posts occupy rare structural configurations. All effects replicate StoryScope's, consistent in direction and larger in magnitude. We release pipeline, instrument, prompts, code, and aggregate artifacts. Subjects: Computation and Language (cs.CL) Cite as: Focus to learn more arXiv-issued DOI via DataCite Submission history From: Jochen Madler [view email] [v1]
Full-text links: View a PDF of the paper titled SlopShape: Identifying AI-Generated Commercial Web Content, by Jochen Madler (Sitefire)View PDFHTML (experimental)TeX Source < prev | new Change to browse by: References & Citations NASA ADSGoogle Scholar export BibTeX citation BibTeX formatted citation loading... Data provided by: Bookmark
Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv (What is alphaXiv?) Links to Code Toggle CatalyzeX Code Finder for Papers (What is CatalyzeX?) DagsHub Toggle DagsHub (What is DagsHub?) GotitPub Toggle Gotit.pub (What is GotitPub?) Huggingface Toggle Hugging Face (What is Huggingface?) ScienceCast Toggle ScienceCast (What is ScienceCast?) Demos Demos Replicate Toggle Replicate (What is Replicate?) Spaces Toggle Hugging Face Spaces (What is Spaces?) Spaces Toggle TXYZ.AI (What is TXYZ.AI?) Related Papers Recommenders and Search Tools Link to Influence Flower Influence Flower (What are Influence Flowers?) Core recommender toggle CORE Recommender (What is CORE?) Author About arXivLabs arXivLabs: experimental projects with community collaborators Which authors of this paper are endorsers? | We gratefully acknowledge support from About Major funding support from |
The research presented by Jochen Madler addresses the limitations of word-level detectors for identifying artificially generated text, noting that while such detectors can identify unedited AI-generated text with high accuracy, their performance is brittle when texts are reworded, and a simple word-level score fails to characterize the text or attribute authorship to a specific AI model. The paper investigates whether AI-generated content can be identified by examining deeper structural signatures, focusing on how information is presented, the sequence of information, the type of evidence provided, and the overall voice. To test this hypothesis, the authors replicated the pattern recognition strategy demonstrated by StoryScope on commercial content. They conducted an experiment using 2,250 human blog posts from 268 company domains juxtaposed against 11,250 AI-generated mirrors produced by five different frontier models. The core of the work involves developing a 214-feature instrument designed to detect these structural patterns, which is applied by a large language model and subsequently validated through a human gold-annotation session. The validation demonstrated strong agreement, achieving human-human kappa scores of 0.928 and human-model scores of 0.946. The instrument successfully detects AI posts based solely on 187 structural features, achieving a macro-F1 score of 98.0 accuracy on held-out company data. Furthermore, the detection mechanism proved robust, retaining a score of 98.1 even when the AI posts were subsequently reworded by their own generative models. This indicates that the structural signals present in the text are intrinsic to the content, not dependent on specific phrasing. The analysis revealed that AI-generated posts exhibit a distinct, tidy, and self-announcing structural shape, contrasting with human posts, which occupy rarer structural configurations. This pattern successfully characterizes and attributes the source of the text: 79.3 percent of the detected content were correctly attributed to the established source against a background chance rate of 16.7 percent. The effects observed in this study were consistent with those found in StoryScope, exhibiting the same directionality and larger magnitude, reinforcing the notion that these structural signatures are a reliable method for discerning AI authorship in commercial web content. The authors release the pipeline, instruments, prompts, code, and aggregate artifacts to facilitate further research in this domain. |