27.5KB language-agnostic WebGPU syntax highlighter27.5KB language-agnostic1 WebGPU syntax highlighterShu Ding at Vercel Labsimport { highlight } from 'gpu-lexer'

const spans = await highlight('source code')
// {
// type: 'plain' | 'comment' | 'string' | 'number' | 'keyword' | 'type' | 'function' | 'constant' | 'operator'
// start: number
// end: number
// }[]gpu-lexer splits source code into simple parts—words, whitespace, newlines, and symbols. Then a tiny WebGPU model combines local and whole-file context to label each part. It is designed for any language: instead of choosing a grammar, it guesses each part's type from the surrounding source, even when it never saw that language or syntax during training. Adjacent labels become the highlight spans returned to your code.This is an experiment, not a grammar-equivalent highlighter. On files kept out of training, 12.57% of the current model's token labels differ from Shiki. This measures agreement with Shiki—not objective correctness—and unseen languages or real-world code may differ more often.1 "Language-agnostic" means one shared tokenizer and classifier, not equal accuracy for every language. 87.43% is the share of held-out token labels that matched Shiki. Mixed-language code is supported too, including embedded <script> and <style> regions in HTML, Vue, and Svelte.Held-out label agreement with Shiki by languageagreementverified languages (90)95–100%actionscript, angular-html, angular-ts, cython, javascript, jsx, python, tsx90–<95%apex, astro, c, go, haxe, hlsl, html, kotlin, odin, plist, protobuf, rss, scala, scss, svelte, swift, typescript, xsl80–<90%bicep, c3, cmake, coffeescript, cpp, csharp, css, d, diff, docker, gdscript, glsl, hcl, java, make, nginx, objective-cpp, php, powershell, qml, rust, shell, shellscript, systemverilog, toml, vala, wgsl70–<80%ada, asm, bat, fortran-fixed-form, fortran-free-form, groovy, hack, less, lua, matlab, objective-c, perl, puppet, ruby, solidity, vue, xml60–<70%dart, emacs-lisp, gherkin, julia, markdown, nim, nix, prisma, razor, shaderlab, sql, webassembly50–<60%awk, graphql, liquid, mdx, r, viml<50%jinja, vbtraining epoch4,723,430 tokensmodel41,321 parameters · 31.0 KB packed int6Each language appears in one band; results with fewer held-out labels are less stable. Training tokens include context-only tokens and replay.Highlight 10× three.min.jswarmed browser time · lower is better0s8.00s16.0s24.0s32.0sgpu-lexer471.7msSugar High859.4msPrism.js1.20sHighlight.js1.32sStarry Night11.1sShiki30.4sOne browser run after one warm-up on September 8, 2026. The input was 10 concatenated copies of three.min.js (5.56M characters). MacBook Pro, Apple M4 Pro, 20-core GPU, 24GB, macOS 26.6.2, Chrome 152. Each engine ran in a dedicated worker; DOM rendering was excluded. gpu-lexer and Shiki returned token data, Starry Night returned a HAST tree, while Sugar High, Prism.js, and Highlight.js returned highlighted HTML. Sugar High 2.3.1, Prism.js 1.30.0, Highlight.js 11.12.0, Starry Night 3.11.0, and Shiki 4.4.3.Loaded library sizeruntime + selected language coverage · lower is better0KB390.6KB781.3KB1.14MB1.53MBSugar High (major 6 web languages)5.4KBPrism.js (major 6 web languages)8.6KBSugar High (all 29 languages)8.8KBHighlight.js (major 6 web languages)14.9KBgpu-lexer (one model)27.5KBPrism.js (all 297 languages)162.1KBStarry Night (major 6 web languages)185.3KBShiki (major 6 web languages)213.9KBHighlight.js (all 193 languages)240.4KBShiki (all 242 grammars)991.5KBStarry Night (all 710 grammars)1.46MBMinified and Brotli-compressed browser bundles measured on September 8, 2026. Major web includes javascript, typescript, css, html, json, and markdown. gpu-lexer uses the same bundle for every language. Starry Night totals include its Oniguruma WASM payload.Top-25 weighted correctnesspopularity-weighted agreement with Shiki · higher is better0%25.00%50.00%75.00%100%Shiki100%gpu-lexer90.20%Prism.js87.01%Starry Night85.07%Highlight.js84.65%Sugar High76.33%Shiki is the 100% normalization reference. Each library's token names are mapped to the same nine classes: plain, comment, string, number, keyword, type, function, constant, and operator. Scores compare non-whitespace source parts across 1,069 held-out files in the GitHub Innovation Graph top 25 for 2026-Q1, weighted by each language's pusher count. Unsupported languages score zero; corpus size does not affect the weights.[ live demos ]75 languages·react.development.js·lodash.js·three.min.js·App.jsx·main.jsx·root.tsx·Layout.vue·ModalDemo.vue·VPHero.vue·App.svelte·Counter.svelte·certs.ts·types.ts·bootstrap.css·tailwind-experimental.css·normalize.css·bootstrap._variables.scss·html5-boilerplate.html·react-fixture.html·test.xslt·sessions.py·pydantic.main.py·gin.go·counter.go·tokio.worker.rs·axum.main.rs·ripgrep.main.rs·worker_runtime.zig·Assertions.java·Application.kt·Session.swift·AFHTTPSessionManager.m·Model.php·redis.server.c·String.cs·active_record.base.rb·framework.dart·ERC20.sol·Sort.cu·hello-triangle.shaders.hlsl·godot.canvas.glsl·github.svg·typescript.package.json·checkout.action.yml·ascii.cpp·node.bookworm.Dockerfile·Makefile·cmake.CMakeLists.txt·vpc.main.tf·blog.index.astro·App.tsx·index.tsx·Sudoku.hs·react.README.md·nvm.install.sh·install-powershell.ps1·cornell.raytracer.wgsl·ant-design.button.less·envoy.bootstrap.proto·scipy.messagestream.pyx·kitchen-sink.graphql·uv.pyproject.toml·spark.pi.scala·gradle.accepted-api-changes.groovy·gnome-calculator.vala·qt.list-properties.qml·now-in-android.AndroidManifest.xml·simplepie.feed.rss.xml·react-native.Info.plist·apex-recipes.DynamicSOQLRecipes.cls·c3.std.hashmap.c3·phobos.json.d·basic_map_example.odin·godot.physics-player.gd·nim.json.nim·aspnetcore.ErrorPage.cshtml·flex.ArrayCollection.as·haxe.JsonParser.hx·bicep.comprehensive.bicep·coffeescript.coffee·opentitan.aes_core.sv·wasmtime.simd.wat·nginx.conf·calcom.schema.prisma·pygments.test_ada2022.adb·awesome-compose.app.component.html·awesome-compose.app.component.ts·linguist.example.R·VBAllInOne.vb·apacheinit.pp·stockcorr.jlreact.development.js0 byteshttps://unpkg.com/react@19.2.8/cjs/react.development.js—Experimental software. Highlighting is probabilistic, may differ from Shiki, and is not a parser or a substitute for compiler, linter, or security analysis.Shu Ding at Vercel Labs.

The document describes gpu-lexer, a system designed as a language-agnostic WebGPU syntax highlighter. This system operates by splitting source code into fundamental components such as words, whitespace, newlines, and symbols. It employs a compact WebGPU model that combines local and whole-file context to label each segment of the code. The core innovation lies in its ability to guess the type of each part based on the surrounding source context, even for languages it has not explicitly seen during training, establishing its functionality as an experimental approach rather than a strict grammar-equivalent highlighter. While it is language-agnostic, the text notes that this means sharing a single tokenizer and classifier, which does not guarantee equal accuracy across all languages. The system achieves an agreement with the established benchmark Shiki; for instance, in files kept out of the training set, the model's token labels sometimes differ from Shiki’s, suggesting that agreement is measured based on correlation rather than objective correctness.

The methodology involves mapping generated token names to nine standardized classes: plain, comment, string, number, keyword, type, function, constant, and operator, which serve as a unified classification framework. The system performs robust analysis across numerous languages, including JavaScript, Python, C++, TypeScript, CSS, Rust, and many others, demonstrating broad applicability. The measure of agreement with Shiki is quantified by comparing token labels across a large set of held-out files, weighted by the number of contributors in those languages. This comparative analysis reveals varying levels of consistency; for instance, agreement scores range from over ninety percent for verified languages like JavaScript and TypeScript to lower percentages for more specialized or less common grammars.

Performance evaluations benchmark gpu-lexer against other highlighting tools such as Sugar High, Prism.js, Highlight.js, Starry Night, and Shiki, focusing on runtime speed and the size of the loaded library. Results indicate that while some systems offer high visual fidelity, models like gpu-lexer focus on efficient tokenization. For large inputs, comparison with a substantial JavaScript file bundle demonstrates performance differences among the engines. The analysis further examines the efficiency of these tools by measuring the required loaded library size, where terms involving specialized or comprehensive coverage, such as Highlight.js for all nineteen languages or Starry Night covering all seven hundred and ten grammars, result in significantly larger footprint compared to gpu-lexer’s comparatively smaller package.

The underlying model for gpu-lexer is characterized by a relatively small parameter count and a packed total size, reflecting an emphasis on efficiency. The training process utilized context-only tokens and replay to build this model. Although the highlighting process is probabilistic and should not be considered a substitute for formal compiler or security analysis, the results provided offer a quantitative measure of cross-language semantic agreement in code annotation tasks. The system’s performance across a wide spectrum of programming languages and file types—ranging from web development assets like HTML and CSS to systems programming languages like Rust and C—underscores its potential as a generalized tool for interpreting code structure and context regardless of the specific syntax encountered.