LLM Attention Visualization

ishamf.dev

Home
Tools
About

Sign in
Subscribe

LLM Attention Visualization
A visualization of the attention mechanism in LLMs.

Isham Faizal

Sep 7, 2026
• 3 min read

One interesting thing about transformer-based large language models are that, during the generation phase, it is able to draw information from any of its previous tokens. But it needs to be selective; if every token affects the generation equally, it won't be very effective. This process needs a mechanism to decide how much a token affects the next token.Turns out, we can visualize this mechanism!You can tap or hover over any of the generated tokens to see the past tokens that affected* the generation.

Loading... (JavaScript required)

* "Affected" might not be fully accurate, as this visualization is highly simplified. It's calculating the attention weight, scaled by the magnitude of the value vector, aggregated across all attention heads, and summed across all layers. This is then used to control the opacity of the previous tokens. The largest values always have an opacity of 1 and the rest are interpolated.
A lot of information had to be thrown away to limit the visualization to just one numeric value per past token. Because of that, when I started implementing this, I actually thought it might not be comprehensible. But it actually can produce some interesting patterns!For example, in the default "Office Move Summary" prompt, you can hover over the text that are copied verbatim like the address and dates. You can then see the original data stand out quite a bit, because the generated token takes up a lot of the information from the source data.This addresses one thing that I've previously found unintuitive about LLMs. If they work by predicting the next tokens probabilistically, why are they somehow so good at copy-pasting stuff? Won't they eventually make a mistake just by random chance?But with this mechanism, you can see that it doesn't predict the entire sequence from some limited internal states. Since it has access to all past tokens, it can just decide which past tokens to draw from when copying, and so the probability of errors can be very low. In the "Debugging an Average Function" example, you can see that this quite small model (600 million parameters) can easily reproduce an entire JS function except for the intended modification. (Although it's not actually capable of finding the issue by itself, so it needed some hints.)Another interesting part is when you hover over the "remain" in "Existing access cards and phone numbers remain" in the "Office Move Summary" prompt. You can see that it draws from "work" in "Existing employee access cards will work" and "stay the same" in "company phone numbers will stay the same". So it's kind of combining the information from the words in both phrases, which I find quite cool.ImplementationThe visualization itself is a pretty basic React app using Transformers.js to generate the text. But, since we need to pull more data out of the model to visualize it, it can't use the regular generation loop. I had to vibe-code the generation loop in the app so we can actually keep track of the values to visualize.Despite using a smaller model for this, it's still hundreds of megabytes, and waiting for it to download before showing anything just won't work. So I pre-generated a bunch of prompts that can be loaded and viewed instantly.Another tricky thing is that some of the things in the visualization are not actually meant to be read, so they're not defined as outputs. I suppose if you're implementing this using Python ML libraries, it would still be easy to access them. But Transformers.js uses .onnx files that contains the entire computation graph. The model loading and computation logic is implemented in wasm, so there's no easy way to access anything other than the predefined outputs, as far as I can tell.In the end, I used a small script to modify the onnx file just enough to expose those internal values. But that means I can't just use the regular .onnx model. Since I want to have a browser-based generation feature, I have to upload a separate instrumented model to my own Hugging Face repo and point the app there.You can find the code in the GitHub repo.

Share this post

Facebook

X

Reddit

LinkedIn

Daylight and Timezone Globe

An interactive map to convert between time zones, with a daylight visualization, rendered on a globe.

May 4, 2025
1 min read

Daylight and Timezone Map

An interactive map to convert between time zones, with a daylight visualization.

Mar 30, 2025
1 min read

IP Block Subtraction Calculator

A tool to subtract a set of IP blocks from another, primarily used to help write WireGuard config.

Dec 30, 2024
1 min read

ishamf.dev © 2026

RSS
Sign up
Email Me

Powered by Ghost

The visualization of attention in large language models addresses the mechanism by which these models draw information from previous tokens during the generation phase, aiming to understand the selective influence of past context on subsequent token prediction. The core hypothesis explored is that for the generation process to be effective, the mechanism must decide how much influence each preceding token exerts; visualizing this mechanism allows users to examine which prior tokens affected specific generated tokens. This visualization achieves this by calculating attention weight, scaling it by the magnitude of the value vector aggregated across all attention heads and layers, and then using these aggregated values to control the opacity of the antecedent tokens, where the largest values are rendered with full opacity while others are interpolated.

This mechanism reveals crucial insights into how LLMs operate. For instance, in prompts like the "Office Move Summary," hovering over verbatim content, such as addresses or dates, demonstrates that the generated token utilized a significant amount of information directly from the source data. This observation challenges the notion that probabilistic prediction alone accounts for this level of fidelity, suggesting that the selective retrieval mechanism minimizes the probability of random errors by allowing the model to deliberately draw specific context when copying. Furthermore, the visualization exposes complex contextual blending; for example, hovering over a word like "remain" can reveal that it drew contextual information from distinct phrases within the input prompt, such as combining concepts from clauses related to access cards and phone numbers.

The implementation of this visualization presented several technical challenges. While the interface itself was constructed using React and Transformers.js, the standard generation loop was insufficient for extracting the necessary internal data. To address this, the author implemented a workaround by pre-generating prompts, allowing for instant loading and viewing of results. A significant hurdle involved accessing the model's internal computations, as the use of onnx files and wasm implementation in Transformers.js made accessing internal values difficult. Consequently, the author was required to modify the onnx file to expose these specific internal values, which necessitated uploading a separately instrumented model to a repository to ensure that the visualization could function correctly within a browser environment. This process underscores the complexity of extracting deep operational details from opaque transformer architectures.