I had Gemini train its own replacement for $9 — Peter Vijeh← petervijeh.comI had Gemini train its own replacement for $9This article was written with the assistance of AI. If that bothers you, stop reading here. The numbers are real: every score comes from the ten training runs described below, and the full run log is in the linked knife.day write-up.The knife.day write-up, with the full run logI like to cook, and somewhere along the way that turned into an obsession with high-end chef's knives. So I scrape the Reddit threads where people argue about them and pull out every brand, model and steel they mention, to see what is getting bought and argued about.Picking product names out of text is a job called named-entity recognition, and small models have done it for a decade. I was doing it with Gemini 3.1 Pro, one paid API call per comment. Overkill, but it worked: from "picked up a Mazaki in white #2, way better than my old Fibrox" it returned Mazaki as a brand, Fibrox as a model and white #2 as a steel, and nothing else. But the scraper pulls every new comment, so the bill grew with how much people posted, and the only way to cap it was to skip comments.The obvious replacement, an open NER model called GLiNER run zero-shot, cut the cost to nothing and the accuracy to about 0.65 F1 against Gemini's answers. That gap is what the rest of this is about: could Gemini label 4,290 comments once and teach GLiNER to close it?What: Fine-tuned GLiNER large v2.5 (459M) to tag brands, models and materials in Reddit comments, on labels Gemini 3.1 Pro wrote once.Why: Zero-shot GLiNER scored about 0.65 F1 (est.). Gemini scored well and billed every comment for as long as the scraper ran.Approach: Ask Gemini for strings, not offsets. Compute offsets in code. Add comments with no products in them as negatives. Lock a 225-comment validation set before the second run.Problems: Five of ten runs produced no usable model. Three failed on configuration. Two failed on a tensor called words_mask that I filled the way you fill an attention mask.Result: 0.83 F1 against Gemini's labels after 24 minutes on a Tesla T4. $9 of labels, about $2.50 of GPU time, and days of debugging.What I set out to doThe plan had three steps. Have Gemini label a few thousand Reddit comments once, marking every brand, model and steel. Train GLiNER on those labels. Then run GLiNER on my own machine for every comment after that, and stop calling Gemini.Gemini labeled 4,290 comments for $9, or $0.0021 a comment. That means the trained model pays for itself at roughly comment 4,291, as long as later comments are about the same length and it runs on a GPU I already own. The test of success was simple: on 225 comments the model had never seen, how often does it tag the same words Gemini tagged? One catch to keep in mind for every score in this article. Nobody checked Gemini's labels by hand, so the model is graded against Gemini, not against the truth. Where Gemini was wrong, the model gets marked right for copying the mistake and wrong for fixing it.The approachGemini labeled the comments through OpenRouter at temperature 0 in 25 minutes. The prompt decision that mattered most was to never ask the model for character offsets. It counts characters badly and returns spans off by two or three positions. The prompt asks for the exact substring and a label, and TypeScript finds the offsets. If the string is not in the comment, the entity is dropped and logged.// The model returns strings. Code computes the offsets.
{ "entities": [
{ "text": "Benchmade", "label": "knife brand" },
{ "text": "940", "label": "knife model" },
{ "text": "S30V", "label": "knife steel" }
] }Product names are full of punctuation a generic tokenizer splits, so a regex keeps VG-10, CPM-154 and 1.4116 whole and emits every other non-space character as its own token. Spans that still miss a token boundary are dropped rather than guessed. About 30% of the training set is comments that contain a known false-positive trigger (gyuto, carbon, handle, patina) and no product, labeled as empty. Before the second run I set aside 225 comments as a validation set and never touched them again. Training ran on a Tesla T4 on Modal with the HF Trainer.per_device_train_batch_size = 2
gradient_accumulation_steps = 8
learning_rate = 1e-5
threshold = 0.45
What the model trained on
What the model trained on
4,290 forum comments annotated by an LLM for $9

1,575 positive (69.6%)
675 negative
2,250 training examples

2,029 train
225 val
split (validation set locked from run 10 on)
3,907 entity spans
brand

1,720
product model

1,345
material spec

842
Roughly 30% of examples deliberately contain no entities at all.
What went wrongFor five runs the model learned nothing. The first three failed on configuration, and anyone using the HF Trainer with GLiNER will hit them in an afternoon.RunWhat went wrongFix1GLiNER's default max_steps=10000 overrode num_train_epochs=3; trained 39 epochsSet max_steps explicitly2load_best_model_at_end without eval_strategy throwsSet eval_strategy="steps"3Trainer saved state-dict keys without the "model." prefix GLiNER's loader expectsPut the prefix back on save4ner_labels missing on negative examplesSet the label list on every example4–5words_mask built as binary; loss flat at 70–130Emit incremental word indicesRuns 4 and 5 were the expensive ones. GLiNER's tokenize_inputs crashed on broken Reddit emoji, so I had patched it, and the patch has to fill a tensor called words_mask. It sits next to attention_mask, has the same shape, and every attention mask I have ever built is ones for real tokens and zeros for padding. I built it that way. Nothing about the name or the shape says it is anything other than an attention mask.Training ran to completion. Loss started around 130, drifted to about 70 and stayed there. No crash, no warning, no NaN, gradients of ordinary size, checkpoints saved on schedule, eval F1 near zero. I blamed the label list first, because run 4 also had negatives with no labels set. Fixing that and rerunning gave the same flat loss. The only thing wrong with run 5 was words_mask, a tensor I had never looked at.
Ten training runs
Ten training runs
The first five produced no usable model — all plumbing, no modeling

0.0

0.2

0.4

0.6

0.8

1.0

failed

scored

best F1

in production

✕
R1
step cap

✕
R2
config

✕
R3
checkpoint

✕
R4
word mask

✕
R5
word mask

not
scored
R6
first success

0.800
R7
tuned 209M

0.879
R8
459M

0.799
R9
too many negs

0.832
R10
locked val

Runs 1–5 failed outright. The dashed line tracks overall F1 after the word-mask fix.
How I fixed itI read GLiNER's training loop instead of its docstrings. words_mask is not a mask. It is a word index: 0 for special, prompt and padding tokens, then 1, 2, 3 for the first sub-token of each real word. The span-scoring head uses it to pool sub-tokens back into words. Filled with ones it says the whole comment is a single word, so the model is asked to find brand and material spans inside one enormous token. It cannot, so the loss stays flat, and a flat loss does not show which input is wrong.# what I wrote # what GLiNER expects
words_mask = [1,1,1,1,1] words_mask = [0,1,2,2,3]
# [CLS] Mazaki wh ##ite #2With the index fixed, run 6 learned on the first try. The rest was tuning against the locked set. The 209M medium model reached 0.800; the 459M large model, which fits a T4 only with gradient accumulation, reached 0.83. Ten times more adversarial negatives (510 instead of 51) dropped F1 to 0.799, so run 10 went back to 51. One threshold per class instead of a global cutoff took material recall from 0.787 to 0.911, because steel names like MagnaCut, S35VN and HAP40 score lower confidence than brands and a single cutoff dropped them. Every large run bottoms out at epoch 2 and overfits after; with 2,000 examples that is a dataset-size problem, and early stopping is the fix.
F1 by entity class
F1 by entity class
Zero-shot baseline vs. fine-tuned checkpoints, same held-out data

Zero-shot (est.)

Fine-tuned 209M

Fine-tuned 459M

0.0

0.2

0.4

0.6

0.8

1.0

~0.65

0.800

0.879
Overall

n/a

0.858

0.904
Brand

n/a

0.775

0.877
Product

n/a

0.712

0.829
Spec

Bigger encoder helps most where the vocabulary is purely domain-specific.
What I learnedIt worked. The model runs locally, matches Gemini's labels at 0.83 F1 on comments it never saw, and knows that "carbon steel" is a category rather than a steel and that PM2 sometimes means the Spyderco Paramilitary 2 and sometimes is just letters. One earlier run scored 0.879 on a random split. I do not count it as the result: on random splits, two drops in F1 that I had blamed on my changes turned out to come from which comments landed in the validation set.On paper the project cost less than lunch: $9 of labels, $2.50 of GPU. What it cost me was the days spent on a tensor that passed every check the code had and was still wrong. I think the lost days are the normal case for small fine-tuning jobs. The model and the data are rarely the problem; the code between them fails, and a flat loss from a wrong input tensor looks the same as a flat loss from hard data. If I had to choose between a better label set and an assertion on every tensor I hand-build, I would take the assertion.This model powers New Knife Day, which tracks what knife people on Reddit are buying and arguing about. The knife-side write-up there has the full run log. Both are linked below.At a glanceProblemFind the brand, model and material names in Reddit comments, and skip the generic words around them, without paying an LLM per commentApproachHave Gemini label the comments once, then fine-tune GLiNER large v2.5 (a DeBERTa-v3-large encoder) on those labels and run it locallyResult0.83 F1 on a 225-comment validation set fixed before training; material recall 0.911 with a per-class thresholdCost$9 in Gemini labels plus about $2.50 of T4 time across ten runsStackTypeScript and MongoDB for the scraper and labels, Python and PyTorch for training on Modal, FastAPI to serve the modelNew Knife Day, the site this model runs on →The knife.day write-up, with the full run log →Personal site. Views my own; not affiliated with or endorsed by my employer.

The work details an approach to automate the extraction of brand, model, and material names from Reddit comments by leveraging a large language model for initial labeling and fine-tuning a smaller Named Entity Recognition model. The initial motivation stemmed from the high cost associated with using an LLM like Gemini for per-comment extraction, leading to a strategy to delegate this task and reduce operational costs.

The core methodology involved having Gemini label a large set of comments, effectively costing nine dollars for the labels, which served as the ground truth for training. This labeled dataset was then used to fine-tune an open NER model named GLiNER, which was intended to perform entity tagging on new, unseen comments locally. A critical aspect of the process involved careful prompt design; the author determined that asking the model for exact substrings rather than character offsets was necessary because the latter was prone to errors due to tokenization issues. Technical challenges arose during the training process, particularly concerning the internal structure of GLiNER. The author discovered that a tensor called words_mask, which was often misinterpreted as a standard attention mask, actually functioned as a word index, which necessitated a modification to the training loop to ensure the model correctly utilized positional information.

The training phase was fraught with difficulties, as five out of ten experimental runs yielded no usable models due to configuration errors or failures in tensor handling. Debugging revealed that the failure modes were often rooted in the implementation details of the training framework rather than the quality of the labels or the data itself. The author learned that performance was sensitive to data partitioning; locking a validation set before the first run was essential, and simply observing the results on random splits could be misleading. Furthermore, the author found that adjustments to the loss calculation, such as applying a threshold per entity class instead of a global cutoff, significantly improved material recall, demonstrated by boosting the confidence in identifying complex material names.

Ultimately, the fine-tuned model demonstrated a strong performance, achieving an F1 score of 0.83 against Gemini's labels on a validation set it had never seen. The performance metrics broke down by entity class, with the model showing a high recall for material specifications. The author concluded that while the process involved significant time spent debugging technical plumbing—specifically managing the input tensors and training configurations—the focus should remain on ensuring the integrity of the code connecting the data and the model. The project successfully developed a system capable of identifying complex product information from unstructured text, validated by its performance on a curated set, and resulting in a system that powers a tracking mechanism for consumer discussions on platforms like Reddit.