Hangar Harness / Model Tests

Hangar Harness / Model Tests
I've been testing a simple prompt with different model and harness combinations to work out which one produces best results. I do this in /goal mode.
Prompt: Build a single-page Three.js sci-fi hangar with hovering drones, animated warning lights, emissive runway strips, and subtle volumetric-style fog planes. Include drone formation toggle and cinematic camera path. Output one self-contained HTML file with inline JavaScript.

Model ↕
Harness ↕
File ↕
Duration ↕
TTFT ↕
Input tokens ↕
Output tokens ↕
Reasoning tokens ↕
Total tokens ↕
Cached input % ↕
Tool calls ↕
Tool errors ↕
Opened in browser ↕
Checked screenshots ↕

GLM 5.3 Flash Max
Codex
Open
9m 0.232s
9.344s
457,685
17,458
7,694
475,143
85.26%
14
1
Blocked
No

Luna 5.6 MaxCodexOpen9m 13.098s6.199s1,146,75525,5126,8791,172,26792.76%322YesYes
SOL 5.6 MaxCodexOpen10m 48.765s8.460s1,069,16328,2787,5151,097,44194.60%215NoNo
Astra 6.0 MaxCodexOpen37m 29.705s3.589s1,292,36643,12915,2711,335,49594.93%205YesNo
GLM 5.3 Flash MaxOMPOpen30m 14.979s6.596s1,678,50963,405—1,741,91478.54%570YesYes
Qwen 3.8 27B x-highOMPOpen41m 25.836s15.275s3,407,45171,93549,1313,479,38686.92%890YesYes
GLM 5.3 Flash MaxOpenCodeOpen20m 28.948s6.284s4,305,44750,46833,4144,355,91596.89%670YesYes
Qwen 3.8 27B x-highOpenCodeOpen8m 48.470s10.182s665,49041,81728,788707,30795.64%130YesYes
Qwen 3.8 27B x-highDSH / PTCOpen24m 32.929s7.724s1,012,49989,894—1,102,39391.69%235YesYes
Qwen 3.8 27B x-highDSHOpen18m 15.239s6.755s2,654,45778,232—2,732,68995.48%422NoNo

All GLM runs are labelled GLM 5.3 Flash Max; input tokens include cached input. Output tokens are the generated total, including reasoning; when a harness reports reasoning separately, the reasoning column shows that subset. The DSH adapter does not report a separate reasoning count. DSH durations sum active turn time across both turns, excluding the pause between turns. Tool errors are recorded failed tool events. A dash means unavailable or not reported.

The provided text details an experimental process conducted to evaluate different combinations of language models and execution harnesses for a specific creative coding prompt. The overall objective of this testing was to ascertain which configuration yields the optimal results when instructing the model to generate a single-page Three.js sci-fi hangar featuring elements such as hovering drones, animated warning lights, emissive runway strips, volumetric fog planes, drone formation toggles, and a cinematic camera path, all encapsulated in a self-contained HTML file with inline JavaScript.

The investigation systematically compares various models against different harnesses across numerous performance metrics. These metrics include overall duration, time to first token, input tokens, output tokens, reasoning tokens, total tokens, cached input percentage, tool calls, and tool errors. The comparison spans several models, including GLM 5.3 Flash Max, Luna 5.6 MaxCodex, SOL 5.6 MaxCodex, Astra 6.0 MaxCodex, GLM 5.3 Flash Max with OMPOpen, Qwen 3.8 27B x-high with OMPOpen, and Qwen 3.8 27B x-high with DSH / PTCOpen, each paired with corresponding harnesses like CodexOpen or OpenCodeOpen.

The data presentation delineates these complex interactions using specific token counts for input, output, reasoning, and total tokens, allowing for an assessment of the computational load and generation efficiency associated with each model-harness pairing. Furthermore, the log attributes specific contextual details to the results, such as noting when a harness reports reasoning separately, indicating how the measurement of reasoning tokens is handled across different setups. The duration recorded for some tests also accounts for active turn time, excluding pauses between turns in the case of the DSH adapter.

The results reflect varying performance profiles when executing the complex generation task. Specific annotations provided clarify that all runs involving GLM models are labelled as GLM 5.3 Flash Max, and input tokens include cached input, while output tokens represent the generated total including reasoning when applicable. The use of dashes in the data signifies unavailability or unreported metrics for certain configurations. In summary, this examination functions as a comparative study on optimizing model selection and harness implementation to maximize efficiency and quality in complex, multimodal generative tasks.