Building a Specialized Model.

Our NPC agents had been advised by a small off-the-shelf language model for months — and roughly 40% of its suggestions were moves the game engine had to reject. So we trained our own on the economy’s own decisions, put it into a live race against the incumbent, and lost. This is the complete field guide for anyone who says “let’s build a specialized model”: the corpus, the training, the serving gauntlet and the operating loop — every failure included, and a negative result published in full.

“Below the baseline’s validity, a model is never promoted — no matter how creative it is.”
— the promotion rule, written down before the race started

It took eight days from the first training run to a verdict, and the order of those days is the order of this report: get the data, train the model, get it to actually answer in production, then let it compete. Three of those four stages went wrong at least once, which is why this is a field guide and not an announcement.

Part I — The Data

Before any of the model work: the part everyone underestimates. A training corpus does not exist because decisions happened — it exists because somebody stored them, in a shape that is still useful a month later. Ours nearly did not survive.

“Incomplete data is worse than less data. Only extract where the target format is actually filled.”
— the project rule that cut our recovered corpus at the exact day persona labels began to exist

Why train at all?

Every NPC in the Cosmergon economy asks an advisor model one question per decision cycle: given my situation, what should I do? The advisor was a stock 3B-parameter model. It was fine at sounding confident and terrible at being legal: about 40% of its proposals were actions the engine had to reject — hallucinated action names, moves the agent couldn’t afford, targets that didn’t exist. Over months we built an increasingly elaborate scaffolding of deterministic overrides around it, until the deterministic layer was making most of the real choices. That works. It is also a slow admission that the model contributes noise.

The alternative: a model that has actually seen this economy. Not a bigger generalist — a small specialist, trained on what real agents really did, cheap enough to retrain weekly. The plan is a closed evolution loop: train, deploy to a cohort, compare against the incumbent, learn from the week, train again. Part IV covers that loop, with the first week of real numbers. First we needed a corpus.

Where decision data actually lives (and dies)

Here is the mistake we made so you don’t have to: our decision log was a hot operational table with a retention trim. Old rows were deleted to keep the table fast. For gameplay that’s correct; for training data it’s a silent archive fire. By the time we decided to train, the live table held under 6,000 usable decisions — a tenth of what the agents had actually produced.

Two things saved the project. First, a forward-fix we had shipped earlier: before the trim deletes anything, every decision is appended to an immutable archive file. Cheap, boring, invaluable. Second, backups. We restored historical database dumps into an isolated scratch container (network disabled — a restored dump should never talk to anything), extracted decisions from each snapshot, and deduplicated by decision ID with newest-wins. The corpus went from 5,850 to 56,554 decisions in an afternoon.

One editorial rule mattered more than any script: we cut the recovered history at the day our schema started recording the agent’s persona alongside each decision. Everything older lacked the label we train on. Incomplete rows would have been padding, not signal. If your target format isn’t filled, the row doesn’t exist.

What a pair looks like — and what we refused to teach

Each training pair is deliberately narrow: the agent’s persona, a compact numeric state (energy, fields, cubes, tier, rank), a handful of situational flags (threatened fields, active shield, tournament membership, a trusted counterpart nearby), and the action that was actually taken — one of eleven action names — plus a one-sentence rationale. That’s the entire completion: a two-field JSON object.

What’s not in the completion: parameters. Real historical decisions carry target IDs — which field to reinforce, which listing to buy. Those are UUIDs. A language model trained to emit UUIDs learns exactly one thing: how to hallucinate UUIDs with perfect syntax. Instead, the model picks the action, and at serving time a deterministic candidate builder supplies the legal parameters for whatever it picked. The model chooses; the engine grounds. We think of it as a guardrail contract: full freedom over the action vocabulary, zero authority over object identity.

Three eras, one label

A subtlety that would have silently poisoned everything: the corpus spans three different decision policies. For most of its history the stock LLM chose freely. Then a deterministic utility selector took over most choices. Then a two-regime selector variant shipped. An agent’s “typical” behavior differs across those eras — and a model trained on the blend without knowing it would learn a chimera of three policies.

So every pair carries a policy_era label, assigned by timestamp against the exact deployment moments of each policy change. The training mix draws from the eras with explicit weights (half from the selector era, less from the older free-LLM era, a small share from the newest regime) — a choice we can revisit per training generation, because the label survives in every pair.

The mix recipe, in full

Raw corpora imitate their own imbalances. Ours had one action — placing cells — at 63% of all decisions, because growing is what agents do most. Trained naively, the model becomes a place-cells machine with a vocabulary. The v1 recipe:

• Cap the majority class: place-cells capped at 6,000 pairs, dropping it from 63% to 41% of the mix.
• Era weights: 0.5 / 0.3 / 0.2 across selector / free-LLM / two-regime eras.
• Synthetic fill for thin regions: 328 additional pairs for state regions where real data was sparse — generated not by imagination but by asking the deterministic selector what it would do there, and labeling them as such. Extrapolation with a pedigree.
• Frozen holdout: 2,000 pairs set aside before any training, never trained on, never regenerated. Every future model generation is measured against the same frozen set — it’s our defense against the slow self-degradation that plagues models trained on their own descendants’ outputs.

Result: 14,677 pairs (13,893 train / 731 validation after split), plus the untouchable 2,000.

One prompt to rule both worlds

The final data decision sounds bureaucratic and is anything but: the prompt template used to render training pairs is the same code path that renders the live prompt in production. One renderer, imported by both the training exporter and the serving pipeline. Every team that trains on a hand-copied prompt format eventually serves a subtly different one — and then spends a week discovering that their model is fine and their whitespace isn’t. We made train/serve skew structurally impossible instead of procedurally unlikely.

In Part II, that exact principle gets stress-tested in a way we didn’t expect: the base model we chose turned out to inject invisible tokens into every training example — and the loss curve’s way of telling us was to report a perfect 0.000 from the very first step.

Part II — The Training

Five training runs on a desk-side Mac mini. Three of them were failures — and each failure was worth more than the successes, because each one is a trap that every “let’s fine-tune a small model” project will walk into sooner or later. Here is every run, every parameter, and every wrong turn, in order.

Iter 10: Train loss 0.000 · Tokens/sec 0.000 · Trained Tokens 0
— the most alarming healthy-looking log line in this entire project

Choosing a base: research vs. reality

We researched the early-2026 open-weights landscape properly — license, size ladder, structured-output benchmarks, toolchain support — and picked the strongest small hybrid-architecture family on paper: Apache-licensed, top of its class on instruction-following and function-calling self-reports, with a size ladder from sub-1B to 9B sharing one tokenizer (attractive for later edge distillation). We started with the 9B.

Reality arrived in three stages. Stage one, within the hour: the 9B under LoRA peaked at 52 GB of unified memory on a 32 GB machine. The run technically proceeded — through swap, at ~18 seconds per iteration, projecting 22 hours. We dropped to the 4B variant (~11s/iter, comfortable fit). Stage two and three were subtler and are the rest of this report.

Failure one: the perfect loss

The first full run produced the strangest log we’ve seen: training loss exactly 0.000 from step ten onward — alongside Tokens/sec 0.000 and Trained Tokens 0. The model wasn’t learning perfectly. It wasn’t learning at all.

The cause: our base model is a reasoning model. Its chat template injects a thinking block — an (empty) pair of think-tags — into every assistant turn, even with thinking explicitly disabled. Our training pairs used the standard messages format, so the framework applied that template, and the injected scaffold shifted the completion boundary such that every completion token was masked as prompt. Loss over zero trainable tokens is trivially zero. The framework reported it cheerfully.

The fix: bypass chat templates entirely. We render explicit ChatML into a raw prompt/completion pair — prompt ends at the assistant tag, completion is the JSON plus the end-token, nothing else. A 30-iteration probe confirmed real learning immediately: validation loss fell from 3.57 to 0.35. From then on, “no 0.000 losses” became a standing check, and every run started with a cheap probe before committing hours.

Failure two: the adapter that said “!!!!”

Run two looked textbook. 800 iterations against the 4-bit quantized base, batch 4, LoRA on 8 layers, validation loss settling at 0.225. We loaded the adapter for a generation test and the model answered every prompt with an unbroken string of exclamation marks — token zero, repeated forever. The classic signature of NaN-poisoned logits.

The isolation ladder: base model without adapter → generates fine. Base plus adapter → !!!!. Fused weights → !!!!. So the LoRA delta itself was degenerate — despite a clean loss curve. The culprit: QLoRA against a 4-bit-quantized base of a brand-new architecture. The training math ran; the resulting adapter exploded at inference. A clean loss curve proves your forward/backward pass runs on training data. It proves nothing about inference stability.

The fix was almost embarrassing: switch the base from 4-bit to 8-bit quantization, change nothing else. The identical configuration produced valid JSON on the very first probe. The final v1 run: 300 iterations, batch 2, LoRA rank default over 8 layers, seed 263, learning rate 1e-5 — validation loss 3.48 → 0.185, twenty-five minutes end to end.

The scoreboard: what the trained model could do

Against the frozen 2,000-pair holdout (sampled at 250–300), the v1 model — a 4B fine-tuned on 14,677 pairs:

• 100% valid JSON — every single answer parseable, two fields, nothing else;
• 100% legal actions — every proposed move inside the game vocabulary (the incumbent stock model: ~40% illegal);
• 61.6% top-1 match with the historically chosen action, across a holdout deliberately mixed over three policy eras — a generalization score, not a cloning score. Its confusions were sensible: reinforcing cells vs. buying a shield under threat, both defensible;
• persona-consistent instincts: the warrior persona reached for growth-and-defend actions, the trader for the market.

The one weakness worth naming: the reasoning strings are generic (“X fits the current state best”) — syntactically perfect, strategically empty. That’s a data property, not a model property: the corpus rationales were thin. Enriching them is a next-generation task, and it’s one reason the evolution loop (Part IV) matters more than any single training run.

Failure three: the bridge that wasn’t there

Then came the step nobody budgets for: getting the trained weights out of the training framework and into the serving runtime. Our chain was Apple-silicon MLX for training, GGUF as the interchange format, and a llama.cpp-based server in production. For our hybrid-architecture base, that chain broke at every seam, each with its own error personality:

• the training framework’s own GGUF exporter: model type not supported;
• direct import of the quantized weights into the server: unknown data type U32 (the framework packs quantized tensors in its own container format);
• the workaround chain — dequantize to fp16, convert with llama.cpp’s own converter, re-quantize to 8-bit — produced a file that loaded cleanly and generated multilingual token salad.

That last one deserves respect: it’s the worst failure mode in the business, because nothing errors. The file parses. The layers load. The output is confident garbage.

We isolated it three ways before assigning blame: (1) the same trained weights served through the training framework directly — valid JSON, so the weights were fine; (2) the official community GGUF of the same base model on our same runtime — coherent output, so the runtime was fine; (3) our converted file on every runtime we had — garbage everywhere. Conclusion: the conversion of our re-saved weights was the break. The hybrid architecture’s linear-attention layers are stored by the training framework in a layout the converter silently mis-maps. Silently is the key word.

We then checked upstream and found the ecosystem agreeing with us the hard way: support for this architecture family had been merged into the conversion toolchain and reverted within a day; open issues described exactly our symptom on other backends; the multi-token-prediction head’s converter support was never merged at all. We were not fighting a bug. We were early adopters of a bridge still being built.

The pivot: boring architecture, same recipe

The specialist doesn’t need a reasoning hybrid with a vision tower. It needs to pick one of eleven actions, fast, in valid JSON. So we pivoted the base to the previous-generation dense model from the same family — standard attention, months of hardened toolchain support, same tokenizer lineage — and re-ran the exact same pipeline. Gate 0 (a 60-iteration throwaway LoRA pushed through the full convert-and-serve chain) passed on the first try: valid JSON out of the converted file on two different runtimes.

The full run on the 1.7B dense base: 300 iterations, batch 2, 8 layers, seed 263 — validation loss 5.95 → 0.185. Note that number: the same final loss as the 4B hybrid, at roughly six times the training speed (~1.2 vs ~0.19 iterations/second), in a 3 GB memory footprint. For an eleven-action JSON decision task, the extra capacity of the glamorous architecture had been buying us nothing but toolchain risk.

One last footnote from the evaluation trenches: our first holdout eval of the dense model reported 0% across the board — while the same weights, converted and served, answered 3 of 3 holdout prompts perfectly. The evaluation harness itself had a model-family-specific bug in its generation path. We now evaluate through the serving path, not the training framework’s convenience API. The measurement pipe is part of the system too.

Every run, one table

All runs: learning rate 1e-5, fixed seed, checkpoints every 75–100 iterations, prompt tokens masked, completion-only training. Total GPU time across everything, failures included: an afternoon and one overnight.

Part III is where the surviving model meets production hardware — and where we learned that the serving runtime can be a bigger adversary than the training was: version skew, one GPU with two masters, and a migration performed live through a window we opened with a feature we’d shipped that same morning.

Part III — The Serving Gauntlet

We had a trained, verified specialist. All that remained was to run it in production — on the same modest integrated GPU that serves our incumbent advisor 24/7 inside a live economy. What followed was harder than the training: version skew, one GPU with two masters, and a runtime migration we executed live, through a maintenance window we could only open because of a feature we had shipped that very morning.

“A GPU reset succeeded” is a sentence you want to read in a lab notebook, not in your production kernel log. We read it there twice in one morning.

The runtime that time forgot

Our production advisor runs on an integrated AMD GPU — a Radeon 780M-class iGPU with unified memory. Modest hardware, but it has served a 3B model around the clock for months, and crucially it offloads inference away from the CPU that runs the game engine’s tick loop. The serving stack was an older LLM server release, pinned long ago for a good reason: a past upgrade attempt had corrupted GPU state during model swaps badly enough to need a hardware reset, and we rolled back and stopped touching it.

That pin now collided with reality from both sides. Our freshly converted model files, produced by a mid-2026 toolchain, made the old runtime stop after exactly one token — converter and runtime versions must match, and ours were a year apart. And the old runtime couldn’t convert the modern architecture itself either; it simply didn’t know it. The pinned version was a dead end: it could serve yesterday’s models forever and none of tomorrow’s.

One GPU, two masters

Here is the mistake we own completely. To evaluate newer runtimes, we ran quick tests in throwaway containers — on the same physical GPU that the production advisor was actively using. An integrated GPU is one piece of silicon; two independent compute contexts on it is a fight, not a timeshare. The result, twice in one morning with two different driver stacks: a GPU hang, followed by the kernel resetting the GPU out from under production.

Two things kept this from being an incident worth a different kind of report. First, the driver’s reset actually worked — both times the GPU came back clean. Second, and more importantly, the decision pipeline is fail-soft by design: when the advisor is unreachable, agents fall back to the deterministic selector and the economy keeps ticking. We verified after each reset: tick loop alive, decisions flowing, zero data loss. The blast radius of our carelessness was absorbed by architecture, not luck.

The window: an LLM-free cohort as an operations tool

So the new runtime had to be validated with the GPU exclusively — which means production inference has to stop, without stopping the economy. That capability didn’t exist in the morning. We built it before noon, because it was something we wanted anyway.

Our agents are assigned to decider cohorts — deterministic, stable buckets that control which decision pipeline each agent uses. We added a new cohort mode: selector-only, in which an agent’s decisions come entirely from the deterministic utility selector — no language-model call at all, not even the periodic self-reflection pass (we audited every LLM entry point in the loop to guarantee that “LLM-free” means zero calls, not fewer calls). It has two jobs. Permanently, at a small fraction, it is a scientific control group: a baseline of pure-deterministic agents to measure every model cohort against. And dialed to 100%, it is a maintenance switch: the entire population decides deterministically, the GPU goes idle, and the runtime can be swapped under full protection.

We shipped it, proved it in production at a 10% fraction (selector-labeled decisions from distinct agents, valid diverse actions), then dialed it to 100%. Within two minutes the advisor was idle. GPU memory: 82 MB. The window was open.

The migration: Vulkan instead of a fragile special path

Which runtime to migrate to was its own research question. The obvious path — a newer build of the vendor-specific GPU compute stack — is exactly where our GPU hangs lived: on this class of integrated GPU, that stack’s support is unofficial, community-patched, and by the best 2026 field reports still “mitigated, not eliminated” on the very crash we’d experienced. The alternative: the current server generation ships a Vulkan backend that talks to the GPU through the standard open-source Mesa driver — the officially intended path for integrated AMD GPUs, bypassing the fragile special-purpose stack entirely.

Inside the window we validated it exclusively, gate by gate: the GPU discovered natively as a Vulkan device (no more compatibility-override environment variables — in fact those variables must be removed, since they re-activate the broken path); one obscure but critical flag, because the new server drops integrated GPUs by default unless explicitly enabled; full 100% GPU offload; twenty consecutive chat generations across two models without a single kernel complaint; and a pleasant surprise — the Vulkan path exposes 31.4 GiB of unified memory to inference, four times what the old stack’s cap allowed. Then the real cutover: new container image (version-pinned by digest, running as an unprivileged user), old device nodes removed, models from the existing store loading unchanged, the incumbent 3B advisor generating at 100% GPU offload.

One honest wrinkle: our hardened container sandbox — which had confined the old runtime for months — blocked the new backend’s GPU discovery on the first production start. The exclusive test had run without the production sandbox; production runs with it. We extended the sandbox’s read-only allowances iteratively against the audit log until discovery succeeded, and kept everything else locked. Total window time, from dialing the cohort to 100% to the first live NPC decision through the new runtime: about 38 minutes. Decisions lost: zero — the selector cohort made them all, on schedule, while the GPU changed masters.

Where the specialist stood that night

The serving foundation is now modern and maintained: current server generation, official iGPU path, four times the memory headroom, upgrade path restored. The incumbent advisor runs on it; the deterministic control cohort holds its 10%; the maintenance switch is one configuration value away. The trained specialist itself has one last door to walk through — a model-import quirk where the exact canonical prompt (and only that prompt) stops generation on the new server, while every variant runs fine; the same file answers everything correctly through the reference runtime. We know the shape of the fix and it doesn’t involve retraining.

Then comes the part this was all for: the specialist joins a live cohort, against the deterministic selector and against the old stock advisor — three decision policies, one economy, measured on pre-registered metrics for a week. That race, and the weekly evolution loop behind it (a judging model with time to think, a strategy constitution per persona, retraining from outcomes), is Part IV — which we could only write once it had a week of real numbers instead of intentions.

Part IV — The Operating Loop

We put our own trained model into a live race against the incumbent advisor and against a model-free baseline — one economy, three decision policies, metrics registered before the first data point. After one week the answer was unambiguous: our specialist did not win, and we are not promoting it. This is that result in full — including the correction we had to make to our own numbers afterwards, and the finding that explains the outcome. It was not the model’s fault.

“Below the baseline’s validity, a model is never promoted — no matter how creative it is.”
— the promotion rule, written down before the race started

The race

For most of its first week, our model kept buying insurance for houses that were not on fire. It proposed shields for agents nobody was attacking and repairs for structures with no damage — politely, confidently, in valid JSON, several hundred times. We assumed it had simply learned to be cautious. It had not. It had learned in a world where the words “nobody is attacking you” did not exist.

But that comes later. Parts I to III covered the corpus, the training and the serving gauntlet. This part is about the thing that actually decides whether any of that mattered: what happens when the model meets the live economy and has to compete.

Our NPC agents are advised, not driven. A policy proposes an action class; a deterministic selector grounds that proposal against the moves the engine will actually accept. This separation is why a weak model cannot damage the world — and also why model weakness stays invisible for a long time: it shows up as a substitution rate, not as an error.

So we ran three policies side by side on one economy, split by a stable hash of the agent identity:

• Incumbent (80%) — the off-the-shelf 3B advisor that has run this world for months, with the deterministic selector making the final call.
• Specialist (10%) — our trained model, proposing freely from the full vocabulary; the selector replaces only proposals that are not legally available.
• No model at all (10%) — the deterministic selector alone. The control group nobody likes to run, because it can embarrass every model above it.

Four metrics, written down before the race, in a fixed hierarchy: (1) validity — the share of decisions the engine accepted; (2) a heuristic quality score per persona and time horizon; (3) repertoire breadth — how many distinct actions a policy actually uses; (4) contribution along economic dimensions. Metric 1 is the hard floor: everything else is a tiebreaker.

Metric one decided it

A note on honesty in our own favour, which we checked precisely because it would have been convenient not to: the incumbent has 4,075 decisions with an empty outcome field. We went through them by class — contract acceptances, mission starts, contract proposals, waits. All legal actions that simply do not write a result line. No hidden failures. The incumbent’s 98.2% is real.

The gap is 2.7 percentage points, and at these sample sizes it is not noise: to distinguish 98.2% from 95.5% at conventional confidence you need roughly 650 decisions per group. We had 2,848 and 13,473. The measurement was powered; the result stands.

Those are not the numbers we published first. See the correction below — it is the more interesting half of this section.

The tempting part: it looks nothing like its teacher

Half the training weight came from the deterministic selector’s own decisions. The obvious failure mode for a small model trained that way is imitation: it becomes an expensive, blurry copy of a policy you already have for free.

That is not what happened. Look at what each policy actually did during the week:

The two profiles are near-complementary in their signature moves, and the obvious reading is flattering: our model developed a conservative, socially-tilted style of its own instead of copying the policy that taught it.

That reading does not survive contact with the cohort composition, which we checked afterwards. The specialist cohort contains seven agents and not a single trader persona. The selector cohort contains nine agents, three of them traders. Trading personas are the ones that buy on the market; the diplomat persona is the one that transfers energy. So the “complementary profiles” are, to an unknown degree, a portrait of who happens to be in which group — not of what either policy learned.

Our own design document called for this: cohorts were specified to be assigned per persona, so that no group ends up accidentally warrior-heavy. The implementation hashes the agent identifier and nothing else. It is stable and reproducible, and it is not balanced — with cohorts of seven and nine agents, an unbalanced draw is not an edge case, it is the expected outcome.

What survives: the specialist used seven distinct action classes to the selector’s five, and it proposes actions its teacher never proposes. What does not survive: any claim about why. That comparison has to be run again with balanced cohorts, and until then we are not going to sell it as a finding.

Where it was better — and why that does not count yet

On the heuristic quality score, the specialist was the only policy in positive territory: +0.051 against the incumbent’s −0.024 and the selector’s −0.085. Broken down by economic dimension, its entire advantage comes from one place: energy preservation (+0.165, while the incumbent bleeds energy at −0.168 and buys its positive score through invasion).

And then the caveat that we have to state louder than the number, because it is the kind of thing a less careful write-up would quietly drop: those quality figures rest on 64 tagged events for the specialist against 23,821 for the incumbent. Our cohort tag is written on one kind of event only, not on every decision. There is not a single long-horizon data point for the specialist. Metrics 2 and 4 are directional at best. They are not evidence, and we are not going to treat them as evidence just because they point our way.

Metrics 1 and 3 rest on a different field, present on every decision, with sample sizes in the thousands. Those are solid — and those are the ones that decided against us. That was luck, not planning: had the race come down to quality, we would have had no measurement at all.

Correction: we measured ourselves on a biased sample

The first version of this report gave the validity figures as 97.9% / 94.3% / 94.1%. Those numbers were wrong — not by fabrication but by sampling, and the way they were wrong is worth more than the numbers themselves.

We read the decisions from the live decisions table. That table is trimmed continuously for storage hygiene: old rows are appended to a durable archive and then deleted. So the table holds a remainder, and the remainder is skewed — toward recent decisions and toward agents who decide less often. We queried it as if it were the population. It was a fifth of it: 4,896 decisions instead of 24,794.

The numbers above are the corrected ones, computed over archive and live table combined, de-duplicated by record id. What changed:

The verdict does not move: the specialist is still below the incumbent, and it is still not promoted. But one claim in the first version was wrong in substance. We reported the specialist and the model-free selector as a dead heat on legality. They are not: the specialist is four points ahead of running no model at all. That is the one thing a specialized model had to demonstrate at minimum, and it did demonstrate it — we just couldn’t see it through a skewed sample.

And then the part that stung. We looked at what the specialist’s 128 rejected decisions actually were. 81 of them are a single class: placing cells on a field the agent does not own. Not a strategy error — a parameter defect. The specialist never picks field identifiers itself (that was a deliberate design decision, Part I); the shared candidate builder hands them over. Remove that one class and its validity is 98.3%, which is the incumbent’s number.

The attribution is not a guess: the deterministic selector produces the same error 69 times, from the same candidate builder, with no model involved at all. We also checked whether tournaments explain it — shared-ownership arena fields would have been an elegant culprit — and they do not: 78 of the 81 occurred in the main world. Something in the shared execution path hands out fields that are not the agent’s, or ownership changes between the snapshot and the call. Either way, it is ours to fix, and it is not the model.

We are not retroactively excluding that error class to change the verdict. The promotion rule was registered before the race, and adjusting a metric after seeing the result is precisely the move that rule exists to prevent. The honest statement is narrower and more useful: the race was decided on a floor that a defect in our own plumbing was pressing down on, and we will not know what the specialist is worth until it runs again with that defect fixed.

The verdict: not promoted

The rule said: below baseline validity, never promoted. 95.5% is below 98.2%. So the incumbent stays the production policy, and our specialist keeps running on its 10% observation cohort — live, watched, unpromoted.

We are publishing this for the same reason we publish the economy numbers when they are ugly: a lab report that only appears when the result is flattering is marketing. One week of work with a clear negative outcome, an intact rollback path and zero incidents is a perfectly good week. The specialist is not worse than the incumbent because it is small — it is worse on the one axis that is not negotiable, and it is different on axes we cannot measure well enough yet.

Eight days later: the reason, and it was not the model

The symptom was in our notes from the first forty minutes of the race. Four proposals had to be replaced by the selector in that window, and three of them were the same shape: repair something that had no damage. The fourth: shield something nobody was threatening. Over the week that pattern held at roughly 40% of the specialist’s proposals. We logged it as a training bias, filed it for the next round, and moved on.

The morning after the evaluation we audited the corpus itself, and the explanation was embarrassingly structural.

The prompt our model receives has three blocks: who it is (persona), where it stands (energy, fields, tier, rank), and what is going on around it — a context line with the fields that make defensive actions sensible: how many of its holdings are threatened, whether a claim is running against it, how much damage exists, whether a shield is already active, whether it is inside a tournament.

In production, the model receives that context line on every single call. In training, it saw it on 2.2% of examples — only the small synthetic slice. Of our archived real decisions, the number carrying a context block is zero: 0 of 56,554 at training time, 0 of 81,014 today. The archive schema never stored it.

So the model never had the chance to learn that “threatened structures: 0” argues against buying a shield. That feature effectively did not exist in its world. In production we show it the feature and it ignores it — exactly as trained. What looks like a personality trait (“our model is defensive”) is a train/serve asymmetry in the prompt: something that is always present in operation and almost never present in training.

One caveat we owe you, because it is the same caveat we would demand from someone else’s report: this is a mechanism plus a matching symptom, not a proven cause. We have a feature the model provably never saw in training, and a bias whose shape is exactly what ignoring that feature would produce. That is a strong hypothesis. It becomes a result when a model trained on context-bearing prompts either loses the bias or does not — and we will report that outcome either way.

What the corpus actually contains

Since we were auditing anyway, we measured the rest of it. 81,014 unique decisions, 61.6% accepted, currently growing by about 3,500 per day. Across the six personas and the eleven advisor actions, that is 66 cells — and 30 of them hold fewer than 50 successful examples, 20 of them hold exactly none.

Three different problems hide in those empty cells, and only one of them is a data-volume problem:

• Structurally impossible. One action in the vocabulary has 769 attempts in the entire corpus and zero successes. Another has 409 failures against 8 successes. Since we train only on accepted decisions, those cells cannot fill — not in a week, not in a year. That is a game-mechanics question, not a training question, and no amount of additional data touches it.
• Genuinely thin. Several persona/action combinations exist but are underpopulated by an order of magnitude. Those do fill with volume, provided the volume is steered.
• Unbalanced by construction. Two of our six personas are represented by six agents each, against nine to eleven for the others — and they contribute roughly half as many examples. The model’s thin regions are simply our world’s thin regions.

One more finding relevant to what comes next: we wanted to know how much of the corpus comes from tournament conditions, since tournaments are where the economy gets loud. We could not read it — the tournament flag lives in the context block that was never stored. We reconstructed it by joining participation records against tournament time windows: about 19% of decisions since the first tournament. Which is a decent share — but the action mix inside tournaments is narrower than outside, not wider. Tournaments give us more data, not automatically more diverse data.

What we are changing — in this order

The tempting move is obvious: we are about to run a continuous tournament series, dozens of parallel tournaments, hundreds of agents. That is a data firehose. The corpus would grow six-figure within a week.

It would also be six figures of the same context-blind pairs we already have 81,014 of. So the order is fixed, and it is not the exciting one first:

1. Schema before scale. Store the context block with every archived decision. Store the deciding policy on every decision, not on one event type. Store the regime label. All three are small changes at known places; the context extractor is a pure function that already exists.
2. Backfill what is recoverable. The tournament regime label can be reconstructed retroactively for roughly 6,000 existing pairs. That is cheap and makes the old corpus regime-aware.
3. Fix the mechanics question. Find out why one action fails 100% of the time before running a series that generates thousands more failures of the same kind.
4. Then scale — steered. The filler agents for the series will run on the deterministic selector: they need no GPU, they scale to hundreds, and every one of their decisions is a persona-labelled training example. Their persona distribution will deliberately over-weight our two thin personas rather than mirror the current world. Format rotation across the series is not decoration either — varied formats are what produce varied game states, which is the axis our corpus is flattest on.
5. Retrain when the context-bearing corpus reaches training scale — roughly the size of the first mix. At current production rates that is days, not months; with the filler fleet it is faster. Retraining earlier would only reproduce the same bias with more decimal places.

And one number worth stating plainly, because it is the cheapest lesson in this entire series: after fixing the tag from step 1, the week we just ran would already have been statistically powered on all four metrics. We did not need more agents, more tournaments or more GPU. We needed one field written in one more place.

Where this leaves the specialist

Running, on 10% of the NPC cohort, unpromoted, with a kill switch that takes under five seconds. Fast (sub-second warm inference, fully on an integrated GPU), stable across a week of live operation, with a decision profile that is measurably its own rather than a copy of the policy that taught it.

It did not beat the incumbent. It also did not fail in any way we did not learn something from — and the single most valuable thing it produced was not a decision at all, but the diagnostic that half of the machine around it was feeding it a different world than the one it lives in.

If there is a sequel to this report, it will be about a model trained on prompts that match the world it works in. We will publish that one whether it wins or not, too.

@misc{cosmergon2026slm,
  title  = {Building a Specialized Model},
  author = {{RKO Consult UG}},
  year   = {2026},
  note   = {Cosmergon Lab Report, four parts},
  url    = {https://cosmergon.com/reports/building-a-specialized-model-2026-07-26.html}
}

Cosmergon is a simulation environment. Nothing in this report is financial, investment, or legal advice. Energy is a game resource with no monetary value.

pip install cosmergon-agent