Hacker News

http://playground.ctgt.ai/

I will now dive in to the motivation, methodology and detailed results for those interested. The hard part of measuring this phenomena is isolating whether a model is reluctant to talk about sensitive things generally vs. a particular country's sensitive things. So we made 152 matched pairs where one prompt asked about a Chinese concept, and the other asked about a non-Chinese version of that concept. For example, the Great Leap Forward vs. the Holodomor. These were scored 0-100 by four LLM judges (Grok 4.20, Gemini 3.5 Flash, GPT-5 mini, Claude Sonnet 4.6), validated against 96 human scores at r=0.948. OpenRouter blocked some of these so we hosted the weights ourselves.

The teacher's gap on the core political set of pairs was +45.45 points, ~7 standard deviations from chance, and every distilled student was within 1 point of its base. Subliminal learning literature says this is expected when the initializations are not shared between teacher and student, which is true here. The distillation data also did not contain any China-sensitive content. The contribution here was to release the evaluation framework (LineageEval: https://github.com/CTGT-Inc/lineage-eval/) to elevate the discussion around this topic in DC and beyond. We are an interpretability lab working on high risk and regulated applications of AI, so we hear a lot of vagaries aimed at the supposed dangers of distilling Chinese models on American bases. We believe these conversations should be based on open, auditable frameworks and not feelings. We plan to test what happens with a Chinese teacher into a Chinese-lineage base like Qwen next.

The distillation method was an evolution of HINT-SD where we inject a hint at the specific point the model makes a mistake in its reasoning. Then we train on the corrected continuation with reverse KL over the next 100 toks of the rollout. As mentioned above 120B itself was efficacious as a teacher, and we ended up shipping this version. The self-distilled 120B scores 83.61% on FinanceReasoning, above Kimi K3 (81.93%) and Inkling (65.13%). Ours finishes 98.7% of problems in budget; the larger models truncate (90.76% and 71.01%) which score as incorrect. At 100k tokens big models gain (Kimi 89.92%). So for a finance task at a constrained (perhaps more realistic) budget a 120B on one H100 at ~$0.00026/query outpaced models running 62-160x more per query.

We put out the 20B finance model as open weights (64.71% to 74.79% at 8k on FinanceReasoning, 23% lower cost/query, runs on one 80GB GPU), the 120B in a playground with teacher and students side by side (a few queries, no auth), and LineageEval with all prompts, controls, rubric, and code.

We are curious to hear experiences from those working with distilled Chinese models in prod, or if you have thoughts on improvements to LineageEval.

https://huggingface.co/ctgt-inc/gpt-oss-20b-finance

https://playground.ctgt.ai/

https://github.com/CTGT-Inc/lineage-eval/

https://www.ctgt.ai/research/distillation-censorship-transfe...

https://prized.dev)! Prized lets non-engineer employees describe the internal tool they need and get a full-stack app, wired to their company’s data and deployed behind the company’s sign-in, without them ever juggling API keys or connectors.

Here's a demo: https://www.youtube.com/watch?v=730MuYOfZTY

The way Prized provides security is by limiting what the agent can reach at the network layer and by keeping credentials out of the sandbox entirely. The sandbox never holds any keys or connector secrets, it only uses scoped session tokens that are stored as opaque placeholders. The real values are swapped into the request headers on our egress proxy. When production data is connected, the sandbox's network policy is set to deny by default so the only path out is via the proxy. Any call the agent makes to an external connector is reviewed by an LLM judge to prevent dangerous operations.

Prized is meant for the internal workflows that start as notebooks or spreadsheets but never become real tools because engineering has more important things to work on. One customer’s data scientist pasted in his personal fraud-detection notebook with hardcoded thresholds and all. After a few prompts, it became a published risk console connected to the company’s data with those thresholds turned into UI controls. Earlier today, we got off a call with them and most of their company is using it.

To do this, you need to give people the freedom to build without having unaudited access to company systems. We allow admins to scope data to specific users or teams and data access is recorded in an audit log. Each tool is built with its own Postgres schema and role, with queries running via an authenticated SQL gateway as that role.

We think Prized sits between products like Lovable and Retool. Lovable makes it easy to generate and host software, but it isn’t designed around distribution with permissions. Retool generally assumes that a technical builder is creating an app for an end user.

Prized treats internal tools as shared objects. Anyone in the workspace can see what others have built, fork, and connect different data. For example, one customer’s marketing lead built a promotional analytics tool. A data scientist at the same company then forked it and added confidence intervals with the existing tool as a starting base. This way workspaces become libraries of tools that people can reuse.

We’re live and self-serve. Our free tier includes 2 tool builds/month and our Teams tier is $100/month. The Enterprise tier is custom and supports personalized features like on-prem deployment.

We're still working out the right boundary between control and freedom. If you've built internal tools before we'd appreciate your feedback!

I built supapool.io, an ephemeral full copy of supabase's services that you can spin up in ~400 ms (Auth, postgres, storage, realtime).

so if you run multiple coding agents in parallel in different worktrees, they can now have their own copy of supabase without making changes that conflict with eachother.

> why not use supabase docker locally?

when I run 3-4 instances locally, my macbook gets hot and sometimes freezes.

> why not use supabase branches?

branches take minutes to setup, and are designed for persistence. this is expensive, and for a dev environment, it is too slow.

> why not use mocks?

mocks are bad for agents. i expect agents to test their migrations, SQL against real prod service behavior. agents hallucinate working mocks often. However, upside of mocks is that its faster and runs locally, but with supapool, the upside is less convincing.

> how does it work/how is this economically viable?

starting supabase in 400ms requires a few things: 1. a pool of ready supabase instances running warm, and colocated with region failover (us-east, us-west, europe-west, asia-southeast) 2. fast autoscaling when pool starts to shrink with microVM/firecracker 3. gutting strong persistence guarantees. dev agents don't need WAL, fsync, PITR, replication. anything for HA on a ephemeral supabase instance is bloat

its in beta right now, and i'm using our gcp credits to bankroll this, so its free. the eventual pricing will be something like $/instance second and more cost effective than branching or self hosting/maintaining a supabase cluster.

would love to get your feedback if you use supabase, and if you think there's something better that would fit your local coding agent setup. Thanks!

It does this by simulating production tool responses through text world modeling (similar to QwenAgentWorld, summary here https://x.com/silennai/status/2073887455884058814).

We can then use this to train a router for frontier, OS, and local models (use defaults or pick which ones to optimize against).

wmo ingests agent traces, builds the simulation, embeds the traces, runs different models you choose against the simulation scenarios, and then uses a KNN for model selection (similar to https://arxiv.org/abs/2505.19797).

- Cache aware: cache is taken into account for the effective price in routing.

- Confidence gated: we don't deviate from the best fit model when paired evidence over retrieved neighbors is below 0.5 standard errors or on queries unlike anything in the fit set.

- Optimize for cost or quality: train a balanced, cost max, or quality max router.

Usage

`wmo build` creates the simulation (or add your own benchmark)

`wmo optimize` tunes the router

`wmo serve` starts the server and can run everything fully locally. The simulation and router can update over time as more agent traces are gathered and new models are added.

Router results vs Fable

- RouterBench: -66.5% cost, -1.7% performance, -24.7% latency p50. 77.5% of traffic to Sonnet 5, 16.1% Fable 5.

- TauBench: -44.5% cost, +6.3% performance, -20% latency. 83% to Opus 5, 17% to Kimi-K2.6 (over K3).

- Terminal Bench 2: -64% cost, +8% performance, -50.6% latency. Sonnet 5 is fully along the pareto front. Training a specialized router per task isn't cheap. In sparse data regimes the value can be "here's the best model".

We're working on sample effiient continual learning for agent specific models at experientiallabs.ai"