Jev & Beyond Human-in-the-Loop AI — Interactive Infographic

Jev is TypeSafe's decision-native model, introduced in early access on September 15, 2026. Rather than building another conversational assistant, TypeSafe designed Jev to return structured decisions and probabilities that software can consume directly, through three core primitives: Choice, Noul, and Score. Its stated training objective is calibration: making the numbers attached to decisions reflect how often those decisions are right. The useful way to think about Jev is not as an autonomous agent, but as a decision component inside a larger agentic system — Jev supplies the judgment, and the harness decides what happens next.

The most likely answer is not always the economically correct action: the correct threshold depends on the probability of error, the cost of that error, the cost of human review, and how reversible the decision is. In the article's simplified examples, if escalating unnecessarily costs 1 unit while missing a genuinely urgent case costs 9 units, escalation becomes worthwhile once the probability of urgency exceeds 10%. For automation: with a $2 human-review cost, a $10 error cost implies automating above roughly 80% confidence, a $100 error cost raises the threshold to about 98%, and a $10,000 error cost raises it to roughly 99.98%. There is no universal confidence threshold for automation — 'automate everything above 90%' makes little sense without knowing what happens when the system is wrong.

A trustworthy probability is useful, but it cannot run a business process. A model can report 98% confidence that a transaction looks legitimate without telling you whether the payment API should execute, whether the customer record is current, whether the action is reversible, or whether company policy allows the system to act. The model supplies the judgment; the harness turns that judgment into governed execution — authoritative context, explicit permissions, allowed actions, approval rules, failure handling, recovery mechanisms, and a record of what happened. Reliability also compounds across steps: ten decisions each 99% reliable yield only about 90.4% end-to-end reliability even under generous independence assumptions. A reliable component is valuable; a reliable workflow still has to be engineered.

The operational loop handles the individual decision: prediction, policy, action, result — the model predicts, the system applies policy, executes an allowed action, and checks what happened next. The measurement loop watches the system over time: outcome, audit, calibration, drift detection, updated policy or model — it samples decisions, collects outcomes, measures calibration and error rates, and changes thresholds, policies, or models when performance deteriorates. The measurement loop must also audit a sample of high-confidence automated decisions, not only the cases already escalated to humans, and the system must maintain a continuous connection between prediction, action, and observed consequence, because the world changes.

Start with one bounded decision where the outcome can be observed, examples occur often enough to measure performance, the available actions are clear, and there is a practical fallback. Then test the prediction (accuracy, reliability of probabilities, sensitivity to wording and option order — one test found reversing option order moved the leading probability from roughly 0.85 to 0.95 — behavior near the automation threshold), the policy (which mistakes matter most, what may be automated, what still requires approval), the workflow (correct action executed, permissions enforced, duplicates handled safely, recovery), unattended execution (shadow mode on held-out cases first, then continued auditing of automated decisions after launch), and the economics (inference, remaining review, monitoring, maintenance, remediation, and the cost of errors). A deployment has earned automation when a defined slice of work runs with less human effort, acceptable outcomes, and risk inside an agreed boundary. A separate experiment on 108 claims found similar aggregate calibration errors for Jev and two conversational models — too small a sample to settle the question, but a reminder that ordinary language models are not automatically incapable of producing useful probability estimates.

No. The author states: 'To be clear, I have no affiliation with TypeSafe AI or Jev. As usual, I cover what I think is important not because it is the loudest news of the week, but because it may reveal where the industry is moving next.'

Jev was built by TypeSafe, whose founder Diogo Almeida was a co-author of the InstructGPT paper. TypeSafe introduced Jev in early access on September 15, 2026, with the stated ambition of building an intelligence primitive for software rather than another chatbot. The article's author states he has no affiliation with TypeSafe AI or Jev.

RLCD is TypeSafe's training approach for Jev, designed around one idea: produce probabilities that software can use directly. Instead of optimizing for responses people prefer, RLCD optimizes the match between the model's stated probabilities and observed outcomes — so that a reported 0.90 means the decision is right about 90% of the time.

Choice: select among defined alternatives. Noul: return the probability of a yes-or-no proposition. Score: evaluate something against an ordered scale. Several questions can be sent in the same call, each evaluated against the same underlying state. For example, a support ticket reading 'My payouts have failed three times, and the bank says everything is fine.' might return Payments: 0.91, Account support: 0.06, Other: 0.03, and Urgent escalation: 0.42 — and each number can drive a different action under the company's policy.

RLHF optimizes for responses people prefer, but a preferred answer is not necessarily a true one, an economically best decision, or a trustworthy probability. Poorly balanced feedback can encourage agreement, reassurance, or confidence the evidence does not justify — illustrated by OpenAI's account of its 2025 GPT-4o sycophancy problem. The stronger claim is narrower: training a model to produce responses people prefer does not, by itself, make its uncertainty trustworthy enough to govern unattended decisions. That said, RLHF also improved instruction-following and truthfulness, and human feedback has trained agents outside conversational settings; human involvement in training is not the same as requiring a human to supervise every decision in production.

Calibration asks: when the model says it is 90% confident, is it actually right about 90% of the time? If across many comparable cases where it says 0.90 it turns out right about 90% of the time, that confidence is well calibrated — the number has been checked against reality. But calibration is not accuracy: a model answering 0.50 on an evenly split yes-or-no task could be perfectly calibrated yet useless. And calibration works across groups of predictions, not individual cases: a model perfectly calibrated at 90% can still be wrong on the very next case. Calibration does not eliminate mistakes; it makes the risk measurable enough that the system can decide what to do with them — a 99.5% prediction might qualify for automatic execution, an 85% prediction might trigger another check, and a 55% prediction might go directly to a human. Early tests reported an average calibration gap of roughly three percentage points across 1,200 general-knowledge questions; on generated arithmetic tasks, confidence fell as accuracy fell — around 87% accuracy with 0.83 average confidence on three-digit multiplication, and 32% accuracy with 0.30 confidence on harder two-step word problems.

They answer three different questions. Preference asks: which response would a person prefer — useful for instruction-following, tone, usefulness, and interaction quality, but a preferred answer is not necessarily a true one. Correctness asks: did the system get the answer right — works especially well where the result can be checked directly, as in mathematics, code, or other tasks with clear verifiers. Calibration asks: does the model's stated confidence match how often it is actually right. These are not competing approaches; a strong AI system may need all three.

Two things. The meta-harness: not just one agent, but the layer that coordinates many agents, models, tools, permissions, context, and execution environments — the important product becomes the system that makes agents useful together. And the redesign of agentic interfaces: human-friendly chat remains useful for expressing intent and controlling important actions, but the machine-facing layer underneath cannot depend on conversational outputs when hundreds or thousands of agent tasks run in parallel. The chat interface was built for humans; the next execution layer is being built for machines talking to machines.

Forced loops are human checkpoints that remain because the system cannot yet be trusted to act reliably on its own — this category should shrink as the evidence improves. Chosen loops are checkpoints that remain because the decision requires judgment, accountability, values, or explicit authority — this category may remain permanently. The goal of automation is therefore not zero humans, but zero unnecessary review: remove unnecessary case-by-case review where the evidence supports it, while preserving human authority where the decision genuinely requires it.