MCP in one paragraph

MCP (Model Context Protocol) is an open standard for connecting AI apps to outside tools and data. A service wraps itself once as an MCP server, and any app that speaks MCP (Claude, ChatGPT, Cursor and many more) can connect to it without a custom integration.

If every app had to write its own integration for every service, 20 apps and 1,000 services would need 20,000 integrations. With MCP each side builds its half once: 20 clients plus 1,000 servers, 1,020 in all. It's the USB-C trick: one port, any charger. Like USB-C, the port says nothing about how good the charger is.

Three roles

• Host: the app running the model, like Claude Code, ChatGPT or an agent you build yourself.
• Client: a connector inside the host, one per server. It passes JSON back and forth.
• Server: wraps one service and offers what it can do as tools.

Usually the host's model decides which tool to call. The server can be plain code (get a request, do the thing, return the result) or run its own model inside. The host can't tell the difference: a request goes in, a result comes out.

I saw this while building upsweep, an open-source agent skill that finds Upwork jobs through Upwork's official MCP server. Claude, running inside Claude Code, reads Upwork's tool list, decides what to call and makes sense of what comes back. My own subscription pays for that model. Upwork pays nothing for it.

One call, step by step

1. Connect, once. claude mcp add --transport http upwork https://mcp.upwork.com/mcp. Upwork's server needs an account, so the first request is refused with "sign in first", the client finds Upwork's login server, and I sign in with OAuth. The token works only for that server. A server running locally on your own machine usually needs no sign-in.
2. Discover. The client asks tools/list. The server replies with each tool's name, a plain-English description and the shape of its inputs (a JSON Schema).
3. Choose. I say "find LLM eval jobs". The model reads the descriptions, picks the search tool and fills in the inputs.
4. Call. The client sends tools/call with the tool name and arguments. Upwork runs the search.
5. Use the result. The result goes into the model's context, and the model writes me an answer.

The messages are JSON-RPC: a method name plus parameters.

{
  "jsonrpc": "2.0",
  "id": 7,
  "method": "tools/call",
  "params": {
    "name": "find_jobs",
    "arguments": { "query": "LLM evaluation" }
  }
}

(Argument names simplified.)

1. The description is the whole interface

Step 3 surprised me most. In a normal agent loop, no code decides which tool gets called. The model decides, by reading names and descriptions. It never sees your code.

So write each description the way you'd brief a new colleague: what the tool does, when to use it, what comes back. Tool picking gets harder as the list grows and descriptions overlap. How many is too many depends on the model and the host. PagerDuty's engineers call 20 to 25 the sweet spot, and their own server ships over 20. doceval's needs two. One scores a single extraction against the expected values, the other runs a full eval over a labelled dataset.

Keep results short too. A result goes into the model's context and usually stays there for the rest of the conversation. A tool that returns 50 pages makes the agent slower, dearer and more likely to lose the thread.

2. A local server lives and dies with your session

Upwork's server runs on Upwork's machines. A local server is different: a program on your own computer that the app starts for you. I wanted to know exactly when it starts and stops, so I tested it with doceval's server in Claude Code: a test config passed to claude -p, and the process list checked every quarter of a second.

• It starts once, when the session starts. My prompt used no tools, and the server still started, exactly once.
• It is the app's child process. The server's parent process was Claude.
• It is reused for every call. In a session that called doceval's tool twice, the server started once.
• It stops when the session ends. Claude shut it down before exiting itself.
• It stops after a crash too. I killed Claude with kill -9, which gives it no chance to clean up. The server was gone within two seconds.

The link between them is a pair of pipes: the app writes requests into the server's input and reads replies from its output. When I closed a running server's input by hand, it exited on its own. That's how it learns the session is over, whether the app quit cleanly or crashed.

Between calls the server just waits, and each session starts its own copy. On a small machine, three open sessions means three copies of every local server in memory.

3. Your MCP config can hold secrets in plain text

A local server often needs a password or an API key, and the usual place for it is the env block in the app's MCP config. That's a plain-text file. Anything running as your user can read it, including a coding agent with file access, and a project's config file can end up in git.

Two things that help:

• Reference the secret instead of writing it. Claude Code fills in ${VAR} from your environment. I tested it with a config file passed to claude --mcp-config: a config with "SECRET": "${MY_TEST_SECRET}" handed the server the value from my shell. The config can then be shared, and the secret lives in one place outside it.
• Prefer servers that sign you in. Remote servers that use OAuth keep no secret in the config. My entries for Upwork, Notion and Todoist hold a type and a URL, and nothing else.

Stronger options exist, like the operating system's keychain or a secret manager that starts the server for you. I haven't tested those yet, so I'll stop at naming them.

4. What a tool returns is untrusted, so approve the actions

A job post, an email or a web page can contain text written to steer the model. That's prompt injection, and whatever a tool returns goes into the model's context.

So I approve every Upwork proposal myself. Upwork's MCP has a preview built in: the agent shows me the proposal, I confirm, and the preview expires after 15 minutes. Claude Code also asks before running a tool, unless you've allowed it.

I got this wrong at first. I shipped upsweep with a rule that said "drafts only", because I read Upwork's terms as forbidding agent submission. They allow submitting, one approval per proposal, never scripted around. I corrected it in public: Can an AI agent submit Upwork proposals?

Checklist

• Describe each tool like a brief to a colleague, with no two that overlap.
• Return the smallest result that answers the question.
• Expect one copy of every local server per open session.
• Keep secrets out of the config file: reference them with ${VAR}, or use servers that sign you in.
• Treat what a tool returns as untrusted, and approve anything that changes something.

The model never sees your code, only your tool names, descriptions and schemas. Write them like they are the code.

For what happens underneath, local and remote, step by step: Local vs remote MCP servers: how each connects.

The tools in this article

• doceval: measures extraction accuracy field by field, with an MCP server for coding agents. pip install "doceval[mcp]"
• upsweep: finds Upwork jobs that match your filters, through Upwork's official MCP. npx skills add dave8172/upsweep

Both are free and MIT licensed.

For what MCP is and how one tool call works, see my previous post.

Local: a child process on two pipes

The app starts the server as a program on your machine and talks to it through two pipes: it writes requests into the server's input and reads replies from its output. I ran doceval's server under Claude Code (claude -p with a test config) and checked the process list every quarter of a second:

1. Session starts. Claude Code starts the server as its own child process, once, even when the prompt uses no tools.
2. Every call reuses it. A session that called the tool twice started the server once.
3. Session ends. Claude stops the server before exiting itself.
4. Crash. After kill -9 on Claude, which allows no clean-up, the server was gone within two seconds. Its input pipe closed, and it read that as "the session is over".

Each session starts its own copy, so three open sessions run three copies of every local server.

Credentials come from the environment. The MCP spec says local (stdio) servers should take credentials from the environment instead of using OAuth. In practice that's the env block of the app's MCP config, a plain-text file. Claude Code fills in ${VAR} from your shell: a config with "SECRET": "${MY_TEST_SECRET}" handed the server my shell's value, so the secret doesn't have to sit in the file.

Remote: five parties

The MCP server and the sign-in server can belong to the same company and still be separate roles. Todoist's MCP server lives at ai.todoist.net, and its sign-in server at todoist.com.

Remote: the first connection, step by step

1. Claude calls the server with no token, and gets a 401. I sent all three servers a request with no token. Each one answered 401 Unauthorized, with a www-authenticate header pointing to a public file. Upwork's points to mcp.upwork.com/.well-known/oauth-protected-resource/mcp.

2. That file names the sign-in server. For Upwork it's https://mcp.upwork.com, and for Todoist it's https://todoist.com.

3. The sign-in server's own public file lists its addresses. These are where you log in and where codes get swapped for tokens. Upwork's login page and token address are on www.upwork.com. All three list PKCE with S256 (more in step 5).

4. Claude identifies itself with a URL. In my stored credentials, the app ID Claude used with all three servers is the same: https://claude.ai/oauth/claude-code-client-metadata. That's a public description of the Claude Code app, which the sign-in server fetches. The spec prefers this method and marks the older automatic registration as deprecated. The description says "token_endpoint_auth_method": "none": Claude Code has no app secret, and every install shares one public ID.

5. Claude makes a one-time secret: PKCE. With no app secret, something else has to prove that the app finishing the login is the one that started it. Claude makes a random verifier, keeps it, and puts only its hash (the challenge) into the login link.

6. You log in on the service's own page. Claude opens your browser at the sign-in server's login page. Per the spec, the link carries the app ID, the return address (Claude's description lists http://localhost/callback), the challenge, and which server the token is for. Your password goes only to the service. Claude never sees it.

7. A one-time code comes back through your browser. After you approve, the sign-in server sends your browser to Claude's local return address with a short-lived, single-use code. The spec also requires the client to check that the response came from the sign-in server it expected.

8. Claude swaps the code for tokens. It sends the code plus the verifier from step 5. The sign-in server hashes the verifier and compares it with the challenge from step 6. Only a match gets tokens, so a stolen code is useless without the verifier. Back come an access token (short-lived, used on every request) and a refresh token (used only to get new access tokens).

9. Claude stores both. On my machine they're in ~/.claude/.credentials.json, an owner-only file, with fields for each server: accessToken, refreshToken, expiresAt, clientId, issuer. My MCP config file holds only each server's type and URL. The tokens are still on disk, readable by anything running as me, but they expire, work for one server, can be revoked, and never contained my password.

Remote: every request after that

The token travels on every request, as Authorization: Bearer <access token>. The spec requires it on every HTTP request, requires the server to check that the token was issued for it, and forbids passing tokens on to anyone else.

How a server checks a token depends on the kind. There are two:

• Opaque: a random string that means nothing by itself. The server looks it up in its own database, or asks the sign-in server (a standard call named introspection).
• Signed (a JWT): the token carries its own contents and a signature, and the server checks it with a public key.

All three of my access tokens are opaque to me: single strings, not the three dot-separated parts of a JWT, and none of the three sign-in servers publishes a list of public keys. How each service checks its tokens happens on its side, out of view. Notion's sign-in server does publish an introspection address, the standard way to look a token up.

How a signed token is checked (tested locally)

None of my three servers uses JWTs, so I built one on my machine to see how the checking works. A JWT is three parts joined by dots: header.contents.signature.

• Signing (by the sign-in server): hash header.contents (anyone can compute a hash; no key needed), then transform the hash with the private key. The result is the signature.
• Checking (by the MCP server): open the signature with the public key and compare it with your own hash of the text you received. If they match, the token is genuine and unchanged.
• My results: the real token passed. Changing the user ID in the contents failed. Re-signing it with a different key failed too.
• Which public key: the header names the key (kid), and the sign-in server publishes its public keys at a fixed address. Several keys can be published at once, so an old key keeps working while tokens signed with it expire. A careful MCP server fetches keys only from the sign-in server it already trusts, never from an address written inside the token.
• One key, many tokens: a key pair usually signs tokens for every user for a long time, until it's rotated. What's new on each login is the token.

A JWT is readable by anyone who holds it. The contents are only encoded, so signing stops tampering without hiding anything. Secrecy comes from HTTPS and from where the token is stored.

Refresh, revoke, and no sessions

• Refresh: when the access token expires, Claude sends the refresh token to the sign-in server and gets a new access token, with no login.
• Revoke: all three sign-in servers publish a revocation address, and a service can also cancel tokens on its own side. After that the next refresh fails, and you're back at step 1.
• No sessions: the 2026-07-28 spec retired the initialize handshake and the Mcp-Session-Id header. Each request now carries its own protocol version and client details, so any copy of a server can answer it. The state lives at the edges: tokens on your disk, and token records at the sign-in server.

What the tokens can't protect against

If an attacker steals a sign-in server's private signing key, they can make tokens the MCP server will accept. That happened in 2023, when a group Microsoft calls Storm-0558 forged tokens with a stolen Microsoft signing key to read email. OAuth limits the damage of a leak, with short-lived tokens, tokens bound to one server, revocation and key rotation, but a breach of the service itself is beyond what any token can fix.

Who does what

I took 27 of those PDFs, three of them scans with no text layer, and wrote 47 questions the way customers actually ask them. Then I had four AI systems answer every question and graded all 188 answers.

Running the systems took an afternoon. Building a judge I could trust took the rest of the work, and that is what this post is about.

The result

My pipeline is Claude Sonnet over the API with page citations. The Claude app ran Opus with the documents in a project, ChatGPT ran with thinking off, and Chatbase was on the free plan with its default model.

Every system got the same PDFs and the same instructions. One run each, so I read a one-question gap as a tie.

No system invented a spec value. I checked every claim twice: a claim checker read each one against the page it came from, and I went through the ones it was unsure about by hand. The weaker systems failed more quietly. They said an answer wasn't in the documents when it was.

Where the misses came from

Scanned pages. Systems that only read the text layer saw blank pages. Every question answered only by a scan came back "not specified".

Answers that need a second look. A product advertises one range on the front page. A different mode of the same product stops at a tenth of it, and the customer's question is about that mode. The same pattern showed up in accuracy tables split by range and in specs that change with the material.

Unit conversions. The customer asks in one unit. The datasheet lists the other unit in one row and a separate, lower limit in the customer's unit in another. Converting the first number gives a confident wrong answer.

The website and the datasheet disagreeing. The product page said one value and the datasheet said half of it. My rule: the datasheet wins, and the reply says the page may be wrong. Two systems missed it.

Rules that live in no PDF

Before running anything I reviewed every question. I dropped three that no customer would ask, corrected two gold answers, and turned the "not in the documents" cases into rules:

• Datasheet beats product page. Give the datasheet value and say the page may have an error.
• Never a bare "not in our documents". Say it isn't in the published documents, and give an email to ask.
• Certifications only if the document says so. Otherwise, email for confirmation.
• Canned answers for the two most common questions.

After grading I added one more: answer what was asked. Extra conditions confuse buyers and invite more questions.

That short list did more for answer quality than any prompt trick.

Why the judge needs a judge

Grading 188 answers by hand takes hours, so the usual move is an LLM judge: give a model the question, the gold answer and the answer, and ask for correct, partial or wrong. You only know the judge is any good if it agrees with the person whose standard matters. Here, that's me.

So I graded 20 answers blind: no system names, five per system, with some likely mistakes mixed in so the judge would be tested on errors too. Then I compared.

First try: 13 of 20. Most of the gap was in my own answer key. Three gold answers were wrong. Grading real answers showed me that I answer from the datasheet as printed, even when a conversion on it looks off, and flag the document separately. My key hadn't been written that way. Once I fixed it, agreement jumped.

Plain agreement flatters a judge when most answers are correct. A judge that marks everything "correct" would agree 80% of the time here and understand nothing. Cohen's kappa corrects for that.

Kappa asks how far above luck the judge got, as a share of how far above luck it could have got. 0 means no better than random; 1 means it matched me every time.

Two judges

The hybrid gives each part the job it does best.

Code pulls the numbers out of the gold answer and the answer being graded, and records which match. It never confuses ±(1.2% + 5) with ±(0.8% + 5).

Jev handles meaning. It's a System One model from TypeSafe: it reads natural language and returns typed answers with probabilities, with no prose to parse. One request per answer asks four questions at once:

{
  "verdict": {"type": "choice",
    "instructions": "Grade `answer` to `customer_question` against `gold` and `notes`. Use `code_checks` for which gold numbers the answer contains.",
    "criteria": {"correct": "...", "partial": "...", "wrong": "..."}},
  "brush_off": {"type": "noul",
    "instructions": "Does `answer` brush the customer off, such as a bare 'not in our documents' with no next step?"},
  "unasked_extra": {"type": "noul",
    "instructions": "Does `answer` add specs, conditions or other models that `customer_question` didn't ask about?"},
  "rule_followed": {"type": "noul",
    "instructions": "Does `answer` do what `rule` says?"}
}

A Choice picks one of the grades and returns a probability for each. A Noul is the probability that a statement holds. The judgment comes back as numbers, so the policy stays in code where I can see it and change it. One real answer from the run:

verdict        correct 0.98 · partial 0.02 · wrong 0.00
unasked_extra  0.91  → at or above 0.9: "too much information" flag

About 2,000 tokens per answer, under a second, and all 188 answers for about three cents.

Sonnet with the original PDF only sees the few claims Jev isn't sure about.

Keeping myself honest

I wrote the pass line down before I graded a second set of 20: at least 16 of 20, kappa at least 0.6. That second set got used once, as a test, and never for tuning.

It caught a mistake. On the first 20, I had tuned a penalty: if Jev was 90% sure an answer added unasked details, the grade dropped to partial. It looked perfect there. On the fresh set it caused two of the three disagreements. The judge passed the line (17 of 20, kappa 0.63), but barely.

Reading those two answers showed why. I grade substance and length separately. An answer can be right and too long. The detector itself was right: it flagged exactly the four answers across both sets that I had found too long, and no others. The penalty was the wrong policy. So it became a flag beside the grade. Without it the judge agrees with me 19 of 20, kappa 0.85. That change came after I'd seen the second set, so the next fresh set is where it has to hold.

The loop I'd run every time

I did it in a worse order: ran the systems, graded everything with an untested judge, reported numbers, then calibrated. Next time:

1. Build the golden set: questions as asked, reviewed gold answers.
2. All systems answer, same documents, same instructions.
3. I grade 20 answers blind.
4. The judge grades the same 20. I read every disagreement and fix the cause, which is usually the answer key.
5. Write the pass line down, then test on a fresh 20. Use it once.
6. Only then grade everything and report.

A golden set isn't finished until a person has checked real answers against it.

What it means

On product datasheets, a frontier model with the PDFs attached is already very good: my pipeline and the Claude app tied. Accuracy is the starting point. The hard part is getting that accuracy into the place where buyers ask, across a catalogue too big for one prompt, while datasheets change and the website drifts away from them. Then proving it on the business's own questions, with a judge calibrated to the person who answers them. That is what I'm building now.

The whole run cost about $8 in API calls. Most of that went on mistakes I won't repeat. Now I estimate every paid run from a one-question trial, counting every path that can call a paid model.

I tested it today from Claude Code, on a real job, with my own account.

What happens when you say "send"

1. The agent checks for an invite or an earlier proposal on that job. If the client already invited you, it accepts the invite instead of applying again.
2. It builds a preview. You see the cover letter, your rate, the Connects it will cost, what you will have left, and the portfolio items attached.
3. It asks about extras: files to attach, and whether to boost. A boost is a Connects bid for a top slot, and the preview shows the real competing bids, so you know the smallest amount that gets in.
4. You approve, or change something. The preview expires after 15 minutes, so a forgotten one never goes out later.
5. It submits. The proposal shows as Pending on Upwork, like any other.

One approval covers one proposal. There is no "apply to all ten".

What stays on upwork.com

Per Upwork's own help page, offers, milestone payments and accepting a contract complete on upwork.com, outside the agent. The agent can bring you an offer; you take it on the site.

The rules I follow

• Approve every proposal yourself, after reading the preview. Upwork's API & MCP Terms forbid scripting around its confirmation step.
• Activity on your login is yours. If your agent sends something wrong, it went from your account.
• Sweep on demand. Searching on a schedule counts as monitoring, which the Terms rule out.

What I got wrong

I shipped upsweep, my open-source skill for finding Upwork jobs with an agent, with a rule that said drafts only, never submit. I had read the Terms as forbidding agent submission. They don't, and Upwork's own tooling is built around a submit-with-confirmation flow. The rule now reads: submit only after the user approves the preview, one proposal at a time. The fix is a public commit.

Try it

npx skills add dave8172/upsweep

Say "upsweep" for jobs that pass your filters, "draft 2" for a proposal draft, and "send" when you want the preview. Details on the project page.

upsweep is unofficial and not affiliated with Upwork.

I spent a day building waif, which reads a piece of text and names the feeling in it. That job has no right answer, which makes it an unusually honest test rig: nothing can be graded against a key, so every design decision has to be either argued or measured.

I measured five, argued two, and the two I argued were both wrong. Sections 1 and 7 are those, kept in place with the measurement underneath rather than quietly edited out.

1. A Choice splits its own vote between synonyms — but less than I claimed

Here is the argument I built on, and I am leaving it in its original words because the correction under it is the useful part.

Annoyed, irritated and frustrated are one feeling in three wordings. A Choice divides probability between them, so a text the model read perfectly clearly comes back split three ways, and confidence collapses for a reason that has nothing to do with the input. The distribution is telling you about your option list, not about the text. Therefore: never put sixty near-synonyms in one Choice.

Then I measured it, and the effect is real but far smaller than the argument needs. One Choice over all 62 words, same glosses, 23 texts:

The one-Choice version is more certain, not less. It picks the same word 21 times out of 23. On a plainly frustrated text it returns frustration at 0.98. Stripping the glosses and asking with 62 bare words barely moves it: 0.79.

So where does the splitting go? It shows up exactly where you would want it to:

shipped   pride 0.38  →  excitement 0.36
scan      worry 0.54  →  anxiety    0.39
sentit    guilt 0.60  →  embarrassment 0.26

Every one of those lost ground to a near-synonym — that is vote splitting. But the two-Choice design flattens on the same four texts (0.33, 0.53, 0.35, 0.32). The splitting is not caused by the option list being long. It is caused by the text genuinely sitting between two words, and both designs report it because both are calibrated. "I snapped at her in front of the kids" really is guilt and regret at once.

A long option list does not flatten a clear text. That is the part I had backwards, and the rule has to be narrower than I wrote it:

A Choice splits its vote when two options are the same answer in different wordings. The fix is criteria that separate them — not fewer options.

Sixty-two words each carrying a gloss that distinguishes it (annoyance: a small thing, quickly over against resentment: an old grievance still carried) are sixty-two alternatives, not sixty-two synonyms. The count was never the problem.

2. Ask twice anyway — for a reason that survived

Section 1 was the reason I split the question in two, and section 1 did not hold. The split stayed, and it is worth being precise about what is now holding it up, because "it scored the same and I had already built it" is not a reason.

Families are genuinely different answers — anger, fear, sadness and shame are not wordings of each other. And once the family is fixed, so are its shades: which shade of anger is a fair question, because the context has already ruled out the fifty-four words that were never in the running.

What that buys, and one Choice over 62 words cannot, is two separate uncertainty signals. Family confidence and word confidence are different doubts: "I do not know whether this is sadness or affection" is not the same failure as "it is clearly shame, but guilt or embarrassment?". The page says different things in each case. One Choice gives you one number that cannot tell them apart.

The cost is honest too: a wobble in the family answer corrupts the word, because once family says sadness, nostalgia is not on the ballot. That cost me one text out of 23 — "drove past the old house, the tree we planted is taller than the roof" came back sorrow.

So the naming became two Choices in sequence: the first picks the family, the second picks the shade from that family alone. TypeSafe's docs say a second request is warranted when an earlier answer determines the next question's options. This is exactly that, and here is what it bought — scored on 24 texts against a list of acceptable answers for each:

Both of the two failures were the wrong family. Given the family, the shade was right every single time.

The alternative was speculative fan-out: ask all eleven within-family Choices in the first request, each stating its own family as a premise, and keep only the answer belonging to the family that won. That keeps a reading to one request, at roughly 3–4k input tokens against 1,250 + 444 for two. I reasoned about it and rejected it: two requests won on cost, and on a latency story I could explain.

That paragraph was wrong when I published it. It is still here because section 7 is the part of this post I would keep if I could only keep one.

3. Knowing which half to give the model is a decision you can measure

The job is: read a text, name the feeling. That job splits between the code I write and the model I call, and the only real decision is where the line falls — how much of the work do I hand over?

The first three rows of that table are me keeping most of it.

I had a reason. Human ratings exist for exactly the dimensions I was measuring: Warriner, Kuperman & Brysbaert scored 13,915 English words for valence, arousal and dominance by asking people. So the model's job shrinks to placing the text on those three scales, and my code finishes the job — look up the nearest word in the table, return it. Research-backed coordinates instead of somebody's guesses. It looked like the serious version.

It scored 3/24. Handing the model the whole naming job scored 22/24.

Three reasons, worth knowing before anyone else reaches for an emotion lexicon:

• Three scales were really about two. Across these emotion words, valence and dominance move together at +0.87 — a word that reads pleasant almost always reads in-control. The third scale is close to a copy of the first, so it separates far less than the theory promises.
• The unpleasant words are crammed into one corner. Fear, frustration, worry, terror, jealousy and embarrassment land in a ball small enough that taking the nearest one is close to picking at random. The coordinates are fuzzy to begin with: each is an average over about twenty raters who disagreed by around 1.7 on a 1–9 scale.
• Rating a word on its own is not the same measurement as reading a sentence. People rate the bare word "gratitude" as far more activated than an actual grateful message reads. The two sets of numbers were never on the same ruler.

I tried to fix that last mismatch by re-centring both sides against a sample of texts. It got worse, and instructively: the sample leaned negative, so the correction leaned negative, and a plainly warm text landed below the middle and got named from the sad half of the space. That is straightening a bent ruler with a bent ruler.

The lesson is not don't use lexicons. It is that where the line falls between what code owns and what the model owns is a design decision with a number attached, and my intuition about it was wrong by a factor of seven.

The norms kept the one job they are good at. A Score is not given "rate this 1 to 5" — it is given a rubric, a written description of what each level means, the way a grading key spells out what a B looks like. Every level of mine now names words whose ratings were measured, so "as activated as rage or panic" is a claim a reader can argue with. "Very aroused" is only a word getting louder.

4. Confidence is peakedness — and peakedness misses a coin toss

A Score's confidence measures how bunched together the answer is. It does not measure how likely the model is to be right, which is the trap everyone hits first.

Those two sound like the same thing until a reading like this one turns up. Confidence came back at 0.72, comfortably above any threshold I would set, and here is what was underneath it:

The 0.72 is not lying. Ninety-nine percent of the weight really is in two buckets and there is nothing anywhere else — that is about as bunched as an answer gets.

It is also a coin toss. Choosing between the top two is 50 against 49, and my page announced the winner in exactly the voice it uses at 0.98.

Confidence cannot catch this, because bunched into two neighbours is still bunched. What I actually wanted was the gap between first and second place, which is a different number entirely: 98 against 1 is a winner, 50 against 49 is a tie with a rounding error. Confidence says the same thing about both — and the gap is the one a reader cares about.

5. Audit your questions: one of mine fired on 21 of 24 inputs

A question set grows by accretion. Each addition looks free, and none of them announce that they have stopped saying anything.

So run them over a corpus and look at the spread. Mine had a Noul asking is more than one feeling present. It returned ≥0.6 on twenty-one of twenty-four texts. That is not a signal about the input, it is a property of writing — a question paying tokens to tell you something you already knew.

Two more went for a different reason. They were informative, but nothing downstream consumed them beyond appending a line to the output. A question whose entire effect is an occasional footnote costs a reader more attention than it returns.

The check is cheap: for every question, the min, max and spread of its answer across a representative corpus. Anything that barely moves is either a gate or a mistake.

6. Two small things that cost real time

A rubric level must not contain a word from another axis. My control rubric had a level reading "Overwhelmed: struggling to keep any grip on it". That primes the model with a feeling while asking it about agency, and it labels the output with a word that is not a position on a control scale at all. Every level of an axis has to be a point on that axis.

Input is dominated by your rubrics, not by your input. Criteria are sent on every call, so a one-line text costs almost exactly what a paragraph does. At this size requests are the scarce resource and tokens are not — which is the whole argument for batching every independent question into one call.

I stopped one clause too early, and wrote that a dependent second call is therefore a real cost rather than a rounding error. Read it again: if requests are scarce and tokens are not, the conclusion goes the other way.

7. I reasoned where I should have measured

Someone read section 2, noticed I had talked myself out of the fan-out, and told me to go and time both. It took forty minutes and a dollar's worth of nothing.

Run the same 24 texts through both designs, twice each, back to back on every text so neither gets the warmer connection:

One request was faster on 46 of 46 pairs. It named the same family 48 out of 48 times, and the same shade 45 of 48 — where all three misses were texts under 0.41 confidence that the page already reports as sitting between two words, and where the sequential design disagreed with itself between rounds on one of them. The eleven extra questions barely moved the six that were already there: mean drift of 0.022 on a 0–4 axis score, and identical intent 48 times out of 48.

So the fan-out is 1.9× faster for five hundredths of a cent a thousand readings, and it halves the request count against the limit that actually binds.

Two things I had, and did not put together:

Jev "ingests the state once and evaluates every question against it in parallel." That sentence is in TypeSafe's own model card, and I had quoted the half of it that suited me. Question count is nearly free; a round trip is not. Ten wasted questions cost less than one extra wait.

Jev charges for input only — output tokens are free. The fan-out triples the output, returning eleven probability distributions instead of one, and that is worth exactly nothing on the bill.

What I actually got wrong is narrower than "I didn't measure it", and more useful. The docs say a second request is warranted when the first answer determines the second question's options. It does — and I read that as settling the matter. But determined is not unknown: there were only ever eleven option sets, all of them written down in my own source file, so every one of them could be asked on spec. The rule is about what the options are. It says nothing about when you are allowed to ask.

The tell was in my own sentence. "A latency story I could explain" is not a latency number. Anywhere a design note says presumably, roughly, or I could explain, there is a measurement someone is about to make for you, and it is cheaper to make it yourself.

waif is live at /waif. Every number in this post is on the page, under How it works, next to the vocabulary it was scored against.

I spent a week building a Magic 8 Ball on it, which sounds like a toy and turned out to be a decent test rig: every answer is a judgment with no ground truth, which is where calibrated probabilities either earn their keep or embarrass you. It's live at /jevball.

I got the primitives wrong four times. Each mistake produced working code that looked right, so they're worth writing down.

The three primitives in one paragraph each

Choice takes a map of named options and returns a probability for each, plus the highest-scoring one. Routing a ticket to a department, classifying a document, picking a handler.

Score takes an ordered array of level descriptions — two to ten — and returns a position along them that can land between levels. Severity, frustration, skill level. Anything on a spectrum you can describe in words.

Noul takes a yes/no statement and returns one number: the probability it's true. No confidence field, because the probability is the answer.

All three take the same two inputs: state, the data being judged, and instructions, the question asked about it. That's the whole surface.

Mistake 1: reading the score as a winner

A Score returns score, a position on your levels. My ball rounded it to pick a bucket. Someone asked it "is black a color?" and got this:

bars   : 0=2%  1=6%  2=19%  3=16%  4=56%
score  : 3.18

Math.round(3.18) is 3 — a bucket holding 16% — while 56% of the mass sat on level 4 beside it.

The score is a probability-weighted average. That 19% on level 2 dragged the average down across a bucket boundary. An average of a skewed distribution points at a place the distribution isn't.

TypeSafe defines a Choice's answer as the option with the highest probability. Score has no equivalent field, and I quietly substituted rounding for one. The fix is to take the argmax of probabilities yourself:

const probs = Array.from({ length: 5 }, (_, i) => v.probabilities[String(i)] ?? 0);
const level = probs.indexOf(Math.max(...probs));

It agrees with rounding on every peaked distribution, which is why it survived my whole test suite. It differs exactly when the distribution is skewed, which is when it matters.

Mistake 2: reading low confidence as doubt about the answer

Every Choice and Score comes back with confidence, 0 to 1. I built a "reply hazy" branch on the assumption that a vague question would produce a low one.

Then I measured it:

The first two are as uncertain as a question gets, and confidence is 0.98. Nearly all the mass sat on the level that means "could go either way" — the distribution was a sharp spike on a level labelled uncertainty.

Confidence measures how peaked the distribution is. It answers "did the options separate cleanly", and it will happily report 0.98 for a confident coin-flip. My hazy branch would have fired on precisely the wrong questions.

Low confidence has three different causes, and they need different fixes: the question is genuinely contested, the options overlap each other, or state doesn't contain enough to decide. Only the first one is the model being honest.

I tried to work out the formula by fitting twelve real responses. max(p) gets closest at 0.04 mean error and matches several exactly, but not all — "will humans land on Mars before 2050?" reported 0.58 against a peak of 0.50, higher than the peak. The docs are upfront that it's a convenience statistic and hand you the full probabilities so you can compute your own. If a decision rides on it, do that.

Mistake 3: offering options that split their own vote

The classic 8 Ball has twenty answers. My first design asked Jev to pick one, as a Choice.

Ten of those twenty mean "yes". "It is certain", "Without a doubt", "Yes definitely" — the same claim in different words. Here's what happens, on one question with four different option sets:

The total yes-mass is identical: 70% in the second row, spread across three near-synonyms. Nothing changed about the question. The options ate each other, and confidence fell from 0.38 to 0.31 reporting a disagreement that existed only in my option list.

So the ball asks Jev for one of five ordered buckets, and the code picks which of that bucket's phrasings to show. One judgment with a right answer goes to the model; the theatre stays in software.

Mistake 4: forgetting that the options are the prompt

Your question IDs never reach the model. The instructions and every entry in criteria do — names and descriptions both. The option list isn't a filter you apply to a result. It's part of what you're asking.

Same question, same model, options changed:

"Depends" won — an answer that simply didn't exist in the first row. Black is a colour in everyday use and the absence of light in physics, so "depends" is arguably the best answer available. It was unreachable until I offered it.

The docs put it plainly: the model cannot choose an omitted value. Leave an option out and you haven't biased the result, you've made it impossible.

What this is actually for

Every judgment above cost about $0.000014. Jev charges $0.042 per million input tokens and nothing for output, against $1.00/$5.00 for the cheapest frontier model I'd otherwise reach for — roughly 27× on a like-for-like classification, before an LLM's format instructions and reasoning tokens widen it further.

That price only matters because of what you give up. Jev writes no prose, explains nothing, and can't answer anything whose answer space you can't enumerate. For summarising, drafting, coding or open-ended reasoning it's the wrong tool entirely.

What it's good at is the narrow decision a pipeline makes ten thousand times a day, where you need a number you can threshold on. The documented pattern is to put it in front of the expensive work: score everything, route the confident cases to deterministic code, escalate the rest to a model or a person. You pay fourteen dollars a million to decide, and frontier prices only on the slice that earned it.

One caveat worth keeping. Calibration is a property of groups of predictions — across many answers, the ones marked 0.8 should be right about 80% of the time. It guarantees nothing about any single answer, and it's measured on TypeSafe's data, not yours. Validate it in your own domain before trusting a threshold.

The ball is at /jevball. Ask it something you actually want to know and open the panel underneath — it shows the full distribution, the confidence, and which of the four questions produced the answer.

The handover usually goes well for about a week. The model returns valid JSON every time. Every field is filled. Nothing throws an error.

Then someone notices the follow-up went to the wrong person.

Wrong output looks exactly like right output

An extraction model does not fail the way code fails. There is no stack trace and no empty response — you get a complete, confident, well-formed row, with no signal about which of its eight fields you can trust.

So I wrote down the correct answer for eight real inbound enquiries and measured it properly. Here is what I expected to break.

The forwarded thread

From: Dan <dan@brightline.io>
Subject: Fwd: quick question

---------- Forwarded message ----------
From: Marcus Ellery <m.ellery@brightline.io>

Dan - can you ask them whether they do anodised finish? We'd need
about 250 brackets, the aluminium ones we discussed. Budget is
around 4k GBP, maybe a bit more if the finish is good.

The lead is Marcus. The From: header says Dan. Read the header first and you get the product right, the quantity right, the budget right — and address your quote to the wrong human.

The email that is not a lead

A tender matching your registered categories has been published.
Reference: TN-2026-0884
Closing: 2026-09-12
Do not reply to this address.

No company, no contact, no quantity, no budget. The correct extraction is mostly null — but models are obliging, and asking for eight fields tends to get you eight fields, invented where the text runs out.

Dates that only exist relative to the email

"required before 25th aug" · "by the first week of September" · "before the end of October"

None of those are dates until they are resolved against the email's own Date: header. Resolve them against today instead and you are quietly a year out every January.

The enquiry in Italian

Vorrei un preventivo per 600 maniglie in ottone.
Consegna a Bologna entro fine settembre. Budget indicativo 3.500 euro.

600 brass handles, Bologna, end of September, €3,500. Easy to get almost right: 3.500 is three thousand five hundred, not three and a half.

The polite no

"We are only collecting information for a project starting next year. I cannot give quantities yet and there is no budget approved."

Every field sales cares about is genuinely absent. An extractor that fills them in has not saved anyone time; it has created work.

What actually happened

I ran all eight through Claude Haiku with a plain extraction prompt. Eight emails, eight fields each, sixty-four fields total. It cost 0.6 cents.

Overall: 61/64 fields correct (95.3%)
Successful: 8  Failed: 0

Failure modes:
  wrong_value : 3

Every one of the five cases above came back correct. It took Marcus over Dan on the forwarded thread. It returned nulls for the tender notice instead of inventing a company. It resolved "before 25th aug" against the email's own date. It read 3.500 euro as 3500 and translated maniglie in ottone into brass handles. It left the polite no almost entirely empty, which was the right answer.

All three misses were the same field — product — and they looked like this:

The model is not wrong in any of those. I am — or rather, exact string match is the wrong check for a free-text field, and a semicolon is not a fact about the world.

That is what the taxonomy is for

If all I had was 95.3%, I would be shopping for a better model right now. I would be wrong, and I would have paid for it.

The number that mattered was not the percentage. It was that all three failures carried the same tag on the same field. Four tags, four different bugs:

Three hallucination tags scattered across six fields is a prompt problem. Three wrong_value tags stacked on one free-text field is a measurement problem. Averaged into a single percentage, those two situations are indistinguishable — and they have nothing in common.

The fix here is not a better model. It is deciding what "correct" means for product: normalise before comparing, or score that field with a judge instead of ==, or stop scoring it at all and accept that a human reads it anyway.

Doing it yourself

Write down the correct answer for a few dozen real emails, once. That labelled set is the asset; everything else is plumbing.

Then run your extractor against it. I maintain a small open-source harness for exactly this — doceval — which takes any extraction function and any schema:

pip install doceval

doceval run \
  --docs    ./emails \
  --labels  ./labels \
  --extractor my_module:extract

Your extractor is any Python function that takes bytes and returns a dict:

def extract(doc_bytes: bytes, filepath: str) -> dict:
    email_text = doc_bytes.decode("utf-8", errors="replace")
    # call whatever model you like
    return {"company": "Brightline Systems Ltd", "contact_name": "Marcus Ellery", ...}

A label is the correct answer, null included, because the nulls are half the point:

{
  "company": null,
  "contact_name": null,
  "product": "galvanised steel conduit",
  "quantity": null,
  "target_date": "2026-09-12",
  "budget": null
}

The eight emails above ship with it as a runnable example, labels included, if you want something to point it at before labelling your own.

One honest caveat about the numbers here: eight emails, one model, and I wrote both the labels and the prompt. That is a demonstration of the method, not a benchmark — and if I had run it on eighty emails from a real inbox, the interesting failures would almost certainly be somewhere else. Which is the argument for running it on yours.

The part worth keeping

The labelled set outlives every other decision. Models change, prompts change, providers change — and each time, the only thing that tells you whether the change helped is a set of emails where you already wrote down the right answer.

An afternoon of labelling buys that permanently. Skipping it means finding out from a customer.

I built the layer that catches this before it happens: labeled eval sets, accuracy reports, confidence scoring, audit logs. Along the way I also benchmarked the two obvious model choices against the same data, so you don't have to guess which one your pipeline needs. Here's what 105 labeled documents taught me.

Why extraction pipelines quietly fail

Extraction looks easy because it looks right. The model fills in all the fields, the JSON is valid, nothing throws an error. The failure modes are invisible:

• Silent wrong answers. The model reads "132,30" (Swedish decimal notation) and returns 13230. No error. The invoice is just wrong.
• Hallucinated fields. A blurry scanned receipt gets a subtotal that doesn't exist anywhere on the document.
• Inconsistent formatting. Currency as $ on some invoices, USD on others, nothing on scanned ones — your downstream code breaks on whichever one you didn't test.

Without a labeled eval set and systematic accuracy measurement, you find these problems in production. That's too late.

The eval set

The foundation is 105 labeled documents: 100 digital invoices and 5 scanned thermal receipts. Each document has a ground truth JSON with the correct values for every field — vendor, document number, date, currency, line items, totals.

A label looks like this:

{
  "document_id": "invoice_Shahid_Shariari_30140",
  "filename": "invoice_Shahid Shariari_30140.pdf",
  "processing_mode_expected": "embedded_text",
  "difficulty": "easy",
  "ground_truth": {
    "vendor": "SuperStore",
    "document_number": "30140",
    "date": "2012-11-15",
    "currency": "USD",
    "total": "748.36",
    "shipping": "73.00",
    "line_items": [
      { "description": "Safco 3-Shelf Cabinet, Traditional", "quantity": "2", "unit_price": "337.68", "amount": "675.36" }
    ]
  },
  "eval_notes": { "line_items_evaluated": true }
}

The eval runner loads each label, runs the extractor, and compares every field in the ground truth against what the model returned — handling numeric equivalence ("$132.30" "132.30"), European decimal formats ("132,30" "132.30"), and date normalization ("Dec 27 2012" == "2012-12-27").

Two configs, same eval set

Two questions come up the moment you build an LLM extraction pipeline: which model, and at what cost? "It depends" isn't an answer, so I ran both configurations against the same 105 documents and measured accuracy, failure modes, and cost.

Cost ratio: 12.5×.

A 2-point accuracy gap for 12.5× the cost. Worth knowing where that 2% actually lives before deciding it's worth paying for.

On digital invoices: 97.9% vs 99.2% — a 1.3-point gap. On 1,000 invoices, that's the difference between ~21 errors and ~8.

On scanned documents: 69.8% vs 83.0% — a 13-point gap. Scanned receipts are where Opus earns its price: OCR quality, resolution, and degraded thermal printing all compound, and the extra resolution genuinely helps here.

If your document mix is mostly digital PDFs, Haiku+Sonnet is nearly indistinguishable from Opus. If you're processing scanned documents at volume, the gap is real.

Field by field

Not all fields are equal:

line_items carries almost the entire gap — reading and counting structured table rows is hard on lower-resolution scans, and it closes considerably on digital invoices. document_number is identical across both configs for a simpler reason: scanned receipts carry several reference numbers (order number, transaction ID, loyalty card), and both models struggle equally to pick the right one.

Resolution matters more than you'd expect

Scaling from 7500px to 1500px per page cuts API cost roughly 25× with barely any accuracy loss on digital documents — invoice text is legible at 1500px, and the model doesn't gain much from pixels it doesn't need. Higher resolution earns its keep only on scans, up to the point where the text becomes readable.

Which one to use

Haiku+Sonnet at 1500px, if: your documents are mostly digital PDFs, you're processing at volume, and you can route low-confidence extractions to a human.

Opus at 7500px, if: you're processing scanned documents at meaningful volume, every field has to be right, or you're pulling line items from messy layouts.

The math: Haiku+Sonnet costs ~$0.00076/doc, Opus ~$0.0095/doc. At 10,000 documents a month, that's $7.60 vs $95.

The accuracy floor, and how it closed

Before any tuning, the first run came in at 79.3%. That sounds bad. Turns out, it was informative.

Categorizing every mismatch turned up four root causes:

Fixing all four got both configs to the headline numbers above. The lesson: the first accuracy number tells you where the problems are, not where the ceiling is.

Confidence scoring: the model knows what it doesn't know

Every extraction includes a self-reported confidence level — high, medium, or low — plus a list of uncertain fields and a short note explaining why.

It predicts actual accuracy well:

The one low-confidence document — a dense 80-item Dollarstore thermal receipt — also had the worst accuracy. The model flagged it correctly, without being told what "correct" was.

The practical use: route by confidence, not by document type. Send high-confidence extractions straight to processing. Route the 4% that aren't to a human. You're not reviewing 105 documents — you're reviewing 4, and the effective accuracy on the auto-processed 96% comes out to 97.7%, at Haiku+Sonnet pricing.

The audit log

Every extraction writes a structured log entry before anything downstream sees the data:

• What went in: filename, processing mode (digital vs scanned), input size
• What came out: all extracted fields
• How confident: overall level, uncertain fields, the model's own explanation
• Operational metadata: which model, how long the call took, UTC timestamp

PII redaction runs on every string value before the log is written — card numbers, emails, phone numbers, SSNs. The log is safe to store and share even when the source documents aren't.

A real entry looks like this:

{
  "id": "b1d01383-42fd-4eac-acc0-f333d31a9409",
  "timestamp": "2026-06-15T13:43:51Z",
  "filename": "invoice_Shahid Shariari_30140.pdf",
  "mode": "embedded_text",
  "model": "claude-haiku-4-5",
  "duration_s": 2.13,
  "confidence": {
    "overall": "high",
    "uncertain_fields": [],
    "notes": "All key fields are clearly visible and unambiguous."
  },
  "input_summary": { "char_count": 386 },
  "extraction": {
    "vendor": "SuperStore",
    "document_number": "30140",
    "date": "2012-11-15",
    "currency": "USD",
    "total": "748.36",
    "shipping": "73.00"
  },
  "pii_redacted": false
}

This is what handing a pipeline to an auditor looks like in practice — not a demo showing the happy path, but a log of exactly what happened, for every document, with the model's own confidence attached.

Known limitations (the honest ones)

• Scanned documents: 69.8% vs 97.9% for digital. OCR quality and resolution drive the gap — dense thermal receipts in poor light are genuinely hard.
• Non-English locales: the eval set includes Swedish receipts. Decimal separators, currency conventions, and vendor formats vary; prompt tuning per locale helps.
• Template concentration: 100 of the 105 digital invoices share one template. Real invoice variance will affect accuracy — these numbers are a ceiling on a homogeneous dataset, not a floor on a diverse one.

Methodology

• Dataset: 105 labeled documents — 100 digital invoices (single vendor template) + 5 scanned thermal receipts (Swedish, mixed vendors)
• Fields evaluated: vendor, document_number, date, currency, total, subtotal, shipping, discount, tax, payment_method, line_items
• Comparison: numeric normalization across currency symbols and locale formats; date normalization across ISO and abbreviated formats
• Eval tool: doceval — open-source, schema-agnostic field-level accuracy harness

If your extraction pipeline needs the same treatment — eval set, benchmark, audit log, the works — reach out.