Why we moved questionnaire mapping to a decision model

01

What mapping is

Tyrvar keeps a customer's security posture as answers to a schema of roughly 160 canonical questions. “Describe your data encryption at rest strategy” is one key. “Do you require multi-factor authentication for access to production systems?” is another. A buyer's questionnaire asks the same things in its own dialect, so the job of the mapper is to take each incoming question and pick the key whose stored answer would answer it.

Until now that has been an embedding-similarity search: embed the incoming question, embed each canonical question, rank by cosine, hand the top ten to a reviewer with the best match pre-selected. We had tuned it twice this summer, once by enriching the text we embed for each key and once by blending in a lexical score. Both helped at the margins. Neither changed the shape of the problem, which is that similarity search rewards questions that are phrased alike, and buyers phrase everything alike. “Do you encrypt data stored on managed endpoints?” and “Describe your data encryption at rest strategy” sit close together in embedding space. They are different facts.

02

A model that only picks

In September, TypeSafe AI released Jev, a model that does not generate text. You give it a piece of state and a typed question, and it returns a decision with a probability for every option and a calibrated confidence, in one forward pass. One of its question types is a choice from a fixed list of up to 255 options.

That is exactly the shape of mapping. The state is the buyer's question. The options are our canonical keys, each rendered as its label and its canonical question, plus “none of these.” The answer is a distribution over keys. No parsing, no prompt asking a language model to please respond in JSON, and no way for it to invent a key that does not exist. We ran the evaluation through OpenRouter's decisions endpoint, which speaks the same wire format as TypeSafe's own API.

03

The experiment

We already had a recall harness: a fixture of 22 hand-written questions, each with the key a person says it should land on, scored against the live schema for every mapping variant we have ever shipped. The first case in the fixture is a real failure from production, where “How do you encrypt customer data?” confidently mapped to endpoint encryption. We added two Jev variants. One hands Jev the entire schema as the option list. The other keeps our embedding shortlist and asks Jev to re-rank the top ten.

The four cases Jev did not put first are all ones where its pick is arguably better than the fixture's. The breach-notification question asks about an SLA and Jev chose the SLA key over the procedures key. The change-management question asks whether two people review production changes and Jev chose the key that says exactly that over the generic SDLC key. Two identical runs produced identical choices on all 44 calls. The whole fixture cost under a cent.

Twenty-two cases is thin, so the second pass used the largest labelled set we had: 325 questions from our own dogfood questionnaires where the stored mapping was marked confirmed, plus every correction a person had ever made in the product. On that set the headline flipped. Embedding similarity got 243 of 325 right. Jev got 184.

04

Our ground truth wasn't

Jev is calibrated, which means its confidence is supposed to track its accuracy. On this set it did: 19 percent right in the lowest confidence band, rising to 76 percent in the highest. So the interesting rows were the ones where Jev contradicted a confirmed label at 0.95 or above. There were 42 of them. I read every one, and in nearly every case the stored label was wrong and Jev's pick was the key any security engineer would choose. “Is cloud data periodically backed up?” had been confirmed to the backup-testing key. A question about VPN access to production had been confirmed to a data-classification key.

That sent me to the provenance of the labels. Of the 365 mappings marked confirmed in production, 261 had no confirmation timestamp and no confirming user. Nobody had confirmed them. When we introduced the “mapping confirmed” flag in August, the migration seeded it from a different fact: whether the answer had been approved. Approving a drafted answer was silently treated as approving the mapping beneath it. The labels on those 261 rows were the embedding mapper's own output from August, wearing a human's signature.

Split the 325 cases by how the label was actually produced and the picture is consistent again.

On the only rows where a person made a mapping judgment against the machine, Jev wins by a wide margin. On the rows where a person clicked accept on the machine's suggestion, similarity wins, which is what you would expect from a label that is the suggestion. On the 236 seeded rows, similarity agreeing with itself is not evidence of anything. I sampled 14 of the high-confidence disagreements there and read 12 of them for Jev, including the patching-timeframe question at the top of this note, which had been mapped to application remediation instead of patch management.

This is the finding I care most about, and it has nothing to do with which model is better. Some of the answers we drafted for our own questionnaires this summer were sourced from a neighbouring fact, and the review workflow did not catch it because the reviewer was reading a fluent answer to a slightly different question. The mapping step needs its own confidence and its own human decision. Approval of the text is not approval of the source.

05

What did not work

Three things, in the interest of not writing a launch post. First, giving Jev the whole schema has weaker recall at depth than our fusion shortlist: when Jev is wrong it puts zero mass on most keys, so the right one can end up unranked instead of at position six. Re-ranking the existing top ten keeps the recall floor we already have. Second, the “none of these” option was a clean signal on the hand-written fixture and a noisy one on real questionnaire text, catching 7 of 27 shortlist misses with 20 false alarms. We are not using it as a routing signal. Third, calibration in the middle bands is rough on real text. The top band is trustworthy. The 0.4 to 0.6 band means “a person should look.”

06

What we changed

The mapper is now a two-stage pipeline: the embedding and lexical search we already ran produces the ten candidates, and Jev re-ranks them. Its confidence is the number the review workflow had been missing, so an admin can say “auto-accept above 0.9, queue the rest” and mean it. Cost is not a factor in either direction. A full 260-question questionnaire re-ranks for about a cent, against several dollars of answer drafting downstream. It is live in production as this is published, alongside the privacy policy update that names the new provider.

Two guardrails sit in front of it. Buyer questions are not secret in the way answers are, but they are customer data, so TypeSafe joins our subprocessor list and our no-training commitment applies to that traffic like any other. And the human stays where they were: nothing exports without a person approving the answer, and now, separately, its source.

We are also going back through every mapping that the August migration marked confirmed and clearing the flag. Those questionnaires were our own, so no customer answer is affected, but a label that no one made should not be allowed to look like one.

07

The first live questionnaire

The day after this note was drafted, the re-rank went live on our testing environment and a 118-question vendor security assessment went through it. The model decided 99 of the rows it saw: 87 with confidence averaging 0.94, and twelve flagged for review at an average of 0.42. Reading the confident picks, they land on the keys the similarity mapper kept missing: a bill-of-materials question on the open-source key, API key storage on secret storage, remote production access on the remote-access key. The flagged twelve are the right rows to flag. Some are correct, and in the others the right key is the runner-up, one click away.

It also contradicted one of my conclusions above. On fourteen rows the model answered “none of these,” and I had decided not to trust that signal. Looking at the fourteen, they are mostly document requests and prompts no trust profile can answer: “please attach,” “provide a copy of the most recent report,” a question about Privacy Shield. The similarity fallback then mapped them anyway, to keys that were wrong at confidences that looked fine. On real questionnaire text, a confident “none” is a better abstention signal than the fixture suggested. We are measuring it on a few more questionnaires before changing the rule.

08

The lesson

When a calibrated model disagrees with your ground truth at 0.95, check the ground truth first. We went looking for a better mapper and found a provenance bug in our own labels, which is a small version of the argument the rest of this site makes: a fact without a record of who established it and how is not really a fact. That applies to a vendor's claim that they encrypt at rest, and it applies just as much to a row in our own database that says a question was mapped correctly.

The obvious next question is whether we need a vendor for this at all. The same week Jev launched, Cua open-sourced a 700k-parameter model that does the same pick-from-a-list job for web forms and beat Jev on their own benchmark. Our corrections history plus paraphrases of the canonical questions is a training set, and a scorer that small would run inside our existing worker for free. That experiment is queued, and this evaluation is the number it has to beat.

The evaluation harness, the 22-case fixture, and the raw result files are available on request. If you run a similar mapping step and want to compare notes, or you have a questionnaire you would like to see go through this pipeline, the address is below.