JEV / DECISION ENGINEERING
Jev confidence: thresholds, review, and failures
Use probabilities to route work without mistaking them for a guarantee of correctness.
Start with the cost of error
Separate safe automation, cases requiring more information, and decisions that need a person. A threshold is not a universal model constant; tune it on labeled examples from your task.
Build edge cases
Include overlapping categories, missing context, negation, multilingual inputs, and unseen classes. Record ground truth, model output, and final action for each example.
Measure the workflow
Track costly misroutes, review rate, end-to-end time, and total cost as well as accuracy. Recheck thresholds after a model update.
Probability, confidence, and accuracy differ
Choice and Score probabilities describe one answer's distribution. Confidence is a single statistic computed from that distribution by TypeSafe. Accuracy requires many labeled cases. A 0.9 output does not prove nine of ten future decisions will be correct. Noul has a noul field, not confidence.
Different actions need different gates
A reversible ticket label and a financial refund have different consequences. A label can use a threshold evaluated on your cases; a refund also needs identity, order, policy, and authorization checks. The same confidence does not imply the same risk.
Tune from errors, not intuition
Group cases by language, business category, and cost of error. Track coverage, review share, and costly mistakes at each threshold; verify on held-out cases. Revisit thresholds after model, option, or traffic changes.
An inspectable action rule
Route low-confidence Choice results to a person. For cases above the tuned threshold, allow only read-only or reversible actions first. Noul does not have a confidence field; define a separate rule based on the noul probability and the statement's direction.
Progress is stored only in this browser