Skip to content
Stephen Kovach
Stephen Kovach

Explore Stephen Kovach

Evaluate AI on the Work It Will Actually Do

A polished demonstration does not establish operational reliability. Useful AI evaluation starts with a defined task, representative examples, and clear failure criteria.

An AI system can give an impressive answer to a carefully chosen question and still be unsuitable for a routine business process. The demonstration shows a possibility. An operational evaluation asks whether that possibility holds across the work the system will actually receive.

The first question should therefore be specific: what task is being delegated, what counts as success, and what happens when the answer is wrong?

Define the output before judging it

“Summarize documents accurately” leaves too much open. A clearer task might require identifying a document’s date, parties, key obligations, and any missing information, with each extracted statement linked to its source passage.

That definition makes several failure types visible. The system could omit an important term, invent one, confuse two documents, or provide a correct statement with the wrong reference. A single overall score can conceal those differences.

NIST’s AI risk guidance connects measurement to the deployment context and calls for appropriate performance criteria and documentation of differences between testing and real-world use. The relevant test is the job being performed, including its consequences. [1]

Build a representative test set

For the hypothetical document task, include short and long files, clear and poor scans, conflicting dates, repeated names, and documents that genuinely lack the requested answer. The system should be allowed to say that information is missing rather than being rewarded for always producing a confident response.

Keep some examples separate from the material used to adjust the system. Google’s evaluation guidance explains why training, validation, and test data serve different purposes, and why repeatedly tuning against the same examples weakens the independence of the final test. [2]

This principle also applies when improving instructions rather than training a model. If every revision is shaped around the same twenty documents, success on those documents may be a poor indication of performance on the next twenty.

Measure the cost of mistakes

Suppose a system handles 100 illustrative cases correctly in 92 instances. That percentage is incomplete. The eight errors might be minor formatting defects, or they might all involve a crucial obligation. The appropriate decision depends on the failure pattern.

Record correctness by field, the time needed for human review, and the frequency of unsupported claims. Compare the whole process with the existing alternative. A model that drafts quickly but requires extensive checking may deliver less improvement than its response speed suggests.

Also test the boundaries around the model: document access, data extraction, output storage, and the permissions associated with any actions. A correct answer written to the wrong record is still a failed workflow.

Make monitoring part of the process

Google’s production guidance emphasizes monitoring inputs, features, and real-world performance, including differences between training and production conditions. A successful initial evaluation does not settle performance indefinitely. [3]

For the example workflow, a practical review could sample completed outputs, record recurring errors, and trigger reassessment when document types or system components change. Keep a clear route for human escalation and a way to return to the prior process if quality deteriorates.

The evaluation should finish with a bounded conclusion: suitable for this task, with these controls, on the evidence collected. That is more useful than declaring a system generally accurate or generally unreliable.

AI becomes easier to judge when the discussion moves from impressive answers to dependable work. The measure of progress is the quality of the completed process, including the effort required to verify it.

Sources

  1. NIST AI Risk Management Framework Playbook: Measure
  2. Google Machine Learning: Dividing datasets for evaluation
  3. Google Machine Learning: Monitoring production pipelines

Research, reflection, and a wider perspective.

Continue exploring

LETTERS FROM STEPHEN

A note from Stephen.

New essays, research, and occasional notes. Choose the subjects that interest you.

Choose your interests
Send me writing about

Leave all unchecked to receive all topics.

Confirm your address by email. Privacy policy

inj[jm