Choosing a task that can be evaluated
| Dimension | What it needs | Why it usually fails |
|---|---|---|
| Drafting from approved templates | A small set of approved sources and a review step that already exists. | Sources change without versioning, so drafts cite retired material. |
| Summarising a defined corpus | A bounded collection with ownership, freshness and a length standard. | The corpus is wider than anyone owns, so quality cannot be attributed. |
| Extracting fields from documents | Labelled examples, a target schema and a review path for low confidence. | No ground truth, so accuracy is judged by anecdote rather than measurement. |
| Answering questions over internal knowledge | Permission-aware retrieval plus citations, refusals and an evaluation set. | Access enforced in the prompt, or retrieval evaluated separately from answers. |
| Routing and classifying requests | A stable label set, balanced examples and a cost of misrouting. | Labels change without a version, so performance silently degrades. |
Choose the task before the tooling
Write down the job, the person who does it today, the input, the acceptable output and the cost of being wrong. A drafting assistant, a summariser, an extraction pipeline and a question-answering interface are four different systems with different data, evaluation and risk. A programme that starts from a model name has to work backwards into a task, and usually invents one.
- Task, consumer and failure cost stated in one paragraph
- A non-generative alternative compared first
- Success defined as a measurable improvement in the task
Workflow fit decides adoption
A system that produces a draft inside an existing tool gets used; one that requires a new destination gets abandoned. Decide where the output lands, who reviews it, what happens when the reviewer disagrees, and how quickly the loop closes. Latency and format matter more than benchmark scores at this stage.
- Output delivered where the work already happens
- Review step designed, not assumed
- Escalation path when confidence is low
Data readiness is the limiting factor
Generation is only as good as the material behind it: which sources are approved for the task, who may see them, how fresh they must be and how deletion works. Permit-list the corpus instead of assuming everything internal is fair game, and treat any retrieved text as untrusted input when it reaches a model with tools.
- Approved source list with an owner per source
- Permission and retention context carried into the pipeline
- Deletion and revocation tested, not assumed
Evaluate on the task, not on a benchmark
Build a small evaluation set from real tasks, including the cases that must be refused, and score the whole system: input handling, output quality, citation support and the human review burden. Re-run it when prompts, models, retrieval configuration or source data change, and treat a quality regression like any other release defect.
A practical first evaluation set is 50 to 200 real examples with expected outputs or grading notes, including the awkward ones: missing information, conflicting sources and requests the system should refuse. Score them against a written rubric, and where an automated grader such as a second model is used, check its scores against human judgement on a sample.
Cost, latency and the human in the loop
Generation cost tracks tokens, retrieval depth and retries; latency tracks the same things plus review. Set an envelope per task, measure cost per completed task rather than per token, and keep a person accountable for consequential outcomes. A system that is cheaper but wrong is not cheaper once the review burden is counted.
- Cost and latency envelope agreed per task
- Human approval required for consequential actions
- Review effort measured, not assumed to fall
Illustrative example · hypothetical organisation
Worked example: summarising support escalations
Take a hypothetical support team that wants a summary of each escalated ticket for the engineer who picks it up.
The task is narrow: from the ticket thread, produce the customer's problem, the steps already tried, the product version and the open question, in that order. The input is a ticket the engineer may already read, so no new permission boundary is created.
Evaluation uses past escalations with summaries written by senior engineers. The rubric checks that nothing is invented, that the version is correct and that the open question matches the thread. The summary appears inside the ticketing tool with a link to the full thread, and engineers can flag a bad summary in one click, which feeds the evaluation set.
Failure modes to design against
The recurring failures are well understood: confident output that is unsupported, drift when a source or model changes, sensitive content leaking into a prompt or a log, and silent cost growth. Each has a design response — citations with refusal, versioned components with re-evaluation, data minimisation with access control, and per-task budgeting.
Questions and trade-offs
Question
Where should we start with generative AI?
With a task that has an owner, a cadence and a measurable failure cost — usually drafting, summarisation or extraction inside a workflow that already has a review step. Answer-generation systems are harder because they need permission-aware retrieval and an evaluation set first. Start where the review burden falls and the benefit is visible.
Question
How do we keep company data out of a model we do not control?
By deciding the data-flow boundary before choosing a model: what may leave the environment, what must stay, what retention applies and which sub-processors are involved. Data minimisation, a permitted-source list and access control at retrieval are the practical controls; a policy statement alone does not enforce them.
Question
Should we build on an API or host an open model?
Start from the data boundary and the task. A managed API is the fastest way to test a task and suits most first use cases when the provider's data terms are acceptable. Hosting an open-weight model gives more control over data location and cost at volume, and brings serving, scaling and update work your team has to run.
