Applied AI Engineering

You’ll learn to: Separate extracted facts, policy decisions and external actions.

Start with an observable job: prepare an invoice review, not “build an intelligent agent.” Define the input documents, the output schema, the reviewer and the cost of a wrong result. Our fictional Northstar workshop needs a review packet: an invoice total, a purchase-order limit, applicable policy, supporting evidence and a status. It never needs this prototype to send money.

Keep three boundaries visible. Extraction proposes facts. Deterministic code compares verified facts under a versioned rule. A human owns any external action. The model can explain a result, but its explanation cannot change the rule result. Write down missing-data and disagreement states before selecting a model. A system that can say what is missing is easier to debug than one forced to answer every request.

Try it: A prototype reads an invoice and immediately calls a payment tool. Describe the boundary you would change.

Check your reasoning

Separates extraction, deterministic validation and human-authorized action; does not merely add a stronger prompt.

You’ll learn to: Make every important field traceable to an immutable source.

A citation is useful only if a reviewer can recover the exact evidence. Store document ID, content hash, version, tenant, locator and the verbatim supporting span with each extracted field. Keep normalized value separate from raw text. A page number is useful for a PDF; a row and column identify a CSV cell. An HTML section ID can identify a policy clause.

A content hash identifies bytes, not truth or authority. Two conflicting documents can both have valid hashes. Decide which source has authority for each field: invoices assert billed amounts; purchase orders define authorized limits; policy documents define rules. Never silently merge disagreement into a single “best” value. A provenance chain should survive conversion, chunking, retrieval and explanation.

Try it: Why is a document URL alone insufficient evidence for an extracted invoice total?

Check your reasoning

Identifies mutable content and lack of a precise supporting location; proposes version/hash plus locator/span.

You’ll learn to: Establish a simple comparison before adding AI.

Build the smallest deterministic pipeline that handles one clean case. Parse a known invoice template, find its exact purchase-order ID, compare integer amounts and emit a review packet. This is a reference point, not a production parser. Measure its weaknesses explicitly: extraction, document selection, policy selection, rule execution, citation support and presentation.

A single end-to-end score hides why a change helped. If a model extracts the right amount from the wrong tenant’s file, the extraction appears correct while the system is unacceptable. Keep a small manually labelled development set with both easy and awkward cases. Save predictions and errors before editing prompts. Change one component at a time and compare on the same frozen inputs.

Try it: Your system cites the correct invoice but reads 1,200 as 12. Which stage should you inspect first?

Check your reasoning

Classifies an extraction/normalization error and checks raw text, locale and parser output before retrieval or prompting.

You’ll learn to: Treat layout, tables and reading order as part of the data.

Document conversion is not just concatenating visible words. A PDF may encode text in a different order from how people read it; a scan may require OCR. Tables carry relationships through row labels and column headings. Repeated headers, footers and signatures can overwhelm retrieval if kept without structure.

Use a common document representation with typed blocks, hierarchy and provenance. Tools such as Docling provide structured conversion options; inspect outputs rather than assuming conversion succeeded. Build a small golden set of pages with manually checked totals and table cells. If a parser cannot recover a field reliably, preserve the source and mark the extraction unavailable. Optional OCR or layout models are extensions to the project, not required services.

Try it: Why can a plain-text extraction that contains every number still be wrong for a table?

Check your reasoning

Explains loss of row/column relationships and recommends preserving headers and locators.

You’ll learn to: Represent missing, ambiguous and conflicting fields explicitly.

Choose types before asking for structured output. Amounts use integer minor units for currencies with a known minor-unit convention; do not assume every currency has two decimal places. Dates need an explicit format and timezone when time matters. Use a decimal parser before converting money, not binary floating-point rounding.

Distinguish present, missing, unreadable and conflicting. A missing total is not zero. An ambiguous date such as 03/04/2026 is not safe to normalize without locale evidence. Schema validation catches wrong types and absent required keys, but it cannot prove that a plausible value was present in the source. Follow structural validation with semantic checks and evidence checks. Retain the original span so a reviewer can resolve ambiguity.

Try it: A JSON response contains amount_minor:0 but the source total is unreadable. What should the pipeline do?

Check your reasoning

Rejects the unsupported zero, preserves uncertainty and routes to needs_information rather than a successful rule comparison.

You’ll learn to: Keep answer-bearing relationships together within a token budget.

A chunk should contain enough context to support a claim while remaining small enough to retrieve precisely. Start with document sections and table rows. Carry parent headings, document identity and policy version into chunk metadata. If a table spans chunks, repeat its headings. Fixed overlap can protect a split sentence, but it also creates duplicates that may crowd the result list.

Test chunk choices against questions, not aesthetics. Ask whether the evidence needed for each labelled question appears together in at least one eligible chunk. Keep a mapping back to raw source locations. Retrieval chunks are representations, not new authorities. Rebuilding chunks must not sever the link to the original document or let a stale version masquerade as current.

Try it: What should a chunk contain if it includes a spending threshold and the next sentence specifies an exception?

Check your reasoning

Keeps the threshold and qualifying exception together or supplies linked context; retains source/version metadata.

You’ll learn to: Use the query’s structure before reaching for embeddings.

Many document tasks have exact handles: invoice ID, purchase-order number, product code or policy section. Resolve these deterministically where possible. Token-based search is a useful baseline for names and rare identifiers. Normalize carefully: stripping leading zeros from an identifier can change its meaning.

For natural-language requests, lexical ranking can miss paraphrases but offers a transparent diagnostic starting point. Keep candidate selection separate from answer generation. Record the query, eligible corpus, returned IDs and ranks. A retrieval miss should not silently become a model-created fact. In the practice pack, exact PO lookup is deliberately enough for several cases; use this baseline to justify any more complex retriever.

Try it: For “What limit applies to PO-007?”, what should the first retrieval operation be?

Check your reasoning

Exact authorized ID lookup with explicit missing/duplicate handling before approximate similarity.

You’ll learn to: Evaluate whether each retrieval stage earns its cost.

Embeddings map text into vectors so similarity can retrieve paraphrases. Similarity is not truth, authorization or proof that a passage answers the question. Hybrid retrieval combines complementary candidate lists, such as lexical and vector results. Reciprocal-rank fusion can combine ranks without assuming the original scores share a scale: sum 1/(k + rank) across lists, with a fixed chosen k.

A reranker reads query-candidate pairs to reorder a bounded shortlist. It cannot recover relevant evidence absent from that shortlist. Compare exact/lexical, semantic, fused and reranked variants on the same labelled queries. Track latency and cost alongside recall. Keep embeddings optional until the simple baseline’s errors demonstrate a need.

Try it: A reranker is excellent but relevant evidence never enters its candidate list. What should you improve first?

Check your reasoning

Identifies candidate recall or chunk/query coverage as the bottleneck, rather than changing the answer prompt.

You’ll learn to: Apply eligibility constraints before ranking.

Retrieval must search an eligible corpus. Tenant access is derived from the authenticated request, not from a model-generated tenant field. Apply authorization before retrieval results reach the model, and recheck it when serving citations. A search score cannot override access control.

Time is another constraint. Policy applicability depends on effective dates and the business event date, not merely the file with the newest modification timestamp. Keep immutable versions and explicit effective intervals. Define precedence when sources conflict; unresolved conflicts should produce an exception for review. Test stale policies, near-duplicate documents and matching IDs across tenants. Do not log another tenant’s document content while diagnosing a filtered query.

Try it: Two tenants have the same PO ID and one result ranks higher. Which result is eligible?

Check your reasoning

Only an authorized tenant’s document with the applicable version; ranking is considered after eligibility.

You’ll learn to: Check support rather than merely checking that links exist.

Build explanations from a decision record containing verified facts, rule IDs and evidence references. Each material factual claim should point to a supporting span. A real link can still be irrelevant or contradict the claim. Separate citation validity, citation coverage and actual support.

Have the renderer reject unknown evidence IDs. Check that numeric claims agree with normalized fields and that quoted spans exist in the referenced source version. Semantic support may require a reviewer or a calibrated evaluator, especially for qualifications and exceptions. Do not ask the language model for hidden reasoning. Ask for a short auditable explanation: the observed facts, the rule applied and the remaining uncertainty.

Try it: A response has three working citations but none contain the claimed total. Is it grounded?

Check your reasoning

No; distinguishes functioning references from evidence that supports the specific claim.

You’ll learn to: Use deterministic logic for explicit comparisons and precedence.

Represent policy as reviewed, versioned code or a small decision table. Evaluate required facts first; only then compare like-for-like quantities. Make boundary conditions explicit. For our synthetic policy, equality with the PO limit is allowed, a greater amount is an exception, and missing required evidence needs information.

Rule order matters. Do not produce a positive result from one passing check if another required check is unknown. Separate status from reason codes so multiple reasons can be recorded without inventing contradictory states. Test exactly below, equal to and above each threshold. Keep the policy version in the output so results remain interpretable after rules change.

Try it: Under a rule allowing amounts up to GBP 150.00, how should GBP 150.01 be handled?

Check your reasoning

Exception for over_limit using integer minor units; includes applicable rule and evidence rather than rounding down.

You’ll learn to: Choose useful fallback states instead of confident guesses.

A model’s fluent tone is not calibrated confidence. Distinguish retrieval failure, extraction ambiguity, source disagreement and an unsupported question. Each requires a different recovery action. “Needs information: no matching PO” is more useful than a generic low-confidence label.

Evaluate abstention jointly with coverage. If a system answers only one easy case correctly, it has low selective error but little usefulness. Define coverage as answered eligible cases divided by all evaluated cases, and selective error as wrong answered cases divided by answered cases. When nothing is answered, selective error is undefined, not zero. Tune thresholds on development data and report them on a separate evaluation split.

Try it: A system answers 20 of 50 cases and gets 2 wrong. What are its coverage and selective error?

Check your reasoning

Coverage 40%, selective error 10%; does not divide errors by all cases and call it selective error.

You’ll learn to: Use autonomy only where the task needs adaptive tool selection.

A fixed workflow is easier to test when the steps are known: ingest, validate, retrieve, compare, explain, review. An agent may help when the next information-gathering step genuinely depends on intermediate results. The tradeoff is a larger action space and more failure modes.

Give each tool a narrow typed contract and explicit allowed inputs. Prefer read-only lookup tools in the first version. The server enforces authorization, limits and output schemas regardless of what the model requests. Set maximum tool calls and a deadline; stop with a useful partial result rather than looping indefinitely. Compare the agent against the fixed workflow before accepting its extra cost.

Try it: Your task always uses the same five steps. What evidence would justify replacing it with an agent?

Check your reasoning

A measured benefit on cases requiring adaptive investigation that exceeds added cost and failure risk, not novelty alone.

You’ll learn to: Keep document content outside the system’s authority boundary.

Documents may contain irrelevant or instruction-like text. Treat all retrieved material as data to inspect, never as authority to change tools, policies or access rules. Prompt separation helps communicate intent but is not a complete security boundary.

Enforce tool permissions and tenant scope in application code. Limit allowed actions, validate tool arguments, and render text safely. A retrieved file must not be able to change which users can read records or where results are sent. Use a harmless fixture containing unrelated instruction-like prose and verify that policy and authorization remain unchanged. Track the result as a contract check, not proof that all prompt injection is solved.

Try it: Where should a system enforce which tenant’s documents a lookup tool may return?

Check your reasoning

In trusted server-side authorization derived from the session, independent of retrieved text and model arguments.

You’ll learn to: Apply minimization to prompts, storage, traces and citations.

Data exposure can happen outside the main model call: parser logs, exception messages, analytics events, embeddings, caches and citation URLs. Inventory these paths. Send only fields needed for the task and define retention and deletion behavior for derived artifacts as well as originals.

Use opaque document IDs rather than embedding sensitive text in URLs. Access checks apply when a citation is opened, not only when its answer was created. Trace metadata should support debugging without copying entire documents by default. For this course, all fixtures are synthetic and local; keep real invoices out of the exercise. Provider and deployment choices require a separate review before handling real data.

Try it: Why is deleting an original document insufficient if its chunks and embeddings remain searchable?

Check your reasoning

Explains derived-data retention and requires deletion/invalidation of indexes, caches and access paths too.

You’ll learn to: Separate development feedback from evidence of generalization.

Write case labels before optimizing the system. Include clean cases, missing fields, contradictory values, threshold boundaries, stale policy, wrong tenant, duplicate IDs and unavailable evidence. Labels should include expected status, reason codes and required evidence, not just a polished ideal answer.

Use development cases to iterate. Reserve a different evaluation split and avoid changing the system after inspecting each individual result. A public practice split is not a genuinely hidden benchmark. Group related templates or suppliers when splitting to reduce leakage. Have a second person inspect ambiguous labels when possible. Report sample size and scenario coverage; a dozen fixtures cannot establish production safety.

Try it: Why is randomly splitting near-identical versions of the same invoice likely to overstate quality?

Check your reasoning

Identifies leakage/shared structure across splits and recommends grouping by source/template or time.

You’ll learn to: Compute rank metrics against declared relevance labels.

For a query with R labelled relevant documents, recall@k is relevant documents found in the first k results divided by R. Precision@k is relevant results in those k slots divided by k when k results are returned. Reciprocal rank is 1 divided by the first relevant result’s rank, or zero if none is retrieved; average it across queries for MRR.

Decide what counts as relevant: a document about the same topic is not necessarily evidence for the answer. Multi-document questions may need both a policy and an invoice, so first-hit metrics alone are insufficient. Evaluate after authorization and version filters. Track empty-result rates and candidate recall before judging the generator.

Try it: Relevant IDs are A and C; results are C, B, A. What is recall@2?

Check your reasoning

One of two relevant documents appears in the first two results, so recall@2 is 0.5.

You’ll learn to: Use component metrics and examine disagreements.

Decision accuracy compares statuses, but a correct status with an invented explanation is still a failure. Measure status accuracy, reason-code correctness, evidence completeness, unsupported claims and review workload. Break results down by scenario instead of reporting only an aggregate.

An LLM judge can help triage explanations, but it is an evaluator with its own errors. Define a rubric, compare its judgments with human labels and record disagreement. Use deterministic checks for schemas, arithmetic and citation IDs first. A stricter judge may lower pass rates without making the underlying system worse. Keep model, rubric and prompt versions fixed for a comparison; include sample size and uncertainty in any improvement claim.

Try it: Two versions have equal status accuracy. Name a measurement that could distinguish their usefulness to reviewers.

Check your reasoning

Offers evidence support/completeness, reason correctness, review time or uncertainty handling with an observable definition.

You’ll learn to: Make failures reproducible across stages.

Give each request a correlation ID and each stage a span. Record input document hashes, parser and index versions, retrieved IDs, policy version, model identifier, status, duration and bounded error category. Keep secrets and full documents out of routine telemetry. A trace should let you locate the failing stage and reproduce it from authorized retained inputs.

Distinguish a timeout from a valid empty result and a schema error from unsupported evidence. Track distributions, such as p95 latency, rather than averages alone. Use sampled content inspection only under an explicit access and retention policy. Version changes must be visible so yesterday’s success can be compared with today’s regression.

Try it: What minimum metadata lets you reproduce a result after a policy update?

Check your reasoning

Includes immutable input/version identifiers, applicable policy version and relevant parser/index/model configuration.

You’ll learn to: Recover from failures without duplicate actions or unbounded spending.

Set deadlines, token limits and maximum tool calls before requests run. Retry only failures likely to be transient, with backoff and a finite attempt limit. A malformed request is not improved by repeating it unchanged. Reserve budget before paid work and reconcile observed usage afterward.

Use idempotency keys for logical operations and durable state transitions so client retries do not create duplicate jobs. Cache entries must include tenant, document version, policy version and relevant model/prompt configuration. An old cached answer is not valid merely because the question text is unchanged. Never cache a temporary upstream failure as a successful answer.

Try it: Why should a cached invoice decision include the policy version in its key?

Check your reasoning

Because a changed applicable rule can change the result even when invoice bytes and question are unchanged.

You’ll learn to: Ship a version only when its evidence supports the change.

Freeze a release bundle: parser, prompt, retrieval settings, policy, model and evaluation data version. Run component and end-to-end checks. Define blocking regressions before seeing results: invalid evidence IDs, cross-tenant retrieval and unsupported positive decisions should fail a release even if aggregate accuracy improves.

Roll out gradually where possible and monitor error categories, review volume, latency and cost. Keep the last known-good configuration available. Rollback must restore compatible indexes and policies, not just application code. A green synthetic test suite is a release input, not a guarantee about unseen documents. Document which production observations would trigger a rollback or disablement.

Try it: What should a rollback plan include when a release changes both chunking and generation prompts?

Check your reasoning

Compatible previous index/source mapping and prompt/model configuration, plus a tested way to select them.

You’ll learn to: Combine extraction and provenance into a reviewable artifact.

Take-home 1 builds ingestion for the provided TXT, CSV and HTML fixtures. Emit a normalized JSON evidence packet with raw spans, locators, hashes and explicit missing states. Reject ambiguous money and duplicate identifiers instead of guessing. Add one new malformed fixture and explain the expected behavior.

Take-home 2 builds retrieval and grounded answers over those packets. Start with exact IDs and lexical search. Label at least six questions, including one unanswerable question and one requiring two documents. Compare candidate recall and citation support. Add embeddings or a reranker only as a measured extension. Deliver code, a reproducible command and an error-analysis table; screenshots alone cannot establish correctness.

Try it: What evidence would show that your extraction project handles a missing total correctly?

Check your reasoning

A reproducible fixture/test and output preserving missingness with provenance, rather than a manually edited screenshot.

You’ll learn to: Turn evidence into deterministic statuses with useful explanations.

Take-home 3 consumes validated facts and a fixed policy version. Return ready_for_review, needs_information or exception, a set of reason codes, and supporting document IDs. The practice policy is deliberately small: a matching same-tenant PO must exist, required fields must be present, currencies must agree and invoice amount must not exceed the limit. Unknown or contradictory facts cannot yield ready_for_review.

Implement boundary tests and compare against the labelled development cases. Generate a short explanation only after the rule result exists; if that explanation contradicts the result, reject or replace it with a deterministic template. Do not add payment or approval integrations. Document policy precedence, limitations and which additional evidence would resolve each exception.

Try it: An explanation says “ready” while deterministic rules return exception. Which result governs?

Check your reasoning

The validated rule result; reject or replace the inconsistent explanation and preserve an auditable discrepancy.

You’ll learn to: Present measured tradeoffs and demonstrate recovery.

Take-home 4 integrates the pipeline and adds a review interface or CLI. Start from the downloadable baseline, improve at least one documented weakness and evaluate on development plus untouched local evaluation cases. The published evaluation split is transparent practice data, not a hidden benchmark. Add three genuinely new cases after freezing the implementation.

Submit a short architecture note, source/evidence examples, component metrics, end-to-end errors, latency/cost measurements and a rollback plan. Demonstrate a missing document, a duplicate request and a stale policy. Your strongest result may be a smaller workflow with clearer limits. Reviewers should be able to reproduce the run offline before any optional paid-model experiment. Passing the exercise is not a certification of real-world readiness.

Try it: What would make a higher-scoring capstone less convincing than a simpler baseline?

Check your reasoning

Data leakage, unsupported evidence, unreproducible results, unreported cost or missing failure handling; connects the weakness to a concrete check.