Evaluation and Guardrails

Traditional software is deterministic: given the same input, a function returns the same output, and a test asserts on that output directly. An LLM does not work this way. The same prompt can produce different wording, different reasoning paths, or an outright wrong answer from one call to the next. This is why AI applications need two things that a typical CRUD app doesn't: a way to measure whether outputs are actually good (evaluation), and a way to stop bad outputs, bad inputs, or bad tool calls from causing harm before they reach a user or an external system (guardrails).

Evaluation

Evaluation answers the question: is this AI application doing what it's supposed to do? It applies at two different points in an application's life.

Offline evaluation happens before a change ships, similar to a test suite. It's typically built around a dataset of representative inputs, sometimes paired with an expected or reference output. Running the application against this dataset after a prompt change, a model swap, or a new tool gives you a way to catch regressions instead of eyeballing a handful of manual test runs.

Online evaluation happens against real production traffic. Since there's rarely a single correct answer to check against, this usually means sampling live requests and responses for review, or tracking proxy metrics like tool-call error rates, response latency, and user feedback signals (thumbs up/down, follow-up questions that suggest confusion).

Scoring individual outputs usually combines a few approaches. Rule-based checks (exact match, regex, JSON schema validation, checking that a required field is present) are cheap and reliable, but only work for outputs with a well-defined shape. Human review, someone reading the output and scoring it, is the most trustworthy option and the least scalable one. LLM-as-judge, where a separate LLM call scores the output against a rubric, scales better than human review but inherits the judge model's own blind spots, so it's worth spot-checking judge scores against human review periodically.

This is a different use of "evaluation" than the one in the Evaluator-Optimizer pattern, where an evaluator is part of the application's runtime flow, improving a single generation through feedback. The evaluation described here happens around the application, to decide whether the whole system is working, not inside a single request. If your stack already has Langfuse or MLflow wired up for observability, that's also where evaluation scores and traces typically get recorded and reviewed over time.

Guardrails

Guardrails are checks applied around a model call, on the way in or the way out, to enforce constraints the model itself can't be trusted to enforce on its own.

On the input side, this includes rejecting or sanitizing user input before it reaches the model, and basic prompt injection defenses that try to detect instructions embedded in user-supplied content aimed at overriding the system prompt. It also includes PII detection, catching personal data such as emails, phone numbers, or ID numbers before it's sent to a third-party model. Content moderation, checking input or output against categories like violence, hate speech, or self-harm content, fits on either side; some model providers expose a dedicated moderation endpoint for this rather than relying on the main chat model.

On the output side, there's PII redaction (the same concern as input-side detection, applied to what the model returns), and output validation, checking that a response conforms to an expected format or set of allowed values, which overlaps with the structured output techniques used to shape model responses in the first place. Topic and scope restriction keeps the model on the topics it was built for, so a customer support agent doesn't start answering unrelated questions. Tool and permission restriction limits which tools a model can call and with what arguments, so it can't take an action outside its intended scope even if it's tricked into trying.

In a Spring AI application, the natural place to implement most of these is an Advisor, since advisors run before and after a ChatClient call and can inspect or reject a request or response there. Tool-level restriction is usually better handled further out, for example by controlling which tools an MCP gateway exposes to a given client, rather than trusting the model to self-restrict.

None of these guardrails are foolproof on their own. A regex-based PII filter misses creative formatting, an LLM-as-judge moderation check can itself be fooled, and no single layer catches every prompt injection technique. The usual approach is layering several cheap, fast checks together rather than relying on one comprehensive one.