Evaluations in your pipeline: testing every change automatically

When a developer adjusts a prompt, upgrades a model version or changes the retrieval parameters in a RAG system, a decision point arises immediately: do we accept this change into the production environment, or do we roll the commit back? To keep this decision from depending on gut feeling or spot-check manual inspection, an automated threshold in your Continuous Integration and Continuous Deployment (CI/CD) pipeline is essential. By automatically running a fixed evaluation set on every change and comparing the outcome quantitatively against the baseline (the current production state), the pipeline can enforce a hard go/no-go on the basis of predetermined quality and tolerance limits.

This article goes beyond the basic principle of repeated testing; where an analysis of regression testing for prompts stops at the concept of repetition, this manual focuses on the technical integration of continuous monitoring within software pipelines. We treat reproducibility explicitly as a technical precondition here, not as a subject in its own right. The goal is to build a robust, automated system that catches quality decay ("drift" or unintended regression) before code or configurations reach the production platform.

The measurement problem: continuous quality assurance with dynamic variables

In traditional software development, the same input under identical circumstances leads to exactly the same output. In applications built on large language models (LLMs), every component introduces variability. A change to the system prompt may raise the accuracy of structured data extraction while at the same time affecting the tone of a summary adversely. Likewise, an update to an embedding model in the retrieval layer can change the relevance of the context, which in turn feeds through into the generative model's eventual answers.

Overseeing all these interdependencies is impossible with manual testing. Without automatic testing, three specific risks arise:

• Invisible regression: An improvement on five specific edge cases quietly causes a drop in quality on 20% of regular user questions.
• Version differences at providers: The model name stays the same (with a rolling release from an API provider, for instance), but the underlying weights change, causing output to shift subtly.
• Lack of an objective decision rule: Members of the development team argue about subjective preferences instead of testing against a quantitative floor.

The solution to this measurement problem is to set up an automated test suite executed on every pull request (PR) or commit, comparable to unit and integration tests in regular software engineering.

Setting up a reproducible measurement design

For an automatic evaluation to function reliably in a CI/CD pipeline, the measurement design must be strictly structured and reproducible. A shaky test setup leads to false alarms (causing the pipeline to block wrongly) or to unwarranted approvals. A complete measurement design within the pipeline requires four core components to be recorded without exception.

1. The fixed variables (determinism)

During an evaluation run, all external factors must be frozen wherever possible. This means the CI/CD script has to specify exactly the following parameters:

• Prompt version and system prompt version: Recorded through semantic version control in Git (see also version control for prompts in code for the implementation in the application layer).
• Exact model identification: Do not use generic aliases such as gpt-4o if the provider also offers specific snapshot versions (such as gpt-4o-2024-08-06).
• Inference parameters: Set the temperature to 0.0 for maximum determinism, or with a higher temperature record a fixed seed value (theseed parameter) if the model supports it. Also record top_p, frequency_penalty and presence_penalty .
• Retrieval configuration: Where applicable to a RAG pipeline: fix the index version, the number of documents retrieved ($k$) and the similarity threshold.

2. Number of repetitions (sample coverage)

Because LLMs can show small variations even at temperature=0 (as a result of floating-point calculations on parallel GPU clusters), a single evaluation run per test case is insufficient for a robust judgment. The measurement design must provide for at least $N=3$ to $N=5$ repetitions per query on the evaluation set in order to calculate the standard deviation. To determine whether a threshold has been crossed without being misled by noise, an understanding of statistics for evaluations is indispensable.

3. Scoring methodology (rubrics and judges)

Quality has to be converted into a quantitative score between 0.0 and 1.0, or a binary label (pass/fail). Depending on the type of task, three evaluation methods are used:

• Deterministic matching: For structured output (JSON, SQL, regex). Here the test suite checks syntactic and semantic validity. An excellent example of this is evaluating JSON validity under load to verify whether the model keeps to the intended schema.
• Traditional NLP metrics: ROUGE, BLEU or BERTScore for comparisons against a reference answer. Very fast and cheap, but limited in recognizing paraphrases that are substantively correct.
• Model judge (LLM-as-a-judge): A more powerful secondary LLM that assesses the output against an explicit quality rubric (on accuracy, completeness and tone, for example).

4. Calibrating the judge

When an LLM is deployed as a judge in the CI/CD pipeline, the reliability of that judge itself must be checked continuously. This is done by regularly comparing the model judge's scores against a human-validated dataset (gold standard) and calculating Cohen's Kappa or the Pearson correlation coefficient. For the precise setup, we refer to the article on LLM-as-a-judge and calibrating judges.

Dutch language and cultural pitfalls in the test set

For applications deployed in the Dutch market, a general (English-oriented) test set offers insufficient guarantees. The automated test set must explicitly contain cases in which specific characteristics of the Dutch language and context are put to the test. If these categories are not tested structurally in the pipeline, unnoticed errors appear in production quickly.

Forms of address (u versus jij)

In Dutch, the distinction between formal (u) and informal (jij/je) is commercially and operationally crucial. A change to the system prompt can cause the model to switch unintentionally from a formal to an informal tone. The test set must contain prompt variations that explicitly check whether the model maintains the instruction about the form of address consistently across multiple conversational turns.

Compounds and word unity

Dutch features compounds written as a single word (for example kwaliteitsbewakingssysteem or integratietestomgeving). Many language models, trained primarily on English tokenization, tend to write these words separately ("kwaliteits bewakings systeem"). This is known in Dutch as the English disease. Automated evaluation must check the model's output for correct spelling of compound nouns, since this directly affects professional presentation.

Regional variants (NL-NL versus BE-NL)

Depending on the audience, the desired terminology can differ between the Netherlands and Flanders (for example vrijblijvende offerte versus offerte zonder verbintenis, or pinnen versus bancontact). A robust test set contains test cases that specifically validate that the terms generated by the application match the right language region.

Entities, legislation and naming

Dutch entities such as BSN numbers, postal codes (1234 AB), Chamber of Commerce numbers and specific legislation (such as the GDPR or particular building decrees) require exact handling. A generic model may tend to format addresses or numbers according to American standards. The test suite in the CI/CD pipeline must verify through regular expressions and qualitative checks that Dutch entities are recognized and generated correctly.

Costs and budgetary control of the evaluation pipeline

Automatically running a complete evaluation set on every Git push or PR update can bring considerable costs. Every test case consumes input and output tokens, and when a model judge (LLM-as-a-judge) is used, those costs double. For a detailed analysis of the base costs, we refer to the overview on the costs of evaluating. In a CI/CD context, these one-off costs turn into a recurring operational item.

To prevent the CI/CD pipeline from consuming unlimited API budget, developers must put budgetary controls in place. The table below gives an overview of the estimated costs per evaluation run for test sets of different sizes.

Strategies for cost control

To keep costs manageable without sacrificing quality, three techniques are combined in practice:

1. Tiered pipeline execution (tiered evaluation):
   • Tier 1 (PR level): Run only a quick smoke test of 15 to 20 critical test cases at low temperature with light assessment (binary regular expressions or JSON schema checks, for example).
   • Tier 2 (merge/main branch): Run the full regression set (100–200 test cases) with $N=3$ repetitions, LLM-as-a-judge assessment included.
   • Tier 3 (nightly build): Extensive stress and load testing with historical data from the entire database. For this, see also the method for reusing evaluation data from production .
2. Caching unchanged components: If only the summary prompt has changed, the retrieval step for the test set need not be executed again provided the context outputs were stored earlier.
3. Programmatic cost limits: Set hard limits on the API keys used for the CI/CD pipeline. Make sure the pipeline fails in a controlled way as soon as a budget threshold is reached. To set up this safeguard, consult the guide on setting hard cost limits and quotas.

The architecture of the automated evaluation pipeline

A complete integration of evaluations into the CI/CD pipeline consists of several successive steps. The flow diagram below illustrates how a code or prompt change moves through the system and leads to a decision.

[ Developer Commit / PR ]
           │
           ▼
[ Step 1: CI Pipeline Triggered ]
           │
           ▼
[ Step 2: Fetch Testset & Baseline Scores ]
           │
           ▼
[ Step 3: Execute LLM Application Run ]
   ├── Fixed Prompts / Models
   ├── Target API Calls (N-iterations)
   └── Log Inputs, Outputs & Latency
           │
           ▼
[ Step 4: Run Evaluators ]
   ├── Deterministic Checks (JSON Schema, Regex)
   ├── Code-based NLP Metrics (ROUGE / BERTScore)
   └── Model Judge (LLM-as-a-Judge)
           │
           ▼
[ Step 5: Statistical Comparison vs. Baseline ]
           │
   ┌───────┴────────────────────────┐
   ▼                                ▼
[ Delta ≥ Threshold ]     [ Delta < Threshold ]
   │                                │
   ▼                                ▼
[ PASS: Merge Allowed ]   [ FAIL: Block Deployment ]
                                    │
                                    ▼
                         [ Trigger Rollback & Log Alert ]

System requirements and integration points

To run this process in GitHub Actions, GitLab CI or Azure DevOps, for example, the following components must be present:

• Test runner script: A Python or TypeScript script that reads in the test suite, executes the API calls to the model under test and captures the results. To detect performance regressions in good time, response time can be monitored within this script as well. Read more about handling response times correctly in measuring latency percentiles.
• Storage of baselines: The scores of the current production state must be stored centrally somewhere (as a JSON artifact in an object storage bucket, for example) so the CI runner can compare the new results against them.
• Logging and observability: All generated answers, API responses and evaluation results from the CI run must be stored traceably for analysis when a build fails. For this, see the guidelines on observability and logging on the API hub.

From evaluation score to automatic PR decision-making

The final step in the pipeline is converting the raw test results automatically into an action: approving or blocking a pull request. Here you must look beyond an average score alone and define specific thresholds (guardrails).

An example of a decision map configuration in a CI script:

# Example evaluation_rules.yaml (Illustratief voorbeeld)
rules:
  - metric: json_validity
    min_threshold: 1.00       # Harde eis: 100% van de outputs moet geldige JSON zijn
    action_on_fail: BLOCK

  - metric: factual_accuracy
    min_threshold: 0.95       # Minimaal 95% nauwkeurigheid vereist
    max_drop_vs_baseline: 0.01 # Maximaal 1% daling toegestaan t.o.v. baseline
    action_on_fail: BLOCK

  - metric: tone_formal_nl
    min_threshold: 0.90
    max_drop_vs_baseline: 0.05
    action_on_fail: WARN      # Geef een waarschuwing op de PR, maar blokkeer niet direct

  - metric: latency_p95
    max_threshold_ms: 2500    # P95 latency mag niet boven 2,5 seconden komen
    action_on_fail: BLOCK

Having these rules tested automatically by the CI runner creates a transparent process. If a developer adjusts a system prompt (see also measuring system prompt variation and consistency) and accuracy on the test set drops by 3%, the pipeline reports immediately which specific test cases the regression occurred on. The team can then repair the change in a targeted way or roll the commit back.

Network bridge: from measuring to building

Setting up continuous evaluations on the subdomain benchmark.llmnet.nl focuses on **measuring** and analyzing model quality. As soon as the evaluation criteria and thresholds have been established, however, the work shifts to **building** and maintaining the actual software architecture.

For technically realizing the test runner, integrating API calls within your existing CI/CD pipeline and handling API errors during test runs, read on in the API guide on automatically testing LLM integrations. Without a solid integration layer in the codebase, after all, an evaluation set remains a theoretical model that cannot be enforced automatically.