Skip to content
NLEN
Illustration: Measuring instruction following with IFEval

Quantifying instruction following with IFEval

By Ivo Donker — compiled with AI support (Claude & Gemini) · Last updated: 6 August 2026

When evaluating large language models (LLMs), the focus often falls on knowledge retention, logical reasoning or linguistic fluency. While these properties are crucial for generic chatbots, traditional benchmarks fall short when a model has to operate in an automated software pipeline. In such scenarios the crucial question is not only whether the model knows the right answer, but whether the answer meets the supplied structure and constraints exactly. This aspect is called instruction following.

Systems depending on structured data extraction, automated code generation or specific formatting require a model that executes instructions to the letter. IFEval (Instruction Following Evaluation) is designed to measure this specific behavior objectively and reproducibly. This article goes deeper into how IFEval works, how it differs from other evaluation methods, and how to apply these principles within your own software architecture.

What is instruction following and why is it separate from knowledge?

It is important to draw a sharp distinction between the factual correctness of an answer and how far a model follows instructions. A model can possess exceptional domain knowledge and still be unusable in a production environment if it ignores structural restrictions. Suppose a prompt asks: "Give a summary of nitrogen policy in exactly three paragraphs, without using the letter 'e', and start the last paragraph with the word 'In conclusion'."

A model that gives a factually perfect summary in four paragraphs containing the letter 'e' dozens of times has delivered a substantively correct answer but fails entirely on instruction following. In applications where an LLM's output is passed directly to a parser or database, such deviations cause system errors.

Within the broader context of model testing, it helps to understand how this type of evaluation relates to general leaderboards. For a broader perspective on interpreting general model performance, consult the article on reading benchmarks . Where classic benchmarks focus on 'what' the model knows, IFEval focuses on 'how' the model structures its output.

The core idea behind IFEval: verifiable instructions

Traditional methods for judging complex text generation often lean on human annotators or larger language models acting as judge (LLM-as-a-judge). Both methods introduce noise, subjectivity and inconsistency. IFEval solves this by using only objectively verifiable instructions. These are tasks whose correctness does not depend on semantic interpretation but can be established deterministically with simple program code.

IFEval focuses on so-called heuristic constraints. These are specific rules checkable with regular expressions (regex) or simple algorithms. Some categories of such instructions are:

Because evaluation runs through code, the measurement is fully reproducible. There is no bias, and the cost of running the benchmark is minimal compared with deploying human assessors or making API calls to commercial judge models.

Strict versus loose scoring

IFEval generally reports performance through two different methodologies: *strict scoring* and *loose scoring*. The difference between the two is essential for judging a model's practical usability.

Aspect Strict scoring Loose scoring
Handling of whitespace Counts exactly; extra line breaks can lead to rejection. Is normalized (trimming spaces and blank lines).
Punctuation and casing Exact match required, including capitals and periods. Often case-insensitive and ignores small punctuation differences.
Conversational filler Model utterances such as "Certainly, here it is:" cause immediate failure. Can be filtered out by heuristics before the check.
Application Critical API integrations and parsers. Interactive chatbots for human end users.

In a production environment where JSON pockets have to be filtered directly out of the text, strict scoring is the only metric that matters. If a model tends to add politeness formulas ("Here is the requested schema: ..."), the strict parser breaks, even if the eventual JSON is substantively correct. Loose scoring, by contrast, is valuable when the output is intended for a human reader who does not stumble over an extra space or a missing capital.

Prompt-level versus instruction-level accuracy

A crucial aspect of IFEval is how scores are computed. The benchmark distinguishes two levels of detail:

1. Prompt-level accuracy

Here you look at whether a model meets *all* the requirements set within one prompt. If a prompt contains three different instructions (for example: write in Dutch, use at most 150 words, and avoid the letter 'e') and the model meets only two of the three, the score for that prompt is zero (0). This metric is binary and indicates how reliable the model is on complex, compound tasks.

2. Instruction-level accuracy

This metric looks at the percentage of individual instructions executed successfully, regardless of whether they appeared in the same prompt. In the example above, the model would score 66.7% (2 out of 3). This score gives a more nuanced picture of the model's capabilities and helps developers understand whether a model is 'nearly' functional or systematically ignores specific types of instruction.

Models often show a sharp drop in prompt-level accuracy as the number of constraints per prompt increases. This phenomenon underlines the need, when designing applications, to break instructions up as much as possible or to use specific techniques for structuring input and output.

Building your own verifiable instruction set

To guarantee LLM reliability within a specific business domain, it is advisable to set up your own domain-specific variant of IFEval. This lets you run regression tests every time a prompt, model version or system architecture changes. For a structured approach to this, see the step-by-step plan for setting up your own benchmark is worth consulting.

Building your own evaluation set consists of three steps: selecting instruction types, writing deterministic verification functions, and setting up the test loop.

Step 1: define the constraints

Focus on constraints that directly affect your application. If you extract data to fill a database, focus on JSON validation, XML tags and data types. If you generate blog posts, focus on word limits and paragraph structures.

Step 2: write the verification code

Below is an example of a Python verification function that checks whether a model output meets three specific criteria: it has to contain exactly three paragraphs, must not contain exclamation marks, and has to end with a specific sentence.

def test_instructie_volgzaamheid(model_output: str) -> dict:
    # Restrictie 1: Exact drie alinea's (gescheiden door dubbele regeleinden)
    paragraphs = [p.strip() for p in model_output.strip().split('\n\n') if p.strip()]
    constraint_paragraphs = len(paragraphs) == 3

    # Restrictie 2: Geen uitroeptekens
    constraint_no_exclamation = "!" not in model_output

    # Restrictie 3: Eindigen met een specifieke zin (negeer trailing whitespace)
    expected_ending = "Dit is het einde van de samenvatting."
    clean_output = model_output.strip().rstrip('.')
    clean_expected = expected_ending.rstrip('.')
    constraint_ending = clean_output.endswith(clean_expected)

    return {
        "score_prompt_niveau": int(constraint_paragraphs and constraint_no_exclamation and constraint_ending),
        "details": {
            "drie_alineas": constraint_paragraphs,
            "geen_uitroeptekens": constraint_no_exclamation,
            "juiste_afsluiting": constraint_ending
        }
    }

By running this type of code across a test set of, say, a hundred unique prompts, you generate a reliable picture of the model's performance without any need for manual checking.

Pitfalls in designing instruction tests

When setting up an evaluation system based on IFEval principles, a few subtle pitfalls lie in wait. Failing to recognize these factors can lead to faulty test results and unwarranted conclusions about model performance.

Contradictory or excessive requirements

It is easy to design prompts that unintentionally contain contradictory tasks. A prompt asking for "a summary of exactly 50 words" while simultaneously demanding that "at least 5 specific core concepts be explained extensively"forces the model into an impossible bind. This measures not instruction following but the model's ability to handle logical paradoxes.

Language-dependent verification rules

Many IFEval implementations were originally written in English. Translating or adapting them to Dutch raises problems with language-specific properties. Consider:

Errors in the verification code itself

A common problem is that the code performing the check (the 'grader') contains bugs itself. If the code fails to parse unicode characters or line endings on Windows systems correctly (`\r\n` versus `\n`), for instance, correct model answers get wrongly marked as errors. Extensive unit tests for the verification functions are therefore a requirement.

Relation to system prompts and output enforcement

The score a model achieves on an instruction test does not stand alone; it depends heavily on how the interaction with the model is designed. Two important factors play a role: the design of the system prompt and the use of programmatic enforcement.

A well-formulated system prompt lays down the ground rules the model has to follow throughout the session. By putting restrictions in the system role rather than the user role, output stability can be raised considerably. For best practices in this area, see the guide on system prompts is worth consulting.

When absolute guarantees about output structure are needed, it is also unwise to rely solely on the model's own instruction following. In critical systems you combine prompts with schema validation and forced JSON structures. Applying these techniques reduces dependence on raw model performance. More information on implementing this can be found in the articles on enforcing output formats and the use of API-based structured output.

What does a low score mean in practice?

If a model scores low on instruction following during an IFEval measurement, this has direct consequences for the architecture of the software the model is integrated into. Simply accepting the output is not an option. Developers then have to take mitigating measures:

  1. Implement retry mechanisms: When the verification code detects that the output does not meet the requirements, the system can automatically submit a new request, possibly with an adjusted temperature parameter.
  2. Self-correction loops: The error message from the verification code can be fed back to the model ("Your output contained 4 paragraphs instead of 3. Correct this."). This raises the chance of correct output on the second attempt, though it brings extra latency and API costs.
  3. Build robust parsers: Instead of requiring the model to deliver exact JSON with no extra text, the parser can be designed to filter JSON pockets out of a larger block of text with regex. This catches errors on the receiving end.

Integration into regression testing

Measuring instruction following is not a one-off task. With LLM provider updates, changes in model routing or adjustments in application code, performance can shift unexpectedly. Including IFEval-style tests in the CI/CD pipeline is therefore advisable.

By firing a fixed set of fifty to a hundred verifiable prompts at the model on every code change, you keep regressions from reaching the production environment unnoticed. This process ties closely to setting up broader prompt management. For a deeper dive into this integration, read the article on regression testing for prompts.

Important trade-off: While larger models generally score better on complex instruction sets, smaller models fine-tuned specifically (through direct preference optimization or reinforcement learning on instruction tasks, for instance) can achieve comparable or better results on specific formatting tasks at a fraction of the cost.

Conclusion

IFEval offers a methodical answer to the challenges of subjective LLM evaluation. By focusing on deterministically verifiable constraints, it lets developers quantify the reliability of language models in software pipelines objectively. Whether it concerns enforcing length limits, excluding specific words or imposing XML structures: measuring and monitoring instruction following is a necessary step for any organization that wants to take LLMs seriously in production.

Further reading