Testing function calling accuracy with complex schemas

Calling external functions reliably forms the backbone of modern AI applications. When a language model has to generate flat parameters such as a search term or a date, most models achieve high success rates. The situation changes drastically as soon as a JSON schema contains deep nesting, dependent objects, arrays with specific restrictions or polymorphic structures. This measurement supports a crucial architectural decision: can a lighter or cheaper model fill this specific function schema flawlessly, or is a heavier reasoning model necessary to prevent runtime errors in backend systems?

This article differs explicitly from general API manuals and generic stress tests. Where the article on JSON validity under load focuses on network latency and syntax corruption under heavy concurrency, this evaluation method analyzes the semantic and structural parsing power of models when faced with complex interfaces. We examine not only whether the JSON validates, but whether the generated values connect logically to the schema and the user instruction.

The anatomy of structural function failure

When a model fails at function calling with complex schemas, it rarely does so with an invalid punctuation mark. The introduction of constrained decoding and grammar-based sampling in leading inference engines ensures that the raw output is almost always syntactically correct JSON. The real errors occur at a more abstract level:

First, we see type confusion within arrays. When an array permits polymorphic objects (such as a list of filters in which each filter type requires a different value type), the model regularly injects properties of one object type into another. Second, hallucinations of optional fields occur: the model fills non-required properties with assumptions rather than leaving them empty when the source text contains no information. Third, context loss in deep nestingarises. From the third level of object structures onward, models tend to repeat global fields within sub-objects, or simply skip required keys in deeper layers.

To understand how this mechanism works under the hood, the guide on how a model calls a function offers in-depth background on token constraints and function definitions. The evaluation method below builds on this by defining systematic measurement criteria.

The measurement dimensions: from syntax to semantic fidelity

A robust benchmark for function calling uses four separate measurement dimensions. A composite binary score (pass/fail) hides where the integration risk actually sits.

Recording these four dimensions strictly separately makes it immediately clear whether a model stumbles structurally over JSON definition logic or simply struggles to read the input text with comprehension.

Defining a benchmark schema

To put a model genuinely to the test, a trivial function description is not enough. We use a composite schema in which nested entities, conditional fields, enums and unique array items all come together. Think of a financial transaction schema for a mortgage quote request:

{
  "name": "verwerk_hypotheek_aanvraag",
  "description": "Verwerkt een samengestelde hypotheekaanvraag met meerdere onderpanden en leningdelen.",
  "parameters": {
    "type": "object",
    "properties": {
      "dossier_id": { "type": "string", "pattern": "^DOS-[0-9]{5}$" },
      "aanvragers": {
        "type": "array",
        "minItems": 1,
        "items": {
          "type": "object",
          "properties": {
            "rol": { "type": "string", "enum": ["hoofdaanvrager", "partner", "borgsteller"] },
            "persoonsgegevens": {
              "type": "object",
              "properties": {
                "achternaam": { "type": "string" },
                "geboortedatum": { "type": "string", "format": "date" },
                "bruto_jaarinkomen": { "type": "number", "minimum": 0 }
              },
              "required": ["achternaam", "geboortedatum", "bruto_jaarinkomen"],
              "additionalProperties": false
            }
          },
          "required": ["rol", "persoonsgegevens"],
          "additionalProperties": false
        }
      },
      "leningdelen": {
        "type": "array",
        "minItems": 1,
        "items": {
          "type": "object",
          "properties": {
            "aflosvorm": { "type": "string", "enum": ["annuitair", "lineair", "aflossingsvrij"] },
            "hoofdsom": { "type": "integer", "minimum": 1000 },
            "looptijd_maanden": { "type": "integer", "minimum": 12, "maximum": 360 }
          },
          "required": ["aflosvorm", "hoofdsom", "looptijd_maanden"],
          "additionalProperties": false
        }
      }
    },
    "required": ["dossier_id", "aanvragers", "leningdelen"],
    "additionalProperties": false
  }
}

To validate and analyze similar definitions interactively, use the JSON schema output validator tool, which tests schema requirements directly against sample outputs.

Building the test collection: synthesis and traps

A reliable test set for complex schemas preferably contains at least 100 to 200 unique test cases. An effective guideline for building this dataset is to divide the prompts across three specific categories. These percentages are a practical rule of thumb that you can adjust to the risks of your own domain:

1. Explicit, complete instructions (guide value around 60%): Messages in which all necessary fields are stated unambiguously. Here you test the basic ability to capture structured data without permutations in field names.

2. Messages with noise and distraction (guide value around 25%): Messages containing superfluous details that look tempting for optional parameters but do not belong in the schema. Consider a fictitious renovation scenario by way of illustration: "The applicant would like to add a dormer window costing 12,000 euros, but for now we are only recording the main application of 350,000 euros." A model that cannot filter noise will try to force the fictitious dormer amount into the schema somewhere anyway.

3. Edge cases and missing data (guide value around 15%): Messages in which a required field is deliberately absent, or which contain a typo that clashes with a regex pattern. Here you measure whether the model still calls the function with invalid dummy data, or refuses and falls back on an explanatory text message.

A reproducible measurement setup and test execution

To keep a comparison between model versions scientifically clean, all non-functional parameters must be fixed strictly. Apply the following preconditions:

Set the sampling temperature to temperature = 0.0 (or the lowest value the provider permits) and fix the seed if supported. Switch the model into the API's official function-calling mode, and avoid instructing JSON manually in a standard system prompt. Anyone wanting to dive deeper into the API parameters and implementation details will find the exact architectural patterns in the article on function calling via the API .

Run each test case at least 3 times to rule out stochastic variation. Even at zero temperature, floating-point rounding on clustered GPU infrastructure can cause minimal differences in token choices.

import json
import jsonschema
from typing import Dict, Any, List

def evalueer_function_call(
    schema: Dict[str, Any],
    model_output_args: str,
    ground_truth: Dict[str, Any]
) -> Dict[str, Any]:
    resultaat = {
        "valid_json": False,
        "schema_conform": False,
        "exact_match": False,
        "fouten": []
    }
    
    # Stap 1: Valideer JSON syntaxis
    try:
        parsed_payload = json.loads(model_output_args)
        resultaat["valid_json"] = True
    except json.JSONDecodeError as e:
        resultaat["fouten"].append(f"JSONDecodeError: {str(e)}")
        return resultaat

    # Stap 2: Valideer tegen JSON Schema
    try:
        jsonschema.validate(instance=parsed_payload, schema=schema)
        resultaat["schema_conform"] = True
    except jsonschema.ValidationError as e:
        resultaat["fouten"].append(f"ValidationError: {e.message} op pad: {list(e.path)}")

    # Stap 3: Semantische vergelijking met Ground Truth
    if resultaat["schema_conform"]:
        if parsed_payload == ground_truth:
            resultaat["exact_match"] = True
        else:
            resultaat["fouten"].append("Payload wijkt af van verwachte waarden.")
            
    return resultaat

Typical Dutch language pitfalls in function calls

Evaluating Dutch-language prompts against structured schemas surfaces specific linguistic frictions that remain invisible in English-language benchmarks:

Decimal notations and digit separators: In Dutch texts, periods are used for thousands and commas for decimals. Take a fictitious amount as an example: with the notation 150.000,50 euro , models trained primarily on English-language data regularly translate this into the number 150 instead of 150000.50. This leads to catastrophic errors in administrative payloads.

Compound words and field names: When a schema uses English field names (such as employment_type) and the prompt speaks of "loondienstverband voor onbepaalde tijd", the model has to make both a translation and an abstraction leap to the correct enum value (e.g. PERMANENT_CONTRACT). Models with a weaker Dutch vocabulary drop this field or enter the literal Dutch term as an invalid string.

Address and personal name conventions: Dutch name particles (such as "van der", "de", "ten") are often split incorrectly by models across fields such as first_name and last_name, unless the schema explicitly accounts for this or the model has been tested sufficiently on Dutch name structures.

Aggregating and analyzing measurement results

After working through the complete test set, we aggregate the outcomes into clear KPIs. Do not look at the average success rate across the entire dataset, but segment the results on the basis of schema depth and degree of complexity.

A breakdown like this shows precisely where a specific model's reliability limit lies. When a system in production requires at least 99% schema conformance, the table demonstrates immediately that this model cannot function autonomously from level 3 onward without additional fallback mechanisms or repair prompts.

Costs and turnaround time of the evaluation

Testing function calling at scale calls for a deliberate budget. Because function descriptions and JSON schemas have to be sent along in the context with every API call, token consumption per test case is considerably higher than with standard text prompts.

An average schema of 50 lines of JSON quickly consumes 400 to 800 system tokens per call, function definitions included. Combined with a user prompt of 150 tokens and a generated payload of 250 tokens, one test case costs roughly 1,000 tokens. A test cycle of 200 unique prompts with 3 repetitions (600 calls in total) therefore consumes around 600,000 tokens per model version tested. With commercial API endpoints, the financial investment for this generally stays limited to a few euros per run, while it fully covers the potential production risk of breaking integrations.

To ensure that new model versions or changes to function schemas cause no unexpected regression, integrate this test suite directly into the continuous integration process. In the guide on automating evaluations in the pipeline you can read how to set breakpoints and threshold values for automated builds.

Securing structural reliability

Function calling with complex schemas must never be judged on the basis of anecdotal tests or superficial demonstrations. Only by quantifying schema conformance, argument extraction and edge cases systematically does a clear picture of operational robustness emerge. By including continuous evaluation in the development cycle, you transform fragile model output into a predictable and reliable software component.