How to Fine-Tune a Voice Agent: The LLM Layer

A voice agent that answers in four polished sentences when the caller needed one, picks the wrong booking tool once in twenty attempts, or drifts off its script by the tenth turn quietly leaks the revenue it was deployed to capture: callers hang up, bookings die half-made, and staff pick up the escalated calls. The usual fight is an ever-growing system prompt, and at some point the patches stop holding – the agent follows the new rule in the morning demo and forgets it on the hundredth call of the day.

Fine-tuning fixes this class of failure at the weights, and it costs less than its reputation suggests: a LoRA pass on a small open-weight model trains in hours for hundreds of dollars, and the tuned model then answers faster and cheaper per call than the frontier API it replaces. The hard part of fine-tuning the LLM in a voice agent is everything around that run – the dataset, the evaluation that proves the tuned model is better than the prompt it replaced, and the inference endpoint the model will run on in production.

The difference is easiest to hear on real caller turns – the base model answers in written English, the tuned model answers the way people talk. The first is a price question: “Do you have anything for two this Friday… the fourteenth I mean, and how much is it?”

The second caller asks: “Can you move my appointment to next week?”

The third changes their mind mid-sentence: “Book it under Peterson… no wait, it’s under my wife’s name, Sanders.”

Identify and lock the issue with your current LLM

Before a failing call goes into the training set, we turn it into an automated test that pins down where the failure lives: the rendered prompt with the tool schemas as the model saw them, the raw tokens before parsing, and the parser output, captured and replayed as assertions against the latest release of the voice agent. If the replay passes, the failure was in the software around the model – a parser bug, a template mismatch, a config value – and it is already fixed without touching the weights. Only the calls that still fail belong in a training set; when we review these reports ourselves, about half of “the model calls tools wrong” cases end here, because the raw tokens show the model was right.

Choosing the base model

A fine-tuned model is one you serve yourself: the weights are yours, and the LLM stage of the agent now points at your own inference endpoint. For a team already running a custom voice stack that is a config change, and even a team on a managed voice platform keeps its setup, since the platforms accept any OpenAI-compatible endpoint as the LLM. Either way, prompt assembly, streaming discipline, and cancellation land in your code – generation has to stop the moment the conversation discards it, or you pay for tokens nobody hears – and that engineering, rather than the GPU bill, is where the budget goes.

The model choice itself comes down to five criteria: the license, total loaded weights rather than active ones, the chat and tool template your runtime supports, adapter support, and results on a held-out set of your own traffic.

The open-weight field has real candidates at every size, and the license is the first filter. Gemma 4 and the Qwen3.5 line both ship under terms that let you deploy commercially without legal review. OpenAI’s GPT-OSS pair covers the same ground.

The shortlist itself comes from the public comparisons, with the caveat that independent tables lag the release calendar: a model that shipped this quarter often publishes its tool-calling scores on its own model card months before anyone else measures it. Artificial Analysis is the closest thing to a current view – it picks up new releases quickly and tracks the latency and hosting cost that decide whether a candidate fits a voice latency budget at all. We use it and the model cards as shortlist filters, and a held-out set of our own schemas and traffic settles the choice.

Building the dataset

The training example and its label

The unit of training data is a complete decision trace, not a question-and-answer pair: the conversation, the tool schemas exactly as the model saw them at that turn, the tool calls and their results. We render every example through the same chat template used at inference, because a dataset that drifts from the production format trains a model for a system that does not exist.

Labelling does not require an annotation team, because the cheapest correct signal already sits in the systems of record. The booking exists, the case was created, the transfer did not happen – compare the end state of the call with the state the call should have produced, and the label is free and objective, even for calls recorded long before the project started. An LLM judge over transcripts comes second, for behaviors with no database trace; it is the pattern behind our automated call evaluation pipeline. Human review comes last, to calibrate the judge rather than label the corpus.

Training only on calls the current agent marked successful preserves its blind spots and removes the failure examples the tune needs to learn from, so the sample mixes verified successes, model-caused failures, correct escalations, mid-call corrections, and calls where the transcript and the end state disagree.

Cleaning and sizing the set

Personal data comes out before anything leaves the call store, but the redacted text is not what we train on – each marker gets a realistic substitute of the same type, so the training data reads like a real call again while carrying none of the real data:

On volume, the two targets sit an order of magnitude apart. Around a hundred well-chosen conversations move style and tone – published work took a small model from under a third of responses in target style to over ninety percent at that size. Tool-call reliability wants low thousands of traces. We stop when held-out performance per behavior levels off, not at a conversation count.

Augmentation is an optional pass, for when the archive is thin exactly where the held-out numbers are weak. We expand real calls rather than inventing new ones: rewording caller turns so the phrasing varies, damaging transcripts the way production does – recognition errors and cut-off turns – and adding extra copies of the rare cases that matter most, escalations and mid-call corrections first. The expansion stays on the training side only, because a reworded evaluation call is still the same call and will make the model look better than it is.

One trap inside the training loop deserves a check of its own: we confirm the model learned from the conversation structure at all, because runs exist where every metric looked healthy while the model had learned to mirror its first response for the rest of the call. Clean loss curves coexist with a model that ignored the conversation. Observability is the prerequisite for all of it.

When no call archive exists

Everything above assumes an archive of production calls, and in most projects that reach us that archive is the missing piece – the agent has not launched yet, or it launched without the logging that would have built it. Three sources stand in for it, ordered by how close each comes to production traffic.

Synthetic traces from the product contract. The scenarios come from the intents, tool schemas, policy states, and escalation rules; caller conversations generated against them run through the real orchestration path, so integration bugs show up early.

Simulated callers. An LLM plays the customer against the agent across scripted scenarios and asserts on tool calls and outcomes turn by turn. Open harnesses cover this; nothing needs to be bought.

Human role-play through the production audio stack. We recruit people who did not design the happy paths, give them goals rather than scripts, and run the calls through the real telephony, recognition, tools, and synthesis, labelling the output as role-play rather than production.

The third source solves a problem in the usual advice: a pre-launch evaluation set cannot contain real calls, because none exist. Role-play through the real stack is the stand-in, and real calls with personal data removed replace it after the first canary.

The main risk in generated data is that it all sounds the same. Synthetic callers are more fluent, more patient, more on-script, and better recorded than real ones. We vary persona, patience, disfluency, and interruption point deliberately, and keep training and evaluation scenarios separated at the seed level, or the test set becomes a rewording of the training data.

Our own default is a fourth path: launch on a frontier model with a prompt, log everything, then tune the small model on the winning calls. Frontier providers’ terms bar training competitors – an intake agent for a law firm is not one. What gets distilled is the application’s own traffic – the transcripts its callers produced and the tool calls the system executed – rather than the teacher’s general capability.

Supervised tuning, and what it quietly breaks

The prompt-only baseline on the held-out set comes first – often that measurement alone ends the project, because it shows the runtime fix already cleared the bar.

Then we train a low-rank adapter on the smallest dataset that covers the behavior, selecting on task metrics rather than token loss, since lower loss coexists with worse tool decisions. LoRA is enough for these behaviors; full-parameter tuning earns its cost only after an adapter stops improving on a failure that matters. One exception from our self-hosting work: mixture-of-experts routing can collapse under a naive low-rank adapter, so an MoE base wants a more careful pass.

Two regressions arrive with every fine-tune and appear in no training set. General capability drops – we record it as a number on a general benchmark and decide whether the loss is acceptable for an agent that must also refuse and escalate. And safety alignment weakens even on completely harmless data: published measurements show harmful responses becoming several times more common after ordinary fine-tuning on ordinary datasets. For a voice agent the behaviors at risk are refusals, escalation rules, and the ban on claiming an action is complete, which is why those rules stay in the inference-time system prompt after the behavior is trained in.

The gate before it takes a live call

Single-trial accuracy is the wrong release gate, because callers do not retry. A model at 90% on one trial is near 40% across eight independent tries; real failures correlate, so we measure the repeated-trial number rather than trusting either intuition. Published voice benchmarks put completion rates at a third to a half of what the same model achieves in text – the fine-tune moves one named behavior, and validation, confirmation flows, and human handoff absorb the remainder, sized for the accuracy the system will actually deliver.

The evaluation runs on the exact build that takes the call – quantized, parser-configured, template-matched – not on the checkpoint from the training run.

The suite size is our estimate, shown with its method so anyone can recalculate it for their own case: eight to twelve task types that carry money or legal risk, five to eight trials each, plus the guardrail and format checks. That lands near sixty to a hundred scored conversations per release – a day of compute and a defensible gate. We count conversations, not utterances, because five hundred utterances drawn from twelve calls is a twelve-sample experiment.

Rollback is an adapter routing change: the previous adapter stays loaded beside the new one, the canary compares like with like, and routing away takes seconds.

The numbers we report are p95 and p99 voice-to-voice latency plus the retry rate, before and after. A tuned model’s latency contribution is mostly fewer tail events: in our own deployment, a malformed-JSON retry cost roughly a second each time it fired, and one turn stalled for twenty-five seconds with no single root cause. A tune that leaves the median flat and removes a retry on one turn in twelve has done more for the caller than one that shaves forty milliseconds. The latency budget sets the targets; the testing guide covers the harness.

How long the LLM fine-tune takes

The training itself takes hours; the calendar goes to the dataset and the infrastructure – reviewing the failing calls, building the baseline and the gate, collecting and cleaning traces, standing up the endpoint, running the canary. How long that takes depends mostly on what already exists: a team with a logged call archive and evaluation in place moves in weeks, while a team starting without logging spends most of the project building that foundation before any training run. The dataset stage then repeats every time the tools change, which is the recurring cost the first run establishes. Two habits keep that cost down: tool schemas stay in the context rather than trained into the weights, so renames and new arguments do not invalidate the tune, and deterministic argument validation stays in the runtime, where it catches drift between the weights and the current tools the day someone ships a schema change.

A release is the whole set of versions together: base model, adapter, chat template, tool schemas, runtime, parser, quantization, generation settings. A runtime or template change triggers the same re-testing as a new adapter, and the set rolls back as one piece. Release approval belongs to someone who did not build the tune.

The sequence end to end

The sequence we run starts with proving the failure is the model’s – the prompt, the providers, and the orchestration checked first, because that is where most voice-agent failures actually live – then builds traces labeled from system-of-record state and trains one adapter on the smallest dataset that covers the behavior. The release depends on repeated trials against calls the training never saw, and goes live as a canary behind an adapter that routes away in seconds.

After that, the tuned model is a system the team operates: a dataset versioned alongside the tool schemas, an evaluation suite that grows with each incident, and a retraining trigger list rather than a calendar. Whether that ownership pays at your volume is what the decision article and the self-hosting analysis answer.

We scope these programs before they start: base model, data plan with or without a call archive, and whether the result returns anything at your volume. Book a consultation.