**End-to-End Fine-Tuning of Tool-Use Models with the XYZ-Aquila-SFT Dataset**
This article presents a complete, reproducible pipeline for supervised fine-tuning (SFT) of language models on complex tool-use data, using the XYZ-Aquila-SFT dataset, Hugging Face Transformers, PEFT, and PyTorch. The goal is to handle multi-turn tool-use trajectories, preserve embedded reasoning, and adapt models effectively with LoRA while maintaining downstream tool-call accuracy.
—
## From Raw Logs to Fine-Tuned Model
The workflow begins by streaming and inspecting the XYZ-Aquila-SFT dataset, which contains conversations with system instructions, tool schemas, multi-turn messages, and tool-call trajectories. We parse each example into structured trajectories that expose:
– The system core and tool blocks (`
– Embedded tool schemas and reasoning blocks (`
– Tool calls, observations, and response patterns
A nesting-safe JSON parser extracts tool definitions from system messages, while regex patterns identify reasoning and observation tokens. This parsing reveals corpus-level statistics such as:
– Average tool calls per trajectory
– Message depth and trajectory length
– Tool usage frequency and argument-key distributions
– Heavily used tools and outliers
These insights inform later decisions on token budgeting, data splits, and evaluation design.
—
## Aligning Formats: Extract → Render
Because Qwen3 models expect tool schemas embedded in system messages, we implement a reversible two-step process:
1. **Extract**: Convert raw trajectories into a structured format containing messages, tools, questions, and answers.
2. **Render**: Re-inject tools into the system prompt using a Qwen3-compatible template, ensuring exact byte-level consistency where possible.
When subtle format drifts occur, we fall back to preserving the original system content via `tools_suffix`, ensuring no loss of grounding.
—
## Tokenization and Supervision Strategy
Instead of relying on automated chat templates, we manually construct ChatML tokens to maintain precise control over:
– Token boundaries between roles
– Presence or absence of “ and `im_end` tokens
– Token-level loss masking
We apply loss **only to assistant tokens**, including tool calls and observations, while marking all other tokens with `-100`. This approach prevents the model from attending to or optimizing over reasoning or observation tokens that should remain frozen.
Sequence length is capped at 2048 tokens using a truncate policy. Trajectories exceeding this budget are filtered or shortened, with special attention to preserving assistant turns that contain tool calls.
—
## Training Setup with LoRA
We load Qwen3-0.6B with:
– LoRA adapters (rank 16, alpha 32)
– Gradient checkpointing
– Input requirement hooks
– Mixed precision (BF16 on supported GPUs, fp32 otherwise)
Training uses:
– AdamW optimizer
– Cosine learning-rate schedule with warmup
– Gradient accumulation (8 steps)
– Token-level clipping
– Batch size 1
The model is trained for 30 steps as a smoke test; the pipeline supports scaling `N_STREAM` and `MAX_STEPS` for production runs.
—
## Evaluation with Teacher-Forced Probes
To measure tool-call fidelity before and after fine-tuning, we construct **teacher-forced probes**:
– Cut trajectories right before assistant turns containing tool calls
– Prompt the model to generate the next tokens
– Parse predicted tool calls and compare:
– Tool-name accuracy
– Argument-key F1
– Parse success rate
This method isolates the model’s ability to reproduce correct function calls under controlled conditions.
—
## Outputs and Reusability
The pipeline exports:
– A structured JSONL dataset (`aquila_{lang}_structured_tools.jsonl`) containing parsed trajectories
– Corpus statistics (`corpus_stats.json`) with distributions, counts, and evaluation scores
– A LoRA adapter directory compatible with Hugging Face Transformers
These artifacts enable:
– Reproducible experiments
– Ablation studies on sequence length, loss masking, and data filtering
– Further fine-tuning on downstream agentic benchmarks
—
## Frequently Asked Questions (FAQ)
**Q1: What is XYZ-Aquila-SFT?**
XYZ-Aquila-SFT is a dataset of multi-turn, tool-use conversations designed for supervised fine-tuning of language models. Each example contains a question, an answer, a trajectory of roles and content, declared tool calls, and embedded tool schemas.
**Q2: Why manual ChatML rendering instead of `tokenizer.apply_chat_template`?**
Automatic templates may silently drop or alter special tokens used in reasoning and observation phases, which would corrupt supervision. Manual rendering guarantees token-level alignment and preserves all reasoning content.
**Q3: How are tool schemas extracted from the system message?**
Tool definitions are enclosed in `
**Q4: What does “teacher-forced probe” mean?**
A probe truncates a trajectory right before an assistant turn with a tool call. The model generates tokens autoregressively, but we evaluate by comparing the predicted tool call against the gold call from the dataset.
**Q5: Can this pipeline work with other models?**
Yes. As long as the target model uses a similar ChatML format and supports LoRA, you can swap `MODEL_ID` and adjust the tool template accordingly.
**Q6: Why limit loss to assistant tokens only?**
Tool calls, observations, and reasoning steps are supervision signals. Optimizing the model to output them correctly requires loss applied only to those positions; otherwise, the model may learn to ignore or corrupt them.
—
## Conclusion
This end-to-end pipeline demonstrates how to analyze, transform, fine-tune, and evaluate complex tool-use data from the XYZ-Aquila-SFT dataset. By preserving embedded reasoning, applying precise token-level supervision, and using LoRA for efficient adaptation, we maintain and even improve tool-call accuracy after fine-tuning.
The workflow produces reusable structured datasets, detailed corpus statistics, and trainable adapters, providing a strong foundation for scaling tool-aware SFT and building more capable agentic language models. Whether you are experimenting with sequence length, loss masking strategies, or new tool-use benchmarks, this pipeline offers a robust starting point for future research.
**Check out the full code implementations, dive deeper into the datasets, and stay updated with the latest advances by following us on Twitter, joining our ML SubReddit, and subscribing to our Newsletter.**
