Two Paths Toward Domain Intelligence

As large language models continue to move from research labs into real production systems, one question keeps coming up again and again: how do we turn a powerful but generic model into something that truly understands a specific domain?

General-purpose LLMs are impressive, but they are rarely precise enough out of the box for technical or enterprise use cases. They speak fluently, but fluency is not the same as expertise. When the task involves strict formats, domain rules, APIs, network configurations, or deterministic reasoning, adaptation becomes necessary.

Two techniques dominate this adaptation process today: fine-tuning and knowledge distillation. They are often mentioned together, sometimes even confused with each other, yet they solve the problem in very different ways. Understanding this distinction is essential before attempting to combine them into more advanced pipelines.

Fine-Tuning: Learning from Data

Fine-tuning is usually the first approach people encounter. The idea is conceptually simple: you take a pretrained language model and continue training it on data that reflects your target domain. If the base model has learned general language patterns from the internet, fine-tuning teaches it how language behaves in a much narrower context. Over time, the model shifts its internal representations to better match the terminology, structure, and expectations of that domain.

In practice, fine-tuning can take multiple forms. Full fine-tuning updates every parameter of the model and can lead to very strong specialization, but it is also computationally expensive and risky, especially when the dataset is limited. This is why parameter-efficient techniques such as LoRA and QLoRA have become so popular. Instead of modifying the entire network, these methods inject small, trainable components that steer the model's behavior while keeping most weights frozen. The result is a significant reduction in compute and memory requirements, without sacrificing too much performance.

The strength of fine-tuning lies in control. When you fine-tune a model, you are explicitly teaching it what to do through examples. If the dataset is well curated and representative, the model can become highly reliable within that scope. However, this strength is also its weakness. Fine-tuning is only as good as the data you provide. High-quality domain data is expensive to create, difficult to maintain, and often incomplete. There is also the risk of overfitting, where the model learns patterns too rigidly and loses some of its generalization ability.

Knowledge Distillation: Learning from Models

Knowledge distillation approaches the same problem from a completely different angle. Instead of learning directly from human-curated data, a model learns by imitating another model. In this setup, a very large and capable language model acts as a teacher. The teacher is already good at reasoning, formatting, and handling complex instructions. The goal is not to replicate its entire knowledge base, but to transfer its behavior to a smaller, more efficient student model.

The process typically involves asking the teacher model to generate outputs for a wide range of inputs. These outputs may include not only final answers, but also intermediate reasoning steps, structured formats, or domain-specific conventions. The student model is then trained to reproduce these outputs as closely as possible. In doing so, it internalizes patterns that would be difficult to encode manually or collect through human annotation.

The appeal of distillation is efficiency. Smaller models are cheaper to run, easier to deploy, and faster at inference time. By learning from a strong teacher, they can achieve a level of performance that would otherwise be unreachable given their size. However, distillation comes with its own constraints. The student is fundamentally limited by the teacher's behavior. Any bias, mistake, or blind spot present in the teacher will be inherited by the student. Moreover, distillation is less flexible than fine-tuning when it comes to introducing genuinely new knowledge, because the student is not learning from raw data but from a filtered, model-generated view of the world.

Comparing the Two Approaches

This is where the comparison between fine-tuning and distillation becomes interesting. Fine-tuning is about grounding a model in data. Distillation is about transferring competence. One emphasizes learning from examples created or selected by humans, while the other emphasizes learning from the behavior of a more capable system. Fine-tuning offers greater flexibility and deeper domain control, but at a higher cost. Distillation offers efficiency and scalability, but at the price of dependency on the teacher.

In many enterprise and technical domains, the problem is not the lack of a powerful teacher model. Large, general-purpose LLMs already exist and perform remarkably well. The real challenge is making that intelligence usable in production, where latency, cost, and determinism matter. This is especially true in structured environments such as network automation, configuration management, or API interaction, where consistency and correctness are far more important than creative language generation.

A Foundation for Hybrid Approaches

Understanding fine-tuning and knowledge distillation as distinct but complementary tools is the foundation for more practical approaches. Once this mental model is clear, it becomes natural to ask whether the two techniques can be combined in a way that leverages their strengths while minimizing their weaknesses. That question opens the door to hybrid pipelines, where large models act as teachers and data generators, and smaller models are efficiently fine-tuned to become domain specialists.

In the next step, I will explore exactly how this hybrid approach works in practice, why it is particularly effective in structured domains, and how to design a pipeline that turns a general-purpose LLM into a production-ready domain expert without massive compute or datasets.

Step One: Choosing the Teacher Model

Every experiment built around distillation and synthetic data generation starts with a crucial decision: choosing the teacher. This step matters more than any architectural trick or optimization that comes later, because the teacher defines the upper bound of what the student can realistically learn. If the teacher is weak, inconsistent, or imprecise, the entire pipeline inherits those flaws.

In this approach, the teacher is not meant to be deployed, optimized, or compressed. Its role is purely epistemic. It acts as a source of knowledge, reasoning patterns, and domain-specific behavior that would be extremely expensive to encode manually. For this reason, the teacher must be significantly stronger than the student model, both in raw capability and in reliability.

In general, suitable teachers belong to the class of very large, state-of-the-art LLMs such as GPT-4–level models or equivalents like Claude 3 Opus. These models share a set of properties that make them ideal teachers rather than candidates for direct fine-tuning. They exhibit strong reasoning abilities, meaning they can follow multi-step logic, respect constraints, and maintain coherence across complex instructions. They also show high technical precision, which is essential in domains where small errors can invalidate an entire output, such as configuration generation or API reasoning. Finally, they are particularly good at producing structured outputs, whether that means JSON schemas, command sequences, formal specifications, or deterministic templates.

Another practical consideration is accessibility. These models are typically available through APIs rather than as downloadable weights. This means there is no access to internal logits or hidden states, and therefore classical logit-based distillation is not an option. Instead, the distillation process necessarily takes the form of supervised fine-tuning on generated outputs. In other words, the teacher provides high-quality input–output pairs, and the student learns to reproduce the same behavior. While this may sound like a limitation, in practice it aligns very well with real-world constraints, where closed models are often the strongest knowledge sources available.

For this specific experimentation, the teacher model used was GPT-5.2, which at the time represents the most advanced version available. The choice was deliberate. The goal was not to test whether distillation works in principle, but to test how far this approach can be pushed when the teacher is exceptionally strong. GPT-5.2 offers a combination of deep reasoning, consistency across long contexts, and robustness in technical domains that makes it particularly well suited for acting as a domain intelligence generator rather than a conversational assistant.

Within the pipeline, the role of the teacher is twofold. First, it is responsible for generating a synthetic dataset of very high quality. This dataset is not generic text, but carefully prompted, domain-specific material that reflects the exact behaviors the student model is expected to learn. Because the teacher already understands both the domain and the desired output structure, it can generate data that would otherwise require extensive human effort to curate.

Second, and more subtly, the teacher transfers implicit knowledge about the domain. This includes conventions, edge cases, typical reasoning paths, and formatting habits that are rarely written down explicitly. When the teacher produces an output, it encodes not just the final answer, but also a set of assumptions about how problems in that domain should be approached. By training on these outputs, the student model begins to internalize those assumptions, effectively absorbing domain expertise without ever being exposed to the full complexity or scale of the original training data.

This first step sets the tone for the entire experiment. Rather than treating the teacher as a black-box oracle to query at inference time, it is treated as a temporary but extremely powerful mentor. Once it has done its job generating and shaping the data, it can be removed from the system entirely, leaving behind a smaller, cheaper, and more controllable model that still reflects much of the teacher's domain intelligence.

Step Two: Defining the Domain of Competence

Once the teacher model has been selected, the next step is to decide what kind of intelligence we actually want to distill. This choice is far from trivial. Not all domains benefit equally from this approach, and selecting the wrong one can lead to disappointing results, even with an excellent teacher.

For this experimentation, the chosen domain was network automation and configuration management. This is a highly technical field, deeply rooted in operational constraints, where correctness and consistency matter far more than stylistic fluency. Network engineers interact daily with structured systems, formal grammars, and deterministic outputs, often through command-line interfaces and configuration files that leave little room for ambiguity. From a distillation perspective, this is an ideal environment.

What makes this domain particularly suitable is its structured nature. Network devices expose information through well-defined commands, and their outputs follow recurring patterns. Configuration syntax is strict, validation rules are explicit, and mappings between different representations, such as CLI commands and data models, are governed by clear semantics. This repeatability is crucial. It allows a teacher model to generate large amounts of high-quality synthetic data that is internally consistent, and it allows a student model to learn stable behaviors rather than brittle heuristics.

Within this domain, the experimentation focused on a set of concrete operational tasks that frequently appear in real automation pipelines. One of these tasks is parsing CLI output, which involves transforming raw, human-readable command output into structured representations that can be consumed by software systems. Another is configuration generation, where the model must produce valid device configurations given a set of parameters or constraints. Closely related to this is parameter validation, which requires understanding both acceptable ranges and interdependencies between configuration options.

The domain also naturally involves translation between different abstractions. Mapping CLI-based configurations to structured data models such as YANG or JSON is a common requirement in modern network management systems. This task is especially interesting from a learning perspective, because it tests whether the model understands the meaning behind commands, not just their surface form. Finally, guided troubleshooting plays an important role. In this case, the model is expected to reason over observed states, identify likely issues, and suggest corrective actions in a controlled and technically accurate way.

To make this more concrete, consider a simple but representative example used during data generation. The input consists of the output of a Cisco IOS XR command such as show interfaces. This output is dense, semi-structured, and designed primarily for human operators. The task given to the teacher model is to extract the operational state of the interfaces and return it as a normalized JSON structure. This requires recognizing interface names, administrative and operational states, counters, and statuses, and then mapping them into a consistent schema.

This kind of prompt encapsulates many of the reasons why the domain is well suited for distillation. The task is deterministic, the expected output format is strict, and there is a clear notion of correctness. A strong teacher model can perform this transformation reliably and consistently across many variations of the input. By collecting thousands of such examples, the student model is not just memorizing patterns, but learning how to systematically translate unstructured operational data into machine-friendly representations.

By clearly defining the domain of competence at this stage, the experiment establishes firm boundaries for what the student model is expected to know and, just as importantly, what it is allowed to ignore. This focus is what enables efficient learning later on. Rather than attempting to compress general intelligence, the goal is to distill a very specific and highly valuable slice of it, tailored to a domain where accuracy and structure are non-negotiable.

Step Three: Building the Synthetic Dataset

With a strong teacher model and a clearly defined domain, the experimentation moves into its most critical operational phase: the creation of the synthetic dataset. This is the point where the abstract idea of "learning from a bigger model" turns into something concrete and trainable.

The guiding principle behind the dataset design is realism. The prompts given to the teacher are intentionally close to what a real system, operator, or automation pipeline would produce. They are not artificially simplified, nor are they designed to showcase the teacher's creativity. Instead, they are meant to elicit precise, repeatable behavior. Variability is introduced, but in a controlled way. CLI outputs differ slightly in formatting, ordering, or verbosity. Configuration requirements change parameters, constraints, or feature combinations. Troubleshooting scenarios vary symptoms while remaining grounded in realistic network behavior. This controlled diversity ensures that the student model learns to generalize within the domain, without being exposed to noise that would dilute the signal.

Another key design choice is determinism in the outputs. For a given class of task, the expected structure of the response is fixed. The teacher is instructed to always return the same schema, the same field names, and the same level of detail. This consistency is not an aesthetic preference; it is a functional requirement. The student model is being trained to act as a component in an automation system, not as a conversational agent. Deterministic outputs make downstream integration possible and reduce the risk of unpredictable behavior at inference time.

Structure plays a central role throughout the dataset. Every example follows a clear instruction–input–output pattern. The instruction defines the task in explicit terms, the input provides the raw material to operate on, and the output represents the normalized, machine-consumable result. This structure mirrors supervised fine-tuning setups and allows the student model to focus on learning the transformation itself, rather than inferring what is being asked.

A simplified example of a single record illustrates this approach clearly:

{
  "instruction": "Parse the following CLI output and extract interface state",
  "input": "GigabitEthernet0/0/0 is up, line protocol is up...",
  "output": {
    "interface": "GigabitEthernet0/0/0",
    "admin_state": "up",
    "oper_state": "up"
  }
}

Although this example is minimal, it captures the essence of the dataset. The task is unambiguous, the input reflects a realistic CLI snippet, and the output adheres to a strict schema. At scale, thousands of such records cover different interface types, states, error conditions, and vendor-specific variations, all while preserving the same structural contract.

In terms of volume, the experimentation deliberately avoids the "more is always better" mindset. The dataset size typically ranges from a few thousand to a few tens of thousands of examples, roughly between five thousand and fifty thousand records. This range is not arbitrary. It reflects a balance between diversity and control. At this scale, the teacher can generate enough variation to cover the domain effectively, while still allowing careful inspection and validation of the data.

Quality is consistently prioritized over quantity. A smaller dataset with clean, accurate, and consistent examples produces better results than a massive dataset polluted with subtle inconsistencies or incorrect edge cases. Because the teacher model is strong, each example carries a high density of useful signal. The role of the experimenter is not to generate as much data as possible, but to curate the prompting strategy so that the teacher's strengths are expressed in a form the student can absorb.

By the end of this step, the teacher model has effectively been converted into a dataset generator. Its knowledge, reasoning patterns, and domain intuition are no longer locked behind an API, but embedded in a static corpus that can be used repeatedly. This dataset becomes the bridge between a powerful but expensive model and a smaller, efficient student that can be trained, evaluated, and deployed with far fewer constraints.

Step Four: Selecting the Student Model

After the synthetic dataset has been created, the focus shifts from knowledge generation to knowledge absorption. This is where the student model enters the picture. Unlike the teacher, the student is not chosen for its absolute performance, but for its suitability as a deployable, efficient, and adaptable system. The entire experiment ultimately succeeds or fails based on how well this model can internalize the distilled domain intelligence while remaining practical to run in real environments.

The models considered for this role fall into a specific category: mid-sized, open-weight language models in the 7–8 billion parameter range. Examples include Mistral 7B, LLaMA 3 8B, and Qwen 7B. These models are large enough to capture non-trivial reasoning patterns and structured transformations, yet small enough to be fine-tuned and deployed without requiring specialized infrastructure. They represent a sweet spot where capability and efficiency meet.

The primary reason for selecting models in this class is the balance between quality and cost. Smaller models struggle with consistency and long-range dependencies, especially in technical domains where subtle details matter. Much larger models, while more capable, quickly become impractical for experimentation and production due to their resource requirements. A 7–8B model, when properly fine-tuned, can deliver surprisingly strong performance on narrow domains while keeping training and inference costs under control.

Another decisive factor is support for parameter-efficient fine-tuning techniques. All the selected models work well with approaches such as LoRA and QLoRA. This means that instead of updating billions of parameters, only a small number of additional weights are trained. As a result, fine-tuning becomes feasible even on limited hardware, and multiple domain-specialized variants can be produced from the same base model without duplicating the full parameter set. This flexibility is essential when iterating quickly or experimenting with different task formulations.

Deployability also plays a central role in the choice of the student model. These models can be run on-premise, without relying on external APIs or cloud-hosted services. For enterprise and infrastructure-related domains, this is often a hard requirement. Being able to deploy the model close to the systems it interacts with reduces latency, improves reliability, and simplifies compliance with security or data governance constraints. It also allows tighter integration with existing automation pipelines and tooling.

Conceptually, the student model is where all previous steps converge. The teacher provides the expertise, the domain definition sets the boundaries, and the synthetic dataset encodes the behavior to be learned. The student's job is not to become a general-purpose conversational model, but to act as a specialized component that reliably performs a defined set of tasks. Its success is measured not by benchmark scores, but by how well it reproduces the teacher's behavior within the chosen domain, using a fraction of the resources.

At this stage, the system is ready for the actual fine-tuning process. The next step is to apply parameter-efficient training on the synthetic dataset and observe how effectively the student model absorbs the distilled knowledge. This is where the theoretical advantages of the approach are finally tested in practice, and where trade-offs between specialization, generalization, and efficiency become visible.

Step Five: Fine-Tuning Strategy with LoRA and QLoRA

With the student model selected, the experimentation reaches the point where theory turns into measurable results. The objective of this step is not to push the model to its absolute limits, but to inject domain-specific behavior as efficiently and safely as possible. This is where parameter-efficient fine-tuning becomes essential.

For this setup, the recommended method is QLoRA. The core idea behind QLoRA is to combine aggressive quantization of the base model with lightweight trainable adapters. The original model weights are loaded in 4-bit precision, drastically reducing memory usage, while small LoRA layers are added on top and trained in higher precision. This allows fine-tuning models with billions of parameters on hardware that would otherwise be insufficient.

The choice of QLoRA is driven by practicality rather than novelty. Quantizing the base model to 4-bit makes it possible to fit a 7–8B parameter model comfortably within a single GPU with limited VRAM. At the same time, LoRA adapters provide enough expressive power to steer the model toward the desired domain behavior without destabilizing its pretrained knowledge. This balance is particularly important when the training data is synthetic and highly structured, as the goal is refinement rather than wholesale rewriting of the model's internal representations.

The configuration used during fine-tuning follows a conservative philosophy. The LoRA rank is kept relatively small, typically between eight and sixteen. This limits the number of additional parameters and reduces the risk of overfitting, while still allowing the model to learn meaningful transformations. The adapters are applied only to selected components of the attention mechanism, most notably the query and value projection layers. These layers play a central role in how the model attends to different parts of the input, and small changes there can have a disproportionately large effect on behavior.

Training is performed with small batch sizes and a low learning rate. This is a deliberate choice. Synthetic datasets generated by a strong teacher tend to be very dense in information, and aggressive optimization can quickly lead to overspecialization or loss of general coherence. A slower, more controlled training process allows the student model to gradually align with the teacher's outputs while preserving the stability of the underlying pretrained model.

The results of this setup are one of the most compelling aspects of the approach. Less than one percent of the model's total parameters are actually trained. Despite this, the behavioral shift toward the target domain is clearly observable. From a resource perspective, the requirements are modest. A GPU with roughly eight to twelve gigabytes of VRAM is sufficient for both training and experimentation. Training itself completes in a matter of hours rather than days or weeks, making rapid iteration possible.

This step is where the promise of the overall approach becomes tangible. A relatively small, open-weight model, fine-tuned with a lightweight strategy and a carefully generated synthetic dataset, begins to exhibit domain expertise that would traditionally require much larger models or far more expensive training setups. The student is no longer a generalist. It has been shaped into a specialist, not by brute force, but by targeted, efficient learning.

With fine-tuning complete, the final step is evaluation: understanding how well the student model has absorbed the teacher's behavior, where it succeeds, where it fails, and how reliably it performs under real-world conditions. That evaluation determines whether the distilled intelligence is ready to move from experiment to production.

Step Six: The End-to-End Training Pipeline

At this point, all the individual components are in place. What remains is to connect them into a single, coherent pipeline that can be reasoned about, reproduced, and iterated on. Thinking in terms of an end-to-end flow is essential, because the real value of this approach does not lie in any single technique, but in how they reinforce each other when combined.

The pipeline starts with the teacher model. This is the only component that needs to be exceptionally powerful, and it is also the only one that is used transiently. The teacher's sole responsibility is to act as a knowledge source. Through carefully designed prompts, it generates synthetic data that reflects both the domain constraints and the desired behavior of the final system. At this stage, the focus is not on speed or cost, but on maximizing correctness, consistency, and coverage of relevant scenarios.

The outputs produced by the teacher are then collected into a domain-specific dataset. This dataset is static, inspectable, and versionable. Once generated, it becomes independent of the teacher model itself. This is an important conceptual shift. The intelligence embodied by the teacher is no longer accessed dynamically through an API, but encoded into examples that can be reused, filtered, augmented, or regenerated as needed. The dataset effectively becomes the contract between the teacher and the student.

From there, the process moves into supervised fine-tuning. The student model is trained on this dataset using a parameter-efficient strategy, typically QLoRA. During this phase, the student is exposed repeatedly to the same kinds of transformations, reasoning steps, and output structures that the teacher produced. Over time, it learns to reproduce these behaviors autonomously. Because only a small fraction of parameters are updated, this learning process is both fast and stable, even when running on modest hardware.

Once fine-tuning is complete, the teacher disappears entirely from the runtime picture. The resulting artifact is a compact, domain-specialized model that can be deployed wherever it is needed. Inference is fast, costs are predictable, and behavior is tightly aligned with the domain it was trained for. Instead of relying on a large, general-purpose model for every request, the system now uses a focused component that does one job well.

Visually, the pipeline can be summarized as a simple flow:

What makes this pipeline powerful is its asymmetry. The most expensive and capable component is used only once, during data generation. Everything downstream is optimized for efficiency, reproducibility, and deployment. If requirements change, the process can be repeated: regenerate part of the dataset, fine-tune a new adapter, and deploy an updated student model without touching the core infrastructure.

This end-to-end view also clarifies why the approach scales so well in practice. Adding a new task or extending the domain does not require retraining from scratch. It often requires only adjusting prompts, generating additional synthetic data, and running another lightweight fine-tuning pass. The result is a workflow that aligns much more closely with real engineering constraints than traditional large-scale training, while still delivering models that behave like domain experts rather than generic assistants.

With the full pipeline in place, the final questions naturally shift from "how does this work?" to "when does this make sense, and when does it not?" Those trade-offs, along with concrete evaluation results, are what ultimately determine whether this approach belongs in production or remains an experimental technique.

In the upcoming posts, I'll dive into the actual training code, configuration details, and evaluation methodology, along with a transparent discussion of what worked, what didn't, and what I would change in a second iteration. Stay tuned for results and implementation details.