Fine-Tune Compare

Documentation & Technical Guide

How It Works

Fine-Tune Compare is a client-side evaluation tool that helps you assess the quality of your fine-tuned LLM by comparing its outputs against a base model. The tool operates in four stages:

Import & Matching: Upload three files (test dataset, base outputs, fine-tuned outputs). The tool matches rows using IDs or normalized prompt hashing.
Validation: Checks for missing fields, mismatches, duplicates, and data quality issues before evaluation.
Evaluation: Computes metrics for each example, including structure compliance, similarity to expected output, and output stability.
Analysis & Export: Classifies results as Improved/Regressed/Neutral, provides filtering and sorting capabilities, and exports reports for further analysis.

Input File Formats

The tool accepts JSONL (JSON Lines) or CSV formats for all three required files:

1. Test Dataset (Required)

Contains the prompts and optionally expected outputs for evaluation.

{
  "id": "unique_identifier",           // Optional but recommended
  "prompt": "Your prompt text here",   // Required (or "instruction")
  "expected_output": "Expected text",  // Optional but recommended
  "category": "task_type",             // Optional
  "tags": ["tag1", "tag2"],            // Optional
  "metadata": {}                       // Optional
}

2. Base Model Outputs (Required)

Contains outputs from your base (non-fine-tuned) model.

{
  "id": "unique_identifier",           // Optional (must match test dataset)
  "prompt": "Your prompt text here",   // Required if id is missing
  "model_output": "Model response"     // Required (or "output"/"completion")
}

3. Fine-Tuned Model Outputs (Required)

Contains outputs from your fine-tuned model. Same format as base outputs.

{
  "id": "unique_identifier",           // Optional (must match test dataset)
  "prompt": "Your prompt text here",   // Required if id is missing
  "model_output": "Model response"     // Required (or "output"/"completion")
}

Matching Strategy: Rows are matched across files using id if present, otherwise by hashing the normalized prompt (trimmed, whitespace collapsed, newlines normalized).

Output File Formats

CSV Export (Metrics Report)

Contains one row per evaluated example with key metrics and flags. Useful for spreadsheet analysis and visualization.

Columns: id, category, prompt, expected, base_output, tuned_output, score_base, score_tuned, delta, status, flags

JSON Export (Full Report)

Contains complete evaluation results including summary statistics and detailed metrics for each example.

Structure: {"summary": {...}, "results": [...]} with full metric breakdowns per example

JSONL Export (Worst Cases for Retraining)

Contains up to 100 worst-performing examples (lowest delta) in JSONL format, ready to be imported into the JSONL Dataset Editor for dataset correction.

Fields: prompt, expected_output, tuned_output, label, delta, reason_flags

Algorithms & Metrics

Structure Compliance Score

Evaluates whether the output follows expected structural patterns:

JSON Validation: Checks if output is valid JSON (+0.5 points)
Code Fence Detection: Detects markdown code blocks with ``` (+0.25 points)
Markdown Headers: Detects markdown headers with # (+0.25 points)

Range: 0.0 to 1.0 (normalized)

Similarity to Expected (Jaccard Index)

Measures token overlap between model output and expected output using the Jaccard similarity coefficient:

J(A,B) = |A ∩ B| / |A ∪ B|

Where:
- A = set of tokens in model output (lowercased, whitespace-split)
- B = set of tokens in expected output (lowercased, whitespace-split)
- ∩ = intersection (common tokens)
- ∪ = union (all unique tokens)

Range: 0.0 (no overlap) to 1.0 (identical token sets)

Repetition Detection

Detects output degradation by identifying repeated 5-word phrases:

Slides a 5-word window across the output
Counts maximum repetitions of any phrase
Normalizes count: min(1.0, max_repeat_count / 5)

Range: 0.0 (no repetition) to 1.0 (severe repetition)

Token Estimation

Estimates token count using the industry-standard approximation:

estimated_tokens ≈ character_count / 4

This approximation is accurate within ~15% for English text and widely used in LLM cost estimation.

Composite Score & Classification

Final score is computed as a weighted average:

If expected_output exists:
  score = 0.4 × adherence + 0.3 × structure + 0.3 × similarity

If no expected_output:
  score = 0.4 × adherence + 0.3 × structure + 0.3 × (1 - repetition)

Delta Classification:

Improved: delta ≥ +0.05 (fine-tuned significantly better)
Regressed: delta ≤ -0.05 (fine-tuned significantly worse)
Neutral: -0.05 < delta < +0.05 (no significant change)

Privacy & Local Processing

All processing happens entirely in your browser using JavaScript. No data is sent to any server. Your test datasets, model outputs, and evaluation results remain completely private on your machine. This tool works offline once loaded and can handle datasets up to 50,000 rows efficiently.

HOWAI