Fine-tuning large language models (LLMs) is a powerful way to adapt them for specific tasks, but it can also introduce unintended changes in behavior—like biases, harmful outputs, or semantic drift. Current evaluation methods rely on predefined metrics, which may miss unexpected shifts. An unsupervised approach could help detect these changes automatically by comparing the outputs of the original and fine-tuned models.

How It Could Work

One way to uncover behavioral differences is by analyzing text outputs from both models. Here’s how it might work:

• Sampling: Generate responses from both models under similar conditions (e.g., the same prompts).
• Clustering: Use embeddings (like sentence transformers) to group similar responses without predefined labels.
• Labeling: Assign human-readable descriptions to each cluster using another LLM (e.g., "positive statements about geese").
• Comparison: Check if the fine-tuned model dominates certain clusters, revealing behavioral shifts.

This method could highlight surprising changes, like a model suddenly favoring certain topics or tones after fine-tuning.

Potential Applications and Challenges

Researchers and companies deploying fine-tuned models could use this to monitor unintended effects without manual effort. For example:

• AI safety teams might detect harmful biases early.
• Model providers could integrate it into their evaluation pipelines.

Challenges include ensuring clusters are meaningful (not just noise) and that automated labels are accurate. A proof-of-concept could start small—comparing GPT-3 and a fine-tuned version—then scale up with better clustering and labeling techniques.

How It Compares to Existing Tools

Current tools, like Hugging Face’s evaluate or OpenAI’s human-reviewed assessments, rely on predefined metrics or manual checks. This approach differs by:

• Discovering behavioral changes without pre-specifying what to look for.
• Using unsupervised methods to reduce human effort.

Future improvements could include adapting the method for multimodal models or reinforcement learning pipelines.