Predicting Maintenance Actions from Historical Logs using Domain-Specific LLMs

Abstract

Maintenance logs of complex specialized equipment capture problem–action records that are essential for building predictive maintenance solutions but remain difficult to utilize due to their terse, abbreviation-heavy style. This work provides the first systematic benchmark and domain-adaptation study of large language models (LLMs) for predicting maintenance actions from free-text problem descriptions in the MaintNet aviation dataset. We evaluate a range of proprietary and open-source LLMs under zero-shot and few-shot prompting and additionally fine-tune selected open models for supervised evaluation. Experiments are conducted on both raw-abbreviation and expanded datasets, using both lexical (ROUGE, BLEU) and semantic (cosine similarity, BERTScore) metrics. Results show that GPT-4o achieves the strongest semantic alignment, while the instruct version of Gemma-3-4B leads in lexical overlap. Few-shot prompting boosts weaker models disproportionately, narrowing the gap with stronger baselines. Fine-tuning delivers the most significant gains, with instruct versions of Gemma-3-4B, LLaMA-3.2-3B, and Phi-4-mini, improving BLEU by up to 90% and ROUGE-2 by 30%. Notably, the fine-tuned Gemma-3-4B surpasses GPT-4o across multiple metrics, demonstrating the effectiveness of domain-specific adaptation. These findings highlight the potential of fine-tuned LLMs to utilize unstructured aviation logs for building reliable maintenance systems.