Bearllm Fault Diagnosis Eval

bearllm-fault-diagnosis-eval

BearLLM: A Prior Knowledge-Enhanced Bearing Health Management Framework with Unified Vibration Signal Representation — Haotian Peng et al. (2024) (arXiv:2408.11281, 2024)

What this evaluates

Evaluates a multimodal LLM framework's ability to perform bearing fault diagnosis, anomaly detection, and maintenance recommendation using vibration signals and textual prompts. It probes cross-condition generalization and zero-shot transfer across diverse industrial bearing datasets.

Datasets

• MBHM — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/SIA-IDE/BearLLM
• JUST — total ?; splits: test (-1)
• IMS — total ?; splits: test (-1)
• CWRU — total ?; splits: test (-1)
• XJTU — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly classified fault types or correctly identified tasks relative to the total number of test instances. Calculated as (number of correct predictions / total number of predictions) × 100.

Input / output format

Input: Paired vibration signal segments (time-domain or frequency-aligned) and natural language task prompts (e.g., fault diagnosis, anomaly detection, maintenance recommendation).

Output: Natural language text describing the fault type, severity, or maintenance action; or fault class labels for baseline comparison.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Class imbalance in datasets like IMS skews overall accuracy, masking poor performance on minority fault classes.
• Zero-shot generalization tests may suffer from domain shift if the training MBHM dataset lacks representative condition coverage.
• Severity underestimation in zero-shot settings can occur when the model lacks fine-tuning on specific fault severity distributions.

Evidence (verbatim from paper)

On the JUST dataset, our method achieves an accuracy of 90.22% without any fine-tuning.

Citation

@misc{peng2024bearllm,
  title={BearLLM: A Prior Knowledge-Enhanced Bearing Health Management Framework with Unified Vibration Signal Representation},
  author={Haotian Peng et al. (2024)},
  year={2024},
  note={arXiv:2408.11281}
}

• arXiv: 2408.11281