Inatag Eval

inatag-eval

iNatAg: Multi-Class Classification Models Enabled by a Large-Scale Benchmark Dataset with 4.7M Images of 2,959 Crop and Weed Species — Jain et al. (2025) (arXiv:2503.20068, 2025)

What this evaluates

Evaluates multi-class image classification capabilities for agricultural species, genus, family, and crop/weed distinction. Probes fine-grained visual recognition and taxonomic hierarchy understanding in plant identification.

Datasets

• iNatAg — total 2059000; splits: train (1544250), val (308850), test (205900); repo https://github.com/Project-AgML/AgML

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly predicted labels out of the total number of predictions. Computed per task (species, genus, family, crop/weed) and reported as a percentage.
• F1 Score — range: percent
  • Harmonic mean of precision and recall. Computed per task and reported as a percentage.

Input / output format

Input: RGB image resized to 384×384 or 224×224 pixels, optionally concatenated with a 32-dimensional embedding derived from normalized latitude and longitude coordinates.

Output: Predicted class index for species (2,059 classes), genus, family, or binary label for crop/weed (0=weed, 1=crop).

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100

def compute_f1(predictions, gold, num_classes):
    tp = fp = fn = 0
    for p, g in zip(predictions, gold):
        if p == g:
            tp += 1
        elif p != g:
            fp += 1
            fn += 1
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return (2 * precision * recall / (precision + recall)) * 100 if (precision + recall) > 0 else 0

Common pitfalls

• The experimental dataset caps images at 1,000 per species (2,059,000 total), which differs from the full 4.7M iNatAg release mentioned in the abstract.
• Genus and family metrics are derived by mapping species predictions to taxonomic groups via a lookup table, not trained directly, which can distort hierarchy accuracy if species predictions are incorrect.
• Geospatial inputs must be normalized and passed through a fully connected layer to produce a 32-dim embedding before concatenation; raw coordinates cannot be fed directly.

Evidence (verbatim from paper)

All trained models were evaluated on a test set consisting of 205,900 images. Evaluation included both fine-grained species classification and binary crop/weed classification, with metrics such as accuracy, precision, recall, and F1 score. To assess taxonomic generalization, genus and family-level accuracy were computed by mapping species predictions to their respective higher-order groups using a predefined lookup table.

Citation

@misc{jain2025inatag,
  title={iNatAg: Multi-Class Classification Models Enabled by a Large-Scale Benchmark Dataset with 4.7M Images of 2,959 Crop and Weed Species},
  author={Jain et al. (2025)},
  year={2025},
  note={arXiv:2503.20068}
}

• arXiv: 2503.20068