# Legal Constitution Eval

> Evaluates a fine-tuned open-source language model on its ability to perform keyword extraction, summarization, and sentiment analysis on the Indian Constitution. The protocol tests whether domain-specific fine-tuning improves the model's capacity to grasp nuanced legal semantics and structural elements. Use when the user wants to benchmark on Indian Constitution, or asks about evaluating this task. Reports precision, recall, and F1 score.

- Skill: `qhjqhj00/legal-constitution-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/legal-constitution-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/legal-constitution-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/legal-constitution-eval

---


# legal-constitution-eval

> Leveraging open-source models for legal language modeling and analysis: a case study on the Indian constitution — Gupta et al. (2024) (arXiv:2404.06751, 2024)

## What this evaluates

Evaluates a fine-tuned open-source language model on its ability to perform keyword extraction, summarization, and sentiment analysis on the Indian Constitution. The protocol tests whether domain-specific fine-tuning improves the model's capacity to grasp nuanced legal semantics and structural elements.

## Datasets

- **Indian Constitution** — total ?; splits: test (-1)

## Metrics

- `precision, recall, and F1 score` **(primary)** — range: [0, 1]
  - Standard precision, recall, and F1 score metrics used to evaluate keyword extraction, summarization, and sentiment analysis tasks.

## Input / output format

**Input**: Text segments extracted from the Indian Constitution PDF.

**Output**: Extracted keywords, generated summaries, or sentiment labels for each segment.

## Scoring recipe

```python
def evaluate(predictions, gold):
    prec = calculate_precision(predictions, gold)
    rec = calculate_recall(predictions, gold)
    f1 = 2 * (prec * rec) / (prec + rec)
    return prec, rec, f1
```

## Common pitfalls

- The evaluation lacks specific dataset splits, annotation guidelines, and quantitative results for the three tasks.
- No baseline models or statistical significance tests are provided to contextualize the reported performance.
- The pipeline is tested only on the Indian Constitution, limiting claims about generalizability to other legal domains.

## Evidence (verbatim from paper)

> performance evaluated via precision, recall, and F1 score, demonstrating effective extraction of legal insights from structured, complex legal texts through fine-tuned transformer-based processing.

## Citation

```bibtex
@misc{gupta2024legalconstitution,
  title={Leveraging open-source models for legal language modeling and analysis: a case study on the Indian constitution},
  author={Gupta et al. (2024)},
  year={2024},
  note={arXiv:2404.06751}
}
```

- arXiv: 2404.06751