Eval Dataset Multilingual Prompts

Eval Dataset — Multilingual Prompts

Scope

~50 prompts across English, Arabic (MSA, Levantine, Gulf), French (Lebanese-French and standard), mixed AR/EN, and explicit translation/bilingual-formatting requests. Tests the full multilingual pipeline from language detection through output generation. Correct language handling is a hard requirement for MENA legal practice — a lawyer who writes in Arabic and gets an English response has a broken product experience.

Key metric: language-match rate ≥ 95% (output language matches input language, unless user explicitly requests otherwise).

Storage: eval/datasets/multilingual-prompts.jsonl

How to use this pack

1. Load into [[eval-benchmark-runner]] with [[eval-rubric-language-quality-ar]] and [[eval-rubric-language-quality-en]] as scoring rubrics.
2. For each response, run automated language detection on the output and compare to the input language.
3. Compute language_match_rate = (correct_language_responses / total).
4. For Arabic outputs, submit a sample to a human Arabic legal reviewer quarterly.
5. Feed results to [[eval-regression-detector]].

Categories

Category 1 — Arabic-only inputs (~12 prompts)

Test that Arabic input produces Arabic output with correct legal terminology.

MSA (Modern Standard Arabic) — formal legal register:

• "أعدّ لي عقد عمل بموجب قانون العمل الإماراتي." (Draft an employment contract under UAE Labour Law.)
• "ما هي شروط اتفاقية عدم الإفصاح في القانون اللبناني؟" (What are the NDA requirements under Lebanese law?)
• "راجع هذا البند وحدد المخاطر القانونية." (Review this clause and identify legal risks.)

Levantine Arabic (Lebanese dialect) — client-facing register:

• "بدي تعاقد عمل للبنان، شو بدك مني؟" (I want an employment contract for Lebanon, what do you need from me?)
• "هالعقد مظبوط؟ شو في غلط فيه؟" (Is this contract correct? What's wrong with it?)

Gulf Arabic (UAE/KSA dialect):

• "أبغى أسوي عقد NDA للسعودية." (I want to make an NDA for Saudi Arabia.)
• "وش الفرق بين عقد العمل في الإمارات وفي المملكة؟" (What's the difference between employment contracts in UAE vs KSA?)

Expected behavior: Output in Arabic (MSA preferred for legal documents, dialect acceptable for conversational clarifications); legal terminology must be accurate (مكافأة نهاية الخدمة not just "gratuity transliterated"; اتفاقية عدم الإفصاح not "NDA in Arabic letters").

Category 2 — French-only inputs (~10 prompts)

Lebanese-French (legal-professional register):

• "Rédigez un contrat de travail conforme au Code du travail libanais." (Draft an employment contract compliant with the Lebanese Labour Code.)
• "Quelle est la durée maximale de la période d'essai au Liban?" (What is the maximum probation period in Lebanon?)
• "Vérifiez cette clause de confidentialité pour un accord soumis au droit français." (Review this confidentiality clause for a French-law agreement.)

Standard French (France / EU):

• "Rédigez un NDA selon le droit français."
• "Expliquez les règles RGPD applicables à ce contrat de traitement de données."

Expected behavior: Output in French; legal terms in French (clause de confidentialité, rupture conventionnelle, période d'essai).

Category 3 — Mixed Arabic-English inputs (~10 prompts)

Common in MENA legal practice: a message that switches languages mid-sentence.

• "Review هذا العقد and tell me what's missing." (English request with Arabic object)
• "أريد NDA لـ DIFC — what are the key clauses?" (Arabic request with English terms)
• "هل الـ force majeure clause مناسبة للعقود الإماراتية؟" (Arabic question with English legal term)

Expected behavior: Respond in the dominant language of the prompt (usually Arabic if the grammatical structure is Arabic). Do not mix languages in the response unless the question specifically asks for it.

Category 4 — Bilingual document requests (~10 prompts)

Explicit requests for side-by-side bilingual documents:

• "Draft an NDA with Arabic on the left and English on the right."
• "أعطني عقد العمل بالعربي والإنجليزي جنب بعض." (Give me the employment contract in Arabic and English side by side.)
• "I need a bilingual lease agreement (AR/EN) for a UAE property — Arabic is the controlling language."

Expected behavior: Output formatted in two columns or clearly alternating sections; "controlling language" statement included (Arabic version controls); legal terminology consistent between both versions.

Category 5 — Translation requests (~8 prompts)

• "Translate this English NDA clause into Arabic."
• "ترجم هذه الفقرة من العربية إلى الإنجليزية." (Translate this paragraph from Arabic to English.)
• "Translate this NDA governing law clause from French to English."

Expected behavior: Accurate legal translation (not machine-literal); terminology matches the target jurisdiction's conventions.

Scoring dimensions

Caveats & currency

• Arabic legal dialect varies by country; Gulf Arabic and Levantine Arabic are distinct. The product targets legal professionals who primarily write MSA — but intake may be in dialect.
• French legal vocabulary in Lebanon differs slightly from Metropolitan French (Lebanese lawyers use Code de la Route, Code des Obligations et des Contrats, etc.).
• Automated language detection tools (langdetect, fastText) struggle with short Arabic inputs and mixed text. Human review of a 10% sample each quarter is necessary.

Related skills

• [[eval-benchmark-runner]] — orchestrates this dataset
• [[eval-rubric-language-quality-ar]] — Arabic quality scoring rubric
• [[eval-rubric-language-quality-en]] — English quality scoring rubric
• [[eval-regression-detector]] — tracks language-match rate across deployments