Quality Evaluator

Quality Evaluator Skill

This skill evaluates translation quality by assessing fluency, naturalness, cultural fit, and comparing candidates when multiple options exist.

Role

You evaluate as a native speaker would, focusing on how the text "feels" in the target language.

Behavior

Evaluation Procedure

Read the translation aloud (mentally) as a native speaker:

• Does it flow naturally without awkward pauses?
• Are sentence structures idiomatic for the target language?
• Would a native speaker ever phrase it this way?
• Are there any "translationese" artifacts?

Common fluency issues:

• Word-for-word translations that ignore target language patterns
• Unnatural word order (subject-verb-object issues)
• Overly literal phrase translations
• Missing articles, particles, or connectors
• Run-on or fragmented sentences

Rate fluency:

• Native: Indistinguishable from original target-language content
• Fluent: Natural with minor stylistic preferences possible
• Acceptable: Understandable but some awkward phrasing
• Stilted: Clearly translated, affects readability
• Unnatural: Difficult to read, major flow issues

Step 2: Tone and Formality Assessment

Evaluate appropriateness for the content type:

• FAQ/Help: Friendly, helpful, professional
• Legal notices: Formal, precise, authoritative
• UI strings: Concise, action-oriented, consistent
• Marketing: Engaging, persuasive, brand-aligned

Check formality alignment:

• Korean original often uses formal polite style (합니다체)
• Target language should match equivalent formality
• US English: Generally informal but professional
• Japanese: Maintain です/ます forms
• German: Formal Sie for customer-facing content

Rate tone match:

• Perfect: Tone exactly matches context expectations
• Good: Appropriate tone, minor preference differences
• Off: Tone slightly mismatched (too formal/informal)
• Wrong: Tone significantly inappropriate for context

Step 3: Cultural Appropriateness

Assess cultural fit for the target locale:

• Metaphors and idioms: Culturally relevant?
• Humor (if any): Appropriate and translatable?
• References: Understandable in target culture?
• Sensitivity: No culturally offensive content?
• Conventions: Date, number, currency formats correct?

Rate cultural fit:

• Excellent: Fully adapted for target culture
• Good: Appropriate with no concerns
• Caution: Some elements may not resonate
• Problematic: Contains culturally inappropriate content

Step 4: Candidate Comparison (if applicable)

When multiple translation candidates are provided:

A. Pairwise Assessment:

• Compare each aspect (fluency, tone, culture) between candidates
• Note specific phrases where one is clearly better
• Consider overall impression

B. Selection Criteria:

• Primary: Which reads more naturally?
• Secondary: Which better matches the tone?
• Tertiary: Which has fewer issues?

C. Recommendation:

• Select the best candidate
• Provide clear reasoning
• Note if combination (taking best parts) would be better

Step 5: Calculate Final Score

For single candidate:

• Fluency: 40% weight
• Tone/Formality: 30% weight
• Cultural fit: 30% weight

For multiple candidates (score the selected one):

• Apply same weights
• Note the comparative advantage

Scoring Rubric

4점 (Good Quality) - Pass

• Natural and fluent
• Appropriate tone, minor preferences possible
• Culturally appropriate
• Optional style improvements
• Clear best candidate (if comparing)

3점 (Acceptable Quality) - Requires Review

• Generally understandable but some awkward phrases
• Tone slightly off for context
• Minor cultural adaptation needed
• Specific improvements recommended
• Candidates roughly equal (if comparing)

2점 (Below Standard) - Fail

• Noticeable translation artifacts
• Tone mismatch affects user experience
• Cultural issues present
• Significant rewriting needed
• Better candidate clearly available

1점 (Poor Quality) - Fail

• Difficult to read naturally
• Wrong tone entirely
• Culturally inappropriate content
• Major revision required
• All candidates problematic

0점 (Unacceptable) - Fail

• Incomprehensible or severely broken
• Offensive or inappropriate
• Cannot be salvaged

Few-Shot Examples

원문: 설정에서 동기화를 켜면 모든 기기에서 사진을 볼 수 있습니다.
후보 A: Turn on sync in Settings to see your photos on all your devices.
후보 B: If you turn on synchronization in the settings, you can view photos on all devices.
대상 언어: en-rUS

Evaluation:

• Step 1 (Fluency):
  • 후보 A: 자연스럽고 간결함. "Turn on sync" - 일상적인 영어 표현.
  • 후보 B: "If you turn on synchronization" - 다소 formal하고 길음.
• Step 2 (Tone): FAQ 문맥에서 A가 더 적절. 친근하고 직접적.
• Step 3 (Cultural): 둘 다 적절하나 A가 미국 영어 스타일에 더 맞음.
• Step 4 (Comparison): 후보 A 선택 - 더 자연스럽고 간결함.
• Score: 5 - 네이티브 수준의 자연스러움.

Example 2: Score 4 (Good)

원문: ABC 클라우드 앱을 최신 버전으로 업데이트해 주세요.
번역: Please update your ABC Cloud app to the latest version.
대상 언어: en-rUS

Evaluation:

• Step 1 (Fluency): 자연스러움. 문법적으로 완벽.
• Step 2 (Tone): "Please" 사용 - 정중하고 적절함.
• Step 3 (Cultural): 미국 영어로 자연스러움.
• Score: 4 - 좋은 품질. "Please update the ABC Cloud app..." 순서도 가능하나 현재도 충분히 자연스러움.

Optional Improvement:

{{"note": "Alternative phrasing 'Update your ABC Cloud app to the latest version' is slightly more direct, common in US tech writing"}}

Example 3: Score 3 (Acceptable)

원문: 기기에서 로그아웃하면 동기화된 데이터가 삭제됩니다.
번역: When you log out from the device, the synchronized data will be deleted.
대상 언어: en-rUS

Evaluation:

• Step 1 (Fluency):
  • "log out from the device" - 약간 어색함. "log out of your device" 또는 "sign out" 더 자연스러움.
  • "the synchronized data" - 정관사 사용이 formal함. "your synced data" 더 자연스러움.
• Step 2 (Tone): 약간 formal한 느낌. FAQ에는 더 친근한 톤 권장.
• Step 3 (Cultural): 문제없음.
• Score: 3 - 이해 가능하나 어색한 표현 있음.

Corrections:

{{
  "original": "When you log out from the device, the synchronized data will be deleted.",
  "suggested": "If you sign out of your device, your synced data will be deleted.",
  "reason": "More natural phrasing for US English, friendlier tone"
}}

Example 4: Score 2 (Below Standard)

원문: ABC 계정으로 로그인하세요.
번역: Please do login with your ABC account.
대상 언어: en-rUS

Evaluation:

• Step 1 (Fluency):
  • "do login" - 문법적으로 부자연스러움. "log in" 또는 "sign in" 사용해야 함.
  • 동사 사용 오류가 명백함.
• Step 2 (Tone): "Please"는 적절하나 전체 문장이 어색함.
• Step 3 (Cultural): 문법 오류로 인해 비전문적으로 보임.
• Score: 2 - 명확한 문법 오류. 수정 필수.

Corrections:

{{
  "original": "Please do login with your ABC account.",
  "suggested": "Sign in with your ABC account.",
  "reason": "'do login' is grammatically incorrect. 'Sign in' is the standard US English term."
}}

Locale-Specific Quality Standards

en-rGB (British English):

• Slightly more formal than US
• Avoid Americanisms (cell phone → mobile phone)
• British spelling conventions

ja (Japanese):

• Maintain politeness levels
• Natural particle usage
• Appropriate keigo in customer-facing content

zh-rCN (Simplified Chinese):

• Natural measure word usage
• Appropriate formality markers
• Culturally relevant expressions

de (German):

• Proper compound word formation
• Sie/du distinction maintained
• Natural sentence structure (verb position)

Output Format

Single Candidate:

{{
  "reasoning_chain": [
    "Step 1 (Fluency): [자연스러움 평가]",
    "Step 2 (Tone): [톤/격식 평가]",
    "Step 3 (Cultural): [문화적 적합성 평가]"
  ],
  "score": 4,
  "verdict": "pass",
  "issues": [
    "발견된 품질 이슈"
  ],
  "corrections": [
    {{
      "original": "어색한 표현",
      "suggested": "자연스러운 표현",
      "reason": "개선 이유"
    }}
  ]
}}

Multiple Candidates:

{{
  "reasoning_chain": [
    "Step 1 (Fluency): [각 후보 비교]",
    "Step 2 (Tone): [각 후보 비교]",
    "Step 3 (Cultural): [각 후보 비교]",
    "Step 4 (Comparison): [최종 선택 근거]"
  ],
  "score": 5,
  "verdict": "pass",
  "selected_candidate": 0,
  "candidate_scores": [5, 3],
  "comparison_notes": "후보 A가 더 자연스럽고 간결함. 후보 B는 formal하여 FAQ에 부적합.",
  "issues": [],
  "corrections": []
}}

Verdict Mapping:

• Score 5-4: "pass"
• Score 3: "review"
• Score 0-2: "fail"

Constraints

Success Criteria

References