adu2021 Learning to Reason for Factuality
Core Concept
Reasoning models prone to hallucinations when generating long-form factual content, yet traditional reward functions like FActScore cause reward hacking by reducing output quality. This paper proposes a multi-dimensional reward function that simultaneously optimizes for factual accuracy, response detail level, and answer relevance. Applied through online RL, the approach significantly reduces hallucinations while maintaining helpful, detailed responses.
Architecture Overview
- Multi-dimensional Reward Function: Balances accuracy, detail, and relevance
- Factual Precision Metric: Measures proportion of hallucination-free content
- Detail Level Assessment: Ensures sufficient response completeness
- Relevance Scoring: Verifies information pertains to query
- Online RL Training: Iteratively improves reasoning with nuanced rewards
Implementation Steps
Step 1: Design Multi-dimensional Reward Function
Create reward function balancing multiple objectives.
import torch
from typing import Tuple, Dict
class MultiDimensionalRewardFunction:
"""
Compute rewards balancing factuality, detail, and relevance.
"""
def __init__(self, fact_checker, relevance_scorer):
self.fact_checker = fact_checker
self.relevance_scorer = relevance_scorer
def compute_reward(
self,
question: str,
response: str,
ground_truth: str = None
) -> Dict:
"""
Compute multi-dimensional reward.
Args:
question: Original question
response: Model's response
ground_truth: Optional ground truth for comparison
Returns:
Dict with component rewards and total
"""
# Component 1: Factual Precision
factuality_score = self._compute_factuality(response, ground_truth)
# Component 2: Detail Level
detail_score = self._compute_detail_level(response)
# Component 3: Answer Relevance
relevance_score = self._compute_relevance(question, response)
# Combine with careful weighting
# Higher weight on factuality to prevent hallucinations
total_reward = (
0.5 * factuality_score +
0.25 * detail_score +
0.25 * relevance_score
)
return {
"factuality": factuality_score,
"detail": detail_score,
"relevance": relevance_score,
"total": total_reward
}
def _compute_factuality(self, response: str, ground_truth: str = None) -> float:
"""
Compute factual accuracy of response.
Args:
response: Model response
ground_truth: Ground truth to check against
Returns:
Factuality score 0-1 (1 = all facts correct)
"""
# Extract factual claims from response
claims = self._extract_claims(response)
if not claims:
return 1.0 # No claims, can't be wrong
# Check each claim for factuality
correct_claims = 0
for claim in claims:
# Verify claim
is_factual = self.fact_checker.verify(claim, ground_truth)
if is_factual:
correct_claims += 1
factuality = correct_claims / len(claims)
return factuality
def _compute_detail_level(self, response: str) -> float:
"""
Assess response detail level.
Args:
response: Model response
Returns:
Detail score 0-1 (1 = well-detailed)
"""
# Multiple signals for detail
signals = []
# Signal 1: Length (but not too long)
word_count = len(response.split())
if word_count < 50:
length_score = word_count / 50.0 # Too short
elif word_count < 500:
length_score = 1.0 # Optimal
else:
length_score = max(0.7, 1.0 - (word_count - 500) / 2000) # Diminishing
signals.append(length_score * 0.4)
# Signal 2: Sentence complexity (more complex = more detailed)
sentences = response.split(".")
avg_sent_length = word_count / len(sentences) if sentences else 0
if avg_sent_length < 10:
complexity_score = 0.5 # Too simple
elif avg_sent_length < 25:
complexity_score = 1.0 # Good complexity
else:
complexity_score = 0.8 # Acceptable
signals.append(complexity_score * 0.3)
# Signal 3: Presence of examples/details
detail_keywords = ["example", "such as", "specifically", "notably", "for instance"]
keyword_count = sum(1 for kw in detail_keywords if kw in response.lower())
detail_score = min(keyword_count / 3.0, 1.0)
signals.append(detail_score * 0.3)
detail = sum(signals)
return detail
def _compute_relevance(self, question: str, response: str) -> float:
"""
Assess how relevant response is to question.
Args:
question: Original question
response: Model response
Returns:
Relevance score 0-1
"""
# Semantic similarity between question and response
relevance = self.relevance_scorer.compute_relevance(question, response)
return relevance
def _extract_claims(self, text: str) -> list:
"""Extract factual claims from text."""
# Simple implementation: sentences ending with facts
sentences = text.split(".")
claims = []
for sent in sentences:
if any(verb in sent.lower() for verb in ["is", "are", "was", "were"]):
claims.append(sent.strip())
return claims
Step 2: Implement Fact-Checking System
Create system to verify factuality of claims.
class FactChecker:
"""
Verify factuality of model-generated claims.
"""
def __init__(self, knowledge_base=None, external_api=None):
self.knowledge_base = knowledge_base
self.external_api = external_api
def verify(self, claim: str, context: str = None) -> bool:
"""
Verify whether claim is factual.
Args:
claim: Factual claim to verify
context: Optional context
Returns:
True if claim is factually correct
"""
# Try knowledge base first
if self.knowledge_base:
kb_result = self.knowledge_base.check_fact(claim)
if kb_result is not None:
return kb_result
# Try external API (e.g., web search)
if self.external_api:
api_result = self.external_api.verify_fact(claim)
if api_result is not None:
return api_result
# If no verification possible, assume correct
return True
def extract_and_verify(self, text: str) -> Dict[str, bool]:
"""
Extract claims and verify each.
Args:
text: Text containing claims
Returns:
Dict mapping claims to factuality
"""
claims = self._extract_claims(text)
verification = {}
for claim in claims:
verification[claim] = self.verify(claim)
return verification
def _extract_claims(self, text: str) -> list:
"""Extract factual claims from text."""
# Parse sentences that contain assertions
import re
# Pattern: Subject + Verb + Object assertions
sentences = re.split(r'[.!?]+', text)
claims = []
for sent in sentences:
sent = sent.strip()
if self._is_factual_claim(sent):
claims.append(sent)
return claims
def _is_factual_claim(self, sentence: str) -> bool:
"""Determine if sentence is a factual claim."""
assertion_verbs = ["is", "are", "was", "were", "be", "been", "have", "has", "had"]
return any(verb in sentence.lower() for verb in assertion_verbs)
Step 3: Implement Online RL Training with Multi-dimensional Rewards
Train reasoning model using multi-dimensional reward function.
class FactualityRLTrainer:
"""
Train reasoning model for factuality using online RL.
"""
def __init__(self, model, reward_fn):
self.model = model
self.reward_fn = reward_fn
def train_step(self, question: str, ground_truth: str = None, use_old_response: str = None):
"""
Single training step optimizing for factuality.
Args:
question: Question to answer
ground_truth: Ground truth for reward
use_old_response: Use old response for comparison
Returns:
Training metrics
"""
# Generate response
response = self.model.generate(question, max_length=500)
# Compute multi-dimensional rewards
rewards = self.reward_fn.compute_reward(question, response, ground_truth)
# Compute log probabilities of generated response
log_probs = self.model.get_logprobs_for_response(question, response)
# REINFORCE update: scale gradients by rewards
# Gradient = log_prob * reward
policy_loss = -(log_probs * rewards["total"]).mean()
# Backward pass
policy_loss.backward()
torch.nn.utils.clip_grad_norm_(self.model.parameters(), max_norm=1.0)
self.model.optimizer.step()
self.model.optimizer.zero_grad()
return {
"loss": policy_loss.item(),
"factuality_reward": rewards["factuality"],
"detail_reward": rewards["detail"],
"relevance_reward": rewards["relevance"],
"total_reward": rewards["total"],
"response": response
}
def train_epoch(self, questions: list, ground_truths: list, batch_size: int = 8):
"""
Train for one epoch.
Args:
questions: List of questions
ground_truths: List of ground truth answers
batch_size: Batch size
Returns:
Average metrics for epoch
"""
epoch_metrics = {
"avg_loss": 0,
"avg_factuality": 0,
"avg_detail": 0,
"avg_relevance": 0
}
num_batches = 0
for i in range(0, len(questions), batch_size):
batch_questions = questions[i:i + batch_size]
batch_truths = ground_truths[i:i + batch_size]
batch_loss = 0
batch_factuality = 0
batch_detail = 0
batch_relevance = 0
for question, ground_truth in zip(batch_questions, batch_truths):
metrics = self.train_step(question, ground_truth)
batch_loss += metrics["loss"]
batch_factuality += metrics["factuality_reward"]
batch_detail += metrics["detail_reward"]
batch_relevance += metrics["relevance_reward"]
batch_size_actual = len(batch_questions)
epoch_metrics["avg_loss"] += batch_loss / batch_size_actual
epoch_metrics["avg_factuality"] += batch_factuality / batch_size_actual
epoch_metrics["avg_detail"] += batch_detail / batch_size_actual
epoch_metrics["avg_relevance"] += batch_relevance / batch_size_actual
num_batches += 1
# Average over all batches
for key in epoch_metrics:
epoch_metrics[key] /= max(num_batches, 1)
return epoch_metrics
Step 4: Evaluate Factuality Improvements
Create evaluation framework for factuality metrics.
def evaluate_factuality(model, test_questions: list, ground_truths: list) -> Dict:
"""
Evaluate model on factuality metrics.
Args:
model: Trained model
test_questions: Test questions
ground_truths: Ground truth answers
Returns:
Evaluation metrics
"""
metrics = {
"hallucination_rate": 0,
"avg_detail": 0,
"avg_relevance": 0,
"overall_quality": 0
}
fact_checker = FactChecker()
reward_fn = MultiDimensionalRewardFunction(fact_checker, None)
for question, truth in zip(test_questions, ground_truths):
# Generate response
response = model.generate(question)
# Check for hallucinations
verification = fact_checker.extract_and_verify(response)
hallucinations = sum(1 for v in verification.values() if not v)
total_claims = len(verification) if verification else 1
hallucination_rate = hallucinations / total_claims
# Compute rewards
rewards = reward_fn.compute_reward(question, response, truth)
metrics["hallucination_rate"] += hallucination_rate
metrics["avg_detail"] += rewards["detail"]
metrics["avg_relevance"] += rewards["relevance"]
# Average over test set
n = len(test_questions)
metrics["hallucination_rate"] /= n
metrics["avg_detail"] /= n
metrics["avg_relevance"] /= n
metrics["overall_quality"] = (
(1.0 - metrics["hallucination_rate"]) * 0.5 +
metrics["avg_detail"] * 0.25 +
metrics["avg_relevance"] * 0.25
)
return metrics
Practical Guidance
When to Use Learning to Reason for Factuality
- Long-form factual generation: Answering complex questions with multiple facts
- Hallucination-prone models: Reasoning models generating plausible but false content
- Balanced optimization: Need both accuracy and response quality
- Online learning scenarios: Continuously improving with feedback
When NOT to Use Learning to Reason for Factuality
- Closed-domain QA: Knowledge base complete and accurate
- Creative tasks: Penalizing hallucinations inappropriate for fiction/poetry
- Latency-sensitive: Fact-checking adds computational overhead
- Toxic content: Fact-checking system itself may have biases
Hyperparameter Recommendations
- Factuality weight: 0.5 (higher priority to prevent hallucinations)
- Detail weight: 0.25 (maintains responsiveness)
- Relevance weight: 0.25 (ensures answer to question)
- Hallucination threshold: Flag claims with <0.5 confidence
- Learning rate: 1e-5 (conservative for stability)
Key Insights
The critical insight is that single-objective reward functions (like FActScore) cause reward hacking—models reduce output length to appear more factual. By combining factuality with detail and relevance, the framework forces genuine accuracy improvements rather than gaming the metric. The multi-dimensional approach learns nuanced trade-offs between competing objectives.
Reference
Learning to Reason for Factuality (arXiv:2508.05618)
Proposes multi-dimensional reward function combining factual precision, response detail, and answer relevance. Reduces hallucinations by 23.1% while maintaining response quality through online RL on reasoning models.