qhjqhj00 embedded-ai-companion-eval
Embedded AI Companion System on Edge Devices — Gupta et al. (2026) (arXiv:2601.08128, 2026)
What this evaluates
Evaluates the conversational quality, memory retrieval, personalization, and long-term memory extraction capabilities of an edge-deployed AI companion system over simulated multi-session interactions.
Datasets
- Synthetic User Simulation — total ?; splits: test (-1)
Metrics
Conversation Quality(primary) — range: other- GPT-5 rates the AI's responses on a scale (observed range ~1-4.2) for naturalness and personalization based on the full conversation history across sessions.
Generated QA— range: percent- GPT-5 scores answers as correct (+1), partially correct (+0.5), or incorrect (0). The final score is the percentage of total points earned relative to the number of questions in each category (Specific, Inferred).
Personalization Hits/Misses— range: other- GPT-5 counts 'hits' (referrals to past information) and 'misses' (forgotten or misremembered information) in a new short conversation session prompted on a previously discussed topic.
Extraction Quality— range: percent- GPT-5 evaluates extracted memories against ground truth characterization and chat history across three sub-metrics: Correctness (factual accuracy %), Coverage (extraction rate of characterization), and Completeness (extraction rate of conversation-present information).
Input / output format
Input: Full conversation history across multiple simulated sessions, user characterizations, generated QA prompts, and new conversation transcripts on past topics.
Output: AI companion dialogue turns, extracted memory profiles, and QA responses. Evaluation yields numerical scores or percentages via GPT-5 rubrics.
Scoring recipe
def score_conversation_quality(history):
return gpt5_judge(history, rubric='naturalness/personalization')
def score_generated_qa(history, questions):
total = sum(gpt5_judge(ans, history, options={'+1':'correct', '+0.5':'partial', '0':'incorrect'}) for ans in [model.respond(history, q) for q in questions])
return (total / len(questions)) * 100
def score_personalization(history, new_session):
hits = gpt5_judge(new_session, history, action='count_referrals')
misses = gpt5_judge(new_session, history, action='count_misremembered')
return hits, misses
def score_extraction(memories, profile, history, ground_truth):
correctness = gpt5_judge(memories, ground_truth, rubric='factual_accuracy')
coverage = gpt5_judge(memories, ground_truth, rubric='extraction_rate')
completeness = gpt5_judge(memories, history, rubric='extraction_rate_present_only')
return correctness, coverage, completeness
Common pitfalls
- Uses GPT-5 for both user simulation and evaluation, risking self-reinforcing bias.
- Personalization 'hits' count multiple referrals to the same past fact as separate instances, inflating scores.
- Extraction metrics are only applicable to the memory-augmented system, not baselines.
- Synthetic user characterizations and scripted topic switches may not reflect real-world conversational dynamics.
Evidence (verbatim from paper)
We evaluate our system and comparison baselines along 3 dimensions: (i.) Conversation Quality (overall quality of AI responses across the conversation), (ii.) Generated QA (quality of AI answers to specific questions we generate based on the conversation), (iii.) Personalization (amount of demonstrated examples of personalization during a short new conversation session).
Citation
@misc{gupta2026embedded,
title={Embedded AI Companion System on Edge Devices},
author={Gupta et al. (2026)},
year={2026},
note={arXiv:2601.08128}
}
- arXiv: 2601.08128