Esv Intervention Eval

esv-intervention-eval

Leveraging AI-Generated Emotional Self-Voice to Nudge People towards their Ideal Selves — Fang et al. (2024) (arXiv:2409.11531, 2024)

What this evaluates

Evaluates the psychological and behavioral impact of an AI-generated emotional self-voice intervention compared to text-only and control conditions on goal-related resilience, confidence, motivation, and emotional states.

Datasets

• Custom Human-Subject Intervention Dataset — total ?; splits: pre-intervention (-1), post-intervention (-1)

Metrics

• VADER compound score — range: [-1, 1]
  • Normalized, weighted composite score of sentiment polarity for free-text responses, ranging from -1 (negative) to 1 (positive).
• Self-report questionnaire scores (primary) — range: [1, 7]
  • Likert-scale ratings (1-7) measuring psychological constructs including valence, arousal, resilience, confidence, motivation, commitment, vividness, similarity, and positive affect.

Input / output format

Input: Participants receive scenario prompts (goal failure or habit struggle) and are assigned to Control, Text, or ESV conditions. They complete pre-intervention and post-intervention questionnaires and provide free-text reflections.

Output: Numeric Likert-scale ratings for psychological outcomes and free-text responses for sentiment analysis.

Scoring recipe

# For continuous outcomes (valence, arousal, resilience):
lme_model = lmer(outcome ~ Test * Condition + covariates + (1|Subject), data=df)
p_val_lme = summary(lme_model).p_value

# For collinear outcomes (confidence, motivation, commitment):
df_art = aligned_rank_transform(df, by=['Test', 'Condition'])
anova_model = aov(outcome ~ Test * Condition, data=df_art)
p_val_anova = anova_model.p_value

# For text responses:
vader_score = vader_sentiment(text_response).compound

Common pitfalls

• Ignoring repeated-measures structure: Standard ANOVA/t-tests on pre/post data violate independence assumptions; LME or ART-ANOVA is required.
• Overlooking collinearity: Pre/post and condition effects were perfectly separated for confidence/motivation/commitment, necessitating Aligned Rank Transform instead of standard LME.
• Misinterpreting VADER: VADER captures general sentiment polarity, not specific psychological constructs, so it should not be directly equated with questionnaire scores.

Evidence (verbatim from paper)

For the quantitative data, we used a linear mixed effects (LME) model (lme4 package in R [6]) to account for the repeated measurement nature of the data, namely each subject has 2 observations: pre-intervention and post-intervention. ... The main outcomes are valence, arousal, resilience, confidence, motivation, and commitment. ... We also used VADER [41] to quantitatively capture the sentiment in the free responses.

Citation

@misc{fang2024leveraging,
  title={Leveraging AI-Generated Emotional Self-Voice to Nudge People towards their Ideal Selves},
  author={Fang et al. (2024)},
  year={2024},
  note={arXiv:2409.11531}
}

• arXiv: 2409.11531