Sentiment Simca

Skill: sentiment-simca

Triggers

• "sentiment and simca analysis tool"
• "sentiment simca"
• "merge comments with posts"
• "create comment context file"
• "build comment post file"
• "comment sentiment"
• "join comments to posts"

What this skill does

Three-phase pipeline — one master file throughout:

1. Merge (Steps 1–5): joins comments to their parent posts so each comment row carries the post text.
2. Sentiment (Steps 6–9): classifies every comment as resonates / deflects / neutral by axis-specific rules.
3. SIMCA (Steps 10–13): classifies resonates-only comments with the 5-tag SIMCA framework (AMV, OGN, CE, DM, NEC). Results are written back into the same master file.

PHASE 1 — Merge comments with posts

Step 1 — Confirm the folder path

Ask the analyst:

Which period's posts\ folder should I use? Default: I:\טאפ\Impact\bi monthly reports\Period 2_May-June\posts (Press Enter to use the default, or paste a different path.)

Wait for the analyst to confirm or provide a path. Store it as FOLDER.

Step 2 — List available files

Run:

python "C:\Users\Yael Mabat\discourse analysis tool\merge_comments_with_posts.py" "<FOLDER>" --list

This prints all .xlsx files in the folder. Try to auto-identify:

• Posts file: a file whose name contains "posts" (case-insensitive)
• Comments file: any other file

If exactly one file matches each role, proceed to Step 3 without asking.

If the script output is ambiguous (zero or multiple matches for either role), show the analyst the file list and ask which file is which.

Step 3 — Run the merge

python "C:\Users\Yael Mabat\discourse analysis tool\merge_comments_with_posts.py" "<FOLDER>" --posts "<POSTS_FILE>" --comments "<COMMENTS_FILE>"

Step 4 — Report results

Report: files used, comments matched, comments unmatched, output file path. If unmatched > 5% of total, flag it with a warning.

Step 5 — Done

Tell the analyst comments_with_post_context.xlsx is ready. Mention that the next step (classification) will read this file.

PHASE 2 — Interactive Classification Loop

Triggers for this phase:

• "classify comments"
• "start classification"
• "next batch"
• "categorize everything" (triggers full automated run — Step 9)

Step 6 — Confirm input and batch size

Confirm:

• Folder path (default: same as Step 1)
• That comments_with_post_context.xlsx exists
• Batch size (default: 20; analyst may say 50 or 100)

Store FOLDER, BATCH_SIZE, OFFSET = 0.

Step 7 — Read and classify a batch

Run Python to extract the next batch, including the Strategic Axis and Narrative Tag columns:

import pandas as pd, json
df = pd.read_excel(r"<FOLDER>\comments_with_post_context.xlsx")
batch = df.iloc[OFFSET : OFFSET + BATCH_SIZE]
rows = []
for i, row in batch.iterrows():
    rows.append({
        'row': i,
        'axis': str(row.get('Strategic Axis', '')),
        'narrative': str(row.get('Narrative Tag', '')),
        'post': str(row.get('Post Text', '')).replace('\n', ' ')[:200],
        'comment': str(row.get('text', '')).replace('\n', ' ')[:200]
    })
with open(r'C:\Users\YAELMA~1\AppData\Local\Temp\batch_current.json', 'w', encoding='utf-8') as f:
    json.dump(rows, f, ensure_ascii=False, indent=2)
print('done')

Read the JSON file, then classify each row using the axis-specific rules below.

Display as a markdown table:

Step 8 — Analyst review

Wait for the analyst to respond:

• "All good" / "next batch" → confirm all, advance OFFSET += BATCH_SIZE, go to Step 7.
• Corrections ("10 = deflects, 14 = neutral") → apply, record corrected examples, advance offset, go to Step 7.
• "Next 50" / "bigger batch" → change BATCH_SIZE, go to Step 7.

Keep a running tally: "Reviewed so far: X | Corrections this session: Y" Accumulate analyst corrections as examples to improve future batches.

Step 9 — Full automated run (only when analyst explicitly says so)

Triggers: "categorize everything", "full run", "tag everything", or similar.

1. Write classify_sentiment.py at the project root using all accumulated corrections as few-shot examples in the prompt. The script:

   • Reads comments_with_post_context.xlsx
   • Writes already-reviewed rows with confirmed labels directly (no API call needed)
   • Classifies remaining rows via Claude API (model: claude-haiku-4-5-20251001, batch 20)
   • Applies axis-specific rules per row based on the Strategic Axis column
   • Saves comments_with_sentiment_and_simca.xlsx with a label column (resonates / deflects / neutral)
   • Generates sentiment_chart.html (Chart.js — breakdown by axis, narrative, network)
   • Supports resume on interruption

2. Run the script and report: total classified, label breakdown (counts + %), output file paths.

Classification Rules by Axis

Labels (all axes)

• Resonates — comment amplifies the axis narrative
• Deflects — comment diverts from / contradicts the axis narrative
• Neutral — formulaic, factual, no signal

ציר כאוס (Chaos axis)

Key principle: the POST context determines the label, not just the comment text. The same generic distress comment ("אני בשוק", "מה קורה כאן") can be Resonates on one post and Deflects on another, depending on the post's political framing.

Resonates:

• Comment expresses distress, anger, or confusion AND the post amplifies chaos against the government (e.g. soldiers killed, state failures, societal breakdown)
• Blame is unattributed OR directed at the government / PM / ministers

Deflects:

• Explicitly blames leftists, opposition, Arabs, antisemitism
• Supports a right-wing figure or activist (e.g. Ziv Barnes)
• Expresses outrage/distress in reaction to a post that defends a right-wing figure — even if the comment sounds generic ("אני בשוק", "מה קורה כאן", "אנחנו האויבים של עצמנו")
• Defends the government / right-wing parties / haredim when a post criticizes them

Neutral:

• Formulaic condolence/well-wish: יהי זכרו/זכרה ברוך, רפואה שלמה, נקווה לטוב
• Purely factual / analytical with no emotional charge
• Tribute / eulogy with no blame or chaos signal
• "עצוב ביותר יהי זכרה ברוך" — sadness + condolence, no chaos signal → Neutral

Confirmed examples (from analyst review, Period 2):

Additional confirmed rules — ציר כאוס

Haredi violence posts: Comments expressing anger directed AT haredim (cursing them, condemning their violence, calling for removing their services) = Resonates. Reasoning: haredi violence is a chaos signal; the government enables it; anger at haredim amplifies the chaos narrative.

"Identity unknown" shooting posts (Arab victim test): When a post withholds a victim's identity (a known social media "test" of whether people will react differently to an Arab victim), meta-comments like "בוחנים אותי", "אני במבחן", "אלוהים מנסה אותי" = Neutral. They are self-aware commentary on their own reaction, not amplifying or deflecting the chaos narrative.

Solidarity with neglected communities (North, South): Comments expressing support/solidarity with communities in the North/South abandoned by the government = Resonates. They validate the chaos narrative of government neglect.

Sarcastic praise of government ("איזה כיף שהמדינה דואגת לנו"): Heavy sarcasm directed at the government's failure to protect = Resonates.

Humor/dismissal on war/security posts (Iran, attacks): Cynical jokes ("Season 2 Episode 4079"), exam-cancellation humor, "נו מה" = Neutral (no signal). Pure distress/tension ("אני במתח ברמות", "its over for me") = Resonates.

Other axes

Rules for additional axes will be added here as the analyst defines them during the review loop. Always ask the analyst for axis-specific rules before classifying a new axis for the first time.

PHASE 3 — SIMCA Classification

Triggers for this phase:

• "simca classification"
• "tag simca"
• "run simca"
• "simca batch"
• "categorize simca everything" (triggers full automated run — Step 13)

SIMCA is a multi-label framework based on the social psychology of collective action (van Zomeren et al. 2008, 2018; Kligler-Vilenchik et al. 2021). Unlike Phase 2 (single label per comment), SIMCA assigns one or more tags per comment.

Single-file design: SIMCA writes its columns directly into the same file as the sentiment labels. There is one master file — no separate SIMCA output file.

Resonates-only scope: SIMCA runs ONLY on rows where label == "resonates" (from Phase 2). Deflecting and neutral comments are already classified and excluded — they don't need SIMCA tagging. This also means the GOVERNMENT_SUPPORTER column is not needed: pro-government/deflecting comments never enter Phase 3.

Step 10 — Confirm input file and batch size

Ask the analyst which folder to use (default: same as Phase 1/2). The master file is comments_with_sentiment_and_simca.xlsx in that folder.

Batch size default: 20.

Store SIMCA_FILE, SIMCA_BATCH_SIZE, SIMCA_OFFSET = 0. Filter: df = df[df['label'] == 'resonates'].reset_index(drop=False)

Step 11 — Read and classify a batch

Extract the batch from resonates-only rows. Include post context — it helps distinguish which group is being named and whether outrage is political or personal:

import pandas as pd, json
df = pd.read_excel(r"<SIMCA_FILE>")
df = df[df['label'] == 'resonates'].reset_index(drop=False)
batch = df.iloc[SIMCA_OFFSET : SIMCA_OFFSET + SIMCA_BATCH_SIZE]
rows = []
for i, row in batch.iterrows():
    rows.append({
        'row': i,
        'post': str(row.get('Post Text', '')).replace('\n', ' ')[:150],
        'comment': str(row.get('text', '')).replace('\n', ' ')[:300]
    })
with open(r'C:\Users\YAELMA~1\AppData\Local\Temp\simca_batch_current.json', 'w', encoding='utf-8') as f:
    json.dump(rows, f, ensure_ascii=False, indent=2)
print('done')

Classify each comment using the SIMCA framework below. Display as a markdown table with one column per tag (✓ = applies, blank = does not):

Column key: AMV = ANGER_MORAL_VIOLATION, OGN = OUTGROUP_NAMING, CE = COLLECTIVE_EFFICACY, DM = DEMOBILIZATION, NEC = NOT_ENOUGH_CONTENT

Step 12 — Analyst review

Same loop as Phase 2:

• "All good" / "next batch" → confirm all, advance SIMCA_OFFSET += SIMCA_BATCH_SIZE, go to Step 11.
• Corrections ("5 = OUTGROUP_NAMING too", "12 = NOT_ENOUGH_CONTENT only") → apply, record.
• Keep running tally: "SIMCA reviewed: X | Corrections: Y"

Step 13 — Full automated SIMCA run (only when analyst explicitly says so)

Triggers: "categorize simca everything", "simca full run", "tag everything simca".

Write classify_simca.py at the project root:

• Reads the master file (comments_with_sentiment_and_simca.xlsx)
• Classifies each comment via Claude API (model: claude-haiku-4-5-20251001, batch 20)
• Uses the full SIMCA system prompt below (verbatim, with worked examples)
• Filters to label == "resonates" rows only before classifying
• Writes the following columns back into the same master file (in-place, resonates rows only):
  • simca_anger_moral_violation (1/0)
  • simca_outgroup_naming (1/0)
  • simca_collective_efficacy (1/0)
  • simca_demobilization (1/0)
  • simca_not_enough_content (1/0)
  • simca_justification
• Supports resume on interruption (skips rows already classified)

SIMCA Classification Rules (embed verbatim in classifier prompt)

Background

You are a research assistant helping with discourse analysis of Hebrew-language social media comments. Your task is to assign one or more tags to each comment based on the framework below. The framework is drawn from research on the social psychology of collective action and political mobilization (van Zomeren et al. 2008, 2018; Kligler-Vilenchik et al. 2021).

Tag 1: ANGER_MORAL_VIOLATION

Apply when the comment expresses affective injustice — anger, outrage, contempt, disgust — or frames an action/actor as a violation of moral beliefs, values, or rights. The key is emotional moral condemnation, not just disagreement.

Look for:

• Insults and condemnatory labels: "גנבים", "שקרנים", "חלאות", "נבלות", "פושעים", "מחבלים" (when used against fellow citizens), "פרימיטיביים", "חולי רוח"
• Moral-violation framing: "חילול השם", "בושה", "אין להם גבול", "פשע", "שוד", "שחיתות"
• Rage/moral outrage: "נשבר הלב" (when tied to injustice, not pure grief), "אי אפשר כבר איתם", "מגעיל"
• Rights/values violation: framing something as a violation of what is deserved, fair, or sacred
• Anger emotion words: "כועס", "זעם", "מתעצבן"

Do NOT apply for:

• Pure grief/condolence ("יהי זכרו ברוך", "ת.נ.צ.ב.ה.") — bereavement, not moral-political anger
• Mild criticism without moral charge or emotional intensity
• Simple factual statements

Tag 2: OUTGROUP_NAMING

Apply when the comment names, labels, or categorizes an out-group in a politically charged way — drawing a boundary between "us" and "them."

Look for:

• Political/sectoral group labels used as accusations or boundary markers: "ביביסטים", "חרדים", "מתנחלים", "שמאלנים", "ימנים", "ערבים" (politicized), "החילונים"
• Recategorization outside the moral community: "לא יהודים", "כמו הערבים", "מחבלים" (applied to citizens)
• Political figures/factions as the enemy: "הממשלה", "הקואליציה", "האופוזיציה", named politicians as stand-ins
• "Them" pronouns with negative attribution: "הם", "אותם", "שלהם" marking a hostile other
• Generalizations: "כולם...", "כל ה...", "תמיד הם..."

Do NOT apply for:

• Neutral mention of a group with no boundary-drawing function
• In-group references without out-group contrast

Tag 3: COLLECTIVE_EFFICACY

Apply when the comment expresses belief that the group can act and achieve change through unified effort. Three things must be present:

1. A "we" subject (group, public, people — NOT authorities, NOT fate)
2. A capability or agency claim ("can," "will," "are able," "in our hands")
3. A change/goal orientation (stopping, winning, changing, removing)

Subtypes:

• Group efficacy: "ביחד ננצח", "יחד אנחנו יכולים", "יש לנו את הכוח"
• Participative efficacy: "כל קול נחשב", "בלעדיך זה לא יקרה", "גם הקול שלי משנה"
• Empowerment: "הכוח בידיים שלנו", "לא נשתוק", "השינוי מתחיל מאיתנו"
• Mobilization calls: "בואו נצא לרחובות", "כולנו לכיכר", "זה הזמן לפעול"
• Past-victory framing: "כבר עשינו את זה...", "כשהעם רוצה הוא מנצח"

Do NOT apply for (most common confusions):

• Demands on authorities: "צריך לעצור אותם", "חייבים לעשות משהו" — agent is "them," not "us"
• Wishes or curses: "שיעופו מהחיים שלנו" — wish, not capability claim
• Normative "should/must" without agentive "we"
• Calls for the government/police/authorities to act

Diagnostic test: Is the agent "we/us/the public" with a can/will verb? Yes → tag. Agent is anyone else, or modality is "should/must/wish" → do not tag.

Tag 4: DEMOBILIZATION

Apply when the comment expresses cynicism, fatalism, resignation, or active discouragement of action. Active opposite of efficacy: undermines the belief that action is worthwhile or possible.

Look for:

• Cynical resignation: "נו, ומה קרה איתם?", "מה כבר אפשר לעשות", "אין מה לעשות", "תמיד ככה"
• Fatalism: "זה לא ישתנה לעולם", "אותו דבר תמיד", "מה זה משנה"
• Normalizing wrongdoing: "מופתעים? כי אנחנו לא", "ידוע מראש", "כרגיל"
• Active discouragement: telling others not to bother, mocking those who try
• Hopelessness: "המדינה אבודה", "אין סיכוי", "לא יעזור כלום"
• Withdrawal: "אני מתייאש", "אין לי כוח לזה יותר"

Do NOT apply for:

• Anger without resignation (that's ANGER_MORAL_VIOLATION)
• Strong critique that still implies action is needed

Tag 5: NOT_ENOUGH_CONTENT

Apply ONLY when no other tag fits:

• Pure grief/condolence: "יהי זכרו ברוך", "ת.נ.צ.ב.ה.", "ברוך דיין האמת"
• Emoji-only or near-empty comments
• Personal sadness without moral-political framing
• Off-topic, spam, or unrelated content
• Too short or ambiguous to interpret

Mutually exclusive: if ANY other tag applies, do not use this one.

Important rules (apply every time)

1. Multiple tags are allowed and expected. Tag ALL that genuinely apply.
2. Tag based on what is in the text, not assumed feelings.
3. Hebrew grief formulas ("יהי זכרו ברוך", "ה' ייקום דמו", "ברוך דיין האמת") = NOT_ENOUGH_CONTENT UNLESS clearly used as part of a political-moral argument.
4. "ה' ייקום דמו" naming enemies = ANGER_MORAL_VIOLATION + OUTGROUP_NAMING; in pure mourning = NOT_ENOUGH_CONTENT.
5. When uncertain between tags, prefer the more specific one.
6. When uncertain whether anything applies, prefer NOT_ENOUGH_CONTENT over guessing.

Worked examples

Comment: "גנבים! שודדים! פושעים! שקרנים!!!"

{"tags": ["ANGER_MORAL_VIOLATION"], "brief_justification": "Stack of moral-condemnation labels with high affective intensity, but no out-group named and no efficacy claim."}

Comment: "הביביסטים שכחו מה זה להיות ציונים"

{"tags": ["ANGER_MORAL_VIOLATION", "OUTGROUP_NAMING"], "brief_justification": "Names 'Bibists' as out-group and accuses them of moral failure (forgetting Zionism)."}

Comment: "צריך לעצור אותם חלאות"

{"tags": ["ANGER_MORAL_VIOLATION"], "brief_justification": "Strong moral condemnation ('scum'), but 'someone must stop them' is a demand on authorities, not collective efficacy."}

Comment: "ביחד ננצח את הממשלה הזאת"

{"tags": ["COLLECTIVE_EFFICACY", "OUTGROUP_NAMING"], "brief_justification": "Classic group-efficacy form ('together we will win') with the government framed as adversary."}

Comment: "נו, ומה קרה איתם? כלום כרגיל"

{"tags": ["DEMOBILIZATION"], "brief_justification": "Cynical resignation, normalizing inaction ('nothing as usual')."}

Comment: "יהי זכרו ברוך"

{"tags": ["NOT_ENOUGH_CONTENT"], "brief_justification": "Pure condolence formula, no political-moral content."}

Comment: "בושה לנו במימון שלנו לחרדים האלה — חייב להיפסק!"

{"tags": ["ANGER_MORAL_VIOLATION", "OUTGROUP_NAMING"], "brief_justification": "Moral-violation framing ('shame'), names Haredim as out-group; the 'must stop' is a normative demand, not collective efficacy."}