Israeli Chatbot Analytics

Israeli Chatbot Analytics

Analyze and optimize Hebrew chatbot performance. This skill covers conversation flow analytics, Hebrew-specific sentiment analysis, drop-off detection, user satisfaction scoring, A/B testing for Hebrew response variants, intent recognition accuracy tracking, anomaly alerting, and reporting dashboards. Use it to understand whether your Hebrew chatbot is actually helping users and where to focus improvements.

Instructions

Step 1: Collect and Structure Conversation Logs

Before analyzing, ensure conversation data is structured consistently. Each conversation session should include:

# Standard conversation log schema
conversation_log = {
    "session_id": "uuid-string",
    "user_id": "anonymous-or-identified",
    "channel": "whatsapp|telegram|web|app",
    "language": "he",           # Primary language detected
    "started_at": "ISO-8601",
    "ended_at": "ISO-8601",
    "messages": [
        {
            "timestamp": "ISO-8601",
            "sender": "user|bot",
            "text": "שלום, אני צריך עזרה",
            "intent": "greeting",           # Detected intent
            "intent_confidence": 0.92,       # Model confidence
            "entities": [],                  # Extracted entities
            "response_time_ms": 340,         # Bot response latency
        }
    ],
    "outcome": "resolved|escalated|abandoned|unknown",
    "satisfaction_score": null,   # CSAT score if collected
    "metadata": {
        "bot_version": "2.1.0",
        "ab_variant": "formal_he",
    }
}

Two fields in that schema carry every headline number in this skill, and neither one arrives in an export. Define both explicitly before you compute anything.

Deriving outcome (do this first). Completion, escalation, abandonment, drop-off, the satisfaction composite and cost per resolved conversation all key off this label. Platform exports do not contain it: the Dialogflow CX parser writes unknown, and a WhatsApp webhook stream has no outcome concept at all. Run the analyzer without deriving it and you get a dashboard of zeros. Write the rule down and version it:

Never take an outcome the bot writes about itself ("flow completed") as an outcome. That is the bot's own belief, and treating it as ground truth is the same error as reading high_confidence_rate as intent_accuracy (Step 8).

Defining the session boundary. session_id, started_at and ended_at are given on Dialogflow CX and Rasa. On WhatsApp Cloud API, the dominant Israeli channel, there is no session object at all, only a flat webhook stream. You must cut it yourself with an inactivity timeout, and the timeout you pick mechanically sets your abandonment rate and average handle time. A user who replies the next morning is one long resolved session under a 24-hour cut and one abandoned session plus a new one under a 30-minute cut. Both numbers get quoted to management. Pick one (30 minutes is a reasonable default for support), write it into the log-normalization step, and never change it without restating the history. The WhatsApp 24-hour customer-service window and the 72-hour free-entry-point window are BILLING windows: do not reuse them as conversation boundaries.

If your platform exports a different shape, normalize it first. Common platforms:

Step 2: Conversation Flow Analysis

Analyze session-level metrics to understand overall chatbot health:

Build a ConversationMetrics dataclass that tracks total_sessions, completed_sessions, escalated_sessions, abandoned_sessions, session_lengths (per-session message count), and session_durations (seconds). Derive rate properties (completion_rate, escalation_rate, abandonment_rate) as count / total_sessions, and avg_session_length / median_session_duration_seconds from the list fields.

compute_flow_metrics(conversations) iterates the structured logs once, increments the right outcome counter (resolved / escalated / abandoned), appends message count and (ended_at - started_at).total_seconds(), and returns the metrics object.

Industry benchmarks for support chatbots (apply with judgment to Hebrew bots):

These are general support-chatbot targets with no Israeli sample behind them. Use them as a starting bar and replace each row with your own baseline after four weeks of data.

Step 3: Drop-off Point Detection

Identify where users abandon. This reveals UX problems, confusing prompts, or missing capabilities:

detect_drop_off_points(conversations) filters to outcome == "abandoned" and returns three Counter.most_common slices: drop-off by conversation depth (message count), by active intent at drop (walking from the tail to the first message that carries an intent), and by last bot message (first 100 chars, walking from the tail for the last sender == "bot").

Keep fallback in the by-intent bucket. Fallback-then-abandon is the most common real drop-off pattern, so filtering it out empties the report for exactly the sessions you most need to see.

detect_conversation_loops(conversations, threshold=3) flags sessions where the bot repeats the same text ≥ threshold times in a row by scanning the bot-message stream and tracking a consecutive-repeat counter; emit {session_id, repeated_message, repeat_count, total_messages} for each looped session.

Step 4: Hebrew Sentiment Analysis

Hebrew sentiment analysis requires special handling due to morphological complexity, negation patterns, and slang. Use DictaBERT (encoder, classification) for production sentiment scoring, AlephBERT (onlplab/alephbert-base from the ONLP Lab at Bar-Ilan University) as an alternative encoder baseline, or a lexicon-based approach for lightweight analysis. When you need one model to classify sentiment AND summarize the conversation in Hebrew prose for the ops team, use Dicta-LM 3.0 (February 2026), the current Hebrew model family from Dicta: 24B (adapted from Mistral-Small-3.1), 12B (from NVIDIA Nemotron Nano V2) and 1.7B (from Qwen3-1.7B), each with a 65k native context and a chat variant with tool-calling support. The 1.7B variant is the practical choice for per-message classification at volume; the 24B for offline summarization. DictaLM 2.0 (July 2024, 7B, Mistral-based) is the previous generation and still works, but new builds should start on 3.0.

Using DictaBERT (recommended for production). The simplest path is pipeline("sentiment-analysis", model="dicta-il/dictabert-sentiment"), which resolves the label names off the model config for you. If you drive the model directly for batching control, wrap AutoTokenizer + AutoModelForSequenceClassification, tokenize with truncation=True, max_length=512, padding=True, softmax the logits, and map each row through id2label.

CRITICAL: read the label names from model.config.id2label, never from a hardcoded list. Label order is model metadata, not a convention, and it is not alphabetical. For dicta-il/dictabert-sentiment it is {0: "Positive", 1: "Negative", 2: "Neutral"}. A version of this skill that hardcoded ["negative","neutral","positive"] was wrong at every index and reported every frustration spike as a satisfaction spike.

Full code, batching wrapper and the lexicon fallback: references/hebrew-sentiment-guide.md.

Hebrew-specific sentiment challenges (summary). Negation flips meaning and "לא רע" reads mildly positive in Israeli usage. Sarcasm is very common ("יופי, בדיוק מה שחיכיתי לו" is deeply negative); DictaBERT catches some, fine-tune on your domain for the rest. Slang moves fast ("אחלה" / "סבבה" / "בומבה" positive, "חרא" / "פאדיחה" negative, "וואלה" context-dependent). And users mix scripts in one message ("ה-support שלכם גרוע"), so your model or lexicon must handle both.

See references/hebrew-sentiment-guide.md for the full treatment of these challenges, including the slang lexicon and negation-handling code.

Step 5: Intent Recognition Accuracy Tracking

Track how well your chatbot understands user requests over time:

Build IntentAccuracyTracker to log (predicted, actual, confidence, timestamp) per prediction and expose:

• confusion_matrix(): 2D {actual: {predicted: count}} over the sorted intent universe.
• misclassification_report(min_count=5): top (actual, predicted) pairs where predicted != actual.
• low_confidence_intents(threshold=0.6): intents whose mean confidence is below threshold, with sample_count and below_threshold_pct.
• accuracy_trend(): daily {date, accuracy, sample_count} series for plotting (bucket by timestamp[:10]).

How to get ground truth labels. Sample 100-200 conversations a week and have Hebrew-speaking annotators label the actual intent; this is the gold standard. Supplement it with escalation signals (a user correcting the bot, "לא, התכוונתי ל...", or asking for a human right after a misunderstanding, flags the prior intent as wrong) and post-chat surveys asking "Did the bot understand what you needed?" correlated with the detected intent.

Step 6: User Satisfaction Measurement

Combine multiple signals to build a satisfaction score:

Build a SatisfactionSignals dataclass carrying direct feedback (csat_score 1-5, thumbs_rating "up"/"down"), behavioural signals (session_resolved, escalated_to_human, abandoned, repeated_fallbacks, loop_detected) and sentiment signals (final_sentiment positive/neutral/negative, sentiment_trend improving/stable/declining).

Its composite_score() -> float returns 0.0-1.0. If csat_score is present, return (csat_score - 1) / 4 directly and stop. Otherwise start at 0.5 (or 0.8 / 0.2 for thumbs up / down), then apply: +0.15 resolved, -0.1 escalated, -0.2 abandoned, -0.15 if repeated_fallbacks > 2, -0.2 loop detected, +/-0.1-0.15 for final_sentiment, +/-0.05-0.1 for sentiment_trend; clamp to [0, 1].

Provide collect_post_chat_survey_he() that returns a Hebrew post-chat survey: title "נשמח לשמוע מה חשבת", a 1-5 rating on "עד כמה הצ'אטבוט עזר לך?", a yes/no on "האם הצ'אטבוט הבין את מה שרצית?", and an optional open "רוצה לשתף עוד משהו?" field. Use "שלח משוב" as the submit label.

Step 7: A/B Testing for Hebrew Response Variants

Test different phrasings, formality levels, and gender handling strategies:

Build HebrewABTestManager with three responsibilities:

1. Register a test. create_test(test_id, variants: {name: response_text}, traffic_split=None). Default split is uniform across variants. Store {variants, traffic_split, created_at} per test_id. Example variants:

{"formal": "שלום וברוכים הבאים. כיצד נוכל לסייע לכם?",
 "casual": "היי! איך אפשר לעזור?",
 "gender_neutral": "שלום! ניתן לבחור מהאפשרויות הבאות:"}

2. Deterministic bucketing. assign_variant(test_id, user_id) hashes f"{user_id}:{test_id}" with hashlib.md5, maps to a bucket in [0, 1), and walks the cumulative traffic_split so the same user always gets the same variant. Use this in get_response(...) and increment an impressions counter at the same time.

3. Outcome tracking. record_outcome(test_id, variant, completed=False, satisfaction=None, escalated=False) and get_test_results(test_id) returning per-variant {impressions, completion_rate, avg_satisfaction, escalation_rate}.

Common Hebrew A/B test dimensions. Formality ("כיצד נוכל לסייע?" vs "איך אפשר לעזור?") against completion rate; gender handling (slash notation "את/ה" vs gender-neutral "ניתן ל...") against satisfaction; response length against drop-off; emoji use against engagement; and error phrasing ("לא הצלחתי להבין" vs "אפשר לנסח אחרת?") against retry rate.

Step 8: Performance Dashboards and KPIs

Build a ChatbotDashboard dataclass grouping the fields below, plus a to_report_dict() that renders them by section (rates as %, times as ms):

Implement build_dashboard(conversations, period_days=7) to populate the dataclass:

• Outcome rates from Counter(c["outcome"]) / n.
• avg_handle_time_seconds from (ended_at - started_at).total_seconds() per session.
• avg_csat from satisfaction_score where present.
• avg_response_time_ms / p95_response_time_ms from bot messages with response_time_ms (p95 by nearest rank, sorted_rts[math.ceil(0.95 * n) - 1]).
• high_confidence_rate = share of user messages with intent_confidence > 0.7. fallback_rate = share of user messages with intent == "fallback".
• intent_accuracy stays None unless you have ground-truth labels. Populate it only from IntentAccuracyTracker (Step 5) and render n/a otherwise. Model confidence is not accuracy: a confident but wrong classifier scores 100% on confidence and can be wrong on every prediction, and this is the number most likely to be quoted to management.
• conversations_per_day = n / period_days. peak_hour and busiest_day from Counter over started_at hour and weekday.

Israeli traffic patterns to expect. These follow from the Sun-Thu work week and are working assumptions, not a measured dataset. Confirm each against your own logs before building a staffing or alerting rule on it.

• Peak hours are typically 10:00-12:00 and 19:00-22:00 (Israel Time, UTC+2/+3)
• Sunday is the busiest day (first workday of the Israeli week)
• Friday afternoon and Saturday see minimal traffic
• Holiday periods (Rosh Hashana, Pesach, Sukkot) show different patterns

Retention and Returning-User Metrics

Session-level metrics tell you how a single conversation went, but not whether the bot earns repeat use. Track these retention dimensions alongside the dashboard above (all require a stable user_id across sessions, pseudonymized per the Privacy and Consent section):

For each user_id, collect the set of distinct dates with a conversation, then compute: D1 return rate (first date + 1 day is also in the set), D7 return rate (any of first date + 2..7 days is in the set, more stable than D1 at Israeli volumes), and repeat-contact rate (more than one distinct date). On a support bot a high repeat rate can mean trust or unresolved issues, so always read it next to first-contact resolution.

Step 9: Hebrew-Specific Analytics Challenges

RTL Text in Charts and Visualizations

When rendering analytics dashboards that display Hebrew text, handle these RTL issues:

Set matplotlib.rcParams["font.family"] = ["DejaVu Sans", "Arial", "Heebo"] so Hebrew glyphs render at all, then apply bidi.algorithm.get_display() (the python-bidi package) to every label before drawing, because matplotlib has no native RTL. Prefer horizontal bar charts so Hebrew labels sit on the y-axis and read naturally. For interactive dashboards Plotly handles RTL better than matplotlib: use font-family: "Heebo, Arial, sans-serif" and add extra inline-start margin for the labels.

Hebrew Word Tokenization for Word Clouds

Whitespace tokenization fails on Hebrew because of the prefix particles (ב, ה, ו, ל, מ, כ, ש). Use the YAP morphological analyzer (https://github.com/OnlpLab/yap) in production, or strip common prefixes, only when the word is longer than 3 characters and the remainder is at least 2. For word clouds, run the bidi algorithm before rendering and drop stopwords (של, את, על, עם, אני, זה, כי, גם, לא, יש, אין, מה). Full tokenizer code: references/hebrew-sentiment-guide.md.

Mixed Hebrew-English Query Handling

Israeli users code-switch constantly, so classify per message and track the mix. Count Hebrew characters ([\u0590-\u05FF]) against Latin ones ([a-zA-Z]): no letters at all is unknown and must NOT count toward the denominator; otherwise he when Hebrew is at least half the letters, en below that. Track the code-switching rate separately as the share of classifiable messages that are 20-80% Hebrew. That band overlaps the primary-language buckets on purpose: a message can be both Hebrew and code-switched. Treating them as three exclusive buckets is what made an earlier version of the bundled script disagree with this rule on the same log.

Step 10: Alerting and Anomaly Detection

Define an AlertRule dataclass with name, metric, operator ("gt" / "lt"), threshold, window_minutes, severity (critical / warning / info) and description_he, the Hebrew text the ops team will actually read. The three that catch the most real incidents are satisfaction_drop (avg_csat lt 3.0 over 120 min, critical), high_abandonment (abandonment_rate gt 0.40 over 60 min, critical) and high_fallback_rate (fallback_rate gt 0.25 over 30 min, warning). The full six-rule starting set with Hebrew descriptions is in references/chatbot-metrics-glossary.md.

These thresholds are starting bars, not benchmarks: replace each with your own baseline after four weeks of data. They also assume a dialogue-managed bot. On an LLM bot the slow_response p95 rule at 3 seconds fires permanently and gets muted, taking the real latency signal with it, so swap it for the time-to-first-token rule in references/llm-bot-observability.md.

AlertManager wraps the rule list. check_metrics(current_metrics: dict) walks every rule, skips when the metric is missing, and triggers when value > threshold (op gt) or value < threshold (op lt). Each triggered alert is a dict with rule_name, severity, metric, current_value, threshold, description_he, and triggered_at.

Step 11: Reporting Templates

Generate periodic reports summarizing chatbot performance:

Implement generate_weekly_report(dashboard, previous_dashboard=None, period_start, period_end):

• Helper trend_arrow(current, previous, higher_is_better): returns (ללא שינוי) for < 1% delta; otherwise emits [v] +X.X% (good direction) or [!] +X.X% (bad direction).
• Emit a # דוח ביצועי צ'אטבוט שבועי header, period subheader, and a | מדד | ערך | שינוי מהשבוע הקודם | markdown table over: שיחות, שיעור פתרון, CSAT, שיעור הסלמה (lower-is-better), שיעור נטישה (lower-is-better), שיעור ניבויים בביטחון גבוה, זמן תגובה ממוצע (lower-is-better). Render a דיוק זיהוי כוונות row only when labelled data produced a real intent_accuracy.
• Append a ## תנועה block with conversations_per_day, peak_hour, busiest_day.

Step 12: Integration with Chatbot Platforms

Each platform exports conversations in its own shape; normalize every one of them to the Step 1 schema and the metrics above run unchanged. Parser recipes for the Conversational Agents (Dialogflow CX) BigQuery export and the Rasa tracker store, plus the Hebrew language-code and Rasa-version caveats, are in references/platform-integrations.md. Botpress, ManyChat, WhatsApp webhook logs and custom bots have no canonical export shape and must be mapped by hand.

WhatsApp Business Platform pricing notes

Meta may revise pricing only on the first day of each quarter. Utility templates sent inside an open customer-service window are currently free, and free-entry-point windows stay open 72 hours. Israel rates are not published per country, so do not hardcode a figure: pull the current rate card. Two 2026 changes to budget for: a separate pricing policy for AI Providers on the platform took effect 16 February 2026 (updated 12 May 2026), so check whether your bot falls under it before modelling cost per conversation; and businesses on the Marketing Messages API can now set a max price per marketing-message delivery, which turns per-message cost into a variable you control and should therefore log. Add template_category (marketing / utility / authentication / service) and a boolean arrived_via_ctw_ad to the Step 1 conversation-log schema, so finance and product can split CSAT and completion by paid versus free interaction. Full breakdown and the remaining cost fields to log: references/chatbot-metrics-glossary.md.

Anti-spam compliance (Israel Communications Law, Section 30A)

If your chatbot sends marketing messages (broadcasts, promotional templates on WhatsApp, Telegram campaigns, SMS retargeting), Section 30A of the Communications Law (Telecom and Broadcasts) 5742-1982 applies. The law requires prior opt-in consent before sending advertising messages. DLA Piper's Israel summary describes the statute as prohibiting "advertising by means of automated dialing, fax or text messages without first obtaining the recipient's initial opt-in prior consent", with a mandatory opt-out in every message. Whether that reaches WhatsApp and Telegram rests on how Israeli courts read "text messages", not on explicit statutory text, and we could not verify a specific ruling. Treat IM broadcasts as in scope for compliance planning and get a lawyer's read before relying on the opposite. The term "advertisement" is interpreted broadly: any message not purely service-related can be treated as advertising.

Tag every send with an opt_in_basis ("explicit_form" / "ctw_ad_click" / "service_reply" / "transactional") as your audit trail, track unsubscribe-path success as a compliance KPI, and split completion and CSAT between opt-in marketing flows and user-initiated service flows, since combining them masks both. Detail and the cross-channel skill pointers: references/chatbot-metrics-glossary.md.

This is engineering guidance, not legal advice. Israeli law provides statutory damages per unsolicited marketing message without proof of damages, so a misconfigured broadcast to even a few hundred non-consenting users can become a meaningful financial event. We could not verify the current per-message cap against a primary source, so confirm the figure and your specific exposure with a privacy lawyer before sizing the risk.

Experimentation and analytics stack

HebrewABTestManager (Step 7) does in-process bucketing with in-memory results. For sequential testing or CUPED variance reduction, move to Statsig, LaunchDarkly or GrowthBook: none of them care what language variant_text is in, and GrowthBook is the one that never ingests your event data, so Hebrew transcripts stay in your own warehouse. Plan on 2+ weeks and 200+ impressions per variant; Israeli user bases are small and the Sun-Thu week makes one-week tests unreliable.

On the analytics side, GA4 has a built-in AI Assistant channel group (Medium ai-assistant) for LLM-referred traffic, and Mixpanel's AI query builder was renamed from Spark to Mixpanel Agent and ships an MCP server your agent can query directly.

Vendor ownership, pricing tiers, current names and the recognized-referrer caveat: references/analytics-stack-notes.md. Statsig changed hands twice between September 2025 and May 2026, so do not trust an older note about who operates it.

LLM and RAG bot observability

If your bot generates answers with an LLM rather than matching intents, Steps 5 and 8 measure the wrong things: there is no intent label to score, no fallback to count, and a fluent wrong answer registers as a completed session. The session-level metrics (drop-off, escalation, CSAT, retention) still apply unchanged; what you add on top is groundedness, retrieval hit rate, tool-call success rate, and cost per resolved conversation. Log retrieved chunk ids against the ids the answer cited. This is a citation-compliance canary, not groundedness. See references/llm-bot-observability.md.

An LLM-as-judge scorer is the standard fallback where you have no ground truth, but calibrate it against Hebrew-speaking human annotators before quoting its output as accuracy, and run it on a weekly sample rather than on every turn. The Step 8 warning about confidence being mistaken for accuracy applies to judge scores with equal force.

Field-level schema additions, the four metrics to build first, judge-calibration procedure, OpenTelemetry GenAI tracing conventions and the LLM-specific alert rules: references/llm-bot-observability.md.

Examples

Example 1: Analyze chatbot performance for the past week

"Analyze my Hebrew chatbot logs from the past week and show me where users are dropping off."

Load the period's logs, run compute_flow_metrics(), detect_drop_off_points() and detect_conversation_loops(), then summarize completion rate, top drop-off points and looping sessions with actionable recommendations.

Example 2: Set up A/B testing for greeting messages

"I want to test whether a formal or casual Hebrew greeting works better."

Create the test with HebrewABTestManager.create_test(), variants formal ("כיצד נוכל לסייע לכם היום?") vs. casual ("היי! מה אפשר לעשות בשבילך?"), 50/50 split, wire it into the greeting handler, and track completion rate, CSAT and escalation per variant.

Example 3: Set up anomaly alerting

"Alert me if chatbot satisfaction drops suddenly."

Configure AlertManager with the satisfaction and escalation rules, compute metrics over rolling windows, route alerts to Slack / email / PagerDuty, and keep the Hebrew description_he text for the ops team.

Example 4: Generate a weekly performance report

"Create a Hebrew weekly report for the chatbot team."

Run build_dashboard() for the current and previous week, pass both to generate_weekly_report() for trend arrows, and add drop-off and intent breakdowns. Output is RTL-compatible Hebrew markdown.

Bundled Resources

Scripts

• scripts/conversation-analyzer.py -- Pure standard library, no pip install. Computes outcome rates, drop-off points, conversation loops, intent confidence, response-time percentiles, Israeli traffic patterns and Hebrew/English code-switching. It does NOT compute sentiment: that needs the DictaBERT path in Step 4. Run: python3 scripts/conversation-analyzer.py --help

References

• references/chatbot-metrics-glossary.md -- Glossary of chatbot analytics metrics with Hebrew translations and industry benchmarks. Consult when defining KPIs or explaining metrics to Hebrew-speaking stakeholders.
• references/analytics-stack-notes.md -- Vendor detail for the experimentation and analytics stack (Statsig / LaunchDarkly / GrowthBook, GA4 AI Assistant channel, Mixpanel Agent and MCP). Consult before picking or pricing a tool.
• references/platform-integrations.md -- Parser recipes folding each vendor's conversation export (Conversational Agents / Dialogflow CX BigQuery, Rasa tracker store) into the standard schema. Consult when onboarding logs from a new platform.
• references/llm-bot-observability.md -- Measurement layer for LLM-backed and RAG chatbots: log-schema additions, groundedness and retrieval metrics, LLM-as-judge calibration, tracing, and LLM-specific alert rules. Consult when the bot generates answers instead of matching intents.
• references/hebrew-sentiment-guide.md -- Guide to Hebrew sentiment analysis challenges including negation, sarcasm, slang, and mixed-language handling. Consult when building or tuning Hebrew sentiment models.

Gotchas

• Hebrew sentiment analysis requires Israeli-specific training data. Standard English sentiment models misclassify Hebrew sarcasm (very common in Israeli communication) as neutral or positive.
• Israeli chatbot usage peaks on Sunday mornings (start of work week), not Monday. Weekly analytics reports should anchor to Sunday-Thursday.
• Hebrew text analytics must handle prefixed particles (ב-, ל-, כ-, מ-) that change word boundaries. Standard tokenizers trained on English split Hebrew words incorrectly.
• Israeli users frequently code-switch between Hebrew and English within a single chatbot conversation. Analytics tools must handle bilingual sessions, not treat them as two separate languages.

Privacy and Consent

This skill ingests full conversation transcripts and user_id values, and runs sentiment analysis on user messages. Conversation text is personal data and often contains sensitive content (health, finances, complaints). Handle it under Israel's Privacy Protection Law, including Amendment 13 (in force August 2025), which tightened consent, notice, accountability, and data-minimization obligations.

Practical rules, in full in references/chatbot-metrics-glossary.md: get consent to store and analyze chat content and disclose sentiment analysis as a processing purpose; pseudonymize user_id before it reaches the pipeline and keep the mapping table separate (retention and A/B bucketing both work fine on a stable pseudonymous id); strip or mask entities you do not need (ID numbers, names, card numbers); set an explicit retention window for raw transcripts, for example 90 days, and keep only aggregates long-term; restrict and log access, and know where the data is stored and processed. Note that the Data Security Regulations require access logs for medium and high security databases to be retained for at least 24 months, which is a separate obligation from transcript retention.

• This is engineering guidance, not legal advice. Confirm your specific obligations with a privacy professional.

Recommended MCP Servers

None is required. The skill operates on exported conversation logs (BigQuery exports, Rasa tracker-store dumps, application log files) loaded from disk and analyzed locally with the bundled script. If your metrics already live in Mixpanel, its MCP server lets you query them conversationally from the agent, but that is optional and sits outside this skill's analysis path.

Reference Links

Troubleshooting

• DictaBERT model not loading: the dicta-il/dictabert-sentiment model needs PyTorch + transformers (~500MB). Run pip install torch transformers (known to work on the transformers 5.x line; pin whichever version you have actually tested); for CPU-only, install torch from https://download.pytorch.org/whl/cpu.
• Sentiment labels look inverted: you hardcoded a label list. Read model.config.id2label instead. For dicta-il/dictabert-sentiment it is {0: "Positive", 1: "Negative", 2: "Neutral"}.
• Timestamps parse to nothing on older Python: datetime.fromisoformat only handles the full ISO-8601 range, including a trailing Z, from Python 3.11. On 3.10 the parse fails and durations and traffic patterns come back empty. Use Python 3.11 or newer.
• Hebrew text appears reversed in charts: matplotlib has no native RTL. Apply python-bidi (bidi.algorithm.get_display()) before rendering, or switch to Plotly.
• Tokenization produces wrong word frequencies: whitespace splitting ignores Hebrew prefix particles. Use the prefix-stripping tokenizer in Step 9, or the YAP morphological analyzer (https://github.com/OnlpLab/yap) for production.
• Sentiment scores unreliable for short messages: 1-3 word messages lack context ("סבבה" can be positive or neutral). Under 4 words, rely on behavioral signals (continued / escalated / abandoned) plus the Step 6 satisfaction signals.
• A/B test results not statistically significant: usually too small a sample, common for Israeli user bases. Run at least 2 weeks, 200+ impressions per variant, target p < 0.05.