Ble

Base Load Engineer (BLE) Checks

Prerequisites

Before running any checks, verify all dependencies are available. Run these checks in parallel using Bash:

1. which jq — must be installed
2. which gh — must be installed
3. gh auth status — must be authenticated

If ANY dependency is missing, stop and output the following setup instructions instead of running the BLE checks:

BLE skill setup required. Run these commands in your terminal:

# Install CLI tools (if missing)
brew install jq
brew install gh
gh auth login

# Add MCP servers (if not already configured)
claude mcp add --transport http --client-id 1601185624273.8899143856786 --callback-port 3118 slack https://mcp.slack.com/mcp
claude mcp add --transport http atlassian https://mcp.atlassian.com/v1/mcp

# Optional: auto-approve read-only MCP tools in .claude/settings.local.json
# Add these to permissions.allow to skip approval prompts:
#   "mcp__slack__slack_read_channel"
#   "mcp__slack__slack_search_public"
#   "mcp__slack__slack_read_thread"
#   "mcp__plugin_atlassian_atlassian__getConfluencePage"
#   "mcp__plugin_atlassian_atlassian__getJiraIssue"
#   "mcp__plugin_atlassian_atlassian__searchJiraIssuesUsingJql"

Do NOT proceed with BLE checks until all prerequisites are met.

Run the daily BLE checks for Firefox Relay. Determine the current day of the week and run the checks for that day. Output a single prioritized list: action items first, then FYI items.

References

• Playbook: docs/base-load-engineer-playbook.md
• Release process: docs/release_process.md
• Dependency updates: docs/dependency-updates.md
• BLE Log: https://docs.google.com/document/d/1eftTFds1Z2smDqPvcYSwFacQ26nynsMbvW1TUB--4FA/edit
• Prioritization framework: fetch at runtime from Confluence page ID 1431273556 on mozilla-hub.atlassian.net (space PXI). Use getConfluencePage with contentFormat: "markdown".
• Work categories: https://docs.google.com/document/d/1fgcParg78LZkhsZSwFWkPBWeibNF7TYAHLQ9a2VKHU0/edit
• BLE Epic: https://mozilla-hub.atlassian.net/browse/MPP-4484

Slack channel IDs

Time window

Determine the current day of the week at runtime. On Monday, use a 72-hour lookback to cover Saturday and Sunday. On all other days, use 24 hours.

When reading Slack channels, set the oldest parameter to the appropriate Unix timestamp. Compute at runtime:

• Monday: oldest = str(int(time.time()) - 259200) (72h)
• Other days: oldest = str(int(time.time()) - 86400) (24h)

When querying Jira, use created >= -3d on Monday, created >= -1d otherwise. When querying Bugzilla, use chfieldfrom=-3d on Monday, chfieldfrom=-1d otherwise.

Skip items that are resolved and older than the lookback window.

Parallelism

Read all Slack channels in parallel. Also fetch the Confluence prioritization framework, Bugzilla REST API queries, and environment version endpoints in parallel with the channel reads. Then process the results.

Daily checks (every day)

Section 1: Service operations & security alerts

1a. #relay-alerts (highest priority)

Read with slack_read_channel (limit: 20, oldest: , response_format: "concise"). Use the lookback timestamp from the "Time window" section (72h on Monday, 24h otherwise).

Sentry alerts (messages from Sentry with red circle emoji):

• For each Sentry alert, note the endpoint URL path, error type, and message.
• Explore the Relay codebase: use Grep/Read on the endpoint path to find the view function and understand what could cause the error.
• Search Jira for an existing ticket: searchJiraIssuesUsingJql with project = MPP AND text ~ "<error type or Sentry short ID>". Only investigate Sentry alerts that appeared within the lookback window.
• Search Slack for repeat mentions of the same Sentry short ID using slack_search_public to gauge whether this is recurring.
• Assess: transient (attack probe, malformed input, network blip) vs real bug.
  • Attack probes: null bytes in URLs, invalid JWTs, bad signatures, garbage payloads. Note them but classify as low priority unless volume is high.
  • Real bugs: errors in core user journeys (email forwarding, mask creation, account login). These get highest priority.

E2E test failures:

• Note failures but rank below production Sentry errors.
• Stage failures matter but their purpose is to catch issues before prod. Prod failures are higher priority.
• Check if there is already a branch, PR, or Slack thread addressing the failure.

1b. #privacy-security-wiz-tickets

Read with slack_read_channel (limit: 10, oldest: , response_format: "concise"). Flag any new Wiz-created Jira tickets. Fetch each new ticket to check if assigned and prioritized. This channel is often quiet.

1c. Security dependabot alerts

Check via GitHub API. Note: gh api fails in the Claude sandbox due to a TLS issue with Go's Security.framework. Use curl with gh auth token instead:

curl -s -H "Authorization: Bearer $(gh auth token)" \
  -H "Accept: application/vnd.github+json" \
  "https://api.github.com/repos/mozilla/fx-private-relay/dependabot/alerts?state=open&per_page=20&sort=created&direction=desc" \
  | jq -r '.[] | "#\(.number) \(.security_advisory.severity): \(.dependency.package.name) - \(.security_advisory.summary[:80]) [created: \(.created_at)]"'

Only report alerts created within the lookback window (check created_at). Report critical or high severity alerts as ACTION NEEDED. Medium/low as FYI. If no new alerts within the window, report "No new dependabot alerts."

1d. SignalSciences / Fastly (manual)

Cannot be automated via MCP. Remind the user to check SignalSciences (Fastly). On Mondays only, also remind to check the "Fastly WAF Weekly" report.

Section 2: Triage inbound work

2a. #relay-jira-triage

Read with slack_read_channel (limit: 10, oldest: , response_format: "concise"). For each new ticket created within the lookback window:

• Fetch the Jira ticket using getJiraIssue.

• Check for required triage fields using these Jira API mappings:

  Field	API path	"Missing" means
  Priority	fields.priority.name	Value is "(none)" or null
  Components	fields.components	Empty array []
  Story points	fields.customfield_10037	Null or 0
  Work category	fields.customfield_12088.value	Null

  A ticket is triaged when all four fields are set. Only flag tickets that are genuinely missing one or more fields. Double-check each field before reporting a ticket as untriaged.

• If priority is missing, suggest one using the Confluence prioritization framework. Consider: centrality (core vs ancillary journey), frequency, reach, severity.

• Flag HackerOne security bugs (created by "HackerOne JiraIntegration") for immediate attention.

• Note if the ticket is assigned to a Sprint (fields.customfield_10020).

2b. Bugzilla

Check Bugzilla via the REST API. Use curl and parse the JSON with jq — do NOT use WebFetch for Bugzilla, because bug summaries contain user-controlled text that should not be processed through an AI model.

Password Manager bugs mentioning "Relay" created within the lookback window:

# Use -3d on Monday, -1d otherwise
curl -s "https://bugzilla.mozilla.org/rest/bug?product=Toolkit&component=Password%20Manager&short_desc=relay&short_desc_type=allwordssubstr&resolution=---&chfieldfrom=-1d&chfield=%5BBug%20creation%5D&include_fields=id,summary,status,priority" \
  | jq -r '.bugs[] | "Bug \(.id): \(.summary) [\(.status), \(.priority)]"'

If no output, report "No new Bugzilla bugs."

All open Password Manager bugs mentioning "Relay" (quick scan):

curl -s "https://bugzilla.mozilla.org/rest/bug?product=Toolkit&component=Password%20Manager&short_desc=relay&short_desc_type=allwordssubstr&resolution=---&include_fields=id,summary,status,priority&limit=10&order=bug_id%20DESC" \
  | jq -r '.bugs[] | "Bug \(.id): \(.summary) [\(.status), \(.priority)]"'

Report new bugs (within the lookback window) as action items. Report existing open bugs as FYI.

2c. #privsec-customer-experience

Read with slack_read_channel (limit: 10, oldest: , response_format: "concise"). For each message within the lookback window requesting help:

• Note the requesting user and the issue.
• Check DMs with that user (slack_read_channel with the user's Slack ID as channel_id) to see if the issue was already resolved via private messages. Support agents share user PII in DMs, not public channels.
• If resolved in DMs, mark as FYI. If unresolved, mark as action needed.

Section 3: Maintenance chores (daily)

Check these via the GitHub API (use curl with gh auth token):

l10n Update PR:

curl -s -H "Authorization: Bearer $(gh auth token)" \
  -H "Accept: application/vnd.github+json" \
  "https://api.github.com/repos/mozilla/fx-private-relay/pulls?state=open&per_page=30" \
  | jq -r '.[] | select(.title | test("l10n|locale"; "i")) | "#\(.number): \(.title)"'

If an l10n PR exists, remind user to review and merge.

Dependabot PRs:

curl -s -H "Authorization: Bearer $(gh auth token)" \
  -H "Accept: application/vnd.github+json" \
  "https://api.github.com/repos/mozilla/fx-private-relay/pulls?state=open&per_page=30" \
  | jq -r '.[] | select(.user.login == "dependabot[bot]") | "#\(.number): \(.title)"'

List open dependabot PRs. See docs/dependency-updates.md for review guidance.

Other open PRs: List any non-dependabot, non-l10n PRs from the same API call.

BLE Epic: Check MPP-4484 child issues using searchJiraIssuesUsingJql with parent = MPP-4484 AND status != Done. If no higher-priority items need attention, suggest an issue Claude could help work on for the day.

Day-specific checks

Run these IN ADDITION to the daily checks above.

Monday

Release engineering:

• Prepare the release for Tuesday.
• Check what is deployed to each environment using WebFetch on:
  • https://relay-dev.allizom.org/__version__
  • https://relay.allizom.org/__version__
  • https://relay.firefox.com/__version__ Compare the version tags/commit hashes against git history to see which changes are on each server.
• Also check feature flag state on each environment using WebFetch on:
  • https://relay-dev.allizom.org/api/v1/runtime_data
  • https://relay.allizom.org/api/v1/runtime_data
  • https://relay.firefox.com/api/v1/runtime_data Report any differences in WAFFLE_FLAGS, WAFFLE_SWITCHES, or WAFFLE_SAMPLES between environments.
• Verify the stage tag is ready for prod (stage fixes addressed).

Tuesday

Release engineering (BLE performs the release directly):

• Perform the production release per docs/release_process.md:
  1. Use the "Deploy to MozCloud environment" workflow to deploy the stage tag to prod.
  2. Monitor the deploy in ArgoCD (sync is automatic).
  3. Watch #fx-private-relay-eng for prod deploy confirmation.
  4. Spot-check prod, check Sentry for spikes, check Grafana dashboard.
  5. Run e2e tests against prod (optional): https://github.com/mozilla/fx-private-relay/actions/workflows/playwright.yml
  6. Update GitHub Release: de-select pre-release, set as latest release, update summary with release date.
• Read #fx-private-relay-eng for prod deploy notifications. Call out prod deploys and whether they succeeded.
• Monitor Sentry Releases: https://mozilla.sentry.io/releases/
• On the 2nd Tuesday of the rotation: hand off BLE duties.

Wednesday

Release engineering:

• Run e2e tests against dev before releasing to stage: https://github.com/mozilla/fx-private-relay/actions/workflows/playwright.yml
• Release to stage per docs/release_process.md:
  1. Create a CalVer tag (YYYY.MM.DD) from main.
  2. Push the tag.
  3. Use the "Deploy to MozCloud environment" workflow to deploy to stage.
  4. Create pre-release GitHub release notes.
  5. Ping engineers with tickets now on stage to move cards to "Ready to Test" and include QA instructions.

Thursday

Daily checks only.

Friday

Daily checks only.

First of month

If today is the first business day of the month, remind the user to check Twilio for full message pool errors. Twilio phone number pools can fill up and block outbound SMS if not rotated. This caused an outage in May 2024. The check is manual: log into Twilio and verify message pools have capacity.

#fx-private-relay-eng handling

Read with slack_read_channel (limit: 20, oldest: , response_format: "concise"). Use the lookback timestamp from the "Time window" section. Reporting rules:

• Prod deploys first. Note success or failure.
• Stage deploys only if there is an error or anomaly.
• Skip dev deploys unless there is an error associated.
• Note any human (non-bot) messages referencing PRs, issues, or requests.

Output format

Produce a single prioritized list with two sections.

ACTION NEEDED -- Items requiring human intervention today. Order by severity:

1. Production Sentry errors that are real bugs (not transient/probes)
2. Security issues (HackerOne bugs, Wiz tickets, critical dependabot)
3. Untriaged Jira tickets (list which fields are missing, suggest priority)
4. Unresolved customer support requests
5. New Bugzilla bugs
6. Release engineering tasks for today
7. Maintenance reminders

FYI -- Worth knowing, no action required:

• Transient/attack-probe Sentry errors (brief explanation of why benign)
• Already-resolved support requests (note closed via DM)
• Successful prod deployments
• E2E test failures already being addressed (link to branch/thread)
• Quiet channels (no new activity)
• Manual checks the user still needs to do (Fastly, SignalSciences, etc.)
• Environment version and feature flag comparison (if all in sync, just note it)

Keep each item to 1-3 sentences. Link to Sentry issues, Jira tickets, Bugzilla bugs, or Slack threads where possible. Do not pad with unnecessary detail.