# Parsing Helper

> Customize and test Grok parsing for USP, Cloud Sensor, and External adapters. Helps generate parsing rules from sample logs, validate against test data, and deploy configurations. Use when setting up new log sources, troubleshooting parsing issues, or modifying field extraction for adapters.

- Skill: `refractionpoint/parsing-helper` (Agent Skill)
- Install (CLI): `npx skillmds@latest add refractionpoint/parsing-helper`
- Raw SKILL.md: https://api.skillmd.com/api/skills/refractionpoint/parsing-helper/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: refractionpoint (https://skillmd.com/u/refractionpoint)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/refractionpoint/parsing-helper

---


# Parsing Helper

A guided workflow for creating, testing, and deploying Grok parsing configurations for LimaCharlie adapters. This skill helps you customize how log data is parsed and normalized as it's ingested into LimaCharlie.

---

## LimaCharlie Integration

> **Prerequisites**: Run `/init-lc` to initialize LimaCharlie context.

### LimaCharlie CLI Access

All LimaCharlie operations use the `limacharlie` CLI directly:

```bash
limacharlie <noun> <verb> --oid <oid> --output yaml [flags]
```

For command help and discovery: `limacharlie <command> --ai-help`

### Critical Rules

| Rule | Wrong | Right |
|------|-------|-------|
| **CLI Access** | Call MCP tools or spawn api-executor | Use `Bash("limacharlie ...")` directly |
| **Output Format** | `--output json` | `--output yaml` (more token-efficient) |
| **Filter Output** | Pipe to jq/yq | Use `--filter JMESPATH` to select fields |
| **LCQL Queries** | Write query syntax manually | Use `limacharlie ai generate-query` first |
| **Timestamps** | Calculate epoch values | Use `date +%s` or `date -d '7 days ago' +%s` |
| **OID** | Use org name | Use UUID (call `limacharlie org list` if needed) |

---

> ⚠️ **TIMEZONE NOTE**: Patterns like `SYSLOGTIMESTAMP` (`Dec 16 17:50:04`) lack timezone info. LimaCharlie assumes UTC. This skill automatically detects timezone mismatches by comparing parsed times to current UTC.

## When to Use

Use this skill when:

- **Setting up new log parsing**: Configure Grok patterns for a new log source
- **Troubleshooting parsing issues**: Logs aren't being parsed correctly or fields are missing
- **Modifying field extraction**: Need to add, remove, or change extracted fields
- **Testing parsing changes**: Validate new parsing rules before deploying to production
- **Generating adapter configurations**: Create YAML/CLI config for local USP adapters

Common scenarios:
- "Help me parse these firewall logs"
- "The timestamp isn't being extracted correctly from my syslog"
- "I need to add a new field to my adapter parsing"
- "Generate a Grok pattern for these log samples"
- "Test this parsing configuration before I deploy it"

## What This Skill Does

This skill provides a 5-phase workflow:

1. **Organization Selection** - Select the LimaCharlie organization
2. **Adapter Discovery** - Identify the adapter type and get current configuration
3. **Sample Log Collection** - Get sample logs from the adapter's sensor or user input
4. **Grok Pattern Generation** - Analyze logs and generate appropriate Grok patterns
5. **Validation & Deployment** - Test parsing and apply configuration

### Supported Adapter Types

| Type | Description | Configuration Method |
|------|-------------|---------------------|
| **External Adapter** | Cloud-managed adapters (syslog, webhook, API) | Update via `set_external_adapter` API |
| **Cloud Sensor** | Virtual sensors for cloud platforms (AWS, Azure, GCP, SaaS) | Update via `set_cloud_sensor` API |
| **One-off/USP Adapter** | On-prem adapters with local configuration | Output YAML/CLI config for user |

## Required Information

Before starting, gather:

- **Organization**: The LimaCharlie org where the adapter is configured
- **Adapter Type**: External Adapter, Cloud Sensor, or One-off/USP Adapter
- **Adapter Name**: For cloud-managed adapters, the name of the existing adapter
- **Sample Logs**: Either from the adapter's sensor or user-provided samples

## How to Use

### Phase 1: Organization Selection

Get the list of available organizations:

```bash
limacharlie org list --output yaml
```

Use `AskUserQuestion` to let the user select an organization if multiple are available.

### Phase 2: Adapter Discovery & Type Selection

Ask the user which adapter type they're working with using `AskUserQuestion`:

**Option A: External Adapter** (cloud-managed syslog, webhook, API)

1. List available external adapters:
```bash
limacharlie external-adapter list --oid <SELECTED_ORG_ID> --output yaml
```

2. Get the selected adapter's current configuration:
```bash
limacharlie external-adapter get --key <ADAPTER_NAME> --oid <SELECTED_ORG_ID> --output yaml
```

**Option B: Cloud Sensor** (AWS, Azure, GCP, SaaS)

1. List available cloud sensors:
```bash
limacharlie cloud-adapter list --oid <SELECTED_ORG_ID> --output yaml
```

2. Get the selected cloud sensor's current configuration:
```bash
limacharlie cloud-adapter get --key <SENSOR_NAME> --oid <SELECTED_ORG_ID> --output yaml
```

**Option C: One-off/USP Adapter** (local adapter, no cloud config)

No API calls needed. Proceed directly to sample log collection.

### Phase 3: Get Sample Log Data

**Option A: Query from adapter's sensor** (for cloud-managed adapters)

If the adapter is already ingesting data, you can fetch sample events:

1. Find the sensor by IID (Installation ID from the adapter's installation key):
```bash
limacharlie sensor list --oid <SELECTED_ORG_ID> --filter "[?iid=='<IID>']" --output yaml
```

2. Get recent events from the sensor:

**IMPORTANT**: Always calculate timestamps dynamically using Bash FIRST:

```bash
# Run this to get current epoch timestamps for the last hour
start=$(date -d '1 hour ago' +%s) && end=$(date +%s) && echo "start=$start end=$end"
```

Then use the actual output values (e.g., `start=1764805928 end=1764809528`) directly in the CLI call - do NOT use placeholder values:

```bash
limacharlie search run --query "event sid = '<SENSOR_ID>'" --start $start --end $end --oid <SELECTED_ORG_ID> --filter "[:10]" --output yaml
```

3. Extract the raw log content from the events for analysis.

**Option B: User provides sample logs**

If no data is available or for one-off adapters, ask the user to paste sample log lines.

Use `AskUserQuestion` or direct text input to collect sample logs.

### Recognizing Unparsed Events

When fetching events from a sensor, **unparsed logs** appear as:
- `event_type: "unknown_event"` in the routing
- Event payload contains only a `text` field with the raw log line

Example of an unparsed event:
```json
{
  "event": {
    "text": "2025-12-03 16:41:19 status installed shared-mime-info:amd64 2.2-1"
  },
  "routing": {
    "event_type": "unknown_event",
    "hostname": "my-adapter"
  }
}
```

This indicates no Grok parsing is configured for the adapter. The raw content in the `text` field is what you need to analyze and create a parsing pattern for.

### Phase 4: Generate Grok Pattern

Analyze the sample logs and generate an appropriate Grok pattern.

**Common Grok Patterns:**

| Pattern | Description | Example Match |
|---------|-------------|---------------|
| `%{TIMESTAMP_ISO8601:timestamp}` | ISO 8601 timestamp (YYYY-MM-DD) | `2024-01-15T12:30:45Z` or `2024-01-15 12:30:45` |
| `%{DATESTAMP:timestamp}` | US/EU date format (MM-DD-YY) | `01-15-24 12:30:45` |
| `%{SYSLOGTIMESTAMP:date}` | Syslog timestamp | `Jan 15 12:30:45` |
| `%{IP:ip_address}` | IPv4/IPv6 address | `192.168.1.100` |
| `%{HOSTNAME:host}` | Hostname | `server01.example.com` |
| `%{NUMBER:value}` | Numeric value | `12345` |
| `%{INT:count}` | Integer | `42` |
| `%{WORD:action}` | Single word | `ACCEPT` |
| `%{DATA:field}` | Non-greedy match (up to delimiter) | `any text` |
| `%{GREEDYDATA:message}` | Greedy match (rest of line) | `everything else` |
| `%{LOGLEVEL:level}` | Log level | `ERROR`, `WARN`, `INFO` |
| `%{POSINT:port}` | Positive integer | `443` |

> **Timestamp Pattern Warning**: Do NOT use `%{DATESTAMP}` for `YYYY-MM-DD` format logs - it will misparse the year (treating `2025-12-03` as day=25, month=12, year=03). Use `%{TIMESTAMP_ISO8601}` instead for any `YYYY-MM-DD` formatted timestamps.

**IMPORTANT - Grok Field Name for Text Platform:**

For text platform adapters, always use `message` as the key in `parsing_grok`:

```yaml
parsing_grok:
  message: '%{PATTERN:field} ...'  # Always use "message" as the key, NOT "text"
```

**Building the Mapping Configuration:**

```yaml
client_options:
  mapping:
    parsing_grok:
      message: '%{SYSLOGTIMESTAMP:date} %{HOSTNAME:host} %{WORD:service}\[%{INT:pid}\]: %{GREEDYDATA:message}'
    event_type_path: "service"
    event_time_path: "date"
    sensor_hostname_path: "host"
```

### Phase 4.5: Automatic Timestamp Timezone Check

> **REQUIRED**: After validation, MUST perform this sanity check for patterns without explicit timezone.

**Applies when Grok pattern uses:**
- `%{SYSLOGTIMESTAMP}` - e.g., `Dec 16 17:50:04`
- `%{DATESTAMP}` - e.g., `12-16-24 17:50:04`
- `%{TIMESTAMP_ISO8601}` without `Z` or offset - e.g., `2024-12-16 17:50:04`

**Automatic Detection Steps:**

1. After `validate_usp_mapping` returns, extract the `event_time` from results (epoch milliseconds)

2. Get current UTC epoch:
```bash
date -u +%s
```

3. Calculate the offset:
```bash
# Example: event_time=1734357004000 (milliseconds)
# current_utc=$(date -u +%s)
# offset_hours=$(( (current_utc - event_time/1000) / 3600 ))
```

4. **If offset is 4-12 hours**: The logs are likely in a local timezone (US timezones are UTC-5 to UTC-8)

   **Automatically warn the user:**
   > ⚠️ **Timezone Warning**: The parsed event_time appears to be ~[X] hours behind current UTC time. This suggests your log timestamps are in local time (e.g., Pacific Time), not UTC.
   >
   > LimaCharlie interprets all timestamps without timezone info as UTC. Your events will appear [X] hours in the past.
   >
   > **Recommendation**: Configure your log source to emit UTC timestamps:
   > - rsyslog: Add `$ActionFileDefaultTemplate RSYSLOG_FileFormat` to rsyslog.conf
   > - systemd-journald: Already uses UTC by default

5. Continue with deployment even if warning is shown (user can address later)

### Phase 5: Validate & Apply

**Step 1: Validate the parsing configuration**

Use `validate_usp_mapping` to test the Grok pattern against sample data:

```bash
# Write adapter config with mapping and sample data to a YAML file:
cat > /tmp/usp-test.yaml << 'EOF'
parsing_grok:
  message: "%{SYSLOGTIMESTAMP:date} %{HOSTNAME:host} %{WORD:service}\\[%{INT:pid}\\]: %{GREEDYDATA:msg}"
event_type_path: service
event_time_path: date
sensor_hostname_path: host
sample_data:
  - "<SAMPLE_LOG_LINES>"
EOF
limacharlie usp validate --oid <SELECTED_ORG_ID> --platform text --input-file /tmp/usp-test.yaml --output yaml
```

> **Note**: The `mapping` object should have `parsing_grok`, `event_type_path`, `event_time_path`, and `sensor_hostname_path` at the same level - NOT nested under a `parsing` key.

**Step 2: Review parsed results**

Show the parsed events to the user:
- Verify all expected fields are extracted
- Confirm event type assignment

**Automatic Timezone Sanity Check:**

If the Grok pattern uses `SYSLOGTIMESTAMP`, `DATESTAMP`, or `TIMESTAMP_ISO8601` without explicit timezone:

1. Get current UTC: `current_utc=$(date -u +%s)`
2. Extract `event_time` from validation result (milliseconds)
3. Calculate offset: `offset_hours=$(( (current_utc - event_time/1000) / 3600 ))`
4. **If offset is 4-12 hours**, display warning:

   "⚠️ **Timezone Mismatch Detected**: Parsed time is ~[offset] hours behind UTC. Your logs appear to be in local time. Events will be timestamped incorrectly in LimaCharlie. Consider configuring your log source to use UTC."

**Step 3: Iterate if needed**

If validation fails or fields are missing, adjust the Grok pattern and re-validate.

**Step 4: Apply configuration**

**For External Adapters:**
```bash
# Write config to temp file
cat > /tmp/adapter-config.yaml << 'EOF'
... existing config with updated parsing_rules ...
EOF
limacharlie external-adapter set --key <ADAPTER_NAME> --input-file /tmp/adapter-config.yaml --oid <SELECTED_ORG_ID>
```

**For Cloud Sensors:**
```bash
# Write config to temp file
cat > /tmp/cloud-adapter-config.yaml << 'EOF'
... existing config with updated parsing ...
EOF
limacharlie cloud-adapter set --key <SENSOR_NAME> --input-file /tmp/cloud-adapter-config.yaml --oid <SELECTED_ORG_ID>
```

**Step 5: Verify no adapter errors**

After applying the configuration for External Adapters or Cloud Sensors, check for any errors:

```bash
limacharlie org errors --oid <SELECTED_ORG_ID> --output yaml
```

Look for errors related to the adapter. If errors appear, review the adapter configuration and address the issues.

**For One-off/USP Adapters:**

Provide the configuration for the user to apply locally:

**YAML Config:**
```yaml
file:
  client_options:
    platform: text
    identity:
      installation_key: <IID>
      oid: <OID>
    sensor_seed_key: my-adapter
    mapping:
      parsing_grok:
        message: '%{SYSLOGTIMESTAMP:date} %{HOSTNAME:host} %{WORD:service}\[%{INT:pid}\]: %{GREEDYDATA:message}'
      event_type_path: "service"
      event_time_path: "date"
      event_time_timezone: "America/New_York"  # Required for SYSLOGTIMESTAMP (no timezone in pattern)
      sensor_hostname_path: "host"
  file_path: "/var/log/messages"
```

**CLI Config:**
```bash
./lc_adapter file \
  client_options.identity.installation_key=<IID> \
  client_options.identity.oid=<OID> \
  client_options.platform=text \
  client_options.sensor_seed_key=my-adapter \
  "client_options.mapping.parsing_grok.message=%{SYSLOGTIMESTAMP:date} %{HOSTNAME:host} %{WORD:service}\[%{INT:pid}\]: %{GREEDYDATA:message}" \
  client_options.mapping.event_type_path=service \
  client_options.mapping.event_time_path=date \
  client_options.mapping.event_time_timezone=America/New_York \
  client_options.mapping.sensor_hostname_path=host \
  file_path=/var/log/messages
```

> **IMPORTANT**: The `event_time_timezone` is REQUIRED when using `SYSLOGTIMESTAMP`, `DATESTAMP`, or `TIMESTAMP_ISO8601` without explicit timezone info. Without it, LimaCharlie assumes UTC and timestamps will be incorrect.

**When outputting CLI config for `test-limacharlie-adapter` skill, use this format:**
```
--grok '<PATTERN>'
--event-type <PATH>
--event-time <PATH>
--event-time-tz <TIMEZONE>  # Include when timestamp pattern lacks timezone
--hostname-path <PATH>
```

## Example Usage

### Example 1: Parse Syslog Logs

**User**: "Help me parse syslog logs from my Linux server"

**Sample log:**
```
Jan 15 12:30:45 server01 sshd[12345]: Accepted publickey for admin from 192.168.1.50 port 54321 ssh2
```

**Generated Grok pattern:**
```
%{SYSLOGTIMESTAMP:date} %{HOSTNAME:host} %{WORD:service}\[%{INT:pid}\]: %{GREEDYDATA:message}
```

**Mapping configuration:**
```yaml
mapping:
  parsing_grok:
    message: '%{SYSLOGTIMESTAMP:date} %{HOSTNAME:host} %{WORD:service}\[%{INT:pid}\]: %{GREEDYDATA:message}'
  event_type_path: "service"
  event_time_path: "date"
  event_time_timezone: "America/New_York"  # Required - SYSLOGTIMESTAMP has no timezone
  sensor_hostname_path: "host"
```

**CLI options for test-limacharlie-adapter:**
```
--grok '%{SYSLOGTIMESTAMP:date} %{HOSTNAME:host} %{WORD:service}\[%{INT:pid}\]: %{GREEDYDATA:message}'
--event-type service
--event-time date
--event-time-tz America/New_York
--hostname-path host
```

**Validation result:**
```json
{
  "valid": true,
  "results": [{
    "event": {
      "timestamp": "Jan 15 12:30:45",
      "event_type": "sshd",
      "message": "Accepted publickey for admin from 192.168.1.50 port 54321 ssh2"
    },
    "routing": {
      "hostname": "server01"
    }
  }]
}
```

### Example 2: Parse Firewall Logs

**User**: "I need to parse these firewall logs and extract the source/destination IPs"

**Sample log:**
```
2024-01-15T12:00:00Z ACCEPT TCP 192.168.1.100:54321 10.0.0.5:443 packets=1 bytes=78
```

**Generated Grok pattern:**
```
%{TIMESTAMP_ISO8601:timestamp} %{WORD:action} %{WORD:protocol} %{IP:src_ip}:%{NUMBER:src_port} %{IP:dst_ip}:%{NUMBER:dst_port} packets=%{NUMBER:packets} bytes=%{NUMBER:bytes}
```

**Mapping configuration:**
```yaml
mapping:
  parsing_grok:
    message: '%{TIMESTAMP_ISO8601:timestamp} %{WORD:action} %{WORD:protocol} %{IP:src_ip}:%{NUMBER:src_port} %{IP:dst_ip}:%{NUMBER:dst_port} packets=%{NUMBER:packets} bytes=%{NUMBER:bytes}'
  event_type_path: "action"
  event_time_path: "timestamp"
```

**Validation result:**
```json
{
  "valid": true,
  "results": [{
    "event": {
      "timestamp": "2024-01-15T12:00:00Z",
      "event_type": "ACCEPT",
      "protocol": "TCP",
      "src_ip": "192.168.1.100",
      "src_port": "54321",
      "dst_ip": "10.0.0.5",
      "dst_port": "443",
      "packets": "1",
      "bytes": "78"
    }
  }]
}
```

### Example 3: One-off USP Adapter Configuration

**User**: "I have a local adapter running, just give me the config to parse Apache access logs"

**Sample log:**
```
192.168.1.50 - admin [15/Jan/2024:12:30:45 +0000] "GET /api/users HTTP/1.1" 200 1234
```

**Generated configuration (YAML):**
```yaml
file:
  client_options:
    platform: text
    identity:
      installation_key: <YOUR_IID>
      oid: <YOUR_OID>
    sensor_seed_key: apache-logs
    mapping:
      parsing_grok:
        message: '%{COMMONAPACHELOG}'
      event_type_path: "verb"
      event_time_path: "timestamp"
  file_path: "/var/log/apache2/access.log"
```

**Generated configuration (CLI):**
```bash
./lc_adapter file \
  client_options.identity.installation_key=<YOUR_IID> \
  client_options.identity.oid=<YOUR_OID> \
  client_options.platform=text \
  client_options.sensor_seed_key=apache-logs \
  "client_options.mapping.parsing_grok.message=%{COMMONAPACHELOG}" \
  client_options.mapping.event_type_path=verb \
  client_options.mapping.event_time_path=timestamp \
  file_path=/var/log/apache2/access.log
```

## Additional Notes

### Pre-built Grok Patterns

LimaCharlie supports these pre-built patterns for common log formats:

| Pattern | Use Case |
|---------|----------|
| `%{COMMONAPACHELOG}` | Apache common log format |
| `%{COMBINEDAPACHELOG}` | Apache combined log format |
| `%{NGINXACCESS}` | Nginx access logs |

### Key/Value Parsing (Alternative to Grok)

For logs with `key=value` format (like CEF), use `parsing_re` with key-value extraction:

```yaml
mapping:
  parsing_re: '(?:<\d+>\s*)?(\w+)=(".*?"|\S+)'
```

This regex captures pairs where:
- Submatch 1 = Key name
- Submatch 2 = Value

### Field Path Syntax

Use `/` as separator for nested JSON navigation:
- `a` → access field `a`
- `a/b/c` → access nested field `a.b.c`

### Event Type Path Templates

The `event_type_path` supports template strings for dynamic event type assignment based on event content.

### Timestamp Timezone Handling

LimaCharlie assumes UTC for timestamps that don't include explicit timezone information.

**Formats with explicit timezone (safe):**
- ISO 8601 with `Z`: `2024-01-15T12:30:45Z`
- ISO 8601 with offset: `2024-01-15T12:30:45-08:00`
- RFC 3339: `2024-01-15T12:30:45.000-08:00`

**Formats without timezone (assumed UTC):**
- Syslog: `Jan 15 12:30:45`
- Simple datetime: `2024-01-15 12:30:45`

**Recommendation**: Configure log sources to emit UTC timestamps when possible. For syslog on Linux:
- rsyslog: Use `$ActionFileDefaultTemplate RSYSLOG_FileFormat` in rsyslog.conf
- journald: Timestamps are already in UTC by default

### Common Issues

1. **Pattern not matching**: Ensure special characters are escaped (e.g., `\[` for literal brackets)
2. **Greedy matching**: Use `%{DATA}` for non-greedy matching, `%{GREEDYDATA}` only at end of pattern
3. **Timestamp not parsing**: Verify timestamp format matches the Grok pattern exactly

### Common Pitfalls

1. **Using stale timestamps**: Always calculate epoch timestamps dynamically with Bash before querying `get_historic_events`. Never use placeholder or hardcoded values - the API will return empty results for timestamps outside the data retention window.

2. **Wrong mapping structure**: The `mapping` object should have `parsing_grok`, `event_type_path`, and `event_time_path` at the same level - NOT nested under a `parsing` key.

3. **Wrong Grok field name**: For text platform adapters, always use `message` as the key in `parsing_grok`, not `text`.

4. **DATESTAMP vs TIMESTAMP_ISO8601**:
   - `YYYY-MM-DD HH:MM:SS` → Use `%{TIMESTAMP_ISO8601}`
   - `MM-DD-YY HH:MM:SS` → Use `%{DATESTAMP}`
   - `Jan 15 12:30:45` → Use `%{SYSLOGTIMESTAMP}`

5. **Not recognizing unparsed events**: If you see `event_type: "unknown_event"` with only a `text` field, that means parsing is not configured - the `text` field contains the raw log data that needs a Grok pattern.

6. **Timezone assumptions for timestamps without timezone info**:
   - `SYSLOGTIMESTAMP`, `DATESTAMP`, and `TIMESTAMP_ISO8601` without `Z` or offset don't include timezone
   - LimaCharlie assumes UTC for all such timestamps
   - If logs are in local time (e.g., Pacific Time), events will appear 7-8 hours in the future
   - **Solution**: Configure log source to emit UTC timestamps, or verify with user that times appear correct

### Indexing Support

For searchable fields, configure custom indexing:

```yaml
indexing:
  - events_included: ["*"]
    path: "src_ip"
    index_type: "ip"
  - events_included: ["*"]
    path: "dst_ip"
    index_type: "ip"
```

Supported index types: `file_hash`, `file_path`, `file_name`, `domain`, `ip`, `user`, `service_name`, `package_name`

## Related Skills

- `detection-engineering`: Create D&R rules to detect patterns in parsed logs
- `test-limacharlie-adapter`: Deploy a test adapter to verify parsing
- `lookup-lc-doc`: Search LimaCharlie documentation for more parsing details

## Reference

For more details on parsing configuration:
- Adapter Usage: `./docs/limacharlie/doc/Sensors/Adapters/adapter-usage.md`
- Log Collection Guide: `./docs/limacharlie/doc/Sensors/Reference/logcollectionguide.md`