tomevault-io Splunk Developer
For monitoring/alerting infrastructure see: rhel-monitoring, ubuntu-monitoring. For log analysis at scale see: large-file-analysis.
Reference Files
Detailed code examples, patterns, and configuration are in the reference files below. Read the relevant file when working on that area.
| File | Covers |
|---|---|
| advanced-spl-apps-api-security.md | advanced SPL patterns, app development, REST API usage, and security/compliance searches |
| dashboards-alerts-inputs.md | knowledge objects, dashboard XML (Simple XML and Dashboard Studio), alerts/scheduled searches, and data inputs configuration |
| spl-commands-extraction.md | SPL query basics, transforming commands (stats, chart, timechart, eval, rex), data extraction patterns, and lookups |
Anti-Patterns
| Anti-Pattern | Why It Fails | Correct Approach |
|---|---|---|
Using wildcards at the beginning of searches (*error*) |
Splunk scans every event; bypasses index-time field extraction; search takes minutes instead of seconds | Use indexed fields, source, sourcetype, or index as primary filters; wildcards only at the end of terms |
| Running real-time searches on dashboards | Real-time searches consume persistent search slots; do not scale; cause scheduler congestion | Use scheduled searches with acceleration or 1-minute interval searches; real-time only for brief triage |
| Building dashboards without data models | Every panel runs its own raw search; inconsistent field names; no search optimization or acceleration | Create a data model with proper field extraction; build dashboards on pivot or tstats for 10-100x faster queries |
| Not using summary indexing for common aggregations | Repeated calculation of the same aggregation across millions of events; slow dashboards, wasted compute | Use collect or tstats summaries for frequently-queried metrics; pre-compute expensive aggregations |
| Extracting fields at search time when index-time would be appropriate | Search-time extraction runs on every query; high-volume fields waste CPU repeatedly | Use transforms.conf for high-value, high-volume fields; search-time extraction for ad-hoc or low-volume fields |
Related Skills
| Workload | Skill |
|---|---|
| RHEL monitoring (Prometheus, Grafana, ELK) | rhel-monitoring |
| Ubuntu monitoring (Prometheus, Grafana, logging) | ubuntu-monitoring |
| Large file / log analysis | large-file-analysis |
| Docker container logging | docker-admin |
| Python SDK development | python-flask-developer |
| Data warehouse / data pipeline design | data-warehouse, data-lake |
Source: joogy06/agent-foundry — distributed by TomeVault.