1---2name: splunk-developer3description: Use when developing Splunk searches, dashboards, and apps — SPL (Search Processing Language) queries, transforming commands (stats, chart, timechart, eval, rex, spath), knowledge objects (fields, lookups, event types, tags, data models), dashboards (Simple XML, Dashboard Studio), alerts and scheduled searches, data inputs (forwarders, HEC, scripted inputs, modular inputs), index-time vs search-time extractions, macros, workflow actions, custom Splunk apps, and REST API integration. Part of the observability-* skill family.4---56# Splunk Developer78For monitoring/alerting infrastructure see: `rhel-monitoring`, `ubuntu-monitoring`. For log analysis at scale see: `large-file-analysis`.910<HARD-RULE>11Always specify index and sourcetype in every search — searching across all indexes wastes resources and may return unexpected data from indexes you lack permissions to see properly.12</HARD-RULE>1314<HARD-RULE>15Never use "join" when stats/eval can accomplish the same result — join is memory-intensive and fails on large datasets; use stats or eventstats instead.16</HARD-RULE>1718<HARD-RULE>19Always use tstats with accelerated data models for high-frequency dashboards — raw searches against large indexes cause search concurrency issues.20</HARD-RULE>2122<HARD-RULE>23Never store sensitive data (passwords, API keys) in dashboard XML or saved searches — use credential storage or encrypted inputs.24</HARD-RULE>2526---2728## Reference Files2930Detailed code examples, patterns, and configuration are in the reference files below. Read the relevant file when working on that area.3132| File | Covers |33|---|---|34| [advanced-spl-apps-api-security.md](advanced-spl-apps-api-security.md) | advanced SPL patterns, app development, REST API usage, and security/compliance searches |35| [dashboards-alerts-inputs.md](dashboards-alerts-inputs.md) | knowledge objects, dashboard XML (Simple XML and Dashboard Studio), alerts/scheduled searches, and data inputs configuration |36| [spl-commands-extraction.md](spl-commands-extraction.md) | SPL query basics, transforming commands (stats, chart, timechart, eval, rex), data extraction patterns, and lookups |3738---3940---4142## Anti-Patterns4344| Anti-Pattern | Why It Fails | Correct Approach |45|---|---|---|46| Using wildcards at the beginning of searches (`*error*`) | Splunk scans every event; bypasses index-time field extraction; search takes minutes instead of seconds | Use indexed fields, source, sourcetype, or index as primary filters; wildcards only at the end of terms |47| Running real-time searches on dashboards | Real-time searches consume persistent search slots; do not scale; cause scheduler congestion | Use scheduled searches with acceleration or 1-minute interval searches; real-time only for brief triage |48| Building dashboards without data models | Every panel runs its own raw search; inconsistent field names; no search optimization or acceleration | Create a data model with proper field extraction; build dashboards on pivot or `tstats` for 10-100x faster queries |49| Not using summary indexing for common aggregations | Repeated calculation of the same aggregation across millions of events; slow dashboards, wasted compute | Use `collect` or `tstats` summaries for frequently-queried metrics; pre-compute expensive aggregations |50| Extracting fields at search time when index-time would be appropriate | Search-time extraction runs on every query; high-volume fields waste CPU repeatedly | Use transforms.conf for high-value, high-volume fields; search-time extraction for ad-hoc or low-volume fields |5152---5354## Related Skills5556| Workload | Skill |57|---|---|58| RHEL monitoring (Prometheus, Grafana, ELK) | `rhel-monitoring` |59| Ubuntu monitoring (Prometheus, Grafana, logging) | `ubuntu-monitoring` |60| Large file / log analysis | `large-file-analysis` |61| Docker container logging | `docker-admin` |62| Python SDK development | `python-flask-developer` |63| Data warehouse / data pipeline design | `data-warehouse`, `data-lake` |
joogy06