Splunk server runbook (Oracle Linux 9 / Splunk 10)

Purpose
- Central visibility for auth, service, and firewall events.

Before changes
- Confirm Splunk is running and accessible.
- Back up `etc/` (configs) before major changes.

Access control
- Rotate the Splunk admin password.
- Restrict the Web UI to jump host IPs only.

Enable receiving (for forwarders)
- Enable the receiving port (default 9997).
- CLI example:
  - `/opt/splunk/bin/splunk enable listen 9997 -auth admin:<password>`
- See `docs/splunk_forwarder.md` for forwarder setup steps.

Indexes (optional)
- Create a dedicated index like `maccdc` for team logs.
- Keep default index if you prefer less complexity.

Minimum dashboards and searches
- Failed logons (Windows): `index=* EventCode=4625`
- New users (Windows): `index=* EventCode=4720`
- Group changes (Windows): `index=* EventCode=4728 OR EventCode=4732`
- Linux auth failures: `index=* ("Failed password" OR "authentication failure")`
- sudo activity: `index=* ("sudo:" AND "COMMAND=")`
- Firewall denies: `index=* ("DENY" OR "BLOCK")`

Verification
- Confirm forwarders are sending data.
- Run a quick search to verify new events appear after a test login.

Rollback
- Restore from the backed-up `etc/` directory if a change breaks inputs.

Attack surface & triage focus
- Web UI (8000) and management API (8089): new users/roles, weak auth, exposed to non-jump IPs.
- Receiving port (9997): unexpected forwarders or data sources.
- Apps and scripted inputs: new apps under `/opt/splunk/etc/apps`, suspicious scripts in `bin/`.
- Config drift: `inputs.conf`, `outputs.conf`, `server.conf`, `web.conf`.
- Large ingestion spikes or disabled indexes.

Manual triage sequences (built-in commands)
```
sudo /opt/splunk/bin/splunk status
sudo /opt/splunk/bin/splunk list user
sudo /opt/splunk/bin/splunk show index
sudo /opt/splunk/bin/splunk btool inputs list --debug | head -n 200
sudo tail -n 200 /opt/splunk/var/log/splunk/splunkd.log
ss -tulpn | grep -E ':(8000|8089|9997)\\b'
systemctl status splunk --no-pager -l | head -n 12
```

Injection response scenarios (built-in commands)
```
# Admin/role changes
sudo /opt/splunk/bin/splunk list user
sudo /opt/splunk/bin/splunk list role

# Auth and REST activity
sudo grep -i "login" /opt/splunk/var/log/splunk/audit.log | tail -n 50
sudo grep -i "rest" /opt/splunk/var/log/splunk/splunkd.log | tail -n 50

# New apps or scripted inputs
ls -lt /opt/splunk/etc/apps | head -n 20
sudo find /opt/splunk/etc/apps -type f -name "*.sh" -o -name "*.py" | head -n 50
sudo /opt/splunk/bin/splunk btool inputs list --debug | grep -i script | head -n 20

# HEC/forwarders exposure
sudo /opt/splunk/bin/splunk btool inputs list --debug | grep -i http | head -n 20
sudo /opt/splunk/bin/splunk list forward-server
```

Command sequence (run from repo root on Splunk host)
1) `sudo bash scripts/probe/linux_splunk.sh --summary`
2) `sudo bash scripts/linux/harden_linux.sh --mode dry-run --sshd-hardening --enable-firewalld --allow-ports 22,8000,8089,9997`
3) `sudo bash scripts/linux/harden_linux.sh --mode apply --sshd-hardening --enable-firewalld --allow-ports 22,8000,8089,9997`
4) `sudo bash scripts/probe/linux_splunk.sh --summary`

Enable receiving (optional)
- `/opt/splunk/bin/splunk enable listen 9997 -auth admin:<password>`
