Openstack Security

OpenStack Security Operations

Security posture management for OpenStack requires defense-in-depth: no single control prevents all threats, so multiple overlapping layers protect the cloud. The defense layers are network segmentation (isolate management from tenant from external traffic), TLS everywhere (encrypt all API communication), RBAC least-privilege (users and services get only the permissions they need), and audit logging (record every authentication and authorization decision).

Security is not a one-time deployment task. Certificates expire. Vulnerabilities are disclosed. Passwords must rotate. Security groups drift. The GUARD agent consumes this skill for continuous security posture assessment, evaluating whether the cloud's security controls remain effective against evolving threats.

In NASA SE terms, security spans multiple phases: Phase B (security design and architecture), Phase C (certificate generation and TLS deployment), Phase D (security audit verification), and Phase E (ongoing security operations). SP-6105 SS 6.4 (Technical Risk Management) provides the framework for identifying, assessing, and mitigating security risks throughout the cloud lifecycle.

Deploy

Security-First Deployment

Kolla-Ansible TLS configuration (globals.yml):

# Enable TLS on all interfaces
kolla_enable_tls_internal: "yes"
kolla_enable_tls_external: "yes"
kolla_copy_ca_into_containers: "yes"

# Certificate paths
kolla_external_fqdn_cert: "/etc/kolla/certificates/haproxy.pem"
kolla_internal_fqdn_cert: "/etc/kolla/certificates/haproxy-internal.pem"
kolla_admin_openrc_cacert: "/etc/kolla/certificates/ca/root.crt"

# Optional: client certificate verification
kolla_enable_tls_backend: "yes"  # Encrypt HAProxy-to-service traffic

Generating certificates:

# Self-signed CA (lab/development)
mkdir -p /etc/kolla/certificates/ca
openssl genrsa -out /etc/kolla/certificates/ca/root.key 4096
openssl req -x509 -new -nodes -key /etc/kolla/certificates/ca/root.key \
  -sha256 -days 3650 -out /etc/kolla/certificates/ca/root.crt \
  -subj "/C=US/ST=Lab/O=OpenStack/CN=Kolla-CA"

# HAProxy certificate (external)
openssl genrsa -out /etc/kolla/certificates/haproxy.key 2048
openssl req -new -key /etc/kolla/certificates/haproxy.key \
  -out /etc/kolla/certificates/haproxy.csr \
  -subj "/C=US/ST=Lab/O=OpenStack/CN=${KOLLA_EXTERNAL_FQDN}" \
  -addext "subjectAltName=DNS:${KOLLA_EXTERNAL_FQDN},IP:${KOLLA_EXTERNAL_VIP}"

openssl x509 -req -in /etc/kolla/certificates/haproxy.csr \
  -CA /etc/kolla/certificates/ca/root.crt \
  -CAkey /etc/kolla/certificates/ca/root.key \
  -CAcreateserial -out /etc/kolla/certificates/haproxy.crt \
  -days 365 -sha256

# Combine into PEM (required by HAProxy)
cat /etc/kolla/certificates/haproxy.crt \
    /etc/kolla/certificates/haproxy.key \
    > /etc/kolla/certificates/haproxy.pem

# Internal certificate (same process with internal FQDN/VIP)
# ... repeat with kolla_internal_fqdn and kolla_internal_vip

HAProxy TLS termination:

HAProxy handles TLS for all OpenStack API endpoints. Kolla-Ansible configures this automatically when TLS is enabled. Verify:

# Check HAProxy is serving TLS
openssl s_client -connect ${KOLLA_EXTERNAL_VIP}:443 -showcerts </dev/null 2>/dev/null | head -20

# Verify certificate chain
openssl s_client -connect ${KOLLA_EXTERNAL_VIP}:5000 -CAfile /etc/kolla/certificates/ca/root.crt </dev/null 2>/dev/null | grep "Verify return code"
# Expected: Verify return code: 0 (ok)

Firewall rules for management network:

# Allow only necessary ports on management interface
firewall-cmd --zone=management --add-service=ssh --permanent
firewall-cmd --zone=management --add-port=5000/tcp --permanent  # Keystone
firewall-cmd --zone=management --add-port=8774/tcp --permanent  # Nova
firewall-cmd --zone=management --add-port=9696/tcp --permanent  # Neutron
firewall-cmd --zone=management --add-port=8776/tcp --permanent  # Cinder
firewall-cmd --zone=management --add-port=9292/tcp --permanent  # Glance
firewall-cmd --zone=management --add-port=8080/tcp --permanent  # Swift
firewall-cmd --zone=management --add-port=8004/tcp --permanent  # Heat
firewall-cmd --zone=management --add-port=443/tcp --permanent   # Horizon
firewall-cmd --zone=management --add-port=3000/tcp --permanent  # Grafana
firewall-cmd --zone=management --add-port=9090/tcp --permanent  # Prometheus
firewall-cmd --reload

# Default deny on all other ports
firewall-cmd --zone=management --set-target=DROP --permanent
firewall-cmd --reload

Security group defaults:

# Create restrictive default security group rules
openstack security group rule delete $(openstack security group rule list default -f value -c ID)
# Start with no rules (default deny)

# Add only SSH and ICMP for management
openstack security group rule create --protocol tcp --dst-port 22 --remote-ip 10.0.0.0/8 default
openstack security group rule create --protocol icmp default

Configure

Certificate Lifecycle Management

Certificate inventory:

Expiry monitoring:

#!/bin/bash
# check-cert-expiry.sh -- alert on certificates expiring within 30 days
THRESHOLD_DAYS=30
THRESHOLD_SECS=$((THRESHOLD_DAYS * 86400))

for cert in /etc/kolla/certificates/*.pem /etc/kolla/certificates/*.crt; do
  [ -f "$cert" ] || continue
  expiry=$(openssl x509 -in "$cert" -noout -enddate 2>/dev/null | cut -d= -f2)
  expiry_epoch=$(date -d "$expiry" +%s 2>/dev/null)
  now_epoch=$(date +%s)
  remaining=$((expiry_epoch - now_epoch))

  if [ $remaining -lt $THRESHOLD_SECS ]; then
    echo "WARNING: $cert expires in $((remaining / 86400)) days ($expiry)"
  fi
done

Rotation procedure:

# 1. Generate new certificate (see Deploy section)
# 2. Replace PEM file
cp /etc/kolla/certificates/haproxy-new.pem /etc/kolla/certificates/haproxy.pem

# 3. Reconfigure HAProxy to pick up new cert
kolla-ansible -i inventory reconfigure --tags haproxy

# 4. Verify
openssl s_client -connect ${KOLLA_EXTERNAL_VIP}:443 </dev/null 2>/dev/null | openssl x509 -noout -dates

RBAC Policy Customization

Per-service policy files:

Each OpenStack service has a policy.yaml (or policy.json) that defines authorization rules:

# Location: /etc/kolla/<service>/policy.yaml
# Kolla-Ansible merges custom policies with code defaults

# Example: Keystone -- restrict user creation to domain admins
# /etc/kolla/keystone/policy.yaml
"identity:create_user": "rule:admin_required and domain_id:%(target.user.domain_id)s"

# Example: Nova -- restrict flavor management to cloud admins
# /etc/kolla/nova/policy.yaml
"os_compute_api:os-flavor-manage": "role:admin and is_admin_project:True"

# Example: Neutron -- restrict external network creation
# /etc/kolla/neutron/policy.yaml
"create_network:router:external": "role:admin"

Best practices:

• Start with defaults; override only what you need
• Document every policy override with rationale
• Test policy changes with a non-admin user before deploying
• Use oslopolicy-checker to validate policy files

# Validate policy syntax
docker exec nova_api oslopolicy-checker --policy /etc/nova/policy.yaml

Network Segmentation

Audit Logging

Keystone CADF events:

# keystone.conf
[DEFAULT]
notification_driver = messagingv2
notification_format = cadf

# CADF events include:
# - authentication.authenticate (success/failure)
# - identity.authenticate.pending
# - identity.create_user
# - identity.update_project
# - identity.delete_role_assignment

Log aggregation for security events:

# Search for failed authentication attempts
docker logs keystone_api 2>&1 | grep -i "authentication failed\|401\|Unauthorized" | tail -20

# Search for RBAC denials
docker logs nova_api 2>&1 | grep -i "policy\|403\|Forbidden" | tail -20

# Aggregate across all services
for svc in keystone nova neutron cinder glance; do
  echo "=== ${svc} ==="
  docker logs ${svc}_api 2>&1 | grep -c "401\|403" || echo 0
done

Password Rotation

# 1. Generate new passwords
kolla-genpwd  # Or manually update /etc/kolla/passwords.yml

# 2. Reconfigure services to use new passwords
kolla-ansible -i inventory reconfigure

# 3. Verify all services authenticate with new credentials
openstack token issue
openstack service list

API Rate Limiting

Configure HAProxy rate limiting to prevent brute-force attacks:

# /etc/kolla/haproxy/haproxy.cfg (via Kolla-Ansible custom config)
frontend keystone_api
  stick-table type ip size 100k expire 30s store http_req_rate(10s)
  http-request deny deny_status 429 if { sc_http_req_rate(0) gt 100 }
  http-request track-sc0 src

Service Account Hardening

# List all service accounts
openstack user list --domain default --project service

# Verify each service account has only the required role
for svc in nova neutron cinder glance swift heat; do
  echo "=== ${svc} ==="
  openstack role assignment list --user ${svc} --names
done
# Each should have exactly "admin" role on "service" project -- no more

Operate

Certificate Rotation Procedures

Planned rotation (before expiry):

1. Generate new certificate with same SAN entries
2. Verify new certificate: openssl x509 -in new.crt -noout -text
3. Create combined PEM: cat new.crt new.key > new.pem
4. Backup current certificate: cp haproxy.pem haproxy.pem.bak
5. Replace: cp new.pem haproxy.pem
6. Reconfigure: kolla-ansible -i inventory reconfigure --tags haproxy
7. Verify all endpoints: openstack token issue

Emergency rotation (compromised key):

1. Revoke compromised certificate (if using CA with CRL)
2. Generate new key pair immediately (do not reuse old key)
3. Issue new certificate from CA
4. Deploy and reconfigure (same as planned rotation steps 3-7)
5. Audit: review logs for unauthorized access during compromise window
6. Rotate all service passwords as precaution

Security Audit Checklist

Weekly:

• Review failed authentication logs (Keystone CADF)
• Check for unauthorized API access patterns
• Verify security group rules unchanged from baseline
• Confirm certificate expiry > 30 days

Monthly:

• Run vulnerability scan against API endpoints
• Review RBAC policies against least-privilege baseline
• Audit service account permissions
• Check for CVEs affecting deployed OpenStack version
• Verify audit logging is active and rotated properly

Quarterly:

• Conduct penetration test scope review
• Review network segmentation effectiveness
• Rotate service account passwords
• Update security baseline documentation

Incident Response Procedures

Compromised credential (user or service account):

# 1. Immediately revoke all tokens for the compromised user
openstack token revoke $(openstack token issue -c id -f value)  # For current token
# Disable user to prevent new token issuance
openstack user set --disable <compromised-user>

# 2. Rotate the compromised password
openstack user set --password-prompt <compromised-user>

# 3. Rotate Fernet keys to invalidate all outstanding tokens
kolla-ansible -i inventory keystone_fernet_rotate

# 4. Audit: review what the compromised credential accessed
docker logs keystone_api 2>&1 | grep "<compromised-user>" | tail -100

# 5. Re-enable user with new password after investigation
openstack user set --enable <compromised-user>

Compromised instance:

# 1. Isolate: remove from all security groups (cuts network)
openstack server remove security group <instance-id> default

# 2. Snapshot for forensics
openstack server image create --name "forensic_$(date +%Y%m%d)" <instance-id>

# 3. Review console log for indicators of compromise
openstack console log show <instance-id> | tail -200

# 4. Terminate if confirmed compromised
openstack server delete <instance-id>

# 5. Audit: check if lateral movement occurred
# Review Neutron flow logs, Keystone auth logs for the instance's IP

Compliance Reporting

Generate compliance evidence for auditors:

# TLS status across all endpoints
for port in 5000 8774 9696 8776 9292 8080 8004 443; do
  echo "=== Port ${port} ==="
  echo | openssl s_client -connect ${KOLLA_EXTERNAL_VIP}:${port} 2>/dev/null | \
    openssl x509 -noout -subject -dates -issuer
done

# RBAC policy summary
for svc in keystone nova neutron cinder glance swift heat horizon; do
  echo "=== ${svc} ==="
  [ -f /etc/kolla/${svc}/policy.yaml ] && wc -l /etc/kolla/${svc}/policy.yaml || echo "Using defaults"
done

# Security group audit
openstack security group list --all-projects -f table
openstack security group rule list default -f table

Security Patch Workflow

# 1. Check for CVEs
# Review: https://security.openstack.org/ossalist.html

# 2. Assess impact against deployed version
openstack versions show  # or check container image tags
docker inspect --format '{{.Config.Labels}}' nova_api | python3 -m json.tool

# 3. If patch required, update container images
kolla-ansible -i inventory pull  # Pull latest images

# 4. Deploy updated containers
kolla-ansible -i inventory upgrade  # Rolling upgrade

# 5. Verify services after upgrade
openstack token issue
openstack server list
openstack network list

Troubleshoot

1. TLS Certificate Expired

Symptoms: All API calls fail with SSL errors. Services cannot communicate. curl: (60) SSL certificate problem: certificate has expired.

2. Certificate Chain Incomplete

Symptoms: Browser warnings on Horizon ("Not Secure"). API clients reject connection with unable to get local issuer certificate.

3. Security Group Blocking Legitimate Traffic

Symptoms: Instances cannot reach expected services. SSH works from some IPs but not others.

4. RBAC Policy Denying Authorized Actions

Symptoms: Users with correct roles get 403 Forbidden. Actions worked previously but stopped.

5. Audit Log Overflow

Symptoms: Disk filling with log data. Log rotation not working. Log queries extremely slow.

6. Service Account Password Expired or Changed

Symptoms: Cross-service calls fail. Token validation works but service-to-service 401. docker logs nova_api shows Unauthorized accessing Keystone.

Integration Points

Security connects to every component of the cloud stack because every component has attack surface:

Keystone skill: RBAC policies are Keystone's domain. Token security (Fernet key management, token lifetime, revocation) is a joint concern. Federation security (SAML assertion validation, OIDC token verification) requires coordinated configuration between Keystone and the external IdP.

Monitoring skill: Security event alerting -- failed authentication spikes trigger monitoring alerts. Certificate expiry countdown is a Prometheus metric. Unauthorized API access patterns detected through metric analysis feed security incident response.

Backup skill: Backup encryption protects sensitive data at rest. Credential backup (passwords.yml, Fernet keys) requires secure storage with access controls separate from general backup storage. Backup access itself is a security-auditable activity.

Networking-debug skill: Security group analysis is a core part of network troubleshooting. When traffic is blocked, the first question is "is it a security group rule?" Network segmentation verification (management vs tenant vs external isolation) is a joint security/networking concern.

All core OpenStack skills: TLS configuration affects every service endpoint. Each service has keystonemiddleware for token validation, service account credentials, and API endpoint security. Security hardening is not a separate layer -- it is woven into every service deployment.

GUARD agent: Primary consumer. GUARD uses this skill for continuous security posture assessment: "Are certificates current? Are RBAC policies least-privilege? Any suspicious authentication patterns? Any unpatched CVEs?" Security data is GUARD's primary input for posture evaluation.

NASA SE Cross-References