Performing Service Account Credential Rotation

Performing Service Account Credential Rotation

Overview

Service accounts are non-human identities used by applications, daemons, CI/CD pipelines, and automated processes to authenticate to systems and APIs. These accounts often have elevated privileges and their credentials (passwords, API keys, certificates, tokens) are frequently long-lived and shared across teams, making them prime targets for attackers. Credential rotation is the systematic process of replacing these secrets on a scheduled basis, propagating new credentials to all dependent systems, and verifying service continuity after rotation.

When to Use

• When conducting security assessments that involve performing service account credential rotation
• When following incident response procedures for related security events
• When performing scheduled security testing or auditing activities
• When validating security controls through hands-on testing

Common Misconfigurations & Verification

• Incomplete consumer propagation: rotation breaks services when the new secret is updated at the source (AD/IAM/DB) but not in every consumer — app config, CI/CD pipeline secrets, Kubernetes secrets, scheduled tasks, connection pools. Map dependencies first; an unmapped consumer is an outage waiting to happen.
• Old credential never revoked: deactivating is not deleting. After the grace period, confirm the prior AWS access key is deleted (not merely set Inactive), the old AD password no longer authenticates, and superseded GCP/Azure keys are removed — a stale-but-valid key defeats the rotation.
• gMSA eligibility assumed: gMSA only fits Windows services that support it; SQL linked servers, DCOM identities, and some third-party agents do not. Verify with Test-ADServiceAccount before cutover.
• No post-rotation verification: rotating without a health check hides breakage until the next business cycle. Run authentication and functional smoke tests against each dependent service after rotation.
• Verify: confirm rotation events are logged and auditable, alerting fires on rotation failure, the rollback procedure is tested, and credential age now sits within the 30–90 day policy window.

Prerequisites

• Inventory of all service accounts across AD, cloud, and applications
• Secrets management platform (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, or CyberArk)
• Service dependency mapping (which services use which credentials)
• Change management process for rotation windows
• Monitoring for service health post-rotation

Core Concepts

Service Account Types

Group Managed Service Accounts (gMSA)

Windows gMSAs provide automatic password management by Active Directory:

• AD automatically rotates the password every 30 days
• Password is 240 bytes, cryptographically random
• Multiple servers can use the same gMSA simultaneously
• No administrator knows or manages the password
• Eliminates manual rotation for Windows services

Rotation Architecture

Secrets Manager / Vault
        │
        ├── Rotation Trigger (schedule or on-demand)
        │
        ├── Generate new credential
        │
        ├── Update credential at source (AD, cloud IAM, database)
        │
        ├── Update credential in all consumers:
        │   ├── Application configuration
        │   ├── CI/CD pipeline secrets
        │   ├── Kubernetes secrets
        │   └── Other dependent services
        │
        ├── Verify service health
        │   ├── Health check endpoints
        │   ├── Authentication test
        │   └── Functional smoke test
        │
        └── Revoke old credential (after grace period)

Workflow

Step 1: Discover and Inventory Service Accounts

Enumerate all service accounts and their dependencies:

# Active Directory: Find all service accounts
Get-ADServiceAccount -Filter * -Properties *
Get-ADUser -Filter {ServicePrincipalName -ne "$null"} -Properties ServicePrincipalName,PasswordLastSet,LastLogonDate

# Find accounts with passwords older than 90 days
$threshold = (Get-Date).AddDays(-90)
Get-ADUser -Filter {PasswordLastSet -lt $threshold -and Enabled -eq $true} -Properties PasswordLastSet,ServicePrincipalName |
    Where-Object {$_.ServicePrincipalName} |
    Select-Object Name, PasswordLastSet, ServicePrincipalName

Step 2: Implement gMSA for Windows Services

# Create KDS Root Key (one-time, domain-wide)
Add-KdsRootKey -EffectiveImmediately

# Create the gMSA account
New-ADServiceAccount -Name "svc-webapp-gmsa" `
    -DNSHostName "svc-webapp-gmsa.corp.example.com" `
    -PrincipalsAllowedToRetrieveManagedPassword "WebServerGroup" `
    -KerberosEncryptionType AES128,AES256

# Install on target server
Install-ADServiceAccount -Identity "svc-webapp-gmsa"

# Test the account
Test-ADServiceAccount -Identity "svc-webapp-gmsa"

# Configure IIS Application Pool to use gMSA
# Set identity to: CORP\svc-webapp-gmsa$

Step 3: AWS Access Key Rotation with Secrets Manager

import boto3
import json

def rotate_iam_access_key(secret_arn, iam_username):
    """Rotate an IAM user's access key via Secrets Manager."""
    iam = boto3.client("iam")
    sm = boto3.client("secretsmanager")

    # Create new access key
    new_key = iam.create_access_key(UserName=iam_username)
    new_access_key = new_key["AccessKey"]["AccessKeyId"]
    new_secret_key = new_key["AccessKey"]["SecretAccessKey"]

    # Store new credentials in Secrets Manager
    sm.put_secret_value(
        SecretId=secret_arn,
        SecretString=json.dumps({
            "accessKeyId": new_access_key,
            "secretAccessKey": new_secret_key,
            "username": iam_username,
        })
    )

    # List old access keys and deactivate them
    keys = iam.list_access_keys(UserName=iam_username)
    for key in keys["AccessKeyMetadata"]:
        if key["AccessKeyId"] != new_access_key and key["Status"] == "Active":
            iam.update_access_key(
                UserName=iam_username,
                AccessKeyId=key["AccessKeyId"],
                Status="Inactive"
            )

    return {"new_key_id": new_access_key, "old_keys_deactivated": True}

Step 4: Database Credential Rotation with Vault

import hvac

def configure_vault_database_rotation(vault_url, vault_token, db_config):
    """Configure HashiCorp Vault for automatic database credential rotation."""
    client = hvac.Client(url=vault_url, token=vault_token)

    # Enable database secrets engine
    client.sys.enable_secrets_engine(
        backend_type="database",
        path="database"
    )

    # Configure database connection
    client.secrets.database.configure(
        name=db_config["name"],
        plugin_name="postgresql-database-plugin",
        connection_url=f"postgresql://{{{{username}}}}:{{{{password}}}}@"
                       f"{db_config['host']}:{db_config['port']}/{db_config['database']}",
        allowed_roles=[db_config["role_name"]],
        username=db_config["admin_user"],
        password=db_config["admin_password"],
    )

    # Create a role for dynamic credentials
    client.secrets.database.create_role(
        name=db_config["role_name"],
        db_name=db_config["name"],
        creation_statements=[
            "CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';",
            f"GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{{{name}}}}\";"
        ],
        default_ttl="1h",
        max_ttl="24h",
    )

    return {"status": "configured", "role": db_config["role_name"]}

Step 5: Post-Rotation Verification

After every rotation, verify service continuity:

import requests
import time

def verify_service_health(service_endpoints, max_retries=3, delay=10):
    """Check that services are healthy after credential rotation."""
    results = []
    for endpoint in service_endpoints:
        for attempt in range(max_retries):
            try:
                response = requests.get(
                    endpoint["health_url"],
                    timeout=10,
                    headers=endpoint.get("headers", {})
                )
                healthy = response.status_code == 200
                results.append({
                    "service": endpoint["name"],
                    "status": "healthy" if healthy else f"unhealthy ({response.status_code})",
                    "attempt": attempt + 1,
                })
                if healthy:
                    break
            except requests.RequestException as e:
                results.append({
                    "service": endpoint["name"],
                    "status": f"error: {str(e)}",
                    "attempt": attempt + 1,
                })
            if attempt < max_retries - 1:
                time.sleep(delay)

    return results

Validation Checklist

• Complete inventory of service accounts with dependency mapping
• gMSA implemented for all eligible Windows service accounts
• Cloud access keys rotated via secrets manager (AWS, GCP, Azure)
• Database credentials managed via dynamic secrets (Vault) or rotation policy
• Rotation schedule defined (30-90 days depending on risk level)
• Post-rotation health checks automated
• Alerting configured for rotation failures
• Old credentials revoked after grace period
• Rotation events logged and auditable
• Rollback procedure documented and tested

References

• Google Cloud Service Account Key Rotation
• AWS Secrets Manager Rotation
• Microsoft gMSA Documentation
• HashiCorp Vault Database Secrets Engine