openstack-keystone

name	openstack-keystone
description	OpenStack Keystone identity service skill for deploying, configuring, operating, and troubleshooting the authentication and authorization backbone of an OpenStack cloud. Covers identity management, token lifecycle (Fernet provider with rotation), service catalog registration, RBAC policy customization, domain/project/user hierarchy, federation basics (SAML/OIDC), credential encryption, and endpoint management. Use when deploying Keystone via Kolla-Ansible, managing users and projects, debugging 401 errors, rotating Fernet keys, configuring RBAC policies, or integrating services through the service catalog.
user-invocable	true
allowed-tools	Read Grep Glob
metadata	{"extensions":{"gsd-skill-creator":{"version":1,"createdAt":"2026-02-22","triggers":{"intents":["keystone","identity","authentication","service catalog","RBAC","token","federation","policy.json","endpoint"],"contexts":["deploying openstack","configuring identity","troubleshooting authentication","managing users and projects"]}}}}

OpenStack Keystone Identity Service

Keystone is the identity service for OpenStack. Every API call to every OpenStack service passes through Keystone for authentication and authorization. It is the first service deployed and the last decommissioned. If Keystone is down, the entire cloud is down.

Keystone provides five core functions: identity (users, groups), resources (projects, domains), assignment (roles mapped to users on projects), token (authentication proof with configurable lifetime), and catalog (service endpoint registry). Understanding Keystone means understanding how every OpenStack service discovers and trusts every other service.

Deploy

Kolla-Ansible Configuration

globals.yml settings:

# Required -- set strong passwords
keystone_admin_password: "{{ vault_keystone_admin_password }}"
keystone_database_password: "{{ vault_keystone_database_password }}"

# TLS (recommended for production)
kolla_enable_tls_internal: "yes"
kolla_enable_tls_external: "yes"
kolla_copy_ca_into_containers: "yes"

# Token provider (Fernet is default and recommended)
keystone_token_provider: "fernet"

# Optional tuning
keystone_token_expiration: 3600  # seconds, default 1 hour

Deployment sequence:

# 1. Bootstrap -- creates databases, service users, initial endpoints
kolla-ansible -i inventory bootstrap-servers

# 2. Deploy Keystone (runs as part of full deploy or targeted)
kolla-ansible -i inventory deploy --tags keystone

# 3. Post-deploy -- creates admin credentials file
kolla-ansible -i inventory post-deploy

Container verification:

# Verify Keystone containers are running
docker ps --filter "name=keystone" --format "table {{.Names}}\t{{.Status}}"
# Expected: keystone_api, keystone_fernet (both Up)

# Verify service responds
source /etc/kolla/admin-openrc.sh
openstack token issue  # Must return a valid token table

Service catalog registration:

Kolla-Ansible auto-registers Keystone in the service catalog. Verify:

openstack service list  # Should show "identity" service
openstack endpoint list --service keystone
# Expected: 3 endpoints (public, internal, admin) per region

Initial admin setup:

# admin-openrc.sh is generated by post-deploy
source /etc/kolla/admin-openrc.sh

# Verify admin project and user exist
openstack project show admin
openstack user show admin
openstack role assignment list --user admin --project admin
# admin should have "admin" role on "admin" project

Configure

Fernet Token Provider

Fernet tokens are cryptographic tokens validated without database lookup. They require synchronized key repositories across Keystone nodes.

Key repository: /etc/kolla/keystone/fernet-keys/ (inside container)

Key types:

• Staging key (index 0): Used for decryption only; becomes primary on rotation
• Primary key (highest index): Used for encryption; tokens it created remain valid
• Secondary keys (middle indices): Decrypt only; pruned after max_active_keys exceeded

Rotation procedure:

# Rotate Fernet keys (Kolla-Ansible managed)
kolla-ansible -i inventory keystone_fernet_rotate

# Manual rotation (if needed)
docker exec keystone_api keystone-manage fernet_rotate \
  --keystone-user keystone --keystone-group keystone

Configuration: keystone.conf [fernet_tokens]

• max_active_keys: Default 3. Set to (token_expiration / rotation_interval) + 2

Domain, Project, and User Hierarchy

Domain (organizational boundary)
  +-- Project (resource container, formerly "tenant")
       +-- User (identity with credentials)
       +-- Group (collection of users)
       +-- Role Assignment (user/group + role on this project)

Best practices:

• Create a domain per organization or department
• Use the "default" domain for service accounts
• Never modify the "admin" project; create separate operator projects
• Assign roles to groups, not individual users, for scalability

RBAC Policies

Policy files: /etc/kolla/keystone/policy.yaml (override defaults)

# Example: restrict user creation to domain-scoped admins
"identity:create_user": "rule:admin_required and domain_id:%(target.user.domain_id)s"

# Example: allow project members to list users in their project
"identity:list_users": "role:member and project_id:%(scope.project.id)s"

Policy evaluation order:

1. Check explicit rule match in policy.yaml
2. Fall back to code default (oslo.policy)
3. Deny if no rule matches

Verify policy changes:

# Test a specific policy rule
openstack --os-auth-url http://keystone:5000/v3 \
  --os-username testuser --os-password testpass \
  --os-project-name testproject --os-user-domain-name default \
  --os-project-domain-name default user list

Service Catalog Management

# List all registered services and endpoints
openstack service list
openstack endpoint list

# Register a new service
openstack service create --name nova --description "Compute" compute
openstack endpoint create --region RegionOne \
  compute public http://controller:8774/v2.1
openstack endpoint create --region RegionOne \
  compute internal http://controller:8774/v2.1
openstack endpoint create --region RegionOne \
  compute admin http://controller:8774/v2.1

Federation (SAML/OIDC)

Federation allows external identity providers to authenticate OpenStack users.

Key concepts:

• Identity Provider (IdP): External auth source (AD, Okta, Keycloak)
• Mapping: Rules that translate IdP assertions to Keystone users/groups/projects
• Protocol: SAML 2.0 or OpenID Connect
• Service Provider (SP): Keystone acting as relying party

Configuration reference: keystone.conf [federation], [saml], [openid]

Credential Encryption

# Initialize credential encryption (first time)
docker exec keystone_api keystone-manage credential_setup \
  --keystone-user keystone --keystone-group keystone

# Rotate credential encryption keys
docker exec keystone_api keystone-manage credential_rotate \
  --keystone-user keystone --keystone-group keystone

# Migrate credentials to new encryption
docker exec keystone_api keystone-manage credential_migrate \
  --keystone-user keystone --keystone-group keystone

Operate

User and Project Management

# Create domain
openstack domain create --description "Engineering" engineering

# Create project in domain
openstack project create --domain engineering --description "Dev team" dev-team

# Create user in domain
openstack user create --domain engineering --password-prompt \
  --email [e-mail] devuser

# Assign role
openstack role add --project dev-team --user devuser --user-domain engineering member

# List role assignments
openstack role assignment list --project dev-team --names

Token Debugging

# Issue a token and inspect it
openstack token issue -f json

# Check token expiration
openstack token issue -c expires -f value
# Compare with system clock -- clock skew causes premature expiration

# Revoke a specific token
openstack token revoke <token-id>

Fernet Key Rotation Procedure

Schedule: Rotate at half the token expiration interval. Default token lifetime is 1 hour, so rotate every 30 minutes in production.

# 1. Check current key count
docker exec keystone_api ls /etc/kolla/keystone/fernet-keys/ | wc -l

# 2. Rotate keys
kolla-ansible -i inventory keystone_fernet_rotate

# 3. Verify rotation
docker exec keystone_api ls -la /etc/kolla/keystone/fernet-keys/
# New highest-numbered key should be primary

# 4. Verify tokens still work
openstack token issue

Backup and Restore

# Backup Keystone database
docker exec mariadb mysqldump -u root -p keystone > keystone_backup.sql

# Backup Fernet keys
docker cp keystone_api:/etc/kolla/keystone/fernet-keys/ ./fernet-keys-backup/

# Restore database
docker exec -i mariadb mysql -u root -p keystone < keystone_backup.sql

# Restore Fernet keys
docker cp ./fernet-keys-backup/ keystone_api:/etc/kolla/keystone/fernet-keys/
docker restart keystone_api

Quota Management

# Keystone itself has no quotas, but controls project-level quotas for other services
# Set Nova quotas for a project
openstack quota set --instances 20 --cores 40 --ram 81920 dev-team

# Set Cinder quotas
openstack quota set --volumes 50 --gigabytes 1000 dev-team

Troubleshoot

1. "401 Unauthorized" on All Services

Symptoms: Every openstack CLI command returns 401. All service-to-service calls fail.

Root causes and resolution:

Cause	Diagnosis	Fix
Expired Fernet keys	docker exec keystone_api ls -la /etc/kolla/keystone/fernet-keys/ -- check modification timestamps	kolla-ansible -i inventory keystone_fernet_rotate
Clock skew	date on controller vs compute nodes -- difference > 30s causes token validation failure	Sync NTP: chronyc sources and chronyc tracking
Wrong endpoint in admin-openrc.sh	cat /etc/kolla/admin-openrc.sh -- verify OS_AUTH_URL matches actual Keystone endpoint	Regenerate: kolla-ansible -i inventory post-deploy
Keystone container down	docker ps --filter name=keystone -- not running	docker restart keystone_api keystone_fernet
MariaDB connection failure	docker logs keystone_api 2>&1 | grep -i "database|maria|mysql"	Check MariaDB: docker exec mariadb mysql -u root -p -e "SELECT 1"

2. Service Catalog Endpoint Mismatch

Symptoms: Services fail to find each other. Errors like "Could not find service endpoint" or wrong URLs in API responses.

Diagnosis:

openstack endpoint list --long
# Check: public/internal/admin URLs match actual service locations
# Check: region names are consistent across all services

Fix: Correct the mismatched endpoint:

openstack endpoint set --url http://correct-host:port/v3 <endpoint-id>

3. Token Validation Failures

Symptoms: Tokens issue successfully but fail validation on other services. Intermittent auth failures.

Root causes:

Cause	Diagnosis	Fix
Fernet key mismatch (multi-node)	Compare key files across Keystone nodes	kolla-ansible -i inventory keystone_fernet_rotate to sync
Memcached connectivity	docker logs keystone_api 2>&1 | grep memcache	Verify memcached: docker exec memcached memcstat --servers=localhost
keystonemiddleware misconfigured	Check [keystone_authtoken] in service configs	Verify auth_url, memcached_servers, project_name

4. Database Connection Failures

Symptoms: Keystone API returns 503. Container logs show database errors.

Diagnosis:

docker logs keystone_api 2>&1 | tail -50
# Look for: "OperationalError", "Can't connect to MySQL", "Access denied"

# Test MariaDB directly
docker exec mariadb mysql -u keystone -p -e "USE keystone; SELECT COUNT(*) FROM project;"

Fix:

• Connection refused: docker restart mariadb, verify MariaDB is listening on expected port
• Access denied: Check credentials in /etc/kolla/keystone/keystone.conf [database] section
• Database missing: kolla-ansible -i inventory deploy --tags mariadb,keystone

5. Federation/SSO Issues

Symptoms: External IdP users cannot authenticate. SAML assertions rejected.

Diagnosis:

docker logs keystone_api 2>&1 | grep -i "federation\|saml\|oidc\|mapping"
# Check: certificate trust chain, metadata freshness, mapping rules

Common fixes:

• Update IdP metadata: re-download and reimport
• Fix certificate chain: ensure CA cert is in Keystone trust store
• Debug mapping rules: use keystone-manage mapping_engine to test rule evaluation

6. RBAC Policy Denials

Symptoms: Users get 403 Forbidden despite having seemingly correct roles.

Diagnosis:

# Enable debug logging temporarily
docker exec keystone_api sed -i 's/^#debug = .*/debug = true/' /etc/keystone/keystone.conf
docker restart keystone_api

# Check policy evaluation
docker logs keystone_api 2>&1 | grep "Policy check"
# Look for the specific rule that denied the request

Common fixes:

• Policy syntax error: validate YAML with python3 -c "import yaml; yaml.safe_load(open('policy.yaml'))"
• Rule evaluation order: explicit rules override code defaults; check for conflicting rules
• Scope mismatch: request is project-scoped but rule requires domain-scope, or vice versa

Integration Points

Every OpenStack service authenticates through Keystone. The integration pattern is consistent:

keystonemiddleware configuration (in each service's config file):

[keystone_authtoken]
www_authenticate_uri = http://keystone:5000
auth_url = http://keystone:5000
memcached_servers = memcached:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = nova          # service-specific
password = nova_pass     # service-specific

Service user pattern: Each service has a dedicated user in the service project with the admin role. Kolla-Ansible creates these automatically.

Endpoint discovery: Services find each other through the Keystone service catalog. When Nova needs to fetch an image from Glance, it queries the catalog for the image service endpoint.

Token flow:

1. User authenticates to Keystone, receives scoped token
2. User sends request to Nova with token in X-Auth-Token header
3. Nova's keystonemiddleware validates token against Keystone (or memcached cache)
4. Nova uses its own service user token to call Glance, Neutron, Cinder as needed

NASA SE Cross-References

SE Phase	Keystone Activity	Reference
Phase C (Final Design)	Configure Keystone in globals.yml: admin passwords, TLS settings, token provider, federation	SP-6105 SS 5.1
Phase D (Integration & Test)	Verify: openstack token issue, openstack endpoint list, openstack user list -- prove identity service is operational	SP-6105 SS 5.2-5.3
Phase E (Operations)	Fernet key rotation, user/project management, credential rotation, RBAC policy updates, service catalog maintenance	SP-6105 SS 5.4-5.5, NPR 7123.1 SS 5.4
Phase F (Closeout)	Export user/project data, decommission service catalog entries, archive Keystone database	SP-6105 SS 6.1