1. Why KMS Exists — The Problem It Solves
Encryption is easy. Key management is hard. A KMS centralizes the lifecycle of cryptographic keys — creation, rotation, access control, auditing, and destruction — so application developers never have to store, rotate, or protect raw key material themselves.
Without KMS — The Problems
Keys live next to data
Encryption key stored in same database / config file as encrypted data. Attacker who reads the DB gets the key too.
No rotation
Rotating keys manually across 20 services is error-prone and rarely done — old compromised keys stay active for years.
No audit trail
You can't answer "who decrypted this data at 3am on Tuesday?" — critical for compliance (PCI-DSS, HIPAA, SOC 2).
Key sprawl
Different teams use different keys with no central inventory. When a breach happens, you don't know what was exposed.
With KMS — The Solutions
Keys never leave the HSM
Raw key material stays inside a Hardware Security Module. Your app sends data to KMS for encryption — the key itself is never transmitted.
Automatic rotation
Enable key rotation with one flag. KMS handles re-wrapping of data keys transparently. Old key versions kept for decryption only.
Every call is logged
CloudTrail / Cloud Audit Logs capture every Encrypt, Decrypt, GenerateDataKey call with who, when, what, from where.
IAM-gated access
Granular permissions: "service A can encrypt, service B can decrypt, only admin can delete". Revoke access instantly.
2. Cryptography Fundamentals You Need
Same key encrypts AND decrypts. Very fast — used for bulk data encryption. The challenge: how do you securely share the key?
- AES-128, AES-256 — standard choices
- GCM mode = encryption + integrity check
- Used for: S3 objects, DB columns, disk volumes
Public key encrypts, private key decrypts (or vice versa for signing). Slower — used for key exchange and digital signatures.
- RSA-2048/4096, EC P-256/P-384
- Used for: TLS, signing JWTs, key wrapping
- Never use for bulk data — use symmetric instead
Private key signs data. Anyone with the public key can verify authenticity. Proves who created the data and that it wasn't tampered with.
- Used for: code signing, JWT signing, document signing
- ECDSA (faster), RSA PKCS#1 v1.5, RSA-PSS
Hash-based Message Authentication Code. Proves integrity + authenticity without public/private key complexity.
- HMAC-SHA256 most common
- Used for: API request signing, webhook verification
- KMS GenerateMac / VerifyMac operations
The Core Principle: Never Encrypt Data Directly with a KMS Key
KMS keys (CMKs/KMS Keys) are not used to encrypt your data directly — they are used to encrypt the data key that encrypts your data. This two-level pattern is called Envelope Encryption and is the most important concept in KMS.
3. Key Hierarchy — Three Levels
Every cloud KMS follows the same three-level hierarchy. Understanding this is the foundation for everything else.
Root of Trust / HSM Master Key
Lives INSIDE the Hardware Security Module (HSM). Never exportable. Never visible to anyone — not even cloud provider employees. Derived from true random number generator at HSM initialization. Protects all KMS keys.
KMS Key / Customer Master Key (CMK)
Managed by YOU via the KMS console/API. Has ARN/resource ID, IAM policies, usage logs. This key never leaves the KMS — you call KMS to use it, the key bytes never come to your app. Used only to wrap/unwrap Data Keys.
Data Encryption Key (DEK)
A temporary AES-256 key generated by KMS on-demand. This key does leave KMS — it comes to your app to encrypt data. Immediately discarded after use. KMS returns BOTH the plaintext DEK (use once, then discard) AND the encrypted DEK (store alongside your data).
Your Data (Ciphertext)
Encrypted by the plaintext DEK. Stored in S3, RDS, DynamoDB, your database, etc. Alongside it, you store the encrypted DEK (ciphertext). To decrypt: call KMS to decrypt the DEK, then use the DEK to decrypt data, then discard the DEK again.
4. Envelope Encryption — The Core Pattern
Envelope encryption solves the problem of encrypting large amounts of data without sending all that data through KMS (which would be slow and expensive).
Encrypt Flow
Generate Data Key
Call kms.generateDataKey(keyId) → KMS returns:
• Plaintext DEK (32 bytes, use once)
• Encrypted DEK (wrapped by CMK, store this)
Encrypt Data Locally
Use plaintext DEK to encrypt your data with AES-256-GCM. This happens entirely in your application — no round-trip to KMS per record.
Discard Plaintext DEK
Zero out the plaintext DEK from memory. Store only: ciphertext data + encrypted DEK side by side.
Decrypt Flow
Retrieve Encrypted DEK
Load the encrypted DEK you stored alongside the ciphertext. It's useless without KMS — an attacker who steals your DB only has encrypted data AND an encrypted key.
Decrypt the DEK
Call kms.decrypt(encryptedDEK) → KMS checks your IAM permissions, logs the call, returns plaintext DEK. One KMS call per batch, not per record.
Decrypt Data Locally + Discard
Use plaintext DEK to decrypt data with AES-256-GCM. Zero out DEK from memory. Your data is now plaintext. KMS was called exactly once, not per record.
ENVELOPE ENCRYPTION — VISUAL FLOW
5. Cloud Providers — Deep Dive
- AES-256, RSA 2048/3072/4096, ECC P-256/P-384/secp256k1, HMAC
- Customer-managed keys (CMK) vs AWS-managed vs AWS-owned
- Multi-region keys: replicate key across regions for DR
- Key aliases: human-readable names (alias/my-key)
- CloudTrail integration: every KMS call logged automatically
- Automatic annual rotation (symmetric keys)
- AWS CloudHSM: dedicated single-tenant HSM hardware
- External key store (XKS): key material hosted outside AWS
- Integration: S3, RDS, EBS, Lambda env vars, Secrets Manager, SSM Parameter Store, DynamoDB
- Symmetric: AES-128/256 GCM, AES-256 CBC/CTR
- Asymmetric: RSA 2048/3072/4096, EC P-256/P-384
- HMAC-SHA256/512
- Key ring → Key → Key version hierarchy
- Cloud HSM: FIPS 140-2 Level 3 (same level as AWS)
- External Key Manager (EKM): keys hosted outside GCP
- Autokey: automatically creates and manages keys per service
- Cloud Audit Logs: Data Access logs per key operation
- Integration: GCS, BigQuery, Cloud SQL, Pub/Sub, Secret Manager
- Keys: RSA 2048/3072/4096, EC P-256/P-256K/P-384/P-521
- Secrets: arbitrary blobs (connection strings, API keys)
- Certificates: full X.509 lifecycle management
- Managed HSM: dedicated single-tenant, FIPS 140-2 Level 3
- Soft-delete + Purge protection: 7–90 day recovery window
- RBAC at vault or individual secret/key level
- Private endpoint: access vault only from inside VNet
- Integration: Azure Disk Encryption, Azure Storage, SQL TDE, AKS
- Transit secrets engine: Encrypt-as-a-Service (no key export)
- Dynamic secrets: generate DB credentials on demand, auto-expire
- PKI engine: internal CA, auto-rotating TLS certificates
- Auth methods: Kubernetes, AWS IAM, OIDC, LDAP, AppRole
- Kubernetes injector: auto-inject secrets as env vars / files
- Seal types: auto-unseal with AWS KMS / GCP KMS / Azure Key Vault
- Audit log: every operation logged to file / syslog / socket
- Namespaces (Enterprise): multi-team isolation
6. Key Types & Algorithms
| Type | Algorithm | Key Size | Use Case | Supported By |
|---|---|---|---|---|
| Symmetric | AES-GCM | 256-bit | Encrypt/decrypt data (envelope encryption) | AWS GCP Azure Vault |
| Asymmetric Encrypt | RSA-OAEP | 2048 / 4096 | Encrypt small data, key wrapping | AWS GCP Azure |
| Asymmetric Sign | RSA-PSS / ECDSA | P-256 / P-384 | Code signing, JWT signing, document integrity | AWS GCP Azure |
| HMAC | HMAC-SHA256/512 | 256–512 bit | Message authentication, token signing | AWS GCP |
| ECC | secp256k1 | 256-bit | Blockchain / Bitcoin-compatible signing | AWS GCP |
7. Core KMS Operations — Code Examples
8. Native Cloud Integrations
Cloud services natively integrate with KMS. You don't write encryption code — you just point the service at a KMS key.
Server-side encryption (SSE-KMS). Per-object or per-bucket. Each object gets unique DEK wrapped by your CMK. Bucket policies can deny non-KMS uploads.
Storage encryption at rest. Select CMK at DB creation. Automated backups and snapshots inherit encryption. Cannot encrypt existing unencrypted DB in-place.
Volume encryption. CMK wraps the volume DEK stored in the EBS service. Can set account-level default encryption for all new EBS volumes.
Stores secrets encrypted by KMS CMK. Automatic rotation of DB credentials, API keys. Pull secrets at runtime — no hardcoding. $0.40/secret/month.
Encrypt Lambda environment variables with CMK. KMS decrypts at cold start. Use KMS instead of plaintext env vars for API keys / DB passwords.
Table encryption at rest with CMK. DynamoDB-owned key (free), AWS-managed ($0), or CMK ($1/month). Encryption is transparent to all API calls.
Customer-managed encryption keys (CMEK) at bucket or object level. Default encryption with Google-managed keys is free; CMEK uses Cloud KMS pricing.
CMEK for database encryption at rest. Specify key during instance creation. Replicas inherit the key. Key disablement immediately blocks all DB operations.
CMEK for dataset and table encryption. Can also use column-level encryption with AEAD functions referencing Cloud KMS keys in queries.
GCP equivalent of AWS Secrets Manager. Stores secrets encrypted with Cloud KMS. IAM-gated access, version history, automatic secret rotation (via Cloud Functions).
Encrypts OS and data disks using BitLocker (Windows) or DM-Crypt (Linux). Keys stored in Key Vault. Enables on running VMs without restart.
Transparent Data Encryption (TDE) with customer-managed keys in Key Vault. Key Vault Managed HSM for highest compliance requirement. Cannot change key after creation.
Customer-managed keys for Blob, Files, Queue, Table. Key rotation automatically re-encrypts storage service encryption keys (not the data) — zero downtime.
AKS: etcd encryption with Key Vault keys, CSI Secrets Store driver mounts Key Vault secrets as volumes. App Service: reference Key Vault secrets in app settings.
9. Key Rotation — How It Actually Works
What rotates and what doesn't
New key material is generated
KMS generates new cryptographic key material for the CMK. The old material is NOT deleted — it's kept for decryption only.
New encryptions use new material
Any new Encrypt or GenerateDataKey call uses the new key material automatically. No code changes needed.
Old ciphertext still decryptable
KMS metadata in the ciphertext blob identifies which key version encrypted it. Old data decrypts with old material, new data with new. Transparent to your app.
Data is NOT re-encrypted automatically
Your S3 objects/DB records still use their old DEKs (wrapped by old key material). You must re-encrypt data yourself if you want forward secrecy at the data layer.
Rotation settings
10. HSM & FIPS 140-2 — Hardware Security
Hardware Security Module — a physical computing device dedicated to managing cryptographic keys and performing crypto operations. Tamper-resistant: physically destroys key material if someone tries to open it.
- True hardware random number generator (TRNG)
- Cryptographic operations happen inside the chip
- Cannot extract keys even with physical access
- FIPS 140-2 Level 3 = highest standard for cloud HSMs
Federal Information Processing Standard for cryptographic modules. 4 security levels:
- Level 1: Software only, basic requirements
- Level 2: Physical tamper evidence (seals)
- Level 3: Tamper resistance + key destruction on breach ← cloud HSMs
- Level 4: Complete physical security envelope
Standard KMS (AWS, GCP, Azure) uses multi-tenant shared HSMs — your key stays isolated but hardware is shared.
- AWS CloudHSM: single-tenant HSM cluster (~$1.60/hour)
- Azure Managed HSM: single-tenant (~$3.20/hour)
- GCP Cloud HSM: per-key HSM protection (no dedicated hardware)
- Required for: PCI-DSS Level 1, some government regulations
- PCI-DSS Level 1 (card brand mandate)
- Government / defense (FedRAMP High, DoD IL4+)
- BYOK with proof key never touched cloud provider
- eIDAS / qualified electronic signatures (EU)
- Own crypto officer and HSM administration controls
11. Common Usage Patterns
Encrypt sensitive columns (PII, PAN, SSN) with field-level encryption. Store encrypted DEK per record for key isolation.
Kubernetes encrypts Secrets at rest in etcd using a KMS plugin (envelope encryption via external KMS).
12. Provider Comparison
| Feature | AWS KMS | GCP Cloud KMS | Azure Key Vault | HashiCorp Vault |
|---|---|---|---|---|
| Deployment | AWS-managed | GCP-managed | Azure-managed | Self-hosted / HCP |
| FIPS Level | Level 2 (std) / Level 3 (HSM) | Level 3 (Cloud HSM) | Level 3 (Premium/Managed) | Depends on backend |
| Multi-cloud | AWS-only | GCP-only | Azure-only | Any cloud / on-prem |
| Secrets engine | Secrets Manager (separate) | Secret Manager (separate) | Key Vault Secrets (built-in) | Built-in (KV, dynamic) |
| Dynamic secrets | Via Secrets Manager rotation | Via Secret Manager + Cloud Functions | Via Key Vault + Functions | Native (DB, cloud, SSH) |
| Audit logging | CloudTrail (all calls) | Cloud Audit Logs | Azure Monitor Diagnostics | Audit log to file/syslog |
| Key pricing | $1/key/month + calls | $0.06/version/month + calls | $0/key (std vault fee) + ops | Open source = infra cost |
| External key material | XKS (bring your own) | EKM | BYOK | Full control |
| Best for | AWS workloads, deep S3/RDS integration | GCP-native, BigQuery encryption | Azure AD integration, certificates | Multi-cloud, Kubernetes, on-prem |
13. Best Practices
Key Design
service=checkout, env=prod, data-class=pci for cost allocation and auditAccess Control
kms:GenerateDataKey + kms:Decrypt only, not kms:DeleteKeykms:ViaService condition to restrict key use to specific AWS services onlykms:ScheduleKeyDeletion and kms:DisableKeyKMSInvalidKeyState and unexpected decrypt volumesApplication Patterns
buffer.fill(0))ThrottlingExceptionThe Mental Model: KMS is a Key Custodian, Not an Encryption Service
Think of KMS like a bank vault for keys, not a processing plant for data. You don't bring all your data to the bank to encrypt it — you go to the bank to get a locked box (encrypted DEK), take it home, unlock it (decrypt DEK via KMS), use what's inside (encrypt/decrypt data locally), put the box back empty, and return the locked box to storage. The bank never sees your data. You never see the master key. The audit log shows every time you visited the bank. That's envelope encryption.
• Storing plaintext secrets in environment variables instead of Secrets Manager / Key Vault
• Using the same CMK for all environments (dev key compromise shouldn't endanger prod)
• Not setting up CloudTrail alerting on unexpected KMS decrypt activity (potential exfiltration indicator)
• Encrypting with CBC mode instead of GCM — CBC has no integrity check, vulnerable to padding oracle attacks
• Re-using the same IV/nonce with the same key — breaks GCM security completely