Cosmos DB Safe Key Rotation Without Costly Application Outages

In this blog post Cosmos DB Safe Key Rotation Without Costly Application Outages we will explain how to retire old database credentials without breaking applications, disrupting customers or creating an after-hours recovery exercise.

The risk usually starts with something ordinary. A Cosmos DB key has been exposed, an audit requires rotation, or nobody can confirm when the credentials were last changed. The technology team regenerates the key, only to discover that a forgotten service, scheduled report or integration was still using it.

Safe rotation avoids that problem by treating a credential change as a controlled business change, not a single technical command. One key remains available while the replacement is distributed, tested and monitored.

What Cosmos DB keys actually do

Azure Cosmos DB is Microsoft’s cloud database service for applications that need fast access to data and the ability to operate across regions. An application must prove that it is authorised before it can read or change information in the database.

Many older deployments do this with an account key. It works like a powerful shared password included in the application’s connection details.

Cosmos DB provides primary and secondary read-write keys, plus a separate pair of read-only keys. Both keys in a pair can remain valid at the same time, allowing one to be replaced while applications continue using the other.

This dual-key design is the foundation of safe rotation. The objective is not to regenerate both keys quickly. It is to move every legitimate user from one working key to another before retiring the old credential.

Why apparently simple rotations cause outages

The database is rarely the difficult part. The real challenge is identifying every application, script and external service holding a copy of the credential.

• Production applications may load the key only when they start.
• Background jobs may run weekly, monthly or at financial year-end.
• Developers may have copied connection strings into deployment settings.
• Reporting platforms and support tools may use read-only keys.
• A disaster recovery environment may remain idle until it is needed.
• Third-party providers may manage integrations outside your normal release process.

Updating Azure Key Vault, which securely stores passwords and other sensitive information, does not guarantee that every application has collected the new value. Some services cache their settings and require a restart, configuration refresh or new deployment.

Microsoft also warns that key regeneration can take from a minute to several hours in some circumstances. That makes an improvised change window a poor substitute for a tested runbook.

A safer seven-step rotation runbook

1. Build an inventory before changing anything

List every workload that connects to the Cosmos DB account. Include production systems, testing environments, data exports, monitoring tools, automation, mobile back ends and vendor-managed services.

For each workload, record which key it uses, where that key is stored, who owns the service and how its configuration is refreshed. If your team cannot answer these questions, the immediate priority is discovery rather than rotation.

2. Confirm which key is currently active

Choose one key as the active credential and the other as the rotation candidate. Do not proceed based on naming conventions alone, because an application labelled “secondary” may still contain the primary key.

Where supported and enabled, Cosmos DB account-key usage information can help identify when a key was last used. Safe-key checks can also block a regeneration attempt when the target credential still appears active. This is valuable protection, particularly for infrequent jobs, but it should support rather than replace your application inventory.

3. Regenerate only the inactive key

Once you have evidence that the secondary key is not being used, regenerate that key. The same process works in reverse if applications currently use the secondary key.

RESOURCE_GROUP="production-apps"
COSMOS_ACCOUNT="customer-platform-db"

az cosmosdb keys regenerate \
 --resource-group "$RESOURCE_GROUP" \
 --name "$COSMOS_ACCOUNT" \
 --key-kind secondary

Run this through an approved administrative process rather than from an engineer’s everyday account. Command output, deployment logs and support tickets must never expose the replacement key.

4. Store and distribute the replacement securely

Place the new credential in Azure Key Vault instead of application code, spreadsheets or configuration files. Give workloads permission to retrieve the secret without giving staff broad access to view or export it.

Our guide on securing API keys in Python covers the broader storage and deployment habits that help prevent credentials from leaking into code repositories and build logs.

Update one low-risk application first. Confirm that it can read and write expected data before progressively moving the remaining services.

5. Force applications to collect the new value

Do not assume that changing the secret changes the running application. Confirm whether each service needs to restart, refresh its configuration or receive a new deployment.

Test normal customer transactions as well as less visible functions such as refunds, overnight processing, reports and support tools. The business outcome is continuity across the whole operating cycle, not simply a successful database connection test.

6. Observe before retiring the old key

Monitor failed authentication attempts, application errors, transaction completion and customer-facing availability. Keep the original key valid while this observation takes place so that rollback remains quick.

The monitoring period should cover your real usage pattern. Waiting two hours may be enough for a continuously used platform, but it will not prove that a month-end reporting job has been migrated.

7. Regenerate the old key and record the evidence

After every known workload has moved to the new key, regenerate the old one. This invalidates any forgotten or unauthorised copies and completes the rotation.

az cosmosdb keys regenerate \
 --resource-group "$RESOURCE_GROUP" \
 --name "$COSMOS_ACCOUNT" \
 --key-kind primary

Record who approved the change, which services were tested, the monitoring results and when each old key was retired. This provides useful evidence for risk reviews, customer assurance and Essential 8-aligned governance. Essential 8 is the Australian Government’s cybersecurity framework that many organisations use to improve their security maturity.

What a controlled rotation looks like in practice

Consider a 180-person business with a customer portal, finance integration and monthly billing job connected to one Cosmos DB account. The technology team updates the customer portal successfully and assumes the work is complete.

A proper inventory reveals that the finance integration uses a connection string maintained by an external provider. It also finds that the billing job loads its credential only when a server starts.

By keeping the primary key active, regenerating the unused secondary key and moving each workload in stages, the company can test the portal, provider integration and billing process independently. Only then is the primary key retired.

The outcome is not merely a rotated credential. It is reduced exposure without lost invoices, customer downtime or emergency consulting costs.

The better long-term answer is fewer shared keys

Regular rotation is necessary when account keys remain in use, but the stronger approach is to remove shared credentials wherever practical. For supported Cosmos DB workloads, Microsoft Entra ID and role-based access control allow each application to receive its own identity and only the permissions it needs.

Managed identity goes a step further by allowing an Azure-hosted application to authenticate without storing a database password. Our practical pattern for using managed identity in production explains how this reduces secrets while keeping local development workable.

Do not confuse account access keys with customer-managed encryption keys. Access keys allow applications to connect, while encryption keys protect stored data. Organisations assessing control over encryption should separately review the costs and operational risks of external key management in Azure.

Questions technology leaders should ask

• Can we identify every system using each Cosmos DB key?
• Are credentials stored centrally rather than embedded in code?
• Can applications refresh secrets without an unplanned outage?
• Does monitoring show which credential is still being used?
• Is there a tested rollback process?
• Could managed identity remove the shared key entirely?
• Can our provider produce evidence from the last rotation?

CloudProInc approaches key rotation as an availability and governance issue, not just a database task. As a Melbourne-based Microsoft Partner and Wiz Security Integrator with more than 20 years of enterprise IT experience, we help organisations connect Azure security controls with practical operating procedures.

If you are not sure whether your Cosmos DB keys can be changed safely, or whether forgotten credentials are still in use, we are happy to review the current setup and identify the lowest-risk next step — no strings attached.