Northflank Secret Rotation: The Incident and Recovery Checks

Northflank is a service for deploying and running containerized applications: web apps, APIs, background workers, scheduled jobs and AI workloads. It also manages databases. Customers run these services on Northflank’s infrastructure or their own cloud, with stored passwords and API keys allowing applications to connect to other services. Northflank product overview

On October 30, 2026, Northflank opened a security investigation after a threat alert from Wiz. Northflank engineers reviewed the alert and requested Wiz’s incident-response support. Later that day, Northflank reported that customers had seen unauthorized use of AI API keys they had supplied to the platform. This was evidence of reported credential misuse, beyond a warning about a possible vulnerability. Northflank incident notice

Northflank told all customers to treat environment variables as compromised, including credentials in workloads, secret groups and global secrets. That is broad precautionary guidance; it does not establish that every customer’s secrets were stolen. The notice does not identify an affected software version, explain how the intrusion happened or establish each customer’s exposure. A connection to a wider campaign remains unconfirmed. Official incident updates

The consequences extend beyond the applications hosted there. An AI API key can authorize requests billed to its owner. A database password can allow access within that account’s permissions. Replacing these credentials can also interrupt applications and jobs that still use the old values. Northflank secret rotation therefore needs four separate checks: replacement credentials deployed, old access disabled, fresh connections working and possible misuse investigated.

What changed during the response

Northflank’s timeline records the following events. All times are UTC. Vendor timeline

These updates expanded recovery options. Audit-log availability does not establish how far back records extend. Check the available history before drawing conclusions about earlier activity.

How to respond if you use Northflank

If you run applications, jobs or database addons on Northflank, the following checks help establish whether replacement credentials reached every consumer, old access was disabled and fresh connections work. Getting an application running again does not show whether its credentials were misused. Review provider activity and available audit records separately from service recovery.

MySQL and Redis rotation work differently

Northflank’s database rotation operations have different effects. Plan for them before starting a change.

MySQL: Usernames and permissions stay the same. The default user-facing mode changes user-facing passwords. The full mode also changes internal passwords and restarts the addon. Both modes keep old passwords valid until finalization. Rotate addon secrets

Finalization permanently removes the old credentials. MySQL user-facing finalization disconnects sessions for the rotated users. Full finalization restarts the addon. These operations change the running database; they are not read-only checks. Finalize addon rotation

Redis: Rotation restarts the addon and revokes old credentials in one step. Clients must reconnect with the new password, and customers must redeploy linked services themselves. Redis without persistence is unsupported. Redis Sentinel requires at least three replicas. Redis does not use the separate finalization endpoint. Redis rotation behavior

Do not apply a MySQL overlap plan to Redis. A Redis client still using the previous password can lose access when rotation completes.

Track each credential and its consumers

Inventory the secrets covered by Northflank’s guidance and the applications that use them. Include background workers and scheduled jobs. They may connect less often than the web application and fail only when their next run begins.

Record the following without copying secret values:

For third-party API keys, revoke the original through the issuing provider and create a replacement. Update every consumer. Changing a stored Northflank secret does not disable the original key at its issuer.

Follow the incident guidance promptly. Do not leave an exposed credential active while waiting for perfect recovery logs. If revocation interrupts a service, record the interruption and repair the consumer with the replacement. Northflank also advised disabling public addon access where enabled. Customer guidance

Keep passwords, tokens and connection strings out of tickets and logs. Revision labels such as db-r2 should be ordinary identifiers, not fragments of the secret.

Verify a fresh database connection

A request can succeed through a connection opened before rotation. That does not show that a new process can authenticate with the replacement password.

After the relevant deployment and rotation steps, check a fresh connection from the affected application environment. Use an approved read-only health check. Confirm which credential revision the process loaded without printing its value. Check workers separately; a healthy web instance does not establish that a scheduled job has updated credentials.

The following lines are synthetic application logs, not Northflank incident evidence or guaranteed database output. They illustrate safe revision labels:

2026-10-31T14:02:10Z service=checkout-api credential_revision=db-r1 event=db_connect result=failure reason=authentication_rejected
2026-10-31T14:05:22Z service=checkout-api credential_revision=db-r2 event=db_connect result=success connection=fresh

The first line records a rejected login. The second records a new successful connection using the application’s replacement configuration. Neither proves that the old credential was revoked. Keep the provider or addon completion record separately.

Distinguish authentication failures from network failures. A rejected password calls for checking the selected credential and its permissions. A timeout, DNS failure or refused connection needs a connectivity check. Preserve the actual error and timestamp rather than treating every database error as a failed rotation.

If successful connections are not logged, keep the approved health-check result as evidence. An empty error search cannot replace a positive check. For a broader method of correlating surrounding events, see how to read logs during an incident.

Watch fresh connection logs in Fluxtail

If the affected workload logs already reach Fluxtail, its live log viewer can show connection attempts as they arrive. Use only fields your collector actually delivers and maps.

For events shaped like the synthetic example:

1. Open the existing stream receiving checkout-api events and start live tail.
2. Select the service through its mapped service field. Filter message text for the connection event your application emits, such as db_connect, if that text is present in the message.
3. Record the UTC start and end of a short observation period around an approved fresh-connection health check. This records when you watched; it does not select historical events.
4. Inspect arriving rows for event times, credential revision labels and connection results. If those values arrive as structured fields, inspect the fields directly.
5. Check workers when they next connect. Record which consumers passed and which still need a check.

These are live-view selections, not a query-language expression. Read the actual row before assigning a cause. The live-tail incident response guide explains how to follow active failures.

Live tail cannot show an event that never reached the receiver or establish provider-side revocation. Missing rows may also mean the application does not log successful connections. Keep health-check results, earlier available logs and revocation confirmations alongside the observed events.

Check misuse separately from service recovery

Preserve available AI-provider usage and billing records, platform audit events and workload logs. Restrict access to exports and set an appropriate retention period; they may contain customer or operational data.

Compare unexpected provider activity with known deployments and legitimate usage. Record the provider’s credential identifier, event time and relevant request identifier where available. Do not copy the credential itself. Report unexplained usage to Northflank support and the issuing provider.

Check the earliest available audit event, recorded event categories and gaps before drawing conclusions. A successful database connection says nothing about earlier API-key misuse. Likewise, an empty search cannot establish that no misuse occurred.

Keep four findings separate: replacement credentials deployed, old credentials disabled, fresh connections verified and misuse investigated. Leave unresolved findings explicit even after the application works again.