Encrypted saved objects unusable after key change: isolating Kibana alerting connector secrets in a test cluster

Answer to the exact question

1. When the encryption key differs or is absent, Kibana starts normally. The encrypted connector objects remain in the database, but decryption is attempted only when the object is read by the UI, the Dev Tools API, or when an action is executed. If the key is wrong, a Failed to decrypt error is logged and the action fails. This behaviour is consistent in Kibana 8.7 and later; earlier 8.x releases behaved the same.
2. Isolation pattern: The supported way to keep test credentials separate is to run the test cluster with its own encryption key and import connector fixtures that were created with that key. Kibana provides no external secret‑injection mechanism for connectors; all secrets must live in the encrypted saved‑objects store.
3. Verifying decryption health: You can look for Failed to decrypt or encryption key missing entries in kibana.log. A quick way to confirm the key works is to retrieve the connector via the Dev Tools console (or curl) and check that the request succeeds without a decryption error. The actual secret is not exposed in the log; only the error message is.

Likely explanation vs confirmed facts

• Likely: The key change might prevent Kibana from starting. Confirmed: Kibana starts; errors surface only when a connector is accessed.
• Likely: An external secret manager can be plugged into Kibana for connectors. Confirmed: No such plug‑in exists; connectors rely solely on encrypted saved objects.

Practical steps for this test scenario

1. Generate a 32‑byte random key and base64‑encode it (the format required by xpack.encryptedSavedObjects.encryptionKey):

   openssl rand -base64 32 > test.key
   

2. Configure the test Kibana instance (e.g., kibana.yml):

   xpack.encryptedSavedObjects.encryptionKey: <contents‑of‑test.key>
   

3. Export connector objects from the production cluster using Dev Tools or curl:

   curl -X GET "http://prod-kibana:5601/api/alerting/connector/{connectorId}" \
        -H 'kbn-xsrf: true' \
        -u admin:changeme > prod-connector.json
   

4. Re‑encrypt the exported JSON with the test key. Kibana ships a CLI helper called bin/kibana-encrypted-saved-objects that can re‑encrypt a JSON file:

   bin/kibana-encrypted-saved-objects reencrypt \
     --key-file test.key \
     --input prod-connector.json \
     --output test-connector.json
   

   (Note: the exact CLI flags may vary; consult the Kibana 8.x documentation for your version.)
5. Import the re‑encrypted connector into the test cluster via Dev Tools or curl:

   curl -X POST "http://test-kibana:5601/api/alerting/connector/_import" \
        -H 'kbn-xsrf: true' \
        -H 'Content-Type: application/json' \
        -d @test-connector.json \
        -u admin:changeme
   

6. Verify decryption health:
   • Check kibana.log for any Failed to decrypt entries.
   • Retrieve the connector again and confirm the request returns 200 OK without a decryption error.
7. Run an end‑to‑end rule that references the test connector and observe the action execution log for any decryption‑related messages.

What’s still unknown?

Could you confirm the exact Kibana version you’re running? The CLI helper names and the exact API paths can differ slightly between 8.7, 8.8, and newer releases.

Safety notes

• Never use the production encryption key in any test cluster; it would expose live credentials if the test environment is compromised.
• Changing the key without re‑encrypting existing objects permanently locks those objects; they cannot be recovered.