UNABLE_TO_LOCK_ROW retries from middleware are duplicating records — is upsert on an External ID the right fix?

UNABLE_TO_LOCK_ROW and retried writes

We have an Apex integration where inbound middleware retries a request whenever the first attempt fails partway through — for example after a row lock contention error such as UNABLE_TO_LOCK_ROW, or when our own Queueable retry logic re-enqueues a job. The Apex side currently performs a plain insert, so every retry after a partially successful call creates a duplicate record.

My understanding is that upsert records External_Id__c can make a retried DML idempotent: the retry matches the existing record by key and updates it instead of inserting. But this only works if the chosen field is marked External ID and Unique, and I am unsure which business attribute is a genuinely stable, immutable idempotency key across our objects.

There is also the question of which failures deserve a retry at all. Using Database.insert(list, false) returns SaveResult objects, so retryable statuses could in principle be separated from validation failures that would loop forever.

Open questions

1. Is upsert on a Unique External ID field sufficient to make middleware and Queueable retries safe, or can concurrent upserts on the same key still raise DUPLICATE_VALUE or lock errors that need their own catch-and-retry path?
2. What is a sound way to choose the idempotency key when no single business field is naturally unique and immutable?
3. Which SaveResult status codes should be classified as retryable versus terminal before re-attempting the write?