Deadlock that only happens under load

They take the same locks in different orders, and only concurrency makes that visible. Establish a global ordering, usually by sorting the ids you touch before touching them. Retry logic is a bandage: correct, necessary, and not a substitute for the ordering.

Something synchronous is running inside the event loop, usually a blocking library call or CPU work. One blocking call stalls every other task on that loop. Run those in a thread executor, and measure with a loop-lag metric so the next one is visible immediately.