What makes an error message good?
It names what failed, what was expected, and what to do next. "Invalid input" fails all three; "expected a date in YYYY-MM-DD, got 12/03/2026" passes the first two and hints at the third, which is usually enough.
Every handled exception ends up in the logs and drowns the real ones.
Log at the level where the decision is made, once. An exception caught, handled and logged as an error is a lie about severity; if the program recovered, it is at most a warning, and if it did not, it should reach the top and be logged there with context.
Our retries turned a blip into a full outage.
Synchronised retries are an attack on your own service. Exponential backoff with jitter and a cap, plus a circuit breaker that stops trying entirely once failures dominate. Without jitter every client retries at the same instant, which is precisely the thundering herd.