Deploys cause a handful of 502s every time.
The old pod stops accepting before the proxy stops sending. Add a readiness probe and a preStop sleep of a few seconds:
lifecycle:
preStop:
exec:
command: ["sleep", "5"]
That gap gives the endpoint controller time to remove the pod before the process exits.
Kubernetes secrets are just base64. Is that a problem?
It is, if that is the only layer. Enable encryption at rest for etcd, restrict RBAC on the secrets resource, and keep them out of the manifests in git. Base64 is an encoding, and treating it as protection is the most common misunderstanding in this area.
A pod restarts every few minutes and the logs end normally.
Look at the previous container, not the current one: kubectl logs pod --previous. A clean ending plus a restart usually means OOMKilled, which shows in kubectl describe pod under Last State. Memory limits are the first thing to check, requests the second.
What would you skip entirely?
The reporting layer, until something forces it. It is the part that feels productive to build and the part nobody opens twice.