Kubernetes secrets are just base64. Is that a problem?
It is, if that is the only layer. Enable encryption at rest for etcd, restrict RBAC on the secrets resource, and keep them out of the manifests in git. Base64 is an encoding, and treating it as protection is the most common misunderstanding in this area.
A pod restarts every few minutes and the logs end normally.
Look at the previous container, not the current one: kubectl logs pod --previous. A clean ending plus a restart usually means OOMKilled, which shows in kubectl describe pod under Last State. Memory limits are the first thing to check, requests the second.
Any downside worth knowing before I commit?
It commits you to a format that is tedious to migrate away from later. The first weeks also look worse than doing nothing, which is when most people abandon it.