Embeddings for search on a small corpus

Often not. Lexical search handles exact terms, names and identifiers, which is most of what people type. Embeddings pay off for paraphrase-heavy queries, and the honest setup is hybrid, with lexical results ranked first when the query looks like an identifier.

When you stop noticing it. If it has not surfaced a problem in six months, it is either solved or it was never the bottleneck you thought.