You
How many users do I need for an A/B test?
ChatGPT
It depends on the effect you would act on, not the effect you hope for. Decide the smallest difference worth shipping, then compute the size for that. Running until significance appears is the most common way to get a result that will not replicate.
You
How long does this usually take in practice?
ChatGPT
An afternoon to do it properly the first time, then minutes when it recurs. Most of the first afternoon goes into discovering which of the defaults do not match your situation.