How many users do I need for an A/B test

It depends on the effect you would act on, not the effect you hope for. Decide the smallest difference worth shipping, then compute the size for that. Running until significance appears is the most common way to get a result that will not replicate.

Yes: do the first step, skip the automation, and revisit in a month. Most of the value is in the first step, and most of the cost is in making it repeatable before you know it is right.