Is removing Bootstrap from a five page site worth the effort?
Usually yes at that size. Grid and flexbox cover the layout, custom properties cover the theming, and you stop shipping 200 KB to style five pages. The part people underestimate is the component behaviour, not the styles: modals and dropdowns have to be rewritten.
Switching to dark theme flashes white on load. How do people avoid it?
Set the theme before first paint with a tiny inline script in the head that reads the stored preference and puts an attribute on the root element. Anything loaded as an external file arrives too late, which is exactly what the flash is showing you.
Any downside worth knowing before I commit?
It commits you to a format that is tedious to migrate away from later. The first weeks also look worse than doing nothing, which is when most people abandon it.