Plan your experiment, gather your data, do your inference for all effects and interactions of interest. When all is said and done, accept some level of uncertainty in your conclusions: you might not be 97.5% sure that the treatment effect is positive, but that’s fine. For one thing, decisions need to be made. You were already going to make some decision with much less information—that is, with much more uncertainty. Now that you have more information, you can make a more informed decision. The other thing is, even if you did have a super-clean experiment with excellent measurements and a large and stable effect, so that you had that 95% interval excluding zero for your quantity of interest . . . so what? Whatever you care about is in the future, so even if your treatment was so great compared to the alternative in your sample being studied, there’s no saying what it will be in future populations under different conditions.
That’s not to say that you can’t learn from data; I’m not saying that at all. You can learn a lot from data. But forget about 95%. Just do your best, live your life, and be open about your uncertainties. You might get run over by a bus tomorrow anyway.