It is possible that being “polite” pushes the model into a different position in the latent space which may produce different quality answers.
The theory would be that when people ask polite questions online they may be likely to get better quality responses, ergo when being polite to the model it may be more inclined to do the same.
That said, I’ve not tested this personally, nor am I aware of any specific research into this area.
If you were interested in knowing for yourself you could take a few example prompts for your use case and run each 50 or so times with and without “please” and see for yourself.
I doubt the difference will be so great that the effect size will be obvious, but if you have a way to objectively and quantifiably rate each response, it’s possible you might be able to identify a small but significant difference in the response.