In my experience a dot can make it harder to shoot small groups. It's target dependent. Target size/shape/aiming point all contribute to my issues.
If the dot is larger than the target it makes it tough to maintain POA - if you are trying to center the dot on the target.
If your POA is the bottom of the target then it's tough to always have the same gap/lack of gap between the dot and the bottom of the target.
Ideally, for me, the dot needs to be slightly smaller than the target (round bulls eye) so I can get a ring of target visible around the dot.
I have, several times over the years, installed a red dot optic on a rifle, only to eventually put a scope on it because the dot holds me back on group size.
We can argue about how important tiny (smallest possible) groups are with a self defense handgun but you are talking group size.
I'm not suggesting you take the dot sight off (I too suffer from inability to focus on that front sight without a good set of glasses) I'm just saying that you may benefit from getting the right sized/shape target for the distance you're shooting and the dot size on your sight.
Now, the flyers. Do you have good enough technique/trigger control that you know, I mean know it, when your pistol goes off and the sights are not lined up? When I let one go and the dot (or front sight) isn't where I wanted it to be I know it. I read you were shooting from a rest, but does the dot still move in relation to your aiming point? Do you see the movement is, sometimes, not quite on the POA when the hammer is released?
How consistent is your powder measure with that powder? Do you weigh out the charges when developing a load? Do you, sometimes, weigh several measured charges to see if it is consistent with a powder or powder charge amount/setting? I've seen some powders hang up below the rotor and result in a charge being "light" and then that powder drops with the next charge and results in a "heavy" load on the very next cartridge. Some powders seem to take a very aggressive smacking/knocking of the charging handle on the powder measure to prevent that. Others flow like water, seemingly.
Have you measure the diameters of some of the bullets? Have you weighed them (sorry if you did and I missed it in your post) to see if they are consistent in weight.
Good luck with it. Sometimes, you just find that some bullets are more accurate than others.