Not trying to outright criticize your shooting technique -- you're likely to be more accurate than I -- but cradling the magazine in the palm of your hand seems to be begging for problems, especially in a military-type rifle with loose tolerances. All the mags in all my VZ's are rattle-loose, but none have yet failed.
If you're touching shots on paper at 100 yards and you ignore the above, you might want to try adding some material to the end of the bolt release that touches the magazine follower, then shape it to the extent that it drives the bolt release pin further up into the receiver. Of course, be sure that it completely retracts into the receiver under spring pressure when the follower is not applying pressure. That mod wouldn't be subject to a lot of stress, so something like JB Weld may work.
It may also be that you need a new bolt release. With and the mix-and-match done on these rifles with parts kits manufactured miles and decades apart, there may be tolerance issues easily corrected with a replacement part.
I've also heard about people welding metal to the inside of the receiver to tighten mag fit, mostly on AK's. For AR's, I've read about the installation of one side of velcro tape. I have no experience with either. The welded metal mod should probably be the last resort.