You may have to angle the speedloader outwards and only put it far enough for the bullets to enter the chambers. Then you unlock it and give it a jiggle to drop the rounds in.
An aftermarket grip with more clearance for the speedloaders might also be necessary. I also prefer the Safariland speedloaders to the HKS. They are a little smaller and don't require a jiggle. You just press it against the cylinder and it unlocks itself.