That was also my experience. Shooting factory 41 Magnum JSP back in the day wasn't a good choice for someone trying to compete in IHMSA matches so I was forced to reload by necessity. Got me a Lyman turret press, a set of RCBS carbide dies and had a friend come over and show me how to do it. Simple and, at the same time, extremely satisfying. If I still had the available time, I would jump right back into it again.
.41 Remington Magnum is a great example (one of many) of an outstanding cartridge that most would probably never even consider, save having either tycoon deep pockets, or a reloading press. Unlike wildly popular 9mm Luger as an example, for which the strict cost savings spread is tight enough to make one question even the utility of loading for it, .41 Mag will very quickly remove any such doubt. And that cartridge in a 57 or a 58 or a Blackhawk is just a treat. OP will find that he'll have possibilities open to him that he likely hasn't even thought of....yet.