If that is the case, and I don't doubt you, then one would think that it wouldn't be hard to carry on your "worthless" permit inside your restrictions.
That being said, restrictions are no less unconstitutional regardless of your specific circumstances. I would strongly suggest that you bone up on the HPRB threads and attend a meeting or two.
It's worthless, IMO, because the activity is one I rarely engage in and thus for perhaps a few hours every couple months I might use it. At that rate, the matter of hours it might be valid aren't worth carrying to and from a single location.
If I could divine a reasonable case, I'd make the effort to appeal to the HPRB.