I reported this bug previously by email in April, and the fix that was applied has not quite addressed the issue.
When non-browser agents (e.g. web crawlers and headless fetchers) access posts via my custom domain they are getting a 401 error, while the same post loads fine via the leaflet.pub subdomain URL. The fix added a regex for specific bot useragents, but that's a rather fragile fix that doesn't really address the underlying issue.
Adding ?auth_completed=true to the end of the URL makes it accessible, but that doesn't really help since ?auth_completed=true is not in the canonical URL. A non-browser agent (or even a browser agent) should not need to be authenticated to access public content. Also, if simply adding that to the end of the URL makes it work, then whatever this auth_completed thing is, it is not actually an authentication flow.
This is particularly problematic because a custom domain becomes the canonical URL, even for the leaflet.pub subdomain version. Therefore, at this point, adding a custom domain, rather than enhancing a Leaflet publication, completely breaks SEO and anything else that requires fetching through anything other than a browser. It's paradoxical that adding an enhancement degrades the functionality.
I have tested this with other Leaflet publications that use custom domains and the behaviour is the same, so it's not an issue with my own DNS setup. Ultimately, request handling should transparently be the same for both the custom domain and the leaflet.pub subdomain.
Here is the commit where this was addressed the first time around:
3ebf9823
I think this patch was treating the symptom rather than the problem.
You shouldn't need an exception that lets bots through, because there shouldn't even be a thing that they need to be given an exception to. A public page shouldn't just be straight up broken for anything that can't load Javascript or cookies. They should at least get something.