macOS is configured to use my custom DNS resolver

Support for platform-specific DNS APIs is currently being worked on in Nightly (Bug 1852752). The setting for this is network.dns.native_https_query in about:config.

Thank you for your response. If I'm understanding that Bugzilla ticket that Bugzilla ticket correctly, it specifically relates to the task of resolving the "HTTPS" resource record type (as opposed to the "A" type, "AAAA" type, and so forth) without DoH. That seems like a different problem from what I'm experiencing.

My problem is that Firefox is not using my network connection's configured DNS server for basic A record resolution. Firefox does work as expected on a Windows machine on the same network.

Hmm, I'm trying to figure out whether Firefox shows which DNS server it is using when it is NOT using DNS over HTTPS. (I'm not seeing it on about:networking#dns or about:networking#dnslookuptool. I don't see it in the log created according to https://developer.mozilla.org/docs/Mozilla/Debugging/HTTP_logging.)

I assume your Firefox is not using a proxy server or VPN, which might bypass system resolution.

Thanks for your response, jscher2000. I actually had the exact same train of thought and tried the HTTP logging feature. However, it turns out that this is not a Firefox problem, but rather a misunderstanding on my part regarding Apple's iCloud Private Relay Feature. Firefox works as expected if I disable iCloud Private Relay and clear my machine's DNS cache.

I had mistakenly believed that iCloud Private Relay would only be used for DNS in Safari and other Apple-developed, Private-Relay-enabled apps. Thus, my assumption was that if I kept Private Relay enabled, cleared the machine's DNS cache, and then immediately requested a site in Firefox, my configured DNS server would be used. However, it appears that the Private Relay DNS system is still used to perform the DNS lookup in that scenario.