תשובות אחרונות על Firefox reports a secure connection without existing certificatehttps://support.mozilla.org/he/questions/13508612021-09-27T18:31:13-07:00Yeah, I admit I was wrong in not hoping on much progress.
November is no problem at all - I am rath2021-09-27T18:31:13-07:00pmc1https://support.mozilla.org/he/questions/1350861?page=2#answer-1446774<p>Yeah, I admit I was wrong in not hoping on much progress.
</p><p>November is no problem at all - I am rather used to OS release cycles where it can take a year or more to the next major version. The important thing is, it goes away from my stack, it is no longer me who must carry it along.
</p><p>So, thanks a lot for Your help!
</p>Looks like some progress on the bug report. However, Firefox 94 release comes in November, so it's s2021-09-27T16:36:45-07:00jscher2000https://support.mozilla.org/he/questions/1350861?page=2#answer-1446764<p>Looks like some progress on the bug report. However, Firefox 94 release comes in November, so it's some time away.
</p>jscher2000 said
You can file a bug report on https://bugzilla.mozilla.org/
Done. #1732476
What are 2021-09-24T07:01:02-07:00pmc1https://support.mozilla.org/he/questions/1350861?page=2#answer-1445984<em><p>jscher2000 <a href="#answer-1445921" rel="nofollow">said</a></p></em>
<blockquote>You can file a bug report on <a href="https://bugzilla.mozilla.org/" rel="nofollow">https://bugzilla.mozilla.org/</a>
</blockquote>
<p>Done. #1732476
</p><p>What are the chances anyone will read it?
(With other products, bug reports -even those with ready-made fixes attached- stay unread for ten or twenty years.)
</p>You can file a bug report on https://bugzilla.mozilla.org/
There might be a mailing list (now a Goog2021-09-24T01:52:22-07:00jscher2000https://support.mozilla.org/he/questions/1350861?page=2#answer-1445921<p>You can file a bug report on <a href="https://bugzilla.mozilla.org/" rel="nofollow">https://bugzilla.mozilla.org/</a>
</p><p>There might be a mailing list (now a Google Group) that others read more regularly, but I'm not sure which one is the most relevant:
</p><p><a href="https://lists.mozilla.org/" rel="nofollow">https://lists.mozilla.org/</a>
</p>The question now is: how do we get this thing fixed?
The current state of affairs is: people may con2021-09-24T01:37:39-07:00pmc1https://support.mozilla.org/he/questions/1350861?page=2#answer-1445917<p>The question now is: <strong>how do we get this thing fixed?</strong>
</p><p>The current state of affairs is: people may connect to my application, then they may want to check the security, and then they may experience that the lock icon has gone dead.
</p><p>People without detailed background knowledge - and even people with some technical knowledge - will then most likely think that my application has somehow <strong>hacked their browser</strong>.
</p><p>So this is <strong>very very bad</strong>, and has some urgency.
</p><p>I have prepared an issue statement for the issue. What is next? Do I need a sponsor, or what?
</p><p>Issue statement:
</p>
<blockquote>
When loading a page with many ressources, firefox opens additional connections to the server, to fetch files in parallel.
These connections do carry the 0x5 TLS extension (OCSP stapling), and they do invoke TLS verification. When verification fails, however, these connections are not necessarily abandoned.
When must_staple is set in the certificate, that TLS verification will also invoke TLSFeaturesSatisfiedInternal().
For TLSv1.2 this works. With TLSv1.3 this results in error
ERROR_REQUIRED_TLS_FEATURE_MISSING
because the OCSPStapling response appears not to be available.
A perceivable difference is that with TLSv1.3 these additional connections are opened with TLS extension 0x29 (pre_shared_key), which is not the case with TLSv1.2.
The consequence of the failing verification is not an abandonment of the connection, neither is it reported to anywhere. Nevertheless, the consequence is that Firefox does not copy the certificate data to the presentation layer.
At a subsequent change of the security context, e.g. when a new cookie is received, in the case when the TCP connections did stay open (a new TLS negotiation did NOT happen), this certificate data (that was NOT copied and therefore is undefined) is relied on from UI code.
At that point it may appear that the lock icon is rendered nonfunctional, and no certificate data at all is shown.
</blockquote>jscher2000 said
That's very helpful.
I wonder whether this is a Let's Encrypt specific issue?
Let'2021-09-23T05:25:17-07:00pmc1https://support.mozilla.org/he/questions/1350861?page=2#answer-1445752<em><p>jscher2000 <a href="#answer-1445742" rel="nofollow">said</a></p></em>
<blockquote>That's very helpful.
I wonder whether this is a Let's Encrypt specific issue? </blockquote>
<p>Let's put this differently:
</p>
<ol><li> does any commercial CA provide certificates with the must-staple feature?
</li><li> would anybody who is willing to pay for a cert, and who therefore most likely expects some return-on-investment, be willing to activate a feature that is not mandatory, and that is known to cause problems in some circumstances?
</li></ol>
<blockquote>Two threads mentioned either them or a "free" certificate:
<ul><li> <a href="https://support.mozilla.org/questions/1290478" rel="nofollow">MOZILLA_PKIX_ERROR_REQUIRED_TLS_FEATURE_MISSING caused by ocsp must staple, SSL labs says stapling is OK</a>
</li><li> <a href="https://support.mozilla.org/questions/1327681" rel="nofollow">MOZILLA_PKIX_ERROR_REQUIRED_TLS_FEATURE_MISSING</a>
</li></ul>
</blockquote>
<p>The main difference here: these people actually <strong>see</strong> this error! Their TLS handshake does effectively fail, because must-staple is engaged and the stapling OCSP response is someway missing (for whatever reason), or Firefox is made believe it would be missing.
And a competent guy from letsencrypt forum stated that it is difficult to get must-staple properly working because of flaws in nginx. No idea how the situation is with apache.
</p><p>But our case here is stranger: the TLS handshake works, the webpage works flawlessly, and only deep in the guts of Firefox is a (second?) verification done, which fails, and that failure is reported to nowhere (and can only be found by spamming the codebase with fprintf), but manages to occasionally(!) disturb the internal certChain store and subsequently the UI.
</p><p>I don't get the rationale of this: when stapling is <em>required</em>, then either the reponse is present and good, or the response is bad or missing, and then the connection must be aborted.
</p>
<em><p>jscher2000 <a href="#answer-1445743" rel="nofollow">said</a></p></em>
<blockquote>However, that still doesn't explain the odd inconsistency between requests where some work (logged out) and other do not (logged in).
</blockquote>
<p>I gave up wondering about such. Today I cannot reproduce it at all, except when in turn accessing two individual applications with different hostnames and different certs.
</p>However, that still doesn't explain the odd inconsistency between requests where some work (logged o2021-09-23T04:28:33-07:00jscher2000https://support.mozilla.org/he/questions/1350861?page=2#answer-1445743<p>However, that still doesn't explain the odd inconsistency between requests where some work (logged out) and other do not (logged in).
</p>That's very helpful.
I wonder whether this is a Let's Encrypt specific issue? Two threads mentioned 2021-09-23T04:27:13-07:00jscher2000https://support.mozilla.org/he/questions/1350861?page=2#answer-1445742<p>That's very helpful.
</p><p>I wonder whether this is a Let's Encrypt specific issue? Two threads mentioned either them or a "free" certificate:
</p>
<ul><li> <a href="https://support.mozilla.org/questions/1290478" rel="nofollow">MOZILLA_PKIX_ERROR_REQUIRED_TLS_FEATURE_MISSING caused by ocsp must staple, SSL labs says stapling is OK</a>
</li><li> <a href="https://support.mozilla.org/questions/1327681" rel="nofollow">MOZILLA_PKIX_ERROR_REQUIRED_TLS_FEATURE_MISSING</a>
</li></ul>(D) OCSP stapling
Recent discussions with the letsencrypt people explained:
The perceived error ERRO2021-09-23T03:59:59-07:00pmc1https://support.mozilla.org/he/questions/1350861#answer-1445740<p><strong>(D) OCSP stapling</strong>
</p><p>Recent discussions with the letsencrypt people explained:
</p><p>The perceived error <em>ERROR_REQUIRED_TLS_FEATURE_MISSING</em> and the message <em>couldn't rebuild verified certificate info</em> are <strong>normal</strong> in the case when the configuration <strong>requires</strong> OCSP stapling and the server does not provide the OCSP data.
</p><p>In our case here, the certificate is configured for <strong>must-staple</strong>, so the server is required to provide the OCSP data.
(According to the logs it does that, but a closer look into that is still required.)
</p><p>Letsencrypt people explained that the most simple solution is to just remove the <strong>must-staple</strong> feature, thereby making OCSP stapling optional instead of mandatory. This is probably what most people do.
</p>Some infoObject gets filled with certificate data only if that call would have been successful.
(We 2021-09-23T03:41:45-07:00pmc1https://support.mozilla.org/he/questions/1350861#answer-1445737<p>Some <em>infoObject</em> gets filled with certificate data <strong>only if</strong> that call would have been successful.
</p><p>(We do not know what this <em>infoObject</em> does already contain in the case when it is not filled from here. We also do not know what this <em>infoObject</em> actually is - but there might be some probability that it has something to do with the perceived failure.)
</p><p>Now findung out <strong>why</strong> the <em>certVerifier-&gt;VerifySSLServerCert</em>() does not return success, is a long and winding road, and finally leads to this piece of code (in security/nss/lib/mozpkix/lib/pkixcheck.cpp):
</p><pre> Result
TLSFeaturesSatisfiedInternal(const Input* requiredTLSFeatures,
const Input* stapledOCSPResponse)
{
if (!requiredTLSFeatures) {
return Success;
}
// RFC 6066 10.2: ExtensionType status_request
const static uint8_t status_request = 5;
const static uint8_t status_request_bytes[] = { status_request };
Reader input(*requiredTLSFeatures);
return der::NestedOf(input, der::SEQUENCE, der::INTEGER,
der::EmptyAllowed::No, [&amp;](Reader&amp; r) {
if (!r.MatchRest(status_request_bytes)) {
return Result::ERROR_REQUIRED_TLS_FEATURE_MISSING;
}
if (!stapledOCSPResponse) {
return Result::ERROR_REQUIRED_TLS_FEATURE_MISSING;
}
</pre>
<pre> return Result::Success;
});
}
</pre>
<p><br>
Honestly, I do not understand this construct. But this <em>if (!stapledOCSPResponse)</em> seems to trigger the former message.
</p>Alright. This may get a bit lengthy now.
TL;DR: the problem appears to be with OCSP stapling.
(A) T2021-09-23T03:32:54-07:00pmc1https://support.mozilla.org/he/questions/1350861#answer-1445731<p>Alright. This may get a bit lengthy now.
</p><p><strong>TL;DR: the problem appears to be with OCSP stapling.</strong>
</p><p><br>
<strong>(A) The observables</strong>
</p><p>We see two effects:
</p>
<ul><li> the lock icon in the address bar becomes nonfunctional (while still being present)
</li><li> the Page Info "Show Certificates" does not show the certificates
</li></ul>
<p>These two effects rely on different data, so they do not always align. In due time Firefox copies one data to the other, and then they will align again.
</p><p>All of this (including the data) belongs to the <em>presentation layer</em>. It is only what we see, it is NOT where the problem happens.
</p><p><br>
<strong>(B) the security context</strong>
</p><p>We must assume that the security context changes when a new cookie is received.
</p><p>In this case, the application uses authenticated encrypted cookies with a lot of cryptography included. (These cookies should be safe to send even over unprotected HTML - which is why the new SameSite regimen, prohibiting this for 3rd-party context, is a nuisance.)
The bottomline is: the cookie is not only sent with every request, it actually changes with every request.
</p><p>The log-in process therefore creates already three security contexts:
</p>
<ul><li> first connect to the page
</li><li> The form sent after &lt;Log In&gt; is clicked.
</li><li> Login process completed.
</li></ul>
<p>But the 'turbolinks' page preloader running in javascript does somehow mangle and merge these requests. They are cleanly observable only with javascript disabled.
</p><p>It is usually the third security context that exposes the problem, but may be the second (if it appears at all).
</p><p><br>
<strong>(C) The internals</strong>
</p><p>At the point where the problem actually appears, nothing significant is reported in debug messages - only that a new security context appeared ("we have a security info").
</p><p>But in the <strong>preceding</strong> security context a message appears, which does not appear with other sites
( visbile with MOZ_LOG=pipnss:4 )
</p>
<blockquote>D/pipnss HandshakeCallback: couldn't rebuild verified certificate info</blockquote>
<p>This happens in <em>RebuildVerifiedCertificateInformation</em>(), and it means that a call to <em>certVerifier-&gt;VerifySSLServerCert</em>() was <strong>not successful</strong>.
</p><p>At the end of that first function we find this:
</p><pre> if (rv == Success) {
uint16_t status =
TransportSecurityInfo::ConvertCertificateTransparencyInfoToStatus(
certificateTransparencyInfo);
infoObject-&gt;SetCertificateTransparencyStatus(status);
nsTArray&lt;nsTArray&lt;uint8_t&gt;&gt; certBytesArray =
TransportSecurityInfo::CreateCertBytesArray(builtChain);
infoObject-&gt;SetSucceededCertChain(std::move(certBytesArray));
infoObject-&gt;SetIsBuiltCertChainRootBuiltInRoot(
isBuiltCertChainRootBuiltInRoot);
}
&lt;/pre&gt;
<p>(something is broken with the text formatting; I'll continue in another message)
</p></pre>I most likely found it.
I'm far from understanding it, I don't even understand what that code is sup2021-09-22T14:13:14-07:00pmc1https://support.mozilla.org/he/questions/1350861#answer-1445524<p>I most likely found it.
</p><p>I'm far from understanding it, I don't even understand what that code is supposed to do and what is wrong with that. I'ts totally cryptic. But it is what is different. It seems to come from the certificate.
</p>
<em><p>jscher2000 <a href="#answer-1445499" rel="nofollow">said</a></p></em>
<blockquote>My assumption is that logging in to a website may change a cookie or other stored data,</blockquote>
<p>Would this trigger a new security context? The cookies are the only thing I can think of. And my app definitely sends a (different) ccokie with every request. See question 42044076 on stackoverflow for explanation.
But this is apparently not the problem. It just triggers it.
</p><p>I go to sleep now. We unravel this in due time.
</p>pmc1 said
But, while for me it still seems impossible to pinpoint which component is to blame for th2021-09-22T11:27:20-07:00jscher2000https://support.mozilla.org/he/questions/1350861#answer-1445499<em><p>pmc1 <a href="#answer-1445486" rel="nofollow">said</a></p></em>
<blockquote>But, while for me it still seems impossible to pinpoint which component is to blame for the root cause of these effects, can we agree that, no matter what actually comes down from the server, Firefox is behaving in a way that is not fully correct?
</blockquote>
<p>My assumption is that logging in to a website may change a cookie or other stored data, but it definitely should not change anything else on the browser side.
</p><p>I can't replicate it with this simplified example:
</p><p><a href="https://www.jeffersonscher.com/res/post302form.php" rel="nofollow">https://www.jeffersonscher.com/res/post302form.php</a>
</p><p>Source:
</p>
<ul><li> <a href="https://www.jeffersonscher.com/res/post302form.php.txt" rel="nofollow">https://www.jeffersonscher.com/res/post302form.php.txt</a>
</li><li> <a href="https://www.jeffersonscher.com/res/signin/post302.php.txt" rel="nofollow">https://www.jeffersonscher.com/res/signin/post302.php.txt</a>
</li></ul>jscher2000 said
I was not able to replicate the problem in my regular profile, but I can replicate i2021-09-22T10:04:15-07:00pmc1https://support.mozilla.org/he/questions/1350861#answer-1445486<em><p>jscher2000 <a href="#answer-1445478" rel="nofollow">said</a></p></em><blockquote>
<p>I was not able to replicate the problem in my regular profile, but I can replicate it in a fresh profile on 93 beta.
</p>
</blockquote>
<p>Same here. It seems to happen a lot quicker with a pristine profile. It will probably happen in the other one also, but maybe only after half an hour of working with the app or after accessing some different apps on that server. (Tracking protection?)
</p>
<blockquote>Firefox validated the site certificate on the first access and is able to show that information (example screenshot attached), but when trying to pull up the about:certificate screen after log in, the page is blank. </blockquote>
<p>Same here. The log-in creates a new security context. I do not know why it does that, but MOZ_LOG=nsSecureBrowserUI:5 reports that it does it:
</p>
<blockquote>[Parent 67891: Main Thread]: D/nsSecureBrowserUI we have a security info 0x8218b3700
[Parent 67891: Main Thread]: D/nsSecureBrowserUI set mTopLevelSecurityInfo</blockquote>
<p>The commentary in the source near that message says this:
</p>
<blockquote> // Our BrowsingContext either has a new WindowGlobalParent, or the
// existing one has mutated its security state.
// Recompute our security state and fire notifications to listeners</blockquote>
<p>I don't see why the security state would change at this point. But it does, and at (or before?) that point the chain is lost. The <strong>server certificate</strong> is still there and is intact, but the <strong>certificate chain</strong> is gone. According to the debug log it has not even been re-evaluated, it has just vanished into thin air. And consequentially the data in the securityInfo structure is now inconsistent:
</p>
<blockquote>isBuiltCertChainRootBuiltInRoot: false
resumed: true
succeededCertChain: Array []</blockquote>
<p>But then, a lot of other sites do the same, and do NOT loose the certChain alongside.
</p>
<blockquote>In the case where it's working, the encoded version of the certificate is passed to the about:certificate page in the URL, and when it fails, it is not. Very puzzling.
Weirdly, if I use Back to return to the Log In page, it's fine. If I log out after logging in, I need to reload the page to view the certificate again.
Also, after testing in a second tab, I can't replicate the error.&nbsp;??
</blockquote>
<p>Same here. And it goes on and on in that fashion - this thing was me a source of infinite bewilderment.
But, while for me it still seems impossible to pinpoint which component is to blame for the root cause of these effects, can we agree that, no matter what actually comes down from the server, Firefox is behaving in a way that is not fully correct?
</p>Sorry, I didn't realize that was the URL.
I was not able to replicate the problem in my regular prof2021-09-22T09:20:21-07:00jscher2000https://support.mozilla.org/he/questions/1350861#answer-1445478<p>Sorry, I didn't realize that was the URL.
</p><p>I was not able to replicate the problem in my regular profile, but I can replicate it in a fresh profile on 93 beta. Firefox validated the site certificate on the first access and is able to show that information (example screenshot attached), but when trying to pull up the about:certificate screen after log in, the page is blank.
</p><p>In the case where it's working, the encoded version of the certificate is passed to the about:certificate page in the URL, and when it fails, it is not. Very puzzling.
</p><p>Weirdly, if I use Back to return to the Log In page, it's fine. If I log out after logging in, I need to reload the page to view the certificate again.
</p><p>Also, after testing in a second tab, I can't replicate the error.&nbsp;??
</p>This is on the web now. It should answer to https, and the letsencrypt people have said it does so c2021-09-22T08:38:59-07:00pmc1https://support.mozilla.org/he/questions/1350861#answer-1445473<p>This is on the web now. It should answer to https, and the letsencrypt people have said it does so correctly.
(It's a rule builder for the berkeley ipfw firewall, but I thought I could just make it multiuser capable - and so I can put it online as well - should have thought that far earlier)
</p>pmc1 said
jscher2000 said
Usually if you click the lock icon, a site information panel drops down. C2021-09-22T08:07:38-07:00jscher2000https://support.mozilla.org/he/questions/1350861#answer-1445469<em><p>pmc1 <a href="#answer-1445461" rel="nofollow">said</a></p></em>
<blockquote><em><p>jscher2000 <a href="#answer-1443888" rel="nofollow">said</a></p></em>
<blockquote>Usually if you click the lock icon, a site information panel drops down. Can you give an example of a page where that does not happen? To avoid the moderation queue, you can type a space in the URL before the TLD (example. com) so the forum doesn't recognize the link.
</blockquote>
<p><em>This is now flag.daemon .contact/flowm/</em>
There click Log In (upper right), enter something ( or enter Guest0 Guest0 ), check 'remember me' and send it - and in reply you should get the second context, which more or less likely triggers the flaw. And then you probably see the same as I see.
</p>
</blockquote>
<p>Sorry, is this on the web now or do I need to install something?
</p>cor-el said
See:
Oh yes, it's beautiful. And so well coloured. But, then, otherwise it is basically2021-09-22T07:35:01-07:00pmc1https://support.mozilla.org/he/questions/1350861#answer-1445461<em><p>cor-el <a href="#answer-1445307" rel="nofollow">said</a></p></em>
<blockquote>See:
</blockquote>
<p>Oh yes, it's beautiful. And so well coloured. But, then, otherwise it is basically the same as my grep does. And what I said above is, I need to know "'where these variables are filled with data". So, sorry, but this does not yet fully helpful.
</p><p><br>
Now for the other workproducts:
</p><p><br>
</p>
<em><p>jscher2000 <a href="#answer-1444949" rel="nofollow">said</a></p></em>
<blockquote>
Firefox 78 is near it's end-of-life. Can you test in Firefox 91esr or in FIrefox 93beta (for example, <a href="https://www.mozilla.org/firefox/developer/" rel="nofollow">https://www.mozilla.org/firefox/developer/</a> )?
</blockquote>
<p>Putting a linux on a stick did not work: it would take 5 minutes to only open "Help/About Firefox" to see the current version. (These sticks may support usb-3, but they are unuseable with journaled filesystems. Wonder for what they are good for at all - but then they cost nothing.)
So now I learned how linux install and configuration works, learned how graphical virtualization works, learned how vnc works and finally have a small kali linux integrated in my desktop (which seems surprizingly well maintained and managed for what I know of linux). And the outcome of all of is this:
</p><p><strong>Firefox 93.0b8 developer shows the exact same behaviour.
</strong>
<strong>
</strong>
</p><p>Next workproduct:
</p><p><br>
</p>
<em><p>jscher2000 <a href="#answer-1443888" rel="nofollow">said</a></p></em>
<blockquote>Usually if you click the lock icon, a site information panel drops down. Can you give an example of a page where that does not happen? To avoid the moderation queue, you can type a space in the URL before the TLD (example. com) so the forum doesn't recognize the link.
</blockquote>
<p><em>This is now flag.daemon .contact/flowm/</em>
There click Log In (upper right), enter something ( or enter Guest0 Guest0 ), check 'remember me' and send it - and in reply you should get the second context, which more or less likely triggers the flaw. And then you probably see the same as I see.
</p>See:
https://searchfox.org/mozilla-release/search?q=succeededCertChain&path=
2021-09-21T15:39:43-07:00cor-elhttps://support.mozilla.org/he/questions/1350861#answer-1445307<p>See:
</p>
<ul><li><a href="https://searchfox.org/mozilla-release/search?q=succeededCertChain&amp;path=" rel="nofollow">https://searchfox.org/mozilla-release/search?q=succeededCertChain&amp;path=</a>
</li></ul>And, btw. if the page does not have any TLS, then you will see this:
gBrowser.securityUI.secInfo.suc2021-09-20T21:21:26-07:00pmc1https://support.mozilla.org/he/questions/1350861#answer-1445157<p>And, btw. if the page does not have any TLS, then you will see this:
</p><p>gBrowser.securityUI.secInfo.succeededCertChain: undefined
gIdentityHandler._secInfo.succeededCertChain: undefined
</p><p>We should NEVER see zero-Length arrays there. They are bogus.
Now please do me a favor and figure out where in the code these arrays are filled with data. Then we will see under which (probably erroneous) conditions they can become zero-length.
</p>