I read you mentioned firefox. I had a similar experience a while ago, related to this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1704774#c13

The nginx documentation for the ssl preread module has an almost identical example.

I am running a similar setup to yours. The issue is that only one server block can listen to an address+port pair. You ought to do something like this:

map $ssl_preread_server_name $proxy_backend_router {
        serviceA.example.com    upstreamA:12346;
        serviceB.example.com    upstreamB:12346;
        default $ssl_preread_server_name.invalid_proxy:443;
}


server {
        listen 443;
        ssl_preread on;
        proxy_pass $proxy_backend_router;
}