.. _troubleshooting:

***************
Troubleshooting
***************

TLS errors
==========

In general, TLS issues can be debugged using Java's very detailed debug output. To enable it, the Java process needs to be started with option ``-Djavax.net.debug=all``. Debug output is typically printed to stdout and is **extremely verbose**. Also it **reveals plain text** of requests and responses! Anyhow this is a very useful tool during integration testing. More details can be found at https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/ReadDebug.html.


Inbound
-------

When configuring TLS it might happen, that the keystore is not configured correctly. Sadly, not every possible error leads to a sane error-output. Most of the time the error messages mean, that no valid certificate could be found for the request. Following are two potential error messages, that mean something different than they say.

  **OpenSSL with SNI**::

          javax.net.ssl.SSLHandshakeException: error:100000b8:SSL routines:OPENSSL_internal:NO_SHARED_CIPHER

**Solution:** Either deactivate OpenSSL directly via ``zuul.ssl.openssl.allow=false`` in ``application.properties`` or reduce the amount of certificates to one, to disable the SNI mechanism.

  **No matching certificate in keystore**::

          io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: no cipher suites in common

**Solution:** This simply means, that no certificate could be found for the requesting domain. Check your keystore and make sure that the certificate, made for the requesting client, is present.


Outbound
--------

Outbound HTTPS connections will fail on untrusted certificates (see also :ref:`Backend Pools::TLS <backend_pools_tls>`. In that case the following will show up in logs as::

    2019-11-23T14:12:42,833+0100 [Salamander-ClientToZuulWorker-0] WARN  c.n.z.filters.endpoint.ProxyEndpoint - FAILURE_LOCAL, origin = upstream, origin channel info = Channel: [id: 0xe1cb165b, L:/127.0.0.1:50142 - R:localhost/127.0.0.1:443], active=true, open=true, registered=true, writable=true, id=e1cb165b, Passport: CurrentPassport {start_ms=1574514762106, [+0=SERVER_CH_ACTIVE, +30887473=IN_REQ_HEADERS_RECEIVED, +60512380=FILTERS_INBOUND_START, +73572694=FILTERS_INBOUND_END, +654109637=ORIGIN_CH_CONNECTING, +680463025=ORIGIN_CONN_ACQUIRE_START, +680628600=IN_REQ_LAST_CONTENT_RECEIVED, +685689009=ORIGIN_CH_CONNECTED, +686688864=ORIGIN_CONN_ACQUIRE_END, +691002152=OUT_REQ_HEADERS_SENDING, +693593741=OUT_REQ_LAST_CONTENT_SENDING, +696344587=ORIGIN_CH_ACTIVE, +724605571=NOW]}
    javax.net.ssl.SSLException: SSLEngine closed already
      at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source)


Unfortunately this error message is very generic and doesn't point to the actual root cause. To confirm, DEBUG logging can be enabled for logger ``com.netflix.zuul.netty.connectionpool.ConnectionPoolHandler``, leading to more detailed log output.

**With OpenSSL enabled**::

    2019-11-23T14:14:05,581+0100 [Salamander-ClientToZuulWorker-1] DEBUG i.n.h.s.ReferenceCountedOpenSslContext - verification of certificate failed
    sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
      at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
      at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
      at sun.security.validator.Validator.validate(Validator.java:262)
      at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
      at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
      at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
      at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:237)
      at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:621)
      at io.netty.internal.tcnative.SSL.readFromSSL(Native Method)
      [...]
      at java.lang.Thread.run(Thread.java:748)
    Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
      at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
      at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
      at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
      at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
      ... 40 common frames omitted



**With normal JRE SSL**::

    2019-11-23T14:15:24,995+0100 [Salamander-ClientToZuulWorker-0] DEBUG c.n.z.n.c.ConnectionPoolHandler - Exception on Origin channel for origin - upstream. Channel: [id: 0x21ce1dae, L:/127.0.0.1:50823 ! R:localhost/127.0.0.1:443], active=false, open=false, registered=true, writable=false, id=21ce1dae, Passport: CurrentPassport {start_ms=1574514924255, [+0=SERVER_CH_ACTIVE, +31071419=IN_REQ_HEADERS_RECEIVED, +62144456=FILTERS_INBOUND_START, +76851580=FILTERS_INBOUND_END, +579132125=ORIGIN_CH_CONNECTING, +604916577=ORIGIN_CONN_ACQUIRE_START, +605204297=IN_REQ_LAST_CONTENT_RECEIVED, +610441322=ORIGIN_CH_CONNECTED, +611889646=ORIGIN_CONN_ACQUIRE_END, +616052674=OUT_REQ_HEADERS_SENDING, +618740568=OUT_REQ_LAST_CONTENT_SENDING, +628692309=ORIGIN_CH_ACTIVE, +659922807=FILTERS_OUTBOUND_START, +721169337=FILTERS_OUTBOUND_END, +723031881=OUT_RESP_HEADERS_SENDING, +725671291=OUT_RESP_LAST_CONTENT_SENDING, +727932501=OUT_RESP_HEADERS_SENT, +727983756=OUT_RESP_LAST_CONTENT_SENT, +736707023=ORIGIN_CH_CLOSE, +738236144=OUT_REQ_HEADERS_ERROR_SENDING, +738657984=OUT_REQ_LAST_CONTENT_ERROR_SENDING, +738853699=ORIGIN_CH_CLOSE, +739460086=NOW]} - io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
    io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
      at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459)
      at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
      [...]
      at java.lang.Thread.run(Thread.java:748)
    Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
      [...]
    Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
      at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
      at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
      at sun.security.validator.Validator.validate(Validator.java:262)
      at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
      at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
      at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
      at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1626)
      ... 34 common frames omitted
    Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
      at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
      at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
      at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
      at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
      ... 40 common frames omitted