Merge pull request #2561 from drwetter/fix_1312

Fix 1312

CHANGELOG.md

@@ -47,6 +47,7 @@
 * Don't use external pwd command anymore
 * Doesn't hang anymore when there's no local resolver
 * Added --mtls feature to support client authentication
+* If a TLS 1.3 host is tested and e.g. /usr/bin/openssl supports it, it'll automagically will switch to it
 
 
 ### Features implemented / improvements in 3.0

doc/testssl.1

@@ -141,7 +141,7 @@ Any single check switch supplied as an argument prevents testssl\.sh from doing
 .P
 \fB\-f, \-\-fs, \-\-nsa, \-\-forward\-secrecy\fR Checks robust forward secrecy key exchange\. "Robust" means that ciphers having intrinsic severe weaknesses like Null Authentication or Encryption, 3DES and RC4 won't be considered here\. There shouldn't be the wrong impression that a secure key exchange has been taking place and everything is fine when in reality the encryption sucks\. Also this section lists the available elliptical curves and Diffie Hellman groups, as well as FFDHE groups (TLS 1\.2 and TLS 1\.3)\.
 .P
-\fB\-p, \-\-protocols\fR checks TLS/SSL protocols SSLv2, SSLv3, TLS 1\.0 through TLS 1\.3 and for HTTP: SPDY (NPN) and ALPN, a\.k\.a\. HTTP/2\. For TLS 1\.3 several drafts (from 18 on) and final are supported and being tested for\.
+\fB\-p, \-\-protocols\fR checks TLS/SSL protocols SSLv2, SSLv3, TLS 1\.0 through TLS 1\.3 and for HTTP: SPDY (NPN) and ALPN, a\.k\.a\. HTTP/2\. For TLS 1\.3 several drafts (from 18 on) and final are supported and being tested for\. Note the supplied openssl\-bad version doesn't support TLS 1\.3 \. As the check for TLS 1\.3 will be done in sockets this normally does not pose a problem\. However if a TLS\-1\.3\-only host is encountered and to have a complete test coverage (e.g. header checks) \fB/usr/bin/openssl\fR (or the content of \fBOPENSSL2\fR) is checked for existence and support of TLS 1\.3 and if those tests succeeded it will be switched to this binary\. A message will notify you\.
 .P
 \fB\-P, \-\-server\-preference, \-\-preference\fR displays the servers preferences: cipher order, with used openssl client: negotiated protocol and cipher\. If there's a cipher order enforced by the server it displays it for each protocol (openssl+sockets)\. If there's not, it displays instead which ciphers from the server were picked with each protocol\.
 .P
@@ -418,6 +418,10 @@ MAX_SOCKET_FAIL: A number which tells testssl\.sh how often a TCP socket connect
 MAX_OSSL_FAIL: A number which tells testssl\.sh how often an OpenSSL s_client connect may fail before the program gives up and terminates\. The default is 2\. You can increase it to a higher value if you frequently see a message like \fIFatal error: repeated TCP connect problems, giving up\fR\.
 .IP "\[ci]" 4
 MAX_HEADER_FAIL: A number which tells testssl\.sh how often a HTTP GET request over OpenSSL may return an empty file before the program gives up and terminates\. The default is 3\. Also here you can increase the threshold when you spot messages like \fIFatal error: repeated HTTP header connect problems, doesn't make sense to continue\fR\.
+.IP "\[ci]" 4
+OPENSSL2 can be used to supply an alternative openssl version\. This only makes sense if you want to amend the supplied version in \fBbin/\fR which lacks TLS 1\.3 support with a version which does not and is not in \fB/usr/bin/openssl\fR.
+.IP "\[ci]" 4
+OSSL_SHORTCUT can be set to true when you run interactively and don't want to switch automatically to \fB/usr/bin/openssl\fR (\fBOPENSSL2\fR) if you encounter a TLS 1\.3-only host\.
 .IP "" 0
 .SS "RATING"
 This program has a near\-complete implementation of SSL Labs's 'SSL Server Rating Guide \fIhttps://github\.com/ssllabs/research/wiki/SSL\-Server\-Rating\-Guide\fR'\.

doc/testssl.1.html

@@ -262,7 +262,7 @@ in <code>/etc/hosts</code>.  The use of the switch is only useful if you either
 
 <p><code>-f, --fs, --nsa, --forward-secrecy</code> Checks robust forward secrecy key exchange. "Robust" means that ciphers having intrinsic severe weaknesses like Null Authentication or Encryption, 3DES and RC4 won't be considered here. There shouldn't be the wrong impression that a secure key exchange has been taking place and everything is fine when in reality the encryption sucks. Also this section lists the available elliptical curves and Diffie Hellman groups, as well as FFDHE groups (TLS 1.2 and TLS 1.3).</p>
 
-<p><code>-p, --protocols</code>  checks TLS/SSL protocols SSLv2, SSLv3, TLS 1.0 through TLS 1.3 and for HTTP: SPDY (NPN) and ALPN, a.k.a. HTTP/2. For TLS 1.3 several drafts (from 18 on) and final are supported and being tested for.</p>
+<p><code>-p, --protocols</code>  checks TLS/SSL protocols SSLv2, SSLv3, TLS 1.0 through TLS 1.3 and for HTTP: SPDY (NPN) and ALPN, a.k.a. HTTP/2. For TLS 1.3 several drafts (from 18 on) and final are supported and being tested for. Note the supplied openssl-bad version doesn't support TLS 1.3 . As the check for TLS 1.3 will be done in sockets this normally does not pose a problem. However if a TLS-1.3-only host is encountered and to have a complete test coverage (e.g. header checks) `/usr/bin/openssl` (or the content of `OPENSSL2`) is checked for existence and support of TLS 1.3 and if those tests succeeded it will be switched to this binary. A message will notify you.</p>
 
 <p><code>-P, --server-preference, --preference</code>  displays the servers preferences: cipher order, with used openssl client: negotiated protocol and cipher. If there's a cipher order enforced by the server it displays it for each protocol (openssl+sockets). If there's not, it displays instead which ciphers from the server were picked with each protocol.</p>
 
@@ -501,6 +501,8 @@ Rating automatically gets disabled, to not give a wrong or misleading grade, whe
   <li>MAX_SOCKET_FAIL: A number which tells testssl.sh how often a TCP socket connection may fail before the program gives up and terminates. The default is 2. You can increase it to a higher value if you frequently see a message like <em>Fatal error: repeated openssl s_client connect problem, doesn't make sense to continue</em>.</li>
   <li>MAX_OSSL_FAIL: A number which tells testssl.sh how often an OpenSSL s_client connect may fail before the program gives up and terminates. The default is 2. You can increase it to a higher value if you frequently see a message like <em>Fatal error: repeated TCP connect problems, giving up</em>.</li>
   <li>MAX_HEADER_FAIL: A number which tells testssl.sh how often a HTTP GET request over OpenSSL may return an empty file before the program gives up and terminates. The default is 3. Also here you can increase the threshold when you spot messages like <em>Fatal error: repeated HTTP header connect problems, doesn't make sense to continue</em>.</li>
+  <li>OPENSSL2 can be used to supply an alternative openssl version. This only makes sense if you want to amend the supplied version in <code>bin/</code> which lacks TLS 1.3 support with a version which does not and is not in <code>/usr/bin/openssl</code>.</li>
+  <li>OSSL_SHORTCUT can be set to true when you run interactively and don't want to switch automatically to <code>/usr/bin/openssl</code> (<code>OPENSSL2</code>) if you encounter a TLS 1.3-only host.
 </ul>
 
 <h3 id="RATING">RATING</h3>

doc/testssl.1.md

@@ -175,7 +175,7 @@ Any single check switch supplied as an argument prevents testssl.sh from doing a
 
 `-f, --fs, --nsa, --forward-secrecy` Checks robust forward secrecy key exchange. "Robust" means that ciphers having intrinsic severe weaknesses like Null Authentication or Encryption, 3DES and RC4 won't be considered here. There shouldn't be the wrong impression that a secure key exchange has been taking place and everything is fine when in reality the encryption sucks. Also this section lists the available elliptical curves and Diffie Hellman groups, as well as FFDHE groups (TLS 1.2 and TLS 1.3).
 
-`-p, --protocols`  checks TLS/SSL protocols SSLv2, SSLv3, TLS 1.0 through TLS 1.3 and for HTTP: SPDY (NPN) and ALPN, a.k.a. HTTP/2. For TLS 1.3 several drafts (from 18 on) and final are supported and being tested for.
+`-p, --protocols`  checks TLS/SSL protocols SSLv2, SSLv3, TLS 1.0 through TLS 1.3 and for HTTP: SPDY (NPN) and ALPN, a.k.a. HTTP/2. For TLS 1.3 several drafts (from 18 on) and final are supported and being tested for. Note the supplied openssl-bad version doesn't support TLS 1.3 . As the check for TLS 1.3 will be done in sockets this normally does not pose a problem. However if a TLS-1.3-only host is encountered and to have a complete test coverage (e.g. header checks) `/usr/bin/openssl` (or the content of `OPENSSL2`) is checked for existence and support of TLS 1.3 and if those tests succeeded it will be switched to this binary. A message will notify you.
 
 `-P, --server-preference, --preference`  displays the servers preferences: cipher order, with used openssl client: negotiated protocol and cipher. If there's a cipher order enforced by the server it displays it for each protocol (openssl+sockets). If there's not, it displays instead which ciphers from the server were picked with each protocol.
 
@@ -403,6 +403,9 @@ Except the environment variables mentioned above which can replace command line
 * MAX_SOCKET_FAIL: A number which tells testssl.sh how often a TCP socket connection may fail before the program gives up and terminates. The default is 2. You can increase it to a higher value if you frequently see a message like *Fatal error: repeated openssl s_client connect problem, doesn't make sense to continue*.
 * MAX_OSSL_FAIL: A number which tells testssl.sh how often an OpenSSL s_client connect may fail before the program gives up and terminates. The default is 2. You can increase it to a higher value if you frequently see a message like *Fatal error: repeated TCP connect problems, giving up*.
 * MAX_HEADER_FAIL: A number which tells testssl.sh how often a HTTP GET request over OpenSSL may return an empty file before the program gives up and terminates. The default is 3. Also here you can increase the threshold when you spot messages like *Fatal error: repeated HTTP header connect problems, doesn't make sense to continue*.
+* OPENSSL2 can be used to supply an alternative openssl version. This only makes sense if you want to amend the supplied version in `bin/` which lacks TLS 1.3 support with a version which doesn not and is not in `/usr/bin/openssl`.
+* OSSL_SHORTCUT can be set to true when you run interactively and don't want to switch automatically to `/usr/bin/openssl` (`OPENSSL2`) if you encounter a TLS 1.3-only host.
+
 
 ### RATING
 

testssl.sh

@@ -243,9 +243,10 @@ SYSTEM2=""                              # currently only being used for WSL = ba
 PRINTF=""                               # which external printf to use. Empty presets the internal one, see #1130
 CIPHERS_BY_STRENGTH_FILE=""
 TLS_DATA_FILE=""                        # mandatory file for socket-based handshakes
-OPENSSL=""                              # If you run this from GitHub it's ~/bin/openssl.$(uname).$(uname -m) otherwise /usr/bin/openssl
+OPENSSL=""                              # ~/bin/openssl.$(uname).$(uname -m) if you run this from GitHub. Linux otherwise probably /usr/bin/openssl
-OPENSSL2=""                             # When running from GitHub, this will be openssl version >=1.1.1 (auto determined)
+OPENSSL2=${OPENSSL2:-/usr/bin/openssl}  # This will be openssl version >=1.1.1 (auto determined) as opposed to openssl-bad (OPENSSL)
-OPENSSL2_HAS_TLS_1_3=false              # If we run with supplied binary AND /usr/bin/openssl supports TLS 1.3 this is set to true
+OPENSSL2_HAS_TLS_1_3=false              # If we run with supplied binary AND $OPENSSL2 supports TLS 1.3 this will be set to true
+OSSL_SHORTCUT=${OSSL_SHORTCUT:-true}    # Hack: if during the scan turns out the OpenSSL binary supports TLS 1.3 would be a better choice
 OPENSSL_LOCATION=""
 IKNOW_FNAME=false
 FIRST_FINDING=true                      # is this the first finding we are outputting to file?
@@ -275,7 +276,6 @@ KNOWN_OSSL_PROB=false                   # We need OpenSSL a few times. This vari
 DETECTED_TLS_VERSION=""                 # .. as hex string, e.g. 0300 or 0303
 APP_TRAF_KEY_INFO=""                    # Information about the application traffic keys for a TLS 1.3 connection.
 TLS13_ONLY=false                        # Does the server support TLS 1.3 ONLY?
-OSSL_SHORTCUT=${OSSL_SHORTCUT:-false}   # Hack: if during the scan turns out the OpenSSL binary supports TLS 1.3 would be a better choice, this enables it.
 TLS_EXTENSIONS=""
 TLS13_CERT_COMPRESS_METHODS=""
 CERTIFICATE_TRANSPARENCY_SOURCE=""
@@ -415,6 +415,7 @@ END_TIME=0                              # .. ended
 SCAN_TIME=0                             # diff of both: total scan time
 LAST_TIME=0                             # only used for performance measurements (MEASURE_TIME=true)
 SERVER_COUNTER=0                        # Counter for multiple servers
+OPEN_MSG=""                             # Null the poor man's implementation of a message stack
 
 TLS_LOW_BYTE=""                         # For "secret" development stuff, see -q below
 HEX_CIPHER=""                           #                -- " --
@@ -2405,7 +2406,7 @@ service_detection() {
                ;;
      esac
 
-     outln "\n"
+     outln
      tmpfile_handle ${FUNCNAME[0]}.txt
      return 0
 }
@@ -20297,7 +20298,6 @@ find_openssl_binary() {
      # not check /usr/bin/openssl -- if available. This is more a kludge which we shouldn't use for
      # every openssl feature. At some point we need to decide which with openssl version we go.
      # We also check, whether there's /usr/bin/openssl which has TLS 1.3
-     OPENSSL2=/usr/bin/openssl
      if [[ ! "$OSSL_NAME" =~ LibreSSL ]] && [[ ! $OSSL_VER =~ 1.1.1 ]] && [[ ! $OSSL_VER_MAJOR =~ 3 ]]; then
           if [[ -x $OPENSSL2 ]]; then
                $OPENSSL2 s_client -help 2>$s_client_has2
@@ -21015,6 +21015,9 @@ EOF
 
 # arg1: text to display before "-->"
 # arg2: arg needed to accept to continue
+# ret=0 : arg was accepted to continue (batch mode doesn't do this,or warnings are turned off)
+#     1 : arg was not accepted by user or we're in bacth mode
+
 ignore_no_or_lame() {
      local a
 
@@ -22033,21 +22036,26 @@ determine_optimal_proto() {
           [[ $? -ne 0 ]] && exit $ERR_CLUELESS
      elif "$all_failed" && ! "$ALL_FAILED_SOCKETS"; then
           if ! "$HAS_TLS13" && "$TLS13_ONLY"; then
-               pr_magenta " $NODE:$PORT appears to support TLS 1.3 ONLY. You better use --openssl=<path_to_openssl_supporting_TLS_1.3>"
+               if "$OPENSSL2_HAS_TLS_1_3"; then
-               if ! "$OSSL_SHORTCUT" || [[ ! -x /usr/bin/openssl ]] || /usr/bin/openssl s_client -tls1_3 2>&1 | grep -aiq "unknown option"; then
+                    if "$OSSL_SHORTCUT" || [[ "$WARNINGS" == batch ]]; then
-                    outln
+                         # switch w/o asking
-                    fileout "$jsonID" "WARN" "$NODE:$PORT appears to support TLS 1.3 ONLY, but $OPENSSL does not support TLS 1.3"
+                         OPEN_MSG=" $NODE:$PORT appeared to support TLS 1.3 ONLY. Thus switched implicitly from\n \"$OPENSSL\" to \"$OPENSSL2\"."
-                    ignore_no_or_lame " Type \"yes\" to proceed with $OPENSSL and accept all scan problems" "yes"
+                         fileout "$jsonID" "INFO" "$NODE:$PORT appears to support TLS 1.3 ONLY, switching from $OPENSSL to $OPENSSL2 was implicitly enforced"
-                    [[ $? -ne 0 ]] && exit $ERR_CLUELESS
+                         OPENSSL="$OPENSSL2"
-                    MAX_OSSL_FAIL=10
-               else
-                    # dirty hack but an idea for the future to be implemented upfront: Now we know, we'll better off
-                    # with the OS supplied openssl binary. We need to initialize variables / arrays again though.
-                    # And the service detection can't be made up for now
-                    outln ", \n proceeding with /usr/bin/openssl"
-                    OPENSSL=/usr/bin/openssl
                          find_openssl_binary
                          prepare_arrays
+                    else
+                         # now we need to ask the user
+                         ignore_no_or_lame " Type \"yes\" to proceed with \"$OPENSSL2\" OR accept all scan problems" "yes"
+                         if [[ $? -eq 0 ]]; then
+                              fileout "$jsonID" "INFO" "$NODE:$PORT appears to support TLS 1.3 ONLY, switching from $OPENSSL to $OPENSSL2 by the user"
+                              OPENSSL="$OPENSSL2"
+                              find_openssl_binary
+                              prepare_arrays
+                         else
+                              fileout "$jsonID" "WARN" "$NODE:$PORT appears to support TLS 1.3 ONLY, switching from $OPENSSL to $OPENSSL2 was denied by user"
+                         fi
+                    fi
                fi
           elif ! "$HAS_SSL3" && [[ "$(has_server_protocol "ssl3")" -eq 0 ]] && [[ "$(has_server_protocol "tls1_3")" -ne 0 ]] && \
                [[ "$(has_server_protocol "tls1_2")" -ne 0 ]] && [[ "$(has_server_protocol "tls1_1")" -ne 0 ]] &&
@@ -22092,6 +22100,18 @@ determine_optimal_proto() {
 }
 
 
+# Check messages which needed to be processed. I.e. those which would have destroyed the nice
+# screen output and thus havve been postponed. This is just an idea and is only used once
+# but can be extended in the future. An array might be more handy
+#
+check_msg() {
+     if [[ -n "$OPEN_MSG" ]]; then
+          outln "$OPEN_MSG"
+          OPEN_MSG=""
+     fi
+}
+
+
 # arg1 (optional): ftp smtp, lmtp, pop3, imap, sieve, xmpp, xmpp-server, telnet, ldap, postgres, mysql, irc, nntp (maybe with trailing s)
 #
 determine_service() {
@@ -22132,6 +22152,7 @@ determine_service() {
           determine_optimal_proto
           # returns always 0:
           service_detection $OPTIMAL_PROTO
+          check_msg
      else # STARTTLS
           if [[ "$1" == postgres ]] || [[ "$1" == sieve ]]; then
                protocol="$1"