Merge pull request #1408 from drwetter/connect-timeout

Connect timeout

doc/testssl.1

@@ -1,7 +1,7 @@
 .\" generated with Ronn/v0.7.3
 .\" http://github.com/rtomayko/ronn/tree/0.7.3
 .
-.TH "TESTSSL" "1" "April 2019" "" ""
+.TH "TESTSSL" "1" "December 2019" "" ""
 .
 .SH "NAME"
 \fBtestssl\fR
@@ -122,7 +122,7 @@ Please note that \fBfname\fR has to be in Unix format\. DOS carriage returns won
 \fB\-\-mode <serial|parallel>\fR\. Mass testing to be done serial (default) or parallel (\fB\-\-parallel\fR is shortcut for the latter, \fB\-\-serial\fR is the opposite option)\. Per default mass testing is being run in serial mode, i\.e\. one line after the other is processed and invoked\. The variable \fBMASS_TESTING_MODE\fR can be defined to be either equal \fBserial\fR or \fBparallel\fR\.
 .
 .SS "SPECIAL INVOCATIONS"
-\fB\-t <protocol>, \-\-starttls <protocol>\fR does a default run against a STARTTLS enabled \fBprotocol\fR\. \fBprotocol\fR must be one of \fBftp\fR, \fBsmtp\fR, \fBpop3\fR, \fBimap\fR, \fBxmpp\fR, \fBtelnet\fR, \fBldap\fR, \fBlirc\fR, \fBlmtp\fR, \fBnntp\fR, \fBpostgres\fR, \fBmysql\fR\. For the latter four you need e\.g\. the supplied OpenSSL or OpenSSL version 1\.1\.1\. Please note: MongoDB doesn\'t offer a STARTTLS connection, LDAP currently only works with \fB--ssl-native\fR\. \fBtelnet\fR and \fBirc\fR is WIP\.
+\fB\-t <protocol>, \-\-starttls <protocol>\fR does a default run against a STARTTLS enabled \fBprotocol\fR\. \fBprotocol\fR must be one of \fBftp\fR, \fBsmtp\fR, \fBpop3\fR, \fBimap\fR, \fBxmpp\fR, \fBtelnet\fR, \fBldap\fR, \fBirc\fR, \fBlmtp\fR, \fBnntp\fR, \fBpostgres\fR, \fBmysql\fR\. For the latter four you need e\.g\. the supplied OpenSSL or OpenSSL version 1\.1\.1\. Please note: MongoDB doesn\'t offer a STARTTLS connection, LDAP currently only works with \fB\-\-ssl\-native\fR\. \fBtelnet\fR and \fBirc\fR is WIP\.
 .
 .P
 \fB\-\-xmpphost <jabber_domain>\fR is an additional option for STARTTLS enabled XMPP: It expects the jabber domain as a parameter\. This is only needed if the domain is different from the URI supplied\.
@@ -323,7 +323,7 @@ Security headers (X\-Frame\-Options, X\-XSS\-Protection, Expect\-CT,\.\.\. , CSP
 \fB\-g, \-\-grease\fR checks several server implementation bugs like tolerance to size limitations and GREASE, see https://www\.ietf\.org/archive/id/draft\-ietf\-tls\-grease\-01\.txt \. This checks doesn\'t run per default\.
 .
 .SS "VULNERABILITIES"
-\fB\-U, \-\-vulnerable, \-\-vulnerablilities\fR Just tests all (of the following) vulnerabilities\. The environment variable \fBVULN_THRESHLD\fR determines after which value a separate headline for each vulnerability is being displayed\. Default is \fB1\fR which means if you check for two vulnerabilities, only the general headline for vulnerabilities section is displayed \-\- in addition to the vulnerability and the result\. Otherwise each vulnerability or vulnerability section gets its own headline in addition to the output of the name of the vulnerabilty and test result\. A vulnerability section is comprised of more than one check, e\.g\. the renegotiation vulnerability check has two checks, so has Logjam\.
+\fB\-U, \-\-vulnerable, \-\-vulnerabilities\fR Just tests all (of the following) vulnerabilities\. The environment variable \fBVULN_THRESHLD\fR determines after which value a separate headline for each vulnerability is being displayed\. Default is \fB1\fR which means if you check for two vulnerabilities, only the general headline for vulnerabilities section is displayed \-\- in addition to the vulnerability and the result\. Otherwise each vulnerability or vulnerability section gets its own headline in addition to the output of the name of the vulnerabilty and test result\. A vulnerability section is comprised of more than one check, e\.g\. the renegotiation vulnerability check has two checks, so has Logjam\.
 .
 .P
 \fB\-H, \-\-heartbleed\fR Checks for Heartbleed, a memory leakage in openssl\. Unless the server side doesn\'t support the heartbeat extension it is likely that this check runs into a timeout\. The seconds to wait for a reply can be adjusted with \fBHEARTBLEED_MAX_WAITSOCK\fR\. 8 is the default\.
@@ -377,6 +377,9 @@ Security headers (X\-Frame\-Options, X\-XSS\-Protection, Expect\-CT,\.\.\. , CSP
 \fB\-\-warnings <batch|off|false>\fR The warnings parameter determines how testssl\.sh will deal with situations where user input normally will be necessary\. There are a couple of options here\. \fBbatch\fR doesn\'t wait for a confirming keypress\. This is automatically being chosen for mass testing (\fB\-\-file\fR)\. \fB\-false\fR just skips the warning AND the confirmation\. Please note that there are conflicts where testssl\.sh will still ask for confirmation which are the ones which otherwise would have a drastic impact on the results\. Almost any other decision will be made as a best guess by testssl\.sh\. The same can be achieved by setting the environment variable \fBWARNINGS\fR\.
 .
 .P
+\fB\-\-connect\-timeout <seconds>\fR This is useful for socket TCP connections to a node\. If the node does not complete a TCP handshake (e\.g\. because it is down or behind a firewall or there\'s an IDS or a tarpit) testssl\.sh may ususally hang for around 2 minutes or even much more\. This parameter instructs testssl\.sh to wait at most \fBseconds\fR for the handshake to complete before giving up\. This option only works if your OS has a timeout binary installed\. CONNECT_TIMEOUT is the corresponding enviroment variable\.
+.
+.P
 \fB\-\-openssl\-timeout <seconds>\fR This is especially useful for all connects using openssl and practically useful for mass testing\. It avoids the openssl connect to hang for ~2 minutes\. The expected parameter \fBseconds\fR instructs testssl\.sh to wait before the openssl connect will be terminated\. The option is only available if your OS has a timeout binary installed\. As there are different implementations of \fBtimeout\fR: It automatically calls the binary with the right parameters\. OPENSSL_TIMEOUT is the equivalent environment variable\.
 .
 .P

doc/testssl.1.md

@@ -165,7 +165,7 @@ Any single check switch supplied as an argument prevents testssl.sh from doing a
 
 * Available TLS extensions,
 * TLS ticket + session ID information/capabilities,
-* session resumption capabilities, 
+* session resumption capabilities,
 * Time skew relative to localhost (most server implementations return random values).
 * Several certificate information
     - signature algorithm,
@@ -179,7 +179,7 @@ Any single check switch supplied as an argument prevents testssl.sh from doing a
     - validity: start + end time, how many days to go (warning for certificate lifetime >=5 years)
     - revocation info (CRL, OCSP, OCSP stapling + must staple). When `--phone-out` supplied it checks against the certificate issuer whether the host certificate has been revoked (plain OCSP, CRL).
     - displaying DNS Certification Authority Authorization resource record
-    - Certificate Transparency info (if provided by server).  
+    - Certificate Transparency info (if provided by server).
 
 For the trust chain check 5 certificate stores are provided. If the test against one of the trust stores failed, the one is being identified and the reason for the failure is displayed - in addition the ones which succeeded are displayed too.
 You can configure your own CA via ADDITIONAL_CA_FILES, see section `FILES` below.  If the server provides no matching record in Subject Alternative Name (SAN) but in Common Name (CN), it will be indicated as this is deprecated.
@@ -247,9 +247,11 @@ Also for multiple server certificates are being checked for as well as for the c
 
 ### OUTPUT OPTIONS
 
-`--warnings <batch|off|false>`     The warnings parameter determines how testssl.sh will deal with situations where user input normally will be necessary. There are a couple of options here.  `batch` doesn't wait for a confirming keypress. This is automatically being chosen for mass testing (`--file`). `-false` just skips the warning AND the confirmation. Please note that there are conflicts where testssl.sh will still ask for confirmation which are the ones which otherwise would have a drastic impact on the results. Almost any other decision will be made as a best guess by testssl.sh.
+`--warnings <batch|off|false>`     The warnings parameter determines how testssl.sh will deal with situations where user input normally will be necessary. There are a couple of options here. `batch` doesn't wait for a confirming keypress. This is automatically being chosen for mass testing (`--file`). `-false` just skips the warning AND the confirmation. Please note that there are conflicts where testssl.sh will still ask for confirmation which are the ones which otherwise would have a drastic impact on the results. Almost any other decision will be made as a best guess by testssl.sh.
 The same can be achieved by setting the environment variable `WARNINGS`.
 
+`--connect-timeout <seconds>`  This is useful for socket TCP connections to a node. If the node does not complete a TCP handshake (e.g. because it is down or behind a firewall or there's an IDS or a tarpit) testssl.sh may ususally hang for around 2 minutes or even much more. This parameter instructs testssl.sh to wait at most `seconds` for the handshake to complete before giving up. This option only works if your OS has a timeout binary installed. CONNECT_TIMEOUT is the corresponding enviroment variable.
+
 `--openssl-timeout <seconds>` This is especially useful for all connects using openssl and practically useful for mass testing. It avoids the openssl connect to hang for ~2 minutes. The expected parameter `seconds` instructs testssl.sh to wait before the openssl connect will be terminated. The option is only available if your OS has a timeout binary installed. As there are different implementations of `timeout`: It automatically calls the binary with the right parameters. OPENSSL_TIMEOUT is the equivalent environment variable.
 
 `-q, --quiet`  Normally testssl.sh displays a banner on stdout with several version information, usage rights and a warning. This option suppresses it. Please note that by choosing this option you acknowledge usage terms and the warning normally appearing in the banner.

testssl.sh

@@ -190,7 +190,9 @@ TERM_CURRPOS=0                                              # custom line wrappi
 ########### Defining (and presetting) variables which can be changed
 #
 # Following variables make use of $ENV and can be used like "OPENSSL=<myprivate_path_to_openssl> ./testssl.sh <URI>"
-declare -x OPENSSL OPENSSL_TIMEOUT
+declare -x OPENSSL
+OPENSSL_TIMEOUT=${OPENSSL_TIMEOUT:-""}  # Default connect timeout with openssl before we call the server side unreachable
+CONNECT_TIMEOUT=${CONNECT_TIMEOUT:-""}  # Default connect timeout with sockets before we call the server side unreachable
 PHONE_OUT=${PHONE_OUT:-false}           # Whether testssl can retrieve CRLs and OCSP
 FAST_SOCKET=${FAST_SOCKET:-false}       # EXPERIMENTAL feature to accelerate sockets -- DO NOT USE it for production
 COLOR=${COLOR:-2}                       # 3: Extra color (ciphers, curves), 2: Full color, 1: B/W only 0: No ESC at all
@@ -280,6 +282,7 @@ GIVE_HINTS=false                        # give an additional info to findings
 SERVER_SIZE_LIMIT_BUG=false             # Some servers have either a ClientHello total size limit or a 128 cipher limit (e.g. old ASAs)
 MULTIPLE_CHECKS=false                   # need to know whether an MX record or a hostname resolves to multiple IPs to check
 CHILD_MASS_TESTING=${CHILD_MASS_TESTING:-false}
+TIMEOUT_CMD=""
 HAD_SLEPT=0
 NR_SOCKET_FAIL=0                        # Counter for socket failures
 NR_OSSL_FAIL=0                          # .. for OpenSSL connects
@@ -10245,12 +10248,22 @@ fd_socket() {
                     break
                fi
           done
-     elif ! exec 5<>/dev/tcp/$nodeip/$PORT; then  #  2>/dev/null would remove an error message, but disables debugging
+     # For the following execs: 2>/dev/null would remove a potential error message, but disables debugging.
+     # First we check whether a socket connect timeout was specified
+     elif [[ -n "$CONNECT_TIMEOUT" ]]; then
+          if ! $TIMEOUT_CMD $CONNECT_TIMEOUT bash -c "exec 5<>/dev/tcp/$nodeip/$PORT"; then
+               ((NR_SOCKET_FAIL++))
+               connectivity_problem $NR_SOCKET_FAIL $MAX_SOCKET_FAIL "TCP connect problem" "repeated TCP connect problems (connect timeout), giving up"
+               outln
+               pr_warning "Unable to open a socket to $NODEIP:$PORT. "
+               return 6
+          fi
+     # Now comes the the usual case
+     elif ! exec 5<>/dev/tcp/$nodeip/$PORT; then
           ((NR_SOCKET_FAIL++))
           connectivity_problem $NR_SOCKET_FAIL $MAX_SOCKET_FAIL "TCP connect problem" "repeated TCP connect problems, giving up"
           outln
           pr_warning "Unable to open a socket to $NODEIP:$PORT. "
-          # It can last ~2 minutes but for for those rare occasions we don't do a timeout handler here, KISS
           return 6
      fi
 
@@ -16299,7 +16312,7 @@ run_robot() {
      local -a response
      local -i i subret len iteration testnum pubkeybits pubkeybytes
      local vulnerable=false send_ccs_finished=true
-     local -i start_time end_time timeout=$MAX_WAITSOCK
+     local -i start_time end_time robottimeout=$MAX_WAITSOCK
      local cve="CVE-2017-17382 CVE-2017-17427 CVE-2017-17428 CVE-2017-13098 CVE-2017-1000385 CVE-2017-13099 CVE-2016-6883 CVE-2012-5081 CVE-2017-6168"
      local cwe="CWE-203"
      local jsonID="ROBOT"
@@ -16464,7 +16477,7 @@ run_robot() {
                fi
                debugme echo "reading server error response..."
                start_time=$(LC_ALL=C date "+%s")
-               sockread_serverhello 32768 $timeout
+               sockread_serverhello 32768 $robottimeout
                subret=$?
                if [[ $subret -eq 0 ]]; then
                     end_time=$(LC_ALL=C date "+%s")
@@ -16474,9 +16487,9 @@ run_robot() {
                     # exchange message, measure the amount of time it took to
                     # receive a response and set the timeout value for future
                     # tests to 2 seconds longer than it took to receive a response.
-                    [[ $iteration -ne 2 ]] && [[ $timeout -eq $MAX_WAITSOCK ]] && \
+                    [[ $iteration -ne 2 ]] && [[ $robottimeout -eq $MAX_WAITSOCK ]] && \
                          [[ $((end_time-start_time)) -lt $((MAX_WAITSOCK-2)) ]] && \
-                         timeout=$((end_time-start_time+2))
+                         robottimeout=$((end_time-start_time+2))
                else
                     response[testnum]="Timeout waiting for alert"
                fi
@@ -16515,14 +16528,14 @@ run_robot() {
                # If the test was run with a short timeout and was found to be
                # potentially vulnerable due to some tests timing out, then
                # verify the results by rerunning with a longer timeout.
-               if [[ $timeout -eq $MAX_WAITSOCK ]]; then
+               if [[ $robottimeout -eq $MAX_WAITSOCK ]]; then
                     break
                elif [[ "${response[0]}" == "Timeout waiting for alert" ]] || \
                     [[ "${response[1]}" == "Timeout waiting for alert" ]] || \
                     [[ "${response[2]}" == "Timeout waiting for alert" ]] || \
                     [[ "${response[3]}" == "Timeout waiting for alert" ]] || \
                     [[ "${response[4]}" == "Timeout waiting for alert" ]]; then
-                    timeout=10
+                    robottimeout=10
                else
                     break
                fi
@@ -16832,23 +16845,27 @@ find_openssl_binary() {
 
      [[ "$(echo -e "\x78\x9C\xAB\xCA\xC9\x4C\xE2\x02\x00\x06\x20\x01\xBC" | $OPENSSL zlib -d 2>/dev/null)" == zlib ]] && HAS_ZLIB=true
 
-     if [[ "$OPENSSL_TIMEOUT" != "" ]]; then
+     if [[ -n "$CONNECT_TIMEOUT" ]] || [[ -n "$OPENSSL_TIMEOUT" ]]; then
+          # We don't set a general timeout as we might not have "timeout" installed and we only
+          # do what is instructed. Thus we check first what the command line params were,
+          # then we proceed
           if type -p timeout >/dev/null 2>&1; then
-               if ! "$do_mass_testing"; then
+               # There are different versions of "timeout". Check whether --preserve-status is supported
-                    # there are different "timeout". Check whether --preserve-status is supported
+               if timeout --help 2>/dev/null | grep -q 'preserve-status'; then
-                    if timeout --help 2>/dev/null | grep -q 'preserve-status'; then
+                    TIMEOUT_CMD="timeout --preserve-status"
-                         OPENSSL="timeout --preserve-status $OPENSSL_TIMEOUT $OPENSSL"
+               else
-                    else
+                    TIMEOUT_CMD="timeout"
-                         OPENSSL="timeout $OPENSSL_TIMEOUT $OPENSSL"
-                    fi
                fi
-               MAX_OSSL_FAIL+=2
           else
+               TIMEOUT_CMD=""
                outln
-               prln_warning " Necessary binary \"timeout\" not found."
+               fatal "You specified a connect or openssl timeout but the binary \"timeout\" couldn't be found " $ERR_RESOURCE
-               ignore_no_or_lame " Continue without timeout? " "yes"
+          fi
-               [[ $? -ne 0 ]] && exit $ERR_OSSLBIN
+     fi
-               unset OPENSSL_TIMEOUT
+
+     if ! "$do_mass_testing"; then
+          if [[ -n $OPENSSL_TIMEOUT ]]; then
+               OPENSSL="$TIMEOUT_CMD $OPENSSL_TIMEOUT $OPENSSL"
           fi
      fi
 
@@ -17010,7 +17027,8 @@ tuning / connect options (most also can be preset via environment variables):
 
 output options (can also be preset via environment variables):
      --warnings <batch|off|false>  "batch" doesn't ask for a confirmation, "off" or "false" skips connection warnings
-     --openssl-timeout <seconds>   useful to avoid hangers. <seconds> to wait before openssl connect will be terminated
+     --connect-timeout <seconds>   useful to avoid hangers. Max <seconds> to wait for the TCP socket connect to return
+     --openssl-timeout <seconds>   useful to avoid hangers. Max <seconds> to wait before openssl connect will be terminated
      --quiet                       don't output the banner. By doing this you acknowledge usage terms normally appearing in the banner
      --wide                        wide output for tests like RC4, BEAST. PFS also with hexcode, kx, strength, RFC name
      --show-each                   for wide outputs: display all ciphers tested -- not only succeeded ones
@@ -18188,7 +18206,7 @@ determine_optimal_proto() {
                [[ $? -ne 0 ]] && exit $ERR_CLUELESS
                MAX_OSSL_FAIL=10
           else
-               prln_bold " Your $OPENSSL cannot connect to $NODEIP:$PORT"
+               prln_bold " Your OpenSSL cannot connect to $NODEIP:$PORT"
                ignore_no_or_lame " The results might look ok but they could be nonsense. Really proceed ? (\"yes\" to continue)" "yes"
                [[ $? -ne 0 ]] && exit $ERR_CLUELESS
           fi
@@ -19581,6 +19599,10 @@ parse_cmd_line() {
                     OPENSSL_TIMEOUT="$(parse_opt_equal_sign "$1" "$2")"
                     [[ $? -eq 0 ]] && shift
                     ;;
+               --connect-timeout|--connect-timeout=*)
+                    CONNECT_TIMEOUT="$(parse_opt_equal_sign "$1" "$2")"
+                    [[ $? -eq 0 ]] && shift
+                    ;;
                --mapping|--mapping=*)
                     cipher_mapping="$(parse_opt_equal_sign "$1" "$2")"
                     [[ $? -eq 0 ]] && shift