This commit adds some more checks to determine_optimal_sockets_params(). These additional checks will almost never need to be run, and so will not slow down the typical run of testssl.sh, but adding them will provide information that can be useful for other parts of testssl.sh.
These additional checks will only be run if the server does not support TLSv1.3 and a TLSv1.2 ClientHello is not successful. This means that either:
* The server is not an TLS/SSL enabled server.
* The server only supports SSLv2.
* The server supports some protocol in SSLv3 - TLSv1.1, but does not handle version negotiation correctly.
Adding these additional checks helps in at least the following ways.
If determine_optimal_proto() (assuming it is reverted to using OpenSSL) is unable to connect to the server using OpenSSL, it will be possible determine whether the problem is the the server does not support TLS/SSL or that a different version of OpenSSL is needed to test the server.
If the code in #1205 for run_server_preference() is unable to connect when checking for a cipher order, the reason for the failure will be known, making it possible to determine the correct response to the failure.
This PR adds a new helper function that is run just prior to determine_optimal_proto() and that determines the what information tls_sockets() should include in a ClientHello.
For a TLSv1.3 ClientHello, determine_optimal_sockets_params() determines whether tls_sockets() should use 0x33 or 0x28 are the extension number for the key_share extension. 0x33 should be used with servers that support RFC 8446 or drafts 23-28. 0x28 should be used with servers that support drafts 18-22.
For a TLSv1.2 ClientHello, determine_optimal_sockets_params() determines what cipher list tls_sockets() should send. For most servers, the list of ciphers in $TLS12_CIPHER works best. But, there are some servers that do not support any ciphers in $TLS12_CIPHER, but do support one or more ciphers in $TLS12_CIPHER_2ND_TRY.
In some rare cases a server does not support any of the ciphers in $TLS12_CIPHER, but does support at least one cipher in $TLS12_CIPHER_2ND_TRY. In such cases, TLS12_CIPHER should be changed to $TLS12_CIPHER_2ND_TRY so that subsequent tests using $TLS12_CIPHER will succeed.
This PR fixes a few issues with run_protocol():
* In the case that the call to `tls_sockets "03" "$TLS12_CIPHER"` had a return value of 2, the code determining what results to print was looking at `$DETECTED_TLS_VERSION`. However, the value of this variable was set by the later call to `tls_sockets "04" "$tls13_ciphers_to_test"`. This caused incorrect results in the case of a server that supports TLSv1.3 and TLS1.1 (or earlier), but not TLSv1.2. This PR saves the value of `$DETECTED_TLS_VERSION` in `$tls12_detected_version` and then uses this variable later rather than `$DETECTED_TLS_VERSION`.
* When running in debug mode with a server that does not support TLSv1.3, testssl.sh was printing
TLS 1.3 -- downgradednot offered and downgraded to a weaker protocol"
This PR fixes the output by not printing the "--downgraded"
* As noted in #1329, run_protocols() was treating a downgrade from TLSv1.2 as less bad if the server supports TLSv1.3. This PR changes this code back to treat any downgrade from TLSv1.2 as equally bad.
* In order to be consistent with the TLSv1.3 test, this PR changes the TLS1.2 test output to say "not offered and downgraded to a weaker protocol" if a TLSv1.2 ClientHello results in a downgraded connection.
This PR fixes a problem in run_protocols() that was introduced by https://github.com/drwetter/testssl.sh/commit/7ec3c6ab997b012b1518903137d84043cf9825a8.
https://github.com/drwetter/testssl.sh/commit/7ec3c6ab997b012b1518903137d84043cf9825a8 changes run_protocols() to perform the initial testing for TLSv1.3 support before testing for TLSv1.2 support. The problem with this is that the code for testing TLSv1.3 makes use of the results of the TLSv1.2 testing.
In the current code, Line 5183 looks at the value of $subret to determine whether the TLSv1.2 ClientHello resulted in a successful connection. However, $subet has not yet been set (it has just been initialized to 0 at the beginning of the function). Since $subret will always be 0, the code will try to extract a cipher from $TEMPDIR/$NODEIP.parse_tls_serverhello.txt. This may work, since $TEMPDIR/$NODEIP.parse_tls_serverhello.txt may have been populated by a prior function call, but this is not how the code was intended to work.
This PR fixes the problem by doing the TLSv1.2 testing before the TLSv1.3 testing is done. It still waits until both have been tested, however, before outputting the results, so that the output for TLSv1.2 can be modified depending on whether TLSv1.3 is supported.
draft-ietf-tls-certificate-compression specifies a new TLS extension that allows a client to indicate support for receiving the server's certificates in compressed form. This PR adds initial support for that extension to testssl.sh. It was developed based on an announcement that facebook.com has implemented support for the extension.
This PR does not add a test for a server's support for the new extension, it just adds code to parse_tls_serverhello() that will parse a compressed certificate message if one is present in the server's response. So, in practice, the code in this PR will not actually be used until additional code has been added that sends a ClientHello with the compress_certificate extension.
The code in this PR can be tested by using the --devel option and by changing line 19347 from
tls_sockets "$TLS_LOW_BYTE" "$HEX_CIPHER" "ephemeralkey"
to
tls_sockets "$TLS_LOW_BYTE" "$HEX_CIPHER" "all+" "00,1b, 00,03, 02, 00,01"
testssl.sh can then be called using
testssl.sh --debug 3 --devel 04 "13,01" facebook.com
Note that this PR adds a test for zlib support to find_openssl_binary(). The test uses $OPENSSL zlib in the same way it is used in parse_tls_serverhello(). The test asks $OPENSSL zlib to uncompress the compressed version of the string "zlib" and the checks to see whether the decompression was performed correctly. There is already a test for zlib support in run_crime(), but I did not check to see whether the check in run_crime() could be replaced with a check of the new $HAS_ZLIB variable.
This PR changes run_protocols() so that, when using tls_sockets(), support for TLSv1.3 is only marked as pr_svrty_best() if the final (RFC 8446) version is supported. It also changed run_protocols() so that support for TLSv1.3 is marked as pr_svrty_best() if OpenSSL is used (i.e., if the --ssl-native option is specified).
One potential issue is that the --ssl-native version assumes that if OpenSSL supports TLSv1.3 it supports the final (RFC 8446) version of the protocol. If the tester is using a development version of OpenSSL 1.1.1 rather than the final version, then the protocol test will actually be indicating whether the server supports the same draft version of OpenSSL as the $OPENSSL being used to perform the tests.
Currently the -outfile, -oa, -outFile, and -oA assume that <fname> being provided is to be used as a filename, unless it is "auto." However, all of the individual options (e.g., --logfile) allow for a directory name to be provided instead of a file name.
This PR changes the handling of the -outfile, -oa, -outFile, and -oA options so that if a directory name is provided, the files are created in that directory.
This PR fixes#1243 by modifying create_mass_testing_cmdline() to handle the --outfile, -oa, --outFile, and -oA options in the case that the filename provided is not "auto." It also modifies create_mass_testing_cmdline() so that in the case of serial testing -oj and -oJ are treated the same as --jsonfile and --jsonfile-pretty, respectively.
This PR addresses the following issues raised by shellcheck:
In ../github/testssl_2.9dev_20190409b.sh line 1133:
if [[ "$BASH_VERSINFO" == 3 ]]; then
^-- SC2128: Expanding an array without an index only gives the first element.
In ../github/testssl_2.9dev_20190409b.sh line 4301:
tmpfile_handle $FUNCNAME.dd
^-- SC2128: Expanding an array without an index only gives the first element.
In ../github/testssl_2.9dev_20190409b.sh line 4388:
tmpfile_handle $FUNCNAME.dd
^-- SC2128: Expanding an array without an index only gives the first element.
In ../github/testssl_2.9dev_20190409b.sh line 4657:
tmpfile_handle $FUNCNAME$1.txt
^-- SC2128: Expanding an array without an index only gives the first element.
In ../github/testssl_2.9dev_20190409b.sh line 5327:
tmpfile_handle $FUNCNAME.${debugname}.txt
^-- SC2128: Expanding an array without an index only gives the first element.
In ../github/testssl_2.9dev_20190409b.sh line 5943:
tmpfile_handle $FUNCNAME.byID.log $tmpfile || \
^-- SC2128: Expanding an array without an index only gives the first element.
In ../github/testssl_2.9dev_20190409b.sh line 5944:
tmpfile_handle $FUNCNAME.byticket.log $tmpfile
^-- SC2128: Expanding an array without an index only gives the first element.
In ../github/testssl_2.9dev_20190409b.sh line 12410:
tmpfile_handle $FUNCNAME.dd $SOCK_REPLY_FILE
^-- SC2128: Expanding an array without an index only gives the first element.
In ../github/testssl_2.9dev_20190409b.sh line 13164:
tmpfile_handle $FUNCNAME.dd
^-- SC2128: Expanding an array without an index only gives the first element.
In ../github/testssl_2.9dev_20190409b.sh line 13284:
tmpfile_handle $FUNCNAME.dd $SOCK_REPLY_FILE
^-- SC2128: Expanding an array without an index only gives the first element.
In ../github/testssl_2.9dev_20190409b.sh line 13388:
tmpfile_handle $FUNCNAME.dd $SOCK_REPLY_FILE
^-- SC2128: Expanding an array without an index only gives the first element.
--
In ../github/testssl_2.9dev_20190409b.sh line 13801:
[[ "$DEBUG" -ge 1 ]] && echo $tls_hello_ascii >$TEMPDIR/$FUNCNAME.tls_hello_ascii${i}.txt
^-- SC2128: Expanding an array without an index only gives the first element.
This PR fixes#1223 by checking whether the stapled OCSP response from the server is an error message.
Another way to fix#1223 would be to just change line 8510 to:
```
if grep -a "OCSP Response Status" <<< "$ocsp_response_status" | grep -q successful || \
[[ "$ocsp_response" =~ Responder\ Error: ]]; then
```
However, I believe this alternative would lead to confusing results, testssl.sh would print
offered, error querying OCSP responder (tryLater)
I'm not sure whether it makes sense to say "offered" when the stapled response that is provided is just an error message, but I think it is important to make clear that the error response was received from the TLS server, and that it wasn't testssl.sh that tried querying the OCSP responder.
There is currently a bug in determine_optimal_proto_sockets_helper(). In two places there is code of the form:
tls_sockets ...
if [[ $? -eq 0 ]]; then
...
elif [[ $? -eq 2 ]]; then
...
fi
This code does not work as intended since the second check ("elif [[ $? -eq 2 ]]") is actually comparing the results of the first check to 2 rather than the results of the call to tls_sockets().
This PR fixes that problem and also speeds up the code. Since tls_sockets() sets $DETECTED_TLS_VERSION to the protocol version that was negotiated, there is no need to scan $TEMPDIR/$NODEIP.parse_tls_serverhello.txt for this information.
This is a minor bug when performing run_server_preference() if the server cannot handle ClientHello messages with more than 128 ciphers (i.e., $SERVER_SIZE_LIMIT_BUG is true) and the server supports at least one cipher in 'CAMELLIA:IDEA:KRB5:PSK:SRP:aNULL:eNULL'.
The problem is that `$OPENSSL s_client` is called with a cipher list such as
ECDHE-RSA-AES256-GCM-SHA384:CAMELLIA256-SHA:AES256-SHA256
then
ECDHE-RSA-AES256-GCM-SHA384:CAMELLIA256-SHA:AES256-SHA256:-CAMELLIA256-SHA
then
ECDHE-RSA-AES256-GCM-SHA384:CAMELLIA256-SHA:AES256-SHA256:-CAMELLIA256-SHA:-AES256-SHA256
and finally
ECDHE-RSA-AES256-GCM-SHA384:CAMELLIA256-SHA:AES256-SHA256:-CAMELLIA256-SHA:-AES256-SHA256:-ECDHE-RSA-AES256-GCM-SHA384
The last call to $OPENSSL s_client produces an error since the list of ciphers to send is empty, and this results in connectivity_problem() being called to print a "openssl s_client connect problem" warning.
This PR fixes the problem by constructing a list of ciphers to test for and by not calling $OPENSSL s_client if the list is empty.
As Dirk commented in #1199, TLS_GOSTR341094_RSA_WITH_28147_CNT_MD5 uses an RSA certificate, not a GOST certificate. So, this PR moves that cipher suite (0xff, 0x00) from the GOST list to the RSA list.
According to etc/cipher-mapping.txt, TLS_GOSTR341094_RSA_WITH_28147_CNT_MD5 uses RSA for both authentication and key exchange, so this PR places it on the list of cipher suites that uses RSA for encryption rather than signatures.
This PR modifies get_server_certificate() to use tls_sockets() rather than $OPENSSL for finding certificates using SSLv3 - TLSv1.2, unless $SSL_NATIVE is true. Using tls_sockets() allows testssl.sh to find certificates used by the server even if the server is only using cipher suites not supported by $OPENSSL. This may happen, for example, if the server only supports TLS_ECDHE_ cipher suites with curve X25519 and a version of OpenSSL prior to 1.1.0 is being used. A less likely possibility would be if the server had a certificate with a DH key, and a newer version of OpenSSL that does not support TLS_DH_ cipher suites is being used.
Since tls_sockets() cannot be used to obtain session tickets from the server, an additional test for session ticket lifetime needed to be added.
In order to reduce the number of times the server needs to be queried for certificates, this PR bundles the testing in a similar way to what is already done to test for cipher suites. Currently, each call to get_server_certificate() only tests for one type of certificate. This PR has each call test for more than one type of certificate. For example, one call is made to test for ECDSA, ECDH, DH, DSA, and GOST certificates. If the test is unsuccessful, then the server has none of these certificates. If the test finds a certificate (e.g., an ECDSA) certificate, then another test is run looking for the remaining types (ECDH, DH, DSA, and GOST) until a test is unsuccessful.
For most servers, this will reduce the number of calls to get_server_certificate() from 8 or 9 to 4 or 6.
This PR makes a few improvements to run_server_defaults() when run on an SSLv2-only server.
First, it uses sslv2_sockets() to test the server rather than $OPENSSL, so that it will work even if $OPENSSL does not support SSLv2.
Second, it changes run_server_defaults() to only call get_server_certificate() once if $OPTIMAL_PROTO is -ssl2, since calling more than once is a waste - SSLv2 only supports ciphers that use RSA key exchange.
Finally, as some code assumes that $TEMPDIR/intermediatecerts.pem will exist, even if it is empty, this PR changes a couple of places that delete $TEMPDIR/intermediatecerts.pem to instead make the file empty.
When run_server_preference() is run on a server that only supports SSLv2 it incorrectly reports that the server has a cipher order. The reason for this is that $list_fwd and $list_reverse only include one SSLv2 cipher.
In SSLv2 the server sends a list of all ciphers it supports in common with the client and the client chooses which cipher to use. As a result, the server cannot enforce a cipher order for SSLv2.
So, this PR fixes the problem in run_server_preference() by skipping the test for whether the server enforces a cipher order if $OPTIMAL_PROTO is -ssl2 and simply declares that the server does not enforce a cipher order.
Note that this PR is somewhat dependent on #1194, as #1194 needs to be applied in order for $OPTIMAL_PROTO to be set to -ssl2 when testing an SSLv2-only server.
This PR reorganizes run_cipher_per_proto(). Currently run_cipher_per_proto() runs a for loop, which loops over each protocol and prints the set of supported ciphers for each protocol. This PR simply places the body of the for loop in a separate function from the loop itself. This allows the body of the loop to be called for just a single protocol.
While this PR does not change the way that testssl.sh functions, it would allow for a future change in which run_server_preferences() called cipher_pref_check() for protocols in which the server enforces a cipher order and calls ciphers_by_strength() for protocols in which the server does not enforce a cipher order.
This PR reorganizes cipher_pref_check(). Currently, cipher_pref_check() runs a for loop, which loops over each protocol and prints the set of supported ciphers for each protocol. This PR simply places the body of the for loop in a separate function from the loop itself. This allows cipher_pref_check() to be called for just a single protocol rather than for all protocols. Another PR will make a similar change to run_cipher_per_proto().
The reason for this change is that cipher_pref_check() was only intended to be used in cases in which the server enforces a cipher preference order. Some servers, however, enforce an order for some protocols, but not for others. The change in this PR will make it possible in the future to call cipher_pref_check() only for protocols in which the server enforces a cipher order.
This PR fixes two bugs in determine_optimal_proto().
First, sslv2_sockets() returns 3 if the connection was successful.
Second, if all connection attempts using tls_sockets() were unsuccessful, it is possible that $TEMPDIR/$NODEIP.parse_tls_serverhello.txt will not exist, so copying it or grepping it will lead to an error. Checking that $proto is not 22 will fix this as $proto will be empty is $OPENSSL s_client was used and it will be 00, 01, 02, 03, or 04 if tls_sockets() was used and the connection was successful with some protocol higher than SSLv2.
This PR fixes a bug in get_cipher() - one that also appears in sclient_connect_successful().
The code currently assumes that cipher names contain only uppercase letters and numbers. However, ciphers that do not provide authentication include "anon" in the name, which is written in lowercase.
This PR fixes the problem by allowing lowercase letters to appear in cipher names (except in the first portion of the name).
Note that no change was made to similar code in get_protocol(), since the line in get_protocol() only matches TLSv1.3 ciphers, which do not contain any lowercase letters.
This PR fixes two problems with modify_clienthello().
First, the function was incorrectly using the variable $key_share instead of $new_key_share. Since $key_share is defined when modify_clienthello() is called from resend_if_hello_retry_request(), but not when it is called from client_simulation_sockets(), this bug does not seem to result in incorrect behavior, but it should still be fixed.
Second, when this function is used to create a second ClientHello in response to a HelloRetryRequest, it removes the key_share extension from the original ClientHello and then appends the new key_share extension at the end. According to https://mailarchive.ietf.org/arch/msg/tls/8ZKCyamcYFaV90h6nf4MUnSPkEE, however, extensions must appear in the second ClientHello in the same order in which they appeared in the first ClientHello.
I am not aware of any servers that will actually complain if the extensions in the second ClientHello do not appear in the same order as in the first ClientHello, bug this fix helps to ensure that the ClientHello messages testssl.sh sends are in compliance with the standard.
This PR is an attempt at addressing #1185. According to https://www.etsi.org/deliver/etsi_ts/103500_103599/10352303/01.01.01_60/ts_10352303v010101p.pdf, if eTLS is in use, then the certificate should contain a subjectAltName extension with one or more "names" containing "visibility information." The "visibility information" is encoded as an otherName with a type-id of 0.4.0.3523.3.1 and a value of
VisibilityInformation ::= SEQUENCE {
fingerprint OCTET STRING (SIZE(10)),
accessDescription UTF8String }
The etsi_etls_visibility_info() function determines whether the certificate includes an "visibility information," and, if it does, extracts the fingerprints and access descriptions.
This PR is a work-in-progress for two reasons. First, it has not been tested against any real certificates that contain "visibility information." Testing against real certificates would be helpful to verify that the parsing of the certificate is correct.
Second, the presentation of the visibility information (both in the printed text and in what is sent to fileout()) may need improvement. Having seen no examples, it is not clear what the contents of accessDescription can be expected to look like. The document says that the contents will be "human-readable text," but it is not clear whether the description will be relatively short or very long.
This PR fixes#1165 by changing resend_if_hello_retry_request() to modify the initial ClientHello rather than having it call prepare_tls_clienthello() to try to generate a new ClientHello that is almost the same as the first. The modification is done using a revised version of create_client_simulation_tls_clienthello(), which is now renamed as modify_clienthello().
Since prepare_tls_clienthello() is no longer used to create a second ClientHello message, argument 7 to that function is no longer needed.
There is at least one extension that will fail on a TLSv1.3 ClientHello if the psk_key_exchange_modes extension is not present (see #990). The PR adds the extension to TLSv1.3 ClientHello messages. OpenSSL, Firefox, and Chrome all include this extension in their ClientHello messages, so including it is unlikely to cause problems for any servers.
This PR provides an additional fix for the issue raised by #1159. It defines a third option for the degree of processing that should be performed by tls_sockets(): "all+". When "all+" is provided, the processing is exactly the same as for "all" with the exception of the creation of the supported_groups extension. For a TLSv1.3 ClientHello, curves that are not supported by $OPENSSL are omitted from the supported_groups extension rather than offering these curves as the least preferred option.
The "all+" option is used in run_server_defaults() where, unlike with almost every other call to tls_sockets(), a successful connection is of no use unless the response can be decrypted. This is also the case for run_alpn(), and so the call to tls_sockets() was also changed to "all+" there. But, the change has no effect at the moment, since run_alpn() sends a TLSv1.2 ClientHello.
As a result of #276, `run_server_defaults()` makes several attempts to find certificates that a server offers if the ClientHello is for TLSv1.2 and no SNI is offered. However, these tests are unnecessary if it is already known that the server does not support TLSv1.1.
This PR modifies `run_server_defaults()` so that the the TLSv1.1-only tests are skipped if the server is known to not support TLSv1.1.
This PR fixes#1159. If tls_sockets() connects to a server using TLSv1.3, it cannot be assumed that the server's certificate is available, as testssl.sh may not have been able to decrypt the server's response. This can happen, for example, if X25519 was used for the key exchange and `$OPENSSL` does not support X25519.
If the connection was successful, but the certificate could not be obtained, then this PR tries again using `$OPENSSL`. However, since `$OPENSSL` does not support TLSv1.3, this will only work if the server supports TLSv1.2 or earlier.
run_logjam() is only related to TLSv1.2 and earlier ciphers. So, run_pfs() should only update $DH_GROUP_OFFERED if a DH group was found using a non-TLSv1.3 cipher.
On the other side, if run_logjam() happened to have been run first, and it found an ffdhe cipher, then there is no need for run_pfs() to test for it.
In run_pfs(), when information about the finite field groups offered is printed, the color used is based on the length of the key. This information should also be conveyed to fileout() in the severity parameter.
For cipher suites that use ephemeral DH groups, run_pfs() currently only displays information about the group(s) used if the server complies with RFC 7919. In the case of TLSv1.3 this is appropriate, since server can only use the values from this RFC and only if they are offered by the client in the supported_groups extension.
For TLSv1.2 and earlier, however, servers are free to use whatever DH group they want, but run_pfs() only provides information about the group the server uses if the server complies with RFC 7919. (The information is, however, provided by run_logjam()). However, so far no servers comply with RFC 7919's requirement to refuse to negotiate a TLS_DHE cipher if the supported groups extension is present, included DH groups, but none that are supported by the server. There is also reason to believe that this will not change: https://www.ietf.org/mail-archive/web/tls/current/msg26378.html.
So, this PR proposes to change the way that run_pfs() searches for DH groups for TLSv1.2 and earlier. (Note that run_pfs() only checks for TLSv1.2 or earlier if the $EXPERIMENTAL flag is set to true.) First, it removes the test to see if the server will reject a ClientHello that only specifies TLS_DHE cipher suites if it includes a supported_groups extension that only specifies an unrecognized DH group. Instead, if the server supports TLS_DHE cipher suites (at TLSv1.2 or earlier) and the $EXPERIMENTAL flag is true, it will try to find out what group(s) the server uses. Second, it will report the group(s) found even if the server uses a group that does not come from RFC 7919.
The result is that if the server supports selecting groups from the supported_groups extension, it will print all of the groups that the server supports. If the server ignores the supported_groups extension and always uses the same group, it will print essentially the same information as is already printed by run_logjam().
One discrepancy, however, is that this code use pr_dh_quality() to determine how good a DH group is, based on the length of the prime, and pr_dh_quality() has differs from run_logjam() in terms of how it rates groups based on the lengths of their primes.
This PR is also based on #1139, but it addresses ECDH keys rather than DH keys. When run_pfs() prints the list of elliptic curves offered, it colors each curve according to its quality (based on key length). However, the severity level used when the list is sent to fileout() is always "INFO". This PR changes the call to fileout() to make the severity level be based on the quality of the shortest curve that the server offers.
For XMPP servers, when extracting the SRV-ID and XmppAddr names from the subjectAltName extension, need to take into account that the subjectAltName extension may be marked as critical, in which case there will be the DER encoding of TRUE (0101FF) between the DER encoding of the subjectAltName extension's OID (0603551D11) and the tag for OCTET STRING (04).
This PR is an attempt to address #1097. I have only been able to test it against jabber.topf.org and against locally created test certificates, so it needs more testing.
The main issue that this addresses is that testssl.sh currently checks against the wrong name for XMPP servers. According to RFC 6120, Section 13.7.2.1:
o The initiating entity sets the source domain of its reference
identifiers to the 'to' address it communicates in the initial
stream header; i.e., this is the identity it expects the receiving
entity to provide in a PKIX certificate.
So, if the --xmpphost option is provided, then the name provided in that option should be compared against the name in the certificate rather than the host name.
compare_server_name_to_cert() currently takes the server name to look for in the certificate as an parameter, but every call to compare_server_name_to_cert() uses $NODE as the argument. So, this PR removes the parameter and sets $servername to either $XMPP_HOST or $NODE as appropriate. This small change alone should fix the problem for most XMPP servers since the server's name SHOULD appear in the certificate encoded as a DNS name. That is the case for the one server I could test - jabber.topf.org.
The majority of the code in this PR is to address the other possibilities noted in RFC 6120, Section 13.7.1.2.1. This section notes that an XMPP server's identity name also appear in the subjectAltName extension as an otherName, either an SRV-ID or an XmppAddr identifier. Unfortunately, OpenSSL's certificate printer does not support otherName and just prints "othername:<unsupported>". So, this PR includes code to manually extract any SRV-ID or XmppAddr names from the certificate. This involves parsing the DER encoding of the certificate to look for the subjectAltName extension, looping through all of the names in the extension, and pulling out the names of these two name forms. This code is only run if the server is an XMPP server and the certificate does not have a matching DNS name. So, this code will rarely be executed.
This PR addresses one other issue. There is code in certificate_info() to set the variables $has_dns_sans and $has_dns_sans_nosni. These variables are needed to address the following requirement:
# Find out if the subjectAltName extension is present and contains
# a DNS name, since Section 6.3 of RFC 6125 says:
# Security Warning: A client MUST NOT seek a match for a reference
# identifier of CN-ID if the presented identifiers include a DNS-ID,
# SRV-ID, URI-ID, or any application-specific identifier types
# supported by the client.
While it is relatively easy to determine whether a certificate includes a DNS name in the subjectAltName extension, as noted above, it is not easy to determine whether it has an SRV-ID or an XmppAddr. So, this PR leverages the work compare_server_name_to_cert() does in parsing the subjectAltName extension by having compare_server_name_to_cert() set a global variable indicating whether the certificate has a subjectAltName extension with a relevant name form (DNS, SRV-ID, or XmppAddr for XMPP, or DNS for other servers). $has_dns_sans and $has_dns_sans_nosni are then just set to the value of this global variable.
This PR removes the changes that were added in #1113. As noted in the conversation for #1113, it was eventually determined that the actual bug was related to the first message fragment being too short rather than the overall length of the message. So, the warning message that is displayed by the code is misleading. In addition, if changes are made to avoid TCP fragmentation, then this code will no longer even test for intolerance to short message fragments.
This commit removes the '0a' character from public keys used in the key_share extension. New key pairs were created by repeatedly generating new keys until one was found that had no '0a' characters in the public key.
As noted in #1130, the current implementation of socksend_tls_clienthello() results in packets being fragmented wherever a '0a' character appears in the message. This cannot be avoided, but there are a few places where a '0a' character appears in which the character could easily be replaced:
* In the session_id for a TLSv1.3 ClientHello.
* In the 32-byte client random value
* In any public key sent in the key_share extension
This PR removes those uses of the '0a' character. While this does not do much to address the problem, it does result in a slight reduction in the amount of fragmentation of messages.
This PR is an attempt to fix the problem identified in #1118.
Currently, get_cipher() and get_protocol() attempt the extract the cipher and protocol from the SSL-Session information printed by OpenSSL s_client. This does not always work for TLSv1.3, however, since OpenSSL 1.1.1 will only print SSL-Session information for a TLSv1.3 connection if the server sends New Session Ticket. If the server doesn't, then get_cipher() and get_protocol() return empty strings.
For TLSv1.3 connections in which the server does not send a New Session Ticket, there seems to be only one other source for this information. A line of the form:
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
[Note that "New" would be "Reused" if the connection were created via session resumption.]
The use of this line seems to be okay for extracting the negotiated cipher, but it cannot be used in general to extract the negotiated protocol. The reason is that this line is created as follows:
c = SSL_get_current_cipher(s);
BIO_printf(bio, "%s, Cipher is %s\n",
SSL_CIPHER_get_version(c), SSL_CIPHER_get_name(c));
While the cipher that is printed seems to be the negotiated cipher, the protocol that is printed is "the SSL/TLS protocol version that first defined the cipher." Since TLS 1.3 ciphers may only be used with TLS 1.3, protocol version printed on this line may be accepted as the negotiated protocol if and only if it is "TLSv1.3."
This PR addresses the problem by modifying get_cipher() and get_protocol() to check the "New, ..., Cipher is ..." line if lines from SSL-Session ("Cipher : ...", "Protocol : ...") cannot be found. In the case of get_protocol() the protocol on the "New, ..., Cipher is ..." will be accepted only if the protocol is "TLSv1.3" and the cipher is a TLSv1.3 cipher.
This PR also adds a check for the "New, ..., Cipher is ..." to sclient_connect_successful(). If this line is present, and the protocol and cipher are not "(NONE)", then this is accepted as an indication that the connection was successful even if the "Master-Key" line does not appear. It is not clear whether this extra test is needed, however, as sclient_connect_successful() will not even look at the text in the output of OpenSSL s_client if function's return value is 0, and OpenSSL s_client should return 0 if the connection was successful.
Most of the curves that were defined for the supported_groups extension in RFC 4492 have been deprecated in RFC 8422 and RFC 8446. Appendix B.3.1.4 of RFC 8446 says that these deprecated values "are used in previous versions of TLS and MUST NOT be offered or negotiated by TLS 1.3 implementations."
According to a recent discussion on the TLS mail list (see, for example, https://www.ietf.org/mail-archive/web/tls/current/msg26974.html and https://www.ietf.org/mail-archive/web/tls/current/msg26980.html) a TLS 1.3 server implementation may choose to reject a TLS 1.3 ClientHello simply because the ClientHello offers one or more of the deprecated curves.
This PR address this issue by no longer offering the deprecated curves in TLS 1.3 ClientHello messages. This only affects run_pfs(), since socksend_tls_clienthello() already does not offer the deprecated curves in TLS 1.3 ClientHello messages.
The change in this PR has no affect on the testing of servers that do not support TLS 1.3. For those that do support TLS 1.3, only the 5 non-deprecated curves are tested with TLS 1.3, but all 30 curves are tested with TLS 1.2.
This PR reduces the number of public keys that are included in the key_share extension for a TLS 1.3 ClientHello.
When creating the key_share extension for a TLS 1.3 ClientHello, generate_key_share_extension() generally omits the public keys for larger finite-field groups (ffdhe3072, ffdhe4096, ffdhe6144, and ffdhe8192) so that the extension will not be overly large. However, the extension that it creates is still much larger than what is created by other software.
For a generic TLS 1.3 ClientHello, socksend_tls_clienthello() offers 7 groups in the supported_groups extension (P-256, P-384, P-521, X25519, X448, ffdhe2048, ffdhe3072) and 6 public keys in the key_share extension (P-256, P-384, P-521, X25519, X448, ffdhe2048). While the largest public key is omitted, this still creates a 665 byte key_share extension.
By contrast, Firefox offers 6 groups in the supported_groups extension (X25519, P-256, P-384, P-521, ffdhe2028, ffdhe3072), but only includes two public keys in the key_share extension (X25519, P-256). OpenSSL 1.1.1 offers 5 groups in the supported_groups extension (X25519, P-256, P-384, P-521, X448) and only includes one key in the key_share extension (X25519). Chrome offers 3 groups in the supported_groups extension (X25519, P-256, P-384) and only includes one key in the key_share extension (X25519).
Following the examples of OpenSSL, Firefox, and Chrome, this PR changes generate_key_share_extension() to include at most two public keys in the key_share extension. In general it will offer the public keys for the first two groups that appear in the supported_groups extension. However, it will still exclude the public key for any ffdhe group larger than ffdhe2048 unless that group appears first in the supported_groups extension.
In most cases this change will simply result in the ClientHello message being smaller. In some unusual cases, this change will force a second round-trip, with the server sending a HelloRetryRequest in order to ask for the key_share that it needs, but this will not affect the results of the testing.
In run_grease() there is a mismatch between the severity level of finds as printed and as sent to fileout(). Problems are labeled as medium when printing, but as CRITICAL in the call to fileout(). This PR fixes the problem by changing CRITICAL to MEDIUM.
As described in #1113, some servers will fail if the length of the ClientHello message is 522, 778, 1034, ... bytes (i.e., if length mod 256 = 10) or 526, 782, 1038, ... bytes (i.e., if length mod 256 = 14). This commit avoid this issue for normal testing by adding a 5-byte padding extension to the message if the length would otherwise be one of these lengths.
socksend_tls_clienthello() does not calculate the length of the ClientHello message in the case of a TLS 1.3 ClientHello, since it does not take into account the inclusion of a 32-byte session id. The length value that is being calculated incorrectly is only used to determine whether to include a padding extension, and if so, how long that extension should be.
This fix was previously included as part of PR #1120, since a correct length calculation is needed to avoid a ClientHello length such that length mod 256 = 10, but I removed it from that PR and am making it a separate PR, since it is a bug that should be fixed even if #1120 isn't adopted.
This commit updates the size bug GREASE test in a few ways:
* It removes the changes to socksend_tls_clienthello() - these will be submitted as a separate PR.
* It adds a test for a ClientHello message length of 266 bytes, but only if the server can generally handle messages with lengths between 256 and 511 bytes.
* It corrects the calculation of the length of the padding extension in cases in which a TLS 1 or TLS 1.1 ClientHello is being sent.
Just as some servers will fail if the length of the ClientHello is between 256 and 511 bytes (see RFC 7685), it seems that some servers (or a middlebox sitting in front of the servers) will fail if the length of the ClientHello is 522, 778, 1034, ... bytes in length (i.e., if length mod 256 = 10). I have also encountered one server that will also fail if the length of the ClientHello is 526, 782, 1038, ... bytes in length (i.e., if length mod 256 = 14).
In the case of that one server, the first ClientHello sent by run_pfs() was exactly 1038 bytes, and so run_pfs() was reporting that the server didn't support any PFS ciphers even though it did..
This PR addresses the problem in two ways. First, it modifies socksend_tls_clienthello() so that if the length of the ClientHello would be more than 511 bytes and length mod 256 would be 10 or 14, it adds a 5-byte padding extension in order to ensure that the final length of the ClientHello will not be a length that could trigger the bug.
Second, this PR adds a test to run_grease() to send ClientHello messages of the exact lengths that do trigger the bug so that users can be made aware that their servers have the problem.
RFC 8446 specifies the following for the list of certificates provided by the server:
The sender's certificate MUST come in the first
CertificateEntry in the list. Each following certificate SHOULD
directly certify the one immediately preceding it.
In RFC 5246 the "SHOULD" was a "MUST". This commit adds a check of whether the certificates provided by the server are in the correct order and issues a low severity warning if they are not.
This PR changes run_logjam() so that it does not warn about the use of 2048-bit DH primes, even if the selected prime is a common prime.
This PR leaves two issues unaddressed. First, it does not detect servers that are vulnerable to Attack IV in https://weakdh.org/logjam.html. These are servers that use DH primes that are of sufficient length, but that are poorly generated, and so are still vulnerable to attack.
Second, it does not address the potential problem that use of a common prime could leak information about what server product is being used, even if this information is not leaked through other means (e.g., HTTP headers). This should not be an issue with common primes from an RFC (2409, 3526, 5114, 7919), but would be an issue with product-specific common primes.
According to Section 7.4.2 of RFC 5246, when a server sends its certificate it MUST send a list in which the first certificate is the sender's certificate and "Each following certificate MUST directly certify the one preceding it." testssl.sh currently assumes that the server has populated the list way and so places the second certificate in the list into $TEMPDIR/hostcert_issuer.pem.
However, not all servers have been following this requirement, and so draft-ietf-tls-tls13 (soon to be RFC 8446) only says that servers SHOULD list the certificates in this way and says that clients "SHOULD be prepared to handle potentially extraneous certificates and arbitrary orderings from any TLS version, with the exception of the end-entity certificate which MUST be first."
testssl.sh needs to place the correct certificate in $TEMPDIR/hostcert_issuer.pem, since otherwise any OCSP request it sends will be incorrect, and any attempt to verify and OCSP response will be incorrect as well.
This PR changes extract_certificates() and parse_tls_serverhello() to populate $TEMPDIR/hostcert_issuer.pem with the first certificate in certificate_list that has a subject DN that matches the issuer DN in the server's certificate, rather than simply populating $TEMPDIR/hostcert_issuer.pem with the second certificate in the list.
In testing a random sampling of U.S. government servers, of 57 servers tested 5 reported "unauthorized" for the OCSP URI using the current testssl.sh and all 5 of these reported "not revoked" with this PR. This PR also corrects the same issue in some servers on the Alexa Top 1000, but this was only a problem for 12 of those 1000 servers.
In cases in which the server offers a stapled OCSP response, this commit extracts the OCSP response and then checks the response for the status of the server's certificate. The check is performed in the same way as when the certificate includes an OCSP URI and the "--phone-out" option is set, except that the OCSP response is received from the TLS server rather than coming directly from the OCSP responder. Since this only involves additional processing of data that testssl.sh is already receiving, the check is performed whether or not the "--phone-out" flag is set.
This PR fixes#615 for the case in which tls_sockets() is used by splitting the list of CBC ciphers into two lists, each with fewer than 128 ciphers and then testing each list separately.
For the --ssl-native case, no changes were needed. Even though $cbc_ciphers contains 154 ciphers, no version of OpenSSL supports all of these ciphers, and so the actual ClientHello sent by every version of OpenSSL contains fewer than 128 ciphers.
I did, however, add the -no_ssl2 flag to the "$OPENSSL s_client" command to prevent OpenSSL from sending an SSLv2-compatible ClientHello. As is noted in a comment in run_server_preference(), "the supplied openssl will send an SSLv2 ClientHello if $SNI is empty and the -no_ssl2 isn't provided."
This PR fixes problems with check_revocation_crl() sometimes reporting that a certificate is revoked even when it isn't, and with check_revocation_ocsp() sometimes reporting "error querying OCSP responder" even if the OCSP responder provided a good response. The most common reason for this to happen is that OpenSSL cannot validate the server's certificate (even without status checking). PR #1051 attempted to get status checking to work even in cases in which the server's certificate could not be validated. This PR instead addresses the problem by not checking status if determine_trust() was unable to validate the server's certificate.
In some cases the server's certificate can be validated using some, but not all of the bundles of trusted certificates. For example, I have encountered some sites that can be validated using the Microsoft and Apple bundles, but not the Linux or Mozilla bundles.
This PR introduces GOOD_CA_BUNDLE to store a bundle that could be used to successfully validate the server's certificate. If there is no such bundle, then neither check_revocation_crl() nor check_revocation_ocsp() is run. When check_revocation_crl() and check_revocation_ocsp() are called, the status checks within them closely match the validation check in determine_trust(), which helps to ensure that if the check fails it is because of the status information.
As noted in #1057, at least one CA provides incorrect information when the CRL is downloaded, so validation could fail for a reason other than the certificate being revoked. So, this PR adds a check of the reason that validation failed and only reports "revoked" if the validation failed for that reason.
As noted in #1056, it is not possible to perform an OCSP query without access to the certificate issuer's public key. So, with this PR check_revocation_ocsp() is only called if the server's provided at least one intermediate certificate (i.e., the issuer's certificate, which contains the issuer's public key).
This PR improves the handling of error responses when checking status using OCSP. It can handle a few types of errors:
* When the responder just returns an error (e.g., "Responder error: unauthorized").
* When the response cannot be verified (e.g., invalid signature, expired certificate).
* When the response is valid ("Response verify OK"), but there is a problem with the response for the individual certificate (e.g., information is too old, or status is "unknown").
This PR fixes two issues with OCSP checking. First, the syntax for specifying a host header changed in OpenSSL between versions 1.0.2 and 1.1.0. With OpenSSL 1.0.2-chacha, 1.0.2o, and LibreSSL the syntax needs to be "-header HOST <hostname>". With OpenSSL 1.1.0h and 1.1.1 the syntax needs to be "-header HOST=<hostname>". I have not been able to test other versions of OpenSSL 1.0.2 or 1.1.0, but am assuming that all versions of OpenSSL 1.1.0 use the same syntax as 1.1.0h.
This PR also fixes a typo in the case of an error, which was causing $code to be set to "empty ocsp response" if the response was not empty rather than if it was empty.
The current code for extracting the validity dates for certificates assumes that the strings "Not Before" and "Not After" will appear exactly once in the pretty-print of the certificate. In most cases that works. However, there are a few server certificates that include the private key usage period extension, which also includes "Not Before" and "Not After" times. The result is that the current code does not correctly extract the start date and end date from any certificates that have private key usage period extensions.
This PR fixes the problem and also speeds up extraction of the dates by only using Bash internal functions.
The pretty-print of a certificate begins as follows:
Certificate:
Data:
Version: 3 (0x2)
Serial Number: ...
Signature Algorithm: ...
Issuer: ...
Validity
Not Before: ... GMT
Not After : ... GMT
...
The code in this PR extracts the start date by first removing from the certificate everything that comes before "Not Before: ". It looks for the shortest string that includes ""Not Before: " in order to ensure it is not getting the date from the private key usage period extension. After that, the longest string that begins with "GMT" is removed so that only the notBefore date remains.
The part that removes the string up to "Not Before: " actually looks for the first instance of "Not Before: " that comes after the "Validity". This is to protect against the unlikely possibility that the string "Not Before: " appears somewhere in the issuer's name.
The extraction of the notAfter date works similarly. It first looks for the first instance of "Not After :" that appears after both "Validity" and "Not Before: " and then takes the date string that appears immediately afterwards, with the assumption that the date string ends in "GMT".
In some cases the OCSP URI contains multiple components in the path (e.g., http://www.example.com/OCSP/myOCSPresponder).
This PR changes check_revocation_ocsp() to remove all components in the path, rather than just the final component, when extracting the host name from the URI for the host header.
David Cooper