The commit adds support for RFC 8998 and draft-yang-tls-hybrid-sm2-mlkem. This includes support for the TLS_SM4_GCM_SM3 and TLS_SM4_CCM_SM3 cipher suites, the key exchange groups curveSM2 and curveSM2MLKEM768, and SM2 public keys and signatures.
While this commit adds support to tls_sockets() to decrypt server responses encrypted under SM4 GCM or CCM, OpenSSL does not support performing key derivation using curveSM2. So, tls_sockets() can not decrypt server responses if the key exchange was performed using curveSM2 or curveSM2MLKEM768.
This commit addresses two issues created by changes in certificate printing in OpenSSL 4 (based on testing with OpenSSL 4.0.0-alpha1).
With OpenSSL 4, the public key type for ML-DSA keys is now shown with a string (e.g., ML-DSA-44) rather than an OID. The first change in this commit ensures that the public key size is set correctly in this case.
Second, different information is printed about the size of elliptic curve public keys. All previous versions of OpenSSL (and LibreSSL) just provided the size of the public key:
Public-Key: (256 bit)
OpenSSL 4.0.0-alpha includes additional information:
Public-Key: (256 bit field, 128 bit security level)
The second change in this commit removes this additional information.
This commit fixes#2959 by modifying TLS12_CIPHER, TLS12_CIPHER_2ND_TRY, and TLS12_CIPHER_3RD_TRY so that they each have 118 ciphers (including "00,ff"). It also modifies run_cipherlists(), run_server_defaults(), and run_beast() so that, when $SERVER_SIZE_LIMIT_BUG is true, no more than 125 ciphers are sent.
This commit fixes#2896. This commit avoids modifying the ADDTL_CA_FILES environment variable, and instead substitutes spaces for commas whenever the variable is used.
This commit fixes#2502 in the 3.2 branch by checking that the key_share extension is at least 4 bytes long (8 in ASCII-HEX). These 4 bytes encode the group value (2 bytes) and the length of the key (2 bytes).
With OpenSSL 1.1.0 (and maybe other versions), the `ciphers` function lists many cipher suites that are not actually supported by the `s_client` option. This PR fixes that by using the `-s` option whenever `$OPENSSL ciphers` is used to obtain a list of cipher suites supported by OpenSSL. According to https://www.openssl.org/docs/manmaster/man1/ciphers.html:
```
-s
Only list supported ciphers: those consistent with the security level, and minimum and
maximum protocol version. This is closer to the actual cipher list an application will
support.
```
When the `-s` option is used along with `-tls1`, OpenSSL 1.1.0 will not list any ciphers that only work with TLSv1.2. So, `prepare_debug()` needed to be changed to correctly populate `ossl_supported_tls`, which is supposed to be a list of all non-SSLv2 ciphers supported by the server.
LibreSSL issues an "unknown option" error if the `-s` option is provided, so the `-s` option is only included in the command line if `$OPENSSL` has been determined to support it.
This PR is needed so that `prepare_debug()` can correctly determine which cipher suites are or are not supported by `$OPENSSL`.
This PR removes an extra call to `$OPENSSL s_client` in `get_server_certificates()` and it also changes `get_server_certificates()` to not collect extensions when SNI is not being provided.
At the beginning of run_server_preference(), if the attempt to connect to the server is unsuccessful, a message is printed listing all of the ciphers in $list_fwd and $tls13_list_fwd:
no matching cipher in this list found (pls report this): DES-CBC3-SHA:RC4-MD5:DES-CBC-SHA:RC4-SHA:AES128-SHA:AES128-SHA256:AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:ECDH-RSA-DES-CBC3-SHA:ECDH-RSA-AES128-SHA:ECDH-RSA-AES256-SHA:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:DHE-DSS-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:AES256-SHA256:ECDHE-RSA-DES-CBC3-SHA:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-SHA256:ADH-AES256-GCM-SHA384:AECDH-AES128-SHA:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-AES128-SHA:TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256
This message can be misleading. I tested a server that only supported TLSv1.3 using the provided OpenSSL 1.0.2-chacha. The server supported TLS_AES_256_GCM_SHA384, but OpenSSL didn't. However, the message implies that the server does not support TLS_AES_256_GCM_SHA384.
This PR changes the message (and the one included in CSV/JSON output) to only list those ciphers in $list_fwd and $tls13_list_fwd that are actually supported by $OPENSSL.
Note that even with this PR, some ciphers are listed that aren't really supported by $OPENSSL, since the `-s` option isn't used. But, that is #663.
https://github.com/tlswg/tls13-spec/wiki/implementations now lists a server that supports TLS 1.3 draft 28, so this PR adds supports for drafts 27 and 28.
Since run_protocols() now checks for 11 different drafts of TLS 1.3 in addition to the final version, performing a separate test for each draft had become far too time consuming. So, this PR rewrites the check for TLS 1.3 versions in run_protocols() so that the number of tests is proportional to the number of drafts that the server supports rather than the number of drafts that testssl.sh can check for.
There is at least one server that will fail under some circumstances if the ClientHello offers a compression method other than null.
In OpenSSL 1.1.0 and 1.1.1, s_client will not offer any other compression methods unless the "-comp" option is provided. However, in earlier versions of OpenSSL, s_client will by default offer the DEFLATE compression method, however, this can be disabled using the "-no_comp" option.
This PR addresses the flaw in this server by having s_client_options() add a "-no_comp" option to the command line if "-no_comp" is supported and the test doesn't require offering compression.
Since run_crime() requires compression to be offered, run_crime() was changed to always add "-comp" to the command line, and then s_client_options() was changed to remove "-comp" from the command line, if that option isn't supported.
At the moment the code for downloading a CRL seems to only work if URL is an HTTP or HTTP URL. It fails if the URL is an LDAP URL. The wget command does not support LDAP and when curl retrieves data from an LDAP URL it stores the result in LDIF format, which http_get() cannot currently convert into a PEM-encoded CRL.
This PR addresses the issue by skipping the revocation check for any URL that does not begin with "http".
In general, a CA only needs to keep the status information for a certificate until it expires. So, once a certificate has expired, the information provided about it in a CRL or OCSP response may no longer be reliable. The certificate may no longer be listed as revoked, even it is had been revoked at some point before it expired.
So, this PR changes certificate_info() to only check CRLs for revocation status if the certificate has not expired.
This PR was developed in response to #845. It adds to the list of ciphers used to determine whether the server has a cipher order in order to help avoid cases in which testssl.sh cannot determine a cipher order.
In order to create this list I scanned thousands of servers in order to determine what ciphers they support, including (1) about 20 thousand U.S. government web sites, (2) all of the sites listed at badssl.com, (3) all of the test servers listed at https://github.com/tlswg/tls13-spec/wiki/implementations, (4) about 30 additional non-U.S. government sites, and (5) one server configured as described in #845. I scanned each of these servers using OpenSSL 1.0.2-chacha, 1.0.2o, and 1.1.1.
Then I ran collection information through a script that created the updated list. For each scanned server, and for each of the 3 versions of OpenSSL, the script checked whether $list_fwd contained at least two ciphers from the list. If it didn't, then it would add one of the ciphers supported by the server (and by OpenSSL) to the list. In choosing among the ciphers supported by the server that were not already in $list_fwd, it would choose the cipher that was supported by the most other servers.
The list contain a few oddities as a result of the servers that I scanned. The script added two TLSv1.3 ciphers, since I scanned at least one server that only supports TLSv1.3. The list also includes ADH-AES256-GCM-SHA384 and AECDH-AES128-SHA, which may only be supported by null.badssl.com.
I made one manual change to the list - adding TLS_CHACHA20_POLY1305_SHA256. I did this since the number of TLSv1.3 servers scanned was so small, I didn't think it was safe to assume that all servers that support TLSv1.3 would support both TLS_AES_256_GCM_SHA384 and TLS_AES_128_GCM_SHA256.
Since most of the servers that I scanned were U.S. government servers, it may not be a representative sample. However, since the new list only adds to the current list, it can only be an improvement. Also, the updated list still only includes 37 ciphers, so many more could be added without creating any problems.
There is currently a problem if mass testing is being performed, JSON and/or CSV output is to be produced, the parent process calls fileout(), and each child process have its own output file for the JSON and/or CSV output. The can be seen, for example, with the following:
testssl.sh --openssl=openssl_1.1.1 --file test_servers.txt --csvfile output_dir --jsonfile output_dir
A call will be made in the parent process to report that openssl_1.1.1 has "No engine or GOST support via engine." fileout() will try to write to output_dir, which will result in an error.
This PR fixes the problem by checking the the file to be written to is not a directory (as is already done in html_out() for HTML output).
https://api.dev.ssllabs.com/api/v3/getClients incorrectly indicates a highestProtocol of 771 (TLS 1.2) for clients that support TLS 1.3, which leads run_client_simulation() to incorrectly report "no connection" if the client would have actually connected using TLS 1.3.
This has been addressed by manually editing etc/client-simulation.txt to set the highest_protocol to 0x0304 for the clients that support TLS 1.3.
This PR modifies update_client_sim_data.pl to automatically apply the fix for clients that support TLS 1.3 in order to avoid a possible regression when etc/client-simulation.txt is updated.
When performing client simulations in "--ssl-native" mode, provide the client's list of supported curves to "$OPENSSL s_client" in order to make the results even more accurate.
This PR improves client simulation in "--ssl-native" mode:
* It changes ${protos[i]} to list the protocols that should be disabled rather than those that should be enabled, except in the case that the client only supports one protocol.
* It sets the values for ${tlsvers[i]}, which is used in run_client_simulation(), but was not defined.
* It adds a new variable, ${ciphersuites[i]}, that lists the TLSv1.3 cipher suites supported by a client.
Client simulation still produces false results in "--ssl-native" mode, but the results are better than before.
This PR fixes three issues related to the testing for RFC 7919 DH groups in run_pfs():
* If the RFC 7919 DH groups are supported for both TLSv1.3 cipher suites and non-TLSv1.3 cipher suites, then the list of supported groups is printed twice.
* The finding that is used for CSV/JSON files includes the word "offered" after the list of groups, which is inconsistent with other findings.
* Since the $ffdhe_offered is only used to determine whether to test for use of RFC 7919 DH groups with non-TLSv1.3 ciphers, this flag should only be set if a non-TLSv1.3 ciphers that uses ephemeral DH is found.
If OpenSSL reports an error, sclient_connect_successful() may incorrectly interpret it as a connectivity problem, leading testssl.sh to stop testing before it has completed.
When not using "--ssl-native" mode, this happens if $OPENSSL does not support SSLv3, as both get_server_certificate() and run_beast() will attempt to connect using SSLv3 even if $OPENSSL does not support it.
When using "--ssl-native" mode, this happens in multiple places if $OPENSSL does not support the protocol being used or if $OPENSSL does not support any of the ciphers that are specified in the command line.
This PR fixes the above problems by adding checks for protocol support or for support for at least one cipher before calling $OPENSSL.
run_client_simulation() also has a problem in "--ssl-native" mode of calling $OPENSSL with parameters that cause $OPENSSL to report an error, but this is already addressed by temporarily setting MAX_OSSL_FAIL to 100 during client simulation tests and then, after client simulation testing is complete, returning $MAX_OSSL_FAIL and $NR_OSSL_FAIL to the values they had before client simulation testing began.
parse_tls_serverhello() checks $TLS_CLIENT_HELLO for a supported_versions extension, and if it contains one, checks that the negotiated version is listed in that extension. However, while $TLS_CLIENT_HELLO is always set in socksend_tls_clienthello() it is not set by client_simulation_sockets() (or any of the functions that client_simulation_sockets() calls). As a result, when the server's response to a client simulation is parsed, parse_tls_serverhello() may compare the negotiated version against the supported_versions extension from a ClientHello message from a previous test.
This PR fixes the problem by having client_simulation_sockets() set $TLS_CLIENT_HELLO.
This PR changes find_openssl_binary() so that $OSSL_VER_APPENDIX contains everything from $OSSL_VER that appears after $OSSL_VER_MAJOR.$OSSL_VER_MINOR.
This PR fixes the issue raised in #1013. It primarily does this in two ways:
* In calls to `$OPENSSL s_client` that specify ciphers, the TLSv1.3 ciphers are provided separately using the `-ciphersuites` option. Then, the `s_client_options()` function manipulates the command-line options as necessary based on the version of OpenSSL being used.
* Calls to `$OPENSSL ciphers` were replaced with calls to `actually_supported_ciphers()`, which calls `$OPENSSL ciphers`. `actually_supported_ciphers()` modifies the parameters for the call to `$OPENSSL ciphers` as necessary, based on the version of OpenSSL being used.
When using testssl.sh with the current development branch of OpenSSL 1.1.1, determine_trust() incorrectly reports that "/openssl <= 1.0.2 might be too unreliable to determine trust." The problem is that find_openssl_binary() is not correctly extracting the version information. The version is "1.1.1-pre3-dev," but find_openssl_binary() extracts:
OSSL_VER_MAJOR: 1
OSSL_VER_MINOR: 1.13
OSSL_VER_APPENDIX: -pre-dev
This PR fixes the problem and also eliminates the use of external functions in extracting version information for $OSSL_VER.
Note that this code makes a change from the current code. Currently, $OSSL_VER_APPENDIX is intended to contain anything from $OSSL_VER that comes after $OSSL_VER_MAJOR.$OSSL_VER_MINOR. For example, if $OSSL_VER is 1.1.0-dev, then $OSSL_VER_APPENDIX is "-dev". In this PR, the "-dev" is dropped and so does not appear in $OSSL_VER_MAJOR, $OSSL_VER_MINOR, or $OSSL_VER_APPENDIX. The reason for this is that testssl.sh is only using $OSSL_VER_APPENDIX in cases in which $OSSL_VER_MAJOR.$OSSL_VER_MINOR is 0.9.8 to determine whether 0.9.8a - 0.9.8l or 0.9.8m - 0.9.8z. So, it seems that testssl.sh isn't interested in things such as "-dev" or "-pre3-dev".
If desired, this PR could be changed to that $OSSL_VER_APPENDIX contains everything in $OSSL_VER that appears after $OSSL_VER_MAJOR.$OSSL_VER_MINOR.
This PR fixes#1015 by adding underscore ('_') to the list of characters that may appear in a domain name label. https://github.com/drwetter/testssl.sh/commit/a178f3e1838f0698e193337e10f0c32eb8c5f34a already addressed this for the check of whether the Common Name field in a certificate is a DNS name. This PR fixes it for checks of whether a domain name is a wildcard name and whether the server's host name is a wildcard match against a name in a certificate.
This PR adds a missing check for local TLSv1.3 support in cipher_pref_check(). If the "--ssl-native" mode is being used and $OPENSSL does not support TLSv1.3, then a warning should be displayed indicating that TLSv1.3 cannot be checked rather than displaying the same results as if the server did not support TLSv1.3.
Drafts 25 and 26 of TLS 1.3 were issued in early March. Neither made any change that is relevant to testssl.sh.
This PR adds checks for these two drafts.
The implementation of read_sigalg_from_file() was changed on January 29 in https://github.com/drwetter/testssl.sh/commit/88cd5528e72a923b26386666d5e76b69849f387b. The new implementation does not work correctly in cases in which read_sigalg_from_file() is called with $TMPFILE as as parameter.
The current implementation of the function is:
```
read_sigalg_from_file() {
local hostcert_txt="${1//pem/txt}"
[[ -r "$hostcert_txt" ]] || $OPENSSL x509 -noout -text -in "$1" 2>/dev/null >$hostcert_txt
awk -F':' '/Signature Algorithm/ { print $2; exit; }' $hostcert_txt
}
```
When called using $TMPFILE (/tmp/testssl.XXXXXX/tempfile.txt), hostcert_txt is set to $TMPFILE, and since this file exists and is readable, the next line does nothing and the final line tries to read the signature algorithm from $TMPFILE rather than from a parsed version of the certificate.
This PR reverts read_sigalg_from_file() to its previous implementation, at least as a temporary solution.
Support for X448 was recently added to the development branch of OpenSSL 1.1.1. This PR adds an X448 key pair to etc/tls_data.txt (that was generated using OpenSSL 1.1.1) and adds X448 to the supported_groups extension for TLS 1.3 ClientHello messages.
This commit adds support for TLSv1.3 draft 24. The only change between draft 23 and draft 24 is that draft 24 requires the version number of the record layer to be "0303" for a second ClientHello, whereas draft 23 allowed the version number for both the initial and second ClientHello to be "0301".
The current text reads:
legacy_record_version This value MUST be set to 0x0303 for all
records generated by a TLS 1.3 implementation other than an
initial ClientHello (i.e., one not generated after a
HelloRetryRequest), where it MAY also be 0x0301 for compatibility
purposes.
If extra extensions are provided to socksend_tls_clienthello(), the socksend_tls_clienthello() needs to determine what extensions were provided so that it doesn't add any of these extensions a second time. The code that was looping through the extra extensions to get the extension IDs was using the value for the length of the string. This commit fixes the error.
Some servers will respond with an alert to a ClientHello that does not indicate support for secure renegotiation, which may be signaled through either an extension or the 0x00,0xff "cipher suite." In some cases testssl.sh calls tls_sockets() without including "00,ff" in the list of cipher suites, which results in some servers rejecting a ClientHello that would otherwise result in a successful connection.
This PR fixes the problem by adding "00,ff" to any ClientHello where it was previously missing, with one exception. If a TLSv1.3 ClientHello is being sent and only TLSv1.3 ciphers are listed, then the "00,ff" cipher suite is not added.
This commit fixes#983 by ensuring that the line printed just before calling "return" prints a newline character. It also fixes the problem that no output is sent to the JSON/CSV file in some cases in which no fallback is possible since the server does not support two different protocols below TLSv1.3.
This commit fixes#981 by using a while loop instead of a for loop to check each DNS name in the SAN extension, copying the syntax used in certificate_info() to display all of the SANs.
This commit adds code to run_server_defaults() so that in debug mode all of the server's certificates are saved in $TMPDIR in both PEM-encoded format and pretty-print text format.
testssl.sh was recently changed to store the text printout of the host's certificate in a file, $HOSTCERT_TXT, and then use this file in some places rather than calling "$OPENSSL x509 -in $HOSTCERT -text -noout". There was a problem, however, in cases in which the server had more than one certificate (including cases in which the server returned an unrelated certificate when sent a ClientHello w/o SNI), since the contents of $HOSTCERT_TXT was not always being updated whenever $HOSTCERT changed.
This commit fixes that problem by replacing the previous solution with a less ambitious one. In this version, the global variable is eliminated and instead run_server_defaults() stores a text version of each certificate it finds in an array. This value is then passed to certificate_transparency() and certificate_info() for use. It is also passed from certificate_info() to must_staple().
determine_trust() uses the output of "$OPENSSL verify" to determine whether OpenSSL can construct a valid certification path for the server's certificate. If it does not find a string of the form "error [1-9][0-9]? at [0-9]+ depth lookup:" in the output, then it assumes that validation was successful. In current versions of OpenSSL, when this error is created it is printed to stdout, but in OpenSSL 1.1.1 is it printed to stderr. Since testssl.sh only checks the output sent to stdout, it incorrectly treats all certificates as valid if OpenSSL 1.1.1 is used.
This commit fixes the problem by checking the text that is sent to both stdout and stderr.
This commit also fixes a typo in the call to "$OPENSSL verify" which resulted in the environment variables SSL_CERT_DIR and SSL_CERT_FILE not being set to "/dev/null".
In the output created by certificate_info(), the "Server key size" line labels an elliptic curve key as "ECDSA." This commit changes the label to "EC." I believe this a more correct label since ECDSA is a signature algorithm, not a key type. Also, while unlikely, an elliptic curve key in a certificate may be used for ECDH (e.g, in TLS_ECDH_RSA_WITH_AES_128_CBC_SHA) rather than ECDSA.
Note that this does not impact the JSON or CSV output, since the corresponding fileout command already uses "$cert_keysize EC bits"
This commit adds TLSv1.3 support for run_server_defaults(). It uses get_server_certificate() to run separate checks for RSA and ECDSA certificates by sending TLSv1.3 ClientHello messages with different signature_algorithms values. A similar change is made to certificate_transparency() in order to check for certificate transparency support for certificates returned over TLSv1.3. It also modifies the check for certificates offered without SNI by including an option to use tls_sockets() with servers that support TLSv1.3 if OpenSSL does not support TLSv1.3.
In TLSv1.3, if a HelloRetryRequest needs to be sent and the call to tls_sockets() includes additional request extensions (other than key_share or cookie), then resend_if_hello_retry_request() adds an extra comma between extensions in the value sent to socksend_tls_clienthello(), which creates errors. This commit fixes the problem by removing the extra comma.
This PR defines an extract_certificates() function in order to remove some redundant code from get_server_certificate(). Currently, nearly identical code appears in two places in get_server_certificate() to extract certificates from the output of `$OPENSSL sclient`, in one place for SSLv2 responses and in another for SSLv3 through TLSv1.2. The code to get the certificates used with TLSv1.3 (see https://github.com/dcooper16/testssl.sh/tree/extended_tls_sockets) would have added a third place where this same code would be needed. This PR allows the code to be written once and used in all three places.
This commit prints the contents of the keyUsage and extended key usage extensions in certificates and checks the public keys in the certificates are not being used in a manner that is inconsistent with these extensions.
This PR is intended to improve the functionality of run_tls_fallback_scsv().
The original goal of this PR was to address servers that support TLSv1.3 when using OpenSSL 1.1.1. That does not seem to be an issue, as using `$OPENSSL s_client` with the `-no_tls1_2` flag results in a TLSv1.1 ClientHello, even if `$OPENSSL` supports TLSv1.3. However, if the server supports TLSv1.3, then a message that says "No fallback possible, TLS 1.2 is the only protocol (OK)" isn't entirely correct.
The main issue this PR fixes is some false positives in servers that do not support TLSv1.2. On a few servers that I tested, the current code incorrectly reports "Downgrade attack prevention NOT supported." Some of the servers only support TLSv1, so it should report that fallback is not possible. Another server supports TLSv1.1 and TLSv1, and it supports fallback protection. In both cases, the current code produces a false positive, since it assumes that TLSv1.2 is supported.
In three different places there is a line that is supposed to check whether the list of ciphers to be tested contains any TLSv1.3 ciphers. This check currently fails if there is only one TLSv1.3 cipher in the list and it is the first cipher in the list. This commit fixes the problem.
This commit adds support for draft 23, which contains 2 changes that are relevant for testssl.sh. It adds a few new values for the signature_algorithms extension and it changes to extension number for the key_share extension from 40 to 51.
With the change in the extension number, it is no longer possible to send a single ClientHello that works for all supported drafts of TLSv1.3. (I tried sending a ClientHello with two key share extensions, 40 and 51, but that didn't work.) So, this commit adds a test to determine_optimal_proto() to determine whether TLSv1.3 is supported and if so whether draft 23 is supported or only some earlier draft (18-22). In subsequent tests, the ClientHello uses the appropriate number for the key share (40 or 51) and specifies the appropriate draft version(s) in the supported_versions extension (either 23 or 18-22). In the case of run_protocols() the test for each draft version uses the appropriate key share extension number so that servers that support both draft 23 and an earlier draft can be detected.
This PR is similar to #944. If using OpenSSL 1.1.1 to connect to a server that supports TLSv1.3, `run_crime()` will connect to the server using TLSv1.3, which does not support TLS-level compression. So, the server will be reported as "not vulnerable" even if would use compression for connections at TLSv1.2 and below.
I have not encountered any "live" servers that support both TLSv1.3 and TLS-level compression. I verified this problem by using OpenSSL 1.1.1 to create a server that supports both TLSv1.3 and TLS-level compression:
```
openssl111 s_server -cert cert.pem -key key.pem -accept 8443 -WWW -comp
```
I then tested the server using `testssl.sh --crime` with both openssl111 and OpenSSL 1.0.2-chacha.
run_renego() appears to produce a false positive if OpenSSL 1.1.1 is used and the server being tested supports TLSv1.3 (i.e., the server supports the same draft version of TLSv1.3 as the version of OpenSSL 1.1.1 being used does). This PR fixes the problem by telling calls to $OPENSSL s_client in run_renego() to not use TLSv1.3.
In TLSv1.3 servers may send a supported_groups extension, which "SHOULD contain all groups the server supports, regardless of whether they are currently supported by the client."
This PR extracts the contents of the supported_groups extension, if `parse_tls_serverhello()` is to process "all" of the server's response. The contents of the extension are also displayed on the terminal if $DEBUG -ge 3.
In TLSv1.2 and below, servers respond to a status_request extension (a request for a stapled OCSP response) by returning an empty status_request extension and then including a CertificateStatus message, which follows the Certificate message. In TLSv1.3 the CertificateStatus response is included as the value of the status_request extension, which now appears as an extension within the Certificate message.
This PR extracts the contents of the status_request extension sent by the server so that it can later be processed in the same way as if it had sent in a TLSv1.2 or below response.
This PR adds code to decrypt the encrypted portion of the server's response for TLSv1.3 and to then process any certificates and encrypted extensions. This code supports all 5 TLSv1.3 cipher suites, and so any response can be decrypted as long as the session key can be derived (which requires OpenSSL to support the ephemeral key that was used - see #938).
For the symmetric decryption, the sym-decrypt() function uses OpenSSL when possible and internal Bash functions when needed.
For AES-GCM and AES-CCM ciphers sym-decrypt() normally uses internal Bash functions, which rely on using "$OPENSSL enc" in AES-ECB mode to generate the key stream and then Bash functionality to XOR the key stream with the ciphertext. With some version of OpenSSL the AES-GCM ciphers are decrypted using "$OPENSSL enc" in AES-GCM mode directly. On my system, however, both methods seem to work about equally fast.
For ChaCha20 ciphers, "$OPENSSL enc -chacha20" is used, if supported (OpenSSL 1.1.x only). and Bash internal functions (without any OpenSSL support) are used otherwise. In this case, if the Bash internal functions need to be used, decryption is very, very, very slow. Fortunately, in a typical run of testssl.sh there won't be many cases in which the connection will be TLSv1.3 with ChaCha20 and the entire response needs to be processed (requiring decryption). In most cases, even if the connection is TLSv1.3 with ChaCha20, will at most need the ephemeral key, which is available in plain text.
This is the first in a series of PRs to add support for processing the encrypted portions of the server's response in a TLSv1.3 handshake.
This PR adds the code to derive the handshake traffic key needed to decrypt the response (the next PR will add the code to perform the symmetric-key decryption of the encrypted portions of the response).
Since this PR does not make use of the traffic key that it derives, it doesn't yet add any new functionality.
Note that testssl.sh will not always be able to derive the session keys. If the version of OpenSSL that is bundled with testssl.sh is used and the server chooses to use an X25519 ephemeral key, OpenSSL will be unable to perform the shared secret in derive-handshake-traffic-secret(). (OpenSSL 1.1.0 supports X25519.) Since X25519 use a different encoding than ECDH keys, the lack of X25519 support will be discovered in parse_tls_serverhello() when $OPENSSL pkey is unable to convert the key from DER to PEM. So, in debugging mode, parse_tls_serverhello() now displays a warning if it receives a key share that $OPENSSL pkey cannot handle.
get_server_certificate() uses an awk script to extract the certificates from the output of OPENSSL s_client and it then uses the following line to determine how many certificates were found:
nrsaved=$(count_words "$(echo level?.crt 2>/dev/null)")
If $nrsaved is 0, then get_server_certificate() returns 1 (indicating failure); otherwise it returns 0 (indicating success).
However, the check for the number of certificates returned doesn't work if no certificates were found, as nrsaved will be set to 1 if no certificates were found:
> touch level0.crt
> echo level?.crt
level0.crt
> touch level1.crt
> echo level?.crt
level0.crt level1.crt
> rm level0.crt level1.crt
> echo level?.crt
level?.crt
This PR fixes the problem by first checking that level0.crt exists (-s is used instead of -e, since an empty file wouldn't have a certificate).
This PR adds an additional COLOR level (3). If color is set to 3 then all ciphers are printed according to pr_cipher_quality() rather than just the "Negotiated cipher" in run_server_preference().
This PR reduces the amount of work parse_tls_serverhello() does when processing alert messages when not in debug mode. It delays writing anything to $TMPFILE unless $DEGUG -ge 1 until it has reason to believe that the response was successful. If $DEBUG is 0 and alert messages are sent, then no file operations are performed processing the alert messages.
In almost every case, there is no attempt to look at the contents of $TEMPDIR/$NODEIP.parse_tls_serverhello.txt unless the connection was successful. So, in most cases, it is okay to not call tmpfile_handle() in parse_tls_serverhello() unless the connection was successful. There is, however. one place in run_grease() where the code reads the contents of $TEMPDIR/$NODEIP.parse_tls_serverhello.txt even if the connection was not successful. In order to address this, the DEBUG level is temporarily set to 1 when performing this test if its value is 0. Also in order to address this, changes were made in parse_tls_serverhello() to ensure that "tmpfile_handle $FUNCNAME.txt" is always called before returning if $DEBUG -ge 1.
If certificate-based client authentication is required by the server, then most HTTP-related checks are skipped, even if the "--assume-http" flag is used. If $CLIENT_AUTH is true, then $ASSUME_HTTP is ignored.
In some cases the checks are appropriately skipped, since the tests cannot be performed. In other places, the value of "$CLIENT_AUTH" is used as a hint as to whether HTTP is being used. For example, in run_tickbleed:
if [[ "$SERVICE" != HTTP ]] && ! "$CLIENT_AUTH"; then
outln "-- (applicable only for HTTPS)"
fileout "ticketbleed" "INFO" "Ticketbleed: not applicable, not HTTP" "$cve" "$cwe"
return 0
fi
There are some places, however, where tests are just skipped, even if both $CLIENT_AUTH and $ASSUME_HTTP are true, even though the test could be performed. For example, run_client_simulation() only simulates generic clients in this case.
This PR attempts to address this:
* In run_client_simulation() it runs all of the tests if $ASSUME_HTTP is true.
* In certificate_transparency() it only says that the lack of CT information is "N/A" it can verify that HTTP is not being used (if $SERVICE is not HTTP and $CLIENT_AUTH is false). Otherwise it just says "no" without flagging it as an issue.
* In certificate_info() it displays additional warnings (about use of SHA-1 or subjectAltName matching) only if it can verify that HTTP is being used ($SERVICE is HTTP or $ASSUME_HTTP is true).
* In run_crime(), if compression is used, it only says " but not using HTTP" if it can verify that HTTP is not being used (if $SERVICE is not HTTP and $CLIENT_AUTH is false).
If a supported_versions extension was included in the ClientHello, then check that the version returned by the server was included in the ClientHello's supported_versions extension.
OpenSSL will respond to a TLSv1.3 ClientHello that only specifies 0304 in its supported_versions extension with a ServerHello that specifies whatever draft of TLSv1.3 it currently supports (e.g., 7F16). The result is that run_protocols() incorrectly reports that OpenSSL supports TLSv1.3 "final" in addition to whatever draft version it supports.
This PR fixes that problem by treating it as a failed connection when the ClientHello offers only 0304 and the ServerHello specifies something else (e.g., 7F16).
Performing this check is actually a requirement for clients in Section 4.2.1 of draft-ietf-tls-tls13-22. So, including this check will also help make client simulations more accurate when clients that support TLSv1.3 are added to client-simulation.txt.
This PR adds support for TLSv1.3 draft 22. This PR has testssl.sh operate in "middlebox compatibility mode" as described in Appendix D.4 of draft-ietf-tls-tls13-22 to maximize the chances of being able to perform a successful test even if there is a misbehaving middlebox between testssl.sh and the server being tested. Support for drafts 18 through 21 is still maintained.
This PR has been tested against a few different implementations of draft 22 that were made available shortly before draft 22 was posted.
In TLSv1.3, when responding to a HelloRetryRequest, the second ClientHello should be sent through the same socket as the first ClientHello.
This PR adds an option to socksend_tls_clienthello() to not open a socket and then uses that option in resend_if_hello_retry_request() when sending the second ClientHello.
This PR adds TLSv1.3 support for the negotiated protocol and cipher in run_server_preference(). This mostly addresses #893, however, run_server_preference() will not work with a TLSv1.3-only server as it will fail when trying to determine whether the server has a cipher order.
Note that with this PR run_server_preference() will not always provide consistent results when testing a server that does not support TLSv1.3 using a version of OpenSSL prior to 1.1.1. If it is determined before running run_server_preference() (using run_protocols()) that the server does not support TLSv1.3, then run_server_preference() will using OpenSSL to determine the negotiated protocol and cipher. However, if it has not yet been determined that the server does not support TLSv1.3, then run_server_preference() will use tls_sockets(), which tries to simulate OpenSSL 1.1.1. Since the list of cipher lists sent will differ the negotiated cipher will sometimes differ. In addition, when a cipher suite that uses an ephemeral ECDH key is selected, the negotiated curve is sometimes different.
This PR fixes#901. When $OPENSSL_TIMEOUT is set and mass testing is being performed, this PR changes find_openssl_binary() so that only child instances modify $OPENSSL to add the call to timeout.
This PR also changes the warning in case $OPENSSL_TIMEOUT is set and the timeout binary cannot be found so that the warning message is printed even if $WARNINGS is "off" or "batch".
When the server's response to the ClientHello message (i.e., the ServerHello, Certificate, ServerHelloDone) is split across more than one packet, the current call to tls_sockets() in run_heartbleed() only retrieves the first packet. As a result, sometimes when run_heartbleed() believes that it is reading the response to the Heartbleed payload it is actually reading the continuation of the response to the ClientHello message.
This PR fixes the problem by indicating in the call to tls_sockets() that the ephemeral key is needed. This causes tls_sockets() to continue requesting additional packets until it receives the ServerHelloDone.
When the certificate signature algorithm is RSA-PSS and OpenSSL 1.1.1 is used $cert_sign_algo contains some trailing space characters, which causes the algorithm not to be recognized in the case statement. This PR fixes the problem by removing any trailing space characters from $cert_sign_algo.
This PR fixes a false positive in std_cipherlists(). Currently, sclient_success is not initialized (so it initially set to 0). If a server is being tested that only supports TLSv1.3, the --ssl-native option is not used, and run_protocols() is run before run_std_cipherlists(), then for many of the calls to std_cipherlists() no tests are run and so sclient_success remains at its initial value (0), which is treated as success (i.e., the server supports at least one of the ciphers in the list).
The reason this happens is that in the testing loop, the TLSv1.3 test is skipped if the list of ciphers doesn't include any TLSv1.3 ciphers (and only the "Strong encryption" test includes TLSv1.3 ciphers) and the tests for each of lower versions of SSL/TLS is skipped since it was already determined in run_protocols() that those versions weren't supported.
This PR adds a check for TLSv1.3 support to run_protocols(), checking for support for the final version of TLSv1.3 (0x0304) as well as drafts 18, 19, 20, and 21 (0x7F12, 0x7F13, 0x7F14, and 0x7F15).
This PR is a continuation of #833.
With additional testing with different options I encountered more places where $OPENSSL was printing "WARNING: can't open config file: /usr/local/etc/ssl/openssl.cnf" where testssl.sh was not suppressing the error message.
This PR redirects stderr to /dev/null or to $ERRFILE for several more calls to $OPENSSL in order to suppress these warning messages.
For ciphers that use the ChaCha20-Poly1305 cipher, LibreSSL shows "Enc=ChaCha20-Poly1305" in the "openssl ciphers -V" command rather than "Enc=ChaCha20(256)" and for some GOST ciphers it shows "Enc=GOST-28178-89-CNT" rather than "Enc=GOST(256)". This causes a problem for neat_list() if information is being obtained from "$OPENSSL ciphers -V" rather than from the cipher-mapping.txt file.
If testssl.sh is used with OpenSSL 1.1.1 and TLSv1.3 support is enabled, then the check for whether the server has a cipher order will always fail. The problem is that since the call to s_client doesn't specify a protocol a TLSv1.3 ClientHello will be sent. However, the call specifies a list of ciphers that doesn't include any TLSv1.3 ciphers. So, OpenSSL will fail with the error: "No ciphers enabled for max supported SSL/TLS version." The solution is to add the "-no_tls1_3" option.
This PR fixes the problem by taking advantage of the recently-added s_client_options() function. It adds a "-no_tls1_3" option whenever:
* $OPENSSL supports TLSv1.3
* The command line doesn't specify any protocol: -ssl2, -ssl3, -tls1, -tls1_1, -tls1_2, or -tls1_3.
* The command line includes the -cipher option
* The list of ciphers that will be sent doesn't include any TLSv1.3 ciphers.
Add TLSv1.3 support to run_cipher_per_proto()
Fix branch
This PR adds support for TLSv1.3 to run_server_preference(). It only provides partial support, as it only works if the support supports and earlier TLS protocol (in order to determine whether the server has a cipher order). It also will only show TLSv1.3 as the "Negotiated protocol" if $OPENSSL supports TLSv1.3.
This PR also fixes a bug in which the variable "proto" was defined as used as both a regular variable and as an array.
If run_server_preference() is performed
* against a server that supports SSLv3 and that does not have a cipher order; and
* using a version of OpenSSL that does not support SSLv3; and
* with the --mapping option set to "rfc" or "no-openssl"
then the "Negotiated cipher per proto" will not show the SSLv3 cipher since cipher[i] will be empty.
This PR addresses issue #660 for run_rc4(), ensuring that support for RC4 ciphers is detected even if no RC4 ciphers are supported with the highest protocol that the server supports.
This PR adds support for TLSv1.3 to run_std_cipherlists().
This PR also provides a partial fix for #660 - addressing the issue only for run_std_cipherlists(). Rather than testing the server once for each cipher list, it tries the server once for each protocol supported by the server. This makes the testing more robust, but adds significantly to the time it takes to perform the tests.
This PR addresses issue #660 for run_sweet32(), detecting if 3DES is used, even if it isn't used with the highest protocol version supported by the server. As with PR #854, this increased robustness comes at the expense of taking addition time to run the test.
David Cooper