This PR uses `tls_sockets()` to determine whether a server supports certain extensions that may not be supported by `$OPENSSL`. At the moment it checks for max_fragment_length, client_certificate_url, truncated_hmac, ALPN, signed_certificate_timestamp, encrypt_then_mac, and extended_master_secret.
In https://github.com/dcooper16/testssl.sh/blob/extended_tls_sockets/testssl.sh, `run_server_defaults()` is re-written to use `tls_sockets()` instead of `$OPENSSL`, with just one call to `$OPENSSL s_client` to get the session ticket, which reduces the dependence on `$OPENSSL`, but this PR limits the number of calls to `tls_sockets()`, which is still slow.
Note: I included ALPN in the `tls_sockets()` ClientHello since a single call to `tls_sockets()` cannot test for both NPN and ALPN, and since support for NPN was added to OpenSSL before support for ALPN was added, I figured it was more likely that `determine_tls_extensions()` had already determined whether the server supported NPN.
This PR makes similar changes to `run_client_simulation()` as were made to `tls_sockets()`, so that `run_client_simulation()` retrieves the entire server response, even if it is split across multiple packets, and it has `parse_tls_serverhello()` extract information about the server's ephemeral public key, if present.
The PR also changes `run_client_simulation()` to use information about the ephemeral public key. It includes the length of the public key in the output and, if it is a DH public key, checks that the size is within the acceptable range (`${minDhBits[i]} <= dh_bits <= ${maxDhBits[i]}`).
This PR adds initial parsing of the ServerKeyExchange message to `parse_tls_serverhello()`. For ephemeral DH keys, it extracts the length of the key. For ephemeral ECDH keys that are encoded using the named_curve option, it extracts the length of the key and the name of the curve.
This PR allows the caller to provide additional extensions to `tls_sockets()` to be included in the ClientHello. If the caller provides an extension that would have already been included in the ClientHello, then the caller's value for the extension is used rather than the default value.
This PR extended `parse_tls_serverhello()` in a few ways:
* If the "full" response is to be parsed, then additional checks are performed to verify that `$tls_hello_ascii` contains the entire response
* The extensions field is parsed and the list of extensions found is placed in `$TLS_EXTENSIONS` (if the "full" response is being parsed).
* Initial support for TLS 1.3 is added:
- Accounts for differences between TLS 1.2 ServerHello and TLS 1.3 ServerHello (as outlined in PR #499).
- Recognizes new alerts and handshake message types.
- Allows for server response to include message fragments of type "application data"
I forgot that `parse_tls_serverhello()` is also called by `client_simulation_sockets()`. Since PR #499 changed the input to `parse_tls_serverhello()`, the change needs to be made in `client_simulation_sockets()` as well.
Now that the mapping file is no longer used, `$ADD_RFC_STR` should not be unset just because the mapping file cannot be found.
In addition, since `show_rfc_style()` is now used in `parse_tls_serverhello()`, it cannot return an empty string just because the user set "--mapping no-rfc" on the command line. Instead, `neat_list()` should check the value of `$ADD_RFC_STR` and not call `show_rfc_style()` if it has been unset.
Finally, since `show_rfc_style()` no longer returns strings with extra spaces, there is no need to call `strip_spaces()`
In some cases the server's response to a ClientHello spans more than one packet. If the goal is just to determine whether the connection was successful and to extract a few pieces of information from the ServerHello message, then this is unlikely to be a problem. However, if there is a desire to extract the server's certificate chain (Certificate message) or to determine the type and size of the server's ephemeral public key (ServerKeyExchange message), then the entire response needs to be obtained, even if it spans multiple packets.
This PR adds a new function, `check_tls_serverhellodone()`, that checks whether the entire response has been received (e.g., whether the ServerHelloDone message has been received). If the response indicates that the response is incomplete, then `tls_sockets()` requests more data from the server until the response is complete or until the server doesn't provide any more data in response.
The PR only changes the behavior of `tls_sockets()` if the caller indicates that it wants to extract the ephemeral key or that it wants the entire response to be parsed. Otherwise, only the first packet returned by the server is sent to `parse_tls_serverhello()`. [The value of `$process_full` is not used at the moment, but will be in a subsequent PR that modifies `parse_tls_serverhello()`.]
This PR also changes `tls_sockets()` to send a close_notify to the server if the connection was successfully established.
PR #346 added a test for version tolerance to `run_protocols()`, but I think it may now be more appropriate to remove that test. Draft -16 of TLS 1.3, which was posted on September 22, changed the way that version negotiation is handled for TLS 1.3 and above. The current version tolerance test sends a ClientHello with the version field set to "03, 05", to represent a TLS 1.4 ClientHello. While this was consistent with RFC 5246 and with drafts of TLS 1.3 up to -15, draft -16 changed the version field to `legacy_version` and declared that its value should be "03, 03" for TLS 1.2 and above. (For TLS 1.3 and above a Supported Versions extension is included to inform the server which versions of TLS the client supports.) The change in draft -16 was made as a result of the problems with servers not handling version negotiation correctly.
Since the current draft suggests that a server should never be presented with a ClientHello with a version higher than "03, 03" (even for clients that support TLS versions higher than 1.2), it seems there is no reason to include the version tolerance test anymore.
For servers that do not support TLS 1.2, the additional checks that were added by PR #346 will already detect if the server cannot perform version negotiation correctly.
This PR adds the option for `parse_sslv2_serverhello()` to extract information from the ServerHello (server key size and cipher suites supported) and write the information to `$TMPFILE` as well as to write the server's certificate to `$HOSTCERT`.
The mapping file is now only used in `show_rfc_style()`. This PR changes `show_rfc_style()` to use the `$TLS_CIPHER_HEXCODE` and `$TLS_CIPHER_RFC_NAME` arrays.
Note that `get_install_dir()` still searches for the mapping-rfc.txt in order to determine `$INSTALL_DIR`. `$INSTALL_DIR` is only used to determine the location of the CA bundles in `determine_trust()`:
```
local ca_bundles="$INSTALL_DIR/etc/*.pem"
```
This PR changes `sslv2_sockets()` so that a list of ciphers may optionally be passed as an argument. This will support the use of `sslv2_sockets()` in some places where `$OPENSSL s_client` is currently used.
Changed `Enc=CHACHA20/POLY1305(256)` to `Enc=ChaCha20(256)` and `Enc=GOST-28178-89-CNT(256)` to `Enc=GOST(256)` in order to shorten the names that are printed, so that they fit in the allocated column.
Added the four experimental post-quantum cipher suites mentioned in #462.
`socksend_tls_clienthello()` always includes a server name extension in the ClientHello (for TLS 1.0 and above), even if `$SNI` is empty. If `$NODE` is an IP address, then the IP address is placed in the extension, even though RFC 6066 says that only DNS names are supported in the extension.
This PR changes `socksend_tls_clienthello()` so that the server name extension is only included in the ClientHello is `$SNI` is not empty.
This PR is an attempt to address issue #447. If more than one certificate is being displayed, then a parenthetical saying "(in response to request w/o SNI)" is added for any certificate that was obtained using `$SNI=""`.
In addition, if the certificate was obtained without SNI, then `certificate_info()` doesn't call `$OPENSSL s_client` in order to obtain the non-SNI host certificate and it does not display a separate "Trust (hostname)" finding for the non-SNI certificate.
When `certificate_info()` is given a certificate with a DH public key it displays something like:
```
Server key size fixme: dhKeyAgreement 3072 bits (FIXME: can't tell whether this is good or not)
```
This PR fixes that so that the output is:
```
Server key size DH 3072 bits
```
This PR is in response to issue #454. I tried repeating the reported problem by creating a certificate in which the extendedKeyUsage extension was present and only included the anyExtendedKeyUsage OID. In running the test, I discovered two problems.
First, when `determine_trust()` is calling `verify_retcode_helper()` to display the reason that path validation failed, it assumes that there are at least two certificate bundles provided. (I was running the test using just one certificate bundle, containing my local root.) So, I changed `determine_trust()` to use `${verify_retcode[1]}` rather than `${verify_retcode[2]}` in the case that all bundles failed (it seems that 2 vs. 1 was an arbitrary choice).
Once that was fixed, testssl.sh output "NOT ok (unknown, pls report) 26". So, the second thing this PR fixes is to output "NOT ok (unsupported certificate purpose)" if OpenSSL responds with an unsupported certificate purpose error.
With OpenSSL 1.1.0, `s_client -no_ssl2` fails with an "unknown option" error. At the moment the `-no_ssl2` option is only used in two functions, `run_client_simulation()` and `run_crime()`. In `run_crime()`, the `-no_ssl2` option is only included if the OpenSSL version is 0.9.8.
This PR checks whether the OpenSSL version in use supports the `-no_ssl2` option, and if it doesn't, it removes it from the calls to `s_client` in `run_client_simulation()`.
If the version of OpenSSL being used doesn't support `s_client -ssl3` (e.g., OpenSSL 1.1.0), `run_beast()` doesn't display a warning that testing for CBC in SSLv3 isn't locally supported.
This PR adds a "Local problem" warning if the OpenSSL being used doesn't support `s_client -ssl3`.
The test for whether a server only supports SSLv2 was broken, since `$OPTIMAL_PROTO` will be `-ssl2` whether SSLv2 is the only protocol that succeeds or no protocol succeeds.
This PR sets $OPTIMAL_PROTO (or $STARTTLS_OPTIMAL_PROTO) to "" if no protocol succeeds.
If the version of OpenSSL being used doesn't support `s_client -ssl3` (e.g., OpenSSL 1.1.0), `run_ssl_poodle()` displays `not vulnerable (OK)` even though it can't test whether the server is vulnerable.
This PR fixes it so that a "Local problem" warning is displayed is `s_client -ssl3` isn't supported.
The PR also removes the `$SNI` from the call to `$OPENSSL s_client` since OpenSSL ignores the `-servername` directive for `-ssl3` anyways.
If testssl.sh is called with `--devel 22` and the response from `sslv2_sockets()` is not 0, then `tls_sockets()` will be called, and the result of the `tls_sockets()` command will be output rather than the result of the `sslv2_sockets()` command.
This PR addresses the "FIXME" in `run_protocols()`:
```
sslv2_sockets #FIXME: messages/output need to be moved to this (higher) level
```
It also changes `run_drown()` to call `sslv2_sockets()` in order to avoid duplicate code.
This PR is in response to issue #352, where it was noted that Bash does not support binary data in strings.
I replaced all calls to `sockread()` with calls to `sockread_serverhello()`, and then, since is now used everywhere and not just to read ServerHello messages, I renamed `sockread_serverhello()` to `sockread()`.
I tested the revised code against several servers, including one that is vulnerable to CCS and Heartbleed, and got the same results as with the current code (although the hexdumps displayed in debug mode differ).
One concern I have is the code in `run_ccs_injection()`. The current code is:
```
byte6=$(echo "$SOCKREPLY" | "${HEXDUMPPLAIN[@]}" | sed 's/^..........//')
lines=$(echo "$SOCKREPLY" | "${HEXDUMP[@]}" | count_lines )
debugme echo "lines: $lines, byte6: $byte6"
if [[ "$byte6" == "0a" ]] || [[ "$lines" -gt 1 ]]; then
pr_done_best "not vulnerable (OK)"
...
```
I revised this to:
```
if [[ -s "$SOCK_REPLY_FILE" ]]; then
byte6=$(hexdump -ve '1/1 "%.2x"' "$SOCK_REPLY_FILE" | sed 's/^..........//')
lines=$(hexdump -ve '16/1 "%02x " " \n"' "$SOCK_REPLY_FILE" | count_lines )
debugme echo "lines: $lines, byte6: $byte6"
fi
rm "$SOCK_REPLY_FILE"
if [[ "$byte6" == "0a" ]] || [[ "$lines" -gt 1 ]]; then
...
```
In the revised code `byte6` is initialized to `0a` so that the response is `not vulnerable (OK)` if `$SOCK_REPLY_FILE` is empty. This has worked okay since for all of the servers that I tested that weren't vulnerable `$SOCK_REPLY_FILE` was empty. Since I haven't seen any other examples, I don't understand why check for vulnerability was written the way it was. So, I'm a bit concerned that the test in the revised code may produce incorrect results now that `hexdump -ve '1/1 "%.2x"' "$SOCK_REPLY_FILE"` is an accurate hexdump of the reply.
In the check for old versions of OpenSSL, the results of the call to `ignore_no_or_lame()` are ignored, and so the program continues even if the user enters `no`.
This PR makes three changes to `determine_optimal_proto()`:
* It no longer tries an empty string for `$OPTIMAL_PROTO` twice.
* It does not include `-servername` for `-ssl2` or `-ssl3`, since some versions of OpenSSL that support SSLv2 will fail if `s_client` is provided both the `-ssl2` and `-servername` options.
* It displays a warning if `$OPTIMAL_PROTO` is `-ssl2`, since some tests in testssl.sh will not work correctly for SSLv2-only servers.
This PR addresses two issues related to SSLv2 for "--server-preference" checks.
First, some versions of OpenSSL that support SSLv2 will fail if `s_client` is provided both the `-ssl2` and `-servername` options.
Second, the line for extracting the chosen cipher,`cipher=$(awk '/Cipher.*:/ { print $3 }' $TMPFILE)`, fails for SSLv2. For SSLv2, the output from `$OPENSSL s_client` is as shown below, and the `cipher=` line extracts the word `between` from `Ciphers common between both SSL endpoints:` rather than `IDEA-CBC-MD5` from ` Cipher : IDEA-CBC-MD5`.
```
...
Ciphers common between both SSL endpoints:
RC4-MD5 RC2-CBC-MD5 IDEA-CBC-MD5
DES-CBC-MD5 DES-CBC3-MD5
---
SSL handshake has read 1191 bytes and written 373 bytes
---
New, SSLv2, Cipher is IDEA-CBC-MD5
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : SSLv2
Cipher : IDEA-CBC-MD5
...
```
This PR changes test_just_one() to correctly handle SSLv2 ciphers.
As with PR #424, this PR addresses the problem in which servers that do not implement SSLv2, but that implement RC4-MD5, EXP-RC2-CBC-MD5, EXP-RC4-MD5, or NULL-MD5 are shown as implementing both the SSLv2 and SSLv3 versions of the ciphers, and that any SSLv2 ciphers that a server does implement are not shown as being implemented.
This PR changes run_rc4() to correctly handle SSLv2 ciphers.
It addresses the problem in which servers that do not implement SSLv2, but that implement SSLv3 ciphers that share an OpenSSL name with an SSLv2 cipher (RC4-MD5 and EXP-RC4-MD5), are not incorrectly shown as having implemented the SSLv2 cipher.
It also addresses the problem that if a server does implement SSLv2 with an RC4 SSLv2-cipher suite, then that cipher suite it not shown as being implemented.
`certificate_info()` does not correctly display the Issuer name for CAs that use domain component attributes.
There is a server on the NIST intra-net that I test against that has a certificate issued by a NIST CA, and the issuer name in the certificate is of the form: `/DC=net/DC=example/DC=internal/CN=CAname`
Since there is no organizational name, testssl.sh displays the name as:
```
Issuer "CAname" ("")
```
In this PR, if the Issuer name has 'DC=' attributes, but does not have an 'O=' attribute, the "DC=" attributes are combined into a DNS name that is used as if it were the organizational name:
```
Issuer "CAname" ("internal.example.net")
```
I should note, however, that I have not been able to find any other examples of TLS server certificates that have been issued by CAs that have domain components ("DC=") in their names. So, it may not be worthwhile to change the code to try to accommodate such CAs.
`certificate_info()` currently outputs `$issuer` to the JSON file, where is should be outputting `$issuer_CN` in order for the information in the JSON file to match the information that is displayed.
This PR also fixes the problem that if an Issuer name contains a domain component attribute (DC=) then it will be mistakenly treated as a country attribute (C=).
Rather than try each curve one at a time, follow model in `cipher_pref_check()`. First include all curves in ClientHello, then successively remove from the ClientHello those curves that have been offered by the server until the connection fails. This makes the number of calls to `$OPENSSL s_client` one more than the number of supported curves rather than the number of curves in NamedCurve supported by $OPENSSL.
Note, however, that OpenSSL defines MAX_CURVELIST as 28 and fails if the `-curves` option includes more than 28 curves. Since OpenSSL 1.1.0 offers 29 curves from NamedCurve, this PR breaks the list of supported curves in 2. At the cost of one additional calls to `$OPENSSL s_client` it ensures that the number of curves provides to the `-curves` option is below the limit.
