In some cases, the "TLS extensions" line output for the "--server-defaults" option will not show `"encrypt-then-mac/#22"` even if the server supports this extension. The reason is that a server will only include this extension in the ServerHello message if it supports the extension and the selected cipher is a CBC cipher. So, if `determine_tls_extensions()` connects to the server with a non-CBC cipher, then it will not detect if the server supports the encrypt-then-mac extension.
It is possible that support for the extension will be detected by `get_server_certificate()`, but only if one of the calls to that function results in a CBC cipher being selected and OpenSSL 1.1.0 is being used (as prior versions did not support the encrypt-then-mac extension).
In this PR, if `determine_tls_extensions()` is called and `$TLS_EXTENSIONS` does not already contain `"encrypt-then-mac/#22"`, then an attempt will be made to connect to the server with only CBC ciphers specified in the ClientHello. If the connection is not successful (presumably because the server does not support any CBC ciphers), then a second connection attempt will be made with the "default" ciphers being specified in the ClientHello.
en.wikipedia.org is an example of a server that supports the encrypt-then-mac extension, but for which the support is not currently detected (unless OpenSSL 1.1.0 is used) since in the call to `determine_tls_extension()` a non-CBC cipher is selected.
There are two places in `run_client_simulation()` in which `$OPENSSL s_client` is called, after which there is a `debugme echo` line to display the `$OPENSSL s_client` command line when testssl.sh is being run in debug mode, and then `sclient_connect_successful $? $TMPFILE` is called to determine whether `$OPENSSL s_client` successfully established a connection.
So, `sclient_connect_successful()` is being passed the result of the `debugme()` call, which always returns 0, rather than the result of the `$OPENSSL s_client` call.
This PR fixes the problem by moving the `debugme()` line to before the call to `$OPENSSL s_client`, so that `sclient_connect_successful()` is passed the results of the `$OPENSSL s_client` call.
This PR adds a new utility that generates the various static cipher lists that appear in testssl.sh.
This utility serves two purposes:
* It can be run whenever new ciphers are added to cipher-mapping.txt to see if any of the lists in testssl.sh need to be updated. (This includes if cipher-mapping.txt is modified to add OpenSSL-style names for ciphers that are currently listed, but that have not yet been assigned such names.)
* It can be used as a reference in order to understand how each of the lists is defined.
This PR fixes a few bugs in `sslv2_sockets()`. The main issue is that a server may not send the entire ServerHello in a single packet. If it doesn't and the full response is being parsed (i.e., certificate and list of ciphers), then `parse_sslv2_serverhello()` will encounter errors, since it assumes that it has the entire ServerHello. This PR compares the length of the response to the length of the ServerHello as specified in the first two bytes of the response and requests more data from the server if the response appears incomplete.
This PR also modifies `parse_sslv2_serverhello()` to check for more errors. It compares the length of the response it has been provided to the specified length (`$v2_hello_length`) and returns an error if the response is shorter than `$v2_hello_length` and the full response is supposed to be parsed. It will also check whether there was an error in converting the certificate from DER to PEM format and will return an error if there was (and it will suppress the error message).
Some servers respond to an SSLv2 ClientHello with a list of all SSLv2 ciphers that the server supports rather than just a list of ciphers that it supports in common with the client (i.e., that appear in the ClientHello). This PR changes the sockets version of `std_cipherlists()` so that, if `sslv2_sockets()` is successful, it checks whether there are any ciphers in common between the ClientHello and the ServerHello before declaring that the server supports the specified cipher list.
If `determine_tls_extensions()` does not create a temporary file (`$TEMPDIR/$NODEIP.determine_tls_extensions.txt`) then `run_server_defaults()` will display error messages when an attempt is made to copy this file or to search (grep) it. This may happen if `$OPTIMAL_PROTO` is `-ssl2` or if `determine_tls_extensions()` uses sockets and `parse_tls_serverhello()` encountered an error and did not create a temporary file (`$TEMPDIR/$NODEIP.parse_tls_serverhello.txt`). This PR fixes this by only trying to copy and search `$TEMPDIR/$NODEIP.determine_tls_extensions.txt` is `$OPTIMAL_PROTO` is not `-ssl2` and `determine_tls_extensions()` was successful (return value 0).
This PR changes `run_http2()` so that it uses `tls_sockets()` rather than failing, if `$OPENSSL` does not support the `-alpn` option. If `$OPENSSL` supports the `-alpn` option (or if `$SSL_NATIVE` is true), then this PR has no effect.
This PR change `run_std_cipherlists()` to use sockets. As noted in isse #554, I have some questions about the definitions of the cipher lists, but I wrote the code so that the ciphers that are tested when using sockets are the same as those that are tested when using OpenSSL. For a few of the cipherlists, the sockets version tests a few additional ciphers; but these are ciphers that are not supported by OpenSSL, and whose definitions are consistent with the ciphers that OpenSSL includes.
As written, `std_cipherlists` will use sockets for testing by default, except in two cases:
* If the `$SSL_NATIVE` is true, then only OpenSSL is used, and if OpenSSL doesn't support any ciphers in the cipherlist, then the test is skipped.
* If `$FAST` is true (but `$SSL_NATIVE` is false), then OpenSSL is used whenever it supports at least one cipher from the cipherlist, and `tls_sockets()` (or `sslv2_sockets()`) is only used when OpenSSL doesn't support any ciphers from the cipherlist.
In a few places testssl.sh tries to determine $OPENSSL s_client's capabilities by calling `$OPENSSL s_client` without specifying a host to which to connect. For example:
```
$OPENSSL s_client -no_ssl2 2>&1
```
This idea is that `$OPENSSL s_client` should reveal something about its capabilities without actually trying to connect to a host.
This works in most cases. However, the manual pages for s_client states:
```
-connect host:port
This specifies the host and optional port to connect to. If not specified then an attempt is made to connect to the local host on port 4433.
```
So, the above call is actually trying to connect to the local host on port 4433. If the local host is running `$OPENSSL s_server`, then `$OPENSSL s_server` will by default be listening on port 4433, and the connection attempt will most likely succeed. Since the `OPENSSL s_client` command does not include a `< /dev/null`, the `OPENSSL s_client` will just hang waiting for additional input.
Adding `-connect x` to the `$OPENSSL s_client` prevents $OPENSSL from trying to connect to a host, but seems to still provide the necessary information about OpenSSL's capabilities.
This PR adds ",exp" to the bits column when `run_rc4()` is run in the "--wide" mode and the cipher is an export cipher. This makes the wide mode of `run_rc4()` align with other functions, such as `run_allciphers()`.
