run_renego() appears to produce a false positive if OpenSSL 1.1.1 is used and the server being tested supports TLSv1.3 (i.e., the server supports the same draft version of TLSv1.3 as the version of OpenSSL 1.1.1 being used does). This PR fixes the problem by telling calls to $OPENSSL s_client in run_renego() to not use TLSv1.3.
.. to check during the default run for server implemenation bugs
and run cipher per procol check instead of cipher check.
Please not that this option could disappear later.
In TLSv1.3 servers may send a supported_groups extension, which "SHOULD contain all groups the server supports, regardless of whether they are currently supported by the client."
This PR extracts the contents of the supported_groups extension, if `parse_tls_serverhello()` is to process "all" of the server's response. The contents of the extension are also displayed on the terminal if $DEBUG -ge 3.
In TLSv1.2 and below, servers respond to a status_request extension (a request for a stapled OCSP response) by returning an empty status_request extension and then including a CertificateStatus message, which follows the Certificate message. In TLSv1.3 the CertificateStatus response is included as the value of the status_request extension, which now appears as an extension within the Certificate message.
This PR extracts the contents of the status_request extension sent by the server so that it can later be processed in the same way as if it had sent in a TLSv1.2 or below response.
This PR adds code to decrypt the encrypted portion of the server's response for TLSv1.3 and to then process any certificates and encrypted extensions. This code supports all 5 TLSv1.3 cipher suites, and so any response can be decrypted as long as the session key can be derived (which requires OpenSSL to support the ephemeral key that was used - see #938).
For the symmetric decryption, the sym-decrypt() function uses OpenSSL when possible and internal Bash functions when needed.
For AES-GCM and AES-CCM ciphers sym-decrypt() normally uses internal Bash functions, which rely on using "$OPENSSL enc" in AES-ECB mode to generate the key stream and then Bash functionality to XOR the key stream with the ciphertext. With some version of OpenSSL the AES-GCM ciphers are decrypted using "$OPENSSL enc" in AES-GCM mode directly. On my system, however, both methods seem to work about equally fast.
For ChaCha20 ciphers, "$OPENSSL enc -chacha20" is used, if supported (OpenSSL 1.1.x only). and Bash internal functions (without any OpenSSL support) are used otherwise. In this case, if the Bash internal functions need to be used, decryption is very, very, very slow. Fortunately, in a typical run of testssl.sh there won't be many cases in which the connection will be TLSv1.3 with ChaCha20 and the entire response needs to be processed (requiring decryption). In most cases, even if the connection is TLSv1.3 with ChaCha20, will at most need the ephemeral key, which is available in plain text.
This is the first in a series of PRs to add support for processing the encrypted portions of the server's response in a TLSv1.3 handshake.
This PR adds the code to derive the handshake traffic key needed to decrypt the response (the next PR will add the code to perform the symmetric-key decryption of the encrypted portions of the response).
Since this PR does not make use of the traffic key that it derives, it doesn't yet add any new functionality.
Note that testssl.sh will not always be able to derive the session keys. If the version of OpenSSL that is bundled with testssl.sh is used and the server chooses to use an X25519 ephemeral key, OpenSSL will be unable to perform the shared secret in derive-handshake-traffic-secret(). (OpenSSL 1.1.0 supports X25519.) Since X25519 use a different encoding than ECDH keys, the lack of X25519 support will be discovered in parse_tls_serverhello() when $OPENSSL pkey is unable to convert the key from DER to PEM. So, in debugging mode, parse_tls_serverhello() now displays a warning if it receives a key share that $OPENSSL pkey cannot handle.
get_server_certificate() uses an awk script to extract the certificates from the output of OPENSSL s_client and it then uses the following line to determine how many certificates were found:
nrsaved=$(count_words "$(echo level?.crt 2>/dev/null)")
If $nrsaved is 0, then get_server_certificate() returns 1 (indicating failure); otherwise it returns 0 (indicating success).
However, the check for the number of certificates returned doesn't work if no certificates were found, as nrsaved will be set to 1 if no certificates were found:
> touch level0.crt
> echo level?.crt
level0.crt
> touch level1.crt
> echo level?.crt
level0.crt level1.crt
> rm level0.crt level1.crt
> echo level?.crt
level?.crt
This PR fixes the problem by first checking that level0.crt exists (-s is used instead of -e, since an empty file wouldn't have a certificate).
Similar to the recently added HAS_PKUTIL (f829878a43), this PR adds HAS_PKEY, which indicates whether OpenSSL has the pkey utility. HAS_PKEY is then checked before attempting to do something that requires the pkey utility.
POP3 STARTTLS handshakes were often unsuccessful as
a regex wasn't properly escaped.
Furthermore if a STARTTLS handshake doesn't succeed, there's
a warning now.
See pending PR #905 / issue #333.
There's still lots of work needed and probably the function
needs to be completely rewritten and to be in sync with
other parts of the program.
This PR fixes one odd formatting of header flags like X-Frame-Options,
where the output header maybe contained a LF "\r". X-XSS-Protection was
also not correctly formatted due to the fact that only a part of it until
the blank was displayed.
Also the file output may contain now 1x less blank, e.g.
"X-Content-Type-Options: nosniff" instead of
"X-Content-Type-Options: nosniff"
This PR adds an additional COLOR level (3). If color is set to 3 then all ciphers are printed according to pr_cipher_quality() rather than just the "Negotiated cipher" in run_server_preference().
This PR reduces the amount of work parse_tls_serverhello() does when processing alert messages when not in debug mode. It delays writing anything to $TMPFILE unless $DEGUG -ge 1 until it has reason to believe that the response was successful. If $DEBUG is 0 and alert messages are sent, then no file operations are performed processing the alert messages.
In almost every case, there is no attempt to look at the contents of $TEMPDIR/$NODEIP.parse_tls_serverhello.txt unless the connection was successful. So, in most cases, it is okay to not call tmpfile_handle() in parse_tls_serverhello() unless the connection was successful. There is, however. one place in run_grease() where the code reads the contents of $TEMPDIR/$NODEIP.parse_tls_serverhello.txt even if the connection was not successful. In order to address this, the DEBUG level is temporarily set to 1 when performing this test if its value is 0. Also in order to address this, changes were made in parse_tls_serverhello() to ensure that "tmpfile_handle $FUNCNAME.txt" is always called before returning if $DEBUG -ge 1.
David Cooper
Dirk
Frank Breedijk