This PR adds code to decrypt the encrypted portion of the server's response for TLSv1.3 and to then process any certificates and encrypted extensions. This code supports all 5 TLSv1.3 cipher suites, and so any response can be decrypted as long as the session key can be derived (which requires OpenSSL to support the ephemeral key that was used - see #938).
For the symmetric decryption, the sym-decrypt() function uses OpenSSL when possible and internal Bash functions when needed.
For AES-GCM and AES-CCM ciphers sym-decrypt() normally uses internal Bash functions, which rely on using "$OPENSSL enc" in AES-ECB mode to generate the key stream and then Bash functionality to XOR the key stream with the ciphertext. With some version of OpenSSL the AES-GCM ciphers are decrypted using "$OPENSSL enc" in AES-GCM mode directly. On my system, however, both methods seem to work about equally fast.
For ChaCha20 ciphers, "$OPENSSL enc -chacha20" is used, if supported (OpenSSL 1.1.x only). and Bash internal functions (without any OpenSSL support) are used otherwise. In this case, if the Bash internal functions need to be used, decryption is very, very, very slow. Fortunately, in a typical run of testssl.sh there won't be many cases in which the connection will be TLSv1.3 with ChaCha20 and the entire response needs to be processed (requiring decryption). In most cases, even if the connection is TLSv1.3 with ChaCha20, will at most need the ephemeral key, which is available in plain text.
This is the first in a series of PRs to add support for processing the encrypted portions of the server's response in a TLSv1.3 handshake.
This PR adds the code to derive the handshake traffic key needed to decrypt the response (the next PR will add the code to perform the symmetric-key decryption of the encrypted portions of the response).
Since this PR does not make use of the traffic key that it derives, it doesn't yet add any new functionality.
Note that testssl.sh will not always be able to derive the session keys. If the version of OpenSSL that is bundled with testssl.sh is used and the server chooses to use an X25519 ephemeral key, OpenSSL will be unable to perform the shared secret in derive-handshake-traffic-secret(). (OpenSSL 1.1.0 supports X25519.) Since X25519 use a different encoding than ECDH keys, the lack of X25519 support will be discovered in parse_tls_serverhello() when $OPENSSL pkey is unable to convert the key from DER to PEM. So, in debugging mode, parse_tls_serverhello() now displays a warning if it receives a key share that $OPENSSL pkey cannot handle.
get_server_certificate() uses an awk script to extract the certificates from the output of OPENSSL s_client and it then uses the following line to determine how many certificates were found:
nrsaved=$(count_words "$(echo level?.crt 2>/dev/null)")
If $nrsaved is 0, then get_server_certificate() returns 1 (indicating failure); otherwise it returns 0 (indicating success).
However, the check for the number of certificates returned doesn't work if no certificates were found, as nrsaved will be set to 1 if no certificates were found:
> touch level0.crt
> echo level?.crt
level0.crt
> touch level1.crt
> echo level?.crt
level0.crt level1.crt
> rm level0.crt level1.crt
> echo level?.crt
level?.crt
This PR fixes the problem by first checking that level0.crt exists (-s is used instead of -e, since an empty file wouldn't have a certificate).
Similar to the recently added HAS_PKUTIL (f829878a43), this PR adds HAS_PKEY, which indicates whether OpenSSL has the pkey utility. HAS_PKEY is then checked before attempting to do something that requires the pkey utility.
POP3 STARTTLS handshakes were often unsuccessful as
a regex wasn't properly escaped.
Furthermore if a STARTTLS handshake doesn't succeed, there's
a warning now.
See pending PR #905 / issue #333.
There's still lots of work needed and probably the function
needs to be completely rewritten and to be in sync with
other parts of the program.
This PR fixes one odd formatting of header flags like X-Frame-Options,
where the output header maybe contained a LF "\r". X-XSS-Protection was
also not correctly formatted due to the fact that only a part of it until
the blank was displayed.
Also the file output may contain now 1x less blank, e.g.
"X-Content-Type-Options: nosniff" instead of
"X-Content-Type-Options: nosniff"
This PR adds an additional COLOR level (3). If color is set to 3 then all ciphers are printed according to pr_cipher_quality() rather than just the "Negotiated cipher" in run_server_preference().
This PR reduces the amount of work parse_tls_serverhello() does when processing alert messages when not in debug mode. It delays writing anything to $TMPFILE unless $DEGUG -ge 1 until it has reason to believe that the response was successful. If $DEBUG is 0 and alert messages are sent, then no file operations are performed processing the alert messages.
In almost every case, there is no attempt to look at the contents of $TEMPDIR/$NODEIP.parse_tls_serverhello.txt unless the connection was successful. So, in most cases, it is okay to not call tmpfile_handle() in parse_tls_serverhello() unless the connection was successful. There is, however. one place in run_grease() where the code reads the contents of $TEMPDIR/$NODEIP.parse_tls_serverhello.txt even if the connection was not successful. In order to address this, the DEBUG level is temporarily set to 1 when performing this test if its value is 0. Also in order to address this, changes were made in parse_tls_serverhello() to ensure that "tmpfile_handle $FUNCNAME.txt" is always called before returning if $DEBUG -ge 1.
If certificate-based client authentication is required by the server, then most HTTP-related checks are skipped, even if the "--assume-http" flag is used. If $CLIENT_AUTH is true, then $ASSUME_HTTP is ignored.
In some cases the checks are appropriately skipped, since the tests cannot be performed. In other places, the value of "$CLIENT_AUTH" is used as a hint as to whether HTTP is being used. For example, in run_tickbleed:
if [[ "$SERVICE" != HTTP ]] && ! "$CLIENT_AUTH"; then
outln "-- (applicable only for HTTPS)"
fileout "ticketbleed" "INFO" "Ticketbleed: not applicable, not HTTP" "$cve" "$cwe"
return 0
fi
There are some places, however, where tests are just skipped, even if both $CLIENT_AUTH and $ASSUME_HTTP are true, even though the test could be performed. For example, run_client_simulation() only simulates generic clients in this case.
This PR attempts to address this:
* In run_client_simulation() it runs all of the tests if $ASSUME_HTTP is true.
* In certificate_transparency() it only says that the lack of CT information is "N/A" it can verify that HTTP is not being used (if $SERVICE is not HTTP and $CLIENT_AUTH is false). Otherwise it just says "no" without flagging it as an issue.
* In certificate_info() it displays additional warnings (about use of SHA-1 or subjectAltName matching) only if it can verify that HTTP is being used ($SERVICE is HTTP or $ASSUME_HTTP is true).
* In run_crime(), if compression is used, it only says " but not using HTTP" if it can verify that HTTP is not being used (if $SERVICE is not HTTP and $CLIENT_AUTH is false).
If a supported_versions extension was included in the ClientHello, then check that the version returned by the server was included in the ClientHello's supported_versions extension.
OpenSSL will respond to a TLSv1.3 ClientHello that only specifies 0304 in its supported_versions extension with a ServerHello that specifies whatever draft of TLSv1.3 it currently supports (e.g., 7F16). The result is that run_protocols() incorrectly reports that OpenSSL supports TLSv1.3 "final" in addition to whatever draft version it supports.
This PR fixes that problem by treating it as a failed connection when the ClientHello offers only 0304 and the ServerHello specifies something else (e.g., 7F16).
Performing this check is actually a requirement for clients in Section 4.2.1 of draft-ietf-tls-tls13-22. So, including this check will also help make client simulations more accurate when clients that support TLSv1.3 are added to client-simulation.txt.
The default STARTTLS_SLEEP timeout was increased to 10 seconds in
d1e7498. This caused MySQL connections to timeout. Quick fix is to
parameterize the timeout and pass in 1 again.
Better future fix is to read MySQL as binary packets, parsing the fixed
sized header, to then read the variable sized payload. Doing this will
also greatly speed up testing.
This fixes issue #914.
It partly addresses #915, supposedly the openssl binary used supports
TLS 1.3 (and the correct draft/final).
It also reduces handshakes by not trying protocols which aren't supported
on the client side.
Dirk
David Cooper
Frank Breedijk
Steven Danneman