As per @krissi's suggestion this is a smart addition to add privat CAs
to all certificate stores -- during runtime only. The switch --add-ca or
--add-CA expects an argument of one additional CA file or a comma separated
lists of them. The enviroment file ADDITIONAL_CA_FILES can be used
alternatively.
This fixes eventually #230.
For recognising error conditions during run any testssl.sh run, the program
returns 0 if all checks have been performed successfully and unambiguously. 1
is returned per error or unambiguous condition and per IP. E.g. ./testssl.sh
--ssl-native testssl.net with two IP addresses returns 2 as each client
simulation gets a "1" (ssl-native returns currently wrong results here).
This fixes#986.
A kind of vulnerability scoring for monitoring tools or CI will be done
later separately, see #985 and #327.
As before (see #971) and as indicated (#970) some minor
changed to the jsonID have been taking place. Some
redundant verbosity has been eliminated, some IDs
changed the name.
For HTTP2 you should now ALPN_HTTP2.
Some obvious CWEs were added in the JSON section.
ke for null cipher list, information leakage in
the http headers. There's probably space for improvements.
A few early time marks were added to debug time spend.
Some servers will respond with an alert to a ClientHello that does not indicate support for secure renegotiation, which may be signaled through either an extension or the 0x00,0xff "cipher suite." In some cases testssl.sh calls tls_sockets() without including "00,ff" in the list of cipher suites, which results in some servers rejecting a ClientHello that would otherwise result in a successful connection.
This PR fixes the problem by adding "00,ff" to any ClientHello where it was previously missing, with one exception. If a TLSv1.3 ClientHello is being sent and only TLSv1.3 ciphers are listed, then the "00,ff" cipher suite is not added.
Error codes now done until run_server_defaults(). Previous
sections modified so that execution errors are added.
Modified in compare_server_name_to_cert() ret --> subret.
From the code perspective a clear distinction between
passing an error code and a functional return code
would be great. Still has to be determined whether it
maybe better to rename ret into something different.
Removed a stale code line in run_server_preference() for STARTTLS.
Following the recommendation from @dcooper16 this commit is addressing
a situation when the scan couldn't finish for external reasons and as
a consequence left a non-valid JSON file behind.
It also starts addressing #986 so that the protcol section only returns
a non-zero value if a check coundn't be performed or gave results which
weren't clear.
It also fixes a typo where in the TLS 1.3 check a status from the TLS 1.2
check was not correctly interpreted (TLS 1.2 not offered).
This commit fixes#983 by ensuring that the line printed just before calling "return" prints a newline character. It also fixes the problem that no output is sent to the JSON/CSV file in some cases in which no fallback is possible since the server does not support two different protocols below TLSv1.3.
In order to be more consistent with the other output functions having
"svrty" in their name, *_done_best and *_done_good were changed to
*_svrty_best and *_svrty_good.
run_spdy/run_http2 were renamed to run_npn and run_alpn as this is
what is actually being tested. Also the terminal and file output
is now reflecting this.
Also #980 was fixed in a sense that (only) for ALPN the protocol
h2 will get a "good". There will be an additional CSV/JSON line
for this.
This commit fixes#981 by using a while loop instead of a for loop to check each DNS name in the SAN extension, copying the syntax used in certificate_info() to display all of the SANs.
This commit adds code to run_server_defaults() so that in debug mode all of the server's certificates are saved in $TMPDIR in both PEM-encoded format and pretty-print text format.
testssl.sh was recently changed to store the text printout of the host's certificate in a file, $HOSTCERT_TXT, and then use this file in some places rather than calling "$OPENSSL x509 -in $HOSTCERT -text -noout". There was a problem, however, in cases in which the server had more than one certificate (including cases in which the server returned an unrelated certificate when sent a ClientHello w/o SNI), since the contents of $HOSTCERT_TXT was not always being updated whenever $HOSTCERT changed.
This commit fixes that problem by replacing the previous solution with a less ambitious one. In this version, the global variable is eliminated and instead run_server_defaults() stores a text version of each certificate it finds in an array. This value is then passed to certificate_transparency() and certificate_info() for use. It is also passed from certificate_info() to must_staple().
determine_trust() uses the output of "$OPENSSL verify" to determine whether OpenSSL can construct a valid certification path for the server's certificate. If it does not find a string of the form "error [1-9][0-9]? at [0-9]+ depth lookup:" in the output, then it assumes that validation was successful. In current versions of OpenSSL, when this error is created it is printed to stdout, but in OpenSSL 1.1.1 is it printed to stderr. Since testssl.sh only checks the output sent to stdout, it incorrectly treats all certificates as valid if OpenSSL 1.1.1 is used.
This commit fixes the problem by checking the text that is sent to both stdout and stderr.
This commit also fixes a typo in the call to "$OPENSSL verify" which resulted in the environment variables SSL_CERT_DIR and SSL_CERT_FILE not being set to "/dev/null".
In ``must_staple()`` and ``certificate_info()`` were the jsonIDs
changed to OpenSSLi / IETF names so that testssl.sh is more compliant
to the rest of the world. There might be still space for improvements
are far as common naming scheme is concerned.
The host certificate is now being delivered in JSON and CSV. For
further usage " " needs to be converted back to linefeeds.
Certificate Expiration was renamed to Certificate Validity.
The order of outputting the certificate serial and SHA1 fingerprint
has been swapped.
Also ``certificate_info()`` makes more use of HOSTCERT_TXT.
For certificate start+end time it is now displaying the
time on UTC and without mentioning the timezone twice.
Also if neither CRL nor OCSP URI is provided it'll appear
on the screen below those two checks. JSON/CSV has then an
additional finding
HOSTCERT_TXT is now generated each time a HOSTCERT is
written. For now it is just being used in ``read_sigalg_from_file()``.
More to come.
Also in JSON output the start date and end date of the certificate
are now in separate objects. (the time format could need some polishing though).
In the output created by certificate_info(), the "Server key size" line labels an elliptic curve key as "ECDSA." This commit changes the label to "EC." I believe this a more correct label since ECDSA is a signature algorithm, not a key type. Also, while unlikely, an elliptic curve key in a certificate may be used for ECDH (e.g, in TLS_ECDH_RSA_WITH_AES_128_CBC_SHA) rather than ECDSA.
Note that this does not impact the JSON or CSV output, since the corresponding fileout command already uses "$cert_keysize EC bits"
This commit adds TLSv1.3 support for run_server_defaults(). It uses get_server_certificate() to run separate checks for RSA and ECDSA certificates by sending TLSv1.3 ClientHello messages with different signature_algorithms values. A similar change is made to certificate_transparency() in order to check for certificate transparency support for certificates returned over TLSv1.3. It also modifies the check for certificates offered without SNI by including an option to use tls_sockets() with servers that support TLSv1.3 if OpenSSL does not support TLSv1.3.
In TLSv1.3, if a HelloRetryRequest needs to be sent and the call to tls_sockets() includes additional request extensions (other than key_share or cookie), then resend_if_hello_retry_request() adds an extra comma between extensions in the value sent to socksend_tls_clienthello(), which creates errors. This commit fixes the problem by removing the extra comma.
``run_tls_fallback_scsv()`` was upgraded to the new scheme.
It also changes a couple of more functions using a variable ``$jsonID`` instead
of a repeating fixed value for `fileout()`.
Also some it adds some ``fileout`` calls which were missing.
In the sense of the previous commits this one adds
further changes to the JSON output parameters ID and findings.
Findings in JSON are now more crunchy and don't repeat parts
of ID. Also ID changed so that it is more reflecting what
has been tested.
As the cipherlist checks in the beginning have been less
and less to do with the OPsnSSL standard lists
a change to remove the word "standard" was long overdue.
That has been addressed now alos in the code and in the
JSON/CSV output.
$HOSTCERT has now an .pem file extension. $HOSTCERT_TXT
will contain the text putput of the x509 openssl operation
on the certificate which enables testssl.sh to remove
some of the redundant "openssl x509 -in $HOSTCERT -text"
calls.
fileout <somestrings> has not been consistently replaced
by fileout $jsonID yet.
This PR defines an extract_certificates() function in order to remove some redundant code from get_server_certificate(). Currently, nearly identical code appears in two places in get_server_certificate() to extract certificates from the output of `$OPENSSL sclient`, in one place for SSLv2 responses and in another for SSLv3 through TLSv1.2. The code to get the certificates used with TLSv1.3 (see https://github.com/dcooper16/testssl.sh/tree/extended_tls_sockets) would have added a third place where this same code would be needed. This PR allows the code to be written once and used in all three places.
This commit just adds more changes to those two parameters.
It is not completely done yet (see downgrade protection via SCSV).
Also json_prefix was changed to jsonID. The complete change of the fist
fileout field is pending.
The server default run had several JSON objects which weren't, looking at just
the ID, either clear or contained a redundant explanation in "finding". Purely
certificate related JSON objects are now having the id "cert_<object>" like
cert_CN or cert_SAN.
This commit changes all this, also it avoids another colon in finding (see #830).
Also the implicit strategy "output for the screen s followed by only one output with
fileout" has been relaxed -- which results on more, better parsable JSON objects.
Some example of the changes:
Old:
----
{
"id" : "Server Certificate #1 fingerprint",
"severity" : "INFO",
"finding" : "Fingerprints / Serial: SHA1 2940BC13ECF7DAF30B9084CC734C3B971D73B3BB / 01BFD1DC15006E0ABBA7C670FF5E1101, SHA256 30BA61012FFE7CEAAF9A148A0CB0C5C852A9C04F4B1C27DB6
EFA9919C7F49CCF"
}
[..]
{
"id" : "Server Certificate #2 ocsp_stapling",
"severity" : "OK",
"finding" : "OCSP stapling : offered"
}
New:
----
{
"id" : "cert_key_size <cert#1>",
"severity" : "INFO",
"finding" : "Server keys 2048 bits"
},{
"id" : "cert_fingerprint_SHA1 <cert#1>",
"severity" : "INFO",
"finding" : "2940BC13ECF7DAF30B9084CC734C3B971D73B3BB"
},{
"id" : "cert_fingerprint_SHA256 <cert#1>",
"severity" : "INFO",
"finding" : "30BA61012FFE7CEAAF9A148A0CB0C5C852A9C04F4B1C27DB6EFA9919C7F49CCF"
},{
"id" : "cert_serial <cert#1>",
"severity" : "INFO",
"finding" : "01BFD1DC15006E0ABBA7C670FF5E1101"
}
[..]
{
"id" : "OCSP_stapling <cert#2>",
"severity" : "OK",
"finding" : "offered"
}
This PR also fixes the JSON output where for "OCSP must staple" the id was just
'id" : "OCSP must staple: ocsp_must_staple",' for multiple server
certificates without the certificate number.
As far as the code is concerned: $json_prefix should be a variable which is
used for the id object. If there was more then one certificates for a single
host detected, $json_postfix carries the certificate number.
Unit tests need to be fixed -- if possible.
Add fileout() to #965. This commit also contains a change which needs
to be commited before: separation of ``json_prefix`` from ``json_postfix``.
Open issue: sed in openssl x509 statments look GNUish ([ \t]). Needs clarification.
This commit prints the contents of the keyUsage and extended key usage extensions in certificates and checks the public keys in the certificates are not being used in a manner that is inconsistent with these extensions.
This PR is intended to improve the functionality of run_tls_fallback_scsv().
The original goal of this PR was to address servers that support TLSv1.3 when using OpenSSL 1.1.1. That does not seem to be an issue, as using `$OPENSSL s_client` with the `-no_tls1_2` flag results in a TLSv1.1 ClientHello, even if `$OPENSSL` supports TLSv1.3. However, if the server supports TLSv1.3, then a message that says "No fallback possible, TLS 1.2 is the only protocol (OK)" isn't entirely correct.
The main issue this PR fixes is some false positives in servers that do not support TLSv1.2. On a few servers that I tested, the current code incorrectly reports "Downgrade attack prevention NOT supported." Some of the servers only support TLSv1, so it should report that fallback is not possible. Another server supports TLSv1.1 and TLSv1, and it supports fallback protection. In both cases, the current code produces a false positive, since it assumes that TLSv1.2 is supported.
In three different places there is a line that is supposed to check whether the list of ciphers to be tested contains any TLSv1.3 ciphers. This check currently fails if there is only one TLSv1.3 cipher in the list and it is the first cipher in the list. This commit fixes the problem.
This commit adds support for draft 23, which contains 2 changes that are relevant for testssl.sh. It adds a few new values for the signature_algorithms extension and it changes to extension number for the key_share extension from 40 to 51.
With the change in the extension number, it is no longer possible to send a single ClientHello that works for all supported drafts of TLSv1.3. (I tried sending a ClientHello with two key share extensions, 40 and 51, but that didn't work.) So, this commit adds a test to determine_optimal_proto() to determine whether TLSv1.3 is supported and if so whether draft 23 is supported or only some earlier draft (18-22). In subsequent tests, the ClientHello uses the appropriate number for the key share (40 or 51) and specifies the appropriate draft version(s) in the supported_versions extension (either 23 or 18-22). In the case of run_protocols() the test for each draft version uses the appropriate key share extension number so that servers that support both draft 23 and an earlier draft can be detected.
support. If the server doesn't supply an session identifier the file
is just empty.
This commit fixes that by adding a separate case for OpenSSL 1.1.1
and an empty file. It is deliberately only changing this as this
was tested to work.
It is prelimary and a save-the-work-patch as it might be better
to catch this earlier.
For JSON pretty the host specific parameters target host + port
could be better placed in the scanResult object.
It is still under discussion as logically the parent object is deduced
the command line.
TLS_RSA_* which don't fall into the aleady mentioned
categories (CBC cipher, export, RC4 etc.) are now
a bit more more penalized. Those are the ones which have
an RSA key exchange AND a modern encryption.
pr_cipher_quality() needs to be redone after carefully
reconsidered which cipher should have which rating.
This PR is similar to #944. If using OpenSSL 1.1.1 to connect to a server that supports TLSv1.3, `run_crime()` will connect to the server using TLSv1.3, which does not support TLS-level compression. So, the server will be reported as "not vulnerable" even if would use compression for connections at TLSv1.2 and below.
I have not encountered any "live" servers that support both TLSv1.3 and TLS-level compression. I verified this problem by using OpenSSL 1.1.1 to create a server that supports both TLSv1.3 and TLS-level compression:
```
openssl111 s_server -cert cert.pem -key key.pem -accept 8443 -WWW -comp
```
I then tested the server using `testssl.sh --crime` with both openssl111 and OpenSSL 1.0.2-chacha.
run_renego() appears to produce a false positive if OpenSSL 1.1.1 is used and the server being tested supports TLSv1.3 (i.e., the server supports the same draft version of TLSv1.3 as the version of OpenSSL 1.1.1 being used does). This PR fixes the problem by telling calls to $OPENSSL s_client in run_renego() to not use TLSv1.3.
.. to check during the default run for server implemenation bugs
and run cipher per procol check instead of cipher check.
Please not that this option could disappear later.
In TLSv1.3 servers may send a supported_groups extension, which "SHOULD contain all groups the server supports, regardless of whether they are currently supported by the client."
This PR extracts the contents of the supported_groups extension, if `parse_tls_serverhello()` is to process "all" of the server's response. The contents of the extension are also displayed on the terminal if $DEBUG -ge 3.
In TLSv1.2 and below, servers respond to a status_request extension (a request for a stapled OCSP response) by returning an empty status_request extension and then including a CertificateStatus message, which follows the Certificate message. In TLSv1.3 the CertificateStatus response is included as the value of the status_request extension, which now appears as an extension within the Certificate message.
This PR extracts the contents of the status_request extension sent by the server so that it can later be processed in the same way as if it had sent in a TLSv1.2 or below response.
This PR adds code to decrypt the encrypted portion of the server's response for TLSv1.3 and to then process any certificates and encrypted extensions. This code supports all 5 TLSv1.3 cipher suites, and so any response can be decrypted as long as the session key can be derived (which requires OpenSSL to support the ephemeral key that was used - see #938).
For the symmetric decryption, the sym-decrypt() function uses OpenSSL when possible and internal Bash functions when needed.
For AES-GCM and AES-CCM ciphers sym-decrypt() normally uses internal Bash functions, which rely on using "$OPENSSL enc" in AES-ECB mode to generate the key stream and then Bash functionality to XOR the key stream with the ciphertext. With some version of OpenSSL the AES-GCM ciphers are decrypted using "$OPENSSL enc" in AES-GCM mode directly. On my system, however, both methods seem to work about equally fast.
For ChaCha20 ciphers, "$OPENSSL enc -chacha20" is used, if supported (OpenSSL 1.1.x only). and Bash internal functions (without any OpenSSL support) are used otherwise. In this case, if the Bash internal functions need to be used, decryption is very, very, very slow. Fortunately, in a typical run of testssl.sh there won't be many cases in which the connection will be TLSv1.3 with ChaCha20 and the entire response needs to be processed (requiring decryption). In most cases, even if the connection is TLSv1.3 with ChaCha20, will at most need the ephemeral key, which is available in plain text.
This is the first in a series of PRs to add support for processing the encrypted portions of the server's response in a TLSv1.3 handshake.
This PR adds the code to derive the handshake traffic key needed to decrypt the response (the next PR will add the code to perform the symmetric-key decryption of the encrypted portions of the response).
Since this PR does not make use of the traffic key that it derives, it doesn't yet add any new functionality.
Note that testssl.sh will not always be able to derive the session keys. If the version of OpenSSL that is bundled with testssl.sh is used and the server chooses to use an X25519 ephemeral key, OpenSSL will be unable to perform the shared secret in derive-handshake-traffic-secret(). (OpenSSL 1.1.0 supports X25519.) Since X25519 use a different encoding than ECDH keys, the lack of X25519 support will be discovered in parse_tls_serverhello() when $OPENSSL pkey is unable to convert the key from DER to PEM. So, in debugging mode, parse_tls_serverhello() now displays a warning if it receives a key share that $OPENSSL pkey cannot handle.
get_server_certificate() uses an awk script to extract the certificates from the output of OPENSSL s_client and it then uses the following line to determine how many certificates were found:
nrsaved=$(count_words "$(echo level?.crt 2>/dev/null)")
If $nrsaved is 0, then get_server_certificate() returns 1 (indicating failure); otherwise it returns 0 (indicating success).
However, the check for the number of certificates returned doesn't work if no certificates were found, as nrsaved will be set to 1 if no certificates were found:
> touch level0.crt
> echo level?.crt
level0.crt
> touch level1.crt
> echo level?.crt
level0.crt level1.crt
> rm level0.crt level1.crt
> echo level?.crt
level?.crt
This PR fixes the problem by first checking that level0.crt exists (-s is used instead of -e, since an empty file wouldn't have a certificate).
Similar to the recently added HAS_PKUTIL (f829878a43), this PR adds HAS_PKEY, which indicates whether OpenSSL has the pkey utility. HAS_PKEY is then checked before attempting to do something that requires the pkey utility.
POP3 STARTTLS handshakes were often unsuccessful as
a regex wasn't properly escaped.
Furthermore if a STARTTLS handshake doesn't succeed, there's
a warning now.
