With OpenSSL 1.1.0 (and maybe other versions), the `ciphers` function lists many cipher suites that are not actually supported by the `s_client` option. This PR fixes that by using the `-s` option whenever `$OPENSSL ciphers` is used to obtain a list of cipher suites supported by OpenSSL. According to https://www.openssl.org/docs/manmaster/man1/ciphers.html:
```
-s
Only list supported ciphers: those consistent with the security level, and minimum and
maximum protocol version. This is closer to the actual cipher list an application will
support.
```
When the `-s` option is used along with `-tls1`, OpenSSL 1.1.0 will not list any ciphers that only work with TLSv1.2. So, `prepare_debug()` needed to be changed to correctly populate `ossl_supported_tls`, which is supposed to be a list of all non-SSLv2 ciphers supported by the server.
LibreSSL issues an "unknown option" error if the `-s` option is provided, so the `-s` option is only included in the command line if `$OPENSSL` has been determined to support it.
This PR is needed so that `prepare_debug()` can correctly determine which cipher suites are or are not supported by `$OPENSSL`.
This PR removes an extra call to `$OPENSSL s_client` in `get_server_certificates()` and it also changes `get_server_certificates()` to not collect extensions when SNI is not being provided.
This PR modifies get_server_certificate() to use tls_sockets() rather than $OPENSSL for finding certificates using SSLv3 - TLSv1.2, unless $SSL_NATIVE is true. Using tls_sockets() allows testssl.sh to find certificates used by the server even if the server is only using cipher suites not supported by $OPENSSL. This may happen, for example, if the server only supports TLS_ECDHE_ cipher suites with curve X25519 and a version of OpenSSL prior to 1.1.0 is being used. A less likely possibility would be if the server had a certificate with a DH key, and a newer version of OpenSSL that does not support TLS_DH_ cipher suites is being used.
Since tls_sockets() cannot be used to obtain session tickets from the server, an additional test for session ticket lifetime needed to be added.
In order to reduce the number of times the server needs to be queried for certificates, this PR bundles the testing in a similar way to what is already done to test for cipher suites. Currently, each call to get_server_certificate() only tests for one type of certificate. This PR has each call test for more than one type of certificate. For example, one call is made to test for ECDSA, ECDH, DH, DSA, and GOST certificates. If the test is unsuccessful, then the server has none of these certificates. If the test finds a certificate (e.g., an ECDSA) certificate, then another test is run looking for the remaining types (ECDH, DH, DSA, and GOST) until a test is unsuccessful.
For most servers, this will reduce the number of calls to get_server_certificate() from 8 or 9 to 4 or 6.
This fixes#1157.
* Move IDEA to the same category as 3DES
* Rename the category to 3DES_IDEA (JSON)
* Rename 128 Bit category to AVERAGE (JSON)
* Move 256 Bit CBC ciphers into this category too
* Remove category HIGH
Furthermore:
* Code readability improvements, especially in run_cipherlists()
* fix minor bugs (e.g. aNULL ciphers were used in higher categories when --ssl-native was supplied)
* rearrange order for sub_cipherlists()
* proper documentation for arguments of sub_cipherlists() in run_cipherlists()
* add "$cve" "$cwe" arguments to fileout in sub_cipherlists() -- (was passed before but not used)
* change debugging leftover filenames for sub_cipherlists to the JSON identifier
This PR makes a few improvements to run_server_defaults() when run on an SSLv2-only server.
First, it uses sslv2_sockets() to test the server rather than $OPENSSL, so that it will work even if $OPENSSL does not support SSLv2.
Second, it changes run_server_defaults() to only call get_server_certificate() once if $OPTIMAL_PROTO is -ssl2, since calling more than once is a waste - SSLv2 only supports ciphers that use RSA key exchange.
Finally, as some code assumes that $TEMPDIR/intermediatecerts.pem will exist, even if it is empty, this PR changes a couple of places that delete $TEMPDIR/intermediatecerts.pem to instead make the file empty.
When run_server_preference() is run on a server that only supports SSLv2 it incorrectly reports that the server has a cipher order. The reason for this is that $list_fwd and $list_reverse only include one SSLv2 cipher.
In SSLv2 the server sends a list of all ciphers it supports in common with the client and the client chooses which cipher to use. As a result, the server cannot enforce a cipher order for SSLv2.
So, this PR fixes the problem in run_server_preference() by skipping the test for whether the server enforces a cipher order if $OPTIMAL_PROTO is -ssl2 and simply declares that the server does not enforce a cipher order.
Note that this PR is somewhat dependent on #1194, as #1194 needs to be applied in order for $OPTIMAL_PROTO to be set to -ssl2 when testing an SSLv2-only server.
This PR reorganizes run_cipher_per_proto(). Currently run_cipher_per_proto() runs a for loop, which loops over each protocol and prints the set of supported ciphers for each protocol. This PR simply places the body of the for loop in a separate function from the loop itself. This allows the body of the loop to be called for just a single protocol.
While this PR does not change the way that testssl.sh functions, it would allow for a future change in which run_server_preferences() called cipher_pref_check() for protocols in which the server enforces a cipher order and calls ciphers_by_strength() for protocols in which the server does not enforce a cipher order.
This PR reorganizes cipher_pref_check(). Currently, cipher_pref_check() runs a for loop, which loops over each protocol and prints the set of supported ciphers for each protocol. This PR simply places the body of the for loop in a separate function from the loop itself. This allows cipher_pref_check() to be called for just a single protocol rather than for all protocols. Another PR will make a similar change to run_cipher_per_proto().
The reason for this change is that cipher_pref_check() was only intended to be used in cases in which the server enforces a cipher preference order. Some servers, however, enforce an order for some protocols, but not for others. The change in this PR will make it possible in the future to call cipher_pref_check() only for protocols in which the server enforces a cipher order.
This PR fixes two bugs in determine_optimal_proto().
First, sslv2_sockets() returns 3 if the connection was successful.
Second, if all connection attempts using tls_sockets() were unsuccessful, it is possible that $TEMPDIR/$NODEIP.parse_tls_serverhello.txt will not exist, so copying it or grepping it will lead to an error. Checking that $proto is not 22 will fix this as $proto will be empty is $OPENSSL s_client was used and it will be 00, 01, 02, 03, or 04 if tls_sockets() was used and the connection was successful with some protocol higher than SSLv2.
This PR fixes a bug in get_cipher() - one that also appears in sclient_connect_successful().
The code currently assumes that cipher names contain only uppercase letters and numbers. However, ciphers that do not provide authentication include "anon" in the name, which is written in lowercase.
This PR fixes the problem by allowing lowercase letters to appear in cipher names (except in the first portion of the name).
Note that no change was made to similar code in get_protocol(), since the line in get_protocol() only matches TLSv1.3 ciphers, which do not contain any lowercase letters.
In case where the OpenSSL version used cannot successfully do openssl s_client
connects there are a few problems, see #1087.
This PR partly addresses them by
* changing the logic of HTTP header failure: we don't terminate anymore but
continue with a warning message
* we try to find out what the reason was: If it is a missing curve we signal
it back to the user
* we keep track in a global variable KNOWN_OSSL_PROB. It's not being used yet
on all connects as it has not been decided whether we do a connect despite
we know if there's a problem or rather not.
* Give hints to the user for resumption tests, secure renegotiation, CRIME and BREACH.
For the latter --assume-http needs to be supplied for any output.
Also: for finding the OPTIMAL_PROTO now (unless --ssl-native is being used)
sockets are the default which removes in cases where an openssl s_client
connect fails, the initial message 'doesn't seem to be a TLS/SSL enabled server'
and prompt 'Really proceed ? ("yes" to continue)'. For STARTTLS this needs
to be done as well.
Here a minor bug was fixed: when openssl s_client connect in determine_optimal_proto()
succeeded without a protocol supplied, OPTIMAL_PROTO wasn't set. A statement was
added but now it is only being used when --ssl-native was supplied.
Leftover for this workaround is to find out why the number of certificate retrieved is
zero in those cases, despite the fact that there's a valid 'host_certificate.pem' from
tls_socket() calls. Thus still run_server_defaults() stops after 'TLS clock skew'
as certificate_info() is not being called in run_server_defaults(). For now in
those cases 'Problem: Host certificate found but we can't continue with "server defaults"'
is being printed.
In general for the future it would be great if we could e.g. retrieve the header over
TLS sockets.
This PR fixes two problems with modify_clienthello().
First, the function was incorrectly using the variable $key_share instead of $new_key_share. Since $key_share is defined when modify_clienthello() is called from resend_if_hello_retry_request(), but not when it is called from client_simulation_sockets(), this bug does not seem to result in incorrect behavior, but it should still be fixed.
Second, when this function is used to create a second ClientHello in response to a HelloRetryRequest, it removes the key_share extension from the original ClientHello and then appends the new key_share extension at the end. According to https://mailarchive.ietf.org/arch/msg/tls/8ZKCyamcYFaV90h6nf4MUnSPkEE, however, extensions must appear in the second ClientHello in the same order in which they appeared in the first ClientHello.
I am not aware of any servers that will actually complain if the extensions in the second ClientHello do not appear in the same order as in the first ClientHello, bug this fix helps to ensure that the ClientHello messages testssl.sh sends are in compliance with the standard.
This PR is an attempt at addressing #1185. According to https://www.etsi.org/deliver/etsi_ts/103500_103599/10352303/01.01.01_60/ts_10352303v010101p.pdf, if eTLS is in use, then the certificate should contain a subjectAltName extension with one or more "names" containing "visibility information." The "visibility information" is encoded as an otherName with a type-id of 0.4.0.3523.3.1 and a value of
VisibilityInformation ::= SEQUENCE {
fingerprint OCTET STRING (SIZE(10)),
accessDescription UTF8String }
The etsi_etls_visibility_info() function determines whether the certificate includes an "visibility information," and, if it does, extracts the fingerprints and access descriptions.
This PR is a work-in-progress for two reasons. First, it has not been tested against any real certificates that contain "visibility information." Testing against real certificates would be helpful to verify that the parsing of the certificate is correct.
Second, the presentation of the visibility information (both in the printed text and in what is sent to fileout()) may need improvement. Having seen no examples, it is not clear what the contents of accessDescription can be expected to look like. The document says that the contents will be "human-readable text," but it is not clear whether the description will be relatively short or very long.
To finalize #1157 following was done
* move 3DES one line above
* put 128 bit CBC ciphers (ARIA, Camellie and AES) SEED and IDEA into 128Bit
* the remaining 256 bit ciphers NOT supporting AEAD Mac into high
Also 128 bit ciphers are getting a small complaint (LOW, yellow) if
available instead of red (for SEED and IDEA before)
To ease the (future) output rated_output() is included, but it's not being used
yet.
Also often I have docker hosts for testing. If I use them while the external
network is down, I still experience DNS timeouts. I added for dig
timeout values which proved to be reasonable in my tests with and
without network.
Also if an IPv4 or an IPv6 address was supplied testssl.sh doesn't
do (futile) DNS lookups anymore.
... to enable checks whether a curve has been detected by sockets
won't be detected and/or makes problems with remaining openssl
s_client + other calls
Related to #1087
This PR addresses the problem (#1037) that if a hostname resolves to multiple IP
addresses only the first one is being scanned as this IP isn't reachable and the
scan terminated here and didn't continue with the 2nd, 3rd etc.
Same applied to scans with --mx.
This based on the global MULTIPLE_CHECKS which is set to true whenever
such a scan is started.
One minor point: Also if the last IP isn't reachable the output will say
"proceeding with next:". I guess that should be clear looking at the (different outputs).
This PR also fixes a scan problem with a single cipher/pattern (-x/--single-cipher):
previously where only one IP was scanned.
Furthermore some redundant quotes were removed.
This commit addresses #934. It adds a line in the log/csv output
and a json object named "scanProblem" when either the function
fatal() is being called and it logs the reason of the fatal error.
.. based on https://github.com:/drwetter/openssl-1.0.2.bad
plus the patch fedora-dirk-ipv6.patch applied.
In addition to Peter Mosman's branch the brauch and as a result the binaries
contain following additional STARTTLS features: LMTP, NNTP, IRC .
See also #741, #559, #1093, #179
As noted in #1157 the logic of different cipher suites put to categories
needed an improvement.
This commit addresses it by moving first RC2 and RC4 cipher suites to
the low category.
More to follow.
This commit fixes#1163 which lead to the misleading output when
a TLS 1.3 enabled server had no preferences for the TLS 1.3 ciphers
but for anything below (like currently for testssl.NET).
The TLS 1.3 handshake in sockets plus the following openssl handshake
was moved to the top in run_server_preference() so that it can be better
determined whether TLS 1.3 is available. If this section's outcome is
TLS 1.3 is negotiated a single TLS 1.3 handshake with 5 ciphers only is
done forward and reverse. The resulting ciphers are later on compared
whether there's a cipher order for TLS 1.3.
Basically this section should be redone, so that all openssl handshakes
are replaced by sockets. As this would consume more time as it appears
reasonable at this point of time, this was not done yet. A starting point
for this would be tls13_list_fwd + reverse. After release of 3.0 90%
of the code will be replaced anyway.
DHE-RSA-SEED-SHA and SEED-SHA was added to the reverse and forward lists
as some old openssl versions + apache use it.
Also:
Googles ALPN_PROTO grpc-exp was added (to be reconsidered at some certain point)
Some redundant quotes in double square brackets were removed.
All "do_*" variables are now in quotes when tested w if or [[
The HTML manual is now post processed through tidy
which removes the problem of ">" not HTML encoded.
--color 0 is now explicitly mentioned to avoid escaped codes in the
output.
Minor changes wrt certificate stores
Reflect we're at 3.0rcX, 2.9.5 is past, 2.8 not supported
Removed features from 2.9.5. Added all(?-->David?) features implemented
in 2.9dev.
Mention docker image
Clarify when testssl.sh should be mentioned (license).
This is an update of the root certificate stores. Date from each store
is from yesterday.
Description update.
Also the Java certificate store was added. Previously Java was omitted
as it appeared not to be complete. I tested successfully this store.
Review: grammar, spelling. Errorneous and obsolete description.
Some items reordered.
Updated: to reflect the current capabilities.
Moreover: (Almost) complete the tuning variables section.
This addresses #1169: When using JSON as output format when mass testing
AND we have a non-fatal condition when e.g. openssl lacks support for
something it led to an invalid JSON as the warning was put into file w/o
a trailing comma.
The commit removes the warning to be put into the output. We still have the
message on screen + in HTML which is not as optimal as it could be.
Also I did some cleanups related to redundant double quotes I stumbled over while
fixing this.
David Cooper
Dirk
David Cooper