This PR reorganizes run_cipher_per_proto(). Currently run_cipher_per_proto() runs a for loop, which loops over each protocol and prints the set of supported ciphers for each protocol. This PR simply places the body of the for loop in a separate function from the loop itself. This allows the body of the loop to be called for just a single protocol.
While this PR does not change the way that testssl.sh functions, it would allow for a future change in which run_server_preferences() called cipher_pref_check() for protocols in which the server enforces a cipher order and calls ciphers_by_strength() for protocols in which the server does not enforce a cipher order.
This PR reorganizes cipher_pref_check(). Currently, cipher_pref_check() runs a for loop, which loops over each protocol and prints the set of supported ciphers for each protocol. This PR simply places the body of the for loop in a separate function from the loop itself. This allows cipher_pref_check() to be called for just a single protocol rather than for all protocols. Another PR will make a similar change to run_cipher_per_proto().
The reason for this change is that cipher_pref_check() was only intended to be used in cases in which the server enforces a cipher preference order. Some servers, however, enforce an order for some protocols, but not for others. The change in this PR will make it possible in the future to call cipher_pref_check() only for protocols in which the server enforces a cipher order.
This PR fixes two bugs in determine_optimal_proto().
First, sslv2_sockets() returns 3 if the connection was successful.
Second, if all connection attempts using tls_sockets() were unsuccessful, it is possible that $TEMPDIR/$NODEIP.parse_tls_serverhello.txt will not exist, so copying it or grepping it will lead to an error. Checking that $proto is not 22 will fix this as $proto will be empty is $OPENSSL s_client was used and it will be 00, 01, 02, 03, or 04 if tls_sockets() was used and the connection was successful with some protocol higher than SSLv2.
This PR fixes a bug in get_cipher() - one that also appears in sclient_connect_successful().
The code currently assumes that cipher names contain only uppercase letters and numbers. However, ciphers that do not provide authentication include "anon" in the name, which is written in lowercase.
This PR fixes the problem by allowing lowercase letters to appear in cipher names (except in the first portion of the name).
Note that no change was made to similar code in get_protocol(), since the line in get_protocol() only matches TLSv1.3 ciphers, which do not contain any lowercase letters.
In case where the OpenSSL version used cannot successfully do openssl s_client
connects there are a few problems, see #1087.
This PR partly addresses them by
* changing the logic of HTTP header failure: we don't terminate anymore but
continue with a warning message
* we try to find out what the reason was: If it is a missing curve we signal
it back to the user
* we keep track in a global variable KNOWN_OSSL_PROB. It's not being used yet
on all connects as it has not been decided whether we do a connect despite
we know if there's a problem or rather not.
* Give hints to the user for resumption tests, secure renegotiation, CRIME and BREACH.
For the latter --assume-http needs to be supplied for any output.
Also: for finding the OPTIMAL_PROTO now (unless --ssl-native is being used)
sockets are the default which removes in cases where an openssl s_client
connect fails, the initial message 'doesn't seem to be a TLS/SSL enabled server'
and prompt 'Really proceed ? ("yes" to continue)'. For STARTTLS this needs
to be done as well.
Here a minor bug was fixed: when openssl s_client connect in determine_optimal_proto()
succeeded without a protocol supplied, OPTIMAL_PROTO wasn't set. A statement was
added but now it is only being used when --ssl-native was supplied.
Leftover for this workaround is to find out why the number of certificate retrieved is
zero in those cases, despite the fact that there's a valid 'host_certificate.pem' from
tls_socket() calls. Thus still run_server_defaults() stops after 'TLS clock skew'
as certificate_info() is not being called in run_server_defaults(). For now in
those cases 'Problem: Host certificate found but we can't continue with "server defaults"'
is being printed.
In general for the future it would be great if we could e.g. retrieve the header over
TLS sockets.
This PR fixes two problems with modify_clienthello().
First, the function was incorrectly using the variable $key_share instead of $new_key_share. Since $key_share is defined when modify_clienthello() is called from resend_if_hello_retry_request(), but not when it is called from client_simulation_sockets(), this bug does not seem to result in incorrect behavior, but it should still be fixed.
Second, when this function is used to create a second ClientHello in response to a HelloRetryRequest, it removes the key_share extension from the original ClientHello and then appends the new key_share extension at the end. According to https://mailarchive.ietf.org/arch/msg/tls/8ZKCyamcYFaV90h6nf4MUnSPkEE, however, extensions must appear in the second ClientHello in the same order in which they appeared in the first ClientHello.
I am not aware of any servers that will actually complain if the extensions in the second ClientHello do not appear in the same order as in the first ClientHello, bug this fix helps to ensure that the ClientHello messages testssl.sh sends are in compliance with the standard.
This PR is an attempt at addressing #1185. According to https://www.etsi.org/deliver/etsi_ts/103500_103599/10352303/01.01.01_60/ts_10352303v010101p.pdf, if eTLS is in use, then the certificate should contain a subjectAltName extension with one or more "names" containing "visibility information." The "visibility information" is encoded as an otherName with a type-id of 0.4.0.3523.3.1 and a value of
VisibilityInformation ::= SEQUENCE {
fingerprint OCTET STRING (SIZE(10)),
accessDescription UTF8String }
The etsi_etls_visibility_info() function determines whether the certificate includes an "visibility information," and, if it does, extracts the fingerprints and access descriptions.
This PR is a work-in-progress for two reasons. First, it has not been tested against any real certificates that contain "visibility information." Testing against real certificates would be helpful to verify that the parsing of the certificate is correct.
Second, the presentation of the visibility information (both in the printed text and in what is sent to fileout()) may need improvement. Having seen no examples, it is not clear what the contents of accessDescription can be expected to look like. The document says that the contents will be "human-readable text," but it is not clear whether the description will be relatively short or very long.
To finalize #1157 following was done
* move 3DES one line above
* put 128 bit CBC ciphers (ARIA, Camellie and AES) SEED and IDEA into 128Bit
* the remaining 256 bit ciphers NOT supporting AEAD Mac into high
Also 128 bit ciphers are getting a small complaint (LOW, yellow) if
available instead of red (for SEED and IDEA before)
To ease the (future) output rated_output() is included, but it's not being used
yet.
Also often I have docker hosts for testing. If I use them while the external
network is down, I still experience DNS timeouts. I added for dig
timeout values which proved to be reasonable in my tests with and
without network.
Also if an IPv4 or an IPv6 address was supplied testssl.sh doesn't
do (futile) DNS lookups anymore.
... to enable checks whether a curve has been detected by sockets
won't be detected and/or makes problems with remaining openssl
s_client + other calls
Related to #1087
This PR addresses the problem (#1037) that if a hostname resolves to multiple IP
addresses only the first one is being scanned as this IP isn't reachable and the
scan terminated here and didn't continue with the 2nd, 3rd etc.
Same applied to scans with --mx.
This based on the global MULTIPLE_CHECKS which is set to true whenever
such a scan is started.
One minor point: Also if the last IP isn't reachable the output will say
"proceeding with next:". I guess that should be clear looking at the (different outputs).
This PR also fixes a scan problem with a single cipher/pattern (-x/--single-cipher):
previously where only one IP was scanned.
Furthermore some redundant quotes were removed.
This commit addresses #934. It adds a line in the log/csv output
and a json object named "scanProblem" when either the function
fatal() is being called and it logs the reason of the fatal error.
.. based on https://github.com:/drwetter/openssl-1.0.2.bad
plus the patch fedora-dirk-ipv6.patch applied.
In addition to Peter Mosman's branch the brauch and as a result the binaries
contain following additional STARTTLS features: LMTP, NNTP, IRC .
See also #741, #559, #1093, #179
As noted in #1157 the logic of different cipher suites put to categories
needed an improvement.
This commit addresses it by moving first RC2 and RC4 cipher suites to
the low category.
More to follow.
This commit fixes#1163 which lead to the misleading output when
a TLS 1.3 enabled server had no preferences for the TLS 1.3 ciphers
but for anything below (like currently for testssl.NET).
The TLS 1.3 handshake in sockets plus the following openssl handshake
was moved to the top in run_server_preference() so that it can be better
determined whether TLS 1.3 is available. If this section's outcome is
TLS 1.3 is negotiated a single TLS 1.3 handshake with 5 ciphers only is
done forward and reverse. The resulting ciphers are later on compared
whether there's a cipher order for TLS 1.3.
Basically this section should be redone, so that all openssl handshakes
are replaced by sockets. As this would consume more time as it appears
reasonable at this point of time, this was not done yet. A starting point
for this would be tls13_list_fwd + reverse. After release of 3.0 90%
of the code will be replaced anyway.
DHE-RSA-SEED-SHA and SEED-SHA was added to the reverse and forward lists
as some old openssl versions + apache use it.
Also:
Googles ALPN_PROTO grpc-exp was added (to be reconsidered at some certain point)
Some redundant quotes in double square brackets were removed.
All "do_*" variables are now in quotes when tested w if or [[
The HTML manual is now post processed through tidy
which removes the problem of ">" not HTML encoded.
--color 0 is now explicitly mentioned to avoid escaped codes in the
output.
Minor changes wrt certificate stores
Reflect we're at 3.0rcX, 2.9.5 is past, 2.8 not supported
Removed features from 2.9.5. Added all(?-->David?) features implemented
in 2.9dev.
Mention docker image
Clarify when testssl.sh should be mentioned (license).
This is an update of the root certificate stores. Date from each store
is from yesterday.
Description update.
Also the Java certificate store was added. Previously Java was omitted
as it appeared not to be complete. I tested successfully this store.
Review: grammar, spelling. Errorneous and obsolete description.
Some items reordered.
Updated: to reflect the current capabilities.
Moreover: (Almost) complete the tuning variables section.
This addresses #1169: When using JSON as output format when mass testing
AND we have a non-fatal condition when e.g. openssl lacks support for
something it led to an invalid JSON as the warning was put into file w/o
a trailing comma.
The commit removes the warning to be put into the output. We still have the
message on screen + in HTML which is not as optimal as it could be.
Also I did some cleanups related to redundant double quotes I stumbled over while
fixing this.
As a kind of a pre-warning this commit allows the n-1 connection problem to
give feedback on the screen (that wasn't working before).
Also the message on the screen is now more clear and the manpage
gives better advice.
Related to #1172
This PR fixes#1165 by changing resend_if_hello_retry_request() to modify the initial ClientHello rather than having it call prepare_tls_clienthello() to try to generate a new ClientHello that is almost the same as the first. The modification is done using a revised version of create_client_simulation_tls_clienthello(), which is now renamed as modify_clienthello().
Since prepare_tls_clienthello() is no longer used to create a second ClientHello message, argument 7 to that function is no longer needed.
There is at least one extension that will fail on a TLSv1.3 ClientHello if the psk_key_exchange_modes extension is not present (see #990). The PR adds the extension to TLSv1.3 ClientHello messages. OpenSSL, Firefox, and Chrome all include this extension in their ClientHello messages, so including it is unlikely to cause problems for any servers.
.. after some discussion. As TLS 1.3 is not tested here
any RFC 7919 primes using this protocol will not show
up (they in in run_pfs() though). To avoid misunderstandings
" DH key detected with <= TLS 1.2" is now being printed.
This PR provides an additional fix for the issue raised by #1159. It defines a third option for the degree of processing that should be performed by tls_sockets(): "all+". When "all+" is provided, the processing is exactly the same as for "all" with the exception of the creation of the supported_groups extension. For a TLSv1.3 ClientHello, curves that are not supported by $OPENSSL are omitted from the supported_groups extension rather than offering these curves as the least preferred option.
The "all+" option is used in run_server_defaults() where, unlike with almost every other call to tls_sockets(), a successful connection is of no use unless the response can be decrypted. This is also the case for run_alpn(), and so the call to tls_sockets() was also changed to "all+" there. But, the change has no effect at the moment, since run_alpn() sends a TLSv1.2 ClientHello.
As a result of #276, `run_server_defaults()` makes several attempts to find certificates that a server offers if the ClientHello is for TLSv1.2 and no SNI is offered. However, these tests are unnecessary if it is already known that the server does not support TLSv1.1.
This PR modifies `run_server_defaults()` so that the the TLSv1.1-only tests are skipped if the server is known to not support TLSv1.1.
This PR fixes#1159. If tls_sockets() connects to a server using TLSv1.3, it cannot be assumed that the server's certificate is available, as testssl.sh may not have been able to decrypt the server's response. This can happen, for example, if X25519 was used for the key exchange and `$OPENSSL` does not support X25519.
If the connection was successful, but the certificate could not be obtained, then this PR tries again using `$OPENSSL`. However, since `$OPENSSL` does not support TLSv1.3, this will only work if the server supports TLSv1.2 or earlier.
This commit addresses #179 and implements NNTP via STARTTLS. I did
a few tests and it did work so far.
However the binary support needs to be done. I backported in my
fork of @PeterMosmans tree the section from OpenSSL 1.1.1 -- but
it didn't work, see https://github.com/openssl/openssl/issues/7722.
I just tried to patch it as I suggested and it worked then. My
patch is pushed soon after to https://github.com/drwetter/openssl-1.0.2.bad,
however I'll better wait for the official OPenSSL 1.1.1 patch.
David Cooper
Dirk Wetter