At the moment the code for downloading a CRL seems to only work if URL is an HTTP or HTTP URL. It fails if the URL is an LDAP URL. The wget command does not support LDAP and when curl retrieves data from an LDAP URL it stores the result in LDIF format, which http_get() cannot currently convert into a PEM-encoded CRL.
This PR addresses the issue by skipping the revocation check for any URL that does not begin with "http".
In general, a CA only needs to keep the status information for a certificate until it expires. So, once a certificate has expired, the information provided about it in a CRL or OCSP response may no longer be reliable. The certificate may no longer be listed as revoked, even it is had been revoked at some point before it expired.
So, this PR changes certificate_info() to only check CRLs for revocation status if the certificate has not expired.
In order to use it one has to use --phone-out (PHONE_OUT
is the respective ENV) like
``./testssl.sh --phone-out --json-pretty -S wikipedia.org``
This makes use of curl (if available) or wget (if available) and
falls back to bash socket GET. The latter uses HTTP/1.0 as
chunked transfers by the server (used for bigger files normally)
can't be reasonably separated from their HTTP header. (HTTP/1.0
doesn't support chunked transfers).
curl and wget use the enviroment variables automatically. Probably
we want to use those proxies only if told by a switch to testssl.sh.
"-crl_download" would have been an option. Support would have
been needed to check beforehand. Alos information on proper
usage seems limited, so for now a solution which works is
preferred.
Open/to be clarified:
* Documentation
* Proxy for curl / proxy needs to come from testssl.sh
* Proxy support for HTTP bash socket GET
* JSON ID is cert_CRLrevoked_ (trailing underscore)
* cert_CRLrevoked_ comes before cert_cRLDistributionPoints
(* reconsider naming of cert_cRLDistributionPoints)
* Unit tests
Still open: OCSP
This PR was developed in response to #845. It adds to the list of ciphers used to determine whether the server has a cipher order in order to help avoid cases in which testssl.sh cannot determine a cipher order.
In order to create this list I scanned thousands of servers in order to determine what ciphers they support, including (1) about 20 thousand U.S. government web sites, (2) all of the sites listed at badssl.com, (3) all of the test servers listed at https://github.com/tlswg/tls13-spec/wiki/implementations, (4) about 30 additional non-U.S. government sites, and (5) one server configured as described in #845. I scanned each of these servers using OpenSSL 1.0.2-chacha, 1.0.2o, and 1.1.1.
Then I ran collection information through a script that created the updated list. For each scanned server, and for each of the 3 versions of OpenSSL, the script checked whether $list_fwd contained at least two ciphers from the list. If it didn't, then it would add one of the ciphers supported by the server (and by OpenSSL) to the list. In choosing among the ciphers supported by the server that were not already in $list_fwd, it would choose the cipher that was supported by the most other servers.
The list contain a few oddities as a result of the servers that I scanned. The script added two TLSv1.3 ciphers, since I scanned at least one server that only supports TLSv1.3. The list also includes ADH-AES256-GCM-SHA384 and AECDH-AES128-SHA, which may only be supported by null.badssl.com.
I made one manual change to the list - adding TLS_CHACHA20_POLY1305_SHA256. I did this since the number of TLSv1.3 servers scanned was so small, I didn't think it was safe to assume that all servers that support TLSv1.3 would support both TLS_AES_256_GCM_SHA384 and TLS_AES_128_GCM_SHA256.
Since most of the servers that I scanned were U.S. government servers, it may not be a representative sample. However, since the new list only adds to the current list, it can only be an improvement. Also, the updated list still only includes 37 ciphers, so many more could be added without creating any problems.
As it would be a possible privacy violation a new flag PHONE_OUTSIDE
is introduced (later accompanied by a switch). It determines whether
the client is allowed to retrieve the CRL specified (HTTP only supported).
Tested ok against wikipedia.de and revoked.badssl.com.
To do:
* look into -crl_download
* fileout
* Unit tests
OCSP verification
There is currently a problem if mass testing is being performed, JSON and/or CSV output is to be produced, the parent process calls fileout(), and each child process have its own output file for the JSON and/or CSV output. The can be seen, for example, with the following:
testssl.sh --openssl=openssl_1.1.1 --file test_servers.txt --csvfile output_dir --jsonfile output_dir
A call will be made in the parent process to report that openssl_1.1.1 has "No engine or GOST support via engine." fileout() will try to write to output_dir, which will result in an error.
This PR fixes the problem by checking the the file to be written to is not a directory (as is already done in html_out() for HTML output).
In certain situations while testting for CCS injection it could have happened
that an error code was sent which was not interpreted properly by testssl.sh.
(https://tools.ietf.org/html/rfc5246#section-7.2)
This has now been fixed and thus addresses #906. Also it has been made sure
that other error codes are reported appropiately.
The case where this test failed before was a non-patched Ubuntu 12.04
with openssl/postfix on port 25.
For the upcoming release this commit initiated the beta phase: important features
will be allowed. On the agenda is otherwise to fix bugs.
I ran shellcheck (see #434), and fixed some complaints and adjusted some coding
style mismatches.
There were some cases where security headers were served two times by the
server. The result (screen+html) wasn't properly formatted in those cases.
match_httpheader_key() was improved so that it keeps track when
a CR or an indentation needs to be done.
Some egrep statements were replaced by grep -E as this has been used
already and it is the thing testssl.sh should settle for. (precursor
to #1022).
run_more_flags was renamed to sun_security_headers and names of
variables is better.
HAS_SPDY is now HAS_NPN (similar to renaming the function a while
back)
mktemp should only be used when not avoidable (performance, code). For
temporarily local variables names can often be borrowed from globals
which were already generated by mktemp (SOCK_REPLY_FILE).
This is a fix for #722. It updates the client simulation data from
the SSLlabs API. As usual data was pulled, resorted and clients
to display were hand-selected.
Wishlist: Missing is Oreo, OpenSSL 1.1.1, Safari on OX 11, Firefox
52.x (ESR)
With the recent PR #1033 from @dcooper it can also show TLS 1.3
handshakes.
https://api.dev.ssllabs.com/api/v3/getClients incorrectly indicates a highestProtocol of 771 (TLS 1.2) for clients that support TLS 1.3, which leads run_client_simulation() to incorrectly report "no connection" if the client would have actually connected using TLS 1.3.
This has been addressed by manually editing etc/client-simulation.txt to set the highest_protocol to 0x0304 for the clients that support TLS 1.3.
This PR modifies update_client_sim_data.pl to automatically apply the fix for clients that support TLS 1.3 in order to avoid a possible regression when etc/client-simulation.txt is updated.
b2be380b54 inadvertently changed MASS_TESTING_CMDLINE to be a read-only variable. This causes mass testing to fail, since in mass testing the value of MASS_TESTING_CMDLINE is set to the command line for each child test.
According to programming standards e.g. C-style defines) testssl.sh has now
internal error variables (ERR_*) which are defined to deal with exit codes in
error conditions. Details see ``testssl.sh(1)``, section exit. Thus exit codes
because of an error are now standardized and if needed can be easily changed to
other values.
This is part of a cleanup mentioned in #985 and #752. Codes for monitoring
tools (#327) which imply some kind of rating are still to be done.
The beginning section was reformatted and some items were reordered to keep
variables and functions together which serve similar purposes.
``readonly`` was replaced by ``declare -r`` (closer to C's define and it
makes more sense to settle on one variable if both are being used
for the same purpose)
... from "Further IP addresses" and before calling ``get_aaaa_record``
in ``determine_ip_addresses()``. Logic appeared needlessly to difficult
and was as far as the "Further IP addresses" line was concerned incomplete.
This PR changes the logic the no-DNS switch works. The switch
now expects a value. "min" does minimum lookups, "none" does
no lookups at all (details see testssl.sh(1) ). "none" is
equivalent to the paranoid (boolean) value "true" before.
When performing client simulations in "--ssl-native" mode, provide the client's list of supported curves to "$OPENSSL s_client" in order to make the results even more accurate.
This PR improves client simulation in "--ssl-native" mode:
* It changes ${protos[i]} to list the protocols that should be disabled rather than those that should be enabled, except in the case that the client only supports one protocol.
* It sets the values for ${tlsvers[i]}, which is used in run_client_simulation(), but was not defined.
* It adds a new variable, ${ciphersuites[i]}, that lists the TLSv1.3 cipher suites supported by a client.
Client simulation still produces false results in "--ssl-native" mode, but the results are better than before.
This PR fixes three issues related to the testing for RFC 7919 DH groups in run_pfs():
* If the RFC 7919 DH groups are supported for both TLSv1.3 cipher suites and non-TLSv1.3 cipher suites, then the list of supported groups is printed twice.
* The finding that is used for CSV/JSON files includes the word "offered" after the list of groups, which is inconsistent with other findings.
* Since the $ffdhe_offered is only used to determine whether to test for use of RFC 7919 DH groups with non-TLSv1.3 ciphers, this flag should only be set if a non-TLSv1.3 ciphers that uses ephemeral DH is found.
If OpenSSL reports an error, sclient_connect_successful() may incorrectly interpret it as a connectivity problem, leading testssl.sh to stop testing before it has completed.
When not using "--ssl-native" mode, this happens if $OPENSSL does not support SSLv3, as both get_server_certificate() and run_beast() will attempt to connect using SSLv3 even if $OPENSSL does not support it.
When using "--ssl-native" mode, this happens in multiple places if $OPENSSL does not support the protocol being used or if $OPENSSL does not support any of the ciphers that are specified in the command line.
This PR fixes the above problems by adding checks for protocol support or for support for at least one cipher before calling $OPENSSL.
run_client_simulation() also has a problem in "--ssl-native" mode of calling $OPENSSL with parameters that cause $OPENSSL to report an error, but this is already addressed by temporarily setting MAX_OSSL_FAIL to 100 during client simulation tests and then, after client simulation testing is complete, returning $MAX_OSSL_FAIL and $NR_OSSL_FAIL to the values they had before client simulation testing began.
``run_logjam()`` contained in certain cases additional quotes
and a typo where only the word comment ended up in JSON/CSV.
Instead of ``$(awk '/Master-Key: / { print $2 }' "$2")`` the
admittedly performance sensitive function ``sclient_connect_successful()``
contains now a bash internal match according to #997 . First
tests didn't show much benefit (only default run with mostly
sockets was tested).
Unit tests showed no problems so far, but coverage is low.
See previous commit
This commit finally fixes#1005 so that either a --ssl-native scan
terminates on the next (defined) occasion if there are network connectivity
problems. It introduces another set of variables (MAX_OSSL_FAIL vs. NR_OSSL_FAIL).
As "openssl s_client connect" is sometimes still being used without --ssl-native
it also shortens the wait for regular scans if an outage is encountered.
To make things easier bot sets (incl. *_SOCKET_FAIL) of variables are independent.
For the seldom case that somebody uses --ssl-native with client checks an exception
had to be made as otherwise only MAX_OSSL_FAIL client check would be performed.
This hasn't been understood yet...
As sometimes HTTP header requests (over OpenSSL) fail repeatedly in a way that an empty
reply is returned, the same strategy of detecting problems is applied here,
using MAX_HEADER_FAIL and NR_HEADER_FAIL.
All three detection mechanisims share a new function connectivity_problem().
This commit finally fixes#1005 so that either a --ssl-native scan
terminates on the next (defined) occasion if there are network connectivity
problems. It introduces another set of variables (MAX_OSSL_FAIL vs. NR_OSSL_FAIL).
As "openssl s_client connect" is sometimes still being used without --ssl-native
it also shortens the wait for regular scans if an outage is encountered.
To make things easier bot sets (incl. *_SOCKET_FAIL) of variables are independent.
For the seldom case that somebody uses --ssl-native with client checks an exception
had to be made as otherwise only MAX_OSSL_FAIL client check would be performed.
This hasn't been understood yet...
As sometimes HTTP header requests (over OpenSSL) fail repeatedly in a way that an empty
reply is returned, the same strategy of detecting problems is applied here,
using MAX_HEADER_FAIL and NR_HEADER_FAIL.
All three detection mechanisims share the new function connectivity_problem().
In addition unit tests showed that some vulnerability checks lost their
CVEs+CWEs whcich have been readded. For ROBOT a CVE was added (F5)
parse_tls_serverhello() checks $TLS_CLIENT_HELLO for a supported_versions extension, and if it contains one, checks that the negotiated version is listed in that extension. However, while $TLS_CLIENT_HELLO is always set in socksend_tls_clienthello() it is not set by client_simulation_sockets() (or any of the functions that client_simulation_sockets() calls). As a result, when the server's response to a client simulation is parsed, parse_tls_serverhello() may compare the negotiated version against the supported_versions extension from a ClientHello message from a previous test.
This PR fixes the problem by having client_simulation_sockets() set $TLS_CLIENT_HELLO.
David Cooper
Dirk Wetter
Karsten Weiss