Commit Graph

2584 Commits

Author SHA1 Message Date
Dirk 168eec447a Merge branch 'crl_ocsp' into 2.9dev 2018-04-28 22:36:27 +02:00
Dirk 0851599a4e Polish existing CRL revocation check code
* Extra function for ldap_get()
  * Hint when curl is not installed and LDAP URI is encountered
  * Rename jsonID cert_cRLDistributionPoints to cert_crlDistributionPoints
  * Fix trailing _ in jsonID

Open/to be clarified:

   * Proxy for curl / proxy needs to come from testssl.sh
   * Proxy support for HTTP bash socket GET
   * cert_CRLrevoked comes before cert_cRLDistributionPoints
   * Unit tests

Still open: OCSP
2018-04-28 22:25:43 +02:00
Dirk 5ff9fe0f0d Merge branch 'dcooper16-ldap_urls' into crl_ocsp 2018-04-27 21:49:00 +02:00
Dirk c3927d00c8 Document --phone-out 2018-04-27 21:37:44 +02:00
David Cooper b4f76845ce
Support LDAP crLDistrubutionPoints
This PR adds support for retrieving CRLs from an LDAP URL in the cRLDistributionPoints extension if curl if available.
2018-04-26 15:20:53 -04:00
Dirk Wetter da49603c46
Merge pull request #1042 from dcooper16/non_http_crldp
non-HTTP cRLDistributionPoints
2018-04-26 17:08:32 +02:00
David Cooper 0b8409e13a non-HTTP cRLDistributionPoints
At the moment the code for downloading a CRL seems to only work if URL is an HTTP or HTTP URL. It fails if the URL is an LDAP URL. The wget command does not support LDAP and when curl retrieves data from an LDAP URL it stores the result in LDIF format, which http_get() cannot currently convert into a PEM-encoded CRL.

This PR addresses the issue by skipping the revocation check for any URL that does not begin with "http".
2018-04-26 11:03:11 -04:00
Dirk Wetter 6d919de0ad
Merge pull request #1043 from dcooper16/expired_certs
Don't check expired certificates
2018-04-26 16:51:20 +02:00
David Cooper 78cb75543f
Don't check expired certificates
In general, a CA only needs to keep the status information for a certificate until it expires. So, once a certificate has expired, the information provided about it in a CRL or OCSP response may no longer be reliable. The certificate may no longer be listed as revoked, even it is had been revoked at some point before it expired.

So, this PR changes certificate_info() to only check CRLs for revocation status if the certificate has not expired.
2018-04-26 10:23:52 -04:00
Dirk ddf5ff6bc9 Minor additions wrt --color=3 and fname prefix 2018-04-26 09:39:30 +02:00
Dirk Wetter af2f6dd353
Merge pull request #1041 from dcooper16/server_preference_list
Update ciphers for server preference
2018-04-25 23:09:30 +02:00
Dirk c4f42fd6c7 Experimental support for certificate check via CRL
In order to use it one has to use --phone-out  (PHONE_OUT
is the respective ENV) like

``./testssl.sh --phone-out --json-pretty -S wikipedia.org``

This makes use of curl (if available) or wget (if available) and
falls back to bash socket GET. The latter uses HTTP/1.0 as
chunked transfers by the server (used for bigger files normally)
can't be reasonably separated from their HTTP header. (HTTP/1.0
doesn't support chunked transfers).

curl and wget use the enviroment variables automatically. Probably
we want to use those proxies only if told by a switch to testssl.sh.

"-crl_download" would have been an option. Support would have
been needed to check beforehand. Alos information on proper
usage seems limited, so for now a solution which works is
preferred.

Open/to be clarified:

  * Documentation
  * Proxy for curl / proxy needs to come from testssl.sh
  * Proxy support for HTTP bash socket GET
  * JSON ID is cert_CRLrevoked_ (trailing underscore)
  * cert_CRLrevoked_ comes before cert_cRLDistributionPoints
  (* reconsider naming of cert_cRLDistributionPoints)
  * Unit tests

Still open: OCSP
2018-04-25 22:34:47 +02:00
David Cooper d257dae5d5
Update ciphers for server preference
This PR was developed in response to #845. It adds to the list of ciphers used to determine whether the server has a cipher order in order to help avoid cases in which testssl.sh cannot determine a cipher order.

In order to create this list I scanned thousands of servers in order to determine what ciphers they support, including (1) about 20 thousand U.S. government web sites, (2) all of the sites listed at badssl.com, (3) all of the test servers listed at https://github.com/tlswg/tls13-spec/wiki/implementations, (4) about 30 additional non-U.S. government sites, and (5) one server configured as described in #845. I scanned each of these servers using OpenSSL 1.0.2-chacha, 1.0.2o, and 1.1.1.

Then I ran collection information through a script that created the updated list. For each scanned server, and for each of the 3 versions of OpenSSL, the script checked whether $list_fwd contained at least two ciphers from the list. If it didn't, then it would add one of the ciphers supported by the server (and by OpenSSL) to the list. In choosing among the ciphers supported by the server that were not already in $list_fwd, it would choose the cipher that was supported by the most other servers.

The list contain a few oddities as a result of the servers that I scanned. The script added two TLSv1.3 ciphers, since I scanned at least one server that only supports TLSv1.3. The list also includes ADH-AES256-GCM-SHA384 and AECDH-AES128-SHA, which may only be supported by null.badssl.com.

I made one manual change to the list - adding TLS_CHACHA20_POLY1305_SHA256. I did this since the number of TLSv1.3 servers scanned was so small, I didn't think it was safe to assume that all servers that support TLSv1.3 would support both TLS_AES_256_GCM_SHA384 and TLS_AES_128_GCM_SHA256.

Since most of the servers that I scanned were U.S. government servers, it may not be a representative sample. However, since the new list only adds to the current list, it can only be an improvement. Also, the updated list still only includes 37 ciphers, so many more could be added without creating any problems.
2018-04-25 16:21:38 -04:00
Dirk e84d0cf170 First prototype of CRL checker
As it would be a possible privacy violation a new flag PHONE_OUTSIDE
is introduced (later accompanied by a switch). It determines whether
the client is allowed to retrieve the CRL specified (HTTP only supported).

Tested ok against wikipedia.de and revoked.badssl.com.

To do:
  * look into -crl_download
  * fileout
 * Unit tests

OCSP verification
2018-04-24 17:52:01 +02:00
Dirk Wetter 90b671fbb3
Merge pull request #1040 from dcooper16/fix_typo
Fix typo
2018-04-23 21:37:13 +02:00
David Cooper 5f1f7273c8
Fix typo
This PR fixes a typo that was causing a "command not found" error.
2018-04-23 15:14:17 -04:00
Dirk Wetter 1a03e31a7d
Merge pull request #1039 from dcooper16/csv_json_mass_testing
JSON and CSV output problem with mass testing
2018-04-23 18:19:47 +02:00
David Cooper 6538d77781
JSON and CSV output problem with mass testing
There is currently a problem if mass testing is being performed, JSON and/or CSV output is to be produced, the parent process calls fileout(), and each child process have its own output file for the JSON and/or CSV output. The can be seen, for example, with the following:

testssl.sh --openssl=openssl_1.1.1 --file test_servers.txt --csvfile output_dir --jsonfile output_dir

A call will be made in the parent process to report that openssl_1.1.1 has "No engine or GOST support via engine." fileout() will try to write to output_dir, which will result in an error.

This PR fixes the problem by checking the the file to be written to is not a directory (as is already done in html_out() for HTML output).
2018-04-23 11:25:58 -04:00
Dirk 8149c2d5cf CCS injection: better handling of TLS alert protocols
In certain situations while testting for CCS injection it could have happened
that an error code was sent which was not interpreted properly by testssl.sh.
(https://tools.ietf.org/html/rfc5246#section-7.2)

This has now been fixed and thus addresses #906. Also it has been made sure
that other error codes are reported appropiately.

The case where this test failed before was a non-patched Ubuntu 12.04
with openssl/postfix on port 25.
2018-04-20 11:31:05 +02:00
Dirk 39647d1703 Shellcheck, bump version to 3.0beta
For the upcoming release this commit initiated the beta phase: important features
will be allowed. On the agenda is otherwise to fix bugs.

I ran shellcheck (see #434), and fixed some complaints and adjusted some coding
style mismatches.
2018-04-19 17:57:52 +02:00
Dirk cd01351f96 Merge branch '2.9dev' of github.com:drwetter/testssl.sh into 2.9dev 2018-04-19 08:26:57 +02:00
Dirk b8e9b09ca7 FIX #592 (double header)
There were some cases where security headers were served two times by the
server. The result (screen+html) wasn't properly formatted in those cases.

match_httpheader_key() was improved so that it keeps track when
a CR or an indentation needs to be done.

Some egrep statements were replaced by grep -E as this has been used
already and it is the thing testssl.sh should settle for. (precursor
to #1022).

run_more_flags was renamed to sun_security_headers and names of
variables is better.

HAS_SPDY is now HAS_NPN (similar to renaming the function a while
back)

mktemp should only be used when not avoidable (performance, code). For
temporarily local variables names can often be borrowed from globals
which were already generated by mktemp (SOCK_REPLY_FILE).
2018-04-19 08:11:28 +02:00
Dirk e8d7ba547a remove old client_simulation.txt 2018-04-18 21:09:31 +02:00
Dirk c15e0425dc Update client simulation
This is a fix for #722. It updates the client simulation data from
the SSLlabs API. As usual data was pulled, resorted and clients
to display were hand-selected.

Wishlist: Missing is Oreo, OpenSSL 1.1.1, Safari on OX 11, Firefox
52.x (ESR)

With the recent PR #1033 from @dcooper it can also show TLS 1.3
handshakes.
2018-04-16 11:13:44 +02:00
Dirk Wetter 55adbf905f
Merge pull request #1033 from dcooper16/client_sim_data_tls13
TLS 1.3 clients in update_client_sim_data.pl
2018-04-16 09:07:35 +02:00
Dirk Wetter e38a30d54e
Merge pull request #1034 from dcooper16/client_sim_grease_ciphers
update_client_sim_data.pl and GREASE ciphers
2018-04-16 08:52:57 +02:00
David Cooper f0ebf0339b
update_client_sim_data.pl and GREASE ciphers
Two GREASE ciphers currently appear in https://api.dev.ssllabs.com/api/v3/getClients: 0x3A3A for Chrome 57 and 0xAAAA for Chrome 65.

update_client_sim_data.pl currently only recognizes 0x3A3A as a GREASE cipher and so prints a "FIXME" for 0xAAAA. This PR fixes the problem by adding all 16 ciphers from https://tools.ietf.org/html/draft-ietf-tls-grease-00 to update_client_sim_data.pl.
2018-04-13 17:19:27 -04:00
David Cooper 639b1af916 TLS 1.3 clients in update_client_sim_data.pl
https://api.dev.ssllabs.com/api/v3/getClients incorrectly indicates a highestProtocol of 771 (TLS 1.2) for clients that support TLS 1.3, which leads run_client_simulation() to incorrectly report "no connection" if the client would have actually connected using TLS 1.3.

This has been addressed by manually editing etc/client-simulation.txt to set the highest_protocol to 0x0304 for the clients that support TLS 1.3.

This PR modifies update_client_sim_data.pl to automatically apply the fix for clients that support TLS 1.3 in order to avoid a possible regression when etc/client-simulation.txt is updated.
2018-04-13 16:51:06 -04:00
Dirk Wetter ea4fb5517f
Merge pull request #1032 from dcooper16/fix_mass_testing
Fix mass testing
2018-04-13 07:01:30 +02:00
David Cooper ac230d9cce
Fix mass testing
b2be380b54 inadvertently changed MASS_TESTING_CMDLINE to be a read-only variable. This causes mass testing to fail, since in mass testing the value of MASS_TESTING_CMDLINE is set to the command line for each child test.
2018-04-12 15:55:52 -04:00
Dirk 42e3f4ea14 "Resource" has only one "s" in English 2018-04-12 18:15:54 +02:00
Dirk e7619fa8d9 Documenting exit error codes improvements
See prevoius commit b2be380b54 and
issue #985 / #752.
2018-04-12 18:14:14 +02:00
Dirk b2be380b54 Exit error codes improvements, some formatting
According to programming standards e.g. C-style defines) testssl.sh has now
internal error variables (ERR_*) which are defined to deal with exit codes in
error conditions. Details see ``testssl.sh(1)``, section exit.  Thus exit codes
because of an error are now standardized and if needed can be easily changed to
other values.

This is part of a cleanup mentioned in #985 and #752. Codes for monitoring
tools (#327) which imply some kind of rating are still to be done.

The beginning section was reformatted and some items were reordered to keep
variables and functions together which serve similar purposes.

``readonly`` was replaced by ``declare -r`` (closer to C's define and it
makes more sense to settle on one variable if both are being used
for the same purpose)
2018-04-12 17:53:53 +02:00
Dirk Wetter 24a392d6eb
Merge pull request #1031 from dcooper16/client_simulation_ssl_native
Improve SSL native client simulation
2018-04-12 12:25:11 +02:00
Dirk 36247fecf2 fix no-DNS related error in documentation 2018-04-12 01:19:02 +02:00
Dirk 4370cb9aa1 no-DNS fixes
Add equal sign parser for command line fix. Readjust help to not-square
brackets as values are not optional (anymore)
2018-04-12 01:11:23 +02:00
Dirk 2a4de68c59 Merge branch 'nodns-935' into 2.9dev 2018-04-12 01:06:33 +02:00
Dirk 9f0e442b9f remove some no-DNS related logic
... from "Further IP addresses" and before calling ``get_aaaa_record``
in ``determine_ip_addresses()``. Logic appeared needlessly to difficult
and was as far as the "Further IP addresses" line was concerned incomplete.
2018-04-12 00:53:26 +02:00
Dirk 557942cb0a Change logic and add conservative value for -n/--nodns (#935)
This PR changes the logic the no-DNS switch works. The switch
now expects a value. "min" does minimum lookups, "none" does
no lookups at all (details see testssl.sh(1) ). "none" is
equivalent to the paranoid (boolean) value "true" before.
2018-04-12 00:19:52 +02:00
David Cooper cd8ceae80e Add curve information to SSL native client simulations
When performing client simulations in "--ssl-native" mode, provide the client's list of supported curves to "$OPENSSL s_client" in order to make the results even more accurate.
2018-04-11 13:48:40 -04:00
David Cooper 39db50eea2 Improve SSL native client simulation
This PR improves client simulation in "--ssl-native" mode:

* It changes ${protos[i]} to list the protocols that should be disabled rather than those that should be enabled, except in the case that the client only supports one protocol.

* It sets the values for ${tlsvers[i]}, which is used in run_client_simulation(), but was not defined.

* It adds a new variable, ${ciphersuites[i]}, that lists the TLSv1.3 cipher suites supported by a client.

Client simulation still produces false results in "--ssl-native" mode, but the results are better than before.
2018-04-10 16:57:24 -04:00
Dirk Wetter c357ea7356
Merge pull request #1030 from dcooper16/fix_dh_groups_output
Fix RFC 7919 DH groups
2018-04-10 18:47:23 +02:00
David Cooper 2c792fdc00
Fix RFC 7919 DH groups
This PR fixes three issues related to the testing for RFC 7919 DH groups in run_pfs():

* If the RFC 7919 DH groups are supported for both TLSv1.3 cipher suites and non-TLSv1.3 cipher suites, then the list of supported groups is printed twice.

* The finding that is used for CSV/JSON files includes the word "offered" after the list of groups, which is inconsistent with other findings.

* Since the $ffdhe_offered is only used to determine whether to test for use of RFC 7919 DH groups with non-TLSv1.3 ciphers, this flag should only be set if a non-TLSv1.3 ciphers that uses ephemeral DH is found.
2018-04-10 12:40:49 -04:00
Dirk Wetter a0c17fda9f
Merge pull request #1029 from knweiss/typos
Fix typos found by codespell
2018-04-10 17:45:17 +02:00
Karsten Weiss eead9f62d9 Fix typos found by codespell 2018-04-10 17:37:04 +02:00
Dirk Wetter 7ac63ec84b
Merge pull request #1028 from dcooper16/fix_false_connectivity_problems
Fix false connectivity problem error
2018-04-10 17:18:12 +02:00
David Cooper d1782284f8
Fix false connectivity problem error
If OpenSSL reports an error, sclient_connect_successful() may incorrectly interpret it as a connectivity problem, leading testssl.sh to stop testing before it has completed.

When not using "--ssl-native" mode, this happens if $OPENSSL does not support SSLv3, as both get_server_certificate() and run_beast() will attempt to connect using SSLv3 even if $OPENSSL does not support it.

When using "--ssl-native" mode, this happens in multiple places if $OPENSSL does not support the protocol being used or if $OPENSSL does not support any of the ciphers that are specified in the command line.

This PR fixes the above problems by adding checks for protocol support or for support for at least one cipher before calling $OPENSSL.

run_client_simulation() also has a problem in "--ssl-native" mode of calling $OPENSSL with parameters that cause $OPENSSL to report an error, but this is already addressed by temporarily setting MAX_OSSL_FAIL to 100 during client simulation tests and then, after client simulation testing is complete, returning $MAX_OSSL_FAIL and $NR_OSSL_FAIL to the values they had before client simulation testing began.
2018-04-09 15:43:26 -04:00
Dirk eb3b3a1988 be more verbose what --warnings=batch means (see #1027) 2018-04-05 22:02:35 +02:00
Dirk 6b8f6f82c3 Fix comments in run_logjam, replace awk by bash internal match expr
``run_logjam()`` contained in certain cases additional quotes
and a typo where only the word comment ended up in JSON/CSV.

Instead of ``$(awk '/Master-Key: / { print $2 }' "$2")`` the
admittedly performance sensitive function ``sclient_connect_successful()``
contains now a bash internal match according to #997 . First
tests didn't show much benefit (only default run with mostly
sockets was tested).

Unit tests showed no problems so far, but coverage is low.
2018-03-28 19:46:55 +02:00
Dirk 1924c9a0a6 Connectivity problems, man page update
See previous commit

This commit finally fixes #1005 so that either a --ssl-native scan
terminates on the next (defined) occasion if there are network connectivity
problems. It introduces another set of variables (MAX_OSSL_FAIL vs. NR_OSSL_FAIL).
As "openssl s_client connect" is sometimes still being used without --ssl-native
it also shortens the wait for regular scans if an outage is encountered.
To make things easier bot sets (incl. *_SOCKET_FAIL) of variables are independent.

For the seldom case that somebody uses --ssl-native with client checks an exception
had to be made as otherwise only MAX_OSSL_FAIL client check would be performed.
This hasn't been understood yet...

As sometimes HTTP header requests (over OpenSSL) fail repeatedly in a way that an empty
reply is returned, the same strategy of detecting problems is applied here,
using MAX_HEADER_FAIL and NR_HEADER_FAIL.

All three detection mechanisims share a new function connectivity_problem().
2018-03-28 17:48:04 +02:00