This PR fixes#615 for the case in which tls_sockets() is used by splitting the list of CBC ciphers into two lists, each with fewer than 128 ciphers and then testing each list separately.
For the --ssl-native case, no changes were needed. Even though $cbc_ciphers contains 154 ciphers, no version of OpenSSL supports all of these ciphers, and so the actual ClientHello sent by every version of OpenSSL contains fewer than 128 ciphers.
I did, however, add the -no_ssl2 flag to the "$OPENSSL s_client" command to prevent OpenSSL from sending an SSLv2-compatible ClientHello. As is noted in a comment in run_server_preference(), "the supplied openssl will send an SSLv2 ClientHello if $SNI is empty and the -no_ssl2 isn't provided."
This commit is a FIX for #1069, thus when running in
wide mode it corrects an additional line feed which
happened sometimes.
As @dcooper16 pointed out it also cleans up the needless
if-statements in run_rc4(), run_lucky13() and run_beast().
It also inserts for wide mode lines a blank so the alignment
is not at the left border anymore (check for leftovers
needed).
This PR fixes problems with check_revocation_crl() sometimes reporting that a certificate is revoked even when it isn't, and with check_revocation_ocsp() sometimes reporting "error querying OCSP responder" even if the OCSP responder provided a good response. The most common reason for this to happen is that OpenSSL cannot validate the server's certificate (even without status checking). PR #1051 attempted to get status checking to work even in cases in which the server's certificate could not be validated. This PR instead addresses the problem by not checking status if determine_trust() was unable to validate the server's certificate.
In some cases the server's certificate can be validated using some, but not all of the bundles of trusted certificates. For example, I have encountered some sites that can be validated using the Microsoft and Apple bundles, but not the Linux or Mozilla bundles.
This PR introduces GOOD_CA_BUNDLE to store a bundle that could be used to successfully validate the server's certificate. If there is no such bundle, then neither check_revocation_crl() nor check_revocation_ocsp() is run. When check_revocation_crl() and check_revocation_ocsp() are called, the status checks within them closely match the validation check in determine_trust(), which helps to ensure that if the check fails it is because of the status information.
As noted in #1057, at least one CA provides incorrect information when the CRL is downloaded, so validation could fail for a reason other than the certificate being revoked. So, this PR adds a check of the reason that validation failed and only reports "revoked" if the validation failed for that reason.
As noted in #1056, it is not possible to perform an OCSP query without access to the certificate issuer's public key. So, with this PR check_revocation_ocsp() is only called if the server's provided at least one intermediate certificate (i.e., the issuer's certificate, which contains the issuer's public key).
This PR improves the handling of error responses when checking status using OCSP. It can handle a few types of errors:
* When the responder just returns an error (e.g., "Responder error: unauthorized").
* When the response cannot be verified (e.g., invalid signature, expired certificate).
* When the response is valid ("Response verify OK"), but there is a problem with the response for the individual certificate (e.g., information is too old, or status is "unknown").
This PR fixes two issues with OCSP checking. First, the syntax for specifying a host header changed in OpenSSL between versions 1.0.2 and 1.1.0. With OpenSSL 1.0.2-chacha, 1.0.2o, and LibreSSL the syntax needs to be "-header HOST <hostname>". With OpenSSL 1.1.0h and 1.1.1 the syntax needs to be "-header HOST=<hostname>". I have not been able to test other versions of OpenSSL 1.0.2 or 1.1.0, but am assuming that all versions of OpenSSL 1.1.0 use the same syntax as 1.1.0h.
This PR also fixes a typo in the case of an error, which was causing $code to be set to "empty ocsp response" if the response was not empty rather than if it was empty.
The current code for extracting the validity dates for certificates assumes that the strings "Not Before" and "Not After" will appear exactly once in the pretty-print of the certificate. In most cases that works. However, there are a few server certificates that include the private key usage period extension, which also includes "Not Before" and "Not After" times. The result is that the current code does not correctly extract the start date and end date from any certificates that have private key usage period extensions.
This PR fixes the problem and also speeds up extraction of the dates by only using Bash internal functions.
The pretty-print of a certificate begins as follows:
Certificate:
Data:
Version: 3 (0x2)
Serial Number: ...
Signature Algorithm: ...
Issuer: ...
Validity
Not Before: ... GMT
Not After : ... GMT
...
The code in this PR extracts the start date by first removing from the certificate everything that comes before "Not Before: ". It looks for the shortest string that includes ""Not Before: " in order to ensure it is not getting the date from the private key usage period extension. After that, the longest string that begins with "GMT" is removed so that only the notBefore date remains.
The part that removes the string up to "Not Before: " actually looks for the first instance of "Not Before: " that comes after the "Validity". This is to protect against the unlikely possibility that the string "Not Before: " appears somewhere in the issuer's name.
The extraction of the notAfter date works similarly. It first looks for the first instance of "Not After :" that appears after both "Validity" and "Not Before: " and then takes the date string that appears immediately afterwards, with the assumption that the date string ends in "GMT".
In some cases the OCSP URI contains multiple components in the path (e.g., http://www.example.com/OCSP/myOCSPresponder).
This PR changes check_revocation_ocsp() to remove all components in the path, rather than just the final component, when extracting the host name from the URI for the host header.
My previous commit added a host header but didn't properly
format the host header (trailing slashes / path). This commit
corrects that so that the 305 times HTTP 400 in #1056
should now be gone (TBC), including Google CA responders.
One issue which needs to be addressed (same as in CRL
revocation checks): Not trusted certificates (zhanqi.tv,
taken from my Alexa scans) fail for obvious reasons.
Error from OCSP responder is now being displayed (and logged to JSON, ...)
Whole replay is kept in $tmpfile for debugging purposes
JSON output added for OCSP responderi query failures
Furtermore wget was replaced by "type -p" and grep by fgrep.
https://github.com/tlswg/tls13-spec/wiki/implementations now lists a server that supports TLS 1.3 draft 28, so this PR adds supports for drafts 27 and 28.
Since run_protocols() now checks for 11 different drafts of TLS 1.3 in addition to the final version, performing a separate test for each draft had become far too time consuming. So, this PR rewrites the check for TLS 1.3 versions in run_protocols() so that the number of tests is proportional to the number of drafts that the server supports rather than the number of drafts that testssl.sh can check for.
testssl.sh was inserting two spaces between the CBC ciphers detected by OpenSSL and those detected to tls_sockets(). This PR fixes the problem.
This issue was previously fixed by 87fe0c15da, but that fix was accidentally removed by the next commit: f3dc53f554.
There is at least one server that will fail under some circumstances if the ClientHello offers a compression method other than null.
In OpenSSL 1.1.0 and 1.1.1, s_client will not offer any other compression methods unless the "-comp" option is provided. However, in earlier versions of OpenSSL, s_client will by default offer the DEFLATE compression method, however, this can be disabled using the "-no_comp" option.
This PR addresses the flaw in this server by having s_client_options() add a "-no_comp" option to the command line if "-no_comp" is supported and the test doesn't require offering compression.
Since run_crime() requires compression to be offered, run_crime() was changed to always add "-comp" to the command line, and then s_client_options() was changed to remove "-comp" from the command line, if that option isn't supported.
* Extra function for ldap_get()
* Hint when curl is not installed and LDAP URI is encountered
* Rename jsonID cert_cRLDistributionPoints to cert_crlDistributionPoints
* Fix trailing _ in jsonID
Open/to be clarified:
* Proxy for curl / proxy needs to come from testssl.sh
* Proxy support for HTTP bash socket GET
* cert_CRLrevoked comes before cert_cRLDistributionPoints
* Unit tests
Still open: OCSP
At the moment the code for downloading a CRL seems to only work if URL is an HTTP or HTTP URL. It fails if the URL is an LDAP URL. The wget command does not support LDAP and when curl retrieves data from an LDAP URL it stores the result in LDIF format, which http_get() cannot currently convert into a PEM-encoded CRL.
This PR addresses the issue by skipping the revocation check for any URL that does not begin with "http".
In general, a CA only needs to keep the status information for a certificate until it expires. So, once a certificate has expired, the information provided about it in a CRL or OCSP response may no longer be reliable. The certificate may no longer be listed as revoked, even it is had been revoked at some point before it expired.
So, this PR changes certificate_info() to only check CRLs for revocation status if the certificate has not expired.
In order to use it one has to use --phone-out (PHONE_OUT
is the respective ENV) like
``./testssl.sh --phone-out --json-pretty -S wikipedia.org``
This makes use of curl (if available) or wget (if available) and
falls back to bash socket GET. The latter uses HTTP/1.0 as
chunked transfers by the server (used for bigger files normally)
can't be reasonably separated from their HTTP header. (HTTP/1.0
doesn't support chunked transfers).
curl and wget use the enviroment variables automatically. Probably
we want to use those proxies only if told by a switch to testssl.sh.
"-crl_download" would have been an option. Support would have
been needed to check beforehand. Alos information on proper
usage seems limited, so for now a solution which works is
preferred.
Open/to be clarified:
* Documentation
* Proxy for curl / proxy needs to come from testssl.sh
* Proxy support for HTTP bash socket GET
* JSON ID is cert_CRLrevoked_ (trailing underscore)
* cert_CRLrevoked_ comes before cert_cRLDistributionPoints
(* reconsider naming of cert_cRLDistributionPoints)
* Unit tests
Still open: OCSP
This PR was developed in response to #845. It adds to the list of ciphers used to determine whether the server has a cipher order in order to help avoid cases in which testssl.sh cannot determine a cipher order.
In order to create this list I scanned thousands of servers in order to determine what ciphers they support, including (1) about 20 thousand U.S. government web sites, (2) all of the sites listed at badssl.com, (3) all of the test servers listed at https://github.com/tlswg/tls13-spec/wiki/implementations, (4) about 30 additional non-U.S. government sites, and (5) one server configured as described in #845. I scanned each of these servers using OpenSSL 1.0.2-chacha, 1.0.2o, and 1.1.1.
Then I ran collection information through a script that created the updated list. For each scanned server, and for each of the 3 versions of OpenSSL, the script checked whether $list_fwd contained at least two ciphers from the list. If it didn't, then it would add one of the ciphers supported by the server (and by OpenSSL) to the list. In choosing among the ciphers supported by the server that were not already in $list_fwd, it would choose the cipher that was supported by the most other servers.
The list contain a few oddities as a result of the servers that I scanned. The script added two TLSv1.3 ciphers, since I scanned at least one server that only supports TLSv1.3. The list also includes ADH-AES256-GCM-SHA384 and AECDH-AES128-SHA, which may only be supported by null.badssl.com.
I made one manual change to the list - adding TLS_CHACHA20_POLY1305_SHA256. I did this since the number of TLSv1.3 servers scanned was so small, I didn't think it was safe to assume that all servers that support TLSv1.3 would support both TLS_AES_256_GCM_SHA384 and TLS_AES_128_GCM_SHA256.
Since most of the servers that I scanned were U.S. government servers, it may not be a representative sample. However, since the new list only adds to the current list, it can only be an improvement. Also, the updated list still only includes 37 ciphers, so many more could be added without creating any problems.
As it would be a possible privacy violation a new flag PHONE_OUTSIDE
is introduced (later accompanied by a switch). It determines whether
the client is allowed to retrieve the CRL specified (HTTP only supported).
Tested ok against wikipedia.de and revoked.badssl.com.
To do:
* look into -crl_download
* fileout
* Unit tests
OCSP verification
There is currently a problem if mass testing is being performed, JSON and/or CSV output is to be produced, the parent process calls fileout(), and each child process have its own output file for the JSON and/or CSV output. The can be seen, for example, with the following:
testssl.sh --openssl=openssl_1.1.1 --file test_servers.txt --csvfile output_dir --jsonfile output_dir
A call will be made in the parent process to report that openssl_1.1.1 has "No engine or GOST support via engine." fileout() will try to write to output_dir, which will result in an error.
This PR fixes the problem by checking the the file to be written to is not a directory (as is already done in html_out() for HTML output).
In certain situations while testting for CCS injection it could have happened
that an error code was sent which was not interpreted properly by testssl.sh.
(https://tools.ietf.org/html/rfc5246#section-7.2)
This has now been fixed and thus addresses #906. Also it has been made sure
that other error codes are reported appropiately.
The case where this test failed before was a non-patched Ubuntu 12.04
with openssl/postfix on port 25.
For the upcoming release this commit initiated the beta phase: important features
will be allowed. On the agenda is otherwise to fix bugs.
I ran shellcheck (see #434), and fixed some complaints and adjusted some coding
style mismatches.
There were some cases where security headers were served two times by the
server. The result (screen+html) wasn't properly formatted in those cases.
match_httpheader_key() was improved so that it keeps track when
a CR or an indentation needs to be done.
Some egrep statements were replaced by grep -E as this has been used
already and it is the thing testssl.sh should settle for. (precursor
to #1022).
run_more_flags was renamed to sun_security_headers and names of
variables is better.
HAS_SPDY is now HAS_NPN (similar to renaming the function a while
back)
mktemp should only be used when not avoidable (performance, code). For
temporarily local variables names can often be borrowed from globals
which were already generated by mktemp (SOCK_REPLY_FILE).
b2be380b54 inadvertently changed MASS_TESTING_CMDLINE to be a read-only variable. This causes mass testing to fail, since in mass testing the value of MASS_TESTING_CMDLINE is set to the command line for each child test.
According to programming standards e.g. C-style defines) testssl.sh has now
internal error variables (ERR_*) which are defined to deal with exit codes in
error conditions. Details see ``testssl.sh(1)``, section exit. Thus exit codes
because of an error are now standardized and if needed can be easily changed to
other values.
This is part of a cleanup mentioned in #985 and #752. Codes for monitoring
tools (#327) which imply some kind of rating are still to be done.
The beginning section was reformatted and some items were reordered to keep
variables and functions together which serve similar purposes.
``readonly`` was replaced by ``declare -r`` (closer to C's define and it
makes more sense to settle on one variable if both are being used
for the same purpose)
... from "Further IP addresses" and before calling ``get_aaaa_record``
in ``determine_ip_addresses()``. Logic appeared needlessly to difficult
and was as far as the "Further IP addresses" line was concerned incomplete.
This PR changes the logic the no-DNS switch works. The switch
now expects a value. "min" does minimum lookups, "none" does
no lookups at all (details see testssl.sh(1) ). "none" is
equivalent to the paranoid (boolean) value "true" before.
When performing client simulations in "--ssl-native" mode, provide the client's list of supported curves to "$OPENSSL s_client" in order to make the results even more accurate.
This PR improves client simulation in "--ssl-native" mode:
* It changes ${protos[i]} to list the protocols that should be disabled rather than those that should be enabled, except in the case that the client only supports one protocol.
* It sets the values for ${tlsvers[i]}, which is used in run_client_simulation(), but was not defined.
* It adds a new variable, ${ciphersuites[i]}, that lists the TLSv1.3 cipher suites supported by a client.
Client simulation still produces false results in "--ssl-native" mode, but the results are better than before.
This PR fixes three issues related to the testing for RFC 7919 DH groups in run_pfs():
* If the RFC 7919 DH groups are supported for both TLSv1.3 cipher suites and non-TLSv1.3 cipher suites, then the list of supported groups is printed twice.
* The finding that is used for CSV/JSON files includes the word "offered" after the list of groups, which is inconsistent with other findings.
* Since the $ffdhe_offered is only used to determine whether to test for use of RFC 7919 DH groups with non-TLSv1.3 ciphers, this flag should only be set if a non-TLSv1.3 ciphers that uses ephemeral DH is found.
If OpenSSL reports an error, sclient_connect_successful() may incorrectly interpret it as a connectivity problem, leading testssl.sh to stop testing before it has completed.
When not using "--ssl-native" mode, this happens if $OPENSSL does not support SSLv3, as both get_server_certificate() and run_beast() will attempt to connect using SSLv3 even if $OPENSSL does not support it.
When using "--ssl-native" mode, this happens in multiple places if $OPENSSL does not support the protocol being used or if $OPENSSL does not support any of the ciphers that are specified in the command line.
This PR fixes the above problems by adding checks for protocol support or for support for at least one cipher before calling $OPENSSL.
run_client_simulation() also has a problem in "--ssl-native" mode of calling $OPENSSL with parameters that cause $OPENSSL to report an error, but this is already addressed by temporarily setting MAX_OSSL_FAIL to 100 during client simulation tests and then, after client simulation testing is complete, returning $MAX_OSSL_FAIL and $NR_OSSL_FAIL to the values they had before client simulation testing began.
``run_logjam()`` contained in certain cases additional quotes
and a typo where only the word comment ended up in JSON/CSV.
Instead of ``$(awk '/Master-Key: / { print $2 }' "$2")`` the
admittedly performance sensitive function ``sclient_connect_successful()``
contains now a bash internal match according to #997 . First
tests didn't show much benefit (only default run with mostly
sockets was tested).
Unit tests showed no problems so far, but coverage is low.
This commit finally fixes#1005 so that either a --ssl-native scan
terminates on the next (defined) occasion if there are network connectivity
problems. It introduces another set of variables (MAX_OSSL_FAIL vs. NR_OSSL_FAIL).
As "openssl s_client connect" is sometimes still being used without --ssl-native
it also shortens the wait for regular scans if an outage is encountered.
To make things easier bot sets (incl. *_SOCKET_FAIL) of variables are independent.
For the seldom case that somebody uses --ssl-native with client checks an exception
had to be made as otherwise only MAX_OSSL_FAIL client check would be performed.
This hasn't been understood yet...
As sometimes HTTP header requests (over OpenSSL) fail repeatedly in a way that an empty
reply is returned, the same strategy of detecting problems is applied here,
using MAX_HEADER_FAIL and NR_HEADER_FAIL.
All three detection mechanisims share the new function connectivity_problem().
In addition unit tests showed that some vulnerability checks lost their
CVEs+CWEs whcich have been readded. For ROBOT a CVE was added (F5)
parse_tls_serverhello() checks $TLS_CLIENT_HELLO for a supported_versions extension, and if it contains one, checks that the negotiated version is listed in that extension. However, while $TLS_CLIENT_HELLO is always set in socksend_tls_clienthello() it is not set by client_simulation_sockets() (or any of the functions that client_simulation_sockets() calls). As a result, when the server's response to a client simulation is parsed, parse_tls_serverhello() may compare the negotiated version against the supported_versions extension from a ClientHello message from a previous test.
This PR fixes the problem by having client_simulation_sockets() set $TLS_CLIENT_HELLO.
In addition to 080840f655 and to
address #1005 eventually this is the first part of dealing
with connectivity problems. As opposed to the commit mentioned
above this is the counterpart for openssl which is useful for
just normal usage and for using --ssl-native.
It adds another global MAX_OSSL_FAIL (preset to 2) representing
a threshold how many unsucessful openssl connections are needed to
quit the whole scan. It should again reduce scan time in those
cases.
This is the framework part in sclient_connect_successful() which
is mainly commented out. The hooks will follow soon.
In certain cases (see e.g. #939) the server side "kind of" falls back
from a TLS channel to the underlying plaintext STARTTLS with a 5xx
or 4xx.
This PR adds the detection of 4xx for all protocols (if STARTTLS has been
used) and labels the detection as "likely not av" (and keeps the warning). This
leads to two JSON/CSV objects in those cases.
The detection isn't quite perfect as the issue #939 shows the first
TLS message fragment resembles a TLS alert.
As noticed it also removes the recommendation to increase the
debug level if the level is already sufficient.
FIX#1010 where a missing HSTS header lead to HIGH severity
whereas screen output was on info level.
In order to emphasize the importance of HSTS a missing header
flag will now be slighly penalized. To be in line with HSTS=0
the latter needed to be readjusted to medium as well.
As suggested by krissi for better readability a missing HSTS
header will now be 'not offered' instead of '--' on screen.
This PR changes find_openssl_binary() so that $OSSL_VER_APPENDIX contains everything from $OSSL_VER that appears after $OSSL_VER_MAJOR.$OSSL_VER_MINOR.
This PR fixes the issue raised in #1013. It primarily does this in two ways:
* In calls to `$OPENSSL s_client` that specify ciphers, the TLSv1.3 ciphers are provided separately using the `-ciphersuites` option. Then, the `s_client_options()` function manipulates the command-line options as necessary based on the version of OpenSSL being used.
* Calls to `$OPENSSL ciphers` were replaced with calls to `actually_supported_ciphers()`, which calls `$OPENSSL ciphers`. `actually_supported_ciphers()` modifies the parameters for the call to `$OPENSSL ciphers` as necessary, based on the version of OpenSSL being used.
When using testssl.sh with the current development branch of OpenSSL 1.1.1, determine_trust() incorrectly reports that "/openssl <= 1.0.2 might be too unreliable to determine trust." The problem is that find_openssl_binary() is not correctly extracting the version information. The version is "1.1.1-pre3-dev," but find_openssl_binary() extracts:
OSSL_VER_MAJOR: 1
OSSL_VER_MINOR: 1.13
OSSL_VER_APPENDIX: -pre-dev
This PR fixes the problem and also eliminates the use of external functions in extracting version information for $OSSL_VER.
Note that this code makes a change from the current code. Currently, $OSSL_VER_APPENDIX is intended to contain anything from $OSSL_VER that comes after $OSSL_VER_MAJOR.$OSSL_VER_MINOR. For example, if $OSSL_VER is 1.1.0-dev, then $OSSL_VER_APPENDIX is "-dev". In this PR, the "-dev" is dropped and so does not appear in $OSSL_VER_MAJOR, $OSSL_VER_MINOR, or $OSSL_VER_APPENDIX. The reason for this is that testssl.sh is only using $OSSL_VER_APPENDIX in cases in which $OSSL_VER_MAJOR.$OSSL_VER_MINOR is 0.9.8 to determine whether 0.9.8a - 0.9.8l or 0.9.8m - 0.9.8z. So, it seems that testssl.sh isn't interested in things such as "-dev" or "-pre3-dev".
If desired, this PR could be changed to that $OSSL_VER_APPENDIX contains everything in $OSSL_VER that appears after $OSSL_VER_MAJOR.$OSSL_VER_MINOR.
This PR fixes#1015 by adding underscore ('_') to the list of characters that may appear in a domain name label. a178f3e183 already addressed this for the check of whether the Common Name field in a certificate is a DNS name. This PR fixes it for checks of whether a domain name is a wildcard name and whether the server's host name is a wildcard match against a name in a certificate.
In cases where HTTP/2 was the only ALPN protocol, the
JSON output was missing the finding as the additional
protocol was empty.
This fix tests whether there's an additional protcol
and only calls fileout() if there's one.
When testssl.sh was using -ssl-native and the local openssl version
didn't support any DH ciphers the deliberately lazy approach was to
not complete the second part of the test: check for common primes.
This is a FIX for issue #1016.
This PR adds a missing check for local TLSv1.3 support in cipher_pref_check(). If the "--ssl-native" mode is being used and $OPENSSL does not support TLSv1.3, then a warning should be displayed indicating that TLSv1.3 cannot be checked rather than displaying the same results as if the server did not support TLSv1.3.
Drafts 25 and 26 of TLS 1.3 were issued in early March. Neither made any change that is relevant to testssl.sh.
This PR adds checks for these two drafts.
The implementation of read_sigalg_from_file() was changed on January 29 in 88cd5528e7. The new implementation does not work correctly in cases in which read_sigalg_from_file() is called with $TMPFILE as as parameter.
The current implementation of the function is:
```
read_sigalg_from_file() {
local hostcert_txt="${1//pem/txt}"
[[ -r "$hostcert_txt" ]] || $OPENSSL x509 -noout -text -in "$1" 2>/dev/null >$hostcert_txt
awk -F':' '/Signature Algorithm/ { print $2; exit; }' $hostcert_txt
}
```
When called using $TMPFILE (/tmp/testssl.XXXXXX/tempfile.txt), hostcert_txt is set to $TMPFILE, and since this file exists and is readable, the next line does nothing and the final line tries to read the signature algorithm from $TMPFILE rather than from a parsed version of the certificate.
This PR reverts read_sigalg_from_file() to its previous implementation, at least as a temporary solution.
Support for X448 was recently added to the development branch of OpenSSL 1.1.1. This PR adds an X448 key pair to etc/tls_data.txt (that was generated using OpenSSL 1.1.1) and adds X448 to the supported_groups extension for TLS 1.3 ClientHello messages.
Previously testssl.sh ran into problems when both Public-Key-Pins
and Public-Key-Pins-Report-Only were present in the header.
Now if both are present only Public-Key-Pins is evalutated and a
hint is given. If Public-Key-Pins-Report-Only is only present
a hint is given and those are evaluated.
If more than 1 from either Public-Key-Pins-Report-Only or
Public-Key-Pins is detected only the first one will be processed
and a warning will be displayed.
This commit improves assessments of server which decides after
a few connects either to drop all tcp packets or send a tcp
reset, see #1005 . It adds another global MAX_SOCKET_FAIL
(preset to 2) representing a threshold how many bash socket
connections are needed to quite the whole scan. It dramatically
reduces scan time and makes sure that the user is clear that
the scan had a problem and the results at maximum partly useful
(CSV, HTML, JSON). It also writes a valid JSON file.
See also #769.
It also fixes non-initialized variables in ``fileout_json_finding()``
and ``fileout()``
If a severity level >= INFO was supplied,JSON* and CSV output were
not writing error conditions in fileout(). As those indicate a problem
of the assessment, this has been fixed so that if any severity is
supplied to testssl.sh it will always write WARN conditions
to file when detected.
Also if the scan was interrupted on the client side where previously
a scanTime of 0 was written to the log file now it will raise a warning
that the scan was interrupted.
See also #769. Error conditions after starting a scan on the server side
(TCP Reset, DROP after a while) still won't be treated accurately in the file output
with an error message but the resulting JSON should be still valid.
OpenBSD doesn't seem to be able to convert from $(date)
output to a more readable format as performed on notBefore
and notAfter from x509(1). Thus an exemption is made and
just display the format like x509(1) shows.
Also newer OpenBSD come with a newer LibreSSL version,
thus the pattern when to display a warning when the
"too unreliable to determine trust" warning is displayed.
This commit adds support for TLSv1.3 draft 24. The only change between draft 23 and draft 24 is that draft 24 requires the version number of the record layer to be "0303" for a second ClientHello, whereas draft 23 allowed the version number for both the initial and second ClientHello to be "0301".
The current text reads:
legacy_record_version This value MUST be set to 0x0303 for all
records generated by a TLS 1.3 implementation other than an
initial ClientHello (i.e., one not generated after a
HelloRetryRequest), where it MAY also be 0x0301 for compatibility
purposes.
If extra extensions are provided to socksend_tls_clienthello(), the socksend_tls_clienthello() needs to determine what extensions were provided so that it doesn't add any of these extensions a second time. The code that was looping through the extra extensions to get the extension IDs was using the value for the length of the string. This commit fixes the error.
As per @krissi's suggestion this is a smart addition to add privat CAs
to all certificate stores -- during runtime only. The switch --add-ca or
--add-CA expects an argument of one additional CA file or a comma separated
lists of them. The enviroment file ADDITIONAL_CA_FILES can be used
alternatively.
This fixes eventually #230.
For recognising error conditions during run any testssl.sh run, the program
returns 0 if all checks have been performed successfully and unambiguously. 1
is returned per error or unambiguous condition and per IP. E.g. ./testssl.sh
--ssl-native testssl.net with two IP addresses returns 2 as each client
simulation gets a "1" (ssl-native returns currently wrong results here).
This fixes#986.
A kind of vulnerability scoring for monitoring tools or CI will be done
later separately, see #985 and #327.
As before (see #971) and as indicated (#970) some minor
changed to the jsonID have been taking place. Some
redundant verbosity has been eliminated, some IDs
changed the name.
For HTTP2 you should now ALPN_HTTP2.
Some obvious CWEs were added in the JSON section.
ke for null cipher list, information leakage in
the http headers. There's probably space for improvements.
A few early time marks were added to debug time spend.
Some servers will respond with an alert to a ClientHello that does not indicate support for secure renegotiation, which may be signaled through either an extension or the 0x00,0xff "cipher suite." In some cases testssl.sh calls tls_sockets() without including "00,ff" in the list of cipher suites, which results in some servers rejecting a ClientHello that would otherwise result in a successful connection.
This PR fixes the problem by adding "00,ff" to any ClientHello where it was previously missing, with one exception. If a TLSv1.3 ClientHello is being sent and only TLSv1.3 ciphers are listed, then the "00,ff" cipher suite is not added.
Error codes now done until run_server_defaults(). Previous
sections modified so that execution errors are added.
Modified in compare_server_name_to_cert() ret --> subret.
From the code perspective a clear distinction between
passing an error code and a functional return code
would be great. Still has to be determined whether it
maybe better to rename ret into something different.
Removed a stale code line in run_server_preference() for STARTTLS.
Following the recommendation from @dcooper16 this commit is addressing
a situation when the scan couldn't finish for external reasons and as
a consequence left a non-valid JSON file behind.
It also starts addressing #986 so that the protcol section only returns
a non-zero value if a check coundn't be performed or gave results which
weren't clear.
It also fixes a typo where in the TLS 1.3 check a status from the TLS 1.2
check was not correctly interpreted (TLS 1.2 not offered).
This commit fixes#983 by ensuring that the line printed just before calling "return" prints a newline character. It also fixes the problem that no output is sent to the JSON/CSV file in some cases in which no fallback is possible since the server does not support two different protocols below TLSv1.3.
In order to be more consistent with the other output functions having
"svrty" in their name, *_done_best and *_done_good were changed to
*_svrty_best and *_svrty_good.
run_spdy/run_http2 were renamed to run_npn and run_alpn as this is
what is actually being tested. Also the terminal and file output
is now reflecting this.
Also #980 was fixed in a sense that (only) for ALPN the protocol
h2 will get a "good". There will be an additional CSV/JSON line
for this.
This commit fixes#981 by using a while loop instead of a for loop to check each DNS name in the SAN extension, copying the syntax used in certificate_info() to display all of the SANs.
This commit adds code to run_server_defaults() so that in debug mode all of the server's certificates are saved in $TMPDIR in both PEM-encoded format and pretty-print text format.
testssl.sh was recently changed to store the text printout of the host's certificate in a file, $HOSTCERT_TXT, and then use this file in some places rather than calling "$OPENSSL x509 -in $HOSTCERT -text -noout". There was a problem, however, in cases in which the server had more than one certificate (including cases in which the server returned an unrelated certificate when sent a ClientHello w/o SNI), since the contents of $HOSTCERT_TXT was not always being updated whenever $HOSTCERT changed.
This commit fixes that problem by replacing the previous solution with a less ambitious one. In this version, the global variable is eliminated and instead run_server_defaults() stores a text version of each certificate it finds in an array. This value is then passed to certificate_transparency() and certificate_info() for use. It is also passed from certificate_info() to must_staple().
determine_trust() uses the output of "$OPENSSL verify" to determine whether OpenSSL can construct a valid certification path for the server's certificate. If it does not find a string of the form "error [1-9][0-9]? at [0-9]+ depth lookup:" in the output, then it assumes that validation was successful. In current versions of OpenSSL, when this error is created it is printed to stdout, but in OpenSSL 1.1.1 is it printed to stderr. Since testssl.sh only checks the output sent to stdout, it incorrectly treats all certificates as valid if OpenSSL 1.1.1 is used.
This commit fixes the problem by checking the text that is sent to both stdout and stderr.
This commit also fixes a typo in the call to "$OPENSSL verify" which resulted in the environment variables SSL_CERT_DIR and SSL_CERT_FILE not being set to "/dev/null".
In ``must_staple()`` and ``certificate_info()`` were the jsonIDs
changed to OpenSSLi / IETF names so that testssl.sh is more compliant
to the rest of the world. There might be still space for improvements
are far as common naming scheme is concerned.
The host certificate is now being delivered in JSON and CSV. For
further usage " " needs to be converted back to linefeeds.
Certificate Expiration was renamed to Certificate Validity.
The order of outputting the certificate serial and SHA1 fingerprint
has been swapped.
Also ``certificate_info()`` makes more use of HOSTCERT_TXT.
For certificate start+end time it is now displaying the
time on UTC and without mentioning the timezone twice.
Also if neither CRL nor OCSP URI is provided it'll appear
on the screen below those two checks. JSON/CSV has then an
additional finding
HOSTCERT_TXT is now generated each time a HOSTCERT is
written. For now it is just being used in ``read_sigalg_from_file()``.
More to come.
Also in JSON output the start date and end date of the certificate
are now in separate objects. (the time format could need some polishing though).
In the output created by certificate_info(), the "Server key size" line labels an elliptic curve key as "ECDSA." This commit changes the label to "EC." I believe this a more correct label since ECDSA is a signature algorithm, not a key type. Also, while unlikely, an elliptic curve key in a certificate may be used for ECDH (e.g, in TLS_ECDH_RSA_WITH_AES_128_CBC_SHA) rather than ECDSA.
Note that this does not impact the JSON or CSV output, since the corresponding fileout command already uses "$cert_keysize EC bits"
This commit adds TLSv1.3 support for run_server_defaults(). It uses get_server_certificate() to run separate checks for RSA and ECDSA certificates by sending TLSv1.3 ClientHello messages with different signature_algorithms values. A similar change is made to certificate_transparency() in order to check for certificate transparency support for certificates returned over TLSv1.3. It also modifies the check for certificates offered without SNI by including an option to use tls_sockets() with servers that support TLSv1.3 if OpenSSL does not support TLSv1.3.
In TLSv1.3, if a HelloRetryRequest needs to be sent and the call to tls_sockets() includes additional request extensions (other than key_share or cookie), then resend_if_hello_retry_request() adds an extra comma between extensions in the value sent to socksend_tls_clienthello(), which creates errors. This commit fixes the problem by removing the extra comma.
``run_tls_fallback_scsv()`` was upgraded to the new scheme.
It also changes a couple of more functions using a variable ``$jsonID`` instead
of a repeating fixed value for `fileout()`.
Also some it adds some ``fileout`` calls which were missing.
In the sense of the previous commits this one adds
further changes to the JSON output parameters ID and findings.
Findings in JSON are now more crunchy and don't repeat parts
of ID. Also ID changed so that it is more reflecting what
has been tested.
As the cipherlist checks in the beginning have been less
and less to do with the OPsnSSL standard lists
a change to remove the word "standard" was long overdue.
That has been addressed now alos in the code and in the
JSON/CSV output.
$HOSTCERT has now an .pem file extension. $HOSTCERT_TXT
will contain the text putput of the x509 openssl operation
on the certificate which enables testssl.sh to remove
some of the redundant "openssl x509 -in $HOSTCERT -text"
calls.
fileout <somestrings> has not been consistently replaced
by fileout $jsonID yet.
This PR defines an extract_certificates() function in order to remove some redundant code from get_server_certificate(). Currently, nearly identical code appears in two places in get_server_certificate() to extract certificates from the output of `$OPENSSL sclient`, in one place for SSLv2 responses and in another for SSLv3 through TLSv1.2. The code to get the certificates used with TLSv1.3 (see https://github.com/dcooper16/testssl.sh/tree/extended_tls_sockets) would have added a third place where this same code would be needed. This PR allows the code to be written once and used in all three places.
This commit just adds more changes to those two parameters.
It is not completely done yet (see downgrade protection via SCSV).
Also json_prefix was changed to jsonID. The complete change of the fist
fileout field is pending.
The server default run had several JSON objects which weren't, looking at just
the ID, either clear or contained a redundant explanation in "finding". Purely
certificate related JSON objects are now having the id "cert_<object>" like
cert_CN or cert_SAN.
This commit changes all this, also it avoids another colon in finding (see #830).
Also the implicit strategy "output for the screen s followed by only one output with
fileout" has been relaxed -- which results on more, better parsable JSON objects.
Some example of the changes:
Old:
----
{
"id" : "Server Certificate #1 fingerprint",
"severity" : "INFO",
"finding" : "Fingerprints / Serial: SHA1 2940BC13ECF7DAF30B9084CC734C3B971D73B3BB / 01BFD1DC15006E0ABBA7C670FF5E1101, SHA256 30BA61012FFE7CEAAF9A148A0CB0C5C852A9C04F4B1C27DB6
EFA9919C7F49CCF"
}
[..]
{
"id" : "Server Certificate #2 ocsp_stapling",
"severity" : "OK",
"finding" : "OCSP stapling : offered"
}
New:
----
{
"id" : "cert_key_size <cert#1>",
"severity" : "INFO",
"finding" : "Server keys 2048 bits"
},{
"id" : "cert_fingerprint_SHA1 <cert#1>",
"severity" : "INFO",
"finding" : "2940BC13ECF7DAF30B9084CC734C3B971D73B3BB"
},{
"id" : "cert_fingerprint_SHA256 <cert#1>",
"severity" : "INFO",
"finding" : "30BA61012FFE7CEAAF9A148A0CB0C5C852A9C04F4B1C27DB6EFA9919C7F49CCF"
},{
"id" : "cert_serial <cert#1>",
"severity" : "INFO",
"finding" : "01BFD1DC15006E0ABBA7C670FF5E1101"
}
[..]
{
"id" : "OCSP_stapling <cert#2>",
"severity" : "OK",
"finding" : "offered"
}
This PR also fixes the JSON output where for "OCSP must staple" the id was just
'id" : "OCSP must staple: ocsp_must_staple",' for multiple server
certificates without the certificate number.
As far as the code is concerned: $json_prefix should be a variable which is
used for the id object. If there was more then one certificates for a single
host detected, $json_postfix carries the certificate number.
Unit tests need to be fixed -- if possible.
Add fileout() to #965. This commit also contains a change which needs
to be commited before: separation of ``json_prefix`` from ``json_postfix``.
Open issue: sed in openssl x509 statments look GNUish ([ \t]). Needs clarification.
This commit prints the contents of the keyUsage and extended key usage extensions in certificates and checks the public keys in the certificates are not being used in a manner that is inconsistent with these extensions.
This PR is intended to improve the functionality of run_tls_fallback_scsv().
The original goal of this PR was to address servers that support TLSv1.3 when using OpenSSL 1.1.1. That does not seem to be an issue, as using `$OPENSSL s_client` with the `-no_tls1_2` flag results in a TLSv1.1 ClientHello, even if `$OPENSSL` supports TLSv1.3. However, if the server supports TLSv1.3, then a message that says "No fallback possible, TLS 1.2 is the only protocol (OK)" isn't entirely correct.
The main issue this PR fixes is some false positives in servers that do not support TLSv1.2. On a few servers that I tested, the current code incorrectly reports "Downgrade attack prevention NOT supported." Some of the servers only support TLSv1, so it should report that fallback is not possible. Another server supports TLSv1.1 and TLSv1, and it supports fallback protection. In both cases, the current code produces a false positive, since it assumes that TLSv1.2 is supported.
In three different places there is a line that is supposed to check whether the list of ciphers to be tested contains any TLSv1.3 ciphers. This check currently fails if there is only one TLSv1.3 cipher in the list and it is the first cipher in the list. This commit fixes the problem.
This commit adds support for draft 23, which contains 2 changes that are relevant for testssl.sh. It adds a few new values for the signature_algorithms extension and it changes to extension number for the key_share extension from 40 to 51.
With the change in the extension number, it is no longer possible to send a single ClientHello that works for all supported drafts of TLSv1.3. (I tried sending a ClientHello with two key share extensions, 40 and 51, but that didn't work.) So, this commit adds a test to determine_optimal_proto() to determine whether TLSv1.3 is supported and if so whether draft 23 is supported or only some earlier draft (18-22). In subsequent tests, the ClientHello uses the appropriate number for the key share (40 or 51) and specifies the appropriate draft version(s) in the supported_versions extension (either 23 or 18-22). In the case of run_protocols() the test for each draft version uses the appropriate key share extension number so that servers that support both draft 23 and an earlier draft can be detected.
support. If the server doesn't supply an session identifier the file
is just empty.
This commit fixes that by adding a separate case for OpenSSL 1.1.1
and an empty file. It is deliberately only changing this as this
was tested to work.
It is prelimary and a save-the-work-patch as it might be better
to catch this earlier.
For JSON pretty the host specific parameters target host + port
could be better placed in the scanResult object.
It is still under discussion as logically the parent object is deduced
the command line.
TLS_RSA_* which don't fall into the aleady mentioned
categories (CBC cipher, export, RC4 etc.) are now
a bit more more penalized. Those are the ones which have
an RSA key exchange AND a modern encryption.
pr_cipher_quality() needs to be redone after carefully
reconsidered which cipher should have which rating.
This PR is similar to #944. If using OpenSSL 1.1.1 to connect to a server that supports TLSv1.3, `run_crime()` will connect to the server using TLSv1.3, which does not support TLS-level compression. So, the server will be reported as "not vulnerable" even if would use compression for connections at TLSv1.2 and below.
I have not encountered any "live" servers that support both TLSv1.3 and TLS-level compression. I verified this problem by using OpenSSL 1.1.1 to create a server that supports both TLSv1.3 and TLS-level compression:
```
openssl111 s_server -cert cert.pem -key key.pem -accept 8443 -WWW -comp
```
I then tested the server using `testssl.sh --crime` with both openssl111 and OpenSSL 1.0.2-chacha.
run_renego() appears to produce a false positive if OpenSSL 1.1.1 is used and the server being tested supports TLSv1.3 (i.e., the server supports the same draft version of TLSv1.3 as the version of OpenSSL 1.1.1 being used does). This PR fixes the problem by telling calls to $OPENSSL s_client in run_renego() to not use TLSv1.3.
.. to check during the default run for server implemenation bugs
and run cipher per procol check instead of cipher check.
Please not that this option could disappear later.
In TLSv1.3 servers may send a supported_groups extension, which "SHOULD contain all groups the server supports, regardless of whether they are currently supported by the client."
This PR extracts the contents of the supported_groups extension, if `parse_tls_serverhello()` is to process "all" of the server's response. The contents of the extension are also displayed on the terminal if $DEBUG -ge 3.
In TLSv1.2 and below, servers respond to a status_request extension (a request for a stapled OCSP response) by returning an empty status_request extension and then including a CertificateStatus message, which follows the Certificate message. In TLSv1.3 the CertificateStatus response is included as the value of the status_request extension, which now appears as an extension within the Certificate message.
This PR extracts the contents of the status_request extension sent by the server so that it can later be processed in the same way as if it had sent in a TLSv1.2 or below response.
This PR adds code to decrypt the encrypted portion of the server's response for TLSv1.3 and to then process any certificates and encrypted extensions. This code supports all 5 TLSv1.3 cipher suites, and so any response can be decrypted as long as the session key can be derived (which requires OpenSSL to support the ephemeral key that was used - see #938).
For the symmetric decryption, the sym-decrypt() function uses OpenSSL when possible and internal Bash functions when needed.
For AES-GCM and AES-CCM ciphers sym-decrypt() normally uses internal Bash functions, which rely on using "$OPENSSL enc" in AES-ECB mode to generate the key stream and then Bash functionality to XOR the key stream with the ciphertext. With some version of OpenSSL the AES-GCM ciphers are decrypted using "$OPENSSL enc" in AES-GCM mode directly. On my system, however, both methods seem to work about equally fast.
For ChaCha20 ciphers, "$OPENSSL enc -chacha20" is used, if supported (OpenSSL 1.1.x only). and Bash internal functions (without any OpenSSL support) are used otherwise. In this case, if the Bash internal functions need to be used, decryption is very, very, very slow. Fortunately, in a typical run of testssl.sh there won't be many cases in which the connection will be TLSv1.3 with ChaCha20 and the entire response needs to be processed (requiring decryption). In most cases, even if the connection is TLSv1.3 with ChaCha20, will at most need the ephemeral key, which is available in plain text.
This is the first in a series of PRs to add support for processing the encrypted portions of the server's response in a TLSv1.3 handshake.
This PR adds the code to derive the handshake traffic key needed to decrypt the response (the next PR will add the code to perform the symmetric-key decryption of the encrypted portions of the response).
Since this PR does not make use of the traffic key that it derives, it doesn't yet add any new functionality.
Note that testssl.sh will not always be able to derive the session keys. If the version of OpenSSL that is bundled with testssl.sh is used and the server chooses to use an X25519 ephemeral key, OpenSSL will be unable to perform the shared secret in derive-handshake-traffic-secret(). (OpenSSL 1.1.0 supports X25519.) Since X25519 use a different encoding than ECDH keys, the lack of X25519 support will be discovered in parse_tls_serverhello() when $OPENSSL pkey is unable to convert the key from DER to PEM. So, in debugging mode, parse_tls_serverhello() now displays a warning if it receives a key share that $OPENSSL pkey cannot handle.
get_server_certificate() uses an awk script to extract the certificates from the output of OPENSSL s_client and it then uses the following line to determine how many certificates were found:
nrsaved=$(count_words "$(echo level?.crt 2>/dev/null)")
If $nrsaved is 0, then get_server_certificate() returns 1 (indicating failure); otherwise it returns 0 (indicating success).
However, the check for the number of certificates returned doesn't work if no certificates were found, as nrsaved will be set to 1 if no certificates were found:
> touch level0.crt
> echo level?.crt
level0.crt
> touch level1.crt
> echo level?.crt
level0.crt level1.crt
> rm level0.crt level1.crt
> echo level?.crt
level?.crt
This PR fixes the problem by first checking that level0.crt exists (-s is used instead of -e, since an empty file wouldn't have a certificate).
Similar to the recently added HAS_PKUTIL (f829878a43), this PR adds HAS_PKEY, which indicates whether OpenSSL has the pkey utility. HAS_PKEY is then checked before attempting to do something that requires the pkey utility.
POP3 STARTTLS handshakes were often unsuccessful as
a regex wasn't properly escaped.
Furthermore if a STARTTLS handshake doesn't succeed, there's
a warning now.
See pending PR #905 / issue #333.
There's still lots of work needed and probably the function
needs to be completely rewritten and to be in sync with
other parts of the program.
This PR fixes one odd formatting of header flags like X-Frame-Options,
where the output header maybe contained a LF "\r". X-XSS-Protection was
also not correctly formatted due to the fact that only a part of it until
the blank was displayed.
Also the file output may contain now 1x less blank, e.g.
"X-Content-Type-Options: nosniff" instead of
"X-Content-Type-Options: nosniff"
This PR adds an additional COLOR level (3). If color is set to 3 then all ciphers are printed according to pr_cipher_quality() rather than just the "Negotiated cipher" in run_server_preference().
This PR reduces the amount of work parse_tls_serverhello() does when processing alert messages when not in debug mode. It delays writing anything to $TMPFILE unless $DEGUG -ge 1 until it has reason to believe that the response was successful. If $DEBUG is 0 and alert messages are sent, then no file operations are performed processing the alert messages.
In almost every case, there is no attempt to look at the contents of $TEMPDIR/$NODEIP.parse_tls_serverhello.txt unless the connection was successful. So, in most cases, it is okay to not call tmpfile_handle() in parse_tls_serverhello() unless the connection was successful. There is, however. one place in run_grease() where the code reads the contents of $TEMPDIR/$NODEIP.parse_tls_serverhello.txt even if the connection was not successful. In order to address this, the DEBUG level is temporarily set to 1 when performing this test if its value is 0. Also in order to address this, changes were made in parse_tls_serverhello() to ensure that "tmpfile_handle $FUNCNAME.txt" is always called before returning if $DEBUG -ge 1.
If certificate-based client authentication is required by the server, then most HTTP-related checks are skipped, even if the "--assume-http" flag is used. If $CLIENT_AUTH is true, then $ASSUME_HTTP is ignored.
In some cases the checks are appropriately skipped, since the tests cannot be performed. In other places, the value of "$CLIENT_AUTH" is used as a hint as to whether HTTP is being used. For example, in run_tickbleed:
if [[ "$SERVICE" != HTTP ]] && ! "$CLIENT_AUTH"; then
outln "-- (applicable only for HTTPS)"
fileout "ticketbleed" "INFO" "Ticketbleed: not applicable, not HTTP" "$cve" "$cwe"
return 0
fi
There are some places, however, where tests are just skipped, even if both $CLIENT_AUTH and $ASSUME_HTTP are true, even though the test could be performed. For example, run_client_simulation() only simulates generic clients in this case.
This PR attempts to address this:
* In run_client_simulation() it runs all of the tests if $ASSUME_HTTP is true.
* In certificate_transparency() it only says that the lack of CT information is "N/A" it can verify that HTTP is not being used (if $SERVICE is not HTTP and $CLIENT_AUTH is false). Otherwise it just says "no" without flagging it as an issue.
* In certificate_info() it displays additional warnings (about use of SHA-1 or subjectAltName matching) only if it can verify that HTTP is being used ($SERVICE is HTTP or $ASSUME_HTTP is true).
* In run_crime(), if compression is used, it only says " but not using HTTP" if it can verify that HTTP is not being used (if $SERVICE is not HTTP and $CLIENT_AUTH is false).
If a supported_versions extension was included in the ClientHello, then check that the version returned by the server was included in the ClientHello's supported_versions extension.
OpenSSL will respond to a TLSv1.3 ClientHello that only specifies 0304 in its supported_versions extension with a ServerHello that specifies whatever draft of TLSv1.3 it currently supports (e.g., 7F16). The result is that run_protocols() incorrectly reports that OpenSSL supports TLSv1.3 "final" in addition to whatever draft version it supports.
This PR fixes that problem by treating it as a failed connection when the ClientHello offers only 0304 and the ServerHello specifies something else (e.g., 7F16).
Performing this check is actually a requirement for clients in Section 4.2.1 of draft-ietf-tls-tls13-22. So, including this check will also help make client simulations more accurate when clients that support TLSv1.3 are added to client-simulation.txt.
The default STARTTLS_SLEEP timeout was increased to 10 seconds in
d1e7498. This caused MySQL connections to timeout. Quick fix is to
parameterize the timeout and pass in 1 again.
Better future fix is to read MySQL as binary packets, parsing the fixed
sized header, to then read the variable sized payload. Doing this will
also greatly speed up testing.
This fixes issue #914.
It partly addresses #915, supposedly the openssl binary used supports
TLS 1.3 (and the correct draft/final).
It also reduces handshakes by not trying protocols which aren't supported
on the client side.
This PR adds support for TLSv1.3 draft 22. This PR has testssl.sh operate in "middlebox compatibility mode" as described in Appendix D.4 of draft-ietf-tls-tls13-22 to maximize the chances of being able to perform a successful test even if there is a misbehaving middlebox between testssl.sh and the server being tested. Support for drafts 18 through 21 is still maintained.
This PR has been tested against a few different implementations of draft 22 that were made available shortly before draft 22 was posted.
In TLSv1.3, when responding to a HelloRetryRequest, the second ClientHello should be sent through the same socket as the first ClientHello.
This PR adds an option to socksend_tls_clienthello() to not open a socket and then uses that option in resend_if_hello_retry_request() when sending the second ClientHello.
Revert the exception from previous commit 20b38d5aa6
that TLS alerts on SSLv2 client hello won't be a proper reply -- due to a reality check,
see #908.
In order to have better debugging info the TLS alert message is printed
in clear. Messages code and text assignements was moveed to a separate
function.
Some servers like the one from the satire magazine "focus.de"
choose to return gzip encoded body during run_http_haders().
This has led sometimes to misintepretation that an IPv4 address
is present in the header.
This commit fixes that by telling the server not to want a gzipped
response and if still returned grep properly the return.
When testing servers which fall back after succeeding the STARTTLS handshake
to the underlying protocol (smtp, ftp etc.) there was often misleading output
=not available instead of notifying the user that there's a problem.
Now it is being tested in parse*serverhello() functions whether the first
by resembles a 5XX code and STARTTLS is supposed to be tested and then
passes an error code back to the caller , i.e. tls_sockets() and then
run_protocols(). Also other error code after the handshake are passed
better.
This is only an addition to the top level function run_protocols().
We might want to look into other top level functions too
In scenarios where --ssl-native was chosen AND the
openssl binary wasn't supporting the protocol
there were two warnings. This has been addressed.
Also tls_sockets() can return different values -- for now: 6 -- as
tested by the caller in run_protocols. In order to make
it more robust a fixme statement was added so that the
user becomes iat least aware of it.
After changing the logic from $PROTOS_OFFERED
contaning each protocol detected to $PROTOS_OFFERED
contaning each protocol + colon yes or no there
wasn't a change for the sanity check that no protocols
are offered. This fixes it.
This PR adds TLSv1.3 support for the negotiated protocol and cipher in run_server_preference(). This mostly addresses #893, however, run_server_preference() will not work with a TLSv1.3-only server as it will fail when trying to determine whether the server has a cipher order.
Note that with this PR run_server_preference() will not always provide consistent results when testing a server that does not support TLSv1.3 using a version of OpenSSL prior to 1.1.1. If it is determined before running run_server_preference() (using run_protocols()) that the server does not support TLSv1.3, then run_server_preference() will using OpenSSL to determine the negotiated protocol and cipher. However, if it has not yet been determined that the server does not support TLSv1.3, then run_server_preference() will use tls_sockets(), which tries to simulate OpenSSL 1.1.1. Since the list of cipher lists sent will differ the negotiated cipher will sometimes differ. In addition, when a cipher suite that uses an ephemeral ECDH key is selected, the negotiated curve is sometimes different.
This PR fixes#901. When $OPENSSL_TIMEOUT is set and mass testing is being performed, this PR changes find_openssl_binary() so that only child instances modify $OPENSSL to add the call to timeout.
This PR also changes the warning in case $OPENSSL_TIMEOUT is set and the timeout binary cannot be found so that the warning message is printed even if $WARNINGS is "off" or "batch".
There's a new socksend() function which gives up to 8% performance
benefit (LAN) as is saves 3 sed and 1 tr command in every socksend call.
It has not been put in production yet as it shows a problem in ticketbleed
check and it is late and I call it a day ;-) and resolve that later (see #902).
For performance tests it's quite handy to also have in the flat json format
a footer with time consumed for each run. fileout_json_footer() was
patched accordingly.
... from "--" to "not offered". Reason: e.g. on a white
terminal background it is hard to detect that there's
an output. Also "not offered" is more consistent with
the value of protocol checks.
When the server's response to the ClientHello message (i.e., the ServerHello, Certificate, ServerHelloDone) is split across more than one packet, the current call to tls_sockets() in run_heartbleed() only retrieves the first packet. As a result, sometimes when run_heartbleed() believes that it is reading the response to the Heartbleed payload it is actually reading the continuation of the response to the ClientHello message.
This PR fixes the problem by indicating in the call to tls_sockets() that the ephemeral key is needed. This causes tls_sockets() to continue requesting additional packets until it receives the ServerHelloDone.
There was an error in e450eb34e4
which addressed #846 which lead to the problem that an output filename
could start with a "." if no FNAME_PREFiX and if just --html, --csv
or --log or --json* was supplied.
This commit fixes the problem.
When the certificate signature algorithm is RSA-PSS and OpenSSL 1.1.1 is used $cert_sign_algo contains some trailing space characters, which causes the algorithm not to be recognized in the case statement. This PR fixes the problem by removing any trailing space characters from $cert_sign_algo.
This PR fixes a false positive in std_cipherlists(). Currently, sclient_success is not initialized (so it initially set to 0). If a server is being tested that only supports TLSv1.3, the --ssl-native option is not used, and run_protocols() is run before run_std_cipherlists(), then for many of the calls to std_cipherlists() no tests are run and so sclient_success remains at its initial value (0), which is treated as success (i.e., the server supports at least one of the ciphers in the list).
The reason this happens is that in the testing loop, the TLSv1.3 test is skipped if the list of ciphers doesn't include any TLSv1.3 ciphers (and only the "Strong encryption" test includes TLSv1.3 ciphers) and the tests for each of lower versions of SSL/TLS is skipped since it was already determined in run_protocols() that those versions weren't supported.
This PR adds a check for TLSv1.3 support to run_protocols(), checking for support for the final version of TLSv1.3 (0x0304) as well as drafts 18, 19, 20, and 21 (0x7F12, 0x7F13, 0x7F14, and 0x7F15).
File names are now auto-generated by using "-oA auto" / -oa "auto"
--similar to --csv and friends.
Also the formerly hidden switches --outFile and --outfile were added in the
help and in the manual.
Former code implied a determination of the TLS time in
every call of tls_sockets() despite the fact that the
value is only needed at one point in the run.
This removes this behaviour by introducing another global
boolean switch TLS_DIFFTIME_SET which determines whether
the additional cost will be paid or not.
The gain in execution time is a bit meager though. At
most it seems it's 1-3 seconds.
For servers with client authentication one would need to supply a x509
certificate to check session resumption by ID or ticket. This is not (yet?)
supported in testssl.sh.
This commit fixes the misleading error message so that it is clear what the
problem is.
This PR is a continuation of #833.
With additional testing with different options I encountered more places where $OPENSSL was printing "WARNING: can't open config file: /usr/local/etc/ssl/openssl.cnf" where testssl.sh was not suppressing the error message.
This PR redirects stderr to /dev/null or to $ERRFILE for several more calls to $OPENSSL in order to suppress these warning messages.
For ciphers that use the ChaCha20-Poly1305 cipher, LibreSSL shows "Enc=ChaCha20-Poly1305" in the "openssl ciphers -V" command rather than "Enc=ChaCha20(256)" and for some GOST ciphers it shows "Enc=GOST-28178-89-CNT" rather than "Enc=GOST(256)". This causes a problem for neat_list() if information is being obtained from "$OPENSSL ciphers -V" rather than from the cipher-mapping.txt file.
If testssl.sh is used with OpenSSL 1.1.1 and TLSv1.3 support is enabled, then the check for whether the server has a cipher order will always fail. The problem is that since the call to s_client doesn't specify a protocol a TLSv1.3 ClientHello will be sent. However, the call specifies a list of ciphers that doesn't include any TLSv1.3 ciphers. So, OpenSSL will fail with the error: "No ciphers enabled for max supported SSL/TLS version." The solution is to add the "-no_tls1_3" option.
This PR fixes the problem by taking advantage of the recently-added s_client_options() function. It adds a "-no_tls1_3" option whenever:
* $OPENSSL supports TLSv1.3
* The command line doesn't specify any protocol: -ssl2, -ssl3, -tls1, -tls1_1, -tls1_2, or -tls1_3.
* The command line includes the -cipher option
* The list of ciphers that will be sent doesn't include any TLSv1.3 ciphers.
Add TLSv1.3 support to run_cipher_per_proto()
Fix branch
This PR adds support for TLSv1.3 to run_server_preference(). It only provides partial support, as it only works if the support supports and earlier TLS protocol (in order to determine whether the server has a cipher order). It also will only show TLSv1.3 as the "Negotiated protocol" if $OPENSSL supports TLSv1.3.
This PR also fixes a bug in which the variable "proto" was defined as used as both a regular variable and as an array.
If run_server_preference() is performed
* against a server that supports SSLv3 and that does not have a cipher order; and
* using a version of OpenSSL that does not support SSLv3; and
* with the --mapping option set to "rfc" or "no-openssl"
then the "Negotiated cipher per proto" will not show the SSLv3 cipher since cipher[i] will be empty.
This PR addresses issue #660 for run_rc4(), ensuring that support for RC4 ciphers is detected even if no RC4 ciphers are supported with the highest protocol that the server supports.
This PR adds support for TLSv1.3 to run_std_cipherlists().
This PR also provides a partial fix for #660 - addressing the issue only for run_std_cipherlists(). Rather than testing the server once for each cipher list, it tries the server once for each protocol supported by the server. This makes the testing more robust, but adds significantly to the time it takes to perform the tests.
This PR addresses issue #660 for run_sweet32(), detecting if 3DES is used, even if it isn't used with the highest protocol version supported by the server. As with PR #854, this increased robustness comes at the expense of taking addition time to run the test.
In client_simulation_sockets() and tls_sockets(), don't work to create a SOCK_REPLY_FILE that contains the entire server's response (in cases where the response was spread across multiple packets) unless $DEBUG is at least 1.
I believe there is a typo in the second definition of DEBUG_ALLINONE. If I run testssl.sh using the -x option for bash I get the following error:
testssl.sh: line 12714: -false: command not found
In order to santize input better there's a fucntion
now which does the work . ``safe_echo()``.
It is actually the same as ``tml_out()`` but is a bit snappier.
This PR adds initial support for TLSv1.3 to tls_sockets() and for run_client_simulation(). It does not change any of the other functions test TLSv1.3. So, with the exception of run_client_simulation(), the functionality added by this PR can only be tested using the --devel option.
This PR does not include the ability to decrypt the encrypted portions of the server's response. So, it does not support functions that need to see such things as the server's certificate, status information, or extensions (other than key share).
This PR fixes the use of has_server_protocol() in two places.
Currently std_ciphersuites() only tries SSLv2 if the server is known to support SSLv2. This changes it to try SSLv2 unless the server is known to not support SSLv2.
In run_beast(), tests against the server are run to determine support for TLSv1.2, TLSv1.1, TLSv1, and SSLv3 unless the server is known to support that protocol (i.e., even if has_server_protocol() reports that the server does not support the protocol). This changes it so that a test is only performed against the server if has_server_protocol() reports that it doesn't know whether the protocol is supported.
This may not be specific to LibreSSL, but just my local setup. However, when I test using LibreSSL testssl.sh still prints the following message several times:
WARNING: can't open config file: /usr/local/etc/ssl/openssl.cnf
This PR suppresses the error message for several calls to $OPENSSL and so fixes the problem.
The logic is complete now, so that if a protocol has been detected as
supported or not supported it will save a few cycles not to test
this protocol again.
There's probably -- also besides #839 space for improvements. The
advantage of this solution is that ``has_server_protocol()`` also
marks a protocol as tested if the result of the test is negative.
MongoDB listens on port 27017 and will respond to a direct TLS
ClientHello without the need for STARTTLS. Thus, testssl.sh already
mostly works to scan a MongoDB server.
The problem is that MongoDB will also reply to an HTTP GET request with
a mocked up HTTP response, which was tricking the tool into believing it
was dealing with an HTTP server. The response looks like this:
---
HTTP/1.0 200 OK
Connection: close
Content-Type: text/plain
Content-Length: 84
It looks like you are trying to access MongoDB over HTTP on the native
driver port.
---
Thus, the detection algorith must first look for HTTP, then dig deeper
into the body looking for "MongoDB".
The tool output now says:
Service detected: MongoDB, thus skipping HTTP specific checks
As dcooper16 noted in #834 there are checks which test for openssl versions
but don't take LibreSSL in account. This adds checks to it for several
LibreSSL versions (>=2.1.x) which are known to support ``determine_trust()``
and it HAS_DH_BITS.
Moreover engine check has been improved. Older LibreSSL versions (2.1 specifically)
had different error messages, so the previous checks failed. There's also
a CMD_LINE flag now where one can switch the engine support off: NO_ENGINE .
run_renogo from #834 is still an open issue.
All three issues fixed. Terminal code were actually
tow problems: Logging in from Linux with a 256 color xterm
makes tput set AF from OpenBSD hiccup. And the detection
of not-ncurses style underline e.g. was not working under OpenBSD.
The engine fix was done by David Cooper (see #831).
There's also a name of the binary now (OpenSSL/LibreSSL) for tracking
the flavor used.
SWURL contained for historical reasons trailing blanks
for released versions.
This caused an error in pr_boldurl --> html_out which
didn't write the trailing style info and didn't close
the href tag (travis complained.)
This patch removes the trailing blank but it doesn't
fix the error.
This is an initial implementation of the idea I proposed in #791. It includes checks based on draft-ietf-tls-grease as well as checks for specific implementation bugs that have previously been encountered.
This code needs testing. While I know of one server that will fail if the ClientHello contains more than 128 ciphers, I haven't been able to test any of the other code against any servers that have the tested-for bugs.
In addition, there is a need for polishing. The tests are referred to as `--grease`, which may not be a very user-friendly name. In addition, there is no output providing information about what tests are being performed. If a server fails a test, then a warning message is printed and is sent to `fileout()`. If the server passes all of the tests, then there is just a single output of "No bugs found."
At the moment, the code is not run by default. Unless `-g` or `--grease` is explicitly included in the command line, the code in this commit is not executed.
Currently the call to `$OPENSSL s_client` to obtain the certificate returned by the server when SNI is not provided is in `certificate_info()`, which means that it is called once for each certificate found the various called to `get_server_certificates()`.
This PR moves the call to `$OPENSSL s_client` to `run_server_defaults()` so that the call is made only once, even if more than one server certificate was found.
In addition, in most cases the certificate returned by the server when SNI is not provided will already have been retrieved by `run_server_defaults()` (in rounds 8-14), in which case `$HOSTCERT.nosni` can just be copied from there rather than making an additional call to `$OPENSSL s_client`.
During protocol check if a sever answered unexpected with
closing the conenction or another malformed reply the
output was not ok as DETECTED_TLS_VERSION was empty.
This fixes it by filling the variable with a string in ``parse_tls_serverhello()``
and then check in higher level (``run_protocols()``) the content.
Also it seems that I forgot in the commit from yesterday one ``&&`` to
commit in ``run_breach()``
This PR addresses the same issue as 6bb3494d98. In its current form, fileout_insert_warning() cannot be used after fileout_section_header() has been called for the first time.
testssl.sh produces an invalid JSON file if the --json-pretty option is used with the --single-cipher option. The reason is that fileout_section_header() isn't called before run_cipher_match() calls fileout() and fileout_section_footer() is not called afterwards.
There is also a problem with MEASURE_TIME, since the "cleanup" at the end of lets_roll() is not performed.
This PR fixes these problems by adding a call to fileout_section_header() before the call to run_cipher_match() and by copying the code from the end of lets_roll() to run_cipher_match() (just before the call to exit).
Extra client side warning led to a non-valid JSON pretty output. This fixes
this bug by adding an extra object. The objects are named "clientProblem${NUMBER}".
By "extra client side" I mean extra warnings which are not happening during regular
tests -- those are no extra ones and should just warn with ``fileout()`` instead
of ``fileout_insert_warning()``.
Also some ``fileout arg1 WARN`` were patched: WARN is not a finding. It is just
a report that either on the client side something doesn't work as expected or
the server could not be checked during a particular test. WARNING doesn't
exist at all, WARn should be used instead.
Some lines where a warning output to JSON or CSV was missing, was added.
Currently, when `$DEBUG` is 2 and the connection is successful, `parse_tls_serverhello()` prints out information about the server's ephermal (EC)DH key, but nothing else. For example:
```
sending client hello... reading server hello...
dh_bits: ECDH, P-256, 256 bits
sending close_notify...
(183 lines returned)
```
This commit changes `parse_tls_serverhello()` so that information about dh_bits is only displayed if `$DEBUG` is at least 3, making it the same as for other information about the server's response.
In addition, it indents the printing of the information about dh_bits in order to better align with other information displayed at this debug level.
- add forgotten servive FTP and XMPP
- polish other services
- after TLS 1.2 run is finished run a check whether no protocol has been detected and ask the user for confirmation to proceed
In `run_protocols()` for TLS 1.2, try one set of 127 ciphers and if the result isn't a connection at TLSv1.2 then try another set of 127 ciphers before giving up and assuming that TLS 1.2 isn't supported.
FIX#786
Fixed all other occurences so that debug level 2 is showing only minimal information like rough status and errors
Better line breaks for level 2
In ``client_simulation_sockets()`` and ``tls_sockets()`` moved debug output into if statements (may save a bit of time)
Replaced "$DEBUG -eq" by "$DEBUG -ge"
Removed obsolete hb_rounds in ``run_heartbleed()``
Adjusted wide output in vulnerabilities
The data for `run_client_simulation()` currently includes two clients that send version 2.0 CLIENT-HELLO messages (see Appendix E.2 of RFC 5246). Each of the CLIENT-HELLO messages advertises support for newer protocol versions (SSLv3 in the case of IE6XP and TLSv1.0 in the case of Java 6u45). A server may reject one of these version 2.0 CLIENT-HELLO messages, or it may respond with an SSLv2, SSLv3, or TLSv1.0 ServerHello.
The current code in `client_simulation_sockets()` assumes that the server's response with be an SSLv3 or later ServerHello. So, it can support cases in which servers respond with an SSLv3 or TLSv1.0 ServerHello (once PR #800 is accepted to undo the mistake in PR #797), but not cases in which servers response with an SSLv2 ServerHello.
This PR adds code to `client_simulation_sockets()` to check if the server's response is an SSLv2 ServerHello, so that it can process such responses with `parse_sslv2_serverhello()` rather than `parse_tls_serverhello()`.
When a connection is made using SSLv3 or later, `run_client_simulation()` will show to the protocol and cipher selected for the connection. With this PR, if the connection is made using SSLv2, `run_client_simulation()` will just show "SSLv2." In the case of SSLv2, the ServerHello contains a list of all ciphers that the server and client have in common, and it is up to the client to choose one. So, if the client and server have more than one cipher in common, more information about the client would be needed to know which cipher it would choose.
`std_cipherlists()` uses `has_server_protocol()` to determine whether to test for ciphers using SSLv2. However, this was resulting in false negatives due to #759. This PR removes the `has_server_protocol()` check so that SSLv2 is checked whenever the connection attempt with the TLSv1.2 ClientHello failed.
This PR fixes the extraction of the cipher suites in SSLv2 ClientHellos in `client_simulation_sockets()`. Since `client_simulation_sockets()` can only handle SSLv3 and above ServerHellos, it removes any SSLv2 ciphers and converts the other cipher from 3-byte format to 2-byte format.
If vulnernable to POODLE and has no TLS_FALLBACK_SCSV ==> HIGH.
If only run the fallback check and it has none, still label it as MEDIUM but issue
a clear warning that test under this circumstances is incomplete.
This PR adds the same additional check to `client_simulation_sockets()` as was previously added to `tls_sockets()`. It extracts the list of cipher suites offered from each ClientHello and passes the list to `parse_tls_serverhello()` so that `parse_tls_serverhello()` can check that the cipher offered in the ServerHello was included in the ClientHello.
This assumes that a real client would abort the connection if it was presented with a cipher in the ServerHello that it didn't offer in its ClientHello.
Unfortunately, most categories were passing incorrectly, since a
tls_socket() error was interpreted as a success for many of the cases.
Now we explicitely check for the known yaSSL failure and raise a warning
that the test is inconclusive.
New output against --starttls=mysql looks like:
Testing ~standard cipher categories
-----------------------------------
NULL ciphers (no encryption) SERVER_ERROR: test inconclusive due to MySQL Community Edition (yaSSL) bug.
Anonymous NULL Ciphers (no authentication) SERVER_ERROR: test inconclusive due to MySQL Community Edition (yaSSL) bug.
Export ciphers (w/o ADH+NULL) SERVER_ERROR: test inconclusive due to MySQL Community Edition (yaSSL) bug.
LOW: 64 Bit + DES encryption (w/o export) offered (NOT ok)
Weak 128 Bit ciphers (SEED, IDEA, RC[2,4]) SERVER_ERROR: test inconclusive due to MySQL Community Edition (yaSSL) bug.
Triple DES Ciphers (Medium) offered
High encryption (AES+Camellia, no AEAD) offered (OK)
Strong encryption (AEAD ciphers) SERVER_ERROR: test inconclusive due to MySQL Community Edition (yaSSL) bug.
Partial workaround for #793
According to a discussion thread on the IETF TLS WG mail list (see https://www.ietf.org/mail-archive/web/tls/current/msg19720.html), there is at least one TLS server that will fail if the last extension in the ClientHello has contains extension_data of length 0.
Currently, `tls_sockets()` will create such a ClientHello if:
* The padding extension is included, and the length of the ClientHello without the padding data would be between 508 and 511 bytes.
* No padding extension is included, and the caller provided `$extra_extensions` in which the last extension in `$extra_extensions` is empty.
* No padding extension is included, `$extra_extensions` is empty, no ECC cipher suites are offered, and the ClientHello is for TLSv1.1 or below (in this case the next protocol extension would be that last one).
This PR avoids the server bug (in nearly all cases) by ensuring the the padding extension (when present) always contains at least one byte, and by ensuring that when the padding extension is not present that the (non-empty) heartbeat extension is the last extension.
This PR does leave one possible scenario in which the last extension would be empty. If the caller provides an `$extra_extensions` in which the last extension in `$extra_extensions` is empty, `tls_sockets()` does not add a padding extension (or a padding extension is included in `$extra_extensions`), and `$extra_extensions` includes a heartbeat extension, then the last extension in the ClientHello would be empty. This, however, is a highly unlikely scenario, and certainly there are currently no such calls to `tls_sockets()` in testssl.sh.
Another yaSSL server incompatiblity. MySQL protects against CCS
Injection by erroring if it sees to CCS requests before the handshake
completes. But instead of returning a TLS alert, it seems to error up
the stack to MySQL which returns it's own error message.
Debug output looks like:
---
sending client hello,
reading server hello
1st reply:
sending payload #2 with TLS version x03, x02:
tls_content_type: 16 | tls_protocol: 0000 | byte6: 04
2nd reply:
00000000 16 00 00 02 ff 13 04 23 30 38 53 30 31 42 61 64 |.......#08S01Bad|
00000010 20 68 61 6e 64 73 68 61 6b 65 | handshake|
0000001a
test failed, probably read buffer too small (16000002FF1304)
---
This patch adds a custom check for this MySQL specific error, as far
down in the error check path as I can.
If $PROTOS_OFFERED was empty, and thus the protocols unknown, this
function would return true for any protocol you passed it. This caused
most callers to assume TLS1.0, even if the server didn't offer it.
Instead return false and make the caller do an extra lookup.
As reported in #782, some servers will return a ServerHello with a cipher not listed in the ClientHello rather than than return an Alert, if the server does not support any of the ciphers listed in the ClientHello.
This commit modifies `tls_sockets()` to check whether the cipher in the ServerHello was one included in the ClientHello and to fail if it wasn't.
This is the simplest direct socket implementation of the MySQL STARTTLS
protocol.
This is a binary protocol, so it requires a new stream based send
(instead of the current line based send).
In `run_hpkp()` there is a call to `$OPENSSL s_client` that uses `${sni[i]}` as one of the command line options, but `sni` is not defined. My guess is that this was a copy/paste error from `run_client_simulation()`, which is the only function where an `sni` array is defined.
I am guessing that the intention was to use `$SNI` in `run_hpkp()`.
openssl/master branch now supports mysql STARTTLS in s_client
This patch adds support to call and use that s_client support to run
most, but not all (pfs, client simulation) tests.
The socket implementation is stubbed, but not yet functional.
testssl.sh is taking an educated guess which port makes sense to scan,
which one not and for which one to use which starttls handshake upfront.
This minimizes needless sscans and error messages.
Parallel mass testing mode is now not anymore experimental. To
use it a separate flag ``--mode=parallel`` was introduced. Serial
is still the default for now to avoid unexpected conditions.
Both the mode arguement and the default is subject to change.
The parallel mass testing mode can now also make use of a
nmap file. Also the functional test for nmap file was put
into a separate function and made more user safe. Open point is
that we better should use the hostname if the forward DNS record matches.
Fixed logical inconsistency: Ticketbleed was not being tested against a server with client authentication
Some variables in the beginning reordered
This PR attempts to address the outstanding issues with respect to issue #733, mainly by addressing the rules for when a certificate is obtained without SNI.
I believe I discovered the reason for issue #757: f2303a0d79.
This commit removed attempted to replace `$cbc_cipher_list_hex` (which was computed on the fly) with `$cbc_ciphers_hex` (which is static). However, the function was still using `$cbc_cipher_list_hex`, and since it wasn't being initialized to "" at the beginning of the function, the second call to `run_beast()` (to handle the second IP address) just appended to the value created by the first. Then, when the first two bytes were removed from the resulting string the result was a malformed cipher suite list, which caused `tls_sockets()` to fail.
Open issues: 1) The SNI logic 2) The fileout logic. 3) another section with ``trust_nosni -eq 4/8``
For 2): fileout is a general finding MEDIUM [1] which isn't in line now with the pr_*finding
in the section above anymore. It would make sense to punish HTTP services more than others.
Unfortunately he fileout statement cannot be moved below pr_svrty_medium/pr_svrty_high as
trustfinding_nosni hasn't been determined yet.
Fast solution would be probably to move the trustfinding_nosni section above the trustfinding
section.
Still 3) and a different trust over non-SNI makes it difficult -- e.g. Server has CN match only over
SNI but without SNI SAN matches. That's an edge case though which probably doesn't exist (like Bielefeld)
[1] That was WARN before. WARN should indicate a status of testssl that it cannot perform a check
This PR implements the suggestion from #753 for a child process in mass testing to send a signal to the parent to exit if the child encounters an error parsing its command line. At the moment, the child only sends the signal if it encounters an error that results in the `help()` function being called, but that could easily be changed (e.g., to also send a signal if `fatal()` is called in the child process).
In the case of parallel mass testing, the cleanup function needs to call `get_next_message_testing_parallel_result()` for the child that sent the signal, since otherwise the child's error message would not be displayed. Since I cannot tell which child sent the signal, I just call `cleanup()`, which displays the output of all completed child processes. Since the child process will send the signal almost immediately after starting, it can be assumed the that process that send the signal will be the last one that completed, and so its output will be displayed last (so it isn't hidden from the user).
Note that PR #753 is still needed, since there are still scenarios in which a child would not produce any JSON output, but the parent testssl.sh would not exit (e.g., the child process cannot open a socket to the server it is supposed to test). In additional, PR #754 would still be useful, since it would be more user friendly to catch the error in the mass testing file immediately (when possible) rather that partway through a potentially time-consuming testing process.
There is a bug in testssl.sh that occurs if mass testing is being performed, there is an error in the command line for one of the child tests, and either a single HTML file or a single JSON file is being created.
If mass testing is being performed and `parse_cmd_line()` detects an error in the command line for one of the child tests, then it will call `help()`, which will exit the program, resulting in `cleanup ()` being called. `cleanup ()` will call `html_footer()` and `fileout_footer()`. Since `html_header()` and `json_header()` have not yet been called, `$HTMLHEADER` and `$JSONHEADER` will both be `true, and so `html_footer()` and `fileout_footer()` will output HTML and JSON footers, even though no headers have been output.
This PR fixes the problem by having `help()` set `$HTMLHEADER` and `$JSONHEADER` to `false` so that no HTML or JSON footers are created.
A related problem is that if a single JSON file is being created, the parent process will insert a separator (a comma) into the JSON file between the outputs of each child process. However, if there is an error in one of the child process's command lines, then this child process will not produce any JSON output and so the JSON file will have two consecutive separators (commas), which is invalid according to http://jsonlint.com.
This PR provides a partial fix for the problem for parallel mass testing by checking whether a child process has created a non-empty JSON output before adding a separator to the JSON file. It leaves two unresolved problems:
* It does not fix the problem at all for `run_mass_testing()`, where the separator is added before the test with the command line error is run.
* It does not fix the problem for parallel mass testing for the case in which the first child test has a command line error.
started to use the ip parameter to decide where the dns resoultion
takes place (see #739 and #748). --ip=proxy or DNS_VIA_PROXY=true
will mean DNS resolutioni is done by proxy,
furthermore: swapped a few pr_magenta by pr_warning. Generally
testssl.sh should use for warnings a warning function and not
directly a color (we want to be flexible). There are still a few
remainders for fatal() which use bold magenta and thus I haven't
changed yet.
Also html_reserved() is being called also if no html output
is being requested. This could be fixed better probably.
This PR introduces the following changes/improvements to running mass testing in parallel:
* Continuous feedback is provided on the progress of testing, so that testssl.sh doesn't appeared to be frozen, even if it is waiting a long time for a child test to complete. [The feedback text is sent to `stderr` so that it doesn't appear in the log file if the `--logging` option is used.]
* Text sent to `stderr` by a child test is captured and then displayed in context with the appropriate test rather than being displayed at the time the error occurs. [Note that this means that if the `--logfile <logfile>` option is used (with `logfile` being the name of a file), then `logfile` will include both `stdout` and `stderr`.]
* If a test is started but it does not complete (because testssl.sh was stopped or because the test timed out), then a message is displayed indicating that the test didn't finish.
I believe that `run_mass_testing_parallel()` is either ready for use or nearly ready for use. So, I changed the program to use `run_mass_testing_parallel()` rather than `run_mass_testing()` if the `$EXPERIMENTAL` flag is set.
was received, testssl.sh tries 3 times to get memory from the server, If
the server returns different memory it's highly likely vulnerable.
(some more vulnerable devices to test against would be appreciated).
This is the default now -- all other hosts are mostly labled as OK.
Parsing SERVICE should be a little faster and more reliable
Increased needed debug level for output @ parse_tls_serverhello
Sometimes it just does not. In those case also sometimes session resumption
via tickets is supported.
This fixes the output for Session Ticket RFC 5077 and doesn't draw the wrong
conclusion from a missing lifetime hint.
It also tests for ticketbleed first whether there's a session ticket TLS
extension.
When the `--log`, `--logging`, or `--logfile <logfile>` option is being used and testssl.sh is stopped, a "printf: write error: Broken pipe" message tends to appear. From what I can tell, this is a result of the `tee` process being killed before the `cleanup ()` function completes. At the moment, `cleanup ()` doesn't write very much to `stdout`, but if parallel mass testing is performed, then `cleanup ()` may try to write the results of several previously completed tests.
This PR fixes the problem by adding the `-i` option ("ignore interrupt signals") to `tee`.
Note that I have tested this on a Linux desktop and an OS X laptop, but don't have a MS Windows computer on which to test this change.
This PR provides improvements to `run_mass_testing_parallel()`. Currently, `run_mass_testing_parallel()` treats `$MAX_PARALLEL` as the maximum difference between the number of the test whose results were last processed and the number of the most recently started test. This means that test #40 will not be started until the results of test #20 have been processed. I've encountered situations in which tests 21 though 39 have completed, but test #20 is still running, and so no new tests are started.
This PR fixes the problem by checking the status of all running child tests to see if any are complete, rather than just looking at `$NEXT_PARALLEL_TEST_TO_FINISH`. This prevents one slow child test (or a few slow child tests) from slowing up the entire mass testing process.
This PR also changes the basis for determining whether a slow child process should be killed. Rather than waiting `$MAX_WAIT_TEST` seconds from the time that the parent started waiting (which is rather arbitrary), it kills the process if `$MAX_WAIT_TEST` seconds have passed since the child test was started. Given this, and that the above change makes it less likely that a slow child test will slow up the overall testing, I increased `$MAX_WAIT_TEST` from 600 seconds to 1200 seconds.
I added some `debugme` statements that provide feedback on the status of testing, but in non-debug mode there may be a perception issue. If one test (e.g., test #20) is very slow, testssl.sh will not display any results from later tests until the slow test finishes, even though testssl.sh will continue running new tests in the background. The user, seeing no output from testssl.sh for an extended period of time, may think that testssl.sh has frozen, even though it is really just holding back on displaying the later results so that the results will be displayed in the order in which the tests were started.
There is a comment in the `run_client_simulation()` function that says "FIXME: printf formatting would look better, especially if we want a wide option here."
This PR is an attempt at addressing that FIXME and adding a wide option. The proposed wide option prints the same information as the non-wide option, just with the columns aligned. I didn't add any of the additional information that is displayed by other functions in wide mode, since I thought that made the output too wide.
-H is now --heartbleed instead of --headers,
-B is now --breach instead of --heartbleed,
-T is now --ticketbleed (was previously --breach)
bugs fix for run_ccs_injection() where the tls protocols wa not properly passed to the ClientHello
Made use of already determined protocol ( this time only from determine_optimal_proto() ) ==> we shpould use this in run_protocols() too!)
for run_ccs_injection + run_ticketbleed(). For achieving this determine_optimal_proto() needed to be modified so that it adds a protocol
to PROTOS_OFFERED (all_failed is now boolean there)
added two easy functions for converting dec to hex
sockread_fast() is for testing which should make socket erads faster -- albeit it could potentially block the whole thing
`emphasize_stuff_in_headers()` only adds color to the text being printed to the terminal if `$COLOR` is 2. So, the same should be the case for the HTML output.
This PR addresses issue #616, changing `run_cipher_match()` so that only those ciphers that are available are shown, unless the `--show-each` flag has been provided.
It also fixes a problem where the signature algorithm isn't being shown, even if `$SHOW_SIGALGO` is true.
This PR just addresses some places where quotes need to be used to avoid word splitting in case the referenced file, or path to the file, contains space characters.
The previous fix did not work if testssl.sh was found via `$PATH`. This seems to work in all cases. If testssl.sh is found via `$PATH` or if the command line includes a path, then `which` returns a non-empty response; otherwise, `$0` does not include any path, but one needs to be provided, so `$RUN_DIR/$PROG_NAME` is used.
Using "$0" as the name of the executable seems to work as long as "$0" contains a directory name (e.g, "workingfiles/testssl.sh"), but not if it is just the name of the executable (e.g., "testssl.sh"). Specifying "$RUN_DIR/$PROG_NAME" seems to work in both cases, since if "$0" doesn't contain any path information, `$RUN_DIR` is `.`
Use the suggestion "If you want to print the argument list as close as possible to what the user probably entered" from http://stackoverflow.com/questions/10835933/preserve-quotes-in-bash-arguments to create `$CMDLINE` and to print the command lines in `run_mass_testing()` and `run_mass_testing_parallel()`.
This PR addresses issue #702. Rather than create the command line for each child process in `run_mass_testing()` as a string, it creates it as an array, with each argument being a separate element in the array. This was done based on http://mywiki.wooledge.org/BashFAQ/050.
The printing of each child's command line done based on http://stackoverflow.com/questions/10835933/preserve-quotes-in-bash-arguments.
The `$CMDLINE` string remains unchanged, even though it isn't entirely "correct," since http://jsonlint.com/ complains if the "Invocation:" string contains backslashes.
I was doing some testing on my extended_tls_sockets branch and discovered that it was not fully working since the `TLS13_KEY_SHARES` array was empty. According to https://lists.gnu.org/archive/html/bug-bash/2012-06/msg00068.html, there is an issue when trying to initialize a global array inside a function. (The current code initializes `TLS12_CIPHER`, `TLS_CIPHER`, and `TLS13_KEY_SHARES` within `get_install_dir()`, since tls_data.txt is read in that function.) In fact, according to http://stackoverflow.com/questions/10806357/associative-arrays-are-local-by-default, in order to initialize a global variable in a function, one needs to provide the `-g` option, which was only added in Bash 4.2.
This PR seems to fix the problem by moving the reading of tls_data.txt to the main body of the code rather than reading it within the `get_install_dir()` function.
If I understand correctly how `run_hpkp()` should be displaying the list of Backups, the problem shown in the photo attached to #696 occurred because the dangling SPKIs should have been printed on the next line. This PR fixes this by changing the code that prints out the "good" backups to include a newline after printing the CA's name.
Another problem this PR fixes is that `tm_italic()` is being called instead of `pr_italic()`, meaning that the italicized text is being printed to the terminal, but is not being included in the HTML output.
Finally, this PR fixes errors in the definitions of `tmln_italic()` and `prln_italic()`. `tmln_italic()` calls `outln()` and `prln_italic()` calls `tmln_out()` instead of the reverse.
This PR fixes issue #695 by changing the call to `out_row_aligned_max_width()` so that the length of the flag is considered in determining the length of the first line of the output.
This PR fixes two issues with HTML generation that were introduced by a commit on March 31, 2016, "[count_ciphers is now un-sed'ed, minor improvements](a480e5f699)."
The first is that in `std_cipherlists()`, `[[ $DEBUG -ge 1 ]] && outln " -- $1" || outln` was changed to `[[ $DEBUG -ge 1 ]] && outln " -- $1" || outln`. The result being that in the HTML output, all of the tests from `run_std_cipherlists()` appear on the same line. This PR changes the line to:
```
[[ $DEBUG -ge 1 ]] && tm_out " -- $1"
outln
``
so that the line break is added to the HTML output, but the debugging information is not.
The second problem is that the commit on March 31 moved the call in main to `html_header()` until after the calls to `get_install_dir()`, `find_openssl_binary()`, `mybanner()`, `check4openssl_oldfarts()`, and `check_bsd_mount()`. The problem is that each of these functions may call an output function that will call `html_out()`.
If `html_out()` is called before `html_header()` and the command line contains `--htmlfile <htmlfile>`, then "htmlfile" will be written to before `html_header()` is called and then `html_header()` will warn that "htmlfile" already exists and then exit the program.
If `html_out()` is called before `html_header()` and the command line contains `--html`, then anything send to `html_out()` before `html_header()` is called (such as the banner) will not appear in the HTML file.
I did some testing with http://jsonlint.com/ and discovered a missing comma when massing testing is being performed and a single JSON file is being created.
In `run_server_defaults()` the variable `success` is defined twice, once an an ordinary variable and once as an array. The PR removes the incorrect definition. It also removes the definitions of some variables that are no longer used and reorganizes the definitions so that each line has only one variable type.
I also noticed a typo later in `run_server_defaults()` and corrected it.
This PR introduces the environment variable `CHILD_MASS_TESTING`, and uses it as an indicator that testssl.sh is running as a child within mass testing rather than using the `$APPEND` flag. It also makes a number of other changes to make the handling, of HTML, CSV, JSON, and log files consistent, and it fixes a number of bugs related to the generation of these files when mass testing is being performed.
Please let me know if you disagree with any of the changes in this PR, or if you would prefer that it be broken up into multiple smaller PRs.
Some of the changes are as follows:
- When the `$APPEND` flag is true, all of these files are appended to and headers and footers are omitted. (Perhaps this should be changed. Appending to a log file isn't an issue, but appending to a JSON or HTML file without including headers or footers seems to just create an improperly formatted file).
- Following the code in `prepare_logging()`, an error is printed and the program stops if the `$APPEND` flag is false and one of the files to be written to already exists.
Some of the bugs fixed:
Creating log files did not work with mass testing:
- If `--logfile <logfile>` is used, then the parent and each child try to write to "logfile".
- If `--logging` is used, then a log file is created for each child, but an oddly-named log file is also created for the parent. The one created by the parent contains the entire output.
Plain JSON files:
- When `--jsonfile <jsonfile>` is run, there is no comma separating the final finding for one child and the first finding for the next child.
Pretty JSON files:
- When `--jsonfile-pretty <jsonfile>` is called without mass testing, the "target host" line is empty, since `$NODE` has not yet been set.
- When `--jsonfile <jsonfile>` is run with mass testing, there is no comma separating the final finding for one child and the first finding for the next child. In addition, `fileout_pretty_json_banner()` is never called, and the entries for individual tests have insufficient information to determine what is being tested (it lists "service" and "ip", but not port number).
For the final issue, when mass testing is being performed and all output is being placed in a single file, I have the parent call `fileout_pretty_json_banner()`, but tell `fileout_pretty_json_banner()` to not include a "target host" or "port", but then have each child include a "target host" or "port" (when the "service" and "ip" are being printed).
This PR improves `out_row_aligned_max_width()` in a few ways:
* It makes better use of bash's string manipulation capabilities in order to simplify the function.
* It improves the function's performance. One of the most costly parts of `out_row_aligned_max_width()` was the while loop to print each entry in the text. Since there is only one place in the code where the the entries are not all printed the same ways (the list of supported curves printed by `run_pfs()`), the PR changes `out_row_aligned_max_width()` to just return a plain text string, which the calling function prints in the appropriate way. For the curves printed by `run_pfs()`, a new function, `out_row_aligned_max_width_by_entry()` takes care of getting the output from `out_row_aligned_max_width()` and then printing each entry appropriately.
* The PR also introduces a trick so that when the TLS extensions are printed, the text for an extension won't get split across two rows. It does this by replacing the space charters within the text for an extension with "}", formatting the result with `out_row_aligned_max_width()`, and then converting the "}" back to space characters.
`$OPENSSL_LOCATION` is used in `fileout_pretty_json_banner()`, `html_banner()`, `mybanner()`, and `prepare_logging()`, but the value of `$OPENSSL_LOCATION` is populated in `mybanner()`. This is usually okay, since `mybanner()` is always called before the other three functions are called. However, if `$QUIET` is `true`, then `mybanner()` returns immediately, without populating `$OPENSSL_LOCATION`, even though the value of `$OPENSSL_LOCATION` may be needed by one or more of the other functions.
This PR addresses this problem by populating `$OPENSSL_LOCATION` in `find_openssl_binary()` rather than `mybanner()`.
With the commit made on March 26, "partly (1/2) fixing #653," an HTML banner isn't added to the HTML files anymore. A banner should be added to the top of the HTML file if mass testing is being performed and a separate HTML file is being created for each test.
The `$APPEND` flag being `true` is an indicator that mass testing is being performed and that this is one of the individual tests being run. Given that `$APPEND` is `true`, `$HTMLHEADER` being `true` indicates that testssl.sh is creating the file name for the HTML output. So, it is when both flags are `true` that the HTML banner should be created.
`std_cipherlists()` does not include line breaks between tests in the output to the terminal when `$DEBUG` is 1, and it does not include line break between tests in the HTML output whenever `$DEBUG` is greater than 0.
In `create_client_simulation_tls_clienthello()` the variable `sni_extension_found` should be set if the ClientHello includes an SNI extension. Instead it was being set if and only if the ClientHello included some extension other than SNI.
This bug wasn't detected before for two reasons:
* It is rare to have a ClientHello that includes an SNI extension, but no other extensions.
* The code still works correctly if `sni_extension_found` is set even if there is no SNI in the ClientHello.
So, the bug only creates a problem if the browser's ClientHello include an SNI extension and no other extensions (see "BingPreview Jun 2014" in the client_simulation branch).
When HTML output is not being created, the print functions last step is to call `html_out()`, which responds to `return` rather than `return 0`. This causes problems for lines of code that rely on receiving a return value of 0. For example:
```
[[ $VULN_COUNT -le $VULN_THRESHLD ]] && outln && pr_headlineln " Testing for LUCKY13 vulnerability " && outln
```
This PR fixes two minor bugs:
* In `run_hpkp()`, the call to `$OPENSSL s_client` includes the option `-showcerts` twice. This PR removes one of them.
* In `get_server_certificate()`, the first call to `$OPENSSL s_client` includes `$addcmd`, but `$addcmd` has not yet been initialized. Instead, `$SNI` should be used.
Currently there is code to extract TLS extensions in three places, in `get_server_certificate()` and two places in `determine_tls_extensions()`. This PR replaces them with one new function, `extract_new_tls_extensions()`.
In order for the new function to work correctly whether OpenSSL or `tls_sockets()` is being used, this PR also changes `parse_tls_serverhello()` so that extensions are formatted in the file it creates in the same way as they are formatted by OpenSSL.
- in gerneral better performance measurements , starts from the real beginning (almost)
- allows results to put into file (MEASURE_TIME_FILE=google.txt testssl.sh google.com)
- PS4 improved: has now a performance debugging options (big solution)
- PS4 with proper alignment
- SCAN_TIME is now global so that it can be used not only by JSON-PRETTY (small performance debugging options uses it)
- prepare_debug() has now debugging stuff only, rest went to prepare_arrays()
This PR is the same as #661, except that it applies to the 2.9dev_html branch. It also fixes a bug in `prettyprint_local()` when a pattern to match is provided as input.
While doing some performance testing I discovered that `normalize_ciphercode()` is very slow. This PR simplifies the function and speeds it up significantly. This PR also addresses the TODO item in `normalize_ciphercode()` by eliminating the global variable HEXC.
When I view testssl.sh in KDE's text editor (kate), the code after "INFO" line in `show_finding()` isn't highlighted correctly, and the highlighting doesn't correct again until `pr_liteblueln()`. This PR fixes the highlighting issue by adding spaces between "]]" and ")". To be "safe," I also added spaces between "(" and "[[", and similarly added spaces in `is_json_format()`.
This PR ensures that each HTML file produced by testssl.sh only includes a single header, at the top, and a single footer, at the end. It also tries to ensure that the short-version banner is only placed at the top of the HTML file if (1) mass testing is being performed and (2) the results of each test is being placed in a separate file.
It also moves some of the logic out of main and into `html_header()`.
So far I haven't seen any HTML reserved characters (&, <, >, ", ') in the strings processed by `emphasize_stuff_in_headers()`, so this PR may be unnecessary. However, this PR will ensure that any such characters will be properly escaped in the HTML output.
"=~" doesn't need quotes if there's a text string one wants to match against (and shellcheck complains
about this)
pr_magenta shouldn't be used anymore as the logic what color we use should be done
some place else.
This branch is for getting the HTML patch from @dcooper16 into 2.9dev
Change to David's PR:
* removed HTMLHEADER. We always want that (in fact for flat JSON this is missing and needs to be added)
* not sure what this change does to --file
* changing of names They were redundant sometimes (pr_*_term )
* some formatting for readbility
Open points:
* there's a loop and a segfault --> tm_done_best
* HTMLHEADER: --file
* the former sed statement aroung L1900 for the header was way more readable. The combined
html+terminal version is just too much. Maybe a switch whether HTML is requested
is better so that this can be separated.
* Then e.g. "<span style=\"color:olive;font-weight:bold" can be kept in a variable
* any reason we need the text length here?
* what went into main here is too much. Actuallly what I put already in there bothered
me as too much logic and not obvious dependencies are in here. Now it's worse :-)
Can't this be just similar to JSON or CSV -- a seperate function with hooks
not in main()?
* minor thing: TERM_WIDTH is for HTML is maybe not the best. But that can be
tackled later
This PR attempts to address #631. It allows four choices for the `--mapping` option. "openssl" (default), "rfc", "no-openssl", and "no-rfc".
* "openssl" is the current default
* "no-rfc" shows only the OpenSSL names (just as it currently does)
* "rfc" shows the RFC name rather than the OpenSSL name for things that are not in wide now (just as it currently does). But now, in wide mode, it shows the RFC name first (further to the left) and the OpenSSL name second.
* "rfc-only" shows only the RFC name.
If the `--mapping` option is not provided, this is the same as "openssl". If the "cipher-mapping.txt" file cannot be found, then testssl.sh runs as if "no-rfc" had been requested.
It seems that the head command on OS X does not accept a negative number as a value for the "-n" parameter. This PR provides an alternative method for removing the "generator: " line without using "head."
Some sites have a long list of IP addresses and some IP addresses have a long list of DNS names that map to them.
This PR changes `display_rdns_etc()` to use `out_row_aligned_max_width()` to print the other IP addresses in `$IP46ADDRs` and to print `$rDNS`.
RFC 7633 introduces the TLS Features certificate extension, which contains "Features:
> The object member "Features" is a sequence of TLS extension identifiers (features, in this specification's terminology) as specified in the IANA Transport Layer Security (TLS) Extensions registry. If these features are requested by the client in its ClientHello message, then the server MUST return a ServerHello message that satisfies this request.
The main purpose of this certificate extension is to implement "must staple." If the extension is present in a TLS server's certificate and it includes status_request, then the server MUST include a stapled OCSP response if the client requests one. (The same applies for the status_request_v2 extension.)
This PR adds a check to `certificate_info()` of whether the server supports must staple (i.e., whether its certificate includes a TLS Features extension with "status_request"). It also changes the output for "OCSP stapling" in the case that the server did not staple an OCSP response. It indicates that:
* it is a critical issue if the certificate specifies "must staple"
* it is a low severity issue if the certificate does not specify "must staple," but the certificate does include an OCSP URI.
* it is not an issue at all if the certificate does not specify "must staple" and certificate does not include an OCSP URI.
`run_server_preference()` prints out the server's Negotiated cipher in a different color depending on the quality of the cipher. However, there is a "FIXME" since CBC ciphers are supposed to be flagged, but it is not easy to identity all CBC ciphers from their OpenSSL names.
This PR partially addresses this. It creates a separate function for printing a cipher based on its quality. Whenever possible it determines the quality of the cipher based on the RFC name. However, if it is provided an OpenSSL name and no cipher-mapping.txt file is available, it will follow the current (imperfect) logic for determining the cipher's quality.
The function also returns a value that indicates the quality of the cipher provided, with higher numbers indicating better ciphers. This return value is used by `run_server_preference()` to determine how to populate the "severity" field when calling `fileout()`.
In the case that `tls_sockets()` is being used and the server incorrectly fails the connection rather than downgrading, testssl.sh is printing "not offered" on one line and then the error message on the next line, but all the text should appear on one line (as it does when testing TLS 1 and TLS 1.1).
Since the test for TLS 1.2 in `run_protocols()` now uses `tls_sockets()` whenever `$ssl_native` is `true` (i.e., there is no longer a requirement for `$EXPERIMENTAL` to be true as well), the `$EXPERIMENTAL` flag should no longer be checked if the return value is 1.
Rearrange code so that in the case of just a single test, `parse_hn_port()` is not called earlier than it was previously unless it needs to be called in order to create the HTML file name.
Doing this ensures that the banner is displayed even if the `$URI` cannot be parsed (except in the case that the `$URI` needs to be parsed in order to create a file name) and that any error messages created by `parse_hn_port()` will be included in the HTML, if possible.
Add option for testssl.sh to create the HTML file name. If testssl.sh creates the file name, then, in the case of mass testing, a separate HTML file is created for each test (i.e., for each line in the file provided to `--file`).
I have a test server that I configured to support only SSLv3 and TLSv1.2. When I set `SSLHonorCipherOrder` to `off` I get the following results:
```
ECDHE-RSA-AES256-SHA: SSLv3 ECDHE-RSA-AES256-GCM-SHA384: TLSv1.2
```
The current code, when printing TLSv1.2 checks whether `${cipher[4]}` is empty, and since it is assume no previous protocol (SSLv2, SSLv3, TLSv1, TLSv1.1) was supported and so doesn't output a newline before outputting the cipher and protocol for TLSv1.2.
This PR fixes that by changing to code to look at the previous non-empty cipher (if there is one), even if that does not come from the previous protocol.
When a list of cipher suites is being displayed using `neat_list()`, testssl.sh shows the cipher suite's OpenSSL name and (in most cases) the RFC name as well. However, in all other cases only the OpenSSL name is shown.
This PR adds the option to have cipher suite's RFC names shown instead of the OpenSSL name, by including `--mapping rfc` in the command line. [Note: if the cipher-mapping.txt file cannot be found, then the `--mapping rfc` option is ignored and the OpenSSL names are shown.]
This PR seems to be related to issue #9, but #9 may be been referring to the output created by `neat_list()`.
Gray should appear for COLOR=1 or COLOR=2.
Since `pr_grey()` is basically the same as `out()` for COLOR=0, `mybanner()` should just call `pr_grey()` without checking the value of `$COLOR`.
Change `emphasize_stuff_in_headers()` to use olive and bold olive rather than brown and yellow. This matches what `aha` creates and appears similar to what is displayed in the terminal on a Mac. Also, yellow text is very difficult to read.
Found more places where output should only go to terminal, or where it was only going to the terminal (e.g., printf) but should also be in the HTML. Also added the ability to include active URLs in the HTML output.
To Do: Handle automatic generation of HTML file name and support for parallel testing.
Introduced "trick" so that if the `--file` option is used, `html_header()` will only be called once before anything is printed and `html_footer()` will only be called once after all printing is complete. With this, `html_header()` now delete the output file if it exists.
Also introduced the `html_reserved()`, which is called for all text to be sent to `out_html()`. `html_reserved()` converts any HTML reserved characters (", ', &, <, >) to their corresponding entity names (", ', &, <, >).
This PR addresses the issue raised in #623. This PR is based on the function `out_row_aligned_max_width()` that I proposed in #623, but the `out_row_aligned_max_width()` in this PR is a little different. It takes a fourth parameter, which is the function to use to print each word in the text string to be printed. This is used in `run_pfs()` so that the "Elliptic curves offered" can be printed using this function (some servers support 25 curves), while still having the curves printed using color-coding to indicate the quality of each curve.
I somewhat arbitrarily have each line wrap at 120 characters, but that could be changed (e.g., to `$TERM_WIDTH`).
For the most part I used the RGB values for xterm from https://en.wikipedia.org/wiki/ANSI_escape_code#Colors for the HTML colors, but with a few exceptions. For example, I did not use "yellow" for `pr_svrty_low()`, since that color is very difficult to read. I also used a different color for `pr_svrty_medium()` so that `pr_svrty_medium()` would appear more red than `pr_svrty_low()`.
These color choices could use more adjustment.
This PR adds the option to generate HTML. The code was created as follows:
* For each output function (`out()`, `outln()`, `pr_liteblue()`, etc.) I created two functions: one that just outputs to the terminal and one that outputs to the terminal and to the HTML file (if an HTML file is to be created).
* I modified the code so that any output that should appear in the HTML file in addition to being displayed on the terminal is sent through one of the display functions: out()`, `outln()`, `pr_liteblue()`, etc.
* I created a new function `retstring()` to use in place of `out()` when a function is creating a string to be "captured" by the calling function.
* I modified the code so that no string returned by a function includes color-coding escape characters.
In the revised code that was created to address #587, nothing is printed after "OCSP URI" if there is neither an OCSP URI nor a CRL URI. Instead, "--" should be printed.
I still believe that there is an inconsistency in the reporting of the output, however. At the moment, the "Certificate Revocation List" and "OCSP URI" lines indicate it is acceptable as long as the certificate contains either a CRL URI or an OCSP URI. However, the "OCSP stapling" line reports a minor finding if an OCSP response was not included in the server's reply. Shouldn't we just assume that if the certificate doesn't include an OCSP URI, then it wouldn't be possible for the server to obtain an OCSP response to staple to its reply? If so, then it seems that no OCSP stapling should only be considered a finding if an OCSP URI is present.
This PR fixes issue #601.
The fix for OCSP URLs was easy. I don't entirely understand the first `awk` command in the line to extract the CRL URLs, but I tested it on several certificates and it seems to work correctly (ensuring that the only "URI" lines in the input to the second `awk` command are from the CRL Distribution Points extension).
RFC 4492 introduced the Supported Elliptic Curves Extension, but this extension was renamed Supported Groups in RFC 7919. Following RFC 7919 (and TLSv1.3), `parse_tls_serverhello()` refers to this extension as "supported groups/#10". Since, at the moment, OpenSSL's s_client refers to this extension as "elliptic curves/#10", the extension sometimes appears twice in the "TLS extensions" line, if it is detected by both OpenSSL (in `get_server_certificate()`) and `tls_sockets()` (in `determine_tls_extensions()`):
```
TLS extensions (standard) "renegotiation info/#65281" "elliptic curves/#10" "EC point formats/#11" "supported groups/#10"
```
This PR fixes the problem of the extension appearing twice in the "TLS extensions" line by replacing any instances of "elliptic curves/#10" with "supported_groups/#10" in the `$tls_extensions` line extracted from `$OPENSSL s_client`. This PR also changes "supported groups/#10" to "supported_groups/#10" in `parse_tls_serverhello()`, since the current development branch of OpenSSL uses "supported_groups" to refer to this extension (see https://github.com/openssl/openssl/pull/1825).
This PR increases the width of the "Encryption" column printed by `neat_list()` in order to allow room to print "CamelliaGCM."
This is the alternative fix to the problem that was first addressed in #524. This PR obsoletes PR #530.
This PR changes testssl.sh so that when ciphers are being listed in wide mode (i.e., using `neat_list()`) and the `--show-each` option is set, ciphers that are not available are printed in light grey, whereas ciphers that are available continue to be printed in black. This makes it easier to distinguish between ciphers that are available and those that are not (the "available/"not a/v" column remains).
This PR does not change the way that ciphers that are available are printed, but it includes a hook that would allow that to change. For example, for ciphers that are available, the name of the cipher suite could be printed in a different color depending on its quality (as is done for the "Negotiated cipher" in `run_server_preference()`). The same could be done for the "Encryption" and "Bits" columns.
- run_logjam(): determine dh bit size and based on this mark the common primes as more or less vulnerable
- run_logjam(): renamed remaining dhe variable to dh
- further house keeping in run_logjam()
In some cases, the "TLS extensions" line output for the "--server-defaults" option will not show `"encrypt-then-mac/#22"` even if the server supports this extension. The reason is that a server will only include this extension in the ServerHello message if it supports the extension and the selected cipher is a CBC cipher. So, if `determine_tls_extensions()` connects to the server with a non-CBC cipher, then it will not detect if the server supports the encrypt-then-mac extension.
It is possible that support for the extension will be detected by `get_server_certificate()`, but only if one of the calls to that function results in a CBC cipher being selected and OpenSSL 1.1.0 is being used (as prior versions did not support the encrypt-then-mac extension).
In this PR, if `determine_tls_extensions()` is called and `$TLS_EXTENSIONS` does not already contain `"encrypt-then-mac/#22"`, then an attempt will be made to connect to the server with only CBC ciphers specified in the ClientHello. If the connection is not successful (presumably because the server does not support any CBC ciphers), then a second connection attempt will be made with the "default" ciphers being specified in the ClientHello.
en.wikipedia.org is an example of a server that supports the encrypt-then-mac extension, but for which the support is not currently detected (unless OpenSSL 1.1.0 is used) since in the call to `determine_tls_extension()` a non-CBC cipher is selected.
This PR changes `read_dhbits_from_file()` so that, when the "quiet" parameter is absent, the selected curve is shown in addition to the number of bits. This PR only affects the output of `run_client_simulation()` and the `Negotiated cipher` in `run_server_preference()`.
There are two places in `run_client_simulation()` in which `$OPENSSL s_client` is called, after which there is a `debugme echo` line to display the `$OPENSSL s_client` command line when testssl.sh is being run in debug mode, and then `sclient_connect_successful $? $TMPFILE` is called to determine whether `$OPENSSL s_client` successfully established a connection.
So, `sclient_connect_successful()` is being passed the result of the `debugme()` call, which always returns 0, rather than the result of the `$OPENSSL s_client` call.
This PR fixes the problem by moving the `debugme()` line to before the call to `$OPENSSL s_client`, so that `sclient_connect_successful()` is passed the results of the `$OPENSSL s_client` call.
Starting with OpenSSL 1.1.0, s_client will not offer TLS compression methods, even if OpenSSL is compiled with zlib support, unless the `-comp` flag is included in the command line.
This PR changes `run_crime()` to use `tls_sockets()` rather than failing if `$OPENSSL` lacks zlib support, unless `$SSL_NATIVE` is `true`.
At the moment, the ClientHello created by `socksend_tls_clienthello()` only specifies the NULL compression method. So, this PR adds a new parameter to `socksend_tls_clienthello()` and `tls_sockets()` to allow to caller to request that additional compression methods (DEFLATE and LZS) be specified in the ClientHello.
This PR makes another change to `run_crime()`. At the moment, if `$OPENSSL s_client` fails to connect to the server, `run_crime()` will report that the server is not vulnerable, since the output from `$OPENSSL s_client` includes the line "Compression: NONE" (see below). This PR changes that by checking whether the connection was successful, and reporting a "test failed (couldn't connect)" warning if it wasn't successful, rather than reporting "not vulnerable (OK)".
```
CONNECTED(00000003)
140338777061024:error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version:s23_clnt.c:769:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 389 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1483645971
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
```
This PR fixes a few bugs in `sslv2_sockets()`. The main issue is that a server may not send the entire ServerHello in a single packet. If it doesn't and the full response is being parsed (i.e., certificate and list of ciphers), then `parse_sslv2_serverhello()` will encounter errors, since it assumes that it has the entire ServerHello. This PR compares the length of the response to the length of the ServerHello as specified in the first two bytes of the response and requests more data from the server if the response appears incomplete.
This PR also modifies `parse_sslv2_serverhello()` to check for more errors. It compares the length of the response it has been provided to the specified length (`$v2_hello_length`) and returns an error if the response is shorter than `$v2_hello_length` and the full response is supposed to be parsed. It will also check whether there was an error in converting the certificate from DER to PEM format and will return an error if there was (and it will suppress the error message).
Some servers respond to an SSLv2 ClientHello with a list of all SSLv2 ciphers that the server supports rather than just a list of ciphers that it supports in common with the client (i.e., that appear in the ClientHello). This PR changes the sockets version of `std_cipherlists()` so that, if `sslv2_sockets()` is successful, it checks whether there are any ciphers in common between the ClientHello and the ServerHello before declaring that the server supports the specified cipher list.
Some servers respond to an SSLv2 ClientHello with a list of all SSLv2 ciphers that the server supports rather than just a list of ciphers that it supports in common with the client (i.e., that appear in the ClientHello). This PR changes the sockets version of `run_freak()` so that, if `sslv2_sockets()` is successful, it checks whether there are any ciphers in common between the ClientHello and the ServerHello before declaring that the server supports an export RSA cipher.
If `determine_tls_extensions()` does not create a temporary file (`$TEMPDIR/$NODEIP.determine_tls_extensions.txt`) then `run_server_defaults()` will display error messages when an attempt is made to copy this file or to search (grep) it. This may happen if `$OPTIMAL_PROTO` is `-ssl2` or if `determine_tls_extensions()` uses sockets and `parse_tls_serverhello()` encountered an error and did not create a temporary file (`$TEMPDIR/$NODEIP.parse_tls_serverhello.txt`). This PR fixes this by only trying to copy and search `$TEMPDIR/$NODEIP.determine_tls_extensions.txt` is `$OPTIMAL_PROTO` is not `-ssl2` and `determine_tls_extensions()` was successful (return value 0).
In response to your request in #572, this PR provides a starting point for addressing #120. It adds code to `run_logjam()` to try connecting to the server using any cipher that uses an ephemeral DH key. If successful, it gets the server's ephemeral key (in OpenSSL's PEM format) and then extracts the prime from the key and places it in `$dh_p`. So, all that needs to be done at this point is to compare `$dh_p` against a set of "bad" primes. I'm not sure if I'll be able to work on that part soon, so if someone else has the time, that would be great.
I actually found the `-msg` option easy to use. I moved the code in `parse_tls_serverhello()` that extracts the DH ephemeral public key from the ServerKeyExchange message into a separate function. Then, if using OpenSSL with the `-msg` option, I extract the ServerKeyExchange message from `$TMPFILE` and call this new function to extract the key and convert it to PEM format. That way the new code in `run_logjam()` can use either `$OPENSSL` or `tls_sockets()`.
