This PR modifes pr_cipher_quality() as proposed in #1548 so that GOST ciphers are handled correctly. It changes pr_cipher_quality() so that the OpenSSL name is used in cases in which no RFC name is defined. It also adds a case statement for GOST so that GOST ciphers (that do not use MD5 or Null encryption) are marked as pr_svrty_low (as they are in run_cipherlists) rather than just being assigned the default rating (5).
This commit fixes the way pr_cipher_quality handles the OpenSSL names of some ARIA ciphers that either provide no authentication or that use CBC padding.
This commit makes several changes to the way that ciphers are rated by pr_cipher_quality:
* It upgrades SEED ciphers to considered as strong as the corresponding AES ciphers.
* It downgrades ciphers that use AEAD, but that use a non-FS key exchange (TLS_DH_*, TLS_ECDH*, TLS_PSK_WITH_*) from best to good, thus giving them the same rating as AEAD ciphers that use static RSA (TLS_RSA_*).
* It downgrades some CBC ciphers to low (4) that are currently rated as neither good nor bad (5).
* It modifies the ratings created using OpenSSL names to provide the same ratings as those created using RFC names.
The permitted values for $DISPLAY_CIPHERNAMES are "rfc-only", "openssl-only", "openssl", and "rfc". However, get_install_dir() incorrectly sets $DISPLAY_CIPHERNAMES to "no-rfc" if it cannot find the $CIPHERS_BY_STRENGTH_FILE. ("no-rfc" is the string users would specify at the command line for the --mapping option, but not the value that $DISPLAY_CIPHERNAMES is set to internally).
As noted in #1481, testssl.sh has a problem with printing percent ('%') characters.
At one point, the function out() was implemented as `/usr/bin/printf -- "${1//%/%%}"`. When this was the case, any '%' needed to be replaced with '%%' since '$1' was being used as the format string. This was changed, however, by 8a2fe5915a. Since the format string is now "%b" rather than '$1', the replacement is not needed anymore. Instead, the replacement now causes any '%' to be printed to be duplicated.
This problem does not happen very often, but does sometimes occur when a '%' character appears in a URI, such as in an HTTP redirect, a certificate revocation list, or an OCSP URI.
.. as it may not be available everywhere, see #1521 (NixOS).
This commit replaces all instances from pwd or /bin/pwd by $PWD.
It is a bash internal and the fastest. Also it added some quotes
to PWD a it may contain white spaces in the future (currently
there's a check for it that it won't)
.. as it may not be everywhere available, see #1521 (NixOS).
This commit replaces all instances from pwd or /bin/pwd by `pwd -P`
(-P -> no symbolic link)
This PR fixes two issues related to the generation of HTML files.
First, text that is to appear in the HTML file is first passed through html_reserved() to replace reserved characters with their corresponding entity names (e.g., '>' becomes '>'). html_reserved() seems to work correctly on Ubuntu Linux, but it does not work as expected on MacOS. On MacOS, rather than converting '>' to '>', it gets converted to '\>', and the backslash is rendered by browsers.
This PR appears to fix the problem. However, given that the original version of html_reserved() was not portable, this revised version should be tested on multiple platforms.
I also noticed that in almost every case in which a string is passed to html_out(), it is first run through html_reserved(), but for some reason that is not the case in out() and outln(). I can't see any reason why html_reserved() is not called first in these two cases, so this PR adds in the calls.
This commit fixes two minor issues related to HSTS_MIN:
* If there is a misconfiguration the recommended max-age should be based on $HSTS_MIN rather than being hardcoded to 15552000 seconds = 180 days.
* If max-age is exactly $HSTS_MIN, testssl.sh shouldn't say that max-age is too short while also say that >= $HSTS_MIN seconds is recommended.
This commit addresses two bugs: #1506 and #1508.
First, the variable rDNS can contain multiple lines due to multiple PTR DNS
records, though this is not recommended. In those cases the multiple PTR DNS
were concatenated on the screen, without any blank.
Secondly - depending on the name server entries and on the output of the DNS
binaries used it can contain non-printable characters or characters which are
printable but later on interpreted on the output device (\032 was mentioned
in #1506) which on the screen was interpreted as octal 32 (decimal 26 = ▒,
try echo "\032"), so basically a terminal escape sequence was smuggled
from the DNS server to the screen of the users. In JSON pretty output we
had also this escape sequence which was fine for jsonlint but caused jq
to hiccup.
Fix: we use a loop to check for each FQDN returned. There we remove chars which
under those circumstances can show up. The blacklist is taken from RFC 1912
("Allowable characters in a label for a host name are only ASCII, letters, digits,
and the `-' character").
Since, in cases in which the server enforces a cipher order, both run_cipher_per_proto() and run_server_preference() list every cipher supported by the server for each protocol, there was a discussion at one point about eliminating run_cipher_per_proto() and extending run_server_preference().
This PR takes a step in that direction by providing the option to present the "Cipher order" in wide mode.
This PR fixes one Shellcheck issue:
In testssl_3.1dev_20200208.sh line 2395:
HEADERVALUE="$(fgrep -Fai "$key:" $HEADERFILE | head -1)"
^-- SC2197: fgrep is non-standard and deprecated. Use grep -F instead.
The man page for grep states that fgrep is the same a grep -F and that grep is deprecated. So, fgrep -F is just redundant.
Currently, the function neat_list() uses the variable "export", but does not define it. The result is that "export" variable in the calling function is used.
This PR fixes that by defining "export" as a local variable in neat_list() and by setting its value via a new parameter to the function.
This PR also removes a "FIXME" from run_rc4() since the problem has already been fixed.
This PR fixes a minor bug in get_pub_key_size(). If the key size is being determined manually and length encoding requires 4 bytes, then the current code computes the length incorrectly. This is a very insignificant bug, since does not apply to RSA or ECC keys, and the key would have to be at least 16 megabytes long for it to require 4 bytes to encode.
This PR also cleans up get_pub_key_size() a bit by replacing `i=$i+...` with `i+=...` and by enclosing math in `$(( ... ))`.
Hostnames can contain a trailing dot (and sometimes they should).
If they are supplied to testssl.sh however they will be also interpreted
as a URL PATH when the servive is HTTP.
This commit fixes that.
This switch had no effect. There was probably a regression
problem as it worked before.
Besides fixing that the large case statement in parse_cmd_line()
was simplified, in a sense that banner and help functions were
moved to a separate case statement.
This PR adds support for post-handshake messages when using sockets with TLS 1.3 connections. If a TLS 1.3 connection is established and the connection is to remain open after tls_sockets() finishes, then after the client's Finished message is sent the master secret and the application traffic keys are computed. This PR also adds two new functions to send and receive application data over a TLS 1.3 connection.
This PR also includes two proofs-of-concept for the use of the new functions. receive_app_data() is called immediately after the client's Finished message is sent. Some server's will send new session tickets immediately after the handshake is complete. If they do, then the code will decrypt and parse the session ticket messages.
This PR also modifies service_detection() to try using sockets if the server only supports TLS 1.3 and $OPENSSL does not support TLS 1.3. After the handshake is complete, this code sends an HTTP GET request and reads the response. The code is fairly slow and it doesn't always work. However, since it is only used in cases in which $OPENSSL cannot work, it can't hurt to try using sockets.
Due to a bug, the shellcheck program will complain if a variable is defined as an array but is later used as an ordinary string, even if the two uses are locally defined variables in different contexts. The error message is:
SC2178: Variable was used as an array but is now assigned a string.
While the warnings are not highlighting any actual problems in testssl.sh, this PR gets rid of the warnings by renaming a few variables.
The implementation of AES-GCM in #1473 is much slower than the original version, even when the authentication tag is not being computed. This PR modifies the code in gcm() in order to significantly speed up the encryption/decryption time (when authentication tags are not being computed).
This PR adds processing of the Finished messages in TLS 1.3 handshakes. It also addresses some shellcheck issues.
If in debug mode, the HMAC of the transcript hash of the handshake context ($msg_transcript) is computed and compared against the Finished message sent by the server.
If the full server response is parsed and the connection with the server is not to be closed when tls_sockets() completes, then the TLS 1.3 handshake is completed by creating the client Finished message and sending it to the server.
This PR reorganizes the code for deriving TLS 1.3 symmetric keys in order to facilitate implementing the full key schedule. For example, rather than having a single function to derive the handshake traffic keys, this PR creates one function to derive the handshake secret and a separate function to derive the handshake traffic keys. The second function has been generalized so that it can derive either client or server traffic keys. Separating into two functions also makes the handshake_secret available for later use to derive the master secret and then the application traffic secrets and the application traffic keys.
This PR also changes where there message transcript is created, a message transcript will also be needed to derive the application traffic secrets. This PR includes the code to add the messages to the initial message transcript that will be needed for the input to the application traffic secret derivation function.
RFC 8446 specifies cipher suites that use three symmetric encryption algorithms, all of which are Authenticated Encryption with Associated Data (AEAD) algorithms. In each of these algorithms when data is encryption an authentication tag is created, which allows the recipient to verify that the data has not been modified. The authentication may also cover some additional data that was not encrypted.
The current implementations of these algorithms in testssl.sh decrypt the ciphertext, but do not check that the authentication tag is correct (which involves the recipient computing the correct tag for the received data and then comparing it to the provided tag). While testssl.sh can get away with not checking authentication tags when receiving data, the ability to compute authentication tags is needed in order to send encrypted data as TLS servers would reject any encrypted data that did not have a correct authentication tag. Being able to send encrypted data is necessary to be able to complete the TLS 1.3 handshake.
This PR replaces the current implementations of the symmetric encryption algorithms with full implementations of each of the algorithms. These full implementations include the ability to encrypt data for sending, and can also verify the authentication tag when decrypting data. Since the Bash implementations of these algorithms is very slow, the decryption code is designed to only compute and check authentication tags in debug mode.
While the implementation of the code to compute authentication tags for AES-CCM was based on NIST Special Publication 800-38C, I was not able to implement the code for AES-GCM or Poly1305 from their specifications (NIST Special Publication 800-38D and RFC 8439, respectively). So, I would very much like to thank the implementers of https://github.com/mko-x/SharedAES-GCM and https://github.com/floodyberry/poly1305-donna. The implementations of AES-GCM and Poly1305 in the PR were developed by translating the C code in https://github.com/mko-x/SharedAES-GCM and https://github.com/floodyberry/poly1305-donna into Bash. I don't understand what that code is doing, but it seems to work. :-)
I have only tested this code on a computer with a 64-bit operating system. While I have not tested it, I believe that the decryption code will work with 32-bit integers if not in debug mode (i.e., if not trying to compute the authentication tags). I also believe that the AES-CCM code for computing authentication tags will work with 32-bit integers. However, AES-GCM and Poly1305 code for computing authentication tags will definitely only work on systems that have 64-bit integers. So, on systems that do not have 64-bit integers, encryption will not work for AES-GCM or ChaCha20-Poly1305, and decryption will not work for these algorithms if in debug mode.
For a fresh start it seemed a good idea to cleanup
the order of functions and some variables so that
those with the same functionality are somewhat grouped.
Some of the functions have now a header and a foooter
to make it easier to spot and use then. Also for added future
functions the hope is that they will be put where they better
fit
This PR simplifies the code for determining which draft version of TLS 1.3 a server is offering by making use of a simple regular expression and $BASH_REMATCH rather than looping through every possible draft version.
PR #1463 changed run_ssl_poodle() to only run the test if it is known that the server supports SSLv3. However, support for SSLv3 may be unknown at the time run_ssl_poodle() is run (e.g., if the server supports TLS 1 and SSLv3, and run_ssl_poodle() is the first test performed). So, run_ssl_poodle() should perform testing unless it is known that SSLv3 is not supported.
There's a check for >825 days certificate lifetime. That
check emits a debug statement when the lifetime is within
this limit. It does that also when the certificate expired.
This commit adds now the word "total"
DEBUG: all is fine with total certificate life time
to make sure the life time left not is what should be understood.
Several vulnerability checks add a time penalty when the server
side only support TLS 1.3 as The TLS 1.3 RFC 8446 and implementations
known so far don't support the flaws being checked for.
This PR adds "shortcut" checks for all TLS 1.3, assuming that the
TLS 1.3 implementation is correct which seems at this time a valid
assumpution. That either saves a TCP connect or at least some logic to
be executed. Also in some cases a TLS 1.3 only server emitted unnecessary
warnings, see #1444.
If $DEBUG -eq 1 then it outputs information that a shortcut was
used. It doesn't do that in other cases because the screen output
seems too obtrusive.
It also adds a shortcut for beast when SSL 3 or TLS 1.0 is is known
not to be supported.
This commit radds 747fb039ed which
was accidenially reverted in 45f28d8166.
It fixes#1462.
See also #1459.
* Replace "! -z" with "-n"
* Replace "openssl' with "$OPENSSL"
* Redirect stderr output of $OPENSSL to /dev/null to supress "WARNING: can't open config file: /usr/local/etc/ssl/openssl.cnf" message (see #833)
* Remove unnecessary spaces from $GET_REQ11 string.
The CA browser form agreed on a validity period of 825 days or less
(https://cabforum.org/wp-content/uploads/CA-Browser-Forum-BR-1.5.3-redlined.pdf,
p4).
PR #1427 addressed that. However when an issuer signed/issued a certificate
with exactly 825 days, the check reported incorrectly that the life time
is too long.
This commit addressed that by adding a second to the calulation. Also the
output takes into account that it must be over ('>') 825 days, not '>='.
See #1070, kudos @poupas.
In addition it checks whether the first result was positive (in
terms of a finding). If so it does 4 rounds and checks the
result. So that other servers won't be penalized with 4 seconds.
derive-handshake-traffic-keys() uses the variables `derived_secret`, `server_write_key`, and `server_write_iv`, but they are not declared as local variables of the function. This PR fixes that.
As noted in #1273, there are some environments that will not allow writing to /dev/stdout. PR #1277 was an attempt to address that problem (along with an unrelated problem), but it appears that work on #1277 has been abandoned.
At the moment, "/dev/stdout" is only used as a parameter to asciihex_to_binary_file (in fact, most calls to asciihex_to_binary_file specify "/dev/stdout" as the file parameter). This PR removes the file parameter from asciihex_to_binary_file (and so renames it asciihex_to_binary). In most cases, this just means removing "/dev/stdout" as a parameter to the function. In the few cases in which a parameter other than "/dev/stdout" was provided to asciihex_to_binary_file, this PR just uses a redirect (">" or ">>") to accomplish the same result as providing the output file to asciihex_to_binary_file().
Note that #1273 and #1277 raised the issue of trying to write to /tmp, and this PR does not attempt to address that.
This PR fixes two problems that occur when testing a server that supports TLSv1.3 using OpenSSL 1.1.1 in --ssl-native mode.
First, when testing whether the server has a cipher order, the value of $sclient_success is checked after each call to tls_sockets(), but $sclient_success. As the goal is just to verify that the connection was successful (and didn't downgrade), $? can be checked rather than $sclient_success. [When not in --ssl-native mode, this problem is masked since $sclient_success is set to 0 earlier in the function.]
The second problem is that line 6646 tries to copy "$TEMPDIR/$NODEIP.parse_tls13_serverhello.txt", but this file is currently only created (on line 6287) if tls_sockets() is used to determine the negotiated protocol. This PR fixes the problem by also populating "$TEMPDIR/$NODEIP.parse_tls13_serverhello.txt" when OpenSSL is used to determine the negotiated protocol.
As stated in #1435 when specifying ``-connect-timeout=20`` AND
``--parallel`` there asa problem with the file handles of child
processes (__testssl.sh: line 10454: 5: Bad file descriptor__).
This commit mitigates that in a sense that both switches can't
be used together. There's a check now in parse_cmd_line().
In addition it addresses a problem when fatal() is called and
e.g. JSON files haven't been created yet (error message ~
__testssl.sh: line 825: : No such file or directory__). It
introduces a global CMDLINE_PARSED which remembers the state
whether ``parse_cmd_line()`` has been fully executed or
not. Only when the former is the case it allows writing to files.
That implies that in main parse_cmd_line() has to be followed
by json_header() and similar.
See #1429.
OpenBSD 6.6 had an offset with the HTTP header time of -3600 seconds.
This PR fixes that by adding the GMT time zone to parse_date()'s
HAS_FREEBSDDATE incarnation. That doesn't matter to FreeBSD.
Also now for older OpenBSDs the local and remote time are now
in the same format:
```
HTTP clock skew remote: Thu, 09 Jan 2020 12:52:32 GMT
local: Thu, 09 Jan 2020 12:52:02 GMT
``
so that a time difference is easier to spot.
OpenBSD's grep seems to interpret "-w <EXPR>:" differently
than Linux or MacOSX/FreeBSD in a sense that this doesn't
matchs, see #1430.
This PR fixes that by squashing the w option in all occuorrences.
In addition it removes the SOCKETHEADER if-statement which was
introduced looking forward a while back. It's not happening soon
at least and the variable was not initialized either.
... and reorder manpages also so that --warnings, --connect-timeout
and --openssl-timeout appear in the "input parameter" section.
The HTML manpage looks in the diff view quite different as previously
another computer was used for converting the source format with ronn(1).
The manpage in (g)roff format was manually edited with .RE / .RS
for provide indented bulletpoints.
See also #1419
* It'll be a warning now when a host certificate was issued after
March 1st, 2018 which has a lifetime >825 days, independent
whether it is an internal certificate or not. This can
change later, as browsers treat those certificates different
as "official ones"
* Still the 5 or 10 year threshold overrides this
* For older OpenBSDs there's now a better date format support
used in the expiration and validity period of a host certificate.
It mimics in bash the conversion of other date binaries. It is
not accurate so it might be off a day or at maximum two, probably
as a month has 30.42 days and not 30.
* The date output for OpenBSD is now in line with other OS. Previously
we just echoed the lines in openssl x509 output whereas now we
convert that
The expression 'grep -aw "Public-Key:"' hiccuped on the colon
under OpenBSD, so that any bitsize on a certificate had wrong
values, see #1425. (FreeBSD was fine)
This PR fixes that. It updates the expression by using awk and
bash internal functions.
The same problem occured in run_robot(). The strong typing of
pubkeybits had to be relaxed to a dynamic typing, unless we
choose to define a second string variable.
* old OpenBSD kinda works
* let's encrypt section moved so that OpenBSD can use it too
* Days are wrong
* Date format is not the same as with e.g. GNUdate (but should be)
* variables y m d not declared
* date warning for openbsd completely missing
As noted in #1418 LLMNR (Link-Local Multicast Name Resolution)
resolution times out when using x as an argument to "-connect".
This commit fixes that by replacing "-connect x" by "-connect invalid."
which is supposedly also generally more "DNS query friendly", see
https://tools.ietf.org/html/rfc6761#section-6.4 .
In addition this commit adds a check in get_common_prime() whether
the openssl version used has pkey support. If not with old openssl
versions and previously testssl.sh terminated after presenting garbledoutput.
(This was found as tested how very old version of openssl versions
handle "-connect invalid.")
If the server is known not to support TLS 1.3 (as well as TLS 1.2, TLS 1.1, and TLS 1), then mention TLS 1.3 in the list of not supported protocols. While lack of TLS 1.3 support is not part of the reason that no fallback is possible, it is part of the reason that the result is reported as prln_svrty_high.
If $high_proto is set to something other than SSLv3, support for SSLv3 will not have been determined by determine_optimal_sockets_params(), but it may have been determined later (e.g., by run_protocols()). So, this commit changes the loop to always check for SSLv3 support (without calling "$OPENSSL s_client" if $HAS_SSL3 is false). The check for whether the fallback test can be performed is moved until after the loop
This PR fixes a couple of places where "$OPENSSL s_client" is called with "-ssl3" even if SSLv3 is not supported.
The fix in ciphers_by_strength() is easy, as the issue only occurs if "$using_sockets" is true. If SSLv3 (or TLSv1.3) is not supported, then testing using "$OPENSSL s_client" is skipped and all of the supported ciphers are found using tls_sockets().
The fix for run_tls_fallback_scsv() is more complicated. While it is easy to avoid calling "$OPENSSL s_client" with "-ssl3" if SSLv3 is not supported, it is not easy to determine the correct message to present to the user if support for SSLv3 (and possibly also TLSv1.3) is unknown.
For the case in which $high_proto cannot be set, I believe that I have covered all of the possibilities, but an not sure if the correct message/rating is used in every case.
For the case in which it is not possible to determine whether SSLv3 is the $low_proto, more could be done. If $high_proto is TLS 1.1 or TLS 1, then this PR is okay, as it is possible that SSLv3 would be the fallback protocol, but there is no way to tell. However, it seems unlikely that a server would support TLS 1.2 and SSLv3, but not TLS 1.1 or TLS 1. So, perhaps if $high_proto is TLS 1.2 and the server does not support TLS 1.1 or TLS 1, it should just be assumed that SSLv3 is not supported, even if it cannot be tested.
At the moment, testssl.sh does not correctly derive the handshake traffic keys in the case that the server responds to the initial ClientHello with a HelloRetryRequest. The handshake traffic keys are computed incorrectly because the wrong messages are provided to derive-handshake-traffic-keys() for input to the Transcript-Hash calculation:
* TLS_CLIENT_HELLO is not updated to contain the value of the second ClientHello, and so the first ClientHello is being provided to derive-handshake-traffic-keys() as both the first and the second ClientHello.
* In middlebox compatibility mode the server may send a dummy ChangeCipherSpec message immediately after the HelloRetryRequest. Since it is part of the server's response to the initial ClientHello, the ChangeCipherSpec message is included in the $hrr that is sent to derive-handshake-traffic-keys(), but it should not be included in the computation of the Transcript-Hash.
This PR fixes the above two problems by updating TLS_CLIENT_HELLO when a second ClientHello is sent and by removing any ChangeCipherSpec message (140303000101) from the end of the server's initial response.
* Severity of RC4 in run_cipherlists() is now high as everywhere else
* Same for RC2 and DES. Only Export, NULL and ADH remain critical
* 3DES and IDEA in run_cipherlists() is now medium (see #1393)
* CBC3 SEED 3DES IDEA added in pr_cipher_quality()
* MD5 added to pr_cipher_quality() and labled as high
* double RC2 switch statemnet removed in pr_cipher_quality()
* timeouts are not default anymore but need a cmdline param (or ENV param)
* check whether "timeout" exists
Also:
* allow OPENSSL_TIMEOUT to be passed via ENV similar to others
* replace timeout variable in run_robot() by robottimeout, to avoid
misunderstandings by a human
ToDos: see inline. Mainly if timeout isn't available, testssl.sh doesn't
work.
Not to self: help function still says 60 second is default.
When printing a long HTTP security header, this commit causes every row after the first one to be indented by two additional spaces. In the case of very long headers, this extra indentation makes it a little easier for readers to see where the next security header begins.
When printing out HTTP security headers, run_security_headers() uses out_row_aligned_max_width(), since some headers are very long and need to be wrapped. At the moment, however, the first line is too long. The problem is that while "$header $HEADERVALUE" is printed in the indented area, only $HEADERVALUE is passed to out_row_aligned_max_width().
This PR fixes the problem by passing "$header $HEADERVALUE" to out_row_aligned_max_width() so that the the first line is wrapped at the correct place.
testssl.sh hiccups when a user supplied after --json*/--html/-csv
a filename instead of using the corresponding --json*file/--htmlfile/-csvfile
arguments, see #1397.
This PR adresses that in a sense that it tries to detect to following
argument of --json*/--html/-csv. If that matches a suspected filename
it bails out using fatal().
This is not intended to be perfect (when the pattern doesn't match)
but catches the user error in an early stage. See also #1398
Currently sub_cipherlists() and pr_cipher_quality() use different numbers for the same cipher quality ratings. sub_cipherlists() uses:
-2 = pr_svrty_critical, -1= pr_svrty_high, 0 = pr_svrty_low, 1 = pr_svrty_good, 2 = pr_svrty_best
while pr_cipher_quality() uses:
1 = pr_svrty_critical, 2 = pr_svrty_high, 3 = pr_svrty_medium, 4 = pr_svrty_low
5 = neither good nor bad, 6 = pr_svrty_good, 7 = pr_svrty_best
This PR changes sub_cipherlists() (and run_cipherlists()) to use the same numbers for cipher quality as pr_cipher_quality(). It does not change any of the ratings assigned to ciphers by run_cipherlists() or pr_cipher_quality(), so the two are still not in alignment. But, hopefully using the same numbering in both functions will make it a bit easier to compare them and bring them into alignment.
This resolves a regression introduced with IDN support (see also #1370).
* in check_resolver_bins() the determination of HAS_DIG_NOIDNOUT=true was wrong
* in get_*_record() the check for the bool variable was wrong
* in get_*_record() we shouldn't use quotes as they might be expand to a quoted arg
This PR fixes some indentation issues. The PR is a bit long, but it only makes changes to indentation (except for one comment line, where a trailing space character is removed).
This PR fixes#1385.
sub_session_resumption() returns 3 when $CLIENT_AUTH is true. However, the comment at the beginning of the function indicates that 6 will be returned. run_server_defaults() is prepared to handle a return value of 6 (to indicate client auth), but is not expecting 3 as a possible return value.
In cases where the probes for reading memory from the server side were not
successful (=not vulnerable) the TCP connection was not shut down properly --
leading to and undefined state and probably causing problems to a consecutive
check. The server side then assumably from time to time just didn't return
anything which caused a integration test (t/08_isHTML_valid.t) to fail
randomly.
This PR properly terminates the TCP socket connection. Also, as sending the
close notification before closing the socket was duplicated in testssl.sh
that went to a separate function.
See comment in #1375:
https://github.com/drwetter/testssl.sh/pull/1375#issuecomment-554424814
This PR enhances support for the latest versions of OpenSSL and LibreSSL.
The development version of OpenSSL at https://github.com/openssl/openssl/ is version 3.0.0-dev. So, checks for OpenSSL versions need to support this version as well. At the same time, the latest versions of LibreSSL are 3.0.0, 3.0.1, and 3.0.2, so version number alone will no longer be sufficient to distinguish between OpenSSL and LibreSSL.
In addition to checks for these new version numbers, this PR addresses a couple of other issues:
- In LibreSSL, the "$OPENSSL ciphers" command will not accept any protocol version other than "-tls1" as a parameter (and even including "-tls1" as an option is described as "deprecated"). So, this PR ensures that "$OPENSSL ciphers" is not passed any protocol version option other than "-tls1" is LibreSSL is being used.
- In OpenSSL 3.0.0-dev, the "$OPENSSL dgst" can no longer be used to compute HMACs, but a new "$OPENSSL mac" function has been created. So, this PR changes hmac() to use "$OPENSSL mac" with OpenSSL 3.0.0-dev.
Note that I have not tested the modified version of sub_session_resumption(). I am just assuming that OpenSSL 3.0.0-dev works the same as OpenSSL 1.1.1 and that all versions of LibreSSL work the same as OpenSSL 1.1.0 and earlier.
When tls_sockets() is used for run_protocols(), for each protocol version the results will indicate whether the server responded to a ClientHello for that protocol version with (1) a ServerHello for that same protocol version, (2) a ServerHello for a different (hopefully lower) protocol version, or (3) a handshake failure.
Currently, however, run_prototest_openssl() does not distinguish between cases in which the server responds with a ServerHello for a different (hopefully lower) protocol version and cases in which the server responds with a handshake failure. This PR changes run_prototest_openssl() so that it distinguishes between these two cases (as long as $OPENSSL supports the protocol version specified in the ServerHello).
Making use of the additional information provided by run_prototest_openssl(), this PR also modifies run_protocols() to check that version negotiation was performed correctly even if $using_sockets is false.
Note that one special case needed to be addressed. If an SSLv3-only server is being tested using an $OPENSSL that does not support SSLv3, then $latest_supported must not be set to SSLv3. In the case of a server like this, it is possible that support for SSLv3 will be determined by determine_optimal_sockets_params(), which will cause run_protocols() to report that the server supports SSLv3, even though $OPENSSL does not support SSLv3 and testing is being performed in --ssl-native mode. If $latest_supported were set, then later tests in run_protocols() would incorrectly report a version negotiation failure, even though the failure to connect was a result of a limitation of $OPENSSL rather than a fault of the server.
When running in debugging mode, HAS_DIG and friends was
still false as check_resolver_bins() was called too late.
This amends basically bac0f66112 .
In cases where a certificate has two CNs, the output contained
a linefeed. This replaces the line feed by a space.
This fixes a bug e.g. when supplying a proxy by a DNS name, testssl couldn't resolve the name as the HAS_ variables initialized by ``check_resolver_bins()`` was done later than ``check_proxy()``.
The patch just puts ``check_resolver_bins()`` earlier in "main"
run_prototest_openssl() currently calls "$OPENSSL s_client" twice, once with $PROXY and once without. The problem is that the results of the first call are just ignored. This commit changes run_prototest_openssl() so that the attempt without $PROXY is only tried if the first attempt was unsuccessful.
If --ssl-native is being used and the server supports SSLv2, but does not support any SSLv2 ciphers, there is a missing line break after the warning message is printed.
run_prototest_openssl() currently checks only stdout for the string "no cipher list", which is an indication that the server supports SSLv2, but no ciphers for that protocol. However, the output that includes "no cipher list" is sent to stderr.
This PR fixes a minor problem with run_protocols() in "--ssl-native" mode if $OPENSSL does not support TLS 1.3. Currently, the warning message that $OPENSSL does not support a protocol is printed when run_prototest_openssl() is called. This causes a problem for the output if $OPENSSL does not support TLS 1.3, since the run_prototest_openssl() is called before the results for TLS 1.2 are printed. The result is something like this:
SSLv2 not offered (OK)
SSLv3 not offered (OK)
TLS 1 offered (deprecated)
TLS 1.1 offered (deprecated)
Local problem: /home/cooper/Desktop/testssl.sh/bin/openssl.Linux.x86_64 doesn't support "s_client -tls1_3"
TLS 1.2 offered (OK)
TLS 1.3 NPN/SPDY not offered
ALPN/HTTP2 http/1.1 (offered)
When processing a command line for parallel mass testing, create_mass_testing_cmdline() did not account for the newer shortened versions of the output file options: -oj, -oJ, -oC, and -oH. This caused the command line for the child processes to be incorrect.
If a server offers TLS 1.3 only and the cipher order is server side this commit changes the severity level to INFO.
Also it changes nope to no in two places
This commit addresses the comments in #1205. If a server only supports TLS 1.3, then it is not considered an issue if the server does not enforce a cipher order. However, if the server does not support a cipher order for TLS 1.2 and below, then that is an issue, even if the server does support a cipher order for TLS 1.3.
This PR is an attempt to fix#1163 by running separate tests for a server cipher order preference to TLSv1.3 and for SSLv3 - TLSv1.2.
If the server supports TLSv1.3, then a test is performed to determine whether the server enforces a cipher order to TLSv1.3. A separate test is performed for SSLv3 - TLSv1.2 unless it is known that the server does not support any of these protocols.
If the server enforces a cipher order for SSLv3 - TLSv1.2, but not for TLSv1.3, then cipher_pref_check() is not called for TLSv1.3, since cipher_pref_check() is intended to show the cipher order that the server enforces. As TLSv1.3 will be the negotiated protocol if it is supported, the negotiated cipher for TLSv1.3 will already be presented.
This PR still has one major flaw, which may create a problem when testing a TLSv1.3-only server. If run_protocols() is run before run_server_preference(), then everything will be okay, as run_server_preference() will be able to determine that SSLv3 - TLSv1.2 are not supported. However, if run_server_preference() is run by itself, run_server_preference() will not know that SSLv3 - TLSv1.2 are not supported and so it will try to determine whether the server enforces a cipher preference order for these protocols. The attempt to connect to the server will fail, but at the moment run_server_preference() doesn't know whether the failure is because the server does not support SSLv3 - TLSv1.2 or because the server supports at least one of these protocols, but does not support any ciphers in $list_fwd. At the moment, run_server_preference() incorrectly flags an error.
One option would be to perform additional tests against the server in this case to determine the reason for the connection failure. Another option would be to have some code that is always run earlier, such as determine_optimal_proto(), test whether a server that supports TLSv1.3 supports any earlier protocols (SSLv3 - TLSv1.2).
In run_drown(), $jsonID is set to "DROWN" and most calls to fileout() are of the form
fileout "$jsonID" ...
However, one call is written as
fileout "DROWN" ...
This PR changes this one call to be consistent with the others. This does not change the functionality of the program.
At the moment, $do_starttls is initialized to true in initialize_globals() and then it is set to true again in parse_cmd_line() if the --starttls command line option is used. Presumably the intention was to set $do_starttls to false in initialize_globals().
This PR fixes a minor bug in parse_tls_serverhello(). In some cases the server's entire response is not retrieved. In these cases, it is possible that the response from the server ends with a portion of a handshake message.
The loop at the beginning of parse_tls_serverhello() extracts the various handshake and alert messages from the server's response. If it gets to the end of the response, and what is at the end is not a complete message, it should just ignore that fragment and break out of the loop. At the moment, however, parse_tls_serverhello() just continues in the loop rather than breaking out. This has not been a problem up to now, since $msg_len is usually set to a positive value from a previous iteration of the loop, which causes the loop to end.
In the case of the server identified in #1353, however, $msg_len is 0 and so the continue rather than break results in an endless loop.
Add another pattern because the SEDs tested so far do not seem to be fine with header containing x0d x0a (CRLF) -- which is the usual case. So we also trigger on any sign on a single line which is not alphanumeric (plus _)
See #1351
This PR fixes a bug in modify_clienthello() that occurs when client simulation is being performed, the ClientHello contain an SNI extension, and $SNI is empty. In the case, modify_clienthello() should just skip over the SNI extension and not include one in the modified ClientHello. However, the code currently only skips over the 2-byte extension type. The result being that the remainder of the extension is included in the modified ClientHello. This PR fixes the problem by ensuring the $offset is advanced whether or not $SNI is empty.
PR #1336 included logic to pre-test the server side with sockets
and/or with openssl. However when the user supplied --ssl-native
sockets were never tested before. As a result ALL_FAILED_SOCKETS
was still true, so that the final eif statement complaint erroneously
that sockets didn't work but openssl does.
Also Travis complaint.
This PR fixes it by checking SSL_NATIVE to the final part of the
if statement.
One could also test sockets before and then set ALL_FAILED_SOCKETS
appropriately but that would only make sense if the socket methods
like run_robot() or run_heartbleed() would check ALL_FAILED_SOCKETS
first.
At the moment I went for this as it is easier and the case that sockets
aren't working but openssl does seems not very likely.
There are a few places where testssl.sh sends a TLS 1.2 (or TLS 1.3) ClientHello and expects the server to respond with a ServerHello as long as it supports TLS 1.2 (or TLS 1.3) or earlier.
run_protocols() performs a fairly thorough check for a server's ability to handle version negotiation, but the problem may also be caught by determine_optimal_sockets_params(), if the server rejects a TLS 1.2 ClientHello even though it supports some earlier protocol version.
In the future, we could try to make use of $OPTIMAL_SOCKETS_PROTO in order to make testssl.sh work a bit better with servers (if any still exist) that don't handle version negotiation correctly. At the moment, though, this PR just prints a warning to the user that the server is buggy, and that this may lead to problems in the scan. It doesn't call fileout() to add anything to the JSON/CSV output, since run_protocols() should already be doing that.
This PR modifies run_protocols() to use the information collected by determine_optimal_sockets_params(). If it has already been determined that a protocol is supported, then no test is run. run_protocols() will still run a test for a protocol even if it has been determined that the server does not support that protocol. The reason for running the test is to verify that the server handles version negotiation correctly. This could be a TLSv1 server that rejects a TLSv1.2 or TLSv1.3 ClientHello, or it could happen in the opposite direction. At one point there was a server that would respond to an SSLv3 ClientHello with a TLSv1.2 ServerHello.
This PR required a couple of changes to determine_optimal_sockets_params() so that additional information could be passed to run_protocols(). If the server supports TLS 1.3, then run_protocols() needs to know which version (RFC 8446, draft 28, draft 27, etc.) rather than just that TLS 1.3 is supported. If the server supports TLS 1.2, but not TLS 1.3, then run_protocols() needs to know about at least one TLS 1.2 cipher that the server supports so that it can form a TLS 1.3 ClientHello that has no more than 128 ciphers and that should result in the server returning a TLS 1.2 ServerHello.
In a PR that I'm developing to to use the results of determine_optimal_sockets_params() in run_protocols() I add specific versions of TLS 1.3 to PROTOS_OFFERED (e.g., tls1_3_rfc8446, tsl1_3_draft28). If that PR is accepted, then the current check for TLS 1.3-only will no longer work. So, this commit changes the way that the check for TLS 1.3-only is performed in order to avoid problems if the other PR is merged.
This PR takes advantage of the testing done by determine_optimal_sockets_params() in order to simplify determine_sizelimitbug().
By the time that determine_sizelimitbug() is called, determine_optimal_sockets_params() has already determined whether TLSv1.2 ClientHello with 128 ciphers (including 00FF) sent by tls_sockets() works, and it has set TLS12_CIPHER to a list of exactly 128 ciphers (including 00FF) that works with the server. So, determine_sizelimitbug() doesn't have to check whether the server supports TLSv1.2 and no longer needs to send tests using 127 or 128 ciphers. determine_sizelimitbug() can just perform one test with 129 ciphers, if the server supports TLSv1.2, and use the results to set $SERVER_SIZE_LIMIT_BUG.
This PR reverts determine_optimal_proto() to use OpenSSL again rather than tls_sockets().
The primary reason for this is that the primary purpose of determine_optimal_proto() is to set OPTIMAL_PROTO, which is only used with $OPENSSL s_client. So, the best way to determine what works best on the $OPENSSL s_client command line is to use $OPENSSL s_client.
In most cases, determine_optimal_proto_sockets_helper() would set OPTIMAL_PROTO to an acceptable value, but it might not always do so. For example, suppose that a server
* supports different cipher suites with different protocols,
* supports TLSv1.2, but only with cipher suites not supported by $OPENSSL, but
* supports TLSv1.1 with at least one cipher suite supported by $OPENSSL.
In the above case, determine_optimal_proto_sockets_helper() would set OPTIMAL_PROTO to "-tls1_2", but testing using $OPENSSL would result in OPTIMAL_PROTO being set to "-tls1_1".
Using $OPENSSL for determine_optimal_proto() also allows for edge cases to be detected earlier:
* If the server only supports TLSv1.3, and $OPENSSL does not support TLSv1.3, then the code in this PR will detect that (rather than waiting until run_protocols() is executed).
* The code in this PR can also detect if the server only supports SSLv3 (and possibly also SSLv2), but $OPENSSL does not support SSLv3.
* This code can also detect the (rare) case in which connections using $OPENSSL succeed, but connections using tls_sockets() fail.
[Note also that in the current code, if $all_failed is true, then a message may be printed that $OPENSSL is not IPv6 aware, even if testing was performed using tls_sockets() rather than $OPENSSL.]
This PR fixes an issue with sub_session_resumption() when using OpenSSL 1.1.1.
As noted in #1335, some servers will return a session ticket for TLSv1.2, but not for TLSv1.3.
OpenSSL 1.1.1 does not support the "-no_ssl2" option, and so when using OpenSSL 1.1.1 sub_session_resumption() adds $OPTIMAL_PROTO to the $OPENSSL s_client command line. When determine_optimal_proto_sockets_helper() is called, $OPTIMAL_PROTO will generally be set to "-tls1_2" (or "-tls1_1" or "-tls1") unless the server is a TLSv1.3-only server. As a result sub_session_resumption() will specify that same protocol on the command line if OpenSSL 1.1.1 is being used.
If "--ssl-native" is used, however, then determine_optimal_proto() will set $OPTIMAL_PROTO to "-tls1_3" if the server supports TLSv1.3 (and doesn't use STARTTLS). Similarly, if the version of determine_optimal_proto() in #1336 is used, then $OPTIMAL_PROTO will usually be empty. In either case, sub_session_resumption() will send a TLSv1.3 ClientHello, even if the server only supports session tickets for TLSv1.2 and below.
This PR appears to fix the problem. This PR makes no changes when using a version of OpenSSL that supports "-no_ssl2". When using a version of OpenSSL that does not support "-no_ssl2", however, rather than using $OPTIMAL_PROTO, this PR has sub_session_resumption() use whatever protocol version the server connected with when $sessticket_lifetime_hint was set.
There is a problem if a TLSv1.3-only server is tested using the OpenSSL 1.0.2-chacha binary and $OSSL_SHORTCUT is true.
$HAS_NO_SSL2 is set to true when find_openssl_binary() is called with OpenSSL 1.0.2-chacha. /usr/bin/openssl does not have the -no_ssl2 option, but the second call to find_openssl_binary(), after setting $OPENSSL to /usr/bin/openssl, does not set $HAS_NO_SSL2 to false. So, later calls to $OPENSSL s_client include the -no_ssl2 option, resulting in connection failures.
This PR fixes the problem by modifying find_openssl_binary() to ensure that every OpenSSL-dependent variable is set by this function.
This PR fixes two issues with finding session tickets when using OpenSSL 1.1.1.
First, if OpenSSL connects to the server using TLSv1.3 and it receives more than one Post-Handshake New Session Ticket, then the "TLS session ticket lifetime hint" will appear more than once in $TMPFILE. This will cause the line to appear more than once in $sessticket_lifetime_hint, which causes problems when trying to extract the $lifetime and $unit from $sessticket_lifetime_hint.
This PR fixes the first problem by changing the awk expression in the lines that set sessticket_lifetime_hint so that only the first line with "session ticket lifetime" is extracted.
The second issue is that some servers (e.g., google.com) return a session ticket for TLSv1.2, but not for TLSv1.3. For such servers, testssl.sh will miss the session ticket if $OPTIMAL_PROTO is empty or "-tls1_3" and the --ssl-native flag is not set.
This PR addresses the second issue with the changes in lines 9047 - 9053 -- the code that is intended to provide a last chance to find a session ticket.
If $OPENSSL supports TLSv.1.3 and the server returns session tickets for TLSv1.3 connections, then the session ticket would have already been found by get_server_certificate(), since get_server_certificate() uses $OPENSSL for TLSv1.3 if $OPENSSL supports TLSv1.3. So, in such circumstances, the code in liens 9047 - 9053 should not try again with TLSv1.3. So, if $OPENSSL supports TLSv1.3 and $OPTIMAL_PROTO is empty or is set to "-tls1_3" (either of which would result in a TLSv1.3 ClientHello), the "$OPENSSL s_client" call is changed to specify -no_tls1_3 rather than $OPTIMAL_PROTO.
The code on line 9047 is also changed to only make this final try is $TLS13_ONLY is false. If $TLS13_ONLY is true, then either:
* $OPENSSL does not support TLSv1.3 and the connection attempt would fail anyway; or
* $OPENSSL supports TLSv1.3, in which case any session ticket would have been found by get_server_certificate(), since get_server_certificate() uses $OPENSSL for TLSv1.3 if $OPENSSL supports TLSv1.3.
In either case, there is no reason to try again to find a session ticket.
This commit adds some more checks to determine_optimal_sockets_params(). These additional checks will almost never need to be run, and so will not slow down the typical run of testssl.sh, but adding them will provide information that can be useful for other parts of testssl.sh.
These additional checks will only be run if the server does not support TLSv1.3 and a TLSv1.2 ClientHello is not successful. This means that either:
* The server is not an TLS/SSL enabled server.
* The server only supports SSLv2.
* The server supports some protocol in SSLv3 - TLSv1.1, but does not handle version negotiation correctly.
Adding these additional checks helps in at least the following ways.
If determine_optimal_proto() (assuming it is reverted to using OpenSSL) is unable to connect to the server using OpenSSL, it will be possible determine whether the problem is the the server does not support TLS/SSL or that a different version of OpenSSL is needed to test the server.
If the code in #1205 for run_server_preference() is unable to connect when checking for a cipher order, the reason for the failure will be known, making it possible to determine the correct response to the failure.
This PR adds a new helper function that is run just prior to determine_optimal_proto() and that determines the what information tls_sockets() should include in a ClientHello.
For a TLSv1.3 ClientHello, determine_optimal_sockets_params() determines whether tls_sockets() should use 0x33 or 0x28 are the extension number for the key_share extension. 0x33 should be used with servers that support RFC 8446 or drafts 23-28. 0x28 should be used with servers that support drafts 18-22.
For a TLSv1.2 ClientHello, determine_optimal_sockets_params() determines what cipher list tls_sockets() should send. For most servers, the list of ciphers in $TLS12_CIPHER works best. But, there are some servers that do not support any ciphers in $TLS12_CIPHER, but do support one or more ciphers in $TLS12_CIPHER_2ND_TRY.
This PR fixes a few issues with run_protocol():
* In the case that the call to `tls_sockets "03" "$TLS12_CIPHER"` had a return value of 2, the code determining what results to print was looking at `$DETECTED_TLS_VERSION`. However, the value of this variable was set by the later call to `tls_sockets "04" "$tls13_ciphers_to_test"`. This caused incorrect results in the case of a server that supports TLSv1.3 and TLS1.1 (or earlier), but not TLSv1.2. This PR saves the value of `$DETECTED_TLS_VERSION` in `$tls12_detected_version` and then uses this variable later rather than `$DETECTED_TLS_VERSION`.
* When running in debug mode with a server that does not support TLSv1.3, testssl.sh was printing
TLS 1.3 -- downgradednot offered and downgraded to a weaker protocol"
This PR fixes the output by not printing the "--downgraded"
* As noted in #1329, run_protocols() was treating a downgrade from TLSv1.2 as less bad if the server supports TLSv1.3. This PR changes this code back to treat any downgrade from TLSv1.2 as equally bad.
* In order to be consistent with the TLSv1.3 test, this PR changes the TLS1.2 test output to say "not offered and downgraded to a weaker protocol" if a TLSv1.2 ClientHello results in a downgraded connection.
This PR fixes a problem in run_protocols() that was introduced by 7ec3c6ab99.
7ec3c6ab99 changes run_protocols() to perform the initial testing for TLSv1.3 support before testing for TLSv1.2 support. The problem with this is that the code for testing TLSv1.3 makes use of the results of the TLSv1.2 testing.
In the current code, Line 5183 looks at the value of $subret to determine whether the TLSv1.2 ClientHello resulted in a successful connection. However, $subet has not yet been set (it has just been initialized to 0 at the beginning of the function). Since $subret will always be 0, the code will try to extract a cipher from $TEMPDIR/$NODEIP.parse_tls_serverhello.txt. This may work, since $TEMPDIR/$NODEIP.parse_tls_serverhello.txt may have been populated by a prior function call, but this is not how the code was intended to work.
This PR fixes the problem by doing the TLSv1.2 testing before the TLSv1.3 testing is done. It still waits until both have been tested, however, before outputting the results, so that the output for TLSv1.2 can be modified depending on whether TLSv1.3 is supported.
In order to not repeatedly call check_resolver_bins() the function
was moved to top level. As each check in check_resolver_bins now
is only executed once, it should also work faster. Each get_*_record()
now uses HAS_ variables only.
Also check_resolver_bins() contain now the check whether
idn/idn2 support is available.
Then the IDN URI conversion snipplet was moved to the final function
parse_hn_port() which does operations in the URI supplied.
This PR adds a few quotes to some arguments which when previous code
was executed properly weren't needed.
Also it improves the IDN code from @teward, so that when idn2 is
available, a conversion will be tried, and when idn is available
and/or idn2 failed, a conversion will be tried.
Finally it'll be tried to continue without conversion, hoping that
the DNS client binaries can cope with the IDN URI.
This is not good enough yet and needs to be complemented, see discussion
@ #1321.
The Finding of other_headers such as "Referrer-Policy" during file output are displayed as `$header: $HEADERVALUE` instead of only `$HEADERVALUE` as the good_headers. This leads to duplicate information e.g. in the JSON output file.
... and thus this commit addresses #916.
It does that via a (quite) pre-test which checks for a general availabilty
of TLS 1.3 before the TLS 1.2 protocol test is being run and decides
based on that how a missing TLS 1.2 will be echoed.
Later on the complete TLS 1.3 test will be continued using the results
from the TLS 1.3 pre-test.
This commit addresses #1251 and gives a slight warning when still
using those protocols as government standards are or are at least
to expect also to deprecate those protocols. PCI DSS requires not
to use TLS 1.0 anymore and browser vendors supposedly will deprecate
TLS 1.0/1.1 next year.
This is a WIP for testing. It was committed already in May
(22ad490ea7b2868a4fd45862ca0bf8a3d8f24ea6) but somehow it was
lost.
Comments would be appeciated.
Open:
- how to treat non-HTTP protocols
- TLS 1.3 only hosts will mark the absence of TLS 1.2 as
a medium finding
In cases where TLS 1.3 is the only protocol supported by the server (as e.g.
in #1312), testssl.sh has some limits with the supplied binary.
For now (3.0) there's no perfect technical solution. This PR however improves
the verbosity what's going on and recommends to use an openssl binary
supporting TLS 1.3. And if the "secret" variable OSSL_SHORTCUT is set to true,
it automatically chooses that if available (it's a hack to do so and not
recommended. I just did it as a PoC).
In the next development we should consider probing this upfront!
Furthermore this PR removes some unnecessary quotes in double square brackets.
This commit proactively tries to address cases where the server side
adds Null Bytes after or during ClientHello in cases where it should be text only.
Now VAR2=$(< $VAR1) is being replaced by VAR2=$(cat -v $VAR1) which is normally
not best practice and also considered a useless use of "cat", see
https://web.archive.org/web/20160711205930/http://porkmail.org/era/unix/award.html#uucaletter.
Especially with bash 3.2 (Mac OS X) AND when on the server side binary chars it
was reported to not work ok, see #1292.
Performance measurements showed no to barely measureable penalty (at max 1s displayed difference
in 9 tries).
Travis updated the container images so that the perl
reference to 5.18 was outdated. We use now 5.26 which
works, however we should consider to be more flexible.
JSON::Validator didn't compile in the container. Thus
we switched to just use 'JSON'. That also supports JSON
pretty. For the future we should just test for valid JSON
in all unit test files as it is more effective.
For some reason CVE-2009-3555 ended up in Secure Renegotiation,
whereas CVE-2009-3555 is in fact the Insecure Client-Side Renegotiation
vulnerability with the MiTM problem <= OpenSSl 0.9.8k.
This fixes that (see also #1086 and #933, #907) by removing the CVE #
from the output. Also tyhe output was changed for Secure Renegotiation
into supported/not vulnerable vs. Not supported / VULNERABLE
Some comments were added.
As noted in #1249 STARTTLS with sockets doesn't
work.
This commit fixes that by correcting the STARTTLS
handshake for postgresql. It has to be send via
sockets instead of echo. The server side then will
respond with "S" when STARTTLS is supported. For this
starttls_io() was slightly modified so that also
an input (from the server perspective) is not necessary.
It's fast too
As noted in #1288 with some terminal settings under Linux there
appeared some ~garbage on the screen.
This fixes that by partly reverting 695d02157a .
At least now and under an older OpenBSD like 6.2 this doesn't seem
to be necessary.
CERT_COMPRESSION was declared always with fast in ... so that the variable
was always false. This PR fixes that. In addition a informational line
that the new TLS extension has been added (if $DEBUG >3).
Also determine_optimal_proto() is not being run if devel mode
($do_tls_sockets) is enabled.
Furthermore as David added HAS_ZLIB as a global run_crime() now makes
use of it too.
This PR addresses #316 and #1280: it implements server name indication
also for STARTTLS which has been supported by a number of server
implemantations, in the meantime.
Also it does a final polish to David's pwnedkeys PR #1274 a while back:
UI improvement and detection of network problems.
In addition to PR #1279 it introduces a env variable to devel
mode so that "CERT_COMPRESSION=true ./testssl.sh --devel <params> <target>"
can be used to explore certificate compression on a host.
draft-ietf-tls-certificate-compression specifies a new TLS extension that allows a client to indicate support for receiving the server's certificates in compressed form. This PR adds initial support for that extension to testssl.sh. It was developed based on an announcement that facebook.com has implemented support for the extension.
This PR does not add a test for a server's support for the new extension, it just adds code to parse_tls_serverhello() that will parse a compressed certificate message if one is present in the server's response. So, in practice, the code in this PR will not actually be used until additional code has been added that sends a ClientHello with the compress_certificate extension.
The code in this PR can be tested by using the --devel option and by changing line 19347 from
tls_sockets "$TLS_LOW_BYTE" "$HEX_CIPHER" "ephemeralkey"
to
tls_sockets "$TLS_LOW_BYTE" "$HEX_CIPHER" "all+" "00,1b, 00,03, 02, 00,01"
testssl.sh can then be called using
testssl.sh --debug 3 --devel 04 "13,01" facebook.com
Note that this PR adds a test for zlib support to find_openssl_binary(). The test uses $OPENSSL zlib in the same way it is used in parse_tls_serverhello(). The test asks $OPENSSL zlib to uncompress the compressed version of the string "zlib" and the checks to see whether the decompression was performed correctly. There is already a test for zlib support in run_crime(), but I did not check to see whether the check in run_crime() could be replaced with a check of the new $HAS_ZLIB variable.
A couple of checks required sockets but e.g. LDAP via STARTTLS
throwed an error (FIXME: LDAP+STARTTLS over sockets not supported yet)
in fd_sockets().
This adds a temporary workaround so that those functions are bypassed
and LDAP via STARTTLS can be used again.
See also #1258
... which led to a false output in OpenSSL based handshake simulations.
secp256r1 is prime256v1
secp192r1 is prime192v1
Also a few varaiables were added in debug output (environment.txt)
This PR changes run_protocols() so that, when using tls_sockets(), support for TLSv1.3 is only marked as pr_svrty_best() if the final (RFC 8446) version is supported. It also changed run_protocols() so that support for TLSv1.3 is marked as pr_svrty_best() if OpenSSL is used (i.e., if the --ssl-native option is specified).
One potential issue is that the --ssl-native version assumes that if OpenSSL supports TLSv1.3 it supports the final (RFC 8446) version of the protocol. If the tester is using a development version of OpenSSL 1.1.1 rather than the final version, then the protocol test will actually be indicating whether the server supports the same draft version of OpenSSL as the $OPENSSL being used to perform the tests.
Currently the -outfile, -oa, -outFile, and -oA assume that <fname> being provided is to be used as a filename, unless it is "auto." However, all of the individual options (e.g., --logfile) allow for a directory name to be provided instead of a file name.
This PR changes the handling of the -outfile, -oa, -outFile, and -oA options so that if a directory name is provided, the files are created in that directory.
This PR fixes#1243 by modifying create_mass_testing_cmdline() to handle the --outfile, -oa, --outFile, and -oA options in the case that the filename provided is not "auto." It also modifies create_mass_testing_cmdline() so that in the case of serial testing -oj and -oJ are treated the same as --jsonfile and --jsonfile-pretty, respectively.
As a result of #1225 every Linux binary needed was not allowed
to come from busybox. Which caused the Dockerfile in this repo
and the image @ dockerhub to fail.
This PR relaxes that so that busybox binaries which proved to
work can be used. A whitelist was defined.
This PR addresses the following issues raised by shellcheck:
In ../github/testssl_2.9dev_20190409b.sh line 1133:
if [[ "$BASH_VERSINFO" == 3 ]]; then
^-- SC2128: Expanding an array without an index only gives the first element.
In ../github/testssl_2.9dev_20190409b.sh line 4301:
tmpfile_handle $FUNCNAME.dd
^-- SC2128: Expanding an array without an index only gives the first element.
In ../github/testssl_2.9dev_20190409b.sh line 4388:
tmpfile_handle $FUNCNAME.dd
^-- SC2128: Expanding an array without an index only gives the first element.
In ../github/testssl_2.9dev_20190409b.sh line 4657:
tmpfile_handle $FUNCNAME$1.txt
^-- SC2128: Expanding an array without an index only gives the first element.
In ../github/testssl_2.9dev_20190409b.sh line 5327:
tmpfile_handle $FUNCNAME.${debugname}.txt
^-- SC2128: Expanding an array without an index only gives the first element.
In ../github/testssl_2.9dev_20190409b.sh line 5943:
tmpfile_handle $FUNCNAME.byID.log $tmpfile || \
^-- SC2128: Expanding an array without an index only gives the first element.
In ../github/testssl_2.9dev_20190409b.sh line 5944:
tmpfile_handle $FUNCNAME.byticket.log $tmpfile
^-- SC2128: Expanding an array without an index only gives the first element.
In ../github/testssl_2.9dev_20190409b.sh line 12410:
tmpfile_handle $FUNCNAME.dd $SOCK_REPLY_FILE
^-- SC2128: Expanding an array without an index only gives the first element.
In ../github/testssl_2.9dev_20190409b.sh line 13164:
tmpfile_handle $FUNCNAME.dd
^-- SC2128: Expanding an array without an index only gives the first element.
In ../github/testssl_2.9dev_20190409b.sh line 13284:
tmpfile_handle $FUNCNAME.dd $SOCK_REPLY_FILE
^-- SC2128: Expanding an array without an index only gives the first element.
In ../github/testssl_2.9dev_20190409b.sh line 13388:
tmpfile_handle $FUNCNAME.dd $SOCK_REPLY_FILE
^-- SC2128: Expanding an array without an index only gives the first element.
--
In ../github/testssl_2.9dev_20190409b.sh line 13801:
[[ "$DEBUG" -ge 1 ]] && echo $tls_hello_ascii >$TEMPDIR/$FUNCNAME.tls_hello_ascii${i}.txt
^-- SC2128: Expanding an array without an index only gives the first element.
When users try to reinvent the wheel and write an own dockerfile
this PR checks when binaries come from busybox -- as it is the
case with Alpine Linux.
This PR fixes#803 and emit an extra warning if the certificate
has a lifetime longer or equal of five years which happens often
on appliances with self signed certificates. (CAs do not offer
such a long certificate lifetime.) This was tested under Linux,
FreeBSD and OpenBSD. On the latter however we only check the
years as opposed to other OS where we have a finer granularity
(seconds).
On the screen there's only an output if the lifetime is too long,
using JSON or CSV formats, it is always displayed (ID: cert_validityPeriod).
Also this PR changes the ID cert_expiration_status to cert_expirationStatus.
Older FreeBSD and OpenBSD can't deal with italics characters but it output
the escape codes which could result in a different markup. This PR detects
such OS and just doesn't dsiplay the escape sequence.
Also the manpage is reflecting the change and has updates in the server
defaults and standard cipher checks section.
This PR fixes#1223 by checking whether the stapled OCSP response from the server is an error message.
Another way to fix#1223 would be to just change line 8510 to:
```
if grep -a "OCSP Response Status" <<< "$ocsp_response_status" | grep -q successful || \
[[ "$ocsp_response" =~ Responder\ Error: ]]; then
```
However, I believe this alternative would lead to confusing results, testssl.sh would print
offered, error querying OCSP responder (tryLater)
I'm not sure whether it makes sense to say "offered" when the stapled response that is provided is just an error message, but I think it is important to make clear that the error response was received from the TLS server, and that it wasn't testssl.sh that tried querying the OCSP responder.
A private ~/.digrc overrides the commandline options from dig. So
we need to make sure that the output is still what is expected.
This commit addresses it by adding additional parameters, mostly
to existing awk commands so that only the fields we want are returned.
see #1220
- write to log file if there's a SERVER_SIZE_LIMIT_BUG
- write to screen if $DEBUG > 1
It's 128 + 00ff when the CISCO ACE hiccups (#1204)
Some minor improvements like removing redundant double quotes
As in #1219 reported it was possible to specify e.g.
--csv and --csvfile which was not intended.
This PR detects those conflicting options and
exists.
Also it removes 637812a022
"&& JSONHEADER=false" as it seems errorneous.
This commit fixes a the regression "Session Ticket RFC 5077 hint missing/incomplete" #1218.
Reason was that in some case where the ticket lifetime hint was not restrieved before, later
$OPENSSL s_client -connect with -cipher ALL:COMPLEMENTOFALL didn't get the ticket either.
Just using "$OPTIMAL_PROTO" instead of -cipher ALL:COMPLEMENTOFALL fixed it in the cases
tested so far.
Then a global variable is instroduced -- TLS_TICKETS. Which keeps in any case the
state whether session tickets are supported. This is being used to fix#1089. It
remains a bit unclear what is meant in https://tools.ietf.org/html/rfc5077#section-5.6
by "TLS clients MAY be given a hint of the lifetime of the ticket". We use this information
to chck for resumption by ticket which seems realistically the best solution.
Sessin resumption was also made a bit more reliably: The ServerHello is now
being tested for "New" also. If this and "Reused" wasn't detected, an error
is raised.
In general we could do better in keeping and reusing information of a ServerHello
in TMPDIR.
There is currently a bug in determine_optimal_proto_sockets_helper(). In two places there is code of the form:
tls_sockets ...
if [[ $? -eq 0 ]]; then
...
elif [[ $? -eq 2 ]]; then
...
fi
This code does not work as intended since the second check ("elif [[ $? -eq 2 ]]") is actually comparing the results of the first check to 2 rather than the results of the call to tls_sockets().
This PR fixes that problem and also speeds up the code. Since tls_sockets() sets $DETECTED_TLS_VERSION to the protocol version that was negotiated, there is no need to scan $TEMPDIR/$NODEIP.parse_tls_serverhello.txt for this information.
* changed = to ==
* fixed emphasize errors in emphasize_stuff_in_headers()
* add new debian version
* prospectively add Alt-Svc header, see #1209 (won't show up in output yet)
In order to handle better Cisco ACE loadbalancers (almost extinct species) which
have a problem with ClientHellos >127 ciphers we have had introduced a variable which
needs to be filled better with some sense.
This commit does that by introducing the function determine_sizelimitbug() which
is called in lets_roll().
It also removes then redundant code in cipher_pref_check().
Open:
* handle run_grease()
* do we want this information at least in a logfile
* or maybe eben on screen?
See also #1202 .
For non-EC public key algorithm of the server certificate
the terminal output contained the algorithm but not the
file out put. This PR fixes that, see also #1187 and puts
the public key algorithm first in the cert_keySize -- also
for EC cerificates.
In addition it fixes the recognition of ECDSA certificates
which were detected as DSA certificates (order in case
statement).
Also there were in a few double sqaure brackets an assignment operator
'=' instead of a test operator '=='
There were a few, mostly less common ciphers in this check missing.
This commit adds them and fixes#208.
It also removes redundant quotes in double square brackets and
updates documentation for determine_optimal_proto().
This is the last fix for #1087.
It determines STARTTLS_OPTIMAL_PROTO (unless --ssl-native is being used) with
sockets per default which removes cases where an openssl s_client
connect using STARTTLS failed with the initial message 'doesn't seem to be a TLS/SSL
enabled server' and prompt 'Really proceed ? ("yes" to continue)' now shouldn't
happen in those cases anymore.
To not have redundant code determine_optimal_proto_sockets_helper() is being
used for not STARTTLS and plain TLS/SSL.
In addition it looked like this determine_optimal_proto() was not always called in
the beginning when a STARTTLS scan was requested. Instead determine_service()
contained an openssl s_client connect called which was the wrong place and thus
removed. Also now determine_optimal_proto() also for STARTTLS will always being called.
The information on 64 Block ciphers using SSLv2 in the SWEET32
paper is sparse. Maybe becuase SSLv2 is the bigger problem.
For completeness also SSLv2 ciphers were added to the SWEET32 check.
It fixes finally #613.
Also with --ssl-native when no ciphers or only a handlful of ciphers
are supplied by openssl there's a "Local problem" warning issued and
the test aborted as the results would make much sense otherwise.
A few peices of documenation of parameters and return conditions
for sslv2_sockets() and has_server_protocol() was added.
This is a minor bug when performing run_server_preference() if the server cannot handle ClientHello messages with more than 128 ciphers (i.e., $SERVER_SIZE_LIMIT_BUG is true) and the server supports at least one cipher in 'CAMELLIA:IDEA:KRB5:PSK:SRP:aNULL:eNULL'.
The problem is that `$OPENSSL s_client` is called with a cipher list such as
ECDHE-RSA-AES256-GCM-SHA384:CAMELLIA256-SHA:AES256-SHA256
then
ECDHE-RSA-AES256-GCM-SHA384:CAMELLIA256-SHA:AES256-SHA256:-CAMELLIA256-SHA
then
ECDHE-RSA-AES256-GCM-SHA384:CAMELLIA256-SHA:AES256-SHA256:-CAMELLIA256-SHA:-AES256-SHA256
and finally
ECDHE-RSA-AES256-GCM-SHA384:CAMELLIA256-SHA:AES256-SHA256:-CAMELLIA256-SHA:-AES256-SHA256:-ECDHE-RSA-AES256-GCM-SHA384
The last call to $OPENSSL s_client produces an error since the list of ciphers to send is empty, and this results in connectivity_problem() being called to print a "openssl s_client connect problem" warning.
This PR fixes the problem by constructing a list of ciphers to test for and by not calling $OPENSSL s_client if the list is empty.
As Dirk commented in #1199, TLS_GOSTR341094_RSA_WITH_28147_CNT_MD5 uses an RSA certificate, not a GOST certificate. So, this PR moves that cipher suite (0xff, 0x00) from the GOST list to the RSA list.
According to etc/cipher-mapping.txt, TLS_GOSTR341094_RSA_WITH_28147_CNT_MD5 uses RSA for both authentication and key exchange, so this PR places it on the list of cipher suites that uses RSA for encryption rather than signatures.
With OpenSSL 1.1.0 (and maybe other versions), the `ciphers` function lists many cipher suites that are not actually supported by the `s_client` option. This PR fixes that by using the `-s` option whenever `$OPENSSL ciphers` is used to obtain a list of cipher suites supported by OpenSSL. According to https://www.openssl.org/docs/manmaster/man1/ciphers.html:
```
-s
Only list supported ciphers: those consistent with the security level, and minimum and
maximum protocol version. This is closer to the actual cipher list an application will
support.
```
When the `-s` option is used along with `-tls1`, OpenSSL 1.1.0 will not list any ciphers that only work with TLSv1.2. So, `prepare_debug()` needed to be changed to correctly populate `ossl_supported_tls`, which is supposed to be a list of all non-SSLv2 ciphers supported by the server.
LibreSSL issues an "unknown option" error if the `-s` option is provided, so the `-s` option is only included in the command line if `$OPENSSL` has been determined to support it.
This PR is needed so that `prepare_debug()` can correctly determine which cipher suites are or are not supported by `$OPENSSL`.
This PR removes an extra call to `$OPENSSL s_client` in `get_server_certificates()` and it also changes `get_server_certificates()` to not collect extensions when SNI is not being provided.
This PR modifies get_server_certificate() to use tls_sockets() rather than $OPENSSL for finding certificates using SSLv3 - TLSv1.2, unless $SSL_NATIVE is true. Using tls_sockets() allows testssl.sh to find certificates used by the server even if the server is only using cipher suites not supported by $OPENSSL. This may happen, for example, if the server only supports TLS_ECDHE_ cipher suites with curve X25519 and a version of OpenSSL prior to 1.1.0 is being used. A less likely possibility would be if the server had a certificate with a DH key, and a newer version of OpenSSL that does not support TLS_DH_ cipher suites is being used.
Since tls_sockets() cannot be used to obtain session tickets from the server, an additional test for session ticket lifetime needed to be added.
In order to reduce the number of times the server needs to be queried for certificates, this PR bundles the testing in a similar way to what is already done to test for cipher suites. Currently, each call to get_server_certificate() only tests for one type of certificate. This PR has each call test for more than one type of certificate. For example, one call is made to test for ECDSA, ECDH, DH, DSA, and GOST certificates. If the test is unsuccessful, then the server has none of these certificates. If the test finds a certificate (e.g., an ECDSA) certificate, then another test is run looking for the remaining types (ECDH, DH, DSA, and GOST) until a test is unsuccessful.
For most servers, this will reduce the number of calls to get_server_certificate() from 8 or 9 to 4 or 6.
This fixes#1157.
* Move IDEA to the same category as 3DES
* Rename the category to 3DES_IDEA (JSON)
* Rename 128 Bit category to AVERAGE (JSON)
* Move 256 Bit CBC ciphers into this category too
* Remove category HIGH
Furthermore:
* Code readability improvements, especially in run_cipherlists()
* fix minor bugs (e.g. aNULL ciphers were used in higher categories when --ssl-native was supplied)
* rearrange order for sub_cipherlists()
* proper documentation for arguments of sub_cipherlists() in run_cipherlists()
* add "$cve" "$cwe" arguments to fileout in sub_cipherlists() -- (was passed before but not used)
* change debugging leftover filenames for sub_cipherlists to the JSON identifier
This PR makes a few improvements to run_server_defaults() when run on an SSLv2-only server.
First, it uses sslv2_sockets() to test the server rather than $OPENSSL, so that it will work even if $OPENSSL does not support SSLv2.
Second, it changes run_server_defaults() to only call get_server_certificate() once if $OPTIMAL_PROTO is -ssl2, since calling more than once is a waste - SSLv2 only supports ciphers that use RSA key exchange.
Finally, as some code assumes that $TEMPDIR/intermediatecerts.pem will exist, even if it is empty, this PR changes a couple of places that delete $TEMPDIR/intermediatecerts.pem to instead make the file empty.
When run_server_preference() is run on a server that only supports SSLv2 it incorrectly reports that the server has a cipher order. The reason for this is that $list_fwd and $list_reverse only include one SSLv2 cipher.
In SSLv2 the server sends a list of all ciphers it supports in common with the client and the client chooses which cipher to use. As a result, the server cannot enforce a cipher order for SSLv2.
So, this PR fixes the problem in run_server_preference() by skipping the test for whether the server enforces a cipher order if $OPTIMAL_PROTO is -ssl2 and simply declares that the server does not enforce a cipher order.
Note that this PR is somewhat dependent on #1194, as #1194 needs to be applied in order for $OPTIMAL_PROTO to be set to -ssl2 when testing an SSLv2-only server.
This PR reorganizes run_cipher_per_proto(). Currently run_cipher_per_proto() runs a for loop, which loops over each protocol and prints the set of supported ciphers for each protocol. This PR simply places the body of the for loop in a separate function from the loop itself. This allows the body of the loop to be called for just a single protocol.
While this PR does not change the way that testssl.sh functions, it would allow for a future change in which run_server_preferences() called cipher_pref_check() for protocols in which the server enforces a cipher order and calls ciphers_by_strength() for protocols in which the server does not enforce a cipher order.
This PR reorganizes cipher_pref_check(). Currently, cipher_pref_check() runs a for loop, which loops over each protocol and prints the set of supported ciphers for each protocol. This PR simply places the body of the for loop in a separate function from the loop itself. This allows cipher_pref_check() to be called for just a single protocol rather than for all protocols. Another PR will make a similar change to run_cipher_per_proto().
The reason for this change is that cipher_pref_check() was only intended to be used in cases in which the server enforces a cipher preference order. Some servers, however, enforce an order for some protocols, but not for others. The change in this PR will make it possible in the future to call cipher_pref_check() only for protocols in which the server enforces a cipher order.
This PR fixes two bugs in determine_optimal_proto().
First, sslv2_sockets() returns 3 if the connection was successful.
Second, if all connection attempts using tls_sockets() were unsuccessful, it is possible that $TEMPDIR/$NODEIP.parse_tls_serverhello.txt will not exist, so copying it or grepping it will lead to an error. Checking that $proto is not 22 will fix this as $proto will be empty is $OPENSSL s_client was used and it will be 00, 01, 02, 03, or 04 if tls_sockets() was used and the connection was successful with some protocol higher than SSLv2.
This PR fixes a bug in get_cipher() - one that also appears in sclient_connect_successful().
The code currently assumes that cipher names contain only uppercase letters and numbers. However, ciphers that do not provide authentication include "anon" in the name, which is written in lowercase.
This PR fixes the problem by allowing lowercase letters to appear in cipher names (except in the first portion of the name).
Note that no change was made to similar code in get_protocol(), since the line in get_protocol() only matches TLSv1.3 ciphers, which do not contain any lowercase letters.
In case where the OpenSSL version used cannot successfully do openssl s_client
connects there are a few problems, see #1087.
This PR partly addresses them by
* changing the logic of HTTP header failure: we don't terminate anymore but
continue with a warning message
* we try to find out what the reason was: If it is a missing curve we signal
it back to the user
* we keep track in a global variable KNOWN_OSSL_PROB. It's not being used yet
on all connects as it has not been decided whether we do a connect despite
we know if there's a problem or rather not.
* Give hints to the user for resumption tests, secure renegotiation, CRIME and BREACH.
For the latter --assume-http needs to be supplied for any output.
Also: for finding the OPTIMAL_PROTO now (unless --ssl-native is being used)
sockets are the default which removes in cases where an openssl s_client
connect fails, the initial message 'doesn't seem to be a TLS/SSL enabled server'
and prompt 'Really proceed ? ("yes" to continue)'. For STARTTLS this needs
to be done as well.
Here a minor bug was fixed: when openssl s_client connect in determine_optimal_proto()
succeeded without a protocol supplied, OPTIMAL_PROTO wasn't set. A statement was
added but now it is only being used when --ssl-native was supplied.
Leftover for this workaround is to find out why the number of certificate retrieved is
zero in those cases, despite the fact that there's a valid 'host_certificate.pem' from
tls_socket() calls. Thus still run_server_defaults() stops after 'TLS clock skew'
as certificate_info() is not being called in run_server_defaults(). For now in
those cases 'Problem: Host certificate found but we can't continue with "server defaults"'
is being printed.
In general for the future it would be great if we could e.g. retrieve the header over
TLS sockets.
This PR fixes two problems with modify_clienthello().
First, the function was incorrectly using the variable $key_share instead of $new_key_share. Since $key_share is defined when modify_clienthello() is called from resend_if_hello_retry_request(), but not when it is called from client_simulation_sockets(), this bug does not seem to result in incorrect behavior, but it should still be fixed.
Second, when this function is used to create a second ClientHello in response to a HelloRetryRequest, it removes the key_share extension from the original ClientHello and then appends the new key_share extension at the end. According to https://mailarchive.ietf.org/arch/msg/tls/8ZKCyamcYFaV90h6nf4MUnSPkEE, however, extensions must appear in the second ClientHello in the same order in which they appeared in the first ClientHello.
I am not aware of any servers that will actually complain if the extensions in the second ClientHello do not appear in the same order as in the first ClientHello, bug this fix helps to ensure that the ClientHello messages testssl.sh sends are in compliance with the standard.
This PR is an attempt at addressing #1185. According to https://www.etsi.org/deliver/etsi_ts/103500_103599/10352303/01.01.01_60/ts_10352303v010101p.pdf, if eTLS is in use, then the certificate should contain a subjectAltName extension with one or more "names" containing "visibility information." The "visibility information" is encoded as an otherName with a type-id of 0.4.0.3523.3.1 and a value of
VisibilityInformation ::= SEQUENCE {
fingerprint OCTET STRING (SIZE(10)),
accessDescription UTF8String }
The etsi_etls_visibility_info() function determines whether the certificate includes an "visibility information," and, if it does, extracts the fingerprints and access descriptions.
This PR is a work-in-progress for two reasons. First, it has not been tested against any real certificates that contain "visibility information." Testing against real certificates would be helpful to verify that the parsing of the certificate is correct.
Second, the presentation of the visibility information (both in the printed text and in what is sent to fileout()) may need improvement. Having seen no examples, it is not clear what the contents of accessDescription can be expected to look like. The document says that the contents will be "human-readable text," but it is not clear whether the description will be relatively short or very long.
To finalize #1157 following was done
* move 3DES one line above
* put 128 bit CBC ciphers (ARIA, Camellie and AES) SEED and IDEA into 128Bit
* the remaining 256 bit ciphers NOT supporting AEAD Mac into high
Also 128 bit ciphers are getting a small complaint (LOW, yellow) if
available instead of red (for SEED and IDEA before)
To ease the (future) output rated_output() is included, but it's not being used
yet.
Also often I have docker hosts for testing. If I use them while the external
network is down, I still experience DNS timeouts. I added for dig
timeout values which proved to be reasonable in my tests with and
without network.
Also if an IPv4 or an IPv6 address was supplied testssl.sh doesn't
do (futile) DNS lookups anymore.
... to enable checks whether a curve has been detected by sockets
won't be detected and/or makes problems with remaining openssl
s_client + other calls
Related to #1087
This PR addresses the problem (#1037) that if a hostname resolves to multiple IP
addresses only the first one is being scanned as this IP isn't reachable and the
scan terminated here and didn't continue with the 2nd, 3rd etc.
Same applied to scans with --mx.
This based on the global MULTIPLE_CHECKS which is set to true whenever
such a scan is started.
One minor point: Also if the last IP isn't reachable the output will say
"proceeding with next:". I guess that should be clear looking at the (different outputs).
This PR also fixes a scan problem with a single cipher/pattern (-x/--single-cipher):
previously where only one IP was scanned.
Furthermore some redundant quotes were removed.
This commit addresses #934. It adds a line in the log/csv output
and a json object named "scanProblem" when either the function
fatal() is being called and it logs the reason of the fatal error.
As noted in #1157 the logic of different cipher suites put to categories
needed an improvement.
This commit addresses it by moving first RC2 and RC4 cipher suites to
the low category.
More to follow.
This commit fixes#1163 which lead to the misleading output when
a TLS 1.3 enabled server had no preferences for the TLS 1.3 ciphers
but for anything below (like currently for testssl.NET).
The TLS 1.3 handshake in sockets plus the following openssl handshake
was moved to the top in run_server_preference() so that it can be better
determined whether TLS 1.3 is available. If this section's outcome is
TLS 1.3 is negotiated a single TLS 1.3 handshake with 5 ciphers only is
done forward and reverse. The resulting ciphers are later on compared
whether there's a cipher order for TLS 1.3.
Basically this section should be redone, so that all openssl handshakes
are replaced by sockets. As this would consume more time as it appears
reasonable at this point of time, this was not done yet. A starting point
for this would be tls13_list_fwd + reverse. After release of 3.0 90%
of the code will be replaced anyway.
DHE-RSA-SEED-SHA and SEED-SHA was added to the reverse and forward lists
as some old openssl versions + apache use it.
Also:
Googles ALPN_PROTO grpc-exp was added (to be reconsidered at some certain point)
Some redundant quotes in double square brackets were removed.
All "do_*" variables are now in quotes when tested w if or [[
This addresses #1169: When using JSON as output format when mass testing
AND we have a non-fatal condition when e.g. openssl lacks support for
something it led to an invalid JSON as the warning was put into file w/o
a trailing comma.
The commit removes the warning to be put into the output. We still have the
message on screen + in HTML which is not as optimal as it could be.
Also I did some cleanups related to redundant double quotes I stumbled over while
fixing this.
As a kind of a pre-warning this commit allows the n-1 connection problem to
give feedback on the screen (that wasn't working before).
Also the message on the screen is now more clear and the manpage
gives better advice.
Related to #1172
This PR fixes#1165 by changing resend_if_hello_retry_request() to modify the initial ClientHello rather than having it call prepare_tls_clienthello() to try to generate a new ClientHello that is almost the same as the first. The modification is done using a revised version of create_client_simulation_tls_clienthello(), which is now renamed as modify_clienthello().
Since prepare_tls_clienthello() is no longer used to create a second ClientHello message, argument 7 to that function is no longer needed.
There is at least one extension that will fail on a TLSv1.3 ClientHello if the psk_key_exchange_modes extension is not present (see #990). The PR adds the extension to TLSv1.3 ClientHello messages. OpenSSL, Firefox, and Chrome all include this extension in their ClientHello messages, so including it is unlikely to cause problems for any servers.
.. after some discussion. As TLS 1.3 is not tested here
any RFC 7919 primes using this protocol will not show
up (they in in run_pfs() though). To avoid misunderstandings
" DH key detected with <= TLS 1.2" is now being printed.
This PR provides an additional fix for the issue raised by #1159. It defines a third option for the degree of processing that should be performed by tls_sockets(): "all+". When "all+" is provided, the processing is exactly the same as for "all" with the exception of the creation of the supported_groups extension. For a TLSv1.3 ClientHello, curves that are not supported by $OPENSSL are omitted from the supported_groups extension rather than offering these curves as the least preferred option.
The "all+" option is used in run_server_defaults() where, unlike with almost every other call to tls_sockets(), a successful connection is of no use unless the response can be decrypted. This is also the case for run_alpn(), and so the call to tls_sockets() was also changed to "all+" there. But, the change has no effect at the moment, since run_alpn() sends a TLSv1.2 ClientHello.
As a result of #276, `run_server_defaults()` makes several attempts to find certificates that a server offers if the ClientHello is for TLSv1.2 and no SNI is offered. However, these tests are unnecessary if it is already known that the server does not support TLSv1.1.
This PR modifies `run_server_defaults()` so that the the TLSv1.1-only tests are skipped if the server is known to not support TLSv1.1.
This PR fixes#1159. If tls_sockets() connects to a server using TLSv1.3, it cannot be assumed that the server's certificate is available, as testssl.sh may not have been able to decrypt the server's response. This can happen, for example, if X25519 was used for the key exchange and `$OPENSSL` does not support X25519.
If the connection was successful, but the certificate could not be obtained, then this PR tries again using `$OPENSSL`. However, since `$OPENSSL` does not support TLSv1.3, this will only work if the server supports TLSv1.2 or earlier.
This commit addresses #179 and implements NNTP via STARTTLS. I did
a few tests and it did work so far.
However the binary support needs to be done. I backported in my
fork of @PeterMosmans tree the section from OpenSSL 1.1.1 -- but
it didn't work, see https://github.com/openssl/openssl/issues/7722.
I just tried to patch it as I suggested and it worked then. My
patch is pushed soon after to https://github.com/drwetter/openssl-1.0.2.bad,
however I'll better wait for the official OPenSSL 1.1.1 patch.
This commit finalizes #1139. It displays the DH groups
in both run_logjam() and run_pfs() in a simlilar manner
(except the FFDHE groups).
A common small function pr_dh() was introduced which prints
out the dh group and in round brackets colored DH bits.
This commit finally fixes#547 and makes XMPP handshakes at least
as fast as the other STARTTLS handshakes.
It utilizes dd to read from the file descriptor. In all tests
I ran so far it didn't cause any problems. There's a potential
problem though that dd might block.
This PR fixes#924 and does some foundation for #547. It's a
somewhat preliminary push of code and further work for #547 is required.
XMPP is now similar programmed as other STARTTLS handshakes with the exception
that it is not line based but stream based. That is still the catch here and
needs to be addressed: STARTTLS protocols like IMAP + SMTP use
starttls_full_read() which reads lines until the line is completely received or
the timeout was encountered.
The new function ``starttls_io()`` however does a wait (fixed value: 1 second)
as there's no lf or terminator.
The XMPP STARTTLS handshakes are now the same as in OpenSSL.
There are redundant functions in this code which will be removed later.
Also at some places a hint for lmtp was missing which was added.
The cipher suites names in the RFCs stem (mostly) from IANA, see
https://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml#tls-parameters-4
This PR corrects that in places visible to the user. For backwards
compatibility the cmd line switches still work as before, but there's
a preference to IANA. The RFC naming is labeled as to be retired
in the future.
In addition to 7d36ba9a2e which
added new SSLv2 ciphers to the ciphers file this commit adds those
ciphers also to those functions where needed.
Also it does some housekeeping. [[ doesn't require strings on
the right hand side to be quoted, see bash hackers wiki.
PR #1114 brought #1139 a good step forward. This commit adds
a few tweaks to it:
* the groups in run_pfs() are now also italic, except FFDHE groups
* renaming FF groups to DH groups to provide consistency with the
remainder of testssl.sh
* JSON identifier was renamed from DHE_groups to DH_GROUPS
Open points:
* in run_logjam() there's no warning at all regarding e.g. dh512.badssl.com.
Reading the Logjam paper in section 3.5., first couple of paragraphs we
should warn at least against 512 bits here too.
* how do we treat/label 768 bit and 1024 bit in run_logjam() which comes from
unknown groups? Looks like the paper only was concerned about precompuation.
* In run_logjam() is the bit length not colored but in run_pfs() it is.
* Notation: when do we label FF groups / DH parameter ephemeral?
* Code in run_pfs() and run_logjam() can be merged more.
run_logjam() is only related to TLSv1.2 and earlier ciphers. So, run_pfs() should only update $DH_GROUP_OFFERED if a DH group was found using a non-TLSv1.3 cipher.
On the other side, if run_logjam() happened to have been run first, and it found an ffdhe cipher, then there is no need for run_pfs() to test for it.
In run_pfs(), when information about the finite field groups offered is printed, the color used is based on the length of the key. This information should also be conveyed to fileout() in the severity parameter.
For cipher suites that use ephemeral DH groups, run_pfs() currently only displays information about the group(s) used if the server complies with RFC 7919. In the case of TLSv1.3 this is appropriate, since server can only use the values from this RFC and only if they are offered by the client in the supported_groups extension.
For TLSv1.2 and earlier, however, servers are free to use whatever DH group they want, but run_pfs() only provides information about the group the server uses if the server complies with RFC 7919. (The information is, however, provided by run_logjam()). However, so far no servers comply with RFC 7919's requirement to refuse to negotiate a TLS_DHE cipher if the supported groups extension is present, included DH groups, but none that are supported by the server. There is also reason to believe that this will not change: https://www.ietf.org/mail-archive/web/tls/current/msg26378.html.
So, this PR proposes to change the way that run_pfs() searches for DH groups for TLSv1.2 and earlier. (Note that run_pfs() only checks for TLSv1.2 or earlier if the $EXPERIMENTAL flag is set to true.) First, it removes the test to see if the server will reject a ClientHello that only specifies TLS_DHE cipher suites if it includes a supported_groups extension that only specifies an unrecognized DH group. Instead, if the server supports TLS_DHE cipher suites (at TLSv1.2 or earlier) and the $EXPERIMENTAL flag is true, it will try to find out what group(s) the server uses. Second, it will report the group(s) found even if the server uses a group that does not come from RFC 7919.
The result is that if the server supports selecting groups from the supported_groups extension, it will print all of the groups that the server supports. If the server ignores the supported_groups extension and always uses the same group, it will print essentially the same information as is already printed by run_logjam().
One discrepancy, however, is that this code use pr_dh_quality() to determine how good a DH group is, based on the length of the prime, and pr_dh_quality() has differs from run_logjam() in terms of how it rates groups based on the lengths of their primes.
.. otherwise we'll hit too soon the threshold: Logic: by specifying
a timeout a user indicates that there might be a problem.
Also fatal() now supports a hint which is printed in normal
text (to stderr)
Some Cyrus IMAD if configured with SSL_CTX_set_cipher_list(context, "!TLSv1")
and similar respond with a plaintext 'a002 NO Starttls negotiation failed"
when a not-supported protocol is detected, see #1082.
This PR fixes this by detecting (also) this downgrade. As a precaution
It still issues a warning as this is seems a special configuration.
Looking @ pending #1114 two improvements were done:
1) Keep the status of DH group detected (<name> or "Unknown DH group")
as well as the bit length
2) move the detection to a separate function get_common_prime()
There's still room for improvements when run_pfs() will take
over a part.
Also double code (my bad) from run_logjam() was move to a separate function.
As #1146 noted some installations miss hexdump. Better practice
is to check before what's needed albeit the error message when
a binary is missing does give the user a hint.
Currently the client simulation is based on the handshake data
from SSLlabs which is purely focussed on HTTP -- as SSLlabs does
HTTP only.
In #540 there was a PR addressing the fact that the data is not
what is claims to be -- the handshake of Android 7 seems to be
Chrome for Android and not Android itself.
This PR tries at least to modify the headline for client simulations.
This PR addresses the remaining TCP fragmentation by piping the line buffered
internal print through cat, see also #1130.
It extends 1b52834 which was the same doing for Linux and
OpenBSD.
This PR also consolidates the last remaining low level socket calls
in client_simulation_sockets() into socksend_clienthello().
An negative performance effect is barely measurable.
It also does a check whether the fd 5 is taken by a tty as
I see this while writing the commit message ;-). We might
want to make that line better instead of just echoing. :-)
At the beginning of run_server_preference(), if the attempt to connect to the server is unsuccessful, a message is printed listing all of the ciphers in $list_fwd and $tls13_list_fwd:
no matching cipher in this list found (pls report this): DES-CBC3-SHA:RC4-MD5:DES-CBC-SHA:RC4-SHA:AES128-SHA:AES128-SHA256:AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:ECDH-RSA-DES-CBC3-SHA:ECDH-RSA-AES128-SHA:ECDH-RSA-AES256-SHA:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:DHE-DSS-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:AES256-SHA256:ECDHE-RSA-DES-CBC3-SHA:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-SHA256:ADH-AES256-GCM-SHA384:AECDH-AES128-SHA:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-AES128-SHA:TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256
This message can be misleading. I tested a server that only supported TLSv1.3 using the provided OpenSSL 1.0.2-chacha. The server supported TLS_AES_256_GCM_SHA384, but OpenSSL didn't. However, the message implies that the server does not support TLS_AES_256_GCM_SHA384.
This PR changes the message (and the one included in CSV/JSON output) to only list those ciphers in $list_fwd and $tls13_list_fwd that are actually supported by $OPENSSL.
Note that even with this PR, some ciphers are listed that aren't really supported by $OPENSSL, since the `-s` option isn't used. But, that is #663.
* Put all low level socket related functions close to each other
* removed socksend2 as it was not used and outdated looking forward
* socksend_sslv2_clienthello() renamed to socksend_clienthello() as
it wasn't particular SSLv2 related
* removed the low level socket calls from socksend_tls_clienthello()
and called socksend_clienthello() instead
* renamed socksend_tls_clienthello() to prepare_tls_clienthello()
as it is not a low level function anymore
This PR is also based on #1139, but it addresses ECDH keys rather than DH keys. When run_pfs() prints the list of elliptic curves offered, it colors each curve according to its quality (based on key length). However, the severity level used when the list is sent to fileout() is always "INFO". This PR changes the call to fileout() to make the severity level be based on the quality of the shortest curve that the server offers.
This commit adds LMTP to the STARTTLS protocols
supported. It requires an openssl version which
supports this which is either openssl 1.1.1
or a backported version 1.0.2 (binary is in
process).
A check is in place whetrher the binary supports
this.
Furthermore some framework additions were made for
further STARTTLS protocols like IRC and NNTP.
For XMPP servers, when extracting the SRV-ID and XmppAddr names from the subjectAltName extension, need to take into account that the subjectAltName extension may be marked as critical, in which case there will be the DER encoding of TRUE (0101FF) between the DER encoding of the subjectAltName extension's OID (0603551D11) and the tag for OCTET STRING (04).
This PR is an attempt to address #1097. I have only been able to test it against jabber.topf.org and against locally created test certificates, so it needs more testing.
The main issue that this addresses is that testssl.sh currently checks against the wrong name for XMPP servers. According to RFC 6120, Section 13.7.2.1:
o The initiating entity sets the source domain of its reference
identifiers to the 'to' address it communicates in the initial
stream header; i.e., this is the identity it expects the receiving
entity to provide in a PKIX certificate.
So, if the --xmpphost option is provided, then the name provided in that option should be compared against the name in the certificate rather than the host name.
compare_server_name_to_cert() currently takes the server name to look for in the certificate as an parameter, but every call to compare_server_name_to_cert() uses $NODE as the argument. So, this PR removes the parameter and sets $servername to either $XMPP_HOST or $NODE as appropriate. This small change alone should fix the problem for most XMPP servers since the server's name SHOULD appear in the certificate encoded as a DNS name. That is the case for the one server I could test - jabber.topf.org.
The majority of the code in this PR is to address the other possibilities noted in RFC 6120, Section 13.7.1.2.1. This section notes that an XMPP server's identity name also appear in the subjectAltName extension as an otherName, either an SRV-ID or an XmppAddr identifier. Unfortunately, OpenSSL's certificate printer does not support otherName and just prints "othername:<unsupported>". So, this PR includes code to manually extract any SRV-ID or XmppAddr names from the certificate. This involves parsing the DER encoding of the certificate to look for the subjectAltName extension, looping through all of the names in the extension, and pulling out the names of these two name forms. This code is only run if the server is an XMPP server and the certificate does not have a matching DNS name. So, this code will rarely be executed.
This PR addresses one other issue. There is code in certificate_info() to set the variables $has_dns_sans and $has_dns_sans_nosni. These variables are needed to address the following requirement:
# Find out if the subjectAltName extension is present and contains
# a DNS name, since Section 6.3 of RFC 6125 says:
# Security Warning: A client MUST NOT seek a match for a reference
# identifier of CN-ID if the presented identifiers include a DNS-ID,
# SRV-ID, URI-ID, or any application-specific identifier types
# supported by the client.
While it is relatively easy to determine whether a certificate includes a DNS name in the subjectAltName extension, as noted above, it is not easy to determine whether it has an SRV-ID or an XmppAddr. So, this PR leverages the work compare_server_name_to_cert() does in parsing the subjectAltName extension by having compare_server_name_to_cert() set a global variable indicating whether the certificate has a subjectAltName extension with a relevant name form (DNS, SRV-ID, or XmppAddr for XMPP, or DNS for other servers). $has_dns_sans and $has_dns_sans_nosni are then just set to the value of this global variable.
Instead of checking via uname for Linux this commit does a check
whether the outcome for an external printf is what is expected. This
makes it more compatible e.g. with OpenBSD which surprisingly works
similar like the GNU counterpart.
Also it checks all external printfs installed wther it's the
"right one" to use. Previously it stopped just at the first one
and if this was "wrong", bash's printf was used.
Linux bash internal printf shortened the string
when using len2twobytes() with 3 chars, FreeBSD
e.g. did not.
It worked under both OS though when piping to
the socket with printf.
This commit makes sure that always 2+2 chars
are returned when a 3 char number is supplied.
Kudos @dcooper16
This commit basically reverses the previous commit 305eefc for
Linux only. Here the external printf is working fine, where as
the BSDish counterpart is not (see e.g. #1134).
For Linux is basically addresses #1130 / #1113.
A compatible solution for all OS needs to be found.
This PR removes the changes that were added in #1113. As noted in the conversation for #1113, it was eventually determined that the actual bug was related to the first message fragment being too short rather than the overall length of the message. So, the warning message that is displayed by the code is misleading. In addition, if changes are made to avoid TCP fragmentation, then this code will no longer even test for intolerance to short message fragments.
As @tomato42 pointed out in #1113 '\x0a' causes the printf buffer
to flush before all data was sent. As a result any '\x0a' in
a ClientHello causes a new TCP fragment.
This commit changes all TCP sockets write to use an external
printf if available which doesn't show this behaviour, see #1130.
It was checked against wireshark.
The external printf was available for Linux, FreeBSD 9 and OpenBSD,
so I do not expect any problems with MacOS X either.
There might be further solutions like 'stdbuf' or 'dd' which
are shown in #1130.
As noted in #1130, the current implementation of socksend_tls_clienthello() results in packets being fragmented wherever a '0a' character appears in the message. This cannot be avoided, but there are a few places where a '0a' character appears in which the character could easily be replaced:
* In the session_id for a TLSv1.3 ClientHello.
* In the 32-byte client random value
* In any public key sent in the key_share extension
This PR removes those uses of the '0a' character. While this does not do much to address the problem, it does result in a slight reduction in the amount of fragmentation of messages.
This PR is an attempt to fix the problem identified in #1118.
Currently, get_cipher() and get_protocol() attempt the extract the cipher and protocol from the SSL-Session information printed by OpenSSL s_client. This does not always work for TLSv1.3, however, since OpenSSL 1.1.1 will only print SSL-Session information for a TLSv1.3 connection if the server sends New Session Ticket. If the server doesn't, then get_cipher() and get_protocol() return empty strings.
For TLSv1.3 connections in which the server does not send a New Session Ticket, there seems to be only one other source for this information. A line of the form:
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
[Note that "New" would be "Reused" if the connection were created via session resumption.]
The use of this line seems to be okay for extracting the negotiated cipher, but it cannot be used in general to extract the negotiated protocol. The reason is that this line is created as follows:
c = SSL_get_current_cipher(s);
BIO_printf(bio, "%s, Cipher is %s\n",
SSL_CIPHER_get_version(c), SSL_CIPHER_get_name(c));
While the cipher that is printed seems to be the negotiated cipher, the protocol that is printed is "the SSL/TLS protocol version that first defined the cipher." Since TLS 1.3 ciphers may only be used with TLS 1.3, protocol version printed on this line may be accepted as the negotiated protocol if and only if it is "TLSv1.3."
This PR addresses the problem by modifying get_cipher() and get_protocol() to check the "New, ..., Cipher is ..." line if lines from SSL-Session ("Cipher : ...", "Protocol : ...") cannot be found. In the case of get_protocol() the protocol on the "New, ..., Cipher is ..." will be accepted only if the protocol is "TLSv1.3" and the cipher is a TLSv1.3 cipher.
This PR also adds a check for the "New, ..., Cipher is ..." to sclient_connect_successful(). If this line is present, and the protocol and cipher are not "(NONE)", then this is accepted as an indication that the connection was successful even if the "Master-Key" line does not appear. It is not clear whether this extra test is needed, however, as sclient_connect_successful() will not even look at the text in the output of OpenSSL s_client if function's return value is 0, and OpenSSL s_client should return 0 if the connection was successful.
Most of the curves that were defined for the supported_groups extension in RFC 4492 have been deprecated in RFC 8422 and RFC 8446. Appendix B.3.1.4 of RFC 8446 says that these deprecated values "are used in previous versions of TLS and MUST NOT be offered or negotiated by TLS 1.3 implementations."
According to a recent discussion on the TLS mail list (see, for example, https://www.ietf.org/mail-archive/web/tls/current/msg26974.html and https://www.ietf.org/mail-archive/web/tls/current/msg26980.html) a TLS 1.3 server implementation may choose to reject a TLS 1.3 ClientHello simply because the ClientHello offers one or more of the deprecated curves.
This PR address this issue by no longer offering the deprecated curves in TLS 1.3 ClientHello messages. This only affects run_pfs(), since socksend_tls_clienthello() already does not offer the deprecated curves in TLS 1.3 ClientHello messages.
The change in this PR has no affect on the testing of servers that do not support TLS 1.3. For those that do support TLS 1.3, only the 5 non-deprecated curves are tested with TLS 1.3, but all 30 curves are tested with TLS 1.2.
This PR reduces the number of public keys that are included in the key_share extension for a TLS 1.3 ClientHello.
When creating the key_share extension for a TLS 1.3 ClientHello, generate_key_share_extension() generally omits the public keys for larger finite-field groups (ffdhe3072, ffdhe4096, ffdhe6144, and ffdhe8192) so that the extension will not be overly large. However, the extension that it creates is still much larger than what is created by other software.
For a generic TLS 1.3 ClientHello, socksend_tls_clienthello() offers 7 groups in the supported_groups extension (P-256, P-384, P-521, X25519, X448, ffdhe2048, ffdhe3072) and 6 public keys in the key_share extension (P-256, P-384, P-521, X25519, X448, ffdhe2048). While the largest public key is omitted, this still creates a 665 byte key_share extension.
By contrast, Firefox offers 6 groups in the supported_groups extension (X25519, P-256, P-384, P-521, ffdhe2028, ffdhe3072), but only includes two public keys in the key_share extension (X25519, P-256). OpenSSL 1.1.1 offers 5 groups in the supported_groups extension (X25519, P-256, P-384, P-521, X448) and only includes one key in the key_share extension (X25519). Chrome offers 3 groups in the supported_groups extension (X25519, P-256, P-384) and only includes one key in the key_share extension (X25519).
Following the examples of OpenSSL, Firefox, and Chrome, this PR changes generate_key_share_extension() to include at most two public keys in the key_share extension. In general it will offer the public keys for the first two groups that appear in the supported_groups extension. However, it will still exclude the public key for any ffdhe group larger than ffdhe2048 unless that group appears first in the supported_groups extension.
In most cases this change will simply result in the ClientHello message being smaller. In some unusual cases, this change will force a second round-trip, with the server sending a HelloRetryRequest in order to ask for the key_share that it needs, but this will not affect the results of the testing.
In run_grease() there is a mismatch between the severity level of finds as printed and as sent to fileout(). Problems are labeled as medium when printing, but as CRITICAL in the call to fileout(). This PR fixes the problem by changing CRITICAL to MEDIUM.
This commit fixes#1123 where a security header containing an asterix lead
to a local filename expansion which was included in the CSV file output.
A new function fileout_csv_finding() addresses this.
Also if "$GIVE_HINTS" isn't true the headline and each line in the CSV file doesn't include
anymore the word hint -- which is more consistent with the JSON output.
As described in #1113, some servers will fail if the length of the ClientHello message is 522, 778, 1034, ... bytes (i.e., if length mod 256 = 10) or 526, 782, 1038, ... bytes (i.e., if length mod 256 = 14). This commit avoid this issue for normal testing by adding a 5-byte padding extension to the message if the length would otherwise be one of these lengths.
socksend_tls_clienthello() does not calculate the length of the ClientHello message in the case of a TLS 1.3 ClientHello, since it does not take into account the inclusion of a 32-byte session id. The length value that is being calculated incorrectly is only used to determine whether to include a padding extension, and if so, how long that extension should be.
This fix was previously included as part of PR #1120, since a correct length calculation is needed to avoid a ClientHello length such that length mod 256 = 10, but I removed it from that PR and am making it a separate PR, since it is a bug that should be fixed even if #1120 isn't adopted.
This commit updates the size bug GREASE test in a few ways:
* It removes the changes to socksend_tls_clienthello() - these will be submitted as a separate PR.
* It adds a test for a ClientHello message length of 266 bytes, but only if the server can generally handle messages with lengths between 256 and 511 bytes.
* It corrects the calculation of the length of the padding extension in cases in which a TLS 1 or TLS 1.1 ClientHello is being sent.
Just as some servers will fail if the length of the ClientHello is between 256 and 511 bytes (see RFC 7685), it seems that some servers (or a middlebox sitting in front of the servers) will fail if the length of the ClientHello is 522, 778, 1034, ... bytes in length (i.e., if length mod 256 = 10). I have also encountered one server that will also fail if the length of the ClientHello is 526, 782, 1038, ... bytes in length (i.e., if length mod 256 = 14).
In the case of that one server, the first ClientHello sent by run_pfs() was exactly 1038 bytes, and so run_pfs() was reporting that the server didn't support any PFS ciphers even though it did..
This PR addresses the problem in two ways. First, it modifies socksend_tls_clienthello() so that if the length of the ClientHello would be more than 511 bytes and length mod 256 would be 10 or 14, it adds a 5-byte padding extension in order to ensure that the final length of the ClientHello will not be a length that could trigger the bug.
Second, this PR adds a test to run_grease() to send ClientHello messages of the exact lengths that do trigger the bug so that users can be made aware that their servers have the problem.
In cases where a finding was empty (error condition), the JSON output
wasn't valid because the finding wasn't printed to file.
This commit makes sure that always a finding is printed,
also if it is empty.
FIX#1112
As #1119 noted, there's a warning for users with an OpenSSL 1.1.1
config file because of #1117 / #1098 .
This commit suppresses the warning on the screen if a config file
from OpenSSL 1.1.1 was detected (kludge from
b524b808a1).
This addresses a bug where openssl s_client connects hiccuped
because of newer config files which our openssl 1.0.2 couldn't
swallow.
It appeared first on Debian.
FIX#1117FIX#1098
RFC 8446 specifies the following for the list of certificates provided by the server:
The sender's certificate MUST come in the first
CertificateEntry in the list. Each following certificate SHOULD
directly certify the one immediately preceding it.
In RFC 5246 the "SHOULD" was a "MUST". This commit adds a check of whether the certificates provided by the server are in the correct order and issues a low severity warning if they are not.
As mentioned in #1106 proxying ocsp protocol doesn't work (yet)
This commit notifies the user that it is not possible. One
can ignore that and try by supplying IGN_OCSP_PROXY=true.
It also fixes a typo I probably introduced (pVULN_THRESHLD).
The standard separator after $FNAME_PREFIX is now '-'.
You can as well supply a different <fname_prefix> ending in '.', '_' or ',' , then
no no additional '-' will be appended.
Also a small bash function get_last_char() has been introduced which returns
the last char from a supplied string.
... for curl, wget and sockets. Tested and worked.
Furthermore: fd_socket() now is a bit more injection safe as
an echo statement was exchange by printf. For possible future
changes fd_socket now also has and arg1 for the file descriptor.
... previously it depended on the order of DNS replies otherwise. This was
one outcome of discussion in #1026 where it seemed more logical
to pick an IPv6 address as opposed to an abitrary (v4/v6) address.
This PR fixes checks where those two cmdline options were supplied
but errorneously also the IPv4 address was tested.
It also lables supplied IPv6 addresses as AAAA records
instead of A records.
Still, determine_ip_addresses() has space for improvements.
Some comparisons fixed strings popped up during debugging were polished
to avoid internal quoting
[[ $VAR == "teststr" ]]
will be otherwise expanded to
[[ $VAR == \t\e\s\t\s\t\r ]]
This PR changes run_logjam() so that it does not warn about the use of 2048-bit DH primes, even if the selected prime is a common prime.
This PR leaves two issues unaddressed. First, it does not detect servers that are vulnerable to Attack IV in https://weakdh.org/logjam.html. These are servers that use DH primes that are of sufficient length, but that are poorly generated, and so are still vulnerable to attack.
Second, it does not address the potential problem that use of a common prime could leak information about what server product is being used, even if this information is not leaked through other means (e.g., HTTP headers). This should not be an issue with common primes from an RFC (2409, 3526, 5114, 7919), but would be an issue with product-specific common primes.
This commit fixes a bug mentioned in #1084 where a server
with multiple host certificates wa missing a certificate
number the the host certificate itself.
It also adds a JSON object for the number of host certificates.
According to Section 7.4.2 of RFC 5246, when a server sends its certificate it MUST send a list in which the first certificate is the sender's certificate and "Each following certificate MUST directly certify the one preceding it." testssl.sh currently assumes that the server has populated the list way and so places the second certificate in the list into $TEMPDIR/hostcert_issuer.pem.
However, not all servers have been following this requirement, and so draft-ietf-tls-tls13 (soon to be RFC 8446) only says that servers SHOULD list the certificates in this way and says that clients "SHOULD be prepared to handle potentially extraneous certificates and arbitrary orderings from any TLS version, with the exception of the end-entity certificate which MUST be first."
testssl.sh needs to place the correct certificate in $TEMPDIR/hostcert_issuer.pem, since otherwise any OCSP request it sends will be incorrect, and any attempt to verify and OCSP response will be incorrect as well.
This PR changes extract_certificates() and parse_tls_serverhello() to populate $TEMPDIR/hostcert_issuer.pem with the first certificate in certificate_list that has a subject DN that matches the issuer DN in the server's certificate, rather than simply populating $TEMPDIR/hostcert_issuer.pem with the second certificate in the list.
In testing a random sampling of U.S. government servers, of 57 servers tested 5 reported "unauthorized" for the OCSP URI using the current testssl.sh and all 5 of these reported "not revoked" with this PR. This PR also corrects the same issue in some servers on the Alexa Top 1000, but this was only a problem for 12 of those 1000 servers.
In cases in which the server offers a stapled OCSP response, this commit extracts the OCSP response and then checks the response for the status of the server's certificate. The check is performed in the same way as when the certificate includes an OCSP URI and the "--phone-out" option is set, except that the OCSP response is received from the TLS server rather than coming directly from the OCSP responder. Since this only involves additional processing of data that testssl.sh is already receiving, the check is performed whether or not the "--phone-out" flag is set.
Reduce the offensive tests to 4: the others are "just" / mostly cipher
based checks which should not cause an IDS to block. (This maybe
subject to reconsider at a later time.)
Added a switch --ids-friendly
Updated VULN_COUNT accordingly
Added this (including PHONE_OUT to env debugging output)
Added help()
Manual section added
