When neat_list() is printing information about a cipher suite that uses (EC)DH key exchange that was obtained using an old version of OpenSSL the rows are not properly aligned, since the key exchange input includes an unexpected trailing space. This commit fixes the problem by removing any trailing spaces from $kx.
determine_cert_compression() and certificate_transparency() do not work in debug mode, since tls_sockets() writes debugging messages to stdout. This commit fixes the problem by having determine_cert_compression() and certificate_transparency() return their results using a global variable rather than writing the results to stdout and having having run_server_defaults() catch the output.
There is at least one server that includes a new session ticket in the same packet as the Finished message. This confuses check_tls_serverhellodone() since the new session ticket is encrypted under the application traffic keys rather than the handshake keys. check_tls_serverhellodone(), being unable to decrypt the new session ticket reports a failure and does not return any of the decrypted data.
This commit fixes the problem by having check_tls_serverhellodone() simply return (or ignore) any data that appears after the Finished message. If such data is returned, then tls_sockets() derives the application traffic keys and decrypts it so that it can be parsed by parse_tls_serverhello().
This commit fixes the way that ciphers are graded by only calling set_ciph_str_score() from neat_list() if the cipher suite is supported (i.e., if how2show is true).
This fixes#2203. When used -E with -9 and --show-each the grading wasn't
correct as all ciphers printed were included in the grading verdict instead
of just the ones available.
This commit modifies ciphers_by_strength() and run_server_preference() so that the message indicating that ciphers are listed by strength is not printed until the list of supported ciphers has been determined. This is in support of #1311, as it will allow the message to be modified based on the set of supported ciphers.
This commit also modifies both ciphers_by_strength() and cipher_pref_check() so that the order in which ciphers are listed (by strength or server preference) is not printed if the server does not support the protocol.
As jsonID is not set by run_crime, make the fileout invocation for
servers supporting only TLS 1.3 use the literal "CRIME_TLS" instead.
Previously running testssl with CSV or JSON output would produce an item
with the wrong ID.
Currently run_server_preference() will stop if it cannot determine whether the server enforces a cipher preference order.
This commit changes run_server_preference() so it will continue running even if this case, so that the list of ciphers supported with each TLS protocol is provided. Since it is not known whether the server enforces a cipher order, the list of supported ciphers is ordered by strength.
See #2127. the line seems very long though.
Note: this was previously commited as #2184 but as there were two mistakes
and one other thing which could be improved I decided to make a hard reset.
Apologize if it caused inconvenience.
OpenSSL 3.0.0 and later supports specifying the FFDHE groups from RFC 7919 in the "-groups" (or "-curves") option of s_client.
This commit modifies find_openssl_binary() to check whether $OPENSSL supports this. This information is then used by run_client_simulation(), if client simulation testing is being performed using $OPENSSL. If the "curves" for a client include FFDHE groups, then they will be included in the simulated ClientHello.
On macOS indark mode the git tag in grey wasn't visible. It was
changed now to light grey but it has to be tested how it looks
on a white terminal background.
Also CVS variables were removed which had no meaning.
In rare? occassions where the STARTTLS FEAT request only displays AUTH instead
of AUTH TLS, testssl.sh fails as it cannot upgrade to TLS.
Required by RFC 4217 is only AUTH ("MUST"), AUTH TLS is optional ("should"), see section 6.
This commit relaxes the presence of TLS after AUTH and it fixes#2132.
This commit fixes testssl.sh's calls to the "$OPENSSL ciphers" command.
The main issue it fixes is when actually_supported_osslciphers() is called to get a list of non-SSLv2 ciphers supported by $OPENSSL. With OpenSSL 1.0.2, the "-tls1" option needs to be used to exclude SSLv2 ciphers. With LibreSSL, the "-tls1" option may be provided, but it has no effect. With OpenSSL 1.1.1 and newer, the "-tls1" option causes TLSv1.2-only ciphers (e.g., AES256-SHA256) to be excluded (when the "-s" option is also used).
This commit fixes the problem by allowing "-no_ssl2" to be provided as an option to actually_supported_osslciphers(). For versions of $OPENSSL that support SSLv2, "-no_ssl2" is replaced by "-tls1". For versions of $OPENSSL that do not support SSLv2, "-no_ssl2" is simply removed.
This commit also changes openssl2hexcode() to include the "-tls1" option when $OPENSSL supports SSLv2, since openssl2hexcode() should only return a non-SSLv2 cipher.
determine_optimal_sockets_params() makes two attempts to send a TLS 1.2 ClientHello, with each attempt trying 127 ciphers. However, this leaves 97 ciphers from etc/cipher-mapping.txt that are not tried, most of which use ARIA or CAMELLIA. This commit adds a third attempt a send a ClientHello that offers these 97 remaining ciphers. This helps to ensure that support for TLS 1.2 is detected and that later calls to tls_sockets() work, even if the server only supports the ARIA/CAMELLIA ciphers that are not included in TLS12_CIPHER or TLS12_CIPHER_2ND_TRY.
In some rare cases, a connection to the server will fail with tls_sockets() but not with $OPENSSL. This can cause determine_optimal_sockets_params() to call add_proto_offered() to indicate that the protocol is not supported, and then determine_optimal_proto() to later call add_proto_offered() to indicate that it is supported. However, PROTOS_OFFERED does not get changed, since add_proto_offered() only modifies PROTOS_OFFERED if the protocol is not already listed.
This commit fixes the problem by allowing add_proto_offered() to change an entry for a protocol from "no" to "yes".
If determine_optimal_proto() happens to connect to the server using TLS 1.2, then this commit will set TLS12_CIPHER_OFFERED to the cipher from that connection, if TLS12_CIPHER_OFFERED was not set in determine_optimal_sockets_params(). This will allow run_protocols()'s test of a TLS 1.3 ClientHello to work better, if the problem is that no cipher supported by the server is included in TLS12_CIPHER or TLS12_CIPHER_2ND_TRY.
run_server_preference() calls "default_proto=$(get_protocol $TMPFILE)" even if all attempts to connect to the server failed. This will result in default_proto incorrectly being set to TLS 1.2. This commit fixes the issue by only calling get_protocol() if an attempt to connect to the server was successful.
When running in --ssl-native mode, run_fs() will not detect ECDHE ciphers if the server supports both DHE and ECDHE ciphers and the ECDHE ciphers are only supported with curves that are not offered by $OPENSSL by default. This commit fixes this by adding extra connection attempts with the -curves parameter explicitly provided.
Older versions of LibreSSL that do not support TLS 1.3 only include a small list of curves in the supported_groups extension by default, so need to retry with curves explicitly defined even with versions of $OPENSSL that do not support TLS 1.3.
This commit fixes#2131 by having run_fs() attempt a TLS 1.2 ClientHello if the initial TLS 1.3 ClientHello fails. The TLS 1.2 ClientHello will offer many more curves than the TLS 1.3 ClientHello offers, and so it may succeed if the server supports ECDHE ciphers, but only with curves that were removed by RFC 8446.
With a TLS 1.3 connection using $OPENSSL, a session ID will only appears as part of a post-handshake session ticket. However, when $OPENSSL s_client is called as in determine_optimal_proto() (i.e., with "< /dev/null"), a post-handshake session ticket will not always be received, even if the server supports it. This can result in NO_SESSION_ID incorrectly being set to true. This commit fixes the issue by setting NO_SESSION_ID to true by default, and then setting it to false if a session ID is returned by any connection to the server.
This commit fixes#2147 by having awk search for additional possible strings to start the CRL Distribution Points output. Unless the CRLDP extension is malformed, it will begin with "Full Name", "Relative Name", "Reasons", or "CRL Issuer".
This commit fixes two more issues with using OpenSSL 3.X. When $OPENSSL is used to obtain a fingerprint, OpenSSL 3.X prepends the fingerprint with "sha1" or "sha256" rather than "SHA1" or "SHA256". In addition, the way that OpenSSL 3.X writes distinguished names causes a space character to appear at the beginning of "$cn" and "$issuer_CN" in certificate_info().
PR #2140 contains a bug when handling flat JSON files. FIRST_FINDING should only be set to true in the case of structured JSON output, since it is only in that case that fileout_insert_warning() appends a comma to the JSON file. This commit fixes the problem.
OpenSSL 3.0.X uses different names for some elliptic cures in the "Server Temp Key" line than previous previous versions. This commit addresses this issue by checking for both names.
This commit fixes#2138 by having testssl.sh not wrap early JSON findings in a clientProblem object if the finding is created by a mass testing child and all findings are being placed in a common file. It also sets FIRST_FINDING to true in case another finding is written before the "service" information is written.
Since fileout_insert_warning() adds a comma after the finding is written, the JSON can become corrupted in mass testing if a clientProblem finding is written and then no additional findings are written for that test. In order to try to prevent this, the commit adds several fileout() calls to determine_optimal_proto() in cases in which testssl.sh might exit before testing begins.
This commit adds the use of tls_sockets() to run_tls_fallback_scsv() to perform testing when the --ssl-native flag is not used. With this commit, run_tls_fallback_scsv() only uses tls_sockets() instead of $OPENSSL if the ClientHello needs to include the TLS_FALLBACK_SCSV flag, but it is not supported by $OPENSSL, or if the protocol that would be negotiated is SSLv3 and $OPENSSL does not support SSLv3.
There is at least one server that will not negotiate TLS_DHE_* cipher suites with TLS 1.2 and below if the supported_groups extension is present but does not include any DH groups. This commit adds the DH groups that are currently in the TLS 1.3 ClientHello to the TLS 1.2 and earlier ClientHello.
If $connect_success is false, then sclient_auth() does not "return" any value, and the calling function treats this as if sclient_auth() had returned 0.
This commit fixes sclient_auth() so that 1 is returned if $client_success is false.
This commit changes prepare_tls_clienthello() so that the RSA-PSS algorithms are offered in the signature algorithms extension of TLS 1.2 and below ClientHello messages.
Under Darwin using LibreSSL it was not possible to test for session
resumption by session ID.
This fixes#2096 by checking not only the return value of the s_client
hello but also whether a probable certificate is being returned.
It is now being tested whether the binary locale exists and
there's a global introduced for that.
Also there's no fileout warning at this early stage anymore
as it leads to non-valid JSON.
This fixes#2103 in 3.1dev.
Therefore a new global function was declared checking whether any of
the known locales work on the client without seeting them.
C / POSIX should work as well for LC_COLLATE.
This fixes#2100 for 3.1dev.
This PR addresses a bug where a user encountered the question "The results
might look ok but they could be nonsense. Really proceed".
That happened under Darwin and probably some LibreSSL versions when
checking some hosts. sclient_auth() returned 1 indicating no SSL/TLS
handshake could be established.
This PR modifies sclient_auth() so that in those cases 0 is returned by
skipping the check for the session ID. As NO_SSL_SESSIONID needs to
be set when there's no session ID, this is done separately.
This commit adds parsing the success value of the STARTTLS upgrade
in LDAP. Only possible values whould be 0 or one according to RFC 2380.
All values not equal to zero will terminate the check.
Also, this PR renames sockread_serverhello() to sockread() as the word
serverhello is pretty misleading. It just reads from ANY socket. (sorry
to confuse people here, that should have gone into a separate PR).
Also sockread() and sockread_fast() are better documented.
See #1258
To do:
* more robustness. At least the success value from the response need to be retrieved and checked via starttls_io().
* double check the pre-handshake before the OID whether it's correct for every case
* documentation
* inline help
It seems to work though against db.debian.org
In run_http_header() the GET command is first sent over TLS using a background process, and then if that does not hang, it is sent again in the foreground. Similarly, service_detection() runs the command in the background.
This commit changes determine_optimal_proto() to follow the example of run_http_header() as protection against the possibility of the HTTP query stalling.
This commit fixes determine_optimal_proto() and run_server_defaults() so that a "Local problem" is reported if a $URL_PATH is specified, the server is TLS 1.3-only, and $OPENSSL does not support -enable_pha (and the server does not offer client authentication as part of the initial TLS handshake).
Checking for client authentication with TLS 1.3 requires post-handshake authentication, which does not appear to be supported by LibreSSL. This commit improves the check for client authentication when testing a TLS 1.3 server using LibreSSL by having determine_optimal_proto() first test for connectivity with TLS 1.3 without checking for client authentication and then performing a separate check for client authentication using a non-TLS 1.3 protocol.
This commit only affects the flow of the program if a $URL_PATH is specified, the server supports TLS 1.3, and $OPENSSL supports TLS 1.3 but not -enable_pha.
testss.sh may still provide incorrect information about client authentication if a $URL_PATH is provided, the server is TLS 1.3-only, and LibreSSL is used.
Based on initial testing, this commit improves the check for client authentication in the case that the server only requests client authentication for specific URLs. However, it does not work correctly if the server supports TLS 1.3 and $OPENSSL is a version of LibreSSL that supports TLS 1.3 in s_client. The problem is that LibreSSL does not support post-handshake authentication with TLS 1.3
As noted in #1709, some servers will only request client authentication if a specific URL is requested. This commit modifies the check for client authentication, in the case that a $URL_PATH is provided, by having testssl.sh perform a GET request on the URL provided on the command line.
As discussed in #2079, this commit adds "</dev/null" to calls "$OPENSSL s_client" that are intended to test whether a certain option is supported. This is done to prevent hanging in the case that the option is supported, a TLS server happens to be listening on the port to which s_client tries to connect, and the connection is successful.
This PR also adds a new helper function, sclient_supported(), which is called from locally_supported() and run_prototest_openssl(). The helper function makes use of the already defined variables $HAS_SSL2, $HAS_SSL3, and $HAS_TLS13 in order to avoid calling "$OPENSSL s_client" when it has already been determined whether $OPENSSL supports the specified protocol.
... for all forward + non-mdns lookups.
This might help to avoid supplying domain names to local hosts (they would need
to have a valid certificate for the short DNS name then).
Fixes#2077
* address my comments
* add json fields HTTP_headerTime + HTTP_headerAge if they exists
* output HTTP_AGE if it was detected
* do stripping of line feeds closer to where variables were set
This commit fixes two issues with the headers in the structured JSON output. First, if run_cipher_match() is performed the header this is used is "pretest" rather than "singleCipher". Second, the headers for "serverPreferences" and "fs" are swapped.
Shellcheck complains on line 2234 that keyopts is referenced but not assigned and there is no explanation in the code (or in the documentation) why "$keyopts" is there. This commit adds a comment so that "$keyopts" isn't deleted as part of a code cleanup.
This commit adds several variables to reset_hostdepended_vars() that are not currently being reset for each host being tested, but that should be reset.
testssl.sh currently calls "git log --format='%h %ci' -1 2>/dev/null" three times. This commits changes testssl.sh to make this call just once and then use Bash string manipulation to extract the necessary information from the result.
Sending the entire log to /dev/null is a rather expensive way of checking whether
we are inside a git working tree. Use `git rev-parse --is-inside-work-tree` instead.
This commit fixes#2049 by converting newline characters to spaces in JSON and CSV findings.
fileout() calls newline_to_spaces() on the $findings that are to be written to JSON and CSV files. However, this only affects actual newline characters in the string, not escaped newline characters (i.e., "\n"). Escaped newline characters pass through this function unchanged, but then get converted to newline characters when they are written to the JSON and/or CSV files. This commit fixes the problem by also converting escaped newline characters ("\n") to spaces.
This fixes#1834 and #1435.
The --connect-timeout option had the problem that under certain circumstances
like parallel mass scanning it didn't work. The culprit was that a subshell command
was used to connect to the target but the file descriptor wasn't exported.
The commit changes tha logic so that this connect is still done in a subshell
as a pre-check if it's possible to connect. If this fails it proceeds with
error handling if NR_SOCKET_FAIL is above threshold. Otherwsie it just connects
again.
When testing of the alexa 500 it worked for me(tm). It would be great if others
can give it a try.
This commit fixes a bug in sub_cipherlists() when using sockets to test whether a server supports a set of ciphers with SSLv2. Code that is supposed to extract the list of SSLv2 ciphers to test for is incorrectly reading from the list of non-SSLv2 ciphers.
As noted in https://github.com/drwetter/testssl.sh/pull/1963#issuecomment-954935175, there are two places in which debugme1() is called in which the command to be run prints something to stdout, which is then redirected to a file. Now that debugme1() includes a redirect from stdout to stderr, these two lines do not work as expected. The output appears in the terminal and an empty file is created.
This commit fixes the issue by not using the debugme1() function in these two places.
This commit changes a few lines of code that checks OpenSSL version numbers to recognize 3.0.1 and 3.1.0 in addtion to 3.0.0.
According to https://www.openssl.org/blog/blog/2018/11/28/version, API/ABI compatibility is guaranteed for the same MAJOR version number (starting with version 3), so it should be sufficient to just check that $OSSL_VER_MAJOR is 3.
This commit adds PrioritzeChaCha detection, as requested in #1733. If the server (1) enforces a cipher order for a given protocol, (2) supports at least one non-ChaCha cipher for that protocol, and (3) supports at least one ChaCha cipher that would normally be less preferred than the ChaCha cipher, then a test is performed to see if the server selects the ChaCha cipher if it is listed first in the ClientHello.
While OpenSSL 1.0.2-chacha supports more than 128 cipher suites, other versions of OpenSSL tend to support far fewer than that.
Calling check_tls12_pref() adds to the run time of cipher_pref_check(), but is only needed if OpenSSL would otherwise send a ClientHello with more than 128 cipher suites. So, this commit adds another check and only calls check_tls12_pref() if the server has the size limit bug and OpenSSL supports more than 127 ciphers.
The code added by #2024 creates a cipher list with two consecutive colons. While this doesn't seem to be a problem, the commit removes the extra colon.
Some servers are configured to prioritize ChaCha ciphers if those ciphers are preferred by the client, even if the server is generally configured to use the server's cipher preferences rather than the client's. As a result of this, if a ChaCha cipher appears in the ClientHello before a non-ChaCha cipher, the server may select the ChaCha cipher even if the server is configured to prefer the non-ChaCha cipher.
In a few cases, e.g., cloudflare.com for TLS 1.2, this affects the ordering of the ciphers presented by cipher_pref_check(). This commit fixes the problem by having cipher_pref_check() (and check_tls12_pref()) always place any ChaCha ciphers at the end of the cipher list in the ClientHello. This ensures that cipher_pref_check() presents the ciphers in the server's preference order.
This commit fixes an error in s_client_options() when the function is called with an empty "-cipher" list and $OPENSSL supports "@SECLEVEL". This happens, for example, when ciphers_by_strength() is called for TLS 1.3.
The call to `openssl s_client` will fail is the cipher option is provided with an empty list or a list that just contains "@SECLEVEL=0". So, "@SECLEVEL=0" should only be added if the "$ciphers" list is non-empty. If "$ciphers" remains empty, then the "-cipher" option will not be added to the command line.
As noted in #2016 the detection of curves in find_openssl_binary()
copuld be improved.
This commit removes another connect call. Also it corrects the port statement
for LibreSSL-like pre-checks as port 0 generally seems to me better suited.
Inline documentation was added.
It worked so far with openssl 1.0.2, 1.1.1 and 3.0 and LibreSSL 3.4.
This needs to be backported to 3.0.
In order to be in line with 3.0 this adds a statement to avoid double port
statements for WSL users in a precheck condition
$OPENSSL s_client -groups $curve -connect 127.0.0.1:0:8443
This commit changes that to $OPENSSL s_client -groups $curve -connect 127.0.0.1:8443
which could cause a delay for WSL users and needs further investigation.
Not clear why port 8443 was chosen.
OpenSSL shows certificate serial numbers >35 with a LF (0A). Testssl.sh
just output that which makes JSON invalid and displays the LF in the terminal
too.
This commit fixes that (#2010) by adding filters so that the
serialnumber is not a multiline string.
Also this commit introduces a new function: a size check of the cert serial.
Below 8 bytes the CAB Forum's lower limit is hit which says the *entropy*
from a CSPRNG should be at least 64 bits. It is assumed that below 8 bytes
length this requirement isn't possible to meet (needs to be clarified with
Shannon, 8 bytes seems to low to me).
The high threshold is according to RFC 5280, Section-4.1.2.2 .
See also #2013.
The output has changed, so that on the terminal the serial has one line,
SHA1 and SHA256 each one line. The new json key is "cert_serialNumberLen".
In order to save time and to spare WSL users lame DNS lookups
I removed a couple of connect calls from 14 to 4:
$OPENSSL s_client -ssl2 -connect invalid.
-->
OPENSSL s_client -ssl2
NXCONNECT to localhost IP and port 0 is now the default when WSL
is detected. Not sure whether this is working under all circumstances,
so this needs some testing.
This commit provides a global variable to the RFC 6761 use of "invalid."
which WSL clients doen't seem to handle very well, see #1738, #1812.
With this commit it e.g. is possible to use
NXCONNECT=localhost:0 ./testssl.sh <TARGET>
to save some time.
This commit will be amended later.
Newer configuration files from openssl may include statements
which aren't compatible with our supplied old openssl version.
This commit adds an autodetection of such a file and uses a
openssl.cnf provided by this project then.
A longer while back the section ~ "Testing standard ciphers" was
renamed to "Testing cipher categories". However the internal help
didn't reflect that.
This fixes that, including an addtion to the documentation.
Note: the help still lists "-s --std, --standard" as a cmd line
switch.
* the ignore ~/.digrc option from dig is now parsed from the builtin help
* there was a potential DNS call which is now avoided
* for +noidnout check however there's a call to invalid. added
* the OPENSSL_CONF="" in check_resolver_bins() was moved a few lines
higher to avoid other errors in the terminal
Tested on (EOL) Ubuntu 14.04 which only has dig in an older version
See also #1950
This commit fixes#1961 in the 3.1dev branch by leaving NODEIP set to the server's IP address rather than changing it to the DNS name in the case of STARTTLS XMPP.
In order to address the problem of $OPENSSL s_client not working with STARTTLS XMPP if an IP address is provided to -connect, the -xmpphost option is used to provide the DNS name.
The fact that debugme1() redirects to stderr and the calls to this functions
redo that is deliberately as in the future we might want to use debugme1
without redirection.
... to address #1956 and other places. Similar to #1957,
only for the 3.1dev rolling release branch.
Also it changes debugme1() back? to output debug
statements only when $DEBUG >= 1. Per default here
also stderr is used.
get_server_certificate() includes a few calls to tls_sockets() in which the response will be TLS 1.3 and in which the response will be useless if it cannot be decrypted (since the goal is to obtain the server's certificate). So, these calls to tls_sockets() should specify "all+" rather than "all".
This commit changes run_server_defaults() so that the test for certificate compression is not run in --ssl-native mode. This fixes an issue that was caught by 21_baseline_starttls.t.
This commit adds a check for whether the server supports certificate compression (RFC 8879). If it does, then the list of supprted compression methods is output in the server's preference order.
If the order of the cmdline is '-U --ids-friendly' then we need to make sure we catch --ids-friendly. Normally we do not,
see #1717. The following statement makes sure. In the do-while + case-esac loop the check for --ids-friendly will be
executed again, but it does not hurt
Newer dig versions have an option to ignore $HOME/.digrc, older don't.
This commit adds a patch checking for the availability of such an option and
uses it by default. See #1894 .
If this option doesn't exist then still dig is used and can still lead to
wrong output. Unfortunately Debian-based distros are not very
good at this. Debian 10, Ubuntu 18.04 still use dig 9.11, whereas
Opensuse 15.2 has 9.16. Debian 11 and Ubuntu 20.04 use that too.
This commit adds a new function, print_n_spaces(), which prints a sequence of (up to 80) space characters.
This new function is used to replace a few places in testssl.sh in which a sequence of space characters is printed by calling 'out " "' in a loop. The new function is much faster than the current code, so it will make testssl.sh run slightly faster.
As mentioned in #1931 the port detection for nmap greppable files
leaves space for improvements.
Ths PR adds a pattern detection of ssl and https in the forth or fifth
parameter of an open port, so those ports will be added to a scan when
a nmap greppable output file is as input to testssl.sh .
Also it does minor code adjustments to utils/gmap2testssl.sh .
This addresses a bug filed in #1935 in 3.1dev when the supplied file
has a .txt extension. In this scenario the input file was nulled
as from the input file in nmap format an internal input file was
generated which has a .txt extension, in the same directory.
The idea was to persist the file for the user.
Now, this internal input file is ephemeral and only written to $TEMPDIR.
