See #2032 / #2067, kudos @Wahnes. This is a backport for 3.0
Currently, when "HTTP clock skew" is calculated, this is taken to be the
difference between the timestamp of the moment the HTTPS request was sent and
the date given in the HTTP Date header. This does not yield valid results in
case a HTTP cache is used, either on the client side or on the server side.
According to the HTTP specs, the Date field will contain the timestamp the
response was created, which may not be the timestamp the response was
delivered.
Consider the following example that queries the Varnish project's web server.
Note that Varnish is a popular HTTP caching server, so HTTP caching will of
course be used when serving HTTP responses from this project's web server.
testssl.sh https://varnish-cache.org/ This will typically output a HTTP clock
skew of some thousand seconds.
The patch takes into account the HTTP Age header that caching servers add to
the HTTP response to signal the response's freshness. As client-side caches
normally do not cache HTTPS requests (except maybe for "enterprise" HTTP proxy
caches that do MITM HTTPS proxying), this is mostly targeted to HTTPS websites
that employ server side HTTP caching.
Addtional polishing:
* address my comments in #2032
* add JSON field HTTP_headerAge if they exists
* output HTTP_AGE if it was detected
* do stripping of line feeds closer to where variables were set
This commit fixes an issue with the headers in the structured JSON output. If run_cipher_match() is performed the header that is used is "pretest" rather than "singleCipher".
This commit adds several variables to reset_hostdepended_vars() that are not currently being reset for each host being tested, but that should be reset.
Sending the entire log to /dev/null is a rather expensive way of checking whether
we are inside a git working tree. Use `git rev-parse --is-inside-work-tree` instead.
(cherry picked from commit 1e3f932932)
This commit fixes#2049 by converting newline characters to spaces in JSON and CSV findings.
fileout() calls newline_to_spaces() on any $finding that is to be written to a JSON or CSV file. However, this only affects actual newline characters in the string, not escaped newline characters (i.e., "\n"). Escaped newline characters pass through this function unchanged, but then get converted to newline characters when they are written to the JSON and/or CSV files. This commit fixes the problem by also converting escaped newline characters ("\n") to spaces.
This fixes#1834 and #1435 for branch 3.0. (3.1dev: #2047)
The --connect-timeout option had the problem that under certain circumstances
like parallel mass scanning didn't work. The culprit was that a subshell command
was used to connect to the target but the file descriptor wasn't exported.
The PR changes the logic so that the connect pre-check is still done in a subshell
If this fails it proceeds with error handling if NR_SOCKET_FAIL is above threshold.
Otherwise it just connects again.
This commit fixes a bug in sub_cipherlists() when using sockets to test whether a server supports a set of ciphers with SSLv2. Code that is supposed to extract the list of SSLv2 ciphers to test for is incorrectly reading from the list of non-SSLv2 ciphers.
Some servers are configured to prioritize ChaCha ciphers if those ciphers are preferred by the client, even if the server is generally configured to use the server's cipher preferences rather than the client's. As a result of this, if a ChaCha cipher appears in the ClientHello before a non-ChaCha cipher, the server may select the ChaCha cipher even if the server is configured to prefer the non-ChaCha cipher.
In a few cases, e.g., cloudflare.com for TLS 1.2, this affects the ordering of the ciphers presented by cipher_pref_check(). This PR fixes the problem by having cipher_pref_check() (and check_tls12_pref()) always place any ChaCha ciphers at the end of the cipher list in the ClientHello. This ensures that cipher_pref_check() presents the ciphers in the server's preference order.
As noted in #2016 the detection of curves in find_openssl_binary()
was not yet perfect.
This commit removes another connect call. Also it corrects the port statement
for LibreSSL-like pre-checks as port 0 generally seems to me better suited.
Inline documentation was added.
It worked so far with OpenSSL 1.0.2, 1.1.1 and 3.0 and LibreSSL 3.4.
This is for 3.0. Similar commit (a11bd1585e)
was for 3.1dev, see PR #2020 .
Same as #2014, this is for 3.0 though.
OpenSSL shows certificate serial numbers >35 with a LF (0A). Testssl.sh
just output that which makes JSON invalid and displays the LF in the terminal
too.
This PR fixes that (#2010) by adding text filters so that the
serial number is not a multiline string.
Also this PR introduces a new function: a size check of the cert serial.
Below 8 bytes the CAB Forum's lower limit is hit which says the entropy
from a CSPRNG should be at least 64 bits. It is assumed that below 8 bytes
length this requirement isn't possible to meet (needs to be clarified with
Shannon, 8 bytes seems to low to me).
The high threshold is according to RFC 5280, Section-4.1.2.2 .
See also #2013.
The output has changed, so that on the terminal the serial has one line,
SHA1 and SHA256 each one line. The new json key is "cert_serialNumberLen".
This commit provides a global variable to the RFC 6761 use of "invalid."
which WSL clients don't seem to handle very well, see #1738, #1812.
"invalid." is used as a target to find out in a couple of pre-checks what
is supported by the openssl version.
This PR reduces the number of ``openssl s_client -connect`` by a huge factor.
For the remaining invocations the OS used is being determined and if WSL is
assumed (the check is probably not 100% accurate) it uses ``127.0.0.1:0`` instead.
In (unfortunately only a few) pre-tests the response was immediate.
Also it is possible to use another target if needed by
NXCONNECT=<mytargethere>:<myport> ./testssl.sh <URL>
This is for 3.0. For 3.1dev see #1988 .
Newer configuration files from openssl may include statements
which aren't compatible with our supplied old openssl version.
This commit adds an autodetection of such a file and uses a
openssl.cnf provided by this project then.
The openssl.cnf is from a Debian Jessie package.
See also #1987
A longer while back the section ~ "Testing standard ciphers" was
renamed to "Testing cipher categories". However the internal help
didn't reflect that.
This fixes that, including an addtion to the documentation.
Note: the help still lists "-s --std, --standard" as a cmd line
switch.
Similar to #1972.
* the ignore ~/.digrc option from dig is now parsed from the builtin help
* there was a potential DNS call which is now avoided
* the +noidnout check is now parsed from the builtin help
* the OPENSSL_CONF="" in check_resolver_bins() was moved a few lines higher to avoid other errors in the terminal
* redo check for noidnout by also using the help function of dig
See also #1949
get_server_certificate() includes a few calls to tls_sockets() in which the response will be TLS 1.3 and in which the response will be useless if it cannot be decrypted (since the goal is to obtain the server's certificate). So, these calls to tls_sockets() should specify "all+" rather than "all".
If the order of the cmdline is '-U --ids-friendly' then we need to make sure we catch --ids-friendly. Normally we do not,
see #1717. The following statement makes sure. In the do-while + case-esac loop the check for --ids-friendly will be
executed again, but it does not hurt
Newer dig versions have an option to ignore $HOME/.digrc, older
don't.
This commit adds a patch checking for the availability of
such an option and uses it by default.
If doesn't exist then still dig is used and can still lead to
wrong output. Unfortunately Debian-based distros are not very
good at this. Debian 10, Ubuntu 18.04 still use dig 9.11, whereas
Opensuse 15.2 has 9.16.
Debian 11 and Ubuntu 20.04 use that too.
Thus to be considered for the stable 3.0-branch: move
dig in the resolver functions a couple of lines down.
As mentioned in #1931 the port detection for nmap greppable files
leaves space for improvements.
Ths PR adds a pattern detection of ssl and https in the forth or fifth
parameter of an open port, so those ports will be added to a scan when
a nmap greppable output file is supplied as input to testssl.sh .
Also it does minor code adjustments to utils/gmap2testssl.sh .
Same as #1938, only for 3.0.
This addresses a bug filed in #1935 in 3.1dev when the supplied file
has a .txt extension. In this scenario the input file was nulled
as from the input file in nmap format an internal input file was
generated which has a .txt extension, in the same directory.
The idea was to persist the file for the user.
Now, this internal input file is ephemeral and only written to $TEMPDIR.
There was by mistake a 179 days threshold and also the error message
was wrong when HSTS was exactly set to 179 days, see #1879.
This commit sets it to 180 days and corrects the error messages on
screen.
For DNS queries a trailing dot in the variable $NODE is always fine. For
HTTP queries it is not. and causes the https request to fail.
Backport from 4f1da9b192
Also: removal of ancient CVS_REL relict in $TEMPDIR/environment.txt
