See #2328, original PR #2295 from @w4ntun .
Formally testssl.sh returned an error when it wasn't not possible to determine IP
addresses through DNS resolution, even if --proxy and --ip=proxy flags are set.
The main function always tried to determine IP addresses via DNS and exits with
a fatal error if it cannot do it. Although the client cannot get the IP, the
proxy could, so the SSL/TLS analysis is still possible.
This PR allows the analysis for an HTTP service via a proxy server and the DNS
traffic can be sent directly or through the proxy using the flag --ip=proxy.
ATTENTION: This may be a breaking change for those who don't have a local resolver.
They now have to add --ip=proxy.
In addition:
* help() was amended to add --ip=proxy (was only in the ~i/doc dir before)
* amending ~/doc dir to document it's better to add --nodns=min when there's
no local resolver
This addresses the bug #2330 by implementing a function which removes
control characters from the file output format html,csv,json at the
output.
In every instance called there's a check before whether the string
contains control chars, hoping it'll save a few milli seconds.
A tr function is used, omitting LF.
It doesn't filter the terminal output and the log file output.
see #2325.
"whenever HTTP/1.1 is used then the Accept header uses "text/*" as a MIME type.
This causes some minor issues with some of the checks we are doing"
As noted in #2304, the way that the '&' character is treated in the string part of a pattern substitution changed in Bash 5.2. As a result, the change that was made in #1481 to accommodate older versions of Bash (e.g., on MacOS) now causes testssl.sh to produce incorrect HTML output when run on Bash 5.2.
This commit encodes the '&' characters in the substitution strings in a way that produces correct results on multiple versions of Bash (3.2 on MacOS, 5.2 on Ubuntu 23.10, 5.0 on Ubuntu 20.04).
There are two different scenarios. x0C is the buffsize reply from openldap-like servers
whereas AD servers probably have x84 and return also the OID. The following is kind of
hackish as ldap_ExtendedResponse_parse() in apps/s_client.c of openssl is kind of hard
to understand. It was deducted from a number of hosts.
Bottom line: We'll look at the 9th byte or at the 17th when retrieving the result code
AD:
30 84 00 00 00 7d 02 01 01 78 84 00 00 00 74 0a 01 34 04 00 04 55 30 30 30 30 30 30 30 30 3a 20 [ failed AD .. LdapErr + OID..]
30 84 00 00 00 28 02 01 01 78 84 00 00 00 1F 0A 01 00 04 00 04 00 8A 16 [.. OID ..]
^^ bufflen ^^ resultcode
30 0C 02 01 01 78 07 0A 01 00 04 00 04 00
^^ bufflen ^^ result code
When a server supports client authentication, extract_calist() extracts the list of supported certification authorities sent by the server. extract_calist() uses different code to extract the list from a TLS 1.3 response than from a TLS 1.2 or earlier response, since the CertificateRequest message was changed for TLS 1.3.
For TLS 1.2 and earlier, extract_calist() assumes that the CertificateRequest message is a sequence of certificate types, signature algorithms, and certification authorities. However, the signature algorithms field was added in TLS 1.2 and does not appear in TLS 1.1 and earlier. So, the current code does not work unless the server supports TLS 1.2 or TLS 1.3.
This commit fixes the problem by checking whether the response is a TLS 1.2 response, and skipping over the extraction of the signature algorithms field if the response is neither TLS 1.2 nor TLS 1.3.
Some servers get confused if the signature_algorithms extension is too large. This commit addresses the problem by:
* For TLS 1.2, generally limiting the signature algoritms to those consistent with the key type being tested.
* For TLS 1.3, breaking the list of signature schemes in two, and testing each half of the list separately.
This commit modifies run_fs() to show the signature algorithms the server supports in the ServerKeyExchange message for TLS 1.2 and in the CertificateVerify message for TLS 1.3.
Signature algorithms are not shown for TLS 1.1 and earlier, since for those protocol versions the signature algorithm to use is specified by the protocol. While the signature algorithm used in TLS 1.1 and earlier is weak, testssl.sh already warns if these protocol versions are supported.
This commit fixes#1747 by converting PEM encoded certificates that are sent to fileout() to a single line. As suggested in #1747, '\n' is added after the '----- BEGIN ... -----' line and before the '------ END ... ------' line.
In order to ensure that '\n' appears in the string in the JSON and CSV files, '\\n' is sent to fileout() so that 'printf -- "%b"' converts '\\n' to '\n' rather than converting '\n' to a newline character.
In order to prevent fileout() from converting '\\n' to '\ ', this commit move the fix for #2049 (see PR #2050) from fileout() to fatal().
As a first cleanup action I removed in run_server_preference()
the line with Negotiated Protocol and Negotiated Cipher as
the don't have any real information, see #2235 , comment below:
https://github.com/drwetter/testssl.sh/pull/2235
This commit fixes#2271 by adding the `-no_ssl2` option to the call to get_host_cert() in run_drown(). There is at least one server that causes OpenSSL to hang if this call to get_host_cert() results in an SSLv2 ClientHello being sent. Since this call to get_host_cert() only needs to find the server's certificate in cases in which the server does not support SSLv2, there is no need to send an SSLv2 ClientHello.
This commit simplifies the adding of padding data in a few places. Rather than adding one or two bytes at a time in a "for" loop, all of the padding is added in one step by extracting it from a long padding string. (The one exception is in run_robot(), where a "for" loop is used to add additional padding in case in which the RSA modulus is longer than the pre-defined padding string.)
Extracting the padding from a long string is faster than using a "for" loop and it makes the debugging file a little cleaner.
The idea is the same as PR #1940.
This commit fixes an infinite loop in run_fs() that occurs in cases in which $OPENSSL supports TLS 1.3 and the server supports all of the non-TLS 1.3 FS ciphers that $OPENSSL supports but not all of the TLS 1.3 ciphers that $OPENSSL supports.
The problem is that testing for supported ciphers using $OPENSSL, testing should stop if there are no more ciphers to test (because all of the ciphers supported by $OPENSSL have been determined to be supported by the server). However, currently testing only stops if both the list of TLS 1.3 ciphers and non-TLS 1.3 ciphers is empty. In the problematic case, only the list of non-TLS 1.3 ciphers is empty. Instead of stopping, s_client_options() is called with a -cipher option with an empty list, and s_client_options() simply removes the -cipher option from the command, resulting in a call to $OPENSSL s_client with a full list of non-TLS 1.3 ciphers. Since this call succeeds, the loop continues.
This commit fixes the problem by stopping TLS 1.3 ClientHello testing when the list of TLS 1.3 ciphers is empty and stopping non-TLS 1.3 ClientHello testing when the list of non-TLS 1.3 ciphers is empty.
This commit fileout() calls to ciphers_by_strength() and cipher_pref_check() to indicate whether or not the server enforces a cipher order for a protocol version.
This commit fixes#1311 by only rating the lack of a server-enforced ciper order negatively if there is a difference in the quality rating of the ciphers offered for a particular protocol.
SC2235 is "Use { ..; } instead of (..) to avoid subshell overhead."
In a large number of places testssl.sh uses paraenthesis in complex boolean expressions in order to specify an evaluation order. The paranthesis results in the expression being evaluated in a subshell, which makes evaluation very expensive. This commit addresses the problem by rewriting any expressions that unnecessarily create subshells.
When neat_list() is printing information about a cipher suite that uses (EC)DH key exchange that was obtained using an old version of OpenSSL the rows are not properly aligned, since the key exchange input includes an unexpected trailing space. This commit fixes the problem by removing any trailing spaces from $kx.
determine_cert_compression() and certificate_transparency() do not work in debug mode, since tls_sockets() writes debugging messages to stdout. This commit fixes the problem by having determine_cert_compression() and certificate_transparency() return their results using a global variable rather than writing the results to stdout and having having run_server_defaults() catch the output.
There is at least one server that includes a new session ticket in the same packet as the Finished message. This confuses check_tls_serverhellodone() since the new session ticket is encrypted under the application traffic keys rather than the handshake keys. check_tls_serverhellodone(), being unable to decrypt the new session ticket reports a failure and does not return any of the decrypted data.
This commit fixes the problem by having check_tls_serverhellodone() simply return (or ignore) any data that appears after the Finished message. If such data is returned, then tls_sockets() derives the application traffic keys and decrypts it so that it can be parsed by parse_tls_serverhello().
This commit fixes the way that ciphers are graded by only calling set_ciph_str_score() from neat_list() if the cipher suite is supported (i.e., if how2show is true).
This fixes#2203. When used -E with -9 and --show-each the grading wasn't
correct as all ciphers printed were included in the grading verdict instead
of just the ones available.
This commit modifies ciphers_by_strength() and run_server_preference() so that the message indicating that ciphers are listed by strength is not printed until the list of supported ciphers has been determined. This is in support of #1311, as it will allow the message to be modified based on the set of supported ciphers.
This commit also modifies both ciphers_by_strength() and cipher_pref_check() so that the order in which ciphers are listed (by strength or server preference) is not printed if the server does not support the protocol.
As jsonID is not set by run_crime, make the fileout invocation for
servers supporting only TLS 1.3 use the literal "CRIME_TLS" instead.
Previously running testssl with CSV or JSON output would produce an item
with the wrong ID.
Currently run_server_preference() will stop if it cannot determine whether the server enforces a cipher preference order.
This commit changes run_server_preference() so it will continue running even if this case, so that the list of ciphers supported with each TLS protocol is provided. Since it is not known whether the server enforces a cipher order, the list of supported ciphers is ordered by strength.
See #2127. the line seems very long though.
Note: this was previously commited as #2184 but as there were two mistakes
and one other thing which could be improved I decided to make a hard reset.
Apologize if it caused inconvenience.
OpenSSL 3.0.0 and later supports specifying the FFDHE groups from RFC 7919 in the "-groups" (or "-curves") option of s_client.
This commit modifies find_openssl_binary() to check whether $OPENSSL supports this. This information is then used by run_client_simulation(), if client simulation testing is being performed using $OPENSSL. If the "curves" for a client include FFDHE groups, then they will be included in the simulated ClientHello.
On macOS indark mode the git tag in grey wasn't visible. It was
changed now to light grey but it has to be tested how it looks
on a white terminal background.
Also CVS variables were removed which had no meaning.
In rare? occassions where the STARTTLS FEAT request only displays AUTH instead
of AUTH TLS, testssl.sh fails as it cannot upgrade to TLS.
Required by RFC 4217 is only AUTH ("MUST"), AUTH TLS is optional ("should"), see section 6.
This commit relaxes the presence of TLS after AUTH and it fixes#2132.
This commit fixes testssl.sh's calls to the "$OPENSSL ciphers" command.
The main issue it fixes is when actually_supported_osslciphers() is called to get a list of non-SSLv2 ciphers supported by $OPENSSL. With OpenSSL 1.0.2, the "-tls1" option needs to be used to exclude SSLv2 ciphers. With LibreSSL, the "-tls1" option may be provided, but it has no effect. With OpenSSL 1.1.1 and newer, the "-tls1" option causes TLSv1.2-only ciphers (e.g., AES256-SHA256) to be excluded (when the "-s" option is also used).
This commit fixes the problem by allowing "-no_ssl2" to be provided as an option to actually_supported_osslciphers(). For versions of $OPENSSL that support SSLv2, "-no_ssl2" is replaced by "-tls1". For versions of $OPENSSL that do not support SSLv2, "-no_ssl2" is simply removed.
This commit also changes openssl2hexcode() to include the "-tls1" option when $OPENSSL supports SSLv2, since openssl2hexcode() should only return a non-SSLv2 cipher.
determine_optimal_sockets_params() makes two attempts to send a TLS 1.2 ClientHello, with each attempt trying 127 ciphers. However, this leaves 97 ciphers from etc/cipher-mapping.txt that are not tried, most of which use ARIA or CAMELLIA. This commit adds a third attempt a send a ClientHello that offers these 97 remaining ciphers. This helps to ensure that support for TLS 1.2 is detected and that later calls to tls_sockets() work, even if the server only supports the ARIA/CAMELLIA ciphers that are not included in TLS12_CIPHER or TLS12_CIPHER_2ND_TRY.
In some rare cases, a connection to the server will fail with tls_sockets() but not with $OPENSSL. This can cause determine_optimal_sockets_params() to call add_proto_offered() to indicate that the protocol is not supported, and then determine_optimal_proto() to later call add_proto_offered() to indicate that it is supported. However, PROTOS_OFFERED does not get changed, since add_proto_offered() only modifies PROTOS_OFFERED if the protocol is not already listed.
This commit fixes the problem by allowing add_proto_offered() to change an entry for a protocol from "no" to "yes".
If determine_optimal_proto() happens to connect to the server using TLS 1.2, then this commit will set TLS12_CIPHER_OFFERED to the cipher from that connection, if TLS12_CIPHER_OFFERED was not set in determine_optimal_sockets_params(). This will allow run_protocols()'s test of a TLS 1.3 ClientHello to work better, if the problem is that no cipher supported by the server is included in TLS12_CIPHER or TLS12_CIPHER_2ND_TRY.
run_server_preference() calls "default_proto=$(get_protocol $TMPFILE)" even if all attempts to connect to the server failed. This will result in default_proto incorrectly being set to TLS 1.2. This commit fixes the issue by only calling get_protocol() if an attempt to connect to the server was successful.
When running in --ssl-native mode, run_fs() will not detect ECDHE ciphers if the server supports both DHE and ECDHE ciphers and the ECDHE ciphers are only supported with curves that are not offered by $OPENSSL by default. This commit fixes this by adding extra connection attempts with the -curves parameter explicitly provided.
Older versions of LibreSSL that do not support TLS 1.3 only include a small list of curves in the supported_groups extension by default, so need to retry with curves explicitly defined even with versions of $OPENSSL that do not support TLS 1.3.
This commit fixes#2131 by having run_fs() attempt a TLS 1.2 ClientHello if the initial TLS 1.3 ClientHello fails. The TLS 1.2 ClientHello will offer many more curves than the TLS 1.3 ClientHello offers, and so it may succeed if the server supports ECDHE ciphers, but only with curves that were removed by RFC 8446.
With a TLS 1.3 connection using $OPENSSL, a session ID will only appears as part of a post-handshake session ticket. However, when $OPENSSL s_client is called as in determine_optimal_proto() (i.e., with "< /dev/null"), a post-handshake session ticket will not always be received, even if the server supports it. This can result in NO_SESSION_ID incorrectly being set to true. This commit fixes the issue by setting NO_SESSION_ID to true by default, and then setting it to false if a session ID is returned by any connection to the server.
This commit fixes#2147 by having awk search for additional possible strings to start the CRL Distribution Points output. Unless the CRLDP extension is malformed, it will begin with "Full Name", "Relative Name", "Reasons", or "CRL Issuer".
This commit fixes two more issues with using OpenSSL 3.X. When $OPENSSL is used to obtain a fingerprint, OpenSSL 3.X prepends the fingerprint with "sha1" or "sha256" rather than "SHA1" or "SHA256". In addition, the way that OpenSSL 3.X writes distinguished names causes a space character to appear at the beginning of "$cn" and "$issuer_CN" in certificate_info().
PR #2140 contains a bug when handling flat JSON files. FIRST_FINDING should only be set to true in the case of structured JSON output, since it is only in that case that fileout_insert_warning() appends a comma to the JSON file. This commit fixes the problem.
OpenSSL 3.0.X uses different names for some elliptic cures in the "Server Temp Key" line than previous previous versions. This commit addresses this issue by checking for both names.
This commit fixes#2138 by having testssl.sh not wrap early JSON findings in a clientProblem object if the finding is created by a mass testing child and all findings are being placed in a common file. It also sets FIRST_FINDING to true in case another finding is written before the "service" information is written.
Since fileout_insert_warning() adds a comma after the finding is written, the JSON can become corrupted in mass testing if a clientProblem finding is written and then no additional findings are written for that test. In order to try to prevent this, the commit adds several fileout() calls to determine_optimal_proto() in cases in which testssl.sh might exit before testing begins.
