This commit adds a new function, print_n_spaces(), which prints a sequence of (up to 80) space characters.
This new function is used to replace a few places in testssl.sh in which a sequence of space characters is printed by calling 'out " "' in a loop. The new function is much faster than the current code, so it will make testssl.sh run slightly faster.
In parse_cmd_line() error messages are usually printed to stderr, but in three places the messages are printed to stdout. This commit modifies those three lines so that they also print to stderr.
The commit also replaces a call to a non-existant function, tmln_magenta_term, with a call to tmln_magenta.
This commit addresses a few issued related to the use of testssl.sh with OpenSSL 3.0.0-alpha14.
First, when pkey is used to print a DH key that uses an unknown group, OpenSSL 3.0.0-alpha14 labels the prime and generator using "P:" and "G:" rather than "prime:" and "generator:". (In PR #1586 it was noted that OpenSSL 3.0.0-alpha1 used "prime P:" and "generator G:". The x509 command in OpenSSL 3.0.0-alpha14 still uses "prime P:" and "generator G:" when printing a DH public key in a certificate, but the pkey command just uses "P:" and "G:").
Second, when the pkey command is used to print a DH key that uses certain common primes (e.g., groups from RFC 3526 or RFC 7919), OpenSSL 3.0.0-alpha14 simply prints "GROUP: " followed by a short name for the group rather than printing the value of the prime and generator.
Finally, the "-text" option no longer works if the input is a public key. Fortunately, the "-text_pub" option provides the expected results with all versions of OpenSSL and LibreSSL.
This commit makes some minor improvements to code that converts ASCII-HEX to binary.
First, testssl.sh currently has two functions that do the same thing: asciihex_to_binary() and hex2ascii(). This commit all calls to either of these functions with calls to hex2binary(), which is based on the current asciihex_to_binary().
This commit also changes direct use of printf to calls to hex2ascii() in generate-ccm-counter-blocks(), ccm-compute-tag(), and generate_gcm_counter().
Finally, this commit cleans up the implemention of hex2binary() a bit and introduces the option to use xxd, if that program is available. Using xxd rather than multiple calls to printf has a couple of advantages. xxd is a bit faster. However, the primary advantage is that when debugging (i.e., using bash -x), each call to hex2binary() only includes a few steps, regardless of how long the string is, whereas using printf the call to hex2binary() could take hundreds of steps.
Modify sclient_auth() to use checks similar to sclient_connect_successful() to determine whether the connection attempt was successful. Replace uses of awk and grep with Bash internals string comparisons.
This commit is a first step towards addressing #1709. It attempts to determime whether certificate-based client authentication is (1) not requested, (2) optional, or (3) required. If it is either optional or required, then it extracts the list of CA names (DNs) that the server sends in its CertificateRequest message.
The code for extracting the CA list from the CertificateRequest message seems to be working correctly. However, this commit is incomplete for a couple of reasons. First, it does not produce any new output, it just collects the information. Second, sclient_auth() needs some work.
The current sclient_auth() simply returns 0 if $OPENSSL returned 0. This may be okay if only trying to determine whether certificate-based client authentication is required. However, if it is optional, then the output will include "CertificateRequest", but $OPENSSL will return 0, since the connection was successful even though the client did not provide a certificates.
If $OPENSSL does not return 0, then sclient_auth() checks whether Master-Key is present. This works for TLS 1.2 and earlier, but not for TLS 1.3. So, sclient_auth() needs to be updated to work correctly with TLS 1.3.
The modified version of sclient_auth() will set CLIENT_AUTH and CLIENT_AUTH_CA_LIST for any version of TLS, but the remaining part of the code needs work. As I am not clear on the reason for this code, I need some help with it. Why does the code only look for "CertificateRequest" if "Master-Key" is present? Why is there a check for Session-ID in a function that is supposed to just be checking for client authentication. Why is CLIENT_AUTH set to false if SESSION-ID is absent (this is a no-op since CLIENT_AUTH would already have been false)?
See #1148 and #1805.
As noted in #1148, testssl.sh is not current designed to handle a mass testing file in which CSV, HTML, LOG, and/or JSON file names are provided in the mass testing file. If a child process receives a command line with one of the files, it assumes the same command-line option was provided to the parent so that the output of every test is being written to this one file. If this assumption is wrong, then either the file will not be created at all or it will be malformed since it will be missing header and/or footer information.
This PR partially addresses the problem by introducing new command-line arguments that are for internal use only. These command line arguments allow a child process to distinguish between a CSV, HTML, LOG, or JSON file that it is supposed to create itself versus one that is to be shared by all of the child processes.
There is one major limitation to this PR. The code for handle command-line arguments in the mass testing file is very simple and cannot handle whitespace characters, whether they are enclosed in quotes or are escaped. So, any file names included in the mass testing file cannot have whitespace characters.
This commit fixes#1699 by setting FIRST_FINDING to true in fileout_banner() if $do_json_pretty is true.
When $do_json_pretty is true, fileout_banner() calls fileout_pretty_json_banner(), which starts a new sectio in the JSON file. Setting FIRST_FINDING to true ensures that a comma is not placed before the first entry in this new section. This is the same as is done in other places when a new section is stated: fileout_section_header() and fileout_insert_warning().
This commit modifies a few functions to use fewer external function calls. In most cases this involves replacing external function calls with Bash internal functions, but in one case it involves replacing multiple external function calls with one call to awk.
This commit makes a few changes to the way that some functions work.
is_ipv4addr() and is_ipv6addr() will now strictly only accept a string that is an IPv4 (or IPv6) address and nothing else.
A couple of changes were also made to match_ipv4_httpheader(). First, lines that match $excluded_header (formerly $whitelisted_header) are not processed in the while loop. This prevents the excluded header from being output in the case that $HEADERFILE includes a non-excluded header with an IPv4 address and an excluded header with a string that looks like an IPv4 address.
The list of excluded headers was also modified to exclude any line that begins "Server: " rather than just lines that begin "Server: PRTG". According to https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Server, the "Server" header describes the software used by the server, so it seems reasonable to expect that this header line will never contain an IPv4 address. Also, looking at some old test results I found cases in which Oracle software version numbers in the Server header were mistakenly matched as IPv4 addresses.
The commit updates the test for the "Visibility Information" certificate extension used in the ETSI Enterprise Transport Security protocol.
The main change is to support OpenSSL 3.0.0, which prints more information about subject alternative names that are encoded as otherName. For otherName types for the OpenSSL has no information, it prints "otherName: <OID>::<unsupported>" rather than just "otherName: <unsupported>". So, testssl.sh needs to account for the possibility that the OID for the visibility information certificate extension will be printed.
This commit also updates the reference for this extension and changes the name of the function from etsi_etls_visibility_info() to etls_ets_visibility_info() since the name of the protocol was changed from Enterprise TLS (eTLS) to Enterprise Transport Security (ETS).
This commit does not change the output to the terminal or to JSON/CSV, even though those outputs use the previous name of eTLS rather than ETS.
The ALPN GREASE test was written based on draft-ietf-tls-grease-01, which reserved all ALPN identifiers beginning with the prefix "ignore/". This commit changes the test to align with RFC 8701, which instead reserves {0x0A,0x0A}, {0x1A,0x1A}, ..., {0xFA,0xFA}.
As there as suggestions to check intermediate certificates for things such as expiration date, this commit saves the text versions of each of the intermediate certificates so that they are available to extract additional information.
This commit checks whether any intermediate certificates provided by the server include an extended key usage extension that asserts the OCSP Signing key purpose.
This commit replaces #1680, which checks for such certificates by comparing the server's intermediate certificates against a fixed list of known bad certificates.
run_cipherlists() checks for support for different groups of ciphers, but does not indicate which ciphers in each group are supported. So, for example, if the JSON file indicates that there is a problem with severity level "HIGH" because the "LOW" ciphers are available, there is no clear indication of which of these ciphers are supported by the server.
If run_server_preference() is run with "--color 3", then there will be a visual indication (via color) of the ciphers the server supports that are considered bad, but this information does not appear in the JSON (or CSV) output. The JSON (or CSV) output will include information about every cipher that is supported, but the severity level is always "INFO".
This commit addresses this problem by changing the fileout() calls in ciphers_by_strength() and cipher_pref_check() that output each supported cipher individually so that the "severity" argument is an indication of the quality of the cipher. With this, information about which bad ciphers are supported can easily be found in the JSON/CSV output.
When testssl.sh is called with an unknown option it prints something like:
0: unrecognized option "--option"
It should be printing the name of the program rather than "0". This commit fixes that.
This commit separates pr_cipher_quality() into two functions, one that returns the quality of a cipher as a numeric rating (get_cipher_quality()) and one that prints a cipher based on its quality (pr_cipher_quality()). This separation allows get_cipher_quality() to be used to determine how good a cipher is without having to print anything. Having this ability would be helpful in implementing the changes suggested in #1311.
PR #1373 changed get_cn_from_cert() to handle certificate subject names that include more than one CN attribute. It did this by converting newline characters to spaces. It seems that this resulted in a space character being added to the end of the string returned by get_cn_from_cert() even in the case that the subject name only included one CN attribute. The presence of the space character in returned value caused compare_server_name_to_cert() to determine that the CN attribute did not contain a DNS name (since DNS names cannot include spaces), and so compare_server_name_to_cert() reports that the server name does not match against the CN in the subject. This may be the reason for the problem noted in #1555.
This commit fixes the above problem and also enhances the matching of the CN in the subject name against the server's name. Currently, compare_server_name_to_cert() assumes that the subject field contains at most one CN attribute. However, as noted in #1373, some certificates include subject names with more than one CN attribute, and RFC 6125 (Section 6.2.2) indicates that the certificate subject name include more than one CN, with each specifying a different DNS name.
So, in addition to fixing the problem with the space character, this commit also enhances the CN matching to work even if the certificate includes more than one CN attribute in the subject name.
In some cases when the Trust finding is printed, there is no space between the results when SNI is used and the results without SNI (which appear in paraenthesis). This commit adds the missing space.
This commit fixes several issues related to Shellcheck issue SC2034: unused variables.
In most cases variables are declared in a function, but are referenced later. The exceptions are:
* SESS_RESUMPTION is declared and values are assigned to it, but it us never used. (Same applies for not_new_reused in sub_seession_resumption().)
* In run_cipherlists(), there is a typo in the declaration of sslv2_tdes_ciphers.
* In get_caa_rr_record(), "hash", "len", and "line" are used but not declared.
This commit adds support for EdDSA (Ed25519 and Ed448). In particular:
* It modifies prepare_tls_clienthello() to include Ed25519 and Ed448 in the signature_algorithms extension of the TLS 1.2 and earlier ClientHello (RFC 8422).
* It modifies run_server_defaults() and get_server_certificate() to check whether the server offers EdDSA certificates with TLS 1.3.
* It modifies certificate_info() to handle certificates signed with EdDSA or with EdDSA public keys, even if $OPENSSL does not support pretty printing such keys and signatures.
* It modifies read_sigalg_from_file() to recognize EdDSA signatures even if $OPENSSL does not.
PR #1519 requested that testssl.sh show the signature algorithm that the server uses during the TLS handshake. In TLS 1.3, this appears in the CertificateVerify message. In TLS 1.2 it appears in the ServerKeyExchange message when the chosen cipher suite uses an ephemeral (DH or ECDH) key, except in the case of cipher suites that provide no authentication. This information is not present in TLS 1.1 and earlier, as the hash algorithm to use in these earlier versions of the protocol is hard coded into the specification.
This commit takes a first step towards being able to show the signature algorithm by extending parse_tls_serverhello() to extract the signature algorithm when it is present. Matching the output produced by OpenSSL, it output two separate lines, the "Peer signature type" (RSA, RSA-PSS, DSA, ECDSA, Ed25519, or Ed448) and the "Peer signing digest" (MD5, SHA1, SHA224, SHA256, SHA384, or SHA512). This will allow the same function to extract the signature algorithm and digest, whether the handshake was performed using "$OPENSSL s_client" or tls_sockets().
This commit fixes two issues related to $SHOW_SIGALGO.
First, cipher_pref_check() does not show the signature algorithm if any of the ciphers were found using tls_sockets(), since the call to tls_sockets() does not specify that the server's certificate should be extracted.
Second, in run_beast() the call to tls_sockets() indicates that the server's certificate should be extracted if "$SHOW_SIGALGO" is true, even if "$WIDE" is false. While this does not cause any problems, extracting the certificate is a waste of effort if "$WIDE" is false, since the signature algorithm is not shown in that case.
The "-sigalgs" option is used in get_server_certificate() to obtain certificates the server uses with TLS 1.3. get_server_certificate() is currently designed to use $OPENSSL, if $OPENSSL supports TLS 1.3.
LibreSSL 3.1.{0,1} has added client support for TLS 1.3, but does not support the "-sigalgs" option. So, this commit determines whether the "-sigalgs" option is supported, and if it isn't, then uses tls_sockets().
When cipher_pref_check() is called in "--ssl-native" mode and the specified protocol is not supported, the message indicating a "local problem" is not properly formatted.
PR #1619 set the grade cap to 'F' is the server has a certificate with an RSA with e=1, however, it did not change the rating in the JSON/CSV output. This commit changes the cert_keySize rating to CRITICAL for an RSA key with e=1, regardless of the size of the modulus. It also uses pr_svrty_critical() to print the exponent in this case.
This commit makes a couple of improvements to set_key_str_score().
It rates (finite-field) DH keys the same as RSA and DSA keys.
Second, in the case of a server that has more than one certificate, the current code sets $KEY_EXCH_SCORE based on the length of the public key in the last certificate that is parsed. This commit changes set_key_str_score() so that $KEY_EXCH_SCORE is set based on the weakest public key.
Note that there is still the issue that the key exchange score does not take into account any ephemeral keys used. However, that needs to be addressed by callling set_key_str_score() from run_fs() and run_logjam(), as certificate_info() cannot provide information about ephemeral keys.
This commit adds additional information to the "Server key size" line for a certificate if the subject public key is RSA, ECDSA, or DH.
For RSA it show the public exponent. For ECDSA, it shows the curve. For DH, it shows the group used, if it is a common prime.
This commit fixes#1433 by adding "@SECLEVEL=0" to the "$OPENSSL s_client" and "$OPENSSL ciphers" command lines if that option is supported. Adding this option configures OpenSSL to support some weak ciphers that it would not use in the default configuration.
There is one place in parse_tls_serverhello() that returns 8 if the server's response is not well-formed TLS. However, there is no code in testssl.sh that is prepared to handle this return value. Every function except run_protocols() only distinguishes between 0, 2, and everything else. run_protocols(), however, gets confused if tls_sockets() returns a value that it is not expecting. So, this commit changes parse_tls_serverhello() to return 1 whenever the server's response can not be parsed.
There is code at the beginning of parse_tls_serverhello() that checks whether the server's response appears to consist of a sequence of messages of the form <protocol><content type><content>. However, at the moment the check is only performed if "$do_starttls" is false. This commit changes parse_tls_serverhello() so that the check is always performed.
This commit addresses two compatibility issues with LibreSSL 3.1.0, which has added client support for TLS 1.3.
The first issue is that LibreSSL has named the TLS 1.3 ciphers that it supports AEAD-AES256-GCM-SHA384, AEAD-CHACHA20-POLY1305-SHA256, and AEAD-AES128-GCM-SHA256, rather than using the OpenSSL names, which are TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, and TLS_AES_128_GCM_SHA256. (Draft versions of OpenSSL 1.1.1 names these ciphers TLS13-AES-256-GCM-SHA384, TLS13-CHACHA20-POLY1305-SHA256, TLS13-AES-128-GCM-SHA256.) There are several places where testssl.sh checks whether a cipher suite is a TLS 1.3 cipher by checking whether its OpenSSL name begins with "TLS_" (or "TLS13"). In order to work with LibreSSL 3.1.0, these checks also need to consider names that begin with "AEAD-" to be TLS 1.3 ciphers.
Second, in sub_session_resumption() there is code that adds "-no_ssl2" to the "$OPENSSL s_client" command line if that option is supported. If "-no_ssl2" is not supported, then other protocol information is added to the command line. I believe this code was written with the assumption that any version of OpenSSL that supports "-no_ssl2" does not support TLS 1.3. However, LibreSSL 3.1.0 supports both. So, this commit changes the code to add the "-no_ssl2" option only if TLS 1.3 is not supported.
This commit addresses two compatibility issues with LibreSSL.
First, with LibreSSL, "$OPENSSL s_client" does not support the "-curves" option, so the "-groups" option needs to be used instead. Note that with LibreSSL, the command line "$OPENSSL s_client -groups $curve -connect invalid." will not work, as it will complain "no port defined," but will not indicate whether the specified curve is supported. Adding a port number fixes that problem. (There does not seem to be a need to include a port number for other tests, such as whether the "-curves" option itself is supported.)
Second, including "-out -" in the command line for "$OPENSSL genpkey" causes LibreSSL to create a file with the name "-" if the algorithm is supported. This is not an issue at the moment, since LibreSSL's genpkey does not support X25519 or X448. However, both genpkey with both OpenSSL and LibreSSL uses stdout as the default output if no "-out" is specified, so the "-out -" is not necessary.
generate_key_share_extension() and prepare_tls_clienthello() currently check the $OPENSSL version number to determine whether X25519 and X448 are supported. The commit changes these functions to use $HAS_X25519 and $HAS_X448.
run_ticketbleed() and sub_session_ticket_tls() each include one call to "$OPENSSL s_client". For each of these calls the expected response is a TLS 1.2 or earlier ServerHello. However, if $OPENSSL supports TLS 1.3, then a TLS 1.3 ClientHello will be sent.
This commit fixes this problem in two ways. For the call in run_ticketbleed(), "-no_tls1_3" is added to the command line if "$OPENSSL" supports TLS 1.3. For the call in sub_session_ticket_tls(), this commit changes the function so that the same ClientHello version is sent as will sent by run_ticketbleed() via sockets.
This commit fixes a couple of issues related to the use of testssl.sh with OpenSSL 3.0.0-alpha1.
First, when the command line includes an unknown option (e.g., -ssl2), OpenSSL 3.0.0-alpha responds with "Unknown option: -ssl2" rather than "Option unknown option -ssl2". This commit addresses this by making the check for "unknown option" case insensitve.
Second, the printing a DH key, OpenSSL 3.0.0-alpha1 labels the prime and the generator using "prime P:" and "generator G:" rather than just "prime:" and "generator:". This commit by changing testssl.sh to match on either string.
This commit fixes a problem with run_logjam() when run in --ssl-native mode. If $OPENSSL does not support any DH export ciphers, then no test for such cipher is performed. However, the results of "test" is still checked, leading to testssl.sh incorrectly reporting that the server supports DH EXPORT ciphers.
This commit extends run_server_preference() to list every cipher supported by each protocol even in cases in which the server does not enforce a preference order.
For protocols where the server enforces a cipher order the list of supported ciphers is ordered by server preference (as now). For protocols where the server does not enforce a cipher order, the ciphers are listed by encryption strength (as run_cipher_per_proto() does).
In order to implement this, ciphers_by_strength() was extended to offer a non-wide mode.
This commit is an attempt to fix#1514. The commit is mostly based on a suggestion at https://unix.stackexchange.com/questions/57940/trap-int-term-exit-really-necessary. Even with that change, it seemed that if testssl.sh were in the middle of executing run_cipher_per_proto() when it received a signal, it would not stop until that function had completed. This seems to have something to do with subshells. Changing the while loop in run_cipher_per_proto() seems to have fixed that issue. So, I also made similar changes to the while loops in prettyprint_local().
This commit fixes#1551 by changing get_cipher() to recognize RFC names that begin with SSL_*. It also modifies run_beast() so that it does not get stuck in an infinite loop if get_cipher() doesn't return a valid cipher name.
This commit modifies run_cipherlists() to align with pr_cipher_quality().
The biggest change made by this commit is that it breaks the current list of STRONG ciphers into two lists: one for AEAD ciphers that offer forward secrecy (STRONG) and one for AEAD ciphers that do not offer forward secrecy (GOOD).
The remaining changes are just minor tweaks:
* A few ciphers that use MD5 are moved from AVERAGE and 3DES to LOW.
* '!AECDH' was added to the OpenSSL description for LOW to catch one cipher in OpenSSL 1.0.2-chacha that offers no authentication that was being included in the LOW list.
This commit also changes sub_cipherlists() to change the output when a cipherlist with a rating of 6 is not present. There was a "FIXME" associated with this output, but it didn't matter before since there were no cipherlists with a rating of 6.
This PR modifes pr_cipher_quality() as proposed in #1548 so that GOST ciphers are handled correctly. It changes pr_cipher_quality() so that the OpenSSL name is used in cases in which no RFC name is defined. It also adds a case statement for GOST so that GOST ciphers (that do not use MD5 or Null encryption) are marked as pr_svrty_low (as they are in run_cipherlists) rather than just being assigned the default rating (5).
This commit fixes the way pr_cipher_quality handles the OpenSSL names of some ARIA ciphers that either provide no authentication or that use CBC padding.
This commit makes several changes to the way that ciphers are rated by pr_cipher_quality:
* It upgrades SEED ciphers to considered as strong as the corresponding AES ciphers.
* It downgrades ciphers that use AEAD, but that use a non-FS key exchange (TLS_DH_*, TLS_ECDH*, TLS_PSK_WITH_*) from best to good, thus giving them the same rating as AEAD ciphers that use static RSA (TLS_RSA_*).
* It downgrades some CBC ciphers to low (4) that are currently rated as neither good nor bad (5).
* It modifies the ratings created using OpenSSL names to provide the same ratings as those created using RFC names.
The permitted values for $DISPLAY_CIPHERNAMES are "rfc-only", "openssl-only", "openssl", and "rfc". However, get_install_dir() incorrectly sets $DISPLAY_CIPHERNAMES to "no-rfc" if it cannot find the $CIPHERS_BY_STRENGTH_FILE. ("no-rfc" is the string users would specify at the command line for the --mapping option, but not the value that $DISPLAY_CIPHERNAMES is set to internally).
As noted in #1481, testssl.sh has a problem with printing percent ('%') characters.
At one point, the function out() was implemented as `/usr/bin/printf -- "${1//%/%%}"`. When this was the case, any '%' needed to be replaced with '%%' since '$1' was being used as the format string. This was changed, however, by https://github.com/drwetter/testssl.sh/commit/8a2fe5915a8d4f70ba423f98eff02a6646935aa9. Since the format string is now "%b" rather than '$1', the replacement is not needed anymore. Instead, the replacement now causes any '%' to be printed to be duplicated.
This problem does not happen very often, but does sometimes occur when a '%' character appears in a URI, such as in an HTTP redirect, a certificate revocation list, or an OCSP URI.
This PR fixes two issues related to the generation of HTML files.
First, text that is to appear in the HTML file is first passed through html_reserved() to replace reserved characters with their corresponding entity names (e.g., '>' becomes '>'). html_reserved() seems to work correctly on Ubuntu Linux, but it does not work as expected on MacOS. On MacOS, rather than converting '>' to '>', it gets converted to '\>', and the backslash is rendered by browsers.
This PR appears to fix the problem. However, given that the original version of html_reserved() was not portable, this revised version should be tested on multiple platforms.
I also noticed that in almost every case in which a string is passed to html_out(), it is first run through html_reserved(), but for some reason that is not the case in out() and outln(). I can't see any reason why html_reserved() is not called first in these two cases, so this PR adds in the calls.
This commit fixes two minor issues related to HSTS_MIN:
* If there is a misconfiguration the recommended max-age should be based on $HSTS_MIN rather than being hardcoded to 15552000 seconds = 180 days.
* If max-age is exactly $HSTS_MIN, testssl.sh shouldn't say that max-age is too short while also say that >= $HSTS_MIN seconds is recommended.
Since, in cases in which the server enforces a cipher order, both run_cipher_per_proto() and run_server_preference() list every cipher supported by the server for each protocol, there was a discussion at one point about eliminating run_cipher_per_proto() and extending run_server_preference().
This PR takes a step in that direction by providing the option to present the "Cipher order" in wide mode.
This PR fixes one Shellcheck issue:
In testssl_3.1dev_20200208.sh line 2395:
HEADERVALUE="$(fgrep -Fai "$key:" $HEADERFILE | head -1)"
^-- SC2197: fgrep is non-standard and deprecated. Use grep -F instead.
The man page for grep states that fgrep is the same a grep -F and that grep is deprecated. So, fgrep -F is just redundant.
Currently, the function neat_list() uses the variable "export", but does not define it. The result is that "export" variable in the calling function is used.
This PR fixes that by defining "export" as a local variable in neat_list() and by setting its value via a new parameter to the function.
This PR also removes a "FIXME" from run_rc4() since the problem has already been fixed.
This PR fixes a minor bug in get_pub_key_size(). If the key size is being determined manually and length encoding requires 4 bytes, then the current code computes the length incorrectly. This is a very insignificant bug, since does not apply to RSA or ECC keys, and the key would have to be at least 16 megabytes long for it to require 4 bytes to encode.
This PR also cleans up get_pub_key_size() a bit by replacing `i=$i+...` with `i+=...` and by enclosing math in `$(( ... ))`.
This PR adds support for post-handshake messages when using sockets with TLS 1.3 connections. If a TLS 1.3 connection is established and the connection is to remain open after tls_sockets() finishes, then after the client's Finished message is sent the master secret and the application traffic keys are computed. This PR also adds two new functions to send and receive application data over a TLS 1.3 connection.
This PR also includes two proofs-of-concept for the use of the new functions. receive_app_data() is called immediately after the client's Finished message is sent. Some server's will send new session tickets immediately after the handshake is complete. If they do, then the code will decrypt and parse the session ticket messages.
This PR also modifies service_detection() to try using sockets if the server only supports TLS 1.3 and $OPENSSL does not support TLS 1.3. After the handshake is complete, this code sends an HTTP GET request and reads the response. The code is fairly slow and it doesn't always work. However, since it is only used in cases in which $OPENSSL cannot work, it can't hurt to try using sockets.
Due to a bug, the shellcheck program will complain if a variable is defined as an array but is later used as an ordinary string, even if the two uses are locally defined variables in different contexts. The error message is:
SC2178: Variable was used as an array but is now assigned a string.
While the warnings are not highlighting any actual problems in testssl.sh, this PR gets rid of the warnings by renaming a few variables.
The implementation of AES-GCM in #1473 is much slower than the original version, even when the authentication tag is not being computed. This PR modifies the code in gcm() in order to significantly speed up the encryption/decryption time (when authentication tags are not being computed).
This PR adds processing of the Finished messages in TLS 1.3 handshakes. It also addresses some shellcheck issues.
If in debug mode, the HMAC of the transcript hash of the handshake context ($msg_transcript) is computed and compared against the Finished message sent by the server.
If the full server response is parsed and the connection with the server is not to be closed when tls_sockets() completes, then the TLS 1.3 handshake is completed by creating the client Finished message and sending it to the server.
This PR reorganizes the code for deriving TLS 1.3 symmetric keys in order to facilitate implementing the full key schedule. For example, rather than having a single function to derive the handshake traffic keys, this PR creates one function to derive the handshake secret and a separate function to derive the handshake traffic keys. The second function has been generalized so that it can derive either client or server traffic keys. Separating into two functions also makes the handshake_secret available for later use to derive the master secret and then the application traffic secrets and the application traffic keys.
This PR also changes where there message transcript is created, a message transcript will also be needed to derive the application traffic secrets. This PR includes the code to add the messages to the initial message transcript that will be needed for the input to the application traffic secret derivation function.
RFC 8446 specifies cipher suites that use three symmetric encryption algorithms, all of which are Authenticated Encryption with Associated Data (AEAD) algorithms. In each of these algorithms when data is encryption an authentication tag is created, which allows the recipient to verify that the data has not been modified. The authentication may also cover some additional data that was not encrypted.
The current implementations of these algorithms in testssl.sh decrypt the ciphertext, but do not check that the authentication tag is correct (which involves the recipient computing the correct tag for the received data and then comparing it to the provided tag). While testssl.sh can get away with not checking authentication tags when receiving data, the ability to compute authentication tags is needed in order to send encrypted data as TLS servers would reject any encrypted data that did not have a correct authentication tag. Being able to send encrypted data is necessary to be able to complete the TLS 1.3 handshake.
This PR replaces the current implementations of the symmetric encryption algorithms with full implementations of each of the algorithms. These full implementations include the ability to encrypt data for sending, and can also verify the authentication tag when decrypting data. Since the Bash implementations of these algorithms is very slow, the decryption code is designed to only compute and check authentication tags in debug mode.
While the implementation of the code to compute authentication tags for AES-CCM was based on NIST Special Publication 800-38C, I was not able to implement the code for AES-GCM or Poly1305 from their specifications (NIST Special Publication 800-38D and RFC 8439, respectively). So, I would very much like to thank the implementers of https://github.com/mko-x/SharedAES-GCM and https://github.com/floodyberry/poly1305-donna. The implementations of AES-GCM and Poly1305 in the PR were developed by translating the C code in https://github.com/mko-x/SharedAES-GCM and https://github.com/floodyberry/poly1305-donna into Bash. I don't understand what that code is doing, but it seems to work. :-)
I have only tested this code on a computer with a 64-bit operating system. While I have not tested it, I believe that the decryption code will work with 32-bit integers if not in debug mode (i.e., if not trying to compute the authentication tags). I also believe that the AES-CCM code for computing authentication tags will work with 32-bit integers. However, AES-GCM and Poly1305 code for computing authentication tags will definitely only work on systems that have 64-bit integers. So, on systems that do not have 64-bit integers, encryption will not work for AES-GCM or ChaCha20-Poly1305, and decryption will not work for these algorithms if in debug mode.
This PR simplifies the code for determining which draft version of TLS 1.3 a server is offering by making use of a simple regular expression and $BASH_REMATCH rather than looping through every possible draft version.
PR #1463 changed run_ssl_poodle() to only run the test if it is known that the server supports SSLv3. However, support for SSLv3 may be unknown at the time run_ssl_poodle() is run (e.g., if the server supports TLS 1 and SSLv3, and run_ssl_poodle() is the first test performed). So, run_ssl_poodle() should perform testing unless it is known that SSLv3 is not supported.
* Replace "! -z" with "-n"
* Replace "openssl' with "$OPENSSL"
* Redirect stderr output of $OPENSSL to /dev/null to supress "WARNING: can't open config file: /usr/local/etc/ssl/openssl.cnf" message (see #833)
* Remove unnecessary spaces from $GET_REQ11 string.
derive-handshake-traffic-keys() uses the variables `derived_secret`, `server_write_key`, and `server_write_iv`, but they are not declared as local variables of the function. This PR fixes that.
As noted in #1273, there are some environments that will not allow writing to /dev/stdout. PR #1277 was an attempt to address that problem (along with an unrelated problem), but it appears that work on #1277 has been abandoned.
At the moment, "/dev/stdout" is only used as a parameter to asciihex_to_binary_file (in fact, most calls to asciihex_to_binary_file specify "/dev/stdout" as the file parameter). This PR removes the file parameter from asciihex_to_binary_file (and so renames it asciihex_to_binary). In most cases, this just means removing "/dev/stdout" as a parameter to the function. In the few cases in which a parameter other than "/dev/stdout" was provided to asciihex_to_binary_file, this PR just uses a redirect (">" or ">>") to accomplish the same result as providing the output file to asciihex_to_binary_file().
Note that #1273 and #1277 raised the issue of trying to write to /tmp, and this PR does not attempt to address that.
This PR fixes two problems that occur when testing a server that supports TLSv1.3 using OpenSSL 1.1.1 in --ssl-native mode.
First, when testing whether the server has a cipher order, the value of $sclient_success is checked after each call to tls_sockets(), but $sclient_success. As the goal is just to verify that the connection was successful (and didn't downgrade), $? can be checked rather than $sclient_success. [When not in --ssl-native mode, this problem is masked since $sclient_success is set to 0 earlier in the function.]
The second problem is that line 6646 tries to copy "$TEMPDIR/$NODEIP.parse_tls13_serverhello.txt", but this file is currently only created (on line 6287) if tls_sockets() is used to determine the negotiated protocol. This PR fixes the problem by also populating "$TEMPDIR/$NODEIP.parse_tls13_serverhello.txt" when OpenSSL is used to determine the negotiated protocol.
The ciphersuites string for Safari 13.0 ends with a colon (':'). which causes OpenSSL to reject the command line when client simulation testing is performed in --ssl-native mode. This PR fixes the problem by removing the trailing colon.
If the server is known not to support TLS 1.3 (as well as TLS 1.2, TLS 1.1, and TLS 1), then mention TLS 1.3 in the list of not supported protocols. While lack of TLS 1.3 support is not part of the reason that no fallback is possible, it is part of the reason that the result is reported as prln_svrty_high.
If $high_proto is set to something other than SSLv3, support for SSLv3 will not have been determined by determine_optimal_sockets_params(), but it may have been determined later (e.g., by run_protocols()). So, this commit changes the loop to always check for SSLv3 support (without calling "$OPENSSL s_client" if $HAS_SSL3 is false). The check for whether the fallback test can be performed is moved until after the loop
This PR fixes a couple of places where "$OPENSSL s_client" is called with "-ssl3" even if SSLv3 is not supported.
The fix in ciphers_by_strength() is easy, as the issue only occurs if "$using_sockets" is true. If SSLv3 (or TLSv1.3) is not supported, then testing using "$OPENSSL s_client" is skipped and all of the supported ciphers are found using tls_sockets().
The fix for run_tls_fallback_scsv() is more complicated. While it is easy to avoid calling "$OPENSSL s_client" with "-ssl3" if SSLv3 is not supported, it is not easy to determine the correct message to present to the user if support for SSLv3 (and possibly also TLSv1.3) is unknown.
For the case in which $high_proto cannot be set, I believe that I have covered all of the possibilities, but an not sure if the correct message/rating is used in every case.
For the case in which it is not possible to determine whether SSLv3 is the $low_proto, more could be done. If $high_proto is TLS 1.1 or TLS 1, then this PR is okay, as it is possible that SSLv3 would be the fallback protocol, but there is no way to tell. However, it seems unlikely that a server would support TLS 1.2 and SSLv3, but not TLS 1.1 or TLS 1. So, perhaps if $high_proto is TLS 1.2 and the server does not support TLS 1.1 or TLS 1, it should just be assumed that SSLv3 is not supported, even if it cannot be tested.
At the moment, testssl.sh does not correctly derive the handshake traffic keys in the case that the server responds to the initial ClientHello with a HelloRetryRequest. The handshake traffic keys are computed incorrectly because the wrong messages are provided to derive-handshake-traffic-keys() for input to the Transcript-Hash calculation:
* TLS_CLIENT_HELLO is not updated to contain the value of the second ClientHello, and so the first ClientHello is being provided to derive-handshake-traffic-keys() as both the first and the second ClientHello.
* In middlebox compatibility mode the server may send a dummy ChangeCipherSpec message immediately after the HelloRetryRequest. Since it is part of the server's response to the initial ClientHello, the ChangeCipherSpec message is included in the $hrr that is sent to derive-handshake-traffic-keys(), but it should not be included in the computation of the Transcript-Hash.
This PR fixes the above two problems by updating TLS_CLIENT_HELLO when a second ClientHello is sent and by removing any ChangeCipherSpec message (140303000101) from the end of the server's initial response.
When printing a long HTTP security header, this commit causes every row after the first one to be indented by two additional spaces. In the case of very long headers, this extra indentation makes it a little easier for readers to see where the next security header begins.
When printing out HTTP security headers, run_security_headers() uses out_row_aligned_max_width(), since some headers are very long and need to be wrapped. At the moment, however, the first line is too long. The problem is that while "$header $HEADERVALUE" is printed in the indented area, only $HEADERVALUE is passed to out_row_aligned_max_width().
This PR fixes the problem by passing "$header $HEADERVALUE" to out_row_aligned_max_width() so that the the first line is wrapped at the correct place.
Currently sub_cipherlists() and pr_cipher_quality() use different numbers for the same cipher quality ratings. sub_cipherlists() uses:
-2 = pr_svrty_critical, -1= pr_svrty_high, 0 = pr_svrty_low, 1 = pr_svrty_good, 2 = pr_svrty_best
while pr_cipher_quality() uses:
1 = pr_svrty_critical, 2 = pr_svrty_high, 3 = pr_svrty_medium, 4 = pr_svrty_low
5 = neither good nor bad, 6 = pr_svrty_good, 7 = pr_svrty_best
This PR changes sub_cipherlists() (and run_cipherlists()) to use the same numbers for cipher quality as pr_cipher_quality(). It does not change any of the ratings assigned to ciphers by run_cipherlists() or pr_cipher_quality(), so the two are still not in alignment. But, hopefully using the same numbering in both functions will make it a bit easier to compare them and bring them into alignment.
This PR fixes some indentation issues. The PR is a bit long, but it only makes changes to indentation (except for one comment line, where a trailing space character is removed).
This PR fixes#1385.
sub_session_resumption() returns 3 when $CLIENT_AUTH is true. However, the comment at the beginning of the function indicates that 6 will be returned. run_server_defaults() is prepared to handle a return value of 6 (to indicate client auth), but is not expecting 3 as a possible return value.
This PR enhances support for the latest versions of OpenSSL and LibreSSL.
The development version of OpenSSL at https://github.com/openssl/openssl/ is version 3.0.0-dev. So, checks for OpenSSL versions need to support this version as well. At the same time, the latest versions of LibreSSL are 3.0.0, 3.0.1, and 3.0.2, so version number alone will no longer be sufficient to distinguish between OpenSSL and LibreSSL.
In addition to checks for these new version numbers, this PR addresses a couple of other issues:
- In LibreSSL, the "$OPENSSL ciphers" command will not accept any protocol version other than "-tls1" as a parameter (and even including "-tls1" as an option is described as "deprecated"). So, this PR ensures that "$OPENSSL ciphers" is not passed any protocol version option other than "-tls1" is LibreSSL is being used.
- In OpenSSL 3.0.0-dev, the "$OPENSSL dgst" can no longer be used to compute HMACs, but a new "$OPENSSL mac" function has been created. So, this PR changes hmac() to use "$OPENSSL mac" with OpenSSL 3.0.0-dev.
Note that I have not tested the modified version of sub_session_resumption(). I am just assuming that OpenSSL 3.0.0-dev works the same as OpenSSL 1.1.1 and that all versions of LibreSSL work the same as OpenSSL 1.1.0 and earlier.
When tls_sockets() is used for run_protocols(), for each protocol version the results will indicate whether the server responded to a ClientHello for that protocol version with (1) a ServerHello for that same protocol version, (2) a ServerHello for a different (hopefully lower) protocol version, or (3) a handshake failure.
Currently, however, run_prototest_openssl() does not distinguish between cases in which the server responds with a ServerHello for a different (hopefully lower) protocol version and cases in which the server responds with a handshake failure. This PR changes run_prototest_openssl() so that it distinguishes between these two cases (as long as $OPENSSL supports the protocol version specified in the ServerHello).
Making use of the additional information provided by run_prototest_openssl(), this PR also modifies run_protocols() to check that version negotiation was performed correctly even if $using_sockets is false.
Note that one special case needed to be addressed. If an SSLv3-only server is being tested using an $OPENSSL that does not support SSLv3, then $latest_supported must not be set to SSLv3. In the case of a server like this, it is possible that support for SSLv3 will be determined by determine_optimal_sockets_params(), which will cause run_protocols() to report that the server supports SSLv3, even though $OPENSSL does not support SSLv3 and testing is being performed in --ssl-native mode. If $latest_supported were set, then later tests in run_protocols() would incorrectly report a version negotiation failure, even though the failure to connect was a result of a limitation of $OPENSSL rather than a fault of the server.
David Cooper