`std_cipherlists()` does not include line breaks between tests in the output to the terminal when `$DEBUG` is 1, and it does not include line break between tests in the HTML output whenever `$DEBUG` is greater than 0.
In `create_client_simulation_tls_clienthello()` the variable `sni_extension_found` should be set if the ClientHello includes an SNI extension. Instead it was being set if and only if the ClientHello included some extension other than SNI.
This bug wasn't detected before for two reasons:
* It is rare to have a ClientHello that includes an SNI extension, but no other extensions.
* The code still works correctly if `sni_extension_found` is set even if there is no SNI in the ClientHello.
So, the bug only creates a problem if the browser's ClientHello include an SNI extension and no other extensions (see "BingPreview Jun 2014" in the client_simulation branch).
When HTML output is not being created, the print functions last step is to call `html_out()`, which responds to `return` rather than `return 0`. This causes problems for lines of code that rely on receiving a return value of 0. For example:
```
[[ $VULN_COUNT -le $VULN_THRESHLD ]] && outln && pr_headlineln " Testing for LUCKY13 vulnerability " && outln
```
This PR fixes two minor bugs:
* In `run_hpkp()`, the call to `$OPENSSL s_client` includes the option `-showcerts` twice. This PR removes one of them.
* In `get_server_certificate()`, the first call to `$OPENSSL s_client` includes `$addcmd`, but `$addcmd` has not yet been initialized. Instead, `$SNI` should be used.
Currently there is code to extract TLS extensions in three places, in `get_server_certificate()` and two places in `determine_tls_extensions()`. This PR replaces them with one new function, `extract_new_tls_extensions()`.
In order for the new function to work correctly whether OpenSSL or `tls_sockets()` is being used, this PR also changes `parse_tls_serverhello()` so that extensions are formatted in the file it creates in the same way as they are formatted by OpenSSL.
- in gerneral better performance measurements , starts from the real beginning (almost)
- allows results to put into file (MEASURE_TIME_FILE=google.txt testssl.sh google.com)
- PS4 improved: has now a performance debugging options (big solution)
- PS4 with proper alignment
- SCAN_TIME is now global so that it can be used not only by JSON-PRETTY (small performance debugging options uses it)
- prepare_debug() has now debugging stuff only, rest went to prepare_arrays()
This PR is the same as #661, except that it applies to the 2.9dev_html branch. It also fixes a bug in `prettyprint_local()` when a pattern to match is provided as input.
While doing some performance testing I discovered that `normalize_ciphercode()` is very slow. This PR simplifies the function and speeds it up significantly. This PR also addresses the TODO item in `normalize_ciphercode()` by eliminating the global variable HEXC.
When I view testssl.sh in KDE's text editor (kate), the code after "INFO" line in `show_finding()` isn't highlighted correctly, and the highlighting doesn't correct again until `pr_liteblueln()`. This PR fixes the highlighting issue by adding spaces between "]]" and ")". To be "safe," I also added spaces between "(" and "[[", and similarly added spaces in `is_json_format()`.
This PR ensures that each HTML file produced by testssl.sh only includes a single header, at the top, and a single footer, at the end. It also tries to ensure that the short-version banner is only placed at the top of the HTML file if (1) mass testing is being performed and (2) the results of each test is being placed in a separate file.
It also moves some of the logic out of main and into `html_header()`.
So far I haven't seen any HTML reserved characters (&, <, >, ", ') in the strings processed by `emphasize_stuff_in_headers()`, so this PR may be unnecessary. However, this PR will ensure that any such characters will be properly escaped in the HTML output.
"=~" doesn't need quotes if there's a text string one wants to match against (and shellcheck complains
about this)
pr_magenta shouldn't be used anymore as the logic what color we use should be done
some place else.
This branch is for getting the HTML patch from @dcooper16 into 2.9dev
Change to David's PR:
* removed HTMLHEADER. We always want that (in fact for flat JSON this is missing and needs to be added)
* not sure what this change does to --file
* changing of names They were redundant sometimes (pr_*_term )
* some formatting for readbility
Open points:
* there's a loop and a segfault --> tm_done_best
* HTMLHEADER: --file
* the former sed statement aroung L1900 for the header was way more readable. The combined
html+terminal version is just too much. Maybe a switch whether HTML is requested
is better so that this can be separated.
* Then e.g. "<span style=\"color:olive;font-weight:bold" can be kept in a variable
* any reason we need the text length here?
* what went into main here is too much. Actuallly what I put already in there bothered
me as too much logic and not obvious dependencies are in here. Now it's worse :-)
Can't this be just similar to JSON or CSV -- a seperate function with hooks
not in main()?
* minor thing: TERM_WIDTH is for HTML is maybe not the best. But that can be
tackled later
This PR attempts to address #631. It allows four choices for the `--mapping` option. "openssl" (default), "rfc", "no-openssl", and "no-rfc".
* "openssl" is the current default
* "no-rfc" shows only the OpenSSL names (just as it currently does)
* "rfc" shows the RFC name rather than the OpenSSL name for things that are not in wide now (just as it currently does). But now, in wide mode, it shows the RFC name first (further to the left) and the OpenSSL name second.
* "rfc-only" shows only the RFC name.
If the `--mapping` option is not provided, this is the same as "openssl". If the "cipher-mapping.txt" file cannot be found, then testssl.sh runs as if "no-rfc" had been requested.
It seems that the head command on OS X does not accept a negative number as a value for the "-n" parameter. This PR provides an alternative method for removing the "generator: " line without using "head."
Some sites have a long list of IP addresses and some IP addresses have a long list of DNS names that map to them.
This PR changes `display_rdns_etc()` to use `out_row_aligned_max_width()` to print the other IP addresses in `$IP46ADDRs` and to print `$rDNS`.
RFC 7633 introduces the TLS Features certificate extension, which contains "Features:
> The object member "Features" is a sequence of TLS extension identifiers (features, in this specification's terminology) as specified in the IANA Transport Layer Security (TLS) Extensions registry. If these features are requested by the client in its ClientHello message, then the server MUST return a ServerHello message that satisfies this request.
The main purpose of this certificate extension is to implement "must staple." If the extension is present in a TLS server's certificate and it includes status_request, then the server MUST include a stapled OCSP response if the client requests one. (The same applies for the status_request_v2 extension.)
This PR adds a check to `certificate_info()` of whether the server supports must staple (i.e., whether its certificate includes a TLS Features extension with "status_request"). It also changes the output for "OCSP stapling" in the case that the server did not staple an OCSP response. It indicates that:
* it is a critical issue if the certificate specifies "must staple"
* it is a low severity issue if the certificate does not specify "must staple," but the certificate does include an OCSP URI.
* it is not an issue at all if the certificate does not specify "must staple" and certificate does not include an OCSP URI.
`run_server_preference()` prints out the server's Negotiated cipher in a different color depending on the quality of the cipher. However, there is a "FIXME" since CBC ciphers are supposed to be flagged, but it is not easy to identity all CBC ciphers from their OpenSSL names.
This PR partially addresses this. It creates a separate function for printing a cipher based on its quality. Whenever possible it determines the quality of the cipher based on the RFC name. However, if it is provided an OpenSSL name and no cipher-mapping.txt file is available, it will follow the current (imperfect) logic for determining the cipher's quality.
The function also returns a value that indicates the quality of the cipher provided, with higher numbers indicating better ciphers. This return value is used by `run_server_preference()` to determine how to populate the "severity" field when calling `fileout()`.
In the case that `tls_sockets()` is being used and the server incorrectly fails the connection rather than downgrading, testssl.sh is printing "not offered" on one line and then the error message on the next line, but all the text should appear on one line (as it does when testing TLS 1 and TLS 1.1).
Since the test for TLS 1.2 in `run_protocols()` now uses `tls_sockets()` whenever `$ssl_native` is `true` (i.e., there is no longer a requirement for `$EXPERIMENTAL` to be true as well), the `$EXPERIMENTAL` flag should no longer be checked if the return value is 1.
Rearrange code so that in the case of just a single test, `parse_hn_port()` is not called earlier than it was previously unless it needs to be called in order to create the HTML file name.
Doing this ensures that the banner is displayed even if the `$URI` cannot be parsed (except in the case that the `$URI` needs to be parsed in order to create a file name) and that any error messages created by `parse_hn_port()` will be included in the HTML, if possible.
Add option for testssl.sh to create the HTML file name. If testssl.sh creates the file name, then, in the case of mass testing, a separate HTML file is created for each test (i.e., for each line in the file provided to `--file`).
I have a test server that I configured to support only SSLv3 and TLSv1.2. When I set `SSLHonorCipherOrder` to `off` I get the following results:
```
ECDHE-RSA-AES256-SHA: SSLv3 ECDHE-RSA-AES256-GCM-SHA384: TLSv1.2
```
The current code, when printing TLSv1.2 checks whether `${cipher[4]}` is empty, and since it is assume no previous protocol (SSLv2, SSLv3, TLSv1, TLSv1.1) was supported and so doesn't output a newline before outputting the cipher and protocol for TLSv1.2.
This PR fixes that by changing to code to look at the previous non-empty cipher (if there is one), even if that does not come from the previous protocol.
When a list of cipher suites is being displayed using `neat_list()`, testssl.sh shows the cipher suite's OpenSSL name and (in most cases) the RFC name as well. However, in all other cases only the OpenSSL name is shown.
This PR adds the option to have cipher suite's RFC names shown instead of the OpenSSL name, by including `--mapping rfc` in the command line. [Note: if the cipher-mapping.txt file cannot be found, then the `--mapping rfc` option is ignored and the OpenSSL names are shown.]
This PR seems to be related to issue #9, but #9 may be been referring to the output created by `neat_list()`.
Gray should appear for COLOR=1 or COLOR=2.
Since `pr_grey()` is basically the same as `out()` for COLOR=0, `mybanner()` should just call `pr_grey()` without checking the value of `$COLOR`.
Change `emphasize_stuff_in_headers()` to use olive and bold olive rather than brown and yellow. This matches what `aha` creates and appears similar to what is displayed in the terminal on a Mac. Also, yellow text is very difficult to read.
Found more places where output should only go to terminal, or where it was only going to the terminal (e.g., printf) but should also be in the HTML. Also added the ability to include active URLs in the HTML output.
To Do: Handle automatic generation of HTML file name and support for parallel testing.
Introduced "trick" so that if the `--file` option is used, `html_header()` will only be called once before anything is printed and `html_footer()` will only be called once after all printing is complete. With this, `html_header()` now delete the output file if it exists.
Also introduced the `html_reserved()`, which is called for all text to be sent to `out_html()`. `html_reserved()` converts any HTML reserved characters (", ', &, <, >) to their corresponding entity names (", ', &, <, >).
This PR addresses the issue raised in #623. This PR is based on the function `out_row_aligned_max_width()` that I proposed in #623, but the `out_row_aligned_max_width()` in this PR is a little different. It takes a fourth parameter, which is the function to use to print each word in the text string to be printed. This is used in `run_pfs()` so that the "Elliptic curves offered" can be printed using this function (some servers support 25 curves), while still having the curves printed using color-coding to indicate the quality of each curve.
I somewhat arbitrarily have each line wrap at 120 characters, but that could be changed (e.g., to `$TERM_WIDTH`).
For the most part I used the RGB values for xterm from https://en.wikipedia.org/wiki/ANSI_escape_code#Colors for the HTML colors, but with a few exceptions. For example, I did not use "yellow" for `pr_svrty_low()`, since that color is very difficult to read. I also used a different color for `pr_svrty_medium()` so that `pr_svrty_medium()` would appear more red than `pr_svrty_low()`.
These color choices could use more adjustment.
This PR adds the option to generate HTML. The code was created as follows:
* For each output function (`out()`, `outln()`, `pr_liteblue()`, etc.) I created two functions: one that just outputs to the terminal and one that outputs to the terminal and to the HTML file (if an HTML file is to be created).
* I modified the code so that any output that should appear in the HTML file in addition to being displayed on the terminal is sent through one of the display functions: out()`, `outln()`, `pr_liteblue()`, etc.
* I created a new function `retstring()` to use in place of `out()` when a function is creating a string to be "captured" by the calling function.
* I modified the code so that no string returned by a function includes color-coding escape characters.
In the revised code that was created to address #587, nothing is printed after "OCSP URI" if there is neither an OCSP URI nor a CRL URI. Instead, "--" should be printed.
I still believe that there is an inconsistency in the reporting of the output, however. At the moment, the "Certificate Revocation List" and "OCSP URI" lines indicate it is acceptable as long as the certificate contains either a CRL URI or an OCSP URI. However, the "OCSP stapling" line reports a minor finding if an OCSP response was not included in the server's reply. Shouldn't we just assume that if the certificate doesn't include an OCSP URI, then it wouldn't be possible for the server to obtain an OCSP response to staple to its reply? If so, then it seems that no OCSP stapling should only be considered a finding if an OCSP URI is present.
This PR fixes issue #601.
The fix for OCSP URLs was easy. I don't entirely understand the first `awk` command in the line to extract the CRL URLs, but I tested it on several certificates and it seems to work correctly (ensuring that the only "URI" lines in the input to the second `awk` command are from the CRL Distribution Points extension).
RFC 4492 introduced the Supported Elliptic Curves Extension, but this extension was renamed Supported Groups in RFC 7919. Following RFC 7919 (and TLSv1.3), `parse_tls_serverhello()` refers to this extension as "supported groups/#10". Since, at the moment, OpenSSL's s_client refers to this extension as "elliptic curves/#10", the extension sometimes appears twice in the "TLS extensions" line, if it is detected by both OpenSSL (in `get_server_certificate()`) and `tls_sockets()` (in `determine_tls_extensions()`):
```
TLS extensions (standard) "renegotiation info/#65281" "elliptic curves/#10" "EC point formats/#11" "supported groups/#10"
```
This PR fixes the problem of the extension appearing twice in the "TLS extensions" line by replacing any instances of "elliptic curves/#10" with "supported_groups/#10" in the `$tls_extensions` line extracted from `$OPENSSL s_client`. This PR also changes "supported groups/#10" to "supported_groups/#10" in `parse_tls_serverhello()`, since the current development branch of OpenSSL uses "supported_groups" to refer to this extension (see https://github.com/openssl/openssl/pull/1825).
This PR increases the width of the "Encryption" column printed by `neat_list()` in order to allow room to print "CamelliaGCM."
This is the alternative fix to the problem that was first addressed in #524. This PR obsoletes PR #530.
This PR changes testssl.sh so that when ciphers are being listed in wide mode (i.e., using `neat_list()`) and the `--show-each` option is set, ciphers that are not available are printed in light grey, whereas ciphers that are available continue to be printed in black. This makes it easier to distinguish between ciphers that are available and those that are not (the "available/"not a/v" column remains).
This PR does not change the way that ciphers that are available are printed, but it includes a hook that would allow that to change. For example, for ciphers that are available, the name of the cipher suite could be printed in a different color depending on its quality (as is done for the "Negotiated cipher" in `run_server_preference()`). The same could be done for the "Encryption" and "Bits" columns.
- run_logjam(): determine dh bit size and based on this mark the common primes as more or less vulnerable
- run_logjam(): renamed remaining dhe variable to dh
- further house keeping in run_logjam()
In some cases, the "TLS extensions" line output for the "--server-defaults" option will not show `"encrypt-then-mac/#22"` even if the server supports this extension. The reason is that a server will only include this extension in the ServerHello message if it supports the extension and the selected cipher is a CBC cipher. So, if `determine_tls_extensions()` connects to the server with a non-CBC cipher, then it will not detect if the server supports the encrypt-then-mac extension.
It is possible that support for the extension will be detected by `get_server_certificate()`, but only if one of the calls to that function results in a CBC cipher being selected and OpenSSL 1.1.0 is being used (as prior versions did not support the encrypt-then-mac extension).
In this PR, if `determine_tls_extensions()` is called and `$TLS_EXTENSIONS` does not already contain `"encrypt-then-mac/#22"`, then an attempt will be made to connect to the server with only CBC ciphers specified in the ClientHello. If the connection is not successful (presumably because the server does not support any CBC ciphers), then a second connection attempt will be made with the "default" ciphers being specified in the ClientHello.
en.wikipedia.org is an example of a server that supports the encrypt-then-mac extension, but for which the support is not currently detected (unless OpenSSL 1.1.0 is used) since in the call to `determine_tls_extension()` a non-CBC cipher is selected.
This PR changes `read_dhbits_from_file()` so that, when the "quiet" parameter is absent, the selected curve is shown in addition to the number of bits. This PR only affects the output of `run_client_simulation()` and the `Negotiated cipher` in `run_server_preference()`.
