* Changed calls to testssl.sh to not include `--quiet` or `--append` flags. Modified perl script to remove HTML header and footer before comparing to terminal output.
* Changed `TERM_WIDTH` to 120 (doesn't affect test, but 80 created too much line wrapping).
* Replace date and time information with X's rather than removing entirely. This should not affect the comparison, but will make the output created displayed in an error message look closer to the actual output of testssl.sh
I've never programmed in perl before, but this script seems to work. It includes two checks:
* I runs testssl.sh without the `--debug` flags and checks that the HTML file is the same as what is sent to the terminal.
* It runs testssl.sh with `--debug 4` and checks that the HTML file created is the same as the one created without the `--debug` flag.
`emphasize_stuff_in_headers()` only adds color to the text being printed to the terminal if `$COLOR` is 2. So, the same should be the case for the HTML output.
This PR addresses issue #616, changing `run_cipher_match()` so that only those ciphers that are available are shown, unless the `--show-each` flag has been provided.
It also fixes a problem where the signature algorithm isn't being shown, even if `$SHOW_SIGALGO` is true.
This PR just addresses some places where quotes need to be used to avoid word splitting in case the referenced file, or path to the file, contains space characters.
The previous fix did not work if testssl.sh was found via `$PATH`. This seems to work in all cases. If testssl.sh is found via `$PATH` or if the command line includes a path, then `which` returns a non-empty response; otherwise, `$0` does not include any path, but one needs to be provided, so `$RUN_DIR/$PROG_NAME` is used.
Using "$0" as the name of the executable seems to work as long as "$0" contains a directory name (e.g, "workingfiles/testssl.sh"), but not if it is just the name of the executable (e.g., "testssl.sh"). Specifying "$RUN_DIR/$PROG_NAME" seems to work in both cases, since if "$0" doesn't contain any path information, `$RUN_DIR` is `.`
Use the suggestion "If you want to print the argument list as close as possible to what the user probably entered" from http://stackoverflow.com/questions/10835933/preserve-quotes-in-bash-arguments to create `$CMDLINE` and to print the command lines in `run_mass_testing()` and `run_mass_testing_parallel()`.
This PR addresses issue #702. Rather than create the command line for each child process in `run_mass_testing()` as a string, it creates it as an array, with each argument being a separate element in the array. This was done based on http://mywiki.wooledge.org/BashFAQ/050.
The printing of each child's command line done based on http://stackoverflow.com/questions/10835933/preserve-quotes-in-bash-arguments.
The `$CMDLINE` string remains unchanged, even though it isn't entirely "correct," since http://jsonlint.com/ complains if the "Invocation:" string contains backslashes.
I was doing some testing on my extended_tls_sockets branch and discovered that it was not fully working since the `TLS13_KEY_SHARES` array was empty. According to https://lists.gnu.org/archive/html/bug-bash/2012-06/msg00068.html, there is an issue when trying to initialize a global array inside a function. (The current code initializes `TLS12_CIPHER`, `TLS_CIPHER`, and `TLS13_KEY_SHARES` within `get_install_dir()`, since tls_data.txt is read in that function.) In fact, according to http://stackoverflow.com/questions/10806357/associative-arrays-are-local-by-default, in order to initialize a global variable in a function, one needs to provide the `-g` option, which was only added in Bash 4.2.
This PR seems to fix the problem by moving the reading of tls_data.txt to the main body of the code rather than reading it within the `get_install_dir()` function.
If I understand correctly how `run_hpkp()` should be displaying the list of Backups, the problem shown in the photo attached to #696 occurred because the dangling SPKIs should have been printed on the next line. This PR fixes this by changing the code that prints out the "good" backups to include a newline after printing the CA's name.
Another problem this PR fixes is that `tm_italic()` is being called instead of `pr_italic()`, meaning that the italicized text is being printed to the terminal, but is not being included in the HTML output.
Finally, this PR fixes errors in the definitions of `tmln_italic()` and `prln_italic()`. `tmln_italic()` calls `outln()` and `prln_italic()` calls `tmln_out()` instead of the reverse.
This PR fixes issue #695 by changing the call to `out_row_aligned_max_width()` so that the length of the flag is considered in determining the length of the first line of the output.
This PR fixes two issues with HTML generation that were introduced by a commit on March 31, 2016, "[count_ciphers is now un-sed'ed, minor improvements](https://github.com/drwetter/testssl.sh/commit/a480e5f699983207651aa0a8717dc395d13e6e52)."
The first is that in `std_cipherlists()`, `[[ $DEBUG -ge 1 ]] && outln " -- $1" || outln` was changed to `[[ $DEBUG -ge 1 ]] && outln " -- $1" || outln`. The result being that in the HTML output, all of the tests from `run_std_cipherlists()` appear on the same line. This PR changes the line to:
```
[[ $DEBUG -ge 1 ]] && tm_out " -- $1"
outln
``
so that the line break is added to the HTML output, but the debugging information is not.
The second problem is that the commit on March 31 moved the call in main to `html_header()` until after the calls to `get_install_dir()`, `find_openssl_binary()`, `mybanner()`, `check4openssl_oldfarts()`, and `check_bsd_mount()`. The problem is that each of these functions may call an output function that will call `html_out()`.
If `html_out()` is called before `html_header()` and the command line contains `--htmlfile <htmlfile>`, then "htmlfile" will be written to before `html_header()` is called and then `html_header()` will warn that "htmlfile" already exists and then exit the program.
If `html_out()` is called before `html_header()` and the command line contains `--html`, then anything send to `html_out()` before `html_header()` is called (such as the banner) will not appear in the HTML file.
I did some testing with http://jsonlint.com/ and discovered a missing comma when massing testing is being performed and a single JSON file is being created.
In `run_server_defaults()` the variable `success` is defined twice, once an an ordinary variable and once as an array. The PR removes the incorrect definition. It also removes the definitions of some variables that are no longer used and reorganizes the definitions so that each line has only one variable type.
I also noticed a typo later in `run_server_defaults()` and corrected it.
This PR introduces the environment variable `CHILD_MASS_TESTING`, and uses it as an indicator that testssl.sh is running as a child within mass testing rather than using the `$APPEND` flag. It also makes a number of other changes to make the handling, of HTML, CSV, JSON, and log files consistent, and it fixes a number of bugs related to the generation of these files when mass testing is being performed.
Please let me know if you disagree with any of the changes in this PR, or if you would prefer that it be broken up into multiple smaller PRs.
Some of the changes are as follows:
- When the `$APPEND` flag is true, all of these files are appended to and headers and footers are omitted. (Perhaps this should be changed. Appending to a log file isn't an issue, but appending to a JSON or HTML file without including headers or footers seems to just create an improperly formatted file).
- Following the code in `prepare_logging()`, an error is printed and the program stops if the `$APPEND` flag is false and one of the files to be written to already exists.
Some of the bugs fixed:
Creating log files did not work with mass testing:
- If `--logfile <logfile>` is used, then the parent and each child try to write to "logfile".
- If `--logging` is used, then a log file is created for each child, but an oddly-named log file is also created for the parent. The one created by the parent contains the entire output.
Plain JSON files:
- When `--jsonfile <jsonfile>` is run, there is no comma separating the final finding for one child and the first finding for the next child.
Pretty JSON files:
- When `--jsonfile-pretty <jsonfile>` is called without mass testing, the "target host" line is empty, since `$NODE` has not yet been set.
- When `--jsonfile <jsonfile>` is run with mass testing, there is no comma separating the final finding for one child and the first finding for the next child. In addition, `fileout_pretty_json_banner()` is never called, and the entries for individual tests have insufficient information to determine what is being tested (it lists "service" and "ip", but not port number).
For the final issue, when mass testing is being performed and all output is being placed in a single file, I have the parent call `fileout_pretty_json_banner()`, but tell `fileout_pretty_json_banner()` to not include a "target host" or "port", but then have each child include a "target host" or "port" (when the "service" and "ip" are being printed).
This PR improves `out_row_aligned_max_width()` in a few ways:
* It makes better use of bash's string manipulation capabilities in order to simplify the function.
* It improves the function's performance. One of the most costly parts of `out_row_aligned_max_width()` was the while loop to print each entry in the text. Since there is only one place in the code where the the entries are not all printed the same ways (the list of supported curves printed by `run_pfs()`), the PR changes `out_row_aligned_max_width()` to just return a plain text string, which the calling function prints in the appropriate way. For the curves printed by `run_pfs()`, a new function, `out_row_aligned_max_width_by_entry()` takes care of getting the output from `out_row_aligned_max_width()` and then printing each entry appropriately.
* The PR also introduces a trick so that when the TLS extensions are printed, the text for an extension won't get split across two rows. It does this by replacing the space charters within the text for an extension with "}", formatting the result with `out_row_aligned_max_width()`, and then converting the "}" back to space characters.
`$OPENSSL_LOCATION` is used in `fileout_pretty_json_banner()`, `html_banner()`, `mybanner()`, and `prepare_logging()`, but the value of `$OPENSSL_LOCATION` is populated in `mybanner()`. This is usually okay, since `mybanner()` is always called before the other three functions are called. However, if `$QUIET` is `true`, then `mybanner()` returns immediately, without populating `$OPENSSL_LOCATION`, even though the value of `$OPENSSL_LOCATION` may be needed by one or more of the other functions.
This PR addresses this problem by populating `$OPENSSL_LOCATION` in `find_openssl_binary()` rather than `mybanner()`.
With the commit made on March 26, "partly (1/2) fixing #653," an HTML banner isn't added to the HTML files anymore. A banner should be added to the top of the HTML file if mass testing is being performed and a separate HTML file is being created for each test.
The `$APPEND` flag being `true` is an indicator that mass testing is being performed and that this is one of the individual tests being run. Given that `$APPEND` is `true`, `$HTMLHEADER` being `true` indicates that testssl.sh is creating the file name for the HTML output. So, it is when both flags are `true` that the HTML banner should be created.
`std_cipherlists()` does not include line breaks between tests in the output to the terminal when `$DEBUG` is 1, and it does not include line break between tests in the HTML output whenever `$DEBUG` is greater than 0.
In `create_client_simulation_tls_clienthello()` the variable `sni_extension_found` should be set if the ClientHello includes an SNI extension. Instead it was being set if and only if the ClientHello included some extension other than SNI.
This bug wasn't detected before for two reasons:
* It is rare to have a ClientHello that includes an SNI extension, but no other extensions.
* The code still works correctly if `sni_extension_found` is set even if there is no SNI in the ClientHello.
So, the bug only creates a problem if the browser's ClientHello include an SNI extension and no other extensions (see "BingPreview Jun 2014" in the client_simulation branch).
When HTML output is not being created, the print functions last step is to call `html_out()`, which responds to `return` rather than `return 0`. This causes problems for lines of code that rely on receiving a return value of 0. For example:
```
[[ $VULN_COUNT -le $VULN_THRESHLD ]] && outln && pr_headlineln " Testing for LUCKY13 vulnerability " && outln
```
This PR fixes two minor bugs:
* In `run_hpkp()`, the call to `$OPENSSL s_client` includes the option `-showcerts` twice. This PR removes one of them.
* In `get_server_certificate()`, the first call to `$OPENSSL s_client` includes `$addcmd`, but `$addcmd` has not yet been initialized. Instead, `$SNI` should be used.
Currently there is code to extract TLS extensions in three places, in `get_server_certificate()` and two places in `determine_tls_extensions()`. This PR replaces them with one new function, `extract_new_tls_extensions()`.
In order for the new function to work correctly whether OpenSSL or `tls_sockets()` is being used, this PR also changes `parse_tls_serverhello()` so that extensions are formatted in the file it creates in the same way as they are formatted by OpenSSL.
This PR is the same as #661, except that it applies to the 2.9dev_html branch. It also fixes a bug in `prettyprint_local()` when a pattern to match is provided as input.
While doing some performance testing I discovered that `normalize_ciphercode()` is very slow. This PR simplifies the function and speeds it up significantly. This PR also addresses the TODO item in `normalize_ciphercode()` by eliminating the global variable HEXC.
When I view testssl.sh in KDE's text editor (kate), the code after "INFO" line in `show_finding()` isn't highlighted correctly, and the highlighting doesn't correct again until `pr_liteblueln()`. This PR fixes the highlighting issue by adding spaces between "]]" and ")". To be "safe," I also added spaces between "(" and "[[", and similarly added spaces in `is_json_format()`.
This PR ensures that each HTML file produced by testssl.sh only includes a single header, at the top, and a single footer, at the end. It also tries to ensure that the short-version banner is only placed at the top of the HTML file if (1) mass testing is being performed and (2) the results of each test is being placed in a separate file.
It also moves some of the logic out of main and into `html_header()`.
So far I haven't seen any HTML reserved characters (&, <, >, ", ') in the strings processed by `emphasize_stuff_in_headers()`, so this PR may be unnecessary. However, this PR will ensure that any such characters will be properly escaped in the HTML output.
This PR attempts to address #631. It allows four choices for the `--mapping` option. "openssl" (default), "rfc", "no-openssl", and "no-rfc".
* "openssl" is the current default
* "no-rfc" shows only the OpenSSL names (just as it currently does)
* "rfc" shows the RFC name rather than the OpenSSL name for things that are not in wide now (just as it currently does). But now, in wide mode, it shows the RFC name first (further to the left) and the OpenSSL name second.
* "rfc-only" shows only the RFC name.
If the `--mapping` option is not provided, this is the same as "openssl". If the "cipher-mapping.txt" file cannot be found, then testssl.sh runs as if "no-rfc" had been requested.
It seems that the head command on OS X does not accept a negative number as a value for the "-n" parameter. This PR provides an alternative method for removing the "generator: " line without using "head."
Some sites have a long list of IP addresses and some IP addresses have a long list of DNS names that map to them.
This PR changes `display_rdns_etc()` to use `out_row_aligned_max_width()` to print the other IP addresses in `$IP46ADDRs` and to print `$rDNS`.
RFC 7633 introduces the TLS Features certificate extension, which contains "Features:
> The object member "Features" is a sequence of TLS extension identifiers (features, in this specification's terminology) as specified in the IANA Transport Layer Security (TLS) Extensions registry. If these features are requested by the client in its ClientHello message, then the server MUST return a ServerHello message that satisfies this request.
The main purpose of this certificate extension is to implement "must staple." If the extension is present in a TLS server's certificate and it includes status_request, then the server MUST include a stapled OCSP response if the client requests one. (The same applies for the status_request_v2 extension.)
This PR adds a check to `certificate_info()` of whether the server supports must staple (i.e., whether its certificate includes a TLS Features extension with "status_request"). It also changes the output for "OCSP stapling" in the case that the server did not staple an OCSP response. It indicates that:
* it is a critical issue if the certificate specifies "must staple"
* it is a low severity issue if the certificate does not specify "must staple," but the certificate does include an OCSP URI.
* it is not an issue at all if the certificate does not specify "must staple" and certificate does not include an OCSP URI.
`run_server_preference()` prints out the server's Negotiated cipher in a different color depending on the quality of the cipher. However, there is a "FIXME" since CBC ciphers are supposed to be flagged, but it is not easy to identity all CBC ciphers from their OpenSSL names.
This PR partially addresses this. It creates a separate function for printing a cipher based on its quality. Whenever possible it determines the quality of the cipher based on the RFC name. However, if it is provided an OpenSSL name and no cipher-mapping.txt file is available, it will follow the current (imperfect) logic for determining the cipher's quality.
The function also returns a value that indicates the quality of the cipher provided, with higher numbers indicating better ciphers. This return value is used by `run_server_preference()` to determine how to populate the "severity" field when calling `fileout()`.
In the case that `tls_sockets()` is being used and the server incorrectly fails the connection rather than downgrading, testssl.sh is printing "not offered" on one line and then the error message on the next line, but all the text should appear on one line (as it does when testing TLS 1 and TLS 1.1).
David Cooper