Currently, when `$DEBUG` is 2 and the connection is successful, `parse_tls_serverhello()` prints out information about the server's ephermal (EC)DH key, but nothing else. For example:
```
sending client hello... reading server hello...
dh_bits: ECDH, P-256, 256 bits
sending close_notify...
(183 lines returned)
```
This commit changes `parse_tls_serverhello()` so that information about dh_bits is only displayed if `$DEBUG` is at least 3, making it the same as for other information about the server's response.
In addition, it indents the printing of the information about dh_bits in order to better align with other information displayed at this debug level.
- add forgotten servive FTP and XMPP
- polish other services
- after TLS 1.2 run is finished run a check whether no protocol has been detected and ask the user for confirmation to proceed
In `run_protocols()` for TLS 1.2, try one set of 127 ciphers and if the result isn't a connection at TLSv1.2 then try another set of 127 ciphers before giving up and assuming that TLS 1.2 isn't supported.
FIX#786
Fixed all other occurences so that debug level 2 is showing only minimal information like rough status and errors
Better line breaks for level 2
In ``client_simulation_sockets()`` and ``tls_sockets()`` moved debug output into if statements (may save a bit of time)
Replaced "$DEBUG -eq" by "$DEBUG -ge"
Removed obsolete hb_rounds in ``run_heartbleed()``
Adjusted wide output in vulnerabilities
The data for `run_client_simulation()` currently includes two clients that send version 2.0 CLIENT-HELLO messages (see Appendix E.2 of RFC 5246). Each of the CLIENT-HELLO messages advertises support for newer protocol versions (SSLv3 in the case of IE6XP and TLSv1.0 in the case of Java 6u45). A server may reject one of these version 2.0 CLIENT-HELLO messages, or it may respond with an SSLv2, SSLv3, or TLSv1.0 ServerHello.
The current code in `client_simulation_sockets()` assumes that the server's response with be an SSLv3 or later ServerHello. So, it can support cases in which servers respond with an SSLv3 or TLSv1.0 ServerHello (once PR #800 is accepted to undo the mistake in PR #797), but not cases in which servers response with an SSLv2 ServerHello.
This PR adds code to `client_simulation_sockets()` to check if the server's response is an SSLv2 ServerHello, so that it can process such responses with `parse_sslv2_serverhello()` rather than `parse_tls_serverhello()`.
When a connection is made using SSLv3 or later, `run_client_simulation()` will show to the protocol and cipher selected for the connection. With this PR, if the connection is made using SSLv2, `run_client_simulation()` will just show "SSLv2." In the case of SSLv2, the ServerHello contains a list of all ciphers that the server and client have in common, and it is up to the client to choose one. So, if the client and server have more than one cipher in common, more information about the client would be needed to know which cipher it would choose.
`std_cipherlists()` uses `has_server_protocol()` to determine whether to test for ciphers using SSLv2. However, this was resulting in false negatives due to #759. This PR removes the `has_server_protocol()` check so that SSLv2 is checked whenever the connection attempt with the TLSv1.2 ClientHello failed.
This PR fixes the extraction of the cipher suites in SSLv2 ClientHellos in `client_simulation_sockets()`. Since `client_simulation_sockets()` can only handle SSLv3 and above ServerHellos, it removes any SSLv2 ciphers and converts the other cipher from 3-byte format to 2-byte format.
If vulnernable to POODLE and has no TLS_FALLBACK_SCSV ==> HIGH.
If only run the fallback check and it has none, still label it as MEDIUM but issue
a clear warning that test under this circumstances is incomplete.
This PR adds the same additional check to `client_simulation_sockets()` as was previously added to `tls_sockets()`. It extracts the list of cipher suites offered from each ClientHello and passes the list to `parse_tls_serverhello()` so that `parse_tls_serverhello()` can check that the cipher offered in the ServerHello was included in the ClientHello.
This assumes that a real client would abort the connection if it was presented with a cipher in the ServerHello that it didn't offer in its ClientHello.
Unfortunately, most categories were passing incorrectly, since a
tls_socket() error was interpreted as a success for many of the cases.
Now we explicitely check for the known yaSSL failure and raise a warning
that the test is inconclusive.
New output against --starttls=mysql looks like:
Testing ~standard cipher categories
-----------------------------------
NULL ciphers (no encryption) SERVER_ERROR: test inconclusive due to MySQL Community Edition (yaSSL) bug.
Anonymous NULL Ciphers (no authentication) SERVER_ERROR: test inconclusive due to MySQL Community Edition (yaSSL) bug.
Export ciphers (w/o ADH+NULL) SERVER_ERROR: test inconclusive due to MySQL Community Edition (yaSSL) bug.
LOW: 64 Bit + DES encryption (w/o export) offered (NOT ok)
Weak 128 Bit ciphers (SEED, IDEA, RC[2,4]) SERVER_ERROR: test inconclusive due to MySQL Community Edition (yaSSL) bug.
Triple DES Ciphers (Medium) offered
High encryption (AES+Camellia, no AEAD) offered (OK)
Strong encryption (AEAD ciphers) SERVER_ERROR: test inconclusive due to MySQL Community Edition (yaSSL) bug.
Partial workaround for #793
According to a discussion thread on the IETF TLS WG mail list (see https://www.ietf.org/mail-archive/web/tls/current/msg19720.html), there is at least one TLS server that will fail if the last extension in the ClientHello has contains extension_data of length 0.
Currently, `tls_sockets()` will create such a ClientHello if:
* The padding extension is included, and the length of the ClientHello without the padding data would be between 508 and 511 bytes.
* No padding extension is included, and the caller provided `$extra_extensions` in which the last extension in `$extra_extensions` is empty.
* No padding extension is included, `$extra_extensions` is empty, no ECC cipher suites are offered, and the ClientHello is for TLSv1.1 or below (in this case the next protocol extension would be that last one).
This PR avoids the server bug (in nearly all cases) by ensuring the the padding extension (when present) always contains at least one byte, and by ensuring that when the padding extension is not present that the (non-empty) heartbeat extension is the last extension.
This PR does leave one possible scenario in which the last extension would be empty. If the caller provides an `$extra_extensions` in which the last extension in `$extra_extensions` is empty, `tls_sockets()` does not add a padding extension (or a padding extension is included in `$extra_extensions`), and `$extra_extensions` includes a heartbeat extension, then the last extension in the ClientHello would be empty. This, however, is a highly unlikely scenario, and certainly there are currently no such calls to `tls_sockets()` in testssl.sh.
Another yaSSL server incompatiblity. MySQL protects against CCS
Injection by erroring if it sees to CCS requests before the handshake
completes. But instead of returning a TLS alert, it seems to error up
the stack to MySQL which returns it's own error message.
Debug output looks like:
---
sending client hello,
reading server hello
1st reply:
sending payload #2 with TLS version x03, x02:
tls_content_type: 16 | tls_protocol: 0000 | byte6: 04
2nd reply:
00000000 16 00 00 02 ff 13 04 23 30 38 53 30 31 42 61 64 |.......#08S01Bad|
00000010 20 68 61 6e 64 73 68 61 6b 65 | handshake|
0000001a
test failed, probably read buffer too small (16000002FF1304)
---
This patch adds a custom check for this MySQL specific error, as far
down in the error check path as I can.
If $PROTOS_OFFERED was empty, and thus the protocols unknown, this
function would return true for any protocol you passed it. This caused
most callers to assume TLS1.0, even if the server didn't offer it.
Instead return false and make the caller do an extra lookup.
As reported in #782, some servers will return a ServerHello with a cipher not listed in the ClientHello rather than than return an Alert, if the server does not support any of the ciphers listed in the ClientHello.
This commit modifies `tls_sockets()` to check whether the cipher in the ServerHello was one included in the ClientHello and to fail if it wasn't.
This is the simplest direct socket implementation of the MySQL STARTTLS
protocol.
This is a binary protocol, so it requires a new stream based send
(instead of the current line based send).
In `run_hpkp()` there is a call to `$OPENSSL s_client` that uses `${sni[i]}` as one of the command line options, but `sni` is not defined. My guess is that this was a copy/paste error from `run_client_simulation()`, which is the only function where an `sni` array is defined.
I am guessing that the intention was to use `$SNI` in `run_hpkp()`.
openssl/master branch now supports mysql STARTTLS in s_client
This patch adds support to call and use that s_client support to run
most, but not all (pfs, client simulation) tests.
The socket implementation is stubbed, but not yet functional.
testssl.sh is taking an educated guess which port makes sense to scan,
which one not and for which one to use which starttls handshake upfront.
This minimizes needless sscans and error messages.
Parallel mass testing mode is now not anymore experimental. To
use it a separate flag ``--mode=parallel`` was introduced. Serial
is still the default for now to avoid unexpected conditions.
Both the mode arguement and the default is subject to change.
The parallel mass testing mode can now also make use of a
nmap file. Also the functional test for nmap file was put
into a separate function and made more user safe. Open point is
that we better should use the hostname if the forward DNS record matches.
Fixed logical inconsistency: Ticketbleed was not being tested against a server with client authentication
Some variables in the beginning reordered
This PR attempts to address the outstanding issues with respect to issue #733, mainly by addressing the rules for when a certificate is obtained without SNI.
I believe I discovered the reason for issue #757: f2303a0d79.
This commit removed attempted to replace `$cbc_cipher_list_hex` (which was computed on the fly) with `$cbc_ciphers_hex` (which is static). However, the function was still using `$cbc_cipher_list_hex`, and since it wasn't being initialized to "" at the beginning of the function, the second call to `run_beast()` (to handle the second IP address) just appended to the value created by the first. Then, when the first two bytes were removed from the resulting string the result was a malformed cipher suite list, which caused `tls_sockets()` to fail.
Open issues: 1) The SNI logic 2) The fileout logic. 3) another section with ``trust_nosni -eq 4/8``
For 2): fileout is a general finding MEDIUM [1] which isn't in line now with the pr_*finding
in the section above anymore. It would make sense to punish HTTP services more than others.
Unfortunately he fileout statement cannot be moved below pr_svrty_medium/pr_svrty_high as
trustfinding_nosni hasn't been determined yet.
Fast solution would be probably to move the trustfinding_nosni section above the trustfinding
section.
Still 3) and a different trust over non-SNI makes it difficult -- e.g. Server has CN match only over
SNI but without SNI SAN matches. That's an edge case though which probably doesn't exist (like Bielefeld)
[1] That was WARN before. WARN should indicate a status of testssl that it cannot perform a check
This PR implements the suggestion from #753 for a child process in mass testing to send a signal to the parent to exit if the child encounters an error parsing its command line. At the moment, the child only sends the signal if it encounters an error that results in the `help()` function being called, but that could easily be changed (e.g., to also send a signal if `fatal()` is called in the child process).
In the case of parallel mass testing, the cleanup function needs to call `get_next_message_testing_parallel_result()` for the child that sent the signal, since otherwise the child's error message would not be displayed. Since I cannot tell which child sent the signal, I just call `cleanup()`, which displays the output of all completed child processes. Since the child process will send the signal almost immediately after starting, it can be assumed the that process that send the signal will be the last one that completed, and so its output will be displayed last (so it isn't hidden from the user).
Note that PR #753 is still needed, since there are still scenarios in which a child would not produce any JSON output, but the parent testssl.sh would not exit (e.g., the child process cannot open a socket to the server it is supposed to test). In additional, PR #754 would still be useful, since it would be more user friendly to catch the error in the mass testing file immediately (when possible) rather that partway through a potentially time-consuming testing process.
There is a bug in testssl.sh that occurs if mass testing is being performed, there is an error in the command line for one of the child tests, and either a single HTML file or a single JSON file is being created.
If mass testing is being performed and `parse_cmd_line()` detects an error in the command line for one of the child tests, then it will call `help()`, which will exit the program, resulting in `cleanup ()` being called. `cleanup ()` will call `html_footer()` and `fileout_footer()`. Since `html_header()` and `json_header()` have not yet been called, `$HTMLHEADER` and `$JSONHEADER` will both be `true, and so `html_footer()` and `fileout_footer()` will output HTML and JSON footers, even though no headers have been output.
This PR fixes the problem by having `help()` set `$HTMLHEADER` and `$JSONHEADER` to `false` so that no HTML or JSON footers are created.
A related problem is that if a single JSON file is being created, the parent process will insert a separator (a comma) into the JSON file between the outputs of each child process. However, if there is an error in one of the child process's command lines, then this child process will not produce any JSON output and so the JSON file will have two consecutive separators (commas), which is invalid according to http://jsonlint.com.
This PR provides a partial fix for the problem for parallel mass testing by checking whether a child process has created a non-empty JSON output before adding a separator to the JSON file. It leaves two unresolved problems:
* It does not fix the problem at all for `run_mass_testing()`, where the separator is added before the test with the command line error is run.
* It does not fix the problem for parallel mass testing for the case in which the first child test has a command line error.
started to use the ip parameter to decide where the dns resoultion
takes place (see #739 and #748). --ip=proxy or DNS_VIA_PROXY=true
will mean DNS resolutioni is done by proxy,
furthermore: swapped a few pr_magenta by pr_warning. Generally
testssl.sh should use for warnings a warning function and not
directly a color (we want to be flexible). There are still a few
remainders for fatal() which use bold magenta and thus I haven't
changed yet.
Also html_reserved() is being called also if no html output
is being requested. This could be fixed better probably.
This PR introduces the following changes/improvements to running mass testing in parallel:
* Continuous feedback is provided on the progress of testing, so that testssl.sh doesn't appeared to be frozen, even if it is waiting a long time for a child test to complete. [The feedback text is sent to `stderr` so that it doesn't appear in the log file if the `--logging` option is used.]
* Text sent to `stderr` by a child test is captured and then displayed in context with the appropriate test rather than being displayed at the time the error occurs. [Note that this means that if the `--logfile <logfile>` option is used (with `logfile` being the name of a file), then `logfile` will include both `stdout` and `stderr`.]
* If a test is started but it does not complete (because testssl.sh was stopped or because the test timed out), then a message is displayed indicating that the test didn't finish.
I believe that `run_mass_testing_parallel()` is either ready for use or nearly ready for use. So, I changed the program to use `run_mass_testing_parallel()` rather than `run_mass_testing()` if the `$EXPERIMENTAL` flag is set.
was received, testssl.sh tries 3 times to get memory from the server, If
the server returns different memory it's highly likely vulnerable.
(some more vulnerable devices to test against would be appreciated).
This is the default now -- all other hosts are mostly labled as OK.
Parsing SERVICE should be a little faster and more reliable
Increased needed debug level for output @ parse_tls_serverhello
Sometimes it just does not. In those case also sometimes session resumption
via tickets is supported.
This fixes the output for Session Ticket RFC 5077 and doesn't draw the wrong
conclusion from a missing lifetime hint.
It also tests for ticketbleed first whether there's a session ticket TLS
extension.
When the `--log`, `--logging`, or `--logfile <logfile>` option is being used and testssl.sh is stopped, a "printf: write error: Broken pipe" message tends to appear. From what I can tell, this is a result of the `tee` process being killed before the `cleanup ()` function completes. At the moment, `cleanup ()` doesn't write very much to `stdout`, but if parallel mass testing is performed, then `cleanup ()` may try to write the results of several previously completed tests.
This PR fixes the problem by adding the `-i` option ("ignore interrupt signals") to `tee`.
Note that I have tested this on a Linux desktop and an OS X laptop, but don't have a MS Windows computer on which to test this change.
This PR provides improvements to `run_mass_testing_parallel()`. Currently, `run_mass_testing_parallel()` treats `$MAX_PARALLEL` as the maximum difference between the number of the test whose results were last processed and the number of the most recently started test. This means that test #40 will not be started until the results of test #20 have been processed. I've encountered situations in which tests 21 though 39 have completed, but test #20 is still running, and so no new tests are started.
This PR fixes the problem by checking the status of all running child tests to see if any are complete, rather than just looking at `$NEXT_PARALLEL_TEST_TO_FINISH`. This prevents one slow child test (or a few slow child tests) from slowing up the entire mass testing process.
This PR also changes the basis for determining whether a slow child process should be killed. Rather than waiting `$MAX_WAIT_TEST` seconds from the time that the parent started waiting (which is rather arbitrary), it kills the process if `$MAX_WAIT_TEST` seconds have passed since the child test was started. Given this, and that the above change makes it less likely that a slow child test will slow up the overall testing, I increased `$MAX_WAIT_TEST` from 600 seconds to 1200 seconds.
I added some `debugme` statements that provide feedback on the status of testing, but in non-debug mode there may be a perception issue. If one test (e.g., test #20) is very slow, testssl.sh will not display any results from later tests until the slow test finishes, even though testssl.sh will continue running new tests in the background. The user, seeing no output from testssl.sh for an extended period of time, may think that testssl.sh has frozen, even though it is really just holding back on displaying the later results so that the results will be displayed in the order in which the tests were started.
There is a comment in the `run_client_simulation()` function that says "FIXME: printf formatting would look better, especially if we want a wide option here."
This PR is an attempt at addressing that FIXME and adding a wide option. The proposed wide option prints the same information as the non-wide option, just with the columns aligned. I didn't add any of the additional information that is displayed by other functions in wide mode, since I thought that made the output too wide.
-H is now --heartbleed instead of --headers,
-B is now --breach instead of --heartbleed,
-T is now --ticketbleed (was previously --breach)
bugs fix for run_ccs_injection() where the tls protocols wa not properly passed to the ClientHello
Made use of already determined protocol ( this time only from determine_optimal_proto() ) ==> we shpould use this in run_protocols() too!)
for run_ccs_injection + run_ticketbleed(). For achieving this determine_optimal_proto() needed to be modified so that it adds a protocol
to PROTOS_OFFERED (all_failed is now boolean there)
added two easy functions for converting dec to hex
sockread_fast() is for testing which should make socket erads faster -- albeit it could potentially block the whole thing
`emphasize_stuff_in_headers()` only adds color to the text being printed to the terminal if `$COLOR` is 2. So, the same should be the case for the HTML output.
This PR addresses issue #616, changing `run_cipher_match()` so that only those ciphers that are available are shown, unless the `--show-each` flag has been provided.
It also fixes a problem where the signature algorithm isn't being shown, even if `$SHOW_SIGALGO` is true.
This PR just addresses some places where quotes need to be used to avoid word splitting in case the referenced file, or path to the file, contains space characters.
The previous fix did not work if testssl.sh was found via `$PATH`. This seems to work in all cases. If testssl.sh is found via `$PATH` or if the command line includes a path, then `which` returns a non-empty response; otherwise, `$0` does not include any path, but one needs to be provided, so `$RUN_DIR/$PROG_NAME` is used.
Using "$0" as the name of the executable seems to work as long as "$0" contains a directory name (e.g, "workingfiles/testssl.sh"), but not if it is just the name of the executable (e.g., "testssl.sh"). Specifying "$RUN_DIR/$PROG_NAME" seems to work in both cases, since if "$0" doesn't contain any path information, `$RUN_DIR` is `.`
Use the suggestion "If you want to print the argument list as close as possible to what the user probably entered" from http://stackoverflow.com/questions/10835933/preserve-quotes-in-bash-arguments to create `$CMDLINE` and to print the command lines in `run_mass_testing()` and `run_mass_testing_parallel()`.
This PR addresses issue #702. Rather than create the command line for each child process in `run_mass_testing()` as a string, it creates it as an array, with each argument being a separate element in the array. This was done based on http://mywiki.wooledge.org/BashFAQ/050.
The printing of each child's command line done based on http://stackoverflow.com/questions/10835933/preserve-quotes-in-bash-arguments.
The `$CMDLINE` string remains unchanged, even though it isn't entirely "correct," since http://jsonlint.com/ complains if the "Invocation:" string contains backslashes.
I was doing some testing on my extended_tls_sockets branch and discovered that it was not fully working since the `TLS13_KEY_SHARES` array was empty. According to https://lists.gnu.org/archive/html/bug-bash/2012-06/msg00068.html, there is an issue when trying to initialize a global array inside a function. (The current code initializes `TLS12_CIPHER`, `TLS_CIPHER`, and `TLS13_KEY_SHARES` within `get_install_dir()`, since tls_data.txt is read in that function.) In fact, according to http://stackoverflow.com/questions/10806357/associative-arrays-are-local-by-default, in order to initialize a global variable in a function, one needs to provide the `-g` option, which was only added in Bash 4.2.
This PR seems to fix the problem by moving the reading of tls_data.txt to the main body of the code rather than reading it within the `get_install_dir()` function.
If I understand correctly how `run_hpkp()` should be displaying the list of Backups, the problem shown in the photo attached to #696 occurred because the dangling SPKIs should have been printed on the next line. This PR fixes this by changing the code that prints out the "good" backups to include a newline after printing the CA's name.
Another problem this PR fixes is that `tm_italic()` is being called instead of `pr_italic()`, meaning that the italicized text is being printed to the terminal, but is not being included in the HTML output.
Finally, this PR fixes errors in the definitions of `tmln_italic()` and `prln_italic()`. `tmln_italic()` calls `outln()` and `prln_italic()` calls `tmln_out()` instead of the reverse.
This PR fixes issue #695 by changing the call to `out_row_aligned_max_width()` so that the length of the flag is considered in determining the length of the first line of the output.
This PR fixes two issues with HTML generation that were introduced by a commit on March 31, 2016, "[count_ciphers is now un-sed'ed, minor improvements](a480e5f699)."
The first is that in `std_cipherlists()`, `[[ $DEBUG -ge 1 ]] && outln " -- $1" || outln` was changed to `[[ $DEBUG -ge 1 ]] && outln " -- $1" || outln`. The result being that in the HTML output, all of the tests from `run_std_cipherlists()` appear on the same line. This PR changes the line to:
```
[[ $DEBUG -ge 1 ]] && tm_out " -- $1"
outln
``
so that the line break is added to the HTML output, but the debugging information is not.
The second problem is that the commit on March 31 moved the call in main to `html_header()` until after the calls to `get_install_dir()`, `find_openssl_binary()`, `mybanner()`, `check4openssl_oldfarts()`, and `check_bsd_mount()`. The problem is that each of these functions may call an output function that will call `html_out()`.
If `html_out()` is called before `html_header()` and the command line contains `--htmlfile <htmlfile>`, then "htmlfile" will be written to before `html_header()` is called and then `html_header()` will warn that "htmlfile" already exists and then exit the program.
If `html_out()` is called before `html_header()` and the command line contains `--html`, then anything send to `html_out()` before `html_header()` is called (such as the banner) will not appear in the HTML file.
I did some testing with http://jsonlint.com/ and discovered a missing comma when massing testing is being performed and a single JSON file is being created.
In `run_server_defaults()` the variable `success` is defined twice, once an an ordinary variable and once as an array. The PR removes the incorrect definition. It also removes the definitions of some variables that are no longer used and reorganizes the definitions so that each line has only one variable type.
I also noticed a typo later in `run_server_defaults()` and corrected it.
This PR introduces the environment variable `CHILD_MASS_TESTING`, and uses it as an indicator that testssl.sh is running as a child within mass testing rather than using the `$APPEND` flag. It also makes a number of other changes to make the handling, of HTML, CSV, JSON, and log files consistent, and it fixes a number of bugs related to the generation of these files when mass testing is being performed.
Please let me know if you disagree with any of the changes in this PR, or if you would prefer that it be broken up into multiple smaller PRs.
Some of the changes are as follows:
- When the `$APPEND` flag is true, all of these files are appended to and headers and footers are omitted. (Perhaps this should be changed. Appending to a log file isn't an issue, but appending to a JSON or HTML file without including headers or footers seems to just create an improperly formatted file).
- Following the code in `prepare_logging()`, an error is printed and the program stops if the `$APPEND` flag is false and one of the files to be written to already exists.
Some of the bugs fixed:
Creating log files did not work with mass testing:
- If `--logfile <logfile>` is used, then the parent and each child try to write to "logfile".
- If `--logging` is used, then a log file is created for each child, but an oddly-named log file is also created for the parent. The one created by the parent contains the entire output.
Plain JSON files:
- When `--jsonfile <jsonfile>` is run, there is no comma separating the final finding for one child and the first finding for the next child.
Pretty JSON files:
- When `--jsonfile-pretty <jsonfile>` is called without mass testing, the "target host" line is empty, since `$NODE` has not yet been set.
- When `--jsonfile <jsonfile>` is run with mass testing, there is no comma separating the final finding for one child and the first finding for the next child. In addition, `fileout_pretty_json_banner()` is never called, and the entries for individual tests have insufficient information to determine what is being tested (it lists "service" and "ip", but not port number).
For the final issue, when mass testing is being performed and all output is being placed in a single file, I have the parent call `fileout_pretty_json_banner()`, but tell `fileout_pretty_json_banner()` to not include a "target host" or "port", but then have each child include a "target host" or "port" (when the "service" and "ip" are being printed).
This PR improves `out_row_aligned_max_width()` in a few ways:
* It makes better use of bash's string manipulation capabilities in order to simplify the function.
* It improves the function's performance. One of the most costly parts of `out_row_aligned_max_width()` was the while loop to print each entry in the text. Since there is only one place in the code where the the entries are not all printed the same ways (the list of supported curves printed by `run_pfs()`), the PR changes `out_row_aligned_max_width()` to just return a plain text string, which the calling function prints in the appropriate way. For the curves printed by `run_pfs()`, a new function, `out_row_aligned_max_width_by_entry()` takes care of getting the output from `out_row_aligned_max_width()` and then printing each entry appropriately.
* The PR also introduces a trick so that when the TLS extensions are printed, the text for an extension won't get split across two rows. It does this by replacing the space charters within the text for an extension with "}", formatting the result with `out_row_aligned_max_width()`, and then converting the "}" back to space characters.
`$OPENSSL_LOCATION` is used in `fileout_pretty_json_banner()`, `html_banner()`, `mybanner()`, and `prepare_logging()`, but the value of `$OPENSSL_LOCATION` is populated in `mybanner()`. This is usually okay, since `mybanner()` is always called before the other three functions are called. However, if `$QUIET` is `true`, then `mybanner()` returns immediately, without populating `$OPENSSL_LOCATION`, even though the value of `$OPENSSL_LOCATION` may be needed by one or more of the other functions.
This PR addresses this problem by populating `$OPENSSL_LOCATION` in `find_openssl_binary()` rather than `mybanner()`.
With the commit made on March 26, "partly (1/2) fixing #653," an HTML banner isn't added to the HTML files anymore. A banner should be added to the top of the HTML file if mass testing is being performed and a separate HTML file is being created for each test.
The `$APPEND` flag being `true` is an indicator that mass testing is being performed and that this is one of the individual tests being run. Given that `$APPEND` is `true`, `$HTMLHEADER` being `true` indicates that testssl.sh is creating the file name for the HTML output. So, it is when both flags are `true` that the HTML banner should be created.
`std_cipherlists()` does not include line breaks between tests in the output to the terminal when `$DEBUG` is 1, and it does not include line break between tests in the HTML output whenever `$DEBUG` is greater than 0.
In `create_client_simulation_tls_clienthello()` the variable `sni_extension_found` should be set if the ClientHello includes an SNI extension. Instead it was being set if and only if the ClientHello included some extension other than SNI.
This bug wasn't detected before for two reasons:
* It is rare to have a ClientHello that includes an SNI extension, but no other extensions.
* The code still works correctly if `sni_extension_found` is set even if there is no SNI in the ClientHello.
So, the bug only creates a problem if the browser's ClientHello include an SNI extension and no other extensions (see "BingPreview Jun 2014" in the client_simulation branch).
When HTML output is not being created, the print functions last step is to call `html_out()`, which responds to `return` rather than `return 0`. This causes problems for lines of code that rely on receiving a return value of 0. For example:
```
[[ $VULN_COUNT -le $VULN_THRESHLD ]] && outln && pr_headlineln " Testing for LUCKY13 vulnerability " && outln
```
This PR fixes two minor bugs:
* In `run_hpkp()`, the call to `$OPENSSL s_client` includes the option `-showcerts` twice. This PR removes one of them.
* In `get_server_certificate()`, the first call to `$OPENSSL s_client` includes `$addcmd`, but `$addcmd` has not yet been initialized. Instead, `$SNI` should be used.
Currently there is code to extract TLS extensions in three places, in `get_server_certificate()` and two places in `determine_tls_extensions()`. This PR replaces them with one new function, `extract_new_tls_extensions()`.
In order for the new function to work correctly whether OpenSSL or `tls_sockets()` is being used, this PR also changes `parse_tls_serverhello()` so that extensions are formatted in the file it creates in the same way as they are formatted by OpenSSL.
- in gerneral better performance measurements , starts from the real beginning (almost)
- allows results to put into file (MEASURE_TIME_FILE=google.txt testssl.sh google.com)
- PS4 improved: has now a performance debugging options (big solution)
- PS4 with proper alignment
- SCAN_TIME is now global so that it can be used not only by JSON-PRETTY (small performance debugging options uses it)
- prepare_debug() has now debugging stuff only, rest went to prepare_arrays()
This PR is the same as #661, except that it applies to the 2.9dev_html branch. It also fixes a bug in `prettyprint_local()` when a pattern to match is provided as input.
While doing some performance testing I discovered that `normalize_ciphercode()` is very slow. This PR simplifies the function and speeds it up significantly. This PR also addresses the TODO item in `normalize_ciphercode()` by eliminating the global variable HEXC.
When I view testssl.sh in KDE's text editor (kate), the code after "INFO" line in `show_finding()` isn't highlighted correctly, and the highlighting doesn't correct again until `pr_liteblueln()`. This PR fixes the highlighting issue by adding spaces between "]]" and ")". To be "safe," I also added spaces between "(" and "[[", and similarly added spaces in `is_json_format()`.
This PR ensures that each HTML file produced by testssl.sh only includes a single header, at the top, and a single footer, at the end. It also tries to ensure that the short-version banner is only placed at the top of the HTML file if (1) mass testing is being performed and (2) the results of each test is being placed in a separate file.
It also moves some of the logic out of main and into `html_header()`.
So far I haven't seen any HTML reserved characters (&, <, >, ", ') in the strings processed by `emphasize_stuff_in_headers()`, so this PR may be unnecessary. However, this PR will ensure that any such characters will be properly escaped in the HTML output.
"=~" doesn't need quotes if there's a text string one wants to match against (and shellcheck complains
about this)
pr_magenta shouldn't be used anymore as the logic what color we use should be done
some place else.
This branch is for getting the HTML patch from @dcooper16 into 2.9dev
Change to David's PR:
* removed HTMLHEADER. We always want that (in fact for flat JSON this is missing and needs to be added)
* not sure what this change does to --file
* changing of names They were redundant sometimes (pr_*_term )
* some formatting for readbility
Open points:
* there's a loop and a segfault --> tm_done_best
* HTMLHEADER: --file
* the former sed statement aroung L1900 for the header was way more readable. The combined
html+terminal version is just too much. Maybe a switch whether HTML is requested
is better so that this can be separated.
* Then e.g. "<span style=\"color:olive;font-weight:bold" can be kept in a variable
* any reason we need the text length here?
* what went into main here is too much. Actuallly what I put already in there bothered
me as too much logic and not obvious dependencies are in here. Now it's worse :-)
Can't this be just similar to JSON or CSV -- a seperate function with hooks
not in main()?
* minor thing: TERM_WIDTH is for HTML is maybe not the best. But that can be
tackled later
This PR attempts to address #631. It allows four choices for the `--mapping` option. "openssl" (default), "rfc", "no-openssl", and "no-rfc".
* "openssl" is the current default
* "no-rfc" shows only the OpenSSL names (just as it currently does)
* "rfc" shows the RFC name rather than the OpenSSL name for things that are not in wide now (just as it currently does). But now, in wide mode, it shows the RFC name first (further to the left) and the OpenSSL name second.
* "rfc-only" shows only the RFC name.
If the `--mapping` option is not provided, this is the same as "openssl". If the "cipher-mapping.txt" file cannot be found, then testssl.sh runs as if "no-rfc" had been requested.
It seems that the head command on OS X does not accept a negative number as a value for the "-n" parameter. This PR provides an alternative method for removing the "generator: " line without using "head."
Some sites have a long list of IP addresses and some IP addresses have a long list of DNS names that map to them.
This PR changes `display_rdns_etc()` to use `out_row_aligned_max_width()` to print the other IP addresses in `$IP46ADDRs` and to print `$rDNS`.
RFC 7633 introduces the TLS Features certificate extension, which contains "Features:
> The object member "Features" is a sequence of TLS extension identifiers (features, in this specification's terminology) as specified in the IANA Transport Layer Security (TLS) Extensions registry. If these features are requested by the client in its ClientHello message, then the server MUST return a ServerHello message that satisfies this request.
The main purpose of this certificate extension is to implement "must staple." If the extension is present in a TLS server's certificate and it includes status_request, then the server MUST include a stapled OCSP response if the client requests one. (The same applies for the status_request_v2 extension.)
This PR adds a check to `certificate_info()` of whether the server supports must staple (i.e., whether its certificate includes a TLS Features extension with "status_request"). It also changes the output for "OCSP stapling" in the case that the server did not staple an OCSP response. It indicates that:
* it is a critical issue if the certificate specifies "must staple"
* it is a low severity issue if the certificate does not specify "must staple," but the certificate does include an OCSP URI.
* it is not an issue at all if the certificate does not specify "must staple" and certificate does not include an OCSP URI.
`run_server_preference()` prints out the server's Negotiated cipher in a different color depending on the quality of the cipher. However, there is a "FIXME" since CBC ciphers are supposed to be flagged, but it is not easy to identity all CBC ciphers from their OpenSSL names.
This PR partially addresses this. It creates a separate function for printing a cipher based on its quality. Whenever possible it determines the quality of the cipher based on the RFC name. However, if it is provided an OpenSSL name and no cipher-mapping.txt file is available, it will follow the current (imperfect) logic for determining the cipher's quality.
The function also returns a value that indicates the quality of the cipher provided, with higher numbers indicating better ciphers. This return value is used by `run_server_preference()` to determine how to populate the "severity" field when calling `fileout()`.
In the case that `tls_sockets()` is being used and the server incorrectly fails the connection rather than downgrading, testssl.sh is printing "not offered" on one line and then the error message on the next line, but all the text should appear on one line (as it does when testing TLS 1 and TLS 1.1).
Since the test for TLS 1.2 in `run_protocols()` now uses `tls_sockets()` whenever `$ssl_native` is `true` (i.e., there is no longer a requirement for `$EXPERIMENTAL` to be true as well), the `$EXPERIMENTAL` flag should no longer be checked if the return value is 1.
Rearrange code so that in the case of just a single test, `parse_hn_port()` is not called earlier than it was previously unless it needs to be called in order to create the HTML file name.
Doing this ensures that the banner is displayed even if the `$URI` cannot be parsed (except in the case that the `$URI` needs to be parsed in order to create a file name) and that any error messages created by `parse_hn_port()` will be included in the HTML, if possible.
Add option for testssl.sh to create the HTML file name. If testssl.sh creates the file name, then, in the case of mass testing, a separate HTML file is created for each test (i.e., for each line in the file provided to `--file`).
I have a test server that I configured to support only SSLv3 and TLSv1.2. When I set `SSLHonorCipherOrder` to `off` I get the following results:
```
ECDHE-RSA-AES256-SHA: SSLv3 ECDHE-RSA-AES256-GCM-SHA384: TLSv1.2
```
The current code, when printing TLSv1.2 checks whether `${cipher[4]}` is empty, and since it is assume no previous protocol (SSLv2, SSLv3, TLSv1, TLSv1.1) was supported and so doesn't output a newline before outputting the cipher and protocol for TLSv1.2.
This PR fixes that by changing to code to look at the previous non-empty cipher (if there is one), even if that does not come from the previous protocol.
When a list of cipher suites is being displayed using `neat_list()`, testssl.sh shows the cipher suite's OpenSSL name and (in most cases) the RFC name as well. However, in all other cases only the OpenSSL name is shown.
This PR adds the option to have cipher suite's RFC names shown instead of the OpenSSL name, by including `--mapping rfc` in the command line. [Note: if the cipher-mapping.txt file cannot be found, then the `--mapping rfc` option is ignored and the OpenSSL names are shown.]
This PR seems to be related to issue #9, but #9 may be been referring to the output created by `neat_list()`.
Gray should appear for COLOR=1 or COLOR=2.
Since `pr_grey()` is basically the same as `out()` for COLOR=0, `mybanner()` should just call `pr_grey()` without checking the value of `$COLOR`.
Change `emphasize_stuff_in_headers()` to use olive and bold olive rather than brown and yellow. This matches what `aha` creates and appears similar to what is displayed in the terminal on a Mac. Also, yellow text is very difficult to read.
Found more places where output should only go to terminal, or where it was only going to the terminal (e.g., printf) but should also be in the HTML. Also added the ability to include active URLs in the HTML output.
To Do: Handle automatic generation of HTML file name and support for parallel testing.
Introduced "trick" so that if the `--file` option is used, `html_header()` will only be called once before anything is printed and `html_footer()` will only be called once after all printing is complete. With this, `html_header()` now delete the output file if it exists.
Also introduced the `html_reserved()`, which is called for all text to be sent to `out_html()`. `html_reserved()` converts any HTML reserved characters (", ', &, <, >) to their corresponding entity names (", ', &, <, >).
This PR addresses the issue raised in #623. This PR is based on the function `out_row_aligned_max_width()` that I proposed in #623, but the `out_row_aligned_max_width()` in this PR is a little different. It takes a fourth parameter, which is the function to use to print each word in the text string to be printed. This is used in `run_pfs()` so that the "Elliptic curves offered" can be printed using this function (some servers support 25 curves), while still having the curves printed using color-coding to indicate the quality of each curve.
I somewhat arbitrarily have each line wrap at 120 characters, but that could be changed (e.g., to `$TERM_WIDTH`).
For the most part I used the RGB values for xterm from https://en.wikipedia.org/wiki/ANSI_escape_code#Colors for the HTML colors, but with a few exceptions. For example, I did not use "yellow" for `pr_svrty_low()`, since that color is very difficult to read. I also used a different color for `pr_svrty_medium()` so that `pr_svrty_medium()` would appear more red than `pr_svrty_low()`.
These color choices could use more adjustment.
This PR adds the option to generate HTML. The code was created as follows:
* For each output function (`out()`, `outln()`, `pr_liteblue()`, etc.) I created two functions: one that just outputs to the terminal and one that outputs to the terminal and to the HTML file (if an HTML file is to be created).
* I modified the code so that any output that should appear in the HTML file in addition to being displayed on the terminal is sent through one of the display functions: out()`, `outln()`, `pr_liteblue()`, etc.
* I created a new function `retstring()` to use in place of `out()` when a function is creating a string to be "captured" by the calling function.
* I modified the code so that no string returned by a function includes color-coding escape characters.
