testssl.sh is taking an educated guess which port makes sense to scan,
which one not and for which one to use which starttls handshake upfront.
This minimizes needless sscans and error messages.
Parallel mass testing mode is now not anymore experimental. To
use it a separate flag ``--mode=parallel`` was introduced. Serial
is still the default for now to avoid unexpected conditions.
Both the mode arguement and the default is subject to change.
The parallel mass testing mode can now also make use of a
nmap file. Also the functional test for nmap file was put
into a separate function and made more user safe. Open point is
that we better should use the hostname if the forward DNS record matches.
Fixed logical inconsistency: Ticketbleed was not being tested against a server with client authentication
Some variables in the beginning reordered
timeout: the TLS ticket check has a timeout, so that early on non-reachable hosts
are determined. If it is running into the timeout, it quits early. The
timeout is configurable via environment e.g. TIMEOUT=16 ./ticketbleed.bash <host>
Also other ports are allowed albeit it probably it is of limited use
Supplying no arg is now more user-friendly
This PR attempts to address the outstanding issues with respect to issue #733, mainly by addressing the rules for when a certificate is obtained without SNI.
I believe I discovered the reason for issue #757: f2303a0d79.
This commit removed attempted to replace `$cbc_cipher_list_hex` (which was computed on the fly) with `$cbc_ciphers_hex` (which is static). However, the function was still using `$cbc_cipher_list_hex`, and since it wasn't being initialized to "" at the beginning of the function, the second call to `run_beast()` (to handle the second IP address) just appended to the value created by the first. Then, when the first two bytes were removed from the resulting string the result was a malformed cipher suite list, which caused `tls_sockets()` to fail.
Open issues: 1) The SNI logic 2) The fileout logic. 3) another section with ``trust_nosni -eq 4/8``
For 2): fileout is a general finding MEDIUM [1] which isn't in line now with the pr_*finding
in the section above anymore. It would make sense to punish HTTP services more than others.
Unfortunately he fileout statement cannot be moved below pr_svrty_medium/pr_svrty_high as
trustfinding_nosni hasn't been determined yet.
Fast solution would be probably to move the trustfinding_nosni section above the trustfinding
section.
Still 3) and a different trust over non-SNI makes it difficult -- e.g. Server has CN match only over
SNI but without SNI SAN matches. That's an edge case though which probably doesn't exist (like Bielefeld)
[1] That was WARN before. WARN should indicate a status of testssl that it cannot perform a check
Open issues: 1) The SNI logic 2) The fileout logic. 3) another section with ``trust_nosni -eq 4/8``
For 2): fileout is a general finding MEDIUM [1] which isn't in line now with the pr_*finding
in the section above anymore. It would make sense to punish HTTP services more than others.
Unfortunately he fileout statement cannot be moved below pr_svrty_medium/pr_svrty_high as
trustfinding_nosni hasn't been determined yet.
Fast solution would be probably to move the trustfinding_nosni section above the trustfinding
section.
Still 3) and a different trust over non-SNI makes it difficult -- e.g. Server has CN match only over
SNI but without SNI SAN matches. That's an edge case though which probably doesn't exist (like Bielefeld)
[1] That was WARN before. WARN should indicate a status of testssl that it cannot perform a check
This PR implements the suggestion from #753 for a child process in mass testing to send a signal to the parent to exit if the child encounters an error parsing its command line. At the moment, the child only sends the signal if it encounters an error that results in the `help()` function being called, but that could easily be changed (e.g., to also send a signal if `fatal()` is called in the child process).
In the case of parallel mass testing, the cleanup function needs to call `get_next_message_testing_parallel_result()` for the child that sent the signal, since otherwise the child's error message would not be displayed. Since I cannot tell which child sent the signal, I just call `cleanup()`, which displays the output of all completed child processes. Since the child process will send the signal almost immediately after starting, it can be assumed the that process that send the signal will be the last one that completed, and so its output will be displayed last (so it isn't hidden from the user).
Note that PR #753 is still needed, since there are still scenarios in which a child would not produce any JSON output, but the parent testssl.sh would not exit (e.g., the child process cannot open a socket to the server it is supposed to test). In additional, PR #754 would still be useful, since it would be more user friendly to catch the error in the mass testing file immediately (when possible) rather that partway through a potentially time-consuming testing process.
There is a bug in testssl.sh that occurs if mass testing is being performed, there is an error in the command line for one of the child tests, and either a single HTML file or a single JSON file is being created.
If mass testing is being performed and `parse_cmd_line()` detects an error in the command line for one of the child tests, then it will call `help()`, which will exit the program, resulting in `cleanup ()` being called. `cleanup ()` will call `html_footer()` and `fileout_footer()`. Since `html_header()` and `json_header()` have not yet been called, `$HTMLHEADER` and `$JSONHEADER` will both be `true, and so `html_footer()` and `fileout_footer()` will output HTML and JSON footers, even though no headers have been output.
This PR fixes the problem by having `help()` set `$HTMLHEADER` and `$JSONHEADER` to `false` so that no HTML or JSON footers are created.
A related problem is that if a single JSON file is being created, the parent process will insert a separator (a comma) into the JSON file between the outputs of each child process. However, if there is an error in one of the child process's command lines, then this child process will not produce any JSON output and so the JSON file will have two consecutive separators (commas), which is invalid according to http://jsonlint.com.
This PR provides a partial fix for the problem for parallel mass testing by checking whether a child process has created a non-empty JSON output before adding a separator to the JSON file. It leaves two unresolved problems:
* It does not fix the problem at all for `run_mass_testing()`, where the separator is added before the test with the command line error is run.
* It does not fix the problem for parallel mass testing for the case in which the first child test has a command line error.
started to use the ip parameter to decide where the dns resoultion
takes place (see #739 and #748). --ip=proxy or DNS_VIA_PROXY=true
will mean DNS resolutioni is done by proxy,
furthermore: swapped a few pr_magenta by pr_warning. Generally
testssl.sh should use for warnings a warning function and not
directly a color (we want to be flexible). There are still a few
remainders for fatal() which use bold magenta and thus I haven't
changed yet.
Also html_reserved() is being called also if no html output
is being requested. This could be fixed better probably.
This PR introduces the following changes/improvements to running mass testing in parallel:
* Continuous feedback is provided on the progress of testing, so that testssl.sh doesn't appeared to be frozen, even if it is waiting a long time for a child test to complete. [The feedback text is sent to `stderr` so that it doesn't appear in the log file if the `--logging` option is used.]
* Text sent to `stderr` by a child test is captured and then displayed in context with the appropriate test rather than being displayed at the time the error occurs. [Note that this means that if the `--logfile <logfile>` option is used (with `logfile` being the name of a file), then `logfile` will include both `stdout` and `stderr`.]
* If a test is started but it does not complete (because testssl.sh was stopped or because the test timed out), then a message is displayed indicating that the test didn't finish.
I believe that `run_mass_testing_parallel()` is either ready for use or nearly ready for use. So, I changed the program to use `run_mass_testing_parallel()` rather than `run_mass_testing()` if the `$EXPERIMENTAL` flag is set.
Dirk
David Cooper