In order to santize input better there's a fucntion
now which does the work . ``safe_echo()``.
It is actually the same as ``tml_out()`` but is a bit snappier.
This PR adds initial support for TLSv1.3 to tls_sockets() and for run_client_simulation(). It does not change any of the other functions test TLSv1.3. So, with the exception of run_client_simulation(), the functionality added by this PR can only be tested using the --devel option.
This PR does not include the ability to decrypt the encrypted portions of the server's response. So, it does not support functions that need to see such things as the server's certificate, status information, or extensions (other than key share).
This PR fixes the use of has_server_protocol() in two places.
Currently std_ciphersuites() only tries SSLv2 if the server is known to support SSLv2. This changes it to try SSLv2 unless the server is known to not support SSLv2.
In run_beast(), tests against the server are run to determine support for TLSv1.2, TLSv1.1, TLSv1, and SSLv3 unless the server is known to support that protocol (i.e., even if has_server_protocol() reports that the server does not support the protocol). This changes it so that a test is only performed against the server if has_server_protocol() reports that it doesn't know whether the protocol is supported.
This may not be specific to LibreSSL, but just my local setup. However, when I test using LibreSSL testssl.sh still prints the following message several times:
WARNING: can't open config file: /usr/local/etc/ssl/openssl.cnf
This PR suppresses the error message for several calls to $OPENSSL and so fixes the problem.
The logic is complete now, so that if a protocol has been detected as
supported or not supported it will save a few cycles not to test
this protocol again.
There's probably -- also besides #839 space for improvements. The
advantage of this solution is that ``has_server_protocol()`` also
marks a protocol as tested if the result of the test is negative.
MongoDB listens on port 27017 and will respond to a direct TLS
ClientHello without the need for STARTTLS. Thus, testssl.sh already
mostly works to scan a MongoDB server.
The problem is that MongoDB will also reply to an HTTP GET request with
a mocked up HTTP response, which was tricking the tool into believing it
was dealing with an HTTP server. The response looks like this:
---
HTTP/1.0 200 OK
Connection: close
Content-Type: text/plain
Content-Length: 84
It looks like you are trying to access MongoDB over HTTP on the native
driver port.
---
Thus, the detection algorith must first look for HTTP, then dig deeper
into the body looking for "MongoDB".
The tool output now says:
Service detected: MongoDB, thus skipping HTTP specific checks
As dcooper16 noted in #834 there are checks which test for openssl versions
but don't take LibreSSL in account. This adds checks to it for several
LibreSSL versions (>=2.1.x) which are known to support ``determine_trust()``
and it HAS_DH_BITS.
Moreover engine check has been improved. Older LibreSSL versions (2.1 specifically)
had different error messages, so the previous checks failed. There's also
a CMD_LINE flag now where one can switch the engine support off: NO_ENGINE .
run_renogo from #834 is still an open issue.
All three issues fixed. Terminal code were actually
tow problems: Logging in from Linux with a 256 color xterm
makes tput set AF from OpenBSD hiccup. And the detection
of not-ncurses style underline e.g. was not working under OpenBSD.
The engine fix was done by David Cooper (see #831).
There's also a name of the binary now (OpenSSL/LibreSSL) for tracking
the flavor used.
SWURL contained for historical reasons trailing blanks
for released versions.
This caused an error in pr_boldurl --> html_out which
didn't write the trailing style info and didn't close
the href tag (travis complained.)
This patch removes the trailing blank but it doesn't
fix the error.
This is an initial implementation of the idea I proposed in #791. It includes checks based on draft-ietf-tls-grease as well as checks for specific implementation bugs that have previously been encountered.
This code needs testing. While I know of one server that will fail if the ClientHello contains more than 128 ciphers, I haven't been able to test any of the other code against any servers that have the tested-for bugs.
In addition, there is a need for polishing. The tests are referred to as `--grease`, which may not be a very user-friendly name. In addition, there is no output providing information about what tests are being performed. If a server fails a test, then a warning message is printed and is sent to `fileout()`. If the server passes all of the tests, then there is just a single output of "No bugs found."
At the moment, the code is not run by default. Unless `-g` or `--grease` is explicitly included in the command line, the code in this commit is not executed.
Currently the call to `$OPENSSL s_client` to obtain the certificate returned by the server when SNI is not provided is in `certificate_info()`, which means that it is called once for each certificate found the various called to `get_server_certificates()`.
This PR moves the call to `$OPENSSL s_client` to `run_server_defaults()` so that the call is made only once, even if more than one server certificate was found.
In addition, in most cases the certificate returned by the server when SNI is not provided will already have been retrieved by `run_server_defaults()` (in rounds 8-14), in which case `$HOSTCERT.nosni` can just be copied from there rather than making an additional call to `$OPENSSL s_client`.
During protocol check if a sever answered unexpected with
closing the conenction or another malformed reply the
output was not ok as DETECTED_TLS_VERSION was empty.
This fixes it by filling the variable with a string in ``parse_tls_serverhello()``
and then check in higher level (``run_protocols()``) the content.
Also it seems that I forgot in the commit from yesterday one ``&&`` to
commit in ``run_breach()``
This PR addresses the same issue as 6bb3494d98. In its current form, fileout_insert_warning() cannot be used after fileout_section_header() has been called for the first time.
testssl.sh produces an invalid JSON file if the --json-pretty option is used with the --single-cipher option. The reason is that fileout_section_header() isn't called before run_cipher_match() calls fileout() and fileout_section_footer() is not called afterwards.
There is also a problem with MEASURE_TIME, since the "cleanup" at the end of lets_roll() is not performed.
This PR fixes these problems by adding a call to fileout_section_header() before the call to run_cipher_match() and by copying the code from the end of lets_roll() to run_cipher_match() (just before the call to exit).
Extra client side warning led to a non-valid JSON pretty output. This fixes
this bug by adding an extra object. The objects are named "clientProblem${NUMBER}".
By "extra client side" I mean extra warnings which are not happening during regular
tests -- those are no extra ones and should just warn with ``fileout()`` instead
of ``fileout_insert_warning()``.
Also some ``fileout arg1 WARN`` were patched: WARN is not a finding. It is just
a report that either on the client side something doesn't work as expected or
the server could not be checked during a particular test. WARNING doesn't
exist at all, WARn should be used instead.
Some lines where a warning output to JSON or CSV was missing, was added.
Currently, when `$DEBUG` is 2 and the connection is successful, `parse_tls_serverhello()` prints out information about the server's ephermal (EC)DH key, but nothing else. For example:
```
sending client hello... reading server hello...
dh_bits: ECDH, P-256, 256 bits
sending close_notify...
(183 lines returned)
```
This commit changes `parse_tls_serverhello()` so that information about dh_bits is only displayed if `$DEBUG` is at least 3, making it the same as for other information about the server's response.
In addition, it indents the printing of the information about dh_bits in order to better align with other information displayed at this debug level.
- add forgotten servive FTP and XMPP
- polish other services
- after TLS 1.2 run is finished run a check whether no protocol has been detected and ask the user for confirmation to proceed
In `run_protocols()` for TLS 1.2, try one set of 127 ciphers and if the result isn't a connection at TLSv1.2 then try another set of 127 ciphers before giving up and assuming that TLS 1.2 isn't supported.
FIX#786
Fixed all other occurences so that debug level 2 is showing only minimal information like rough status and errors
Better line breaks for level 2
In ``client_simulation_sockets()`` and ``tls_sockets()`` moved debug output into if statements (may save a bit of time)
Replaced "$DEBUG -eq" by "$DEBUG -ge"
Removed obsolete hb_rounds in ``run_heartbleed()``
Adjusted wide output in vulnerabilities
The data for `run_client_simulation()` currently includes two clients that send version 2.0 CLIENT-HELLO messages (see Appendix E.2 of RFC 5246). Each of the CLIENT-HELLO messages advertises support for newer protocol versions (SSLv3 in the case of IE6XP and TLSv1.0 in the case of Java 6u45). A server may reject one of these version 2.0 CLIENT-HELLO messages, or it may respond with an SSLv2, SSLv3, or TLSv1.0 ServerHello.
The current code in `client_simulation_sockets()` assumes that the server's response with be an SSLv3 or later ServerHello. So, it can support cases in which servers respond with an SSLv3 or TLSv1.0 ServerHello (once PR #800 is accepted to undo the mistake in PR #797), but not cases in which servers response with an SSLv2 ServerHello.
This PR adds code to `client_simulation_sockets()` to check if the server's response is an SSLv2 ServerHello, so that it can process such responses with `parse_sslv2_serverhello()` rather than `parse_tls_serverhello()`.
When a connection is made using SSLv3 or later, `run_client_simulation()` will show to the protocol and cipher selected for the connection. With this PR, if the connection is made using SSLv2, `run_client_simulation()` will just show "SSLv2." In the case of SSLv2, the ServerHello contains a list of all ciphers that the server and client have in common, and it is up to the client to choose one. So, if the client and server have more than one cipher in common, more information about the client would be needed to know which cipher it would choose.
`std_cipherlists()` uses `has_server_protocol()` to determine whether to test for ciphers using SSLv2. However, this was resulting in false negatives due to #759. This PR removes the `has_server_protocol()` check so that SSLv2 is checked whenever the connection attempt with the TLSv1.2 ClientHello failed.
This PR fixes the extraction of the cipher suites in SSLv2 ClientHellos in `client_simulation_sockets()`. Since `client_simulation_sockets()` can only handle SSLv3 and above ServerHellos, it removes any SSLv2 ciphers and converts the other cipher from 3-byte format to 2-byte format.
If vulnernable to POODLE and has no TLS_FALLBACK_SCSV ==> HIGH.
If only run the fallback check and it has none, still label it as MEDIUM but issue
a clear warning that test under this circumstances is incomplete.
This PR adds the same additional check to `client_simulation_sockets()` as was previously added to `tls_sockets()`. It extracts the list of cipher suites offered from each ClientHello and passes the list to `parse_tls_serverhello()` so that `parse_tls_serverhello()` can check that the cipher offered in the ServerHello was included in the ClientHello.
This assumes that a real client would abort the connection if it was presented with a cipher in the ServerHello that it didn't offer in its ClientHello.
Unfortunately, most categories were passing incorrectly, since a
tls_socket() error was interpreted as a success for many of the cases.
Now we explicitely check for the known yaSSL failure and raise a warning
that the test is inconclusive.
New output against --starttls=mysql looks like:
Testing ~standard cipher categories
-----------------------------------
NULL ciphers (no encryption) SERVER_ERROR: test inconclusive due to MySQL Community Edition (yaSSL) bug.
Anonymous NULL Ciphers (no authentication) SERVER_ERROR: test inconclusive due to MySQL Community Edition (yaSSL) bug.
Export ciphers (w/o ADH+NULL) SERVER_ERROR: test inconclusive due to MySQL Community Edition (yaSSL) bug.
LOW: 64 Bit + DES encryption (w/o export) offered (NOT ok)
Weak 128 Bit ciphers (SEED, IDEA, RC[2,4]) SERVER_ERROR: test inconclusive due to MySQL Community Edition (yaSSL) bug.
Triple DES Ciphers (Medium) offered
High encryption (AES+Camellia, no AEAD) offered (OK)
Strong encryption (AEAD ciphers) SERVER_ERROR: test inconclusive due to MySQL Community Edition (yaSSL) bug.
Partial workaround for #793
According to a discussion thread on the IETF TLS WG mail list (see https://www.ietf.org/mail-archive/web/tls/current/msg19720.html), there is at least one TLS server that will fail if the last extension in the ClientHello has contains extension_data of length 0.
Currently, `tls_sockets()` will create such a ClientHello if:
* The padding extension is included, and the length of the ClientHello without the padding data would be between 508 and 511 bytes.
* No padding extension is included, and the caller provided `$extra_extensions` in which the last extension in `$extra_extensions` is empty.
* No padding extension is included, `$extra_extensions` is empty, no ECC cipher suites are offered, and the ClientHello is for TLSv1.1 or below (in this case the next protocol extension would be that last one).
This PR avoids the server bug (in nearly all cases) by ensuring the the padding extension (when present) always contains at least one byte, and by ensuring that when the padding extension is not present that the (non-empty) heartbeat extension is the last extension.
This PR does leave one possible scenario in which the last extension would be empty. If the caller provides an `$extra_extensions` in which the last extension in `$extra_extensions` is empty, `tls_sockets()` does not add a padding extension (or a padding extension is included in `$extra_extensions`), and `$extra_extensions` includes a heartbeat extension, then the last extension in the ClientHello would be empty. This, however, is a highly unlikely scenario, and certainly there are currently no such calls to `tls_sockets()` in testssl.sh.
Another yaSSL server incompatiblity. MySQL protects against CCS
Injection by erroring if it sees to CCS requests before the handshake
completes. But instead of returning a TLS alert, it seems to error up
the stack to MySQL which returns it's own error message.
Debug output looks like:
---
sending client hello,
reading server hello
1st reply:
sending payload #2 with TLS version x03, x02:
tls_content_type: 16 | tls_protocol: 0000 | byte6: 04
2nd reply:
00000000 16 00 00 02 ff 13 04 23 30 38 53 30 31 42 61 64 |.......#08S01Bad|
00000010 20 68 61 6e 64 73 68 61 6b 65 | handshake|
0000001a
test failed, probably read buffer too small (16000002FF1304)
---
This patch adds a custom check for this MySQL specific error, as far
down in the error check path as I can.
If $PROTOS_OFFERED was empty, and thus the protocols unknown, this
function would return true for any protocol you passed it. This caused
most callers to assume TLS1.0, even if the server didn't offer it.
Instead return false and make the caller do an extra lookup.
As reported in #782, some servers will return a ServerHello with a cipher not listed in the ClientHello rather than than return an Alert, if the server does not support any of the ciphers listed in the ClientHello.
This commit modifies `tls_sockets()` to check whether the cipher in the ServerHello was one included in the ClientHello and to fail if it wasn't.
This is the simplest direct socket implementation of the MySQL STARTTLS
protocol.
This is a binary protocol, so it requires a new stream based send
(instead of the current line based send).
In `run_hpkp()` there is a call to `$OPENSSL s_client` that uses `${sni[i]}` as one of the command line options, but `sni` is not defined. My guess is that this was a copy/paste error from `run_client_simulation()`, which is the only function where an `sni` array is defined.
I am guessing that the intention was to use `$SNI` in `run_hpkp()`.
openssl/master branch now supports mysql STARTTLS in s_client
This patch adds support to call and use that s_client support to run
most, but not all (pfs, client simulation) tests.
The socket implementation is stubbed, but not yet functional.
testssl.sh is taking an educated guess which port makes sense to scan,
which one not and for which one to use which starttls handshake upfront.
This minimizes needless sscans and error messages.
Parallel mass testing mode is now not anymore experimental. To
use it a separate flag ``--mode=parallel`` was introduced. Serial
is still the default for now to avoid unexpected conditions.
Both the mode arguement and the default is subject to change.
The parallel mass testing mode can now also make use of a
nmap file. Also the functional test for nmap file was put
into a separate function and made more user safe. Open point is
that we better should use the hostname if the forward DNS record matches.
Fixed logical inconsistency: Ticketbleed was not being tested against a server with client authentication
Some variables in the beginning reordered
This PR attempts to address the outstanding issues with respect to issue #733, mainly by addressing the rules for when a certificate is obtained without SNI.
I believe I discovered the reason for issue #757: f2303a0d79.
This commit removed attempted to replace `$cbc_cipher_list_hex` (which was computed on the fly) with `$cbc_ciphers_hex` (which is static). However, the function was still using `$cbc_cipher_list_hex`, and since it wasn't being initialized to "" at the beginning of the function, the second call to `run_beast()` (to handle the second IP address) just appended to the value created by the first. Then, when the first two bytes were removed from the resulting string the result was a malformed cipher suite list, which caused `tls_sockets()` to fail.
Open issues: 1) The SNI logic 2) The fileout logic. 3) another section with ``trust_nosni -eq 4/8``
For 2): fileout is a general finding MEDIUM [1] which isn't in line now with the pr_*finding
in the section above anymore. It would make sense to punish HTTP services more than others.
Unfortunately he fileout statement cannot be moved below pr_svrty_medium/pr_svrty_high as
trustfinding_nosni hasn't been determined yet.
Fast solution would be probably to move the trustfinding_nosni section above the trustfinding
section.
Still 3) and a different trust over non-SNI makes it difficult -- e.g. Server has CN match only over
SNI but without SNI SAN matches. That's an edge case though which probably doesn't exist (like Bielefeld)
[1] That was WARN before. WARN should indicate a status of testssl that it cannot perform a check
This PR implements the suggestion from #753 for a child process in mass testing to send a signal to the parent to exit if the child encounters an error parsing its command line. At the moment, the child only sends the signal if it encounters an error that results in the `help()` function being called, but that could easily be changed (e.g., to also send a signal if `fatal()` is called in the child process).
In the case of parallel mass testing, the cleanup function needs to call `get_next_message_testing_parallel_result()` for the child that sent the signal, since otherwise the child's error message would not be displayed. Since I cannot tell which child sent the signal, I just call `cleanup()`, which displays the output of all completed child processes. Since the child process will send the signal almost immediately after starting, it can be assumed the that process that send the signal will be the last one that completed, and so its output will be displayed last (so it isn't hidden from the user).
Note that PR #753 is still needed, since there are still scenarios in which a child would not produce any JSON output, but the parent testssl.sh would not exit (e.g., the child process cannot open a socket to the server it is supposed to test). In additional, PR #754 would still be useful, since it would be more user friendly to catch the error in the mass testing file immediately (when possible) rather that partway through a potentially time-consuming testing process.
There is a bug in testssl.sh that occurs if mass testing is being performed, there is an error in the command line for one of the child tests, and either a single HTML file or a single JSON file is being created.
If mass testing is being performed and `parse_cmd_line()` detects an error in the command line for one of the child tests, then it will call `help()`, which will exit the program, resulting in `cleanup ()` being called. `cleanup ()` will call `html_footer()` and `fileout_footer()`. Since `html_header()` and `json_header()` have not yet been called, `$HTMLHEADER` and `$JSONHEADER` will both be `true, and so `html_footer()` and `fileout_footer()` will output HTML and JSON footers, even though no headers have been output.
This PR fixes the problem by having `help()` set `$HTMLHEADER` and `$JSONHEADER` to `false` so that no HTML or JSON footers are created.
A related problem is that if a single JSON file is being created, the parent process will insert a separator (a comma) into the JSON file between the outputs of each child process. However, if there is an error in one of the child process's command lines, then this child process will not produce any JSON output and so the JSON file will have two consecutive separators (commas), which is invalid according to http://jsonlint.com.
This PR provides a partial fix for the problem for parallel mass testing by checking whether a child process has created a non-empty JSON output before adding a separator to the JSON file. It leaves two unresolved problems:
* It does not fix the problem at all for `run_mass_testing()`, where the separator is added before the test with the command line error is run.
* It does not fix the problem for parallel mass testing for the case in which the first child test has a command line error.
started to use the ip parameter to decide where the dns resoultion
takes place (see #739 and #748). --ip=proxy or DNS_VIA_PROXY=true
will mean DNS resolutioni is done by proxy,
furthermore: swapped a few pr_magenta by pr_warning. Generally
testssl.sh should use for warnings a warning function and not
directly a color (we want to be flexible). There are still a few
remainders for fatal() which use bold magenta and thus I haven't
changed yet.
Also html_reserved() is being called also if no html output
is being requested. This could be fixed better probably.
This PR introduces the following changes/improvements to running mass testing in parallel:
* Continuous feedback is provided on the progress of testing, so that testssl.sh doesn't appeared to be frozen, even if it is waiting a long time for a child test to complete. [The feedback text is sent to `stderr` so that it doesn't appear in the log file if the `--logging` option is used.]
* Text sent to `stderr` by a child test is captured and then displayed in context with the appropriate test rather than being displayed at the time the error occurs. [Note that this means that if the `--logfile <logfile>` option is used (with `logfile` being the name of a file), then `logfile` will include both `stdout` and `stderr`.]
* If a test is started but it does not complete (because testssl.sh was stopped or because the test timed out), then a message is displayed indicating that the test didn't finish.
I believe that `run_mass_testing_parallel()` is either ready for use or nearly ready for use. So, I changed the program to use `run_mass_testing_parallel()` rather than `run_mass_testing()` if the `$EXPERIMENTAL` flag is set.
was received, testssl.sh tries 3 times to get memory from the server, If
the server returns different memory it's highly likely vulnerable.
(some more vulnerable devices to test against would be appreciated).
This is the default now -- all other hosts are mostly labled as OK.
Parsing SERVICE should be a little faster and more reliable
Increased needed debug level for output @ parse_tls_serverhello
Sometimes it just does not. In those case also sometimes session resumption
via tickets is supported.
This fixes the output for Session Ticket RFC 5077 and doesn't draw the wrong
conclusion from a missing lifetime hint.
It also tests for ticketbleed first whether there's a session ticket TLS
extension.
When the `--log`, `--logging`, or `--logfile <logfile>` option is being used and testssl.sh is stopped, a "printf: write error: Broken pipe" message tends to appear. From what I can tell, this is a result of the `tee` process being killed before the `cleanup ()` function completes. At the moment, `cleanup ()` doesn't write very much to `stdout`, but if parallel mass testing is performed, then `cleanup ()` may try to write the results of several previously completed tests.
This PR fixes the problem by adding the `-i` option ("ignore interrupt signals") to `tee`.
Note that I have tested this on a Linux desktop and an OS X laptop, but don't have a MS Windows computer on which to test this change.
This PR provides improvements to `run_mass_testing_parallel()`. Currently, `run_mass_testing_parallel()` treats `$MAX_PARALLEL` as the maximum difference between the number of the test whose results were last processed and the number of the most recently started test. This means that test #40 will not be started until the results of test #20 have been processed. I've encountered situations in which tests 21 though 39 have completed, but test #20 is still running, and so no new tests are started.
This PR fixes the problem by checking the status of all running child tests to see if any are complete, rather than just looking at `$NEXT_PARALLEL_TEST_TO_FINISH`. This prevents one slow child test (or a few slow child tests) from slowing up the entire mass testing process.
This PR also changes the basis for determining whether a slow child process should be killed. Rather than waiting `$MAX_WAIT_TEST` seconds from the time that the parent started waiting (which is rather arbitrary), it kills the process if `$MAX_WAIT_TEST` seconds have passed since the child test was started. Given this, and that the above change makes it less likely that a slow child test will slow up the overall testing, I increased `$MAX_WAIT_TEST` from 600 seconds to 1200 seconds.
I added some `debugme` statements that provide feedback on the status of testing, but in non-debug mode there may be a perception issue. If one test (e.g., test #20) is very slow, testssl.sh will not display any results from later tests until the slow test finishes, even though testssl.sh will continue running new tests in the background. The user, seeing no output from testssl.sh for an extended period of time, may think that testssl.sh has frozen, even though it is really just holding back on displaying the later results so that the results will be displayed in the order in which the tests were started.
There is a comment in the `run_client_simulation()` function that says "FIXME: printf formatting would look better, especially if we want a wide option here."
This PR is an attempt at addressing that FIXME and adding a wide option. The proposed wide option prints the same information as the non-wide option, just with the columns aligned. I didn't add any of the additional information that is displayed by other functions in wide mode, since I thought that made the output too wide.
-H is now --heartbleed instead of --headers,
-B is now --breach instead of --heartbleed,
-T is now --ticketbleed (was previously --breach)
bugs fix for run_ccs_injection() where the tls protocols wa not properly passed to the ClientHello
Made use of already determined protocol ( this time only from determine_optimal_proto() ) ==> we shpould use this in run_protocols() too!)
for run_ccs_injection + run_ticketbleed(). For achieving this determine_optimal_proto() needed to be modified so that it adds a protocol
to PROTOS_OFFERED (all_failed is now boolean there)
added two easy functions for converting dec to hex
sockread_fast() is for testing which should make socket erads faster -- albeit it could potentially block the whole thing
`emphasize_stuff_in_headers()` only adds color to the text being printed to the terminal if `$COLOR` is 2. So, the same should be the case for the HTML output.
This PR addresses issue #616, changing `run_cipher_match()` so that only those ciphers that are available are shown, unless the `--show-each` flag has been provided.
It also fixes a problem where the signature algorithm isn't being shown, even if `$SHOW_SIGALGO` is true.
This PR just addresses some places where quotes need to be used to avoid word splitting in case the referenced file, or path to the file, contains space characters.
The previous fix did not work if testssl.sh was found via `$PATH`. This seems to work in all cases. If testssl.sh is found via `$PATH` or if the command line includes a path, then `which` returns a non-empty response; otherwise, `$0` does not include any path, but one needs to be provided, so `$RUN_DIR/$PROG_NAME` is used.
Using "$0" as the name of the executable seems to work as long as "$0" contains a directory name (e.g, "workingfiles/testssl.sh"), but not if it is just the name of the executable (e.g., "testssl.sh"). Specifying "$RUN_DIR/$PROG_NAME" seems to work in both cases, since if "$0" doesn't contain any path information, `$RUN_DIR` is `.`
Use the suggestion "If you want to print the argument list as close as possible to what the user probably entered" from http://stackoverflow.com/questions/10835933/preserve-quotes-in-bash-arguments to create `$CMDLINE` and to print the command lines in `run_mass_testing()` and `run_mass_testing_parallel()`.
This PR addresses issue #702. Rather than create the command line for each child process in `run_mass_testing()` as a string, it creates it as an array, with each argument being a separate element in the array. This was done based on http://mywiki.wooledge.org/BashFAQ/050.
The printing of each child's command line done based on http://stackoverflow.com/questions/10835933/preserve-quotes-in-bash-arguments.
The `$CMDLINE` string remains unchanged, even though it isn't entirely "correct," since http://jsonlint.com/ complains if the "Invocation:" string contains backslashes.
I was doing some testing on my extended_tls_sockets branch and discovered that it was not fully working since the `TLS13_KEY_SHARES` array was empty. According to https://lists.gnu.org/archive/html/bug-bash/2012-06/msg00068.html, there is an issue when trying to initialize a global array inside a function. (The current code initializes `TLS12_CIPHER`, `TLS_CIPHER`, and `TLS13_KEY_SHARES` within `get_install_dir()`, since tls_data.txt is read in that function.) In fact, according to http://stackoverflow.com/questions/10806357/associative-arrays-are-local-by-default, in order to initialize a global variable in a function, one needs to provide the `-g` option, which was only added in Bash 4.2.
This PR seems to fix the problem by moving the reading of tls_data.txt to the main body of the code rather than reading it within the `get_install_dir()` function.
If I understand correctly how `run_hpkp()` should be displaying the list of Backups, the problem shown in the photo attached to #696 occurred because the dangling SPKIs should have been printed on the next line. This PR fixes this by changing the code that prints out the "good" backups to include a newline after printing the CA's name.
Another problem this PR fixes is that `tm_italic()` is being called instead of `pr_italic()`, meaning that the italicized text is being printed to the terminal, but is not being included in the HTML output.
Finally, this PR fixes errors in the definitions of `tmln_italic()` and `prln_italic()`. `tmln_italic()` calls `outln()` and `prln_italic()` calls `tmln_out()` instead of the reverse.
This PR fixes issue #695 by changing the call to `out_row_aligned_max_width()` so that the length of the flag is considered in determining the length of the first line of the output.
This PR fixes two issues with HTML generation that were introduced by a commit on March 31, 2016, "[count_ciphers is now un-sed'ed, minor improvements](a480e5f699)."
The first is that in `std_cipherlists()`, `[[ $DEBUG -ge 1 ]] && outln " -- $1" || outln` was changed to `[[ $DEBUG -ge 1 ]] && outln " -- $1" || outln`. The result being that in the HTML output, all of the tests from `run_std_cipherlists()` appear on the same line. This PR changes the line to:
```
[[ $DEBUG -ge 1 ]] && tm_out " -- $1"
outln
``
so that the line break is added to the HTML output, but the debugging information is not.
The second problem is that the commit on March 31 moved the call in main to `html_header()` until after the calls to `get_install_dir()`, `find_openssl_binary()`, `mybanner()`, `check4openssl_oldfarts()`, and `check_bsd_mount()`. The problem is that each of these functions may call an output function that will call `html_out()`.
If `html_out()` is called before `html_header()` and the command line contains `--htmlfile <htmlfile>`, then "htmlfile" will be written to before `html_header()` is called and then `html_header()` will warn that "htmlfile" already exists and then exit the program.
If `html_out()` is called before `html_header()` and the command line contains `--html`, then anything send to `html_out()` before `html_header()` is called (such as the banner) will not appear in the HTML file.
I did some testing with http://jsonlint.com/ and discovered a missing comma when massing testing is being performed and a single JSON file is being created.
In `run_server_defaults()` the variable `success` is defined twice, once an an ordinary variable and once as an array. The PR removes the incorrect definition. It also removes the definitions of some variables that are no longer used and reorganizes the definitions so that each line has only one variable type.
I also noticed a typo later in `run_server_defaults()` and corrected it.
This PR introduces the environment variable `CHILD_MASS_TESTING`, and uses it as an indicator that testssl.sh is running as a child within mass testing rather than using the `$APPEND` flag. It also makes a number of other changes to make the handling, of HTML, CSV, JSON, and log files consistent, and it fixes a number of bugs related to the generation of these files when mass testing is being performed.
Please let me know if you disagree with any of the changes in this PR, or if you would prefer that it be broken up into multiple smaller PRs.
Some of the changes are as follows:
- When the `$APPEND` flag is true, all of these files are appended to and headers and footers are omitted. (Perhaps this should be changed. Appending to a log file isn't an issue, but appending to a JSON or HTML file without including headers or footers seems to just create an improperly formatted file).
- Following the code in `prepare_logging()`, an error is printed and the program stops if the `$APPEND` flag is false and one of the files to be written to already exists.
Some of the bugs fixed:
Creating log files did not work with mass testing:
- If `--logfile <logfile>` is used, then the parent and each child try to write to "logfile".
- If `--logging` is used, then a log file is created for each child, but an oddly-named log file is also created for the parent. The one created by the parent contains the entire output.
Plain JSON files:
- When `--jsonfile <jsonfile>` is run, there is no comma separating the final finding for one child and the first finding for the next child.
Pretty JSON files:
- When `--jsonfile-pretty <jsonfile>` is called without mass testing, the "target host" line is empty, since `$NODE` has not yet been set.
- When `--jsonfile <jsonfile>` is run with mass testing, there is no comma separating the final finding for one child and the first finding for the next child. In addition, `fileout_pretty_json_banner()` is never called, and the entries for individual tests have insufficient information to determine what is being tested (it lists "service" and "ip", but not port number).
For the final issue, when mass testing is being performed and all output is being placed in a single file, I have the parent call `fileout_pretty_json_banner()`, but tell `fileout_pretty_json_banner()` to not include a "target host" or "port", but then have each child include a "target host" or "port" (when the "service" and "ip" are being printed).
This PR improves `out_row_aligned_max_width()` in a few ways:
* It makes better use of bash's string manipulation capabilities in order to simplify the function.
* It improves the function's performance. One of the most costly parts of `out_row_aligned_max_width()` was the while loop to print each entry in the text. Since there is only one place in the code where the the entries are not all printed the same ways (the list of supported curves printed by `run_pfs()`), the PR changes `out_row_aligned_max_width()` to just return a plain text string, which the calling function prints in the appropriate way. For the curves printed by `run_pfs()`, a new function, `out_row_aligned_max_width_by_entry()` takes care of getting the output from `out_row_aligned_max_width()` and then printing each entry appropriately.
* The PR also introduces a trick so that when the TLS extensions are printed, the text for an extension won't get split across two rows. It does this by replacing the space charters within the text for an extension with "}", formatting the result with `out_row_aligned_max_width()`, and then converting the "}" back to space characters.
`$OPENSSL_LOCATION` is used in `fileout_pretty_json_banner()`, `html_banner()`, `mybanner()`, and `prepare_logging()`, but the value of `$OPENSSL_LOCATION` is populated in `mybanner()`. This is usually okay, since `mybanner()` is always called before the other three functions are called. However, if `$QUIET` is `true`, then `mybanner()` returns immediately, without populating `$OPENSSL_LOCATION`, even though the value of `$OPENSSL_LOCATION` may be needed by one or more of the other functions.
This PR addresses this problem by populating `$OPENSSL_LOCATION` in `find_openssl_binary()` rather than `mybanner()`.
With the commit made on March 26, "partly (1/2) fixing #653," an HTML banner isn't added to the HTML files anymore. A banner should be added to the top of the HTML file if mass testing is being performed and a separate HTML file is being created for each test.
The `$APPEND` flag being `true` is an indicator that mass testing is being performed and that this is one of the individual tests being run. Given that `$APPEND` is `true`, `$HTMLHEADER` being `true` indicates that testssl.sh is creating the file name for the HTML output. So, it is when both flags are `true` that the HTML banner should be created.
`std_cipherlists()` does not include line breaks between tests in the output to the terminal when `$DEBUG` is 1, and it does not include line break between tests in the HTML output whenever `$DEBUG` is greater than 0.
In `create_client_simulation_tls_clienthello()` the variable `sni_extension_found` should be set if the ClientHello includes an SNI extension. Instead it was being set if and only if the ClientHello included some extension other than SNI.
This bug wasn't detected before for two reasons:
* It is rare to have a ClientHello that includes an SNI extension, but no other extensions.
* The code still works correctly if `sni_extension_found` is set even if there is no SNI in the ClientHello.
So, the bug only creates a problem if the browser's ClientHello include an SNI extension and no other extensions (see "BingPreview Jun 2014" in the client_simulation branch).
When HTML output is not being created, the print functions last step is to call `html_out()`, which responds to `return` rather than `return 0`. This causes problems for lines of code that rely on receiving a return value of 0. For example:
```
[[ $VULN_COUNT -le $VULN_THRESHLD ]] && outln && pr_headlineln " Testing for LUCKY13 vulnerability " && outln
```
This PR fixes two minor bugs:
* In `run_hpkp()`, the call to `$OPENSSL s_client` includes the option `-showcerts` twice. This PR removes one of them.
* In `get_server_certificate()`, the first call to `$OPENSSL s_client` includes `$addcmd`, but `$addcmd` has not yet been initialized. Instead, `$SNI` should be used.
Currently there is code to extract TLS extensions in three places, in `get_server_certificate()` and two places in `determine_tls_extensions()`. This PR replaces them with one new function, `extract_new_tls_extensions()`.
In order for the new function to work correctly whether OpenSSL or `tls_sockets()` is being used, this PR also changes `parse_tls_serverhello()` so that extensions are formatted in the file it creates in the same way as they are formatted by OpenSSL.
- in gerneral better performance measurements , starts from the real beginning (almost)
- allows results to put into file (MEASURE_TIME_FILE=google.txt testssl.sh google.com)
- PS4 improved: has now a performance debugging options (big solution)
- PS4 with proper alignment
- SCAN_TIME is now global so that it can be used not only by JSON-PRETTY (small performance debugging options uses it)
- prepare_debug() has now debugging stuff only, rest went to prepare_arrays()
This PR is the same as #661, except that it applies to the 2.9dev_html branch. It also fixes a bug in `prettyprint_local()` when a pattern to match is provided as input.
While doing some performance testing I discovered that `normalize_ciphercode()` is very slow. This PR simplifies the function and speeds it up significantly. This PR also addresses the TODO item in `normalize_ciphercode()` by eliminating the global variable HEXC.
When I view testssl.sh in KDE's text editor (kate), the code after "INFO" line in `show_finding()` isn't highlighted correctly, and the highlighting doesn't correct again until `pr_liteblueln()`. This PR fixes the highlighting issue by adding spaces between "]]" and ")". To be "safe," I also added spaces between "(" and "[[", and similarly added spaces in `is_json_format()`.
This PR ensures that each HTML file produced by testssl.sh only includes a single header, at the top, and a single footer, at the end. It also tries to ensure that the short-version banner is only placed at the top of the HTML file if (1) mass testing is being performed and (2) the results of each test is being placed in a separate file.
It also moves some of the logic out of main and into `html_header()`.
