For JSON pretty the host specific parameters target host + port
could be better placed in the scanResult object.
It is still under discussion as logically the parent object is deduced
the command line.
TLS_RSA_* which don't fall into the aleady mentioned
categories (CBC cipher, export, RC4 etc.) are now
a bit more more penalized. Those are the ones which have
an RSA key exchange AND a modern encryption.
pr_cipher_quality() needs to be redone after carefully
reconsidered which cipher should have which rating.
This PR is similar to #944. If using OpenSSL 1.1.1 to connect to a server that supports TLSv1.3, `run_crime()` will connect to the server using TLSv1.3, which does not support TLS-level compression. So, the server will be reported as "not vulnerable" even if would use compression for connections at TLSv1.2 and below.
I have not encountered any "live" servers that support both TLSv1.3 and TLS-level compression. I verified this problem by using OpenSSL 1.1.1 to create a server that supports both TLSv1.3 and TLS-level compression:
```
openssl111 s_server -cert cert.pem -key key.pem -accept 8443 -WWW -comp
```
I then tested the server using `testssl.sh --crime` with both openssl111 and OpenSSL 1.0.2-chacha.
run_renego() appears to produce a false positive if OpenSSL 1.1.1 is used and the server being tested supports TLSv1.3 (i.e., the server supports the same draft version of TLSv1.3 as the version of OpenSSL 1.1.1 being used does). This PR fixes the problem by telling calls to $OPENSSL s_client in run_renego() to not use TLSv1.3.
.. to check during the default run for server implemenation bugs
and run cipher per procol check instead of cipher check.
Please not that this option could disappear later.
In TLSv1.3 servers may send a supported_groups extension, which "SHOULD contain all groups the server supports, regardless of whether they are currently supported by the client."
This PR extracts the contents of the supported_groups extension, if `parse_tls_serverhello()` is to process "all" of the server's response. The contents of the extension are also displayed on the terminal if $DEBUG -ge 3.
In TLSv1.2 and below, servers respond to a status_request extension (a request for a stapled OCSP response) by returning an empty status_request extension and then including a CertificateStatus message, which follows the Certificate message. In TLSv1.3 the CertificateStatus response is included as the value of the status_request extension, which now appears as an extension within the Certificate message.
This PR extracts the contents of the status_request extension sent by the server so that it can later be processed in the same way as if it had sent in a TLSv1.2 or below response.
This PR adds code to decrypt the encrypted portion of the server's response for TLSv1.3 and to then process any certificates and encrypted extensions. This code supports all 5 TLSv1.3 cipher suites, and so any response can be decrypted as long as the session key can be derived (which requires OpenSSL to support the ephemeral key that was used - see #938).
For the symmetric decryption, the sym-decrypt() function uses OpenSSL when possible and internal Bash functions when needed.
For AES-GCM and AES-CCM ciphers sym-decrypt() normally uses internal Bash functions, which rely on using "$OPENSSL enc" in AES-ECB mode to generate the key stream and then Bash functionality to XOR the key stream with the ciphertext. With some version of OpenSSL the AES-GCM ciphers are decrypted using "$OPENSSL enc" in AES-GCM mode directly. On my system, however, both methods seem to work about equally fast.
For ChaCha20 ciphers, "$OPENSSL enc -chacha20" is used, if supported (OpenSSL 1.1.x only). and Bash internal functions (without any OpenSSL support) are used otherwise. In this case, if the Bash internal functions need to be used, decryption is very, very, very slow. Fortunately, in a typical run of testssl.sh there won't be many cases in which the connection will be TLSv1.3 with ChaCha20 and the entire response needs to be processed (requiring decryption). In most cases, even if the connection is TLSv1.3 with ChaCha20, will at most need the ephemeral key, which is available in plain text.
This is the first in a series of PRs to add support for processing the encrypted portions of the server's response in a TLSv1.3 handshake.
This PR adds the code to derive the handshake traffic key needed to decrypt the response (the next PR will add the code to perform the symmetric-key decryption of the encrypted portions of the response).
Since this PR does not make use of the traffic key that it derives, it doesn't yet add any new functionality.
Note that testssl.sh will not always be able to derive the session keys. If the version of OpenSSL that is bundled with testssl.sh is used and the server chooses to use an X25519 ephemeral key, OpenSSL will be unable to perform the shared secret in derive-handshake-traffic-secret(). (OpenSSL 1.1.0 supports X25519.) Since X25519 use a different encoding than ECDH keys, the lack of X25519 support will be discovered in parse_tls_serverhello() when $OPENSSL pkey is unable to convert the key from DER to PEM. So, in debugging mode, parse_tls_serverhello() now displays a warning if it receives a key share that $OPENSSL pkey cannot handle.
get_server_certificate() uses an awk script to extract the certificates from the output of OPENSSL s_client and it then uses the following line to determine how many certificates were found:
nrsaved=$(count_words "$(echo level?.crt 2>/dev/null)")
If $nrsaved is 0, then get_server_certificate() returns 1 (indicating failure); otherwise it returns 0 (indicating success).
However, the check for the number of certificates returned doesn't work if no certificates were found, as nrsaved will be set to 1 if no certificates were found:
> touch level0.crt
> echo level?.crt
level0.crt
> touch level1.crt
> echo level?.crt
level0.crt level1.crt
> rm level0.crt level1.crt
> echo level?.crt
level?.crt
This PR fixes the problem by first checking that level0.crt exists (-s is used instead of -e, since an empty file wouldn't have a certificate).
Similar to the recently added HAS_PKUTIL (f829878a43), this PR adds HAS_PKEY, which indicates whether OpenSSL has the pkey utility. HAS_PKEY is then checked before attempting to do something that requires the pkey utility.
POP3 STARTTLS handshakes were often unsuccessful as
a regex wasn't properly escaped.
Furthermore if a STARTTLS handshake doesn't succeed, there's
a warning now.
See pending PR #905 / issue #333.
There's still lots of work needed and probably the function
needs to be completely rewritten and to be in sync with
other parts of the program.
This PR fixes one odd formatting of header flags like X-Frame-Options,
where the output header maybe contained a LF "\r". X-XSS-Protection was
also not correctly formatted due to the fact that only a part of it until
the blank was displayed.
Also the file output may contain now 1x less blank, e.g.
"X-Content-Type-Options: nosniff" instead of
"X-Content-Type-Options: nosniff"
This PR adds an additional COLOR level (3). If color is set to 3 then all ciphers are printed according to pr_cipher_quality() rather than just the "Negotiated cipher" in run_server_preference().
This PR reduces the amount of work parse_tls_serverhello() does when processing alert messages when not in debug mode. It delays writing anything to $TMPFILE unless $DEGUG -ge 1 until it has reason to believe that the response was successful. If $DEBUG is 0 and alert messages are sent, then no file operations are performed processing the alert messages.
In almost every case, there is no attempt to look at the contents of $TEMPDIR/$NODEIP.parse_tls_serverhello.txt unless the connection was successful. So, in most cases, it is okay to not call tmpfile_handle() in parse_tls_serverhello() unless the connection was successful. There is, however. one place in run_grease() where the code reads the contents of $TEMPDIR/$NODEIP.parse_tls_serverhello.txt even if the connection was not successful. In order to address this, the DEBUG level is temporarily set to 1 when performing this test if its value is 0. Also in order to address this, changes were made in parse_tls_serverhello() to ensure that "tmpfile_handle $FUNCNAME.txt" is always called before returning if $DEBUG -ge 1.
If certificate-based client authentication is required by the server, then most HTTP-related checks are skipped, even if the "--assume-http" flag is used. If $CLIENT_AUTH is true, then $ASSUME_HTTP is ignored.
In some cases the checks are appropriately skipped, since the tests cannot be performed. In other places, the value of "$CLIENT_AUTH" is used as a hint as to whether HTTP is being used. For example, in run_tickbleed:
if [[ "$SERVICE" != HTTP ]] && ! "$CLIENT_AUTH"; then
outln "-- (applicable only for HTTPS)"
fileout "ticketbleed" "INFO" "Ticketbleed: not applicable, not HTTP" "$cve" "$cwe"
return 0
fi
There are some places, however, where tests are just skipped, even if both $CLIENT_AUTH and $ASSUME_HTTP are true, even though the test could be performed. For example, run_client_simulation() only simulates generic clients in this case.
This PR attempts to address this:
* In run_client_simulation() it runs all of the tests if $ASSUME_HTTP is true.
* In certificate_transparency() it only says that the lack of CT information is "N/A" it can verify that HTTP is not being used (if $SERVICE is not HTTP and $CLIENT_AUTH is false). Otherwise it just says "no" without flagging it as an issue.
* In certificate_info() it displays additional warnings (about use of SHA-1 or subjectAltName matching) only if it can verify that HTTP is being used ($SERVICE is HTTP or $ASSUME_HTTP is true).
* In run_crime(), if compression is used, it only says " but not using HTTP" if it can verify that HTTP is not being used (if $SERVICE is not HTTP and $CLIENT_AUTH is false).
If a supported_versions extension was included in the ClientHello, then check that the version returned by the server was included in the ClientHello's supported_versions extension.
OpenSSL will respond to a TLSv1.3 ClientHello that only specifies 0304 in its supported_versions extension with a ServerHello that specifies whatever draft of TLSv1.3 it currently supports (e.g., 7F16). The result is that run_protocols() incorrectly reports that OpenSSL supports TLSv1.3 "final" in addition to whatever draft version it supports.
This PR fixes that problem by treating it as a failed connection when the ClientHello offers only 0304 and the ServerHello specifies something else (e.g., 7F16).
Performing this check is actually a requirement for clients in Section 4.2.1 of draft-ietf-tls-tls13-22. So, including this check will also help make client simulations more accurate when clients that support TLSv1.3 are added to client-simulation.txt.
The default STARTTLS_SLEEP timeout was increased to 10 seconds in
d1e7498. This caused MySQL connections to timeout. Quick fix is to
parameterize the timeout and pass in 1 again.
Better future fix is to read MySQL as binary packets, parsing the fixed
sized header, to then read the variable sized payload. Doing this will
also greatly speed up testing.
This fixes issue #914.
It partly addresses #915, supposedly the openssl binary used supports
TLS 1.3 (and the correct draft/final).
It also reduces handshakes by not trying protocols which aren't supported
on the client side.
This PR adds support for TLSv1.3 draft 22. This PR has testssl.sh operate in "middlebox compatibility mode" as described in Appendix D.4 of draft-ietf-tls-tls13-22 to maximize the chances of being able to perform a successful test even if there is a misbehaving middlebox between testssl.sh and the server being tested. Support for drafts 18 through 21 is still maintained.
This PR has been tested against a few different implementations of draft 22 that were made available shortly before draft 22 was posted.
In TLSv1.3, when responding to a HelloRetryRequest, the second ClientHello should be sent through the same socket as the first ClientHello.
This PR adds an option to socksend_tls_clienthello() to not open a socket and then uses that option in resend_if_hello_retry_request() when sending the second ClientHello.
Revert the exception from previous commit 20b38d5aa6
that TLS alerts on SSLv2 client hello won't be a proper reply -- due to a reality check,
see #908.
In order to have better debugging info the TLS alert message is printed
in clear. Messages code and text assignements was moveed to a separate
function.
Some servers like the one from the satire magazine "focus.de"
choose to return gzip encoded body during run_http_haders().
This has led sometimes to misintepretation that an IPv4 address
is present in the header.
This commit fixes that by telling the server not to want a gzipped
response and if still returned grep properly the return.
When testing servers which fall back after succeeding the STARTTLS handshake
to the underlying protocol (smtp, ftp etc.) there was often misleading output
=not available instead of notifying the user that there's a problem.
Now it is being tested in parse*serverhello() functions whether the first
by resembles a 5XX code and STARTTLS is supposed to be tested and then
passes an error code back to the caller , i.e. tls_sockets() and then
run_protocols(). Also other error code after the handshake are passed
better.
This is only an addition to the top level function run_protocols().
We might want to look into other top level functions too
In scenarios where --ssl-native was chosen AND the
openssl binary wasn't supporting the protocol
there were two warnings. This has been addressed.
Also tls_sockets() can return different values -- for now: 6 -- as
tested by the caller in run_protocols. In order to make
it more robust a fixme statement was added so that the
user becomes iat least aware of it.
After changing the logic from $PROTOS_OFFERED
contaning each protocol detected to $PROTOS_OFFERED
contaning each protocol + colon yes or no there
wasn't a change for the sanity check that no protocols
are offered. This fixes it.
This PR adds TLSv1.3 support for the negotiated protocol and cipher in run_server_preference(). This mostly addresses #893, however, run_server_preference() will not work with a TLSv1.3-only server as it will fail when trying to determine whether the server has a cipher order.
Note that with this PR run_server_preference() will not always provide consistent results when testing a server that does not support TLSv1.3 using a version of OpenSSL prior to 1.1.1. If it is determined before running run_server_preference() (using run_protocols()) that the server does not support TLSv1.3, then run_server_preference() will using OpenSSL to determine the negotiated protocol and cipher. However, if it has not yet been determined that the server does not support TLSv1.3, then run_server_preference() will use tls_sockets(), which tries to simulate OpenSSL 1.1.1. Since the list of cipher lists sent will differ the negotiated cipher will sometimes differ. In addition, when a cipher suite that uses an ephemeral ECDH key is selected, the negotiated curve is sometimes different.
This PR fixes#901. When $OPENSSL_TIMEOUT is set and mass testing is being performed, this PR changes find_openssl_binary() so that only child instances modify $OPENSSL to add the call to timeout.
This PR also changes the warning in case $OPENSSL_TIMEOUT is set and the timeout binary cannot be found so that the warning message is printed even if $WARNINGS is "off" or "batch".
There's a new socksend() function which gives up to 8% performance
benefit (LAN) as is saves 3 sed and 1 tr command in every socksend call.
It has not been put in production yet as it shows a problem in ticketbleed
check and it is late and I call it a day ;-) and resolve that later (see #902).
For performance tests it's quite handy to also have in the flat json format
a footer with time consumed for each run. fileout_json_footer() was
patched accordingly.
... from "--" to "not offered". Reason: e.g. on a white
terminal background it is hard to detect that there's
an output. Also "not offered" is more consistent with
the value of protocol checks.
When the server's response to the ClientHello message (i.e., the ServerHello, Certificate, ServerHelloDone) is split across more than one packet, the current call to tls_sockets() in run_heartbleed() only retrieves the first packet. As a result, sometimes when run_heartbleed() believes that it is reading the response to the Heartbleed payload it is actually reading the continuation of the response to the ClientHello message.
This PR fixes the problem by indicating in the call to tls_sockets() that the ephemeral key is needed. This causes tls_sockets() to continue requesting additional packets until it receives the ServerHelloDone.
There was an error in e450eb34e4
which addressed #846 which lead to the problem that an output filename
could start with a "." if no FNAME_PREFiX and if just --html, --csv
or --log or --json* was supplied.
This commit fixes the problem.
When the certificate signature algorithm is RSA-PSS and OpenSSL 1.1.1 is used $cert_sign_algo contains some trailing space characters, which causes the algorithm not to be recognized in the case statement. This PR fixes the problem by removing any trailing space characters from $cert_sign_algo.
This PR fixes a false positive in std_cipherlists(). Currently, sclient_success is not initialized (so it initially set to 0). If a server is being tested that only supports TLSv1.3, the --ssl-native option is not used, and run_protocols() is run before run_std_cipherlists(), then for many of the calls to std_cipherlists() no tests are run and so sclient_success remains at its initial value (0), which is treated as success (i.e., the server supports at least one of the ciphers in the list).
The reason this happens is that in the testing loop, the TLSv1.3 test is skipped if the list of ciphers doesn't include any TLSv1.3 ciphers (and only the "Strong encryption" test includes TLSv1.3 ciphers) and the tests for each of lower versions of SSL/TLS is skipped since it was already determined in run_protocols() that those versions weren't supported.
This PR adds a check for TLSv1.3 support to run_protocols(), checking for support for the final version of TLSv1.3 (0x0304) as well as drafts 18, 19, 20, and 21 (0x7F12, 0x7F13, 0x7F14, and 0x7F15).
File names are now auto-generated by using "-oA auto" / -oa "auto"
--similar to --csv and friends.
Also the formerly hidden switches --outFile and --outfile were added in the
help and in the manual.
Former code implied a determination of the TLS time in
every call of tls_sockets() despite the fact that the
value is only needed at one point in the run.
This removes this behaviour by introducing another global
boolean switch TLS_DIFFTIME_SET which determines whether
the additional cost will be paid or not.
The gain in execution time is a bit meager though. At
most it seems it's 1-3 seconds.
For servers with client authentication one would need to supply a x509
certificate to check session resumption by ID or ticket. This is not (yet?)
supported in testssl.sh.
This commit fixes the misleading error message so that it is clear what the
problem is.
This PR is a continuation of #833.
With additional testing with different options I encountered more places where $OPENSSL was printing "WARNING: can't open config file: /usr/local/etc/ssl/openssl.cnf" where testssl.sh was not suppressing the error message.
This PR redirects stderr to /dev/null or to $ERRFILE for several more calls to $OPENSSL in order to suppress these warning messages.
For ciphers that use the ChaCha20-Poly1305 cipher, LibreSSL shows "Enc=ChaCha20-Poly1305" in the "openssl ciphers -V" command rather than "Enc=ChaCha20(256)" and for some GOST ciphers it shows "Enc=GOST-28178-89-CNT" rather than "Enc=GOST(256)". This causes a problem for neat_list() if information is being obtained from "$OPENSSL ciphers -V" rather than from the cipher-mapping.txt file.
If testssl.sh is used with OpenSSL 1.1.1 and TLSv1.3 support is enabled, then the check for whether the server has a cipher order will always fail. The problem is that since the call to s_client doesn't specify a protocol a TLSv1.3 ClientHello will be sent. However, the call specifies a list of ciphers that doesn't include any TLSv1.3 ciphers. So, OpenSSL will fail with the error: "No ciphers enabled for max supported SSL/TLS version." The solution is to add the "-no_tls1_3" option.
This PR fixes the problem by taking advantage of the recently-added s_client_options() function. It adds a "-no_tls1_3" option whenever:
* $OPENSSL supports TLSv1.3
* The command line doesn't specify any protocol: -ssl2, -ssl3, -tls1, -tls1_1, -tls1_2, or -tls1_3.
* The command line includes the -cipher option
* The list of ciphers that will be sent doesn't include any TLSv1.3 ciphers.
Add TLSv1.3 support to run_cipher_per_proto()
Fix branch
This PR adds support for TLSv1.3 to run_server_preference(). It only provides partial support, as it only works if the support supports and earlier TLS protocol (in order to determine whether the server has a cipher order). It also will only show TLSv1.3 as the "Negotiated protocol" if $OPENSSL supports TLSv1.3.
This PR also fixes a bug in which the variable "proto" was defined as used as both a regular variable and as an array.
If run_server_preference() is performed
* against a server that supports SSLv3 and that does not have a cipher order; and
* using a version of OpenSSL that does not support SSLv3; and
* with the --mapping option set to "rfc" or "no-openssl"
then the "Negotiated cipher per proto" will not show the SSLv3 cipher since cipher[i] will be empty.
This PR addresses issue #660 for run_rc4(), ensuring that support for RC4 ciphers is detected even if no RC4 ciphers are supported with the highest protocol that the server supports.
This PR adds support for TLSv1.3 to run_std_cipherlists().
This PR also provides a partial fix for #660 - addressing the issue only for run_std_cipherlists(). Rather than testing the server once for each cipher list, it tries the server once for each protocol supported by the server. This makes the testing more robust, but adds significantly to the time it takes to perform the tests.
This PR addresses issue #660 for run_sweet32(), detecting if 3DES is used, even if it isn't used with the highest protocol version supported by the server. As with PR #854, this increased robustness comes at the expense of taking addition time to run the test.
In client_simulation_sockets() and tls_sockets(), don't work to create a SOCK_REPLY_FILE that contains the entire server's response (in cases where the response was spread across multiple packets) unless $DEBUG is at least 1.
I believe there is a typo in the second definition of DEBUG_ALLINONE. If I run testssl.sh using the -x option for bash I get the following error:
testssl.sh: line 12714: -false: command not found
In order to santize input better there's a fucntion
now which does the work . ``safe_echo()``.
It is actually the same as ``tml_out()`` but is a bit snappier.
This PR adds initial support for TLSv1.3 to tls_sockets() and for run_client_simulation(). It does not change any of the other functions test TLSv1.3. So, with the exception of run_client_simulation(), the functionality added by this PR can only be tested using the --devel option.
This PR does not include the ability to decrypt the encrypted portions of the server's response. So, it does not support functions that need to see such things as the server's certificate, status information, or extensions (other than key share).
This PR fixes the use of has_server_protocol() in two places.
Currently std_ciphersuites() only tries SSLv2 if the server is known to support SSLv2. This changes it to try SSLv2 unless the server is known to not support SSLv2.
In run_beast(), tests against the server are run to determine support for TLSv1.2, TLSv1.1, TLSv1, and SSLv3 unless the server is known to support that protocol (i.e., even if has_server_protocol() reports that the server does not support the protocol). This changes it so that a test is only performed against the server if has_server_protocol() reports that it doesn't know whether the protocol is supported.
This may not be specific to LibreSSL, but just my local setup. However, when I test using LibreSSL testssl.sh still prints the following message several times:
WARNING: can't open config file: /usr/local/etc/ssl/openssl.cnf
This PR suppresses the error message for several calls to $OPENSSL and so fixes the problem.
The logic is complete now, so that if a protocol has been detected as
supported or not supported it will save a few cycles not to test
this protocol again.
There's probably -- also besides #839 space for improvements. The
advantage of this solution is that ``has_server_protocol()`` also
marks a protocol as tested if the result of the test is negative.
MongoDB listens on port 27017 and will respond to a direct TLS
ClientHello without the need for STARTTLS. Thus, testssl.sh already
mostly works to scan a MongoDB server.
The problem is that MongoDB will also reply to an HTTP GET request with
a mocked up HTTP response, which was tricking the tool into believing it
was dealing with an HTTP server. The response looks like this:
---
HTTP/1.0 200 OK
Connection: close
Content-Type: text/plain
Content-Length: 84
It looks like you are trying to access MongoDB over HTTP on the native
driver port.
---
Thus, the detection algorith must first look for HTTP, then dig deeper
into the body looking for "MongoDB".
The tool output now says:
Service detected: MongoDB, thus skipping HTTP specific checks
As dcooper16 noted in #834 there are checks which test for openssl versions
but don't take LibreSSL in account. This adds checks to it for several
LibreSSL versions (>=2.1.x) which are known to support ``determine_trust()``
and it HAS_DH_BITS.
Moreover engine check has been improved. Older LibreSSL versions (2.1 specifically)
had different error messages, so the previous checks failed. There's also
a CMD_LINE flag now where one can switch the engine support off: NO_ENGINE .
run_renogo from #834 is still an open issue.
All three issues fixed. Terminal code were actually
tow problems: Logging in from Linux with a 256 color xterm
makes tput set AF from OpenBSD hiccup. And the detection
of not-ncurses style underline e.g. was not working under OpenBSD.
The engine fix was done by David Cooper (see #831).
There's also a name of the binary now (OpenSSL/LibreSSL) for tracking
the flavor used.
SWURL contained for historical reasons trailing blanks
for released versions.
This caused an error in pr_boldurl --> html_out which
didn't write the trailing style info and didn't close
the href tag (travis complained.)
This patch removes the trailing blank but it doesn't
fix the error.
This is an initial implementation of the idea I proposed in #791. It includes checks based on draft-ietf-tls-grease as well as checks for specific implementation bugs that have previously been encountered.
This code needs testing. While I know of one server that will fail if the ClientHello contains more than 128 ciphers, I haven't been able to test any of the other code against any servers that have the tested-for bugs.
In addition, there is a need for polishing. The tests are referred to as `--grease`, which may not be a very user-friendly name. In addition, there is no output providing information about what tests are being performed. If a server fails a test, then a warning message is printed and is sent to `fileout()`. If the server passes all of the tests, then there is just a single output of "No bugs found."
At the moment, the code is not run by default. Unless `-g` or `--grease` is explicitly included in the command line, the code in this commit is not executed.
Currently the call to `$OPENSSL s_client` to obtain the certificate returned by the server when SNI is not provided is in `certificate_info()`, which means that it is called once for each certificate found the various called to `get_server_certificates()`.
This PR moves the call to `$OPENSSL s_client` to `run_server_defaults()` so that the call is made only once, even if more than one server certificate was found.
In addition, in most cases the certificate returned by the server when SNI is not provided will already have been retrieved by `run_server_defaults()` (in rounds 8-14), in which case `$HOSTCERT.nosni` can just be copied from there rather than making an additional call to `$OPENSSL s_client`.
During protocol check if a sever answered unexpected with
closing the conenction or another malformed reply the
output was not ok as DETECTED_TLS_VERSION was empty.
This fixes it by filling the variable with a string in ``parse_tls_serverhello()``
and then check in higher level (``run_protocols()``) the content.
Also it seems that I forgot in the commit from yesterday one ``&&`` to
commit in ``run_breach()``
This PR addresses the same issue as 6bb3494d98. In its current form, fileout_insert_warning() cannot be used after fileout_section_header() has been called for the first time.
testssl.sh produces an invalid JSON file if the --json-pretty option is used with the --single-cipher option. The reason is that fileout_section_header() isn't called before run_cipher_match() calls fileout() and fileout_section_footer() is not called afterwards.
There is also a problem with MEASURE_TIME, since the "cleanup" at the end of lets_roll() is not performed.
This PR fixes these problems by adding a call to fileout_section_header() before the call to run_cipher_match() and by copying the code from the end of lets_roll() to run_cipher_match() (just before the call to exit).
Extra client side warning led to a non-valid JSON pretty output. This fixes
this bug by adding an extra object. The objects are named "clientProblem${NUMBER}".
By "extra client side" I mean extra warnings which are not happening during regular
tests -- those are no extra ones and should just warn with ``fileout()`` instead
of ``fileout_insert_warning()``.
Also some ``fileout arg1 WARN`` were patched: WARN is not a finding. It is just
a report that either on the client side something doesn't work as expected or
the server could not be checked during a particular test. WARNING doesn't
exist at all, WARn should be used instead.
Some lines where a warning output to JSON or CSV was missing, was added.
Currently, when `$DEBUG` is 2 and the connection is successful, `parse_tls_serverhello()` prints out information about the server's ephermal (EC)DH key, but nothing else. For example:
```
sending client hello... reading server hello...
dh_bits: ECDH, P-256, 256 bits
sending close_notify...
(183 lines returned)
```
This commit changes `parse_tls_serverhello()` so that information about dh_bits is only displayed if `$DEBUG` is at least 3, making it the same as for other information about the server's response.
In addition, it indents the printing of the information about dh_bits in order to better align with other information displayed at this debug level.
- add forgotten servive FTP and XMPP
- polish other services
- after TLS 1.2 run is finished run a check whether no protocol has been detected and ask the user for confirmation to proceed
In `run_protocols()` for TLS 1.2, try one set of 127 ciphers and if the result isn't a connection at TLSv1.2 then try another set of 127 ciphers before giving up and assuming that TLS 1.2 isn't supported.
FIX#786
Fixed all other occurences so that debug level 2 is showing only minimal information like rough status and errors
Better line breaks for level 2
In ``client_simulation_sockets()`` and ``tls_sockets()`` moved debug output into if statements (may save a bit of time)
Replaced "$DEBUG -eq" by "$DEBUG -ge"
Removed obsolete hb_rounds in ``run_heartbleed()``
Adjusted wide output in vulnerabilities
The data for `run_client_simulation()` currently includes two clients that send version 2.0 CLIENT-HELLO messages (see Appendix E.2 of RFC 5246). Each of the CLIENT-HELLO messages advertises support for newer protocol versions (SSLv3 in the case of IE6XP and TLSv1.0 in the case of Java 6u45). A server may reject one of these version 2.0 CLIENT-HELLO messages, or it may respond with an SSLv2, SSLv3, or TLSv1.0 ServerHello.
The current code in `client_simulation_sockets()` assumes that the server's response with be an SSLv3 or later ServerHello. So, it can support cases in which servers respond with an SSLv3 or TLSv1.0 ServerHello (once PR #800 is accepted to undo the mistake in PR #797), but not cases in which servers response with an SSLv2 ServerHello.
This PR adds code to `client_simulation_sockets()` to check if the server's response is an SSLv2 ServerHello, so that it can process such responses with `parse_sslv2_serverhello()` rather than `parse_tls_serverhello()`.
When a connection is made using SSLv3 or later, `run_client_simulation()` will show to the protocol and cipher selected for the connection. With this PR, if the connection is made using SSLv2, `run_client_simulation()` will just show "SSLv2." In the case of SSLv2, the ServerHello contains a list of all ciphers that the server and client have in common, and it is up to the client to choose one. So, if the client and server have more than one cipher in common, more information about the client would be needed to know which cipher it would choose.
`std_cipherlists()` uses `has_server_protocol()` to determine whether to test for ciphers using SSLv2. However, this was resulting in false negatives due to #759. This PR removes the `has_server_protocol()` check so that SSLv2 is checked whenever the connection attempt with the TLSv1.2 ClientHello failed.
This PR fixes the extraction of the cipher suites in SSLv2 ClientHellos in `client_simulation_sockets()`. Since `client_simulation_sockets()` can only handle SSLv3 and above ServerHellos, it removes any SSLv2 ciphers and converts the other cipher from 3-byte format to 2-byte format.
If vulnernable to POODLE and has no TLS_FALLBACK_SCSV ==> HIGH.
If only run the fallback check and it has none, still label it as MEDIUM but issue
a clear warning that test under this circumstances is incomplete.
This PR adds the same additional check to `client_simulation_sockets()` as was previously added to `tls_sockets()`. It extracts the list of cipher suites offered from each ClientHello and passes the list to `parse_tls_serverhello()` so that `parse_tls_serverhello()` can check that the cipher offered in the ServerHello was included in the ClientHello.
This assumes that a real client would abort the connection if it was presented with a cipher in the ServerHello that it didn't offer in its ClientHello.
Unfortunately, most categories were passing incorrectly, since a
tls_socket() error was interpreted as a success for many of the cases.
Now we explicitely check for the known yaSSL failure and raise a warning
that the test is inconclusive.
New output against --starttls=mysql looks like:
Testing ~standard cipher categories
-----------------------------------
NULL ciphers (no encryption) SERVER_ERROR: test inconclusive due to MySQL Community Edition (yaSSL) bug.
Anonymous NULL Ciphers (no authentication) SERVER_ERROR: test inconclusive due to MySQL Community Edition (yaSSL) bug.
Export ciphers (w/o ADH+NULL) SERVER_ERROR: test inconclusive due to MySQL Community Edition (yaSSL) bug.
LOW: 64 Bit + DES encryption (w/o export) offered (NOT ok)
Weak 128 Bit ciphers (SEED, IDEA, RC[2,4]) SERVER_ERROR: test inconclusive due to MySQL Community Edition (yaSSL) bug.
Triple DES Ciphers (Medium) offered
High encryption (AES+Camellia, no AEAD) offered (OK)
Strong encryption (AEAD ciphers) SERVER_ERROR: test inconclusive due to MySQL Community Edition (yaSSL) bug.
Partial workaround for #793
According to a discussion thread on the IETF TLS WG mail list (see https://www.ietf.org/mail-archive/web/tls/current/msg19720.html), there is at least one TLS server that will fail if the last extension in the ClientHello has contains extension_data of length 0.
Currently, `tls_sockets()` will create such a ClientHello if:
* The padding extension is included, and the length of the ClientHello without the padding data would be between 508 and 511 bytes.
* No padding extension is included, and the caller provided `$extra_extensions` in which the last extension in `$extra_extensions` is empty.
* No padding extension is included, `$extra_extensions` is empty, no ECC cipher suites are offered, and the ClientHello is for TLSv1.1 or below (in this case the next protocol extension would be that last one).
This PR avoids the server bug (in nearly all cases) by ensuring the the padding extension (when present) always contains at least one byte, and by ensuring that when the padding extension is not present that the (non-empty) heartbeat extension is the last extension.
This PR does leave one possible scenario in which the last extension would be empty. If the caller provides an `$extra_extensions` in which the last extension in `$extra_extensions` is empty, `tls_sockets()` does not add a padding extension (or a padding extension is included in `$extra_extensions`), and `$extra_extensions` includes a heartbeat extension, then the last extension in the ClientHello would be empty. This, however, is a highly unlikely scenario, and certainly there are currently no such calls to `tls_sockets()` in testssl.sh.
Another yaSSL server incompatiblity. MySQL protects against CCS
Injection by erroring if it sees to CCS requests before the handshake
completes. But instead of returning a TLS alert, it seems to error up
the stack to MySQL which returns it's own error message.
Debug output looks like:
---
sending client hello,
reading server hello
1st reply:
sending payload #2 with TLS version x03, x02:
tls_content_type: 16 | tls_protocol: 0000 | byte6: 04
2nd reply:
00000000 16 00 00 02 ff 13 04 23 30 38 53 30 31 42 61 64 |.......#08S01Bad|
00000010 20 68 61 6e 64 73 68 61 6b 65 | handshake|
0000001a
test failed, probably read buffer too small (16000002FF1304)
---
This patch adds a custom check for this MySQL specific error, as far
down in the error check path as I can.
If $PROTOS_OFFERED was empty, and thus the protocols unknown, this
function would return true for any protocol you passed it. This caused
most callers to assume TLS1.0, even if the server didn't offer it.
Instead return false and make the caller do an extra lookup.
As reported in #782, some servers will return a ServerHello with a cipher not listed in the ClientHello rather than than return an Alert, if the server does not support any of the ciphers listed in the ClientHello.
This commit modifies `tls_sockets()` to check whether the cipher in the ServerHello was one included in the ClientHello and to fail if it wasn't.
This is the simplest direct socket implementation of the MySQL STARTTLS
protocol.
This is a binary protocol, so it requires a new stream based send
(instead of the current line based send).
In `run_hpkp()` there is a call to `$OPENSSL s_client` that uses `${sni[i]}` as one of the command line options, but `sni` is not defined. My guess is that this was a copy/paste error from `run_client_simulation()`, which is the only function where an `sni` array is defined.
I am guessing that the intention was to use `$SNI` in `run_hpkp()`.
openssl/master branch now supports mysql STARTTLS in s_client
This patch adds support to call and use that s_client support to run
most, but not all (pfs, client simulation) tests.
The socket implementation is stubbed, but not yet functional.
testssl.sh is taking an educated guess which port makes sense to scan,
which one not and for which one to use which starttls handshake upfront.
This minimizes needless sscans and error messages.
Parallel mass testing mode is now not anymore experimental. To
use it a separate flag ``--mode=parallel`` was introduced. Serial
is still the default for now to avoid unexpected conditions.
Both the mode arguement and the default is subject to change.
The parallel mass testing mode can now also make use of a
nmap file. Also the functional test for nmap file was put
into a separate function and made more user safe. Open point is
that we better should use the hostname if the forward DNS record matches.
Fixed logical inconsistency: Ticketbleed was not being tested against a server with client authentication
Some variables in the beginning reordered
This PR attempts to address the outstanding issues with respect to issue #733, mainly by addressing the rules for when a certificate is obtained without SNI.
I believe I discovered the reason for issue #757: f2303a0d79.
This commit removed attempted to replace `$cbc_cipher_list_hex` (which was computed on the fly) with `$cbc_ciphers_hex` (which is static). However, the function was still using `$cbc_cipher_list_hex`, and since it wasn't being initialized to "" at the beginning of the function, the second call to `run_beast()` (to handle the second IP address) just appended to the value created by the first. Then, when the first two bytes were removed from the resulting string the result was a malformed cipher suite list, which caused `tls_sockets()` to fail.
Open issues: 1) The SNI logic 2) The fileout logic. 3) another section with ``trust_nosni -eq 4/8``
For 2): fileout is a general finding MEDIUM [1] which isn't in line now with the pr_*finding
in the section above anymore. It would make sense to punish HTTP services more than others.
Unfortunately he fileout statement cannot be moved below pr_svrty_medium/pr_svrty_high as
trustfinding_nosni hasn't been determined yet.
Fast solution would be probably to move the trustfinding_nosni section above the trustfinding
section.
Still 3) and a different trust over non-SNI makes it difficult -- e.g. Server has CN match only over
SNI but without SNI SAN matches. That's an edge case though which probably doesn't exist (like Bielefeld)
[1] That was WARN before. WARN should indicate a status of testssl that it cannot perform a check
This PR implements the suggestion from #753 for a child process in mass testing to send a signal to the parent to exit if the child encounters an error parsing its command line. At the moment, the child only sends the signal if it encounters an error that results in the `help()` function being called, but that could easily be changed (e.g., to also send a signal if `fatal()` is called in the child process).
In the case of parallel mass testing, the cleanup function needs to call `get_next_message_testing_parallel_result()` for the child that sent the signal, since otherwise the child's error message would not be displayed. Since I cannot tell which child sent the signal, I just call `cleanup()`, which displays the output of all completed child processes. Since the child process will send the signal almost immediately after starting, it can be assumed the that process that send the signal will be the last one that completed, and so its output will be displayed last (so it isn't hidden from the user).
Note that PR #753 is still needed, since there are still scenarios in which a child would not produce any JSON output, but the parent testssl.sh would not exit (e.g., the child process cannot open a socket to the server it is supposed to test). In additional, PR #754 would still be useful, since it would be more user friendly to catch the error in the mass testing file immediately (when possible) rather that partway through a potentially time-consuming testing process.
There is a bug in testssl.sh that occurs if mass testing is being performed, there is an error in the command line for one of the child tests, and either a single HTML file or a single JSON file is being created.
If mass testing is being performed and `parse_cmd_line()` detects an error in the command line for one of the child tests, then it will call `help()`, which will exit the program, resulting in `cleanup ()` being called. `cleanup ()` will call `html_footer()` and `fileout_footer()`. Since `html_header()` and `json_header()` have not yet been called, `$HTMLHEADER` and `$JSONHEADER` will both be `true, and so `html_footer()` and `fileout_footer()` will output HTML and JSON footers, even though no headers have been output.
This PR fixes the problem by having `help()` set `$HTMLHEADER` and `$JSONHEADER` to `false` so that no HTML or JSON footers are created.
A related problem is that if a single JSON file is being created, the parent process will insert a separator (a comma) into the JSON file between the outputs of each child process. However, if there is an error in one of the child process's command lines, then this child process will not produce any JSON output and so the JSON file will have two consecutive separators (commas), which is invalid according to http://jsonlint.com.
This PR provides a partial fix for the problem for parallel mass testing by checking whether a child process has created a non-empty JSON output before adding a separator to the JSON file. It leaves two unresolved problems:
* It does not fix the problem at all for `run_mass_testing()`, where the separator is added before the test with the command line error is run.
* It does not fix the problem for parallel mass testing for the case in which the first child test has a command line error.
started to use the ip parameter to decide where the dns resoultion
takes place (see #739 and #748). --ip=proxy or DNS_VIA_PROXY=true
will mean DNS resolutioni is done by proxy,
furthermore: swapped a few pr_magenta by pr_warning. Generally
testssl.sh should use for warnings a warning function and not
directly a color (we want to be flexible). There are still a few
remainders for fatal() which use bold magenta and thus I haven't
changed yet.
Also html_reserved() is being called also if no html output
is being requested. This could be fixed better probably.
This PR introduces the following changes/improvements to running mass testing in parallel:
* Continuous feedback is provided on the progress of testing, so that testssl.sh doesn't appeared to be frozen, even if it is waiting a long time for a child test to complete. [The feedback text is sent to `stderr` so that it doesn't appear in the log file if the `--logging` option is used.]
* Text sent to `stderr` by a child test is captured and then displayed in context with the appropriate test rather than being displayed at the time the error occurs. [Note that this means that if the `--logfile <logfile>` option is used (with `logfile` being the name of a file), then `logfile` will include both `stdout` and `stderr`.]
* If a test is started but it does not complete (because testssl.sh was stopped or because the test timed out), then a message is displayed indicating that the test didn't finish.
I believe that `run_mass_testing_parallel()` is either ready for use or nearly ready for use. So, I changed the program to use `run_mass_testing_parallel()` rather than `run_mass_testing()` if the `$EXPERIMENTAL` flag is set.
was received, testssl.sh tries 3 times to get memory from the server, If
the server returns different memory it's highly likely vulnerable.
(some more vulnerable devices to test against would be appreciated).
This is the default now -- all other hosts are mostly labled as OK.
Parsing SERVICE should be a little faster and more reliable
Increased needed debug level for output @ parse_tls_serverhello
Sometimes it just does not. In those case also sometimes session resumption
via tickets is supported.
This fixes the output for Session Ticket RFC 5077 and doesn't draw the wrong
conclusion from a missing lifetime hint.
It also tests for ticketbleed first whether there's a session ticket TLS
extension.
When the `--log`, `--logging`, or `--logfile <logfile>` option is being used and testssl.sh is stopped, a "printf: write error: Broken pipe" message tends to appear. From what I can tell, this is a result of the `tee` process being killed before the `cleanup ()` function completes. At the moment, `cleanup ()` doesn't write very much to `stdout`, but if parallel mass testing is performed, then `cleanup ()` may try to write the results of several previously completed tests.
This PR fixes the problem by adding the `-i` option ("ignore interrupt signals") to `tee`.
Note that I have tested this on a Linux desktop and an OS X laptop, but don't have a MS Windows computer on which to test this change.
This PR provides improvements to `run_mass_testing_parallel()`. Currently, `run_mass_testing_parallel()` treats `$MAX_PARALLEL` as the maximum difference between the number of the test whose results were last processed and the number of the most recently started test. This means that test #40 will not be started until the results of test #20 have been processed. I've encountered situations in which tests 21 though 39 have completed, but test #20 is still running, and so no new tests are started.
This PR fixes the problem by checking the status of all running child tests to see if any are complete, rather than just looking at `$NEXT_PARALLEL_TEST_TO_FINISH`. This prevents one slow child test (or a few slow child tests) from slowing up the entire mass testing process.
This PR also changes the basis for determining whether a slow child process should be killed. Rather than waiting `$MAX_WAIT_TEST` seconds from the time that the parent started waiting (which is rather arbitrary), it kills the process if `$MAX_WAIT_TEST` seconds have passed since the child test was started. Given this, and that the above change makes it less likely that a slow child test will slow up the overall testing, I increased `$MAX_WAIT_TEST` from 600 seconds to 1200 seconds.
I added some `debugme` statements that provide feedback on the status of testing, but in non-debug mode there may be a perception issue. If one test (e.g., test #20) is very slow, testssl.sh will not display any results from later tests until the slow test finishes, even though testssl.sh will continue running new tests in the background. The user, seeing no output from testssl.sh for an extended period of time, may think that testssl.sh has frozen, even though it is really just holding back on displaying the later results so that the results will be displayed in the order in which the tests were started.
There is a comment in the `run_client_simulation()` function that says "FIXME: printf formatting would look better, especially if we want a wide option here."
This PR is an attempt at addressing that FIXME and adding a wide option. The proposed wide option prints the same information as the non-wide option, just with the columns aligned. I didn't add any of the additional information that is displayed by other functions in wide mode, since I thought that made the output too wide.
-H is now --heartbleed instead of --headers,
-B is now --breach instead of --heartbleed,
-T is now --ticketbleed (was previously --breach)
bugs fix for run_ccs_injection() where the tls protocols wa not properly passed to the ClientHello
Made use of already determined protocol ( this time only from determine_optimal_proto() ) ==> we shpould use this in run_protocols() too!)
for run_ccs_injection + run_ticketbleed(). For achieving this determine_optimal_proto() needed to be modified so that it adds a protocol
to PROTOS_OFFERED (all_failed is now boolean there)
added two easy functions for converting dec to hex
sockread_fast() is for testing which should make socket erads faster -- albeit it could potentially block the whole thing
`emphasize_stuff_in_headers()` only adds color to the text being printed to the terminal if `$COLOR` is 2. So, the same should be the case for the HTML output.
This PR addresses issue #616, changing `run_cipher_match()` so that only those ciphers that are available are shown, unless the `--show-each` flag has been provided.
It also fixes a problem where the signature algorithm isn't being shown, even if `$SHOW_SIGALGO` is true.
This PR just addresses some places where quotes need to be used to avoid word splitting in case the referenced file, or path to the file, contains space characters.
The previous fix did not work if testssl.sh was found via `$PATH`. This seems to work in all cases. If testssl.sh is found via `$PATH` or if the command line includes a path, then `which` returns a non-empty response; otherwise, `$0` does not include any path, but one needs to be provided, so `$RUN_DIR/$PROG_NAME` is used.
Using "$0" as the name of the executable seems to work as long as "$0" contains a directory name (e.g, "workingfiles/testssl.sh"), but not if it is just the name of the executable (e.g., "testssl.sh"). Specifying "$RUN_DIR/$PROG_NAME" seems to work in both cases, since if "$0" doesn't contain any path information, `$RUN_DIR` is `.`
Use the suggestion "If you want to print the argument list as close as possible to what the user probably entered" from http://stackoverflow.com/questions/10835933/preserve-quotes-in-bash-arguments to create `$CMDLINE` and to print the command lines in `run_mass_testing()` and `run_mass_testing_parallel()`.
This PR addresses issue #702. Rather than create the command line for each child process in `run_mass_testing()` as a string, it creates it as an array, with each argument being a separate element in the array. This was done based on http://mywiki.wooledge.org/BashFAQ/050.
The printing of each child's command line done based on http://stackoverflow.com/questions/10835933/preserve-quotes-in-bash-arguments.
The `$CMDLINE` string remains unchanged, even though it isn't entirely "correct," since http://jsonlint.com/ complains if the "Invocation:" string contains backslashes.
I was doing some testing on my extended_tls_sockets branch and discovered that it was not fully working since the `TLS13_KEY_SHARES` array was empty. According to https://lists.gnu.org/archive/html/bug-bash/2012-06/msg00068.html, there is an issue when trying to initialize a global array inside a function. (The current code initializes `TLS12_CIPHER`, `TLS_CIPHER`, and `TLS13_KEY_SHARES` within `get_install_dir()`, since tls_data.txt is read in that function.) In fact, according to http://stackoverflow.com/questions/10806357/associative-arrays-are-local-by-default, in order to initialize a global variable in a function, one needs to provide the `-g` option, which was only added in Bash 4.2.
This PR seems to fix the problem by moving the reading of tls_data.txt to the main body of the code rather than reading it within the `get_install_dir()` function.
If I understand correctly how `run_hpkp()` should be displaying the list of Backups, the problem shown in the photo attached to #696 occurred because the dangling SPKIs should have been printed on the next line. This PR fixes this by changing the code that prints out the "good" backups to include a newline after printing the CA's name.
Another problem this PR fixes is that `tm_italic()` is being called instead of `pr_italic()`, meaning that the italicized text is being printed to the terminal, but is not being included in the HTML output.
Finally, this PR fixes errors in the definitions of `tmln_italic()` and `prln_italic()`. `tmln_italic()` calls `outln()` and `prln_italic()` calls `tmln_out()` instead of the reverse.
This PR fixes issue #695 by changing the call to `out_row_aligned_max_width()` so that the length of the flag is considered in determining the length of the first line of the output.
This PR fixes two issues with HTML generation that were introduced by a commit on March 31, 2016, "[count_ciphers is now un-sed'ed, minor improvements](a480e5f699)."
The first is that in `std_cipherlists()`, `[[ $DEBUG -ge 1 ]] && outln " -- $1" || outln` was changed to `[[ $DEBUG -ge 1 ]] && outln " -- $1" || outln`. The result being that in the HTML output, all of the tests from `run_std_cipherlists()` appear on the same line. This PR changes the line to:
```
[[ $DEBUG -ge 1 ]] && tm_out " -- $1"
outln
``
so that the line break is added to the HTML output, but the debugging information is not.
The second problem is that the commit on March 31 moved the call in main to `html_header()` until after the calls to `get_install_dir()`, `find_openssl_binary()`, `mybanner()`, `check4openssl_oldfarts()`, and `check_bsd_mount()`. The problem is that each of these functions may call an output function that will call `html_out()`.
If `html_out()` is called before `html_header()` and the command line contains `--htmlfile <htmlfile>`, then "htmlfile" will be written to before `html_header()` is called and then `html_header()` will warn that "htmlfile" already exists and then exit the program.
If `html_out()` is called before `html_header()` and the command line contains `--html`, then anything send to `html_out()` before `html_header()` is called (such as the banner) will not appear in the HTML file.
I did some testing with http://jsonlint.com/ and discovered a missing comma when massing testing is being performed and a single JSON file is being created.
In `run_server_defaults()` the variable `success` is defined twice, once an an ordinary variable and once as an array. The PR removes the incorrect definition. It also removes the definitions of some variables that are no longer used and reorganizes the definitions so that each line has only one variable type.
I also noticed a typo later in `run_server_defaults()` and corrected it.
This PR introduces the environment variable `CHILD_MASS_TESTING`, and uses it as an indicator that testssl.sh is running as a child within mass testing rather than using the `$APPEND` flag. It also makes a number of other changes to make the handling, of HTML, CSV, JSON, and log files consistent, and it fixes a number of bugs related to the generation of these files when mass testing is being performed.
Please let me know if you disagree with any of the changes in this PR, or if you would prefer that it be broken up into multiple smaller PRs.
Some of the changes are as follows:
- When the `$APPEND` flag is true, all of these files are appended to and headers and footers are omitted. (Perhaps this should be changed. Appending to a log file isn't an issue, but appending to a JSON or HTML file without including headers or footers seems to just create an improperly formatted file).
- Following the code in `prepare_logging()`, an error is printed and the program stops if the `$APPEND` flag is false and one of the files to be written to already exists.
Some of the bugs fixed:
Creating log files did not work with mass testing:
- If `--logfile <logfile>` is used, then the parent and each child try to write to "logfile".
- If `--logging` is used, then a log file is created for each child, but an oddly-named log file is also created for the parent. The one created by the parent contains the entire output.
Plain JSON files:
- When `--jsonfile <jsonfile>` is run, there is no comma separating the final finding for one child and the first finding for the next child.
Pretty JSON files:
- When `--jsonfile-pretty <jsonfile>` is called without mass testing, the "target host" line is empty, since `$NODE` has not yet been set.
- When `--jsonfile <jsonfile>` is run with mass testing, there is no comma separating the final finding for one child and the first finding for the next child. In addition, `fileout_pretty_json_banner()` is never called, and the entries for individual tests have insufficient information to determine what is being tested (it lists "service" and "ip", but not port number).
For the final issue, when mass testing is being performed and all output is being placed in a single file, I have the parent call `fileout_pretty_json_banner()`, but tell `fileout_pretty_json_banner()` to not include a "target host" or "port", but then have each child include a "target host" or "port" (when the "service" and "ip" are being printed).
This PR improves `out_row_aligned_max_width()` in a few ways:
* It makes better use of bash's string manipulation capabilities in order to simplify the function.
* It improves the function's performance. One of the most costly parts of `out_row_aligned_max_width()` was the while loop to print each entry in the text. Since there is only one place in the code where the the entries are not all printed the same ways (the list of supported curves printed by `run_pfs()`), the PR changes `out_row_aligned_max_width()` to just return a plain text string, which the calling function prints in the appropriate way. For the curves printed by `run_pfs()`, a new function, `out_row_aligned_max_width_by_entry()` takes care of getting the output from `out_row_aligned_max_width()` and then printing each entry appropriately.
* The PR also introduces a trick so that when the TLS extensions are printed, the text for an extension won't get split across two rows. It does this by replacing the space charters within the text for an extension with "}", formatting the result with `out_row_aligned_max_width()`, and then converting the "}" back to space characters.
`$OPENSSL_LOCATION` is used in `fileout_pretty_json_banner()`, `html_banner()`, `mybanner()`, and `prepare_logging()`, but the value of `$OPENSSL_LOCATION` is populated in `mybanner()`. This is usually okay, since `mybanner()` is always called before the other three functions are called. However, if `$QUIET` is `true`, then `mybanner()` returns immediately, without populating `$OPENSSL_LOCATION`, even though the value of `$OPENSSL_LOCATION` may be needed by one or more of the other functions.
This PR addresses this problem by populating `$OPENSSL_LOCATION` in `find_openssl_binary()` rather than `mybanner()`.
With the commit made on March 26, "partly (1/2) fixing #653," an HTML banner isn't added to the HTML files anymore. A banner should be added to the top of the HTML file if mass testing is being performed and a separate HTML file is being created for each test.
The `$APPEND` flag being `true` is an indicator that mass testing is being performed and that this is one of the individual tests being run. Given that `$APPEND` is `true`, `$HTMLHEADER` being `true` indicates that testssl.sh is creating the file name for the HTML output. So, it is when both flags are `true` that the HTML banner should be created.
`std_cipherlists()` does not include line breaks between tests in the output to the terminal when `$DEBUG` is 1, and it does not include line break between tests in the HTML output whenever `$DEBUG` is greater than 0.
In `create_client_simulation_tls_clienthello()` the variable `sni_extension_found` should be set if the ClientHello includes an SNI extension. Instead it was being set if and only if the ClientHello included some extension other than SNI.
This bug wasn't detected before for two reasons:
* It is rare to have a ClientHello that includes an SNI extension, but no other extensions.
* The code still works correctly if `sni_extension_found` is set even if there is no SNI in the ClientHello.
So, the bug only creates a problem if the browser's ClientHello include an SNI extension and no other extensions (see "BingPreview Jun 2014" in the client_simulation branch).
