From 647aeae20528a27caf8f0938e959ecc40305cb74 Mon Sep 17 00:00:00 2001
From: Dirk <dirk@testssl.sh>
Date: Wed, 9 Apr 2025 20:00:47 +0200
Subject: [PATCH] Update docu and (futile) perl script
---
etc/client-simulation.wiresharked.md | 28 +++++++++++++++-------------
utils/update_client_sim_data.pl | 6 ++++--
2 files changed, 19 insertions(+), 15 deletions(-)
diff --git a/etc/client-simulation.wiresharked.md b/etc/client-simulation.wiresharked.md
index 5d5e69b..c7a2c34 100644
--- a/etc/client-simulation.wiresharked.md
+++ b/etc/client-simulation.wiresharked.md
@@ -1,6 +1,6 @@
-The file `client-simulation.wiresharked.txt` contains client handshake data manually harvested from a network capture and displayed by Wireshark.
-The content needs to be added to `client-simulation.txt` which other part comes from the SSLlabs client API via `update_client_sim_data.pl`
-The whole process is manual but not too difficult.
+The file `client-simulation.wiresharked.txt` contains client handshake data manually harvested from a network capture and displayed by wireshark.
+testssl.sh uses the file `client-simulation.txt`. Previously we queried the SSLlabs client API via `update_client_sim_data.pl` and added the data into `client-simulation.txt`. For quite some while we don't use the data from SSLlabs anymore as they haven't changed and they are outdated. That reduces the work to editing `client-simulation.wiresharked.txt` and `client-simulation.txt`.
+
## Instructions how to add a client simulation:
@@ -9,19 +9,21 @@ The whole process is manual but not too difficult.
* Make sure the client traffic is specific: For just "Android" do not use an Android browser! Be also careful with factory installed Google Apps, especially on older devices as they might come with a different TLS stack.
* Stop recording.
* If needed sort for ClientHello.
-* Look for the ClientHello which matches the source IP + destination you had in mind. Check the destination hostname in the SNI extension so that you can be sure, it's the right traffic.
-* Edit `client-simulation.wiresharked.txt` and insert a new section, preferably by copying a previous version of the client from it.
+* Look for the ClientHello which matches the source IP + destination you had in mind. Check the destination hostname in the SNI extension so that you can be sure it's the right traffic.
+* Edit `client-simulation.wiresharked.txt` and insert a new section, preferably by copying a previous version of the client.
* Edit the *names* accordingly and the *short* description. The latter must not contain blanks.
-* Retrieve *handshakebytes* by marking the "TLS 1.x Record Layer" --> Copy --> As a hex stream.
-* For *ch_ciphers*: mark "Cipher Suites" --> Copy --> As a hex stream, remove any leading GREASE ciphers (?a?a) and supply it to `~/utils/hexstream2cipher.sh`. For consistency reasons it is preferred you remove the TLS 1.3 ciphers before which start with TLS\*.
-* *ciphersuites* are TLS 1.3 ciphersuites. You can identify them as they currently are normallky like 0x13\*\*. Retrieve them from above see `~/utils/hexstream2cipher.sh`. They start with TLS\*.
+* Retrieve *handshakebytes* by marking the *TLS 1.x Record Layer* --> Copy --> As a hex stream.
+* For *ch_ciphers*: mark *Cipher Suites* --> Copy --> As a hex stream and supply it to `~/utils/hexstream2cipher.sh`. The last line contains the ciphers which you need to copy. For consistency reasons it is preferred you remove the TLS 1.3 ciphers before which start with TLS\*. . The GREASE "ciphers" (?a?a) which you may see in the very beginning don't show up here.
+* *ciphersuites* are TLS 1.3 ciphersuites which you omitted previously. You can identify them as they currently are normallky like 0x13\*\*. Retrieve them from above see `~/utils/hexstream2cipher.sh`. As said, they start with TLS\*.
+* For *curves* mark the *Supported Groups* TLS extension --> Copy --> As a hex stream, remove any leading GREASE ciphers (?a?a) and supply it to `~/utils/hexstream2curves.sh`. Copy the last line into *curves*.
* Figure out *protos* and *tlsvers* by looking at the *supported_versions* TLS extension (43=0x002b). May work only with recent clients. Be careful as some do not list all TLS versions here (OpenSSL 1.1.1 listed only TLS 1.2/1.3).
-* Adjust *lowest_protocol* and *highest_protocol* accordingly.
-* For *curves* mark the "supported groups" TLS extension --> Copy --> As a hex stream, remove any leading GREASE ciphers (?a?a) and supply it to `~/utils/hexstream2curves.sh`.
-* Retrieve *alpn* by looking at the "alpn" TLS extension 16 (=0x0010).
+* Adjust *lowest_protocol* and *highest_protocol* accordingly (0301=TLS 1.0, 0302=TLS 1.1, 0303=TLS 1.2, 0304=TLS 1.3)
* Review TLS extension 13 (=0x000d) "signature_algorithm" whether any SHA1 signature algorithm is listed. If not *requiresSha2* is true.
* Leave *maxDhBits*/*minDhBits* and *minRsaBits*/*maxRsaBit* at -1, unless you know for sure what the client can handle.
-* When using wireshark, copy also the ja3 and ja4 values accordingly, see e.g. like *java80442*. This could be used in the future.
-* Figure out the *services* by applying a good piece of human logic or have a look at a different version of the client. A (modern) browser is probably "HTTP", OpenSSL or Java "ANY" whereas Thunderbird supports a variety of protocols.
+* Retrieve *alpn* by looking at the *application_layer_protocol_negotiation* TLS extension 16 (=0x0010).
+* When using wireshark, copy also the ja3 and ja4 values accordingly (copy --> value), see e.g. like *java_80442*. This could be used in the future.
+* Figure out the *services* by applying a good piece of human logic. Or have a look at a different version of the client. Any (modern) browser is probably "HTTP", OpenSSL or Java "ANY" whereas mail clients as Thunderbird support a variety of protocols.
* When you're done copy your inserted section from `client-simulation.wiresharked.txt` into `client-simulation.txt`.
* Before submitting a PR: test it yourself! You can also watch it again via wireshark.
+
+
diff --git a/utils/update_client_sim_data.pl b/utils/update_client_sim_data.pl
index 700f553..65774ac 100755
--- a/utils/update_client_sim_data.pl
+++ b/utils/update_client_sim_data.pl
@@ -1,5 +1,7 @@
#!/usr/bin/perl
+### This is futile as the data havent changed for an eternity
+
use strict;
use Data::Dumper;
use JSON;
@@ -15,8 +17,8 @@ $ossl =~ s/\R//g; # remove LFs
die "Unable to open $ossl" unless -f $ossl;
my $ossl = "$ossl" . " ciphers -V 'ALL:COMPLEMENTOFALL:\@STRENGTH'";
-# we get all data from here
-my $json = `curl 'https://api.dev.ssllabs.com/api/v3/getClients'`;
+# we got all data from here : 'https://api.dev.ssllabs.com/api/v3/getClients'. New URL is below but the output hasn't changed either
+my $json = `curl 'https://api.ssllabs.com/api/v3/getClients'`;
foreach my $line ( split /\n/, `$ossl`) {
my @fields = split /\s+/, $line;