Issue
When you connect to an unknown host using SSH with most clients, they detect what I assume is an unknown/self-signed certificate, and ask whether you want to accept the connection anyway.
For fun purposes I decided to write a local file transfer program, and for learning purposes I'm now adding encryption to it.
I would like to achieve the same behavior as those SSH clients.
For starters, I'm looking at OpenSSL's echo demo.
The 2 questions come to mind:
- If certificates must have a domain name, how would the client make it's certificate? Does it create a certificate for each network interface's IP, or is the domain name simply ignored?
- How is the certificate transferred to the client?
To be clear, I'm not talking about SSH certificate user authentication. I'm talking about handshake and data encryption. Essentially replicating how SSH does trust-on-first-use and prompt to verify
I'm also aware that SSH does not use SSL under the hood. But some implementations do (or did) make use of OpenSSL for cryptography, and I'm approaching this (perhaps wrongly) from a perspective of "use SSL so you lower your chances of making stupid cryptography mistakes. a few tweaks are still better than doing it all from scratch".
The question pretty much ends here but I'm going to attempt my own answer, as perhaps it may make it easier if answerers want to tell me where I'm wrong. And to try to show this isn't a low-effort question with zero research behind it.
For the first question, I think the domain name may simply be ignored.
It's not particularly relevant in the context of SSH as it can change with time (e.g. DHCP), and malicious actors on the network can spoof IPs anyway.
In the echo example (second link, to current version as of writing this question) it seems like they essentially bypass the domain name checking (certificate's 'Common Name' as they call it in the readme, to which they assign 'localhost') by calling SSL_set_tlsext_host_name and SSL_set1_host with the same argument (the hostname the client is connecting to, which may be anything).
For the second question, I think there are 3 possible options.
First: SSL handshake fails -> client now has certificate and asks user if they want to store it -> client stores certificate, restarts connection -> ignores certificate check if the certificate is the same as the one stored -> SSL session established.
Second: SSL handshake fails -> client asks server for the CA cert it used to self-sign the certificate (happens in the unprotected socket) -> user-permitting, stores that cert in a truststore -> restarts SSL session and now the certificate works because it can be verified.
Third: SSL handshake succeeds because SSL_CTX_set_verify is called with SSL_VERIFY_NONE -> if verification failed, either option 1 happens without reconnecting, or option 2 happens but the CA cert is transferred through the 'secure' channel.
All of these are by definition unsecure to a MITM attack unless the user makes sure the certificate is the same as it is in the server. But this is clearly also the case for SSH clients.
Solution
Question 1
OpenSSL does not check the domain names on certificates by default. As such, no changes are required to the demo code to make it work with IPs or even arbitrary domains. In fact, if you do want to have domain checking you may have to implement it, more information in this post.
notes
In the echo example (second link, to current version as of writing this question) it seems like they essentially bypass the domain name checking (certificate's 'Common Name' as they call it in the readme, to which they assign 'localhost') by calling SSL_set_tlsext_host_name and SSL_set1_host with the same argument (the hostname the client is connecting to, which may be anything).
This is not true, as the documentation for SSL_set_tlsext_host_name states:
The behaviour of SSL_get_servername() depends on a number of different factors. (...)
On the client, before the handshake
If a servername has been set via a call to SSL_set_tlsext_host_name() then it will return that servername.
If one has not been set, but a TLSv1.2 resumption is being attempted and the session from the original handshake had a servername accepted by the server then it will return that servername.
Otherwise it returns NULL.
(...)
So it is entirely self-contained for the current application where we have control over the client and the server.
For SSL_set1_host the case is different as the documentation states it is used for actually checking against subject CommonName. So for purposes of skipping the checks, we may simply not call it or call it with a NULL argument.
Question 2
For this particular intended use, the certificate will simply be transferred to the client with the TLS handshake. There is no need to transfer it beforehand in the socket or custom-encrypt it or do any such inanity.
The TLS handshake gets the server's certificate automatically (unencrypted for TLS1.2, encrypted for TLS1.3). And we have access to it through OpenSSL.
I first assumed the handshake failing would prevent us from being able to get our hands on the certificate. This isn't the case as we have the SSL_CTX_set_verify function, allowing us not only access to the certificate, but also so at a time of our choosing.
By passing SSL_VERIFY_NONE to the mode parameter, we can keep OpenSSL from terminating the session due to a self-signed certificate (or a bad certificate in general that it can't match to any of the CAs in the system).
Afterwards we would have the ability to prompt the user, and terminate the session in the event of a negative response.
A better way however is to pass SSL_VERIFY_PEER and a callback function. That way we can determine the certificate is bad and terminate the connection (saving the certificate). After prompting the user, on a positive response we may then restart the handshake where the certificate will then be accepted.
The advantage to this approach over the first one is that we won't have an open insecure session that could be dangerous if we don't make absolutely sure there's no data going in or out as we await the user's response.
Answered By - TrisT Answer Checked By - Gilberto Lyons (WPSolving Admin)