Issue
I'm writing code with the curl_easy interface of libcurl, and am having a hard time
getting it to reuse connections. Details below.
curl --version
curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.1 zlib/1.2.11
libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
Release-Date: 2018-01-24
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3
pop3s rtmp rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB
SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL
My code runs through a series of unit tests all of which have to connect to
the same host over H2, so I've set up persistent connections. I call
curl_easy_init once, then all the global curl_easy_setopt values, then in a
loop, I make the requests by calling curl_easy_setopt to set post fields,
url, content length, and then curl_easy_perform.
I only call curl_easy_cleanup after all requests have been made.
I have the following options set:
CURLOPT_FOLLOWLOCATION => 1
CURLOPT_SSL_VERIFYPEER => 1
CURLOPT_SSL_VERIFYHOST => 2
CURLOPT_SSLVERSION 7<<16
CURLOPT_HTTP_VERSION = 4
CURLOPT_TCP_FASTOPEN = 1
CURLOPT_TCP_KEEPALIVE = 1
CURLOPT_ACCEPT_ENCODING = ""
CURLOPT_TRANSFER_ENCODING = 1
CURLOPT_POST = 1
CURLOPT_DNS_CACHE_TIMEOUT = 0
CURLOPT_VERBOSE = 1
CURLOPT_SSLCERT
CURLOPT_SSLKEY
CURLOPT_CAINFO
Verbose output looks like this:
* TCP_NODELAY set
* TCP_FASTOPEN_CONNECT set
* Connected to XYZ () port 443 (#0)
* found 3 certificates in /data/certs/ca_chain.pem
* found 401 certificates in /etc/ssl/certs
* ALPN, offering h2
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification OK
* server certificate status verification SKIPPED
* error fetching CN from cert:The requested data were not available.
* common name: (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject:
* start date: Mon, 13 Jul 2020 08:04:30 GMT
* expire date: Mon, 20 Jul 2020 08:05:00 GMT
* issuer: C=US,ST=MA,L=Cambridge,O=XXXXX,CN=YYYYY
* compression: NULL
* ALPN, server accepted to use h2
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade:
len=0
* Using Stream ID: 1 (easy handle 0x9036570)
> POST /foo/ HTTP/2
<Request>
* We are completely uploaded and fine
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
<Response>
* Connection #0 to host XYZ left intact
* Found bundle for host XYZ: 0xe141520 [can multiplex]
* Connection #0 is still name resolving, can't reuse
* Trying IP...
* TCP_NODELAY set
* TCP_FASTOPEN_CONNECT set
* Connected to XYZ () port 443 (#1)
* found 3 certificates in /data/certs/ca_chain.pem
* found 401 certificates in /etc/ssl/certs
* ALPN, offering h2
* ALPN, offering http/1.1
* SSL re-using session ID
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification OK
* server certificate status verification SKIPPED
* error fetching CN from cert:The requested data were not available.
* common name: (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject:
* start date: Mon, 13 Jul 2020 08:04:30 GMT
* expire date: Mon, 20 Jul 2020 08:05:00 GMT
* issuer: C=US,ST=MA,L=Cambridge,O=XXXXX,CN=YYYYY
* compression: NULL
* ALPN, server accepted to use h2
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade:
len=0
* Using Stream ID: 1 (easy handle 0x9036570)
<Request>
* We are completely uploaded and fine
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< vary: accept-encoding
< content-encoding: gzip
< content-type: text/plain
< date: Mon, 13 Jul 2020 19:30:36 GMT
< server: abcdefg
<Response>
* Connection #1 to host XYZ left intact
* Found bundle for host XYZ: 0xe141520 [can multiplex]
* Connection #0 is still name resolving, can't reuse
* Connection #1 is still name resolving, can't reuse
And so on...
It fairly soon gets to this:
* Connection cache is full, closing the oldest one.
* Closing connection 0
* Connection #5 to host XYZ left intact
This closing of the oldest connection continues from here on in until either everything is done, or a TLS handshake error puts the connection into an inconsistent state.
Any idea what I can do to get curl to reuse the H2 connection to the host?
Thanks for your time.
Solution
The issue is with CURLOPT_TCP_FASTOPEN. Turning off that option will allow connection reuse. FAST_OPEN is only needed if using multiple TCP connections to avoid a roundtrip during handshake.
Traditional TCP handshake is a 3-way protocol.
- The initiator sends a SYN packet to the peer,
- The peer responds with a SYN-ACK packet to the initiator,
- Finally the initiator sends its own ACK.
After sending the ACK, the initiator sends the first data packet. In all, 4 packets to send data.
With FAST_OPEN, data transfer (on connections after the first connection) can begin from the first packet. It uses a TCP Cookie (TFO) to establish that the two hosts have recently successfully connected to each other.
See https://en.wikipedia.org/wiki/TCP_Fast_Open for more information.
Answered By - bluesmoon
