Fix S3 not interruptting

[RossComputerGuy]

Motivation

Fixes #11678

Context

This is my attempt to try and get S3 transfers to interrupt properly. The only problem now is retrying doesn't interrupt but it seems the retry strategy doesn't work if you don't already have a connection.

---

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

[NaN-git]

AFAIK Curl transfers are interruptible already.

checkInterrupt() throws an exception, i.e. it shouldn't be used within a C framework, if there should be a proper shutdown.

IMHO this patch does not make S3 transfers interruptible because it can't interrupt running transfer.

[edolstra]

This PR didn't work for me, at least this command:

nix eval --impure --expr 'builtins.fetchurl "s3://highlighter-public/405b3ba02ccb2f07ec6aba9f233d4194.mp4?region=ap-southeast-2"'

still doesn't respond to Ctrl-C.

Note that src/libstore/filetransfer.cc isn't involved in S3 transfers since aws-sdk-cpp uses libcurl directly. Since S3Helper::getObject() doesn't (AFAIK) have a callback to check for interrupts, the only way I can think of is to pass it a wrapper std::stringstream that calls checkInterrupt(). (That still wouldn't be perfect, since it would only check for interrupts when it receives data, but it's better than nothing.)

[RossComputerGuy]

Yeah, I think the best option might actually be to implement an HTTP client using AWS's API for one and then we could throw some checkInterrupts in it. I tried the PR with that example expression but connecting to my S3 server but it timed out. During the timeout, Ctrl-C didn't work and the retry strategy wasn't even called during that.

[NaN-git]

I have this code, which might fix the issue. But I cannot test it. It's not ready to be merged yet, e.g. aborting multipart uploads might also be possible with transferManager->CancelAll(). Also act.progress() could throw an exception, which is not caught.

[edolstra]

Apparently aws-cpp-sdk has a HttpClientFactory interface that allows overriding the HTTP client (see https://sdk.amazonaws.com/cpp/api/LATEST/aws-cpp-sdk-core/html/class_aws_1_1_http_1_1_http_client_factory.html). Maybe we can use that to make it use libstore's FileTransfer class, which hopefully would make it interruptable automatically.

[RossComputerGuy]

I've started overriding the HttpClientFactory, it does seem like the right direction. I just need to know how to push a FileTransfer request.

[RossComputerGuy]

I pushed my current progress on it, still figuring out how FileTransfer works.