Why is tokio multi threaded runtime so much slower than current thread?

[bennetthardwick]

I was working on a project that downloads large files with reqwest when I noticed that the multi threaded runtime was downloading files significantly slower than the current thread runtime. I understand that there is going to be some extra overhead with multiple threads - but how significant a drop in performance would be reasonable?

I've written some simple benchmarks and put the code up here: https://github.com/bennetthardwick/reqwest-benches.

With a local file server that serves a 16 GiB/s I get the following speeds:

bench	speed
reqwest_current_thread	3276 MiB/s
reqwest_multi_thread	680 MiB/s
go	2804 MiB/s

The code looks like this with the only difference between the two being flavor = "current_thread" set for one.

#[tokio::main]
async fn main() {
    println!("Starting reqwest_multi_thread...");

    let mut stream = reqwest::get("http://localhost:3000")
        .await
        .unwrap()
        .bytes_stream();

    let mut byte_count: usize = 0;
    let start = Instant::now();

    while let Some(next) = stream.next().await {
        let bytes = next.unwrap();
        byte_count += bytes.len();
        black_box(bytes);
    }

    // server will return 16GiB of ones
    assert_eq!(byte_count, 16 * 1024 * 1024 * 1024);

    println!(
        "reqwest_multi_thread ran at an average of {} MiB/s",
        (byte_count as f64) / 1024. / 1024. / start.elapsed().as_secs() as f64
    );
}

Is this kind of slowdown to be expected or am I doing something wrong somewhere? I would appreciate any help.

[Darksonn]

The main thread is not a worker thread when using the multi-thread runtime. Wrap the body in tokio::spawn to actually execute the code on a worker thread.