feat(taskworker) Add concurrent worker

[markstory]

Move the taskworker process to be a multiprocess concurrent worker. This will help enable higher CPU usage in worker pods, as we can pack more concurrent CPU operations into each pod (at the cost of memory).

The main process is responsible for:

• Spawning children
• Making RPC requests to fill child queues and submit results.

Each child process handles:

• Resolving task names
• Checking at_most_once keys
• Enforcing processing deadlines
• Executing task functions

Instead of using more child processes to enforce timeouts, I've used SIGALRM. I've verified that tasks like

@exampletasks.register(name="examples.infinite", retry=Retry(times=2))
def infinite_task() -> None:
    try:
        while True:
            pass
    except Exception as e:
        print("haha caught exception", e)

Do not paralyze workers with infinite loops.

When a worker is terminated, it uses an Event to have children exit, and then drains any results. If there are tasks in the _child_tasks queue will not be completed, and instead will sent to another worker when the processing_deadline on the activations expires.

[codecov]

Codecov Report

Attention: Patch coverage is 87.98450% with 31 lines in your changes missing coverage. Please review.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/sentry/taskworker/worker.py	80.39%	30 Missing ⚠️
tests/sentry/taskworker/test_worker.py	99.04%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #83254       +/-   ##
===========================================
+ Coverage   42.17%   87.56%   +45.39%     
===========================================
  Files        9440     9471       +31     
  Lines      535567   537610     +2043     
  Branches    21142    21142               
===========================================
+ Hits       225853   470754   +244901     
+ Misses     309356    66498   -242858     
  Partials      358      358