Expose job controller's workqueue rate limiting configs

[roteme-runai]

Issue:

The mpi-operator does not handle a large number of MPI jobs effectively. Specifically, when creating a significant number of jobs (e.g., 100), the operator experiences delays in job management, leading to the following issues:

• Excessive time taken to create all the pods required for the jobs.
• Significant delay between a job's completion and the corresponding status update.
• Delayed cleanup of pods (per CleanPodPolicy) due to the lag in status updates.

Root Cause:

The controller uses a workqueue with a default rate limiter configuration that is not adjustable via operator options. This is in contrast to other load-related configurations (e.g., threadiness, qps, etc.), which are user-configurable. The low default rate-limiting settings result in insufficient parallel processing, thereby delaying job handling.

Proposed Solution:

To address this, I propose exposing the controller's rate-limiting settings as user-configurable options. This change would allow users to adjust the rate limiter based on their specific usage requirements, expected scale, and system capabilities.
The solution has been tested and verified in a production environment and has demonstrated improved handling of bigger-scale MPI jobs.

Backporting Request:

If this fix is approved, I kindly request its inclusion in a new release, ideally in versions from v0.6 onward.

Thank you for your time!

[mlsorensen]

LGTM. Confirmed the defaults for the new flags match the previous default limit and burst, shouldn't change behavior.

[mchmarny]

@tenzen-y @jlewi @rongou can you help review this?

[jlewi]

I'm not involved anymore so I'm probably not the right person to be reviewing this.

[rongou]

/lgtm
/approve

@terrytangyuan @alculquicondor

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mlsorensen, rongou

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rongou]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[alculquicondor]

/lgtm

[google-oss-prow]

New changes are detected. LGTM label has been removed.

[roteme-runai]

@rongou, @alculquicondor re-generated since verify-generate failed.
There are also failing e2e tests, but they are failing for other, older PRs too. I'm also unable to build the intel images locally to debug.
How do we proceed?

[alculquicondor]

@tenzen-y has been trying to fix that failure without success.

If you have any free cycles, we would appreciate help on that front. #671

[tenzen-y]

Thank you for this contribution.
Could you rebase on the master branch to take the CI issue fix patch?