[Feature] model-level retry

[TomWhite-MedStar]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Config option to specify number of times that a model should be auto-retried before failing that model. Default is 0 times, but can specify a positive number. Optionally also specify a retry-timeout (e.g. waiting X seconds before auto-retry). This could be model-level or global config.

Conceptually, this would be implemented as an option within the try/catch statement with the code that executes the model.

Describe alternatives you've considered

I created custom on-run-end macros that log results details to a table so that I can create a new dbt run --select statement of just the models that failed (e.g. dbt run --select failedModelA+ failedModelB+). However, this requires issuing an entirely new dbt run command. I'd much prefer to conditionally retry failed models a couple times before failing the entire model. This

Who will this benefit?

We use several dbt-core workflows to run several thousand SQL jobs per day to populate our warehouse. Several days per week we run into intermittent cloud vendor issues that cause a failure to schedule a given model (which then kills the entire dbt pipeline).

By contrast, we use Azure Data Factory (ADF) to load data from source to target. It does support task/model-level conditional retry (specifying number of times to retry, and delay in seconds between retry). We have never had data loss or pipeline failures in ADF, largely because of the success of those task-level retries.

Adding this feature to dbt-core would make it much more robust for critical production operations.

Are you interested in contributing this feature?

Yes - but I would need guidance on where to make changes. If I knew where, I could fork project, build unit tests, and submit pull request

Anything else?

No response