How Does Flyte 2 Recover from Infrastructure and Code Failures?

Flyte 2 is a durable AI runtime that automatically recovers from infrastructure and code failures without manual intervention. When a task fails — whether from an API rate limit, network error, OOM kill, node interruption, a disappearing GPU, or a code exception — Flyte catches the failure, applies the retry policy you declared, and resumes from where it left off rather than restarting the whole pipeline. This applies to workloads like batch inference, fine-tuning, pretraining, RL, and agent loops.

The failure modes Flyte 2 is built to handle

Flyte 2's site lists the specific failures it targets, and they map to the two categories in the question:

Infrastructure failures

• API rate limit (e.g., a provider returns HTTP 429 mid-shard)
• Network error
• OOM kill
• Node interruption
• GPU disappears

Code failures

• Code exception

The design premise is that these are "the failure modes you actually hit" in production AI workloads — not edge cases.

How recovery works: retry the failed task, not the whole job

The key mechanism is task-level retry with backoff. When a failure occurs:

1. Flyte catches the failure at the task level.
2. It backs off exponentially according to the RetryStrategy you declared.
3. It retries only the failed task.
4. Completed work is never rerun.

The site's own example makes this concrete: a provider returns a 429 mid-shard during batch inference. Flyte catches it, backs off exponentially per the RetryStrategy, and retries the one failed task — completed shards are never rerun.

This matters for cost and time. In a fan-out workload with many shards, restarting the entire pipeline on a single transient error wastes all the compute already spent. Retrying only the failed unit preserves that work.

How recovery works: resume from the last recorded step

The second mechanism is durable execution via recorded I/O. The runtime records the input and output for every step. If a crash occurs, the workflow resumes from the breakpoint instead of restarting from the beginning.

This is what "durable runtime" means in practice: the runtime holds the state of each step, so a crash is a pause, not a reset. Combined with task-level retry, this covers both transient failures (retry the task) and process-level crashes (resume from the last completed step).

What you declare, and what Flyte handles

You declare the recovery policy in a TaskEnvironment — images, resources, secrets, and retry policy. Tasks themselves are plain async functions, so branching, looping, and fan-out use ordinary control flow rather than a DSL or YAML.

The result, per the site, is recovery "no manual intervention" — you press run, break the pipeline with a chosen failure, and watch Flyte recover.

What this means for your decision

Flyte 2 is a fit if your workloads are long-running or expensive enough that a mid-pipeline failure is costly — batch inference across many shards, fine-tuning, pretraining, RL, or agent loops. The value proposition is strongest when failures are frequent and partial progress is worth preserving.

It is less differentiated if your tasks are short, cheap, and idempotent enough that a full restart is acceptable.

Two things the source does not specify, so verify before relying on them: the exact retry defaults and backoff parameters (the site says backoff follows the RetryStrategy, but not the default values), and whether recovery semantics differ across the supported SDKs (Python, TypeScript, Rust, Go, Mojo). The site describes the runtime behavior generically; check the SDK docs for your language.