What Is Flyte 2 and What Does It Do?

Updated

Flyte 2 is an open-source, durable AI runtime for orchestrating AI workloads — training, batch inference, fine-tuning, model serving, and agent loops. Its defining feature is automatic recovery: when infrastructure or code failures interrupt a workload, Flyte resumes from the last recorded step instead of restarting the whole job. It is licensed under Apache 2.0 and hosted by the Linux Foundation AI & Data. If you run long or expensive AI pipelines on variable compute (cloud, on-prem, or bursty GPU clusters), that recovery behavior is the main reason to consider it.

The core idea: durability

Most orchestrators re-run a failed pipeline from the beginning. Flyte records the inputs and outputs of every step, so a crash becomes a resume point rather than a restart.

The site frames this around the failure modes teams actually hit:

Failure What happens in Flyte
API rate limit (e.g. a provider returns 429 mid-shard) Catches the error, backs off exponentially per the RetryStrategy, and retries only the failed task — completed shards are never rerun
Network error Retried per policy without re-running finished work
OOM kill The failed task is retried; prior steps stay cached
Node interruption / GPU disappears Workload resumes from recorded state
Code exception Surfaces at the failing task, not the whole pipeline

The practical effect: a 15-minute batch job that dies at minute 12 doesn't cost you the first 12 minutes again.

How you write workflows

Flyte 2 uses pure Python — no DSL and no YAML. A TaskEnvironment declares the image, resources, secrets, and retry policy; tasks themselves are plain async functions. You branch, loop, and fan out with ordinary control flow, and the runtime records I/O for each step.

pip install flyte

SDKs are also available for TypeScript (flyte-sdk-ts), Rust (flyte-sdk-rs), Go (flyte-sdk-go), and Mojo (flyte-sdk-mojo), with Python as the primary path.

Two properties worth noting:

  • Type-checked I/O between tasks — language-native types, files, DataFrames, and dataclasses pass between steps.
  • Native agents — wrap tools as tasks, and every LLM call is traced.

The same code runs locally, on a devbox, or on your cluster, which shortens the loop between writing and deploying.

Serving and dashboards

The same Python API that defines workflows also serves long-running applications. Native integrations wrap Streamlit, FastAPI, vLLM, SGLang, and Ollama: you declare the app, image, resources, and scaling, and Flyte handles the serving plumbing.

  • OpenAI-compatible LLM endpoints with model streaming and tensor parallelism
  • Interactive dashboards served as long-running apps, with auth included
  • Scale to zero when idle — replicas and scaledown are one line of config

This means serving and orchestration live in one platform rather than two stacks.

Who it's for

Flyte 2 fits teams running AI workloads where a restart is expensive or where infrastructure is unreliable: batch inference, fine-tuning, pretraining, RL, and agent loops. It's a reasonable fit if you already write Python and want recovery, typed data flow, and serving without adopting a separate DSL.

It's less obviously a fit if your jobs are short and idempotent, or if you're standardized on a different orchestration ecosystem and don't need step-level recovery.

Getting started

You can try it without local setup via GitHub Codespaces, or deploy either the OSS or Enterprise edition. The project lives at flyteorg/flyte-sdk on GitHub, and installation starts with pip install flyte.