flyte.org

Yes. Flyte 2 serves long-running applications — model endpoints and interactive dashboards — through the same Python API you use for workflows. You declare the app, its image, resources, and scaling, and Flyte handles the serving plumbing. This applies when you want your serving layer to live next to your pipelines rather than in a separate stack.

What "same Python API" means in practice

Flyte 2's serving capability is not a separate product with its own config language. The page describes it as "the same Python API serves long-running apps next to your workflows." So a workflow task and a served app are both declared in Python, using the same SDK surface. That matters if you already write tasks as plain async functions — you don't switch mental models or toolchains to add a service.

Native integrations

Flyte 2 ships native integrations that wrap common serving and UI frameworks. Per the page, these include:

• vLLM — for LLM serving
• SGLang — for LLM serving
• FastAPI — for general HTTP services
• Streamlit — for interactive dashboards
• Ollama — for local model serving

The integration pattern is declarative: you declare the app, image, resources, and scaling, and Flyte manages the serving plumbing underneath. The page links a flyteplugins-vllm example (serve_llm.py) as a concrete reference.

What the endpoints give you

For LLM serving specifically, the page lists these capabilities:

• OpenAI-compatible endpoints — clients that speak the OpenAI API can call your served model directly.
• Model streaming — token-by-token output rather than waiting for the full response.
• Tensor parallelism — splitting a model across multiple GPUs for larger models or higher throughput.

For dashboards, the page states they run as long-running apps with auth included — you don't build authentication separately.

Scaling behavior

Scaling is described as configuration, not infrastructure work:

• Scale to zero when idle — replicas drop to zero when there's no traffic.
• Replicas and scaledown are one line of config — both the replica count and the scale-down behavior are set in a single line.

This is the part that most affects cost and operational load: an idle dashboard or endpoint doesn't hold compute.

A concrete scenario

Say you fine-tune a model in a Flyte workflow, then want to serve it for a demo or internal tool. With this setup you would:

1. Declare the serving app in Python, pointing at your model and choosing the vLLM integration.
2. Set the image and resources (including GPU count for tensor parallelism).
3. Set replicas and scaledown in one line of config.
4. Get an OpenAI-compatible endpoint that streams responses, plus a Streamlit dashboard if you want a UI — both with auth, both scaling to zero when idle.

The workflow that produced the model and the endpoint that serves it live in the same system, so there's no export step or second deployment pipeline to maintain.

What the page does not specify

The source material does not state pricing, whether login is required, or deployment prerequisites beyond the OSS/Enterprise split (the page shows "Deploy Enterprise Flyte" and "Deploy OSS" as separate paths, and notes Apache 2.0 OSS under the Linux Foundation AI & Data). If you need those details, check the deployment pages directly rather than assuming either path is free or open.

Flyte 2 is an open-source, durable AI runtime for orchestrating AI workloads — training, batch inference, fine-tuning, model serving, and agent loops. Its defining feature is automatic recovery: when infrastructure or code failures interrupt a workload, Flyte resumes from the last recorded step instead of restarting the whole job. It is licensed under Apache 2.0 and hosted by the Linux Foundation AI & Data. If you run long or expensive AI pipelines on variable compute (cloud, on-prem, or bursty GPU clusters), that recovery behavior is the main reason to consider it.

The core idea: durability

Most orchestrators re-run a failed pipeline from the beginning. Flyte records the inputs and outputs of every step, so a crash becomes a resume point rather than a restart.

The site frames this around the failure modes teams actually hit:

The practical effect: a 15-minute batch job that dies at minute 12 doesn't cost you the first 12 minutes again.

How you write workflows

Flyte 2 uses pure Python — no DSL and no YAML. A TaskEnvironment declares the image, resources, secrets, and retry policy; tasks themselves are plain async functions. You branch, loop, and fan out with ordinary control flow, and the runtime records I/O for each step.

pip install flyte

SDKs are also available for TypeScript (flyte-sdk-ts), Rust (flyte-sdk-rs), Go (flyte-sdk-go), and Mojo (flyte-sdk-mojo), with Python as the primary path.

Two properties worth noting:

• Type-checked I/O between tasks — language-native types, files, DataFrames, and dataclasses pass between steps.
• Native agents — wrap tools as tasks, and every LLM call is traced.

The same code runs locally, on a devbox, or on your cluster, which shortens the loop between writing and deploying.

Serving and dashboards

The same Python API that defines workflows also serves long-running applications. Native integrations wrap Streamlit, FastAPI, vLLM, SGLang, and Ollama: you declare the app, image, resources, and scaling, and Flyte handles the serving plumbing.

• OpenAI-compatible LLM endpoints with model streaming and tensor parallelism
• Interactive dashboards served as long-running apps, with auth included
• Scale to zero when idle — replicas and scaledown are one line of config

This means serving and orchestration live in one platform rather than two stacks.

Who it's for

Flyte 2 fits teams running AI workloads where a restart is expensive or where infrastructure is unreliable: batch inference, fine-tuning, pretraining, RL, and agent loops. It's a reasonable fit if you already write Python and want recovery, typed data flow, and serving without adopting a separate DSL.

It's less obviously a fit if your jobs are short and idempotent, or if you're standardized on a different orchestration ecosystem and don't need step-level recovery.

Getting started

You can try it without local setup via GitHub Codespaces, or deploy either the OSS or Enterprise edition. The project lives at flyteorg/flyte-sdk on GitHub, and installation starts with pip install flyte.

Flyte 2 is designed to run the same workflow code across local machines, devboxes, and Kubernetes clusters, and it can make parts of an existing stack durable without a rewrite. The page evidence shows environment options including Devbox, AWS, and Kind, plus a GitHub Codespaces trial path. If your stack is Python-based and you want failure recovery, dynamic orchestration, and production scaling, Flyte 2 is a candidate. If you need a non-Python DSL or a fully managed service with published pricing, the available page evidence does not confirm those details.

What "works with existing stacks" means in Flyte 2

The core idea is that Flyte 2 wraps your existing code and infrastructure rather than replacing them. According to the page, you can "make your existing stack durable" and run workloads for training, serving, and agents with auto-recovery from failures.

The mechanism is a TaskEnvironment that declares images, resources, secrets, and retry policy. Tasks themselves are plain async functions. That means:

• You keep writing Python, TypeScript, Rust, Go, or Mojo using the respective SDKs.
• You do not need a separate DSL or YAML to define branching, looping, or fan-out; ordinary control flow is used.
• The runtime records input/output for every step, so a crash resumes instead of restarting from the beginning.

This matters if your current stack already has Python services, model servers, or data pipelines. Flyte 2 sits alongside them and adds durability and orchestration rather than forcing a migration.

Where Flyte 2 can run

The page lists several environments for running the same code:

The page also states that "the same code runs locally, on the devbox, or on your cluster." That consistency is the practical benefit: you can develop and test in a lightweight environment, then move to a cluster without rewriting the workflow.

For a concrete scenario: if you have a batch inference script that calls an external API, you can run it on a devbox to iterate, then deploy the same task to an AWS cluster for production scale. The retry and recovery behavior applies in both cases.

How existing tools fit in

Flyte 2 provides native integrations that wrap common serving and app frameworks:

• Streamlit for interactive dashboards
• FastAPI for services
• vLLM, SGLang, and Ollama for model endpoints

The page describes declaring the app, image, resources, and scaling, with Flyte handling the serving plumbing. It also mentions OpenAI-compatible LLM endpoints with model streaming and tensor parallelism, dashboards served as long-running apps with auth included, and scale-to-zero when idle via one line of config.

If your stack already uses these tools, the integration path is to declare them within Flyte rather than rebuild them. The page does not specify version compatibility or configuration details, so those would need to be checked against current documentation.

What to verify before committing

The page evidence supports these claims, but a few practical questions remain open:

• Pricing and licensing: The page shows "APACHE 2.0 OSS" and "LINUX FOUNDATION AI & DATA," indicating an open-source option. It also references "Deploy Enterprise Flyte," but no pricing signals or payment platforms are listed. Do not assume the enterprise tier is free or that login is not required.
• SDK maturity: Python, TypeScript, Rust, Go, and Mojo SDKs are listed. The page does not indicate which are production-ready versus experimental.
• Cluster requirements: Kind and AWS are mentioned, but specific Kubernetes versions, node types, or GPU configurations are not detailed.

A reasonable next step is to try the GitHub Codespaces option or install via pip install flyte and run a sample task locally. That will confirm whether the SDK fits your language and whether the local-to-cluster path matches your deployment model.

Flyte 2 is a durable AI runtime that automatically recovers from infrastructure and code failures without manual intervention. When a task fails — whether from an API rate limit, network error, OOM kill, node interruption, a disappearing GPU, or a code exception — Flyte catches the failure, applies the retry policy you declared, and resumes from where it left off rather than restarting the whole pipeline. This applies to workloads like batch inference, fine-tuning, pretraining, RL, and agent loops.

The failure modes Flyte 2 is built to handle

Flyte 2's site lists the specific failures it targets, and they map to the two categories in the question:

Infrastructure failures

• API rate limit (e.g., a provider returns HTTP 429 mid-shard)
• Network error
• OOM kill
• Node interruption
• GPU disappears

Code failures

• Code exception

The design premise is that these are "the failure modes you actually hit" in production AI workloads — not edge cases.

How recovery works: retry the failed task, not the whole job

The key mechanism is task-level retry with backoff. When a failure occurs:

1. Flyte catches the failure at the task level.
2. It backs off exponentially according to the RetryStrategy you declared.
3. It retries only the failed task.
4. Completed work is never rerun.

The site's own example makes this concrete: a provider returns a 429 mid-shard during batch inference. Flyte catches it, backs off exponentially per the RetryStrategy, and retries the one failed task — completed shards are never rerun.

This matters for cost and time. In a fan-out workload with many shards, restarting the entire pipeline on a single transient error wastes all the compute already spent. Retrying only the failed unit preserves that work.

How recovery works: resume from the last recorded step

The second mechanism is durable execution via recorded I/O. The runtime records the input and output for every step. If a crash occurs, the workflow resumes from the breakpoint instead of restarting from the beginning.

This is what "durable runtime" means in practice: the runtime holds the state of each step, so a crash is a pause, not a reset. Combined with task-level retry, this covers both transient failures (retry the task) and process-level crashes (resume from the last completed step).

What you declare, and what Flyte handles

You declare the recovery policy in a TaskEnvironment — images, resources, secrets, and retry policy. Tasks themselves are plain async functions, so branching, looping, and fan-out use ordinary control flow rather than a DSL or YAML.

The result, per the site, is recovery "no manual intervention" — you press run, break the pipeline with a chosen failure, and watch Flyte recover.

What this means for your decision

Flyte 2 is a fit if your workloads are long-running or expensive enough that a mid-pipeline failure is costly — batch inference across many shards, fine-tuning, pretraining, RL, or agent loops. The value proposition is strongest when failures are frequent and partial progress is worth preserving.

It is less differentiated if your tasks are short, cheap, and idempotent enough that a full restart is acceptable.

Two things the source does not specify, so verify before relying on them: the exact retry defaults and backoff parameters (the site says backoff follows the RetryStrategy, but not the default values), and whether recovery semantics differ across the supported SDKs (Python, TypeScript, Rust, Go, Mojo). The site describes the runtime behavior generically; check the SDK docs for your language.

Flyte 2 workflows are written as plain async functions in a supported language SDK — no DSL and no YAML. You declare a TaskEnvironment (image, resources, secrets, retry policy) once, then write tasks as ordinary functions and compose them with normal control flow. The runtime records inputs and outputs for every step, so a crash resumes from the last completed step instead of restarting the whole workflow. This applies if you are on Flyte 2 with one of the supported SDKs; the examples below use Python.

The two building blocks

The environment is defined once and reused, so image and resource settings are not repeated per task. Tasks stay ordinary functions — you can call them, test them, and run them locally without a cluster.

Writing a task

A task is an async function. Its signature is the contract: typed parameters in, typed return value out. Because the runtime records I/O per step, a failure mid-workflow resumes rather than re-running completed work.

env = TaskEnvironment(
    image="my-image:latest",
    resources=...,
    secrets=...,
    retry_policy=...,
)

@env.task
async def fetch_shard(shard_id: int) -> DataFrame:
    ...

@env.task
async def train(data: DataFrame) -> Model:
    ...

Composing with normal control flow

Branching, looping, and fan-out use the language's own constructs — if, for, asyncio.gather, and similar. There is no separate graph syntax to learn, and the runtime still records every step's I/O.

@env.task
async def pipeline(shards: list[int]) -> Model:
    results = await asyncio.gather(*(fetch_shard(s) for s in shards))
    merged = merge(results)
    if needs_more_data(merged):
        merged = await fetch_shard(extra_shard)
    return await train(merged)

Fan-out across shards, a conditional re-fetch, and a final training step are all expressed as ordinary Python.

Typed I/O between tasks

Data passed between tasks is type-checked using language-native types. Supported values include native types, files, DataFrames, and dataclasses. This means a mismatch between what one task returns and what the next expects surfaces as a type error rather than a runtime surprise.

Supported SDKs

Flyte 2 ships SDKs for several languages, so the same model — environment plus plain functions — applies beyond Python:

• Python — flyte-sdk
• TypeScript — flyte-sdk-ts
• Rust — flyte-sdk-rs
• Go — flyte-sdk-go
• Mojo — flyte-sdk-mojo

The Python SDK installs with pip install flyte.

Agents and serving use the same API

Native agents wrap tools as tasks, and every LLM call is traced. The same Python API also serves long-running apps next to your workflows, with native integrations for Streamlit, FastAPI, vLLM, SGLang, and Ollama — you declare the app, image, resources, and scaling, and Flyte handles the serving plumbing. That includes OpenAI-compatible LLM endpoints with model streaming and tensor parallelism, interactive dashboards served as long-running apps with auth, and scale-to-zero when idle via one line of config.

Verifying it works

The same code runs locally, on the devbox, or on your cluster, so you can validate a workflow before deploying. Run it locally first to confirm types and control flow, then move it to a cluster without rewriting it. If a task fails — an API rate limit, a network error, an OOM kill, a node interruption, a missing GPU, or a code exception — the retry policy in the TaskEnvironment governs recovery, and completed steps are not re-run.

Common sticking points

• Putting config on tasks instead of the environment. Image, resources, secrets, and retry policy belong in the TaskEnvironment so they are declared once.
• Expecting a graph definition. There is no separate workflow DSL to write; composition is just control flow.
• Assuming a crash restarts everything. Because I/O is recorded per step, recovery resumes from the failed step — but only if the work is expressed as tasks within the environment.