简体中文
English
Website review
This review may include AI-generated content and is not individually verified. Domain, certificate and other facts come from the observed data on this page.
What is MMDeploy?
MMDeploy is the deployment toolkit in the OpenMMLab ecosystem: it takes models trained with OpenMMLab toolkits and converts them into formats that run on a range of inference backends and devices. Its documentation organizes this around a practical pipeline — convert a model, run inference, evaluate the converted result, and optionally quantize it — plus SDK usage, benchmarks, and support for specific backend features.
What it covers in practice
- Model conversion: guides on converting models, writing configs, and exporting to backends such as ONNX Runtime, TensorRT, ncnn, OpenVINO, PPLNN, SNPE, RKNN, TVM, Core ML, and TorchScript.
- Deployment targets: OpenMMLab codebase support spans MMPretrain, MMDetection, MMSegmentation, MMagic, MMOCR, MMPose, MMDetection3d, MMRotate, and MMAction2.
- Verification and tuning: model evaluation, quantization, benchmark results, and tests on embedded devices.
- Extending it: developer material on architecture, adding new models or backends, custom operators per backend, and regression testing.
Who it suits, and the trade-off
It is aimed at teams already using OpenMMLab models who need to move them from training code to production or edge hardware. The benefit is a single conversion-and-test workflow across many backends; the cost is that each backend has its own supported-feature list and operator set, so conversion success depends on your model's operators and your target device. Expect to check the per-backend support pages and benchmark results before committing to a deployment path.
A useful next step: pick your target backend first, then read its feature-support page alongside the "How to convert model" guide, and confirm your model's operators are covered before building the full pipeline.
How do I convert an OpenMMLab model to a deployable format with MMDeploy?
Use MMDeploy’s Convert Model workflow: install MMDeploy and the relevant backend, then run the conversion tool against your OpenMMLab model config and checkpoint to produce a deployable model, and verify it by running inference on the converted result. The documentation lists this as a distinct step alongside inference, evaluation and SDK usage, so treat conversion as one stage in a pipeline rather than the end goal.
A practical sequence
- Check prerequisites and install. Follow the Prerequisites and Installation pages for the MMDeploy version and the deployment backend you intend to use. Backend choice matters: the docs cover ONNX Runtime, TensorRT, ncnn, OpenVINO, PPLNN, SNPE, RKNN, TVM, TorchScript and Core ML, each with its own feature support.
- Convert. Use the “How to convert model” guide with your model’s config and checkpoint. Conversion typically produces an intermediate ONNX model plus backend-specific artifacts, depending on the backend.
- Run inference. Use the “Inference Model” guide to confirm the converted model loads and produces sensible outputs.
- Evaluate. Use the “Evaluate Model” guide to compare accuracy against the original PyTorch model before shipping.
Choosing a backend
| Situation | Sensible starting point | Trade-off |
|---|---|---|
| First conversion, quick validation | ONNX Runtime | Broad support, simpler setup; not always the fastest on target hardware |
| NVIDIA GPU deployment | TensorRT | Strong performance; more setup and version sensitivity |
| Embedded or ARM devices | ncnn, RKNN or SNPE | Fits constrained hardware; feature coverage varies by backend |
| Intel hardware | OpenVINO | Good fit for Intel CPUs/accelerators; model coverage depends on ops |
For a concrete case: if you have an MMDetection model and a Jetson device, convert to ONNX first to validate the graph, then target TensorRT and check the TensorRT feature-support page for any custom ops your model uses.
Where conversions usually go wrong
- Unsupported operators. Check the backend’s supported-features page; custom ops may need the Custom Ops guidance for ONNX Runtime, TensorRT or ncnn.
- Preprocessing mismatches. Normalization and resize settings in the deploy config must match training; otherwise evaluation accuracy drops even when inference runs.
- Skipping evaluation. Always compare metrics on a small validation set after conversion.
If you are deploying an OpenMMLab model family other than detection, the docs include per-codebase deployment pages for MMPretrain, MMSegmentation, MMOCR, MMPose, MMAction2 and others, which give model-specific conversion notes. The official OpenMMLab project pages, such as GitHub repositories, are also useful for version compatibility questions.
Which inference backends does MMDeploy support, and how do I choose between TensorRT, ONNX Runtime, ncnn, and OpenVINO?
MMDeploy supports multiple inference backends, and its documentation lists dedicated support pages for TensorRT, ONNX Runtime, ncnn, OpenVINO, PPLNN, SNPE, RKNN, TVM, TorchScript, and Core ML, along with custom operator support for several of them (ONNX Runtime, TensorRT, ncnn). The right choice depends on your target hardware and whether you need maximum speed, broad portability, or a small runtime footprint.
How the four common backends differ
| Backend | Typical hardware | Best suited for | Main trade-off |
|---|---|---|---|
| TensorRT | NVIDIA GPUs (desktop, server, Jetson) | Lowest latency on NVIDIA hardware; production GPU serving | NVIDIA-only; engine builds are tied to GPU/version; conversion can be fiddly |
| ONNX Runtime | CPUs, GPUs, and many accelerators via providers | Portability and quick cross-platform deployment | Usually slower than vendor-specific engines on the same GPU |
| ncnn | ARM CPUs and mobile/embedded devices | Phone and edge deployment with no heavy dependencies | Limited operator coverage; some models need custom ops |
| OpenVINO | Intel CPUs, integrated GPUs, VPUs | Intel-based servers, laptops, and edge boxes | Best results mostly on Intel hardware |
How to choose
Start from the deployment target, not from benchmarks. If you are shipping to an NVIDIA GPU and latency matters, TensorRT is the natural first candidate; if you need one pipeline that runs on CPUs and several accelerators, ONNX Runtime is the safer default. For Android or ARM edge devices, ncnn keeps the runtime light. For Intel hardware, OpenVINO is the intended path.
A practical next step: pick your target device, check the corresponding backend page in MMDeploy documentation for supported operators and conversion steps, then run the documented benchmark on your own model before committing. If your model uses operators a backend lacks, compare the custom-op guides for ONNX Runtime, TensorRT, and ncnn rather than switching backends blindly.
How do I install MMDeploy and build it from source or with Docker?
MMDeploy's documentation puts installation and building in separate sections: a Get Started → Installation page for the basic setup, and a Build section with two parallel routes — Build from Source and Use Docker Image — plus Build from Script and CMake Build Option Spec pages for finer control. The practical path is to read Installation first, then pick the build route that matches your environment.
Choosing a build route
| Route | Suits | Trade-off |
|---|---|---|
| Build from source | You need a specific backend, CUDA/cuDNN version or compiler toolchain, or plan to modify the code | Most control, but you must satisfy the prerequisites yourself and rebuild when dependencies change |
| Docker image | You want a reproducible environment quickly, or are deploying on a clean machine or server | Faster to a working state; you inherit whatever versions the image pins and must mount your models and data into the container |
| Build from script | You want the source build without invoking CMake by hand | Convenient wrapper, but you still need the same prerequisites in place |
Suggested next step
Read Prerequisites before either route, because the backend you target (for example ONNX Runtime, TensorRT, ncnn, OpenVINO, PPLNN, SNPE, RKNN, TVM or Core ML) determines which SDKs and drivers must already be installed. Then open Build from Source for the CMake steps, or Use Docker Image if you prefer a preconfigured environment. Use CMake Build Option Spec when you need to turn specific backends or features on or off.
For a concrete case: if you are deploying a detector to an NVIDIA GPU server with a fixed driver, a source build lets you match TensorRT and CUDA exactly; if you are handing the setup to a colleague, the Docker route removes "works on my machine" problems.
The docs also cover converting, inferring and evaluating models after the build, and there are deployment guides for OpenMMLab projects such as MMDetection and MMSegmentation, which are worth reading once the build succeeds.
How can I benchmark and test a deployed model on embedded devices or with TVM?
MMDeploy documents both tasks under its Benchmark section: "Benchmark Test on embedded device" and "Benchmark Test on TVM." The embedded-device page is the place to start for hardware deployment testing, while the TVM page covers the compiler-based backend path. Both sit alongside the general benchmark and quantization test results, so you can compare a deployed model's numbers against reference results rather than judging them in isolation.
What each path is for
- Embedded device testing suits fixed-function boards where you need to confirm the converted model actually runs and performs acceptably on the target hardware. Use it when your deployment target is the device itself and you want latency or throughput evidence before shipping.
- TVM testing suits teams evaluating TVM as a backend, for example when comparing it against TensorRT or ONNX Runtime for the same model. It is more of an evaluation path than a production default.
A practical sequence
- Convert the model following "How to convert model," and confirm inference works on a desktop backend first.
- Quantize if the device is resource-constrained; the quantization test results give a reference point for accuracy trade-offs.
- Run the benchmark on the embedded device or with TVM, and compare against the supported-model and benchmark tables to see whether your numbers are in the expected range.
- Check the backend feature support pages (for example RKNN, SNPE or ncnn) for any operator gaps that could explain unexpected results.
Deciding between them
Choose embedded-device testing when the hardware is already fixed and you need deployment evidence. Choose TVM when you are still selecting a backend and want a compiler-based option in the comparison. For background on the conversion and SDK layers these tests depend on, the OpenMMLab ecosystem documentation at OpenMMLab and the upstream PyTorch project at PyTorch are useful references.
How do I add support for a new model or a new backend in MMDeploy?
MMDeploy treats these as two separate extension paths. Adding a new model means writing a deployment config and, where needed, a rewrite that maps the model's graph to backend-friendly operators. Adding a new backend means implementing the backend's op converters and registration so existing model configs can target it.
The documentation's Developer Guide lists both tasks directly: "How to support new models" and "How to support new backends," alongside "mmdeploy Architecture" and "How to add test units for backend ops." Read the architecture page first, since both paths depend on the same pipeline: export, rewrite, partition, convert to a backend SDK model, then inference and evaluation.
Adding a new model
- Confirm the model can be exported to ONNX (or TorchScript) and note the input/output names.
- Write a deployment config that names the model, its inputs, and the target backend.
- Run the rewrite step and inspect the partitioned graph for unsupported operators.
- Add op converters or a rewrite rule for anything the backend cannot handle.
- Validate with inference on sample inputs and compare outputs against the original framework.
- Add evaluation and a regression test so later changes do not silently break it.
Adding a new backend
- Implement the backend's operator converters, following the pattern of an existing backend such as ONNX Runtime or TensorRT.
- Register the backend so it can be selected in configs.
- Add test units for each supported op, as the guide's "How to add test units for backend ops" page describes.
- Run the regression test to check that supported models still convert and produce matching results.
Choosing where to start
| Situation | Start with | Main cost |
|---|---|---|
| Your model is not yet deployable | Model support path | Config and rewrite work; op coverage depends on the chosen backend |
| Your target hardware SDK is not listed | Backend support path | Implementing and testing op converters across many operators |
| A model converts but fails at runtime | Op-level debugging | Isolating unsupported ops and adding converters |
A practical reader scenario: a team has a detector trained in an OpenMMLab codebase and a new edge accelerator. They should first try converting with an existing backend that resembles the target (for example ONNX Runtime) to expose unsupported ops early, then implement only those ops for the new backend rather than attempting full coverage at once.
As a next step, open the Developer Guide section of MMDeploy documentation and read the architecture page before either how-to, then use the "How to get partitioned ONNX models" page to see exactly which operators need backend support.
Cloudflare
Google Workspace
User reviews (0)