GPUWorks launches 40 MB open-source AI inference platform for heterogeneous GPU clusters

The code is hosted on GitHub under the FlitAI organization, and the announcement was reported by IT Home and republished by Pandaily AI & Tech China (source). The platform pools Huawei Ascend, T-Head PPU, Hygon DCU and NVIDIA GPUs into a single cluster, matching models to hardware and exposing an OpenAI-compatible API. It runs on bare-metal servers, VMs or cloud containers across private and public clouds.

The installer is just over 40 MB, installs with a single command in about 60 seconds, and does not require Kubernetes. The catalog includes more than 1,000 launch configurations for nearly 400 models, bundling images, inference engines, weight versions and launch flags for each chip-model pair. A built-in AI assistant diagnoses faults, and the scheduler explains placement decisions.

In the developers’ internal tests, one GPUWorks server managed 1,003 GPU machines, brought up over 500 inference replicas in four minutes, and achieved 69,000 tokens per second while using nine CPU cores and 1.6 GB RAM. A wake-up from level-one sleep took 3.6 seconds for a DeepSeek-V4.1-Flash model, and a cold-start to NFS storage on eight NVIDIA H20 cards dropped from 16-17.5 minutes to about four seconds, a 200-fold improvement.

Our read: The platform’s tiny footprint and zero-Kubernetes requirement lower the barrier for enterprises to assemble heterogeneous accelerator pools, while the fast wake-up and detailed telemetry address common operational pain points in multi-vendor AI serving.