Homeglossary

Multi-Instance GPU

Multi-instance GPU is a hardware partitioning technology that divides one physical GPU into isolated GPU instances for separate workloads. Each instance receives dedicated compute and memory resources, helping multiple users or applications share a supported GPU predictably. The term usually refers to NVIDIA Multi-Instance GPU, or MIG, although the broader concept is GPU hardware partitioning.

What Is a Multi-Instance GPU?

Multi-instance GPU is a hardware partitioning technology that divides one physical graphics processing unit, or GPU, into isolated GPU instances for separate workloads. Each instance receives defined compute and memory resources, allowing multiple users or applications to share one supported GPU with predictable performance.

The term usually means NVIDIA Multi-Instance GPU, or MIG. Rather than making programs compete for a full GPU, MIG creates smaller logical GPUs that software can use independently. Hardware-level boundaries separate compute and memory resources. Not every GPU supports MIG, and instance sizes depend on the model and architecture. Some supported models can create up to seven isolated instances, while others offer fewer options.

Why GPU Partitioning Matters

Organizations use powerful GPUs for AI, scientific, and visual computing, but smaller inference services, notebooks, and test jobs may need only part of a GPU. Unused capacity wastes expensive hardware.

Simple sharing can create contention. A spike from one job can slow others or cause memory errors. Partitioning gives each workload a known boundary, making capacity planning easier and reducing disruptions. This matters when response times must remain consistent.

How Multi-Instance GPU Works

MIG is configured through supported drivers, management tools, and scheduling software. Commands and profiles vary by GPU generation and operating environment.

  1. Verify that the GPU, operating system, driver, and CUDA software stack support MIG.
  2. Enable MIG mode when required. This may require stopping workloads or restarting the host.
  3. Choose an approved instance profile with the needed memory and compute capacity.
  4. Create a GPU instance, which reserves a hardware-defined part of the physical GPU.
  5. Create compute instances within that allocation when the architecture supports this level of division.
  6. Expose the logical device to the operating system, container runtime, or cluster scheduler.
  7. Assign workloads to designated devices and monitor memory use, utilization, errors, and latency.
  8. Reconfigure during a maintenance window, because changing the layout can interrupt jobs.

Each instance receives portions of resources such as streaming multiprocessors, GPU memory, memory bandwidth, and cache. A streaming multiprocessor is a block of GPU cores for parallel calculations. Allocation is not a free-form percentage slider. Manufacturers define valid profile combinations for each architecture in the MIG User Guide.

Key Components and Terms

A physical GPU is the installed hardware accelerator. A GPU instance is an isolated hardware partition. A compute instance is a schedulable processing partition within a GPU instance on supported systems. An instance profile is the approved resource package used to create a partition.

GPU memory stores model weights, input data, and intermediate results. Memory bandwidth is the rate data moves to and from memory, while cache is fast local storage. A CUDA device is the logical GPU visible to compatible software. The driver lets the operating system communicate with the GPU, while a container runtime passes an assigned device into a container. In Kubernetes, a device plugin advertises GPU resources so pods can request them.

Multi-Instance GPU vs Time Slicing, vGPU, and Multiple Physical GPUs

These approaches let more than one workload access GPU capacity, but offer different tradeoffs. Multiple physical GPUs run independently in one server rather than splitting one GPU.

ApproachIsolation and guaranteesBest fitCommon limitation
Multi-instance GPUHardware partitions with defined compute and memory resourcesStable concurrent workloads needing predictable boundariesOnly fixed profiles on supported GPUs
Time slicingJobs take turns on a shared GPU, with weaker resource guaranteesShort or tolerant workloadsLatency can vary under contention
Virtual GPU softwareSoftware-managed virtual devices, often with licensing and platform optionsVirtual desktops and broader virtualization needsAdded operational complexity and overhead
Multiple physical GPUsComplete hardware separation per deviceLarge jobs or independently growing teamsHigher hardware, power, and server costs

Common Multi-Instance GPU Use Cases

MIG works best when several workloads run at once and their needs are reasonably steady.

  • Concurrent AI inference services running several small models.
  • Data science notebooks and model experiments requiring dependable shared access.
  • Research systems allocating fixed GPU capacity to projects or departments.
  • Continuous integration testing that should not interfere with production inference.
  • Media processing and edge services on supported hardware and software stacks.
  • Multi-tenant platforms needing clearer resource boundaries between customers, a concern related to a multi-tenant application.
  • AI services using a multimodal model, embeddings, or coordinated agent workloads that fit an available profile.

Benefits of Multi-Instance GPU

The main value is turning one large accelerator into predictable capacity for several smaller jobs. Results depend on choosing profiles that match demand.

  • Improves utilization when no workload needs the whole GPU.
  • Allows workloads to run concurrently rather than wait for exclusive access.
  • Creates compute and memory boundaries that reduce noisy-neighbor effects.
  • Provides more predictable latency and throughput than opportunistic sharing for properly sized workloads.
  • Simplifies allocation by giving teams or services named logical devices.
  • Can delay the need to purchase separate GPUs for small, independent jobs.

Practical Limits and Challenges

MIG does not create capacity. It divides existing GPU resources, so a poor layout can strand capacity or make an instance too small for reliable operation.

  • Only certain GPUs, drivers, operating systems, and software combinations support it.
  • Profile sizes are fixed, so an ideal custom split may not exist.
  • Free slices may not match the next workload's required profile.
  • Changing the layout can disrupt jobs and requires maintenance planning.
  • Demand can be uneven, leaving one instance overloaded while another is idle.
  • Each instance has a memory ceiling, and models that barely fit can fail on larger inputs or traffic spikes.
  • Feature availability, including media and monitoring capabilities, varies by architecture and profile.
  • Schedulers, drivers, containers, and monitoring tools must understand assigned devices.
  • Hardware isolation does not replace application security, identity controls, network policy, or secure data handling.

How to Choose the Right MIG Profile

Choose profiles using measured behavior, not a model's minimum published requirement. Memory headroom is often decisive.

  1. Measure peak memory use, including weights, input batches, caches, and runtime overhead.
  2. Identify whether the workload is limited by compute speed, memory capacity, memory bandwidth, or latency.
  3. Group workloads with stable, similar demand instead of mixing unpredictable jobs.
  4. Reserve headroom for bursts, larger requests, updates, and memory fragmentation.
  5. Avoid tiny slices for models that only barely fit.
  6. Test under realistic concurrent traffic, not only isolated benchmarks.
  7. Set scheduler policies that prevent one team from consuming all suitable profiles.
  8. Review the layout as models, traffic, and priorities change.

For example, several small, steady inference APIs can use smaller slices while a batch scoring job uses a larger slice. If batch work runs overnight, its profile can be scheduled around daytime inference demand.

Deployment and Operations Best Practices

Reliable MIG operation depends on process as well as hardware.

  • Document each profile layout, workload owner, expected memory use, and allocation reason.
  • Use containers with compatible libraries and expose only the assigned logical GPU.
  • Pin and test GPU drivers, CUDA libraries, and container runtimes before upgrades.
  • Monitor utilization, memory use, errors, queue length, and latency for every instance.
  • Alert on out-of-memory errors, failed jobs, device loss, and persistently idle allocations.
  • Schedule maintenance windows for profile changes and communicate service effects.
  • In Kubernetes, verify that the device plugin advertises intended MIG resources and pod requests map to the correct profile.
  • Test isolation with concurrent workloads before making service-level commitments.

When Not to Use Multi-Instance GPU

MIG is a weak fit for a large training run that needs the full GPU, workloads requiring the whole memory pool, or highly variable jobs whose resource needs constantly change. It may also be unsuitable on unsupported hardware or when staff cannot safely manage drivers, schedulers, and maintenance.

In those cases, an unpartitioned GPU with batch scheduling, time slicing for tolerant work, virtual GPU software for desktop-oriented environments, or additional physical GPUs may be more practical. Choose multi-instance GPU for stable concurrent workloads needing firm resource boundaries, and another approach when flexibility or full-device scale matters more.

Start Building
on Emergent today
Start Building