VFIO is an important technology to give user space (applications or virtual machines) access to devices such as NICs or GPUs. Here is the blurb from the Linux kernel documentation:

Many modern systems now provide DMA and interrupt remapping facilities to help ensure I/O devices behave within the boundaries they’ve been allotted. This includes x86 hardware with AMD-Vi and Intel VT-d, POWER systems with Partitionable Endpoints (PEs), and embedded PowerPC systems such as Freescale PAMU. The VFIO driver is an IOMMU/device-agnostic framework for exposing direct device access to userspace, in a secure, IOMMU-protected environment. In other words, this allows safe [2], non-privileged, userspace drivers.

Why do we want that? Virtual machines often make use of direct device access (“device assignment”) when configured for the highest possible I/O performance. From a device and host perspective, this simply turns the VM into a userspace driver, with the benefits of significantly reduced latency, higher bandwidth, and direct use of bare-metal device drivers.

Prep Work Link to heading

First, list all network devices to locate the target PCI address. This gives the PCI ID 0000:00:14.3:

lspci -nn | grep -i Net
0000:00:14.3 Network controller: Intel Corporation Wi-Fi 6 AX201

Make sure to load the VFIO module:

modprobe vfio-pci
lsmod | grep vfio

Bind Device to User Space VFIO Link to heading

To allow VFIO to take over the device, unbind it from its current driver. Now bind the device to the vfio-pci driver:

echo "vfio-pci" > /sys/bus/pci/devices/0000:00:14.3/driver_override
echo "0000:00:14.3" > /sys/bus/pci/devices/0000:00:14.3/driver/unbind
echo "0000:00:14.3" > /sys/bus/pci/drivers/vfio-pci/bind

The device is now under user-space control and not managed by the kernel network stack.

Compile and Run Simple VFIO Application Link to heading

This is a simple VFIO application that does the following steps:

Container from /dev/vio/vfio
Group based on the number with the VFIO driver that took over the NIC
Device
Memory region for BAR
mmap the memory region to user space

#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdint.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <linux/vfio.h>

int main() {
    int container, group, device;
    int group_id = 0; // Replace with actual group ID
    const char *device_name = "0000:00:14.3"; // Replace with device ID

    // Open the VFIO container
    container = open("/dev/vfio/vfio", O_RDWR);
    if (container < 0) {
        perror("Failed to open /dev/vfio/vfio");
        return 1;
    }

    // Check extension support
    if (!ioctl(container, VFIO_CHECK_EXTENSION, VFIO_TYPE1_IOMMU)) {
        fprintf(stderr, "VFIO_TYPE1_IOMMU not supported\n");
        return 1;
    }

    // Open the IOMMU group
    char group_path[64];
    snprintf(group_path, sizeof(group_path), "/dev/vfio/%d", group_id);
    group = open(group_path, O_RDWR);
    if (group < 0) {
        perror("Failed to open VFIO group");
        return 1;
    }

    // Set the group to the container
    if (ioctl(group, VFIO_GROUP_SET_CONTAINER, &container)) {
        perror("Failed to set group container");
        return 1;
    }

    // Enable IOMMU
    if (ioctl(container, VFIO_SET_IOMMU, VFIO_TYPE1_IOMMU)) {
        perror("Failed to set IOMMU");
        return 1;
    }

    // Get device FD
    device = ioctl(group, VFIO_GROUP_GET_DEVICE_FD, device_name);
    if (device < 0) {
        perror("Failed to get device FD");
        return 1;
    }

    // Get region info
    struct vfio_region_info reg = {
        .argsz = sizeof(reg),
        .index = VFIO_PCI_BAR0_REGION_INDEX,
    };

    if (ioctl(device, VFIO_DEVICE_GET_REGION_INFO, &reg)) {
        perror("Failed to get BAR0 info");
        return 1;
    }

    // Map BAR0 region
    void *bar0 = mmap(0, reg.size, PROT_READ | PROT_WRITE, MAP_SHARED, device, reg.offset);
    if (bar0 == MAP_FAILED) {
        perror("mmap failed");
        return 1;
    }

    // Read first few bytes
    uint32_t *ptr = (uint32_t *)bar0;
    printf("BAR0 first dword: 0x%08x\n", ptr[0]);

    // Cleanup
    munmap(bar0, reg.size);
    close(device);
    close(group);
    close(container);

    return 0;
}

Build and run the application (as root!) and it will read BAR0:

gcc -o vfio_nic vfio_nic.c
sudo ./vfio_nic

BAR0 first dword: 0x18880000

Release Device Link to heading

Finally, we can return the device to iwlwifi:

echo "0000:00:14.3" > /sys/bus/pci/drivers/iwlwifi/bind