Background and motivation
...
- qemu NVMe driver that will route guest's NVMe traffic to virtio fabric (although previous attempts at upstreaming this exist)
- virtio fabric driver in the Linux kernel's NVMe target (although previous attempts at upstreaming this exist and there's a very similar SCSI driver already upstreamed)
Development steps
- Introduce HV_EXPERIMENTAL (exactly what HV_HVM, but with NVMe-OF for disk) - Roman
- Start with nullblk as a backend (potentially loopback) - Vitaliy
- Resurrect vhost PRs in pillar, etc. - Vitaliy
- Add checksums to fio test container - Vitaliy
- Implement vhost/nvme.c in Linux kernel - Sergey
- Implement vhost_nvme.c driver in qemu - Vitaliy
- Build a test rig for actual NVMe drives - Dima
Discussion
The nice property of this design is that we can start with independent bits and pieces and keen keep an eye on performance. For example, without writing any additional code we can send NVMe traffic via TCP from our guests to the host helping us test assumptions about scalability . This (of course, this is expected to be much slower than virtio/vhost implementation, but is available in a stock kernel today).
It must also be noted, that while experimenting with vhost and SPDK a few EVE PRs that can be re-used were generated and will be repurposed for this work.
It was suggested by one of our community members that "would be great to avoid any bio repacking on host userspace side and make it possible to fetch nvmeof requests directly on the host kernel side (since guest and host share the same memory, that can be doable). Yes, that means you write your own driver for the host as well, but still you do that for the guest, so should not be a big problem"