I've done some initial review on your patches. Your NVMe changes look fine to me - but
I'm not the one to review that.
I have some comments regarding your memory management refactor. I posted all of them on a
single patch - https://review.gerrithub.io/c/397038/
I'm thinking the g_phys_regions could be a global vtophys override map.
We could expose public spdk_vtophys_override(vaddr, len, paddr) API -> then we
wouldn't need to modify spdk_mem_register.
NVMe could do:
spdk_vtophys_override(cmb_vaddr, len, cmb_paddr);
From: SPDK [mailto:firstname.lastname@example.org] On Behalf Of Stephen Bates
Sent: Tuesday, January 30, 2018 4:35 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] SPDK support for NVMe CMBs and PMRs with WDS/RDS
I just uploaded a patchset to Gerrit that adds support for NVMe controllers with
CMBs/PMRs that support WDS and RDS (my full branch of the code is at ).
This allows NVMe controllers to move/copy data from a namespace on one
controller to a namespace on a different controller without requiring a system
memory buffer. There are lots of interesting use cases for such data-
The biggest issue with using a CMB or PMR for data copies is getting a vtophys
translation. This series adds a new vtophys method for physical memory
regions that fail via the existing methods (e.g. a PCIe BAR). This new method
using a linked list of physical regions that can be added/deleted via
spdk_reg_memory calls. We can then allocate/free memory from these
regions and the current maps can handle both the reference counting and store
the vtophys translations.
The hello_world example is updated to utilize the CMB WDS/RDS capability if
the associated controller supports it. It addition a new example application
called cmb_copy is included that performs the aforementioned offloaded copy
when a CMB is available.
We have confirmed both cmb_copy and the new hello_world work as expected
on both hardware from Eiditicom and Everspin. We plan to do more testing as
more drives with WDS/RDS capable CMBs become available. We used PCIe
packet counters in the Microsemi PCIe switch to confirm traffic is moving
directly between the two NVMe SSDs and not being routed to the root complex
on the CPU.
Feedback on the patches is gratefully received!
SPDK mailing list