10:16 a.m.
hi,
we use sysfs-pci to access PLX NTB directly and want to use SPDK IOAT modue (VFIO) to
copy data.
How to register the NTB bar address into SPDK/DPDK?
Thanks
2:53 a.m.
Hi,
we change to use SPDK pci APIs to access NTB.
After spdk_pci_device_map_bar(), it can get the "mapped address" for accessing
remote memory.
It's okay that using memcpy to copy "data" to "mapped address":
memcpy(mapped_addr, data_buf, size);
However, when using SPDK IOAT:
spdk_ioat_build_copy(ioat, cb_arg, cb_fn, mapped_addr, data_buf, size);
It will fail to translate mapped_addr by spdk_vtophys().
3:08 a.m.
Hi,
Can you try using spdk_mem_register() after you have called spdk_pci_device_map_bar()?
-Jim
On 3/22/21, 7:05 PM, "wiz.chen(a)gmail.com" <wiz.chen(a)gmail.com> wrote:
Hi,
we change to use SPDK pci APIs to access NTB.
After spdk_pci_device_map_bar(), it can get the "mapped address" for
accessing remote memory.
It's okay that using memcpy to copy "data" to "mapped
address":
memcpy(mapped_addr, data_buf, size);
However, when using SPDK IOAT:
spdk_ioat_build_copy(ioat, cb_arg, cb_fn, mapped_addr, data_buf, size);
It will fail to translate mapped_addr by spdk_vtophys().
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
3:45 a.m.
hi,
I check spdk_mem_register(), it needs 2-MB aligned address.
After some investigation:
1. nvme cmb --> nvme_pcie_ctrlr_map_io_cmb():
it do 2MB alignment here.
2. vfio: map all PCI BARs in 2MB-aligned virtual addresses
https://staging.gerrithub.io/c/spdk/dpdk/+/441070
Does the patch: "vfio: map all PCI BARs in 2MB-aligned virtual addresses"
have any side effect?
Or I do the 2MB alignment by myself is a batter choice?
3:04 p.m.
Hi,
You'll need to do the alignment yourself for now. I cannot vouch for the correctness
of the vfio patch.
-Jim
On 3/22/21, 8:25 PM, "wiz.chen(a)gmail.com" <wiz.chen(a)gmail.com> wrote:
hi,
I check spdk_mem_register(), it needs 2-MB aligned address.
After some investigation:
1. nvme cmb --> nvme_pcie_ctrlr_map_io_cmb():
it do 2MB alignment here.
2. vfio: map all PCI BARs in 2MB-aligned virtual addresses
https://staging.gerrithub.io/c/spdk/dpdk/+/441070
Does the patch: "vfio: map all PCI BARs in 2MB-aligned virtual
addresses" have any side effect?
Or I do the 2MB alignment by myself is a batter choice?
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
9:22 a.m.
Hi,
Because changing ntb bar size, It needs to upgrade EEPROM.
Thus, I use the patch: https://staging.gerrithub.io/c/spdk/dpdk/+/441070
to check if spdk_mem_register can work or not.
However, if I create a nvmf target, it will cause segfault.
After investigation, create nvmf target will call mem_map_notify_walk() to register
all
mapped memory.
It seems that the mapped memory of NTB PCI bar cannot be registered into RDMA
(Note: I use 'mlx5_dv' provider.)
Do you have any suggestion?
Thanks.
-yufan
418
days inactive
425
days old
5 comments
3 participants
participants (3)
-
Harris, James R -
wiz.chen＠gmail.com -
Yufan Chen