I will work on your suggestion for issue 1.
For now, in my experiment I would need FreeBSD as a initiator. So, I do not
require any devices that need nic_uio.ko. I have commented out
"configure_freebsd_pci" in scripts/setup.sh to proceed forward.
For issue 2, I resolved the problem my linking RDMA userspace libraries
with SPDK. I changed spdk/mk/spdk.common.mk, and added following lines.
ifeq ($(CONFIG_RDMA), y)
LDFLAGS+= -libverbs -lrdmacm -lmlx4 -lpthread
I have created issue # 253 for same. Hope this fix would help other FreeBSD
On Thu, Mar 1, 2018 at 7:04 PM Harris, James R <james.r.harris(a)intel.com>
For the first failure – do you have a kernel w/o INVARIANTS that you could
try? The FreeBSD NVMe driver is setting the ETERNAL flag when registering
the device meaning the expectation is that it won’t be removed – this needs
to be removed. I’m guessing the systems in the SPDK test pool are running
a STABLE branch which is why this hasn’t come up before.
So you could either try with most recent FreeBSD release, rebuild your
current kernel and disable INVARIANTS, or rebuild your kernel and make nvme
a loadable module and then don’t load it. A FreeBSD kernel patch will also
be needed but that won’t help you in the short term.
Regarding NVMe-oF, I don’t believe this has been tested on FreeBSD. This
is a bug in the SPDK build environment – it should not be building the RDMA
transport on FreeBSD. If you have any experience with using userspace RDMA
on FreeBSD, and could provide any pointers, it would be appreciated. I’m
guessing the error message about not finding the config directory has
something to do with it.
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Sai Rajesh <
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Wednesday, February 28, 2018 at 6:55 PM
*To: *"spdk(a)lists.01.org" <spdk(a)lists.01.org>
*Subject: *[SPDK] SPDK on FreeBSD
I'm running SPDK on FreeBSD master branch. When I try to run setup.sh it
crashes my kernel. For now, I hardcoded kenv variable *hw.nic_uio.bdfs*
to my nvme disk PCI address. I grabbed the address from command "pciconf -l
| grep nvme".
Looks like nic_uio driver is trying to unload/destroy something its not
I plan to run this node as only initiator node, so I donot need any fancy
setup. All I need is able to setup SPDK on it and RDMA connect to target
node which is a Linux node.
If I skip setup.sh, initiator complains SPDK/DPDK environment is not set.
Is there a way to setup SPDK/DPDK without any overhead?
kldunload: can't find file nic_uio.ko
hw.nnic_uio: hw.nic_uio.bdfs = '5:0:0'
nic_uio_load: detaching and storing dev=0xfffff80108c1e100
panic: WARNING: Driver mistake: destroy_dev on eternal 0
cpuid = 19
time = 1519867718
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
vpanic() at vpanic+0x18d/frame 0xfffffe01a1161a80
vpanic() at vpanic/frame 0xfffffe01a1161b00
destroy_devl() at destroy_devl+0x541/frame 0xfffffe01a1161b40
destroy_dev() at destroy_dev+0x40/frame 0xfffffe01a1161b60
nvme_ctrlr_destruct() at nvme_ctrlr_destruct+0x38/frame 0xfffffe01a1161b90
nvme_detach() at nvme_detach+0x1a/frame 0xfffffe01a1161bb0
device_detach() at device_detach+0x167/frame 0xfffffe01a1161bf0
nic_uio_load() at nic_uio_load+0x2e4/frame 0xfffffe01a1161db0
nic_uio_modevent() at nic_uio_modevent+0x53/frame 0xfffffe01a1161df0
driver_module_handler() at driver_module_handler+0x5b/frame
module_register_init() at module_register_init+0xc0/frame
linker_load_module() at linker_load_module+0xb78/frame 0xfffffe01a1162180
kern_kldload() at kern_kldload+0xf1/frame 0xfffffe01a11621d0
sys_kldload() at sys_kldload+0x5b/frame 0xfffffe01a1162200
amd64_syscall() at amd64_syscall+0x79b/frame 0xfffffe01a1162330
fast_syscall_common() at fast_syscall_common+0x101/frame 0x7fffffffeb60
KDB: enter: panic
[ thread pid 1609 tid 100605 ]
Stopped at kdb_enter+0x3b: movq $0,kdb_why
And if I comment out kldload nic_uio.ko from setup.sh, it fails to create
rdma event channel.
root@rajesh:~/spdk # ./examples/nvme/perf/perf -q 1 -s 4096 -w randread
-t 10 -r "trtype:RDMA adrfam:IPv4 traddr:192.168.22.55 trsvcid:9797"
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --no-pci ]
EAL: Sysctl reports 20 cpus
EAL: Detected 20 lcore(s)
EAL: Contigmem driver has 8 buffers, each of size 256MB
EAL: Mapped memory segment 1 @ 0x801400000: physaddr:0x390000000, len
EAL: Mapped memory segment 2 @ 0x811400000: physaddr:0x3a0000000, len
EAL: Mapped memory segment 3 @ 0x821400000: physaddr:0x3b0000000, len
EAL: Mapped memory segment 4 @ 0x831400000: physaddr:0x3c0000000, len
EAL: Mapped memory segment 5 @ 0x841400000: physaddr:0x3d0000000, len
EAL: Mapped memory segment 6 @ 0x851400000: physaddr:0x3e0000000, len
EAL: Mapped memory segment 7 @ 0x861400000: physaddr:0x3f0000000, len
EAL: Mapped memory segment 8 @ 0x871400000: physaddr:0x400000000, len
Unable to unlink shared memory file: /var/run/.spdk_pid1425_config. Error
Unable to unlink shared memory file: /var/run/.spdk_pid1425_hugepage_info.
Error code: 2
Initializing NVMe Controllers
libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'.
nvme_rdma.c: 736:nvme_rdma_qpair_connect: *ERROR*:
nvme_rdma.c:1339:nvme_rdma_ctrlr_construct: *ERROR*: failed to create
No valid NVMe controllers or AIO devices found
Feb 28 20:47:24 saber1b perf: nvme_rdma.c: 736:nvme_rdma_qpair_connect:
*ERROR*: rdma_create_event_channel() failed
Feb 28 20:47:24 saber1b perf: nvme_rdma.c:1339:nvme_rdma_ctrlr_construct:
*ERROR*: failed to create admin qpair
Any help is appreciated to get me started as FreeBSD SPDK initiator.
SPDK mailing list