6:46 a.m.
Hi Joe,
I can help you out more at the end of the week when I'm in the office as I just went
through this not too long ago. FYI here's the sequence that I used, looks pretty close
to what you're doing... if someone doesn't jump in between now and Thu I'll
run through this again on my setup and make sure it works with 18.10.1
sudo ~/spdk/app/nvmf_tgt/nvmf_tgt
sudo ~/spdk/scripts/rpc.py nvmf_create_transport -t RDMA -u 8192 -p 4 -c 0
sudo ~/spdk/scripts/rpc.py construct_nvme_bdev -b NVMe1 -t PCIe -a 0000:0d:00.0
sudo ~/spdk/scripts/rpc.py nvmf_subsystem_create nqn.2016-06.io.spdk:cnode1 -a -s
SPDK00000000000001
sudo ~/spdk/scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 NVMe1n1
sudo ~/spdk/scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t rdma
-a 192.168.100.8 -s 4420
Happy Holidays!
Paul
-----Original Message-----
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Gruher, Joseph R
Sent: Monday, December 24, 2018 3:34 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Trouble Setting Up NVMeoF Target
Hi everyone. I'm running Ubuntu 16.04 with SPDK v18.10.1. I'm having trouble
setting up the NVMeoF target. I can't seem to spot the problem, so maybe someone can
help me out here. I'm pretty sure I'm either formatting the command incorrectly
or I've screwed up the IB/RDMA related packages in my Ubuntu install.
I have a Mellanox CX5 NIC at 192.168.200.4, interface is up and linked:
rsd@nvme:~$ ifconfig enp6s0
enp6s0 Link encap:Ethernet HWaddr 98:03:9b:1e:de:ec
inet addr:192.168.200.4 Bcast:192.168.200.255 Mask:255.255.255.0
inet6 addr: fe80::9a03:9bff:fe1e:deec/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:138 errors:0 dropped:0 overruns:0 frame:0
TX packets:135 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:19029 (19.0 KB) TX bytes:24279 (24.2 KB)
rsd@nvme:~$ sudo ethtool enp6s0
Settings for enp6s0:
Supported ports: [ Backplane ]
Supported link modes: 1000baseKX/Full
10000baseKR/Full
40000baseKR4/Full
40000baseCR4/Full
40000baseSR4/Full
40000baseLR4/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Advertised link modes: 1000baseKX/Full
10000baseKR/Full
40000baseKR4/Full
40000baseCR4/Full
40000baseSR4/Full
40000baseLR4/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Link partner advertised link modes: Not reported
Link partner advertised pause frame use: No
Link partner advertised auto-negotiation: Yes
Speed: 100000Mb/s
Duplex: Full
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000004 (4)
link
Link detected: yes
I ran setup and then started the target and gave it four cores:
rsd@nvme:~$ sudo ~/install/spdk/scripts/setup.sh
[sudo] password for rsd:
0000:d8:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:d9:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
rsd@nvme:~$ sudo /home/rsd/install/spdk/app/nvmf_tgt/nvmf_tgt -m 0xF
Starting SPDK v18.10.1 / DPDK 18.08.0 initialization...
[ DPDK EAL parameters: nvmf --no-shconf -c 0xF --file-prefix=spdk_pid42493 ]
EAL: Detected 80 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
app.c: 602:spdk_app_start: *NOTICE*: Total cores available: 4
reactor.c: 703:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1
reactor.c: 490:_spdk_reactor_run: *NOTICE*: Reactor started on core 1 on socket 0
reactor.c: 490:_spdk_reactor_run: *NOTICE*: Reactor started on core 2 on socket 0
reactor.c: 490:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 on socket 0
reactor.c: 490:_spdk_reactor_run: *NOTICE*: Reactor started on core 3 on socket 0
I created my bdevs from my local NVMe devices:
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py construct_nvme_bdev -b d0 -t pcie
-a 0000:d8:00.0
d0n1
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py construct_nvme_bdev -b d1 -t pcie
-a 0000:d9:00.0
d1n1
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py construct_nvme_bdev -b d2 -t pcie
-a 0000:da:00.0
d2n1
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py construct_nvme_bdev -b d3 -t pcie
-a 0000:db:00.0
d3n1
Matching prints from the target:
EAL: PCI device 0000:d8:00.0 on NUMA socket 1
EAL: probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:d9:00.0 on NUMA socket 1
EAL: probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:da:00.0 on NUMA socket 1
EAL: probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:db:00.0 on NUMA socket 1
EAL: probe driver: 8086:a54 spdk_nvme
Then the problem starts when I try to create my NVMeoF subsystem. If I use this
deprecated method I get an invalid parameters error:
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py construct_nvmf_subsystem -a -n d0n1
-m 1 nqn.2018-12.io.spdk:nqn0 "trtype:RDMA traddr:192.168.200.4 trsvcid:4420"
""
Got JSON-RPC error response
request:
{
"jsonrpc": "2.0",
"params": {
"max_namespaces": 1,
"nqn": "nqn.2018-12.io.spdk:nqn0",
"allow_any_host": true,
"serial_number": "00000000000000000000",
"listen_addresses": [
{
"traddr": "192.168.200.4",
"trtype": "RDMA",
"trsvcid": "4420"
}
],
"namespaces": [
{
"bdev_name": "d0n1"
}
]
},
"method": "construct_nvmf_subsystem",
"id": 1
}
response:
{
"message": "Invalid parameters",
"code": -32602
}
Matching unhappy prints on the target:
nvmf_rpc_deprecated.c: 490:spdk_rpc_construct_nvmf_subsystem: *WARNING*: The
construct_nvmf_subsystem RPC is deprecated. Use nvmf_subsystem_create instead.
libibverbs: Warning: no userspace device-specific driver found for
/sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for
/sys/class/infiniband_verbs/uverbs0
rdma.c:1595:spdk_nvmf_rdma_create: *ERROR*: rdma_create_event_channel() failed, No such
device
transport.c: 93:spdk_nvmf_transport_create: *ERROR*: Unable to create new transport of
type RDMA
nvmf.c: 526:spdk_nvmf_tgt_listen: *ERROR*: Transport initialization failed
If I try to use the newer method, something similar happens, this part goes OK:
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_create
nqn.2018-12.io.spdk:nqn0
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_create
nqn.2018-12.io.spdk:nqn1
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_create
nqn.2018-12.io.spdk:nqn2
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_create
nqn.2018-12.io.spdk:nqn3
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_add_ns
nqn.2018-12.io.spdk:nqn0 d0n1
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_add_ns
nqn.2018-12.io.spdk:nqn1 d1n1
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_add_ns
nqn.2018-12.io.spdk:nqn2 d2n1
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_add_ns
nqn.2018-12.io.spdk:nqn3 d3n1
But then I can't add the transport:
rsd@nvme:~/install/nvme-cli-1.6$ sudo /home/rsd/install/spdk/scripts/rpc.py
nvmf_subsystem_add_listener -t rdma -a 192.168.200.4 nqn.2018-12.io.spdk:nqn0
Got JSON-RPC error response
request:
{
"method": "nvmf_subsystem_add_listener",
"params": {
"nqn": "nqn.2018-12.io.spdk:nqn0",
"listen_address": {
"trtype": "rdma",
"traddr": "192.168.200.4",
"trsvcid": null
}
},
"jsonrpc": "2.0",
"id": 1
}
response:
{
"code": -32602,
"message": "Invalid parameters"
}
Which causes these prints on the target:
nvmf_rpc.c: 517:decode_rpc_listen_address: *ERROR*: spdk_json_decode_object failed
nvmf_rpc.c: 703:nvmf_rpc_subsystem_add_listener: *ERROR*: spdk_json_decode_object failed
Any ideas where I'm going wrong here?
I also tried providing more of the optional arguments:
rsd@nvme:~/install/nvme-cli-1.6$ sudo /home/rsd/install/spdk/scripts/rpc.py
nvmf_subsystem_add_listener -t rdma -a 192.168.200.4 -f ipv4 -s 4420
nqn.2018-12.io.spdk:nqn0
Got JSON-RPC error response
request:
{
"params": {
"listen_address": {
"adrfam": "ipv4",
"trsvcid": "4420",
"traddr": "192.168.200.4",
"trtype": "rdma"
},
"nqn": "nqn.2018-12.io.spdk:nqn0"
},
"method": "nvmf_subsystem_add_listener",
"id": 1,
"jsonrpc": "2.0"
}
response:
{
"code": -32602,
"message": "Invalid parameters"
}
Matching prints on target:
libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
/usr/lib/libibverbs/libmlx4-rdmav2.so: cannot open shared object file: No such file or
directory
libibverbs: Warning: no userspace device-specific driver found for
/sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for
/sys/class/infiniband_verbs/uverbs0
rdma.c:1595:spdk_nvmf_rdma_create: *ERROR*: rdma_create_event_channel() failed, No such
device
transport.c: 93:spdk_nvmf_transport_create: *ERROR*: Unable to create new transport of
type RDMA
nvmf.c: 526:spdk_nvmf_tgt_listen: *ERROR*: Transport initialization failed
The no userspace driver message is certainly concerning. I did tinker about with the
Ubuntu packages a bit trying to get ib_send_bw to work at one point and I wonder if I
broke something along the way. Any ideas how to recover if that is the case (other than a
clean new install of Ubuntu)?
The SPDK pkgdep script does run to completion without any obvious issues and doesn't
resolve the problem.
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/pkgdep.sh
Reading package lists... Done
Building dependency tree
Reading state information... Done
g++ is already the newest version (4:5.3.1-1ubuntu1).
gcc is already the newest version (4:5.3.1-1ubuntu1).
libaio-dev is already the newest version (0.3.110-2).
libiscsi-dev is already the newest version (1.12.0-2).
make is already the newest version (4.1-6).
sg3-utils is already the newest version (1.40-0ubuntu1).
astyle is already the newest version (2.05.1-0ubuntu1).
lcov is already the newest version (1.12-2).
libcunit1-dev is already the newest version (2.1-3-dfsg-2).
pep8 is already the newest version (1.7.0-2).
git is already the newest version (1:2.7.4-0ubuntu1.6).
libssl-dev is already the newest version (1.0.2g-1ubuntu4.14).
pciutils is already the newest version (1:3.3.1-1.1ubuntu1.2).
uuid-dev is already the newest version (2.27.1-6ubuntu3.6).
clang is already the newest version (1:3.8-33ubuntu3.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree
Reading state information... Done
libunwind-dev is already the newest version (1.1-4.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree
Reading state information... Done
libibverbs-dev is already the newest version (1.1.8-1.1ubuntu2).
librdmacm-dev is already the newest version (1.0.21-1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree
Reading state information... Done
libnuma-dev is already the newest version (2.0.11-1ubuntu1.1).
nasm is already the newest version (2.11.08-1ubuntu0.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree
Reading state information... Done
doxygen is already the newest version (1.8.11-1).
mscgen is already the newest version (0.20-5).
graphviz is already the newest version (2.38.0-12ubuntu2.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree
Reading state information... Done
python-pip is already the newest version (8.1.1-2ubuntu0.4).
python3-pip is already the newest version (8.1.1-2ubuntu0.4).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
The directory '/home/rsd/.cache/pip/http' or its parent directory is not owned by
the current user and the cache has been disabled. Please check the permissions and owner
of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/rsd/.cache/pip' or its parent directory is not owned by the
current user and caching wheels has been disabled. check the permissions and owner of that
directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied (use --upgrade to upgrade): configshell_fb in
/usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pexpect in
/usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pyparsing in
/usr/lib/python2.7/dist-packages (from configshell_fb)
Requirement already satisfied (use --upgrade to upgrade): six in
/usr/lib/python2.7/dist-packages (from configshell_fb)
Requirement already satisfied (use --upgrade to upgrade): ptyprocess>=0.5 in
/usr/local/lib/python2.7/dist-packages (from pexpect)
You are using pip version 8.1.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The directory '/home/rsd/.cache/pip/http' or its parent directory is not owned by
the current user and the cache has been disabled. Please check the permissions and owner
of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/rsd/.cache/pip' or its parent directory is not owned by the
current user and caching wheels has been disabled. check the permissions and owner of that
directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied (use --upgrade to upgrade): configshell_fb in
/usr/local/lib/python3.5/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pexpect in
/usr/local/lib/python3.5/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pyparsing in
/usr/local/lib/python3.5/dist-packages (from configshell_fb)
Requirement already satisfied (use --upgrade to upgrade): six in
/usr/lib/python3/dist-packages (from configshell_fb)
Requirement already satisfied (use --upgrade to upgrade): ptyprocess>=0.5 in
/usr/local/lib/python3.5/dist-packages (from pexpect)
You are using pip version 8.1.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Crypto requires NASM version 2.12.02 or newer. Please install
or upgrade and re-run this script if you are going to use Crypto.
Thanks,
Joe
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk