Trouble Setting Up NVMeoF Target

Tuesday, 25 December 2018

Hi everyone.  I'm running Ubuntu 16.04 with SPDK v18.10.1.  I'm having trouble
setting up the NVMeoF target.  I can't seem to spot the problem, so maybe someone can
help me out here.  I'm pretty sure I'm either formatting the command incorrectly
or I've screwed up the IB/RDMA related packages in my Ubuntu install.

I have a Mellanox CX5 NIC at 192.168.200.4, interface is up and linked:

rsd@nvme:~$ ifconfig enp6s0
enp6s0    Link encap:Ethernet  HWaddr 98:03:9b:1e:de:ec
          inet addr:192.168.200.4  Bcast:192.168.200.255  Mask:255.255.255.0
          inet6 addr: fe80::9a03:9bff:fe1e:deec/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:138 errors:0 dropped:0 overruns:0 frame:0
          TX packets:135 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:19029 (19.0 KB)  TX bytes:24279 (24.2 KB)
rsd@nvme:~$ sudo ethtool enp6s0
Settings for enp6s0:
        Supported ports: [ Backplane ]
        Supported link modes:   1000baseKX/Full
                                10000baseKR/Full
                                40000baseKR4/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                40000baseLR4/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Advertised link modes:  1000baseKX/Full
                                10000baseKR/Full
                                40000baseKR4/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                40000baseLR4/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
       Link partner advertised link modes:  Not reported
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Speed: 100000Mb/s
        Duplex: Full
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000004 (4)
                               link
        Link detected: yes

I ran setup and then started the target and gave it four cores:
rsd@nvme:~$ sudo ~/install/spdk/scripts/setup.sh
[sudo] password for rsd:
0000:d8:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:d9:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
rsd@nvme:~$ sudo /home/rsd/install/spdk/app/nvmf_tgt/nvmf_tgt -m 0xF
Starting SPDK v18.10.1 / DPDK 18.08.0 initialization...
[ DPDK EAL parameters: nvmf --no-shconf -c 0xF --file-prefix=spdk_pid42493 ]
EAL: Detected 80 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
app.c: 602:spdk_app_start: *NOTICE*: Total cores available: 4
reactor.c: 703:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1
reactor.c: 490:_spdk_reactor_run: *NOTICE*: Reactor started on core 1 on socket 0
reactor.c: 490:_spdk_reactor_run: *NOTICE*: Reactor started on core 2 on socket 0
reactor.c: 490:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 on socket 0
reactor.c: 490:_spdk_reactor_run: *NOTICE*: Reactor started on core 3 on socket 0

I created my bdevs from my local NVMe devices:
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py construct_nvme_bdev -b d0 -t pcie
-a 0000:d8:00.0
d0n1
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py construct_nvme_bdev -b d1 -t pcie
-a 0000:d9:00.0
d1n1
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py construct_nvme_bdev -b d2 -t pcie
-a 0000:da:00.0
d2n1
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py construct_nvme_bdev -b d3 -t pcie
-a 0000:db:00.0
d3n1

Matching prints from the target:
EAL: PCI device 0000:d8:00.0 on NUMA socket 1
EAL:   probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:d9:00.0 on NUMA socket 1
EAL:   probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:da:00.0 on NUMA socket 1
EAL:   probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:db:00.0 on NUMA socket 1
EAL:   probe driver: 8086:a54 spdk_nvme

Then the problem starts when I try to create my NVMeoF subsystem.  If I use this
deprecated method I get an invalid parameters error:
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py construct_nvmf_subsystem -a -n d0n1
-m 1 nqn.2018-12.io.spdk:nqn0 "trtype:RDMA traddr:192.168.200.4 trsvcid:4420"
""
Got JSON-RPC error response
request:
{
  "jsonrpc": "2.0",
  "params": {
    "max_namespaces": 1,
    "nqn": "nqn.2018-12.io.spdk:nqn0",
    "allow_any_host": true,
    "serial_number": "00000000000000000000",
    "listen_addresses": [
      {
        "traddr": "192.168.200.4",
        "trtype": "RDMA",
        "trsvcid": "4420"
      }
    ],
    "namespaces": [
      {
        "bdev_name": "d0n1"
      }
    ]
  },
  "method": "construct_nvmf_subsystem",
  "id": 1
}
response:
{
  "message": "Invalid parameters",
  "code": -32602
}

Matching unhappy prints on the target:
nvmf_rpc_deprecated.c: 490:spdk_rpc_construct_nvmf_subsystem: *WARNING*: The
construct_nvmf_subsystem RPC is deprecated. Use nvmf_subsystem_create instead.
libibverbs: Warning: no userspace device-specific driver found for
/sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for
/sys/class/infiniband_verbs/uverbs0
rdma.c:1595:spdk_nvmf_rdma_create: *ERROR*: rdma_create_event_channel() failed, No such
device
transport.c:  93:spdk_nvmf_transport_create: *ERROR*: Unable to create new transport of
type RDMA
nvmf.c: 526:spdk_nvmf_tgt_listen: *ERROR*: Transport initialization failed

If I try to use the newer method, something similar happens, this part goes OK:
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_create
nqn.2018-12.io.spdk:nqn0
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_create
nqn.2018-12.io.spdk:nqn1
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_create
nqn.2018-12.io.spdk:nqn2
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_create
nqn.2018-12.io.spdk:nqn3
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_add_ns
nqn.2018-12.io.spdk:nqn0 d0n1
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_add_ns
nqn.2018-12.io.spdk:nqn1 d1n1
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_add_ns
nqn.2018-12.io.spdk:nqn2 d2n1
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_add_ns
nqn.2018-12.io.spdk:nqn3 d3n1

But then I can't add the transport:
rsd@nvme:~/install/nvme-cli-1.6$ sudo /home/rsd/install/spdk/scripts/rpc.py
nvmf_subsystem_add_listener -t rdma -a 192.168.200.4 nqn.2018-12.io.spdk:nqn0
Got JSON-RPC error response
request:
{
  "method": "nvmf_subsystem_add_listener",
  "params": {
    "nqn": "nqn.2018-12.io.spdk:nqn0",
    "listen_address": {
      "trtype": "rdma",
      "traddr": "192.168.200.4",
      "trsvcid": null
    }
  },
  "jsonrpc": "2.0",
  "id": 1
}
response:
{
  "code": -32602,
  "message": "Invalid parameters"
}

Which causes these prints on the target:
nvmf_rpc.c: 517:decode_rpc_listen_address: *ERROR*: spdk_json_decode_object failed
nvmf_rpc.c: 703:nvmf_rpc_subsystem_add_listener: *ERROR*: spdk_json_decode_object failed

Any ideas where I'm going wrong here?

I also tried providing more of the optional arguments:
rsd@nvme:~/install/nvme-cli-1.6$ sudo /home/rsd/install/spdk/scripts/rpc.py
nvmf_subsystem_add_listener -t rdma -a 192.168.200.4 -f ipv4 -s 4420
nqn.2018-12.io.spdk:nqn0
Got JSON-RPC error response
request:
{
  "params": {
    "listen_address": {
      "adrfam": "ipv4",
      "trsvcid": "4420",
      "traddr": "192.168.200.4",
      "trtype": "rdma"
    },
    "nqn": "nqn.2018-12.io.spdk:nqn0"
  },
  "method": "nvmf_subsystem_add_listener",
  "id": 1,
  "jsonrpc": "2.0"
}
response:
{
  "code": -32602,
  "message": "Invalid parameters"
}

Matching prints on target:
libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4':
/usr/lib/libibverbs/libmlx4-rdmav2.so: cannot open shared object file: No such file or
directory
libibverbs: Warning: no userspace device-specific driver found for
/sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for
/sys/class/infiniband_verbs/uverbs0
rdma.c:1595:spdk_nvmf_rdma_create: *ERROR*: rdma_create_event_channel() failed, No such
device
transport.c:  93:spdk_nvmf_transport_create: *ERROR*: Unable to create new transport of
type RDMA
nvmf.c: 526:spdk_nvmf_tgt_listen: *ERROR*: Transport initialization failed

The no userspace driver message is certainly concerning.  I did tinker about with the
Ubuntu packages a bit trying to get ib_send_bw to work at one point and I wonder if I
broke something along the way.  Any ideas how to recover if that is the case (other than a
clean new install of Ubuntu)?

The SPDK pkgdep script does run to completion without any obvious issues and doesn't
resolve the problem.

rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/pkgdep.sh
Reading package lists... Done
Building dependency tree
Reading state information... Done
g++ is already the newest version (4:5.3.1-1ubuntu1).
gcc is already the newest version (4:5.3.1-1ubuntu1).
libaio-dev is already the newest version (0.3.110-2).
libiscsi-dev is already the newest version (1.12.0-2).
make is already the newest version (4.1-6).
sg3-utils is already the newest version (1.40-0ubuntu1).
astyle is already the newest version (2.05.1-0ubuntu1).
lcov is already the newest version (1.12-2).
libcunit1-dev is already the newest version (2.1-3-dfsg-2).
pep8 is already the newest version (1.7.0-2).
git is already the newest version (1:2.7.4-0ubuntu1.6).
libssl-dev is already the newest version (1.0.2g-1ubuntu4.14).
pciutils is already the newest version (1:3.3.1-1.1ubuntu1.2).
uuid-dev is already the newest version (2.27.1-6ubuntu3.6).
clang is already the newest version (1:3.8-33ubuntu3.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree
Reading state information... Done
libunwind-dev is already the newest version (1.1-4.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree
Reading state information... Done
libibverbs-dev is already the newest version (1.1.8-1.1ubuntu2).
librdmacm-dev is already the newest version (1.0.21-1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree
Reading state information... Done
libnuma-dev is already the newest version (2.0.11-1ubuntu1.1).
nasm is already the newest version (2.11.08-1ubuntu0.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree
Reading state information... Done
doxygen is already the newest version (1.8.11-1).
mscgen is already the newest version (0.20-5).
graphviz is already the newest version (2.38.0-12ubuntu2.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree
Reading state information... Done
python-pip is already the newest version (8.1.1-2ubuntu0.4).
python3-pip is already the newest version (8.1.1-2ubuntu0.4).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
The directory '/home/rsd/.cache/pip/http' or its parent directory is not owned by
the current user and the cache has been disabled. Please check the permissions and owner
of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/rsd/.cache/pip' or its parent directory is not owned by the
current user and caching wheels has been disabled. check the permissions and owner of that
directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied (use --upgrade to upgrade): configshell_fb in
/usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pexpect in
/usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pyparsing in
/usr/lib/python2.7/dist-packages (from configshell_fb)
Requirement already satisfied (use --upgrade to upgrade): six in
/usr/lib/python2.7/dist-packages (from configshell_fb)
Requirement already satisfied (use --upgrade to upgrade): ptyprocess>=0.5 in
/usr/local/lib/python2.7/dist-packages (from pexpect)
You are using pip version 8.1.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The directory '/home/rsd/.cache/pip/http' or its parent directory is not owned by
the current user and the cache has been disabled. Please check the permissions and owner
of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/rsd/.cache/pip' or its parent directory is not owned by the
current user and caching wheels has been disabled. check the permissions and owner of that
directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied (use --upgrade to upgrade): configshell_fb in
/usr/local/lib/python3.5/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pexpect in
/usr/local/lib/python3.5/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pyparsing in
/usr/local/lib/python3.5/dist-packages (from configshell_fb)
Requirement already satisfied (use --upgrade to upgrade): six in
/usr/lib/python3/dist-packages (from configshell_fb)
Requirement already satisfied (use --upgrade to upgrade): ptyprocess>=0.5 in
/usr/local/lib/python3.5/dist-packages (from pexpect)
You are using pip version 8.1.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Crypto requires NASM version 2.12.02 or newer. Please install
or upgrade and re-run this script if you are going to use Crypto.

Thanks,
Joe

2021

2020

2019

2018

2017

2016