Hi Kiru, Yes, actually SPDK nvme library already support NVM-oF initiator, please see examples/nvme/perf as an example. It can connect local PCI NVMe or remote NVM-oF target. From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Kirubakaran Kaliannan Sent: Friday, September 22, 2017 12:57 PM To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; Subject: [SPDK] spdk user mode initiator Hi, We are evaluating the spdk-target with NVMf initiator (installing linux 4.9 only on the initator but 3.18 on the target). This is good so for, and we do have customers don’t want to upgrade 3.18 kernel even on the initiator side. So, Similarly do we have SPDK initiator (at user land) so I can have this configured on 3.18 kernel ? If we have one, any pointer to the documentation will help ? Thanks -kiru

Just to cross check, we need to build a plugin (to use the spdk SDK) for the application we have and to compile it with the spdk to use it as the initiator. Right ? As I understand there is no traditional way of exposing the device for the application to consume. Please confirm. Thanks, -kiru *From:* Kirubakaran Kaliannan [mailto:kirubak@zadarastorage.com] *Sent:* Friday, September 22, 2017 2:43 PM *To:* 'Storage Performance Development Kit' *Subject:* RE: [SPDK] spdk user mode initiator Thanks Liu. Now i can run perf test using ./perf -q 1 -s 1048576 -w read -M 100 -t 10 -r 'trtype:rdma adrfam:IPv4 traddr:10.3.7.2 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1' How can I import this as a block deivce (/dev/sd? Or /dev/nvme0?? Or some other device format), so I can use this for the application locally ? Thanks, -kiru *From:* SPDK [mailto:spdk-bounces@lists.01.org <spdk-bounces(a)lists.01.org>] *On Behalf Of *Liu, Changpeng *Sent:* Friday, September 22, 2017 10:42 AM *To:* Storage Performance Development Kit *Subject:* Re: [SPDK] spdk user mode initiator Hi Kiru, Yes, actually SPDK nvme library already support NVM-oF initiator, please see examples/nvme/perf as an example. It can connect local PCI NVMe or remote NVM-oF target. *From:* SPDK [mailto:spdk-bounces@lists.01.org <spdk-bounces(a)lists.01.org>] *On Behalf Of *Kirubakaran Kaliannan *Sent:* Friday, September 22, 2017 12:57 PM *To:* Storage Performance Development Kit <spdk(a)lists.01.org&gt; *Subject:* [SPDK] spdk user mode initiator Hi, We are evaluating the spdk-target with NVMf initiator (installing linux 4.9 only on the initator but 3.18 on the target). This is good so for, and we do have customers don’t want to upgrade 3.18 kernel even on the initiator side. So, Similarly do we have SPDK initiator (at user land) so I can have this configured on 3.18 kernel ? If we have one, any pointer to the documentation will help ? Thanks -kiru

One thing is that you can try the nvme cli with SPDK support. Reference here http://www.spdk.io/doc/nvme-cli.html You can also use nvme connect command which through spdk user mode initiator to connect the spdk nvme-of target and expose the /dev/nvmeX and /dev/nvmeXn1 devices locally. Once these devices are listed, can run the application on it. If you have some issue when using SPDK supported nvme cli, please let us know. Thanks, Gang From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Kirubakaran Kaliannan Sent: Friday, September 22, 2017 2:19 AM To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; Subject: Re: [SPDK] spdk user mode initiator Just to cross check, we need to build a plugin (to use the spdk SDK) for the application we have and to compile it with the spdk to use it as the initiator. Right ? As I understand there is no traditional way of exposing the device for the application to consume. Please confirm. Thanks, -kiru From: Kirubakaran Kaliannan [mailto:kirubak@zadarastorage.com<mailto:kirubak@zadarastorage.com>] Sent: Friday, September 22, 2017 2:43 PM To: 'Storage Performance Development Kit' Subject: RE: [SPDK] spdk user mode initiator Thanks Liu. Now i can run perf test using ./perf -q 1 -s 1048576 -w read -M 100 -t 10 -r 'trtype:rdma adrfam:IPv4 traddr:10.3.7.2 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1' How can I import this as a block deivce (/dev/sd? Or /dev/nvme0?? Or some other device format), so I can use this for the application locally ? Thanks, -kiru From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Liu, Changpeng Sent: Friday, September 22, 2017 10:42 AM To: Storage Performance Development Kit Subject: Re: [SPDK] spdk user mode initiator Hi Kiru, Yes, actually SPDK nvme library already support NVM-oF initiator, please see examples/nvme/perf as an example. It can connect local PCI NVMe or remote NVM-oF target. From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Kirubakaran Kaliannan Sent: Friday, September 22, 2017 12:57 PM To: Storage Performance Development Kit <spdk@lists.01.org<mailto:spdk@lists.01.org>> Subject: [SPDK] spdk user mode initiator Hi, We are evaluating the spdk-target with NVMf initiator (installing linux 4.9 only on the initator but 3.18 on the target). This is good so for, and we do have customers don’t want to upgrade 3.18 kernel even on the initiator side. So, Similarly do we have SPDK initiator (at user land) so I can have this configured on 3.18 kernel ? If we have one, any pointer to the documentation will help ? Thanks -kiru

Here nvme-cli (with SPDK support) and SPDK NVMe-oF initiator will both run at the user mode state. We have previously verified that SPDK NVMe-oF initiator can run on kernel 3.10 with some required steps. And use the SPDK perf tool to connect the remote SPDK NVMe-oF target and do the IOs. Below are the major steps at that time we verified on the OS 3.10 (and for 3.18, please check for the similar steps): Please keep us updated whether this works for you. Thanks. Gang Steps to run SPDK nvmf target/host using CentOS7.2 and 3.10 kernel The 3.10 kernel can’t run SPDK nvmf target/host since the lack of dependent driver and library. So we need to install Mellanox official driver to get the latest driver and library. Steps to install latest Mellanox driver The web site to download the Mellanox official driver: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linu... Choose the version which support RHEL/CentOS 7.2. The latest version may not support RHEL/CentOS 7.2. Download the ISO file since it contains a script which can do installation job. Mount the ISO file to a directory. Get into the directory. Run mlnxofedinstall scripts. Follow the steps to install it. To make the driver work, it is better to reboot the machine. Run nvmf target in target machine Clone latest spdk codes from github into the target machines. Run “git submodule update --init” command to get the DPDK codes. Run following command in SPDK directory. [localhost]./configure --with-rdma [localshot]make Run SPDK nvmf host in host machine <<<<<< Here you can use the SPDK supported nvme-cli to connect Clone latest spdk codes from github into the target machines. Run “git submodule update --init” command to get the DPDK codes. Run following command in SPDK directory. [localhost]./configure --with-rdma [localshot]make Assign IP to the Mellanox NIC port. The IP should can ping the SPDK nvmf target’s IP. Run the following command: [localhost]NRHUGE=8192 /path/to/spdk/scripts/setup.sh From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Kirubakaran Kaliannan Sent: Sunday, September 24, 2017 10:29 PM To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; Subject: Re: [SPDK] spdk user mode initiator Hi Gang, I am on 3.18 kernel on initiator. I assume, I cannot use the nvme-cli command on the initiator with 3.18 kernel. Right (as this needs nvme_fabrics driver !! )? or am I missing some option in the command to expose the nvme devices through spdk user-mode with out any support from the nvme kernel divers ? Thanks -kiru From: SPDK [mailto:spdk-bounces@lists.01.org<mailto:spdk-bounces@lists.01.org>] On Behalf Of Cao, Gang Sent: Friday, September 22, 2017 9:16 PM To: Storage Performance Development Kit Subject: Re: [SPDK] spdk user mode initiator One thing is that you can try the nvme cli with SPDK support. Reference here http://www.spdk.io/doc/nvme-cli.html You can also use nvme connect command which through spdk user mode initiator to connect the spdk nvme-of target and expose the /dev/nvmeX and /dev/nvmeXn1 devices locally. Once these devices are listed, can run the application on it. If you have some issue when using SPDK supported nvme cli, please let us know. Thanks, Gang From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Kirubakaran Kaliannan Sent: Friday, September 22, 2017 2:19 AM To: Storage Performance Development Kit <spdk@lists.01.org<mailto:spdk@lists.01.org>> Subject: Re: [SPDK] spdk user mode initiator Just to cross check, we need to build a plugin (to use the spdk SDK) for the application we have and to compile it with the spdk to use it as the initiator. Right ? As I understand there is no traditional way of exposing the device for the application to consume. Please confirm. Thanks, -kiru From: Kirubakaran Kaliannan [mailto:kirubak@zadarastorage.com<mailto:kirubak@zadarastorage.com>] Sent: Friday, September 22, 2017 2:43 PM To: 'Storage Performance Development Kit' Subject: RE: [SPDK] spdk user mode initiator Thanks Liu. Now i can run perf test using ./perf -q 1 -s 1048576 -w read -M 100 -t 10 -r 'trtype:rdma adrfam:IPv4 traddr:10.3.7.2 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1' How can I import this as a block deivce (/dev/sd? Or /dev/nvme0?? Or some other device format), so I can use this for the application locally ? Thanks, -kiru From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Liu, Changpeng Sent: Friday, September 22, 2017 10:42 AM To: Storage Performance Development Kit Subject: Re: [SPDK] spdk user mode initiator Hi Kiru, Yes, actually SPDK nvme library already support NVM-oF initiator, please see examples/nvme/perf as an example. It can connect local PCI NVMe or remote NVM-oF target. From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Kirubakaran Kaliannan Sent: Friday, September 22, 2017 12:57 PM To: Storage Performance Development Kit <spdk@lists.01.org<mailto:spdk@lists.01.org>> Subject: [SPDK] spdk user mode initiator Hi, We are evaluating the spdk-target with NVMf initiator (installing linux 4.9 only on the initator but 3.18 on the target). This is good so for, and we do have customers don’t want to upgrade 3.18 kernel even on the initiator side. So, Similarly do we have SPDK initiator (at user land) so I can have this configured on 3.18 kernel ? If we have one, any pointer to the documentation will help ? Thanks -kiru