Hi, You can follow this guide: NVMe over Fabrics Target Getting Started Guide http://www.spdk.io/spdk/doc/nvmf_getting_started.html 发件人: SPDK [mailto:spdk-bounces@lists.01.org] 代表 Wang, Teng 发送时间: 2016年11月30日 6:45 收件人: spdk(a)lists.01.org 主题: [SPDK] how to use spdk's nvme over fabrics Hi all, Is there any detailed instruction or sample code for using SPDK's NVMe over fabrics? How to set up the experiment? I would really appreciate if anyone give me some Instructions or materials on this. Thanks, Teng

Teng Here are some notes that I took on my system when starting the nvmf target. I put steps 1 � 3 and 5 in a script that I run whenever my system reboots. 1. Load ib_rdma_modules sudo modprobe ib_cm sudo modprobe ib_core sudo modprobe ib_ucm sudo modprobe ib_umad sudo modprobe ib_uverbs sudo modprobe iw_cm sudo modprobe rdma_cm sudo modprobe rdma_ucm 2. Detect Mellanox NICs sudo modprobe mlx5_core sudo modprobe mlx5_ib 3. Assign NICs ip addresses a. sudo ifconfig eth1 192.168.100.8 netmask 255.255.255.0 up b. sudo ifconfig eth2 192.168.100.9 netmask 255.255.255.0 up 4. Start the nvmf target cd ~/spdk/app/nvmf_tgt sudo ./nvmf_tgt -c nvmf.conf.in & My configuration file has the following Subsystems [Subsystem1] NQN nqn.2016-06.io.spdk:cnode1 Core 0 Mode Direct Listen RDMA 192.168.100.8:4420 Host nqn.2016-06.io.spdk:init NVMe 0000:81:00.0 # Multiple subsystems are allowed. [Subsystem2] NQN nqn.2016-06.io.spdk:cnode2 Core 0 Mode Direct Listen RDMA 192.168.100.9:4420 Host nqn.2016-06.io.spdk:init NVMe 0000:86:00.0 5. On the client load nvme-rdma kernel module sudo modprobe -v nvme-rdma 6. Use the cliet to discover the nvme target sudo nvme discover -t rdma -a 192.168.100.8 -s 4420 From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Walker, Benjamin Sent: Wednesday, November 30, 2016 9:57 AM To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; Subject: Re: [SPDK] how to use spdk's nvme over fabrics The guide linked below is our best attempt at documentation so far. I do really need to write more. The general outline is the following, assuming you have two systems (initiator and target) connected via RDMA NICs: 1) On the target system, compile SPDK. Follow the guide linked below to start the SPDK NVMe-oF target application. A fully documented example configuration file is in <spdk repo>/etc/nvmf/nvmf.conf.in � use that as the starting point. 2) On the initiator system, make sure you are running Linux kernel 4.8 and have nvme-cli installed. This is outside of SPDK, so we don’t provide instructions on how to do this. By far, the easiest way is to just install Fedora 24/25 on the initiator. It has a suitable kernel by default and the nvme-cli tool is packaged, so you have zero additional configuration to do. 3) The nvme-cli tool is capable of configuring the Linux kernel’s built-in NVMe-oF initiator to connect to a target. You simply type something like “nvme connect �t rdma �a <ip> -s <port> -n <nqn>”, where the values match the ones you put into the SPDK NVMe-oF target’s configuration file. The initiator will then connect and a block device will appear on the initiator system at /dev/nvmeX. Only step #1 is SPDK-specific. Step #2 and #3 are the same if you are using the Linux kernel’s NVMe-oF target � that’s why we don’t really document those today. From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Lei Zhu Sent: Tuesday, November 29, 2016 7:54 PM To: Storage Performance Development Kit <spdk@lists.01.org<mailto:spdk@lists.01.org>> Subject: [SPDK] 答复: how to use spdk's nvme over fabrics Hi, You can follow this guide: NVMe over Fabrics Target Getting Started Guide http://www.spdk.io/spdk/doc/nvmf_getting_started.html 发件人: SPDK [mailto:spdk-bounces@lists.01.org] 代表 Wang, Teng 发送时间: 2016年11月30日 6:45 收件人: spdk@lists.01.org<mailto:spdk@lists.01.org> 主题: [SPDK] how to use spdk's nvme over fabrics Hi all, Is there any detailed instruction or sample code for using SPDK's NVMe over fabrics? How to set up the experiment? I would really appreciate if anyone give me some Instructions or materials on this. Thanks, Teng

The NVMe-oF initiator will be released by the end of this month. From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Wang, Teng Sent: Wednesday, November 30, 2016 7:07 PM To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; Subject: Re: [SPDK] how to use spdk's nvme over fabrics Hi Lei, Ben and John, Thanks so much for your instructions! I heard that intel is going to release its own SPDK initiator, may I know the approximate time for this release? Thanks, Teng From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Kariuki, John K Sent: Wednesday, November 30, 2016 2:21 PM To: Storage Performance Development Kit <spdk@lists.01.org<mailto:spdk@lists.01.org>> Subject: Re: [SPDK] how to use spdk's nvme over fabrics Teng Here are some notes that I took on my system when starting the nvmf target. I put steps 1 – 3 and 5 in a script that I run whenever my system reboots. 1. Load ib_rdma_modules sudo modprobe ib_cm sudo modprobe ib_core sudo modprobe ib_ucm sudo modprobe ib_umad sudo modprobe ib_uverbs sudo modprobe iw_cm sudo modprobe rdma_cm sudo modprobe rdma_ucm 2. Detect Mellanox NICs sudo modprobe mlx5_core sudo modprobe mlx5_ib 3. Assign NICs ip addresses a. sudo ifconfig eth1 192.168.100.8 netmask 255.255.255.0 up b. sudo ifconfig eth2 192.168.100.9 netmask 255.255.255.0 up 4. Start the nvmf target cd ~/spdk/app/nvmf_tgt sudo ./nvmf_tgt -c nvmf.conf.in & My configuration file has the following Subsystems [Subsystem1] NQN nqn.2016-06.io.spdk:cnode1 Core 0 Mode Direct Listen RDMA 192.168.100.8:4420 Host nqn.2016-06.io.spdk:init NVMe 0000:81:00.0 # Multiple subsystems are allowed. [Subsystem2] NQN nqn.2016-06.io.spdk:cnode2 Core 0 Mode Direct Listen RDMA 192.168.100.9:4420 Host nqn.2016-06.io.spdk:init NVMe 0000:86:00.0 5. On the client load nvme-rdma kernel module sudo modprobe -v nvme-rdma 6. Use the cliet to discover the nvme target sudo nvme discover -t rdma -a 192.168.100.8 -s 4420 From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Walker, Benjamin Sent: Wednesday, November 30, 2016 9:57 AM To: Storage Performance Development Kit <spdk@lists.01.org<mailto:spdk@lists.01.org>> Subject: Re: [SPDK] how to use spdk's nvme over fabrics The guide linked below is our best attempt at documentation so far. I do really need to write more. The general outline is the following, assuming you have two systems (initiator and target) connected via RDMA NICs: 1) On the target system, compile SPDK. Follow the guide linked below to start the SPDK NVMe-oF target application. A fully documented example configuration file is in <spdk repo>/etc/nvmf/nvmf.conf.in – use that as the starting point. 2) On the initiator system, make sure you are running Linux kernel 4.8 and have nvme-cli installed. This is outside of SPDK, so we don’t provide instructions on how to do this. By far, the easiest way is to just install Fedora 24/25 on the initiator. It has a suitable kernel by default and the nvme-cli tool is packaged, so you have zero additional configuration to do. 3) The nvme-cli tool is capable of configuring the Linux kernel’s built-in NVMe-oF initiator to connect to a target. You simply type something like “nvme connect –t rdma –a <ip> -s <port> -n <nqn>”, where the values match the ones you put into the SPDK NVMe-oF target’s configuration file. The initiator will then connect and a block device will appear on the initiator system at /dev/nvmeX. Only step #1 is SPDK-specific. Step #2 and #3 are the same if you are using the Linux kernel’s NVMe-oF target – that’s why we don’t really document those today. From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Lei Zhu Sent: Tuesday, November 29, 2016 7:54 PM To: Storage Performance Development Kit <spdk@lists.01.org<mailto:spdk@lists.01.org>> Subject: [SPDK] 答复: how to use spdk's nvme over fabrics Hi, You can follow this guide: NVMe over Fabrics Target Getting Started Guide http://www.spdk.io/spdk/doc/nvmf_getting_started.html 发件人: SPDK [mailto:spdk-bounces@lists.01.org] 代表 Wang, Teng 发送时间: 2016年11月30日 6:45 收件人: spdk@lists.01.org<mailto:spdk@lists.01.org> 主题: [SPDK] how to use spdk's nvme over fabrics Hi all, Is there any detailed instruction or sample code for using SPDK's NVMe over fabrics? How to set up the experiment? I would really appreciate if anyone give me some Instructions or materials on this. Thanks, Teng