May 2017 - SPDK - Ml01.01.Org

nvme_ctrlr.c:1224:nvme_ctrlr_process_init: ***ERROR*** Initialization timed out in state 3

by Oza Oza

I have ported SPDK for ARMv8. And DPDK is compiled with version: 16.11.1 init controller is failing. root@ubuntu:/home/oza/SPDK/spdk# odepth=128 --size=4G --readwrite=read --filename=0000.01.00.00/1 --bs=4096 --i /home/oza/fio /home/oza/SPDK/spdk EAL: pci driver is being registered 0x1nreadtest: (g=0): rw=read, bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=spdk_fio, iodepth=128 fio-2.17-29-gf0ac1 Starting 1 process Starting Intel(R) DPDK initialization ... [ DPDK EAL parameters: fio -c 1 --file-prefix=spdk_pid6448 --base-virtaddr=0x1000000000 --proc-type=auto ] EAL: Detected 8 lcore(s) EAL: Auto-detected process type: PRIMARY EAL: Probing VFIO support... EAL: VFIO support initialized EAL: cannot open /proc/self/numa_maps, consider that all memory is in socket_id 0 EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 8086:953 spdk_nvme EAL: using IOMMU type 1 (Type 1) EAL: vfio_group_fd=11 iommu_group_no=3 *vfio_dev_fd=13 EAL: reg=0x2000 fd=13 cap_offset=0x50 EAL: the msi-x bar number is 0 0x2000 0x200 EAL: Hotplug doesn't support vfio yet spdk_fio_setup() is being called EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 8086:953 spdk_nvme EAL: vfio_group_fd=11 iommu_group_no=3 *vfio_dev_fd=16 EAL: reg=0x2000 fd=16 cap_offset=0x50 EAL: the msi-x bar number is 0 0x2000 0x200 EAL: inside pci_vfio_write_config offset=4 nvme_ctrlr.c:1224:nvme_ctrlr_process_init: ***ERROR*** Initialization timed out in state 3 nvme_ctrlr.c: 403:nvme_ctrlr_shutdown: ***ERROR*** did not shutdown within 5 seconds EAL: Hotplug doesn't support vfio yet EAL: PCI device 0000:01:00.0 on NUMA socket 0 EAL: probe driver: 8086:953 spdk_nvme EAL: vfio_group_fd=11 iommu_group_no=3 *vfio_dev_fd=18 EAL: reg=0x2000 fd=18 cap_offset=0x50 EAL: the msi-x bar number is 0 0x2000 0x200 EAL: Hotplug doesn't support vfio yet EAL: Requested device 0000:01:00.0 cannot be used spdk_nvme_probe() failed Regards, Oza.

5 years, 1 month

3
15
0 / 0

VM oops while testing SPDK hotplug.sh

by Isaac Otsiabah

Daniel. I have done more testing using hotplug.sh and here is why one does not see the problem by running hotplug.sh as is. When hotplug.sh is run, it always creates a new VM to run the test on and, destroys the VM when completed. Therefore, it uses a fresh VM all the time and never gets the chance to do the inserts and remove on a VM that has run the same test several times (ie. 4 to 5 times) before. When I run hotplug.sh with a fresh VM, it passes, On the other hand, when I use the same VM that has run the test several times before, the VM oops, this is the problem. I also think this is the likely scenario the customer will experience. I wasn't clear in my earlier emails because i was still determining the cause of the problem (at the higher level at least). I think you will see the problem if you do this i. In an xterm window, Create the VM separately (without the -daemon flag). for example IMAGE=/home/fedora24/fedora24-2.img qemu-img create -f qcow2 $IMAGE 50G MEM=8192M FEDORA_ISO=/tmp/Fedora-Server-dvd-x86_64-24-1.2.iso qemu_pidfile=/tmp/qemu_pidfile qemu-2.7.1/x86_64-softmmu/qemu-system-x86_64 \ -hda $IMAGE \ -net user,hostfwd=tcp::10022-:22 \ -net nic \ -cpu host \ -m ${MEM} \ -pidfile "/tmp/qemu_pidfile" \ --enable-kvm \ -chardev socket,id=mon0,host=localhost,port=4444,ipv4,server,nowait \ -mon chardev=mon0,mode=readline \ -cdrom $FEDORA_ISO ii. Then comment out in hotplug.sh the portion that creates the VM. iii. Also comment out these 4 lines at the bottom in hotplug.sh to avoid killing the VM. qemupid=`cat "$qemu_pidfile" | awk '{printf $0}'` kill -9 $qemupid rm "$qemu_pidfile" rm "$test_img" iv. Run hotplug.sh (about 5 times and you will see the oops on VM console) The host system I am using is a Centos 7.2 Isaac From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Verkamp, Daniel Sent: Wednesday, May 17, 2017 1:09 PM To: Storage Performance Development Kit <spdk(a)lists.01.org> Subject: Re: [SPDK] VM crashes while testing SPDK hotplug Hi Isaac, The version of the hotplug script in the repository (test/lib/nvme/hotplug.sh) is the current version we are running in our automated test pool. We haven't hit the -net/--netdev issue that you mentioned yet because the version of qemu we are using is older (the current host system running this test is Fedora 25 with qemu 2.7.1). It looks like we'll need to update the script for that. We would be happy to accept a patch to hotplug.sh if the --netdev option also works on older qemu. If the kernel crashes due to user program behavior, it sounds like there is a bug in the kernel uio driver. We haven't seen this crash in our automated testing, so I am not sure what the cause could be. It is also worth trying a newer kernel version (we are just using Linux 4.5.5 because the test VM image hasn't been updated in a while). -- Daniel From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Isaac Otsiabah Sent: Monday, May 15, 2017 1:43 PM To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk@lists.01.org>> Subject: Re: [SPDK] VM crashes while testing SPDK hotplug Daniel, please, do you have an updated version of the hotplug.sh script you can share with us? I created the VM using this exact command on my Centos 7 host IMAGE=/home/fedora24/fedora24.img qemu-img create -f qcow2 $IMAGE 50G MEM=8192M FEDORA_ISO=/tmp/Fedora-Server-dvd-x86_64-24-1.2.iso /tmp/qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \ -hda $IMAGE \ -net nic,model=virtio \ -net bridge,br=br1 \ -netdev user,id=hotplug,hostfwd=tcp::10022-:22 \ -m ${MEM} \ -pidfile "/tmp/qemu_pid_fedora.txt" \ --enable-kvm \ -cpu host \ -chardev socket,id=mon0,host=localhost,port=4444,ipv4,server,nowait \ -mon chardev=mon0,mode=readline \ -cdrom $FEDORA_ISO After the install is complete, I setup the guest IP address in /etc/sysconfig/network-scripts/ifcfg-ens3 and brings up the interface with ./ifup ens3 >From the VM, I clone spdk and build. Then I run the group of test in hotplug.sh skipping the VM creation and copying spdk to the VM sections. I mentioned the -netdev flag in my earlier email Isaac From: Isaac Otsiabah Sent: Monday, May 15, 2017 1:08 PM To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk@lists.01.org>> Cc: Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com<mailto:IOtsiabah@us.fujitsu.com>>; Paul Von-Stamwitz <PVonStamwitz(a)us.fujitsu.com<mailto:PVonStamwitz@us.fujitsu.com>>; Edward Yang <eyang(a)us.fujitsu.com<mailto:eyang@us.fujitsu.com>> Subject: RE: VM crashes while testing SPDK hotplug Daniel, i installed a Fedora 24 VM and test it. After running the test twice or more, the VM oops. Unlike the previous failure on Centos, this failure does not reboot but VM oops after two or more test run. My host is a Centos machine. I found the qemu-kvm which comes with the OS installation does not support nvme so I build qemu-system-x86_64 version 2.9. [root@host1 spdk]# /tmp/qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 -version QEMU emulator version 2.9.0 Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers One observation on (although this is not the problem because I executed the scripts/setup.sh and the hotplug binary from vm console during appropriate breakpoints because local port 10022 was not responsive), the hotplug.h has the flag "-net user,hostfwd=tcp::10022-:22 \" to redirect gust ssh port 22 to host port 10022. However, qemu-system-x86_64 version 2.9 does not have this option but it has -netdev option but is is different. The qemu-system-86_64 man page on -netdev flag is as follows: -netdev user,id=str[,ipv4[=on|off]][,net=addr[/mask]][,host=addr] [,ipv6[=on|off]][,ipv6-net=addr[/int]][,ipv6-host=addr] [,restrict=on|off][,hostname=host][,dhcpstart=addr] [,dns=addr][,ipv6-dns=addr][,dnssearch=domain][,tftp=dir] [,bootfile=f][,hostfwd=rule][,guestfwd=rule][,smb=dir[,smbserver=addr]] configure a user mode network backend with ID 'str', its DHCP server and optional services It says hostfwd=rule and does not give detail of the rule. I used tcp so I specified it as -netdev user,id=hotplug,hostfwd=tcp::10022-:22 \ >From the host "netstat -an |egrep -I listen |less" I see local port 10022 is being listened on. I installed sshpass and tested this -netdev flag redirection with a simple sshpass command to the vm but got no response. Therefore, i bypassed executing scripts/setup.sh and the hotplug binary using sshpass command. So I can test it without executing setup.sh and the hotplug binary through sshpass on port 10022. The main issue is why does it oops after I run it 2 or more times? Isaac From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Verkamp, Daniel Sent: Tuesday, May 09, 2017 3:33 PM To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk@lists.01.org>> Subject: Re: [SPDK] VM crashes while testing SPDK hotplug Hi Isaac, Our hotplug tests with a VM (test/lib/nvme/hotplug.sh) are working with a Fedora 24 VM guest running kernel 4.5.5. I suspect there is a bug in the CentOS kernel version (3.10 is fairly old and is probably missing uio/hotplug-related bug fixes from the mainline kernels). Can you try to reproduce your problem on a newer kernel version and see if that is the cause of the issue? Thanks, -- Daniel From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Isaac Otsiabah Sent: Tuesday, May 9, 2017 2:11 PM To: spdk(a)lists.01.org<mailto:spdk@lists.01.org> Subject: [SPDK] VM crashes while testing SPDK hotplug I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug. 1. VM creation is as follows IMAGE=/home/centos7/centos72.img qemu-img create -f qcow2 $IMAGE 50G MEM=8192M ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso [root@host1]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root@host1]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso -r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \ -hda $IMAGE \ -net nic,model=virtio \ -net bridge,br=br1 \ -m ${MEM} \ -pidfile "/tmp/qemu_pid2.txt" \ --enable-kvm \ -cpu host \ -chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \ -mon chardev=mon0,mode=readline \ -cdrom $ISO 2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/ echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449 echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449 echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449 echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449 echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449 echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449 echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449 echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449 Also, commands to delete the devices work without crashing the VM echo "device_del nvme0" | nc localhost 4449 echo "device_del nvme1" | nc localhost 4449 echo "device_del nvme2" | nc localhost 4449 echo "device_del nvme3" | nc localhost 4449 3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. /var/log/message and I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug. 1. VM creation is as follows IMAGE=/home/centos7/centos72.img qemu-img create -f qcow2 $IMAGE 50G MEM=8192M ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso [root@host1]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root@host1]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso -r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \ -hda $IMAGE \ -net nic,model=virtio \ -net bridge,br=br1 \ -m ${MEM} \ -pidfile "/tmp/qemu_pid2.txt" \ --enable-kvm \ -cpu host \ -chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \ -mon chardev=mon0,mode=readline \ -cdrom $ISO 2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/ echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449 echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449 echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449 echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449 echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449 echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449 echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449 echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449 Also, commands to delete the devices work without crashing the VM echo "device_del nvme0" | nc localhost 4449 echo "device_del nvme1" | nc localhost 4449 echo "device_del nvme2" | nc localhost 4449 echo "device_del nvme3" | nc localhost 4449 3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. The /var/log/message and vmcore-dmesg.txt files are in the attached tar file. Would appreciate any help in why a bug in SPDK crashes the VM. Thanks. Isaac

5 years, 3 months

2
5
0 / 0

Need help for iSCSI configuration for NVMe disc

by Kiran Dikshit

Hi All, I am trying to configure iSCSI for a NVMe drive and getting the below error’s. I have unbound the NVMe drive from kernel using the setup.sh. tgt_node.c: 718:spdk_iscsi_tgt_node_construct: ***ERROR*** Could not construct SCSI device tgt_node.c: 984:spdk_cf_add_iscsi_tgt_node: ***ERROR*** tgt_node1: add_iscsi_target_node error tgt_node.c:1010:spdk_iscsi_init_tgt_nodes: ***ERROR*** spdk_cf_add_iscsi_tgt_node() failed iscsi_subsystem.c: 965:spdk_iscsi_subsystem_init: ***ERROR*** spdk_iscsi_init_tgt_nodes() failed app.c: 404:spdk_app_init: ***ERROR*** spdk_subsystem_init() failed Below is the section of my iscsi.conf.in <http://iscsi.conf.in/> file for reference. # NVMe configuration options [Nvme] # NVMe Device Whitelist # Users may specify which NVMe devices to claim by their PCI # domain, bus, device, and function. The format is dddd:bb:dd.f, which is # the same format displayed by lspci or in /sys/bus/pci/devices. The second # argument is a "name" for the device that can be anything. The name # is referenced later in the Subsystem section. # # Alternatively, the user can specify ClaimAllDevices. All # NVMe devices will be claimed and named Nvme0, Nvme1, etc. # BDF 0000:00:00.0 Nvme0 BDF 0000:0a:00.0 Nvme3 [TargetNode1] TargetName disk2 TargetAlias "Data Disk2" Mapping PortalGroup1 InitiatorGroup1 AuthMethod Auto AuthGroup AuthGroup1 UseDigest Auto LUN0 Nvme3 QueueDepth 64 Thank you Kiran

5 years, 3 months

3
11
0 / 0

VM oops while testing SPDK hotplug.sh

by Isaac Otsiabah

Hi Daniel, how are you? Daniel i performed two types of hotplug test using Fedora 25 (kernel 4.8.6-300, VM and host running the same kernel version). 1. Test 1 (Running hotplug.sh in once in a new VM): I created a Fedora25 image for the VM and used it to test hotplug.sh with no modification and it worked fine. With no modification, hotplug.sh performs the test on a newly installed VM every time and destroys the VM at the end. While this is ok, i think we should also run the test in an environment where the VM continue to run while NVMe hotplug operation is performed several times. This is a practical scenario at the customer environment so I tried to test it in Test 2 below. 2. Test 2 (Running hotplug.sh several times while VM continue to run): I created this small script to start the VM (Fedora25) on a Fedora 25 host, kernel-4.8.6. I commented out the portion that creates and destroys the VM in hotplug.sh so that the VM will keep running after a hotplug.sh run. [root@localhost ~]# cat c_kvm_with_qemu-2-7.1_helper_33_fedora.sh #!/bin/bash base_img=/home/fedora25/fedora25-1.img IMAGE=/home/fedora25/test_fedora25-1.img rm -f $IMAGE MEM=8192M qemu-img create -b $base_img -f qcow2 $IMAGE 50G /usr/bin/qemu-system-x86_64 \ -hda $IMAGE \ -net user,hostfwd=tcp::10022-:22 \ -net nic \ -cpu host \ -m ${MEM} \ -pidfile "/tmp/qemu-2-7-1_pid_fedora.txt" \ --enable-kvm \ -chardev socket,id=mon0,host=localhost,port=4444,ipv4,server,nowait \ -mon chardev=mon0,mode=readline Then I ran the modified hotpluig.sh (as mentioned above) and this is what I found. i. The VM oops as it did before in my previous report. ii. In insert_devices(), if i comment out the "ssh_vm scripts/setup.sh" to prevent setup.sh from running, it did not oops, however, inserts of new devices were not found. It appears, the problem may not be the kernel version but running setup.sh more than once is the issue. Daniel, each time setup.sh is ran, it writes 1024 to memory on the proc filesystem (/proc/sys/vm/nr_hugepages) to be used for the allocation of huge pages, could this be the issue since a prior run had already allocated huge pages? Isaac From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Isaac Otsiabah Sent: Friday, May 19, 2017 11:33 AM To: spdk(a)lists.01.org<mailto:spdk@lists.01.org> Subject: [SPDK] VM oops while testing SPDK hotplug.sh Daniel. I have done more testing using hotplug.sh and here is why one does not see the problem by running hotplug.sh as is. When hotplug.sh is run, it always creates a new VM to run the test on and, destroys the VM when completed. Therefore, it uses a fresh VM all the time and never gets the chance to do the inserts and remove on a VM that has run the same test several times (ie. 4 to 5 times) before. When I run hotplug.sh with a fresh VM, it passes, On the other hand, when I use the same VM that has run the test several times before, the VM oops, this is the problem. I also think this is the likely scenario the customer will experience. I wasn't clear in my earlier emails because i was still determining the cause of the problem (at the higher level at least). I think you will see the problem if you do this i. In an xterm window, Create the VM separately (without the -daemon flag). for example IMAGE=/home/fedora24/fedora24-2.img qemu-img create -f qcow2 $IMAGE 50G MEM=8192M FEDORA_ISO=/tmp/Fedora-Server-dvd-x86_64-24-1.2.iso qemu_pidfile=/tmp/qemu_pidfile qemu-2.7.1/x86_64-softmmu/qemu-system-x86_64 \ -hda $IMAGE \ -net user,hostfwd=tcp::10022-:22 \ -net nic \ -cpu host \ -m ${MEM} \ -pidfile "/tmp/qemu_pidfile" \ --enable-kvm \ -chardev socket,id=mon0,host=localhost,port=4444,ipv4,server,nowait \ -mon chardev=mon0,mode=readline \ -cdrom $FEDORA_ISO ii. Then comment out in hotplug.sh the portion that creates the VM. iii. Also comment out these 4 lines at the bottom in hotplug.sh to avoid killing the VM. qemupid=`cat "$qemu_pidfile" | awk '{printf $0}'` kill -9 $qemupid rm "$qemu_pidfile" rm "$test_img" iv. Run hotplug.sh (about 5 times and you will see the oops on VM console) The host system I am using is a Centos 7.2 Isaac From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Verkamp, Daniel Sent: Wednesday, May 17, 2017 1:09 PM To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk@lists.01.org>> Subject: Re: [SPDK] VM crashes while testing SPDK hotplug Hi Isaac, The version of the hotplug script in the repository (test/lib/nvme/hotplug.sh) is the current version we are running in our automated test pool. We haven't hit the -net/--netdev issue that you mentioned yet because the version of qemu we are using is older (the current host system running this test is Fedora 25 with qemu 2.7.1). It looks like we'll need to update the script for that. We would be happy to accept a patch to hotplug.sh if the --netdev option also works on older qemu. If the kernel crashes due to user program behavior, it sounds like there is a bug in the kernel uio driver. We haven't seen this crash in our automated testing, so I am not sure what the cause could be. It is also worth trying a newer kernel version (we are just using Linux 4.5.5 because the test VM image hasn't been updated in a while). -- Daniel From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Isaac Otsiabah Sent: Monday, May 15, 2017 1:43 PM To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk@lists.01.org>> Subject: Re: [SPDK] VM crashes while testing SPDK hotplug Daniel, please, do you have an updated version of the hotplug.sh script you can share with us? I created the VM using this exact command on my Centos 7 host IMAGE=/home/fedora24/fedora24.img qemu-img create -f qcow2 $IMAGE 50G MEM=8192M FEDORA_ISO=/tmp/Fedora-Server-dvd-x86_64-24-1.2.iso /tmp/qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \ -hda $IMAGE \ -net nic,model=virtio \ -net bridge,br=br1 \ -netdev user,id=hotplug,hostfwd=tcp::10022-:22 \ -m ${MEM} \ -pidfile "/tmp/qemu_pid_fedora.txt" \ --enable-kvm \ -cpu host \ -chardev socket,id=mon0,host=localhost,port=4444,ipv4,server,nowait \ -mon chardev=mon0,mode=readline \ -cdrom $FEDORA_ISO After the install is complete, I setup the guest IP address in /etc/sysconfig/network-scripts/ifcfg-ens3 and brings up the interface with ./ifup ens3 >From the VM, I clone spdk and build. Then I run the group of test in hotplug.sh skipping the VM creation and copying spdk to the VM sections. I mentioned the -netdev flag in my earlier email Isaac From: Isaac Otsiabah Sent: Monday, May 15, 2017 1:08 PM To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk@lists.01.org>> Cc: Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com<mailto:IOtsiabah@us.fujitsu.com>>; Paul Von-Stamwitz <PVonStamwitz(a)us.fujitsu.com<mailto:PVonStamwitz@us.fujitsu.com>>; Edward Yang <eyang(a)us.fujitsu.com<mailto:eyang@us.fujitsu.com>> Subject: RE: VM crashes while testing SPDK hotplug Daniel, i installed a Fedora 24 VM and test it. After running the test twice or more, the VM oops. Unlike the previous failure on Centos, this failure does not reboot but VM oops after two or more test run. My host is a Centos machine. I found the qemu-kvm which comes with the OS installation does not support nvme so I build qemu-system-x86_64 version 2.9. [root@host1 spdk]# /tmp/qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 -version QEMU emulator version 2.9.0 Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers One observation on (although this is not the problem because I executed the scripts/setup.sh and the hotplug binary from vm console during appropriate breakpoints because local port 10022 was not responsive), the hotplug.h has the flag "-net user,hostfwd=tcp::10022-:22 \" to redirect gust ssh port 22 to host port 10022. However, qemu-system-x86_64 version 2.9 does not have this option but it has -netdev option but is is different. The qemu-system-86_64 man page on -netdev flag is as follows: -netdev user,id=str[,ipv4[=on|off]][,net=addr[/mask]][,host=addr] [,ipv6[=on|off]][,ipv6-net=addr[/int]][,ipv6-host=addr] [,restrict=on|off][,hostname=host][,dhcpstart=addr] [,dns=addr][,ipv6-dns=addr][,dnssearch=domain][,tftp=dir] [,bootfile=f][,hostfwd=rule][,guestfwd=rule][,smb=dir[,smbserver=addr]] configure a user mode network backend with ID 'str', its DHCP server and optional services It says hostfwd=rule and does not give detail of the rule. I used tcp so I specified it as -netdev user,id=hotplug,hostfwd=tcp::10022-:22 \ >From the host "netstat -an |egrep -I listen |less" I see local port 10022 is being listened on. I installed sshpass and tested this -netdev flag redirection with a simple sshpass command to the vm but got no response. Therefore, i bypassed executing scripts/setup.sh and the hotplug binary using sshpass command. So I can test it without executing setup.sh and the hotplug binary through sshpass on port 10022. The main issue is why does it oops after I run it 2 or more times? Isaac From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Verkamp, Daniel Sent: Tuesday, May 09, 2017 3:33 PM To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk@lists.01.org>> Subject: Re: [SPDK] VM crashes while testing SPDK hotplug Hi Isaac, Our hotplug tests with a VM (test/lib/nvme/hotplug.sh) are working with a Fedora 24 VM guest running kernel 4.5.5. I suspect there is a bug in the CentOS kernel version (3.10 is fairly old and is probably missing uio/hotplug-related bug fixes from the mainline kernels). Can you try to reproduce your problem on a newer kernel version and see if that is the cause of the issue? Thanks, -- Daniel From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Isaac Otsiabah Sent: Tuesday, May 9, 2017 2:11 PM To: spdk(a)lists.01.org<mailto:spdk@lists.01.org> Subject: [SPDK] VM crashes while testing SPDK hotplug I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug. 1. VM creation is as follows IMAGE=/home/centos7/centos72.img qemu-img create -f qcow2 $IMAGE 50G MEM=8192M ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso [root@host1]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root@host1]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso -r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \ -hda $IMAGE \ -net nic,model=virtio \ -net bridge,br=br1 \ -m ${MEM} \ -pidfile "/tmp/qemu_pid2.txt" \ --enable-kvm \ -cpu host \ -chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \ -mon chardev=mon0,mode=readline \ -cdrom $ISO 2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/ echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449 echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449 echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449 echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449 echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449 echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449 echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449 echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449 Also, commands to delete the devices work without crashing the VM echo "device_del nvme0" | nc localhost 4449 echo "device_del nvme1" | nc localhost 4449 echo "device_del nvme2" | nc localhost 4449 echo "device_del nvme3" | nc localhost 4449 3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. /var/log/message and I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug. 1. VM creation is as follows IMAGE=/home/centos7/centos72.img qemu-img create -f qcow2 $IMAGE 50G MEM=8192M ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso [root@host1]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root@host1]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso -r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \ -hda $IMAGE \ -net nic,model=virtio \ -net bridge,br=br1 \ -m ${MEM} \ -pidfile "/tmp/qemu_pid2.txt" \ --enable-kvm \ -cpu host \ -chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \ -mon chardev=mon0,mode=readline \ -cdrom $ISO 2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/ echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449 echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449 echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449 echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449 echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449 echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449 echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449 echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449 Also, commands to delete the devices work without crashing the VM echo "device_del nvme0" | nc localhost 4449 echo "device_del nvme1" | nc localhost 4449 echo "device_del nvme2" | nc localhost 4449 echo "device_del nvme3" | nc localhost 4449 3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. The /var/log/message and vmcore-dmesg.txt files are in the attached tar file. Would appreciate any help in why a bug in SPDK crashes the VM. Thanks. Isaac

5 years, 3 months

1
0
0 / 0

An Open, Transparent Patch Submission Process

by Walker, Benjamin

Hi all, We're excited to announce a new patch submission process for SPDK utilizing GerritHub, a hosted code review service integrated with GitHub. All patches, even from SPDK maintainers, will now use the new process, which means all code reviews and discussions are now fully open and transparent to everyone. We encourage everyone, even if you aren't an expert, to review patches and vote with a +1 or -1! A quick query for all open patches is here (better queries and dashboard set up are in the guide at the bottom): https://review.gerrithub.io/#/q/project:spdk/spdk+status:open Further, the SPDK continuous integration tests are hooked up to GerritHub, which means that when you submit a patch a whole battery of tests on a wide range of systems will be run with results reported directly to the code review. You can view the current status of the continuous integration system at: http://ci.spdk.io A comprehensive guide to the new patch submission process can be found here: http://www.spdk.io/development A quick start guide is still in the works. We're thrilled to open this system up to everyone, giving all community members access to the same tools the SPDK maintainers have. Thanks, The SPDK maintainers

5 years, 4 months

1
0
0 / 0

VM not booting up with SPDK vhost

by Abhik Sarkar

Hi, This is actually a follow-up email as I was suggested to post the query on the mailing list. So, we are running into an issue while setting up SPDK vhost. Our guest OS does not seem to boot-up after it looks like Qemu has setup communication with the vhost app. We first faced the problem with guest running Scientific Linux 6.1, kernel version 2.6.32-696.1.1.el6. On James Harris’ recommendation, we tried running guest with CentOS 7, kernel version 3.10.0-514.el7.x86_64. But we see the same issue. I have attached 2 different config files. One with the vhost setting and one without. centos7.1.xml : This does not have any vhost related command line arguments while launching the VM Status: The VM launches just fine and we are able to login to the VM vcentos7.1.xml: This one has the vhost setting appended Status: The vhost-target output seems to show that the vhost-client (qemu) has been setup properly (vhost_logs.txt) but we can’t login to the VM. Follwing is the arp output # arp -an centos7.1 -->? (192.168.122.24) at 52:54:00:47:0c:3d [ether] on virbr0 vcentos7.1 -->? (192.168.122.204) at <incomplete> on virbr0 Also the console does not show any boot messages or login prompt. # virsh ttyconsole vcentos7.1 /dev/pts/0 #virsh console vcentos7.1 Connected to domain vcentos7.1 Escape character is ^] The qemu build used is the spdk branch with the latest commit ID being 6eed2b580b6d5bc853b8efc15931eaf0c0de68a2 Thanks/Regards -a

5 years, 4 months

2
1
0 / 0

"Open file failed: No such file or directory" error in examples/nvme/nvme_manage/nvme_manage tool

by Isaac Otsiabah

I experimented with examples/nvme/nvme_manage/nvme_manage tool. After the prompt to enter the firmware path name, I entered the firmware full path name and got "Open file failed: No such file or directory" error. After a little debug I found in update_firmware_image(void) function line 775, fgets(..) reads the new line character and puts it at the end of the input buffer. This new line character must be removed from the input buffer before the buffer contents is used. 775 if (fgets(path, 256, stdin) == NULL) { 776 printf("Invalid path setting\n"); 777 while (getchar() != '\n'); 778 return; 779 } The same problem is also in the get_controller() function after fgets is used to read the PCI Address from standard input. 300 while ((ch = getchar()) != '\n' && ch != EOF); 301 p = fgets(address, 64, stdin); 302 if (p == NULL) { 303 return NULL; 304 } Both can be fixed by removing the new line character from the input buffer after fgets return Ie. if (path[strlen(path)-1]=='\n') path[strlen(path)-1]='\0'; Isaac [root@spdk2 spdk]# ./examples/nvme/nvme_manage/nvme_manage Starting DPDK 17.02.0 initialization... [ DPDK EAL parameters: nvme_manage -c 0x1 --file-prefix=spdk_pid19289 ] EAL: Detected 16 lcore(s) EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: PCI device 0000:02:00.0 on NUMA socket 0 EAL: probe driver: 8086:953 spdk_nvme EAL: PCI device 0000:03:00.0 on NUMA socket 0 EAL: probe driver: 8086:953 spdk_nvme EAL: PCI device 0000:04:00.0 on NUMA socket 0 EAL: probe driver: 8086:953 spdk_nvme EAL: PCI device 0000:05:00.0 on NUMA socket 0 EAL: probe driver: 8086:953 spdk_nvme IDENTIFY (06) sqid:0 cid:59 nsid:ffffffff cdw10:00000000 cdw11:00000000 INVALID NAMESPACE OR FORMAT (00/0b) sqid:0 cid:59 cdw0:0 sqhd:000a p:1 m:0 dnr:0 IDENTIFY (06) sqid:0 cid:59 nsid:ffffffff cdw10:00000000 cdw11:00000000 INVALID NAMESPACE OR FORMAT (00/0b) sqid:0 cid:59 cdw0:0 sqhd:000a p:1 m:0 dnr:0 IDENTIFY (06) sqid:0 cid:59 nsid:ffffffff cdw10:00000000 cdw11:00000000 INVALID NAMESPACE OR FORMAT (00/0b) sqid:0 cid:59 cdw0:0 sqhd:000a p:1 m:0 dnr:0 IDENTIFY (06) sqid:0 cid:59 nsid:ffffffff cdw10:00000000 cdw11:00000000 INVALID NAMESPACE OR FORMAT (00/0b) sqid:0 cid:59 cdw0:0 sqhd:000a p:1 m:0 dnr:0 NVMe Management Options [1: list controllers] [2: create namespace] [3: delete namespace] [4: attach namespace to controller] [5: detach namespace from controller] [6: format namespace or controller] [7: firmware update] [8: quit] 7 0000:02:00.00 INTEL SSDPEDMD800G4 CVFT4216000E800CGN 0 0000:03:00.00 INTEL SSDPEDMD800G4 CVFT4216000G800CGN 0 0000:04:00.00 INTEL SSDPEDMD800G4 CVFT421500DG800CGN 0 0000:05:00.00 INTEL SSDPEDMD800G4 CVFT421500DE800CGN 0 Please Input PCI Address(domain:bus:dev.func): 0000:05:00.00 Please Input The Path Of Firmware Image /root/spdk/image/8DV101B0_8B1B0133_signed.bin Open file failed: No such file or directory press Enter to display cmd menu ...

5 years, 4 months

2
2
0 / 0

valgrind issues related to spdk

by Rukhsana Ansari

Hi, I tried using valgrind-3.11.0-24.el7.x86_64 for my app based on spdk and found that rte_eal_init->eal_hugepage_info_init->clear_hugedir takes an extremely long time to complete and rte_eal_pci_init() seems to hang. Is valgrind-3.11.0-24.el7.x86_64 supported on spdk? As specified in: https://stackoverflow.com/questions/37386738/error-when-running-dpdk-app-... I built dpdk with set CONFIG_RTE_MACHINE="snb". Based on this other email thread, https://lists.01.org/pipermail/spdk/2017-March/000409.html I tried using *https://github.com/bluca/valgrind-dpdk* <https://github.com/bluca/valgrind-dpdk> valgrind --leak-check=full --show-leak-kinds=all -v -v -v -d -d -d myprog but ran into this error: valgrind: m_deduppoolalloc.c:258 (vgPlain_allocEltDedupPA): Assertion 'eltSzB <= ddpa->poolSzB' failed. Segmentation fault (core dumped) Any pointers to help running valgrind on spdk will be much appreciated. Thanks, -- -Rukhsana

5 years, 4 months

1
0
0 / 0

Could not find NVMe controller at PCI address 0000:06:00.0

by Hugh Daschbach

I'm having issues getting app/nvmf_tgt/nvmf_tgt running. I've configured four of five local NVMe devices in nvmf.conf. The devices are discovered during early initialization, but spdk_nvme_probe() fails to find a previously probed device. It looks like nvme_pci_ctrlr_scan() expects a hotplug event that it never sees. What is supposed to trigger the hotplug event? Here's the output of nvmf_tgt: Starting DPDK 17.08.0-rc0 initialization... [ DPDK EAL parameters: nvmf -c 0x5555 --file-prefix=spdk_pid184703 ] EAL: Detected 72 lcore(s) EAL: No free hugepages reported in hugepages-1048576kB EAL: Debug dataplane logs available - lower performance EAL: Probing VFIO support... EAL: VFIO support initialized Occupied cpu socket mask is 0x1 Ioat Copy Engine Offload Enabled EAL: PCI device 0000:06:00.0 on NUMA socket 0 EAL: using IOMMU type 1 (Type 1) EAL: PCI device 0000:07:00.0 on NUMA socket 0 EAL: PCI device 0000:08:00.0 on NUMA socket 0 EAL: PCI device 0000:09:00.0 on NUMA socket 0 EAL: PCI device 0000:84:00.0 on NUMA socket 1 EAL: Releasing pci mapped resource for 0000:84:00.0 EAL: Calling pci_unmap_resource for 0000:84:00.0 at 0x7f194ae10000 Reactor started on core 2 on socket 0 Reactor started on core 4 on socket 0 Reactor started on core 6 on socket 0 Reactor started on core 8 on socket 0 Reactor started on core 10 on socket 0 Reactor started on core 12 on socket 0 Reactor started on core 14 on socket 0 Reactor started on core 0 on socket 0 *** RDMA Transport Init *** allocated subsystem nqn.2014-08.org.nvmexpress.discovery on lcore 0 on socket 0 allocated subsystem nqn.2016-06.io.spdk:cnode0 on lcore 2 on socket 0 Total cores available: 8 EAL: PCI device 0000:06:00.0 on NUMA socket 0 conf.c: 578:spdk_nvmf_construct_subsystem: ***ERROR*** Could not find NVMe controller at PCI address 0000:06:00.0 nvmf_tgt.c: 279:spdk_nvmf_startup: ***ERROR*** spdk_nvmf_parse_conf() failed EAL: Releasing pci mapped resource for 0000:06:00.0 EAL: Calling pci_unmap_resource for 0000:06:00.0 at 0x7f194ae00000 EAL: Releasing pci mapped resource for 0000:07:00.0 EAL: Calling pci_unmap_resource for 0000:07:00.0 at 0x7f194ae04000 EAL: Releasing pci mapped resource for 0000:08:00.0 EAL: Calling pci_unmap_resource for 0000:08:00.0 at 0x7f194ae08000 EAL: Releasing pci mapped resource for 0000:09:00.0 EAL: Calling pci_unmap_resource for 0000:09:00.0 at 0x7f194ae0c000 Select bits from nvmf.conf: [Nvme] TransportID "trtype:PCIe traddr:0000:05:00.0" Nvme0 TransportID "trtype:PCIe traddr:0000:06:00.0" Nvme1 TransportID "trtype:PCIe traddr:0000:07:00.0" Nvme2 TransportID "trtype:PCIe traddr:0000:08:00.0" Nvme3 TransportID "trtype:PCIe traddr:0000:09:00.0" Nvme4 [Subsystem0] NQN nqn.2016-06.io.spdk:cnode0 Core 2 Mode Direct Listen RDMA 192.168.10.1:4420 SN SPDK00000000000001 NVMe 0000:06:00.0 [Subsystem1] NQN nqn.2016-06.io.spdk:cnode1 Core 4 Mode Direct Listen RDMA 192.168.10.1:4420 SN SPDK00000000000002 NVMe 0000:07:00.0 [Subsystem2] NQN nqn.2016-06.io.spdk:cnode2 Core 6 Mode Direct Listen RDMA 192.168.10.1:4420 SN SPDK00000000000003 NVMe 0000:08:00.0 [Subsystem3] NQN nqn.2016-06.io.spdk:cnode3 Core 8 Mode Direct Listen RDMA 192.168.10.1:4420 SN SPDK00000000000004 NVMe 0000:09:00.0 and lspci output following the errror: [root@localhost spdk]# lspci -v -s 0:6:0.0 06:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 171X (rev 03) (prog-if 02 [NVM Express]) Subsystem: Dell Express Flash NVMe XS1715 SSD 800GB Physical Slot: 180 Flags: fast devsel, IRQ 31, NUMA node 0 Memory at 9a400000 (64-bit, non-prefetchable) [disabled] [size=16K] Capabilities: [c0] Power Management version 3 Capabilities: [c8] MSI: Enable- Count=1/32 Maskable+ 64bit+ Capabilities: [e0] MSI-X: Enable- Count=129 Masked- Capabilities: [70] Express Endpoint, MSI 00 Capabilities: [40] Vendor Specific Information: Len=24 <?> Capabilities: [100] Advanced Error Reporting Capabilities: [180] #19 Capabilities: [150] Vendor Specific Information: ID=0001 Rev=1 Len=02c <?> Kernel driver in use: vfio-pci Kernel modules: nvme The above is from branch master (both SPDK and DPDK). I've tested v17.03 with similar results. What am I missing? Thanks, Hugh

5 years, 4 months

2
2
0 / 0

New feature - Blobs as bdevs

by Pelplinski, Piotr

Hi everyone! Today we'll be discussing and reviewing new early concept for spdk - using blobs as bdevs in vhost. You're welcome to join our meeting at +1(916)356-2663. Choose bridge 5, Conference ID: 656560618. Meeting starts today, 05/18/2017 at 08:00 AM MST (UTC-7). -- Best Regards, Piotr Pelpliński -------------------------------------------------------------------- Intel Technology Poland sp. z o.o. ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN. Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek przegladanie lub rozpowszechnianie jest zabronione. This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.

5 years, 4 months

1
1
0 / 0

2022

2021

2020

2019

2018

2017

2016

SPDK May 2017