nvme_ctrlr.c:1224:nvme_ctrlr_process_init: ***ERROR*** Initialization timed out in state 3
by Oza Oza
I have ported SPDK for ARMv8.
And DPDK is compiled with version: 16.11.1
init controller is failing.
root@ubuntu:/home/oza/SPDK/spdk#
odepth=128 --size=4G --readwrite=read --filename=0000.01.00.00/1 --bs=4096
--i
/home/oza/fio /home/oza/SPDK/spdk
EAL: pci driver is being registered 0x1nreadtest: (g=0): rw=read,
bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=spdk_fio, iodepth=128
fio-2.17-29-gf0ac1
Starting 1 process
Starting Intel(R) DPDK initialization ...
[ DPDK EAL parameters: fio -c 1 --file-prefix=spdk_pid6448
--base-virtaddr=0x1000000000 --proc-type=auto ]
EAL: Detected 8 lcore(s)
EAL: Auto-detected process type: PRIMARY
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: cannot open /proc/self/numa_maps, consider that all memory is in
socket_id 0
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
EAL: using IOMMU type 1 (Type 1)
EAL: vfio_group_fd=11 iommu_group_no=3 *vfio_dev_fd=13
EAL: reg=0x2000 fd=13 cap_offset=0x50
EAL: the msi-x bar number is 0 0x2000 0x200
EAL: Hotplug doesn't support vfio yet
spdk_fio_setup() is being called
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
EAL: vfio_group_fd=11 iommu_group_no=3 *vfio_dev_fd=16
EAL: reg=0x2000 fd=16 cap_offset=0x50
EAL: the msi-x bar number is 0 0x2000 0x200
EAL: inside pci_vfio_write_config offset=4
nvme_ctrlr.c:1224:nvme_ctrlr_process_init: ***ERROR*** Initialization
timed out in state 3
nvme_ctrlr.c: 403:nvme_ctrlr_shutdown: ***ERROR*** did not shutdown within
5 seconds
EAL: Hotplug doesn't support vfio yet
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
EAL: vfio_group_fd=11 iommu_group_no=3 *vfio_dev_fd=18
EAL: reg=0x2000 fd=18 cap_offset=0x50
EAL: the msi-x bar number is 0 0x2000 0x200
EAL: Hotplug doesn't support vfio yet
EAL: Requested device 0000:01:00.0 cannot be used
spdk_nvme_probe() failed
Regards,
Oza.
3 years, 5 months
VM oops while testing SPDK hotplug.sh
by Isaac Otsiabah
Daniel. I have done more testing using hotplug.sh and here is why one does not see the problem by running hotplug.sh as is. When hotplug.sh is run, it always creates a new VM to run the test on and, destroys the VM when completed. Therefore, it uses a fresh VM all the time and never gets the chance to do the inserts and remove on a VM that has run the same test several times (ie. 4 to 5 times) before. When I run hotplug.sh with a fresh VM, it passes, On the other hand, when I use the same VM that has run the test several times before, the VM oops, this is the problem. I also think this is the likely scenario the customer will experience.
I wasn't clear in my earlier emails because i was still determining the cause of the problem (at the higher level at least). I think you will see the problem if you do this
i. In an xterm window, Create the VM separately (without the -daemon flag). for example
IMAGE=/home/fedora24/fedora24-2.img
qemu-img create -f qcow2 $IMAGE 50G
MEM=8192M
FEDORA_ISO=/tmp/Fedora-Server-dvd-x86_64-24-1.2.iso
qemu_pidfile=/tmp/qemu_pidfile
qemu-2.7.1/x86_64-softmmu/qemu-system-x86_64 \
-hda $IMAGE \
-net user,hostfwd=tcp::10022-:22 \
-net nic \
-cpu host \
-m ${MEM} \
-pidfile "/tmp/qemu_pidfile" \
--enable-kvm \
-chardev socket,id=mon0,host=localhost,port=4444,ipv4,server,nowait \
-mon chardev=mon0,mode=readline \
-cdrom $FEDORA_ISO
ii. Then comment out in hotplug.sh the portion that creates the VM.
iii. Also comment out these 4 lines at the bottom in hotplug.sh to avoid killing the VM.
qemupid=`cat "$qemu_pidfile" | awk '{printf $0}'`
kill -9 $qemupid
rm "$qemu_pidfile"
rm "$test_img"
iv. Run hotplug.sh (about 5 times and you will see the oops on VM console)
The host system I am using is a Centos 7.2
Isaac
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Verkamp, Daniel
Sent: Wednesday, May 17, 2017 1:09 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] VM crashes while testing SPDK hotplug
Hi Isaac,
The version of the hotplug script in the repository (test/lib/nvme/hotplug.sh) is the current version we are running in our automated test pool.
We haven't hit the -net/--netdev issue that you mentioned yet because the version of qemu we are using is older (the current host system running this test is Fedora 25 with qemu 2.7.1). It looks like we'll need to update the script for that. We would be happy to accept a patch to hotplug.sh if the --netdev option also works on older qemu.
If the kernel crashes due to user program behavior, it sounds like there is a bug in the kernel uio driver. We haven't seen this crash in our automated testing, so I am not sure what the cause could be. It is also worth trying a newer kernel version (we are just using Linux 4.5.5 because the test VM image hasn't been updated in a while).
-- Daniel
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Isaac Otsiabah
Sent: Monday, May 15, 2017 1:43 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk@lists.01.org>>
Subject: Re: [SPDK] VM crashes while testing SPDK hotplug
Daniel, please, do you have an updated version of the hotplug.sh script you can share with us? I created the VM using this exact command on my Centos 7 host
IMAGE=/home/fedora24/fedora24.img
qemu-img create -f qcow2 $IMAGE 50G
MEM=8192M
FEDORA_ISO=/tmp/Fedora-Server-dvd-x86_64-24-1.2.iso
/tmp/qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \
-hda $IMAGE \
-net nic,model=virtio \
-net bridge,br=br1 \
-netdev user,id=hotplug,hostfwd=tcp::10022-:22 \
-m ${MEM} \
-pidfile "/tmp/qemu_pid_fedora.txt" \
--enable-kvm \
-cpu host \
-chardev socket,id=mon0,host=localhost,port=4444,ipv4,server,nowait \
-mon chardev=mon0,mode=readline \
-cdrom $FEDORA_ISO
After the install is complete, I setup the guest IP address in /etc/sysconfig/network-scripts/ifcfg-ens3 and brings up the interface with ./ifup ens3
>From the VM, I clone spdk and build.
Then I run the group of test in hotplug.sh skipping the VM creation and copying spdk to the VM sections.
I mentioned the -netdev flag in my earlier email
Isaac
From: Isaac Otsiabah
Sent: Monday, May 15, 2017 1:08 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk@lists.01.org>>
Cc: Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com<mailto:IOtsiabah@us.fujitsu.com>>; Paul Von-Stamwitz <PVonStamwitz(a)us.fujitsu.com<mailto:PVonStamwitz@us.fujitsu.com>>; Edward Yang <eyang(a)us.fujitsu.com<mailto:eyang@us.fujitsu.com>>
Subject: RE: VM crashes while testing SPDK hotplug
Daniel, i installed a Fedora 24 VM and test it. After running the test twice or more, the VM oops. Unlike the previous failure on Centos, this failure does not reboot but VM oops after two or more test run. My host is a Centos machine. I found the qemu-kvm which comes with the OS installation does not support nvme so I build qemu-system-x86_64 version 2.9.
[root@host1 spdk]# /tmp/qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 -version
QEMU emulator version 2.9.0
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
One observation on (although this is not the problem because I executed the scripts/setup.sh and the hotplug binary from vm console during appropriate breakpoints because local port 10022 was not responsive), the hotplug.h has the flag "-net user,hostfwd=tcp::10022-:22 \" to redirect gust ssh port 22 to host port 10022. However, qemu-system-x86_64 version 2.9 does not have this option but it has -netdev option but is is different. The qemu-system-86_64 man page on -netdev flag is as follows:
-netdev user,id=str[,ipv4[=on|off]][,net=addr[/mask]][,host=addr]
[,ipv6[=on|off]][,ipv6-net=addr[/int]][,ipv6-host=addr]
[,restrict=on|off][,hostname=host][,dhcpstart=addr]
[,dns=addr][,ipv6-dns=addr][,dnssearch=domain][,tftp=dir]
[,bootfile=f][,hostfwd=rule][,guestfwd=rule][,smb=dir[,smbserver=addr]]
configure a user mode network backend with ID 'str',
its DHCP server and optional services
It says hostfwd=rule and does not give detail of the rule. I used tcp so I specified it as
-netdev user,id=hotplug,hostfwd=tcp::10022-:22 \
>From the host "netstat -an |egrep -I listen |less" I see local port 10022 is being listened on. I installed sshpass and tested this -netdev flag redirection with a simple sshpass command to the vm but got no response. Therefore, i bypassed executing scripts/setup.sh and the hotplug binary using sshpass command.
So I can test it without executing setup.sh and the hotplug binary through sshpass on port 10022. The main issue is why does it oops after I run it 2 or more times?
Isaac
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Verkamp, Daniel
Sent: Tuesday, May 09, 2017 3:33 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk@lists.01.org>>
Subject: Re: [SPDK] VM crashes while testing SPDK hotplug
Hi Isaac,
Our hotplug tests with a VM (test/lib/nvme/hotplug.sh) are working with a Fedora 24 VM guest running kernel 4.5.5. I suspect there is a bug in the CentOS kernel version (3.10 is fairly old and is probably missing uio/hotplug-related bug fixes from the mainline kernels).
Can you try to reproduce your problem on a newer kernel version and see if that is the cause of the issue?
Thanks,
-- Daniel
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Isaac Otsiabah
Sent: Tuesday, May 9, 2017 2:11 PM
To: spdk(a)lists.01.org<mailto:spdk@lists.01.org>
Subject: [SPDK] VM crashes while testing SPDK hotplug
I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug.
1. VM creation is as follows
IMAGE=/home/centos7/centos72.img
qemu-img create -f qcow2 $IMAGE 50G
MEM=8192M
ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso
[root@host1]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[root@host1]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso
-r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso
qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \
-hda $IMAGE \
-net nic,model=virtio \
-net bridge,br=br1 \
-m ${MEM} \
-pidfile "/tmp/qemu_pid2.txt" \
--enable-kvm \
-cpu host \
-chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \
-mon chardev=mon0,mode=readline \
-cdrom $ISO
2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/
echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449
echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449
echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449
echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449
echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449
echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449
echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449
echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449
Also, commands to delete the devices work without crashing the VM
echo "device_del nvme0" | nc localhost 4449
echo "device_del nvme1" | nc localhost 4449
echo "device_del nvme2" | nc localhost 4449
echo "device_del nvme3" | nc localhost 4449
3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. /var/log/message and I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug.
1. VM creation is as follows
IMAGE=/home/centos7/centos72.img
qemu-img create -f qcow2 $IMAGE 50G
MEM=8192M
ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso
[root@host1]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[root@host1]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso
-r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso
qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \
-hda $IMAGE \
-net nic,model=virtio \
-net bridge,br=br1 \
-m ${MEM} \
-pidfile "/tmp/qemu_pid2.txt" \
--enable-kvm \
-cpu host \
-chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \
-mon chardev=mon0,mode=readline \
-cdrom $ISO
2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/
echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449
echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449
echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449
echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449
echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449
echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449
echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449
echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449
Also, commands to delete the devices work without crashing the VM
echo "device_del nvme0" | nc localhost 4449
echo "device_del nvme1" | nc localhost 4449
echo "device_del nvme2" | nc localhost 4449
echo "device_del nvme3" | nc localhost 4449
3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. The /var/log/message and vmcore-dmesg.txt files are in the attached tar file. Would appreciate any help in why a bug in SPDK crashes the VM. Thanks.
Isaac
3 years, 7 months
Need help for iSCSI configuration for NVMe disc
by Kiran Dikshit
Hi All,
I am trying to configure iSCSI for a NVMe drive and getting the below error’s. I have unbound the NVMe drive from kernel using the setup.sh.
tgt_node.c: 718:spdk_iscsi_tgt_node_construct: ***ERROR*** Could not construct SCSI device
tgt_node.c: 984:spdk_cf_add_iscsi_tgt_node: ***ERROR*** tgt_node1: add_iscsi_target_node error
tgt_node.c:1010:spdk_iscsi_init_tgt_nodes: ***ERROR*** spdk_cf_add_iscsi_tgt_node() failed
iscsi_subsystem.c: 965:spdk_iscsi_subsystem_init: ***ERROR*** spdk_iscsi_init_tgt_nodes() failed
app.c: 404:spdk_app_init: ***ERROR*** spdk_subsystem_init() failed
Below is the section of my iscsi.conf.in <http://iscsi.conf.in/> file for reference.
# NVMe configuration options
[Nvme]
# NVMe Device Whitelist
# Users may specify which NVMe devices to claim by their PCI
# domain, bus, device, and function. The format is dddd:bb:dd.f, which is
# the same format displayed by lspci or in /sys/bus/pci/devices. The second
# argument is a "name" for the device that can be anything. The name
# is referenced later in the Subsystem section.
#
# Alternatively, the user can specify ClaimAllDevices. All
# NVMe devices will be claimed and named Nvme0, Nvme1, etc.
# BDF 0000:00:00.0 Nvme0
BDF 0000:0a:00.0 Nvme3
[TargetNode1]
TargetName disk2
TargetAlias "Data Disk2"
Mapping PortalGroup1 InitiatorGroup1
AuthMethod Auto
AuthGroup AuthGroup1
UseDigest Auto
LUN0 Nvme3
QueueDepth 64
Thank you
Kiran
3 years, 7 months
VM oops while testing SPDK hotplug.sh
by Isaac Otsiabah
Hi Daniel, how are you? Daniel i performed two types of hotplug test using Fedora 25 (kernel 4.8.6-300, VM and host running the same kernel version).
1. Test 1 (Running hotplug.sh in once in a new VM): I created a Fedora25 image for the VM and used it to test hotplug.sh with no modification and it worked fine. With no modification, hotplug.sh performs the test on a newly installed VM every time and destroys the VM at the end. While this is ok, i think we should also run the test in an environment where the VM continue to run while NVMe hotplug operation is performed several times. This is a practical scenario at the customer environment so I tried to test it in Test 2 below.
2. Test 2 (Running hotplug.sh several times while VM continue to run): I created this small script to start the VM (Fedora25) on a Fedora 25 host, kernel-4.8.6. I commented out the portion that creates and destroys the VM in hotplug.sh so that the VM will keep running after a hotplug.sh run.
[root@localhost ~]# cat c_kvm_with_qemu-2-7.1_helper_33_fedora.sh
#!/bin/bash
base_img=/home/fedora25/fedora25-1.img
IMAGE=/home/fedora25/test_fedora25-1.img
rm -f $IMAGE
MEM=8192M
qemu-img create -b $base_img -f qcow2 $IMAGE 50G
/usr/bin/qemu-system-x86_64 \
-hda $IMAGE \
-net user,hostfwd=tcp::10022-:22 \
-net nic \
-cpu host \
-m ${MEM} \
-pidfile "/tmp/qemu-2-7-1_pid_fedora.txt" \
--enable-kvm \
-chardev socket,id=mon0,host=localhost,port=4444,ipv4,server,nowait \
-mon chardev=mon0,mode=readline
Then I ran the modified hotpluig.sh (as mentioned above) and this is what I found.
i. The VM oops as it did before in my previous report.
ii. In insert_devices(), if i comment out the "ssh_vm scripts/setup.sh" to prevent setup.sh from running, it did not oops, however, inserts of new devices were not found. It appears, the problem may not be the kernel version but running setup.sh more than once is the issue.
Daniel, each time setup.sh is ran, it writes 1024 to memory on the proc filesystem (/proc/sys/vm/nr_hugepages) to be used for the allocation of huge pages, could this be the issue since a prior run had already allocated huge pages?
Isaac
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Isaac Otsiabah
Sent: Friday, May 19, 2017 11:33 AM
To: spdk(a)lists.01.org<mailto:spdk@lists.01.org>
Subject: [SPDK] VM oops while testing SPDK hotplug.sh
Daniel. I have done more testing using hotplug.sh and here is why one does not see the problem by running hotplug.sh as is. When hotplug.sh is run, it always creates a new VM to run the test on and, destroys the VM when completed. Therefore, it uses a fresh VM all the time and never gets the chance to do the inserts and remove on a VM that has run the same test several times (ie. 4 to 5 times) before. When I run hotplug.sh with a fresh VM, it passes, On the other hand, when I use the same VM that has run the test several times before, the VM oops, this is the problem. I also think this is the likely scenario the customer will experience.
I wasn't clear in my earlier emails because i was still determining the cause of the problem (at the higher level at least). I think you will see the problem if you do this
i. In an xterm window, Create the VM separately (without the -daemon flag). for example
IMAGE=/home/fedora24/fedora24-2.img
qemu-img create -f qcow2 $IMAGE 50G
MEM=8192M
FEDORA_ISO=/tmp/Fedora-Server-dvd-x86_64-24-1.2.iso
qemu_pidfile=/tmp/qemu_pidfile
qemu-2.7.1/x86_64-softmmu/qemu-system-x86_64 \
-hda $IMAGE \
-net user,hostfwd=tcp::10022-:22 \
-net nic \
-cpu host \
-m ${MEM} \
-pidfile "/tmp/qemu_pidfile" \
--enable-kvm \
-chardev socket,id=mon0,host=localhost,port=4444,ipv4,server,nowait \
-mon chardev=mon0,mode=readline \
-cdrom $FEDORA_ISO
ii. Then comment out in hotplug.sh the portion that creates the VM.
iii. Also comment out these 4 lines at the bottom in hotplug.sh to avoid killing the VM.
qemupid=`cat "$qemu_pidfile" | awk '{printf $0}'`
kill -9 $qemupid
rm "$qemu_pidfile"
rm "$test_img"
iv. Run hotplug.sh (about 5 times and you will see the oops on VM console)
The host system I am using is a Centos 7.2
Isaac
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Verkamp, Daniel
Sent: Wednesday, May 17, 2017 1:09 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk@lists.01.org>>
Subject: Re: [SPDK] VM crashes while testing SPDK hotplug
Hi Isaac,
The version of the hotplug script in the repository (test/lib/nvme/hotplug.sh) is the current version we are running in our automated test pool.
We haven't hit the -net/--netdev issue that you mentioned yet because the version of qemu we are using is older (the current host system running this test is Fedora 25 with qemu 2.7.1). It looks like we'll need to update the script for that. We would be happy to accept a patch to hotplug.sh if the --netdev option also works on older qemu.
If the kernel crashes due to user program behavior, it sounds like there is a bug in the kernel uio driver. We haven't seen this crash in our automated testing, so I am not sure what the cause could be. It is also worth trying a newer kernel version (we are just using Linux 4.5.5 because the test VM image hasn't been updated in a while).
-- Daniel
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Isaac Otsiabah
Sent: Monday, May 15, 2017 1:43 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk@lists.01.org>>
Subject: Re: [SPDK] VM crashes while testing SPDK hotplug
Daniel, please, do you have an updated version of the hotplug.sh script you can share with us? I created the VM using this exact command on my Centos 7 host
IMAGE=/home/fedora24/fedora24.img
qemu-img create -f qcow2 $IMAGE 50G
MEM=8192M
FEDORA_ISO=/tmp/Fedora-Server-dvd-x86_64-24-1.2.iso
/tmp/qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \
-hda $IMAGE \
-net nic,model=virtio \
-net bridge,br=br1 \
-netdev user,id=hotplug,hostfwd=tcp::10022-:22 \
-m ${MEM} \
-pidfile "/tmp/qemu_pid_fedora.txt" \
--enable-kvm \
-cpu host \
-chardev socket,id=mon0,host=localhost,port=4444,ipv4,server,nowait \
-mon chardev=mon0,mode=readline \
-cdrom $FEDORA_ISO
After the install is complete, I setup the guest IP address in /etc/sysconfig/network-scripts/ifcfg-ens3 and brings up the interface with ./ifup ens3
>From the VM, I clone spdk and build.
Then I run the group of test in hotplug.sh skipping the VM creation and copying spdk to the VM sections.
I mentioned the -netdev flag in my earlier email
Isaac
From: Isaac Otsiabah
Sent: Monday, May 15, 2017 1:08 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk@lists.01.org>>
Cc: Isaac Otsiabah <IOtsiabah(a)us.fujitsu.com<mailto:IOtsiabah@us.fujitsu.com>>; Paul Von-Stamwitz <PVonStamwitz(a)us.fujitsu.com<mailto:PVonStamwitz@us.fujitsu.com>>; Edward Yang <eyang(a)us.fujitsu.com<mailto:eyang@us.fujitsu.com>>
Subject: RE: VM crashes while testing SPDK hotplug
Daniel, i installed a Fedora 24 VM and test it. After running the test twice or more, the VM oops. Unlike the previous failure on Centos, this failure does not reboot but VM oops after two or more test run. My host is a Centos machine. I found the qemu-kvm which comes with the OS installation does not support nvme so I build qemu-system-x86_64 version 2.9.
[root@host1 spdk]# /tmp/qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 -version
QEMU emulator version 2.9.0
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
One observation on (although this is not the problem because I executed the scripts/setup.sh and the hotplug binary from vm console during appropriate breakpoints because local port 10022 was not responsive), the hotplug.h has the flag "-net user,hostfwd=tcp::10022-:22 \" to redirect gust ssh port 22 to host port 10022. However, qemu-system-x86_64 version 2.9 does not have this option but it has -netdev option but is is different. The qemu-system-86_64 man page on -netdev flag is as follows:
-netdev user,id=str[,ipv4[=on|off]][,net=addr[/mask]][,host=addr]
[,ipv6[=on|off]][,ipv6-net=addr[/int]][,ipv6-host=addr]
[,restrict=on|off][,hostname=host][,dhcpstart=addr]
[,dns=addr][,ipv6-dns=addr][,dnssearch=domain][,tftp=dir]
[,bootfile=f][,hostfwd=rule][,guestfwd=rule][,smb=dir[,smbserver=addr]]
configure a user mode network backend with ID 'str',
its DHCP server and optional services
It says hostfwd=rule and does not give detail of the rule. I used tcp so I specified it as
-netdev user,id=hotplug,hostfwd=tcp::10022-:22 \
>From the host "netstat -an |egrep -I listen |less" I see local port 10022 is being listened on. I installed sshpass and tested this -netdev flag redirection with a simple sshpass command to the vm but got no response. Therefore, i bypassed executing scripts/setup.sh and the hotplug binary using sshpass command.
So I can test it without executing setup.sh and the hotplug binary through sshpass on port 10022. The main issue is why does it oops after I run it 2 or more times?
Isaac
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Verkamp, Daniel
Sent: Tuesday, May 09, 2017 3:33 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk@lists.01.org>>
Subject: Re: [SPDK] VM crashes while testing SPDK hotplug
Hi Isaac,
Our hotplug tests with a VM (test/lib/nvme/hotplug.sh) are working with a Fedora 24 VM guest running kernel 4.5.5. I suspect there is a bug in the CentOS kernel version (3.10 is fairly old and is probably missing uio/hotplug-related bug fixes from the mainline kernels).
Can you try to reproduce your problem on a newer kernel version and see if that is the cause of the issue?
Thanks,
-- Daniel
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Isaac Otsiabah
Sent: Tuesday, May 9, 2017 2:11 PM
To: spdk(a)lists.01.org<mailto:spdk@lists.01.org>
Subject: [SPDK] VM crashes while testing SPDK hotplug
I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug.
1. VM creation is as follows
IMAGE=/home/centos7/centos72.img
qemu-img create -f qcow2 $IMAGE 50G
MEM=8192M
ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso
[root@host1]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[root@host1]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso
-r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso
qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \
-hda $IMAGE \
-net nic,model=virtio \
-net bridge,br=br1 \
-m ${MEM} \
-pidfile "/tmp/qemu_pid2.txt" \
--enable-kvm \
-cpu host \
-chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \
-mon chardev=mon0,mode=readline \
-cdrom $ISO
2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/
echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449
echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449
echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449
echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449
echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449
echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449
echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449
echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449
Also, commands to delete the devices work without crashing the VM
echo "device_del nvme0" | nc localhost 4449
echo "device_del nvme1" | nc localhost 4449
echo "device_del nvme2" | nc localhost 4449
echo "device_del nvme3" | nc localhost 4449
3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. /var/log/message and I created a VM on a Centos 7 with a listening socket on port 4449 and tested the hotplug.
1. VM creation is as follows
IMAGE=/home/centos7/centos72.img
qemu-img create -f qcow2 $IMAGE 50G
MEM=8192M
ISO=/tmp/CentOS-7-x86_64-Everything-1611.iso
[root@host1]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[root@host1]# ls -l /tmp/CentOS-7-x86_64-Everything-1611.iso
-r--------. 1 qemu qemu 8280604672 Apr 12 13:37 /tmp/CentOS-7-x86_64-Everything-1611.iso
qemu-2.9.0/x86_64-softmmu/qemu-system-x86_64 \
-hda $IMAGE \
-net nic,model=virtio \
-net bridge,br=br1 \
-m ${MEM} \
-pidfile "/tmp/qemu_pid2.txt" \
--enable-kvm \
-cpu host \
-chardev socket,id=mon0,host=localhost,port=4449,ipv4,server,nowait \
-mon chardev=mon0,mode=readline \
-cdrom $ISO
2. Without running the SPDK ( ie. examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8 ), the qemu commands to insert fake nvme devices work, i can see the nvme devices in /dev/
echo " drive_add 0 file=/root/test0,format=raw,id=drive0,if=none" | nc localhost 4449
echo " drive_add 1 file=/root/test1,format=raw,id=drive1,if=none" | nc localhost 4449
echo "drive_add 2 file=/root/test2,format=raw,id=drive2,if=none" | nc localhost 4449
echo "drive_add 3 file=/root/test3,format=raw,id=drive3,if=none" | nc localhost 4449
echo "device_add nvme,drive=drive0,id=nvme0,serial=nvme0" |nc localhost 4449
echo "device_add nvme,drive=drive1,id=nvme1,serial=nvme1" |nc localhost 4449
echo "device_add nvme,drive=drive2,id=nvme2,serial=nvme2" |nc localhost 4449
echo "device_add nvme,drive=drive3,id=nvme3,serial=nvme3" |nc localhost 4449
Also, commands to delete the devices work without crashing the VM
echo "device_del nvme0" | nc localhost 4449
echo "device_del nvme1" | nc localhost 4449
echo "device_del nvme2" | nc localhost 4449
echo "device_del nvme3" | nc localhost 4449
3. However, with the SPDK hotplug test application (examples/nvme/hotplug/hotplug -i 0 -t 15 -n 4 -r 8), the device_del command causes a fault and crashes the VM and it reboot as a result. The /var/log/message and vmcore-dmesg.txt files are in the attached tar file. Would appreciate any help in why a bug in SPDK crashes the VM. Thanks.
Isaac
3 years, 7 months
An Open, Transparent Patch Submission Process
by Walker, Benjamin
Hi all,
We're excited to announce a new patch submission process for SPDK utilizing
GerritHub, a hosted code review service integrated with GitHub. All patches,
even from SPDK maintainers, will now use the new process, which means all code
reviews and discussions are now fully open and transparent to everyone. We
encourage everyone, even if you aren't an expert, to review patches and vote
with a +1 or -1! A quick query for all open patches is here (better queries and
dashboard set up are in the guide at the bottom):
https://review.gerrithub.io/#/q/project:spdk/spdk+status:open
Further, the SPDK continuous integration tests are hooked up to GerritHub, which
means that when you submit a patch a whole battery of tests on a wide range of
systems will be run with results reported directly to the code review. You can
view the current status of the continuous integration system at:
http://ci.spdk.io
A comprehensive guide to the new patch submission process can be found here:
http://www.spdk.io/development
A quick start guide is still in the works.
We're thrilled to open this system up to everyone, giving all community members
access to the same tools the SPDK maintainers have.
Thanks,
The SPDK maintainers
3 years, 8 months
VM not booting up with SPDK vhost
by Abhik Sarkar
Hi,
This is actually a follow-up email as I was suggested to post the query on the mailing list.
So, we are running into an issue while setting up SPDK vhost.
Our guest OS does not seem to boot-up after it looks like Qemu has setup communication with the vhost app.
We first faced the problem with guest running Scientific Linux 6.1, kernel version 2.6.32-696.1.1.el6.
On James Harris’ recommendation, we tried running guest with CentOS 7, kernel version 3.10.0-514.el7.x86_64. But we see the same issue.
I have attached 2 different config files. One with the vhost setting and one without.
centos7.1.xml : This does not have any vhost related command line arguments while launching the VM
Status: The VM launches just fine and we are able to login to the VM
vcentos7.1.xml: This one has the vhost setting appended
Status: The vhost-target output seems to show that the vhost-client (qemu) has been setup properly (vhost_logs.txt) but we can’t login to the VM.
Follwing is the arp output
# arp -an
centos7.1 -->? (192.168.122.24) at 52:54:00:47:0c:3d [ether] on virbr0
vcentos7.1 -->? (192.168.122.204) at <incomplete> on virbr0
Also the console does not show any boot messages or login prompt.
# virsh ttyconsole vcentos7.1
/dev/pts/0
#virsh console vcentos7.1
Connected to domain vcentos7.1
Escape character is ^]
The qemu build used is the spdk branch with the latest commit ID being 6eed2b580b6d5bc853b8efc15931eaf0c0de68a2
Thanks/Regards
-a
3 years, 8 months
"Open file failed: No such file or directory" error in examples/nvme/nvme_manage/nvme_manage tool
by Isaac Otsiabah
I experimented with examples/nvme/nvme_manage/nvme_manage tool. After the prompt to enter the firmware path name, I entered the firmware full path name and got "Open file failed: No such file or directory" error.
After a little debug I found in update_firmware_image(void) function line 775, fgets(..) reads the new line character and puts it at the end of the input buffer. This new line character must be removed from the input buffer before the buffer contents is used.
775 if (fgets(path, 256, stdin) == NULL) {
776 printf("Invalid path setting\n");
777 while (getchar() != '\n');
778 return;
779 }
The same problem is also in the get_controller() function after fgets is used to read the PCI Address from standard input.
300 while ((ch = getchar()) != '\n' && ch != EOF);
301 p = fgets(address, 64, stdin);
302 if (p == NULL) {
303 return NULL;
304 }
Both can be fixed by removing the new line character from the input buffer after fgets return
Ie.
if (path[strlen(path)-1]=='\n')
path[strlen(path)-1]='\0';
Isaac
[root@spdk2 spdk]# ./examples/nvme/nvme_manage/nvme_manage
Starting DPDK 17.02.0 initialization...
[ DPDK EAL parameters: nvme_manage -c 0x1 --file-prefix=spdk_pid19289 ]
EAL: Detected 16 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:02:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
EAL: PCI device 0000:03:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
EAL: PCI device 0000:04:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
EAL: PCI device 0000:05:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
IDENTIFY (06) sqid:0 cid:59 nsid:ffffffff cdw10:00000000 cdw11:00000000
INVALID NAMESPACE OR FORMAT (00/0b) sqid:0 cid:59 cdw0:0 sqhd:000a p:1 m:0 dnr:0
IDENTIFY (06) sqid:0 cid:59 nsid:ffffffff cdw10:00000000 cdw11:00000000
INVALID NAMESPACE OR FORMAT (00/0b) sqid:0 cid:59 cdw0:0 sqhd:000a p:1 m:0 dnr:0
IDENTIFY (06) sqid:0 cid:59 nsid:ffffffff cdw10:00000000 cdw11:00000000
INVALID NAMESPACE OR FORMAT (00/0b) sqid:0 cid:59 cdw0:0 sqhd:000a p:1 m:0 dnr:0
IDENTIFY (06) sqid:0 cid:59 nsid:ffffffff cdw10:00000000 cdw11:00000000
INVALID NAMESPACE OR FORMAT (00/0b) sqid:0 cid:59 cdw0:0 sqhd:000a p:1 m:0 dnr:0
NVMe Management Options
[1: list controllers]
[2: create namespace]
[3: delete namespace]
[4: attach namespace to controller]
[5: detach namespace from controller]
[6: format namespace or controller]
[7: firmware update]
[8: quit]
7
0000:02:00.00 INTEL SSDPEDMD800G4 CVFT4216000E800CGN 0
0000:03:00.00 INTEL SSDPEDMD800G4 CVFT4216000G800CGN 0
0000:04:00.00 INTEL SSDPEDMD800G4 CVFT421500DG800CGN 0
0000:05:00.00 INTEL SSDPEDMD800G4 CVFT421500DE800CGN 0
Please Input PCI Address(domain:bus:dev.func):
0000:05:00.00
Please Input The Path Of Firmware Image
/root/spdk/image/8DV101B0_8B1B0133_signed.bin
Open file failed: No such file or directory
press Enter to display cmd menu ...
3 years, 8 months
Could not find NVMe controller at PCI address 0000:06:00.0
by Hugh Daschbach
I'm having issues getting app/nvmf_tgt/nvmf_tgt running. I've
configured four of five local NVMe devices in nvmf.conf. The devices
are discovered during early initialization, but spdk_nvme_probe()
fails to find a previously probed device. It looks like
nvme_pci_ctrlr_scan() expects a hotplug event that it never sees.
What is supposed to trigger the hotplug event?
Here's the output of nvmf_tgt:
Starting DPDK 17.08.0-rc0 initialization...
[ DPDK EAL parameters: nvmf -c 0x5555 --file-prefix=spdk_pid184703 ]
EAL: Detected 72 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Debug dataplane logs available - lower performance
EAL: Probing VFIO support...
EAL: VFIO support initialized
Occupied cpu socket mask is 0x1
Ioat Copy Engine Offload Enabled
EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL: using IOMMU type 1 (Type 1)
EAL: PCI device 0000:07:00.0 on NUMA socket 0
EAL: PCI device 0000:08:00.0 on NUMA socket 0
EAL: PCI device 0000:09:00.0 on NUMA socket 0
EAL: PCI device 0000:84:00.0 on NUMA socket 1
EAL: Releasing pci mapped resource for 0000:84:00.0
EAL: Calling pci_unmap_resource for 0000:84:00.0 at 0x7f194ae10000
Reactor started on core 2 on socket 0
Reactor started on core 4 on socket 0
Reactor started on core 6 on socket 0
Reactor started on core 8 on socket 0
Reactor started on core 10 on socket 0
Reactor started on core 12 on socket 0
Reactor started on core 14 on socket 0
Reactor started on core 0 on socket 0
*** RDMA Transport Init ***
allocated subsystem nqn.2014-08.org.nvmexpress.discovery on lcore 0 on
socket 0
allocated subsystem nqn.2016-06.io.spdk:cnode0 on lcore 2 on socket 0
Total cores available: 8
EAL: PCI device 0000:06:00.0 on NUMA socket 0
conf.c: 578:spdk_nvmf_construct_subsystem: ***ERROR*** Could not find NVMe
controller at PCI address 0000:06:00.0
nvmf_tgt.c: 279:spdk_nvmf_startup: ***ERROR*** spdk_nvmf_parse_conf() failed
EAL: Releasing pci mapped resource for 0000:06:00.0
EAL: Calling pci_unmap_resource for 0000:06:00.0 at 0x7f194ae00000
EAL: Releasing pci mapped resource for 0000:07:00.0
EAL: Calling pci_unmap_resource for 0000:07:00.0 at 0x7f194ae04000
EAL: Releasing pci mapped resource for 0000:08:00.0
EAL: Calling pci_unmap_resource for 0000:08:00.0 at 0x7f194ae08000
EAL: Releasing pci mapped resource for 0000:09:00.0
EAL: Calling pci_unmap_resource for 0000:09:00.0 at 0x7f194ae0c000
Select bits from nvmf.conf:
[Nvme]
TransportID "trtype:PCIe traddr:0000:05:00.0" Nvme0
TransportID "trtype:PCIe traddr:0000:06:00.0" Nvme1
TransportID "trtype:PCIe traddr:0000:07:00.0" Nvme2
TransportID "trtype:PCIe traddr:0000:08:00.0" Nvme3
TransportID "trtype:PCIe traddr:0000:09:00.0" Nvme4
[Subsystem0]
NQN nqn.2016-06.io.spdk:cnode0
Core 2
Mode Direct
Listen RDMA 192.168.10.1:4420
SN SPDK00000000000001
NVMe 0000:06:00.0
[Subsystem1]
NQN nqn.2016-06.io.spdk:cnode1
Core 4
Mode Direct
Listen RDMA 192.168.10.1:4420
SN SPDK00000000000002
NVMe 0000:07:00.0
[Subsystem2]
NQN nqn.2016-06.io.spdk:cnode2
Core 6
Mode Direct
Listen RDMA 192.168.10.1:4420
SN SPDK00000000000003
NVMe 0000:08:00.0
[Subsystem3]
NQN nqn.2016-06.io.spdk:cnode3
Core 8
Mode Direct
Listen RDMA 192.168.10.1:4420
SN SPDK00000000000004
NVMe 0000:09:00.0
and lspci output following the errror:
[root@localhost spdk]# lspci -v -s 0:6:0.0
06:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD
Controller 171X (rev 03) (prog-if 02 [NVM Express])
Subsystem: Dell Express Flash NVMe XS1715 SSD 800GB
Physical Slot: 180
Flags: fast devsel, IRQ 31, NUMA node 0
Memory at 9a400000 (64-bit, non-prefetchable) [disabled] [size=16K]
Capabilities: [c0] Power Management version 3
Capabilities: [c8] MSI: Enable- Count=1/32 Maskable+ 64bit+
Capabilities: [e0] MSI-X: Enable- Count=129 Masked-
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [40] Vendor Specific Information: Len=24 <?>
Capabilities: [100] Advanced Error Reporting
Capabilities: [180] #19
Capabilities: [150] Vendor Specific Information: ID=0001 Rev=1
Len=02c <?>
Kernel driver in use: vfio-pci
Kernel modules: nvme
The above is from branch master (both SPDK and DPDK). I've tested v17.03
with similar results. What am I missing?
Thanks,
Hugh
3 years, 8 months
New feature - Blobs as bdevs
by Pelplinski, Piotr
Hi everyone!
Today we'll be discussing and reviewing new early concept for spdk - using blobs as bdevs in vhost.
You're welcome to join our meeting at +1(916)356-2663. Choose bridge 5, Conference ID: 656560618.
Meeting starts today, 05/18/2017 at 08:00 AM MST (UTC-7).
--
Best Regards,
Piotr Pelpliński
--------------------------------------------------------------------
Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.
Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.
3 years, 8 months