vhost is destroyed many times when starting vhost device
by Li Feng
Hi,
In rte_vhost_compat.c, spdk_extern_vhost_pre_msg_handler will call
destroy_device when some conditions meet.
I tested and found when a vhost-blk setup, the vhost device will be
destroyed 9 times.
And the bdev(no matter which type) will be affected. I'm using the aio
bdev, and the io_setup/io_destroy will be called 9 times when a disk
is inserted to a VM.
My concern is:
Is there any solution to reduce the call times(at least: disks nums * 9)?
The call will slow down the VM boot time if VM has multiple disks.
There are some related logs:
Starting SPDK v20.01-pre git sha1 93eecf627 / DPDK 19.11.0 initialization...
[ DPDK EAL parameters: spdk_tgt --no-shconf -c 1 -m 1
--log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6
--iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations
--file-prefix=spdk_pid2004865 ]
EAL: No available hugepages reported in hugepages-1048576kB
app.c: 645:spdk_app_start: *NOTICE*: Total cores available: 1
reactor.c: 346:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
destroy device
rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
destroy device
rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
destroy device
rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
destroy device
rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
rte_vhost_compat.c: 174:spdk_extern_vhost_pre_msg_handler: *ERROR*:
destroy device
rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
rte_vhost_compat.c: 193:spdk_extern_vhost_pre_msg_handler: *ERROR*:
destroy device
rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
destroy device
rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
destroy device
rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
destroy device
rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
Thanks,
Feng Li
--
The SmartX email address is only for business purpose. Any sent message
that is not related to the business is not authorized or permitted by
SmartX.
本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
2 years, 3 months
Can I use spdk_pci_hook_device to implement a virtual NVMe device?
by oscar.huang＠microchip.com
I'd like to create a virtual NVMe device and use SPDK NVMe APIs to operate it. It looks like spdk_pci_hook_device() can add a virtual PCI device to SPDK PCI system:
void spdk_pci_hook_device(struct spdk_pci_driver *drv, struct spdk_pci_device *dev);
But the struct spdk_pci_device only contains the following callbacks:
int (*map_bar)(struct spdk_pci_device *dev, uint32_t bar,
void **mapped_addr, uint64_t *phys_addr, uint64_t *size);
int (*unmap_bar)(struct spdk_pci_device *dev, uint32_t bar,
void *addr);
int (*cfg_read)(struct spdk_pci_device *dev, void *value,
uint32_t len, uint32_t offset);
int (*cfg_write)(struct spdk_pci_device *dev, void *value,
uint32_t len, uint32_t offset);
void (*detach)(struct spdk_pci_device *dev);
It does not have mmio callbacks. When applications write a device memory, sometimes it needs to trigger some actions on the device. Without mmio callbacks, how to make a useful virtual device with spdk_pci_hook_device? Pls suggest. Much appreciated!
2 years, 3 months
spdk_nvme_ns_cmd_read returning -ENOMEM
by trevordkramer＠gmail.com
Hello,
I have a system with 8 1GB hugepages configured at boot time split between 2 sockets. I reserve all 8 for spdk with
struct spdk_env_opts opts{};
spdk_env_opts_init(&opts);
opts.mem_size = 1024 * 8;
When I call spdk_nvme_ns_cmd_read I get -ENOMEM if my buffer (which I allocated with spdk_zmalloc) is too large. The call to spdk_zmalloc succeeds so the buffer is created. I don't understand what the buffer size would have to do with spdk_nvme_ns_cmd_read failing. Can anyone shed any light? I've looked though the code a bit and it looks like nvme_allocate_request must be failing but I don't understand why.
Thanks,
Trevor
2 years, 3 months
SPDK v19.10/DPDK v19.05.0 eal_adjust_config issue, Illegal instruction causing application to abort
by asprasannavenkatesh＠gmail.com
Hi guys,
I am running SPDK version 19.10 and DPDK version 19.05.0 in my initiator application and I am running into an issue that seems to be sporadic in nature. I am running into this issue, where the application aborts saying Illegal instruction, with Signal 4, SIGILL (code: illegal operand). Issue seems to be more of a library related/EAL config related issue.
I am using the following DPDK EAL parameters in the application. I am also attaching the topmost information from backtrace, as the other things are company confidential.
Starting SPDK v19.10 / DPDK 19.05.0 initialization...
[ DPDK EAL parameters: nvmf --no-shconf -c 0x1 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid40756 ]
*** Aborted at 1587486078 (Unix time, try 'date -d @1587486078') ***
*** Signal 4 (SIGILL) (0x7f2e40ec986d) received by PID 40756 (pthread TID 0x7f2e17fff700) (linux TID 40767) (maybe from PID 1089247341, UID 32558) (code: illegal operand), stack trace: ***
@ 00007f2e3fb2938f (unknown)
@ 00007f2e40ec986d eal_adjust_config -> <wokspace>/spdk/spdk-src/lib/json/json_write.c
@ 00007f2e40eb03a5 rte_eal_init -> <wokspace>/spdk/spdk-src/lib/json/json_write.c
@ 00007f2e40e906e6 spdk_env_init <wokspace>/spdk/spdk-src/lib/env_dpdk/init.c:456
I am trying to debug this issue. But adding -g and -ggdb compilation flags to the cmake list file makes the issue to vanish. Have any of you seen this kind of crash and how should we go about fixing this? Is adding -g and -gdb a workaround or is that the required fix. Is this a DPDK issue or just a gcc issue?
Also here is the system information that I am using
Initiator_system:~$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 56
On-line CPU(s) list: 0-55
Thread(s) per core: 2
Core(s) per socket: 14
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Model name: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
Stepping: 1
CPU MHz: 1236.281
CPU max MHz: 3300.0000
CPU min MHz: 1200.0000
BogoMIPS: 4801.15
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 35840K
NUMA node0 CPU(s): 0-13,28-41
NUMA node1 CPU(s): 14-27,42-55
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts
Kindly keep us posted if you guys have any inputs.
Thanks and Regards,
Prasanna
2 years, 3 months
Spdk nvme CLI - block device
by Gili Buzaglo
Hi
I tried to understand if nvme CLI based on spdk-1.6 can somehow generate a block device without using the nvme fabrics kernel module.
I managed to add tcp support and connect ,but then the code goes to the kernel module and no block device was added.
In general , I know that it is possible to create block devices which are implemented in user space (for example with nbd) .. and I know spdk has block device layer..
Does someone think that what I did is somehow possible ?
The motivation for this is easy host installation without the requirement to replace kernel or application changes.
I'm aware there will be a performance impact..
Thanks
-gili
2 years, 3 months
Re: Set SPDK log level and log flags
by Wenhua Liu
Thanks Gang for answering my question.
I had tried setting log level and log print level to "DEBUG" but I did not see effect. That's why I asked if the debug build is needed.
Do you know the syntax setting log flags?
Thanks,
-Wenhua
On 4/21/20, 11:20 PM, "Cao, Gang" <gang.cao(a)intel.com> wrote:
I think it is needed to have the Debug build first and then change the log level for Notice and Debug.
-----Original Message-----
From: Wenhua Liu <liuw(a)vmware.com>
Sent: Wednesday, April 22, 2020 14:09
To: spdk(a)lists.01.org
Subject: [SPDK] Set SPDK log level and log flags
Hi
Looking at SPDK log related command options in scripts/rpc.py, I see the following command options
log_set_flag (set_log_flag)
set log flag
log_clear_flag (clear_log_flag)
clear log flag
log_get_flags (get_log_flags)
get log flags
log_set_level (set_log_level)
set log level
log_get_level (get_log_level)
get log level
log_set_print_level (set_log_print_level)
set log print level
log_get_print_level (get_log_print_level)
get log print level
Running log_get_flags, I see the following log flags:
~/spdk/spdk$ sudo scripts/rpc.py log_get_flags
{
"aio": false,
"app_config": false,
"bdev": false,
"bdev_ftl": false,
"bdev_malloc": false,
"bdev_null": false,
"bdev_nvme": false,
"bdev_ocssd": false,
"bdev_raid": false,
"bdev_raid0": false,
"blob": false,
"blob_rw": false,
"blobfs": false,
"blobfs_rw": false,
"copy_ioat": false,
"ftl_core": false,
"ftl_init": false,
"gpt_parse": false,
"ioat": false,
"json_util": false,
"log": false,
"lvol": false,
"lvolrpc": false,
"nbd": false,
"net": false,
"notify_rpc": false,
"nvme": false,
"nvmf": false,
"nvmf_tcp": false,
"opal": false,
"raidrpc": false,
"reactor": false,
"rpc": false,
"rpc_client": false,
"thread": false,
"trace": false,
"vbdev_delay": false,
"vbdev_gpt": false,
"vbdev_lvol": false,
"vbdev_opal": false,
"vbdev_passthru": false,
"vbdev_split": false,
"vbdev_zone_block": false,
"virtio": false,
"virtio_blk": false,
"virtio_dev": false,
"virtio_pci": false,
"virtio_user": false,
"vmd": false
}
What are the relationship between log level, log print level and log flags. How do I use them? I tried to change log level and log print level from “NOTICE” to “DEBUG” but did not see any change. Also to use them, do I need to make any special SPDK build?
Thanks,
-Wenhua Liu
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
2 years, 3 months
Set SPDK log level and log flags
by Wenhua Liu
Hi
Looking at SPDK log related command options in scripts/rpc.py, I see the following command options
log_set_flag (set_log_flag)
set log flag
log_clear_flag (clear_log_flag)
clear log flag
log_get_flags (get_log_flags)
get log flags
log_set_level (set_log_level)
set log level
log_get_level (get_log_level)
get log level
log_set_print_level (set_log_print_level)
set log print level
log_get_print_level (get_log_print_level)
get log print level
Running log_get_flags, I see the following log flags:
~/spdk/spdk$ sudo scripts/rpc.py log_get_flags
{
"aio": false,
"app_config": false,
"bdev": false,
"bdev_ftl": false,
"bdev_malloc": false,
"bdev_null": false,
"bdev_nvme": false,
"bdev_ocssd": false,
"bdev_raid": false,
"bdev_raid0": false,
"blob": false,
"blob_rw": false,
"blobfs": false,
"blobfs_rw": false,
"copy_ioat": false,
"ftl_core": false,
"ftl_init": false,
"gpt_parse": false,
"ioat": false,
"json_util": false,
"log": false,
"lvol": false,
"lvolrpc": false,
"nbd": false,
"net": false,
"notify_rpc": false,
"nvme": false,
"nvmf": false,
"nvmf_tcp": false,
"opal": false,
"raidrpc": false,
"reactor": false,
"rpc": false,
"rpc_client": false,
"thread": false,
"trace": false,
"vbdev_delay": false,
"vbdev_gpt": false,
"vbdev_lvol": false,
"vbdev_opal": false,
"vbdev_passthru": false,
"vbdev_split": false,
"vbdev_zone_block": false,
"virtio": false,
"virtio_blk": false,
"virtio_dev": false,
"virtio_pci": false,
"virtio_user": false,
"vmd": false
}
What are the relationship between log level, log print level and log flags. How do I use them? I tried to change log level and log print level from “NOTICE” to “DEBUG” but did not see any change. Also to use them, do I need to make any special SPDK build?
Thanks,
-Wenhua Liu
2 years, 3 months
Using cmb_copy in QEMU environment
by archiemorningstar88＠gmail.com
Hi,
I've been trying to get cmb_copy (examples/nvme/cmb_copy) to work using an emulated SSD with CMB in a QEMU environment, but cmb_copy cannot allocate a CMB buffer. The identify (examples/nvme/identify) reports the two emulated SSD's I created have 128MB of CMB. I'm running QEMU v4.2.0 with the guest OS Ubuntu 18.04. I've also tried various revisions of SPDK (20.01.1, 19.10.1, 19.04.1, 18.04.1) with similar results.
I see these two errors print out which I believe couldn't register the memory so the call to spdk_nvme_ctrlr_alloc_cmb_io_buffer() fails (pctrlr->cmb_bar_virt_addr is NULL):
could not get phys addr for 0x1088200000
nvme_pcie.c: 554:nvme_pcie_ctrlr_map_cmb: *ERROR*: spdk_mem_register() failed
I tried to trace the code and see that when it searches for the physical address g_vtophys_pci_devices list is empty when trying to register the CMB memory space. It only gets later populated with the two SSDs when cleanup_pci_devices() is called at the end of spdk_pci_device_attach().
My qemu startup:
x86_64-softmmu/qemu-system-x86_64 \
-cpu host \
-machine q35,accel=kvm,kernel-irqchip=split \
-smp 8 \
-m 32768 \
-enable-kvm \
-drive if=virtio,file=test1.qcow2,cache=none \
-vnc :1 \
-device e1000,netdev=net0 \
-netdev user,id=net0,hostfwd=tcp::5555-:22 \
-drive file=nvme0.img,if=none,id=D22,format=raw \
-device nvme,drive=D22,serial=1234,cmb_size_mb=128 \
-drive file=nvme1.img,if=none,id=D23,format=raw \
-device nvme,drive=D23,serial=5678,cmb_size_mb=128 \
-device intel-iommu,intremap=on \
If anyone has any suggestions it would be greatly appreciated.
Archie
2 years, 3 months
Re: how to deploy and run spdk in centos7
by Felipe Franciosi
> On Apr 14, 2020, at 5:04 PM, Alexey Marchuk <alexeymar(a)mellanox.com> wrote:
>
> I tried to compile SPDK on Ubuntu 18.04 and run it on Centos 7.7 (CPU arch was the same), at least spdk_tgt failed due to libcrypto dependency:
> error while loading shared libraries: libcrypto.so.1.1: cannot open shared object file: No such file or directory
> Centos: $ locate libcrypto
> /usr/lib64/libcrypto.so
> /usr/lib64/libcrypto.so.1.0.2k
> /usr/lib64/libcrypto.so.10
>
> As you can Centos provides libcrypto.so.10 while Ubuntu uses libcrypro.so.1.1. If you install/compile the same version of this library then I believe it should work. Or you may use another Centos version which provides the same libcrypto version as Ubuntu
> Ubuntu: $ ldd spdk_tgt
> libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
> Centos: $ ldd spdk_tgt
> libcrypto.so.1.1 => not found
Alternatively, you can setup a docker container of centos7 and build
it in there on your Ubuntu machine. In combination with systems like
"mock", that is a much preferred way to cross-compile across Linux
distros. For example:
$ docker run -ti centos:centos7
Unable to find image 'centos:centos7' locally
centos7: Pulling from library/centos
Digest: sha256:4a701376d03f6b39b8c2a8f4a8e499441b0d567f9ab9d58e4991de4472fb813c
Status: Downloaded newer image for centos:centos7
[root@6c4ab3740af9 /]# cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
[root@6c4ab3740af9 /]#
And then:
[root@6c4ab3740af9 /]# yum install -y libssh-devel
...
[root@6c4ab3740af9 /]# ls /lib64/libcrypto*
/lib64/libcrypto.so.1.0.2k /lib64/libcrypto.so.10
[root@6c4ab3740af9 /]#
F.
>
> Best regards,
> Alexey Marchuk
>
> -----Original Message-----
> From: Walker, Benjamin <benjamin.walker(a)intel.com>
> Sent: Tuesday, April 14, 2020 5:26 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] Re: how to deploy and run spdk in centos7
>
>
>
>> -----Original Message-----
>> From: Yang, Ziye <ziye.yang(a)intel.com>
>> Sent: Monday, April 13, 2020 8:05 PM
>> To: Storage Performance Development Kit <spdk(a)lists.01.org>
>> Subject: [SPDK] Re: how to deploy and run spdk in centos7
>>
>> Hi Xianghong,
>>
>> Can you make sure that the machine which has ubuntu is same as the
>> machine which has centos? If not, the complied binary may fail to run
>> on centos. Because when you build SPDK/DPDK on ubuntu, it detects the
>> CPU type and leverages some features from CPU to optimize some
>> functions with CPU instructions. If there are no such instructions on
>> the machine which has centos, it will fail to execute.
>
> If this does end up being the problem, you can add the --target-arch parameter to SPDK's configure script to choose a specific architecture to build for.
>
>>
>>
>>
>>
>> Best Regards
>> Ziye Yang
>>
>> -----Original Message-----
>> From: mengxianghong2008(a)126.com <mengxianghong2008(a)126.com>
>> Sent: Tuesday, April 14, 2020 10:33 AM
>> To: spdk(a)lists.01.org
>> Subject: [SPDK] how to deploy and run spdk in centos7
>>
>> I build spdk at unbuntu, after build, how can I deploy it and run at centos7?
>> Anyone can give me a doc or link to instructure me, best wishes!!!
>> _______________________________________________
>> SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to
>> spdk-leave(a)lists.01.org
>> _______________________________________________
>> SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to
>> spdk-leave(a)lists.01.org
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
2 years, 4 months
Mellanox build bot
by Alexey Marchuk
Hello,
We enabled verification of SPDK patches on ARM. It is triggered automatically on each patchset - performs compilation and runs unit tests. We plan to extend testing scope with new functional tests.
Mellanox build bot puts Community-CI +/-1 votes, logs are available at the link in the comment. If you think that -1 vote was due to "false positive" error, you can retrigger a new build by leaving a comment with "mellanox:retest" phrase.
We are open for feedback and improvements.
Best regards,
Alexey Marchuk
2 years, 4 months
