thoughts of vhost-user disks + RDMA
by Li Feng
Hi experts,
I have questions about getting data from the vhost-user and sending
out through RDMA with zero-copy.
When using vhost-user-blk/scsi, all VM's memory range is mapped to target.
For example, if a VM has a very big memory, like 100G, and it connects
to 10 vhost-user-blk disks.
Then the vhost-user target needs mmap 10*100G virtual memory.
If the data needs to transfer to another host through RDMA, and don't
want to copy the data out of vhost-user, you should register the all
virtual memory of vhost-user as RDMA MR at initialization. Because you
don't know the virtio data is put in which memory range, and only
register part of virtual memory. right?
There will have so big range of memory that needs to be registered.
With some tests, the `ibv_reg_mr` call will make the host hang when
registering serval TB virtual memory.
If registering MR at the IO path, the performance is not acceptable.
Any ideas about this issue if don't want to copy data?
Thanks,
Feng Li
--
The SmartX email address is only for business purpose. Any sent message
that is not related to the business is not authorized or permitted by
SmartX.
本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
1 year
performing i/o to a disk while removing cause the cpu to stuck
by Avner Taieb
Hi,
We are using SPDK 19.07, nvme device, vfio driver.
We got into a situation that removing disk, while performing I/O to it,
stall the CPU in the function spdk_mmio_write_4, when accessing the PCI
register, in the line *addr = val
any idea ?
Thanks,
Avner
1 year
Failed to link spdk library
by gaiaismus＠outlook.com
HI, After I updated the SPDK version to v20.01, my project failed link to spdk library.
This is part of my CMakeList.txt
----------------------------------------------------------------------------------
include_directories(${PROJECT_SOURCE_DIR}/include)
include_directories(/usr/local/include)
include_directories(${PROJECT_SOURCE_DIR}/../spdk/include)
link_directories(/usr/local/lib)
link_directories(${PROJECT_SOURCE_DIR}/../spdk/dpdk/build/lib)
link_directories(${PROJECT_SOURCE_DIR}/../spdk/build/lib)
set(EXECUTABLE_OUTPUT_PATH ${PROJECT_SOURCE_DIR}/build)
set(LIBRARY_OUTPUT_PATH ${PROJECT_SOURCE_DIR}/buid/lib)
add_executable(io_demo ${DIR_SRCS})
target_link_libraries(
io_demo
dpdk pthread abt dl numa uuid m rt
spdk_app_rpc spdk_bdev spdk_bdev_aio spdk_bdev_delay spdk_bdev_error spdk_bdev_ftl spdk_bdev_gpt spdk_bdev_lvol
spdk_bdev_malloc spdk_bdev_null spdk_bdev_nvme spdk_bdev_passthru spdk_bdev_raid spdk_bdev_rpc
spdk_bdev_split spdk_bdev_virtio spdk_bdev_zone_block spdk_blob spdk_blob_bdev
spdk_blobfs spdk_blobfs_bdev spdk_conf spdk_copy spdk_copy_ioat spdk_env_dpdk spdk_env_dpdk_rpc
spdk_event spdk_event_bdev spdk_event_copy spdk_event_iscsi spdk_event_nbd spdk_event_net spdk_event_nvmf
spdk_event_scsi spdk_event_vhost spdk_event_vmd spdk_ftl spdk_ioat spdk_iscsi spdk_json spdk_jsonrpc
spdk_log spdk_log_rpc spdk_lvol spdk_nbd spdk_net spdk_notify spdk_nvme spdk_nvmf spdk_rpc spdk_scsi
spdk_sock spdk_sock_posix spdk_thread
spdk_vhost spdk_virtio spdk_vmd
spdk_ut_mock
rte_bus_vdev rte_compressdev rte_eal rte_fib rte_kvargs rte_mempool rte_mempool_ring
rte_net rte_rib rte_vhost rte_bus_pci rte_cmdline rte_cryptodev rte_ethdev rte_hash rte_mbuf
rte_mempool_bucket rte_meter rte_pci rte_ring
spdk_trace
spdk_util
spdk_env_dpdk spdk_iscsi spdk_event_iscsi spdk_scsi
)
This is my error message
We are running with CentOS Linux release 7.4.1708 (Core)
----------------------------------------------------------------------------------
/home/xxx/io-test/io/../spdk/build/lib/libspdk_util.a(crc32c.o): In function `spdk_crc32c_update':
/home/xxx/io-test/spdk/lib/util/crc32c.c:52: undefined reference to `crc32_iscsi'
collect2: error: ld returned 1 exit status
make[2]: *** [io_demo] Error 1
make[1]: *** [CMakeFiles/io_demo.dir/all] Error 2
make: *** [all] Error 2
----------------------------------------------------------------------------------
If someone can help me resolve this error or provide a correct CMakeList example, will be much appreciated.
1 year
128k sequential read / write (fio) performance with spdk+vpp is not as good as the one with "kernel TCP"
by sejun.kwon＠samsung.com
Hello,
I'm working on SPDK library + VPP, because some report said that VPP reduces the overhead of network.
When I test with VPP (with mlx5 poll mode driver) and null device with spdk, 4k performance with VPP is much better than the default(kTCP).
But, 128k write performance with VPP is 30 percent lower than the one with kTCP.
The configuration when VPP starts is as below.
Is there anyone why 128k write performance with VPP is not good as kernel ?
I increase num-rx-desc and there is improvment, but it still lower than kernel.
Thanks in advance.
unix {
nodaemon
cli-listen /run/vpp/cli.sock
# cli-listen localhost:5002
full-coredump
}
session {
evt_qs_memfd_seg
}
dpdk {
dev 0000:03:00.0
{
}
}
socksvr {
# socket-name /tmp/vpp-api.sock
socket-name /run/vpp-api.sock
}
plugins {
plugin default { disable }
plugin dpdk_plugin.so { enable }
}
1 year
Expand your horizons with reviews!
by Luse, Paul E
Hello Everyone,
Wanted to take a minute to recognize those who spend so much time reviewing patches. Reviews are one of the most important things all of us can do to help build a vibrant community and continue to keep the quality level as high as it has been for years. It’s easy to fall into a “trap” though, where you tend to review only patches in areas of code that you know.
We don’t want to discourage building expertise, of course, but it’s also critically important to review patches in areas of code that you don’t know. You’ll learn more about SPDK while at the same time providing potentially very valuable feedback. Don’t be shy about asking questions to learn, and let them know you are curious about the code; you’ll be amazed at how many times the author will rethink their implementation after someone with fresh eyes has taken a look.
So remember, it’s the maintainers’ job review & approve, but it’s everyone’s job to review ☺
Thanks!!
Paul
1 year
ask for help about SPDK CI
by Richael Zhuang
Hi,
I got one problem about the new CI system. The CI is not triggered after I submit a patch to review.spdk.io.
So I'm wondering whether I miss something about the process to submit a patch?
Best Regards,
Richael
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
1 year
Schedule for SPDK 20.04 release
by Zawadzki, Tomasz
Hello all,
In order for the release process progress as smooth as possible,
the merge window for SPDK 20.04 release will close by April 17th.
Please ensure all patches you believe should be included in the release
are merged to master branch by this date.
Then, by end of April, a formal release will take place.
Extended period between merge window closing and release is
in anticipation of the 2020 US SPDK, PMDK and VTune Summit held on April 21-22.
You can find out more about the summit and register to attend:
https://spdk.io/news/2020/01/28/us_summit_announce/
Thanks,
Tomek
1 year
vfio multiple processes
by Nabarro, Tom
Can't run multiple SPDK processes each accessing disjoint set of NVMe drives using VFIO driver (even as root). Each process seems to try to access all /dev/vfio/* device files and get lots of resource busy messages. Sometimes one of the processes complaint it can't find the device it's been assigned in nvme.conf.
Have any ideas?
(processes are started in normal, non-multiprocess mode)
Thanks in advance
Best regards,
Tom Nabarro BEng (hons)
Extreme Storage Architecture & Development
Intel Corporation
E: tom.nabarro(a)intel.com<mailto:tom.nabarro@intel.com>
M: +44 (0)7786 260986
Skype: tom.nabarro
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
1 year
