Reporting intermittent test failures
by Harris, James R
Hi all,
I’ve seen a lot of cases recently where -1 votes from the test pool have been removed from a patch due to a failure unrelated to the patch, but then nothing was filed in GitHub for that failure. The filing in GitHub could be a new issue, or a comment on an existing issue.
Please make those GitHub updates a priority. It’s the only way the project can understand the frequency of those intermittent failures and gather to get them fixed. If you’re not sure if a failure has been seen before, search GitHub issues with the “Intermittent Failure” label, or ask on Slack if anyone else has seen the issue. There is no harm in filing a new issue that may be a duplicate – we can always clean these up later during the next bug scrub meeting. The important thing is that we get the failure tracked.
Thanks,
-Jim
5 months
SPDK socket abstraction layer
by Zawadzki, Tomasz
Hello everyone,
Summary:
With this message I wanted to update SPDK community on state of VPP socket abstraction as of SPDK 19.07 release.
At this time there does not seem to be a clear efficiency improvements with VPP. There is no further work planned on SPDK and VPP integration.
Details:
As some of you may remember, SPDK 18.04 release introduced support for alternative socket types. Along with that release, Vector Packet Processing (VPP)<https://wiki.fd.io/view/VPP> 18.01 was integrated with SPDK, by expanding socket abstraction to use VPP Communications Library (VCL). TCP/IP stack in VPP<https://wiki.fd.io/view/VPP/HostStack> was in early stages back then and has seen improvements throughout the last year.
To better use VPP capabilities, following fruitful collaboration with VPP team, in SPDK 19.07, this implementation was changed from VCL to VPP Session API from VPP 19.04.2.
VPP socket abstraction has met some challenges due to inherent design of both projects, in particular related to running separate processes and memory copies.
Seeing improvements from original implementation was encouraging, yet measuring against posix socket abstraction (taking into consideration entire system, i.e. both processes), results are comparable. In other words, at this time there does not seem to be a clear benefit of either socket abstraction from standpoint of CPU efficiency or IOPS.
With this message I just wanted to update SPDK community on state of socket abstraction layers as of SPDK 19.07 release. Each SPDK release always brings improvements to the abstraction and its implementations, with exciting work on more efficient use of kernel TCP stack - changes in SPDK 19.10 and SPDK 20.01.
However there is no active involvement at this point around VPP implementation of socket abstraction in SPDK. Contributions in this area are always welcome. In case you're interested in implementing further enhancements of VPP and SPDK integration feel free to reply, or to use one of the many SPDK community communications channels<https://spdk.io/community/>.
Thanks,
Tomek
10 months, 2 weeks
vfio multiple processes
by Nabarro, Tom
Can't run multiple SPDK processes each accessing disjoint set of NVMe drives using VFIO driver (even as root). Each process seems to try to access all /dev/vfio/* device files and get lots of resource busy messages. Sometimes one of the processes complaint it can't find the device it's been assigned in nvme.conf.
Have any ideas?
(processes are started in normal, non-multiprocess mode)
Thanks in advance
Best regards,
Tom Nabarro BEng (hons)
Extreme Storage Architecture & Development
Intel Corporation
E: tom.nabarro(a)intel.com<mailto:tom.nabarro@intel.com>
M: +44 (0)7786 260986
Skype: tom.nabarro
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
1 year, 2 months
[Release] 20.01: Optimized thin provisioning, FTL bdev, VMD hot plug, FUSED support
by Zawadzki, Tomasz
On behalf of the SPDK community I'm pleased to announce the release of SPDK 20.01!
The 20.01 release will be SPDK's first long term support release. This release will be supported for 1 year and will have bug fix releases quarterly. This release will guarantee API stability (but not ABI stability) for its lifetime.
This release contains the following new features:
- Optimized thin provisioning: Metadata format in blobstore changed to allow for more optimized cluster allocation with thin provisioned blobs. This improves write performance and reduces the number of writes on each cluster allocation.
- Flash Translation Layer bdev: FTL library is now based on zoned bdev API, instead of Open Channel interface. Added FTL virtual bdev, that can be created on top of any bdev supporting zoned bdev API.
- VMD hot plug: Added support for hot plug and hot remove detection for devices behind the VMD.
- Compare and write: Support for NVMe FUSED commands in NVMe-oF target and NVMe driver has been added. Along with compare and write operations on bdev layer this allows for atomic operations on supported hardware or via emulation.
- NVMe-oF custom NVMe admin command: Added support for implementing handler for specific NVMe admin commands in NVMe-oF target. This can be used to implement or replace NVMe admin commands that are currently not handled by the NVMe-oF subsystem.
- NVMe-oF pluggable transports: The NVMe-oF target now supports plugging out of tree NVMe-oF transports.
- NVMe-oF/TCP and iSCSI performance improvements: The posix module in SPDK's socket abstraction layer has received a number of major improvements, including an asynchronous writev that automatically batches writes to minimize syscalls and support for MSG_ZEROCOPY. Further improvements have been made specifically to NVMe-oF/TCP to batch receives. These improvements yield as much as 3x improvements in I/Ops per core on some workloads.
- Crypto bdev: Significant performance improvement in encryption (writes) for both AESNI and QAT polled mode drivers. Additionally, QAT queue pair load balancing improves scaling with cores in all workloads.
- DPDK: Added support for DPDK 19.11.
The full changelog for this release is available at:
https://github.com/spdk/spdk/releases/tag/v20.01
This release contains 935 commits from 42 authors with over 52k lines of code changed.
We'd especially like to recognize all of our first time contributors:
Alok Kataria
Artur Paszkiewicz
Ivan Betsis
Łukasz Radomski
Suhua
Zhang Junfeng
Thanks to everyone for your contributions, participation, and effort!
Thanks,
Tomek
1 year, 2 months
Preparation for SPDK 20.01 release
by Zawadzki, Tomasz
Hello all,
The merge window for SPDK 20.01 release will close by January 24th.
Please ensure all patches you believe should be included in the release are merged to master branch by this date.
You can do it by adding a hashtag '20.01' in GerritHub on those patches.
The current set of patches that are tagged and need to be reviewed can be seen here:
https://review.gerrithub.io/q/hashtag:%2220.01%22+(status:open)
As with previous release, process used so far for Maintenance releases (ex. 19.10.1) will be applied to main releases. On January 24th new branch 'v20.01.x' will be created, and a patch on it will be tagged as release candidate.
Then, on January 31th, a formal release will take place tagging the last patch on the branch as SPDK 20.01.
Between release candidate and formal release, only critical fixes shall be backported to the 'v20.01.x' branch.
Development can continue without disruptions on 'master' branch.
Thanks,
Tomek
1 year, 2 months
vfio
by Nabarro, Tom
I'm trying to switch from UIO to VFIO and the spdk/examples/nvme examples work on my test node when I've performed the switch.
[ DPDK EAL parameters: daos --no-shconf -c 0x1 --log-level=lib.eal:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid182014 ]
ERROR: daos_io_server:0 EAL: No free hugepages reported in hugepages-1048576kB
daos_io_server:0 EAL: VFIO support initialized
daos_io_server:0 EAL: using IOMMU type 1 (Type 1)
daos_io_server:0 EAL: Cannot open /dev/vfio/47: Permission denied
EAL: Failed to open group 47
I've tried to follow the env init including opts as per the examples and the DPDK EAL parameters look the same as when running examples.
Any ideas?
Best regards,
Tom Nabarro BEng (hons)
Extreme Storage Architecture & Development
Intel Corporation
E: tom.nabarro(a)intel.com<mailto:tom.nabarro@intel.com>
M: +44 (0)7786 260986
Skype: tom.nabarro
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
1 year, 2 months
spdk_thread_{exit,destroy} safety
by Andrey Kuzmin
Hi guys,
I'm using spdk_thread_create to provide single-threaded context for the
operations that otherwise would require locking. Everything runs nice up to
exiting/destroying such a thread. In particular, after calling
spdk_thread_exit I'm witnessing thread being unable to process pending
messages (such as _spdk_put_io_channel which are generated by
spdk_put_io_channel issued on the exiting thread before calling
spdk_thread_exit), and destroying thread at this doesn't seem safe.
Is there any recommended procedure to be followed before calling
thread_exit so that the thread consumes all outstanding messages and is safe
to destroy? I would assume something like
while (spdk_thread_poll(thread) > 0);
spdk_thread_exit(thread);
spdk_thread_destroy(thread);
should do the trick, but I'd prefer to rely on a recommended procedure.
Thanks,
Andrey
1 year, 2 months
