ARM64 and coherency with SPDK
by John Linn
Are there any assumptions about the SPDK on ARM64 with h/w coherency of the system such that s/w coherency is not an option?
We might have configured it wrong, but without h/w coherency we are not seeing the NVMe examples work correctly.
I'm newer to huge pages and pinned pages which sound like they should be coherent.
Or maybe I missed this in the documentation and you can point me to the right place.
Thanks
John
1 year, 7 months
FW: SPDK with SoftRoce and RSD.
by Lipka, Kamil
Hello,
we are developing virtual environment for deploying Intel Rack Scale Design with SPDK as a storage backend and SoftRoce for RDMA capabilities.
We are experiencing some difficulties with 'nvme discovery' and 'nvme connect' commands.
After we successfully create volumes and subsystems on target with spdk, we then invoke nvme discover command on initiator the response is as expected:
vagrant@vagrant:~$ sudo nvme discover -t rdma -s 4450 -a 192.168.121.86 -q 3010b188-7c46-4219-838c-a5900d1c1a57
Discovery Log Number of Records 1, Generation counter 4
=====Discovery Log Entry 0======
trtype: rdma
adrfam: ipv4
subtype: nvme subsystem
treq: not specified
portid: 0
trsvcid: 4450
subnqn: nqn.2014-08.org.nvmexpress:uuid:822aa643-034c-4af5-8be7-0087585376af
traddr: 192.168.121.86
rdma_prtype: not specified
rdma_qptype: connected
rdma_cms: rdma-cm
rdma_pkey: 0x0000
but when we invoke it one more time the response is:
vagrant@vagrant:~$ sudo nvme discover -t rdma -s 4450 -a 192.168.121.86 -q 3010b188-7c46-4219-838c-a5900d1c1a57
Failed to write to /dev/nvme-fabrics: Connection reset by peer
dmesg logs for these commands:
[ 2347.388124] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 192.168.121.86:4450
[ 2347.389248] nvme nvme0: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[ 2348.071467] nvme nvme0: Connect rejected: status 10 (stale conn).
[ 2348.071493] nvme nvme0: rdma connection establishment failed (-104)
Then after around 5 seconds response for 'nvme discovery' is again as expected (logs from dmesg):
[ 2347.388124] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 192.168.121.86:4450
[ 2347.389248] nvme nvme0: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[ 2348.071467] nvme nvme0: Connect rejected: status 10 (stale conn).
[ 2348.071493] nvme nvme0: rdma connection establishment failed (-104)
[ 2349.551415] nvme nvme0: Connect rejected: status 10 (stale conn).
[ 2349.551443] nvme nvme0: rdma connection establishment failed (-104)
[ 2349.767590] nvme nvme0: Connect rejected: status 10 (stale conn).
[ 2349.767620] nvme nvme0: rdma connection establishment failed (-104)
[ 2349.983495] nvme nvme0: Connect rejected: status 10 (stale conn).
[ 2349.983521] nvme nvme0: rdma connection establishment failed (-104)
[ 2350.191459] nvme nvme0: Connect rejected: status 10 (stale conn).
[ 2350.191486] nvme nvme0: rdma connection establishment failed (-104)
[ 2350.383501] nvme nvme0: Connect rejected: status 10 (stale conn).
[ 2350.383529] nvme nvme0: rdma connection establishment failed (-104)
[ 2350.599496] nvme nvme0: Connect rejected: status 10 (stale conn).
[ 2350.599523] nvme nvme0: rdma connection establishment failed (-104)
[ 2350.831376] nvme nvme0: Connect rejected: status 10 (stale conn).
[ 2350.831403] nvme nvme0: rdma connection establishment failed (-104)
[ 2351.627940] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 192.168.121.86:4450
[ 2351.629125] nvme nvme0: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
[ 2351.855416] nvme nvme0: Connect rejected: status 10 (stale conn).
[ 2351.855443] nvme nvme0: rdma connection establishment failed (-104)
Logs from SPDK contain and client are attached.
Can you help us with this?
Do you need more info?
Best regards
Kamil
--------------------------------------------------------------------
Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.
Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.
1 year, 7 months
assigning name errors
by Nabarro, Tom
Hello all
It doesn't seem to cause any functional problems but these errors keep appearing:
"bdev_nvme.c: 977:attach_cb: *ERROR*: Failed to assign name to NVMe device"
Can someone shed light on why that might be happening and what it means.
Thanks
Tom
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
1 year, 7 months
Re: [SPDK] blobstore io unit size
by Niu, Yawei
Got it, thanks.
On 24/07/2019, 7:58 PM, "SPDK on behalf of Zawadzki, Tomasz" <spdk-bounces(a)lists.01.org on behalf of tomasz.zawadzki(a)intel.com> wrote:
Hi Niu,
Thanks for upgrading to the SPDK v19.04.1 ! On related note 19.07 is coming.
io_unit was introduced with SPDK v18.10 as a way to expose differently sized devices (512/4k) as blobs/logical volumes.
Blobstore backwards compatibility for pre-18.10 blobs is handled by assuming SPDK_BS_PAGE_SIZE as io_unit_size for those.
As for 'forwards compatibility' - loading new blob with old SPDK app - only certain configurations might work (ex. 4k io_unit_size) by accident
and should not be depended upon.
Applications based on SPDK post 18.10 should always check the spdk_bs_get_io_unit_size(), to know the proper size for submissions.
Same as applications based on SPDK pre 18.10, should have been checking for spdk_bs_get_page_size().
This means that old blobs will work with old and new apps, as long as the apps check for correct size as appropriate for corresponding SPDK version.
Description of spdk_blob_io_unmap() in the documentation is incorrect, all spdk_blob_io_*() calls use io_units as ... well units for io :)
Thanks for noticing, please see following patch correcting the typo:
https://review.gerrithub.io/c/spdk/spdk/+/463067
Thanks,
Tomek
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Niu, Yawei
> Sent: Wednesday, July 24, 2019 10:37 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] blobstore io unit size
>
> Hi, All
>
> Since I upgraded to SPDK v19.04.1, I realized there is an io unit size
> introduced for blobstore (the blobsotre io unit was fixed 4k page size
> before), and seems that caused a nasty compatibility issue: when I call
> spdk_bs_get_io_unit_size() against an old blobstore, it returns 4k, however,
> the blobstore on NVMe device actually uses 512 bytes io unit size internally.
> To my understanding, if 512 unit size is used in new SPDK, the old app will
> break inevitably (old app usually uses fixed 4k size), we’d just return error
> when loading blobstore to report this incompatible issue, or, new SPDK have
> to keep using 4k unit to support old app, right?
>
> Another thing I want to get confirmed is the parameters used for
> spdk_blob_io_unmap(), as document mentioned: ‘offset’ is the io units from
> the beginning of blob, ‘length’ is the size of unmap area in pages, which
> means for a 512 io unit size blobstore, the ‘offset’ should be calculated by
> ‘off_bytes / 512’, and the ‘length’ is still calculated by ‘size_bytes / 4k’?
>
> Thanks
> -Niu
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
1 year, 7 months
blobstore io unit size
by Niu, Yawei
Hi, All
Since I upgraded to SPDK v19.04.1, I realized there is an io unit size introduced for blobstore (the blobsotre io unit was fixed 4k page size before), and seems that caused a nasty compatibility issue: when I call spdk_bs_get_io_unit_size() against an old blobstore, it returns 4k, however, the blobstore on NVMe device actually uses 512 bytes io unit size internally. To my understanding, if 512 unit size is used in new SPDK, the old app will break inevitably (old app usually uses fixed 4k size), we’d just return error when loading blobstore to report this incompatible issue, or, new SPDK have to keep using 4k unit to support old app, right?
Another thing I want to get confirmed is the parameters used for spdk_blob_io_unmap(), as document mentioned: ‘offset’ is the io units from the beginning of blob, ‘length’ is the size of unmap area in pages, which means for a 512 io unit size blobstore, the ‘offset’ should be calculated by ‘off_bytes / 512’, and the ‘length’ is still calculated by ‘size_bytes / 4k’?
Thanks
-Niu
1 year, 7 months
Re: [SPDK] shared lib and constructor functions
by Niu, Yawei
I see the problem now, the libspdk.so is actually not a shared lib at all, it's very small (1066 bytes in my build) and not recognizable by nm or ldd, but putting '-lspdk' to link option can pacify compiler not complaining 'undefined reference to...'. I tried to link each individual shared libs, like spdk_bdev_nvme, spdk_bdev_aio, etc. the constructors are successfully called at the end.
So what's the libspdk.so generated by 'configure --with-shared; make'? It suppress the 'undefined reference' errors but actually doesn't provide anything? What's the proper way to link SPDK shared libs?
Thanks
-Niu
On 12/07/2019, 2:45 PM, "SPDK on behalf of Niu, Yawei" <spdk-bounces(a)lists.01.org on behalf of yawei.niu(a)intel.com> wrote:
Hi,
I tried to build my app with SPDK shared lib and found that the constructor functions (for SPDK module registration) don’t work anymore, it worked before when I was using old SPDK which producing single shared lib ‘spdk’, the gcc command for old SPDK looks like:
gcc -o mylib.so -shared -lspdk …
For new SPDK:
gcc -o mylib.so -Wl,--whole-archive -lrte_mempool … (all the DPDK static libs)… -Wl,--no-whole-archive -shared -lspdk -lspdk_env_dpdk ….
Is this the correct way to link the new shared libs?
Thanks
-Niu
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
1 year, 7 months
Nvme Drive hot insertion detection time (github issue #589)
by Szwed, Maciej
Hi,
I want do some cleanup in our github issues and noticed that issue #589<https://github.com/spdk/spdk/issues/589> have a fix that hasn't been confirmed. Please let me know if somebody encountered such issue lately or had a chance to verify provided fix. My own tests confirmed that vfio hotplug works correctly. Short reminder of the issue topic: "If drive is hot inserted with nvmf_tgt running, it takes 2 to 10 minutes for new drives to be detected by utilities identify or nvme cli.'
Thanks,
Maciek
1 year, 7 months
Registration Open! 2019 SPDK PMDK & VTune™ PRC Summit – September 5-6 in Beijing, China
by Cao, Gang
Hi all,
We are pleased to announce that 2019 PRC SPDK PMDK & VTune™ Amplifier Summit at September 5-6 in Beijing has been open for registration.
Chinese version registration link here: https://www.bagevent.com/event/2659663
English version registration link here: https://www.bagevent.com/event/2659838
You can pick either one of them to register.
Below is some additional information. More please refer to the registration website on the two-days agenda, speakers and so on.
Join Intel Customer’s developers and architects in this external showcase of some of the top Intel Software Solutions. A great opportunity to meet with the SPDK, PMDK and VTune™ user communities and listen to a new series of talks from early adopters and long-term users. Topics will include case studies, future deep dives and tech tutorials. There will also be demos, hands on labs and informal technical discussions with the experts.
Storage Performance Development Kit (SPDK)
SPDK(https://spdk.io/) is a storage reference software development kit designed for latest non-volatile media, designed to allow for millions of IOPS per core, while reducing both average and tail latency, all with an open source license.
Persistent Memory Development Kit (PMDK)
PMDK(https://pmem.io/pmdk/) is a vendor-neutral, growing collection of libraries started by Intel, motivated by the introduction of Intel® Optane™ DC Persistent Memory. PMDK will work with any NVDIMMs that support the SNIA NVM Programming Model. These libraries have been developed for various use cases, tuned, validated to production quality, and thoroughly documented.
Intel® VTune™ Amplifier
Intel® VTune™ Amplifier(https://software.intel.com/en-us/vtune) collects key profiling data needed to optimize modern polled I/O that uses SPDK or PMDK. It also has analyses to optimize the use and configuration of the latest Intel® Optane™ DC persistent memory. Also featuring Intel® Inspector - Persistence Inspector(https://software.intel.com/en-us/articles/detect-persistent-mem....
Thanks,
Gang
1 year, 7 months
