Announcing the 2018 SPDK Developer Meetup hosted by NetApp
by Luse, Paul E
Come and join the 2nd annual SPDK Developer Meetup, this year hosted by NetApp in beautiful Sunnyvale CA! It's an excellent opportunity for networking, learning and making forward progress on the code and generally making the community more productive. We'll be covering all sorts of topics but a major theme this year will be NVMeOF (including FC support) so if you're interested in these topics you won't want to miss this meet-up.
Please see the SPDK blog for complete details including a registration link:
http://www.spdk.io/blog/
Thanks!
Paul
2 years, 10 months
Bug Scrub meeting is canceled today <Sent from email again>
by Wan, Qun
Dear all
As there are some logistic issue for Webex that not fixed yet, you will find that the Webex will report "ROOM ID invalid", so today's Bug Scrub will be canceled, we are updating the status in the github issue tracker and pls. feel free to update in the github if you have any questions or help needed from us. Sorry for any inconvenience. See you next time.
The calendar has been canceled before the meeting, and now there are some people ask this in the community, I guess someone Didn't receive it, just sent by email again. Sorry for any inconvenience.
Best Regards,
Anna
2 years, 10 months
Using SPDK as QEMU's rootfs disk && A patch for libguestfs to support SPDK
by Bob Chen
Hi guys,
During the last few weeks, I've been looking at how to use SPDK as QEMU's
bootable rootfs disk.
I managed to boot up a SPDK rootfs disk by using OVMF UEFI boot-loader. And
in order to deploy the guest OS before start-up(which has a unrecognizable
filesystem to the host), I have written a small patch for the libguestfs.
It was based on Redhat's CentOS libguestfs-1.36.10 RPM, hopefully somebody
could add this SPDK support into the mainline code.
What I'm asking here is a new problem I encountered yesterday. When I
increased the guest memory to more than 8GB, the guest kernel would crash
at boot-up (see attached screen-shot). This issue would not be reproduced
if I only use SPDK as the data disk.
Maybe there is something wrong with the boot-loader?
Thanks, Bob
<0083-Add-SPDK-drive-support.patch>
------------------------------------------------------
>From 7927b0fa080db923989cef7181531dfaa09ccf6c Mon Sep 17 00:00:00 2001
From: Bob Chen <a175818323(a)gmail.com>
Date: Fri, 27 Jul 2018 12:55:28 +0800
Subject: [PATCH 1/1] Add SPDK drive support
---
lib/drives.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
lib/guestfs-internal.h | 2 ++
lib/launch-direct.c | 20 ++++++++++++++++++--
lib/launch-libvirt.c | 3 +++
lib/qemu.c | 3 +++
5 files changed, 71 insertions(+), 2 deletions(-)
diff --git a/lib/drives.c b/lib/drives.c
index d3887c1..780b1b2 100644
--- a/lib/drives.c
+++ b/lib/drives.c
@@ -396,6 +396,45 @@ create_drive_iscsi (guestfs_h *g,
}
#endif /* DISABLED IN RHEL 7 */
+static struct drive *
+create_drive_spdk (guestfs_h *g,
+ const struct drive_create_data *data)
+{
+ if (data->username != NULL) {
+ error (g, _("spdk: you cannot specify a username with this protocol"));
+ return NULL;
+ }
+ if (data->secret != NULL) {
+ error (g, _("spdk: you cannot specify a secret with this protocol"));
+ return NULL;
+ }
+
+ if (data->nr_servers != 1) {
+ error (g, _("spdk: you must specify exactly one server"));
+ return NULL;
+ }
+
+ if (data->servers[0].transport != drive_transport_unix) {
+ error (g, _("spdk: only unix transport is supported"));
+ return NULL;
+ }
+
+ return create_drive_non_file (g, data);
+}
+
+bool
+guestfs_int_has_spdk_drive (guestfs_h *g)
+{
+ size_t i;
+ struct drive *drv;
+ ITER_DRIVES(g, i, drv) {
+ if (drv->src.protocol == drive_protocol_spdk) {
+ return true;
+ }
+ }
+ return false;
+}
+
/**
* Create the special F</dev/null> drive.
*
@@ -494,6 +533,7 @@ guestfs_int_drive_protocol_to_string (enum
drive_protocol protocol)
case drive_protocol_sheepdog: return "sheepdog";
case drive_protocol_ssh: return "ssh";
case drive_protocol_tftp: return "tftp";
+ case drive_protocol_spdk: return "spdk";
}
abort ();
}
@@ -878,6 +918,11 @@ guestfs_impl_add_drive_opts (guestfs_h *g, const char
*filename,
drv = create_drive_curl (g, &data);
}
#endif /* DISABLED IN RHEL 7 */
+ else if (STREQ (protocol, "spdk")) {
+ data.protocol = drive_protocol_spdk;
+ data.iface = safe_strdup(g, "vhost-user-scsi-pci");
+ drv = create_drive_spdk(g, &data);
+ }
else {
error (g, _("unknown protocol '%s'"), protocol);
drv = NULL; /*FALLTHROUGH*/
diff --git a/lib/guestfs-internal.h b/lib/guestfs-internal.h
index 3bae02b..a0e5573 100644
--- a/lib/guestfs-internal.h
+++ b/lib/guestfs-internal.h
@@ -199,6 +199,7 @@ enum drive_protocol {
drive_protocol_sheepdog,
drive_protocol_ssh,
drive_protocol_tftp,
+ drive_protocol_spdk,
};
enum drive_transport {
@@ -829,6 +830,7 @@ extern void guestfs_int_rollback_drives (guestfs_h *g,
size_t);
extern void guestfs_int_add_dummy_appliance_drive (guestfs_h *g);
extern void guestfs_int_free_drives (guestfs_h *g);
extern const char *guestfs_int_drive_protocol_to_string (enum
drive_protocol protocol);
+extern bool guestfs_int_has_spdk_drive (guestfs_h *g);
/* appliance.c */
extern int guestfs_int_build_appliance (guestfs_h *g, char **kernel, char
**initrd, char **appliance);
diff --git a/lib/launch-direct.c b/lib/launch-direct.c
index 3d6e72e..f157324 100644
--- a/lib/launch-direct.c
+++ b/lib/launch-direct.c
@@ -409,8 +409,17 @@ launch_direct (guestfs_h *g, void *datav, const char
*arg)
ADD_CMDLINE_PRINTF ("%d", g->smp);
}
- ADD_CMDLINE ("-m");
- ADD_CMDLINE_PRINTF ("%d", g->memsize);
+ if (guestfs_int_has_spdk_drive(g)) {
+ ADD_CMDLINE ("-m");
+ ADD_CMDLINE ("1G");
+ ADD_CMDLINE ("-object");
+ ADD_CMDLINE
("memory-backend-file,id=mem0,size=1G,mem-path=/dev/hugepages,share=on");
+ ADD_CMDLINE ("-numa");
+ ADD_CMDLINE ("node,memdev=mem0");
+ } else {
+ ADD_CMDLINE ("-m");
+ ADD_CMDLINE_PRINTF ("%d", g->memsize);
+ }
/* Force exit instead of reboot on panic */
ADD_CMDLINE ("-no-reboot");
@@ -567,6 +576,13 @@ launch_direct (guestfs_h *g, void *datav, const char
*arg)
goto cleanup0;
}
#endif
+ else if (drv->iface && STREQ(drv->iface, "vhost-user-scsi-pci")) {
+ /* SPDK */
+ ADD_CMDLINE ("-chardev");
+ ADD_CMDLINE_PRINTF ("socket,id=vhost,path=%s", escaped_file);
+ ADD_CMDLINE ("-device");
+ ADD_CMDLINE ("vhost-user-scsi-pci,chardev=vhost");
+ }
else if (drv->iface) {
ADD_CMDLINE ("-drive");
ADD_CMDLINE_PRINTF ("%s,if=%s", param, drv->iface);
diff --git a/lib/launch-libvirt.c b/lib/launch-libvirt.c
index 05c398b..468ece9 100644
--- a/lib/launch-libvirt.c
+++ b/lib/launch-libvirt.c
@@ -1548,6 +1548,9 @@ construct_libvirt_xml_disk (guestfs_h *g,
case drive_protocol_tftp:
error (g, _("libvirt does not support the qemu curl driver
protocols (ftp, http, etc.); try setting LIBGUESTFS_BACKEND=direct"));
return -1;
+ case drive_protocol_spdk:
+ error (g, _("libvirt does not support SPDK driver protocol yet;
try setting LIBGUESTFS_BACKEND=direct"));
+ return -1;
}
if (construct_libvirt_xml_disk_target (g, xo, drv_index) == -1)
diff --git a/lib/qemu.c b/lib/qemu.c
index 887e31b..a77ea87 100644
--- a/lib/qemu.c
+++ b/lib/qemu.c
@@ -942,6 +942,8 @@ guestfs_int_drive_source_qemu_param (guestfs_h *g,
case drive_protocol_tftp:
return make_uri (g, "tftp", src->username, src->secret,
&src->servers[0], src->u.exportname);
+ case drive_protocol_spdk:
+ return safe_strdup(g, src->servers[0].u.socket);
}
abort ();
@@ -1016,6 +1018,7 @@ guestfs_int_discard_possible (guestfs_h *g, struct
drive *drv,
case drive_protocol_https:
case drive_protocol_ssh:
case drive_protocol_tftp:
+ case drive_protocol_spdk:
NOT_SUPPORTED (g, -1,
_("discard cannot be enabled on this drive: "
"protocol '%s' does not support discard"),
--
1.8.3.1
2 years, 10 months
Jenkins and other repos
by Luse, Paul E
Guys-
Is Jenkins picking up any of our other repos like spdk.github.io for webpage updates (posts, etc)? Doesn't look like it saw mine from this morning but the CH test pool did
Thx
Paul
2 years, 10 months
Git submodule dirty
by John Meneghini
Has anyone seen this problem before?
After updating the master branch I see that my dpdk submodule is dirty.
Fedora:vagrant:spdk(vagrant_hugepgs) > git status
On branch vagrant_hugepgs
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
(commit or discard the untracked or modified content in submodules)
modified: dpdk (modified content)
no changes added to commit (use "git add" and/or "git commit -a")
Fedora:vagrant:spdk(vagrant_hugepgs) > git diff
diff --git a/dpdk b/dpdk
--- a/dpdk
+++ b/dpdk
@@ -1 +1 @@
-Subproject commit b6ae5bcff6ca09a7e1536eaa449aa6f4e704a6d9
+Subproject commit b6ae5bcff6ca09a7e1536eaa449aa6f4e704a6d9-dirty
Fedora:vagrant:spdk(vagrant_hugepgs) > cd dpdk
Fedora:vagrant:dpdk((b6ae5bcff...)) > git status
HEAD detached at b6ae5bcff
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
typechange: app/test-bbdev/turbo_dec_default.data
typechange: app/test-bbdev/turbo_enc_default.data
More at:
https://stackoverflow.com/questions/4873980/git-diff-says-subproject-is-d...
John Meneghini
john.a.meneghini(a)gmail.com
2 years, 10 months
Proposal to drop test from CH test pool
by Luse, Paul E
All-
This came up in the community meeting this morning and was discussed. Wanted to throw it out for the maintainers to decide. There's coverage here already in Jenkins and with CH pool EOL'ing soon (month or so) seems like it makes more sense just to drop the test as opposed to adding more resources.
Thx
Paul
1) Chandler test pool runs lvol tests with vhost - test time per patch currently over 10 minutes - should break up lvol tests onto separate system in Chandler test pool?
a) Note: Jenkins runs lvol tests separately from vhost
2 years, 10 months
uio crash when hot-remove NVME SSD disk
by Vincent
Hello all,
We just test the hotplug function of SPDK,
The kernel will crash sometimes when we remove disk.
the kdump crash call stack is attached below.
The kernel version is "4.16.11-1.el7.elrepo.x86_64"
Does anyone can give us a hint of how to solve this problem ?
Thank you so much.
[ 3813.935443] BUG: unable to handle kernel NULL pointer dereference at
0000000000000173
[ 3813.935466] IP: 0x173
[ 3813.935472] PGD 8000001fe7558067 P4D 8000001fe7558067 PUD 1ff332d067 PMD
0
[ 3813.935490] Oops: 0010 [#1] SMP PTI
[ 3813.935496] Modules linked in: virtio_pci virtio_ring virtio
uio_pci_generic uio ipmi_si ipmi_devintf ipmi_msghandler sr_mod cdrom
joydev uas usb_storage fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE
nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4
nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun
ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi binfmt_misc sha512_ssse3
sha512_generic skx_edac x86_pkg_temp_thermal intel_powerclamp coretemp
kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
pcbc aesni_intel crypto_simd glue_helper cryptd intel_cstate qat_c62x
intel_rapl_perf intel_qat wdat_wdt pcspkr dh_generic input_leds cdc_acm
authenc sg mei_me lpc_ich mei ioatdma
[ 3813.935638] shpchp i2c_i801 mfd_core acpi_power_meter acpi_pad tpm_crb
nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod
crc32c_intel ast i40e drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops igb ttm ahci ptp libahci pps_core drm dca i2c_algo_bit libata
dm_mirror dm_region_hash dm_log dm_mod dax [last unloaded: ipmi_msghandler]
[ 3813.935725] CPU: 35 PID: 21251 Comm: reactor_35 Not tainted
4.16.11-1.el7.elrepo.x86_64 #1
[ 3813.935731] Hardware name: AIC HA202-PV/PAVO, BIOS PAVH_0.02.1 02/27/2018
[ 3813.935738] RIP: 0010:0x173
[ 3813.935745] RSP: 0018:ffffc900216d7e40 EFLAGS: 00010202
[ 3813.935753] RAX: ffff883fe34a7850 RBX: ffff881fd5331570 RCX:
0000000000000001
[ 3813.935760] RDX: 0000000000000173 RSI: ffff881ff4835de8 RDI:
ffff883fe34a7850
[ 3813.935766] RBP: ffffc900216d7e60 R08: 0000000000000000 R09:
0000000000000000
[ 3813.935773] R10: ffff881ff4835de8 R11: ffff881ff1c13310 R12:
ffff883ff462ed98
[ 3813.935779] R13: 0000000000000000 R14: ffff881ff499c320 R15:
ffff881fb97e9200
[ 3813.935788] FS: 00007efa9d5f9700(0000) GS:ffff883fff3c0000(0000)
knlGS:0000000000000000
[ 3813.935795] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3813.935802] CR2: 0000000000000173 CR3: 0000001fc27f8003 CR4:
00000000007606e0
[ 3813.935809] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 3813.935816] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 3813.935821] PKRU: 55555554
[ 3813.935826] Call Trace:
[ 3813.935843] ? uio_release+0x37/0x60 [uio]
[ 3813.935859] __fput+0xea/0x220
[ 3813.935871] ____fput+0xe/0x10
[ 3813.935881] task_work_run+0x8c/0xb0
[ 3813.935895] exit_to_usermode_loop+0x6b/0x95
[ 3813.935905] do_syscall_64+0x182/0x1b0
[ 3813.935920] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[ 3813.935927] RIP: 0033:0x7f02aa71c76d
[ 3813.935934] RSP: 002b:00007efa9d5f88b0 EFLAGS: 00000293 ORIG_RAX:
0000000000000003
[ 3813.935943] RAX: 0000000000000000 RBX: 00007efea2c2ec80 RCX:
00007f02aa71c76d
2 years, 10 months
[Release] 18.07: RAID, Infrastructure Improvements, Bug Fixes
by Walker, Benjamin
On behalf of the SPDK community I'm pleased to announce the release of SPDK
18.07! The changelog for this release is available at:
https://github.com/spdk/spdk/releases/tag/v18.07
This quarterly release contains 757 commits from 51 different authors - a new
record for contributors to a single release! We'd especially like to recognize
all of our first time contributors:
Aneesh Pachilangottil
Jakub Radtke
John Barnard
Kunal Sablok
Leonid Ravich
Mike Playle
Patrick Ohly
Sai Tallamraju
Senthil Kumar V
Shahar Salzman
Srikanth Kaligotla
Thanks to everyone for your contributions, participation, and effort!
Thanks,
Ben
2 years, 10 months
Re: [SPDK] Crypto pool sizes
by Harris, James R
(Note: GerritHub URL is https://review.gerrithub.io/#/c/spdk/spdk/+/403107/ for those that are interested.)
Hey Paul,
I think the pool sizes are fine. My question (and I think Ben’s question also) is about the arrays of mbuf pointers in the crypto_io_channel. Those arrays are used for individual operations so we shouldn’t need 40K mbuf pointers for a single operation.
Thanks,
-Jim
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Paul E Luse <paul.e.luse(a)intel.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Wednesday, August 1, 2018 at 6:28 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Crypto pool sizes
For those who have commented on why I had such huge arrays of pointers for crypto operations, thought it’d be easier to explain it in email rather than in the review. Feel free to ignore if you’re not paying attention to that patch series.
Below are the calculated numbers for each pool size (crypto ops, source mbufs, dest mbufs). Pretty easy to follow, because we use LBA as IV we break every IO in 512B crypto operations each needing one op from each pool (unless it’s a read then is doesn’t need a dest mbuf).
QD
IO size
Per Pool
3 pools
16
4096
128
384
32
4096
256
768
64
4096
512
1536
128
4096
1024
3072
256
4096
2048
6144
16
65536
2048
6144
32
65536
4096
12288
64
65536
8192
24576
128
65536
16384
49152
256
65536
32768
98304
16
1048576
32768
98304
32
1048576
65536
196608
64
1048576
131072
393216
128
1048576
262144
786432
256
1048576
524288
1572864
The API I’m using to submit requires passing in an array of these buffers (or operations) as a parameter. So there are 4 of these arrays of pointers in the channel structure, the memory is allocated from the heap:
struct rte_crypto_op *crypto_ops[NUM_MBUFS];
struct rte_mbuf *src_mbufs[NUM_MBUFS];
struct rte_mbuf *dst_mbufs[NUM_MBUFS];
struct rte_crypto_op *dequeued_ops[NUM_MBUFS];
There are two arrays of crypto operations, one for enqueue and one for de-queue. There’s a rule on the size of these pools too, they must be powers of 2. So, for example, in order to support 64K IO and 32 QD I need pool sizes of 4096 (see above) however that’s not sufficient because running something bdevperf (or anything really) will quickly get us to that Q depth plus there’s other IO from other modules like LVS that cause me to run out of mbufs unless I bump it up like another 8 which isn’t possible so the next option is 8192. Even at 8192 we have pretty limited QD (4) at 1MB IO.
Will discuss some more on IRC, what’s in the patch now is probably too high for normal use but am going to need to come up with something here that makes sense. At 8192 we’re only talking about 2MB of heap memory for ptr storage in these arrays but even then for large IO (1MB>) this probably isn’t acceptable.
2 years, 10 months
