Resolved by adding pci whitelist into spdk_env_opts. Adding whitelist into nvme.conf was not sufficient. Regards, Tom Nabarro - DCG/ESAD M: +44 (0)7786 260986 Skype: tom.nabarro From: Nabarro, Tom Sent: Friday, January 31, 2020 12:42 PM To: 'Storage Performance Development Kit' <spdk(a)lists.01.org&gt; Subject: vfio multiple processes Can't run multiple SPDK processes each accessing disjoint set of NVMe drives using VFIO driver (even as root). Each process seems to try to access all /dev/vfio/* device files and get lots of resource busy messages. Sometimes one of the processes complaint it can't find the device it's been assigned in nvme.conf. Have any ideas? (processes are started in normal, non-multiprocess mode) Thanks in advance Best regards, Tom Nabarro BEng (hons) Extreme Storage Architecture & Development Intel Corporation E: tom.nabarro@intel.com<mailto:tom.nabarro@intel.com> M: +44 (0)7786 260986 Skype: tom.nabarro --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

Ah, that's a limitation I introduced together with DPDK 18.11 support. SPDK keeps vfio handles open because they may be required for a secondary process. There's room for improvement, but the current state is probably going to stay for a while. D. > -----Original Message----- > From: Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com&gt; > Sent: Monday, February 3, 2020 10:14 AM > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > Subject: [SPDK] Re: vfio multiple processes > > TBH I'm not sure the whitelist should be required. Sounds like a bug, I'll take a > look. > > D. > > > -----Original Message----- > > From: Nabarro, Tom <tom.nabarro(a)intel.com&gt; > > Sent: Saturday, February 1, 2020 3:00 PM > > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > > Subject: [SPDK] Re: vfio multiple processes > > > > Resolved by adding pci whitelist into spdk_env_opts. > > Adding whitelist into nvme.conf was not sufficient. > > > > Regards, > > Tom Nabarro - DCG/ESAD > > M: +44 (0)7786 260986 > > Skype: tom.nabarro > > > > From: Nabarro, Tom > > Sent: Friday, January 31, 2020 12:42 PM > > To: 'Storage Performance Development Kit' <spdk(a)lists.01.org&gt; > > Subject: vfio multiple processes > > > > Can't run multiple SPDK processes each accessing disjoint set of NVMe drives > > using VFIO driver (even as root). Each process seems to try to access all > > /dev/vfio/* device files and get lots of resource busy messages. Sometimes > > one of the processes complaint it can't find the device it's been assigned in > > nvme.conf. > > > > Have any ideas? > > (processes are started in normal, non-multiprocess mode) > > > > Thanks in advance > > > > Best regards, > > Tom Nabarro BEng (hons) > > Extreme Storage Architecture & Development > > Intel Corporation > > E: tom.nabarro@intel.com<mailto:tom.nabarro@intel.com> > > M: +44 (0)7786 260986 > > Skype: tom.nabarro > > > > --------------------------------------------------------------------- > > Intel Corporation (UK) Limited > > Registered No. 1134945 (England) > > Registered Office: Pipers Way, Swindon SN3 1RJ > > VAT No: 860 2173 47 > > > > This e-mail and any attachments may contain confidential material for > > the sole use of the intended recipient(s). Any review or distribution > > by others is strictly prohibited. If you are not the intended > > recipient, please contact the sender and delete all copies. > > _______________________________________________ > > SPDK mailing list -- spdk(a)lists.01.org > > To unsubscribe send an email to spdk-leave(a)lists.01.org > _______________________________________________ > SPDK mailing list -- spdk(a)lists.01.org > To unsubscribe send an email to spdk-leave(a)lists.01.org