From: SPDK <spdk-bounces@lists.01.org> on behalf of Lance Hartmann ORACLE <lance.hartmann@oracle.com>
Reply-To: Storage Performance Development Kit <spdk@lists.01.org>
Date: Tuesday, June 26, 2018 at 9:56 AM
To: Storage Performance Development Kit <spdk@lists.01.org>
Subject: [SPDK] Best practices on driver binding for SPDK in production environments

 

 

<snip>

 

4.   With my new udev rules in place, I was successful getting specific NVMe controllers (based on bus-device-function) to unbind from the Linux nvme driver and bind to vfio-pci.   However, I made a couple of observations in the kernel log (dmesg).   In particular, I was drawn to the following for an NVMe controller at BDF:  0000:40:00.0 for which I had a udev rule to unbind from nvme and bind to vfio-pci:

 

[   35.534279] nvme nvme1: pci function 0000:40:00.0

[   37.964945] nvme nvme1: failed to mark controller live

[   37.964947] nvme nvme1: Removing after probe failure status: 0

 

One theory I have for the above is that my udev RUN rule was invoked while the nvme driver’s probe() was still running on this controller, and perhaps the unbind request came in before the probe() completed hence this “name1: failed to mark controller live”.   This has left lingering in my mind that maybe instead of triggering on (Event 2) when the bind occurs, that perhaps I should instead try to derive a trigger on the “last" udev event, an “add”, where the NVMe namespace’s are instantiated.   Of course, I’d need to know ahead of time just how many namespaces exist on that controller if I were to do that so I’d trigger on the last one.   I’m wondering if that may help to avoid what looks like a complaint during the middle of probe() of that particular controller.   Then, again, maybe I can just safely ignore that and not worry about it at all?    Thoughts?

 

[Jim]  Can you confirm your suspicion - maybe add a 1 or 2 second delay after detecting Event 2 before unbinding – and see if that eliminates the probe failures?  I’m not suggesting that as a workaround or solution – just want to know for sure if we need to worry about deferring the unbind until after the kernel driver’s probe has completed.  It sounds like these error messages are benign but would be nice to avoid them.

 

Overall this seems like a reasonable approach though.  How do you see this working if a system has multiple NVMe SSDs – one of which has the OS install, and the rest should be assigned to uio/vfio?

 

I discovered another issue during this experimentation that is somewhat tangential to this task, but I’ll write a separate email on that topic.

 

thanks for any feedback,

--
Lance Hartmann
lance.hartmann@oracle.com