Hi All, Is there any benchmarking done to prove that SPDK is better than the Kernel driver? We did run the sample driver provided from the SPDK and compared with the kernel interface and found that SPDK is very slow. We did run FIO.

Can you describe the setup for the performance measurement between Kernel and SPDK driver. Regards, Param. _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk

The setup is the default setup of NVMf target with Direct attached Subsystem. The FIO plugin was at the target side and the IOPS from SPDK was found to be around 2.5K and the IOPS from kernel was found to be 100k.

Regards, Param. On 06-Dec-2016, at 5:57 PM, Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com&gt; wrote: On Tue, Dec 6, 2016, 15:24 Kumaraparameshwaran Rathnavel < krath(a)cloudsimple.com&gt; wrote: Hi All, Is there any benchmarking done to prove that SPDK is better than the Kernel driver? We did run the sample driver provided from the SPDK and compared with the kernel interface and found that SPDK is very slow. We did run FIO. It's worth publishing your setup and results first. Regards, Andrey Can you describe the setup for the performance measurement between Kernel and SPDK driver. Regards, Param. _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk -- Regards, Andrey _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk

Can you briefly describe your setup if you don’t mind. It will be of great help. Regards, Param. On 06-Dec-2016, at 6:24 PM, Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com&gt; wrote: On Tue, Dec 6, 2016, 15:49 Kumaraparameshwaran Rathnavel < krath(a)cloudsimple.com&gt; wrote: > The setup is the default setup of NVMf target with Direct attached > Subsystem. The FIO plugin was at the target side and the IOPS from SPDK was > found to be around 2.5K and the IOPS from kernel was found to be 100k. > Sounds like a setup issue to me. In my tests, SPDK typically matched or surpassed kernel stack in IOPS while using 2-3x less CPU. Regards, Andrey > > Regards, > Param. > > On 06-Dec-2016, at 5:57 PM, Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com&gt; > wrote: > > > > On Tue, Dec 6, 2016, 15:24 Kumaraparameshwaran Rathnavel < > krath(a)cloudsimple.com&gt; wrote: > > Hi All, > > Is there any benchmarking done to prove that SPDK is better than the > Kernel driver? > > We did run the sample driver provided from the SPDK and compared with the > kernel interface and found that SPDK is very slow. We did run FIO. > > > It's worth publishing your setup and results first. > > Regards, > Andrey > > > Can you describe the setup for the performance measurement between Kernel > and SPDK driver. > > Regards, > Param. > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > > -- > > Regards, > Andrey > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > > > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > -- Regards, Andrey _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk

What was the Reactor mask and did you assign different cores ? What was the queues per connection number in the target side? I had all as default values. Regards, Param. On 06-Dec-2016, at 6:35 PM, Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com&gt; wrote: Same as yours, essentially, although run about a year ago. On Dec 6, 2016 15:57, "Kumaraparameshwaran Rathnavel" < krath(a)cloudsimple.com&gt; wrote: > Can you briefly describe your setup if you don’t mind. > > It will be of great help. > > Regards, > Param. > > On 06-Dec-2016, at 6:24 PM, Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com&gt; > wrote: > > > > On Tue, Dec 6, 2016, 15:49 Kumaraparameshwaran Rathnavel < > krath(a)cloudsimple.com&gt; wrote: > >> The setup is the default setup of NVMf target with Direct attached >> Subsystem. The FIO plugin was at the target side and the IOPS from SPDK was >> found to be around 2.5K and the IOPS from kernel was found to be 100k. >> > > Sounds like a setup issue to me. In my tests, SPDK typically matched or > surpassed kernel stack in IOPS while using 2-3x less CPU. > > Regards, > Andrey > >> >> Regards, >> Param. >> >> On 06-Dec-2016, at 5:57 PM, Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com&gt; >> wrote: >> >> >> >> On Tue, Dec 6, 2016, 15:24 Kumaraparameshwaran Rathnavel < >> krath(a)cloudsimple.com&gt; wrote: >> >> Hi All, >> >> Is there any benchmarking done to prove that SPDK is better than the >> Kernel driver? >> >> We did run the sample driver provided from the SPDK and compared with >> the kernel interface and found that SPDK is very slow. We did run FIO. >> >> >> It's worth publishing your setup and results first. >> >> Regards, >> Andrey >> >> >> Can you describe the setup for the performance measurement between >> Kernel and SPDK driver. >> >> Regards, >> Param. >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> >> -- >> >> Regards, >> Andrey >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> >> >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> > -- > > Regards, > Andrey > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > > > > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > > _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk

> On Dec 6, 2016, at 5:49 AM, Kumaraparameshwaran Rathnavel <krath@cloudsimple > .com> wrote: > > The setup is the default setup of NVMf target with Direct attached > Subsystem. The FIO plugin was at the target side and the IOPS from SPDK was > found to be around 2.5K and the IOPS from kernel was found to be 100k. Hi Param, Can you check the AcceptorPollRate setting in your configuration file?  There was a recent fix to the default value for this parameter.  You should set it to at least 10000 (10ms). https://github.com/spdk/spdk/commit/5f3761cf7ec50cf03183203dc50657a9f2a18908 Thanks,

-Jim > > Regards, > Param. > > On 06-Dec-2016, at 5:57 PM, Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com&gt; > > wrote: > > > > > > > > On Tue, Dec 6, 2016, 15:24 Kumaraparameshwaran Rathnavel <krath@cloudsimpl > > e.com> wrote: > > > Hi All, > > > > > > Is there any benchmarking done to prove that SPDK is better than the > > > Kernel driver? > > > > > > We did run the sample driver provided from the SPDK and compared with > > > the kernel interface and found that SPDK is very slow. We did run FIO. > > > > > > > It's worth publishing your setup and results first. > > > > Regards, > > Andrey > > > > > > Can you describe the setup for the performance measurement between > > > Kernel and SPDK driver. > > > > > > Regards, > > > Param. > > > _______________________________________________ > > > SPDK mailing list > > > SPDK(a)lists.01.org > > > https://lists.01.org/mailman/listinfo/spdk > > > > > --  > > Regards, > > Andrey > > _______________________________________________ > > SPDK mailing list > > SPDK(a)lists.01.org > > https://lists.01.org/mailman/listinfo/spdk > > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk

On Tue, 2016-12-06 at 16:35 +0000, Harris, James R wrote: > > > On Dec 6, 2016, at 5:49 AM, Kumaraparameshwaran Rathnavel <krath@ > > cloudsimple > > .com> wrote: > > > > The setup is the default setup of NVMf target with Direct > > attached > > Subsystem. The FIO plugin was at the target side and the IOPS > > from SPDK was > > found to be around 2.5K and the IOPS from kernel was found to be > > 100k. > > > Hi Param, > > Can you check the AcceptorPollRate setting in your configuration > file?  There > was a recent fix to the default value for this parameter.  You > should set it > to at least 10000 (10ms). > > https://github.com/spdk/spdk/commit/5f3761cf7ec50cf03183203dc50657a > 9f2a18908 > > Thanks, Just to reinforce here - this is the solution. We saw this internally a few weeks ago. Increasing the AcceptorPollRate to at least 10ms (you can go as much as 1 second if you want, but the performance benefit decreases the larger you make it), or moving the accceptor to a different core using the AcceptorCore option will fix the problem.