I tested SPDK's thread in this command:
sudo ./bdevperf -c bdev.conf -q 8 -o 512 -w write -t 1 -T Malloc1
It is a single thread and pass messages to itself. According to the VTune
result, *spdk_ring_enqueue* is the top hottest function: used about 20% of
CPU time. After adding the SP ring into thread's message queue, the
overhead of *spdk_ring_enqueue* drops to 4%, and the IOPS is improved from
5.67M to 6.88M.
I agree that an SLIST can provide better performance, but I still think
DPDK's SP ring is better choice. It provides good enough performance,
highly optimized, and easy to make the change.
Furthermore, not only for single thread applications, it can also optimize
performance in other scenarios. For example, suppose we have 3 threads A, B
and C. Thread A sends most of the messages to B, B to C, and C to A. The
added SP ring can send most messages much faster. Because we keep the MP
ring, the change can still work well with existed libraries and
On Wed, Oct 30, 2019 at 1:47 AM Walker, Benjamin <benjamin.walker(a)intel.com>
On Tue, 2019-10-29 at 04:42 +0000, cranechu(a)gmail.com wrote:
> Hi, All,
> In current SPDK thread, it has a single message ring, which is
> MP/SC. It does work for both SP and MP rings, but it also introduces high
> overhead even when the application uses SP/SC only. For the example of
> test bdevperf on one Malloc bdev, the cost of ring enqueue is as high as
> of total CPU time.
Are you using just a single SPDK thread? I believe the malloc bdev does a
of the data, then queues a message to itself to let the stack unwind to
the I/O. See _spdk_bdev_io_complete. I think this is primarily what's
your performance. I think a better optimization may be to specialize the
where the thread is passing a message to itself to not use a ring at all
replace it with an SLIST.
> My mitigation way is to add another SP/SC message ring into the thread.
> one thread receives its first message, it reserves this SP/SC message
> the first provider. Messages from other providers are still sent to the
> existed MP/SC message ring.
> In this way, applications who take SP/SC into design considerations can
> the benefits as expected. In the above example, we observe 15%
> Here is the code diff:
> Any comments are welcome.
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org