1:42 p.m.
Hi, All,
In current SPDK thread, it has a single message ring, which is configured as MP/SC. It
does work for both SP and MP rings, but it also introduces high overhead even when the
application uses SP/SC only. For the example of the test bdevperf on one Malloc bdev, the
cost of ring enqueue is as high as 15% of total CPU time.
My mitigation way is to add another SP/SC message ring into the thread. When one thread
receives its first message, it reserves this SP/SC message ring to the first provider.
Messages from other providers are still sent to the existed MP/SC message ring.
In this way, applications who take SP/SC into design considerations can get the benefits
as expected. In the above example, we observe 15% performance improvement.
Here is the code diff:
https://github.com/cranechu/spdk/commit/2704ab4902b2b7743a9b269350a9e0f95...
Any comments are welcome.
Thanks,
-Crane
2:47 a.m.
On Tue, 2019-10-29 at 04:42 +0000, cranechu(a)gmail.com wrote:
Hi, All,
In current SPDK thread, it has a single message ring, which is configured as
MP/SC. It does work for both SP and MP rings, but it also introduces high
overhead even when the application uses SP/SC only. For the example of the
test bdevperf on one Malloc bdev, the cost of ring enqueue is as high as 15%
of total CPU time.
Are you using just a single SPDK thread? I believe the malloc bdev does a memcpy
of the data, then queues a message to itself to let the stack unwind to complete
the I/O. See _spdk_bdev_io_complete. I think this is primarily what's hurting
your performance. I think a better optimization may be to specialize the case
where the thread is passing a message to itself to not use a ring at all and
replace it with an SLIST.
My mitigation way is to add another SP/SC message ring into the thread. When
one thread receives its first message, it reserves this SP/SC message ring to
the first provider. Messages from other providers are still sent to the
existed MP/SC message ring.
In this way, applications who take SP/SC into design considerations can get
the benefits as expected. In the above example, we observe 15% performance
improvement.
Here is the code diff:
https://github.com/cranechu/spdk/commit/2704ab4902b2b7743a9b269350a9e0f95...
Any comments are welcome.
Thanks,
-Crane
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
3:54 p.m.
Hi, Benjamin,
I tested SPDK's thread in this command:
sudo ./bdevperf -c bdev.conf -q 8 -o 512 -w write -t 1 -T Malloc1
It is a single thread and pass messages to itself. According to the VTune
result, *spdk_ring_enqueue* is the top hottest function: used about 20% of
CPU time. After adding the SP ring into thread's message queue, the
overhead of *spdk_ring_enqueue* drops to 4%, and the IOPS is improved from
5.67M to 6.88M.
I agree that an SLIST can provide better performance, but I still think
DPDK's SP ring is better choice. It provides good enough performance,
highly optimized, and easy to make the change.
Furthermore, not only for single thread applications, it can also optimize
performance in other scenarios. For example, suppose we have 3 threads A, B
and C. Thread A sends most of the messages to B, B to C, and C to A. The
added SP ring can send most messages much faster. Because we keep the MP
ring, the change can still work well with existed libraries and
applications.
Thanks,
-Crane
On Wed, Oct 30, 2019 at 1:47 AM Walker, Benjamin <benjamin.walker(a)intel.com>
wrote:
On Tue, 2019-10-29 at 04:42 +0000, cranechu(a)gmail.com wrote:
> Hi, All,
>
> In current SPDK thread, it has a single message ring, which is
configured as
> MP/SC. It does work for both SP and MP rings, but it also introduces high
> overhead even when the application uses SP/SC only. For the example of
the
> test bdevperf on one Malloc bdev, the cost of ring enqueue is as high as
15%
> of total CPU time.
Are you using just a single SPDK thread? I believe the malloc bdev does a
memcpy
of the data, then queues a message to itself to let the stack unwind to
complete
the I/O. See _spdk_bdev_io_complete. I think this is primarily what's
hurting
your performance. I think a better optimization may be to specialize the
case
where the thread is passing a message to itself to not use a ring at all
and
replace it with an SLIST.
>
> My mitigation way is to add another SP/SC message ring into the thread.
When
> one thread receives its first message, it reserves this SP/SC message
ring to
> the first provider. Messages from other providers are still sent to the
> existed MP/SC message ring.
>
> In this way, applications who take SP/SC into design considerations can
get
> the benefits as expected. In the above example, we observe 15%
performance
> improvement.
>
> Here is the code diff:
>
https://github.com/cranechu/spdk/commit/2704ab4902b2b7743a9b269350a9e0f95...
> Any comments are welcome.
>
> Thanks,
> -Crane
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
12:37 a.m.
On 10/31/19, 11:55 PM, "Crane Chu" <cranechu(a)gmail.com> wrote:
Hi, Benjamin,
I tested SPDK's thread in this command:
sudo ./bdevperf -c bdev.conf -q 8 -o 512 -w write -t 1 -T Malloc1
It is a single thread and pass messages to itself. According to the VTune
result, *spdk_ring_enqueue* is the top hottest function: used about 20% of
CPU time. After adding the SP ring into thread's message queue, the
overhead of *spdk_ring_enqueue* drops to 4%, and the IOPS is improved from
5.67M to 6.88M.
I agree that an SLIST can provide better performance, but I still think
DPDK's SP ring is better choice. It provides good enough performance,
highly optimized, and easy to make the change.
Whether we use an SLIST or a SP ring to optimize messages sent by a thread to itself, we
will need to think about the order in which self-sent messages are processed in relation
to messages from other threads. I think we likely want to prioritize messages from other
threads. That's the nice thing about using a MP ring - we don't need to worry
about the fairness aspect.
It's also important to note that this test case (bdevperf w/ malloc backend using
memcpy) is the worst case scenario. In almost all other cases - even malloc backend with
CB-DMA offload - completions are all processed inline without using spdk_thread_send_msg.
I'm not saying we shouldn't look at optimizations for this case, but it won't
provide any meaningful efficiency improvements for the vast majority of workloads.
-Jim
484
days inactive
487
days old
3 comments
4 participants
participants (4)
-
Crane Chu -
cranechu＠gmail.com
-
Harris, James R -
Walker, Benjamin