Hello Charlie, Could you add more details about the performance drop? Are you using SPDK target and/or initiator and what is payload size? I would expect some degradation with small payloads, as written it this doc https://www.kernel.org/doc/html/v4.15/networking/msg_zerocopy.html#opport... : "MSG_ZEROCOPY is generally only effective at writes over around 10 KB. " Best regards, Alexey Marchuk -----Original Message----- From: Charlie Li <charlie.li(a)gmail.com&gt; Sent: Wednesday, January 20, 2021 12:41 AM To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; Subject: [SPDK] Performance impact with "enable_zerocopy_send" Hello, In SPDK 20.10, "enable_zerocopy_send" is set to "false" by default. I tried to set "enable_zerocopy_send" to true, but noticed the performance dropped. Is this as expected? My understanding is that setting "enable_zerocopy_send" to true should improve performance. Thanks, Charlie _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org

Charlie, Could you clarify where did you enable ZCOPY - on target side or on initiator side? If you enable it on target side then it is expected that Read (Controller to Host) has higher IOPS since in Read target writes data to a socket and ZCOPY takes effect on write operation. In Write (Host to Controller) target reads data from a socket and only writes responses back to the initiator. Best regards, Alexey Marchuk -----Original Message----- From: Charlie Li <charlie.li(a)gmail.com&gt; Sent: Wednesday, March 3, 2021 7:28 PM To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; Subject: [SPDK] Re: Performance impact with "enable_zerocopy_send" Hi Alexey, Looks like you are right that ZEROCOPY is not effective for smaller buffer sizes like 4KB. Here are the performance changes I got after enabling ZEROCOPY Random Read in 4KB: Dropped from 5042 KIOPS to 3080 KIOPS; -38.91% Random Write in 4KB: Dropped from 6301 KIOPS to 5899 KIOPS; -6.38% Sequential Read in 128KB: Increased from 158.3 KIOPS to 262 KIOPS; +65.51% Sequential Write in 128KB: Increased from 161.6 KIOPS to 167.5 KIOPS; +3.65% Why does XEROCOPY have a much bigger impact on Read than on Write? Thanks, Charlie On Mon, Feb 8, 2021 at 11:37 AM Alexey Marchuk <alexeymar(a)nvidia.com&gt; wrote: _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org