Hi Andrey, Will look at this, but for the write operation. There may be the following reasons: 1 For the write operation. If the data is not capsulated in the command buffer (In our NVMe-oF initiator for the data write on I/O qpair, we will not use that), the NVMe-oF target will send the R2T PDU to the host. So compared with read operation, there will be additional PDU exchange. 2 There are restrictions for the active R2T pdus defined in the spec. Currently, we use this Macro in the nvme_tcp.c (#define NVME_TCP_MAX_R2T_DEFAULT 16). When the host sends the ic_req pdu, it will tell the target that there will be at most so many active R2T pdus. So if you want to improve the performance, you need to increase this value. Currently it is not configurable, you may need to update it with a larger value. Thanks. Best Regards Ziye Yang -----Original Message----- From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Andrey Kuzmin Sent: Friday, November 30, 2018 1:03 AM To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; Subject: [SPDK] Null bdev read/write performance with NVMf/TCP target I'm getting some rather counter-intuitive results in the first hand-on with the recently announced SPDK NVMoF/TCP target. The only difference between the two runs below (over a 10GigE link) is that the first one is doing sequential read, while the second one - sequential write, and notice that both run against the same null bdev target. Any ideas as to why null bdev writes are 2x slower than reads and where that extra 1ms of latency is coming from? Regards, Andrey sudo ./examples/nvme/perf/perf -c 0x1 -q 256 -o 4096 -w read -t 10 -r "trtype:TCP adrfam:IPv4 traddr:192.168.100.105 trsvcid:4420 subnqn:nqn.test_tgt" Starting SPDK v19.01-pre / DPDK 18.08.0 initialization... [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci --base-virtaddr=0x200000000000 --file-prefix=spdk_pid3870 ] EAL: Detected 56 lcore(s) EAL: Detected 2 NUMA nodes EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... Initializing NVMe Controllers Attaching to NVMe over Fabrics controller at 192.168.100.105:4420: nqn.test_tgt Attached to NVMe over Fabrics controller at 192.168.100.105:4420: nqn.test_tgt Associating SPDK bdev Controller (TESTTGTCTL) with lcore 0 Initialization complete. Launching workers. Starting thread on core 0 ======================================================== Latency(us) Device Information : IOPS MB/s Average min max SPDK bdev Controller (TESTTGTCTL ) from core 0: 203888.90 796.44 1255.62 720.37 1967.79 ======================================================== Total : 203888.90 796.44 1255.62 720.37 1967.79 sudo ./examples/nvme/perf/perf -c 0x1 -q 256 -o 4096 -w write -t 10 -r "trtype:TCP adrfam:IPv4 traddr:192.168.100.105 trsvcid:4420 subnqn:nqn.test_tgt" Starting SPDK v19.01-pre / DPDK 18.08.0 initialization... [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci --base-virtaddr=0x200000000000 --file-prefix=spdk_pid3873 ] EAL: Detected 56 lcore(s) EAL: Detected 2 NUMA nodes EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... Initializing NVMe Controllers Attaching to NVMe over Fabrics controller at 192.168.100.105:4420: nqn.test_tgt Attached to NVMe over Fabrics controller at 192.168.100.105:4420: nqn.test_tgt Associating SPDK bdev Controller (TESTTGTCTL) with lcore 0 Initialization complete. Launching workers. Starting thread on core 0 ======================================================== Latency(us) Device Information : IOPS MB/s Average min max SPDK bdev Controller (TESTTGTCTL ) from core 0: 92426.00 361.04 2770.15 773.45 5265.16 ======================================================== Total : 92426.00 361.04 2770.15 773.45 5265.16 _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk

Hi Andrey, For (1) Actually, we have that feature, but the SPDK initiator currently does not use it. Usually, the capsule data size is reported by the target to the initiator. For large I/O size, R2T will still be needed.

patch): https://review.gerrithub.io/#/c/spdk/spdk/+/435398 To see whether the sequential write for 4KB or 8KB will be improved.

Best Regards Ziye Yang -----Original Message----- From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Andrey Kuzmin Sent: Friday, November 30, 2018 3:18 PM To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; Subject: Re: [SPDK] Null bdev read/write performance with NVMf/TCP target Thankd Ziye, I'll give a shot to expanding the max R2T PDU and get back to you in this. Re (1) below, are there any plans to support immediate data in the nvmf initiator in the future, to avoid the extra R2T exchange (similar to iSCSI)? Thanks, Andrey On Fri, Nov 30, 2018, 04:28 Yang, Ziye <ziye.yang(a)intel.com&gt; wrote: > Hi Andrey, > > Will look at this, but for the write operation. There may be the > following > reasons: > > 1 For the write operation. If the data is not capsulated in the > command buffer (In our NVMe-oF initiator for the data write on I/O > qpair, we will not use that), the NVMe-oF target will send the R2T PDU > to the host. So compared with read operation, there will be additional PDU exchange. > 2 There are restrictions for the active R2T pdus defined in the spec. > Currently, we use this Macro in the nvme_tcp.c > (#define NVME_TCP_MAX_R2T_DEFAULT 16). > > When the host sends the ic_req pdu, it will tell the target that there > will be at most so many active R2T pdus. So if you want to improve the > performance, you need to increase this value. Currently it is not > configurable, you may need to update it with a larger value. > > Thanks. > > > > > > > > Best Regards > Ziye Yang > > -----Original Message----- > From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Andrey > Kuzmin > Sent: Friday, November 30, 2018 1:03 AM > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > Subject: [SPDK] Null bdev read/write performance with NVMf/TCP target > > I'm getting some rather counter-intuitive results in the first hand-on > with the recently announced SPDK NVMoF/TCP target. The only difference > between the two runs below (over a 10GigE link) is that the first one > is doing sequential read, while the second one - sequential write, and > notice that both run against the same null bdev target. Any ideas as > to why null bdev writes are 2x slower than reads and where that extra > 1ms of latency is coming from? > > Regards, > Andrey > > sudo ./examples/nvme/perf/perf -c 0x1 -q 256 -o 4096 -w read -t 10 -r > "trtype:TCP adrfam:IPv4 traddr:192.168.100.105 trsvcid:4420 > subnqn:nqn.test_tgt" > Starting SPDK v19.01-pre / DPDK 18.08.0 initialization... > [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci > --base-virtaddr=0x200000000000 --file-prefix=spdk_pid3870 ] > EAL: Detected 56 lcore(s) > EAL: Detected 2 NUMA nodes > EAL: No free hugepages reported in hugepages-1048576kB > EAL: Probing VFIO support... > Initializing NVMe Controllers > Attaching to NVMe over Fabrics controller at 192.168.100.105:4420: > nqn.test_tgt > Attached to NVMe over Fabrics controller at 192.168.100.105:4420: > nqn.test_tgt > Associating SPDK bdev Controller (TESTTGTCTL) with lcore 0 > Initialization complete. Launching workers. > Starting thread on core 0 > ======================================================== > > Latency(us) > Device Information : IOPS > MB/s Average min max > SPDK bdev Controller (TESTTGTCTL ) from core 0: 203888.90 > 796.44 1255.62 720.37 1967.79 > ======================================================== > Total : 203888.90 > 796.44 1255.62 720.37 1967.79 > > sudo ./examples/nvme/perf/perf -c 0x1 -q 256 -o 4096 -w write -t 10 -r > "trtype:TCP adrfam:IPv4 traddr:192.168.100.105 trsvcid:4420 > subnqn:nqn.test_tgt" > Starting SPDK v19.01-pre / DPDK 18.08.0 initialization... > [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci > --base-virtaddr=0x200000000000 --file-prefix=spdk_pid3873 ] > EAL: Detected 56 lcore(s) > EAL: Detected 2 NUMA nodes > EAL: No free hugepages reported in hugepages-1048576kB > EAL: Probing VFIO support... > Initializing NVMe Controllers > Attaching to NVMe over Fabrics controller at 192.168.100.105:4420: > nqn.test_tgt > Attached to NVMe over Fabrics controller at 192.168.100.105:4420: > nqn.test_tgt > Associating SPDK bdev Controller (TESTTGTCTL) with lcore 0 > Initialization complete. Launching workers. > Starting thread on core 0 > ======================================================== > > Latency(us) > Device Information : IOPS > MB/s Average min max > SPDK bdev Controller (TESTTGTCTL ) from core 0: 92426.00 > 361.04 2770.15 773.45 5265.16 > ======================================================== > Total : 92426.00 > 361.04 2770.15 773.45 5265.16 > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > -- Regards, Andrey _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk

Ziye, I tried out both of your suggestions. Increasing MX_R2T brought home substantial improvement (~50%), while allowing in-capsule data - not so much, just an extra 10% surprisingly. With both fixes, write throughput in the runs I reported earlier increased to some 620MB/s, that is close to 80% of the read throughput (latency P50 now stands at 1.33 for reads ms vs. 1.57ms for writes). Also, while playing with 8K transfers, I noticed that MAX_IN_CAPSULE_DATA_SIZE is defined in both nvmf/tcp.c and nvme/nvme_tcp.c, and they do not agree :) (4K vs 8K). Worth a fix introducing a single definition, I believe. Regards, Andrey On Fri, Nov 30, 2018 at 2:33 PM Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com&gt; wrote: > > > On Fri, Nov 30, 2018, 10:27 Yang, Ziye <ziye.yang(a)intel.com&gt; wrote: > >> Hi Andrey, >> >> For (1) Actually, we have that feature, but the SPDK initiator >> currently does not use it. Usually, the capsule data size is reported by >> the target to the initiator. For large I/O size, R2T will still be needed. > > > I guess, for larger io sizes, r2t could be piggy-backed on a tcp ack, so > to say. Looks like quite natural optimization. > > If you want to, you can use the following branch to test (Not a formal >> patch): >> https://review.gerrithub.io/#/c/spdk/spdk/+/435398 >> >> To see whether the sequential write for 4KB or 8KB will be improved. >> > Thanks, will do. > > Regards, > Andrey > >> >> >> >> >> Best Regards >> Ziye Yang >> >> >> -----Original Message----- >> From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Andrey Kuzmin >> Sent: Friday, November 30, 2018 3:18 PM >> To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; >> Subject: Re: [SPDK] Null bdev read/write performance with NVMf/TCP target >> >> Thankd Ziye, I'll give a shot to expanding the max R2T PDU and get back >> to you in this. >> >> Re (1) below, are there any plans to support immediate data in the nvmf >> initiator in the future, to avoid the extra R2T exchange (similar to iSCSI)? >> >> Thanks, >> Andrey >> >> On Fri, Nov 30, 2018, 04:28 Yang, Ziye <ziye.yang(a)intel.com&gt; wrote: >> >> > Hi Andrey, >> > >> > Will look at this, but for the write operation. There may be the >> > following >> > reasons: >> > >> > 1 For the write operation. If the data is not capsulated in the >> > command buffer (In our NVMe-oF initiator for the data write on I/O >> > qpair, we will not use that), the NVMe-oF target will send the R2T PDU >> > to the host. So compared with read operation, there will be additional >> PDU exchange. >> > 2 There are restrictions for the active R2T pdus defined in the spec. >> > Currently, we use this Macro in the nvme_tcp.c >> > (#define NVME_TCP_MAX_R2T_DEFAULT 16). >> > >> > When the host sends the ic_req pdu, it will tell the target that there >> > will be at most so many active R2T pdus. So if you want to improve the >> > performance, you need to increase this value. Currently it is not >> > configurable, you may need to update it with a larger value. >> > >> > Thanks. >> > >> > >> > >> > >> > >> > >> > >> > Best Regards >> > Ziye Yang >> > >> > -----Original Message----- >> > From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Andrey >> > Kuzmin >> > Sent: Friday, November 30, 2018 1:03 AM >> > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; >> > Subject: [SPDK] Null bdev read/write performance with NVMf/TCP target >> > >> > I'm getting some rather counter-intuitive results in the first hand-on >> > with the recently announced SPDK NVMoF/TCP target. The only difference >> > between the two runs below (over a 10GigE link) is that the first one >> > is doing sequential read, while the second one - sequential write, and >> > notice that both run against the same null bdev target. Any ideas as >> > to why null bdev writes are 2x slower than reads and where that extra >> > 1ms of latency is coming from? >> > >> > Regards, >> > Andrey >> > >> > sudo ./examples/nvme/perf/perf -c 0x1 -q 256 -o 4096 -w read -t 10 -r >> > "trtype:TCP adrfam:IPv4 traddr:192.168.100.105 trsvcid:4420 >> > subnqn:nqn.test_tgt" >> > Starting SPDK v19.01-pre / DPDK 18.08.0 initialization... >> > [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci >> > --base-virtaddr=0x200000000000 --file-prefix=spdk_pid3870 ] >> > EAL: Detected 56 lcore(s) >> > EAL: Detected 2 NUMA nodes >> > EAL: No free hugepages reported in hugepages-1048576kB >> > EAL: Probing VFIO support... >> > Initializing NVMe Controllers >> > Attaching to NVMe over Fabrics controller at 192.168.100.105:4420: >> > nqn.test_tgt >> > Attached to NVMe over Fabrics controller at 192.168.100.105:4420: >> > nqn.test_tgt >> > Associating SPDK bdev Controller (TESTTGTCTL) with lcore 0 >> > Initialization complete. Launching workers. >> > Starting thread on core 0 >> > ======================================================== >> > >> > Latency(us) >> > Device Information : IOPS >> > MB/s Average min max >> > SPDK bdev Controller (TESTTGTCTL ) from core 0: 203888.90 >> > 796.44 1255.62 720.37 1967.79 >> > ======================================================== >> > Total : 203888.90 >> > 796.44 1255.62 720.37 1967.79 >> > >> > sudo ./examples/nvme/perf/perf -c 0x1 -q 256 -o 4096 -w write -t 10 -r >> > "trtype:TCP adrfam:IPv4 traddr:192.168.100.105 trsvcid:4420 >> > subnqn:nqn.test_tgt" >> > Starting SPDK v19.01-pre / DPDK 18.08.0 initialization... >> > [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci >> > --base-virtaddr=0x200000000000 --file-prefix=spdk_pid3873 ] >> > EAL: Detected 56 lcore(s) >> > EAL: Detected 2 NUMA nodes >> > EAL: No free hugepages reported in hugepages-1048576kB >> > EAL: Probing VFIO support... >> > Initializing NVMe Controllers >> > Attaching to NVMe over Fabrics controller at 192.168.100.105:4420: >> > nqn.test_tgt >> > Attached to NVMe over Fabrics controller at 192.168.100.105:4420: >> > nqn.test_tgt >> > Associating SPDK bdev Controller (TESTTGTCTL) with lcore 0 >> > Initialization complete. Launching workers. >> > Starting thread on core 0 >> > ======================================================== >> > >> > Latency(us) >> > Device Information : IOPS >> > MB/s Average min max >> > SPDK bdev Controller (TESTTGTCTL ) from core 0: 92426.00 >> > 361.04 2770.15 773.45 5265.16 >> > ======================================================== >> > Total : 92426.00 >> > 361.04 2770.15 773.45 5265.16 >> > _______________________________________________ >> > SPDK mailing list >> > SPDK(a)lists.01.org >> > https://lists.01.org/mailman/listinfo/spdk >> > _______________________________________________ >> > SPDK mailing list >> > SPDK(a)lists.01.org >> > https://lists.01.org/mailman/listinfo/spdk >> > >> -- >> >> Regards, >> Andrey >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> > -- > > Regards, > Andrey >

Hi Andrey, You can support this formal patch, it will support the incapsuleDataSize better. https://review.gerrithub.io/#/c/spdk/spdk/+/435826/ BTW, you need to configure InCapsuleDataSize in the Transport section in your target configuration file or use RPC. Thanks. Best Regards Ziye Yang -----Original Message----- From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Andrey Kuzmin Sent: Saturday, December 1, 2018 12:29 AM To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; Subject: Re: [SPDK] Null bdev read/write performance with NVMf/TCP target Also FYI, 50/50 rw mix outperforms 100% read workload by more than 20% (950 MB/s vs. 750), also something not exactly intuitive. Nice first impression, nonetheless, thanks for the great job. Regards, Andrey On Fri, Nov 30, 2018 at 6:47 PM Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com&gt; wrote: > Ziye, > > I tried out both of your suggestions. Increasing MX_R2T brought home > substantial improvement (~50%), while allowing in-capsule data - not > so much, just an extra 10% surprisingly. With both fixes, write > throughput in the runs I reported earlier increased to some 620MB/s, > that is close to 80% of the read throughput (latency P50 now stands at 1.33 for reads ms vs. > 1.57ms for writes). > > Also, while playing with 8K transfers, I noticed that > MAX_IN_CAPSULE_DATA_SIZE is defined in both nvmf/tcp.c and > nvme/nvme_tcp.c, and they do not agree :) (4K vs 8K). Worth a fix > introducing a single definition, I believe. > > Regards, > Andrey > > > On Fri, Nov 30, 2018 at 2:33 PM Andrey Kuzmin > <andrey.v.kuzmin(a)gmail.com&gt; > wrote: > >> >> >> On Fri, Nov 30, 2018, 10:27 Yang, Ziye <ziye.yang(a)intel.com&gt; wrote: >> >>> Hi Andrey, >>> >>> For (1) Actually, we have that feature, but the SPDK initiator >>> currently does not use it. Usually, the capsule data size is >>> reported by the target to the initiator. For large I/O size, R2T will still be needed. >> >> >> I guess, for larger io sizes, r2t could be piggy-backed on a tcp ack, >> so to say. Looks like quite natural optimization. >> >> If you want to, you can use the following branch to test (Not a >> formal >>> patch): >>> https://review.gerrithub.io/#/c/spdk/spdk/+/435398 >>> >>> To see whether the sequential write for 4KB or 8KB will be improved. >>> >> Thanks, will do. >> >> Regards, >> Andrey >> >>> >>> >>> >>> >>> Best Regards >>> Ziye Yang >>> >>> >>> -----Original Message----- >>> From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Andrey >>> Kuzmin >>> Sent: Friday, November 30, 2018 3:18 PM >>> To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; >>> Subject: Re: [SPDK] Null bdev read/write performance with NVMf/TCP >>> target >>> >>> Thankd Ziye, I'll give a shot to expanding the max R2T PDU and get >>> back to you in this. >>> >>> Re (1) below, are there any plans to support immediate data in the >>> nvmf initiator in the future, to avoid the extra R2T exchange (similar to iSCSI)? >>> >>> Thanks, >>> Andrey >>> >>> On Fri, Nov 30, 2018, 04:28 Yang, Ziye <ziye.yang(a)intel.com&gt; wrote: >>> >>> > Hi Andrey, >>> > >>> > Will look at this, but for the write operation. There may be the >>> > following >>> > reasons: >>> > >>> > 1 For the write operation. If the data is not capsulated in the >>> > command buffer (In our NVMe-oF initiator for the data write on I/O >>> > qpair, we will not use that), the NVMe-oF target will send the R2T >>> > PDU to the host. So compared with read operation, there will be >>> > additional >>> PDU exchange. >>> > 2 There are restrictions for the active R2T pdus defined in the spec. >>> > Currently, we use this Macro in the nvme_tcp.c >>> > (#define NVME_TCP_MAX_R2T_DEFAULT 16). >>> > >>> > When the host sends the ic_req pdu, it will tell the target that >>> > there will be at most so many active R2T pdus. So if you want to >>> > improve the performance, you need to increase this value. >>> > Currently it is not configurable, you may need to update it with a larger value. >>> > >>> > Thanks. >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > Best Regards >>> > Ziye Yang >>> > >>> > -----Original Message----- >>> > From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Andrey >>> > Kuzmin >>> > Sent: Friday, November 30, 2018 1:03 AM >>> > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; >>> > Subject: [SPDK] Null bdev read/write performance with NVMf/TCP >>> > target >>> > >>> > I'm getting some rather counter-intuitive results in the first >>> > hand-on with the recently announced SPDK NVMoF/TCP target. The >>> > only difference between the two runs below (over a 10GigE link) is >>> > that the first one is doing sequential read, while the second one >>> > - sequential write, and notice that both run against the same null >>> > bdev target. Any ideas as to why null bdev writes are 2x slower >>> > than reads and where that extra 1ms of latency is coming from? >>> > >>> > Regards, >>> > Andrey >>> > >>> > sudo ./examples/nvme/perf/perf -c 0x1 -q 256 -o 4096 -w read -t 10 >>> > -r "trtype:TCP adrfam:IPv4 traddr:192.168.100.105 trsvcid:4420 >>> > subnqn:nqn.test_tgt" >>> > Starting SPDK v19.01-pre / DPDK 18.08.0 initialization... >>> > [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci >>> > --base-virtaddr=0x200000000000 --file-prefix=spdk_pid3870 ] >>> > EAL: Detected 56 lcore(s) >>> > EAL: Detected 2 NUMA nodes >>> > EAL: No free hugepages reported in hugepages-1048576kB >>> > EAL: Probing VFIO support... >>> > Initializing NVMe Controllers >>> > Attaching to NVMe over Fabrics controller at 192.168.100.105:4420: >>> > nqn.test_tgt >>> > Attached to NVMe over Fabrics controller at 192.168.100.105:4420: >>> > nqn.test_tgt >>> > Associating SPDK bdev Controller (TESTTGTCTL) with lcore 0 >>> > Initialization complete. Launching workers. >>> > Starting thread on core 0 >>> > ======================================================== >>> > >>> > Latency(us) >>> > Device Information : IOPS >>> > MB/s Average min max >>> > SPDK bdev Controller (TESTTGTCTL ) from core 0: 203888.90 >>> > 796.44 1255.62 720.37 1967.79 >>> > ======================================================== >>> > Total : 203888.90 >>> > 796.44 1255.62 720.37 1967.79 >>> > >>> > sudo ./examples/nvme/perf/perf -c 0x1 -q 256 -o 4096 -w write -t >>> > 10 -r "trtype:TCP adrfam:IPv4 traddr:192.168.100.105 trsvcid:4420 >>> > subnqn:nqn.test_tgt" >>> > Starting SPDK v19.01-pre / DPDK 18.08.0 initialization... >>> > [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci >>> > --base-virtaddr=0x200000000000 --file-prefix=spdk_pid3873 ] >>> > EAL: Detected 56 lcore(s) >>> > EAL: Detected 2 NUMA nodes >>> > EAL: No free hugepages reported in hugepages-1048576kB >>> > EAL: Probing VFIO support... >>> > Initializing NVMe Controllers >>> > Attaching to NVMe over Fabrics controller at 192.168.100.105:4420: >>> > nqn.test_tgt >>> > Attached to NVMe over Fabrics controller at 192.168.100.105:4420: >>> > nqn.test_tgt >>> > Associating SPDK bdev Controller (TESTTGTCTL) with lcore 0 >>> > Initialization complete. Launching workers. >>> > Starting thread on core 0 >>> > ======================================================== >>> > >>> > Latency(us) >>> > Device Information : IOPS >>> > MB/s Average min max >>> > SPDK bdev Controller (TESTTGTCTL ) from core 0: 92426.00 >>> > 361.04 2770.15 773.45 5265.16 >>> > ======================================================== >>> > Total : 92426.00 >>> > 361.04 2770.15 773.45 5265.16 >>> > _______________________________________________ >>> > SPDK mailing list >>> > SPDK(a)lists.01.org >>> > https://lists.01.org/mailman/listinfo/spdk >>> > _______________________________________________ >>> > SPDK mailing list >>> > SPDK(a)lists.01.org >>> > https://lists.01.org/mailman/listinfo/spdk >>> > >>> -- >>> >>> Regards, >>> Andrey >>> _______________________________________________ >>> SPDK mailing list >>> SPDK(a)lists.01.org >>> https://lists.01.org/mailman/listinfo/spdk >>> _______________________________________________ >>> SPDK mailing list >>> SPDK(a)lists.01.org >>> https://lists.01.org/mailman/listinfo/spdk >>> >> -- >> >> Regards, >> Andrey >> > _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk

Hi Andrey, Thanks for your report. Best Regards Ziye Yang -----Original Message----- From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Andrey Kuzmin Sent: Tuesday, December 4, 2018 1:16 AM To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; Subject: Re: [SPDK] Null bdev read/write performance with NVMf/TCP target Thanks Ziye, the patch works for me. Notice also that, once the multiple lcore issue raised by Sasha has been addressed, I was able to retest performance with multiple connections and found out that the asymmetry I reported is gone if one runs over at least 4 connections (with the same aggregate queue depth). With 4 lcores and 256 aggregate QD, I see ~1050 MB/s for all three workloads (read, writes, and 50/50 mix). Regards, Andrey On Mon, Dec 3, 2018 at 5:54 AM Yang, Ziye <ziye.yang(a)intel.com&gt; wrote: > Hi Andrey, > > You can support this formal patch, it will support the > incapsuleDataSize better. > https://review.gerrithub.io/#/c/spdk/spdk/+/435826/ > > BTW, you need to configure InCapsuleDataSize in the Transport > section in your target configuration file or use RPC. > > Thanks. > > > Best Regards > Ziye Yang > > > -----Original Message----- > From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Andrey > Kuzmin > Sent: Saturday, December 1, 2018 12:29 AM > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > Subject: Re: [SPDK] Null bdev read/write performance with NVMf/TCP > target > > Also FYI, 50/50 rw mix outperforms 100% read workload by more than 20% > (950 MB/s vs. 750), also something not exactly intuitive. Nice first > impression, nonetheless, thanks for the great job. > > Regards, > Andrey > > > On Fri, Nov 30, 2018 at 6:47 PM Andrey Kuzmin > <andrey.v.kuzmin(a)gmail.com&gt; > wrote: > > > Ziye, > > > > I tried out both of your suggestions. Increasing MX_R2T brought home > > substantial improvement (~50%), while allowing in-capsule data - not > > so much, just an extra 10% surprisingly. With both fixes, write > > throughput in the runs I reported earlier increased to some 620MB/s, > > that is close to 80% of the read throughput (latency P50 now stands > > at > 1.33 for reads ms vs. > > 1.57ms for writes). > > > > Also, while playing with 8K transfers, I noticed that > > MAX_IN_CAPSULE_DATA_SIZE is defined in both nvmf/tcp.c and > > nvme/nvme_tcp.c, and they do not agree :) (4K vs 8K). Worth a fix > > introducing a single definition, I believe. > > > > Regards, > > Andrey > > > > > > On Fri, Nov 30, 2018 at 2:33 PM Andrey Kuzmin > > <andrey.v.kuzmin(a)gmail.com&gt; > > wrote: > > > >> > >> > >> On Fri, Nov 30, 2018, 10:27 Yang, Ziye <ziye.yang(a)intel.com&gt; wrote: > >> > >>> Hi Andrey, > >>> > >>> For (1) Actually, we have that feature, but the SPDK initiator > >>> currently does not use it. Usually, the capsule data size is > >>> reported by the target to the initiator. For large I/O size, R2T > >>> will > still be needed. > >> > >> > >> I guess, for larger io sizes, r2t could be piggy-backed on a tcp > >> ack, so to say. Looks like quite natural optimization. > >> > >> If you want to, you can use the following branch to test (Not a > >> formal > >>> patch): > >>> https://review.gerrithub.io/#/c/spdk/spdk/+/435398 > >>> > >>> To see whether the sequential write for 4KB or 8KB will be improved. > >>> > >> Thanks, will do. > >> > >> Regards, > >> Andrey > >> > >>> > >>> > >>> > >>> > >>> Best Regards > >>> Ziye Yang > >>> > >>> > >>> -----Original Message----- > >>> From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Andrey > >>> Kuzmin > >>> Sent: Friday, November 30, 2018 3:18 PM > >>> To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > >>> Subject: Re: [SPDK] Null bdev read/write performance with NVMf/TCP > >>> target > >>> > >>> Thankd Ziye, I'll give a shot to expanding the max R2T PDU and get > >>> back to you in this. > >>> > >>> Re (1) below, are there any plans to support immediate data in the > >>> nvmf initiator in the future, to avoid the extra R2T exchange > >>> (similar > to iSCSI)? > >>> > >>> Thanks, > >>> Andrey > >>> > >>> On Fri, Nov 30, 2018, 04:28 Yang, Ziye <ziye.yang(a)intel.com&gt; wrote: > >>> > >>> > Hi Andrey, > >>> > > >>> > Will look at this, but for the write operation. There may be the > >>> > following > >>> > reasons: > >>> > > >>> > 1 For the write operation. If the data is not capsulated in the > >>> > command buffer (In our NVMe-oF initiator for the data write on > >>> > I/O qpair, we will not use that), the NVMe-oF target will send > >>> > the R2T PDU to the host. So compared with read operation, there > >>> > will be additional > >>> PDU exchange. > >>> > 2 There are restrictions for the active R2T pdus defined in the spec. > >>> > Currently, we use this Macro in the nvme_tcp.c > >>> > (#define NVME_TCP_MAX_R2T_DEFAULT 16). > >>> > > >>> > When the host sends the ic_req pdu, it will tell the target that > >>> > there will be at most so many active R2T pdus. So if you want to > >>> > improve the performance, you need to increase this value. > >>> > Currently it is not configurable, you may need to update it > >>> > with a > larger value. > >>> > > >>> > Thanks. > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > Best Regards > >>> > Ziye Yang > >>> > > >>> > -----Original Message----- > >>> > From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of > >>> > Andrey Kuzmin > >>> > Sent: Friday, November 30, 2018 1:03 AM > >>> > To: Storage Performance Development Kit <spdk(a)lists.01.org&gt; > >>> > Subject: [SPDK] Null bdev read/write performance with NVMf/TCP > >>> > target > >>> > > >>> > I'm getting some rather counter-intuitive results in the first > >>> > hand-on with the recently announced SPDK NVMoF/TCP target. The > >>> > only difference between the two runs below (over a 10GigE link) > >>> > is that the first one is doing sequential read, while the second > >>> > one > >>> > - sequential write, and notice that both run against the same > >>> > null bdev target. Any ideas as to why null bdev writes are 2x > >>> > slower than reads and where that extra 1ms of latency is coming from? > >>> > > >>> > Regards, > >>> > Andrey > >>> > > >>> > sudo ./examples/nvme/perf/perf -c 0x1 -q 256 -o 4096 -w read -t > >>> > 10 -r "trtype:TCP adrfam:IPv4 traddr:192.168.100.105 > >>> > trsvcid:4420 subnqn:nqn.test_tgt" > >>> > Starting SPDK v19.01-pre / DPDK 18.08.0 initialization... > >>> > [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci > >>> > --base-virtaddr=0x200000000000 --file-prefix=spdk_pid3870 ] > >>> > EAL: Detected 56 lcore(s) > >>> > EAL: Detected 2 NUMA nodes > >>> > EAL: No free hugepages reported in hugepages-1048576kB > >>> > EAL: Probing VFIO support... > >>> > Initializing NVMe Controllers > >>> > Attaching to NVMe over Fabrics controller at 192.168.100.105:4420: > >>> > nqn.test_tgt > >>> > Attached to NVMe over Fabrics controller at 192.168.100.105:4420: > >>> > nqn.test_tgt > >>> > Associating SPDK bdev Controller (TESTTGTCTL) with lcore 0 > >>> > Initialization complete. Launching workers. > >>> > Starting thread on core 0 > >>> > ======================================================== > >>> > > >>> > Latency(us) > >>> > Device Information : IOPS > >>> > MB/s Average min max > >>> > SPDK bdev Controller (TESTTGTCTL ) from core 0: 203888.90 > >>> > 796.44 1255.62 720.37 1967.79 > >>> > ======================================================== > >>> > Total : 203888.90 > >>> > 796.44 1255.62 720.37 1967.79 > >>> > > >>> > sudo ./examples/nvme/perf/perf -c 0x1 -q 256 -o 4096 -w write -t > >>> > 10 -r "trtype:TCP adrfam:IPv4 traddr:192.168.100.105 > >>> > trsvcid:4420 subnqn:nqn.test_tgt" > >>> > Starting SPDK v19.01-pre / DPDK 18.08.0 initialization... > >>> > [ DPDK EAL parameters: perf --no-shconf -c 0x1 --no-pci > >>> > --base-virtaddr=0x200000000000 --file-prefix=spdk_pid3873 ] > >>> > EAL: Detected 56 lcore(s) > >>> > EAL: Detected 2 NUMA nodes > >>> > EAL: No free hugepages reported in hugepages-1048576kB > >>> > EAL: Probing VFIO support... > >>> > Initializing NVMe Controllers > >>> > Attaching to NVMe over Fabrics controller at 192.168.100.105:4420: > >>> > nqn.test_tgt > >>> > Attached to NVMe over Fabrics controller at 192.168.100.105:4420: > >>> > nqn.test_tgt > >>> > Associating SPDK bdev Controller (TESTTGTCTL) with lcore 0 > >>> > Initialization complete. Launching workers. > >>> > Starting thread on core 0 > >>> > ======================================================== > >>> > > >>> > Latency(us) > >>> > Device Information : IOPS > >>> > MB/s Average min max > >>> > SPDK bdev Controller (TESTTGTCTL ) from core 0: 92426.00 > >>> > 361.04 2770.15 773.45 5265.16 > >>> > ======================================================== > >>> > Total : 92426.00 > >>> > 361.04 2770.15 773.45 5265.16 > >>> > _______________________________________________ > >>> > SPDK mailing list > >>> > SPDK(a)lists.01.org > >>> > https://lists.01.org/mailman/listinfo/spdk > >>> > _______________________________________________ > >>> > SPDK mailing list > >>> > SPDK(a)lists.01.org > >>> > https://lists.01.org/mailman/listinfo/spdk > >>> > > >>> -- > >>> > >>> Regards, > >>> Andrey > >>> _______________________________________________ > >>> SPDK mailing list > >>> SPDK(a)lists.01.org > >>> https://lists.01.org/mailman/listinfo/spdk > >>> _______________________________________________ > >>> SPDK mailing list > >>> SPDK(a)lists.01.org > >>> https://lists.01.org/mailman/listinfo/spdk > >>> > >> -- > >> > >> Regards, > >> Andrey > >> > > > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk