Our results were also pretty much align with your findings. But CPU utilization was way less in SPDK target compared to Kernel mode target. It would be nice to learn if anyone else has seen better results on kernel mode initiator with SPDK target. From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Rajat Maheshwari Sent: Thursday, March 09, 2017 12:53 AM To: spdk(a)lists.01.org Subject: Re: [SPDK] NVMeoF IO latency performance apps Thanks. We are using SPDK on target end, but kernel Initiator. We used fio, but found some unexpected results. (latency figures almost similar for SPDK target & Kernel Target, besides we are using NULL block devices too, so media latency is not involved here) So, I would like to check for certain fio parameters, if in case we are missing something. For ex.: couple of samples from fio "read I/O" run (SPDK NVMe-oF target, Kernel host) (fio command line: fio --bs=4k --numjobs=16 --iodepth=4 --loops=1 --ioengine=libaio --direct=1 --invalidate=1 --fsync_on_close=1 --randrepeat=1 --norandommap --time_based --runtime=10 --filename=/dev/nvme0n1 --name=read-phase --rw=read ) sample #1 slat (usec): min=1 , max=43 , avg= 3.61, stdev= 1.77 clat (usec): min=40 , max=414 , avg=233.02, stdev=28.75 lat (usec): min=47 , max=416 , avg=236.73, stdev=28.79 sample #2 slat (usec): min=1 , max=41 , avg= 3.63, stdev= 1.77 clat (usec): min=71 , max=473 , avg=232.60, stdev=28.65 lat (usec): min=74 , max=476 , avg=236.33, stdev=28.70 sample #3 slat (usec): min=1 , max=48 , avg= 3.64, stdev= 1.82 clat (usec): min=17 , max=477 , avg=233.10, stdev=28.87 lat (usec): min=21 , max=479 , avg=236.84, stdev=28.93 All above samples (infact every sample) shows maximum clat above 380us. Also, the average clat is never below 230us. We also compared it using kernel target (& Kernel host) and apparently we didn't see much difference between two runs. Max clat range was almost in the range (only 10-20usec higher than SDPK target), but average clat was, almost, exactly the same (230-240us). Is it really expected? (I believe, its not!) So, are we missing any fio flag(s)? On Mon, 2017-03-06 at 23:04 +0530, Rajat Maheshwari wrote: The majority of people use fio. Are you planning to use the SPDK NVMe- oF target, the NVMe-oF host, or both? If you are using just the SPDK NVMe-oF target with the kernel host, fio is an excellent tool. SPDK mailing list SPDK@lists.01.org<mailto:SPDK@lists.01.org> https://lists.01.org/mailman/listinfo/spdk