Hi, I ran some tests for nvmf target using various configurations in host side: perf, fio (SPDK plugin), fio (libaio). It looks like, fio+SPDK produces non-uniform load on cores in nvmf application. SPDK's FIO plugin opens 3 QPs for each fio job. I suppose, they are: admin QP for discovery, admin QP and IO QP. On the other side, nvmf target, by default, use round-robin policy for QP assignment to cores. As result, in some configurations, some cores are busy with IO and other handle admin commands only. In opposite, in tests with fio above libaio, a single admin QP is opened at connect and all target's core are busy with IO. For example, nvmf target rans on 3 cores and then I ran 2 fio jobs with SPDK initiator. Core #14 handles IO from 2 jobs. nvmf_tgt.c: 212:nvmf_tgt_get_qpair_core: *NOTICE*: Assigning QP 0x10295e0 to core 12 nvmf_tgt.c: 212:nvmf_tgt_get_qpair_core: *NOTICE*: Assigning QP 0x102cfd0 to core 13 nvmf_tgt.c: 212:nvmf_tgt_get_qpair_core: *NOTICE*: Assigning QP 0x102d580 to core 14 rdma.c: 526:spdk_nvmf_rdma_set_ibv_state: *NOTICE*: IBV QP#0 changed to: IBV_QPS_ERR nvmf_tgt.c: 212:nvmf_tgt_get_qpair_core: *NOTICE*: Assigning QP 0x10295e0 to core 12 nvmf_tgt.c: 212:nvmf_tgt_get_qpair_core: *NOTICE*: Assigning QP 0x102daf0 to core 13 nvmf_tgt.c: 212:nvmf_tgt_get_qpair_core: *NOTICE*: Assigning QP 0x102e0a0 to core 14 rdma.c: 526:spdk_nvmf_rdma_set_ibv_state: *NOTICE*: IBV QP#0 changed to: IBV_QPS_ERR rdma.c: 526:spdk_nvmf_rdma_set_ibv_state: *NOTICE*: IBV QP#2 changed to: IBV_QPS_ERR rdma.c: 526:spdk_nvmf_rdma_set_ibv_state: *NOTICE*: IBV QP#0 changed to: IBV_QPS_ERR rdma.c: 526:spdk_nvmf_rdma_set_ibv_state: *NOTICE*: IBV QP#1 changed to: IBV_QPS_ERR rdma.c: 526:spdk_nvmf_rdma_set_ibv_state: *NOTICE*: IBV QP#0 changed to: IBV_QPS_ERR If the same test runs above libaio, 24 IO QPs and 1 admin QP is opened at connect, and the test utilizes two cores in nvmf target. Have I got that right? Is there any reason to open admin QPs in each fio job ? (I understand, from implementation perspective that's is simplest way). If you ask me, round-robin approach in target can be improved. I think, admin and IO QPs should be handled independently. IO QPs should be spread among cores regardless admin QPs. What do you think?

Best regards Sasha _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk

On 1/14/19, 11:55 AM, "SPDK on behalf of Walker, Benjamin" < spdk-bounces(a)lists.01.org on behalf of benjamin.walker(a)intel.com&gt; wrote: > > If you ask me, round-robin approach in target can be improved. I think, > admin and IO QPs should be handled independently. IO QPs should be > spread among cores regardless admin QPs. What do you think? I think this is the best solution - we should make the distribution of the qpairs smarter such that admin queues don't count when advancing the round-robin pointer. Do we know that a queue is admin or IO when we make the decision on advancing the round-robin pointer?

_______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk

> > On 1/14/19, 11:55 AM, "SPDK on behalf of Walker, Benjamin" < > spdk-bounces(a)lists.01.org on behalf of benjamin.walker(a)intel.com&gt; wrote: > > > > > If you ask me, round-robin approach in target can be improved. I think, > > admin and IO QPs should be handled independently. IO QPs should be > > spread among cores regardless admin QPs. What do you think? > > I think this is the best solution - we should make the distribution of the > qpairs smarter such that admin queues don't count when advancing the > round-robin > pointer. > > Do we know that a queue is admin or IO when we make the decision on advancing > the round-robin pointer? Fortunately, it's one of the few things we do actually know up front. This is what the target knows when a new connection is requested: struct spdk_nvmf_rdma_request_private_data { uint16_t recfmt; /* record format */ uint16_t qid; /* queue id */ uint16_t hrqsize; /* host receive queue size */ uint16_t hsqsize; /* host send queue size */ uint16_t cntlid; /* controller id */ uint8_t reserved[22]; }; So qid of 0 means they want to request an admin queue. Perfect. Maybe we keep separate admin and IO round-robin pointers? To avoid the case where some non-trivial number of admin queues are established between I/O queue connections. It's not a likely scenario, and probably wouldn't make much of a performance difference if it did, but keeping separate pointers should be easy to implement. -Jim