Hi, Jim Our threading model looks like: On server start, the service process creates all service threads one by one, each thread function will set CPU affinity before doing anything, and only the first created thread will call spdk_env_init(). So in my case, will the following created threads inherit the affinity set by spdk_env_init() from the first thread? And I suppose calling spdk_unaffinitize_thread() after spdk_env_init() could break our CPU affinity as well, so we need re-set affinity after the spdk_unaffinitize_thread() call? Thanks -Niu On 2020/2/27, 11:11 PM, "Harris, James R" <james.r.harris(a)intel.com&gt; wrote: Hi Niu, The core mask is used to specify which cores will be used for polling threads managed by the SPDK/DPDK application framework. There really isn't a way to disable the core mask, but there are some ways to minimize its effects. I'm curious - in your environment, when do you create your DAOS threads? And you create them from the same thread that is calling spdk_env_init()? Note that the thread calling spdk_env_init() will be pinned to that single core you've specified in your core mask. And threads created after spdk_env_init() will inherit that affinity. You can call spdk_unaffinitize_thread() immediately after spdk_env_init() returns to eliminate that affinity. Could you try that and see if it helps when specifying core_mask 0x1 in your environment? -Jim On 2/27/20, 2:02 AM, "Niu, Yawei" <yawei.niu(a)intel.com&gt; wrote: Hi, We (DAOS) used to call spdk_env_init() with default ‘core_mask’ “0x1” to initialize DPDK EAL, during recent tests, we realized that performance can be boosted a lot by changing the ‘core_mask’ to some other value like ‘0x8’, so I’m wondering if it because some sort of DPDK service thread was stealing CPU cycles from DAOS service thread, and when the “core_mask” is changed, the DPDK thread is happen to be created on some unused core? But I didn’t observe any suspicious thread by ‘ps’ command (with core_mask ‘0x1’ or ‘0x8’). So my question is what this ‘core_mask’ is used for? I suppose it’s not useful for our server (we have our own polling thread and user level scheduler), is there any way to disable it? What’s the best practice of this ‘core_mask’ configuration for SPDK apps? Thanks -Niu _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org

I see, I'll reorg the code a bit. Thanks for your help! On 2020/2/28, 12:02 AM, "Harris, James R" <james.r.harris(a)intel.com&gt; wrote: Hi Niu, Do you have the ability to call spdk_env_init() before setting the CPU affinity? If not, you'll likely need to save the CPU affinity before calling spdk_env_init(), and then set it back again after spdk_env_init() returns. -Jim On 2/27/20, 8:54 AM, "Niu, Yawei" <yawei.niu(a)intel.com&gt; wrote: Hi, Jim Our threading model looks like: On server start, the service process creates all service threads one by one, each thread function will set CPU affinity before doing anything, and only the first created thread will call spdk_env_init(). So in my case, will the following created threads inherit the affinity set by spdk_env_init() from the first thread? And I suppose calling spdk_unaffinitize_thread() after spdk_env_init() could break our CPU affinity as well, so we need re-set affinity after the spdk_unaffinitize_thread() call? Thanks -Niu On 2020/2/27, 11:11 PM, "Harris, James R" <james.r.harris(a)intel.com&gt; wrote: Hi Niu, The core mask is used to specify which cores will be used for polling threads managed by the SPDK/DPDK application framework. There really isn't a way to disable the core mask, but there are some ways to minimize its effects. I'm curious - in your environment, when do you create your DAOS threads? And you create them from the same thread that is calling spdk_env_init()? Note that the thread calling spdk_env_init() will be pinned to that single core you've specified in your core mask. And threads created after spdk_env_init() will inherit that affinity. You can call spdk_unaffinitize_thread() immediately after spdk_env_init() returns to eliminate that affinity. Could you try that and see if it helps when specifying core_mask 0x1 in your environment? -Jim On 2/27/20, 2:02 AM, "Niu, Yawei" <yawei.niu(a)intel.com&gt; wrote: Hi, We (DAOS) used to call spdk_env_init() with default ‘core_mask’ “0x1” to initialize DPDK EAL, during recent tests, we realized that performance can be boosted a lot by changing the ‘core_mask’ to some other value like ‘0x8’, so I’m wondering if it because some sort of DPDK service thread was stealing CPU cycles from DAOS service thread, and when the “core_mask” is changed, the DPDK thread is happen to be created on some unused core? But I didn’t observe any suspicious thread by ‘ps’ command (with core_mask ‘0x1’ or ‘0x8’). So my question is what this ‘core_mask’ is used for? I suppose it’s not useful for our server (we have our own polling thread and user level scheduler), is there any way to disable it? What’s the best practice of this ‘core_mask’ configuration for SPDK apps? Thanks -Niu _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org