Hi Andrey, I've never seen that issue on any of the Xeon platforms I've tested with. Not sure about Epyc though. I can't figure out where "set_mempolicy failed" gets printed by DPDK though. I see where DPDK prints messages when set_mempolicy fails, but it's in a different format.

On 2/18/20, 1:19 PM, "Andrey Kuzmin" <andrey.v.kuzmin(a)gmail.com&gt; wrote: On Wed, Feb 19, 2020, 00:11 Harris, James R <james.r.harris(a)intel.com&gt; wrote: > Hi Andrey, > > I've never seen that issue on any of the Xeon platforms I've tested with. > Not sure about Epyc though. > > I can't figure out where "set_mempolicy failed" gets printed by DPDK > though. I see where DPDK prints messages when set_mempolicy fails, but > it's in a different format. Hi Jim, I have also spent some time scratching my head on this until I realized that DPDK overrides numa_error with its own logging, and that's exactly where it gets printed. Ah - thanks for teaching me something new today. Now I understand why my simple grep failed to find it! As to the cause, it appears that the kernel is unhappy with the mode/nodes combination DPDK tries to give it. The kernel is 3.10, BTW. Do you by chance have any local edits in your DPDK submodule? > No, I'm running stock SPDK/DPDK. Regards, Andrey > -Jim > > > On 2/18/20, 8:28 AM, "Andrey Kuzmin" <andrey.v.kuzmin(a)gmail.com&gt; wrote: > > Hi guys, > > I'm running 20.01 (default configuration) on a CentOS 7 Epyc-based > system, and getting a lot of subj warnings like in the below example. > Does that point to a system configuration issue that needs to be > fixed? > > Thanks, > Andrey > > Starting SPDK v20.01 git sha1 5ffffe9 / DPDK 19.11.0 initialization... > [ DPDK EAL parameters: hotplug --no-shconf -c 0x1 > --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 > --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations > --file-prefix=spdk_pid109760 ] > EAL: No available hugepages reported in hugepages-1048576kB > EAL: VFIO support initialized > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > Initializing NVMe Controllers > EAL: set_mempolicy failed: Invalid argument > EAL: using IOMMU type 1 (Type 1) > EAL: set_mempolicy failed: Invalid argument > Attaching to 0000:45:00.0 > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > Attaching to 0000:46:00.0 > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > Attaching to 0000:4b:00.0 > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > Attaching to 0000:4c:00.0 > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > Attaching to 0000:4d:00.0 > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > Attaching to 0000:4e:00.0 > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > Attached to 0000:45:00.0 > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > Attached to 0000:46:00.0 > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > Attached to 0000:4b:00.0 > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > Attached to 0000:4c:00.0 > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > Attached to 0000:4d:00.0 > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > Attached to 0000:4e:00.0 > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > EAL: set_mempolicy failed: Invalid argument > Initialization complete. Starting I/O... > _______________________________________________ > SPDK mailing list -- spdk(a)lists.01.org > To unsubscribe send an email to spdk-leave(a)lists.01.org > > > _______________________________________________ > SPDK mailing list -- spdk(a)lists.01.org > To unsubscribe send an email to spdk-leave(a)lists.01.org > _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org

Hi. I hit the same problem on an AMD EPYC 7301 server. Can you please share details how you fix this?

On-line CPU(s) list: 0-63 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 2 NUMA node(s): 8 Thanks, Jin _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org