August 2020 - LKP - Ml01.01.Org

[sched/fair] 3c29e651e1: hackbench.throughput -15.2% regression

by kernel test robot

Greeting, FYI, we noticed a -15.2% regression of hackbench.throughput due to commit: commit: 3c29e651e16dd3b3179cfb2d055ee9538e37515c ("sched/fair: Fall back to sched-idle CPU if idle CPU isn't found") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: hackbench on test machine: 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory with following parameters: nr_threads: 100% mode: threads ipc: pipe cpufreq_governor: performance ucode: 0xca test-description: Hackbench is both a benchmark and a stress test for the Linux kernel scheduler. test-url: https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/sc... In addition to that, the commit also has significant impact on the following tests: +------------------+-----------------------------------------------------------------------+ | testcase: change | hackbench: hackbench.throughput -11.3% regression | | test machine | 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory | | test parameters | cpufreq_governor=performance | | | ipc=pipe | | | iterations=18 | | | mode=threads | | | nr_threads=1600% | | | ucode=0x43 | +------------------+-----------------------------------------------------------------------+ | testcase: change | hackbench: hackbench.throughput -10.6% regression | | test machine | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory | | test parameters | cpufreq_governor=performance | | | ipc=pipe | | | mode=process | | | nr_threads=50% | | | ucode=0xb000038 | +------------------+-----------------------------------------------------------------------+ | testcase: change | hackbench: hackbench.throughput -6.8% regression | | test machine | 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory | | test parameters | cpufreq_governor=performance | | | ipc=pipe | | | mode=threads | | | nr_threads=100% | | | ucode=0xb8 | +------------------+-----------------------------------------------------------------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/ipc/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode: gcc-7/performance/pipe/x86_64-rhel-7.6/threads/100%/debian-x86_64-20191114.cgz/lkp-cfl-e1/hackbench/0xca commit: 43e9f7f231 ("sched/fair: Start tracking SCHED_IDLE tasks count in cfs_rq") 3c29e651e1 ("sched/fair: Fall back to sched-idle CPU if idle CPU isn't found") 43e9f7f231e40e45 3c29e651e16dd3b3179cfb2d055 ---------------- --------------------------- %stddev %change %stddev \ | \ 69492 ± 3% -15.2% 58899 hackbench.throughput 7.727e+08 ± 7% +38.3% 1.069e+09 ± 2% hackbench.time.involuntary_context_switches 131708 ± 4% -15.4% 111382 hackbench.time.minor_page_faults 684.98 ± 2% -2.5% 667.72 hackbench.time.user_time 1.755e+09 ± 2% +11.2% 1.953e+09 hackbench.time.voluntary_context_switches 4.32e+08 ± 4% -15.6% 3.648e+08 hackbench.workload 186.50 ± 12% -16.6% 155.50 ± 11% interrupts.TLB:TLB_shootdowns 4147892 ± 5% +19.6% 4961643 vmstat.system.cs 0.72 ± 10% -0.1 0.58 ± 17% mpstat.cpu.all.idle% 0.00 ±156% +0.0 0.01 ± 62% mpstat.cpu.all.soft% 1038 ± 9% +18.5% 1230 ± 4% slabinfo.avc_xperms_data.active_objs 1038 ± 9% +18.5% 1230 ± 4% slabinfo.avc_xperms_data.num_objs 40647036 ± 10% -34.6% 26589504 proc-vmstat.numa_hit 40647036 ± 10% -34.6% 26589504 proc-vmstat.numa_local 40724029 ± 10% -34.5% 26661546 proc-vmstat.pgalloc_normal 763906 ± 2% -7.5% 706781 proc-vmstat.pgfault 40701242 ± 10% -34.6% 26637947 proc-vmstat.pgfree 1636058 ± 11% -45.0% 899629 ± 11% turbostat.C1 0.10 ± 7% -0.0 0.07 ± 7% turbostat.C1% 15201 ± 25% +113.3% 32422 ± 8% turbostat.C1E 0.01 ± 57% +0.0 0.03 ± 13% turbostat.C1E% 9093 ± 11% +217.2% 28841 ± 11% turbostat.C3 0.00 ±173% +0.0 0.03 ± 13% turbostat.C3% 16977 ± 2% -92.8% 1222 ± 98% turbostat.C8 0.17 ± 4% -0.2 0.01 ±110% turbostat.C8% 0.13 -100.0% 0.00 turbostat.CPU%c7 9623106 ± 5% -35.1% 6241651 ± 8% cpuidle.C1.time 1636668 ± 11% -45.0% 899926 ± 11% cpuidle.C1.usage 899494 ± 33% +272.6% 3351412 ± 5% cpuidle.C1E.time 15327 ± 25% +112.6% 32592 ± 8% cpuidle.C1E.usage 435674 ± 20% +643.5% 3239351 ± 14% cpuidle.C3.time 9179 ± 11% +215.2% 28933 ± 11% cpuidle.C3.usage 16850924 ± 2% -94.2% 979516 ± 86% cpuidle.C8.time 17150 ± 2% -92.0% 1378 ± 88% cpuidle.C8.usage 6022328 ± 18% -57.1% 2584773 ± 7% cpuidle.POLL.time 5339703 ± 19% -60.1% 2130089 ± 7% cpuidle.POLL.usage 14.06 ± 25% +64.0% 23.07 ± 13% sched_debug.cfs_rq:/.runnable_load_avg.min 25.10 ± 5% -15.8% 21.12 ± 7% sched_debug.cfs_rq:/.runnable_load_avg.stddev 13339 ± 24% +45.7% 19433 ± 7% sched_debug.cfs_rq:/.runnable_weight.min 166.51 ± 24% +48.9% 247.86 ± 7% sched_debug.cfs_rq:/.util_est_enqueued.min 253.56 ± 2% -12.3% 222.30 ± 9% sched_debug.cfs_rq:/.util_est_enqueued.stddev 159602 ± 23% +55.6% 248411 ± 6% sched_debug.cpu.avg_idle.max 51567 ± 24% +49.1% 76905 ± 11% sched_debug.cpu.avg_idle.stddev 11576 ± 5% -9.5% 10478 sched_debug.cpu.curr->pid.avg 9976 ± 8% -16.8% 8305 ± 9% sched_debug.cpu.curr->pid.min 4.81 ± 15% +43.3% 6.89 ± 8% sched_debug.cpu.nr_running.min 6.68 ± 4% -17.0% 5.54 ± 8% sched_debug.cpu.nr_running.stddev 76003928 ± 2% +21.0% 91975904 sched_debug.cpu.nr_switches.avg 78115404 ± 2% +23.4% 96397551 sched_debug.cpu.nr_switches.max 74275755 ± 2% +19.0% 88359903 sched_debug.cpu.nr_switches.min 1043054 ± 15% +83.9% 1917752 ± 10% sched_debug.cpu.nr_switches.stddev -685.26 +64.8% -1129 sched_debug.cpu.nr_uninterruptible.min 15470 ± 9% -31.0% 10674 ± 9% softirqs.CPU0.SCHED 13756 ± 10% -40.8% 8140 ± 4% softirqs.CPU1.SCHED 13645 ± 11% -39.7% 8227 ± 3% softirqs.CPU10.SCHED 14065 ± 12% -38.8% 8609 ± 2% softirqs.CPU11.SCHED 13948 ± 11% -39.9% 8381 ± 6% softirqs.CPU12.SCHED 13823 ± 12% -39.7% 8332 ± 9% softirqs.CPU13.SCHED 73323 +13.4% 83165 ± 11% softirqs.CPU14.RCU 13872 ± 12% -36.5% 8809 ± 7% softirqs.CPU14.SCHED 13726 ± 12% -40.4% 8175 ± 4% softirqs.CPU15.SCHED 13780 ± 11% -34.8% 8987 ± 12% softirqs.CPU2.SCHED 13895 ± 10% -40.5% 8266 ± 6% softirqs.CPU3.SCHED 221879 +13.4% 251589 ± 14% softirqs.CPU3.TIMER 13873 ± 10% -40.4% 8262 ± 6% softirqs.CPU4.SCHED 72625 +12.0% 81355 ± 10% softirqs.CPU5.RCU 13876 ± 13% -41.4% 8137 ± 3% softirqs.CPU5.SCHED 14118 ± 12% -40.8% 8354 ± 3% softirqs.CPU6.SCHED 13503 ± 12% -40.0% 8099 softirqs.CPU7.SCHED 13549 ± 11% -38.2% 8374 ± 8% softirqs.CPU8.SCHED 14273 ± 10% -40.1% 8548 ± 6% softirqs.CPU9.SCHED 223184 ± 11% -38.9% 136383 ± 5% softirqs.SCHED 55.80 +9.9% 61.34 perf-stat.i.MPKI 1.32 +0.1 1.39 perf-stat.i.branch-miss-rate% 1.066e+08 +3.7% 1.105e+08 perf-stat.i.branch-misses 0.15 ± 8% -0.0 0.11 ± 3% perf-stat.i.cache-miss-rate% 2521584 ± 9% -26.5% 1854353 ± 3% perf-stat.i.cache-misses 2.27e+09 +8.0% 2.451e+09 perf-stat.i.cache-references 4162783 ± 4% +19.7% 4981708 perf-stat.i.context-switches 1.52 +2.0% 1.55 perf-stat.i.cpi 227633 ± 14% +75.1% 398694 ± 3% perf-stat.i.cpu-migrations 33931 ± 12% +42.7% 48428 ± 2% perf-stat.i.cycles-between-cache-misses 0.01 -0.0 0.01 ± 4% perf-stat.i.dTLB-load-miss-rate% 1408442 ± 2% -16.4% 1177727 ± 4% perf-stat.i.dTLB-load-misses 1.229e+10 -2.1% 1.203e+10 perf-stat.i.dTLB-loads 0.00 ± 2% -0.0 0.00 ± 4% perf-stat.i.dTLB-store-miss-rate% 35757 ± 2% -16.0% 30032 ± 3% perf-stat.i.dTLB-store-misses 7.449e+09 -2.3% 7.279e+09 perf-stat.i.dTLB-stores 39246970 -5.5% 37079604 perf-stat.i.iTLB-load-misses 76813 ± 8% -26.2% 56724 ± 6% perf-stat.i.iTLB-loads 4.096e+10 -1.9% 4.02e+10 perf-stat.i.instructions 1062 +4.1% 1105 perf-stat.i.instructions-per-iTLB-miss 0.66 -1.9% 0.65 perf-stat.i.ipc 1210 -7.1% 1124 perf-stat.i.minor-faults 105889 ± 10% -24.8% 79596 ± 5% perf-stat.i.node-loads 174508 ± 7% -24.4% 131915 perf-stat.i.node-stores 1210 -7.1% 1124 perf-stat.i.page-faults 55.42 ± 2% +10.0% 60.98 perf-stat.overall.MPKI 1.31 +0.1 1.38 perf-stat.overall.branch-miss-rate% 0.11 ± 11% -0.0 0.08 ± 3% perf-stat.overall.cache-miss-rate% 1.51 +1.9% 1.54 perf-stat.overall.cpi 24740 ± 9% +34.9% 33377 ± 3% perf-stat.overall.cycles-between-cache-misses 0.01 -0.0 0.01 ± 4% perf-stat.overall.dTLB-load-miss-rate% 0.00 ± 2% -0.0 0.00 ± 4% perf-stat.overall.dTLB-store-miss-rate% 1043 +3.9% 1084 perf-stat.overall.instructions-per-iTLB-miss 0.66 -1.9% 0.65 perf-stat.overall.ipc 57826 ± 3% +15.6% 66874 perf-stat.overall.path-length 1.064e+08 +3.7% 1.104e+08 perf-stat.ps.branch-misses 2517524 ± 9% -26.5% 1851335 ± 3% perf-stat.ps.cache-misses 2.266e+09 +8.0% 2.447e+09 perf-stat.ps.cache-references 4155898 ± 4% +19.7% 4973458 perf-stat.ps.context-switches 227258 ± 14% +75.1% 398041 ± 3% perf-stat.ps.cpu-migrations 1406119 ± 2% -16.4% 1175777 ± 4% perf-stat.ps.dTLB-load-misses 1.227e+10 -2.1% 1.201e+10 perf-stat.ps.dTLB-loads 35699 ± 2% -16.0% 29983 ± 3% perf-stat.ps.dTLB-store-misses 7.437e+09 -2.3% 7.267e+09 perf-stat.ps.dTLB-stores 39182154 -5.5% 37018206 perf-stat.ps.iTLB-load-misses 76695 ± 8% -26.2% 56638 ± 6% perf-stat.ps.iTLB-loads 4.09e+10 -1.9% 4.013e+10 perf-stat.ps.instructions 1208 -7.1% 1122 perf-stat.ps.minor-faults 105716 ± 10% -24.8% 79465 ± 5% perf-stat.ps.node-loads 174234 ± 7% -24.4% 131703 perf-stat.ps.node-stores 1208 -7.1% 1122 perf-stat.ps.page-faults 29.39 ±104% -29.4 0.00 perf-profile.calltrace.cycles-pp.start_thread 15.86 ±105% -15.9 0.00 perf-profile.calltrace.cycles-pp.__GI___libc_write.start_thread 14.88 ±105% -14.9 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_write.start_thread 14.75 ±105% -14.8 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write.start_thread 13.53 ±104% -13.5 0.00 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write.start_thread 12.51 ±104% -12.5 0.00 perf-profile.calltrace.cycles-pp.__GI___libc_read.start_thread 12.45 ±105% -12.4 0.00 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write 11.21 ±105% -11.2 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_read.start_thread 11.08 ±105% -11.1 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read.start_thread 10.47 ±105% -10.5 0.00 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read.start_thread 9.73 ±105% -9.7 0.00 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read 3.09 ± 12% -0.6 2.50 ± 2% perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_read.new_sync_read.vfs_read.ksys_read 2.45 ± 14% -0.5 1.95 ± 2% perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.new_sync_write.vfs_write.ksys_write 1.15 ± 13% -0.2 0.95 ± 4% perf-profile.calltrace.cycles-pp.copyin.copy_page_from_iter.pipe_write.new_sync_write.vfs_write 0.99 ± 13% -0.2 0.83 ± 5% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.copy_page_from_iter.pipe_write.new_sync_write 0.92 ± 15% +0.2 1.10 perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.84 ± 23% +0.3 1.10 ± 2% perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.pipe_wait 0.99 ± 19% +0.3 1.26 ± 3% perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 0.97 ± 23% +0.3 1.26 perf-profile.calltrace.cycles-pp.__enqueue_entity.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 0.29 ±100% +0.3 0.60 ± 5% perf-profile.calltrace.cycles-pp.reschedule_interrupt.__lock_text_start.__wake_up_common_lock.pipe_write.new_sync_write 0.55 ± 62% +0.3 0.89 ± 7% perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.dequeue_task_fair.__schedule.schedule 0.28 ±100% +0.4 0.63 ± 7% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__schedule.schedule.exit_to_usermode_loop 0.33 ±100% +0.4 0.69 ± 2% perf-profile.calltrace.cycles-pp.prepare_to_wait.pipe_wait.pipe_read.new_sync_read.vfs_read 0.29 ±100% +0.4 0.65 ± 4% perf-profile.calltrace.cycles-pp.___perf_sw_event.__schedule.schedule.pipe_wait.pipe_read 1.00 ± 27% +0.4 1.37 ± 2% perf-profile.calltrace.cycles-pp.check_preempt_wakeup.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function 0.57 ± 63% +0.4 0.95 ± 5% perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate 1.46 ± 31% +0.4 1.86 ± 2% perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.72 ± 60% +0.4 1.14 ± 2% perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.29 ±100% +0.4 0.70 ± 6% perf-profile.calltrace.cycles-pp.pick_next_entity.pick_next_task_fair.__schedule.schedule.pipe_wait 0.34 ±100% +0.4 0.77 ± 7% perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task 1.26 ± 26% +0.5 1.72 perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common 1.13 ± 26% +0.5 1.59 perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 0.35 ±100% +0.5 0.83 ± 8% perf-profile.calltrace.cycles-pp._raw_spin_lock.__schedule.schedule.exit_to_usermode_loop.do_syscall_64 1.28 ± 28% +0.5 1.77 ± 3% perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.pipe_wait 1.40 ± 26% +0.5 1.89 perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 0.44 ±100% +0.5 0.95 ± 7% perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop 2.33 ± 17% +0.5 2.88 ± 2% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.pipe_wait.pipe_read 1.33 ± 28% +0.6 1.89 ± 2% perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 1.40 ± 24% +0.6 1.97 ± 4% perf-profile.calltrace.cycles-pp.available_idle_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function 0.83 ± 66% +0.6 1.40 ± 5% perf-profile.calltrace.cycles-pp.native_write_msr 5.01 ± 8% +0.8 5.81 ± 2% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 1.75 ± 39% +0.9 2.61 ± 6% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop.do_syscall_64 1.33 ± 67% +1.0 2.28 ± 6% perf-profile.calltrace.cycles-pp.switch_fpu_return.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.56 ± 12% +1.2 7.78 perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_wait.pipe_read 8.96 ± 9% +1.6 10.56 perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function 9.23 ± 10% +1.6 10.87 perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 9.17 ± 10% +1.6 10.81 perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common 3.72 ± 31% +1.7 5.44 ± 6% perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.37 ± 8% +1.9 5.32 ± 4% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common 4.37 ± 10% +2.3 6.65 ± 5% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 15.27 ± 10% +2.3 17.59 perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_wait.pipe_read.new_sync_read 15.59 ± 10% +2.4 17.96 perf-profile.calltrace.cycles-pp.schedule.pipe_wait.pipe_read.new_sync_read.vfs_read 3.00 ± 71% +2.6 5.60 ± 6% perf-profile.calltrace.cycles-pp.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.10 ± 71% +2.7 5.77 ± 6% perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 18.05 ± 10% +2.7 20.73 perf-profile.calltrace.cycles-pp.pipe_wait.pipe_read.new_sync_read.vfs_read.ksys_read 28.21 ± 5% +3.7 31.87 perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write 24.29 ± 8% +4.7 28.98 perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write 23.38 ± 8% +4.7 28.08 perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write 23.72 ± 8% +4.7 28.43 perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write 31.18 ± 41% +12.7 43.86 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 33.44 ± 40% +13.2 46.61 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 64.81 ± 43% +27.5 92.34 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 65.30 ± 43% +27.7 92.99 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 29.39 ±104% -29.4 0.00 perf-profile.children.cycles-pp.start_thread 16.09 ±105% -16.1 0.00 perf-profile.children.cycles-pp.__GI___libc_write 12.73 ±104% -12.7 0.00 perf-profile.children.cycles-pp.__GI___libc_read 5.80 ± 10% -1.2 4.65 ± 3% perf-profile.children.cycles-pp.security_file_permission 4.10 ± 16% -1.0 3.10 ± 6% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 3.21 ± 9% -0.7 2.55 ± 3% perf-profile.children.cycles-pp.copy_page_to_iter 1.45 ± 32% -0.6 0.83 perf-profile.children.cycles-pp.entry_SYSCALL_64 3.04 ± 8% -0.6 2.44 ± 3% perf-profile.children.cycles-pp.selinux_file_permission 2.56 ± 11% -0.6 1.99 ± 2% perf-profile.children.cycles-pp.copy_page_from_iter 2.80 ± 9% -0.5 2.25 ± 2% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 3.38 ± 7% -0.5 2.87 ± 2% perf-profile.children.cycles-pp.mutex_lock 1.62 ± 12% -0.4 1.23 ± 3% perf-profile.children.cycles-pp.___might_sleep 1.82 ± 9% -0.4 1.43 ± 4% perf-profile.children.cycles-pp.syscall_return_via_sysret 2.09 ± 12% -0.4 1.72 ± 6% perf-profile.children.cycles-pp.file_has_perm 0.90 ± 20% -0.3 0.57 ± 9% perf-profile.children.cycles-pp.__mutex_lock 1.14 ± 12% -0.3 0.82 ± 3% perf-profile.children.cycles-pp.__inode_security_revalidate 1.26 ± 10% -0.3 0.97 ± 4% perf-profile.children.cycles-pp.copyin 1.70 ± 8% -0.3 1.42 ± 3% perf-profile.children.cycles-pp.copyout 1.25 ± 11% -0.3 0.98 ± 3% perf-profile.children.cycles-pp.fsnotify 1.04 ± 9% -0.3 0.79 ± 3% perf-profile.children.cycles-pp._cond_resched 1.72 ± 7% -0.2 1.48 ± 5% perf-profile.children.cycles-pp.fput_many 1.16 ± 10% -0.2 0.92 ± 4% perf-profile.children.cycles-pp.__might_sleep 1.09 ± 8% -0.2 0.85 ± 3% perf-profile.children.cycles-pp.__fsnotify_parent 0.83 ± 13% -0.2 0.63 ± 3% perf-profile.children.cycles-pp.__might_fault 0.69 ± 11% -0.1 0.56 perf-profile.children.cycles-pp.current_time 0.99 ± 4% -0.1 0.89 ± 4% perf-profile.children.cycles-pp.touch_atime 0.65 ± 6% -0.1 0.55 ± 6% perf-profile.children.cycles-pp.atime_needs_update 0.39 ± 4% -0.1 0.29 perf-profile.children.cycles-pp.rcu_all_qs 0.13 ± 39% -0.1 0.04 ± 59% perf-profile.children.cycles-pp.mutex_spin_on_owner 0.34 ± 11% -0.1 0.26 ± 9% perf-profile.children.cycles-pp.preempt_schedule_common 0.49 ± 5% -0.1 0.41 ± 2% perf-profile.children.cycles-pp.wake_up_q 0.24 ± 2% -0.1 0.17 ± 4% perf-profile.children.cycles-pp.fpregs_assert_state_consistent 0.30 ± 11% -0.1 0.23 ± 3% perf-profile.children.cycles-pp.inode_has_perm 0.23 ± 8% -0.1 0.18 ± 6% perf-profile.children.cycles-pp.__sb_end_write 0.21 ± 9% -0.1 0.16 ± 9% perf-profile.children.cycles-pp.__x64_sys_read 0.29 ± 8% -0.0 0.24 ± 2% perf-profile.children.cycles-pp.timespec64_trunc 0.21 ± 8% -0.0 0.17 ± 8% perf-profile.children.cycles-pp.__x64_sys_write 0.22 ± 7% -0.0 0.19 ± 2% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.08 ± 16% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.bpf_fd_pass 0.10 ± 4% +0.0 0.11 ± 3% perf-profile.children.cycles-pp.perf_exclude_event 0.08 ± 24% +0.0 0.13 ± 5% perf-profile.children.cycles-pp.rcu_note_context_switch 0.17 ± 19% +0.1 0.24 ± 8% perf-profile.children.cycles-pp.wakeup_preempt_entity 0.23 ± 19% +0.1 0.32 ± 3% perf-profile.children.cycles-pp.resched_curr 0.61 ± 8% +0.1 0.72 ± 6% perf-profile.children.cycles-pp.sched_clock 0.67 ± 8% +0.1 0.78 ± 6% perf-profile.children.cycles-pp.sched_clock_cpu 0.50 ± 9% +0.1 0.62 ± 3% perf-profile.children.cycles-pp.account_entity_enqueue 0.61 ± 10% +0.2 0.79 perf-profile.children.cycles-pp.account_entity_dequeue 0.43 ± 19% +0.2 0.62 ± 7% perf-profile.children.cycles-pp.clear_buddies 1.42 ± 14% +0.2 1.66 ± 4% perf-profile.children.cycles-pp.native_write_msr 1.17 ± 13% +0.3 1.42 ± 2% perf-profile.children.cycles-pp.check_preempt_wakeup 0.85 ± 23% +0.3 1.12 ± 7% perf-profile.children.cycles-pp.put_prev_entity 0.94 ± 14% +0.3 1.22 ± 5% perf-profile.children.cycles-pp.___perf_sw_event 1.45 ± 11% +0.3 1.73 perf-profile.children.cycles-pp.check_preempt_curr 1.45 ± 12% +0.3 1.76 perf-profile.children.cycles-pp.__enqueue_entity 1.60 ± 10% +0.3 1.91 perf-profile.children.cycles-pp.ttwu_do_wakeup 1.12 ± 9% +0.3 1.43 ± 3% perf-profile.children.cycles-pp.__update_load_avg_se 1.64 ± 8% +0.4 2.00 ± 4% perf-profile.children.cycles-pp.available_idle_cpu 0.96 ± 19% +0.4 1.34 ± 7% perf-profile.children.cycles-pp.pick_next_entity 1.85 ± 16% +0.4 2.28 ± 6% perf-profile.children.cycles-pp.switch_fpu_return 1.90 ± 9% +0.6 2.50 ± 3% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 3.22 ± 13% +0.6 3.87 perf-profile.children.cycles-pp.update_load_avg 4.03 ± 10% +0.6 4.68 ± 2% perf-profile.children.cycles-pp.update_curr 3.01 ± 14% +0.7 3.72 ± 2% perf-profile.children.cycles-pp.reweight_entity 5.10 ± 8% +0.8 5.87 ± 2% perf-profile.children.cycles-pp.enqueue_entity 4.71 ± 15% +1.1 5.77 ± 4% perf-profile.children.cycles-pp.pick_next_task_fair 6.73 ± 11% +1.1 7.87 perf-profile.children.cycles-pp.dequeue_task_fair 9.07 ± 9% +1.5 10.61 perf-profile.children.cycles-pp.enqueue_task_fair 9.32 ± 9% +1.6 10.89 perf-profile.children.cycles-pp.ttwu_do_activate 91.46 +1.6 93.03 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 9.27 ± 9% +1.6 10.84 perf-profile.children.cycles-pp.activate_task 4.16 ± 26% +1.6 5.78 ± 6% perf-profile.children.cycles-pp.exit_to_usermode_loop 90.79 +1.7 92.47 perf-profile.children.cycles-pp.do_syscall_64 3.48 ± 8% +1.9 5.36 ± 4% perf-profile.children.cycles-pp.select_idle_sibling 4.44 ± 10% +2.2 6.67 ± 5% perf-profile.children.cycles-pp.select_task_rq_fair 18.46 ± 9% +2.3 20.81 perf-profile.children.cycles-pp.pipe_wait 19.93 ± 11% +3.5 23.46 ± 2% perf-profile.children.cycles-pp.__schedule 28.82 ± 5% +3.6 32.45 perf-profile.children.cycles-pp.__wake_up_common_lock 19.89 ± 12% +3.7 23.62 ± 2% perf-profile.children.cycles-pp.schedule 23.96 ± 8% +4.5 28.45 perf-profile.children.cycles-pp.try_to_wake_up 24.53 ± 8% +4.5 29.06 perf-profile.children.cycles-pp.__wake_up_common 23.91 ± 8% +4.6 28.46 perf-profile.children.cycles-pp.autoremove_wake_function 2.76 ± 9% -0.5 2.23 ± 2% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 1.46 ± 18% -0.4 1.06 ± 4% perf-profile.self.cycles-pp.pipe_write 1.81 ± 9% -0.4 1.43 ± 4% perf-profile.self.cycles-pp.syscall_return_via_sysret 1.56 ± 12% -0.4 1.19 ± 3% perf-profile.self.cycles-pp.___might_sleep 2.09 ± 7% -0.3 1.81 ± 2% perf-profile.self.cycles-pp.mutex_lock 1.86 ± 8% -0.3 1.59 ± 3% perf-profile.self.cycles-pp.selinux_file_permission 1.21 ± 11% -0.3 0.96 ± 2% perf-profile.self.cycles-pp.fsnotify 1.70 ± 7% -0.2 1.46 ± 5% perf-profile.self.cycles-pp.fput_many 1.04 ± 11% -0.2 0.82 ± 3% perf-profile.self.cycles-pp.__might_sleep 1.01 ± 8% -0.2 0.80 ± 3% perf-profile.self.cycles-pp.__fsnotify_parent 1.04 ± 10% -0.2 0.83 perf-profile.self.cycles-pp.entry_SYSCALL_64 0.67 ± 10% -0.2 0.51 ± 5% perf-profile.self.cycles-pp.new_sync_write 0.64 ± 12% -0.2 0.48 ± 5% perf-profile.self.cycles-pp.vfs_write 1.67 -0.2 1.51 ± 2% perf-profile.self.cycles-pp.pipe_read 0.58 ± 17% -0.1 0.44 ± 7% perf-profile.self.cycles-pp.file_has_perm 0.44 ± 18% -0.1 0.31 ± 7% perf-profile.self.cycles-pp.__mutex_lock 0.49 ± 13% -0.1 0.35 ± 3% perf-profile.self.cycles-pp.copy_page_from_iter 0.64 ± 8% -0.1 0.52 ± 4% perf-profile.self.cycles-pp.copy_page_to_iter 0.67 ± 7% -0.1 0.55 ± 3% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.30 ± 5% -0.1 0.20 ± 4% perf-profile.self.cycles-pp.ksys_write 0.40 ± 12% -0.1 0.30 ± 7% perf-profile.self.cycles-pp.security_file_permission 0.13 ± 38% -0.1 0.04 ± 59% perf-profile.self.cycles-pp.mutex_spin_on_owner 0.36 ± 11% -0.1 0.28 ± 3% perf-profile.self.cycles-pp.__inode_security_revalidate 0.76 ± 5% -0.1 0.68 ± 5% perf-profile.self.cycles-pp.do_syscall_64 0.21 ± 2% -0.1 0.14 ± 8% perf-profile.self.cycles-pp.fpregs_assert_state_consistent 0.26 ± 12% -0.1 0.20 ± 4% perf-profile.self.cycles-pp.inode_has_perm 0.29 ± 11% -0.1 0.23 ± 6% perf-profile.self.cycles-pp.current_time 0.28 ± 4% -0.1 0.21 ± 2% perf-profile.self.cycles-pp.rcu_all_qs 0.22 ± 7% -0.1 0.17 ± 7% perf-profile.self.cycles-pp.__sb_end_write 0.31 -0.1 0.26 ± 7% perf-profile.self.cycles-pp.ksys_read 0.20 ± 11% -0.0 0.16 ± 9% perf-profile.self.cycles-pp.__x64_sys_read 0.26 ± 8% -0.0 0.22 perf-profile.self.cycles-pp.timespec64_trunc 0.20 ± 5% -0.0 0.16 ± 9% perf-profile.self.cycles-pp.__x64_sys_write 0.24 ± 9% -0.0 0.21 ± 6% perf-profile.self.cycles-pp.__might_fault 0.09 ± 9% -0.0 0.06 perf-profile.self.cycles-pp.copyin 0.14 ± 5% -0.0 0.12 ± 3% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax 0.07 ± 10% +0.0 0.09 ± 4% perf-profile.self.cycles-pp.sched_clock 0.16 ± 5% +0.0 0.18 ± 2% perf-profile.self.cycles-pp.ttwu_do_wakeup 0.20 ± 12% +0.0 0.24 ± 5% perf-profile.self.cycles-pp.__list_add_valid 0.10 ± 26% +0.0 0.14 ± 10% perf-profile.self.cycles-pp.update_cfs_rq_h_load 0.15 ± 16% +0.1 0.21 ± 11% perf-profile.self.cycles-pp.wakeup_preempt_entity 0.35 ± 7% +0.1 0.43 ± 4% perf-profile.self.cycles-pp.prepare_to_wait 0.55 ± 2% +0.1 0.63 ± 3% perf-profile.self.cycles-pp.__fget_light 0.22 ± 18% +0.1 0.30 ± 3% perf-profile.self.cycles-pp.resched_curr 0.48 ± 11% +0.1 0.58 ± 3% perf-profile.self.cycles-pp.check_preempt_wakeup 0.39 ± 11% +0.1 0.50 perf-profile.self.cycles-pp.account_entity_enqueue 0.85 ± 7% +0.1 0.97 ± 4% perf-profile.self.cycles-pp.enqueue_entity 0.43 ± 16% +0.1 0.55 ± 7% perf-profile.self.cycles-pp.pick_next_entity 0.86 ± 10% +0.1 0.99 perf-profile.self.cycles-pp.dequeue_task_fair 0.37 ± 21% +0.2 0.55 ± 6% perf-profile.self.cycles-pp.clear_buddies 0.45 ± 14% +0.2 0.65 perf-profile.self.cycles-pp.account_entity_dequeue 1.41 ± 14% +0.2 1.65 ± 4% perf-profile.self.cycles-pp.native_write_msr 0.80 ± 16% +0.3 1.06 ± 4% perf-profile.self.cycles-pp.___perf_sw_event 1.09 ± 9% +0.3 1.39 ± 2% perf-profile.self.cycles-pp.__update_load_avg_se 0.84 ± 16% +0.3 1.14 ± 7% perf-profile.self.cycles-pp.select_task_rq_fair 1.45 ± 12% +0.3 1.75 perf-profile.self.cycles-pp.__enqueue_entity 1.62 ± 8% +0.4 1.99 ± 4% perf-profile.self.cycles-pp.available_idle_cpu 1.83 ± 16% +0.4 2.25 ± 6% perf-profile.self.cycles-pp.switch_fpu_return 2.15 ± 11% +0.5 2.60 ± 5% perf-profile.self.cycles-pp.update_curr 1.88 ± 9% +0.6 2.45 ± 3% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 1.21 ± 9% +0.7 1.92 ± 3% perf-profile.self.cycles-pp._raw_spin_lock 0.80 ± 11% +1.4 2.17 ± 3% perf-profile.self.cycles-pp.select_idle_sibling hackbench.throughput 74000 +-+-----------------------------------------------------------------+ 72000 +-+.+ .+ | |.+ : .+. .+. + | 70000 +-+ : + + +. : | 68000 +-+ + + : | | :: | 66000 +-+ + | 64000 +-+ | 62000 +-+ | | | 60000 +-+ O O OO O O O O O O O OO | 58000 +-+ O O O O O O O O O O O | O O OO O O | 56000 O-+ O O O O O O | 54000 +-O-----------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-hsw-ep4: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory ========================================================================================= compiler/cpufreq_governor/ipc/iterations/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode: gcc-7/performance/pipe/18/x86_64-rhel-7.6/threads/1600%/debian-x86_64-2019-11-14.cgz/lkp-hsw-ep4/hackbench/0x43 commit: 43e9f7f231 ("sched/fair: Start tracking SCHED_IDLE tasks count in cfs_rq") 3c29e651e1 ("sched/fair: Fall back to sched-idle CPU if idle CPU isn't found") 43e9f7f231e40e45 3c29e651e16dd3b3179cfb2d055 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:4 dmesg.WARNING:at#for_ip_interrupt_entry/0x %stddev %change %stddev \ | \ 176075 ± 2% -11.3% 156152 hackbench.throughput 891.79 +12.3% 1001 hackbench.time.elapsed_time 891.79 +12.3% 1001 hackbench.time.elapsed_time.max 1.889e+09 ± 9% +33.1% 2.514e+09 ± 3% hackbench.time.involuntary_context_switches 42939 ± 3% +17.0% 50220 hackbench.time.system_time 18978 +3.2% 19592 hackbench.time.user_time 2.558e+09 ± 8% +33.5% 3.415e+09 ± 2% hackbench.time.voluntary_context_switches 43133704 ± 3% +39.2% 60035429 ± 33% cpuidle.C1.time 48661 +17.7% 57274 ± 5% meminfo.Shmem 19043 ± 2% -14.0% 16377 meminfo.max_used_kB 2502 +1.3% 2535 turbostat.Avg_MHz 3.368e+08 ± 5% +30.6% 4.398e+08 turbostat.IRQ 66.25 +4.5% 69.25 vmstat.cpu.sy 29.25 ± 2% -8.5% 26.75 vmstat.cpu.us 5000443 ± 7% +18.6% 5931772 vmstat.system.cs 370173 ± 3% +17.1% 433616 vmstat.system.in 98786923 ± 7% +74.5% 1.724e+08 numa-numastat.node0.local_node 98791606 ± 7% +74.5% 1.724e+08 numa-numastat.node0.numa_hit 4686 ± 99% +297.8% 18642 ± 24% numa-numastat.node0.other_node 1.818e+08 -51.4% 88447082 ± 3% numa-numastat.node1.local_node 1.819e+08 -51.4% 88451833 ± 3% numa-numastat.node1.numa_hit 18694 ± 25% -74.6% 4753 ± 97% numa-numastat.node1.other_node 155316 +1.5% 157632 proc-vmstat.nr_active_anon 99.50 +5.3% 104.75 ± 3% proc-vmstat.nr_dirtied 4440 +4.9% 4656 proc-vmstat.nr_inactive_anon 6283 +1.9% 6399 proc-vmstat.nr_mapped 12248 ± 2% +16.6% 14276 ± 5% proc-vmstat.nr_shmem 98.75 +5.3% 104.00 proc-vmstat.nr_written 155316 +1.5% 157632 proc-vmstat.nr_zone_active_anon 4440 +4.9% 4656 proc-vmstat.nr_zone_inactive_anon 2.807e+08 -7.1% 2.609e+08 proc-vmstat.numa_hit 2.807e+08 -7.1% 2.609e+08 proc-vmstat.numa_local 2.83e+08 -7.0% 2.632e+08 proc-vmstat.pgalloc_normal 3212744 +2.3% 3286078 proc-vmstat.pgfault 2.828e+08 -7.0% 2.631e+08 proc-vmstat.pgfree 39216 ± 35% +153.3% 99342 ± 4% numa-vmstat.node0.nr_active_anon 31864 ± 44% +211.0% 99111 ± 4% numa-vmstat.node0.nr_anon_pages 11183 ± 3% +4212.8% 482321 ± 2% numa-vmstat.node0.nr_kernel_stack 499.50 ± 23% +66.9% 833.75 ± 8% numa-vmstat.node0.nr_page_table_pages 11417 -63.5% 4168 ± 5% numa-vmstat.node0.nr_shmem 15905 ± 4% -12.5% 13919 ± 2% numa-vmstat.node0.nr_slab_reclaimable 17996 ± 3% +587.4% 123708 ± 2% numa-vmstat.node0.nr_slab_unreclaimable 39216 ± 35% +153.3% 99342 ± 4% numa-vmstat.node0.nr_zone_active_anon 52471566 ± 6% +73.3% 90915199 numa-vmstat.node0.numa_hit 52466577 ± 6% +73.2% 90896366 numa-vmstat.node0.numa_local 4990 ± 92% +277.5% 18836 ± 24% numa-vmstat.node0.numa_other 114712 ± 12% -50.1% 57234 ± 7% numa-vmstat.node1.nr_active_anon 114545 ± 12% -58.2% 47874 ± 9% numa-vmstat.node1.nr_anon_pages 462075 -96.6% 15717 ± 67% numa-vmstat.node1.nr_kernel_stack 871.50 ± 13% -37.9% 541.00 ± 13% numa-vmstat.node1.nr_page_table_pages 691.25 ± 22% +1359.2% 10086 ± 8% numa-vmstat.node1.nr_shmem 11357 ± 5% +18.1% 13414 ± 4% numa-vmstat.node1.nr_slab_reclaimable 118594 -85.5% 17243 ± 16% numa-vmstat.node1.nr_slab_unreclaimable 114712 ± 12% -50.1% 57234 ± 7% numa-vmstat.node1.nr_zone_active_anon 93308245 ± 2% -49.7% 46921467 ± 4% numa-vmstat.node1.numa_hit 93128224 ± 2% -49.8% 46755145 ± 4% numa-vmstat.node1.numa_local 157762 ± 35% +154.4% 401333 ± 4% numa-meminfo.node0.Active 157721 ± 35% +154.4% 401249 ± 4% numa-meminfo.node0.Active(anon) 128232 ± 44% +212.2% 400337 ± 4% numa-meminfo.node0.AnonPages 63487 ± 4% -12.0% 55850 ± 2% numa-meminfo.node0.KReclaimable 11691 ± 3% +4087.3% 489553 numa-meminfo.node0.KernelStack 1170596 ± 4% +104.2% 2390602 numa-meminfo.node0.MemUsed 2003 ± 23% +67.3% 3351 ± 8% numa-meminfo.node0.PageTables 63487 ± 4% -12.0% 55850 ± 2% numa-meminfo.node0.SReclaimable 72301 ± 3% +591.9% 500285 numa-meminfo.node0.SUnreclaim 45749 -63.5% 16679 ± 5% numa-meminfo.node0.Shmem 135789 ± 3% +309.6% 556136 numa-meminfo.node0.Slab 460025 ± 11% -50.2% 228882 ± 7% numa-meminfo.node1.Active 459902 ± 11% -50.2% 228802 ± 7% numa-meminfo.node1.Active(anon) 459237 ± 11% -58.3% 191354 ± 9% numa-meminfo.node1.AnonPages 45568 ± 5% +17.8% 53657 ± 4% numa-meminfo.node1.KReclaimable 468955 -96.7% 15461 ± 67% numa-meminfo.node1.KernelStack 2241450 -51.9% 1078199 ± 3% numa-meminfo.node1.MemUsed 3493 ± 13% -38.0% 2167 ± 13% numa-meminfo.node1.PageTables 45568 ± 5% +17.8% 53657 ± 4% numa-meminfo.node1.SReclaimable 479010 -85.6% 68750 ± 16% numa-meminfo.node1.SUnreclaim 2779 ± 23% +1354.1% 40419 ± 8% numa-meminfo.node1.Shmem 524580 -76.7% 122408 ± 8% numa-meminfo.node1.Slab 250988 ± 19% -52.0% 120407 ± 33% sched_debug.cfs_rq:/.MIN_vruntime.avg 1469922 ± 14% -32.1% 998119 ± 33% sched_debug.cfs_rq:/.MIN_vruntime.stddev 20766 ± 9% -25.3% 15519 ± 10% sched_debug.cfs_rq:/.load.avg 43820 ± 19% -40.6% 26044 ± 36% sched_debug.cfs_rq:/.load.stddev 250988 ± 19% -52.0% 120407 ± 33% sched_debug.cfs_rq:/.max_vruntime.avg 1469922 ± 14% -32.1% 998119 ± 33% sched_debug.cfs_rq:/.max_vruntime.stddev 10990567 ± 9% +22.9% 13509735 ± 7% sched_debug.cfs_rq:/.min_vruntime.stddev 102.40 -33.3% 68.27 ± 57% sched_debug.cfs_rq:/.removed.load_avg.max 4711 -33.4% 3136 ± 57% sched_debug.cfs_rq:/.removed.runnable_sum.max 19930 ± 10% -25.1% 14917 ± 10% sched_debug.cfs_rq:/.runnable_weight.avg -5713498 -349.0% 14226386 ± 8% sched_debug.cfs_rq:/.spread0.avg 8106715 ± 13% +283.1% 31057182 ± 6% sched_debug.cfs_rq:/.spread0.max -18025645 -99.7% -45393 sched_debug.cfs_rq:/.spread0.min 11027552 ± 9% +21.5% 13398318 ± 7% sched_debug.cfs_rq:/.spread0.stddev 570.46 ± 5% +109.2% 1193 ± 21% sched_debug.cfs_rq:/.util_est_enqueued.avg 2898 ± 11% +90.0% 5507 ± 10% sched_debug.cfs_rq:/.util_est_enqueued.max 563.27 ± 13% +149.3% 1404 ± 18% sched_debug.cfs_rq:/.util_est_enqueued.stddev 155578 ± 22% +84.1% 286457 ± 28% sched_debug.cpu.avg_idle.min 1678 ± 43% +105.9% 3457 ± 31% sched_debug.cpu.clock.stddev 1678 ± 43% +105.9% 3457 ± 31% sched_debug.cpu.clock_task.stddev 50207 -12.3% 44017 ± 7% sched_debug.cpu.curr->pid.avg 82673 -19.5% 66514 ± 5% sched_debug.cpu.curr->pid.max 746.05 ± 11% +554.3% 4881 ± 8% sched_debug.cpu.curr->pid.min 30728 ± 2% -40.8% 18187 ± 10% sched_debug.cpu.curr->pid.stddev 0.00 ± 43% +106.0% 0.00 ± 31% sched_debug.cpu.next_balance.stddev 20.49 ± 6% +149.0% 51.03 ± 23% sched_debug.cpu.nr_running.avg 157.83 ± 9% +77.0% 279.37 ± 8% sched_debug.cpu.nr_running.max 29.95 ± 11% +137.1% 70.99 ± 16% sched_debug.cpu.nr_running.stddev 29803727 ± 9% +29.6% 38623870 ± 3% sched_debug.cpu.nr_switches.avg 51549476 ± 10% +22.5% 63171598 ± 3% sched_debug.cpu.nr_switches.max 12792570 ± 3% +43.5% 18362678 ± 4% sched_debug.cpu.nr_switches.min 2.03 ± 80% +430.9% 10.76 ± 38% sched_debug.cpu.nr_uninterruptible.avg 6.99 ± 4% +48.7% 10.39 ± 5% perf-stat.i.MPKI 7.49 ± 5% -1.2 6.29 ± 5% perf-stat.i.cache-miss-rate% 27202880 ± 2% +46.9% 39966416 ± 2% perf-stat.i.cache-misses 4.662e+08 ± 5% +71.5% 7.998e+08 ± 5% perf-stat.i.cache-references 5989366 ± 4% +12.9% 6760958 ± 5% perf-stat.i.context-switches 197319 ± 12% -19.2% 159504 ± 3% perf-stat.i.cpu-migrations 7415 ± 2% -23.3% 5686 ± 3% perf-stat.i.cycles-between-cache-misses 0.61 ± 35% -0.3 0.35 ± 4% perf-stat.i.dTLB-load-miss-rate% 1.506e+08 ± 38% -44.9% 82967539 ± 5% perf-stat.i.dTLB-load-misses 0.22 ± 4% -0.0 0.20 ± 4% perf-stat.i.dTLB-store-miss-rate% 35974976 ± 5% -13.4% 31146077 ± 6% perf-stat.i.dTLB-store-misses 1.627e+10 -3.7% 1.566e+10 perf-stat.i.dTLB-stores 46665620 ± 3% -14.6% 39838962 ± 5% perf-stat.i.iTLB-load-misses 2191 ± 5% +21.4% 2660 ± 7% perf-stat.i.instructions-per-iTLB-miss 0.47 -2.5% 0.46 perf-stat.i.ipc 6660 ± 8% -19.1% 5391 ± 8% perf-stat.i.minor-faults 62.93 ± 2% +11.1 74.06 perf-stat.i.node-load-miss-rate% 12537290 ± 4% +73.5% 21750457 ± 4% perf-stat.i.node-load-misses 5459308 ± 3% -7.4% 5057404 ± 3% perf-stat.i.node-loads 29.39 ± 2% -4.1 25.29 ± 2% perf-stat.i.node-store-miss-rate% 2863317 +13.6% 3252632 ± 2% perf-stat.i.node-store-misses 6381925 ± 2% +53.5% 9796918 ± 4% perf-stat.i.node-stores 6638 ± 8% -19.4% 5348 ± 8% perf-stat.i.page-faults 4.89 ± 4% +78.4% 8.73 perf-stat.overall.MPKI 1.29 +0.0 1.31 perf-stat.overall.branch-miss-rate% 6.13 ± 4% -1.3 4.85 ± 3% perf-stat.overall.cache-miss-rate% 2.15 +2.0% 2.19 perf-stat.overall.cpi 7187 -27.9% 5182 ± 3% perf-stat.overall.cycles-between-cache-misses 0.63 ± 39% -0.3 0.36 ± 5% perf-stat.overall.dTLB-load-miss-rate% 0.23 ± 5% -0.0 0.21 ± 3% perf-stat.overall.dTLB-store-miss-rate% 75.65 -1.1 74.59 perf-stat.overall.iTLB-load-miss-rate% 1604 ± 3% +15.9% 1859 perf-stat.overall.instructions-per-iTLB-miss 0.47 -2.0% 0.46 perf-stat.overall.ipc 63.14 ± 2% +13.6 76.70 perf-stat.overall.node-load-miss-rate% 30.74 ± 2% -5.1 25.62 ± 2% perf-stat.overall.node-store-miss-rate% 47754 ± 2% +11.7% 53321 perf-stat.overall.path-length 24893507 +40.6% 34993733 ± 2% perf-stat.ps.cache-misses 4.072e+08 ± 5% +77.1% 7.212e+08 perf-stat.ps.cache-references 4968881 ± 7% +19.0% 5911450 perf-stat.ps.context-switches 1.789e+11 +1.3% 1.812e+11 perf-stat.ps.cpu-cycles 1.59e+08 ± 39% -44.1% 88801351 ± 6% perf-stat.ps.dTLB-load-misses 2.502e+10 -2.0% 2.453e+10 perf-stat.ps.dTLB-loads 39138578 ± 5% -12.6% 34190129 ± 3% perf-stat.ps.dTLB-store-misses 1.675e+10 -3.7% 1.613e+10 perf-stat.ps.dTLB-stores 51901139 ± 2% -14.3% 44461197 perf-stat.ps.iTLB-load-misses 16711264 ± 2% -9.3% 15153636 ± 2% perf-stat.ps.iTLB-loads 3573 -9.1% 3247 perf-stat.ps.minor-faults 10427704 ± 3% +73.2% 18055799 ± 3% perf-stat.ps.node-load-misses 6085553 ± 4% -10.0% 5479559 ± 2% perf-stat.ps.node-loads 2566511 +13.4% 2911367 ± 2% perf-stat.ps.node-store-misses 5784179 ± 2% +46.2% 8454210 ± 3% perf-stat.ps.node-stores 3573 -9.1% 3247 perf-stat.ps.page-faults 7.427e+13 ± 2% +11.7% 8.293e+13 perf-stat.total.instructions 127394 ± 5% +19.1% 151665 softirqs.CPU0.RCU 321761 ± 3% +12.8% 363107 ± 3% softirqs.CPU0.TIMER 124526 ± 5% +20.2% 149697 ± 3% softirqs.CPU1.RCU 123768 ± 4% +24.7% 154332 ± 2% softirqs.CPU10.RCU 130315 ± 3% +22.1% 159111 softirqs.CPU11.RCU 126684 ± 4% +26.1% 159755 ± 2% softirqs.CPU12.RCU 124649 ± 9% +27.8% 159298 ± 3% softirqs.CPU13.RCU 124528 ± 4% +23.2% 153372 ± 2% softirqs.CPU14.RCU 326150 ± 2% +9.8% 357991 ± 4% softirqs.CPU14.TIMER 121923 ± 6% +17.4% 143119 softirqs.CPU15.RCU 122114 ± 5% +16.7% 142560 softirqs.CPU16.RCU 121425 ± 6% +14.4% 138861 ± 3% softirqs.CPU17.RCU 330948 ± 2% +22.2% 404570 ± 16% softirqs.CPU17.TIMER 312317 ± 4% +18.2% 369091 ± 2% softirqs.CPU18.TIMER 9626 ± 2% +15.0% 11067 ± 3% softirqs.CPU19.SCHED 322308 ± 3% +21.1% 390454 ± 2% softirqs.CPU19.TIMER 124281 ± 5% +22.8% 152670 ± 2% softirqs.CPU2.RCU 9452 ± 2% +15.6% 10927 ± 3% softirqs.CPU20.SCHED 318575 ± 6% +22.3% 389741 ± 3% softirqs.CPU20.TIMER 9327 ± 3% +16.7% 10886 ± 3% softirqs.CPU21.SCHED 306628 ± 4% +23.3% 377962 ± 3% softirqs.CPU21.TIMER 9424 ± 2% +17.6% 11084 ± 3% softirqs.CPU22.SCHED 302796 ± 4% +24.2% 376126 ± 2% softirqs.CPU22.TIMER 9798 ± 2% +11.4% 10918 ± 4% softirqs.CPU23.SCHED 312009 ± 4% +23.6% 385626 ± 2% softirqs.CPU23.TIMER 9559 ± 3% +13.2% 10822 softirqs.CPU24.SCHED 311772 ± 4% +23.0% 383504 ± 2% softirqs.CPU24.TIMER 9514 ± 4% +13.2% 10766 ± 3% softirqs.CPU25.SCHED 307640 ± 4% +22.4% 376479 ± 2% softirqs.CPU25.TIMER 9573 ± 3% +13.8% 10897 ± 3% softirqs.CPU26.SCHED 314607 ± 9% +21.1% 380918 ± 2% softirqs.CPU26.TIMER 9385 +13.7% 10668 ± 2% softirqs.CPU27.SCHED 301949 ± 4% +25.2% 378140 ± 3% softirqs.CPU27.TIMER 9483 ± 2% +14.7% 10874 ± 2% softirqs.CPU28.SCHED 299462 ± 3% +24.9% 373896 ± 2% softirqs.CPU28.TIMER 9505 ± 3% +13.3% 10769 ± 3% softirqs.CPU29.SCHED 307087 ± 4% +23.8% 380066 ± 3% softirqs.CPU29.TIMER 124992 ± 4% +19.4% 149253 ± 3% softirqs.CPU3.RCU 9630 ± 4% +13.6% 10942 ± 3% softirqs.CPU30.SCHED 306382 ± 4% +25.4% 384355 ± 2% softirqs.CPU30.TIMER 9496 ± 6% +12.6% 10689 ± 2% softirqs.CPU31.SCHED 308689 ± 4% +24.5% 384419 ± 2% softirqs.CPU31.TIMER 9714 +12.6% 10937 ± 4% softirqs.CPU32.SCHED 304137 ± 3% +35.9% 413240 ± 13% softirqs.CPU32.TIMER 9598 ± 2% +13.4% 10882 ± 5% softirqs.CPU33.SCHED 9622 +13.8% 10949 ± 3% softirqs.CPU34.SCHED 305882 ± 4% +27.3% 389315 ± 4% softirqs.CPU34.TIMER 9409 ± 4% +15.3% 10853 ± 4% softirqs.CPU35.SCHED 306215 ± 4% +23.8% 379222 ± 2% softirqs.CPU35.TIMER 321617 ± 4% +12.9% 362966 ± 3% softirqs.CPU36.TIMER 134843 ± 6% +19.4% 161053 ± 2% softirqs.CPU37.RCU 130954 ± 6% +20.4% 157716 softirqs.CPU38.RCU 129884 ± 5% +17.8% 152978 softirqs.CPU39.RCU 124744 ± 4% +20.1% 149790 ± 3% softirqs.CPU4.RCU 130593 ± 5% +17.2% 153092 ± 2% softirqs.CPU40.RCU 134185 ± 5% +19.9% 160871 softirqs.CPU41.RCU 133336 ± 5% +18.1% 157409 softirqs.CPU42.RCU 134486 ± 4% +18.7% 159613 softirqs.CPU43.RCU 129857 ± 6% +18.5% 153887 softirqs.CPU44.RCU 131816 ± 5% +20.0% 158144 softirqs.CPU45.RCU 132691 ± 5% +22.4% 162412 softirqs.CPU46.RCU 138736 ± 3% +18.7% 164668 softirqs.CPU47.RCU 132971 ± 6% +24.2% 165142 softirqs.CPU48.RCU 136142 ± 3% +20.8% 164493 softirqs.CPU49.RCU 125011 ± 4% +23.6% 154462 ± 2% softirqs.CPU5.RCU 133660 ± 4% +19.5% 159753 ± 2% softirqs.CPU50.RCU 140405 ± 4% +19.5% 167772 softirqs.CPU51.RCU 140523 ± 4% +18.5% 166576 softirqs.CPU52.RCU 138849 ± 5% +14.6% 159157 ± 4% softirqs.CPU53.RCU 332171 ± 2% +21.4% 403260 ± 15% softirqs.CPU53.TIMER 133037 ± 4% +14.0% 151688 ± 2% softirqs.CPU54.RCU 9584 +14.3% 10959 ± 2% softirqs.CPU54.SCHED 310603 ± 4% +18.0% 366472 ± 3% softirqs.CPU54.TIMER 9565 ± 2% +13.8% 10889 ± 5% softirqs.CPU55.SCHED 321231 ± 3% +21.3% 389753 ± 2% softirqs.CPU55.TIMER 136633 ± 4% +12.6% 153854 softirqs.CPU56.RCU 9430 ± 2% +15.6% 10898 ± 5% softirqs.CPU56.SCHED 319477 ± 6% +22.1% 390107 ± 2% softirqs.CPU56.TIMER 128672 ± 4% +14.1% 146791 ± 3% softirqs.CPU57.RCU 9493 ± 2% +13.1% 10740 ± 3% softirqs.CPU57.SCHED 305670 ± 4% +23.5% 377597 ± 2% softirqs.CPU57.TIMER 127129 ± 4% +15.6% 146993 ± 2% softirqs.CPU58.RCU 9529 ± 3% +13.6% 10824 ± 4% softirqs.CPU58.SCHED 302317 ± 4% +23.8% 374371 ± 3% softirqs.CPU58.TIMER 9427 ± 2% +14.0% 10743 ± 2% softirqs.CPU59.SCHED 310985 ± 4% +23.5% 384119 ± 3% softirqs.CPU59.TIMER 124937 ± 4% +21.6% 151962 ± 3% softirqs.CPU6.RCU 9435 ± 2% +14.1% 10762 ± 3% softirqs.CPU60.SCHED 311646 ± 4% +22.8% 382731 ± 3% softirqs.CPU60.TIMER 9585 +13.5% 10877 ± 2% softirqs.CPU61.SCHED 306857 ± 4% +22.8% 376944 ± 2% softirqs.CPU61.TIMER 9483 ± 2% +12.5% 10671 ± 5% softirqs.CPU62.SCHED 9620 ± 4% +9.7% 10556 ± 4% softirqs.CPU63.SCHED 301085 ± 3% +24.7% 375397 ± 3% softirqs.CPU63.TIMER 300443 ± 4% +23.7% 371620 ± 2% softirqs.CPU64.TIMER 9667 +11.2% 10751 ± 4% softirqs.CPU65.SCHED 307194 ± 4% +23.4% 379020 ± 3% softirqs.CPU65.TIMER 9493 ± 3% +14.9% 10909 ± 3% softirqs.CPU66.SCHED 306226 ± 4% +25.1% 383140 ± 3% softirqs.CPU66.TIMER 9674 ± 3% +13.8% 11010 ± 3% softirqs.CPU67.SCHED 307870 ± 4% +25.0% 384793 ± 2% softirqs.CPU67.TIMER 9451 ± 2% +14.2% 10792 ± 4% softirqs.CPU68.SCHED 301873 ± 3% +26.1% 380748 softirqs.CPU68.TIMER 9542 +13.8% 10863 ± 3% softirqs.CPU69.SCHED 316206 ± 8% +20.5% 380893 ± 2% softirqs.CPU69.TIMER 128158 ± 4% +20.4% 154294 ± 2% softirqs.CPU7.RCU 9456 ± 3% +13.5% 10734 ± 3% softirqs.CPU70.SCHED 305361 ± 4% +36.4% 416546 ± 15% softirqs.CPU70.TIMER 9520 ± 3% +15.4% 10985 ± 4% softirqs.CPU71.SCHED 305605 ± 4% +24.6% 380653 ± 2% softirqs.CPU71.TIMER 122831 ± 4% +21.6% 149424 ± 2% softirqs.CPU8.RCU 123870 ± 4% +23.0% 152396 ± 2% softirqs.CPU9.RCU 9357957 ± 5% +15.4% 10794953 softirqs.RCU 23209193 ± 3% +15.5% 26806870 ± 3% softirqs.TIMER 4.79 ± 4% -3.1 1.71 ± 16% perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.85 ± 4% -2.4 1.41 ± 16% perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.39 ± 4% -2.0 1.35 ± 16% perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_read.new_sync_read.vfs_read.ksys_read 2.93 ± 5% -1.9 0.99 ± 18% perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.new_sync_write.vfs_write.ksys_write 3.25 ± 4% -1.9 1.31 ± 10% perf-profile.calltrace.cycles-pp.mutex_unlock.pipe_write.new_sync_write.vfs_write.ksys_write 2.14 ± 4% -1.2 0.91 ± 14% perf-profile.calltrace.cycles-pp.selinux_file_permission.security_file_permission.vfs_read.ksys_read.do_syscall_64 2.06 ± 4% -1.2 0.85 ± 13% perf-profile.calltrace.cycles-pp.selinux_file_permission.security_file_permission.vfs_write.ksys_write.do_syscall_64 1.52 ± 3% -1.0 0.49 ± 59% perf-profile.calltrace.cycles-pp.copyout.copy_page_to_iter.pipe_read.new_sync_read.vfs_read 1.46 ± 3% -1.0 0.46 ± 59% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout.copy_page_to_iter.pipe_read.new_sync_read 1.53 ± 8% -0.8 0.68 ± 12% perf-profile.calltrace.cycles-pp.touch_atime.pipe_read.new_sync_read.vfs_read.ksys_read 1.65 ± 4% -0.8 0.81 ± 10% perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.45 ± 5% -0.7 0.74 ± 10% perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.35 ± 5% -0.7 0.68 ± 10% perf-profile.calltrace.cycles-pp.__fget.__fget_light.__fdget_pos.ksys_write.do_syscall_64 1.28 ± 8% -0.7 0.62 ± 12% perf-profile.calltrace.cycles-pp.file_update_time.pipe_write.new_sync_write.vfs_write.ksys_write 1.29 ± 3% -0.6 0.67 ± 9% perf-profile.calltrace.cycles-pp.mutex_lock.pipe_write.new_sync_write.vfs_write.ksys_write 0.64 ± 11% +0.3 0.96 ± 7% perf-profile.calltrace.cycles-pp.__lock_text_start.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write 0.00 +0.5 0.55 ± 3% perf-profile.calltrace.cycles-pp.pick_next_entity.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop 0.00 +0.6 0.55 ± 5% perf-profile.calltrace.cycles-pp.pick_next_entity.pick_next_task_fair.__schedule.schedule.pipe_wait 0.00 +0.6 0.57 ± 6% perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate 0.00 +0.6 0.61 ± 7% perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.dequeue_task_fair.__schedule.schedule 0.00 +0.7 0.69 ± 8% perf-profile.calltrace.cycles-pp.finish_task_switch.__schedule.schedule.pipe_wait.pipe_read 0.00 +0.7 0.70 ± 6% perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.pipe_wait 0.00 +0.7 0.71 ± 11% perf-profile.calltrace.cycles-pp.__enqueue_entity.put_prev_entity.pick_next_task_fair.__schedule.schedule 0.00 +0.7 0.72 ± 7% perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 0.00 +0.7 0.72 ± 6% perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.dequeue_task_fair.__schedule.schedule.pipe_wait 0.00 +0.8 0.76 ± 10% perf-profile.calltrace.cycles-pp.__enqueue_entity.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 0.00 +0.8 0.77 ± 2% perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule 0.00 +0.8 0.77 ± 5% perf-profile.calltrace.cycles-pp.__switch_to 0.00 +0.8 0.77 ± 6% perf-profile.calltrace.cycles-pp.___perf_sw_event.__schedule.schedule.pipe_wait.pipe_read 0.00 +0.8 0.79 ± 5% perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_entity.dequeue_task_fair.__schedule.schedule 0.00 +0.8 0.81 ± 4% perf-profile.calltrace.cycles-pp.mutex_lock.pipe_wait.pipe_read.new_sync_read.vfs_read 0.00 +0.8 0.82 ± 6% perf-profile.calltrace.cycles-pp.update_rq_clock.__schedule.schedule.pipe_wait.pipe_read 0.00 +0.8 0.82 ± 6% perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 0.00 +0.9 0.90 ± 3% perf-profile.calltrace.cycles-pp.find_next_bit.cpumask_next_wrap.select_idle_sibling.select_task_rq_fair.try_to_wake_up 0.00 +0.9 0.91 ± 4% perf-profile.calltrace.cycles-pp.update_rq_clock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 0.00 +0.9 0.91 ± 6% perf-profile.calltrace.cycles-pp.check_preempt_wakeup.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function 0.00 +0.9 0.92 ± 2% perf-profile.calltrace.cycles-pp.native_write_msr 0.00 +1.1 1.05 ± 5% perf-profile.calltrace.cycles-pp.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common 0.00 +1.1 1.08 ± 8% perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop 0.00 +1.1 1.13 ± 5% perf-profile.calltrace.cycles-pp.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 0.00 +1.2 1.17 ± 9% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common 0.00 +1.2 1.18 ± 2% perf-profile.calltrace.cycles-pp.__switch_to_asm 0.00 +1.2 1.20 ± 9% perf-profile.calltrace.cycles-pp._raw_spin_lock.__schedule.schedule.pipe_wait.pipe_read 0.00 +1.3 1.30 ± 5% perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 0.00 +1.3 1.34 ± 6% perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.pipe_wait 0.00 +1.6 1.62 ± 4% perf-profile.calltrace.cycles-pp.switch_fpu_return.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +1.9 1.88 ± 5% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.pipe_wait.pipe_read 0.00 +1.9 1.93 ± 8% perf-profile.calltrace.cycles-pp._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 0.13 ±173% +2.0 2.11 ± 5% perf-profile.calltrace.cycles-pp.cpumask_next_wrap.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function 0.00 +2.1 2.06 ± 4% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.pipe_wait 0.00 +2.4 2.38 ± 4% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop.do_syscall_64 0.00 +2.7 2.65 ± 4% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 0.35 ±103% +4.3 4.67 ± 4% perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.22 ±173% +4.6 4.81 ± 4% perf-profile.calltrace.cycles-pp.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.23 ±173% +4.7 4.97 ± 4% perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.72 ± 23% +5.0 5.72 ± 5% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_wait.pipe_read 0.69 ± 17% +5.6 6.25 ± 4% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function 0.74 ± 17% +5.8 6.50 ± 4% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common 0.74 ± 18% +5.8 6.55 ± 5% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 15.99 ± 54% +7.5 23.51 ± 6% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 13.13 +7.7 20.86 ± 3% perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 11.49 +8.7 20.19 ± 3% perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64 1.31 ± 27% +8.9 10.21 ± 7% perf-profile.calltrace.cycles-pp.available_idle_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function 1.81 ± 22% +11.2 12.98 ± 5% perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_wait.pipe_read.new_sync_read 1.89 ± 22% +11.4 13.28 ± 5% perf-profile.calltrace.cycles-pp.schedule.pipe_wait.pipe_read.new_sync_read.vfs_read 2.18 ± 21% +13.1 15.28 ± 4% perf-profile.calltrace.cycles-pp.pipe_wait.pipe_read.new_sync_read.vfs_read.ksys_read 2.01 ± 25% +19.1 21.10 ± 7% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common 2.38 ± 23% +20.0 22.40 ± 7% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 19.01 ± 3% +23.4 42.38 ± 4% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 21.13 ± 53% +23.8 44.92 ± 5% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 17.85 ± 4% +23.9 41.76 ± 4% perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64 19.16 ± 53% +24.6 43.74 ± 5% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.79 ± 22% +30.7 34.45 ± 6% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write 3.84 ± 23% +30.9 34.69 ± 6% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write 4.13 ± 21% +31.0 35.10 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write 5.77 ± 14% +31.0 36.76 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write 9.20 ± 6% -6.0 3.16 ± 12% perf-profile.children.cycles-pp.syscall_return_via_sysret 8.88 ± 3% -5.8 3.03 ± 12% perf-profile.children.cycles-pp.entry_SYSCALL_64 8.72 ± 4% -5.3 3.39 ± 11% perf-profile.children.cycles-pp.security_file_permission 4.27 ± 3% -2.4 1.89 ± 9% perf-profile.children.cycles-pp.selinux_file_permission 3.15 ± 5% -2.1 1.07 ± 12% perf-profile.children.cycles-pp.file_has_perm 3.48 ± 3% -2.0 1.46 ± 10% perf-profile.children.cycles-pp.copy_page_to_iter 3.80 ± 3% -2.0 1.79 ± 6% perf-profile.children.cycles-pp.mutex_unlock 3.02 ± 5% -1.9 1.11 ± 11% perf-profile.children.cycles-pp.copy_page_from_iter 3.31 ± 4% -1.6 1.75 ± 6% perf-profile.children.cycles-pp.__fdget_pos 2.55 ± 3% -1.5 1.02 ± 10% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 3.17 ± 4% -1.5 1.70 ± 6% perf-profile.children.cycles-pp.__fget_light 2.10 ± 7% -1.4 0.71 ± 12% perf-profile.children.cycles-pp.fsnotify 2.52 ± 6% -1.3 1.23 ± 8% perf-profile.children.cycles-pp.__fget 1.77 ± 6% -1.1 0.66 ± 9% perf-profile.children.cycles-pp.avc_has_perm 1.89 ± 5% -1.1 0.78 ± 10% perf-profile.children.cycles-pp.___might_sleep 1.44 ± 4% -1.0 0.48 ± 12% perf-profile.children.cycles-pp.__inode_security_revalidate 1.33 ± 10% -0.9 0.40 ± 16% perf-profile.children.cycles-pp.current_time 1.56 ± 3% -0.9 0.66 ± 9% perf-profile.children.cycles-pp.copyout 1.56 ± 8% -0.8 0.73 ± 7% perf-profile.children.cycles-pp.touch_atime 1.33 ± 4% -0.8 0.53 ± 10% perf-profile.children.cycles-pp.__might_sleep 1.15 ± 9% -0.8 0.36 ± 12% perf-profile.children.cycles-pp.atime_needs_update 1.17 ± 4% -0.7 0.43 ± 13% perf-profile.children.cycles-pp.copyin 1.23 ± 5% -0.7 0.54 ± 9% perf-profile.children.cycles-pp.__fsnotify_parent 1.32 ± 7% -0.6 0.68 ± 6% perf-profile.children.cycles-pp.file_update_time 1.02 ± 3% -0.6 0.38 ± 12% perf-profile.children.cycles-pp.__might_fault 2.54 ± 2% -0.5 2.03 perf-profile.children.cycles-pp.mutex_lock 0.78 ± 4% -0.4 0.36 ± 10% perf-profile.children.cycles-pp.fput_many 0.53 ± 4% -0.4 0.15 ± 21% perf-profile.children.cycles-pp.inode_has_perm 0.44 ± 12% -0.3 0.11 ± 22% perf-profile.children.cycles-pp.iov_iter_init 0.76 ± 4% -0.3 0.44 ± 8% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.53 ± 3% -0.3 0.21 ± 6% perf-profile.children.cycles-pp.rcu_all_qs 0.41 ± 9% -0.3 0.12 ± 15% perf-profile.children.cycles-pp.timespec64_trunc 0.97 -0.3 0.71 ± 3% perf-profile.children.cycles-pp._cond_resched 0.47 ± 7% -0.3 0.21 ± 11% perf-profile.children.cycles-pp.__x64_sys_read 0.45 ± 7% -0.3 0.20 ± 10% perf-profile.children.cycles-pp.__x64_sys_write 0.37 ± 4% -0.2 0.17 ± 16% perf-profile.children.cycles-pp.generic_pipe_buf_confirm 0.44 ± 12% -0.2 0.24 ± 3% perf-profile.children.cycles-pp.__sb_start_write 0.31 ± 6% -0.2 0.12 ± 13% perf-profile.children.cycles-pp.__sb_end_write 0.36 ± 7% -0.2 0.18 ± 6% perf-profile.children.cycles-pp.rw_verify_area 0.26 ± 14% -0.2 0.09 ± 14% perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64 0.29 ± 8% -0.2 0.14 ± 16% perf-profile.children.cycles-pp.fpregs_assert_state_consistent 0.22 ± 39% -0.2 0.06 ± 11% perf-profile.children.cycles-pp.__vfs_read 0.21 ± 19% -0.1 0.11 ± 9% perf-profile.children.cycles-pp.__mutex_unlock_slowpath 0.12 ± 9% -0.1 0.03 ±100% perf-profile.children.cycles-pp.__vfs_write 0.12 ± 17% -0.0 0.08 ± 11% perf-profile.children.cycles-pp.wake_up_q 0.08 ± 10% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.kill_fasync 0.11 ± 7% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode 0.09 ± 5% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.prepare_exit_to_usermode 0.01 ±173% +0.1 0.06 ± 6% perf-profile.children.cycles-pp.__mnt_want_write 0.07 ± 25% +0.1 0.12 ± 27% perf-profile.children.cycles-pp.__softirqentry_text_start 0.08 ± 30% +0.1 0.14 ± 26% perf-profile.children.cycles-pp.irq_exit 0.06 ± 6% +0.1 0.12 ± 10% perf-profile.children.cycles-pp.__mark_inode_dirty 0.00 +0.1 0.06 perf-profile.children.cycles-pp.preempt_schedule_common 0.00 +0.1 0.07 ± 6% perf-profile.children.cycles-pp.is_cpu_allowed 0.03 ±100% +0.1 0.10 ± 8% perf-profile.children.cycles-pp.native_apic_msr_eoi_write 0.00 +0.1 0.07 ± 5% perf-profile.children.cycles-pp.__x2apic_send_IPI_dest 0.01 ±173% +0.1 0.09 ± 32% perf-profile.children.cycles-pp.migrate_task_rq_fair 0.00 +0.1 0.07 ± 14% perf-profile.children.cycles-pp.update_cfs_rq_h_load 0.05 ± 61% +0.1 0.13 ± 5% perf-profile.children.cycles-pp.native_irq_return_iret 0.00 +0.1 0.08 ± 8% perf-profile.children.cycles-pp.rcu_note_context_switch 0.00 +0.1 0.08 ± 13% perf-profile.children.cycles-pp.smp_reschedule_interrupt 0.00 +0.1 0.09 ± 4% perf-profile.children.cycles-pp.native_load_tls 0.03 ±102% +0.1 0.12 ± 30% perf-profile.children.cycles-pp.set_task_cpu 0.00 +0.1 0.09 ± 11% perf-profile.children.cycles-pp.check_cfs_rq_runtime 0.00 +0.1 0.10 ± 5% perf-profile.children.cycles-pp.rb_next 0.10 ± 10% +0.1 0.20 ± 9% perf-profile.children.cycles-pp.generic_update_time 0.00 +0.1 0.10 ± 29% perf-profile.children.cycles-pp.__x64_sys_exit 0.00 +0.1 0.10 ± 28% perf-profile.children.cycles-pp.do_exit 0.00 +0.1 0.11 ± 8% perf-profile.children.cycles-pp.rb_insert_color 0.00 +0.1 0.11 ± 7% perf-profile.children.cycles-pp.__list_add_valid 0.00 +0.1 0.12 ± 7% perf-profile.children.cycles-pp.resched_curr 0.00 +0.1 0.12 ± 6% perf-profile.children.cycles-pp.switch_mm_irqs_off 0.13 ± 10% +0.1 0.26 ± 4% perf-profile.children.cycles-pp.__list_del_entry_valid 0.00 +0.1 0.15 ± 4% perf-profile.children.cycles-pp.set_next_buddy 0.00 +0.2 0.16 ± 5% perf-profile.children.cycles-pp.deactivate_task 0.05 ± 9% +0.2 0.22 ± 7% perf-profile.children.cycles-pp.rb_erase 0.06 ± 13% +0.2 0.23 ± 9% perf-profile.children.cycles-pp.finish_wait 0.17 ± 12% +0.2 0.35 ± 2% perf-profile.children.cycles-pp.anon_pipe_buf_release 0.01 ±173% +0.2 0.20 ± 8% perf-profile.children.cycles-pp.wakeup_preempt_entity 0.00 +0.2 0.21 ± 4% perf-profile.children.cycles-pp.cpumask_next 0.07 ± 22% +0.3 0.40 ± 2% perf-profile.children.cycles-pp.prepare_to_wait 0.03 ±100% +0.4 0.39 ± 5% perf-profile.children.cycles-pp.update_min_vruntime 0.03 ±100% +0.4 0.39 ± 12% perf-profile.children.cycles-pp.cpus_share_cache 0.06 ± 14% +0.4 0.43 perf-profile.children.cycles-pp.clear_buddies 0.11 ± 15% +0.4 0.50 ± 3% perf-profile.children.cycles-pp.cpuacct_charge 0.10 ± 26% +0.4 0.49 ± 4% perf-profile.children.cycles-pp.native_sched_clock 0.32 ± 31% +0.4 0.75 ± 14% perf-profile.children.cycles-pp.sched_ttwu_pending 0.11 ± 26% +0.4 0.56 ± 3% perf-profile.children.cycles-pp.sched_clock 0.34 ± 32% +0.5 0.80 ± 15% perf-profile.children.cycles-pp.scheduler_ipi 0.08 ± 12% +0.5 0.56 ± 7% perf-profile.children.cycles-pp.account_entity_enqueue 0.13 ± 26% +0.5 0.62 ± 3% perf-profile.children.cycles-pp.sched_clock_cpu 0.14 ± 26% +0.6 0.77 ± 4% perf-profile.children.cycles-pp.__calc_delta 0.08 ± 16% +0.7 0.76 ± 3% perf-profile.children.cycles-pp.account_entity_dequeue 0.84 ± 5% +0.7 1.54 ± 5% perf-profile.children.cycles-pp.__lock_text_start 1.61 ± 16% +0.7 2.31 ± 9% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.42 ± 31% +0.7 1.12 ± 13% perf-profile.children.cycles-pp.reschedule_interrupt 0.24 ± 28% +0.9 1.11 perf-profile.children.cycles-pp.native_write_msr 0.14 ± 22% +0.9 1.01 ± 5% perf-profile.children.cycles-pp.check_preempt_wakeup 0.24 ± 32% +0.9 1.12 ± 7% perf-profile.children.cycles-pp.finish_task_switch 0.22 ± 22% +0.9 1.10 ± 3% perf-profile.children.cycles-pp.find_next_bit 0.22 ± 23% +0.9 1.12 ± 3% perf-profile.children.cycles-pp.set_next_entity 0.21 ± 32% +0.9 1.14 ± 3% perf-profile.children.cycles-pp.update_cfs_group 0.16 ± 23% +1.0 1.15 ± 4% perf-profile.children.cycles-pp.check_preempt_curr 0.21 ± 30% +1.0 1.21 ± 2% perf-profile.children.cycles-pp.__switch_to 0.23 ± 21% +1.0 1.24 ± 2% perf-profile.children.cycles-pp.__switch_to_asm 0.16 ± 38% +1.0 1.20 ± 4% perf-profile.children.cycles-pp.___perf_sw_event 0.15 ± 26% +1.0 1.20 ± 3% perf-profile.children.cycles-pp.pick_next_entity 0.15 ± 20% +1.1 1.21 ± 7% perf-profile.children.cycles-pp.put_prev_entity 0.17 ± 22% +1.1 1.23 ± 4% perf-profile.children.cycles-pp.ttwu_do_wakeup 0.19 ± 10% +1.3 1.52 ± 10% perf-profile.children.cycles-pp.__enqueue_entity 0.24 ± 23% +1.3 1.59 ± 6% perf-profile.children.cycles-pp.__update_load_avg_se 0.28 ± 23% +1.4 1.64 ± 3% perf-profile.children.cycles-pp.switch_fpu_return 0.31 ± 29% +1.6 1.88 ± 5% perf-profile.children.cycles-pp.update_rq_clock 0.44 ± 21% +1.8 2.24 ± 4% perf-profile.children.cycles-pp.cpumask_next_wrap 0.33 ± 20% +1.8 2.15 ± 4% perf-profile.children.cycles-pp.dequeue_entity 23.28 ± 2% +2.0 25.27 ± 2% perf-profile.children.cycles-pp.ksys_read 0.46 ± 25% +2.1 2.57 ± 5% perf-profile.children.cycles-pp.update_load_avg 0.20 ± 21% +2.1 2.34 ± 4% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.91 ± 29% +2.2 3.12 ± 11% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 0.38 ± 23% +2.4 2.80 ± 4% perf-profile.children.cycles-pp.reweight_entity 0.52 ± 18% +2.4 2.96 ± 3% perf-profile.children.cycles-pp.enqueue_entity 0.49 ± 23% +2.6 3.10 ± 4% perf-profile.children.cycles-pp.update_curr 20.92 ± 2% +3.2 24.12 ± 2% perf-profile.children.cycles-pp.vfs_read 0.21 ± 23% +3.4 3.65 ± 7% perf-profile.children.cycles-pp._raw_spin_lock 0.76 ± 24% +3.9 4.67 ± 3% perf-profile.children.cycles-pp.pick_next_task_fair 0.66 ± 34% +4.4 5.06 ± 2% perf-profile.children.cycles-pp.exit_to_usermode_loop 0.81 ± 22% +5.0 5.83 ± 4% perf-profile.children.cycles-pp.dequeue_task_fair 1.01 ± 21% +5.8 6.86 ± 4% perf-profile.children.cycles-pp.enqueue_task_fair 1.06 ± 21% +6.0 7.10 ± 4% perf-profile.children.cycles-pp.activate_task 1.07 ± 21% +6.1 7.15 ± 4% perf-profile.children.cycles-pp.ttwu_do_activate 13.19 +7.8 20.94 ± 3% perf-profile.children.cycles-pp.new_sync_read 11.61 +8.7 20.29 ± 3% perf-profile.children.cycles-pp.pipe_read 1.45 ± 25% +8.9 10.35 ± 7% perf-profile.children.cycles-pp.available_idle_cpu 78.85 ± 2% +9.8 88.66 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 77.90 ± 2% +10.1 88.00 perf-profile.children.cycles-pp.do_syscall_64 2.42 ± 21% +12.9 15.36 ± 4% perf-profile.children.cycles-pp.pipe_wait 2.71 ± 26% +15.4 18.12 ± 4% perf-profile.children.cycles-pp.__schedule 2.76 ± 25% +15.6 18.37 ± 4% perf-profile.children.cycles-pp.schedule 27.59 ± 2% +18.6 46.17 ± 3% perf-profile.children.cycles-pp.ksys_write 2.26 ± 23% +19.0 21.30 ± 7% perf-profile.children.cycles-pp.select_idle_sibling 24.98 ± 2% +19.9 44.88 ± 3% perf-profile.children.cycles-pp.vfs_write 2.48 ± 23% +20.0 22.46 ± 7% perf-profile.children.cycles-pp.select_task_rq_fair 19.06 ± 3% +23.3 42.41 ± 4% perf-profile.children.cycles-pp.new_sync_write 17.98 ± 4% +23.9 41.93 ± 4% perf-profile.children.cycles-pp.pipe_write 4.03 ± 21% +30.6 34.60 ± 6% perf-profile.children.cycles-pp.try_to_wake_up 3.97 ± 22% +30.8 34.74 ± 6% perf-profile.children.cycles-pp.autoremove_wake_function 4.30 ± 20% +30.9 35.19 ± 6% perf-profile.children.cycles-pp.__wake_up_common 6.06 ± 14% +31.2 37.21 ± 5% perf-profile.children.cycles-pp.__wake_up_common_lock 24.41 ± 6% -15.5 8.96 ± 10% perf-profile.self.cycles-pp.do_syscall_64 9.18 ± 6% -6.0 3.14 ± 12% perf-profile.self.cycles-pp.syscall_return_via_sysret 8.63 ± 5% -5.6 3.00 ± 12% perf-profile.self.cycles-pp.entry_SYSCALL_64 3.71 ± 3% -2.0 1.73 ± 6% perf-profile.self.cycles-pp.mutex_unlock 2.47 ± 3% -1.5 0.96 ± 10% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 2.76 ± 3% -1.4 1.35 ± 8% perf-profile.self.cycles-pp.selinux_file_permission 2.02 ± 7% -1.3 0.68 ± 12% perf-profile.self.cycles-pp.fsnotify 2.46 ± 6% -1.3 1.20 ± 8% perf-profile.self.cycles-pp.__fget 1.75 ± 6% -1.1 0.64 ± 10% perf-profile.self.cycles-pp.avc_has_perm 1.83 ± 5% -1.1 0.75 ± 10% perf-profile.self.cycles-pp.___might_sleep 1.75 -1.0 0.77 ± 7% perf-profile.self.cycles-pp.pipe_write 1.20 ± 4% -0.7 0.46 ± 10% perf-profile.self.cycles-pp.__might_sleep 1.29 ± 11% -0.7 0.58 ± 10% perf-profile.self.cycles-pp.new_sync_read 1.63 ± 4% -0.7 0.97 ± 5% perf-profile.self.cycles-pp.pipe_read 1.15 ± 5% -0.7 0.49 ± 8% perf-profile.self.cycles-pp.__fsnotify_parent 0.89 ± 8% -0.6 0.27 ± 15% perf-profile.self.cycles-pp.copy_page_from_iter 0.89 ± 4% -0.6 0.29 ± 14% perf-profile.self.cycles-pp.security_file_permission 0.83 ± 5% -0.6 0.24 ± 21% perf-profile.self.cycles-pp.file_has_perm 0.90 ± 2% -0.5 0.41 ± 6% perf-profile.self.cycles-pp.new_sync_write 0.67 ± 10% -0.5 0.20 ± 15% perf-profile.self.cycles-pp.current_time 0.83 ± 4% -0.4 0.41 ± 10% perf-profile.self.cycles-pp.copy_page_to_iter 0.72 ± 7% -0.4 0.31 ± 9% perf-profile.self.cycles-pp.vfs_write 0.75 ± 4% -0.4 0.35 ± 9% perf-profile.self.cycles-pp.fput_many 1.01 ± 4% -0.3 0.69 ± 5% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.45 ± 4% -0.3 0.14 ± 19% perf-profile.self.cycles-pp.inode_has_perm 0.54 ± 8% -0.3 0.23 ± 9% perf-profile.self.cycles-pp.file_update_time 0.41 ± 10% -0.3 0.11 ± 20% perf-profile.self.cycles-pp.iov_iter_init 0.49 ± 7% -0.3 0.19 ± 8% perf-profile.self.cycles-pp.atime_needs_update 0.44 ± 7% -0.3 0.15 ± 14% perf-profile.self.cycles-pp.ksys_read 0.44 ± 7% -0.3 0.17 ± 9% perf-profile.self.cycles-pp.__inode_security_revalidate 0.41 ± 2% -0.3 0.15 ± 8% perf-profile.self.cycles-pp.rcu_all_qs 0.43 ± 8% -0.3 0.17 ± 11% perf-profile.self.cycles-pp.ksys_write 0.37 ± 9% -0.3 0.11 ± 17% perf-profile.self.cycles-pp.timespec64_trunc 0.42 ± 14% -0.2 0.18 ± 8% perf-profile.self.cycles-pp.__sb_start_write 0.39 ± 7% -0.2 0.17 ± 11% perf-profile.self.cycles-pp.__x64_sys_write 0.40 ± 8% -0.2 0.17 ± 8% perf-profile.self.cycles-pp.__x64_sys_read 0.41 ± 7% -0.2 0.20 ± 11% perf-profile.self.cycles-pp.__wake_up_common_lock 0.60 ± 4% -0.2 0.39 ± 7% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax 1.25 -0.2 1.04 ± 2% perf-profile.self.cycles-pp.mutex_lock 0.30 ± 6% -0.2 0.12 ± 15% perf-profile.self.cycles-pp.__sb_end_write 0.35 ± 5% -0.2 0.18 ± 3% perf-profile.self.cycles-pp.rw_verify_area 0.34 ± 7% -0.2 0.18 ± 8% perf-profile.self.cycles-pp.touch_atime 0.32 ± 4% -0.2 0.15 ± 14% perf-profile.self.cycles-pp.generic_pipe_buf_confirm 0.23 ± 16% -0.2 0.08 ± 11% perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64 0.74 ± 3% -0.2 0.59 ± 5% perf-profile.self.cycles-pp.vfs_read 0.60 ± 3% -0.2 0.45 ± 4% perf-profile.self.cycles-pp.__fget_light 0.27 ± 8% -0.1 0.12 ± 15% perf-profile.self.cycles-pp.fpregs_assert_state_consistent 0.18 ± 45% -0.1 0.04 ± 58% perf-profile.self.cycles-pp.__vfs_read 0.18 ± 5% -0.1 0.06 ± 17% perf-profile.self.cycles-pp.__fdget_pos 0.25 ± 5% -0.1 0.14 ± 11% perf-profile.self.cycles-pp.__might_fault 0.07 ± 7% -0.0 0.04 ± 57% perf-profile.self.cycles-pp.kill_fasync 0.01 ±173% +0.1 0.06 ± 17% perf-profile.self.cycles-pp.sched_ttwu_pending 0.00 +0.1 0.05 ± 8% perf-profile.self.cycles-pp.scheduler_ipi 0.00 +0.1 0.05 ± 8% perf-profile.self.cycles-pp.rcu_note_context_switch 0.01 ±173% +0.1 0.07 ± 12% perf-profile.self.cycles-pp.generic_update_time 0.00 +0.1 0.06 ± 7% perf-profile.self.cycles-pp.cpumask_next 0.00 +0.1 0.06 ± 11% perf-profile.self.cycles-pp.ttwu_do_activate 0.00 +0.1 0.06 perf-profile.self.cycles-pp.__mnt_want_write 0.06 ± 14% +0.1 0.12 ± 10% perf-profile.self.cycles-pp.__mark_inode_dirty 0.00 +0.1 0.06 ± 13% perf-profile.self.cycles-pp.is_cpu_allowed 0.00 +0.1 0.07 ± 13% perf-profile.self.cycles-pp.sched_clock_cpu 0.00 +0.1 0.07 ± 6% perf-profile.self.cycles-pp.sched_clock 0.03 ±100% +0.1 0.10 ± 11% perf-profile.self.cycles-pp.native_apic_msr_eoi_write 0.00 +0.1 0.07 ± 14% perf-profile.self.cycles-pp.update_cfs_rq_h_load 0.00 +0.1 0.08 perf-profile.self.cycles-pp.native_load_tls 0.05 ± 60% +0.1 0.13 ± 5% perf-profile.self.cycles-pp.native_irq_return_iret 0.00 +0.1 0.09 ± 9% perf-profile.self.cycles-pp.ttwu_do_wakeup 0.00 +0.1 0.09 ± 11% perf-profile.self.cycles-pp.rb_next 0.00 +0.1 0.10 ± 8% perf-profile.self.cycles-pp.__list_add_valid 0.00 +0.1 0.10 ± 10% perf-profile.self.cycles-pp.rb_insert_color 0.00 +0.1 0.11 ± 9% perf-profile.self.cycles-pp.resched_curr 0.00 +0.1 0.11 ± 7% perf-profile.self.cycles-pp.switch_mm_irqs_off 0.32 ± 7% +0.1 0.45 ± 5% perf-profile.self.cycles-pp.__wake_up_common 0.13 ± 10% +0.1 0.25 ± 5% perf-profile.self.cycles-pp.__list_del_entry_valid 0.00 +0.1 0.13 ± 5% perf-profile.self.cycles-pp.autoremove_wake_function 0.00 +0.1 0.13 ± 8% perf-profile.self.cycles-pp.check_preempt_curr 0.04 ± 58% +0.1 0.18 ± 10% perf-profile.self.cycles-pp.finish_wait 0.00 +0.1 0.14 ± 5% perf-profile.self.cycles-pp.set_next_buddy 0.00 +0.1 0.15 ± 3% perf-profile.self.cycles-pp.put_prev_entity 0.00 +0.2 0.15 ± 5% perf-profile.self.cycles-pp.deactivate_task 0.05 ± 9% +0.2 0.21 ± 5% perf-profile.self.cycles-pp.rb_erase 0.00 +0.2 0.15 ± 10% perf-profile.self.cycles-pp.exit_to_usermode_loop 0.00 +0.2 0.18 ± 6% perf-profile.self.cycles-pp.wakeup_preempt_entity 0.15 ± 7% +0.2 0.34 ± 2% perf-profile.self.cycles-pp.anon_pipe_buf_release 0.03 ±100% +0.2 0.21 ± 8% perf-profile.self.cycles-pp.pipe_wait 0.03 ±102% +0.2 0.24 ± 3% perf-profile.self.cycles-pp.activate_task 0.01 ±173% +0.2 0.23 ± 3% perf-profile.self.cycles-pp.prepare_to_wait 0.03 ±100% +0.2 0.25 ± 4% perf-profile.self.cycles-pp.set_next_entity 0.02 ±173% +0.2 0.26 ± 9% perf-profile.self.cycles-pp.dequeue_entity 0.04 ±100% +0.3 0.29 ± 5% perf-profile.self.cycles-pp.enqueue_entity 0.08 ± 31% +0.3 0.34 ± 2% perf-profile.self.cycles-pp.schedule 0.78 ± 3% +0.3 1.09 perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.09 ± 24% +0.3 0.42 ± 7% perf-profile.self.cycles-pp.finish_task_switch 0.03 ±100% +0.3 0.37 ± 6% perf-profile.self.cycles-pp.update_min_vruntime 0.03 ±102% +0.4 0.38 perf-profile.self.cycles-pp.clear_buddies 0.03 ±100% +0.4 0.39 ± 12% perf-profile.self.cycles-pp.cpus_share_cache 0.07 ± 22% +0.4 0.44 ± 7% perf-profile.self.cycles-pp.check_preempt_wakeup 0.09 ± 27% +0.4 0.47 ± 4% perf-profile.self.cycles-pp.native_sched_clock 0.07 ± 26% +0.4 0.45 ± 4% perf-profile.self.cycles-pp.pick_next_entity 0.11 ± 15% +0.4 0.50 ± 3% perf-profile.self.cycles-pp.cpuacct_charge 0.07 ± 14% +0.5 0.53 ± 8% perf-profile.self.cycles-pp.account_entity_enqueue 0.10 ± 21% +0.5 0.56 ± 8% perf-profile.self.cycles-pp.try_to_wake_up 0.10 ± 18% +0.5 0.62 ± 6% perf-profile.self.cycles-pp.dequeue_task_fair 0.14 ± 26% +0.6 0.75 ± 4% perf-profile.self.cycles-pp.__calc_delta 0.06 ± 13% +0.6 0.68 ± 4% perf-profile.self.cycles-pp.account_entity_dequeue 0.15 ± 27% +0.6 0.79 ± 6% perf-profile.self.cycles-pp.enqueue_task_fair 0.17 ± 30% +0.7 0.89 ± 4% perf-profile.self.cycles-pp.update_load_avg 0.15 ± 20% +0.7 0.88 ± 4% perf-profile.self.cycles-pp.reweight_entity 0.15 ± 23% +0.8 0.96 ± 3% perf-profile.self.cycles-pp.pick_next_task_fair 0.24 ± 30% +0.9 1.11 perf-profile.self.cycles-pp.native_write_msr 0.20 ± 23% +0.9 1.07 ± 5% perf-profile.self.cycles-pp.select_task_rq_fair 0.20 ± 22% +0.9 1.07 ± 3% perf-profile.self.cycles-pp.find_next_bit 0.20 ± 31% +0.9 1.11 ± 3% perf-profile.self.cycles-pp.update_cfs_group 0.20 ± 28% +0.9 1.14 ± 2% perf-profile.self.cycles-pp.__switch_to 0.14 ± 37% +1.0 1.13 ± 5% perf-profile.self.cycles-pp.___perf_sw_event 0.21 ± 24% +1.0 1.22 ± 2% perf-profile.self.cycles-pp.__switch_to_asm 0.24 ± 23% +1.1 1.31 ± 5% perf-profile.self.cycles-pp.cpumask_next_wrap 0.23 ± 32% +1.3 1.49 ± 5% perf-profile.self.cycles-pp.update_rq_clock 0.19 ± 28% +1.3 1.45 ± 6% perf-profile.self.cycles-pp.update_curr 0.23 ± 23% +1.3 1.55 ± 6% perf-profile.self.cycles-pp.__update_load_avg_se 0.19 ± 10% +1.3 1.51 ± 10% perf-profile.self.cycles-pp.__enqueue_entity 0.28 ± 22% +1.3 1.63 ± 3% perf-profile.self.cycles-pp.switch_fpu_return 0.11 ± 19% +1.6 1.72 ± 5% perf-profile.self.cycles-pp._raw_spin_lock 0.42 ± 26% +1.7 2.12 ± 2% perf-profile.self.cycles-pp.__schedule 0.19 ± 20% +2.0 2.20 ± 4% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.91 ± 29% +2.2 3.11 ± 12% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 0.32 ± 19% +7.7 8.05 ± 7% perf-profile.self.cycles-pp.select_idle_sibling 1.44 ± 25% +8.8 10.23 ± 7% perf-profile.self.cycles-pp.available_idle_cpu 439.00 +11.7% 490.50 interrupts.100:IR-PCI-MSI.1572923-edge.eth0-TxRx-59 436.25 +13.1% 493.25 interrupts.101:IR-PCI-MSI.1572924-edge.eth0-TxRx-60 436.25 +12.4% 490.50 interrupts.102:IR-PCI-MSI.1572925-edge.eth0-TxRx-61 436.25 +12.4% 490.50 interrupts.103:IR-PCI-MSI.1572926-edge.eth0-TxRx-62 509.75 ± 3% +545.1% 3288 ±120% interrupts.42:IR-PCI-MSI.1572867-edge.eth0-TxRx-3 474.00 ± 3% +49.7% 709.75 ± 24% interrupts.46:IR-PCI-MSI.1572871-edge.eth0-TxRx-7 588.00 ± 35% +196.0% 1740 ± 50% interrupts.51:IR-PCI-MSI.1572876-edge.eth0-TxRx-12 436.75 ± 2% +12.8% 492.50 interrupts.55:IR-PCI-MSI.1572880-edge.eth0-TxRx-16 438.00 +39.5% 611.00 ± 33% interrupts.57:IR-PCI-MSI.1572882-edge.eth0-TxRx-18 436.25 +13.8% 496.25 interrupts.58:IR-PCI-MSI.1572883-edge.eth0-TxRx-19 436.25 +14.3% 498.50 interrupts.59:IR-PCI-MSI.1572884-edge.eth0-TxRx-20 437.00 +12.2% 490.50 interrupts.60:IR-PCI-MSI.1572885-edge.eth0-TxRx-21 436.25 +12.4% 490.50 interrupts.61:IR-PCI-MSI.1572886-edge.eth0-TxRx-22 436.25 +12.4% 490.50 interrupts.62:IR-PCI-MSI.1572887-edge.eth0-TxRx-23 441.00 +11.8% 493.00 interrupts.63:IR-PCI-MSI.1572888-edge.eth0-TxRx-24 437.25 +13.3% 495.25 interrupts.64:IR-PCI-MSI.1572889-edge.eth0-TxRx-25 436.25 +12.4% 490.50 interrupts.65:IR-PCI-MSI.1572890-edge.eth0-TxRx-26 436.25 +12.4% 490.50 interrupts.68:IR-PCI-MSI.1572893-edge.eth0-TxRx-29 436.25 +12.4% 490.50 interrupts.69:IR-PCI-MSI.1572894-edge.eth0-TxRx-30 436.25 +12.4% 490.50 interrupts.71:IR-PCI-MSI.1572896-edge.eth0-TxRx-32 437.25 ± 2% +12.2% 490.50 interrupts.74:IR-PCI-MSI.1572897-edge.eth0-TxRx-33 436.25 +12.6% 491.25 interrupts.75:IR-PCI-MSI.1572898-edge.eth0-TxRx-34 440.75 +12.4% 495.25 ± 2% interrupts.76:IR-PCI-MSI.1572899-edge.eth0-TxRx-35 436.25 +12.6% 491.00 interrupts.77:IR-PCI-MSI.1572900-edge.eth0-TxRx-36 436.75 +13.1% 493.75 interrupts.78:IR-PCI-MSI.1572901-edge.eth0-TxRx-37 436.25 +12.8% 492.25 interrupts.79:IR-PCI-MSI.1572902-edge.eth0-TxRx-38 436.25 +12.8% 492.25 interrupts.80:IR-PCI-MSI.1572903-edge.eth0-TxRx-39 436.25 +12.4% 490.50 interrupts.81:IR-PCI-MSI.1572904-edge.eth0-TxRx-40 436.25 +12.4% 490.50 interrupts.82:IR-PCI-MSI.1572905-edge.eth0-TxRx-41 436.25 +12.4% 490.50 interrupts.83:IR-PCI-MSI.1572906-edge.eth0-TxRx-42 436.25 +14.6% 499.75 ± 2% interrupts.84:IR-PCI-MSI.1572907-edge.eth0-TxRx-43 436.25 +12.4% 490.50 interrupts.85:IR-PCI-MSI.1572908-edge.eth0-TxRx-44 436.50 +12.4% 490.50 interrupts.86:IR-PCI-MSI.1572909-edge.eth0-TxRx-45 436.25 +12.4% 490.50 interrupts.87:IR-PCI-MSI.1572910-edge.eth0-TxRx-46 436.25 +12.4% 490.50 interrupts.88:IR-PCI-MSI.1572911-edge.eth0-TxRx-47 436.25 +12.6% 491.00 interrupts.89:IR-PCI-MSI.1572912-edge.eth0-TxRx-48 441.25 ± 3% +14.1% 503.25 ± 3% interrupts.90:IR-PCI-MSI.1572913-edge.eth0-TxRx-49 436.25 +12.4% 490.50 interrupts.91:IR-PCI-MSI.1572914-edge.eth0-TxRx-50 436.25 +14.4% 499.25 ± 3% interrupts.92:IR-PCI-MSI.1572915-edge.eth0-TxRx-51 439.25 ± 2% +11.7% 490.50 interrupts.94:IR-PCI-MSI.1572917-edge.eth0-TxRx-53 437.75 +12.1% 490.50 interrupts.95:IR-PCI-MSI.1572918-edge.eth0-TxRx-54 436.50 +12.4% 490.50 interrupts.96:IR-PCI-MSI.1572919-edge.eth0-TxRx-55 436.25 +12.4% 490.50 interrupts.97:IR-PCI-MSI.1572920-edge.eth0-TxRx-56 439.00 ± 2% +11.7% 490.50 interrupts.98:IR-PCI-MSI.1572921-edge.eth0-TxRx-57 442.25 ± 3% +10.9% 490.50 interrupts.99:IR-PCI-MSI.1572922-edge.eth0-TxRx-58 1989 +12.1% 2230 interrupts.9:IR-IO-APIC.9-fasteoi.acpi 6596036 ± 10% -21.1% 5206443 ± 6% interrupts.CAL:Function_call_interrupts 86097 ± 13% -18.4% 70229 ± 7% interrupts.CPU0.CAL:Function_call_interrupts 1785369 +12.4% 2006294 interrupts.CPU0.LOC:Local_timer_interrupts 102823 ± 12% -18.3% 84031 ± 5% interrupts.CPU0.TLB:TLB_shootdowns 388425 ± 12% +27.8% 496548 ± 4% interrupts.CPU0.TRM:Thermal_event_interrupts 1989 +12.1% 2230 interrupts.CPU1.9:IR-IO-APIC.9-fasteoi.acpi 90370 ± 8% -21.4% 70995 ± 6% interrupts.CPU1.CAL:Function_call_interrupts 1784834 +12.4% 2006046 interrupts.CPU1.LOC:Local_timer_interrupts 3088391 ± 15% -20.9% 2441882 ± 7% interrupts.CPU1.RES:Rescheduling_interrupts 105391 ± 8% -18.6% 85793 ± 6% interrupts.CPU1.TLB:TLB_shootdowns 388373 ± 12% +27.9% 496558 ± 4% interrupts.CPU1.TRM:Thermal_event_interrupts 88236 ± 11% -19.5% 71001 ± 7% interrupts.CPU10.CAL:Function_call_interrupts 1785471 +12.3% 2004740 interrupts.CPU10.LOC:Local_timer_interrupts 3040774 ± 4% -21.1% 2398396 ± 8% interrupts.CPU10.RES:Rescheduling_interrupts 104483 ± 10% -18.6% 85025 ± 5% interrupts.CPU10.TLB:TLB_shootdowns 388417 ± 12% +27.8% 496560 ± 4% interrupts.CPU10.TRM:Thermal_event_interrupts 1785039 +12.3% 2005274 interrupts.CPU11.LOC:Local_timer_interrupts 925.75 ±173% +461.3% 5196 ± 58% interrupts.CPU11.NMI:Non-maskable_interrupts 925.75 ±173% +461.3% 5196 ± 58% interrupts.CPU11.PMI:Performance_monitoring_interrupts 3094193 ± 14% -21.5% 2430228 ± 8% interrupts.CPU11.RES:Rescheduling_interrupts 102398 ± 10% -16.1% 85884 ± 5% interrupts.CPU11.TLB:TLB_shootdowns 388413 ± 12% +27.8% 496474 ± 4% interrupts.CPU11.TRM:Thermal_event_interrupts 588.00 ± 35% +196.0% 1740 ± 50% interrupts.CPU12.51:IR-PCI-MSI.1572876-edge.eth0-TxRx-12 87438 ± 9% -18.1% 71580 ± 7% interrupts.CPU12.CAL:Function_call_interrupts 1785376 +12.3% 2005068 interrupts.CPU12.LOC:Local_timer_interrupts 103057 ± 10% -16.4% 86186 ± 6% interrupts.CPU12.TLB:TLB_shootdowns 388416 ± 12% +27.8% 496490 ± 4% interrupts.CPU12.TRM:Thermal_event_interrupts 88735 ± 9% -19.4% 71496 ± 9% interrupts.CPU13.CAL:Function_call_interrupts 1785460 +12.3% 2005445 interrupts.CPU13.LOC:Local_timer_interrupts 104230 ± 9% -17.5% 86038 ± 6% interrupts.CPU13.TLB:TLB_shootdowns 388422 ± 12% +27.8% 496563 ± 4% interrupts.CPU13.TRM:Thermal_event_interrupts 88327 ± 12% -19.0% 71507 ± 7% interrupts.CPU14.CAL:Function_call_interrupts 1785094 +12.3% 2005441 interrupts.CPU14.LOC:Local_timer_interrupts 103317 ± 11% -18.1% 84581 ± 6% interrupts.CPU14.TLB:TLB_shootdowns 388412 ± 12% +27.8% 496564 ± 4% interrupts.CPU14.TRM:Thermal_event_interrupts 86972 ± 10% -17.3% 71898 ± 7% interrupts.CPU15.CAL:Function_call_interrupts 1784668 +12.4% 2005375 interrupts.CPU15.LOC:Local_timer_interrupts 3039799 ± 9% -21.1% 2397505 ± 9% interrupts.CPU15.RES:Rescheduling_interrupts 103367 ± 12% -16.9% 85892 ± 6% interrupts.CPU15.TLB:TLB_shootdowns 388342 ± 12% +27.9% 496559 ± 4% interrupts.CPU15.TRM:Thermal_event_interrupts 436.75 ± 2% +12.8% 492.50 interrupts.CPU16.55:IR-PCI-MSI.1572880-edge.eth0-TxRx-16 1784750 +12.4% 2005330 interrupts.CPU16.LOC:Local_timer_interrupts 55.25 ±171% +7696.8% 4307 ± 64% interrupts.CPU16.NMI:Non-maskable_interrupts 55.25 ±171% +7696.8% 4307 ± 64% interrupts.CPU16.PMI:Performance_monitoring_interrupts 2969996 ± 10% -18.7% 2413141 ± 9% interrupts.CPU16.RES:Rescheduling_interrupts 102367 ± 11% -13.7% 88305 ± 3% interrupts.CPU16.TLB:TLB_shootdowns 388423 ± 12% +27.8% 496563 ± 4% interrupts.CPU16.TRM:Thermal_event_interrupts 1785363 +12.3% 2005849 interrupts.CPU17.LOC:Local_timer_interrupts 388418 ± 12% +27.8% 496560 ± 4% interrupts.CPU17.TRM:Thermal_event_interrupts 438.00 +39.5% 611.00 ± 33% interrupts.CPU18.57:IR-PCI-MSI.1572882-edge.eth0-TxRx-18 88992 ± 12% -22.3% 69145 ± 6% interrupts.CPU18.CAL:Function_call_interrupts 1785398 +12.1% 2001316 interrupts.CPU18.LOC:Local_timer_interrupts 2199105 ± 7% +115.2% 4731679 ± 7% interrupts.CPU18.RES:Rescheduling_interrupts 108465 ± 8% -23.0% 83467 ± 5% interrupts.CPU18.TLB:TLB_shootdowns 329294 ± 3% +15.4% 380131 ± 2% interrupts.CPU18.TRM:Thermal_event_interrupts 436.25 +13.8% 496.25 interrupts.CPU19.58:IR-PCI-MSI.1572883-edge.eth0-TxRx-19 92059 ± 6% -24.2% 69776 ± 5% interrupts.CPU19.CAL:Function_call_interrupts 1784665 +12.2% 2001579 interrupts.CPU19.LOC:Local_timer_interrupts 1284622 ± 7% +225.8% 4185208 ± 4% interrupts.CPU19.RES:Rescheduling_interrupts 108492 ± 7% -23.4% 83059 ± 4% interrupts.CPU19.TLB:TLB_shootdowns 329166 ± 3% +15.5% 380141 ± 2% interrupts.CPU19.TRM:Thermal_event_interrupts 1784512 +12.4% 2005572 interrupts.CPU2.LOC:Local_timer_interrupts 3091132 ± 8% -22.3% 2401342 ± 4% interrupts.CPU2.RES:Rescheduling_interrupts 102562 ± 9% -16.5% 85643 ± 5% interrupts.CPU2.TLB:TLB_shootdowns 388380 ± 12% +27.8% 496544 ± 4% interrupts.CPU2.TRM:Thermal_event_interrupts 436.25 +14.3% 498.50 interrupts.CPU20.59:IR-PCI-MSI.1572884-edge.eth0-TxRx-20 91785 ± 9% -23.9% 69853 ± 4% interrupts.CPU20.CAL:Function_call_interrupts 1785284 +12.1% 2001952 interrupts.CPU20.LOC:Local_timer_interrupts 1281621 ± 12% +203.6% 3890516 ± 7% interrupts.CPU20.RES:Rescheduling_interrupts 106297 ± 9% -19.9% 85180 ± 3% interrupts.CPU20.TLB:TLB_shootdowns 329293 ± 3% +15.4% 380131 ± 2% interrupts.CPU20.TRM:Thermal_event_interrupts 437.00 +12.2% 490.50 interrupts.CPU21.60:IR-PCI-MSI.1572885-edge.eth0-TxRx-21 92635 ± 7% -23.9% 70512 ± 6% interrupts.CPU21.CAL:Function_call_interrupts 1785480 +12.1% 2001723 interrupts.CPU21.LOC:Local_timer_interrupts 1595447 ± 6% +187.8% 4591004 interrupts.CPU21.RES:Rescheduling_interrupts 107575 ± 7% -21.6% 84296 ± 5% interrupts.CPU21.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380086 ± 2% interrupts.CPU21.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU22.61:IR-PCI-MSI.1572886-edge.eth0-TxRx-22 92674 ± 10% -24.4% 70047 ± 6% interrupts.CPU22.CAL:Function_call_interrupts 1784881 +12.1% 2001482 interrupts.CPU22.LOC:Local_timer_interrupts 1605143 ± 6% +179.5% 4486945 ± 5% interrupts.CPU22.RES:Rescheduling_interrupts 108749 ± 10% -23.3% 83409 ± 5% interrupts.CPU22.TLB:TLB_shootdowns 329295 ± 3% +15.4% 380136 ± 2% interrupts.CPU22.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU23.62:IR-PCI-MSI.1572887-edge.eth0-TxRx-23 90975 ± 11% -23.6% 69492 ± 5% interrupts.CPU23.CAL:Function_call_interrupts 1785236 +12.1% 2001811 interrupts.CPU23.LOC:Local_timer_interrupts 1600988 ± 2% +180.2% 4486667 ± 5% interrupts.CPU23.RES:Rescheduling_interrupts 106819 ± 10% -22.3% 83007 ± 4% interrupts.CPU23.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380146 ± 2% interrupts.CPU23.TRM:Thermal_event_interrupts 441.00 +11.8% 493.00 interrupts.CPU24.63:IR-PCI-MSI.1572888-edge.eth0-TxRx-24 91919 ± 11% -23.1% 70689 ± 5% interrupts.CPU24.CAL:Function_call_interrupts 1784996 +12.1% 2000922 interrupts.CPU24.LOC:Local_timer_interrupts 1612237 ± 5% +188.4% 4649702 ± 2% interrupts.CPU24.RES:Rescheduling_interrupts 108187 ± 8% -22.5% 83831 ± 3% interrupts.CPU24.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380122 ± 2% interrupts.CPU24.TRM:Thermal_event_interrupts 437.25 +13.3% 495.25 interrupts.CPU25.64:IR-PCI-MSI.1572889-edge.eth0-TxRx-25 88417 ± 12% -20.8% 70029 ± 4% interrupts.CPU25.CAL:Function_call_interrupts 1785404 +12.0% 1999622 interrupts.CPU25.LOC:Local_timer_interrupts 1612074 ± 2% +182.4% 4551699 interrupts.CPU25.RES:Rescheduling_interrupts 105324 ± 10% -20.1% 84201 ± 4% interrupts.CPU25.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380141 ± 2% interrupts.CPU25.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU26.65:IR-PCI-MSI.1572890-edge.eth0-TxRx-26 92632 ± 12% -23.5% 70907 ± 5% interrupts.CPU26.CAL:Function_call_interrupts 1784730 +12.1% 2000264 interrupts.CPU26.LOC:Local_timer_interrupts 1639148 ± 6% +188.0% 4720149 ± 4% interrupts.CPU26.RES:Rescheduling_interrupts 108099 ± 11% -21.6% 84727 ± 4% interrupts.CPU26.TLB:TLB_shootdowns 329271 ± 3% +15.4% 379984 ± 2% interrupts.CPU26.TRM:Thermal_event_interrupts 91312 ± 12% -23.4% 69963 ± 6% interrupts.CPU27.CAL:Function_call_interrupts 1785171 +12.1% 2001743 interrupts.CPU27.LOC:Local_timer_interrupts 1704847 ± 11% +179.4% 4764019 ± 4% interrupts.CPU27.RES:Rescheduling_interrupts 106515 ± 11% -21.6% 83491 ± 5% interrupts.CPU27.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380080 ± 2% interrupts.CPU27.TRM:Thermal_event_interrupts 89178 ± 8% -22.0% 69565 ± 6% interrupts.CPU28.CAL:Function_call_interrupts 1785194 +12.1% 2001836 interrupts.CPU28.LOC:Local_timer_interrupts 1593025 ± 8% +185.7% 4551782 ± 5% interrupts.CPU28.RES:Rescheduling_interrupts 104408 ± 10% -19.2% 84365 ± 6% interrupts.CPU28.TLB:TLB_shootdowns 329294 ± 3% +15.4% 380142 ± 2% interrupts.CPU28.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU29.68:IR-PCI-MSI.1572893-edge.eth0-TxRx-29 90850 ± 12% -21.9% 70993 ± 6% interrupts.CPU29.CAL:Function_call_interrupts 1785398 +12.1% 2000545 interrupts.CPU29.LOC:Local_timer_interrupts 1657508 ± 5% +179.4% 4631109 ± 3% interrupts.CPU29.RES:Rescheduling_interrupts 106383 ± 13% -20.8% 84284 ± 4% interrupts.CPU29.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380133 ± 2% interrupts.CPU29.TRM:Thermal_event_interrupts 509.75 ± 3% +545.1% 3288 ±120% interrupts.CPU3.42:IR-PCI-MSI.1572867-edge.eth0-TxRx-3 87008 ± 11% -19.9% 69730 ± 5% interrupts.CPU3.CAL:Function_call_interrupts 1784446 +12.4% 2005368 interrupts.CPU3.LOC:Local_timer_interrupts 3169674 ± 9% -20.4% 2521963 ± 8% interrupts.CPU3.RES:Rescheduling_interrupts 103703 ± 12% -17.1% 85958 ± 7% interrupts.CPU3.TLB:TLB_shootdowns 388356 ± 12% +27.9% 496551 ± 4% interrupts.CPU3.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU30.69:IR-PCI-MSI.1572894-edge.eth0-TxRx-30 92043 ± 12% -24.2% 69765 ± 5% interrupts.CPU30.CAL:Function_call_interrupts 1785175 +12.1% 2001248 interrupts.CPU30.LOC:Local_timer_interrupts 1637465 ± 6% +191.3% 4769701 ± 3% interrupts.CPU30.RES:Rescheduling_interrupts 106836 ± 11% -22.5% 82810 ± 5% interrupts.CPU30.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380130 ± 2% interrupts.CPU30.TRM:Thermal_event_interrupts 1785301 +12.0% 1999270 interrupts.CPU31.LOC:Local_timer_interrupts 62.75 ±173% +4261.4% 2736 ±113% interrupts.CPU31.NMI:Non-maskable_interrupts 62.75 ±173% +4261.4% 2736 ±113% interrupts.CPU31.PMI:Performance_monitoring_interrupts 1589500 ± 5% +184.1% 4515325 interrupts.CPU31.RES:Rescheduling_interrupts 106576 ± 11% -21.5% 83698 ± 5% interrupts.CPU31.TLB:TLB_shootdowns 329293 ± 3% +15.4% 380131 ± 2% interrupts.CPU31.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU32.71:IR-PCI-MSI.1572896-edge.eth0-TxRx-32 93348 ± 12% -24.4% 70595 ± 8% interrupts.CPU32.CAL:Function_call_interrupts 1785245 +12.1% 2001145 interrupts.CPU32.LOC:Local_timer_interrupts 1626883 ± 6% +176.3% 4494456 ± 5% interrupts.CPU32.RES:Rescheduling_interrupts 107151 ± 12% -21.4% 84236 ± 5% interrupts.CPU32.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380131 ± 2% interrupts.CPU32.TRM:Thermal_event_interrupts 437.25 ± 2% +12.2% 490.50 interrupts.CPU33.74:IR-PCI-MSI.1572897-edge.eth0-TxRx-33 91487 ± 12% -22.7% 70758 ± 3% interrupts.CPU33.CAL:Function_call_interrupts 1785371 +12.0% 1999987 interrupts.CPU33.LOC:Local_timer_interrupts 1578761 ± 6% +201.2% 4754762 ± 6% interrupts.CPU33.RES:Rescheduling_interrupts 106071 ± 12% -21.3% 83430 ± 2% interrupts.CPU33.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380130 ± 2% interrupts.CPU33.TRM:Thermal_event_interrupts 436.25 +12.6% 491.25 interrupts.CPU34.75:IR-PCI-MSI.1572898-edge.eth0-TxRx-34 91116 ± 11% -24.3% 68943 ± 7% interrupts.CPU34.CAL:Function_call_interrupts 1785353 +12.1% 2001525 interrupts.CPU34.LOC:Local_timer_interrupts 1553828 ± 4% +186.0% 4444035 ± 3% interrupts.CPU34.RES:Rescheduling_interrupts 105575 ± 10% -20.7% 83762 ± 4% interrupts.CPU34.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380139 ± 2% interrupts.CPU34.TRM:Thermal_event_interrupts 440.75 +12.4% 495.25 ± 2% interrupts.CPU35.76:IR-PCI-MSI.1572899-edge.eth0-TxRx-35 91396 ± 11% -25.6% 67986 ± 11% interrupts.CPU35.CAL:Function_call_interrupts 1785237 +12.0% 2000288 interrupts.CPU35.LOC:Local_timer_interrupts 1602616 ± 10% +189.7% 4642115 ± 3% interrupts.CPU35.RES:Rescheduling_interrupts 107669 ± 12% -21.3% 84736 ± 4% interrupts.CPU35.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380145 ± 2% interrupts.CPU35.TRM:Thermal_event_interrupts 436.25 +12.6% 491.00 interrupts.CPU36.77:IR-PCI-MSI.1572900-edge.eth0-TxRx-36 93553 ± 8% -19.8% 75044 ± 6% interrupts.CPU36.CAL:Function_call_interrupts 1784636 +12.4% 2005389 interrupts.CPU36.LOC:Local_timer_interrupts 2588984 ± 6% +25.8% 3256348 ± 6% interrupts.CPU36.RES:Rescheduling_interrupts 107053 ± 7% -19.0% 86754 ± 5% interrupts.CPU36.TLB:TLB_shootdowns 388423 ± 12% +27.8% 496560 ± 4% interrupts.CPU36.TRM:Thermal_event_interrupts 436.75 +13.1% 493.75 interrupts.CPU37.78:IR-PCI-MSI.1572901-edge.eth0-TxRx-37 89518 ± 10% -18.9% 72619 ± 5% interrupts.CPU37.CAL:Function_call_interrupts 1784648 +12.4% 2005651 interrupts.CPU37.LOC:Local_timer_interrupts 3112476 ± 10% -20.8% 2464103 ± 5% interrupts.CPU37.RES:Rescheduling_interrupts 102576 ± 10% -14.4% 87794 ± 6% interrupts.CPU37.TLB:TLB_shootdowns 388412 ± 12% +27.8% 496563 ± 4% interrupts.CPU37.TRM:Thermal_event_interrupts 436.25 +12.8% 492.25 interrupts.CPU38.79:IR-PCI-MSI.1572902-edge.eth0-TxRx-38 92566 ± 10% -20.6% 73466 ± 6% interrupts.CPU38.CAL:Function_call_interrupts 1785132 +12.4% 2005883 interrupts.CPU38.LOC:Local_timer_interrupts 3163520 ± 10% -26.0% 2340820 ± 11% interrupts.CPU38.RES:Rescheduling_interrupts 105823 ± 10% -18.0% 86785 ± 4% interrupts.CPU38.TLB:TLB_shootdowns 388401 ± 12% +27.8% 496554 ± 4% interrupts.CPU38.TRM:Thermal_event_interrupts 436.25 +12.8% 492.25 interrupts.CPU39.80:IR-PCI-MSI.1572903-edge.eth0-TxRx-39 93197 ± 11% -21.8% 72854 ± 7% interrupts.CPU39.CAL:Function_call_interrupts 1784624 +12.4% 2005934 interrupts.CPU39.LOC:Local_timer_interrupts 105953 ± 11% -20.8% 83925 ± 6% interrupts.CPU39.TLB:TLB_shootdowns 388418 ± 12% +27.8% 496548 ± 4% interrupts.CPU39.TRM:Thermal_event_interrupts 86883 ± 13% -17.8% 71449 ± 7% interrupts.CPU4.CAL:Function_call_interrupts 1784433 +12.4% 2005462 interrupts.CPU4.LOC:Local_timer_interrupts 3119739 ± 9% -21.1% 2461192 ± 4% interrupts.CPU4.RES:Rescheduling_interrupts 104829 ± 11% -19.1% 84802 ± 5% interrupts.CPU4.TLB:TLB_shootdowns 388417 ± 12% +27.8% 496563 ± 4% interrupts.CPU4.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU40.81:IR-PCI-MSI.1572904-edge.eth0-TxRx-40 91881 ± 12% -21.4% 72188 ± 8% interrupts.CPU40.CAL:Function_call_interrupts 1784831 +12.4% 2005991 interrupts.CPU40.LOC:Local_timer_interrupts 3101288 ± 10% -17.2% 2569221 ± 7% interrupts.CPU40.RES:Rescheduling_interrupts 105884 ± 10% -19.0% 85772 ± 5% interrupts.CPU40.TLB:TLB_shootdowns 388415 ± 12% +27.8% 496564 ± 4% interrupts.CPU40.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU41.82:IR-PCI-MSI.1572905-edge.eth0-TxRx-41 91739 ± 12% -19.9% 73475 ± 7% interrupts.CPU41.CAL:Function_call_interrupts 1785231 +12.3% 2005398 interrupts.CPU41.LOC:Local_timer_interrupts 3223405 ± 11% -27.9% 2325498 ± 5% interrupts.CPU41.RES:Rescheduling_interrupts 104410 ± 11% -18.7% 84846 ± 6% interrupts.CPU41.TLB:TLB_shootdowns 388408 ± 12% +27.8% 496557 ± 4% interrupts.CPU41.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU42.83:IR-PCI-MSI.1572906-edge.eth0-TxRx-42 1785354 +12.4% 2006034 interrupts.CPU42.LOC:Local_timer_interrupts 3068304 ± 9% -22.1% 2390347 ± 6% interrupts.CPU42.RES:Rescheduling_interrupts 107102 ± 9% -20.7% 84890 ± 4% interrupts.CPU42.TLB:TLB_shootdowns 388423 ± 12% +27.8% 496564 ± 4% interrupts.CPU42.TRM:Thermal_event_interrupts 436.25 +14.6% 499.75 ± 2% interrupts.CPU43.84:IR-PCI-MSI.1572907-edge.eth0-TxRx-43 92178 ± 9% -18.3% 75306 ± 7% interrupts.CPU43.CAL:Function_call_interrupts 1784759 +12.4% 2005986 interrupts.CPU43.LOC:Local_timer_interrupts 6434 ± 7% -64.3% 2297 ±106% interrupts.CPU43.NMI:Non-maskable_interrupts 6434 ± 7% -64.3% 2297 ±106% interrupts.CPU43.PMI:Performance_monitoring_interrupts 104773 ± 9% -17.6% 86369 ± 6% interrupts.CPU43.TLB:TLB_shootdowns 388425 ± 12% +27.8% 496556 ± 4% interrupts.CPU43.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU44.85:IR-PCI-MSI.1572908-edge.eth0-TxRx-44 92387 ± 10% -21.9% 72174 ± 9% interrupts.CPU44.CAL:Function_call_interrupts 1784603 +12.4% 2006064 interrupts.CPU44.LOC:Local_timer_interrupts 106179 ± 9% -18.6% 86461 ± 6% interrupts.CPU44.TLB:TLB_shootdowns 388426 ± 12% +27.8% 496562 ± 4% interrupts.CPU44.TRM:Thermal_event_interrupts 436.50 +12.4% 490.50 interrupts.CPU45.86:IR-PCI-MSI.1572909-edge.eth0-TxRx-45 93641 ± 7% -21.9% 73155 ± 6% interrupts.CPU45.CAL:Function_call_interrupts 1784811 +12.4% 2005654 interrupts.CPU45.LOC:Local_timer_interrupts 3093195 ± 11% -15.6% 2609354 ± 4% interrupts.CPU45.RES:Rescheduling_interrupts 106463 ± 7% -19.6% 85593 ± 5% interrupts.CPU45.TLB:TLB_shootdowns 388408 ± 12% +27.8% 496564 ± 4% interrupts.CPU45.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU46.87:IR-PCI-MSI.1572910-edge.eth0-TxRx-46 91739 ± 10% -19.0% 74279 ± 8% interrupts.CPU46.CAL:Function_call_interrupts 1784609 +12.4% 2005522 interrupts.CPU46.LOC:Local_timer_interrupts 2970471 ± 9% -21.0% 2345439 ± 8% interrupts.CPU46.RES:Rescheduling_interrupts 108158 ± 10% -19.8% 86743 ± 4% interrupts.CPU46.TLB:TLB_shootdowns 388368 ± 12% +27.8% 496469 ± 4% interrupts.CPU46.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU47.88:IR-PCI-MSI.1572911-edge.eth0-TxRx-47 92856 ± 9% -20.0% 74327 ± 8% interrupts.CPU47.CAL:Function_call_interrupts 1784835 +12.4% 2005700 interrupts.CPU47.LOC:Local_timer_interrupts 3116242 ± 10% -20.4% 2480398 ± 5% interrupts.CPU47.RES:Rescheduling_interrupts 104927 ± 9% -18.6% 85393 ± 6% interrupts.CPU47.TLB:TLB_shootdowns 388403 ± 12% +27.8% 496554 ± 4% interrupts.CPU47.TRM:Thermal_event_interrupts 436.25 +12.6% 491.00 interrupts.CPU48.89:IR-PCI-MSI.1572912-edge.eth0-TxRx-48 92889 ± 11% -21.5% 72890 ± 8% interrupts.CPU48.CAL:Function_call_interrupts 1784772 +12.4% 2006205 interrupts.CPU48.LOC:Local_timer_interrupts 3175210 ± 12% -19.7% 2548339 ± 13% interrupts.CPU48.RES:Rescheduling_interrupts 105183 ± 11% -18.6% 85618 ± 6% interrupts.CPU48.TLB:TLB_shootdowns 388426 ± 12% +27.8% 496563 ± 4% interrupts.CPU48.TRM:Thermal_event_interrupts 441.25 ± 3% +14.1% 503.25 ± 3% interrupts.CPU49.90:IR-PCI-MSI.1572913-edge.eth0-TxRx-49 90134 ± 10% -18.3% 73603 ± 6% interrupts.CPU49.CAL:Function_call_interrupts 1785145 +12.4% 2005932 interrupts.CPU49.LOC:Local_timer_interrupts 3185609 ± 11% -17.6% 2626502 ± 4% interrupts.CPU49.RES:Rescheduling_interrupts 102737 ± 10% -17.3% 84932 ± 5% interrupts.CPU49.TLB:TLB_shootdowns 388425 ± 12% +27.8% 496560 ± 4% interrupts.CPU49.TRM:Thermal_event_interrupts 92114 ± 9% -23.2% 70721 ± 6% interrupts.CPU5.CAL:Function_call_interrupts 1785135 +12.3% 2005424 interrupts.CPU5.LOC:Local_timer_interrupts 3210646 ± 12% -24.8% 2415935 ± 4% interrupts.CPU5.RES:Rescheduling_interrupts 107349 ± 9% -18.9% 87034 ± 6% interrupts.CPU5.TLB:TLB_shootdowns 388418 ± 12% +27.8% 496564 ± 4% interrupts.CPU5.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU50.91:IR-PCI-MSI.1572914-edge.eth0-TxRx-50 92406 ± 10% -20.2% 73771 ± 8% interrupts.CPU50.CAL:Function_call_interrupts 1785200 +12.4% 2006488 interrupts.CPU50.LOC:Local_timer_interrupts 106635 ± 9% -19.9% 85431 ± 6% interrupts.CPU50.TLB:TLB_shootdowns 388419 ± 12% +27.8% 496564 ± 4% interrupts.CPU50.TRM:Thermal_event_interrupts 436.25 +14.4% 499.25 ± 3% interrupts.CPU51.92:IR-PCI-MSI.1572915-edge.eth0-TxRx-51 91491 ± 9% -20.8% 72473 ± 6% interrupts.CPU51.CAL:Function_call_interrupts 1784683 +12.4% 2005527 interrupts.CPU51.LOC:Local_timer_interrupts 3051494 ± 8% -24.6% 2300032 ± 10% interrupts.CPU51.RES:Rescheduling_interrupts 104954 ± 10% -20.2% 83775 ± 5% interrupts.CPU51.TLB:TLB_shootdowns 388264 ± 12% +27.9% 496462 ± 4% interrupts.CPU51.TRM:Thermal_event_interrupts 92938 ± 11% -19.7% 74594 ± 7% interrupts.CPU52.CAL:Function_call_interrupts 1785451 +12.4% 2006171 interrupts.CPU52.LOC:Local_timer_interrupts 3025681 ± 11% -17.1% 2506800 ± 10% interrupts.CPU52.RES:Rescheduling_interrupts 107100 ± 11% -19.6% 86156 ± 6% interrupts.CPU52.TLB:TLB_shootdowns 388422 ± 12% +27.8% 496563 ± 4% interrupts.CPU52.TRM:Thermal_event_interrupts 439.25 ± 2% +11.7% 490.50 interrupts.CPU53.94:IR-PCI-MSI.1572917-edge.eth0-TxRx-53 93000 ± 12% -20.9% 73549 ± 7% interrupts.CPU53.CAL:Function_call_interrupts 1785290 +12.3% 2005441 interrupts.CPU53.LOC:Local_timer_interrupts 3212623 ± 7% -18.2% 2629478 ± 10% interrupts.CPU53.RES:Rescheduling_interrupts 107303 ± 11% -17.5% 88567 ± 10% interrupts.CPU53.TLB:TLB_shootdowns 388411 ± 12% +27.8% 496557 ± 4% interrupts.CPU53.TRM:Thermal_event_interrupts 437.75 +12.1% 490.50 interrupts.CPU54.95:IR-PCI-MSI.1572918-edge.eth0-TxRx-54 1785170 +12.0% 2000090 interrupts.CPU54.LOC:Local_timer_interrupts 5351 ± 26% -64.5% 1900 ± 99% interrupts.CPU54.NMI:Non-maskable_interrupts 5351 ± 26% -64.5% 1900 ± 99% interrupts.CPU54.PMI:Performance_monitoring_interrupts 2066106 ± 9% +92.7% 3982033 ± 6% interrupts.CPU54.RES:Rescheduling_interrupts 108150 ± 12% -21.2% 85173 ± 5% interrupts.CPU54.TLB:TLB_shootdowns 329294 ± 3% +15.4% 380146 ± 2% interrupts.CPU54.TRM:Thermal_event_interrupts 436.50 +12.4% 490.50 interrupts.CPU55.96:IR-PCI-MSI.1572919-edge.eth0-TxRx-55 91164 ± 12% -19.9% 73014 ± 6% interrupts.CPU55.CAL:Function_call_interrupts 1785472 +12.0% 1999115 interrupts.CPU55.LOC:Local_timer_interrupts 1286654 ± 6% +228.2% 4223034 ± 5% interrupts.CPU55.RES:Rescheduling_interrupts 104943 ± 11% -19.5% 84466 ± 5% interrupts.CPU55.TLB:TLB_shootdowns 329292 ± 3% +15.4% 379918 ± 2% interrupts.CPU55.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU56.97:IR-PCI-MSI.1572920-edge.eth0-TxRx-56 96934 ± 13% -23.4% 74215 ± 6% interrupts.CPU56.CAL:Function_call_interrupts 1785010 +12.1% 2000169 interrupts.CPU56.LOC:Local_timer_interrupts 1312758 ± 17% +194.5% 3866522 ± 3% interrupts.CPU56.RES:Rescheduling_interrupts 109307 ± 12% -22.1% 85139 ± 4% interrupts.CPU56.TLB:TLB_shootdowns 329273 ± 3% +15.4% 380131 ± 2% interrupts.CPU56.TRM:Thermal_event_interrupts 439.00 ± 2% +11.7% 490.50 interrupts.CPU57.98:IR-PCI-MSI.1572921-edge.eth0-TxRx-57 94960 ± 11% -22.9% 73246 ± 5% interrupts.CPU57.CAL:Function_call_interrupts 1785297 +12.0% 2000019 interrupts.CPU57.LOC:Local_timer_interrupts 1666475 ± 10% +169.6% 4492937 interrupts.CPU57.RES:Rescheduling_interrupts 107534 ± 10% -22.0% 83871 ± 5% interrupts.CPU57.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380136 ± 2% interrupts.CPU57.TRM:Thermal_event_interrupts 442.25 ± 3% +10.9% 490.50 interrupts.CPU58.99:IR-PCI-MSI.1572922-edge.eth0-TxRx-58 92818 ± 12% -21.4% 72925 ± 5% interrupts.CPU58.CAL:Function_call_interrupts 1785429 +12.0% 1999528 interrupts.CPU58.LOC:Local_timer_interrupts 1606064 ± 5% +176.8% 4446179 ± 6% interrupts.CPU58.RES:Rescheduling_interrupts 103696 ± 11% -19.5% 83458 ± 4% interrupts.CPU58.TLB:TLB_shootdowns 329293 ± 3% +15.4% 380022 ± 2% interrupts.CPU58.TRM:Thermal_event_interrupts 439.00 +11.7% 490.50 interrupts.CPU59.100:IR-PCI-MSI.1572923-edge.eth0-TxRx-59 93876 ± 13% -20.5% 74669 ± 6% interrupts.CPU59.CAL:Function_call_interrupts 1785085 +12.0% 1999992 interrupts.CPU59.LOC:Local_timer_interrupts 1634812 ± 5% +174.5% 4487501 ± 4% interrupts.CPU59.RES:Rescheduling_interrupts 106929 ± 13% -21.0% 84522 ± 5% interrupts.CPU59.TLB:TLB_shootdowns 329291 ± 3% +15.4% 380111 ± 2% interrupts.CPU59.TRM:Thermal_event_interrupts 89930 ± 9% -20.5% 71520 ± 7% interrupts.CPU6.CAL:Function_call_interrupts 1785220 +12.3% 2005468 interrupts.CPU6.LOC:Local_timer_interrupts 106307 ± 7% -19.9% 85162 ± 6% interrupts.CPU6.TLB:TLB_shootdowns 388421 ± 12% +27.8% 496564 ± 4% interrupts.CPU6.TRM:Thermal_event_interrupts 436.25 +13.1% 493.25 interrupts.CPU60.101:IR-PCI-MSI.1572924-edge.eth0-TxRx-60 98595 ± 11% -25.6% 73379 ± 5% interrupts.CPU60.CAL:Function_call_interrupts 1785116 +12.1% 2000324 interrupts.CPU60.LOC:Local_timer_interrupts 1604314 ± 8% +186.8% 4601614 ± 5% interrupts.CPU60.RES:Rescheduling_interrupts 110442 ± 10% -24.0% 83932 ± 5% interrupts.CPU60.TLB:TLB_shootdowns 329256 ± 3% +15.5% 380140 ± 2% interrupts.CPU60.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU61.102:IR-PCI-MSI.1572925-edge.eth0-TxRx-61 93035 ± 11% -21.4% 73124 ± 5% interrupts.CPU61.CAL:Function_call_interrupts 1785176 +12.1% 2000960 interrupts.CPU61.LOC:Local_timer_interrupts 4567 ± 67% -79.2% 952.00 ±171% interrupts.CPU61.NMI:Non-maskable_interrupts 4567 ± 67% -79.2% 952.00 ±171% interrupts.CPU61.PMI:Performance_monitoring_interrupts 1575186 ± 6% +194.9% 4644760 ± 4% interrupts.CPU61.RES:Rescheduling_interrupts 103665 ± 11% -18.7% 84259 ± 5% interrupts.CPU61.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380127 ± 2% interrupts.CPU61.TRM:Thermal_event_interrupts 436.25 +12.4% 490.50 interrupts.CPU62.103:IR-PCI-MSI.1572926-edge.eth0-TxRx-62 95188 ± 10% -21.9% 74376 ± 6% interrupts.CPU62.CAL:Function_call_interrupts 1785028 +12.0% 1999262 interrupts.CPU62.LOC:Local_timer_interrupts 1613834 ± 8% +184.7% 4594499 ± 3% interrupts.CPU62.RES:Rescheduling_interrupts 106856 ± 10% -21.4% 83987 ± 5% interrupts.CPU62.TLB:TLB_shootdowns 329270 ± 3% +15.4% 380120 ± 2% interrupts.CPU62.TRM:Thermal_event_interrupts 94411 ± 11% -21.3% 74269 ± 5% interrupts.CPU63.CAL:Function_call_interrupts 1785322 +12.0% 1999237 interrupts.CPU63.LOC:Local_timer_interrupts 1669218 ± 6% +187.2% 4794684 ± 3% interrupts.CPU63.RES:Rescheduling_interrupts 105402 ± 10% -18.3% 86076 ± 8% interrupts.CPU63.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380141 ± 2% interrupts.CPU63.TRM:Thermal_event_interrupts 96060 ± 11% -22.6% 74364 ± 6% interrupts.CPU64.CAL:Function_call_interrupts 1785304 +12.0% 1999571 interrupts.CPU64.LOC:Local_timer_interrupts 1681501 ± 7% +164.0% 4438647 ± 3% interrupts.CPU64.RES:Rescheduling_interrupts 108534 ± 10% -22.8% 83813 ± 5% interrupts.CPU64.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380142 ± 2% interrupts.CPU64.TRM:Thermal_event_interrupts 94178 ± 9% -22.7% 72758 ± 8% interrupts.CPU65.CAL:Function_call_interrupts 1785338 +12.0% 1999408 interrupts.CPU65.LOC:Local_timer_interrupts 1674108 ± 4% +182.2% 4724262 ± 3% interrupts.CPU65.RES:Rescheduling_interrupts 107519 ± 10% -22.6% 83259 ± 5% interrupts.CPU65.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380128 ± 2% interrupts.CPU65.TRM:Thermal_event_interrupts 95135 ± 8% -22.4% 73795 ± 5% interrupts.CPU66.CAL:Function_call_interrupts 1785413 +12.0% 2000352 interrupts.CPU66.LOC:Local_timer_interrupts 616.75 ±171% +902.4% 6182 ± 28% interrupts.CPU66.NMI:Non-maskable_interrupts 616.75 ±171% +902.4% 6182 ± 28% interrupts.CPU66.PMI:Performance_monitoring_interrupts 1646166 ± 6% +169.3% 4433267 ± 6% interrupts.CPU66.RES:Rescheduling_interrupts 107190 ± 7% -22.5% 83034 ± 4% interrupts.CPU66.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380118 ± 2% interrupts.CPU66.TRM:Thermal_event_interrupts 95627 ± 10% -22.4% 74249 ± 5% interrupts.CPU67.CAL:Function_call_interrupts 1785314 +12.1% 2001128 interrupts.CPU67.LOC:Local_timer_interrupts 1611146 ± 4% +183.8% 4572204 ± 7% interrupts.CPU67.RES:Rescheduling_interrupts 109283 ± 11% -22.5% 84676 ± 4% interrupts.CPU67.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380142 ± 2% interrupts.CPU67.TRM:Thermal_event_interrupts 95479 ± 11% -20.6% 75791 ± 5% interrupts.CPU68.CAL:Function_call_interrupts 1785192 +12.0% 1999411 interrupts.CPU68.LOC:Local_timer_interrupts 1575394 ± 7% +189.9% 4567277 ± 4% interrupts.CPU68.RES:Rescheduling_interrupts 106972 ± 12% -18.6% 87039 ± 5% interrupts.CPU68.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380129 ± 2% interrupts.CPU68.TRM:Thermal_event_interrupts 93067 ± 14% -19.9% 74538 ± 4% interrupts.CPU69.CAL:Function_call_interrupts 1785257 +12.1% 2000636 interrupts.CPU69.LOC:Local_timer_interrupts 1578663 ± 9% +192.9% 4624518 ± 3% interrupts.CPU69.RES:Rescheduling_interrupts 104271 ± 12% -18.5% 84989 ± 2% interrupts.CPU69.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380066 ± 2% interrupts.CPU69.TRM:Thermal_event_interrupts 474.00 ± 3% +49.7% 709.75 ± 24% interrupts.CPU7.46:IR-PCI-MSI.1572871-edge.eth0-TxRx-7 92062 ± 9% -19.7% 73884 ± 5% interrupts.CPU7.CAL:Function_call_interrupts 1784266 +12.4% 2004980 interrupts.CPU7.LOC:Local_timer_interrupts 2505703 ± 11% -19.0% 2029187 ± 5% interrupts.CPU7.RES:Rescheduling_interrupts 107698 ± 8% -18.4% 87903 ± 6% interrupts.CPU7.TLB:TLB_shootdowns 388281 ± 12% +27.8% 496347 ± 4% interrupts.CPU7.TRM:Thermal_event_interrupts 92337 ± 12% -17.5% 76148 ± 2% interrupts.CPU70.CAL:Function_call_interrupts 1785161 +12.0% 1999757 interrupts.CPU70.LOC:Local_timer_interrupts 1562360 ± 4% +170.8% 4231208 ± 9% interrupts.CPU70.RES:Rescheduling_interrupts 103862 ± 10% -18.8% 84371 ± 3% interrupts.CPU70.TLB:TLB_shootdowns 329296 ± 3% +15.4% 380141 ± 2% interrupts.CPU70.TRM:Thermal_event_interrupts 95829 ± 12% -21.2% 75509 ± 5% interrupts.CPU71.CAL:Function_call_interrupts 1784894 +12.1% 2001287 interrupts.CPU71.LOC:Local_timer_interrupts 1533567 ± 7% +196.9% 4553440 interrupts.CPU71.RES:Rescheduling_interrupts 105883 ± 11% -20.4% 84248 ± 6% interrupts.CPU71.TLB:TLB_shootdowns 329247 ± 3% +15.5% 380146 ± 2% interrupts.CPU71.TRM:Thermal_event_interrupts 88372 ± 11% -20.3% 70389 ± 7% interrupts.CPU8.CAL:Function_call_interrupts 1784452 +12.4% 2005305 interrupts.CPU8.LOC:Local_timer_interrupts 103992 ± 10% -17.4% 85887 ± 5% interrupts.CPU8.TLB:TLB_shootdowns 388422 ± 12% +27.8% 496557 ± 4% interrupts.CPU8.TRM:Thermal_event_interrupts 89290 ± 10% -19.1% 72221 ± 6% interrupts.CPU9.CAL:Function_call_interrupts 1784716 +12.4% 2005710 interrupts.CPU9.LOC:Local_timer_interrupts 104340 ± 10% -16.0% 87647 ± 3% interrupts.CPU9.TLB:TLB_shootdowns 388408 ± 12% +27.8% 496557 ± 4% interrupts.CPU9.TRM:Thermal_event_interrupts 1.285e+08 +12.2% 1.442e+08 interrupts.LOC:Local_timer_interrupts 144.00 +50.0% 216.00 interrupts.MCP:Machine_check_polls 1.677e+08 ± 8% +50.4% 2.522e+08 ± 2% interrupts.RES:Rescheduling_interrupts 7624145 ± 10% -19.7% 6125835 ± 5% interrupts.TLB:TLB_shootdowns 25836833 ± 8% +22.2% 31559779 ± 2% interrupts.TRM:Thermal_event_interrupts *************************************************************************************************** lkp-bdw-ep6: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory ========================================================================================= compiler/cpufreq_governor/ipc/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode: gcc-7/performance/pipe/x86_64-rhel-7.6/process/50%/debian-x86_64-2019-11-14.cgz/lkp-bdw-ep6/hackbench/0xb000038 commit: 43e9f7f231 ("sched/fair: Start tracking SCHED_IDLE tasks count in cfs_rq") 3c29e651e1 ("sched/fair: Fall back to sched-idle CPU if idle CPU isn't found") 43e9f7f231e40e45 3c29e651e16dd3b3179cfb2d055 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 -25% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x %stddev %change %stddev \ | \ 134091 -10.6% 119937 hackbench.throughput 3.352e+09 ± 2% +16.1% 3.892e+09 ± 5% hackbench.time.involuntary_context_switches 535571 ± 2% +10.6% 592420 ± 4% numa-meminfo.node0.FilePages 12614 ± 18% -12.2% 11078 ± 18% numa-meminfo.node1.Mapped 3472 ± 6% +18.0% 4096 slabinfo.skbuff_head_cache.active_objs 3568 ± 6% +15.7% 4128 slabinfo.skbuff_head_cache.num_objs 730128 ± 5% -52.2% 349073 ± 25% turbostat.C3 14.77 +5.9% 15.64 turbostat.RAMWatt 24613783 ± 5% -32.6% 16595193 ± 23% numa-numastat.node0.local_node 24632941 ± 5% -32.6% 16605181 ± 23% numa-numastat.node0.numa_hit 25827961 -29.9% 18097150 ± 19% numa-numastat.node1.local_node 25836948 -29.9% 18115676 ± 19% numa-numastat.node1.numa_hit 50467170 ± 2% -31.2% 34733456 ± 10% proc-vmstat.numa_hit 50439016 ± 2% -31.2% 34704933 ± 10% proc-vmstat.numa_local 50622650 ± 2% -31.1% 34896854 ± 10% proc-vmstat.pgalloc_normal 50548381 ± 2% -31.1% 34834391 ± 10% proc-vmstat.pgfree 79.00 +2.2% 80.75 vmstat.cpu.sy 19.00 -7.9% 17.50 ± 2% vmstat.cpu.us 13450755 +4.6% 14066265 vmstat.system.cs 1122772 -12.1% 986681 vmstat.system.in 133891 ± 2% +10.6% 148112 ± 4% numa-vmstat.node0.nr_file_pages 12301995 ± 4% -30.2% 8590153 ± 22% numa-vmstat.node0.numa_hit 12282910 ± 4% -30.1% 8580139 ± 22% numa-vmstat.node0.numa_local 12964354 ± 5% -31.2% 8914408 ± 18% numa-vmstat.node1.numa_hit 12806275 ± 5% -31.7% 8746928 ± 19% numa-vmstat.node1.numa_local 33816695 ± 10% +17.8% 39824104 ± 7% cpuidle.C1.time 21015951 ± 3% +19.6% 25127666 ± 7% cpuidle.C1E.time 730991 ± 5% -52.1% 350128 ± 25% cpuidle.C3.usage 1.497e+08 ± 28% +41.6% 2.119e+08 ± 15% cpuidle.C6.time 6771631 ± 4% -45.2% 3713804 ± 15% cpuidle.POLL.time 5209842 ± 4% -44.1% 2913245 ± 15% cpuidle.POLL.usage 26965506 +5.5% 28444381 ± 4% sched_debug.cfs_rq:/.min_vruntime.avg 28257967 +6.6% 30126596 ± 4% sched_debug.cfs_rq:/.min_vruntime.max 582203 ± 9% +37.0% 797353 ± 8% sched_debug.cfs_rq:/.min_vruntime.stddev 4.65 ± 8% -10.7% 4.16 ± 6% sched_debug.cfs_rq:/.runnable_load_avg.stddev 4620 ± 6% -10.7% 4125 ± 4% sched_debug.cfs_rq:/.runnable_weight.stddev 582831 ± 9% +36.6% 796226 ± 7% sched_debug.cfs_rq:/.spread0.stddev 154.73 ± 2% -14.7% 132.00 ± 3% sched_debug.cfs_rq:/.util_avg.stddev 120.93 ± 25% +43.5% 173.50 ± 19% sched_debug.cfs_rq:/.util_est_enqueued.min 184.93 ± 2% -11.0% 164.54 ± 3% sched_debug.cfs_rq:/.util_est_enqueued.stddev 182135 ± 18% +57.1% 286098 ± 6% sched_debug.cpu.avg_idle.avg 185926 ± 19% +49.0% 276991 ± 9% sched_debug.cpu.avg_idle.stddev 3.32 ± 4% -10.8% 2.96 ± 3% sched_debug.cpu.nr_running.stddev 46710201 +9.0% 50934866 ± 5% sched_debug.cpu.nr_switches.avg 50053735 +16.9% 58491253 ± 7% sched_debug.cpu.nr_switches.max 43751407 +6.7% 46698664 ± 4% sched_debug.cpu.nr_switches.min 1344975 ± 10% +78.9% 2406504 ± 16% sched_debug.cpu.nr_switches.stddev 1.63 ± 3% -32.8% 1.09 ± 11% sched_debug.cpu.nr_uninterruptible.avg 1017 ± 27% -59.5% 411.75 ± 16% interrupts.40:IR-PCI-MSI.1572871-edge.eth0-TxRx-6 6547829 -14.4% 5603910 ± 5% interrupts.CPU16.RES:Rescheduling_interrupts 1017 ± 27% -59.5% 411.75 ± 16% interrupts.CPU19.40:IR-PCI-MSI.1572871-edge.eth0-TxRx-6 6801598 -14.7% 5801847 ± 5% interrupts.CPU19.RES:Rescheduling_interrupts 6609756 -13.7% 5703084 ± 8% interrupts.CPU20.RES:Rescheduling_interrupts 6446878 ± 2% -11.8% 5689145 ± 7% interrupts.CPU21.RES:Rescheduling_interrupts 6634825 -9.4% 6008203 ± 6% interrupts.CPU24.RES:Rescheduling_interrupts 6801182 -9.8% 6137528 ± 3% interrupts.CPU25.RES:Rescheduling_interrupts 6687252 -12.9% 5821967 ± 6% interrupts.CPU28.RES:Rescheduling_interrupts 6583387 -10.9% 5868659 ± 6% interrupts.CPU29.RES:Rescheduling_interrupts 6585631 -16.6% 5490726 ± 9% interrupts.CPU37.RES:Rescheduling_interrupts 6759973 -13.2% 5867800 ± 7% interrupts.CPU57.RES:Rescheduling_interrupts 6459616 ± 2% -10.5% 5783047 ± 6% interrupts.CPU58.RES:Rescheduling_interrupts 6580818 ± 2% -16.9% 5468945 ± 3% interrupts.CPU60.RES:Rescheduling_interrupts 6723178 ± 2% -12.4% 5887778 ± 3% interrupts.CPU63.RES:Rescheduling_interrupts 6480129 -16.2% 5427168 ± 7% interrupts.CPU64.RES:Rescheduling_interrupts 6439558 ± 2% -11.6% 5691160 ± 5% interrupts.CPU65.RES:Rescheduling_interrupts 6662822 -12.0% 5866390 ± 7% interrupts.CPU68.RES:Rescheduling_interrupts 6602178 -12.0% 5809760 ± 7% interrupts.CPU7.RES:Rescheduling_interrupts 6616172 -14.1% 5685889 ± 7% interrupts.CPU73.RES:Rescheduling_interrupts 6591477 -13.8% 5682655 ± 8% interrupts.CPU81.RES:Rescheduling_interrupts 13.97 +15.8% 16.18 perf-stat.i.MPKI 1.70 -0.1 1.65 perf-stat.i.branch-miss-rate% 3.607e+08 -1.6% 3.549e+08 perf-stat.i.branch-misses 1.22 ± 6% +0.8 2.01 ± 18% perf-stat.i.cache-miss-rate% 17650711 ± 6% +98.0% 34953649 ± 18% perf-stat.i.cache-misses 1.496e+09 +17.3% 1.755e+09 perf-stat.i.cache-references 13503062 +4.6% 14122231 perf-stat.i.context-switches 2946654 ± 2% +18.5% 3491269 ± 3% perf-stat.i.cpu-migrations 29365 ± 8% -30.6% 20387 ± 31% perf-stat.i.cycles-between-cache-misses 0.35 ± 2% +0.0 0.36 ± 2% perf-stat.i.dTLB-store-miss-rate% 2.048e+10 -2.1% 2.006e+10 perf-stat.i.dTLB-stores 55.17 -1.4 53.74 perf-stat.i.iTLB-load-miss-rate% 1.019e+08 +5.5% 1.074e+08 perf-stat.i.iTLB-loads 3068 -3.0% 2975 perf-stat.i.minor-faults 9195204 ± 6% +113.0% 19589023 ± 18% perf-stat.i.node-load-misses 28132 ± 6% +125.3% 63396 ± 13% perf-stat.i.node-loads 58.92 -9.8 49.16 ± 4% perf-stat.i.node-store-miss-rate% 3588196 ± 6% +60.5% 5760265 ± 19% perf-stat.i.node-store-misses 2415953 ± 5% +145.2% 5924042 ± 18% perf-stat.i.node-stores 3068 -3.0% 2975 perf-stat.i.page-faults 13.96 +16.0% 16.19 perf-stat.overall.MPKI 1.70 -0.1 1.65 perf-stat.overall.branch-miss-rate% 1.18 ± 6% +0.8 1.99 ± 18% perf-stat.overall.cache-miss-rate% 13823 ± 6% -47.8% 7221 ± 20% perf-stat.overall.cycles-between-cache-misses 0.35 ± 2% +0.0 0.36 ± 2% perf-stat.overall.dTLB-store-miss-rate% 55.11 -1.4 53.70 perf-stat.overall.iTLB-load-miss-rate% 59.75 -10.5 49.23 perf-stat.overall.node-store-miss-rate% 78192 +13.0% 88370 perf-stat.overall.path-length 3.601e+08 -1.6% 3.543e+08 perf-stat.ps.branch-misses 17622620 ± 6% +98.0% 34899005 ± 18% perf-stat.ps.cache-misses 1.494e+09 +17.3% 1.752e+09 perf-stat.ps.cache-references 13480657 +4.6% 14099389 perf-stat.ps.context-switches 2941749 ± 2% +18.5% 3485631 ± 3% perf-stat.ps.cpu-migrations 2.045e+10 -2.1% 2.003e+10 perf-stat.ps.dTLB-stores 1.017e+08 +5.5% 1.072e+08 perf-stat.ps.iTLB-loads 3064 -3.0% 2972 perf-stat.ps.minor-faults 9180406 ± 6% +113.0% 19558101 ± 18% perf-stat.ps.node-load-misses 28101 ± 6% +125.3% 63314 ± 13% perf-stat.ps.node-loads 3582467 ± 6% +60.5% 5751138 ± 19% perf-stat.ps.node-store-misses 2412086 ± 5% +145.2% 5914636 ± 18% perf-stat.ps.node-stores 3064 -3.0% 2972 perf-stat.ps.page-faults 223485 +9.1% 243918 ± 5% softirqs.CPU0.TIMER 11887 ± 11% -18.2% 9723 ± 6% softirqs.CPU1.SCHED 215411 +9.2% 235267 ± 6% softirqs.CPU1.TIMER 11293 ± 3% -14.3% 9678 ± 10% softirqs.CPU10.SCHED 213916 +9.8% 234908 ± 5% softirqs.CPU10.TIMER 212758 +7.7% 229141 ± 5% softirqs.CPU11.TIMER 11098 ± 2% -15.9% 9337 ± 10% softirqs.CPU12.SCHED 213397 +9.0% 232706 ± 6% softirqs.CPU12.TIMER 212419 +8.3% 230115 ± 6% softirqs.CPU13.TIMER 10777 ± 4% -12.5% 9426 ± 6% softirqs.CPU14.SCHED 11241 ± 3% -11.5% 9950 ± 6% softirqs.CPU15.SCHED 214604 +10.0% 236116 ± 8% softirqs.CPU15.TIMER 11147 -14.9% 9482 ± 8% softirqs.CPU16.SCHED 216343 +8.4% 234558 ± 6% softirqs.CPU16.TIMER 214680 +7.0% 229748 ± 6% softirqs.CPU17.TIMER 212791 +8.6% 230988 ± 6% softirqs.CPU19.TIMER 11211 ± 4% -10.4% 10047 ± 6% softirqs.CPU2.SCHED 219918 +9.8% 241513 ± 6% softirqs.CPU2.TIMER 10825 ± 4% -14.3% 9273 ± 7% softirqs.CPU20.SCHED 215209 +8.8% 234149 ± 6% softirqs.CPU20.TIMER 10869 ± 4% -11.4% 9626 ± 4% softirqs.CPU21.SCHED 216225 +8.7% 235105 ± 5% softirqs.CPU21.TIMER 13013 ± 7% -14.0% 11192 ± 3% softirqs.CPU22.SCHED 220406 +8.9% 239975 ± 4% softirqs.CPU22.TIMER 11925 -14.9% 10145 ± 3% softirqs.CPU23.SCHED 220231 +7.6% 237056 ± 5% softirqs.CPU23.TIMER 11140 ± 2% -11.5% 9861 ± 7% softirqs.CPU24.SCHED 216091 +8.9% 235338 ± 5% softirqs.CPU24.TIMER 11438 ± 3% -16.3% 9578 ± 2% softirqs.CPU25.SCHED 214880 +9.4% 235143 ± 5% softirqs.CPU25.TIMER 214888 +8.8% 233874 ± 5% softirqs.CPU26.TIMER 11410 ± 2% -13.7% 9844 ± 10% softirqs.CPU27.SCHED 215012 +8.7% 233790 ± 4% softirqs.CPU27.TIMER 220618 ± 2% +7.8% 237788 ± 5% softirqs.CPU28.TIMER 11096 ± 5% -14.3% 9513 ± 6% softirqs.CPU29.SCHED 217194 +8.2% 235108 ± 5% softirqs.CPU29.TIMER 11445 ± 2% -16.7% 9536 ± 10% softirqs.CPU3.SCHED 214692 +9.1% 234221 ± 5% softirqs.CPU3.TIMER 11127 ± 5% -10.5% 9954 ± 6% softirqs.CPU30.SCHED 214861 +8.2% 232532 ± 5% softirqs.CPU30.TIMER 217582 +7.8% 234493 ± 4% softirqs.CPU31.TIMER 214606 +8.8% 233473 ± 4% softirqs.CPU32.TIMER 11299 ± 2% -11.7% 9975 ± 6% softirqs.CPU33.SCHED 11503 ± 2% -16.0% 9658 ± 10% softirqs.CPU34.SCHED 213233 +10.9% 236580 ± 6% softirqs.CPU34.TIMER 11162 ± 2% -11.6% 9863 ± 7% softirqs.CPU35.SCHED 11241 -13.9% 9678 ± 6% softirqs.CPU36.SCHED 215681 +9.2% 235446 ± 4% softirqs.CPU36.TIMER 11358 -15.2% 9632 ± 6% softirqs.CPU37.SCHED 219691 ± 2% +18.5% 260407 ± 13% softirqs.CPU37.TIMER 11570 ± 2% -13.1% 10049 ± 6% softirqs.CPU38.SCHED 215559 +8.2% 233189 ± 4% softirqs.CPU38.TIMER 11295 ± 4% -13.1% 9819 ± 10% softirqs.CPU39.SCHED 215781 ± 2% +7.6% 232108 ± 5% softirqs.CPU39.TIMER 11506 ± 7% -13.0% 10006 ± 6% softirqs.CPU4.SCHED 214973 +8.9% 234140 ± 6% softirqs.CPU4.TIMER 11304 ± 2% -13.7% 9759 ± 10% softirqs.CPU40.SCHED 213827 +10.1% 235341 ± 4% softirqs.CPU40.TIMER 11462 ± 4% -13.5% 9913 ± 8% softirqs.CPU41.SCHED 215879 ± 2% +7.2% 231499 ± 5% softirqs.CPU41.TIMER 11418 ± 3% -14.0% 9819 ± 8% softirqs.CPU43.SCHED 214864 +8.3% 232698 ± 4% softirqs.CPU43.TIMER 10706 ± 5% -13.9% 9223 ± 11% softirqs.CPU44.SCHED 217386 +9.4% 237916 ± 5% softirqs.CPU44.TIMER 215191 +9.7% 236017 ± 5% softirqs.CPU45.TIMER 214214 +9.9% 235466 ± 6% softirqs.CPU46.TIMER 214471 +9.9% 235675 ± 5% softirqs.CPU47.TIMER 11251 ± 2% -13.0% 9785 ± 8% softirqs.CPU48.SCHED 212500 +9.9% 233451 ± 6% softirqs.CPU48.TIMER 213609 +9.9% 234657 ± 5% softirqs.CPU49.TIMER 216836 +8.5% 235225 ± 5% softirqs.CPU5.TIMER 215854 +9.0% 235383 ± 5% softirqs.CPU50.TIMER 11468 -14.4% 9820 ± 9% softirqs.CPU51.SCHED 215402 +8.8% 234360 ± 5% softirqs.CPU51.TIMER 214152 +8.8% 233038 ± 5% softirqs.CPU52.TIMER 212843 +8.4% 230716 ± 5% softirqs.CPU53.TIMER 11373 ± 2% -14.9% 9679 ± 7% softirqs.CPU54.SCHED 214008 +9.5% 234380 ± 5% softirqs.CPU54.TIMER 212619 +7.5% 228539 ± 5% softirqs.CPU55.TIMER 11136 ± 3% -16.4% 9312 ± 10% softirqs.CPU56.SCHED 213404 +9.0% 232542 ± 6% softirqs.CPU56.TIMER 11171 ± 4% -11.9% 9837 ± 7% softirqs.CPU57.SCHED 212283 +8.6% 230474 ± 6% softirqs.CPU57.TIMER 11137 ± 4% -14.0% 9578 ± 6% softirqs.CPU58.SCHED 11226 ± 4% -17.2% 9289 ± 4% softirqs.CPU59.SCHED 214193 +23.1% 263668 ± 22% softirqs.CPU59.TIMER 217195 +8.2% 234975 ± 5% softirqs.CPU6.TIMER 11051 ± 4% -14.4% 9460 ± 9% softirqs.CPU60.SCHED 217131 +7.9% 234363 ± 5% softirqs.CPU60.TIMER 11206 ± 3% -12.6% 9793 ± 9% softirqs.CPU61.SCHED 211791 +9.2% 231302 ± 5% softirqs.CPU61.TIMER 214956 ± 2% +8.8% 233770 ± 5% softirqs.CPU62.TIMER 11180 ± 3% -14.8% 9526 ± 8% softirqs.CPU63.SCHED 213413 +8.7% 232008 ± 5% softirqs.CPU63.TIMER 215674 +8.1% 233086 ± 6% softirqs.CPU64.TIMER 11032 ± 3% -15.7% 9302 ± 7% softirqs.CPU65.SCHED 215870 +9.0% 235386 ± 5% softirqs.CPU65.TIMER 11403 ± 3% -14.2% 9786 ± 6% softirqs.CPU66.SCHED 215968 +9.4% 236175 ± 5% softirqs.CPU66.TIMER 11098 ± 3% -7.4% 10281 ± 4% softirqs.CPU67.SCHED 217846 +8.3% 236023 ± 4% softirqs.CPU67.TIMER 11094 -12.2% 9740 ± 7% softirqs.CPU68.SCHED 215512 +9.4% 235796 ± 5% softirqs.CPU68.TIMER 11321 ± 3% -14.1% 9724 ± 3% softirqs.CPU69.SCHED 213564 +9.0% 232867 ± 5% softirqs.CPU69.TIMER 218822 +8.8% 238080 ± 6% softirqs.CPU7.TIMER 11507 ± 2% -12.9% 10026 ± 4% softirqs.CPU70.SCHED 213928 +9.4% 233986 ± 4% softirqs.CPU70.TIMER 213528 +8.5% 231730 ± 4% softirqs.CPU71.TIMER 11262 ± 2% -11.1% 10013 ± 5% softirqs.CPU72.SCHED 11145 ± 3% -14.5% 9533 ± 8% softirqs.CPU73.SCHED 216467 +9.2% 236409 ± 5% softirqs.CPU73.TIMER 11447 ± 4% -13.2% 9935 ± 4% softirqs.CPU74.SCHED 212578 +9.1% 232022 ± 5% softirqs.CPU74.TIMER 215256 +7.8% 232072 ± 5% softirqs.CPU75.TIMER 11324 ± 4% -11.8% 9985 ± 3% softirqs.CPU76.SCHED 214170 +8.6% 232516 ± 5% softirqs.CPU76.TIMER 11646 ± 5% -15.7% 9824 ± 7% softirqs.CPU77.SCHED 213706 +8.0% 230880 ± 5% softirqs.CPU77.TIMER 11397 ± 6% -13.9% 9810 ± 10% softirqs.CPU78.SCHED 11179 ± 3% -11.5% 9894 ± 5% softirqs.CPU79.SCHED 213215 +8.4% 231081 ± 5% softirqs.CPU79.TIMER 11165 ± 4% -16.2% 9352 ± 9% softirqs.CPU8.SCHED 216508 ± 2% +7.9% 233692 ± 6% softirqs.CPU8.TIMER 11203 ± 2% -13.9% 9643 ± 4% softirqs.CPU80.SCHED 216446 +8.3% 234469 ± 5% softirqs.CPU80.TIMER 11151 ± 2% -12.8% 9726 ± 6% softirqs.CPU81.SCHED 11373 ± 3% -13.0% 9891 ± 6% softirqs.CPU82.SCHED 213597 +8.9% 232608 ± 5% softirqs.CPU82.TIMER 214308 +7.9% 231148 ± 5% softirqs.CPU83.TIMER 214384 +20.9% 259149 ± 15% softirqs.CPU84.TIMER 11458 -13.7% 9887 ± 7% softirqs.CPU86.SCHED 215459 +8.2% 233058 ± 5% softirqs.CPU86.TIMER 11303 ± 4% -12.4% 9904 ± 8% softirqs.CPU87.SCHED 214185 +7.9% 231116 ± 5% softirqs.CPU87.TIMER 11030 ± 2% -13.0% 9593 ± 8% softirqs.CPU9.SCHED 213229 +8.7% 231788 ± 6% softirqs.CPU9.TIMER 995635 -12.9% 867688 ± 5% softirqs.SCHED 19052052 +8.6% 20690944 ± 5% softirqs.TIMER 31.32 -1.6 29.74 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 30.24 -1.5 28.76 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 26.06 -0.9 25.17 perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.35 ± 5% -0.9 1.48 ± 12% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common 25.36 -0.8 24.54 perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64 2.75 ± 4% -0.7 2.08 ± 7% perf-profile.calltrace.cycles-pp._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 4.24 ± 2% -0.6 3.60 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64 3.84 ± 2% -0.6 3.23 ± 2% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret 7.83 -0.5 7.30 ± 2% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function 1.68 ± 6% -0.5 1.17 ± 10% perf-profile.calltrace.cycles-pp.update_cfs_group.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 8.02 -0.5 7.53 ± 2% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common 8.08 -0.5 7.60 ± 2% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 1.63 ± 4% -0.4 1.19 ± 6% perf-profile.calltrace.cycles-pp.mutex_unlock.pipe_write.new_sync_write.vfs_write.ksys_write 1.71 ± 6% -0.4 1.28 ± 8% perf-profile.calltrace.cycles-pp.update_cfs_group.dequeue_task_fair.__schedule.schedule.pipe_wait 2.60 -0.4 2.19 ± 4% perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.03 ± 2% -0.4 1.64 ± 4% perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.47 -0.4 3.08 ± 4% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 1.95 ± 2% -0.3 1.61 ± 3% perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.new_sync_write.vfs_write.ksys_write 2.67 -0.3 2.35 ± 3% perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_read.new_sync_read.vfs_read.ksys_read 0.56 -0.3 0.26 ±100% perf-profile.calltrace.cycles-pp.fsnotify.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.22 ± 4% -0.3 0.94 ± 2% perf-profile.calltrace.cycles-pp.mutex_lock.pipe_write.new_sync_write.vfs_write.ksys_write 1.25 ± 2% -0.3 1.00 ± 5% perf-profile.calltrace.cycles-pp.selinux_file_permission.security_file_permission.vfs_write.ksys_write.do_syscall_64 2.64 -0.2 2.41 ± 2% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.pipe_wait 1.53 -0.2 1.34 ± 7% perf-profile.calltrace.cycles-pp.selinux_file_permission.security_file_permission.vfs_read.ksys_read.do_syscall_64 0.59 ± 4% -0.2 0.40 ± 57% perf-profile.calltrace.cycles-pp.__enqueue_entity.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 1.34 -0.2 1.16 ± 6% perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 1.07 ± 2% -0.2 0.90 ± 4% perf-profile.calltrace.cycles-pp.copyin.copy_page_from_iter.pipe_write.new_sync_write.vfs_write 1.05 -0.2 0.88 ± 6% perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_entity.dequeue_task_fair.__schedule.schedule 1.56 ± 2% -0.2 1.38 ± 2% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout.copy_page_to_iter.pipe_read.new_sync_read 1.17 ± 4% -0.2 1.01 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write 0.98 ± 2% -0.2 0.82 ± 4% perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.66 ± 2% -0.2 1.50 ± 5% perf-profile.calltrace.cycles-pp.copyout.copy_page_to_iter.pipe_read.new_sync_read.vfs_read 0.73 ± 3% -0.2 0.57 ± 2% perf-profile.calltrace.cycles-pp.file_has_perm.security_file_permission.vfs_read.ksys_read.do_syscall_64 0.96 ± 2% -0.1 0.82 ± 3% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.copy_page_from_iter.pipe_write.new_sync_write 0.89 ± 2% -0.1 0.77 ± 3% perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.66 ± 6% -0.1 0.54 ± 2% perf-profile.calltrace.cycles-pp.file_has_perm.security_file_permission.vfs_write.ksys_write.do_syscall_64 2.42 -0.1 2.32 perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.pipe_wait.pipe_read 0.89 ± 4% -0.1 0.79 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__wake_up_common_lock.pipe_write.new_sync_write 0.74 ± 2% -0.1 0.66 ± 3% perf-profile.calltrace.cycles-pp.update_curr.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate 0.84 -0.1 0.79 ± 3% perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule.pipe_wait 0.56 -0.0 0.54 ± 2% perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate 0.77 ± 2% +0.1 0.82 perf-profile.calltrace.cycles-pp.check_preempt_wakeup.check_preempt_curr.ttwu_do_wakeup.try_to_wake_up.autoremove_wake_function 0.90 ± 3% +0.1 0.97 ± 2% perf-profile.calltrace.cycles-pp.native_write_msr 0.69 ± 3% +0.1 0.77 ± 3% perf-profile.calltrace.cycles-pp.update_rq_clock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 0.80 ± 4% +0.1 0.89 ± 2% perf-profile.calltrace.cycles-pp.___perf_sw_event.__schedule.schedule.pipe_wait.pipe_read 1.10 +0.1 1.20 perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 0.73 ± 3% +0.1 0.83 ± 2% perf-profile.calltrace.cycles-pp.__switch_to 1.05 +0.1 1.16 perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.pipe_wait 0.63 ± 4% +0.1 0.75 ± 2% perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop 0.99 ± 5% +0.1 1.12 ± 5% perf-profile.calltrace.cycles-pp.set_task_cpu.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 1.07 ± 3% +0.1 1.21 ± 2% perf-profile.calltrace.cycles-pp.__switch_to_asm 0.64 ± 6% +0.2 0.80 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock.__schedule.schedule.pipe_wait.pipe_read 1.41 ± 3% +0.2 1.59 ± 2% perf-profile.calltrace.cycles-pp.switch_fpu_return.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.77 ± 5% +0.2 1.00 ± 5% perf-profile.calltrace.cycles-pp.load_new_mm_cr3.switch_mm_irqs_off.__schedule.schedule.exit_to_usermode_loop 1.73 ± 3% +0.3 2.06 ± 3% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop.do_syscall_64 0.14 ±173% +0.5 0.63 ± 5% perf-profile.calltrace.cycles-pp.update_rq_clock.__schedule.schedule.pipe_wait.pipe_read 0.13 ±173% +0.5 0.64 ± 5% perf-profile.calltrace.cycles-pp.migrate_task_rq_fair.set_task_cpu.try_to_wake_up.autoremove_wake_function.__wake_up_common 1.85 ± 5% +0.5 2.38 ± 3% perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.exit_to_usermode_loop.do_syscall_64 0.00 +0.6 0.59 ± 4% perf-profile.calltrace.cycles-pp.mutex_lock.pipe_wait.pipe_read.new_sync_read.vfs_read 0.00 +0.6 0.60 ± 5% perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.dequeue_task_fair.__schedule.schedule.pipe_wait 0.00 +0.7 0.67 ± 5% perf-profile.calltrace.cycles-pp.__update_load_avg_cfs_rq.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 0.15 ±173% +1.0 1.14 ± 12% perf-profile.calltrace.cycles-pp.cpumask_next_wrap.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function 87.66 +1.1 88.78 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 86.87 +1.1 88.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.59 ± 4% +1.4 6.96 ± 3% perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.72 ± 4% +1.4 7.12 ± 3% perf-profile.calltrace.cycles-pp.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.87 ± 4% +1.4 7.28 ± 3% perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.70 ± 30% +3.3 4.98 ± 13% perf-profile.calltrace.cycles-pp.available_idle_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function 35.54 +3.3 38.87 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 34.28 +3.5 37.82 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 31.12 +4.1 35.22 ± 2% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 30.49 +4.1 34.63 ± 2% perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64 22.54 ± 2% +6.0 28.53 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write 20.58 ± 2% +6.2 26.76 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write 19.97 ± 2% +6.2 26.16 ± 3% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write 19.73 ± 3% +6.2 25.92 ± 3% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write 3.39 ± 20% +7.0 10.39 ± 12% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common 4.89 ± 15% +7.1 11.98 ± 10% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 31.34 -1.6 29.75 perf-profile.children.cycles-pp.ksys_read 30.28 -1.5 28.79 perf-profile.children.cycles-pp.vfs_read 4.14 ± 5% -1.3 2.85 ± 7% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 3.51 ± 6% -1.0 2.54 ± 8% perf-profile.children.cycles-pp.update_cfs_group 26.10 -0.9 25.20 perf-profile.children.cycles-pp.new_sync_read 25.42 -0.8 24.59 perf-profile.children.cycles-pp.pipe_read 4.67 -0.8 3.86 ± 4% perf-profile.children.cycles-pp.security_file_permission 4.33 ± 2% -0.7 3.65 ± 2% perf-profile.children.cycles-pp.syscall_return_via_sysret 4.24 ± 2% -0.6 3.60 ± 3% perf-profile.children.cycles-pp.entry_SYSCALL_64 2.18 ± 4% -0.5 1.64 ± 6% perf-profile.children.cycles-pp.mutex_unlock 7.92 -0.5 7.40 ± 3% perf-profile.children.cycles-pp.enqueue_task_fair 8.10 -0.5 7.61 ± 3% perf-profile.children.cycles-pp.activate_task 8.16 -0.5 7.67 ± 3% perf-profile.children.cycles-pp.ttwu_do_activate 4.12 ± 4% -0.5 3.64 ± 5% perf-profile.children.cycles-pp._raw_spin_lock 2.83 -0.4 2.38 ± 6% perf-profile.children.cycles-pp.selinux_file_permission 3.53 -0.4 3.13 ± 4% perf-profile.children.cycles-pp.enqueue_entity 1.99 ± 2% -0.4 1.64 ± 3% perf-profile.children.cycles-pp.copy_page_from_iter 3.34 -0.4 2.99 ± 4% perf-profile.children.cycles-pp.update_load_avg 2.70 -0.3 2.39 ± 3% perf-profile.children.cycles-pp.copy_page_to_iter 2.66 -0.3 2.35 ± 3% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 0.93 ± 3% -0.3 0.64 ± 7% perf-profile.children.cycles-pp.__mutex_unlock_slowpath 1.41 ± 4% -0.3 1.13 perf-profile.children.cycles-pp.file_has_perm 2.75 -0.3 2.49 ± 2% perf-profile.children.cycles-pp.dequeue_entity 2.29 ± 2% -0.3 2.04 perf-profile.children.cycles-pp.mutex_lock 0.85 ± 3% -0.2 0.61 ± 4% perf-profile.children.cycles-pp.__inode_security_revalidate 0.59 ± 3% -0.2 0.39 ± 6% perf-profile.children.cycles-pp.__mutex_lock 1.79 ± 2% -0.2 1.59 ± 4% perf-profile.children.cycles-pp.__fdget_pos 1.70 ± 2% -0.2 1.52 ± 3% perf-profile.children.cycles-pp.__fget_light 0.93 ± 3% -0.2 0.75 perf-profile.children.cycles-pp.avc_has_perm 1.09 ± 2% -0.2 0.92 ± 4% perf-profile.children.cycles-pp.copyin 2.33 ± 2% -0.2 2.16 perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.96 ± 2% -0.2 0.81 ± 2% perf-profile.children.cycles-pp.___might_sleep 1.67 ± 2% -0.2 1.52 ± 5% perf-profile.children.cycles-pp.copyout 0.83 ± 4% -0.1 0.68 ± 2% perf-profile.children.cycles-pp._cond_resched 0.33 ± 5% -0.1 0.19 ± 10% perf-profile.children.cycles-pp.preempt_schedule_common 0.42 ± 2% -0.1 0.28 ± 8% perf-profile.children.cycles-pp.wake_up_q 0.70 ± 3% -0.1 0.59 ± 3% perf-profile.children.cycles-pp.__might_sleep 0.65 ± 5% -0.1 0.54 ± 7% perf-profile.children.cycles-pp.__fsnotify_parent 0.52 ± 2% -0.1 0.43 perf-profile.children.cycles-pp.__might_fault 0.53 ± 5% -0.1 0.45 perf-profile.children.cycles-pp.current_time 0.84 ± 3% -0.1 0.77 perf-profile.children.cycles-pp.fsnotify 0.16 ± 7% -0.1 0.09 ± 13% perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode 0.15 ± 8% -0.1 0.08 ± 15% perf-profile.children.cycles-pp.prepare_exit_to_usermode 0.22 ± 3% -0.1 0.16 ± 7% perf-profile.children.cycles-pp.wake_q_add 0.49 -0.1 0.44 perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.26 ± 4% -0.1 0.21 ± 2% perf-profile.children.cycles-pp.rcu_all_qs 0.08 ± 14% -0.1 0.03 ±100% perf-profile.children.cycles-pp.__vfs_read 0.25 ± 4% -0.0 0.20 ± 5% perf-profile.children.cycles-pp.finish_wait 0.38 -0.0 0.34 ± 2% perf-profile.children.cycles-pp.__x64_sys_read 0.43 ± 3% -0.0 0.39 ± 2% perf-profile.children.cycles-pp.prepare_to_wait 0.14 ± 20% -0.0 0.10 ± 4% perf-profile.children.cycles-pp.task_tick_fair 0.16 ± 19% -0.0 0.12 ± 3% perf-profile.children.cycles-pp.scheduler_tick 0.10 ± 8% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.mutex_spin_on_owner 0.18 ± 8% -0.0 0.14 ± 3% perf-profile.children.cycles-pp.inode_has_perm 0.21 ± 2% -0.0 0.18 ± 6% perf-profile.children.cycles-pp.deactivate_task 0.20 ± 2% -0.0 0.17 ± 4% perf-profile.children.cycles-pp.generic_pipe_buf_confirm 0.13 ± 3% -0.0 0.11 ± 4% perf-profile.children.cycles-pp.__sb_end_write 0.14 ± 9% -0.0 0.12 perf-profile.children.cycles-pp.timespec64_trunc 0.14 ± 3% -0.0 0.11 ± 3% perf-profile.children.cycles-pp.native_irq_return_iret 0.11 ± 4% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.smp_reschedule_interrupt 0.17 ± 4% -0.0 0.15 perf-profile.children.cycles-pp.iov_iter_init 0.11 ± 4% -0.0 0.09 ± 10% perf-profile.children.cycles-pp.native_apic_msr_eoi_write 0.09 ± 4% -0.0 0.07 perf-profile.children.cycles-pp.__x2apic_send_IPI_dest 0.15 -0.0 0.13 ± 3% perf-profile.children.cycles-pp.rb_insert_color 0.18 ± 2% -0.0 0.17 ± 2% perf-profile.children.cycles-pp.set_next_buddy 0.08 ± 5% -0.0 0.07 perf-profile.children.cycles-pp.interrupt_entry 0.23 ± 2% +0.0 0.24 perf-profile.children.cycles-pp.cpumask_next 0.67 +0.0 0.69 perf-profile.children.cycles-pp.__calc_delta 0.15 ± 3% +0.0 0.18 ± 6% perf-profile.children.cycles-pp.wakeup_preempt_entity 0.17 ± 6% +0.1 0.23 ± 3% perf-profile.children.cycles-pp.remove_entity_load_avg 0.14 ± 6% +0.1 0.19 ± 7% perf-profile.children.cycles-pp.attach_entity_load_avg 0.80 ± 3% +0.1 0.86 perf-profile.children.cycles-pp.check_preempt_wakeup 0.37 ± 3% +0.1 0.45 ± 4% perf-profile.children.cycles-pp.account_entity_enqueue 0.79 ± 4% +0.1 0.88 ± 5% perf-profile.children.cycles-pp.finish_task_switch 0.80 ± 2% +0.1 0.89 ± 2% perf-profile.children.cycles-pp.put_prev_entity 0.54 ± 2% +0.1 0.65 perf-profile.children.cycles-pp.account_entity_dequeue 1.01 ± 5% +0.1 1.13 ± 5% perf-profile.children.cycles-pp.set_task_cpu 0.49 ± 7% +0.2 0.65 ± 5% perf-profile.children.cycles-pp.migrate_task_rq_fair 1.23 ± 4% +0.2 1.40 ± 2% perf-profile.children.cycles-pp.__switch_to_asm 1.15 ± 2% +0.2 1.33 ± 2% perf-profile.children.cycles-pp.__switch_to 1.42 ± 3% +0.2 1.59 ± 2% perf-profile.children.cycles-pp.switch_fpu_return 0.76 ± 3% +0.2 0.96 ± 3% perf-profile.children.cycles-pp.pick_next_entity 2.21 +0.2 2.42 perf-profile.children.cycles-pp.reweight_entity 2.20 ± 2% +0.2 2.43 perf-profile.children.cycles-pp.load_new_mm_cr3 1.24 ± 5% +0.2 1.48 ± 4% perf-profile.children.cycles-pp.update_rq_clock 1.43 ± 5% +0.3 1.73 ± 2% perf-profile.children.cycles-pp.___perf_sw_event 0.38 ± 15% +0.3 0.68 ± 9% perf-profile.children.cycles-pp.find_next_bit 3.96 +0.3 4.28 perf-profile.children.cycles-pp.pick_next_task_fair 4.43 ± 2% +0.4 4.82 perf-profile.children.cycles-pp.switch_mm_irqs_off 1.11 +0.7 1.83 ± 4% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.46 ± 26% +0.7 1.20 ± 12% perf-profile.children.cycles-pp.cpumask_next_wrap 87.76 +1.1 88.86 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 86.98 +1.1 88.10 perf-profile.children.cycles-pp.do_syscall_64 21.71 +1.3 23.02 perf-profile.children.cycles-pp.__schedule 5.98 ± 4% +1.4 7.34 ± 3% perf-profile.children.cycles-pp.exit_to_usermode_loop 21.59 +1.4 22.99 perf-profile.children.cycles-pp.schedule 1.73 ± 29% +3.3 5.03 ± 13% perf-profile.children.cycles-pp.available_idle_cpu 35.55 +3.3 38.88 perf-profile.children.cycles-pp.ksys_write 34.31 +3.5 37.84 perf-profile.children.cycles-pp.vfs_write 31.14 +4.1 35.23 ± 2% perf-profile.children.cycles-pp.new_sync_write 30.56 +4.2 34.74 ± 2% perf-profile.children.cycles-pp.pipe_write 22.96 ± 2% +6.0 28.93 ± 3% perf-profile.children.cycles-pp.__wake_up_common_lock 20.11 ± 2% +6.1 26.19 ± 3% perf-profile.children.cycles-pp.try_to_wake_up 20.66 ± 2% +6.2 26.82 ± 3% perf-profile.children.cycles-pp.__wake_up_common 20.01 ± 2% +6.2 26.19 ± 3% perf-profile.children.cycles-pp.autoremove_wake_function 3.46 ± 20% +7.0 10.48 ± 12% perf-profile.children.cycles-pp.select_idle_sibling 4.92 ± 15% +7.1 12.01 ± 10% perf-profile.children.cycles-pp.select_task_rq_fair 11.85 ± 3% -2.1 9.72 ± 2% perf-profile.self.cycles-pp.do_syscall_64 4.13 ± 5% -1.3 2.85 ± 8% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 3.49 ± 6% -1.0 2.52 ± 9% perf-profile.self.cycles-pp.update_cfs_group 4.33 ± 2% -0.7 3.64 ± 2% perf-profile.self.cycles-pp.syscall_return_via_sysret 4.24 ± 2% -0.6 3.60 ± 3% perf-profile.self.cycles-pp.entry_SYSCALL_64 2.12 ± 4% -0.5 1.59 ± 6% perf-profile.self.cycles-pp.mutex_unlock 2.61 -0.3 2.31 ± 3% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 1.60 ± 2% -0.3 1.30 ± 5% perf-profile.self.cycles-pp.update_load_avg 1.48 ± 3% -0.2 1.25 ± 2% perf-profile.self.cycles-pp.mutex_lock 1.95 -0.2 1.75 ± 7% perf-profile.self.cycles-pp.selinux_file_permission 1.06 ± 5% -0.2 0.86 ± 5% perf-profile.self.cycles-pp.pipe_write 1.46 ± 2% -0.2 1.28 ± 3% perf-profile.self.cycles-pp.pipe_read 1.65 ± 2% -0.2 1.49 ± 4% perf-profile.self.cycles-pp.__fget_light 0.91 ± 3% -0.2 0.74 perf-profile.self.cycles-pp.avc_has_perm 0.94 ± 2% -0.2 0.79 ± 2% perf-profile.self.cycles-pp.___might_sleep 1.84 -0.1 1.72 ± 2% perf-profile.self.cycles-pp.update_curr 1.05 -0.1 0.93 perf-profile.self.cycles-pp.enqueue_task_fair 0.63 ± 3% -0.1 0.52 ± 2% perf-profile.self.cycles-pp.__might_sleep 0.60 ± 6% -0.1 0.51 ± 7% perf-profile.self.cycles-pp.__fsnotify_parent 0.29 ± 3% -0.1 0.19 ± 5% perf-profile.self.cycles-pp.__mutex_lock 0.38 ± 4% -0.1 0.29 ± 4% perf-profile.self.cycles-pp.vfs_write 0.51 -0.1 0.43 ± 4% perf-profile.self.cycles-pp.copy_page_to_iter 0.49 -0.1 0.41 ± 2% perf-profile.self.cycles-pp.new_sync_write 0.35 ± 4% -0.1 0.28 ± 2% perf-profile.self.cycles-pp.copy_page_from_iter 0.81 ± 3% -0.1 0.75 perf-profile.self.cycles-pp.fsnotify 0.26 -0.1 0.19 ± 3% perf-profile.self.cycles-pp.__inode_security_revalidate 0.20 ± 4% -0.1 0.13 ± 6% perf-profile.self.cycles-pp.__mutex_unlock_slowpath 0.29 ± 5% -0.1 0.23 ± 3% perf-profile.self.cycles-pp.file_has_perm 0.22 -0.1 0.16 ± 8% perf-profile.self.cycles-pp.wake_q_add 0.57 ± 2% -0.1 0.52 perf-profile.self.cycles-pp.new_sync_read 0.34 ± 2% -0.1 0.29 ± 3% perf-profile.self.cycles-pp.dequeue_entity 0.26 ± 3% -0.1 0.21 ± 3% perf-profile.self.cycles-pp.ksys_read 0.28 ± 2% -0.0 0.23 perf-profile.self.cycles-pp.security_file_permission 0.22 ± 3% -0.0 0.17 ± 4% perf-profile.self.cycles-pp.finish_wait 0.25 -0.0 0.21 ± 4% perf-profile.self.cycles-pp.ksys_write 0.88 -0.0 0.83 perf-profile.self.cycles-pp.__lock_text_start 0.66 -0.0 0.61 perf-profile.self.cycles-pp.vfs_read 1.37 -0.0 1.32 perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.29 ± 4% -0.0 0.25 perf-profile.self.cycles-pp.current_time 0.20 ± 3% -0.0 0.16 ± 4% perf-profile.self.cycles-pp.rcu_all_qs 0.10 ± 8% -0.0 0.06 ± 6% perf-profile.self.cycles-pp.mutex_spin_on_owner 0.42 -0.0 0.39 ± 2% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax 0.34 -0.0 0.31 ± 2% perf-profile.self.cycles-pp.__x64_sys_read 0.09 ± 5% -0.0 0.05 ± 9% perf-profile.self.cycles-pp.wake_up_q 0.15 ± 8% -0.0 0.12 ± 4% perf-profile.self.cycles-pp.inode_has_perm 0.14 ± 5% -0.0 0.11 ± 9% perf-profile.self.cycles-pp.rb_next 0.20 ± 2% -0.0 0.17 ± 4% perf-profile.self.cycles-pp.deactivate_task 0.11 ± 4% -0.0 0.08 ± 8% perf-profile.self.cycles-pp.__fdget_pos 0.24 ± 2% -0.0 0.22 ± 3% perf-profile.self.cycles-pp.prepare_to_wait 0.14 ± 3% -0.0 0.11 ± 3% perf-profile.self.cycles-pp.native_irq_return_iret 0.13 ± 7% -0.0 0.11 ± 4% perf-profile.self.cycles-pp.timespec64_trunc 0.06 -0.0 0.04 ± 57% perf-profile.self.cycles-pp.copyout 0.11 ± 4% -0.0 0.09 ± 10% perf-profile.self.cycles-pp.native_apic_msr_eoi_write 0.17 ± 4% -0.0 0.15 ± 2% perf-profile.self.cycles-pp.generic_pipe_buf_confirm 0.18 ± 2% -0.0 0.16 ± 4% perf-profile.self.cycles-pp.__might_fault 0.17 ± 2% -0.0 0.16 ± 2% perf-profile.self.cycles-pp.set_next_buddy 0.15 ± 2% -0.0 0.13 ± 3% perf-profile.self.cycles-pp.rb_insert_color 0.12 ± 3% -0.0 0.11 ± 4% perf-profile.self.cycles-pp.__sb_end_write 0.10 ± 8% -0.0 0.08 ± 5% perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64 0.06 -0.0 0.05 perf-profile.self.cycles-pp.kill_fasync 0.14 ± 3% +0.0 0.16 ± 5% perf-profile.self.cycles-pp.exit_to_usermode_loop 0.66 +0.0 0.68 perf-profile.self.cycles-pp.__calc_delta 0.16 ± 5% +0.0 0.18 ± 2% perf-profile.self.cycles-pp.native_load_tls 0.14 ± 3% +0.0 0.17 ± 4% perf-profile.self.cycles-pp.wakeup_preempt_entity 0.67 +0.0 0.70 perf-profile.self.cycles-pp.dequeue_task_fair 0.23 ± 3% +0.0 0.27 ± 3% perf-profile.self.cycles-pp._cond_resched 0.17 ± 4% +0.0 0.21 ± 6% perf-profile.self.cycles-pp.activate_task 0.35 ± 3% +0.0 0.40 perf-profile.self.cycles-pp.pick_next_entity 1.20 +0.1 1.25 perf-profile.self.cycles-pp.select_task_rq_fair 0.13 ± 6% +0.1 0.18 ± 6% perf-profile.self.cycles-pp.attach_entity_load_avg 0.87 ± 3% +0.1 0.96 ± 3% perf-profile.self.cycles-pp.pick_next_task_fair 0.27 ± 6% +0.1 0.36 ± 7% perf-profile.self.cycles-pp.migrate_task_rq_fair 0.32 ± 2% +0.1 0.41 ± 4% perf-profile.self.cycles-pp.account_entity_enqueue 0.42 +0.1 0.56 ± 2% perf-profile.self.cycles-pp.account_entity_dequeue 2.24 ± 3% +0.1 2.39 ± 2% perf-profile.self.cycles-pp.switch_mm_irqs_off 1.09 ± 2% +0.2 1.26 ± 2% perf-profile.self.cycles-pp.__switch_to 1.20 ± 4% +0.2 1.37 ± 2% perf-profile.self.cycles-pp.__switch_to_asm 1.40 ± 3% +0.2 1.58 ± 2% perf-profile.self.cycles-pp.switch_fpu_return 0.89 ± 6% +0.2 1.10 ± 5% perf-profile.self.cycles-pp.update_rq_clock 2.19 ± 2% +0.2 2.42 perf-profile.self.cycles-pp.load_new_mm_cr3 1.33 ± 4% +0.3 1.60 ± 3% perf-profile.self.cycles-pp.___perf_sw_event 0.34 ± 14% +0.3 0.66 ± 9% perf-profile.self.cycles-pp.find_next_bit 0.28 ± 23% +0.4 0.72 ± 12% perf-profile.self.cycles-pp.cpumask_next_wrap 0.89 +0.7 1.58 ± 4% perf-profile.self.cycles-pp._raw_spin_lock 1.04 +0.7 1.76 ± 4% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.65 ± 12% +3.0 3.61 ± 13% perf-profile.self.cycles-pp.select_idle_sibling 1.71 ± 29% +3.3 4.97 ± 13% perf-profile.self.cycles-pp.available_idle_cpu *************************************************************************************************** lkp-cfl-e1: 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory ========================================================================================= compiler/cpufreq_governor/ipc/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode: gcc-7/performance/pipe/x86_64-rhel-7.6/threads/100%/debian-x86_64-2018-04-03.cgz/lkp-cfl-e1/hackbench/0xb8 commit: 43e9f7f231 ("sched/fair: Start tracking SCHED_IDLE tasks count in cfs_rq") 3c29e651e1 ("sched/fair: Fall back to sched-idle CPU if idle CPU isn't found") 43e9f7f231e40e45 3c29e651e16dd3b3179cfb2d055 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:2 -50% :4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x 1:2 -50% :4 dmesg.WARNING:stack_recursion %stddev %change %stddev \ | \ 49845 -6.8% 46466 hackbench.throughput 8.776e+08 +13.0% 9.915e+08 hackbench.time.involuntary_context_switches 93623 -6.3% 87707 hackbench.time.minor_page_faults 740.08 -1.4% 729.54 hackbench.time.user_time 1.569e+09 +3.5% 1.623e+09 hackbench.time.voluntary_context_switches 3.072e+08 -6.2% 2.88e+08 hackbench.workload 21.45 -1.7% 21.09 boot-time.boot 1747212 ± 3% -31.0% 1205072 ± 14% cpuidle.POLL.time 1289000 ± 4% -36.6% 817407 ± 13% cpuidle.POLL.usage 4038486 +6.3% 4293331 vmstat.system.cs 394711 -10.2% 354353 ± 2% vmstat.system.in 10059 -15.4% 8506 ± 5% softirqs.CPU0.SCHED 309881 -27.3% 225159 ± 4% softirqs.CPU2.TIMER 220105 +20.8% 265807 ± 16% softirqs.CPU6.TIMER 120937 ± 3% -17.6% 99667 ± 2% softirqs.SCHED 18883 -8.9% 17200 ± 3% slabinfo.kmalloc-32.active_objs 18883 -8.9% 17200 ± 3% slabinfo.kmalloc-32.num_objs 8573 ± 2% -7.6% 7924 slabinfo.lsm_file_cache.active_objs 8573 ± 2% -7.6% 7924 slabinfo.lsm_file_cache.num_objs 5280 -8.8% 4814 slabinfo.task_delay_info.active_objs 5280 -8.8% 4814 slabinfo.task_delay_info.num_objs 65295 -1.9% 64034 proc-vmstat.nr_active_anon 60605 -1.5% 59671 proc-vmstat.nr_anon_pages 12935 -0.9% 12820 proc-vmstat.nr_slab_unreclaimable 65295 -1.9% 64034 proc-vmstat.nr_zone_active_anon 24774010 -16.6% 20673098 proc-vmstat.numa_hit 24774010 -16.6% 20673098 proc-vmstat.numa_local 24839243 -16.5% 20735391 proc-vmstat.pgalloc_normal 673024 -2.4% 657142 proc-vmstat.pgfault 24822112 -16.5% 20717243 proc-vmstat.pgfree 247.95 ± 14% -19.1% 200.59 ± 4% sched_debug.cfs_rq:/.load_avg.max 43.55 ± 3% -19.9% 34.86 ± 6% sched_debug.cfs_rq:/.load_avg.min 55.51 ± 6% -13.2% 48.19 ± 7% sched_debug.cfs_rq:/.load_avg.stddev 20116 ± 8% -12.8% 17531 ± 5% sched_debug.cfs_rq:/.runnable_weight.min 1288 +7.5% 1384 ± 3% sched_debug.cfs_rq:/.util_avg.max 54763 ± 9% +130.8% 126390 ± 25% sched_debug.cpu.avg_idle.avg 301502 ± 18% +84.9% 557406 ± 14% sched_debug.cpu.avg_idle.max 82233 ± 22% +122.7% 183153 ± 17% sched_debug.cpu.avg_idle.stddev 0.00 ± 12% -12.0% 0.00 ± 8% sched_debug.cpu.next_balance.stddev 16.92 ± 4% -7.5% 15.66 ± 3% sched_debug.cpu.nr_running.avg 2.92 ± 10% -20.9% 2.31 ± 16% sched_debug.cpu.nr_uninterruptible.avg 2977 ± 50% -73.0% 803.75 ± 43% interrupts.133:IR-PCI-MSI.2097154-edge.eth1-TxRx-1 546.00 ± 23% +119.5% 1198 ± 16% interrupts.135:IR-PCI-MSI.2097156-edge.eth1-TxRx-3 13831842 -11.5% 12245495 ± 2% interrupts.CPU0.RES:Rescheduling_interrupts 45259 ± 5% +68.9% 76440 ± 15% interrupts.CPU0.TRM:Thermal_event_interrupts 2977 ± 50% -73.0% 803.75 ± 43% interrupts.CPU1.133:IR-PCI-MSI.2097154-edge.eth1-TxRx-1 45283 ± 5% +68.9% 76474 ± 15% interrupts.CPU1.TRM:Thermal_event_interrupts 13947887 -11.0% 12413074 interrupts.CPU10.RES:Rescheduling_interrupts 45275 ± 5% +68.9% 76468 ± 15% interrupts.CPU10.TRM:Thermal_event_interrupts 45286 ± 5% +68.9% 76468 ± 15% interrupts.CPU11.TRM:Thermal_event_interrupts 13579741 -11.8% 11974820 ± 5% interrupts.CPU12.RES:Rescheduling_interrupts 45286 ± 5% +68.9% 76471 ± 15% interrupts.CPU12.TRM:Thermal_event_interrupts 13668073 -11.3% 12119346 interrupts.CPU13.RES:Rescheduling_interrupts 45286 ± 5% +68.9% 76474 ± 15% interrupts.CPU13.TRM:Thermal_event_interrupts 13578574 -7.8% 12518854 interrupts.CPU14.RES:Rescheduling_interrupts 45286 ± 5% +68.9% 76467 ± 15% interrupts.CPU14.TRM:Thermal_event_interrupts 13689117 -10.1% 12310290 ± 2% interrupts.CPU15.RES:Rescheduling_interrupts 45286 ± 5% +68.9% 76473 ± 15% interrupts.CPU15.TRM:Thermal_event_interrupts 13924470 -10.5% 12461613 interrupts.CPU2.RES:Rescheduling_interrupts 45264 ± 5% +68.9% 76467 ± 15% interrupts.CPU2.TRM:Thermal_event_interrupts 546.00 ± 23% +119.5% 1198 ± 16% interrupts.CPU3.135:IR-PCI-MSI.2097156-edge.eth1-TxRx-3 13951884 -10.5% 12480739 interrupts.CPU3.RES:Rescheduling_interrupts 45286 ± 5% +68.9% 76469 ± 15% interrupts.CPU3.TRM:Thermal_event_interrupts 45285 ± 5% +68.9% 76471 ± 15% interrupts.CPU4.TRM:Thermal_event_interrupts 14030690 -10.7% 12530992 ± 3% interrupts.CPU5.RES:Rescheduling_interrupts 45286 ± 5% +68.9% 76471 ± 15% interrupts.CPU5.TRM:Thermal_event_interrupts 45285 ± 5% +68.9% 76474 ± 15% interrupts.CPU6.TRM:Thermal_event_interrupts 45285 ± 5% +68.9% 76472 ± 15% interrupts.CPU7.TRM:Thermal_event_interrupts 13319036 -10.2% 11966842 ± 2% interrupts.CPU8.RES:Rescheduling_interrupts 45280 ± 5% +68.7% 76405 ± 15% interrupts.CPU8.TRM:Thermal_event_interrupts 13401328 -11.8% 11816711 ± 3% interrupts.CPU9.RES:Rescheduling_interrupts 45279 ± 5% +68.9% 76467 ± 15% interrupts.CPU9.TRM:Thermal_event_interrupts 2.19e+08 -9.8% 1.974e+08 ± 2% interrupts.RES:Rescheduling_interrupts 724498 ± 5% +68.9% 1223435 ± 15% interrupts.TRM:Thermal_event_interrupts 53.44 +4.2% 55.68 perf-stat.i.MPKI 0.12 -0.0 0.10 ± 4% perf-stat.i.cache-miss-rate% 1764490 -13.1% 1533362 ± 3% perf-stat.i.cache-misses 1.767e+09 +3.6% 1.832e+09 perf-stat.i.cache-references 4055785 +6.2% 4306694 perf-stat.i.context-switches 336154 +21.5% 408363 perf-stat.i.cpu-migrations 34528 ± 5% +10.3% 38089 ± 3% perf-stat.i.cycles-between-cache-misses 0.01 ± 6% -0.0 0.01 ± 6% perf-stat.i.dTLB-load-miss-rate% 899577 ± 6% -15.2% 762553 ± 5% perf-stat.i.dTLB-load-misses 0.00 ± 7% -0.0 0.00 ± 4% perf-stat.i.dTLB-store-miss-rate% 22816 ± 7% -13.2% 19813 ± 4% perf-stat.i.dTLB-store-misses 6.044e+09 -1.2% 5.969e+09 perf-stat.i.dTLB-stores 31331199 -3.4% 30272777 perf-stat.i.iTLB-load-misses 1061 +3.1% 1094 perf-stat.i.instructions-per-iTLB-miss 1077 -2.9% 1046 perf-stat.i.minor-faults 73018 -10.9% 65028 ± 2% perf-stat.i.node-loads 122632 ± 2% -9.3% 111181 ± 4% perf-stat.i.node-stores 1077 -2.9% 1046 perf-stat.i.page-faults 53.29 +4.2% 55.53 perf-stat.overall.MPKI 0.10 -0.0 0.08 ± 3% perf-stat.overall.cache-miss-rate% 25264 +15.3% 29120 ± 3% perf-stat.overall.cycles-between-cache-misses 0.01 ± 6% -0.0 0.01 ± 6% perf-stat.overall.dTLB-load-miss-rate% 0.00 ± 7% -0.0 0.00 ± 4% perf-stat.overall.dTLB-store-miss-rate% 1058 +2.9% 1089 perf-stat.overall.instructions-per-iTLB-miss 65158 +6.6% 69487 perf-stat.overall.path-length 1761814 -13.1% 1530883 ± 3% perf-stat.ps.cache-misses 1.765e+09 +3.7% 1.829e+09 perf-stat.ps.cache-references 4048770 +6.2% 4299578 perf-stat.ps.context-switches 335565 +21.5% 407697 perf-stat.ps.cpu-migrations 898100 ± 6% -15.2% 761293 ± 5% perf-stat.ps.dTLB-load-misses 22777 ± 7% -13.2% 19781 ± 4% perf-stat.ps.dTLB-store-misses 6.034e+09 -1.2% 5.96e+09 perf-stat.ps.dTLB-stores 31277565 -3.4% 30222833 perf-stat.ps.iTLB-load-misses 1075 -2.9% 1044 perf-stat.ps.minor-faults 72896 -10.9% 64922 ± 2% perf-stat.ps.node-loads 122424 ± 2% -9.3% 111005 ± 4% perf-stat.ps.node-stores 1075 -2.9% 1044 perf-stat.ps.page-faults 39.37 ±100% -36.4 2.98 ±173% perf-profile.calltrace.cycles-pp.start_thread 21.68 ±100% -20.0 1.64 ±173% perf-profile.calltrace.cycles-pp.__GI___libc_write.start_thread 20.14 ±100% -18.6 1.52 ±173% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_write.start_thread 20.00 ±100% -18.5 1.51 ±173% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write.start_thread 18.05 ±100% -16.7 1.39 ±173% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write.start_thread 16.75 ±100% -15.5 1.22 ±173% perf-profile.calltrace.cycles-pp.__GI___libc_read.start_thread 16.70 ±100% -15.4 1.26 ±173% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write 14.98 ±100% -13.9 1.09 ±173% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_read.start_thread 14.82 ±100% -13.7 1.08 ±173% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read.start_thread 13.95 ±100% -12.9 1.03 ±173% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read.start_thread 12.97 ±100% -12.0 0.95 ±173% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read 3.69 ± 6% -1.1 2.58 ± 7% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common 4.25 ± 6% -1.0 3.29 ± 6% perf-profile.calltrace.cycles-pp._raw_spin_lock.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 1.54 ± 10% -0.5 1.05 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write 1.07 ± 16% -0.5 0.60 ± 5% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__wake_up_common_lock.pipe_write.new_sync_write 1.14 ± 2% -0.3 0.80 ± 7% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__schedule.schedule.pipe_wait 0.55 ± 4% -0.3 0.26 ±100% perf-profile.calltrace.cycles-pp.mutex_unlock.pipe_read.new_sync_read.vfs_read.ksys_read 1.90 ± 9% -0.2 1.67 ± 7% perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.71 ± 7% -0.2 1.48 ± 7% perf-profile.calltrace.cycles-pp.__fget.__fget_light.__fdget_pos.ksys_write.do_syscall_64 0.68 ± 19% +0.2 0.86 perf-profile.calltrace.cycles-pp.update_cfs_group.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 0.67 ± 16% +0.2 0.88 ± 4% perf-profile.calltrace.cycles-pp.update_cfs_group.dequeue_task_fair.__schedule.schedule.pipe_wait 0.29 ±100% +0.4 0.64 ± 4% perf-profile.calltrace.cycles-pp.___perf_sw_event.__schedule.schedule.pipe_wait.pipe_read 0.31 ±100% +0.4 0.67 ± 4% perf-profile.calltrace.cycles-pp.mutex_lock.pipe_wait.pipe_read.new_sync_read.vfs_read 0.32 ±100% +0.4 0.68 ± 2% perf-profile.calltrace.cycles-pp.pick_next_entity.pick_next_task_fair.__schedule.schedule.pipe_wait 0.29 ±100% +0.4 0.68 ± 3% perf-profile.calltrace.cycles-pp._raw_spin_lock.__schedule.schedule.exit_to_usermode_loop.do_syscall_64 0.35 ±100% +0.4 0.78 perf-profile.calltrace.cycles-pp.update_curr.reweight_entity.dequeue_task_fair.__schedule.schedule 1.23 ± 19% +0.4 1.68 ± 2% perf-profile.calltrace.cycles-pp.reweight_entity.dequeue_task_fair.__schedule.schedule.pipe_wait 1.29 ± 18% +0.5 1.76 ± 5% perf-profile.calltrace.cycles-pp.reweight_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up 0.00 +0.5 0.53 ± 2% perf-profile.calltrace.cycles-pp.update_load_avg.set_next_entity.pick_next_task_fair.__schedule.schedule 0.00 +0.6 0.59 ± 7% perf-profile.calltrace.cycles-pp.__enqueue_entity.put_prev_entity.pick_next_task_fair.__schedule.schedule 8.82 +0.6 9.46 ± 2% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function 0.47 ±100% +0.7 1.14 ± 6% perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop 2.21 ± 10% +0.7 2.88 ± 4% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop.do_syscall_64 9.10 +0.7 9.79 perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 9.04 +0.7 9.74 ± 2% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common 0.71 ±100% +0.9 1.61 ± 3% perf-profile.calltrace.cycles-pp.__switch_to 0.68 ± 99% +0.9 1.60 ± 3% perf-profile.calltrace.cycles-pp.native_write_msr 0.86 ±100% +0.9 1.79 ± 2% perf-profile.calltrace.cycles-pp.__switch_to_asm 6.54 +1.0 7.53 ± 2% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_wait.pipe_read 26.96 +1.0 27.95 perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64 1.42 ± 63% +1.0 2.47 ± 3% perf-profile.calltrace.cycles-pp.switch_fpu_return.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.54 ± 8% +1.2 5.79 ± 4% perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 14.99 +1.3 16.27 perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_wait.pipe_read.new_sync_read 15.35 +1.3 16.66 perf-profile.calltrace.cycles-pp.schedule.pipe_wait.pipe_read.new_sync_read.vfs_read 27.37 +1.3 28.70 ± 2% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write 17.81 +1.6 19.38 perf-profile.calltrace.cycles-pp.pipe_wait.pipe_read.new_sync_read.vfs_read.ksys_read 3.95 ± 11% +1.8 5.70 ± 6% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common 23.38 +2.0 25.33 ± 2% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write 24.30 +2.0 26.27 ± 2% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write 23.73 +2.0 25.71 ± 2% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write 5.24 ± 10% +2.0 7.25 ± 6% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 3.23 ± 57% +2.7 5.98 ± 4% perf-profile.calltrace.cycles-pp.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.34 ± 57% +2.8 6.18 ± 4% perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 39.37 ±100% -36.4 2.98 ±173% perf-profile.children.cycles-pp.start_thread 21.68 ±100% -20.0 1.67 ±173% perf-profile.children.cycles-pp.__GI___libc_write 16.99 ±100% -15.7 1.24 ±173% perf-profile.children.cycles-pp.__GI___libc_read 6.84 ± 10% -2.1 4.71 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 45.35 -1.0 44.33 perf-profile.children.cycles-pp.ksys_write 6.70 ± 5% -1.0 5.70 ± 2% perf-profile.children.cycles-pp._raw_spin_lock 5.90 ± 4% -0.6 5.34 ± 6% perf-profile.children.cycles-pp.security_file_permission 3.35 ± 9% -0.6 2.80 ± 3% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 42.14 -0.5 41.63 perf-profile.children.cycles-pp.vfs_write 3.41 -0.4 3.01 perf-profile.children.cycles-pp.__fdget_pos 3.30 -0.4 2.91 perf-profile.children.cycles-pp.__fget_light 2.71 ± 2% -0.4 2.32 perf-profile.children.cycles-pp.__fget 3.23 -0.3 2.93 ± 3% perf-profile.children.cycles-pp.mutex_lock 2.76 -0.3 2.47 ± 5% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 2.21 ± 3% -0.3 1.93 ± 10% perf-profile.children.cycles-pp.file_has_perm 3.14 -0.2 2.90 ± 4% perf-profile.children.cycles-pp.copy_page_to_iter 3.03 ± 5% -0.2 2.79 ± 5% perf-profile.children.cycles-pp.selinux_file_permission 2.10 ± 5% -0.2 1.86 ± 2% perf-profile.children.cycles-pp.mutex_unlock 1.50 ± 5% -0.2 1.27 ± 2% perf-profile.children.cycles-pp.fput_many 0.68 ± 17% -0.2 0.48 ± 4% perf-profile.children.cycles-pp.__mutex_lock 1.66 ± 2% -0.2 1.47 ± 4% perf-profile.children.cycles-pp.copyout 1.73 -0.2 1.57 ± 2% perf-profile.children.cycles-pp.update_rq_clock 1.30 -0.1 1.17 ± 6% perf-profile.children.cycles-pp.copyin 0.27 ± 11% -0.1 0.15 ± 9% perf-profile.children.cycles-pp.preempt_schedule_common 1.25 ± 3% -0.1 1.15 ± 3% perf-profile.children.cycles-pp.fsnotify 0.79 -0.1 0.69 ± 7% perf-profile.children.cycles-pp.reschedule_interrupt 0.91 -0.1 0.82 ± 5% perf-profile.children.cycles-pp._cond_resched 0.80 ± 2% -0.1 0.71 ± 3% perf-profile.children.cycles-pp.file_update_time 0.29 ± 10% -0.1 0.24 ± 5% perf-profile.children.cycles-pp.rb_insert_color 0.26 -0.0 0.21 ± 2% perf-profile.children.cycles-pp.native_irq_return_iret 0.07 ± 14% -0.0 0.03 ±100% perf-profile.children.cycles-pp.scheduler_ipi 0.22 ± 6% -0.0 0.18 ± 4% perf-profile.children.cycles-pp.__x64_sys_read 0.25 ± 4% -0.0 0.21 ± 2% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.27 -0.0 0.23 ± 9% perf-profile.children.cycles-pp.timespec64_trunc 0.23 ± 2% -0.0 0.20 ± 5% perf-profile.children.cycles-pp.smp_reschedule_interrupt 0.22 -0.0 0.20 ± 5% perf-profile.children.cycles-pp.__x64_sys_write 0.24 -0.0 0.22 ± 3% perf-profile.children.cycles-pp.__list_add_valid 0.14 -0.0 0.12 ± 10% perf-profile.children.cycles-pp.kill_fasync 0.11 ± 4% +0.0 0.12 ± 3% perf-profile.children.cycles-pp.perf_exclude_event 0.10 +0.0 0.12 ± 8% perf-profile.children.cycles-pp.is_cpu_allowed 0.09 ± 11% +0.0 0.12 ± 7% perf-profile.children.cycles-pp.rcu_note_context_switch 0.07 +0.0 0.10 ± 10% perf-profile.children.cycles-pp.pkg_thermal_notify 0.17 ± 3% +0.0 0.20 ± 9% perf-profile.children.cycles-pp.deactivate_task 0.16 +0.0 0.21 ± 10% perf-profile.children.cycles-pp.update_cfs_rq_h_load 0.08 ± 6% +0.0 0.12 ± 10% perf-profile.children.cycles-pp.intel_thermal_interrupt 0.08 ± 6% +0.0 0.12 ± 6% perf-profile.children.cycles-pp.smp_thermal_interrupt 0.08 +0.1 0.13 ± 9% perf-profile.children.cycles-pp.thermal_interrupt 0.00 +0.1 0.06 ± 7% perf-profile.children.cycles-pp.default_wake_function 0.19 ± 5% +0.1 0.25 ± 5% perf-profile.children.cycles-pp.wakeup_preempt_entity 0.19 ± 5% +0.1 0.27 ± 4% perf-profile.children.cycles-pp.finish_wait 0.89 ± 8% +0.1 0.98 perf-profile.children.cycles-pp.__calc_delta 0.63 ± 4% +0.1 0.74 ± 4% perf-profile.children.cycles-pp.native_sched_clock 0.33 ± 4% +0.1 0.44 ± 5% perf-profile.children.cycles-pp.set_next_buddy 1.23 ± 3% +0.1 1.34 ± 3% perf-profile.children.cycles-pp.check_preempt_wakeup 0.69 ± 4% +0.1 0.81 ± 3% perf-profile.children.cycles-pp.sched_clock 1.52 ± 3% +0.1 1.65 ± 2% perf-profile.children.cycles-pp.set_next_entity 0.73 ± 4% +0.1 0.88 ± 3% perf-profile.children.cycles-pp.sched_clock_cpu 0.57 ± 4% +0.2 0.74 ± 4% perf-profile.children.cycles-pp.account_entity_dequeue 1.21 ± 4% +0.2 1.39 ± 2% perf-profile.children.cycles-pp.__update_load_avg_se 1.99 ± 3% +0.2 2.20 ± 3% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 1.64 ± 6% +0.2 1.85 perf-profile.children.cycles-pp.__switch_to_asm 1.60 ± 4% +0.2 1.85 ± 3% perf-profile.children.cycles-pp.update_cfs_group 3.91 +0.2 4.15 perf-profile.children.cycles-pp.update_curr 1.04 ± 5% +0.3 1.30 ± 4% perf-profile.children.cycles-pp.___perf_sw_event 1.05 ± 9% +0.3 1.33 ± 2% perf-profile.children.cycles-pp.pick_next_entity 1.57 ± 3% +0.3 1.89 perf-profile.children.cycles-pp.native_write_msr 1.01 ± 9% +0.3 1.33 ± 5% perf-profile.children.cycles-pp.put_prev_entity 2.17 ± 7% +0.3 2.51 ± 2% perf-profile.children.cycles-pp.switch_fpu_return 1.93 ± 7% +0.4 2.36 perf-profile.children.cycles-pp.__switch_to 2.99 ± 4% +0.6 3.57 ± 2% perf-profile.children.cycles-pp.reweight_entity 8.89 +0.6 9.50 ± 2% perf-profile.children.cycles-pp.enqueue_task_fair 9.16 +0.7 9.82 perf-profile.children.cycles-pp.ttwu_do_activate 9.09 +0.7 9.76 ± 2% perf-profile.children.cycles-pp.activate_task 5.18 ± 6% +0.9 6.04 ± 2% perf-profile.children.cycles-pp.pick_next_task_fair 6.71 +0.9 7.63 ± 2% perf-profile.children.cycles-pp.dequeue_task_fair 28.01 +1.4 29.36 ± 2% perf-profile.children.cycles-pp.__wake_up_common_lock 4.91 ± 7% +1.4 6.28 ± 4% perf-profile.children.cycles-pp.exit_to_usermode_loop 18.19 +1.4 19.56 perf-profile.children.cycles-pp.pipe_wait 4.03 ± 11% +1.7 5.75 ± 6% perf-profile.children.cycles-pp.select_idle_sibling 23.86 ± 2% +1.8 25.71 ± 2% perf-profile.children.cycles-pp.try_to_wake_up 24.44 +1.9 26.36 ± 2% perf-profile.children.cycles-pp.__wake_up_common 23.84 +1.9 25.77 ± 2% perf-profile.children.cycles-pp.autoremove_wake_function 5.29 ± 10% +2.0 7.29 ± 6% perf-profile.children.cycles-pp.select_task_rq_fair 20.27 ± 2% +2.3 22.53 ± 2% perf-profile.children.cycles-pp.__schedule 20.36 ± 2% +2.5 22.82 ± 2% perf-profile.children.cycles-pp.schedule 6.82 ± 10% -2.1 4.69 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 2.67 ± 2% -0.4 2.29 perf-profile.self.cycles-pp.__fget 2.72 -0.3 2.44 ± 5% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 1.22 -0.2 0.98 ± 2% perf-profile.self.cycles-pp.update_rq_clock 1.36 ± 3% -0.2 1.13 ± 6% perf-profile.self.cycles-pp.pipe_write 1.48 ± 5% -0.2 1.24 ± 3% perf-profile.self.cycles-pp.fput_many 2.04 ± 5% -0.2 1.81 ± 2% perf-profile.self.cycles-pp.mutex_unlock 1.96 -0.2 1.73 ± 2% perf-profile.self.cycles-pp.mutex_lock 0.66 ± 12% -0.1 0.53 ± 10% perf-profile.self.cycles-pp.file_has_perm 0.38 ± 10% -0.1 0.28 ± 9% perf-profile.self.cycles-pp.ksys_read 0.34 ± 7% -0.1 0.25 ± 7% perf-profile.self.cycles-pp.__mutex_lock 1.21 ± 3% -0.1 1.11 ± 2% perf-profile.self.cycles-pp.fsnotify 1.00 -0.1 0.94 ± 5% perf-profile.self.cycles-pp.__might_sleep 0.60 ± 4% -0.1 0.55 ± 8% perf-profile.self.cycles-pp.vfs_write 0.29 ± 10% -0.1 0.23 ± 3% perf-profile.self.cycles-pp.rb_insert_color 0.32 ± 6% -0.0 0.28 ± 6% perf-profile.self.cycles-pp.ksys_write 0.26 -0.0 0.21 ± 2% perf-profile.self.cycles-pp.native_irq_return_iret 0.07 ± 14% -0.0 0.03 ±100% perf-profile.self.cycles-pp.scheduler_ipi 0.36 ± 5% -0.0 0.32 ± 8% perf-profile.self.cycles-pp.__sb_start_write 0.29 -0.0 0.24 ± 4% perf-profile.self.cycles-pp.check_preempt_curr 0.22 ± 6% -0.0 0.17 ± 6% perf-profile.self.cycles-pp.__x64_sys_read 0.25 -0.0 0.21 ± 8% perf-profile.self.cycles-pp.timespec64_trunc 0.22 ± 4% -0.0 0.19 ± 11% perf-profile.self.cycles-pp.wake_q_add 0.21 ± 2% -0.0 0.18 ± 6% perf-profile.self.cycles-pp.__x64_sys_write 0.21 ± 2% -0.0 0.20 perf-profile.self.cycles-pp.__list_add_valid 0.11 ± 4% -0.0 0.10 ± 7% perf-profile.self.cycles-pp.kill_fasync 0.06 ± 9% +0.0 0.07 ± 5% perf-profile.self.cycles-pp.rcu_note_context_switch 0.15 +0.0 0.18 ± 6% perf-profile.self.cycles-pp.autoremove_wake_function 0.16 +0.0 0.19 ± 9% perf-profile.self.cycles-pp.deactivate_task 0.26 +0.0 0.29 ± 5% perf-profile.self.cycles-pp.rcu_all_qs 0.33 ± 3% +0.0 0.37 ± 4% perf-profile.self.cycles-pp.set_next_entity 0.18 ± 5% +0.0 0.22 ± 6% perf-profile.self.cycles-pp.wakeup_preempt_entity 0.03 ±100% +0.0 0.07 perf-profile.self.cycles-pp.check_cfs_rq_runtime 0.16 ± 6% +0.0 0.20 ± 6% perf-profile.self.cycles-pp.exit_to_usermode_loop 0.16 +0.0 0.20 ± 12% perf-profile.self.cycles-pp.update_cfs_rq_h_load 0.20 ± 2% +0.1 0.25 ± 2% perf-profile.self.cycles-pp.activate_task 0.41 ± 6% +0.1 0.46 perf-profile.self.cycles-pp.schedule 1.96 +0.1 2.02 perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.15 ± 6% +0.1 0.22 perf-profile.self.cycles-pp.finish_wait 0.41 ± 4% +0.1 0.48 ± 6% perf-profile.self.cycles-pp.account_entity_enqueue 0.61 ± 4% +0.1 0.71 ± 3% perf-profile.self.cycles-pp.native_sched_clock 0.83 ± 2% +0.1 0.93 perf-profile.self.cycles-pp.dequeue_task_fair 0.29 ± 3% +0.1 0.40 ± 5% perf-profile.self.cycles-pp.set_next_buddy 1.02 +0.1 1.15 ± 7% perf-profile.self.cycles-pp.enqueue_task_fair 0.46 ± 12% +0.2 0.61 ± 3% perf-profile.self.cycles-pp.pick_next_entity 1.17 ± 4% +0.2 1.35 ± 2% perf-profile.self.cycles-pp.__update_load_avg_se 0.42 ± 8% +0.2 0.61 ± 5% perf-profile.self.cycles-pp.account_entity_dequeue 1.64 ± 6% +0.2 1.85 perf-profile.self.cycles-pp.__switch_to_asm 1.23 ± 4% +0.2 1.46 ± 3% perf-profile.self.cycles-pp.reweight_entity 1.07 ± 5% +0.2 1.31 ± 4% perf-profile.self.cycles-pp.select_task_rq_fair 0.90 ± 5% +0.2 1.14 ± 5% perf-profile.self.cycles-pp.___perf_sw_event 1.57 ± 4% +0.2 1.82 ± 2% perf-profile.self.cycles-pp.update_cfs_group 1.17 ± 3% +0.3 1.42 ± 7% perf-profile.self.cycles-pp.update_load_avg 1.56 ± 4% +0.3 1.88 perf-profile.self.cycles-pp.native_write_msr 2.16 ± 7% +0.3 2.50 ± 2% perf-profile.self.cycles-pp.switch_fpu_return 1.84 ± 8% +0.4 2.25 ± 2% perf-profile.self.cycles-pp.__switch_to 1.22 +0.5 1.73 ± 4% perf-profile.self.cycles-pp._raw_spin_lock 0.92 ± 11% +1.3 2.27 ± 6% perf-profile.self.cycles-pp.select_idle_sibling Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

1 month, 4 weeks

4
6
0 / 0

[btrfs] c75e839414: aim7.jobs-per-min -9.1% regression

by kernel test robot

Greeting, FYI, we noticed a -9.1% regression of aim7.jobs-per-min due to commit: commit: c75e839414d3610e6487ae3145199c500d55f7f7 ("btrfs: kill the subvol_srcu") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: aim7 on test machine: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory with following parameters: disk: 4BRD_12G md: RAID0 fs: btrfs test: disk_wrt load: 1500 cpufreq_governor: performance ucode: 0x500002c test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system. test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/ If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/4BRD_12G/btrfs/x86_64-rhel-7.6/1500/RAID0/debian-x86_64-20191114.cgz/lkp-csl-2ap2/disk_wrt/aim7/0x500002c commit: efc3453494 ("btrfs: make btrfs_cleanup_fs_roots use the radix tree lock") c75e839414 ("btrfs: kill the subvol_srcu") efc3453494af7818 c75e839414d3610e6487ae31451 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 3:9 -33% :8 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x %stddev %change %stddev \ | \ 29509 ± 2% -9.1% 26837 ± 2% aim7.jobs-per-min 305.28 ± 2% +10.0% 335.72 ± 2% aim7.time.elapsed_time 305.28 ± 2% +10.0% 335.72 ± 2% aim7.time.elapsed_time.max 4883135 ± 10% +37.9% 6735464 ± 7% aim7.time.involuntary_context_switches 56288 ± 2% +10.5% 62202 ± 2% aim7.time.system_time 2344783 +6.5% 2497364 ± 2% aim7.time.voluntary_context_switches 62337721 ± 2% +9.8% 68456490 ± 2% turbostat.IRQ 431.56 ± 6% +22.3% 527.88 ± 4% vmstat.procs.r 27340 ± 2% +11.2% 30397 ± 2% vmstat.system.cs 226804 ± 6% +21.7% 276057 ± 4% meminfo.Active(file) 221309 ± 6% +22.3% 270668 ± 4% meminfo.Dirty 720.89 ±111% +49.3% 1076 ± 73% meminfo.Mlocked 14278 ± 2% -8.3% 13094 ± 2% meminfo.max_used_kB 57228 ± 6% +22.7% 70195 ± 5% numa-meminfo.node0.Active(file) 55433 ± 6% +21.6% 67431 ± 4% numa-meminfo.node0.Dirty 56152 ± 6% +21.4% 68180 ± 5% numa-meminfo.node1.Active(file) 55001 ± 6% +22.5% 67397 ± 4% numa-meminfo.node1.Dirty 56373 ± 6% +21.7% 68594 ± 4% numa-meminfo.node2.Active(file) 55222 ± 7% +22.6% 67726 ± 4% numa-meminfo.node2.Dirty 56671 ± 6% +20.5% 68317 ± 3% numa-meminfo.node3.Active(file) 55285 ± 6% +21.8% 67355 ± 4% numa-meminfo.node3.Dirty 56694 ± 6% +21.7% 69019 ± 4% proc-vmstat.nr_active_file 55342 ± 6% +22.3% 67662 ± 4% proc-vmstat.nr_dirty 402316 +2.1% 410951 proc-vmstat.nr_file_pages 180.22 ±111% +49.4% 269.25 ± 73% proc-vmstat.nr_mlock 56694 ± 6% +21.7% 69019 ± 4% proc-vmstat.nr_zone_active_file 54680 ± 6% +22.8% 67168 ± 4% proc-vmstat.nr_zone_write_pending 3144381 ± 2% +6.1% 3335275 proc-vmstat.pgactivate 1387558 ± 2% +7.9% 1496754 ± 2% proc-vmstat.pgfault 983.33 ± 4% +5.4% 1036 proc-vmstat.unevictable_pgs_culled 14331 ± 6% +22.6% 17566 ± 5% numa-vmstat.node0.nr_active_file 13884 ± 6% +21.6% 16884 ± 4% numa-vmstat.node0.nr_dirty 14330 ± 6% +22.6% 17566 ± 5% numa-vmstat.node0.nr_zone_active_file 13714 ± 6% +22.2% 16755 ± 4% numa-vmstat.node0.nr_zone_write_pending 14047 ± 6% +21.3% 17043 ± 4% numa-vmstat.node1.nr_active_file 13763 ± 6% +22.3% 16838 ± 4% numa-vmstat.node1.nr_dirty 14047 ± 6% +21.3% 17043 ± 4% numa-vmstat.node1.nr_zone_active_file 13599 ± 6% +23.0% 16726 ± 4% numa-vmstat.node1.nr_zone_write_pending 14074 ± 5% +21.7% 17130 ± 4% numa-vmstat.node2.nr_active_file 13786 ± 6% +22.6% 16906 ± 4% numa-vmstat.node2.nr_dirty 14074 ± 5% +21.7% 17130 ± 4% numa-vmstat.node2.nr_zone_active_file 13617 ± 7% +23.3% 16789 ± 4% numa-vmstat.node2.nr_zone_write_pending 14151 ± 6% +20.7% 17074 ± 3% numa-vmstat.node3.nr_active_file 13790 ± 6% +22.1% 16832 ± 4% numa-vmstat.node3.nr_dirty 14152 ± 6% +20.7% 17074 ± 3% numa-vmstat.node3.nr_zone_active_file 13627 ± 6% +22.5% 16696 ± 4% numa-vmstat.node3.nr_zone_write_pending 828631 ± 16% +49.9% 1241934 ± 8% sched_debug.cfs_rq:/.min_vruntime.stddev 1.33 ± 30% +96.9% 2.61 ± 14% sched_debug.cfs_rq:/.nr_spread_over.avg 831152 ± 16% +49.7% 1244586 ± 8% sched_debug.cfs_rq:/.spread0.stddev 1.92 ± 14% +28.4% 2.47 ± 8% sched_debug.cpu.nr_running.avg 5.00 ± 9% +29.7% 6.48 ± 9% sched_debug.cpu.nr_running.max 0.90 ± 9% +16.0% 1.05 ± 3% sched_debug.cpu.nr_running.stddev 21790 ± 11% +16.7% 25440 ± 2% sched_debug.cpu.nr_switches.avg 19761 ± 11% +17.9% 23289 ± 2% sched_debug.cpu.nr_switches.min 20403 ± 12% +18.1% 24101 ± 2% sched_debug.cpu.sched_count.avg 27825 ± 10% +15.8% 32227 ± 4% sched_debug.cpu.sched_count.max 18995 ± 13% +18.3% 22463 ± 2% sched_debug.cpu.sched_count.min 1143 ± 11% +15.6% 1322 ± 4% sched_debug.cpu.sched_count.stddev 1404 ± 7% -31.0% 969.66 ± 11% sched_debug.cpu.sched_goidle.avg 1186 ± 8% -36.0% 759.25 ± 14% sched_debug.cpu.sched_goidle.min 16821 ± 14% +20.6% 20284 ± 6% sched_debug.cpu.ttwu_count.max 2790 ± 11% +21.7% 3395 ± 7% sched_debug.cpu.ttwu_count.stddev 398.19 ± 16% +28.3% 510.73 ± 10% sched_debug.cpu.ttwu_local.stddev 43238516 ± 5% +19.0% 51459028 ± 4% perf-stat.i.cache-misses 83870573 ± 4% +13.8% 95436555 ± 4% perf-stat.i.cache-references 27181 ± 2% +11.1% 30198 ± 2% perf-stat.i.context-switches 3008 ± 2% -7.1% 2794 perf-stat.i.cpu-migrations 12612 ± 5% -15.6% 10641 ± 3% perf-stat.i.cycles-between-cache-misses 1.1e+09 ± 2% -7.3% 1.019e+09 ± 2% perf-stat.i.dTLB-stores 10684629 ± 2% +10.8% 11842781 ± 2% perf-stat.i.node-load-misses 641429 ± 2% +24.4% 798190 ± 2% perf-stat.i.node-loads 5915955 ± 2% -6.3% 5543368 ± 2% perf-stat.i.node-store-misses 380971 -6.7% 355629 ± 2% perf-stat.i.node-stores 1.66 ± 4% +14.4% 1.90 ± 3% perf-stat.overall.MPKI 13064 ± 5% -15.8% 10999 ± 4% perf-stat.overall.cycles-between-cache-misses 43484558 ± 5% +19.2% 51837476 ± 4% perf-stat.ps.cache-misses 83278267 ± 4% +14.4% 95253129 ± 3% perf-stat.ps.cache-references 27414 ± 2% +11.2% 30486 ± 2% perf-stat.ps.context-switches 3028 ± 2% -7.0% 2817 perf-stat.ps.cpu-migrations 1.104e+09 ± 2% -7.3% 1.023e+09 ± 2% perf-stat.ps.dTLB-stores 4394 -1.8% 4315 perf-stat.ps.minor-faults 10766873 ± 2% +11.0% 11947268 ± 2% perf-stat.ps.node-load-misses 645940 ± 2% +24.7% 805392 ± 2% perf-stat.ps.node-loads 5965618 ± 2% -6.2% 5595562 ± 2% perf-stat.ps.node-store-misses 383092 -6.5% 358124 ± 2% perf-stat.ps.node-stores 4395 -1.8% 4315 perf-stat.ps.page-faults 1.538e+13 ± 2% +10.0% 1.692e+13 ± 2% perf-stat.total.instructions 27.39 -0.1 27.28 perf-profile.calltrace.cycles-pp.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write 35.13 +0.1 35.23 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write 35.23 +0.1 35.34 perf-profile.calltrace.cycles-pp._raw_spin_lock.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter 35.48 +0.1 35.60 perf-profile.calltrace.cycles-pp.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write 35.08 +0.1 35.22 perf-profile.calltrace.cycles-pp._raw_spin_lock.btrfs_block_rsv_release.btrfs_inode_rsv_release.btrfs_buffered_write.btrfs_file_write_iter 34.97 +0.1 35.12 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.btrfs_block_rsv_release.btrfs_inode_rsv_release.btrfs_buffered_write 35.57 +0.1 35.72 perf-profile.calltrace.cycles-pp.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write 35.10 +0.2 35.26 perf-profile.calltrace.cycles-pp.btrfs_block_rsv_release.btrfs_inode_rsv_release.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write 35.11 +0.2 35.27 perf-profile.calltrace.cycles-pp.btrfs_inode_rsv_release.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write 98.71 +0.2 98.88 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 98.69 +0.2 98.87 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 98.74 +0.2 98.91 perf-profile.calltrace.cycles-pp.write 98.71 +0.2 98.89 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write 98.68 +0.2 98.86 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 98.64 +0.2 98.82 perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 98.63 +0.2 98.81 perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64 98.58 +0.2 98.77 perf-profile.calltrace.cycles-pp.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write 27.39 -0.1 27.28 perf-profile.children.cycles-pp.btrfs_dirty_pages 0.56 ± 5% -0.1 0.46 ± 9% perf-profile.children.cycles-pp.btrfs_get_extent 0.51 ± 5% -0.1 0.41 ± 9% perf-profile.children.cycles-pp.btrfs_lookup_file_extent 0.52 ± 5% -0.1 0.43 ± 9% perf-profile.children.cycles-pp.btrfs_search_slot 0.44 ± 6% -0.1 0.35 ± 11% perf-profile.children.cycles-pp.btrfs_read_lock_root_node 0.43 ± 6% -0.1 0.34 ± 11% perf-profile.children.cycles-pp.btrfs_tree_read_lock 0.19 ± 15% -0.1 0.11 ± 27% perf-profile.children.cycles-pp.start_secondary 0.19 ± 16% -0.1 0.10 ± 27% perf-profile.children.cycles-pp.cpuidle_enter 0.19 ± 16% -0.1 0.10 ± 27% perf-profile.children.cycles-pp.cpuidle_enter_state 0.41 ± 7% -0.1 0.33 ± 8% perf-profile.children.cycles-pp.rwsem_optimistic_spin 0.46 ± 7% -0.1 0.38 ± 8% perf-profile.children.cycles-pp.rwsem_down_write_slowpath 0.19 ± 15% -0.1 0.11 ± 25% perf-profile.children.cycles-pp.secondary_startup_64 0.19 ± 15% -0.1 0.11 ± 25% perf-profile.children.cycles-pp.cpu_startup_entry 0.19 ± 15% -0.1 0.11 ± 25% perf-profile.children.cycles-pp.do_idle 0.18 ± 15% -0.1 0.10 ± 24% perf-profile.children.cycles-pp.intel_idle 0.29 ± 9% -0.1 0.22 ± 11% perf-profile.children.cycles-pp.osq_lock 0.19 ± 6% -0.0 0.15 ± 16% perf-profile.children.cycles-pp.finish_wait 0.39 ± 5% -0.0 0.35 ± 6% perf-profile.children.cycles-pp.do_unlinkat 0.39 ± 6% -0.0 0.35 ± 7% perf-profile.children.cycles-pp.unlink 0.26 ± 5% -0.0 0.22 ± 6% perf-profile.children.cycles-pp.do_sys_open 0.26 ± 5% -0.0 0.22 ± 6% perf-profile.children.cycles-pp.do_sys_openat2 0.26 ± 4% -0.0 0.22 ± 6% perf-profile.children.cycles-pp.do_filp_open 0.26 ± 4% -0.0 0.22 ± 6% perf-profile.children.cycles-pp.path_openat 0.26 ± 4% -0.0 0.22 ± 6% perf-profile.children.cycles-pp.creat 0.00 +0.0 0.05 perf-profile.children.cycles-pp.btrfs_calculate_inode_block_rsv_size 99.65 +0.1 99.73 perf-profile.children.cycles-pp.do_syscall_64 99.66 +0.1 99.74 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 35.62 +0.1 35.74 perf-profile.children.cycles-pp.btrfs_reserve_metadata_bytes 35.57 +0.1 35.72 perf-profile.children.cycles-pp.btrfs_delalloc_reserve_metadata 97.55 +0.2 97.72 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 98.74 +0.2 98.92 perf-profile.children.cycles-pp.write 98.70 +0.2 98.88 perf-profile.children.cycles-pp.ksys_write 98.69 +0.2 98.87 perf-profile.children.cycles-pp.vfs_write 98.65 +0.2 98.83 perf-profile.children.cycles-pp.new_sync_write 98.63 +0.2 98.81 perf-profile.children.cycles-pp.btrfs_file_write_iter 98.59 +0.2 98.78 perf-profile.children.cycles-pp.btrfs_buffered_write 97.60 +0.2 97.79 perf-profile.children.cycles-pp._raw_spin_lock 0.18 ± 15% -0.1 0.10 ± 24% perf-profile.self.cycles-pp.intel_idle 0.28 ± 9% -0.1 0.21 ± 12% perf-profile.self.cycles-pp.osq_lock 0.45 ± 2% -0.0 0.41 ± 2% perf-profile.self.cycles-pp._raw_spin_lock 135788 ± 2% +13.1% 153639 ± 3% softirqs.CPU0.RCU 115547 ± 3% +8.0% 124768 ± 4% softirqs.CPU0.TIMER 135243 ± 3% +14.0% 154188 ± 2% softirqs.CPU1.RCU 136056 ± 3% +12.4% 152960 ± 3% softirqs.CPU10.RCU 134253 ± 3% +14.2% 153260 ± 3% softirqs.CPU100.RCU 134668 ± 4% +13.7% 153157 ± 2% softirqs.CPU101.RCU 109995 ± 3% +7.9% 118637 ± 3% softirqs.CPU101.TIMER 134989 ± 4% +13.9% 153700 ± 2% softirqs.CPU102.RCU 109876 ± 3% +8.9% 119608 ± 2% softirqs.CPU102.TIMER 134868 ± 4% +13.8% 153427 ± 3% softirqs.CPU103.RCU 109933 ± 3% +8.2% 118946 ± 3% softirqs.CPU103.TIMER 134454 ± 3% +14.0% 153285 ± 3% softirqs.CPU104.RCU 109731 ± 2% +8.4% 118965 ± 3% softirqs.CPU104.TIMER 134878 ± 3% +13.6% 153283 ± 3% softirqs.CPU105.RCU 134096 ± 3% +14.1% 153022 ± 3% softirqs.CPU106.RCU 109821 ± 2% +8.2% 118845 ± 3% softirqs.CPU106.TIMER 133905 ± 3% +15.0% 154004 ± 3% softirqs.CPU107.RCU 134117 ± 3% +14.3% 153232 ± 3% softirqs.CPU108.RCU 109545 ± 2% +8.8% 119182 ± 3% softirqs.CPU108.TIMER 134574 ± 3% +13.8% 153202 ± 3% softirqs.CPU109.RCU 109804 ± 3% +8.2% 118760 ± 3% softirqs.CPU109.TIMER 135106 ± 3% +14.8% 155124 ± 3% softirqs.CPU11.RCU 111746 ± 4% +7.8% 120460 ± 3% softirqs.CPU11.TIMER 134482 ± 4% +14.3% 153687 ± 3% softirqs.CPU110.RCU 110314 ± 3% +8.1% 119241 ± 3% softirqs.CPU110.TIMER 134464 ± 3% +14.3% 153634 ± 3% softirqs.CPU111.RCU 110626 ± 3% +8.3% 119815 ± 3% softirqs.CPU111.TIMER 134895 ± 3% +14.4% 154257 ± 3% softirqs.CPU112.RCU 110040 ± 3% +14.6% 126058 ± 16% softirqs.CPU112.TIMER 134644 ± 3% +13.6% 152993 ± 3% softirqs.CPU113.RCU 109693 ± 3% +8.1% 118569 ± 3% softirqs.CPU113.TIMER 134313 ± 3% +13.9% 152993 ± 3% softirqs.CPU114.RCU 109711 ± 3% +8.6% 119143 ± 3% softirqs.CPU114.TIMER 134559 ± 3% +13.8% 153115 ± 3% softirqs.CPU115.RCU 110226 ± 3% +8.2% 119221 ± 3% softirqs.CPU115.TIMER 134087 ± 3% +14.2% 153077 ± 3% softirqs.CPU116.RCU 110225 ± 3% +9.0% 120184 ± 3% softirqs.CPU116.TIMER 134271 ± 3% +14.1% 153220 ± 3% softirqs.CPU117.RCU 109179 ± 2% +8.3% 118274 ± 3% softirqs.CPU117.TIMER 134281 ± 3% +14.1% 153230 ± 3% softirqs.CPU118.RCU 134415 ± 3% +14.0% 153215 ± 3% softirqs.CPU119.RCU 109514 ± 2% +8.3% 118643 ± 3% softirqs.CPU119.TIMER 135560 ± 3% +13.5% 153874 ± 3% softirqs.CPU12.RCU 110316 ± 2% +8.5% 119734 ± 3% softirqs.CPU12.TIMER 134177 ± 3% +14.6% 153815 ± 3% softirqs.CPU120.RCU 134367 ± 3% +15.0% 154493 ± 3% softirqs.CPU121.RCU 133972 ± 3% +14.3% 153173 ± 3% softirqs.CPU122.RCU 134801 ± 4% +13.6% 153136 ± 3% softirqs.CPU123.RCU 134709 ± 3% +13.9% 153440 ± 3% softirqs.CPU124.RCU 135185 ± 3% +13.1% 152925 ± 3% softirqs.CPU125.RCU 134977 ± 3% +13.8% 153608 ± 3% softirqs.CPU126.RCU 133862 ± 4% +14.7% 153558 ± 3% softirqs.CPU127.RCU 134554 ± 3% +13.9% 153237 ± 3% softirqs.CPU128.RCU 134254 ± 3% +14.1% 153130 ± 3% softirqs.CPU129.RCU 135133 ± 3% +14.0% 154103 ± 3% softirqs.CPU13.RCU 110389 ± 2% +8.1% 119382 ± 3% softirqs.CPU13.TIMER 134319 ± 3% +13.8% 152909 ± 3% softirqs.CPU130.RCU 133590 ± 3% +14.5% 152925 ± 3% softirqs.CPU131.RCU 134416 ± 4% +14.0% 153223 ± 3% softirqs.CPU132.RCU 133931 ± 3% +14.5% 153359 ± 3% softirqs.CPU133.RCU 134031 ± 3% +14.5% 153527 ± 3% softirqs.CPU134.RCU 133851 ± 4% +14.3% 153052 ± 3% softirqs.CPU135.RCU 134235 ± 4% +13.9% 152959 ± 3% softirqs.CPU136.RCU 134003 ± 3% +14.2% 153037 ± 3% softirqs.CPU137.RCU 133453 ± 3% +14.4% 152625 ± 3% softirqs.CPU138.RCU 133514 ± 3% +14.8% 153289 ± 3% softirqs.CPU139.RCU 134952 ± 3% +13.6% 153372 ± 3% softirqs.CPU14.RCU 134045 ± 3% +13.7% 152361 ± 3% softirqs.CPU140.RCU 133662 ± 3% +14.1% 152528 ± 3% softirqs.CPU141.RCU 133643 ± 3% +14.7% 153283 ± 2% softirqs.CPU142.RCU 133313 ± 3% +15.0% 153254 ± 3% softirqs.CPU143.RCU 133585 ± 3% +14.4% 152837 ± 3% softirqs.CPU144.RCU 108590 ± 3% +12.7% 122339 ± 5% softirqs.CPU144.TIMER 133471 ± 3% +14.5% 152883 ± 3% softirqs.CPU145.RCU 108095 ± 3% +14.5% 123726 ± 6% softirqs.CPU145.TIMER 133588 ± 3% +15.4% 154173 ± 3% softirqs.CPU146.RCU 108268 ± 3% +19.8% 129684 ± 16% softirqs.CPU146.TIMER 134060 ± 3% +14.3% 153270 ± 3% softirqs.CPU147.RCU 108731 ± 3% +13.0% 122902 ± 5% softirqs.CPU147.TIMER 133778 ± 3% +14.1% 152681 ± 3% softirqs.CPU148.RCU 108208 ± 3% +12.6% 121875 ± 5% softirqs.CPU148.TIMER 133514 ± 3% +14.2% 152460 ± 3% softirqs.CPU149.RCU 108054 ± 3% +15.6% 124938 ± 9% softirqs.CPU149.TIMER 134677 ± 3% +14.5% 154138 ± 2% softirqs.CPU15.RCU 133245 ± 3% +14.4% 152373 ± 3% softirqs.CPU150.RCU 107788 ± 3% +13.1% 121957 ± 5% softirqs.CPU150.TIMER 133747 ± 3% +14.0% 152473 ± 3% softirqs.CPU151.RCU 108258 ± 4% +12.4% 121662 ± 5% softirqs.CPU151.TIMER 133819 ± 3% +14.1% 152726 ± 3% softirqs.CPU152.RCU 108163 ± 3% +12.7% 121940 ± 5% softirqs.CPU152.TIMER 134057 ± 3% +14.0% 152844 ± 3% softirqs.CPU153.RCU 107680 ± 3% +13.1% 121787 ± 5% softirqs.CPU153.TIMER 133660 ± 3% +14.2% 152675 ± 3% softirqs.CPU154.RCU 107911 ± 3% +15.4% 124485 ± 9% softirqs.CPU154.TIMER 133349 ± 3% +14.6% 152811 ± 3% softirqs.CPU155.RCU 107513 ± 3% +15.1% 123737 ± 6% softirqs.CPU155.TIMER 133901 ± 3% +14.0% 152615 ± 3% softirqs.CPU156.RCU 107853 ± 3% +13.0% 121904 ± 5% softirqs.CPU156.TIMER 133621 ± 3% +14.2% 152615 ± 3% softirqs.CPU157.RCU 108040 ± 3% +12.6% 121684 ± 5% softirqs.CPU157.TIMER 133891 ± 3% +14.2% 152862 ± 3% softirqs.CPU158.RCU 108534 ± 3% +12.7% 122300 ± 5% softirqs.CPU158.TIMER 133925 ± 3% +14.5% 153347 ± 3% softirqs.CPU159.RCU 108099 ± 3% +12.9% 122015 ± 5% softirqs.CPU159.TIMER 134893 ± 3% +14.5% 154389 ± 3% softirqs.CPU16.RCU 134169 ± 3% +13.9% 152812 ± 3% softirqs.CPU160.RCU 108300 ± 3% +12.9% 122318 ± 5% softirqs.CPU160.TIMER 133818 ± 3% +15.1% 154048 ± 4% softirqs.CPU161.RCU 107919 ± 3% +13.6% 122606 ± 6% softirqs.CPU161.TIMER 133867 ± 3% +14.2% 152856 ± 3% softirqs.CPU162.RCU 107991 ± 3% +13.2% 122209 ± 5% softirqs.CPU162.TIMER 133579 ± 3% +14.3% 152654 ± 3% softirqs.CPU163.RCU 108317 ± 3% +12.9% 122271 ± 5% softirqs.CPU163.TIMER 133914 ± 3% +14.4% 153158 ± 3% softirqs.CPU164.RCU 133684 ± 3% +14.4% 152895 ± 3% softirqs.CPU165.RCU 107435 ± 3% +12.5% 120877 ± 5% softirqs.CPU165.TIMER 133983 ± 3% +13.9% 152582 ± 3% softirqs.CPU166.RCU 107432 ± 3% +13.2% 121641 ± 5% softirqs.CPU166.TIMER 133947 ± 3% +13.6% 152137 ± 3% softirqs.CPU167.RCU 134005 ± 3% +14.5% 153385 ± 4% softirqs.CPU168.RCU 115752 ± 4% +12.0% 129634 ± 4% softirqs.CPU168.TIMER 133557 ± 3% +15.2% 153865 ± 4% softirqs.CPU169.RCU 115884 ± 3% +11.7% 129481 ± 4% softirqs.CPU169.TIMER 135348 ± 3% +13.8% 153977 ± 3% softirqs.CPU17.RCU 110589 ± 3% +7.9% 119298 ± 3% softirqs.CPU17.TIMER 134463 ± 3% +14.0% 153229 ± 3% softirqs.CPU170.RCU 133740 ± 3% +14.1% 152630 ± 3% softirqs.CPU171.RCU 116524 ± 4% +10.9% 129237 ± 3% softirqs.CPU171.TIMER 133716 ± 3% +13.8% 152196 ± 3% softirqs.CPU172.RCU 116393 ± 4% +10.9% 129087 ± 4% softirqs.CPU172.TIMER 133564 ± 3% +13.9% 152165 ± 3% softirqs.CPU173.RCU 116237 ± 3% +11.4% 129460 ± 3% softirqs.CPU173.TIMER 133411 ± 3% +14.5% 152764 ± 3% softirqs.CPU174.RCU 116018 ± 3% +11.2% 129006 ± 4% softirqs.CPU174.TIMER 133147 ± 3% +15.0% 153104 ± 2% softirqs.CPU175.RCU 116052 ± 4% +11.7% 129664 ± 3% softirqs.CPU175.TIMER 133989 ± 3% +13.8% 152504 ± 3% softirqs.CPU176.RCU 116071 ± 4% +10.9% 128756 ± 3% softirqs.CPU176.TIMER 133902 ± 3% +14.1% 152811 ± 3% softirqs.CPU177.RCU 115994 ± 3% +11.1% 128849 ± 4% softirqs.CPU177.TIMER 133679 ± 3% +14.4% 152887 ± 3% softirqs.CPU178.RCU 115942 ± 4% +13.4% 131439 ± 5% softirqs.CPU178.TIMER 133985 ± 3% +14.0% 152762 ± 3% softirqs.CPU179.RCU 116347 ± 4% +11.2% 129422 ± 4% softirqs.CPU179.TIMER 134982 ± 3% +14.0% 153838 ± 3% softirqs.CPU18.RCU 110310 ± 3% +8.6% 119851 ± 3% softirqs.CPU18.TIMER 133236 ± 3% +14.7% 152870 ± 3% softirqs.CPU180.RCU 115901 ± 4% +11.3% 129054 ± 4% softirqs.CPU180.TIMER 133615 ± 3% +14.4% 152876 ± 3% softirqs.CPU181.RCU 115505 ± 4% +11.3% 128560 ± 3% softirqs.CPU181.TIMER 133171 ± 3% +14.7% 152810 ± 3% softirqs.CPU182.RCU 115955 ± 4% +11.0% 128687 ± 3% softirqs.CPU182.TIMER 133413 ± 3% +14.4% 152671 ± 3% softirqs.CPU183.RCU 116313 ± 4% +11.0% 129140 ± 3% softirqs.CPU183.TIMER 133598 ± 3% +14.8% 153388 ± 3% softirqs.CPU184.RCU 115462 ± 3% +13.0% 130430 ± 6% softirqs.CPU184.TIMER 133884 ± 3% +13.9% 152525 ± 3% softirqs.CPU185.RCU 115598 ± 4% +11.1% 128397 ± 4% softirqs.CPU185.TIMER 133262 ± 3% +14.5% 152646 ± 3% softirqs.CPU186.RCU 115745 ± 3% +11.1% 128612 ± 4% softirqs.CPU186.TIMER 133430 ± 3% +14.5% 152767 ± 3% softirqs.CPU187.RCU 115769 ± 4% +11.0% 128464 ± 3% softirqs.CPU187.TIMER 133259 ± 3% +14.8% 153036 ± 3% softirqs.CPU188.RCU 116108 ± 4% +11.3% 129232 ± 3% softirqs.CPU188.TIMER 133609 ± 3% +14.1% 152436 ± 3% softirqs.CPU189.RCU 115984 ± 4% +11.0% 128710 ± 3% softirqs.CPU189.TIMER 135090 ± 3% +13.9% 153802 ± 3% softirqs.CPU19.RCU 110865 ± 2% +8.2% 119936 ± 3% softirqs.CPU19.TIMER 133685 ± 3% +14.2% 152631 ± 3% softirqs.CPU190.RCU 115642 ± 3% +11.2% 128581 ± 4% softirqs.CPU190.TIMER 131031 ± 3% +14.6% 150126 ± 3% softirqs.CPU191.RCU 114275 ± 4% +11.4% 127249 ± 4% softirqs.CPU191.TIMER 134928 ± 3% +14.3% 154165 ± 3% softirqs.CPU2.RCU 135126 ± 3% +14.0% 154083 ± 3% softirqs.CPU20.RCU 111013 ± 3% +8.6% 120549 ± 3% softirqs.CPU20.TIMER 134663 ± 3% +14.5% 154251 ± 3% softirqs.CPU21.RCU 109671 ± 2% +8.2% 118708 ± 3% softirqs.CPU21.TIMER 135047 ± 3% +13.9% 153766 ± 3% softirqs.CPU22.RCU 134437 ± 3% +14.2% 153492 ± 3% softirqs.CPU23.RCU 109675 ± 2% +8.3% 118824 ± 3% softirqs.CPU23.TIMER 136114 ± 3% +13.3% 154277 ± 3% softirqs.CPU24.RCU 134825 ± 3% +14.2% 154007 ± 3% softirqs.CPU25.RCU 139234 ± 4% +13.8% 158515 ± 3% softirqs.CPU26.RCU 135056 ± 3% +14.1% 154062 ± 3% softirqs.CPU27.RCU 135307 ± 3% +13.7% 153897 ± 3% softirqs.CPU28.RCU 134686 ± 3% +14.7% 154497 ± 3% softirqs.CPU29.RCU 134688 ± 3% +14.4% 154115 ± 2% softirqs.CPU3.RCU 134642 ± 3% +14.5% 154226 ± 3% softirqs.CPU30.RCU 134773 ± 3% +14.2% 153887 ± 3% softirqs.CPU31.RCU 134523 ± 3% +14.3% 153733 ± 3% softirqs.CPU32.RCU 134488 ± 3% +14.4% 153801 ± 3% softirqs.CPU33.RCU 134483 ± 3% +14.1% 153380 ± 3% softirqs.CPU34.RCU 134886 ± 3% +13.6% 153180 ± 3% softirqs.CPU35.RCU 135021 ± 3% +13.6% 153340 ± 3% softirqs.CPU36.RCU 134064 ± 3% +15.1% 154277 ± 3% softirqs.CPU37.RCU 134166 ± 3% +14.8% 153967 ± 3% softirqs.CPU38.RCU 134488 ± 3% +14.4% 153871 ± 3% softirqs.CPU39.RCU 135294 ± 2% +14.6% 155025 ± 3% softirqs.CPU4.RCU 134598 ± 4% +14.3% 153848 ± 3% softirqs.CPU40.RCU 134255 ± 3% +14.8% 154099 ± 3% softirqs.CPU41.RCU 134334 ± 3% +14.3% 153491 ± 3% softirqs.CPU42.RCU 134201 ± 3% +14.5% 153654 ± 3% softirqs.CPU43.RCU 134260 ± 3% +14.2% 153359 ± 3% softirqs.CPU44.RCU 134005 ± 3% +14.4% 153367 ± 3% softirqs.CPU45.RCU 134236 ± 3% +14.3% 153419 ± 3% softirqs.CPU46.RCU 134552 ± 3% +13.8% 153056 ± 3% softirqs.CPU47.RCU 134623 ± 3% +14.4% 154028 ± 3% softirqs.CPU48.RCU 134348 ± 3% +14.7% 154080 ± 3% softirqs.CPU49.RCU 109020 ± 3% +18.6% 129279 ± 14% softirqs.CPU49.TIMER 134334 ± 3% +15.0% 154455 ± 3% softirqs.CPU5.RCU 110260 ± 2% +8.8% 119951 ± 4% softirqs.CPU5.TIMER 134377 ± 3% +13.8% 152954 ± 3% softirqs.CPU50.RCU 109169 ± 3% +15.0% 125548 ± 9% softirqs.CPU50.TIMER 134780 ± 3% +14.2% 153967 ± 3% softirqs.CPU51.RCU 109951 ± 3% +12.5% 123641 ± 5% softirqs.CPU51.TIMER 133929 ± 3% +15.0% 154012 ± 3% softirqs.CPU52.RCU 108665 ± 3% +12.8% 122613 ± 5% softirqs.CPU52.TIMER 133761 ± 3% +14.9% 153675 ± 3% softirqs.CPU53.RCU 108606 ± 3% +19.6% 129861 ± 16% softirqs.CPU53.TIMER 134369 ± 3% +14.1% 153255 ± 3% softirqs.CPU54.RCU 108917 ± 3% +12.5% 122585 ± 5% softirqs.CPU54.TIMER 133810 ± 3% +14.6% 153282 ± 3% softirqs.CPU55.RCU 108875 ± 3% +12.5% 122468 ± 5% softirqs.CPU55.TIMER 134038 ± 3% +14.3% 153248 ± 3% softirqs.CPU56.RCU 108569 ± 3% +12.8% 122502 ± 5% softirqs.CPU56.TIMER 133993 ± 4% +14.5% 153426 ± 3% softirqs.CPU57.RCU 108465 ± 3% +12.8% 122399 ± 5% softirqs.CPU57.TIMER 134049 ± 3% +14.7% 153708 ± 3% softirqs.CPU58.RCU 108634 ± 4% +18.8% 129107 ± 16% softirqs.CPU58.TIMER 134025 ± 3% +14.9% 154015 ± 3% softirqs.CPU59.RCU 108501 ± 3% +18.6% 128721 ± 13% softirqs.CPU59.TIMER 135247 ± 3% +14.5% 154804 ± 3% softirqs.CPU6.RCU 133745 ± 4% +14.7% 153452 ± 3% softirqs.CPU60.RCU 108526 ± 3% +13.0% 122688 ± 5% softirqs.CPU60.TIMER 133931 ± 3% +14.6% 153549 ± 3% softirqs.CPU61.RCU 108764 ± 3% +12.9% 122787 ± 5% softirqs.CPU61.TIMER 134682 ± 3% +13.8% 153266 ± 3% softirqs.CPU62.RCU 109188 ± 3% +12.5% 122806 ± 5% softirqs.CPU62.TIMER 134190 ± 3% +14.6% 153761 ± 2% softirqs.CPU63.RCU 108489 ± 3% +12.8% 122426 ± 5% softirqs.CPU63.TIMER 134280 ± 3% +14.3% 153538 ± 3% softirqs.CPU64.RCU 108493 ± 3% +13.3% 122953 ± 5% softirqs.CPU64.TIMER 134074 ± 3% +14.7% 153765 ± 3% softirqs.CPU65.RCU 108331 ± 3% +13.0% 122440 ± 5% softirqs.CPU65.TIMER 134424 ± 4% +14.3% 153692 ± 3% softirqs.CPU66.RCU 108874 ± 3% +12.5% 122490 ± 5% softirqs.CPU66.TIMER 133801 ± 3% +14.9% 153746 ± 3% softirqs.CPU67.RCU 108903 ± 3% +13.2% 123240 ± 5% softirqs.CPU67.TIMER 134085 ± 3% +14.4% 153371 ± 3% softirqs.CPU68.RCU 109310 ± 3% +12.3% 122810 ± 5% softirqs.CPU68.TIMER 134091 ± 3% +14.5% 153515 ± 3% softirqs.CPU69.RCU 108193 ± 3% +12.5% 121716 ± 5% softirqs.CPU69.TIMER 134883 ± 3% +13.6% 153181 ± 3% softirqs.CPU7.RCU 134392 ± 3% +14.8% 154240 ± 3% softirqs.CPU70.RCU 108056 ± 3% +13.4% 122578 ± 5% softirqs.CPU70.TIMER 133901 ± 3% +14.2% 152873 ± 3% softirqs.CPU71.RCU 134273 ± 3% +14.4% 153630 ± 3% softirqs.CPU72.RCU 116189 ± 3% +11.5% 129495 ± 3% softirqs.CPU72.TIMER 134231 ± 3% +14.5% 153682 ± 3% softirqs.CPU73.RCU 116348 ± 3% +11.3% 129453 ± 3% softirqs.CPU73.TIMER 134235 ± 3% +14.1% 153158 ± 3% softirqs.CPU74.RCU 133896 ± 3% +14.6% 153477 ± 3% softirqs.CPU75.RCU 116933 ± 4% +11.2% 129980 ± 3% softirqs.CPU75.TIMER 134014 ± 3% +14.3% 153198 ± 3% softirqs.CPU76.RCU 116889 ± 3% +11.0% 129768 ± 3% softirqs.CPU76.TIMER 134608 ± 3% +13.7% 152994 ± 3% softirqs.CPU77.RCU 117342 ± 3% +11.0% 130274 ± 3% softirqs.CPU77.TIMER 134551 ± 3% +13.9% 153253 ± 3% softirqs.CPU78.RCU 117229 ± 4% +10.5% 129549 ± 4% softirqs.CPU78.TIMER 134118 ± 3% +14.3% 153325 ± 3% softirqs.CPU79.RCU 116982 ± 3% +11.0% 129802 ± 3% softirqs.CPU79.TIMER 135303 ± 3% +13.9% 154129 ± 3% softirqs.CPU8.RCU 134699 ± 3% +13.6% 153047 ± 3% softirqs.CPU80.RCU 117058 ± 4% +10.6% 129470 ± 3% softirqs.CPU80.TIMER 134088 ± 3% +15.6% 155063 ± 3% softirqs.CPU81.RCU 116539 ± 4% +12.3% 130886 ± 5% softirqs.CPU81.TIMER 133989 ± 3% +14.6% 153526 ± 3% softirqs.CPU82.RCU 116614 ± 4% +16.8% 136238 ± 12% softirqs.CPU82.TIMER 133966 ± 3% +14.3% 153142 ± 3% softirqs.CPU83.RCU 117140 ± 3% +11.2% 130205 ± 4% softirqs.CPU83.TIMER 133561 ± 3% +15.0% 153661 ± 3% softirqs.CPU84.RCU 116648 ± 4% +11.3% 129844 ± 4% softirqs.CPU84.TIMER 134217 ± 3% +14.2% 153330 ± 3% softirqs.CPU85.RCU 116247 ± 4% +11.3% 129336 ± 3% softirqs.CPU85.TIMER 133956 ± 3% +14.5% 153338 ± 3% softirqs.CPU86.RCU 116654 ± 4% +11.1% 129560 ± 3% softirqs.CPU86.TIMER 133834 ± 3% +15.1% 154049 ± 2% softirqs.CPU87.RCU 116898 ± 4% +12.0% 130912 ± 3% softirqs.CPU87.TIMER 133951 ± 3% +15.1% 154211 ± 3% softirqs.CPU88.RCU 116139 ± 3% +16.0% 134676 ± 13% softirqs.CPU88.TIMER 134288 ± 3% +14.3% 153433 ± 2% softirqs.CPU89.RCU 116227 ± 3% +11.0% 129064 ± 3% softirqs.CPU89.TIMER 135289 ± 3% +13.6% 153744 ± 3% softirqs.CPU9.RCU 133598 ± 3% +14.5% 153019 ± 3% softirqs.CPU90.RCU 116395 ± 3% +10.9% 129084 ± 4% softirqs.CPU90.TIMER 133854 ± 4% +14.9% 153799 ± 3% softirqs.CPU91.RCU 116562 ± 4% +11.1% 129452 ± 3% softirqs.CPU91.TIMER 134107 ± 3% +14.1% 152958 ± 3% softirqs.CPU92.RCU 117107 ± 4% +10.6% 129559 ± 3% softirqs.CPU92.TIMER 133800 ± 3% +14.7% 153465 ± 3% softirqs.CPU93.RCU 116415 ± 3% +11.5% 129823 ± 4% softirqs.CPU93.TIMER 134092 ± 3% +14.0% 152835 ± 3% softirqs.CPU94.RCU 116063 ± 3% +11.4% 129252 ± 4% softirqs.CPU94.TIMER 134284 ± 3% +14.1% 153230 ± 4% softirqs.CPU95.RCU 117908 ± 3% +11.0% 130904 ± 4% softirqs.CPU95.TIMER 134264 ± 3% +14.3% 153455 ± 3% softirqs.CPU96.RCU 109819 ± 3% +8.6% 119258 ± 3% softirqs.CPU96.TIMER 135000 ± 3% +13.7% 153529 ± 3% softirqs.CPU97.RCU 134163 ± 3% +14.2% 153253 ± 3% softirqs.CPU98.RCU 109990 ± 3% +8.1% 118896 ± 3% softirqs.CPU98.TIMER 134966 ± 3% +13.5% 153245 ± 3% softirqs.CPU99.RCU 25776297 ± 3% +14.3% 29452390 ± 3% softirqs.RCU 21445010 ± 3% +9.5% 23488096 ± 3% softirqs.TIMER 614.00 ± 2% +9.9% 674.50 ± 2% interrupts.9:IO-APIC.9-fasteoi.acpi 611854 ± 2% +10.1% 673945 ± 2% interrupts.CPU0.LOC:Local_timer_interrupts 614.00 ± 2% +9.9% 674.50 ± 2% interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi 611835 ± 2% +10.1% 673883 ± 2% interrupts.CPU1.LOC:Local_timer_interrupts 611859 ± 2% +10.2% 673975 ± 2% interrupts.CPU10.LOC:Local_timer_interrupts 7508 ± 5% +15.5% 8675 ± 4% interrupts.CPU10.RES:Rescheduling_interrupts 611802 ± 2% +10.2% 673904 ± 2% interrupts.CPU100.LOC:Local_timer_interrupts 7323 ± 5% +16.1% 8503 ± 3% interrupts.CPU100.RES:Rescheduling_interrupts 611804 ± 2% +10.2% 673917 ± 2% interrupts.CPU101.LOC:Local_timer_interrupts 7342 ± 3% +13.8% 8352 ± 3% interrupts.CPU101.RES:Rescheduling_interrupts 611780 ± 2% +10.2% 673877 ± 2% interrupts.CPU102.LOC:Local_timer_interrupts 7506 ± 5% +12.2% 8424 ± 4% interrupts.CPU102.RES:Rescheduling_interrupts 611734 ± 2% +10.2% 673851 ± 2% interrupts.CPU103.LOC:Local_timer_interrupts 7436 ± 3% +15.8% 8607 ± 4% interrupts.CPU103.RES:Rescheduling_interrupts 611800 ± 2% +10.2% 673908 ± 2% interrupts.CPU104.LOC:Local_timer_interrupts 7519 ± 5% +13.1% 8502 ± 2% interrupts.CPU104.RES:Rescheduling_interrupts 611790 ± 2% +10.1% 673864 ± 2% interrupts.CPU105.LOC:Local_timer_interrupts 7346 ± 5% +16.4% 8553 ± 2% interrupts.CPU105.RES:Rescheduling_interrupts 611801 ± 2% +10.1% 673859 ± 2% interrupts.CPU106.LOC:Local_timer_interrupts 7352 ± 5% +16.5% 8567 ± 2% interrupts.CPU106.RES:Rescheduling_interrupts 611797 ± 2% +10.1% 673890 ± 2% interrupts.CPU107.LOC:Local_timer_interrupts 7238 ± 6% +15.3% 8344 ± 4% interrupts.CPU107.RES:Rescheduling_interrupts 611778 ± 2% +10.2% 673882 ± 2% interrupts.CPU108.LOC:Local_timer_interrupts 7310 ± 5% +16.3% 8503 ± 5% interrupts.CPU108.RES:Rescheduling_interrupts 611788 ± 2% +10.1% 673881 ± 2% interrupts.CPU109.LOC:Local_timer_interrupts 7324 ± 6% +13.4% 8307 interrupts.CPU109.RES:Rescheduling_interrupts 611827 ± 2% +10.2% 673960 ± 2% interrupts.CPU11.LOC:Local_timer_interrupts 7598 ± 6% +12.6% 8557 ± 5% interrupts.CPU11.RES:Rescheduling_interrupts 611796 ± 2% +10.1% 673865 ± 2% interrupts.CPU110.LOC:Local_timer_interrupts 7326 ± 4% +16.2% 8513 ± 4% interrupts.CPU110.RES:Rescheduling_interrupts 611804 ± 2% +10.1% 673890 ± 2% interrupts.CPU111.LOC:Local_timer_interrupts 7312 ± 3% +14.4% 8365 ± 3% interrupts.CPU111.RES:Rescheduling_interrupts 611801 ± 2% +10.1% 673890 ± 2% interrupts.CPU112.LOC:Local_timer_interrupts 7277 ± 2% +14.6% 8337 ± 5% interrupts.CPU112.RES:Rescheduling_interrupts 611789 ± 2% +10.2% 673896 ± 2% interrupts.CPU113.LOC:Local_timer_interrupts 7346 ± 4% +15.2% 8461 ± 4% interrupts.CPU113.RES:Rescheduling_interrupts 611759 ± 2% +10.2% 673861 ± 2% interrupts.CPU114.LOC:Local_timer_interrupts 7290 ± 4% +17.9% 8594 ± 4% interrupts.CPU114.RES:Rescheduling_interrupts 611783 ± 2% +10.2% 673883 ± 2% interrupts.CPU115.LOC:Local_timer_interrupts 7470 ± 4% +13.6% 8489 ± 5% interrupts.CPU115.RES:Rescheduling_interrupts 611807 ± 2% +10.2% 673925 ± 2% interrupts.CPU116.LOC:Local_timer_interrupts 7336 ± 4% +15.3% 8455 ± 2% interrupts.CPU116.RES:Rescheduling_interrupts 611778 ± 2% +10.2% 673890 ± 2% interrupts.CPU117.LOC:Local_timer_interrupts 7326 ± 5% +15.9% 8493 ± 3% interrupts.CPU117.RES:Rescheduling_interrupts 611782 ± 2% +10.1% 673849 ± 2% interrupts.CPU118.LOC:Local_timer_interrupts 7239 ± 4% +18.4% 8574 ± 5% interrupts.CPU118.RES:Rescheduling_interrupts 611778 ± 2% +10.2% 673881 ± 2% interrupts.CPU119.LOC:Local_timer_interrupts 7327 ± 4% +13.7% 8335 ± 6% interrupts.CPU119.RES:Rescheduling_interrupts 611853 ± 2% +10.1% 673944 ± 2% interrupts.CPU12.LOC:Local_timer_interrupts 7516 ± 4% +17.0% 8798 ± 6% interrupts.CPU12.RES:Rescheduling_interrupts 611417 ± 2% +10.2% 673606 ± 2% interrupts.CPU120.LOC:Local_timer_interrupts 7392 ± 4% +14.2% 8444 ± 5% interrupts.CPU120.RES:Rescheduling_interrupts 611457 ± 2% +10.2% 673608 ± 2% interrupts.CPU121.LOC:Local_timer_interrupts 7376 ± 3% +15.6% 8525 ± 4% interrupts.CPU121.RES:Rescheduling_interrupts 611426 ± 2% +10.2% 673610 ± 2% interrupts.CPU122.LOC:Local_timer_interrupts 7466 ± 5% +11.7% 8336 ± 4% interrupts.CPU122.RES:Rescheduling_interrupts 611384 ± 2% +10.2% 673577 ± 2% interrupts.CPU123.LOC:Local_timer_interrupts 7240 ± 5% +18.2% 8559 ± 3% interrupts.CPU123.RES:Rescheduling_interrupts 611388 ± 2% +10.2% 673597 ± 2% interrupts.CPU124.LOC:Local_timer_interrupts 7245 ± 5% +18.0% 8548 ± 5% interrupts.CPU124.RES:Rescheduling_interrupts 611391 ± 2% +10.2% 673567 ± 2% interrupts.CPU125.LOC:Local_timer_interrupts 7318 ± 3% +15.3% 8436 ± 5% interrupts.CPU125.RES:Rescheduling_interrupts 611421 ± 2% +10.2% 673570 ± 2% interrupts.CPU126.LOC:Local_timer_interrupts 7258 ± 5% +16.4% 8448 ± 4% interrupts.CPU126.RES:Rescheduling_interrupts 611468 ± 2% +10.2% 673614 ± 2% interrupts.CPU127.LOC:Local_timer_interrupts 7276 ± 5% +16.9% 8508 ± 5% interrupts.CPU127.RES:Rescheduling_interrupts 611422 ± 2% +10.2% 673591 ± 2% interrupts.CPU128.LOC:Local_timer_interrupts 7278 ± 4% +17.0% 8518 ± 3% interrupts.CPU128.RES:Rescheduling_interrupts 611407 ± 2% +10.2% 673562 ± 2% interrupts.CPU129.LOC:Local_timer_interrupts 7284 ± 3% +17.0% 8519 ± 5% interrupts.CPU129.RES:Rescheduling_interrupts 611825 ± 2% +10.1% 673920 ± 2% interrupts.CPU13.LOC:Local_timer_interrupts 7550 ± 4% +16.2% 8771 ± 3% interrupts.CPU13.RES:Rescheduling_interrupts 611428 ± 2% +10.2% 673571 ± 2% interrupts.CPU130.LOC:Local_timer_interrupts 7337 ± 5% +16.4% 8541 ± 5% interrupts.CPU130.RES:Rescheduling_interrupts 611406 ± 2% +10.2% 673582 ± 2% interrupts.CPU131.LOC:Local_timer_interrupts 7205 ± 5% +17.1% 8440 ± 6% interrupts.CPU131.RES:Rescheduling_interrupts 611449 ± 2% +10.2% 673586 ± 2% interrupts.CPU132.LOC:Local_timer_interrupts 7431 ± 4% +12.7% 8372 ± 3% interrupts.CPU132.RES:Rescheduling_interrupts 611464 ± 2% +10.2% 673588 ± 2% interrupts.CPU133.LOC:Local_timer_interrupts 7289 ± 4% +15.9% 8446 ± 3% interrupts.CPU133.RES:Rescheduling_interrupts 611437 ± 2% +10.2% 673583 ± 2% interrupts.CPU134.LOC:Local_timer_interrupts 7321 ± 4% +17.7% 8616 ± 4% interrupts.CPU134.RES:Rescheduling_interrupts 611401 ± 2% +10.2% 673595 ± 2% interrupts.CPU135.LOC:Local_timer_interrupts 7300 ± 4% +15.7% 8446 ± 4% interrupts.CPU135.RES:Rescheduling_interrupts 611462 ± 2% +10.2% 673602 ± 2% interrupts.CPU136.LOC:Local_timer_interrupts 7389 ± 4% +13.9% 8417 ± 5% interrupts.CPU136.RES:Rescheduling_interrupts 611424 ± 2% +10.2% 673588 ± 2% interrupts.CPU137.LOC:Local_timer_interrupts 7126 ± 4% +20.0% 8550 ± 5% interrupts.CPU137.RES:Rescheduling_interrupts 611383 ± 2% +10.2% 673556 ± 2% interrupts.CPU138.LOC:Local_timer_interrupts 7297 ± 5% +16.1% 8475 ± 5% interrupts.CPU138.RES:Rescheduling_interrupts 611422 ± 2% +10.2% 673603 ± 2% interrupts.CPU139.LOC:Local_timer_interrupts 7160 ± 4% +17.6% 8423 ± 4% interrupts.CPU139.RES:Rescheduling_interrupts 611815 ± 2% +10.1% 673901 ± 2% interrupts.CPU14.LOC:Local_timer_interrupts 7576 ± 5% +16.3% 8810 ± 3% interrupts.CPU14.RES:Rescheduling_interrupts 611388 ± 2% +10.2% 673577 ± 2% interrupts.CPU140.LOC:Local_timer_interrupts 7342 ± 6% +12.9% 8293 ± 4% interrupts.CPU140.RES:Rescheduling_interrupts 611394 ± 2% +10.2% 673596 ± 2% interrupts.CPU141.LOC:Local_timer_interrupts 7257 ± 4% +16.1% 8423 ± 5% interrupts.CPU141.RES:Rescheduling_interrupts 611411 ± 2% +10.2% 673590 ± 2% interrupts.CPU142.LOC:Local_timer_interrupts 7300 ± 6% +16.1% 8475 ± 3% interrupts.CPU142.RES:Rescheduling_interrupts 611396 ± 2% +10.2% 673622 ± 2% interrupts.CPU143.LOC:Local_timer_interrupts 7399 ± 5% +14.0% 8435 ± 4% interrupts.CPU143.RES:Rescheduling_interrupts 610929 ± 2% +10.3% 673686 ± 2% interrupts.CPU144.LOC:Local_timer_interrupts 7281 ± 5% +13.4% 8257 ± 5% interrupts.CPU144.RES:Rescheduling_interrupts 610918 ± 2% +10.3% 673755 ± 2% interrupts.CPU145.LOC:Local_timer_interrupts 7343 ± 5% +14.5% 8408 ± 4% interrupts.CPU145.RES:Rescheduling_interrupts 610910 ± 2% +10.3% 673705 ± 2% interrupts.CPU146.LOC:Local_timer_interrupts 7524 ± 4% +13.6% 8550 ± 6% interrupts.CPU146.RES:Rescheduling_interrupts 610858 ± 2% +10.3% 673769 ± 2% interrupts.CPU147.LOC:Local_timer_interrupts 7433 ± 4% +10.8% 8237 ± 3% interrupts.CPU147.RES:Rescheduling_interrupts 610872 ± 2% +10.3% 673727 ± 2% interrupts.CPU148.LOC:Local_timer_interrupts 7354 ± 3% +13.1% 8320 ± 4% interrupts.CPU148.RES:Rescheduling_interrupts 610909 ± 2% +10.3% 673750 ± 2% interrupts.CPU149.LOC:Local_timer_interrupts 7365 ± 3% +16.5% 8582 ± 3% interrupts.CPU149.RES:Rescheduling_interrupts 611818 ± 2% +10.2% 673937 ± 2% interrupts.CPU15.LOC:Local_timer_interrupts 7632 ± 4% +12.5% 8587 ± 4% interrupts.CPU15.RES:Rescheduling_interrupts 610898 ± 2% +10.3% 673770 ± 2% interrupts.CPU150.LOC:Local_timer_interrupts 7369 ± 4% +12.4% 8282 ± 3% interrupts.CPU150.RES:Rescheduling_interrupts 610887 ± 2% +10.3% 673754 ± 2% interrupts.CPU151.LOC:Local_timer_interrupts 7322 ± 4% +15.3% 8445 ± 3% interrupts.CPU151.RES:Rescheduling_interrupts 610849 ± 2% +10.3% 673762 ± 2% interrupts.CPU152.LOC:Local_timer_interrupts 7305 ± 4% +15.5% 8439 ± 5% interrupts.CPU152.RES:Rescheduling_interrupts 610909 ± 2% +10.3% 673759 ± 2% interrupts.CPU153.LOC:Local_timer_interrupts 7363 ± 6% +14.5% 8432 ± 2% interrupts.CPU153.RES:Rescheduling_interrupts 610878 ± 2% +10.3% 673730 ± 2% interrupts.CPU154.LOC:Local_timer_interrupts 7373 ± 4% +14.1% 8414 ± 4% interrupts.CPU154.RES:Rescheduling_interrupts 610891 ± 2% +10.3% 673720 ± 2% interrupts.CPU155.LOC:Local_timer_interrupts 7385 ± 5% +11.8% 8260 ± 4% interrupts.CPU155.RES:Rescheduling_interrupts 610909 ± 2% +10.3% 673753 ± 2% interrupts.CPU156.LOC:Local_timer_interrupts 7377 ± 5% +14.0% 8410 ± 5% interrupts.CPU156.RES:Rescheduling_interrupts 610886 ± 2% +10.3% 673754 ± 2% interrupts.CPU157.LOC:Local_timer_interrupts 7399 ± 5% +13.1% 8370 ± 4% interrupts.CPU157.RES:Rescheduling_interrupts 610854 ± 2% +10.3% 673699 ± 2% interrupts.CPU158.LOC:Local_timer_interrupts 7489 ± 4% +10.5% 8278 ± 2% interrupts.CPU158.RES:Rescheduling_interrupts 610921 ± 2% +10.3% 673726 ± 2% interrupts.CPU159.LOC:Local_timer_interrupts 7281 ± 5% +15.4% 8402 ± 4% interrupts.CPU159.RES:Rescheduling_interrupts 611836 ± 2% +10.1% 673889 ± 2% interrupts.CPU16.LOC:Local_timer_interrupts 7548 ± 4% +15.6% 8723 ± 5% interrupts.CPU16.RES:Rescheduling_interrupts 610870 ± 2% +10.3% 673742 ± 2% interrupts.CPU160.LOC:Local_timer_interrupts 610856 ± 2% +10.3% 673722 ± 2% interrupts.CPU161.LOC:Local_timer_interrupts 7472 ± 5% +13.4% 8475 ± 5% interrupts.CPU161.RES:Rescheduling_interrupts 610891 ± 2% +10.3% 673708 ± 2% interrupts.CPU162.LOC:Local_timer_interrupts 7177 ± 5% +14.8% 8239 ± 5% interrupts.CPU162.RES:Rescheduling_interrupts 610864 ± 2% +10.3% 673696 ± 2% interrupts.CPU163.LOC:Local_timer_interrupts 7337 ± 5% +12.7% 8266 ± 4% interrupts.CPU163.RES:Rescheduling_interrupts 610858 ± 2% +10.3% 673724 ± 2% interrupts.CPU164.LOC:Local_timer_interrupts 7269 ± 5% +15.2% 8376 ± 4% interrupts.CPU164.RES:Rescheduling_interrupts 610861 ± 2% +10.3% 673711 ± 2% interrupts.CPU165.LOC:Local_timer_interrupts 7431 ± 4% +9.4% 8131 ± 4% interrupts.CPU165.RES:Rescheduling_interrupts 610867 ± 2% +10.3% 673684 ± 2% interrupts.CPU166.LOC:Local_timer_interrupts 7339 ± 2% +14.1% 8374 ± 4% interrupts.CPU166.RES:Rescheduling_interrupts 610913 ± 2% +10.3% 673730 ± 2% interrupts.CPU167.LOC:Local_timer_interrupts 7174 ± 3% +17.5% 8427 ± 4% interrupts.CPU167.RES:Rescheduling_interrupts 611289 ± 2% +10.2% 673888 ± 2% interrupts.CPU168.LOC:Local_timer_interrupts 7311 ± 5% +14.7% 8382 ± 3% interrupts.CPU168.RES:Rescheduling_interrupts 611333 ± 2% +10.2% 673887 ± 2% interrupts.CPU169.LOC:Local_timer_interrupts 7232 ± 2% +17.8% 8517 ± 5% interrupts.CPU169.RES:Rescheduling_interrupts 611838 ± 2% +10.1% 673879 ± 2% interrupts.CPU17.LOC:Local_timer_interrupts 7539 ± 2% +17.7% 8873 ± 10% interrupts.CPU17.RES:Rescheduling_interrupts 611349 ± 2% +10.2% 673889 ± 2% interrupts.CPU170.LOC:Local_timer_interrupts 7374 ± 2% +14.2% 8424 ± 5% interrupts.CPU170.RES:Rescheduling_interrupts 611297 ± 2% +10.2% 673873 ± 2% interrupts.CPU171.LOC:Local_timer_interrupts 7438 ± 4% +13.9% 8471 ± 5% interrupts.CPU171.RES:Rescheduling_interrupts 611304 ± 2% +10.2% 673845 ± 2% interrupts.CPU172.LOC:Local_timer_interrupts 7317 ± 3% +16.6% 8529 ± 5% interrupts.CPU172.RES:Rescheduling_interrupts 611297 ± 2% +10.2% 673887 ± 2% interrupts.CPU173.LOC:Local_timer_interrupts 611309 ± 2% +10.2% 673858 ± 2% interrupts.CPU174.LOC:Local_timer_interrupts 7369 ± 5% +15.8% 8531 ± 5% interrupts.CPU174.RES:Rescheduling_interrupts 611311 ± 2% +10.2% 673872 ± 2% interrupts.CPU175.LOC:Local_timer_interrupts 7352 ± 4% +15.1% 8465 ± 4% interrupts.CPU175.RES:Rescheduling_interrupts 611309 ± 2% +10.2% 673886 ± 2% interrupts.CPU176.LOC:Local_timer_interrupts 7395 ± 2% +14.0% 8434 ± 5% interrupts.CPU176.RES:Rescheduling_interrupts 611322 ± 2% +10.2% 673868 ± 2% interrupts.CPU177.LOC:Local_timer_interrupts 611248 ± 2% +10.2% 673805 ± 2% interrupts.CPU178.LOC:Local_timer_interrupts 7355 ± 5% +14.1% 8393 ± 4% interrupts.CPU178.RES:Rescheduling_interrupts 611319 ± 2% +10.2% 673906 ± 2% interrupts.CPU179.LOC:Local_timer_interrupts 7269 ± 4% +15.2% 8374 ± 6% interrupts.CPU179.RES:Rescheduling_interrupts 611799 ± 2% +10.1% 673864 ± 2% interrupts.CPU18.LOC:Local_timer_interrupts 7585 ± 4% +14.5% 8683 ± 4% interrupts.CPU18.RES:Rescheduling_interrupts 611330 ± 2% +10.2% 673869 ± 2% interrupts.CPU180.LOC:Local_timer_interrupts 7404 ± 4% +14.0% 8442 ± 5% interrupts.CPU180.RES:Rescheduling_interrupts 611324 ± 2% +10.2% 673889 ± 2% interrupts.CPU181.LOC:Local_timer_interrupts 7435 ± 2% +13.0% 8399 ± 3% interrupts.CPU181.RES:Rescheduling_interrupts 611322 ± 2% +10.2% 673874 ± 2% interrupts.CPU182.LOC:Local_timer_interrupts 7250 ± 4% +14.6% 8305 ± 4% interrupts.CPU182.RES:Rescheduling_interrupts 611307 ± 2% +10.2% 673852 ± 2% interrupts.CPU183.LOC:Local_timer_interrupts 7320 ± 4% +13.2% 8290 ± 5% interrupts.CPU183.RES:Rescheduling_interrupts 611329 ± 2% +10.2% 673877 ± 2% interrupts.CPU184.LOC:Local_timer_interrupts 7377 ± 4% +12.5% 8299 ± 5% interrupts.CPU184.RES:Rescheduling_interrupts 611312 ± 2% +10.2% 673866 ± 2% interrupts.CPU185.LOC:Local_timer_interrupts 7333 ± 4% +13.9% 8351 ± 5% interrupts.CPU185.RES:Rescheduling_interrupts 611290 ± 2% +10.2% 673868 ± 2% interrupts.CPU186.LOC:Local_timer_interrupts 7426 ± 4% +14.1% 8471 ± 3% interrupts.CPU186.RES:Rescheduling_interrupts 611304 ± 2% +10.2% 673853 ± 2% interrupts.CPU187.LOC:Local_timer_interrupts 7380 ± 3% +12.5% 8304 ± 5% interrupts.CPU187.RES:Rescheduling_interrupts 611307 ± 2% +10.2% 673881 ± 2% interrupts.CPU188.LOC:Local_timer_interrupts 7277 ± 5% +16.3% 8467 ± 6% interrupts.CPU188.RES:Rescheduling_interrupts 611299 ± 2% +10.2% 673879 ± 2% interrupts.CPU189.LOC:Local_timer_interrupts 7436 ± 3% +14.5% 8514 ± 6% interrupts.CPU189.RES:Rescheduling_interrupts 611761 ± 2% +10.2% 673928 ± 2% interrupts.CPU19.LOC:Local_timer_interrupts 7515 ± 6% +12.7% 8467 ± 4% interrupts.CPU19.RES:Rescheduling_interrupts 611321 ± 2% +10.2% 673856 ± 2% interrupts.CPU190.LOC:Local_timer_interrupts 7188 ± 3% +16.4% 8366 ± 5% interrupts.CPU190.RES:Rescheduling_interrupts 611544 ± 2% +10.2% 674120 ± 2% interrupts.CPU191.LOC:Local_timer_interrupts 7207 ± 3% +15.3% 8312 ± 3% interrupts.CPU191.RES:Rescheduling_interrupts 611805 ± 2% +10.2% 673920 ± 2% interrupts.CPU2.LOC:Local_timer_interrupts 611781 ± 2% +10.2% 673897 ± 2% interrupts.CPU20.LOC:Local_timer_interrupts 7531 ± 5% +16.4% 8768 ± 3% interrupts.CPU20.RES:Rescheduling_interrupts 611715 ± 2% +10.2% 673897 ± 2% interrupts.CPU21.LOC:Local_timer_interrupts 7467 ± 3% +14.9% 8577 ± 4% interrupts.CPU21.RES:Rescheduling_interrupts 611834 ± 2% +10.1% 673900 ± 2% interrupts.CPU22.LOC:Local_timer_interrupts 7427 ± 6% +17.9% 8754 ± 4% interrupts.CPU22.RES:Rescheduling_interrupts 611788 ± 2% +10.1% 673869 ± 2% interrupts.CPU23.LOC:Local_timer_interrupts 7592 ± 4% +12.2% 8518 ± 4% interrupts.CPU23.RES:Rescheduling_interrupts 611426 ± 2% +10.2% 673619 ± 2% interrupts.CPU24.LOC:Local_timer_interrupts 7457 ± 4% +14.7% 8551 ± 3% interrupts.CPU24.RES:Rescheduling_interrupts 611439 ± 2% +10.2% 673636 ± 2% interrupts.CPU25.LOC:Local_timer_interrupts 611402 ± 2% +10.2% 673620 ± 2% interrupts.CPU26.LOC:Local_timer_interrupts 7463 ± 3% +18.4% 8838 ± 7% interrupts.CPU26.RES:Rescheduling_interrupts 611413 ± 2% +10.2% 673615 ± 2% interrupts.CPU27.LOC:Local_timer_interrupts 7412 ± 4% +16.1% 8607 ± 4% interrupts.CPU27.RES:Rescheduling_interrupts 611409 ± 2% +10.2% 673625 ± 2% interrupts.CPU28.LOC:Local_timer_interrupts 7478 ± 5% +15.6% 8648 ± 3% interrupts.CPU28.RES:Rescheduling_interrupts 611415 ± 2% +10.2% 673620 ± 2% interrupts.CPU29.LOC:Local_timer_interrupts 7431 ± 2% +15.5% 8585 ± 2% interrupts.CPU29.RES:Rescheduling_interrupts 611807 ± 2% +10.1% 673896 ± 2% interrupts.CPU3.LOC:Local_timer_interrupts 7659 ± 5% +20.9% 9264 ± 10% interrupts.CPU3.RES:Rescheduling_interrupts 611433 ± 2% +10.2% 673520 ± 2% interrupts.CPU30.LOC:Local_timer_interrupts 7432 ± 3% +16.5% 8659 ± 4% interrupts.CPU30.RES:Rescheduling_interrupts 611450 ± 2% +10.2% 673604 ± 2% interrupts.CPU31.LOC:Local_timer_interrupts 7490 ± 4% +14.6% 8585 ± 5% interrupts.CPU31.RES:Rescheduling_interrupts 611444 ± 2% +10.2% 673588 ± 2% interrupts.CPU32.LOC:Local_timer_interrupts 7393 ± 4% +19.0% 8799 ± 3% interrupts.CPU32.RES:Rescheduling_interrupts 611407 ± 2% +10.2% 673606 ± 2% interrupts.CPU33.LOC:Local_timer_interrupts 7398 ± 4% +16.8% 8644 ± 4% interrupts.CPU33.RES:Rescheduling_interrupts 611452 ± 2% +10.2% 673609 ± 2% interrupts.CPU34.LOC:Local_timer_interrupts 7414 ± 5% +18.8% 8806 ± 4% interrupts.CPU34.RES:Rescheduling_interrupts 611433 ± 2% +10.2% 673588 ± 2% interrupts.CPU35.LOC:Local_timer_interrupts 7486 ± 5% +18.6% 8878 ± 3% interrupts.CPU35.RES:Rescheduling_interrupts 611450 ± 2% +10.2% 673603 ± 2% interrupts.CPU36.LOC:Local_timer_interrupts 7503 ± 3% +15.0% 8627 ± 3% interrupts.CPU36.RES:Rescheduling_interrupts 611465 ± 2% +10.2% 673611 ± 2% interrupts.CPU37.LOC:Local_timer_interrupts 7416 ± 3% +17.6% 8719 ± 2% interrupts.CPU37.RES:Rescheduling_interrupts 611445 ± 2% +10.2% 673575 ± 2% interrupts.CPU38.LOC:Local_timer_interrupts 7477 ± 5% +14.2% 8540 ± 3% interrupts.CPU38.RES:Rescheduling_interrupts 611420 ± 2% +10.2% 673600 ± 2% interrupts.CPU39.LOC:Local_timer_interrupts 7562 ± 5% +14.0% 8620 ± 4% interrupts.CPU39.RES:Rescheduling_interrupts 611823 ± 2% +10.2% 673949 ± 2% interrupts.CPU4.LOC:Local_timer_interrupts 7579 ± 4% +14.9% 8708 ± 3% interrupts.CPU4.RES:Rescheduling_interrupts 611390 ± 2% +10.2% 673618 ± 2% interrupts.CPU40.LOC:Local_timer_interrupts 7444 ± 4% +15.9% 8626 ± 3% interrupts.CPU40.RES:Rescheduling_interrupts 611429 ± 2% +10.2% 673612 ± 2% interrupts.CPU41.LOC:Local_timer_interrupts 7464 ± 5% +15.3% 8605 ± 2% interrupts.CPU41.RES:Rescheduling_interrupts 611418 ± 2% +10.2% 673597 ± 2% interrupts.CPU42.LOC:Local_timer_interrupts 7394 ± 4% +18.6% 8772 ± 5% interrupts.CPU42.RES:Rescheduling_interrupts 611364 ± 2% +10.2% 673601 ± 2% interrupts.CPU43.LOC:Local_timer_interrupts 7483 ± 5% +15.3% 8630 ± 3% interrupts.CPU43.RES:Rescheduling_interrupts 611421 ± 2% +10.2% 673593 ± 2% interrupts.CPU44.LOC:Local_timer_interrupts 7463 ± 2% +16.0% 8656 ± 4% interrupts.CPU44.RES:Rescheduling_interrupts 611423 ± 2% +10.2% 673593 ± 2% interrupts.CPU45.LOC:Local_timer_interrupts 7455 ± 4% +14.8% 8558 ± 3% interrupts.CPU45.RES:Rescheduling_interrupts 611333 ± 2% +10.2% 673616 ± 2% interrupts.CPU46.LOC:Local_timer_interrupts 7406 ± 3% +17.4% 8696 ± 3% interrupts.CPU46.RES:Rescheduling_interrupts 611421 ± 2% +10.2% 673605 ± 2% interrupts.CPU47.LOC:Local_timer_interrupts 7482 ± 5% +15.7% 8660 ± 2% interrupts.CPU47.RES:Rescheduling_interrupts 610910 ± 2% +10.3% 673737 ± 2% interrupts.CPU48.LOC:Local_timer_interrupts 7521 ± 6% +15.1% 8655 ± 4% interrupts.CPU48.RES:Rescheduling_interrupts 610921 ± 2% +10.3% 673770 ± 2% interrupts.CPU49.LOC:Local_timer_interrupts 7594 ± 5% +12.7% 8558 ± 3% interrupts.CPU49.RES:Rescheduling_interrupts 611796 ± 2% +10.1% 673831 ± 2% interrupts.CPU5.LOC:Local_timer_interrupts 610930 ± 2% +10.3% 673671 ± 2% interrupts.CPU50.LOC:Local_timer_interrupts 7504 ± 2% +17.6% 8824 ± 4% interrupts.CPU50.RES:Rescheduling_interrupts 610882 ± 2% +10.3% 673738 ± 2% interrupts.CPU51.LOC:Local_timer_interrupts 610884 ± 2% +10.3% 673731 ± 2% interrupts.CPU52.LOC:Local_timer_interrupts 7436 ± 4% +18.1% 8783 ± 3% interrupts.CPU52.RES:Rescheduling_interrupts 610885 ± 2% +10.3% 673774 ± 2% interrupts.CPU53.LOC:Local_timer_interrupts 7544 ± 4% +15.3% 8700 ± 4% interrupts.CPU53.RES:Rescheduling_interrupts 610902 ± 2% +10.3% 673763 ± 2% interrupts.CPU54.LOC:Local_timer_interrupts 7534 ± 4% +15.6% 8709 ± 6% interrupts.CPU54.RES:Rescheduling_interrupts 610927 ± 2% +10.3% 673739 ± 2% interrupts.CPU55.LOC:Local_timer_interrupts 610883 ± 2% +10.3% 673719 ± 2% interrupts.CPU56.LOC:Local_timer_interrupts 610921 ± 2% +10.3% 673689 ± 2% interrupts.CPU57.LOC:Local_timer_interrupts 7473 ± 5% +14.8% 8577 ± 4% interrupts.CPU57.RES:Rescheduling_interrupts 610893 ± 2% +10.3% 673781 ± 2% interrupts.CPU58.LOC:Local_timer_interrupts 7407 ± 5% +16.6% 8635 ± 3% interrupts.CPU58.RES:Rescheduling_interrupts 610898 ± 2% +10.3% 673763 ± 2% interrupts.CPU59.LOC:Local_timer_interrupts 7642 ± 5% +13.1% 8643 ± 3% interrupts.CPU59.RES:Rescheduling_interrupts 611823 ± 2% +10.1% 673914 ± 2% interrupts.CPU6.LOC:Local_timer_interrupts 7542 ± 3% +17.3% 8845 ± 3% interrupts.CPU6.RES:Rescheduling_interrupts 610869 ± 2% +10.3% 673784 ± 2% interrupts.CPU60.LOC:Local_timer_interrupts 7535 ± 4% +14.8% 8647 ± 5% interrupts.CPU60.RES:Rescheduling_interrupts 610906 ± 2% +10.3% 673701 ± 2% interrupts.CPU61.LOC:Local_timer_interrupts 7631 ± 4% +13.5% 8660 ± 4% interrupts.CPU61.RES:Rescheduling_interrupts 610863 ± 2% +10.3% 673705 ± 2% interrupts.CPU62.LOC:Local_timer_interrupts 7634 ± 4% +11.8% 8532 ± 4% interrupts.CPU62.RES:Rescheduling_interrupts 610920 ± 2% +10.3% 673730 ± 2% interrupts.CPU63.LOC:Local_timer_interrupts 7549 ± 5% +14.8% 8663 ± 5% interrupts.CPU63.RES:Rescheduling_interrupts 610884 ± 2% +10.3% 673727 ± 2% interrupts.CPU64.LOC:Local_timer_interrupts 7644 ± 3% +13.4% 8672 ± 3% interrupts.CPU64.RES:Rescheduling_interrupts 610853 ± 2% +10.3% 673708 ± 2% interrupts.CPU65.LOC:Local_timer_interrupts 7555 ± 5% +14.8% 8672 ± 3% interrupts.CPU65.RES:Rescheduling_interrupts 610950 ± 2% +10.3% 673804 ± 2% interrupts.CPU66.LOC:Local_timer_interrupts 7457 ± 5% +16.8% 8710 ± 4% interrupts.CPU66.RES:Rescheduling_interrupts 610888 ± 2% +10.3% 673735 ± 2% interrupts.CPU67.LOC:Local_timer_interrupts 7550 ± 4% +11.5% 8415 ± 5% interrupts.CPU67.RES:Rescheduling_interrupts 610891 ± 2% +10.3% 673727 ± 2% interrupts.CPU68.LOC:Local_timer_interrupts 7612 ± 4% +12.1% 8534 ± 3% interrupts.CPU68.RES:Rescheduling_interrupts 610901 ± 2% +10.3% 673659 ± 2% interrupts.CPU69.LOC:Local_timer_interrupts 7493 ± 4% +14.8% 8601 ± 4% interrupts.CPU69.RES:Rescheduling_interrupts 611808 ± 2% +10.1% 673844 ± 2% interrupts.CPU7.LOC:Local_timer_interrupts 7673 ± 5% +15.0% 8821 ± 4% interrupts.CPU7.RES:Rescheduling_interrupts 610906 ± 2% +10.3% 673733 ± 2% interrupts.CPU70.LOC:Local_timer_interrupts 7412 ± 3% +16.9% 8664 ± 4% interrupts.CPU70.RES:Rescheduling_interrupts 610871 ± 2% +10.3% 673671 ± 2% interrupts.CPU71.LOC:Local_timer_interrupts 7519 ± 4% +13.8% 8557 ± 5% interrupts.CPU71.RES:Rescheduling_interrupts 611325 ± 2% +10.2% 673906 ± 2% interrupts.CPU72.LOC:Local_timer_interrupts 7720 ± 4% +9.6% 8459 ± 4% interrupts.CPU72.RES:Rescheduling_interrupts 611328 ± 2% +10.2% 673908 ± 2% interrupts.CPU73.LOC:Local_timer_interrupts 7535 ± 4% +14.6% 8634 ± 4% interrupts.CPU73.RES:Rescheduling_interrupts 611311 ± 2% +10.2% 673883 ± 2% interrupts.CPU74.LOC:Local_timer_interrupts 7526 ± 5% +14.7% 8635 ± 6% interrupts.CPU74.RES:Rescheduling_interrupts 611315 ± 2% +10.2% 673868 ± 2% interrupts.CPU75.LOC:Local_timer_interrupts 7712 ± 5% +9.8% 8469 ± 4% interrupts.CPU75.RES:Rescheduling_interrupts 611304 ± 2% +10.2% 673860 ± 2% interrupts.CPU76.LOC:Local_timer_interrupts 7476 ± 5% +14.2% 8537 ± 5% interrupts.CPU76.RES:Rescheduling_interrupts 611322 ± 2% +10.2% 673834 ± 2% interrupts.CPU77.LOC:Local_timer_interrupts 7670 ± 3% +12.5% 8630 ± 5% interrupts.CPU77.RES:Rescheduling_interrupts 611326 ± 2% +10.2% 673851 ± 2% interrupts.CPU78.LOC:Local_timer_interrupts 7417 ± 3% +15.6% 8578 ± 4% interrupts.CPU78.RES:Rescheduling_interrupts 611328 ± 2% +10.2% 673820 ± 2% interrupts.CPU79.LOC:Local_timer_interrupts 7545 ± 4% +13.9% 8590 ± 4% interrupts.CPU79.RES:Rescheduling_interrupts 611842 ± 2% +10.1% 673928 ± 2% interrupts.CPU8.LOC:Local_timer_interrupts 7673 ± 3% +14.9% 8818 ± 4% interrupts.CPU8.RES:Rescheduling_interrupts 611366 ± 2% +10.2% 673868 ± 2% interrupts.CPU80.LOC:Local_timer_interrupts 7513 ± 3% +16.1% 8723 ± 5% interrupts.CPU80.RES:Rescheduling_interrupts 611322 ± 2% +10.2% 673860 ± 2% interrupts.CPU81.LOC:Local_timer_interrupts 611299 ± 2% +10.2% 673868 ± 2% interrupts.CPU82.LOC:Local_timer_interrupts 7523 ± 4% +13.8% 8562 ± 4% interrupts.CPU82.RES:Rescheduling_interrupts 611320 ± 2% +10.2% 673727 ± 2% interrupts.CPU83.LOC:Local_timer_interrupts 7518 ± 5% +12.4% 8454 ± 5% interrupts.CPU83.RES:Rescheduling_interrupts 611326 ± 2% +10.2% 673899 ± 2% interrupts.CPU84.LOC:Local_timer_interrupts 7452 ± 6% +12.3% 8365 ± 5% interrupts.CPU84.RES:Rescheduling_interrupts 611329 ± 2% +10.2% 673885 ± 2% interrupts.CPU85.LOC:Local_timer_interrupts 7399 ± 5% +11.1% 8218 ± 5% interrupts.CPU85.RES:Rescheduling_interrupts 611278 ± 2% +10.2% 673875 ± 2% interrupts.CPU86.LOC:Local_timer_interrupts 7376 ± 4% +16.3% 8581 ± 3% interrupts.CPU86.RES:Rescheduling_interrupts 611313 ± 2% +10.2% 673874 ± 2% interrupts.CPU87.LOC:Local_timer_interrupts 7544 ± 4% +12.8% 8508 ± 4% interrupts.CPU87.RES:Rescheduling_interrupts 611351 ± 2% +10.2% 673928 ± 2% interrupts.CPU88.LOC:Local_timer_interrupts 7418 ± 6% +13.3% 8406 ± 4% interrupts.CPU88.RES:Rescheduling_interrupts 611324 ± 2% +10.2% 673795 ± 2% interrupts.CPU89.LOC:Local_timer_interrupts 7426 ± 4% +13.6% 8439 ± 5% interrupts.CPU89.RES:Rescheduling_interrupts 611832 ± 2% +10.2% 673968 ± 2% interrupts.CPU9.LOC:Local_timer_interrupts 7538 ± 3% +15.6% 8715 ± 4% interrupts.CPU9.RES:Rescheduling_interrupts 611298 ± 2% +10.2% 673878 ± 2% interrupts.CPU90.LOC:Local_timer_interrupts 7421 ± 3% +12.7% 8363 ± 4% interrupts.CPU90.RES:Rescheduling_interrupts 611305 ± 2% +10.2% 673874 ± 2% interrupts.CPU91.LOC:Local_timer_interrupts 7377 ± 5% +15.6% 8528 ± 5% interrupts.CPU91.RES:Rescheduling_interrupts 611288 ± 2% +10.2% 673876 ± 2% interrupts.CPU92.LOC:Local_timer_interrupts 7349 ± 4% +12.0% 8234 ± 4% interrupts.CPU92.RES:Rescheduling_interrupts 611329 ± 2% +10.2% 673857 ± 2% interrupts.CPU93.LOC:Local_timer_interrupts 7438 ± 3% +14.7% 8530 ± 4% interrupts.CPU93.RES:Rescheduling_interrupts 611276 ± 2% +10.2% 673866 ± 2% interrupts.CPU94.LOC:Local_timer_interrupts 7363 ± 4% +12.7% 8300 ± 5% interrupts.CPU94.RES:Rescheduling_interrupts 611381 ± 2% +10.2% 673922 ± 2% interrupts.CPU95.LOC:Local_timer_interrupts 10549 ± 3% +14.9% 12117 ± 4% interrupts.CPU95.RES:Rescheduling_interrupts 611824 ± 2% +10.1% 673847 ± 2% interrupts.CPU96.LOC:Local_timer_interrupts 7416 ± 3% +12.9% 8375 ± 5% interrupts.CPU96.RES:Rescheduling_interrupts 611751 ± 2% +10.2% 673870 ± 2% interrupts.CPU97.LOC:Local_timer_interrupts 7437 ± 4% +14.0% 8477 ± 4% interrupts.CPU97.RES:Rescheduling_interrupts 611787 ± 2% +10.1% 673877 ± 2% interrupts.CPU98.LOC:Local_timer_interrupts 7555 ± 4% +13.0% 8534 ± 3% interrupts.CPU98.RES:Rescheduling_interrupts 611783 ± 2% +10.1% 673872 ± 2% interrupts.CPU99.LOC:Local_timer_interrupts 7452 ± 3% +12.8% 8404 ± 5% interrupts.CPU99.RES:Rescheduling_interrupts 1.174e+08 ± 2% +10.2% 1.294e+08 ± 2% interrupts.LOC:Local_timer_interrupts 1433875 ± 3% +14.6% 1643307 ± 3% interrupts.RES:Rescheduling_interrupts aim7.jobs-per-min 30500 +-------------------------------------------------------------------+ | ..+.... .| 30000 |-+ ...... ... ... | 29500 |................ ...... .. ... | | +.. ... .. | 29000 |-+ ... ... | 28500 |-+ . ... | | +. | 28000 |-+ O O | 27500 |-+ | | | 27000 |-+ | 26500 |-+ | | O | 26000 +-------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

2 months, 2 weeks

2
3
0 / 0

[sched] c3a340f7e7: invalid_opcode:#[##]

by kernel test robot

Greeting, FYI, we noticed the following commit (built with gcc-4.9): commit: c3a340f7e7eadac7662ab104ceb16432e5a4c6b2 ("sched: Have sched_class_highest define by vmlinux.lds.h") https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/core in testcase: trinity with following parameters: runtime: 300s test-description: Trinity is a linux system call fuzz tester. test-url: http://codemonkey.org.uk/projects/trinity/ on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): +------------------------------------------+------------+------------+ | | 590d697963 | c3a340f7e7 | +------------------------------------------+------------+------------+ | boot_successes | 4 | 0 | | boot_failures | 0 | 4 | | invalid_opcode:#[##] | 0 | 4 | | EIP:sched_init | 0 | 4 | | Kernel_panic-not_syncing:Fatal_exception | 0 | 4 | | kernel_BUG_at_kernel/sched/core.c | 0 | 4 | +------------------------------------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <lkp(a)intel.com> [ 1.813939] ** If you see this message and you are not debugging ** [ 1.814781] ** the kernel, report this immediately to your vendor! ** [ 1.815661] ** ** [ 1.816509] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 1.817362] ********************************************************** [ 1.820914] invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC [ 1.821634] CPU: 0 PID: 0 Comm: swapper Not tainted 5.8.0-rc1-00024-gc3a340f7e7ead #2 [ 1.822630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [ 1.823710] EIP: sched_init+0x3d/0x2fb [ 1.824191] Code: 20 a6 e5 c1 3d 40 a6 e5 c1 0f 95 c0 08 c2 75 1a b8 a0 a6 e5 c1 3d c0 a6 e5 c1 75 0e b8 20 a7 e5 c1 3d 40 a7 e5 c1 74 04 0f 0b <0f> 0b 55 89 e5 57 56 53 83 ec 08 e8 67 04 00 00 8b 3d 80 ad ec c1 [ 1.826607] EAX: c1e5a601 EBX: c1dac484 ECX: 00000000 EDX: 00000001 [ 1.827434] ESI: 00000000 EDI: 000003f8 EBP: c1e93f9c ESP: c1e93f78 [ 1.828241] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00210002 [ 1.829112] CR0: 80050033 CR2: ffc70000 CR3: 021ce000 CR4: 00000690 [ 1.829919] Call Trace: [ 1.830240] ? start_kernel+0x43f/0x677 [ 1.830738] i386_start_kernel+0x48/0x4a [ 1.831261] startup_32_smp+0x15f/0x164 [ 1.831757] Modules linked in: [ 1.832165] ---[ end trace db9d28bea02c9e7f ]--- [ 1.832761] EIP: sched_init+0x3d/0x2fb [ 1.833243] Code: 20 a6 e5 c1 3d 40 a6 e5 c1 0f 95 c0 08 c2 75 1a b8 a0 a6 e5 c1 3d c0 a6 e5 c1 75 0e b8 20 a7 e5 c1 3d 40 a7 e5 c1 74 04 0f 0b <0f> 0b 55 89 e5 57 56 53 83 ec 08 e8 67 04 00 00 8b 3d 80 ad ec c1 [ 1.835656] EAX: c1e5a601 EBX: c1dac484 ECX: 00000000 EDX: 00000001 [ 1.836463] ESI: 00000000 EDI: 000003f8 EBP: c1e93f9c ESP: c1e93f78 [ 1.838603] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00210002 [ 1.839496] CR0: 80050033 CR2: ffc70000 CR3: 021ce000 CR4: 00000690 [ 1.840300] Kernel panic - not syncing: Fatal exception [ 1.840969] ------------[ cut here ]------------ [ 1.840970] kernel BUG at kernel/sched/core.c:6652! Elapsed time: 60 qemu-img create -f qcow2 disk-vm-snb-94-0 256G qemu-img create -f qcow2 disk-vm-snb-94-1 256G To reproduce: # build kernel cd linux cp config-5.8.0-rc1-00024-gc3a340f7e7ead .config make HOSTCC=gcc-4.9 CC=gcc-4.9 ARCH=i386 olddefconfig prepare modules_prepare bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email Thanks, lkp

2 months, 3 weeks

6
12
0 / 0

[hugetlbfs] c0d0381ade: vm-scalability.throughput -33.4% regression

by kernel test robot

Greeting, FYI, we noticed a -33.4% regression of vm-scalability.throughput due to commit: commit: c0d0381ade79885c04a04c303284b040616b116e ("hugetlbfs: use i_mmap_rwsem for more pmd sharing synchronization") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: vm-scalability on test machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory with following parameters: runtime: 300s size: 8T test: anon-cow-seq-hugetlb cpufreq_governor: performance ucode: 0x11 test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us. test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/ If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-20191114.cgz/300s/8T/lkp-knm01/anon-cow-seq-hugetlb/vm-scalability/0x11 commit: 49aef7175c ("mm/memblock.c: remove redundant assignment to variable max_addr") c0d0381ade ("hugetlbfs: use i_mmap_rwsem for more pmd sharing synchronization") 49aef7175cc6eb70 c0d0381ade79885c04a04c30328 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 -25% :4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x :4 25% 1:4 dmesg.WARNING:at_ip__mutex_lock/0x :4 25% 1:4 dmesg.WARNING:at_ip_perf_event_mmap_output/0x %stddev %change %stddev \ | \ 37643 -29.5% 26549 ± 4% vm-scalability.median 9.18 ± 16% +9.3 18.48 ± 21% vm-scalability.median_stddev% 12666048 -33.4% 8429873 ± 4% vm-scalability.throughput 1701516 -19.5% 1369683 ± 5% vm-scalability.time.minor_page_faults 13447 ± 3% -19.3% 10848 vm-scalability.time.percent_of_cpu_this_job_got 26620 ± 4% -10.5% 23830 vm-scalability.time.system_time 14297 ± 3% -35.3% 9255 ± 4% vm-scalability.time.user_time 125377 ± 2% +621.3% 904348 ± 4% vm-scalability.time.voluntary_context_switches 2.468e+09 -19.6% 1.984e+09 ± 5% vm-scalability.workload 3.807e+10 ± 4% +20.9% 4.605e+10 ± 2% cpuidle.C1.time 9524 ± 10% -17.3% 7873 ± 4% numa-vmstat.node0.nr_mapped 2494252 -12.4% 2184278 ± 3% numa-numastat.node0.local_node 2494194 -12.4% 2184223 ± 3% numa-numastat.node0.numa_hit 49.26 ± 3% +9.4 58.62 mpstat.cpu.all.idle% 32.89 ± 4% -3.2 29.70 mpstat.cpu.all.sys% 17.79 ± 3% -6.2 11.64 ± 4% mpstat.cpu.all.usr% 4856 +47.6% 7169 numa-meminfo.node0.HugePages_Surp 4856 +47.6% 7169 numa-meminfo.node0.HugePages_Total 37725 ± 10% -17.7% 31054 ± 4% numa-meminfo.node0.Mapped 12274212 +38.3% 16981169 numa-meminfo.node0.MemUsed 49.00 ± 3% +18.9% 58.25 vmstat.cpu.id 17.00 ± 4% -33.8% 11.25 ± 3% vmstat.cpu.us 138.25 ± 2% -19.5% 111.25 ± 2% vmstat.procs.r 5836 ± 2% +88.0% 10969 ± 3% vmstat.system.cs 4855 +48.0% 7186 meminfo.HugePages_Surp 4856 +48.0% 7186 meminfo.HugePages_Total 9946017 +48.0% 14718559 meminfo.Hugetlb 48593 ± 8% -13.9% 41859 ± 3% meminfo.Mapped 12877361 +36.8% 17619364 meminfo.Memused 81607 ± 2% +29.1% 105324 meminfo.max_used_kB 2890 ±100% +285.4% 11139 ± 3% perf-stat.i.context-switches 298.31 ±100% +243.7% 1025 ± 2% perf-stat.i.cpu-migrations 0.11 ±100% +119.3% 0.24 perf-stat.i.ipc 0.04 ±100% +129.7% 0.10 ± 4% perf-stat.i.metric.K/sec 0.11 ±100% +122.1% 0.24 perf-stat.overall.ipc 3107 ±100% +137.3% 7372 ± 3% perf-stat.overall.path-length 2836 ±100% +284.4% 10902 ± 3% perf-stat.ps.context-switches 275.01 ±100% +257.9% 984.31 ± 2% perf-stat.ps.cpu-migrations 111986 -11.7% 98924 ± 2% slabinfo.anon_vma.active_objs 2434 -11.7% 2150 ± 2% slabinfo.anon_vma.active_slabs 111986 -11.7% 98924 ± 2% slabinfo.anon_vma.num_objs 2434 -11.7% 2150 ± 2% slabinfo.anon_vma.num_slabs 258615 -9.1% 235038 ± 2% slabinfo.anon_vma_chain.active_objs 4041 -9.1% 3672 ± 2% slabinfo.anon_vma_chain.active_slabs 258670 -9.1% 235049 ± 2% slabinfo.anon_vma_chain.num_objs 4041 -9.1% 3672 ± 2% slabinfo.anon_vma_chain.num_slabs 1500 -17.8% 1232 ± 5% slabinfo.hugetlbfs_inode_cache.active_objs 1500 -17.8% 1232 ± 5% slabinfo.hugetlbfs_inode_cache.num_objs 682.00 ± 6% +15.9% 790.50 ± 6% slabinfo.numa_policy.active_objs 682.00 ± 6% +15.9% 790.50 ± 6% slabinfo.numa_policy.num_objs 1082230 -19.6% 869890 ± 5% proc-vmstat.htlb_buddy_alloc_success 1721553 -6.8% 1604364 proc-vmstat.nr_dirty_background_threshold 3447317 -6.8% 3212652 proc-vmstat.nr_dirty_threshold 17371789 -6.8% 16197198 proc-vmstat.nr_free_pages 351.00 +3.8% 364.25 proc-vmstat.nr_inactive_file 39705 +1.4% 40269 proc-vmstat.nr_kernel_stack 12252 ± 7% -13.7% 10570 ± 3% proc-vmstat.nr_mapped 2632 +8.8% 2863 proc-vmstat.nr_page_table_pages 98880 -1.6% 97341 proc-vmstat.nr_slab_unreclaimable 351.00 +3.8% 364.25 proc-vmstat.nr_zone_inactive_file 6285 ± 50% -90.8% 577.50 ± 47% proc-vmstat.numa_hint_faults 6285 ± 50% -90.8% 577.50 ± 47% proc-vmstat.numa_hint_faults_local 2514799 -12.3% 2205532 ± 3% proc-vmstat.numa_hit 2514798 -12.3% 2205531 ± 3% proc-vmstat.numa_local 5.557e+08 -19.6% 4.469e+08 ± 5% proc-vmstat.pgalloc_normal 2510883 -12.9% 2187221 ± 3% proc-vmstat.pgfault 5.553e+08 -19.5% 4.468e+08 ± 5% proc-vmstat.pgfree 60107 ± 2% -19.7% 48269 ± 17% sched_debug.cfs_rq:/.exec_clock.avg 71511 ± 2% -23.4% 54794 ± 16% sched_debug.cfs_rq:/.exec_clock.max 44464 ± 7% -16.9% 36954 ± 18% sched_debug.cfs_rq:/.exec_clock.min 4619 ± 4% -47.8% 2412 ± 11% sched_debug.cfs_rq:/.exec_clock.stddev 614.00 ± 10% +371.9% 2897 ±124% sched_debug.cfs_rq:/.load_avg.max 49.58 ± 11% +321.1% 208.79 ±123% sched_debug.cfs_rq:/.load_avg.stddev 11526482 ± 4% -37.6% 7189313 ± 17% sched_debug.cfs_rq:/.min_vruntime.avg 13399855 ± 4% -40.0% 8039878 ± 15% sched_debug.cfs_rq:/.min_vruntime.max 8425626 ± 8% -37.2% 5294770 ± 18% sched_debug.cfs_rq:/.min_vruntime.min 788640 ± 4% -56.4% 343805 ± 11% sched_debug.cfs_rq:/.min_vruntime.stddev 0.37 ± 8% -40.0% 0.22 ± 10% sched_debug.cfs_rq:/.nr_running.avg 21.17 ± 5% +134.9% 49.72 ± 21% sched_debug.cfs_rq:/.nr_spread_over.avg 56.15 ± 13% +60.7% 90.25 ± 19% sched_debug.cfs_rq:/.nr_spread_over.max 6.95 ± 7% +289.4% 27.06 ± 23% sched_debug.cfs_rq:/.nr_spread_over.min 7.32 ± 5% +39.4% 10.20 ± 19% sched_debug.cfs_rq:/.nr_spread_over.stddev 413.38 ± 11% -27.5% 299.85 ± 14% sched_debug.cfs_rq:/.runnable_avg.avg 834.44 ± 53% -65.7% 286.48 ± 5% sched_debug.cfs_rq:/.runnable_avg.stddev 2625208 ± 11% -29.7% 1845657 ± 13% sched_debug.cfs_rq:/.spread0.avg 4483653 ± 6% -39.8% 2699533 ± 11% sched_debug.cfs_rq:/.spread0.max -475138 -94.3% -26940 sched_debug.cfs_rq:/.spread0.min 801419 ± 4% -56.8% 345835 ± 11% sched_debug.cfs_rq:/.spread0.stddev 371.84 ± 8% -22.5% 288.10 ± 14% sched_debug.cfs_rq:/.util_avg.avg 382.64 ± 4% -32.5% 258.09 ± 4% sched_debug.cfs_rq:/.util_avg.stddev 330.75 ± 10% -54.8% 149.57 ± 7% sched_debug.cfs_rq:/.util_est_enqueued.avg 358.21 ± 6% -27.6% 259.25 ± 2% sched_debug.cfs_rq:/.util_est_enqueued.stddev 4047967 ± 4% +17.6% 4760305 ± 4% sched_debug.cpu.avg_idle.avg 561715 ± 52% +116.4% 1215479 ± 7% sched_debug.cpu.avg_idle.min 731.80 ± 6% -38.6% 449.23 ± 8% sched_debug.cpu.clock.stddev 731.80 ± 6% -38.6% 449.23 ± 8% sched_debug.cpu.clock_task.stddev 4741 ± 13% -43.0% 2702 ± 18% sched_debug.cpu.curr->pid.avg 5519 ± 7% -15.8% 4650 ± 11% sched_debug.cpu.curr->pid.stddev 2119875 ± 5% +16.3% 2465873 ± 4% sched_debug.cpu.max_idle_balance_cost.avg 1107931 ± 2% +24.6% 1380330 ± 3% sched_debug.cpu.max_idle_balance_cost.min 581556 ± 5% +11.1% 646020 ± 5% sched_debug.cpu.max_idle_balance_cost.stddev 0.00 ± 4% -30.8% 0.00 ± 11% sched_debug.cpu.next_balance.stddev 0.31 ± 12% -32.6% 0.21 ± 10% sched_debug.cpu.nr_running.avg 5116 +46.1% 7477 ± 11% sched_debug.cpu.nr_switches.avg 2301 ± 5% +85.0% 4257 ± 22% sched_debug.cpu.nr_switches.min 0.09 ± 44% +300.2% 0.34 ± 17% sched_debug.cpu.nr_uninterruptible.avg 296.40 ± 3% +24.1% 367.95 ± 8% sched_debug.cpu.nr_uninterruptible.max -50.40 +38.8% -69.98 sched_debug.cpu.nr_uninterruptible.min 23.04 ± 3% +53.3% 35.32 ± 12% sched_debug.cpu.nr_uninterruptible.stddev 2343 +98.2% 4646 ± 18% sched_debug.cpu.sched_count.avg 1110 ± 2% +193.0% 3252 ± 23% sched_debug.cpu.sched_count.min 913.60 ± 2% +125.6% 2060 ± 18% sched_debug.cpu.sched_goidle.avg 356.95 +296.7% 1416 ± 25% sched_debug.cpu.sched_goidle.min 1148 +102.6% 2326 ± 18% sched_debug.cpu.ttwu_count.avg 510.15 ± 5% +80.8% 922.26 ± 21% sched_debug.cpu.ttwu_count.min 148.25 ± 6% -11.3% 131.52 ± 6% sched_debug.cpu.ttwu_local.stddev 21.96 ± 6% -10.0 11.99 ± 11% perf-profile.calltrace.cycles-pp.hugetlb_cow.hugetlb_fault.handle_mm_fault.do_page_fault.page_fault 19.05 ± 5% -8.1 10.96 ± 11% perf-profile.calltrace.cycles-pp.copy_user_huge_page.hugetlb_cow.hugetlb_fault.handle_mm_fault.do_page_fault 18.72 ± 5% -8.0 10.76 ± 11% perf-profile.calltrace.cycles-pp.copy_subpage.copy_user_huge_page.hugetlb_cow.hugetlb_fault.handle_mm_fault 17.82 ± 4% -7.6 10.17 ± 10% perf-profile.calltrace.cycles-pp.copy_page.copy_subpage.copy_user_huge_page.hugetlb_cow.hugetlb_fault 2.22 ± 44% -1.5 0.68 ± 23% perf-profile.calltrace.cycles-pp.alloc_huge_page.hugetlb_cow.hugetlb_fault.handle_mm_fault.do_page_fault 1.03 ± 22% -0.6 0.40 ±102% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.do_access 0.91 ± 23% -0.6 0.32 ±103% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.do_access 0.81 ± 19% -0.2 0.60 ± 14% perf-profile.calltrace.cycles-pp.tlb_finish_mmu.exit_mmap.mmput.do_exit.do_group_exit 0.81 ± 19% -0.2 0.60 ± 14% perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.exit_mmap.mmput.do_exit 0.80 ± 19% -0.2 0.60 ± 14% perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.tlb_finish_mmu.exit_mmap.mmput 0.79 ± 19% -0.2 0.59 ± 15% perf-profile.calltrace.cycles-pp.__free_huge_page.release_pages.tlb_flush_mmu.tlb_finish_mmu.exit_mmap 1.44 ± 22% +0.6 2.00 ± 28% perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 0.71 ± 80% +1.5 2.16 ± 33% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.osq_lock 1.03 ± 55% +1.6 2.67 ± 32% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.osq_lock.__mutex_lock 1.24 ± 57% +2.0 3.29 ± 26% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.osq_lock.__mutex_lock.hugetlb_fault 1.38 ± 55% +2.3 3.66 ± 25% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.osq_lock.__mutex_lock.hugetlb_fault.handle_mm_fault 29.76 ± 21% +20.1 49.83 ± 13% perf-profile.calltrace.cycles-pp.__mutex_lock.hugetlb_fault.handle_mm_fault.do_page_fault.page_fault 15.71 ± 27% +23.2 38.88 ± 18% perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.hugetlb_fault.handle_mm_fault.do_page_fault 21.96 ± 6% -10.0 12.00 ± 11% perf-profile.children.cycles-pp.hugetlb_cow 19.36 ± 5% -8.2 11.11 ± 11% perf-profile.children.cycles-pp.copy_user_huge_page 18.74 ± 5% -8.0 10.77 ± 11% perf-profile.children.cycles-pp.copy_subpage 18.31 ± 5% -7.8 10.54 ± 11% perf-profile.children.cycles-pp.copy_page 16.95 ± 20% -7.0 10.00 ± 53% perf-profile.children.cycles-pp.do_rw_once 2.46 ± 41% -1.7 0.80 ± 14% perf-profile.children.cycles-pp.alloc_huge_page 3.11 ± 36% -1.6 1.48 ± 4% perf-profile.children.cycles-pp._raw_spin_lock 2.64 ± 44% -1.5 1.10 ± 6% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 1.60 ± 38% -1.0 0.57 ± 14% perf-profile.children.cycles-pp.alloc_surplus_huge_page 1.81 ± 12% -0.5 1.34 ± 4% perf-profile.children.cycles-pp.do_syscall_64 1.81 ± 12% -0.5 1.35 ± 4% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 0.76 ± 28% -0.4 0.38 ± 12% perf-profile.children.cycles-pp.alloc_fresh_huge_page 1.21 ± 10% -0.4 0.85 ± 11% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 0.40 ± 52% -0.3 0.09 ± 20% perf-profile.children.cycles-pp.prep_new_huge_page 0.81 ± 19% -0.2 0.60 ± 14% perf-profile.children.cycles-pp.tlb_finish_mmu 0.81 ± 19% -0.2 0.60 ± 14% perf-profile.children.cycles-pp.tlb_flush_mmu 0.80 ± 18% -0.2 0.60 ± 15% perf-profile.children.cycles-pp.release_pages 0.79 ± 19% -0.2 0.59 ± 15% perf-profile.children.cycles-pp.__free_huge_page 0.44 ± 13% -0.2 0.26 ± 14% perf-profile.children.cycles-pp.___might_sleep 0.20 ± 14% -0.1 0.06 ± 6% perf-profile.children.cycles-pp.update_and_free_page 0.32 ± 12% -0.1 0.21 ± 6% perf-profile.children.cycles-pp.ksys_read 0.29 ± 13% -0.1 0.18 ± 10% perf-profile.children.cycles-pp.read 0.31 ± 11% -0.1 0.20 ± 8% perf-profile.children.cycles-pp.vfs_read 0.18 ± 17% -0.1 0.09 ± 8% perf-profile.children.cycles-pp._cond_resched 0.16 ± 9% -0.1 0.09 ± 17% perf-profile.children.cycles-pp.__libc_start_main 0.16 ± 11% -0.1 0.08 ± 17% perf-profile.children.cycles-pp.main 0.19 ± 14% -0.1 0.12 ± 17% perf-profile.children.cycles-pp.x86_pmu_disable 0.26 ± 8% -0.1 0.19 ± 6% perf-profile.children.cycles-pp.ksys_write 0.26 ± 9% -0.1 0.19 ± 8% perf-profile.children.cycles-pp.vfs_write 0.15 ± 12% -0.1 0.09 ± 20% perf-profile.children.cycles-pp.new_sync_read 0.15 ± 11% -0.1 0.09 ± 17% perf-profile.children.cycles-pp.pipe_read 0.24 ± 9% -0.1 0.18 ± 7% perf-profile.children.cycles-pp.new_sync_write 0.08 ± 6% -0.1 0.03 ±100% perf-profile.children.cycles-pp.__perf_sw_event 0.11 ± 14% -0.0 0.06 ± 14% perf-profile.children.cycles-pp.__wake_up_common 0.12 ± 3% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.schedule_timeout 0.12 ± 13% -0.0 0.07 ± 17% perf-profile.children.cycles-pp.__wake_up_common_lock 0.10 ± 15% -0.0 0.06 ± 15% perf-profile.children.cycles-pp.autoremove_wake_function 0.11 ± 29% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode 0.26 ± 6% -0.0 0.22 ± 5% perf-profile.children.cycles-pp.tick_program_event 0.09 ± 31% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.prepare_exit_to_usermode 0.09 ± 9% -0.0 0.06 ± 13% perf-profile.children.cycles-pp.rcu_gp_kthread 0.08 ± 10% -0.0 0.06 ± 14% perf-profile.children.cycles-pp.cmd_stat 0.01 ±173% +0.0 0.06 ± 11% perf-profile.children.cycles-pp.select_task_rq_fair 0.11 ± 11% +0.0 0.16 ± 9% perf-profile.children.cycles-pp.finish_task_switch 0.00 +0.1 0.06 ± 14% perf-profile.children.cycles-pp.rwsem_wake 0.00 +0.1 0.06 ± 7% perf-profile.children.cycles-pp.arch_stack_walk 0.00 +0.1 0.06 ± 13% perf-profile.children.cycles-pp.dequeue_task_fair 0.00 +0.1 0.07 ± 13% perf-profile.children.cycles-pp.stack_trace_save_tsk 0.13 ± 6% +0.1 0.21 ± 10% perf-profile.children.cycles-pp.schedule_idle 0.06 ± 11% +0.1 0.14 ± 10% perf-profile.children.cycles-pp.enqueue_entity 0.03 ±100% +0.1 0.11 ± 7% perf-profile.children.cycles-pp.update_load_avg 0.00 +0.1 0.09 ± 12% perf-profile.children.cycles-pp.__account_scheduler_latency 0.07 ± 17% +0.1 0.16 ± 11% perf-profile.children.cycles-pp.activate_task 0.07 ± 17% +0.1 0.16 ± 11% perf-profile.children.cycles-pp.ttwu_do_activate 0.07 ± 17% +0.1 0.16 ± 11% perf-profile.children.cycles-pp.enqueue_task_fair 0.00 +0.1 0.09 ± 8% perf-profile.children.cycles-pp.wake_up_q 0.14 ± 24% +0.1 0.27 ± 13% perf-profile.children.cycles-pp.schedule_preempt_disabled 0.00 +0.1 0.14 ± 15% perf-profile.children.cycles-pp.sched_ttwu_pending 1.17 ± 13% +0.4 1.56 ± 8% perf-profile.children.cycles-pp.load_balance 0.99 ± 11% +0.4 1.41 ± 9% perf-profile.children.cycles-pp.find_busiest_group 0.96 ± 11% +0.4 1.38 ± 9% perf-profile.children.cycles-pp.update_sd_lb_stats 0.66 ± 9% +0.5 1.20 ± 11% perf-profile.children.cycles-pp.newidle_balance 0.53 ± 7% +0.6 1.11 ± 12% perf-profile.children.cycles-pp.pick_next_task_fair 0.71 ± 8% +0.6 1.35 ± 10% perf-profile.children.cycles-pp.schedule 0.00 +0.7 0.68 ± 17% perf-profile.children.cycles-pp.rwsem_down_read_slowpath 0.86 ± 7% +0.7 1.58 ± 9% perf-profile.children.cycles-pp.__schedule 52.14 ± 14% +10.7 62.86 ± 9% perf-profile.children.cycles-pp.page_fault 52.09 ± 14% +10.7 62.83 ± 9% perf-profile.children.cycles-pp.do_page_fault 51.96 ± 14% +10.8 62.74 ± 9% perf-profile.children.cycles-pp.handle_mm_fault 51.86 ± 14% +10.8 62.67 ± 9% perf-profile.children.cycles-pp.hugetlb_fault 29.77 ± 21% +20.1 49.84 ± 13% perf-profile.children.cycles-pp.__mutex_lock 15.83 ± 27% +23.4 39.18 ± 18% perf-profile.children.cycles-pp.osq_lock 16.53 ± 4% -7.1 9.48 ± 9% perf-profile.self.cycles-pp.copy_page 15.27 ± 22% -6.4 8.90 ± 53% perf-profile.self.cycles-pp.do_rw_once 2.38 ± 42% -1.4 1.02 ± 7% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 0.70 ± 9% -0.2 0.48 ± 14% perf-profile.self.cycles-pp.perf_mux_hrtimer_handler 0.38 ± 2% -0.2 0.21 ± 11% perf-profile.self.cycles-pp.copy_subpage 0.33 ± 4% -0.2 0.17 ± 11% perf-profile.self.cycles-pp.copy_user_huge_page 0.18 ± 18% -0.1 0.05 ± 8% perf-profile.self.cycles-pp.update_and_free_page 0.19 ± 14% -0.1 0.11 ± 18% perf-profile.self.cycles-pp.x86_pmu_disable 0.19 ± 6% -0.1 0.12 ± 10% perf-profile.self.cycles-pp.___might_sleep 0.35 ± 14% -0.1 0.28 ± 8% perf-profile.self.cycles-pp.run_timer_softirq 0.18 ± 10% -0.1 0.12 ± 6% perf-profile.self.cycles-pp.get_page_from_freelist 0.26 ± 6% -0.0 0.22 ± 5% perf-profile.self.cycles-pp.tick_program_event 0.07 -0.0 0.06 ± 7% perf-profile.self.cycles-pp.try_to_wake_up 0.00 +0.1 0.06 ± 9% perf-profile.self.cycles-pp.update_load_avg 0.73 ± 12% +0.3 1.06 ± 8% perf-profile.self.cycles-pp.update_sd_lb_stats 14.37 ± 25% +21.2 35.60 ± 18% perf-profile.self.cycles-pp.osq_lock 22773 ± 3% +11.6% 25420 ± 2% softirqs.CPU0.SCHED 15462 ± 6% +24.5% 19248 ± 5% softirqs.CPU1.SCHED 8331 ± 11% +27.1% 10590 ± 11% softirqs.CPU10.RCU 14783 ± 4% +16.0% 17145 ± 4% softirqs.CPU10.SCHED 14236 ± 3% +15.6% 16462 ± 2% softirqs.CPU100.SCHED 14117 ± 3% +18.2% 16683 ± 3% softirqs.CPU101.SCHED 13918 ± 4% +18.4% 16472 ± 5% softirqs.CPU102.SCHED 14183 ± 6% +18.0% 16742 ± 3% softirqs.CPU103.SCHED 13795 ± 5% +19.8% 16528 softirqs.CPU104.SCHED 14326 ± 6% +17.7% 16859 softirqs.CPU105.SCHED 13842 ± 5% +19.4% 16530 ± 2% softirqs.CPU106.SCHED 14155 ± 4% +20.0% 16983 ± 2% softirqs.CPU107.SCHED 13907 ± 3% +18.6% 16492 ± 2% softirqs.CPU108.SCHED 14411 ± 7% +16.1% 16726 ± 2% softirqs.CPU109.SCHED 14354 +19.0% 17085 ± 2% softirqs.CPU11.SCHED 13689 ± 6% +21.8% 16667 ± 3% softirqs.CPU110.SCHED 13461 ± 6% +25.6% 16907 ± 3% softirqs.CPU111.SCHED 13738 ± 9% +23.2% 16925 softirqs.CPU112.SCHED 14135 ± 3% +20.5% 17032 softirqs.CPU113.SCHED 14477 ± 4% +20.8% 17491 ± 4% softirqs.CPU114.SCHED 123003 ± 2% -7.8% 113426 ± 2% softirqs.CPU114.TIMER 14014 ± 4% +23.4% 17299 softirqs.CPU115.SCHED 13528 ± 6% +24.4% 16824 ± 2% softirqs.CPU116.SCHED 13530 ± 6% +24.5% 16840 ± 2% softirqs.CPU117.SCHED 14263 ± 3% +16.1% 16557 ± 2% softirqs.CPU119.SCHED 7902 ± 2% +22.1% 9648 ± 6% softirqs.CPU12.RCU 14332 ± 3% +19.5% 17130 ± 4% softirqs.CPU12.SCHED 13476 ± 5% +26.5% 17047 softirqs.CPU120.SCHED 14240 ± 6% +20.7% 17183 softirqs.CPU121.SCHED 13161 ± 5% +27.0% 16708 ± 2% softirqs.CPU122.SCHED 13037 ± 7% +28.8% 16786 softirqs.CPU123.SCHED 14229 ± 5% +16.5% 16577 ± 3% softirqs.CPU124.SCHED 14427 ± 4% +15.4% 16652 softirqs.CPU125.SCHED 14426 ± 4% +16.3% 16771 softirqs.CPU126.SCHED 6680 +39.3% 9305 ± 22% softirqs.CPU127.RCU 13659 ± 3% +23.7% 16890 ± 3% softirqs.CPU128.SCHED 14396 ± 5% +17.9% 16979 ± 3% softirqs.CPU129.SCHED 14183 ± 7% +20.2% 17048 ± 2% softirqs.CPU13.SCHED 6700 ± 2% +35.8% 9097 ± 17% softirqs.CPU130.RCU 13949 ± 4% +22.2% 17051 ± 4% softirqs.CPU130.SCHED 14271 ± 4% +21.8% 17379 ± 2% softirqs.CPU131.SCHED 129315 ± 3% -7.0% 120290 softirqs.CPU131.TIMER 14135 ± 2% +16.0% 16396 ± 2% softirqs.CPU132.SCHED 13567 ± 4% +22.5% 16617 ± 3% softirqs.CPU133.SCHED 13912 ± 5% +20.1% 16712 ± 2% softirqs.CPU134.SCHED 14309 ± 5% +18.9% 17016 ± 2% softirqs.CPU135.SCHED 13637 ± 4% +20.3% 16402 ± 5% softirqs.CPU136.SCHED 13843 ± 4% +19.3% 16518 ± 2% softirqs.CPU137.SCHED 14049 ± 5% +19.2% 16750 ± 2% softirqs.CPU138.SCHED 14244 ± 7% +18.1% 16816 softirqs.CPU139.SCHED 14139 +22.1% 17271 ± 2% softirqs.CPU14.SCHED 13460 ± 8% +22.9% 16546 softirqs.CPU140.SCHED 13645 ± 6% +21.7% 16601 ± 2% softirqs.CPU141.SCHED 13278 ± 6% +25.5% 16661 ± 3% softirqs.CPU142.SCHED 12487 ± 5% +29.2% 16136 ± 2% softirqs.CPU143.SCHED 13440 ± 9% +22.8% 16502 softirqs.CPU144.SCHED 126064 ± 2% -8.1% 115836 softirqs.CPU144.TIMER 13995 ± 6% +21.2% 16966 ± 3% softirqs.CPU145.SCHED 13834 ± 3% +14.8% 15883 ± 6% softirqs.CPU146.SCHED 14297 ± 9% +17.1% 16735 softirqs.CPU147.SCHED 14500 ± 3% +15.2% 16711 softirqs.CPU148.SCHED 14419 ± 6% +17.5% 16945 softirqs.CPU149.SCHED 8018 ± 6% +24.4% 9977 ± 7% softirqs.CPU15.RCU 13832 ± 5% +28.2% 17737 ± 5% softirqs.CPU15.SCHED 14538 ± 5% +15.0% 16723 ± 2% softirqs.CPU150.SCHED 15092 +9.5% 16526 ± 4% softirqs.CPU152.SCHED 14533 ± 2% +15.6% 16804 softirqs.CPU153.SCHED 14415 ± 2% +14.8% 16554 ± 2% softirqs.CPU154.SCHED 14583 ± 2% +15.4% 16833 ± 2% softirqs.CPU155.SCHED 14851 ± 2% +11.8% 16598 ± 2% softirqs.CPU156.SCHED 6747 +35.2% 9124 ± 19% softirqs.CPU158.RCU 13839 ± 4% +25.9% 17429 ± 4% softirqs.CPU158.SCHED 15020 ± 5% +11.8% 16785 ± 2% softirqs.CPU159.SCHED 13468 ± 5% +25.8% 16945 ± 2% softirqs.CPU16.SCHED 128305 ± 3% -8.1% 117855 softirqs.CPU160.TIMER 15111 ± 2% +13.7% 17184 softirqs.CPU162.SCHED 14788 ± 2% +12.8% 16685 ± 3% softirqs.CPU163.SCHED 13903 ± 5% +21.5% 16889 softirqs.CPU164.SCHED 14547 ± 5% +13.5% 16510 ± 2% softirqs.CPU165.SCHED 14440 ± 4% +13.3% 16354 ± 3% softirqs.CPU166.SCHED 14450 ± 5% +14.1% 16494 ± 2% softirqs.CPU167.SCHED 131454 ± 3% -6.9% 122396 ± 2% softirqs.CPU167.TIMER 13683 ± 6% +23.0% 16824 ± 3% softirqs.CPU168.SCHED 14468 ± 5% +18.1% 17085 ± 3% softirqs.CPU169.SCHED 7703 +22.5% 9437 ± 4% softirqs.CPU17.RCU 14392 +19.8% 17237 softirqs.CPU17.SCHED 14570 ± 6% +16.9% 17038 ± 3% softirqs.CPU170.SCHED 14172 ± 5% +19.3% 16905 softirqs.CPU171.SCHED 14176 ± 3% +17.9% 16709 softirqs.CPU172.SCHED 127159 ± 2% -8.4% 116497 ± 2% softirqs.CPU172.TIMER 15034 ± 2% +11.1% 16698 ± 2% softirqs.CPU174.SCHED 14923 ± 8% +17.0% 17458 ± 5% softirqs.CPU175.SCHED 14164 ± 5% +18.8% 16833 ± 2% softirqs.CPU176.SCHED 127027 ± 2% -8.4% 116378 ± 2% softirqs.CPU176.TIMER 14359 ± 3% +16.9% 16788 ± 4% softirqs.CPU177.SCHED 14466 ± 3% +18.7% 17169 softirqs.CPU178.SCHED 14632 ± 2% +15.1% 16837 softirqs.CPU179.SCHED 123562 ± 3% -8.2% 113392 ± 2% softirqs.CPU179.TIMER 7677 +27.6% 9794 ± 2% softirqs.CPU18.RCU 14177 ± 2% +21.6% 17233 softirqs.CPU18.SCHED 14429 ± 7% +15.8% 16707 softirqs.CPU180.SCHED 15194 ± 5% +12.4% 17084 softirqs.CPU181.SCHED 13917 ± 3% +20.2% 16723 ± 3% softirqs.CPU182.SCHED 14587 ± 2% +15.1% 16786 ± 3% softirqs.CPU183.SCHED 14266 ± 3% +18.0% 16827 ± 2% softirqs.CPU184.SCHED 14661 ± 3% +16.3% 17053 softirqs.CPU185.SCHED 14222 ± 5% +21.0% 17202 ± 3% softirqs.CPU186.SCHED 15059 ± 2% +13.0% 17022 ± 3% softirqs.CPU187.SCHED 125388 ± 2% -7.6% 115887 ± 2% softirqs.CPU187.TIMER 13435 ± 4% +25.0% 16801 ± 3% softirqs.CPU188.SCHED 13434 ± 2% +28.8% 17297 ± 2% softirqs.CPU189.SCHED 7719 +27.0% 9806 ± 9% softirqs.CPU19.RCU 14075 ± 2% +20.7% 16986 softirqs.CPU19.SCHED 13735 ± 3% +20.4% 16536 ± 2% softirqs.CPU190.SCHED 13772 ± 3% +19.8% 16494 softirqs.CPU192.SCHED 13911 ± 2% +17.4% 16327 ± 2% softirqs.CPU193.SCHED 6417 +34.3% 8616 ± 17% softirqs.CPU194.RCU 14008 +17.6% 16468 ± 8% softirqs.CPU194.SCHED 14094 ± 4% +19.9% 16905 softirqs.CPU195.SCHED 14387 ± 7% +16.1% 16698 ± 5% softirqs.CPU196.SCHED 13704 ± 3% +23.7% 16952 softirqs.CPU198.SCHED 8214 ± 10% +23.7% 10157 ± 7% softirqs.CPU2.RCU 14431 +17.4% 16946 ± 4% softirqs.CPU2.SCHED 7770 +30.4% 10135 ± 8% softirqs.CPU20.RCU 13996 ± 2% +23.7% 17320 softirqs.CPU20.SCHED 14611 ± 5% +12.7% 16463 ± 3% softirqs.CPU200.SCHED 14897 ± 3% +15.5% 17199 softirqs.CPU201.SCHED 14105 ± 6% +21.7% 17167 ± 2% softirqs.CPU202.SCHED 127698 ± 3% -7.1% 118685 softirqs.CPU202.TIMER 14369 ± 6% +18.6% 17048 ± 2% softirqs.CPU203.SCHED 126668 ± 3% -8.8% 115539 softirqs.CPU203.TIMER 14461 ± 4% +13.1% 16356 ± 3% softirqs.CPU205.SCHED 13139 ± 10% +26.6% 16634 ± 3% softirqs.CPU206.SCHED 14808 ± 2% +11.8% 16550 ± 2% softirqs.CPU207.SCHED 13744 ± 5% +21.0% 16624 ± 4% softirqs.CPU208.SCHED 14039 ± 5% +18.7% 16661 ± 2% softirqs.CPU209.SCHED 127488 ± 2% -8.2% 116998 softirqs.CPU209.TIMER 7755 ± 2% +30.5% 10122 ± 12% softirqs.CPU21.RCU 13731 ± 5% +25.2% 17185 ± 2% softirqs.CPU21.SCHED 14755 ± 6% +16.1% 17123 ± 4% softirqs.CPU211.SCHED 13920 ± 3% +18.5% 16501 ± 2% softirqs.CPU212.SCHED 14435 ± 4% +16.1% 16765 ± 3% softirqs.CPU213.SCHED 13857 ± 6% +21.6% 16844 ± 2% softirqs.CPU214.SCHED 13417 ± 8% +16.5% 15635 ± 2% softirqs.CPU215.SCHED 127636 ± 3% -7.4% 118224 softirqs.CPU215.TIMER 13715 ± 3% +21.7% 16686 ± 3% softirqs.CPU216.SCHED 126421 ± 2% -7.8% 116544 softirqs.CPU216.TIMER 13885 ± 5% +18.2% 16416 softirqs.CPU217.SCHED 14003 ± 8% +20.5% 16875 softirqs.CPU218.SCHED 13891 ± 4% +23.9% 17210 ± 4% softirqs.CPU22.SCHED 14901 ± 2% +12.0% 16694 ± 2% softirqs.CPU220.SCHED 14623 ± 6% +17.3% 17153 softirqs.CPU221.SCHED 14613 ± 8% +13.9% 16643 ± 2% softirqs.CPU224.SCHED 14745 ± 4% +12.9% 16643 softirqs.CPU225.SCHED 126232 -8.3% 115717 softirqs.CPU225.TIMER 14290 ± 2% +17.2% 16753 softirqs.CPU226.SCHED 15229 ± 3% +10.3% 16800 ± 3% softirqs.CPU228.SCHED 14282 ± 6% +15.4% 16479 ± 3% softirqs.CPU23.SCHED 13785 ± 9% +24.2% 17126 ± 4% softirqs.CPU230.SCHED 126791 ± 4% -9.4% 114889 ± 3% softirqs.CPU231.TIMER 14749 ± 3% +15.0% 16958 ± 3% softirqs.CPU232.SCHED 15099 ± 5% +11.9% 16893 ± 2% softirqs.CPU233.SCHED 14825 ± 6% +12.9% 16738 softirqs.CPU234.SCHED 14272 ± 5% +18.7% 16945 ± 2% softirqs.CPU236.SCHED 15187 ± 2% +10.9% 16846 softirqs.CPU237.SCHED 15267 ± 7% +11.1% 16955 ± 3% softirqs.CPU239.SCHED 14156 ± 4% +16.3% 16466 ± 3% softirqs.CPU24.SCHED 14671 ± 3% +13.9% 16711 ± 3% softirqs.CPU240.SCHED 127318 ± 2% -7.6% 117651 ± 2% softirqs.CPU240.TIMER 14218 ± 5% +20.2% 17097 softirqs.CPU241.SCHED 14554 ± 3% +15.2% 16771 softirqs.CPU242.SCHED 14003 +19.9% 16785 softirqs.CPU243.SCHED 13761 ± 6% +21.5% 16724 ± 2% softirqs.CPU244.SCHED 14418 ± 6% +16.4% 16781 softirqs.CPU245.SCHED 14760 ± 7% +14.8% 16940 ± 4% softirqs.CPU246.SCHED 14891 ± 3% +12.3% 16722 ± 4% softirqs.CPU249.SCHED 13823 ± 4% +24.1% 17148 softirqs.CPU25.SCHED 14201 ± 4% +18.1% 16774 ± 3% softirqs.CPU250.SCHED 14620 ± 10% +18.5% 17331 softirqs.CPU251.SCHED 14877 ± 3% +11.3% 16564 ± 2% softirqs.CPU252.SCHED 14517 ± 6% +16.3% 16877 ± 3% softirqs.CPU254.SCHED 14284 ± 7% +18.0% 16858 ± 4% softirqs.CPU255.SCHED 121538 ± 3% -7.5% 112473 softirqs.CPU255.TIMER 14582 +14.8% 16736 softirqs.CPU256.SCHED 14057 ± 3% +22.8% 17259 softirqs.CPU257.SCHED 15076 ± 3% +13.4% 17097 ± 3% softirqs.CPU258.SCHED 14356 ± 3% +20.2% 17251 ± 3% softirqs.CPU259.SCHED 13998 ± 3% +20.1% 16806 ± 2% softirqs.CPU26.SCHED 13899 ± 4% +19.0% 16534 ± 5% softirqs.CPU260.SCHED 128277 ± 4% -7.4% 118758 softirqs.CPU260.TIMER 14288 ± 5% +20.7% 17251 softirqs.CPU261.SCHED 14456 ± 5% +15.1% 16644 softirqs.CPU262.SCHED 14114 ± 2% +17.8% 16624 ± 4% softirqs.CPU263.SCHED 13328 ± 4% +25.0% 16657 softirqs.CPU264.SCHED 13328 ± 4% +25.1% 16676 ± 4% softirqs.CPU266.SCHED 14367 ± 3% +19.5% 17165 ± 2% softirqs.CPU267.SCHED 122663 ± 3% -7.6% 113293 ± 2% softirqs.CPU268.TIMER 13527 ± 6% +25.2% 16940 softirqs.CPU27.SCHED 14341 ± 5% +19.8% 17187 softirqs.CPU270.SCHED 14345 ± 5% +14.3% 16403 ± 5% softirqs.CPU271.SCHED 14990 ± 3% +14.4% 17152 ± 4% softirqs.CPU272.SCHED 15007 ± 2% +13.4% 17012 softirqs.CPU273.SCHED 14668 ± 8% +17.0% 17157 ± 4% softirqs.CPU274.SCHED 14287 ± 6% +17.2% 16749 ± 4% softirqs.CPU277.SCHED 6301 +41.8% 8938 ± 16% softirqs.CPU278.RCU 13980 ± 10% +24.6% 17424 ± 2% softirqs.CPU278.SCHED 14763 ± 7% +16.0% 17119 softirqs.CPU279.SCHED 13682 ± 5% +23.9% 16950 softirqs.CPU28.SCHED 14204 ± 3% +17.0% 16617 ± 3% softirqs.CPU280.SCHED 14202 ± 3% +14.4% 16240 ± 2% softirqs.CPU281.SCHED 14629 ± 7% +16.3% 17013 ± 3% softirqs.CPU283.SCHED 13692 ± 6% +22.7% 16795 softirqs.CPU284.SCHED 14194 ± 7% +17.5% 16679 ± 3% softirqs.CPU285.SCHED 123721 ± 3% -8.1% 113741 ± 3% softirqs.CPU285.TIMER 14757 ± 7% +15.5% 17050 softirqs.CPU286.SCHED 13560 ± 6% +17.2% 15894 softirqs.CPU287.SCHED 14201 ± 4% +19.9% 17020 ± 2% softirqs.CPU29.SCHED 14345 ± 3% +20.5% 17292 softirqs.CPU3.SCHED 13910 ± 4% +20.9% 16814 ± 4% softirqs.CPU30.SCHED 7587 +41.8% 10759 ± 23% softirqs.CPU31.RCU 13401 ± 4% +26.6% 16967 ± 2% softirqs.CPU31.SCHED 13593 +24.6% 16937 softirqs.CPU32.SCHED 13175 ± 2% +29.0% 16989 softirqs.CPU33.SCHED 13491 ± 5% +25.8% 16974 ± 3% softirqs.CPU34.SCHED 13520 ± 7% +25.9% 17018 ± 2% softirqs.CPU35.SCHED 13723 ± 6% +22.1% 16763 ± 2% softirqs.CPU36.SCHED 13449 ± 3% +27.1% 17091 softirqs.CPU37.SCHED 7405 +40.0% 10368 ± 17% softirqs.CPU38.RCU 14299 ± 6% +22.4% 17509 ± 3% softirqs.CPU38.SCHED 14269 ± 3% +18.6% 16918 softirqs.CPU39.SCHED 7977 ± 2% +23.6% 9862 ± 5% softirqs.CPU4.RCU 14055 ± 2% +23.4% 17349 ± 3% softirqs.CPU4.SCHED 14257 ± 2% +19.4% 17030 ± 2% softirqs.CPU40.SCHED 13774 ± 2% +24.1% 17090 softirqs.CPU41.SCHED 14425 ± 8% +17.7% 16976 softirqs.CPU42.SCHED 14314 ± 2% +21.2% 17345 ± 2% softirqs.CPU43.SCHED 13274 ± 4% +27.7% 16954 softirqs.CPU44.SCHED 135612 ± 10% -15.7% 114264 ± 3% softirqs.CPU44.TIMER 13919 ± 2% +20.9% 16829 softirqs.CPU45.SCHED 13156 +27.9% 16834 ± 2% softirqs.CPU46.SCHED 13617 ± 2% +22.0% 16614 softirqs.CPU47.SCHED 13565 ± 3% +25.7% 17054 ± 2% softirqs.CPU48.SCHED 13791 ± 4% +21.6% 16769 softirqs.CPU49.SCHED 14260 ± 3% +20.1% 17126 ± 4% softirqs.CPU5.SCHED 13510 ± 7% +25.3% 16934 softirqs.CPU50.SCHED 12790 ± 10% +30.3% 16669 softirqs.CPU51.SCHED 13989 +19.9% 16774 ± 3% softirqs.CPU52.SCHED 14279 ± 3% +17.1% 16722 ± 2% softirqs.CPU53.SCHED 7263 +36.9% 9940 ± 21% softirqs.CPU54.RCU 14088 ± 2% +20.1% 16918 softirqs.CPU54.SCHED 13508 ± 6% +23.1% 16629 softirqs.CPU55.SCHED 130302 ± 6% -9.4% 118014 ± 7% softirqs.CPU55.TIMER 13313 ± 3% +27.9% 17030 ± 2% softirqs.CPU56.SCHED 7360 ± 2% +33.2% 9803 ± 12% softirqs.CPU57.RCU 13818 ± 2% +21.3% 16768 ± 2% softirqs.CPU57.SCHED 7296 +27.1% 9274 ± 7% softirqs.CPU58.RCU 13922 ± 3% +23.4% 17181 ± 2% softirqs.CPU58.SCHED 7271 +30.3% 9476 ± 11% softirqs.CPU59.RCU 14169 ± 5% +19.7% 16956 ± 2% softirqs.CPU59.SCHED 8176 ± 7% +16.5% 9524 ± 3% softirqs.CPU6.RCU 13689 ± 3% +25.2% 17132 ± 5% softirqs.CPU6.SCHED 13645 ± 2% +23.3% 16825 softirqs.CPU60.SCHED 13441 ± 6% +24.9% 16786 ± 2% softirqs.CPU61.SCHED 13497 ± 5% +25.7% 16959 softirqs.CPU62.SCHED 13682 ± 4% +25.5% 17170 softirqs.CPU63.SCHED 13656 ± 6% +22.3% 16695 ± 3% softirqs.CPU64.SCHED 13704 ± 7% +22.7% 16819 softirqs.CPU65.SCHED 13683 ± 3% +23.0% 16826 softirqs.CPU66.SCHED 14076 ± 2% +19.3% 16797 ± 2% softirqs.CPU67.SCHED 13713 ± 4% +18.2% 16215 ± 2% softirqs.CPU68.SCHED 13396 ± 3% +25.1% 16754 ± 2% softirqs.CPU69.SCHED 8042 ± 3% +21.0% 9730 ± 7% softirqs.CPU7.RCU 13604 ± 3% +27.3% 17320 ± 5% softirqs.CPU7.SCHED 13023 ± 2% +29.2% 16828 ± 2% softirqs.CPU70.SCHED 12307 ± 3% +32.1% 16255 softirqs.CPU71.SCHED 7727 ± 22% +37.3% 10612 ± 22% softirqs.CPU72.RCU 13320 ± 3% +28.0% 17050 ± 2% softirqs.CPU72.SCHED 13838 ± 3% +22.0% 16883 ± 3% softirqs.CPU73.SCHED 14604 ± 2% +14.4% 16707 ± 4% softirqs.CPU74.SCHED 14184 ± 4% +19.1% 16892 ± 2% softirqs.CPU75.SCHED 14138 ± 4% +19.0% 16830 softirqs.CPU76.SCHED 7098 +28.1% 9090 ± 12% softirqs.CPU77.RCU 13082 ± 4% +28.0% 16741 softirqs.CPU77.SCHED 126355 ± 4% -10.8% 112765 ± 2% softirqs.CPU78.TIMER 14718 ± 5% +12.6% 16576 ± 3% softirqs.CPU79.SCHED 7959 ± 2% +27.1% 10119 ± 10% softirqs.CPU8.RCU 13775 ± 4% +25.1% 17232 ± 5% softirqs.CPU8.SCHED 14397 +13.6% 16355 ± 2% softirqs.CPU80.SCHED 13999 ± 6% +20.7% 16896 ± 2% softirqs.CPU81.SCHED 14129 ± 7% +18.2% 16706 softirqs.CPU82.SCHED 13931 +17.6% 16385 ± 3% softirqs.CPU83.SCHED 14455 ± 5% +16.1% 16776 ± 2% softirqs.CPU84.SCHED 14426 ± 5% +16.4% 16799 softirqs.CPU86.SCHED 7049 +28.2% 9036 ± 10% softirqs.CPU88.RCU 13901 ± 3% +20.1% 16697 ± 2% softirqs.CPU88.SCHED 127258 ± 2% -7.3% 118017 softirqs.CPU88.TIMER 14353 ± 5% +17.4% 16855 ± 2% softirqs.CPU89.SCHED 14823 ± 4% +14.7% 17004 softirqs.CPU9.SCHED 14278 ± 4% +18.4% 16908 softirqs.CPU90.SCHED 14225 ± 2% +18.8% 16894 ± 2% softirqs.CPU91.SCHED 7091 +26.7% 8987 ± 10% softirqs.CPU92.RCU 13726 ± 3% +24.0% 17014 ± 3% softirqs.CPU92.SCHED 14051 ± 5% +20.6% 16939 softirqs.CPU93.SCHED 14123 ± 4% +17.9% 16649 ± 3% softirqs.CPU94.SCHED 130109 ± 5% -8.2% 119400 softirqs.CPU95.TIMER 13638 ± 4% +23.5% 16847 ± 2% softirqs.CPU96.SCHED 13134 ± 2% +29.4% 17001 ± 3% softirqs.CPU97.SCHED 13357 ± 3% +25.7% 16791 ± 2% softirqs.CPU98.SCHED 14112 ± 6% +20.0% 16938 softirqs.CPU99.SCHED 1991578 +21.8% 2426642 ± 3% softirqs.RCU 4100803 +18.4% 4855612 softirqs.SCHED 454.00 ± 27% +675.8% 3522 ±139% interrupts.30:IR-PCI-MSI.2097153-edge.eth0-TxRx-0 384622 -1.6% 378485 interrupts.CAL:Function_call_interrupts 930.25 ± 8% -46.7% 496.00 ± 10% interrupts.CPU0.TLB:TLB_shootdowns 758.50 ± 9% -44.8% 418.75 ± 11% interrupts.CPU1.TLB:TLB_shootdowns 454.00 ± 27% +675.8% 3522 ±139% interrupts.CPU10.30:IR-PCI-MSI.2097153-edge.eth0-TxRx-0 886.75 ± 5% -39.4% 537.00 ± 3% interrupts.CPU10.TLB:TLB_shootdowns 169.50 ± 11% +67.0% 283.00 ± 9% interrupts.CPU100.TLB:TLB_shootdowns 178.50 ± 20% +44.0% 257.00 ± 9% interrupts.CPU101.TLB:TLB_shootdowns 169.00 ± 23% +62.7% 275.00 ± 20% interrupts.CPU104.TLB:TLB_shootdowns 1463 ± 28% +88.1% 2753 ± 22% interrupts.CPU105.NMI:Non-maskable_interrupts 1463 ± 28% +88.1% 2753 ± 22% interrupts.CPU105.PMI:Performance_monitoring_interrupts 158.75 ± 21% +73.4% 275.25 ± 4% interrupts.CPU106.TLB:TLB_shootdowns 157.50 ± 24% +67.3% 263.50 ± 13% interrupts.CPU109.TLB:TLB_shootdowns 2884 ± 27% -43.7% 1625 ± 6% interrupts.CPU11.NMI:Non-maskable_interrupts 2884 ± 27% -43.7% 1625 ± 6% interrupts.CPU11.PMI:Performance_monitoring_interrupts 713.50 ± 5% -38.8% 436.50 ± 13% interrupts.CPU11.TLB:TLB_shootdowns 146.00 ± 38% +79.3% 261.75 ± 14% interrupts.CPU110.TLB:TLB_shootdowns 162.00 ± 14% +82.6% 295.75 ± 12% interrupts.CPU112.TLB:TLB_shootdowns 445.75 ± 52% +170.1% 1204 ± 91% interrupts.CPU114.RES:Rescheduling_interrupts 160.25 ± 37% +70.8% 273.75 ± 14% interrupts.CPU114.TLB:TLB_shootdowns 1417 ± 17% +66.0% 2352 ± 34% interrupts.CPU115.NMI:Non-maskable_interrupts 1417 ± 17% +66.0% 2352 ± 34% interrupts.CPU115.PMI:Performance_monitoring_interrupts 380.75 ± 9% +37.9% 525.00 ± 15% interrupts.CPU116.RES:Rescheduling_interrupts 183.00 ± 9% +48.2% 271.25 ± 14% interrupts.CPU116.TLB:TLB_shootdowns 170.50 ± 15% +74.5% 297.50 ± 5% interrupts.CPU118.TLB:TLB_shootdowns 454.25 ± 38% +251.3% 1596 ±103% interrupts.CPU12.RES:Rescheduling_interrupts 847.00 ± 12% -36.9% 534.25 ± 10% interrupts.CPU12.TLB:TLB_shootdowns 173.25 ± 20% +65.7% 287.00 ± 13% interrupts.CPU122.TLB:TLB_shootdowns 420.50 ± 32% +80.5% 759.00 ± 37% interrupts.CPU123.RES:Rescheduling_interrupts 247.75 ± 22% +75.9% 435.75 ± 14% interrupts.CPU124.RES:Rescheduling_interrupts 181.75 ± 33% +58.7% 288.50 ± 15% interrupts.CPU126.TLB:TLB_shootdowns 364.75 ± 21% +39.1% 507.50 ± 11% interrupts.CPU128.RES:Rescheduling_interrupts 337.75 ± 24% +248.5% 1177 ± 74% interrupts.CPU13.RES:Rescheduling_interrupts 783.25 ± 14% -47.8% 409.00 ± 3% interrupts.CPU13.TLB:TLB_shootdowns 277.00 ± 14% +84.9% 512.25 ± 10% interrupts.CPU131.RES:Rescheduling_interrupts 155.75 ± 24% +71.7% 267.50 ± 21% interrupts.CPU132.TLB:TLB_shootdowns 174.25 ± 24% +50.4% 262.00 ± 19% interrupts.CPU133.TLB:TLB_shootdowns 137.00 ± 7% +116.4% 296.50 ± 11% interrupts.CPU134.TLB:TLB_shootdowns 152.00 ± 24% +86.3% 283.25 ± 17% interrupts.CPU138.TLB:TLB_shootdowns 820.00 ± 10% -32.5% 553.75 ± 3% interrupts.CPU14.TLB:TLB_shootdowns 277.75 ± 19% +81.1% 503.00 ± 7% interrupts.CPU140.RES:Rescheduling_interrupts 298.25 ± 31% +59.3% 475.00 ± 15% interrupts.CPU142.RES:Rescheduling_interrupts 192.50 ± 22% +38.6% 266.75 ± 4% interrupts.CPU143.TLB:TLB_shootdowns 290.00 ± 38% +103.4% 590.00 ± 18% interrupts.CPU145.RES:Rescheduling_interrupts 104.50 ± 34% +90.4% 199.00 ± 16% interrupts.CPU145.TLB:TLB_shootdowns 226.50 ± 10% +96.9% 446.00 ± 15% interrupts.CPU146.RES:Rescheduling_interrupts 95.75 ± 8% +145.4% 235.00 ± 12% interrupts.CPU146.TLB:TLB_shootdowns 145.00 ± 22% +65.2% 239.50 ± 16% interrupts.CPU147.TLB:TLB_shootdowns 90.25 ± 23% +212.2% 281.75 ± 19% interrupts.CPU148.TLB:TLB_shootdowns 104.25 ± 22% +142.7% 253.00 ± 11% interrupts.CPU149.TLB:TLB_shootdowns 2888 ± 16% -35.6% 1860 ± 30% interrupts.CPU15.NMI:Non-maskable_interrupts 2888 ± 16% -35.6% 1860 ± 30% interrupts.CPU15.PMI:Performance_monitoring_interrupts 472.25 ± 31% +275.2% 1771 ±106% interrupts.CPU15.RES:Rescheduling_interrupts 816.50 ± 16% -50.0% 408.50 ± 11% interrupts.CPU15.TLB:TLB_shootdowns 84.75 ± 24% +207.4% 260.50 ± 10% interrupts.CPU150.TLB:TLB_shootdowns 133.50 ± 16% +108.6% 278.50 ± 17% interrupts.CPU151.TLB:TLB_shootdowns 103.75 ± 25% +132.0% 240.75 ± 8% interrupts.CPU152.TLB:TLB_shootdowns 105.00 ± 60% +155.0% 267.75 ± 7% interrupts.CPU153.TLB:TLB_shootdowns 100.50 ± 28% +133.1% 234.25 ± 8% interrupts.CPU154.TLB:TLB_shootdowns 73.50 ± 36% +220.4% 235.50 ± 11% interrupts.CPU155.TLB:TLB_shootdowns 229.50 ± 18% +182.1% 647.50 ± 8% interrupts.CPU156.RES:Rescheduling_interrupts 114.50 ± 20% +110.9% 241.50 ± 29% interrupts.CPU156.TLB:TLB_shootdowns 125.75 ± 27% +79.7% 226.00 ± 8% interrupts.CPU157.TLB:TLB_shootdowns 97.50 ± 21% +143.1% 237.00 ± 16% interrupts.CPU158.TLB:TLB_shootdowns 94.75 ± 49% +136.4% 224.00 ± 21% interrupts.CPU159.TLB:TLB_shootdowns 452.00 ± 36% +89.2% 855.25 ± 39% interrupts.CPU16.RES:Rescheduling_interrupts 959.00 ± 16% -54.3% 438.25 ± 8% interrupts.CPU16.TLB:TLB_shootdowns 96.75 ± 9% +145.0% 237.00 ± 9% interrupts.CPU161.TLB:TLB_shootdowns 94.25 ± 24% +169.5% 254.00 ± 10% interrupts.CPU162.TLB:TLB_shootdowns 100.50 ± 8% +141.8% 243.00 ± 14% interrupts.CPU163.TLB:TLB_shootdowns 87.00 ± 42% +186.5% 249.25 ± 8% interrupts.CPU164.TLB:TLB_shootdowns 93.50 ± 29% +145.7% 229.75 ± 9% interrupts.CPU165.TLB:TLB_shootdowns 110.50 ± 33% +165.4% 293.25 ± 15% interrupts.CPU166.TLB:TLB_shootdowns 323.25 ± 37% +105.7% 665.00 ± 39% interrupts.CPU167.RES:Rescheduling_interrupts 96.75 ± 48% +140.3% 232.50 ± 14% interrupts.CPU167.TLB:TLB_shootdowns 224.00 ± 22% +140.8% 539.50 ± 40% interrupts.CPU168.RES:Rescheduling_interrupts 87.50 ± 26% +151.1% 219.75 ± 14% interrupts.CPU168.TLB:TLB_shootdowns 1120 ± 47% +79.9% 2015 ± 27% interrupts.CPU169.NMI:Non-maskable_interrupts 1120 ± 47% +79.9% 2015 ± 27% interrupts.CPU169.PMI:Performance_monitoring_interrupts 100.25 ± 22% +117.5% 218.00 ± 9% interrupts.CPU169.TLB:TLB_shootdowns 667.75 ± 12% -30.3% 465.50 ± 14% interrupts.CPU17.TLB:TLB_shootdowns 75.00 ± 28% +252.7% 264.50 ± 22% interrupts.CPU170.TLB:TLB_shootdowns 94.25 ± 19% +126.0% 213.00 ± 13% interrupts.CPU171.TLB:TLB_shootdowns 318.25 ± 28% +53.3% 488.00 ± 18% interrupts.CPU172.RES:Rescheduling_interrupts 85.25 ± 33% +221.1% 273.75 ± 19% interrupts.CPU172.TLB:TLB_shootdowns 82.25 ± 18% +130.7% 189.75 ± 20% interrupts.CPU173.TLB:TLB_shootdowns 67.75 ± 55% +261.6% 245.00 ± 12% interrupts.CPU174.TLB:TLB_shootdowns 227.75 ± 41% +588.6% 1568 ± 95% interrupts.CPU175.RES:Rescheduling_interrupts 91.50 ± 10% +150.8% 229.50 ± 10% interrupts.CPU175.TLB:TLB_shootdowns 99.50 ± 17% +147.2% 246.00 ± 22% interrupts.CPU176.TLB:TLB_shootdowns 79.25 ± 18% +200.0% 237.75 ± 27% interrupts.CPU177.TLB:TLB_shootdowns 87.50 ± 36% +129.7% 201.00 ± 4% interrupts.CPU178.TLB:TLB_shootdowns 127.25 ± 44% +82.7% 232.50 ± 15% interrupts.CPU179.TLB:TLB_shootdowns 1161 ± 15% +137.6% 2758 ± 27% interrupts.CPU18.NMI:Non-maskable_interrupts 1161 ± 15% +137.6% 2758 ± 27% interrupts.CPU18.PMI:Performance_monitoring_interrupts 425.75 ± 15% +149.5% 1062 ± 64% interrupts.CPU18.RES:Rescheduling_interrupts 985.50 ± 13% -50.1% 492.25 ± 13% interrupts.CPU18.TLB:TLB_shootdowns 112.25 ± 33% +120.7% 247.75 ± 8% interrupts.CPU180.TLB:TLB_shootdowns 126.25 ± 15% +96.0% 247.50 ± 9% interrupts.CPU181.TLB:TLB_shootdowns 229.25 ± 25% +74.6% 400.25 ± 7% interrupts.CPU182.RES:Rescheduling_interrupts 102.50 ± 41% +173.4% 280.25 ± 4% interrupts.CPU182.TLB:TLB_shootdowns 253.25 ± 13% +52.6% 386.50 ± 8% interrupts.CPU183.RES:Rescheduling_interrupts 97.00 ± 50% +153.4% 245.75 ± 4% interrupts.CPU183.TLB:TLB_shootdowns 99.00 ± 14% +143.7% 241.25 ± 18% interrupts.CPU184.TLB:TLB_shootdowns 287.50 ± 37% +57.0% 451.25 ± 32% interrupts.CPU185.RES:Rescheduling_interrupts 92.50 ± 27% +139.2% 221.25 ± 5% interrupts.CPU185.TLB:TLB_shootdowns 2302 ± 26% +37.4% 3162 ± 5% interrupts.CPU186.NMI:Non-maskable_interrupts 2302 ± 26% +37.4% 3162 ± 5% interrupts.CPU186.PMI:Performance_monitoring_interrupts 87.75 ± 16% +204.0% 266.75 ± 20% interrupts.CPU186.TLB:TLB_shootdowns 97.00 ± 31% +157.0% 249.25 ± 10% interrupts.CPU187.TLB:TLB_shootdowns 261.50 ± 10% +54.2% 403.25 ± 15% interrupts.CPU188.RES:Rescheduling_interrupts 108.50 ± 42% +103.7% 221.00 ± 26% interrupts.CPU188.TLB:TLB_shootdowns 223.75 ± 21% +88.4% 421.50 ± 18% interrupts.CPU189.RES:Rescheduling_interrupts 119.25 ± 11% +108.4% 248.50 ± 25% interrupts.CPU189.TLB:TLB_shootdowns 400.25 ± 27% +117.4% 870.25 ± 52% interrupts.CPU19.RES:Rescheduling_interrupts 699.50 ± 16% -31.8% 476.75 ± 10% interrupts.CPU19.TLB:TLB_shootdowns 99.75 ± 43% +148.6% 248.00 ± 12% interrupts.CPU190.TLB:TLB_shootdowns 98.25 ± 10% +146.6% 242.25 ± 17% interrupts.CPU191.TLB:TLB_shootdowns 105.25 ± 29% +145.4% 258.25 ± 19% interrupts.CPU192.TLB:TLB_shootdowns 89.50 ± 23% +191.3% 260.75 ± 17% interrupts.CPU193.TLB:TLB_shootdowns 66.25 ± 25% +285.7% 255.50 ± 17% interrupts.CPU194.TLB:TLB_shootdowns 278.75 ± 10% +84.9% 515.50 ± 32% interrupts.CPU195.RES:Rescheduling_interrupts 80.00 ± 18% +247.8% 278.25 ± 14% interrupts.CPU195.TLB:TLB_shootdowns 70.25 ± 12% +246.3% 243.25 ± 11% interrupts.CPU196.TLB:TLB_shootdowns 104.75 ± 26% +147.0% 258.75 ± 23% interrupts.CPU197.TLB:TLB_shootdowns 86.75 ± 62% +188.2% 250.00 ± 20% interrupts.CPU198.TLB:TLB_shootdowns 94.25 ± 17% +150.1% 235.75 ± 19% interrupts.CPU199.TLB:TLB_shootdowns 3084 ± 4% -52.9% 1453 ± 6% interrupts.CPU2.NMI:Non-maskable_interrupts 3084 ± 4% -52.9% 1453 ± 6% interrupts.CPU2.PMI:Performance_monitoring_interrupts 556.25 ± 46% +179.6% 1555 ± 83% interrupts.CPU2.RES:Rescheduling_interrupts 891.00 ± 15% -42.6% 511.50 ± 13% interrupts.CPU2.TLB:TLB_shootdowns 364.75 ± 16% +193.9% 1072 ± 40% interrupts.CPU20.RES:Rescheduling_interrupts 825.75 ± 13% -35.7% 530.75 ± 6% interrupts.CPU20.TLB:TLB_shootdowns 129.25 ± 17% +83.0% 236.50 ± 7% interrupts.CPU200.TLB:TLB_shootdowns 88.50 ± 58% +200.0% 265.50 ± 12% interrupts.CPU201.TLB:TLB_shootdowns 116.75 ± 24% +99.8% 233.25 ± 24% interrupts.CPU202.TLB:TLB_shootdowns 254.25 ± 35% +121.8% 564.00 ± 14% interrupts.CPU203.RES:Rescheduling_interrupts 83.25 ± 27% +203.0% 252.25 ± 18% interrupts.CPU203.TLB:TLB_shootdowns 110.25 ± 16% +195.2% 325.50 ± 35% interrupts.CPU204.TLB:TLB_shootdowns 332.00 ± 44% +60.5% 532.75 ± 15% interrupts.CPU205.RES:Rescheduling_interrupts 89.00 ± 32% +165.4% 236.25 ± 11% interrupts.CPU205.TLB:TLB_shootdowns 236.25 ± 23% +112.8% 502.75 ± 10% interrupts.CPU206.RES:Rescheduling_interrupts 82.75 ± 32% +195.2% 244.25 ± 13% interrupts.CPU206.TLB:TLB_shootdowns 121.25 ± 15% +86.6% 226.25 ± 18% interrupts.CPU207.TLB:TLB_shootdowns 83.75 ± 24% +170.7% 226.75 ± 5% interrupts.CPU208.TLB:TLB_shootdowns 330.25 ± 13% +131.5% 764.50 ± 44% interrupts.CPU209.RES:Rescheduling_interrupts 75.75 ± 20% +217.8% 240.75 ± 9% interrupts.CPU209.TLB:TLB_shootdowns 778.00 ± 16% -39.2% 473.00 ± 34% interrupts.CPU21.TLB:TLB_shootdowns 268.75 ± 27% +178.4% 748.25 ± 37% interrupts.CPU210.RES:Rescheduling_interrupts 114.75 ± 16% +100.0% 229.50 ± 13% interrupts.CPU210.TLB:TLB_shootdowns 112.00 ± 29% +101.1% 225.25 ± 15% interrupts.CPU211.TLB:TLB_shootdowns 97.00 ± 58% +159.3% 251.50 ± 21% interrupts.CPU212.TLB:TLB_shootdowns 263.75 ± 29% +80.3% 475.50 ± 15% interrupts.CPU213.RES:Rescheduling_interrupts 101.25 ± 25% +98.5% 201.00 ± 28% interrupts.CPU215.TLB:TLB_shootdowns 1275 ± 12% +70.6% 2177 ± 30% interrupts.CPU216.NMI:Non-maskable_interrupts 1275 ± 12% +70.6% 2177 ± 30% interrupts.CPU216.PMI:Performance_monitoring_interrupts 278.75 ± 15% +48.7% 414.50 ± 14% interrupts.CPU216.RES:Rescheduling_interrupts 95.50 ± 26% +118.3% 208.50 ± 11% interrupts.CPU216.TLB:TLB_shootdowns 1391 ± 30% +54.2% 2146 ± 33% interrupts.CPU217.NMI:Non-maskable_interrupts 1391 ± 30% +54.2% 2146 ± 33% interrupts.CPU217.PMI:Performance_monitoring_interrupts 114.50 ± 10% +84.9% 211.75 ± 16% interrupts.CPU217.TLB:TLB_shootdowns 77.00 ± 48% +308.4% 314.50 ± 55% interrupts.CPU218.TLB:TLB_shootdowns 70.75 ± 41% +259.0% 254.00 ± 21% interrupts.CPU219.TLB:TLB_shootdowns 884.75 ± 18% -36.6% 561.25 ± 12% interrupts.CPU22.TLB:TLB_shootdowns 58.25 ± 36% +315.9% 242.25 ± 13% interrupts.CPU220.TLB:TLB_shootdowns 230.50 ± 35% +103.9% 470.00 ± 18% interrupts.CPU221.RES:Rescheduling_interrupts 61.00 ± 40% +254.1% 216.00 ± 3% interrupts.CPU221.TLB:TLB_shootdowns 114.75 ± 47% +102.6% 232.50 ± 22% interrupts.CPU223.TLB:TLB_shootdowns 94.25 ± 19% +146.9% 232.75 ± 15% interrupts.CPU224.TLB:TLB_shootdowns 92.75 ± 47% +142.6% 225.00 ± 12% interrupts.CPU225.TLB:TLB_shootdowns 84.75 ± 19% +182.0% 239.00 ± 4% interrupts.CPU226.TLB:TLB_shootdowns 91.00 ± 34% +164.8% 241.00 ± 20% interrupts.CPU227.TLB:TLB_shootdowns 2345 ± 32% -40.0% 1407 ± 17% interrupts.CPU228.NMI:Non-maskable_interrupts 2345 ± 32% -40.0% 1407 ± 17% interrupts.CPU228.PMI:Performance_monitoring_interrupts 61.75 ± 24% +232.0% 205.00 ± 10% interrupts.CPU228.TLB:TLB_shootdowns 75.00 ± 49% +188.7% 216.50 ± 14% interrupts.CPU229.TLB:TLB_shootdowns 706.25 ± 17% -38.5% 434.00 ± 19% interrupts.CPU23.TLB:TLB_shootdowns 78.50 ± 20% +208.6% 242.25 ± 11% interrupts.CPU230.TLB:TLB_shootdowns 67.50 ± 35% +214.4% 212.25 ± 20% interrupts.CPU231.TLB:TLB_shootdowns 204.25 ± 15% +141.9% 494.00 ± 11% interrupts.CPU232.RES:Rescheduling_interrupts 65.25 ± 19% +254.0% 231.00 ± 13% interrupts.CPU232.TLB:TLB_shootdowns 104.75 ± 28% +106.9% 216.75 ± 26% interrupts.CPU233.TLB:TLB_shootdowns 65.00 ± 44% +226.9% 212.50 ± 8% interrupts.CPU234.TLB:TLB_shootdowns 72.00 ± 53% +186.8% 206.50 ± 20% interrupts.CPU235.TLB:TLB_shootdowns 262.50 ± 38% +75.3% 460.25 ± 13% interrupts.CPU236.RES:Rescheduling_interrupts 72.75 ± 61% +195.5% 215.00 ± 7% interrupts.CPU236.TLB:TLB_shootdowns 78.25 ± 21% +187.2% 224.75 ± 32% interrupts.CPU237.TLB:TLB_shootdowns 92.25 ± 27% +166.9% 246.25 ± 17% interrupts.CPU238.TLB:TLB_shootdowns 120.50 ± 37% +118.5% 263.25 ± 19% interrupts.CPU239.TLB:TLB_shootdowns 871.75 ± 7% -42.3% 502.75 ± 12% interrupts.CPU24.TLB:TLB_shootdowns 63.75 ± 34% +271.0% 236.50 ± 6% interrupts.CPU240.TLB:TLB_shootdowns 77.25 ± 29% +207.4% 237.50 ± 15% interrupts.CPU241.TLB:TLB_shootdowns 49.75 ± 29% +310.6% 204.25 ± 13% interrupts.CPU242.TLB:TLB_shootdowns 76.00 ± 19% +194.7% 224.00 ± 17% interrupts.CPU243.TLB:TLB_shootdowns 80.75 ± 10% +183.6% 229.00 ± 17% interrupts.CPU244.TLB:TLB_shootdowns 86.75 ± 57% +171.8% 235.75 ± 8% interrupts.CPU245.TLB:TLB_shootdowns 56.50 ± 51% +344.2% 251.00 ± 13% interrupts.CPU246.TLB:TLB_shootdowns 270.50 ± 24% +153.1% 684.75 ± 34% interrupts.CPU247.RES:Rescheduling_interrupts 88.25 ± 24% +180.2% 247.25 ± 18% interrupts.CPU247.TLB:TLB_shootdowns 81.25 ± 37% +167.1% 217.00 ± 20% interrupts.CPU248.TLB:TLB_shootdowns 67.00 ± 26% +216.4% 212.00 ± 11% interrupts.CPU249.TLB:TLB_shootdowns 740.75 ± 11% -36.9% 467.25 ± 7% interrupts.CPU25.TLB:TLB_shootdowns 67.50 ± 29% +247.0% 234.25 ± 11% interrupts.CPU250.TLB:TLB_shootdowns 92.00 ± 63% +111.1% 194.25 ± 24% interrupts.CPU251.TLB:TLB_shootdowns 72.25 ± 45% +223.5% 233.75 ± 16% interrupts.CPU252.TLB:TLB_shootdowns 78.75 ± 29% +178.7% 219.50 ± 12% interrupts.CPU253.TLB:TLB_shootdowns 64.25 ± 58% +258.4% 230.25 ± 11% interrupts.CPU254.TLB:TLB_shootdowns 267.00 ± 17% +44.3% 385.25 ± 13% interrupts.CPU255.RES:Rescheduling_interrupts 114.25 ± 36% +106.1% 235.50 ± 12% interrupts.CPU255.TLB:TLB_shootdowns 74.00 ± 33% +213.9% 232.25 ± 13% interrupts.CPU256.TLB:TLB_shootdowns 86.00 ± 81% +158.4% 222.25 ± 19% interrupts.CPU257.TLB:TLB_shootdowns 1565 ± 38% +54.4% 2417 ± 27% interrupts.CPU258.NMI:Non-maskable_interrupts 1565 ± 38% +54.4% 2417 ± 27% interrupts.CPU258.PMI:Performance_monitoring_interrupts 252.00 ± 25% +118.6% 550.75 ± 54% interrupts.CPU258.RES:Rescheduling_interrupts 81.75 ± 21% +192.7% 239.25 ± 9% interrupts.CPU258.TLB:TLB_shootdowns 106.75 ± 37% +109.1% 223.25 ± 20% interrupts.CPU259.TLB:TLB_shootdowns 899.75 ± 8% -42.9% 514.00 ± 6% interrupts.CPU26.TLB:TLB_shootdowns 67.25 ± 41% +291.8% 263.50 ± 22% interrupts.CPU260.TLB:TLB_shootdowns 59.25 ± 34% +289.0% 230.50 ± 8% interrupts.CPU261.TLB:TLB_shootdowns 292.75 ± 12% +61.0% 471.25 ± 16% interrupts.CPU262.RES:Rescheduling_interrupts 49.75 ± 45% +400.0% 248.75 ± 14% interrupts.CPU262.TLB:TLB_shootdowns 325.00 ± 17% +52.1% 494.25 ± 27% interrupts.CPU263.RES:Rescheduling_interrupts 87.00 ± 32% +143.1% 211.50 ± 11% interrupts.CPU263.TLB:TLB_shootdowns 218.25 ± 14% +71.7% 374.75 ± 5% interrupts.CPU264.RES:Rescheduling_interrupts 75.75 ± 28% +188.8% 218.75 ± 9% interrupts.CPU264.TLB:TLB_shootdowns 56.25 ± 55% +306.2% 228.50 ± 20% interrupts.CPU265.TLB:TLB_shootdowns 80.75 ± 53% +221.1% 259.25 ± 12% interrupts.CPU266.TLB:TLB_shootdowns 271.25 ± 27% +56.2% 423.75 ± 22% interrupts.CPU267.RES:Rescheduling_interrupts 60.25 ± 48% +352.7% 272.75 ± 16% interrupts.CPU267.TLB:TLB_shootdowns 56.00 ± 43% +295.5% 221.50 ± 20% interrupts.CPU268.TLB:TLB_shootdowns 97.25 ± 14% +151.2% 244.25 ± 17% interrupts.CPU269.TLB:TLB_shootdowns 688.00 ± 13% -32.2% 466.25 ± 11% interrupts.CPU27.TLB:TLB_shootdowns 208.00 ± 27% +93.6% 402.75 ± 8% interrupts.CPU270.RES:Rescheduling_interrupts 57.50 ± 30% +368.3% 269.25 ± 19% interrupts.CPU270.TLB:TLB_shootdowns 73.75 ± 34% +288.5% 286.50 ± 14% interrupts.CPU271.TLB:TLB_shootdowns 279.25 ± 32% +37.3% 383.50 ± 10% interrupts.CPU272.RES:Rescheduling_interrupts 79.25 ± 14% +189.6% 229.50 ± 28% interrupts.CPU272.TLB:TLB_shootdowns 216.50 ± 15% +201.8% 653.50 ± 66% interrupts.CPU273.RES:Rescheduling_interrupts 65.50 ± 22% +256.1% 233.25 ± 11% interrupts.CPU273.TLB:TLB_shootdowns 74.75 ± 68% +198.3% 223.00 ± 17% interrupts.CPU274.TLB:TLB_shootdowns 68.50 ± 42% +204.4% 208.50 ± 10% interrupts.CPU275.TLB:TLB_shootdowns 59.75 ± 35% +226.8% 195.25 ± 13% interrupts.CPU276.TLB:TLB_shootdowns 292.25 ± 35% +80.2% 526.75 ± 32% interrupts.CPU277.RES:Rescheduling_interrupts 73.25 ± 39% +233.8% 244.50 ± 14% interrupts.CPU277.TLB:TLB_shootdowns 61.50 ± 60% +248.0% 214.00 ± 9% interrupts.CPU278.TLB:TLB_shootdowns 163.25 ± 15% +188.1% 470.25 ± 18% interrupts.CPU279.RES:Rescheduling_interrupts 81.50 ± 10% +142.0% 197.25 ± 8% interrupts.CPU279.TLB:TLB_shootdowns 372.75 ± 20% +139.3% 892.00 ± 59% interrupts.CPU28.RES:Rescheduling_interrupts 891.00 ± 4% -41.6% 520.50 ± 10% interrupts.CPU28.TLB:TLB_shootdowns 75.75 ± 23% +223.8% 245.25 ± 12% interrupts.CPU280.TLB:TLB_shootdowns 66.25 ± 39% +234.0% 221.25 ± 12% interrupts.CPU281.TLB:TLB_shootdowns 239.25 ± 24% +90.9% 456.75 ± 15% interrupts.CPU282.RES:Rescheduling_interrupts 63.50 ± 36% +281.9% 242.50 ± 11% interrupts.CPU282.TLB:TLB_shootdowns 75.50 ± 39% +215.2% 238.00 ± 10% interrupts.CPU283.TLB:TLB_shootdowns 269.75 ± 41% +72.2% 464.50 ± 20% interrupts.CPU284.RES:Rescheduling_interrupts 66.25 ± 51% +251.7% 233.00 ± 22% interrupts.CPU284.TLB:TLB_shootdowns 103.75 ± 33% +117.3% 225.50 ± 6% interrupts.CPU285.TLB:TLB_shootdowns 274.75 ± 25% +42.6% 391.75 ± 9% interrupts.CPU286.RES:Rescheduling_interrupts 102.50 ± 22% +106.1% 211.25 ± 3% interrupts.CPU286.TLB:TLB_shootdowns 1604 ± 9% -17.7% 1320 interrupts.CPU287.CAL:Function_call_interrupts 1860 ± 18% -26.0% 1376 ± 16% interrupts.CPU287.NMI:Non-maskable_interrupts 1860 ± 18% -26.0% 1376 ± 16% interrupts.CPU287.PMI:Performance_monitoring_interrupts 245.50 ± 19% +54.3% 378.75 ± 8% interrupts.CPU287.RES:Rescheduling_interrupts 685.00 ± 13% -35.5% 441.75 ± 4% interrupts.CPU29.TLB:TLB_shootdowns 689.75 ± 18% -32.4% 466.00 ± 6% interrupts.CPU3.TLB:TLB_shootdowns 399.75 ± 15% +149.8% 998.75 ± 69% interrupts.CPU30.RES:Rescheduling_interrupts 793.00 ± 16% -36.1% 506.75 ± 7% interrupts.CPU30.TLB:TLB_shootdowns 3182 ± 24% -56.9% 1372 ± 6% interrupts.CPU31.NMI:Non-maskable_interrupts 3182 ± 24% -56.9% 1372 ± 6% interrupts.CPU31.PMI:Performance_monitoring_interrupts 283.00 ± 10% +134.5% 663.50 ± 31% interrupts.CPU31.RES:Rescheduling_interrupts 783.50 ± 20% -43.8% 440.25 ± 3% interrupts.CPU31.TLB:TLB_shootdowns 905.25 ± 8% -39.7% 546.25 ± 19% interrupts.CPU32.TLB:TLB_shootdowns 709.25 ± 4% -35.0% 461.00 ± 14% interrupts.CPU33.TLB:TLB_shootdowns 474.50 ± 31% +92.7% 914.50 ± 50% interrupts.CPU34.RES:Rescheduling_interrupts 879.50 ± 11% -43.0% 501.50 ± 9% interrupts.CPU34.TLB:TLB_shootdowns 750.00 ± 12% -42.5% 431.25 ± 10% interrupts.CPU35.TLB:TLB_shootdowns 889.00 ± 5% -45.4% 485.25 ± 6% interrupts.CPU36.TLB:TLB_shootdowns 415.25 ± 26% +34.3% 557.75 ± 9% interrupts.CPU37.RES:Rescheduling_interrupts 818.50 ± 14% -45.1% 449.25 ± 14% interrupts.CPU37.TLB:TLB_shootdowns 453.50 ± 19% +23.7% 561.00 ± 11% interrupts.CPU38.RES:Rescheduling_interrupts 899.00 ± 5% -37.8% 558.75 ± 7% interrupts.CPU38.TLB:TLB_shootdowns 752.25 ± 14% -41.3% 441.25 ± 11% interrupts.CPU39.TLB:TLB_shootdowns 1502 ± 7% +48.7% 2234 ± 24% interrupts.CPU4.NMI:Non-maskable_interrupts 1502 ± 7% +48.7% 2234 ± 24% interrupts.CPU4.PMI:Performance_monitoring_interrupts 1012 ± 26% -47.3% 534.00 ± 6% interrupts.CPU4.TLB:TLB_shootdowns 882.25 ± 14% -41.4% 517.00 ± 17% interrupts.CPU40.TLB:TLB_shootdowns 762.50 ± 13% -35.8% 489.25 ± 4% interrupts.CPU41.TLB:TLB_shootdowns 995.75 ± 12% -49.2% 506.00 ± 8% interrupts.CPU42.TLB:TLB_shootdowns 652.50 ± 12% -35.4% 421.25 ± 11% interrupts.CPU43.TLB:TLB_shootdowns 421.25 ± 18% +102.1% 851.25 ± 47% interrupts.CPU44.RES:Rescheduling_interrupts 819.00 ± 10% -39.3% 497.25 ± 7% interrupts.CPU44.TLB:TLB_shootdowns 369.00 ± 21% +79.1% 660.75 ± 22% interrupts.CPU45.RES:Rescheduling_interrupts 801.00 ± 7% -41.6% 467.75 ± 8% interrupts.CPU45.TLB:TLB_shootdowns 786.75 ± 15% -38.5% 483.75 ± 12% interrupts.CPU46.TLB:TLB_shootdowns 716.75 ± 13% -35.4% 463.25 ± 11% interrupts.CPU47.TLB:TLB_shootdowns 813.25 ± 9% -34.6% 531.75 ± 6% interrupts.CPU48.TLB:TLB_shootdowns 658.50 ± 6% -26.2% 486.25 ± 6% interrupts.CPU49.TLB:TLB_shootdowns 735.25 ± 20% -36.6% 466.00 ± 18% interrupts.CPU5.TLB:TLB_shootdowns 852.50 ± 17% -43.0% 485.75 ± 9% interrupts.CPU50.TLB:TLB_shootdowns 815.75 ± 8% -40.3% 487.25 ± 12% interrupts.CPU51.TLB:TLB_shootdowns 899.50 ± 9% -42.5% 517.50 ± 10% interrupts.CPU52.TLB:TLB_shootdowns 654.50 ± 9% -33.8% 433.00 ± 19% interrupts.CPU53.TLB:TLB_shootdowns 920.00 ± 8% -40.9% 543.50 ± 6% interrupts.CPU54.TLB:TLB_shootdowns 687.25 ± 19% -31.5% 470.75 ± 16% interrupts.CPU55.TLB:TLB_shootdowns 2467 ± 22% -41.5% 1442 ± 13% interrupts.CPU56.NMI:Non-maskable_interrupts 2467 ± 22% -41.5% 1442 ± 13% interrupts.CPU56.PMI:Performance_monitoring_interrupts 342.00 ± 7% +106.1% 705.00 ± 30% interrupts.CPU56.RES:Rescheduling_interrupts 884.75 ± 19% -45.2% 485.25 ± 8% interrupts.CPU56.TLB:TLB_shootdowns 772.00 ± 11% -43.0% 440.00 ± 11% interrupts.CPU57.TLB:TLB_shootdowns 942.50 ± 11% -46.3% 506.00 ± 10% interrupts.CPU58.TLB:TLB_shootdowns 754.25 ± 5% -38.0% 467.50 interrupts.CPU59.TLB:TLB_shootdowns 393.75 ± 9% +311.4% 1619 ±109% interrupts.CPU6.RES:Rescheduling_interrupts 860.25 ± 12% -46.6% 459.00 ± 7% interrupts.CPU6.TLB:TLB_shootdowns 894.50 ± 16% -43.4% 506.00 ± 13% interrupts.CPU60.TLB:TLB_shootdowns 1684 ± 32% +78.1% 2999 ± 14% interrupts.CPU61.NMI:Non-maskable_interrupts 1684 ± 32% +78.1% 2999 ± 14% interrupts.CPU61.PMI:Performance_monitoring_interrupts 693.50 ± 17% -34.9% 451.75 ± 9% interrupts.CPU61.TLB:TLB_shootdowns 936.25 ± 16% -49.3% 474.75 ± 8% interrupts.CPU62.TLB:TLB_shootdowns 421.25 ± 44% +112.9% 897.00 ± 11% interrupts.CPU63.RES:Rescheduling_interrupts 731.25 ± 8% -41.1% 431.00 ± 7% interrupts.CPU63.TLB:TLB_shootdowns 883.00 ± 15% -43.7% 496.75 ± 7% interrupts.CPU64.TLB:TLB_shootdowns 736.00 ± 11% -36.8% 465.50 ± 5% interrupts.CPU65.TLB:TLB_shootdowns 880.00 ± 7% -42.6% 505.25 ± 8% interrupts.CPU66.TLB:TLB_shootdowns 943.50 ± 11% -40.5% 561.75 ± 10% interrupts.CPU68.TLB:TLB_shootdowns 393.25 ± 36% +100.5% 788.50 ± 26% interrupts.CPU69.RES:Rescheduling_interrupts 790.25 ± 9% -38.3% 487.50 ± 8% interrupts.CPU69.TLB:TLB_shootdowns 415.50 ± 24% +407.0% 2106 ±123% interrupts.CPU7.RES:Rescheduling_interrupts 679.00 ± 14% -36.8% 429.25 interrupts.CPU7.TLB:TLB_shootdowns 350.25 ± 13% +92.1% 672.75 ± 12% interrupts.CPU70.RES:Rescheduling_interrupts 778.75 ± 3% -41.8% 453.50 ± 11% interrupts.CPU70.TLB:TLB_shootdowns 673.50 ± 8% -39.8% 405.75 ± 11% interrupts.CPU71.TLB:TLB_shootdowns 460.75 ± 74% +95.7% 901.75 ± 43% interrupts.CPU74.RES:Rescheduling_interrupts 160.75 ± 33% +82.1% 292.75 ± 15% interrupts.CPU76.TLB:TLB_shootdowns 198.00 ± 22% +45.8% 288.75 ± 9% interrupts.CPU77.TLB:TLB_shootdowns 2305 ± 26% -35.5% 1486 ± 12% interrupts.CPU79.NMI:Non-maskable_interrupts 2305 ± 26% -35.5% 1486 ± 12% interrupts.CPU79.PMI:Performance_monitoring_interrupts 155.25 ± 31% +72.9% 268.50 ± 8% interrupts.CPU79.TLB:TLB_shootdowns 887.25 ± 17% -45.4% 484.50 ± 10% interrupts.CPU8.TLB:TLB_shootdowns 192.00 ± 18% +32.9% 255.25 ± 12% interrupts.CPU80.TLB:TLB_shootdowns 158.75 ± 30% +95.1% 309.75 ± 12% interrupts.CPU82.TLB:TLB_shootdowns 265.75 ± 35% +120.8% 586.75 ± 30% interrupts.CPU84.RES:Rescheduling_interrupts 146.75 ± 29% +80.2% 264.50 ± 20% interrupts.CPU84.TLB:TLB_shootdowns 186.00 ± 31% +56.0% 290.25 ± 9% interrupts.CPU86.TLB:TLB_shootdowns 169.25 ± 15% +66.5% 281.75 ± 2% interrupts.CPU88.TLB:TLB_shootdowns 879.75 ± 10% -47.5% 462.00 ± 4% interrupts.CPU9.TLB:TLB_shootdowns 161.00 ± 20% +75.8% 283.00 ± 5% interrupts.CPU90.TLB:TLB_shootdowns 194.75 ± 12% +50.3% 292.75 ± 11% interrupts.CPU91.TLB:TLB_shootdowns 2722 ± 34% -37.8% 1692 ± 6% interrupts.CPU92.NMI:Non-maskable_interrupts 2722 ± 34% -37.8% 1692 ± 6% interrupts.CPU92.PMI:Performance_monitoring_interrupts 190.75 ± 14% +36.3% 260.00 ± 22% interrupts.CPU92.TLB:TLB_shootdowns 188.00 ± 15% +57.0% 295.25 ± 10% interrupts.CPU94.TLB:TLB_shootdowns 200.25 ± 11% +59.6% 319.50 ± 16% interrupts.CPU95.TLB:TLB_shootdowns 316.25 ± 55% +105.1% 648.50 ± 19% interrupts.CPU96.RES:Rescheduling_interrupts 143.75 ± 42% +91.0% 274.50 ± 8% interrupts.CPU96.TLB:TLB_shootdowns 2409 ± 41% -34.2% 1586 ± 46% interrupts.CPU97.NMI:Non-maskable_interrupts 2409 ± 41% -34.2% 1586 ± 46% interrupts.CPU97.PMI:Performance_monitoring_interrupts 186.00 ± 33% +56.0% 290.25 ± 16% interrupts.CPU98.TLB:TLB_shootdowns vm-scalability.time.user_time 16000 +-------------------------------------------------------------------+ | + | 15000 |-+ :+ + + + | 14000 |-+ .++. +. + + +.+ :+ .++ ++. +.++ + + :+ + +| |+.+++ ++ + + + ++.+++ +.+++.++.+ ++ + +.+ + :+ | 13000 |-+ + | | | 12000 |-+ | | | 11000 |-+ | 10000 |-+ | | OO O O | 9000 |-+ OO O OO O OO O | |O OOO O O OO | 8000 +-------------------------------------------------------------------+ vm-scalability.time.voluntary_context_switches 1e+06 +------------------------------------------------------------------+ | O O | 900000 |O+OOO OO O OOO OOO O O OO | 800000 |-+ | | | 700000 |-+ | 600000 |-+ | | | 500000 |-+ | 400000 |-+ | | | 300000 |-+ | 200000 |-+ | |+.+++.+++.+ + +.+++.++.+++.+++.+++.+++. .+ +. | 100000 +------------------------------------------------------------------+ vm-scalability.throughput 1.6e+07 +-----------------------------------------------------------------+ | | 1.5e+07 |-+ + + + | 1.4e+07 |+. + + +. + + +. + + | | +++ +.+ +.+++ .++ +++ +.++ +++ +.+ +. | 1.3e+07 |-+ + +.+++ +++.++ +++.+++.+| 1.2e+07 |-+ | | | 1.1e+07 |-+ | 1e+07 |-+ | | | 9e+06 |-+ O O O O | 8e+06 |-+ O OO O OO O OO OO O | |O O O O O | 7e+06 +-----------------------------------------------------------------+ vm-scalability.median 44000 +-------------------------------------------------------------------+ 42000 |-+ + + + | | : : : : :: | 40000 |+. : +. + ++.+ .++. : + .+++. : +. + | 38000 |-++++ + +.+ ++.+++ +++ + ++.+ + +. .+++. +. ++. | | +++ + + +| 36000 |-+ | 34000 |-+ | 32000 |-+ | | | 30000 |-+ | 28000 |-+ O O O | | O O O O | 26000 |O+OOO O O OO OOO O O O | 24000 +-------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

3 months

3
10
0 / 0

[fs] b6509f6a8c: will-it-scale.per_thread_ops -12.6% regression

by kernel test robot

Greeting, FYI, we noticed a -12.6% regression of will-it-scale.per_thread_ops due to commit: commit: b6509f6a8c4313c068c69785c001451415969e44 ("Revert "fs: Do not check if there is a fsnotify watcher on pseudo inodes"") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: will-it-scale on test machine: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory with following parameters: nr_task: 100% mode: thread test: eventfd1 cpufreq_governor: performance ucode: 0x5002f01 test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two. test-url: https://github.com/antonblanchard/will-it-scale In addition to that, the commit also has significant impact on the following tests: +------------------+---------------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_process_ops -6.4% regression | | test machine | 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory | | test parameters | cpufreq_governor=performance | | | mode=process | | | nr_task=100% | | | test=unix1 | | | ucode=0x5002f01 | +------------------+---------------------------------------------------------------------------+ | testcase: change | will-it-scale: will-it-scale.per_thread_ops -2.3% regression | | test machine | 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory | | test parameters | cpufreq_governor=performance | | | mode=thread | | | nr_task=100% | | | test=pipe1 | | | ucode=0x5002f01 | +------------------+---------------------------------------------------------------------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-csl-2ap4/eventfd1/will-it-scale/0x5002f01 commit: v5.8-rc3 b6509f6a8c ("Revert "fs: Do not check if there is a fsnotify watcher on pseudo inodes"") v5.8-rc3 b6509f6a8c4313c068c69785c00 ---------------- --------------------------- %stddev %change %stddev \ | \ 1652352 -12.6% 1444002 ± 2% will-it-scale.per_thread_ops 3.173e+08 -12.6% 2.772e+08 ± 2% will-it-scale.workload 23.95 -3.1 20.90 ± 2% mpstat.cpu.all.usr% 23.00 -12.0% 20.25 ± 4% vmstat.cpu.us 578849 ± 4% +8.6% 628428 ± 4% sched_debug.cpu.max_idle_balance_cost.max 0.00 ± 9% -14.4% 0.00 ± 13% sched_debug.cpu.next_balance.stddev 2207 ± 24% +34.7% 2973 ± 20% numa-vmstat.node0.nr_mapped 6949 ± 22% +43.7% 9984 ± 10% numa-vmstat.node0.nr_slab_reclaimable 9739 ± 13% -29.0% 6913 ± 12% numa-vmstat.node1.nr_slab_reclaimable 7247 ± 13% -28.4% 5190 ± 19% slabinfo.eventpoll_pwq.active_objs 7247 ± 13% -28.4% 5190 ± 19% slabinfo.eventpoll_pwq.num_objs 960.00 ± 11% -40.0% 576.00 ± 26% slabinfo.kmalloc-rcl-128.active_objs 960.00 ± 11% -40.0% 576.00 ± 26% slabinfo.kmalloc-rcl-128.num_objs 27801 ± 22% +43.7% 39940 ± 10% numa-meminfo.node0.KReclaimable 8718 ± 25% +35.0% 11773 ± 20% numa-meminfo.node0.Mapped 27801 ± 22% +43.7% 39940 ± 10% numa-meminfo.node0.SReclaimable 38959 ± 13% -29.0% 27656 ± 12% numa-meminfo.node1.KReclaimable 38959 ± 13% -29.0% 27656 ± 12% numa-meminfo.node1.SReclaimable 107406 ± 2% -13.6% 92844 ± 6% numa-meminfo.node1.Slab 0.00 +1.1 1.09 ± 16% perf-profile.calltrace.cycles-pp.fsnotify_parent.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +1.3 1.26 ± 18% perf-profile.calltrace.cycles-pp.fsnotify_parent.security_file_permission.vfs_read.ksys_read.do_syscall_64 0.00 +1.8 1.76 ± 21% perf-profile.calltrace.cycles-pp.fsnotify.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +3.7 3.70 ± 20% perf-profile.calltrace.cycles-pp.fsnotify.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +3.0 3.02 ± 17% perf-profile.children.cycles-pp.fsnotify_parent 0.00 +5.5 5.55 ± 20% perf-profile.children.cycles-pp.fsnotify 0.00 +2.7 2.66 ± 17% perf-profile.self.cycles-pp.fsnotify_parent 0.00 +5.4 5.35 ± 20% perf-profile.self.cycles-pp.fsnotify 10890 ± 21% +53.7% 16738 ± 12% interrupts.CPU168.CAL:Function_call_interrupts 4602 ± 5% +22.7% 5649 ± 13% interrupts.CPU169.CAL:Function_call_interrupts 4920 ± 13% -21.3% 3874 ± 2% interrupts.CPU170.CAL:Function_call_interrupts 463.80 ± 11% -25.1% 347.50 ± 4% interrupts.CPU170.RES:Rescheduling_interrupts 3619 ± 2% +10.2% 3990 ± 9% interrupts.CPU36.CAL:Function_call_interrupts 5768 ± 34% +43.3% 8267 ± 2% interrupts.CPU74.NMI:Non-maskable_interrupts 5768 ± 34% +43.3% 8267 ± 2% interrupts.CPU74.PMI:Performance_monitoring_interrupts 5769 ± 34% +43.3% 8268 ± 2% interrupts.CPU75.NMI:Non-maskable_interrupts 5769 ± 34% +43.3% 8268 ± 2% interrupts.CPU75.PMI:Performance_monitoring_interrupts 5769 ± 34% +43.3% 8267 ± 2% interrupts.CPU76.NMI:Non-maskable_interrupts 5769 ± 34% +43.3% 8267 ± 2% interrupts.CPU76.PMI:Performance_monitoring_interrupts 5768 ± 34% +43.3% 8268 ± 2% interrupts.CPU77.NMI:Non-maskable_interrupts 5768 ± 34% +43.3% 8268 ± 2% interrupts.CPU77.PMI:Performance_monitoring_interrupts 3602 ± 2% +12.9% 4068 ± 13% interrupts.CPU81.CAL:Function_call_interrupts 0.05 ± 32% +121.5% 0.10 ± 39% perf-stat.i.MPKI 1.06 -0.1 0.94 perf-stat.i.branch-miss-rate% 1.107e+09 -13.3% 9.596e+08 ± 2% perf-stat.i.branch-misses 199835 -11.5% 176905 ± 2% perf-stat.i.dTLB-store-misses 9.558e+10 +9.4% 1.046e+11 ± 2% perf-stat.i.dTLB-stores 9.687e+08 -13.8% 8.345e+08 ± 2% perf-stat.i.iTLB-load-misses 535.53 +20.3% 644.34 perf-stat.i.instructions-per-iTLB-miss 1.10 ± 12% +79.6% 1.98 ± 2% perf-stat.i.metric.K/sec 1.05 -0.1 0.92 perf-stat.overall.branch-miss-rate% 0.00 ± 13% -0.0 0.00 ± 4% perf-stat.overall.dTLB-load-miss-rate% 0.00 -0.0 0.00 perf-stat.overall.dTLB-store-miss-rate% 528.82 +20.8% 638.59 perf-stat.overall.instructions-per-iTLB-miss 487060 +20.3% 586095 perf-stat.overall.path-length 1.103e+09 -13.3% 9.564e+08 ± 2% perf-stat.ps.branch-misses 199661 -11.5% 176727 ± 2% perf-stat.ps.dTLB-store-misses 9.525e+10 +9.5% 1.043e+11 ± 2% perf-stat.ps.dTLB-stores 9.67e+08 -13.9% 8.33e+08 ± 3% perf-stat.ps.iTLB-load-misses will-it-scale.per_thread_ops 1.7e+06 +----------------------------------------------------------------+ | .+.. .+.+.. .+.. .+ | 1.65e+06 |.+ .+.+ + +.+. .+.+ +. .+.+. | | +. +. + | 1.6e+06 |-+ | | | 1.55e+06 |-+ | | | 1.5e+06 |-+ | | O O O O O O | 1.45e+06 |-O O O O O O O O O O O O O O O O O O O O | | O | 1.4e+06 |-+ | | | 1.35e+06 +----------------------------------------------------------------+ will-it-scale.workload 3.3e+08 +-----------------------------------------------------------------+ | | 3.2e+08 |.+ .+.+..+.+..+.+.+.. .+..+.+. .+.+ | | +..+ +.+ +..+ | 3.1e+08 |-+ | | | 3e+08 |-+ | | | 2.9e+08 |-+ | | | 2.8e+08 |-O O O O O O O O O O O O O O O O O O O O O O O O | | O O O | 2.7e+08 |-+ | | | 2.6e+08 +-----------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-csl-2ap4: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-20191114.cgz/lkp-csl-2ap4/unix1/will-it-scale/0x5002f01 commit: v5.8-rc3 b6509f6a8c ("Revert "fs: Do not check if there is a fsnotify watcher on pseudo inodes"") v5.8-rc3 b6509f6a8c4313c068c69785c00 ---------------- --------------------------- %stddev %change %stddev \ | \ 720286 -6.4% 674507 will-it-scale.per_process_ops 1.383e+08 -6.4% 1.295e+08 will-it-scale.workload 97977 +1.4% 99342 proc-vmstat.nr_anon_pages 8.31 ± 41% +48.5% 12.33 ± 3% sched_debug.cpu.sched_goidle.min 1097 ± 8% +19.8% 1315 ± 9% slabinfo.task_group.active_objs 1097 ± 8% +19.8% 1315 ± 9% slabinfo.task_group.num_objs 13194 ± 25% -29.6% 9295 ± 24% softirqs.CPU124.RCU 9178 ± 8% -7.8% 8467 ± 4% softirqs.CPU48.RCU 1229 ± 68% +156.3% 3151 ± 70% interrupts.CPU0.CAL:Function_call_interrupts 8259 -37.6% 5157 ± 34% interrupts.CPU163.NMI:Non-maskable_interrupts 8259 -37.6% 5157 ± 34% interrupts.CPU163.PMI:Performance_monitoring_interrupts 15337 ± 81% +146.4% 37788 ± 45% numa-vmstat.node2.nr_active_anon 15187 ± 81% +147.5% 37594 ± 45% numa-vmstat.node2.nr_anon_pages 15337 ± 81% +146.4% 37788 ± 45% numa-vmstat.node2.nr_zone_active_anon 127540 ± 6% -8.0% 117311 numa-vmstat.node2.numa_other 62308 ± 81% +144.0% 152033 ± 45% numa-meminfo.node2.Active 61336 ± 81% +146.7% 151288 ± 45% numa-meminfo.node2.Active(anon) 15647 ±195% +497.8% 93536 ± 57% numa-meminfo.node2.AnonHugePages 60733 ± 82% +147.8% 150509 ± 45% numa-meminfo.node2.AnonPages 803985 ± 8% +10.5% 888495 ± 9% numa-meminfo.node2.MemUsed 1.14 -0.1 1.07 perf-stat.i.branch-miss-rate% 1.223e+09 -6.3% 1.146e+09 perf-stat.i.branch-misses 0.99 -1.5% 0.98 perf-stat.i.cpi 1.619e+11 +1.4% 1.641e+11 perf-stat.i.dTLB-loads 1e+11 +4.0% 1.04e+11 perf-stat.i.dTLB-stores 1.06e+09 -7.8% 9.779e+08 perf-stat.i.iTLB-load-misses 5.531e+11 +1.3% 5.603e+11 perf-stat.i.instructions 527.69 +9.7% 578.78 perf-stat.i.instructions-per-iTLB-miss 1.01 +1.4% 1.02 perf-stat.i.ipc 1931 +1.4% 1958 perf-stat.i.metric.M/sec 1.12 -0.1 1.06 perf-stat.overall.branch-miss-rate% 0.99 -1.4% 0.97 perf-stat.overall.cpi 0.00 -0.0 0.00 perf-stat.overall.dTLB-store-miss-rate% 521.71 +9.8% 572.92 perf-stat.overall.instructions-per-iTLB-miss 1.01 +1.4% 1.03 perf-stat.overall.ipc 1206072 +8.1% 1303735 perf-stat.overall.path-length 1.218e+09 -6.3% 1.142e+09 perf-stat.ps.branch-misses 1.613e+11 +1.4% 1.635e+11 perf-stat.ps.dTLB-loads 9.97e+10 +4.0% 1.036e+11 perf-stat.ps.dTLB-stores 1.057e+09 -7.8% 9.747e+08 perf-stat.ps.iTLB-load-misses 5.512e+11 +1.3% 5.584e+11 perf-stat.ps.instructions 1.668e+14 +1.2% 1.688e+14 perf-stat.total.instructions 38.39 -2.2 36.14 perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 36.80 -2.1 34.66 perf-profile.calltrace.cycles-pp.sock_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64 34.46 -2.0 32.44 perf-profile.calltrace.cycles-pp.sock_sendmsg.sock_write_iter.new_sync_write.vfs_write.ksys_write 33.29 -1.8 31.47 perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 30.96 -1.8 29.19 perf-profile.calltrace.cycles-pp.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write.vfs_write 31.72 -1.7 30.05 perf-profile.calltrace.cycles-pp.sock_read_iter.new_sync_read.vfs_read.ksys_read.do_syscall_64 25.95 -1.5 24.46 perf-profile.calltrace.cycles-pp.unix_stream_recvmsg.sock_read_iter.new_sync_read.vfs_read.ksys_read 53.97 -1.5 52.49 perf-profile.calltrace.cycles-pp.write 24.74 -1.4 23.30 perf-profile.calltrace.cycles-pp.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read.vfs_read 47.12 -1.0 46.10 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write 46.34 -1.0 45.38 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 44.29 -0.9 43.43 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 42.43 -0.8 41.64 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 13.05 -0.8 12.28 perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write 11.31 -0.7 10.65 perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter 10.31 -0.6 9.71 perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg 8.93 -0.5 8.39 perf-profile.calltrace.cycles-pp.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read 8.64 -0.5 8.14 perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter 8.31 -0.5 7.85 perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg 7.91 -0.5 7.45 perf-profile.calltrace.cycles-pp.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write 6.55 -0.4 6.19 perf-profile.calltrace.cycles-pp.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read 0.54 ± 2% -0.3 0.25 ±100% perf-profile.calltrace.cycles-pp.___might_sleep.__might_fault._copy_from_iter.skb_copy_datagram_from_iter.unix_stream_sendmsg 3.81 -0.2 3.58 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.write 3.98 -0.2 3.75 perf-profile.calltrace.cycles-pp.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg 3.63 -0.2 3.41 perf-profile.calltrace.cycles-pp._copy_from_iter.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg.sock_write_iter 3.88 -0.2 3.65 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.read 3.82 -0.2 3.60 perf-profile.calltrace.cycles-pp.skb_release_all.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter 2.87 -0.2 2.66 perf-profile.calltrace.cycles-pp.security_socket_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write.vfs_write 3.71 -0.2 3.50 perf-profile.calltrace.cycles-pp.skb_release_head_state.skb_release_all.consume_skb.unix_stream_read_generic.unix_stream_recvmsg 3.62 -0.2 3.42 perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic 3.91 -0.2 3.71 perf-profile.calltrace.cycles-pp.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic 2.95 -0.2 2.76 perf-profile.calltrace.cycles-pp.security_socket_recvmsg.sock_recvmsg.sock_read_iter.new_sync_read.vfs_read 3.50 -0.2 3.31 perf-profile.calltrace.cycles-pp.__check_object_size.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg.sock_write_iter 3.33 -0.2 3.15 perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb 3.44 -0.2 3.25 perf-profile.calltrace.cycles-pp.unix_destruct_scm.skb_release_head_state.skb_release_all.consume_skb.unix_stream_read_generic 3.09 -0.2 2.91 perf-profile.calltrace.cycles-pp.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg 3.65 -0.2 3.47 perf-profile.calltrace.cycles-pp.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor 2.38 -0.2 2.21 perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.08 -0.1 1.94 perf-profile.calltrace.cycles-pp.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write 2.63 -0.1 2.50 perf-profile.calltrace.cycles-pp.sock_wfree.unix_destruct_scm.skb_release_head_state.skb_release_all.consume_skb 3.44 -0.1 3.31 perf-profile.calltrace.cycles-pp.sock_recvmsg.sock_read_iter.new_sync_read.vfs_read.ksys_read 1.71 -0.1 1.59 perf-profile.calltrace.cycles-pp.common_file_perm.security_file_permission.vfs_read.ksys_read.do_syscall_64 1.69 -0.1 1.58 perf-profile.calltrace.cycles-pp.common_file_perm.security_file_permission.vfs_write.ksys_write.do_syscall_64 1.96 -0.1 1.85 perf-profile.calltrace.cycles-pp.aa_sk_perm.security_socket_recvmsg.sock_recvmsg.sock_read_iter.new_sync_read 1.44 -0.1 1.34 perf-profile.calltrace.cycles-pp.wait_for_unix_gc.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write 1.57 -0.1 1.48 perf-profile.calltrace.cycles-pp.copyin._copy_from_iter.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg 1.47 -0.1 1.39 perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor 1.20 -0.1 1.13 perf-profile.calltrace.cycles-pp.sock_def_readable.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write 0.70 -0.1 0.63 ± 4% perf-profile.calltrace.cycles-pp.__might_sleep.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb 1.10 ± 2% -0.1 1.03 ± 2% perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 0.98 -0.1 0.91 perf-profile.calltrace.cycles-pp.skb_set_owner_w.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter 1.01 ± 2% -0.1 0.94 ± 3% perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.01 -0.1 0.94 perf-profile.calltrace.cycles-pp.__might_fault._copy_from_iter.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg 1.13 -0.1 1.07 perf-profile.calltrace.cycles-pp.unix_write_space.sock_wfree.unix_destruct_scm.skb_release_head_state.skb_release_all 1.05 -0.1 0.99 ± 3% perf-profile.calltrace.cycles-pp.__ksize.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg 0.96 -0.1 0.91 perf-profile.calltrace.cycles-pp._raw_spin_unlock_irqrestore.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read 0.96 -0.1 0.90 perf-profile.calltrace.cycles-pp._raw_spin_unlock_irqrestore.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write 0.98 -0.1 0.93 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.write 1.04 -0.1 0.98 perf-profile.calltrace.cycles-pp.skb_unlink.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read 1.00 -0.1 0.94 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.read 0.80 -0.1 0.75 perf-profile.calltrace.cycles-pp._raw_spin_lock.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read 1.02 -0.0 0.97 perf-profile.calltrace.cycles-pp.skb_queue_tail.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write 0.85 -0.0 0.80 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.skb_unlink.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter 0.68 ± 2% -0.0 0.64 perf-profile.calltrace.cycles-pp.__prepare_exit_to_usermode.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 0.71 -0.0 0.67 perf-profile.calltrace.cycles-pp.___might_sleep.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags 0.70 ± 2% -0.0 0.66 ± 2% perf-profile.calltrace.cycles-pp.copy_user_generic_unrolled.copyin._copy_from_iter.skb_copy_datagram_from_iter.unix_stream_sendmsg 0.85 -0.0 0.81 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.skb_queue_tail.unix_stream_sendmsg.sock_sendmsg.sock_write_iter 0.56 -0.0 0.52 ± 2% perf-profile.calltrace.cycles-pp.apparmor_socket_recvmsg.security_socket_recvmsg.sock_recvmsg.sock_read_iter.new_sync_read 0.57 -0.0 0.54 perf-profile.calltrace.cycles-pp.___might_sleep.wait_for_unix_gc.unix_stream_sendmsg.sock_sendmsg.sock_write_iter 0.68 -0.0 0.64 ± 2% perf-profile.calltrace.cycles-pp.copy_user_generic_unrolled.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter 0.57 -0.0 0.54 perf-profile.calltrace.cycles-pp.__might_sleep.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags 1.68 -0.0 1.64 perf-profile.calltrace.cycles-pp.mutex_lock.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read 0.65 -0.0 0.62 perf-profile.calltrace.cycles-pp.mutex_unlock.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read 0.00 +0.6 0.57 perf-profile.calltrace.cycles-pp.fsnotify_parent.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +0.6 0.62 perf-profile.calltrace.cycles-pp.fsnotify_parent.security_file_permission.vfs_read.ksys_read.do_syscall_64 0.00 +0.6 0.64 perf-profile.calltrace.cycles-pp.fsnotify_parent.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.50 +0.7 3.20 perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +0.9 0.93 perf-profile.calltrace.cycles-pp.fsnotify.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 48.49 +1.3 49.77 perf-profile.calltrace.cycles-pp.read 41.56 +1.7 43.30 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read 40.78 +1.8 42.56 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 38.81 +1.9 40.68 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 37.05 +1.9 38.94 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 0.00 +1.9 1.90 perf-profile.calltrace.cycles-pp.fsnotify.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 38.49 -2.2 36.24 perf-profile.children.cycles-pp.new_sync_write 36.86 -2.1 34.71 perf-profile.children.cycles-pp.sock_write_iter 34.57 -2.0 32.53 perf-profile.children.cycles-pp.sock_sendmsg 33.36 -1.8 31.55 perf-profile.children.cycles-pp.new_sync_read 31.28 -1.8 29.50 perf-profile.children.cycles-pp.unix_stream_sendmsg 31.77 -1.7 30.10 perf-profile.children.cycles-pp.sock_read_iter 26.00 -1.5 24.50 perf-profile.children.cycles-pp.unix_stream_recvmsg 54.00 -1.5 52.53 perf-profile.children.cycles-pp.write 24.99 -1.5 23.53 perf-profile.children.cycles-pp.unix_stream_read_generic 44.34 -0.9 43.48 perf-profile.children.cycles-pp.ksys_write 13.14 -0.8 12.36 perf-profile.children.cycles-pp.sock_alloc_send_pskb 42.57 -0.7 41.85 perf-profile.children.cycles-pp.vfs_write 11.37 -0.7 10.71 perf-profile.children.cycles-pp.alloc_skb_with_frags 10.51 -0.6 9.90 perf-profile.children.cycles-pp.__alloc_skb 9.00 -0.5 8.46 perf-profile.children.cycles-pp.unix_stream_read_actor 8.72 -0.5 8.21 perf-profile.children.cycles-pp.skb_copy_datagram_iter 8.00 -0.5 7.53 perf-profile.children.cycles-pp.skb_copy_datagram_from_iter 8.38 -0.5 7.91 perf-profile.children.cycles-pp.__skb_datagram_iter 7.43 -0.4 7.04 perf-profile.children.cycles-pp.__check_object_size 6.57 -0.4 6.21 perf-profile.children.cycles-pp.consume_skb 5.14 -0.3 4.83 perf-profile.children.cycles-pp.entry_SYSCALL_64 4.60 -0.3 4.32 perf-profile.children.cycles-pp.syscall_return_via_sysret 4.16 -0.3 3.89 perf-profile.children.cycles-pp.aa_sk_perm 4.22 -0.3 3.96 perf-profile.children.cycles-pp.___might_sleep 3.50 -0.2 3.26 perf-profile.children.cycles-pp.common_file_perm 4.07 -0.2 3.83 perf-profile.children.cycles-pp.__kmalloc_reserve 3.71 -0.2 3.48 perf-profile.children.cycles-pp._copy_from_iter 3.85 -0.2 3.63 perf-profile.children.cycles-pp.skb_release_all 2.95 -0.2 2.73 perf-profile.children.cycles-pp.__might_sleep 2.91 -0.2 2.70 perf-profile.children.cycles-pp.security_socket_sendmsg 3.72 -0.2 3.51 perf-profile.children.cycles-pp._copy_to_iter 3.76 -0.2 3.55 perf-profile.children.cycles-pp.skb_release_head_state 3.01 -0.2 2.81 perf-profile.children.cycles-pp.security_socket_recvmsg 3.54 -0.2 3.33 perf-profile.children.cycles-pp.__kmalloc_node_track_caller 3.97 -0.2 3.77 perf-profile.children.cycles-pp.simple_copy_to_iter 3.28 -0.2 3.09 perf-profile.children.cycles-pp.kmem_cache_alloc_node 3.54 -0.2 3.35 perf-profile.children.cycles-pp.unix_destruct_scm 2.36 ± 2% -0.2 2.21 ± 2% perf-profile.children.cycles-pp.__fdget_pos 2.64 -0.1 2.50 perf-profile.children.cycles-pp.sock_wfree 3.50 -0.1 3.37 perf-profile.children.cycles-pp.sock_recvmsg 2.01 ± 2% -0.1 1.88 ± 3% perf-profile.children.cycles-pp.__fget_light 2.10 ± 2% -0.1 1.97 ± 2% perf-profile.children.cycles-pp.__might_fault 1.97 -0.1 1.85 perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 1.54 -0.1 1.44 perf-profile.children.cycles-pp.wait_for_unix_gc 1.95 -0.1 1.85 perf-profile.children.cycles-pp._raw_spin_lock_irqsave 1.43 -0.1 1.34 perf-profile.children.cycles-pp._cond_resched 1.60 -0.1 1.51 perf-profile.children.cycles-pp.copyin 1.52 -0.1 1.43 perf-profile.children.cycles-pp.__prepare_exit_to_usermode 1.67 ± 3% -0.1 1.58 ± 2% perf-profile.children.cycles-pp._raw_spin_lock 1.64 -0.1 1.55 perf-profile.children.cycles-pp.copy_user_generic_unrolled 1.50 -0.1 1.42 perf-profile.children.cycles-pp.copyout 1.21 -0.1 1.13 perf-profile.children.cycles-pp.sock_def_readable 0.98 -0.1 0.91 perf-profile.children.cycles-pp.skb_set_owner_w 0.95 -0.1 0.89 perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 1.05 -0.1 0.99 ± 2% perf-profile.children.cycles-pp.__ksize 0.84 -0.1 0.78 ± 2% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 1.13 -0.1 1.08 perf-profile.children.cycles-pp.unix_write_space 1.05 -0.1 0.99 perf-profile.children.cycles-pp.skb_unlink 0.74 ± 2% -0.1 0.69 perf-profile.children.cycles-pp.apparmor_file_permission 1.02 -0.1 0.97 perf-profile.children.cycles-pp.skb_queue_tail 1.75 -0.0 1.70 perf-profile.children.cycles-pp.mutex_lock 0.56 -0.0 0.52 ± 2% perf-profile.children.cycles-pp.apparmor_socket_recvmsg 0.71 -0.0 0.67 perf-profile.children.cycles-pp.rcu_all_qs 0.67 -0.0 0.63 perf-profile.children.cycles-pp.__syscall_return_slowpath 0.67 -0.0 0.64 perf-profile.children.cycles-pp.check_stack_object 0.44 -0.0 0.41 perf-profile.children.cycles-pp.apparmor_socket_sendmsg 0.67 -0.0 0.65 perf-profile.children.cycles-pp.mutex_unlock 0.30 -0.0 0.28 ± 2% perf-profile.children.cycles-pp.put_pid 0.23 -0.0 0.21 ± 2% perf-profile.children.cycles-pp.skb_release_data 0.43 +0.0 0.46 perf-profile.children.cycles-pp.apparmor_socket_getpeersec_dgram 4.96 +0.6 5.57 perf-profile.children.cycles-pp.security_file_permission 88.79 +0.7 89.51 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 87.30 +0.8 88.11 perf-profile.children.cycles-pp.do_syscall_64 48.55 +1.3 49.85 perf-profile.children.cycles-pp.read 0.00 +1.9 1.87 perf-profile.children.cycles-pp.fsnotify_parent 38.85 +1.9 40.72 perf-profile.children.cycles-pp.ksys_read 37.17 +2.0 39.17 perf-profile.children.cycles-pp.vfs_read 0.00 +2.9 2.90 perf-profile.children.cycles-pp.fsnotify 4.56 -0.3 4.28 perf-profile.self.cycles-pp.syscall_return_via_sysret 4.10 -0.2 3.85 perf-profile.self.cycles-pp.___might_sleep 2.96 -0.2 2.75 perf-profile.self.cycles-pp.unix_stream_read_generic 2.65 -0.2 2.46 perf-profile.self.cycles-pp.__might_sleep 2.94 -0.2 2.75 perf-profile.self.cycles-pp.unix_stream_sendmsg 2.85 ± 2% -0.2 2.67 perf-profile.self.cycles-pp.common_file_perm 1.40 ± 4% -0.2 1.22 perf-profile.self.cycles-pp.new_sync_read 2.32 -0.2 2.17 perf-profile.self.cycles-pp.aa_sk_perm 2.56 -0.1 2.41 perf-profile.self.cycles-pp.entry_SYSCALL_64 1.90 ± 2% -0.1 1.77 ± 3% perf-profile.self.cycles-pp.__fget_light 2.13 -0.1 2.00 perf-profile.self.cycles-pp.__alloc_skb 2.29 -0.1 2.17 perf-profile.self.cycles-pp.sock_write_iter 1.93 -0.1 1.81 perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 1.79 -0.1 1.67 perf-profile.self.cycles-pp.write 1.83 -0.1 1.72 perf-profile.self.cycles-pp.read 1.52 -0.1 1.42 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 1.71 -0.1 1.61 perf-profile.self.cycles-pp.__kmalloc_node_track_caller 1.93 -0.1 1.84 perf-profile.self.cycles-pp._raw_spin_lock_irqsave 1.60 ± 3% -0.1 1.52 ± 2% perf-profile.self.cycles-pp._raw_spin_lock 1.81 -0.1 1.73 perf-profile.self.cycles-pp.kmem_cache_alloc_node 1.17 -0.1 1.09 perf-profile.self.cycles-pp.sock_def_readable 1.55 -0.1 1.47 perf-profile.self.cycles-pp.copy_user_generic_unrolled 1.25 ± 2% -0.1 1.17 ± 2% perf-profile.self.cycles-pp._copy_to_iter 0.61 -0.1 0.54 perf-profile.self.cycles-pp.ksys_write 1.44 -0.1 1.37 perf-profile.self.cycles-pp.sock_wfree 0.95 -0.1 0.88 perf-profile.self.cycles-pp.skb_set_owner_w 1.06 -0.1 0.99 ± 2% perf-profile.self.cycles-pp._copy_from_iter 0.58 ± 5% -0.1 0.52 ± 5% perf-profile.self.cycles-pp.aa_file_perm 1.04 -0.1 0.97 ± 2% perf-profile.self.cycles-pp.__ksize 1.10 -0.1 1.04 perf-profile.self.cycles-pp.unix_write_space 0.68 ± 2% -0.1 0.62 perf-profile.self.cycles-pp.__skb_datagram_iter 0.93 -0.1 0.87 perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 0.52 -0.1 0.47 perf-profile.self.cycles-pp.ksys_read 0.68 ± 2% -0.1 0.63 ± 2% perf-profile.self.cycles-pp.apparmor_file_permission 0.83 -0.0 0.78 perf-profile.self.cycles-pp.unix_destruct_scm 0.87 -0.0 0.82 perf-profile.self.cycles-pp.consume_skb 0.83 -0.0 0.79 perf-profile.self.cycles-pp.sock_alloc_send_pskb 0.71 -0.0 0.67 perf-profile.self.cycles-pp._cond_resched 0.82 -0.0 0.78 perf-profile.self.cycles-pp.alloc_skb_with_frags 0.29 ± 3% -0.0 0.25 perf-profile.self.cycles-pp.unix_stream_read_actor 0.66 -0.0 0.62 perf-profile.self.cycles-pp.skb_copy_datagram_from_iter 0.53 -0.0 0.49 ± 2% perf-profile.self.cycles-pp.apparmor_socket_recvmsg 0.64 -0.0 0.60 perf-profile.self.cycles-pp.__syscall_return_slowpath 0.60 ± 2% -0.0 0.56 ± 4% perf-profile.self.cycles-pp.__might_fault 0.50 -0.0 0.46 perf-profile.self.cycles-pp.__kmalloc_reserve 0.37 -0.0 0.34 perf-profile.self.cycles-pp.security_socket_recvmsg 2.25 -0.0 2.21 perf-profile.self.cycles-pp.sock_read_iter 0.54 -0.0 0.51 perf-profile.self.cycles-pp.rcu_all_qs 0.37 ± 5% -0.0 0.34 ± 4% perf-profile.self.cycles-pp.sock_sendmsg 0.43 -0.0 0.40 perf-profile.self.cycles-pp.__x86_indirect_thunk_rax 0.98 -0.0 0.95 perf-profile.self.cycles-pp.unix_stream_recvmsg 0.35 -0.0 0.32 perf-profile.self.cycles-pp.skb_copy_datagram_iter 0.41 -0.0 0.38 perf-profile.self.cycles-pp.apparmor_socket_sendmsg 0.56 -0.0 0.54 perf-profile.self.cycles-pp.check_stack_object 0.44 -0.0 0.41 perf-profile.self.cycles-pp.wait_for_unix_gc 0.29 ± 3% -0.0 0.26 ± 2% perf-profile.self.cycles-pp.security_socket_sendmsg 0.42 -0.0 0.40 perf-profile.self.cycles-pp.security_socket_getpeersec_dgram 0.63 -0.0 0.61 perf-profile.self.cycles-pp.mutex_unlock 0.40 ± 2% -0.0 0.38 perf-profile.self.cycles-pp.__fdget_pos 0.31 -0.0 0.29 perf-profile.self.cycles-pp.copyin 0.23 -0.0 0.21 ± 2% perf-profile.self.cycles-pp.skb_release_data 0.19 -0.0 0.17 ± 2% perf-profile.self.cycles-pp.skb_release_head_state 0.24 -0.0 0.23 perf-profile.self.cycles-pp.copyout 0.21 ± 2% -0.0 0.19 ± 2% perf-profile.self.cycles-pp.simple_copy_to_iter 0.26 -0.0 0.25 perf-profile.self.cycles-pp.kmalloc_slab 0.17 ± 2% +0.0 0.20 ± 2% perf-profile.self.cycles-pp.__vfs_read 0.40 +0.0 0.43 perf-profile.self.cycles-pp.apparmor_socket_getpeersec_dgram 0.73 +0.0 0.77 perf-profile.self.cycles-pp.mutex_lock 0.51 +0.1 0.57 perf-profile.self.cycles-pp.sock_recvmsg 1.33 +0.2 1.57 perf-profile.self.cycles-pp.vfs_write 0.75 ± 6% +0.4 1.10 ± 3% perf-profile.self.cycles-pp.security_file_permission 0.97 +0.5 1.49 ± 2% perf-profile.self.cycles-pp.vfs_read 0.00 +1.7 1.65 perf-profile.self.cycles-pp.fsnotify_parent 0.00 +2.8 2.80 perf-profile.self.cycles-pp.fsnotify *************************************************************************************************** lkp-csl-2ap4: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-csl-2ap4/pipe1/will-it-scale/0x5002f01 commit: v5.8-rc3 b6509f6a8c ("Revert "fs: Do not check if there is a fsnotify watcher on pseudo inodes"") v5.8-rc3 b6509f6a8c4313c068c69785c00 ---------------- --------------------------- %stddev %change %stddev \ | \ 1045643 -2.3% 1021331 will-it-scale.per_thread_ops 2.008e+08 -2.3% 1.961e+08 will-it-scale.workload 29490176 ± 3% -9.8% 26607616 ± 3% meminfo.DirectMap2M 202721 ± 4% +11.5% 225943 ± 3% numa-numastat.node2.local_node 25892 ± 28% -75.5% 6341 ± 97% numa-numastat.node2.other_node 1843 +8.9% 2007 ± 5% numa-vmstat.node2.nr_mapped 130681 ± 5% -14.7% 111436 ± 5% numa-vmstat.node2.numa_other 9801 ± 5% +16.0% 11370 ± 7% sched_debug.cpu.ttwu_count.max 3052 ± 12% +23.6% 3771 ± 11% sched_debug.cpu.ttwu_local.max 136156 +1.4% 138002 proc-vmstat.nr_active_anon 109009 +1.1% 110229 proc-vmstat.nr_anon_pages 136156 +1.4% 138002 proc-vmstat.nr_zone_active_anon 1120468 +1.6% 1137854 proc-vmstat.pgalloc_normal 337.83 ± 61% +53.8% 519.75 ± 47% interrupts.39:PCI-MSI.524290-edge.eth0-TxRx-1 337.83 ± 61% +53.8% 519.75 ± 47% interrupts.CPU11.39:PCI-MSI.524290-edge.eth0-TxRx-1 303.33 +45.7% 442.00 ± 44% interrupts.CPU148.RES:Rescheduling_interrupts 5509 ± 35% +50.1% 8271 interrupts.CPU155.NMI:Non-maskable_interrupts 5509 ± 35% +50.1% 8271 interrupts.CPU155.PMI:Performance_monitoring_interrupts 5509 ± 35% +50.1% 8271 interrupts.CPU157.NMI:Non-maskable_interrupts 5509 ± 35% +50.1% 8271 interrupts.CPU157.PMI:Performance_monitoring_interrupts 5509 ± 35% +50.1% 8271 interrupts.CPU158.NMI:Non-maskable_interrupts 5509 ± 35% +50.1% 8271 interrupts.CPU158.PMI:Performance_monitoring_interrupts 15130 ± 14% -18.1% 12396 ± 8% interrupts.CPU168.CAL:Function_call_interrupts 4137 ± 15% +44.3% 5970 ± 26% interrupts.CPU48.CAL:Function_call_interrupts 336.00 ± 6% +60.0% 537.75 ± 31% interrupts.CPU48.RES:Rescheduling_interrupts 0.75 -0.0 0.73 perf-stat.i.branch-miss-rate% 8.192e+08 -2.2% 8.01e+08 perf-stat.i.branch-misses 0.98 -1.4% 0.97 perf-stat.i.cpi 1.7e+11 +1.5% 1.725e+11 perf-stat.i.dTLB-loads 1.055e+11 +2.7% 1.084e+11 perf-stat.i.dTLB-stores 7.304e+08 -2.5% 7.123e+08 perf-stat.i.iTLB-load-misses 5.6e+11 +1.6% 5.688e+11 perf-stat.i.instructions 775.04 +4.2% 807.76 perf-stat.i.instructions-per-iTLB-miss 1.02 +1.5% 1.04 perf-stat.i.ipc 2015 +1.5% 2046 perf-stat.i.metric.M/sec 0.73 -0.0 0.72 perf-stat.overall.branch-miss-rate% 0.98 -1.5% 0.96 perf-stat.overall.cpi 0.00 -0.0 0.00 perf-stat.overall.dTLB-store-miss-rate% 764.55 +4.1% 796.24 perf-stat.overall.instructions-per-iTLB-miss 1.02 +1.5% 1.04 perf-stat.overall.ipc 841233 +4.0% 874479 perf-stat.overall.path-length 8.164e+08 -2.2% 7.983e+08 perf-stat.ps.branch-misses 1.695e+11 +1.5% 1.72e+11 perf-stat.ps.dTLB-loads 1.052e+11 +2.7% 1.08e+11 perf-stat.ps.dTLB-stores 7.301e+08 -2.5% 7.119e+08 perf-stat.ps.iTLB-load-misses 5.582e+11 +1.6% 5.668e+11 perf-stat.ps.instructions 1.689e+14 +1.5% 1.715e+14 perf-stat.total.instructions 50.20 -1.4 48.82 perf-profile.calltrace.cycles-pp.__GI___libc_read 40.68 -1.2 39.52 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_read 39.53 -1.1 38.39 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read 36.75 -1.1 35.70 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read 32.27 -0.9 31.36 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read 24.69 -0.6 24.14 perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 22.23 -0.5 21.72 perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64 21.04 -0.5 20.55 perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 18.70 -0.4 18.27 perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64 6.80 -0.1 6.65 perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.new_sync_write.vfs_write.ksys_write 5.39 -0.1 5.26 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__GI___libc_write 2.96 -0.1 2.84 ± 2% perf-profile.calltrace.cycles-pp.fsnotify.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.27 -0.1 2.15 perf-profile.calltrace.cycles-pp.copyin.copy_page_from_iter.pipe_write.new_sync_write.vfs_write 2.42 -0.1 2.31 ± 2% perf-profile.calltrace.cycles-pp.copyout.copy_page_to_iter.pipe_read.new_sync_read.vfs_read 5.38 -0.1 5.28 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__GI___libc_read 2.61 -0.1 2.52 perf-profile.calltrace.cycles-pp.mutex_lock.pipe_read.new_sync_read.vfs_read.ksys_read 3.46 -0.1 3.38 perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write 2.62 -0.1 2.55 perf-profile.calltrace.cycles-pp.mutex_lock.pipe_write.new_sync_write.vfs_write.ksys_write 2.08 -0.1 2.02 perf-profile.calltrace.cycles-pp.__pthread_disable_asynccancel 0.99 -0.1 0.94 perf-profile.calltrace.cycles-pp.fput_many.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write 1.04 -0.0 1.00 perf-profile.calltrace.cycles-pp.__prepare_exit_to_usermode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write 0.97 -0.0 0.93 ± 2% perf-profile.calltrace.cycles-pp.copy_user_generic_unrolled.copyin.copy_page_from_iter.pipe_write.new_sync_write 1.23 -0.0 1.19 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write 1.11 -0.0 1.08 perf-profile.calltrace.cycles-pp.copy_user_generic_unrolled.copyout.copy_page_to_iter.pipe_read.new_sync_read 1.20 -0.0 1.17 perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.pipe_write.new_sync_write.vfs_write.ksys_write 1.01 -0.0 0.98 perf-profile.calltrace.cycles-pp.__prepare_exit_to_usermode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read 0.79 -0.0 0.76 perf-profile.calltrace.cycles-pp.___might_sleep.mutex_lock.pipe_read.new_sync_read.vfs_read 1.50 -0.0 1.47 perf-profile.calltrace.cycles-pp.__might_fault.copy_page_from_iter.pipe_write.new_sync_write.vfs_write 1.13 -0.0 1.11 perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.pipe_read.new_sync_read.vfs_read.ksys_read 0.79 -0.0 0.77 perf-profile.calltrace.cycles-pp.___might_sleep.copy_page_from_iter.pipe_write.new_sync_write.vfs_write 0.72 -0.0 0.70 perf-profile.calltrace.cycles-pp.__sb_end_write.pipe_write.new_sync_write.vfs_write.ksys_write 0.99 -0.0 0.97 perf-profile.calltrace.cycles-pp.mutex_unlock.pipe_read.new_sync_read.vfs_read.ksys_read 0.00 +0.9 0.92 perf-profile.calltrace.cycles-pp.fsnotify_parent.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 49.09 +1.4 50.48 perf-profile.calltrace.cycles-pp.__GI___libc_write 0.00 +1.5 1.50 ± 5% perf-profile.calltrace.cycles-pp.fsnotify.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 39.33 +1.6 40.95 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_write 38.18 +1.7 39.84 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write 35.19 +1.8 37.00 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write 30.77 +1.8 32.61 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write 50.18 -1.4 48.80 perf-profile.children.cycles-pp.__GI___libc_read 36.87 -1.1 35.81 perf-profile.children.cycles-pp.ksys_read 32.66 -0.9 31.73 perf-profile.children.cycles-pp.vfs_read 24.84 -0.6 24.29 perf-profile.children.cycles-pp.new_sync_write 22.51 -0.5 22.00 perf-profile.children.cycles-pp.pipe_write 21.18 -0.5 20.69 perf-profile.children.cycles-pp.new_sync_read 18.94 -0.4 18.50 perf-profile.children.cycles-pp.pipe_read 5.44 -0.2 5.27 perf-profile.children.cycles-pp.mutex_lock 7.06 -0.2 6.90 perf-profile.children.cycles-pp.copy_page_from_iter 6.34 -0.2 6.19 perf-profile.children.cycles-pp.syscall_return_via_sysret 7.28 -0.1 7.13 perf-profile.children.cycles-pp.entry_SYSCALL_64 2.33 -0.1 2.21 perf-profile.children.cycles-pp.copyin 7.14 -0.1 7.03 perf-profile.children.cycles-pp.copy_page_to_iter 2.48 -0.1 2.36 ± 2% perf-profile.children.cycles-pp.copyout 4.78 -0.1 4.67 perf-profile.children.cycles-pp.___might_sleep 2.48 -0.1 2.39 perf-profile.children.cycles-pp.copy_user_generic_unrolled 2.23 -0.1 2.14 perf-profile.children.cycles-pp.__prepare_exit_to_usermode 3.56 -0.1 3.48 perf-profile.children.cycles-pp.__wake_up_common_lock 2.69 -0.1 2.63 perf-profile.children.cycles-pp.__might_sleep 2.41 -0.1 2.35 perf-profile.children.cycles-pp._raw_spin_lock_irq 2.08 -0.1 2.02 perf-profile.children.cycles-pp.__pthread_disable_asynccancel 1.88 -0.1 1.82 perf-profile.children.cycles-pp.fput_many 1.40 -0.0 1.35 perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.34 -0.0 0.29 perf-profile.children.cycles-pp.fput 1.38 -0.0 1.34 perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 0.24 ± 7% -0.0 0.22 perf-profile.children.cycles-pp.tick_sched_handle 0.92 -0.0 0.90 perf-profile.children.cycles-pp.fpregs_assert_state_consistent 1.28 -0.0 1.26 perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.75 -0.0 0.73 perf-profile.children.cycles-pp.__sb_end_write 80.17 +0.5 80.63 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 77.95 +0.5 78.47 perf-profile.children.cycles-pp.do_syscall_64 1.98 +0.9 2.88 perf-profile.children.cycles-pp.fsnotify_parent 3.04 +1.4 4.42 ± 2% perf-profile.children.cycles-pp.fsnotify 49.09 +1.4 50.48 perf-profile.children.cycles-pp.__GI___libc_write 35.45 +1.8 37.22 perf-profile.children.cycles-pp.ksys_write 30.96 +2.0 32.93 perf-profile.children.cycles-pp.vfs_write 6.28 -0.1 6.13 perf-profile.self.cycles-pp.syscall_return_via_sysret 4.68 -0.1 4.58 perf-profile.self.cycles-pp.___might_sleep 2.35 -0.1 2.26 perf-profile.self.cycles-pp.copy_user_generic_unrolled 2.26 -0.1 2.18 perf-profile.self.cycles-pp.mutex_lock 2.61 -0.1 2.54 perf-profile.self.cycles-pp.__GI___libc_write 3.09 -0.1 3.02 perf-profile.self.cycles-pp.pipe_write 2.57 -0.1 2.50 perf-profile.self.cycles-pp.__GI___libc_read 2.26 -0.1 2.19 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 3.74 -0.1 3.68 perf-profile.self.cycles-pp.entry_SYSCALL_64 0.93 -0.1 0.87 perf-profile.self.cycles-pp.ksys_write 1.38 -0.1 1.32 perf-profile.self.cycles-pp.__prepare_exit_to_usermode 2.31 -0.1 2.25 perf-profile.self.cycles-pp._raw_spin_lock_irq 1.78 -0.1 1.72 perf-profile.self.cycles-pp.fput_many 1.99 -0.1 1.94 perf-profile.self.cycles-pp.__pthread_disable_asynccancel 2.37 -0.0 2.32 perf-profile.self.cycles-pp.__might_sleep 1.14 -0.0 1.10 perf-profile.self.cycles-pp.do_syscall_64 1.36 -0.0 1.31 perf-profile.self.cycles-pp._raw_spin_lock_irqsave 2.22 -0.0 2.18 perf-profile.self.cycles-pp.__pthread_enable_asynccancel 1.87 -0.0 1.83 perf-profile.self.cycles-pp.mutex_unlock 1.34 -0.0 1.31 perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 0.91 -0.0 0.89 perf-profile.self.cycles-pp.file_update_time 0.85 -0.0 0.82 perf-profile.self.cycles-pp.fpregs_assert_state_consistent 0.44 -0.0 0.41 ± 2% perf-profile.self.cycles-pp.touch_atime 0.73 -0.0 0.71 perf-profile.self.cycles-pp.__sb_end_write 1.23 -0.0 1.21 perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.21 -0.0 0.20 perf-profile.self.cycles-pp.kill_fasync 2.08 ± 2% +0.4 2.49 ± 4% perf-profile.self.cycles-pp.vfs_write 1.71 +0.8 2.51 perf-profile.self.cycles-pp.fsnotify_parent 2.94 +1.3 4.25 ± 2% perf-profile.self.cycles-pp.fsnotify Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

3 months

3
3
0 / 0

[x86/copy_mc] a0ac629ebe: fio.read_iops -43.3% regression

by kernel test robot

Greeting, FYI, we noticed a -43.3% regression of fio.read_iops due to commit: commit: a0ac629ebe7b3d248cb93807782a00d9142fdb98 ("x86/copy_mc: Introduce copy_mc_generic()") url: https://github.com/0day-ci/linux/commits/Dan-Williams/Renovate-memcpy_mcs... in testcase: fio-basic on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory with following parameters: disk: 2pmem fs: xfs mount_option: dax runtime: 200s nr_task: 50% time_based: tb rw: read bs: 2M ioengine: libaio test_size: 200G cpufreq_governor: performance ucode: 0x5002f01 test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user. test-url: https://github.com/axboe/fio In addition to that, the commit also has significant impact on the following tests: +------------------+----------------------------------------------------------------------+ | testcase: change | fio-basic: fio.read_iops -55.6% regression | | test machine | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory | | test parameters | bs=2M | | | cpufreq_governor=performance | | | disk=2pmem | | | fs=xfs | | | ioengine=sync | | | mount_option=dax | | | nr_task=50% | | | runtime=200s | | | rw=read | | | test_size=200G | | | time_based=tb | | | ucode=0x5002f01 | +------------------+----------------------------------------------------------------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/mount_option/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode: 2M/gcc-9/performance/2pmem/xfs/libaio/x86_64-rhel-8.3/dax/50%/debian-10.4-x86_64-20200603.cgz/200s/read/lkp-csl-2sp6/200G/fio-basic/tb/0x5002f01 commit: 7476b91d4d ("x86, powerpc: Rename memcpy_mcsafe() to copy_mc_to_{user, kernel}()") a0ac629ebe ("x86/copy_mc: Introduce copy_mc_generic()") 7476b91d4db369d8 a0ac629ebe7b3d248cb93807782 ---------------- --------------------------- %stddev %change %stddev \ | \ 97.22 -96.0 1.19 ± 21% fio.latency_100ms% 0.14 -0.1 0.05 fio.latency_10ms% 0.27 ± 13% -0.1 0.14 fio.latency_20ms% 0.04 ± 6% -0.0 0.03 ± 12% fio.latency_20us% 1.00 ± 28% +96.6 97.57 fio.latency_250ms% 0.05 -0.0 0.05 fio.latency_4ms% 0.02 ± 48% +0.3 0.31 ± 15% fio.latency_500ms% 1.25 ± 47% -0.6 0.63 ± 11% fio.latency_50ms% 0.01 ± 9% +0.0 0.02 ± 24% fio.latency_50us% 44292 -43.3% 25124 fio.read_bw_MBps 67895296 +76.8% 1.201e+08 fio.read_clat_90%_us 68681728 +76.7% 1.214e+08 fio.read_clat_95%_us 98304000 ± 19% +80.3% 1.772e+08 ± 4% fio.read_clat_99%_us 66674508 +76.2% 1.175e+08 fio.read_clat_mean_us 9950116 ± 12% +80.3% 17935634 fio.read_clat_stddev 22146 -43.3% 12562 fio.read_iops 2152824 +76.8% 3805428 fio.read_slat_mean_us 291719 ± 14% +86.6% 544324 fio.read_slat_stddev 12923 -2.5% 12594 fio.time.involuntary_context_switches 77.65 ± 3% -39.1% 47.29 fio.time.user_time 4429275 -43.3% 2512537 fio.workload 0.14 ± 3% +0.0 0.16 ± 4% mpstat.cpu.all.soft% 0.47 ± 3% -0.2 0.31 mpstat.cpu.all.usr% 53185 ± 91% +121.2% 117642 ± 40% numa-vmstat.node0.numa_other 122640 ± 39% -52.6% 58092 ± 81% numa-vmstat.node1.numa_other 60096 +1.5% 61021 proc-vmstat.nr_slab_unreclaimable 20103 ± 5% -17.9% 16495 ± 12% proc-vmstat.pgactivate 49.00 -2.0% 48.00 vmstat.cpu.id 1612 -1.6% 1587 vmstat.system.cs 2713 ± 4% +8.0% 2931 ± 4% slabinfo.PING.active_objs 2713 ± 4% +8.0% 2931 ± 4% slabinfo.PING.num_objs 1164 ± 9% +16.8% 1360 ± 6% slabinfo.task_group.active_objs 1164 ± 9% +16.8% 1360 ± 6% slabinfo.task_group.num_objs 379.25 ± 85% +279.7% 1439 ± 75% sched_debug.cfs_rq:/.exec_clock.min 29948 ± 5% -15.5% 25309 ± 5% sched_debug.cfs_rq:/.exec_clock.stddev 21606 ± 7% +25.1% 27034 ± 7% sched_debug.cfs_rq:/.min_vruntime.min 33321 ± 6% -16.5% 27820 ± 6% sched_debug.cfs_rq:/.min_vruntime.stddev 13783 ±109% +184.1% 39158 ± 20% sched_debug.cfs_rq:/.spread0.avg -38497 -76.6% -9012 sched_debug.cfs_rq:/.spread0.min 33321 ± 6% -16.5% 27820 ± 6% sched_debug.cfs_rq:/.spread0.stddev 12.22 ± 10% +27.9% 15.62 ± 3% sched_debug.cpu.clock.stddev 3716 ±173% -100.0% 1.50 ± 57% softirqs.CPU10.NET_RX 17411 ± 36% -41.8% 10126 ± 19% softirqs.CPU24.SCHED 9179 ± 67% +87.1% 17173 ± 23% softirqs.CPU35.SCHED 9611 ± 34% -58.9% 3951 ± 10% softirqs.CPU48.SCHED 17177 ± 30% -42.6% 9864 ± 37% softirqs.CPU69.SCHED 86644 ± 29% -22.3% 67339 ± 5% softirqs.CPU76.TIMER 6339 ± 66% +115.9% 13686 ± 31% softirqs.CPU78.SCHED 10156 ± 64% +91.8% 19477 ± 25% softirqs.CPU81.SCHED 1239 ±172% -100.0% 0.00 interrupts.62:PCI-MSI.31981595-edge.i40e-eth0-TxRx-26 47482 +5.4% 50055 ± 4% interrupts.CAL:Function_call_interrupts 209.00 ± 23% -50.4% 103.75 ± 8% interrupts.CPU0.RES:Rescheduling_interrupts 146.25 ± 16% -27.4% 106.25 ± 16% interrupts.CPU15.RES:Rescheduling_interrupts 168.75 ± 81% -64.6% 59.75 ± 33% interrupts.CPU15.TLB:TLB_shootdowns 7321 ± 5% -52.7% 3461 ± 39% interrupts.CPU20.NMI:Non-maskable_interrupts 7321 ± 5% -52.7% 3461 ± 39% interrupts.CPU20.PMI:Performance_monitoring_interrupts 6665 ± 14% -61.2% 2586 ± 26% interrupts.CPU21.NMI:Non-maskable_interrupts 6665 ± 14% -61.2% 2586 ± 26% interrupts.CPU21.PMI:Performance_monitoring_interrupts 64.50 ± 23% +41.9% 91.50 ± 22% interrupts.CPU21.TLB:TLB_shootdowns 100.00 ± 41% +66.0% 166.00 ± 9% interrupts.CPU24.RES:Rescheduling_interrupts 1238 ±173% -100.0% 0.00 interrupts.CPU26.62:PCI-MSI.31981595-edge.i40e-eth0-TxRx-26 438.25 ± 4% +16.1% 509.00 ± 18% interrupts.CPU28.CAL:Function_call_interrupts 145.50 ± 20% -34.4% 95.50 ± 25% interrupts.CPU35.RES:Rescheduling_interrupts 7134 ± 11% -28.3% 5118 ± 19% interrupts.CPU41.NMI:Non-maskable_interrupts 7134 ± 11% -28.3% 5118 ± 19% interrupts.CPU41.PMI:Performance_monitoring_interrupts 107.75 ± 34% -47.3% 56.75 ± 40% interrupts.CPU93.RES:Rescheduling_interrupts 63.18 ± 12% -26.1 37.12 ± 15% perf-profile.calltrace.cycles-pp.copy_mc_fragile.copy_mc_to_user.copyout_mc._copy_mc_to_iter.dax_iomap_actor 0.00 +3.7 3.72 ± 52% perf-profile.calltrace.cycles-pp.copy_mc_generic.copy_mc_to_user.copyout_mc._copy_mc_to_iter.dax_iomap_actor 0.00 +37.8 37.83 ± 12% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.copy_mc_generic.copy_mc_to_user.copyout_mc._copy_mc_to_iter 63.34 ± 12% -26.2 37.14 ± 15% perf-profile.children.cycles-pp.copy_mc_fragile 2.41 ±112% -2.2 0.25 ±108% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 2.26 ±109% -2.0 0.29 ± 89% perf-profile.children.cycles-pp.asm_call_on_stack 2.15 ±112% -1.9 0.23 ±110% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 2.12 ±113% -1.9 0.23 ±110% perf-profile.children.cycles-pp.hrtimer_interrupt 1.68 ±114% -1.5 0.17 ±119% perf-profile.children.cycles-pp.__hrtimer_run_queues 1.48 ±123% -1.3 0.15 ±121% perf-profile.children.cycles-pp.tick_sched_timer 1.34 ±120% -1.2 0.14 ±122% perf-profile.children.cycles-pp.tick_sched_handle 1.28 ±119% -1.1 0.14 ±122% perf-profile.children.cycles-pp.update_process_times 0.70 ±107% -0.6 0.10 ±120% perf-profile.children.cycles-pp.scheduler_tick 2.65 ±106% +16.5 19.13 ± 12% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 0.00 +22.6 22.58 ± 7% perf-profile.children.cycles-pp.copy_mc_generic 62.52 ± 12% -25.5 37.00 ± 15% perf-profile.self.cycles-pp.copy_mc_fragile 0.00 +22.4 22.41 ± 6% perf-profile.self.cycles-pp.copy_mc_generic 42.43 +68.7% 71.58 perf-stat.i.MPKI 5.949e+09 -42.2% 3.44e+09 perf-stat.i.branch-instructions 0.07 +0.0 0.10 ± 5% perf-stat.i.branch-miss-rate% 3554006 ± 2% -7.5% 3286479 ± 3% perf-stat.i.branch-misses 95.02 -2.4 92.63 perf-stat.i.cache-miss-rate% 1.444e+09 -5.2% 1.369e+09 perf-stat.i.cache-misses 1.513e+09 -2.8% 1.471e+09 perf-stat.i.cache-references 3.81 +72.5% 6.58 perf-stat.i.cpi 102.49 +4.5% 107.13 perf-stat.i.cycles-between-cache-misses 0.00 ± 4% +0.0 0.00 ± 41% perf-stat.i.dTLB-load-miss-rate% 6.03e+09 -42.0% 3.495e+09 perf-stat.i.dTLB-loads 0.00 ± 5% +0.0 0.00 ± 7% perf-stat.i.dTLB-store-miss-rate% 5.909e+09 -42.5% 3.4e+09 perf-stat.i.dTLB-stores 47.00 +1.4 48.45 perf-stat.i.iTLB-load-miss-rate% 2270674 -11.0% 2021114 perf-stat.i.iTLB-load-misses 2563127 -16.0% 2151931 perf-stat.i.iTLB-loads 3.548e+10 -42.4% 2.044e+10 perf-stat.i.instructions 15634 -35.2% 10127 perf-stat.i.instructions-per-iTLB-miss 0.26 -41.6% 0.15 perf-stat.i.ipc 207.77 -37.5% 129.85 perf-stat.i.metric.M/sec 78061415 ± 13% +98.0% 1.546e+08 ± 20% perf-stat.i.node-load-misses 85582855 ± 11% +58.1% 1.353e+08 ± 20% perf-stat.i.node-loads 3.817e+08 -2.8% 3.709e+08 perf-stat.i.node-stores 42.66 +68.7% 71.96 perf-stat.overall.MPKI 0.06 +0.0 0.09 ± 3% perf-stat.overall.branch-miss-rate% 95.45 -2.4 93.07 perf-stat.overall.cache-miss-rate% 3.81 +73.0% 6.59 perf-stat.overall.cpi 93.55 +5.2% 98.41 perf-stat.overall.cycles-between-cache-misses 0.00 ± 5% +0.0 0.00 ± 13% perf-stat.overall.dTLB-load-miss-rate% 0.00 ± 5% +0.0 0.00 ± 2% perf-stat.overall.dTLB-store-miss-rate% 46.98 +1.5 48.43 perf-stat.overall.iTLB-load-miss-rate% 15639 -35.2% 10127 perf-stat.overall.instructions-per-iTLB-miss 0.26 -42.2% 0.15 perf-stat.overall.ipc 1605743 +1.5% 1630326 perf-stat.overall.path-length 5.919e+09 -42.2% 3.422e+09 perf-stat.ps.branch-instructions 3519866 ± 2% -7.8% 3245208 ± 3% perf-stat.ps.branch-misses 1.437e+09 -5.2% 1.362e+09 perf-stat.ps.cache-misses 1.506e+09 -2.8% 1.463e+09 perf-stat.ps.cache-references 1552 -1.4% 1530 perf-stat.ps.context-switches 6e+09 -42.1% 3.477e+09 perf-stat.ps.dTLB-loads 5.88e+09 -42.5% 3.382e+09 perf-stat.ps.dTLB-stores 2257568 -11.0% 2008542 perf-stat.ps.iTLB-load-misses 2547705 -16.1% 2138603 perf-stat.ps.iTLB-loads 3.53e+10 -42.4% 2.034e+10 perf-stat.ps.instructions 77685715 ± 13% +97.9% 1.538e+08 ± 20% perf-stat.ps.node-load-misses 85143339 ± 11% +58.1% 1.346e+08 ± 20% perf-stat.ps.node-loads 3.797e+08 -2.8% 3.69e+08 perf-stat.ps.node-stores 7.112e+12 -42.4% 4.096e+12 perf-stat.total.instructions fio.read_bw_MBps 46000 +-------------------------------------------------------------------+ 44000 |..+.+..+.+..+..+.+..+..+.+.. .+.. .+.+..+.+..+ | | + +. | 42000 |-+ | 40000 |-+ | 38000 |-+ | 36000 |-+ | | | 34000 |-+ | 32000 |-+ | 30000 |-+ | 28000 |-+ | | | 26000 |-+O O O O O O O O O O O O O O O O O O O O O O O O O | 24000 +-------------------------------------------------------------------+ fio.read_iops 23000 +-------------------------------------------------------------------+ 22000 |..+.+..+.+..+..+.+..+..+.+.. .+.. .+.+..+.+..+ | | + +. | 21000 |-+ | 20000 |-+ | 19000 |-+ | 18000 |-+ | | | 17000 |-+ | 16000 |-+ | 15000 |-+ | 14000 |-+ | | | 13000 |-+O O O O O O O O O O O O O O O O O O O O O O O O O | 12000 +-------------------------------------------------------------------+ fio.read_clat_mean_us 1.2e+08 +-----------------------------------------------------------------+ | O O O O O O O O O O O O O O O O O O O O O O O O | 1.1e+08 |-+ | | | | | 1e+08 |-+ | | | 9e+07 |-+ | | | 8e+07 |-+ | | | | | 7e+07 |..+.+.. .+..+. .+..+.+..+..+.+..+.+.. | | +.+..+ +..+ + | 6e+07 +-----------------------------------------------------------------+ fio.read_clat_90__us 1.3e+08 +-----------------------------------------------------------------+ | | 1.2e+08 |-+O O O O O O O O O O O O O O O O O O O O O O O O O | | | 1.1e+08 |-+ | | | 1e+08 |-+ | | | 9e+07 |-+ | | | 8e+07 |-+ | | | 7e+07 |..+.+..+.+..+.+..+.+..+.+..+.+..+..+.+..+.+..+ | | | 6e+07 +-----------------------------------------------------------------+ fio.read_clat_95__us 1.3e+08 +-----------------------------------------------------------------+ | O O O O O | 1.2e+08 |-+O O O O O O O O O O O O O O O O O O O O | | | 1.1e+08 |-+ | | | 1e+08 |-+ | | | 9e+07 |-+ | | | 8e+07 |-+ | | .+. .+.. | 7e+07 |..+.+..+.+..+.+..+.+..+.+. +. +.+..+.+..+ | | | 6e+07 +-----------------------------------------------------------------+ fio.read_slat_mean_us 4e+06 +-----------------------------------------------------------------+ 3.8e+06 |-+O O O O O O O O O O O O O O O O O O O O O O O O O | | | 3.6e+06 |-+ | 3.4e+06 |-+ | | | 3.2e+06 |-+ | 3e+06 |-+ | 2.8e+06 |-+ | | | 2.6e+06 |-+ | 2.4e+06 |-+ | | | 2.2e+06 |..+.+..+.+..+.+..+.+..+.+..+.+..+..+.+..+.+..+ | 2e+06 +-----------------------------------------------------------------+ fio.latency_10ms_ 0.15 +--------------------------------------------------------------------+ 0.14 |..+.+..+..+.+..+..+.+..+..+.+..+..+.+..+.+..+..+ | | | 0.13 |-+ | 0.12 |-+ | 0.11 |-+ | 0.1 |-+ | | | 0.09 |-+ | 0.08 |-+ | 0.07 |-+ | 0.06 |-+ | | | 0.05 |-+O O O O O O O O O O O O O O O O O O O O O O O O O | 0.04 +--------------------------------------------------------------------+ fio.latency_20ms_ 0.55 +--------------------------------------------------------------------+ | + + + | 0.5 |-+ : + + :: | 0.45 |++ : + + + : : | | + : + + : | 0.4 |-+ : : : : : | 0.35 |-+ : : : : : | | : : : : : + | 0.3 |-+ : : : +. : + + | 0.25 |-+ : : .. +.. : + + | | + +..+ +..+ +..+ | 0.2 |-+ | 0.15 |-+ | | O O O O O O O O O O O O O O O O O O O O O O O O O | 0.1 +--------------------------------------------------------------------+ fio.latency_100ms_ 100 +---------------------------------------------------------------------+ 90 |-+ + +.+..+..+ +. | | | 80 |-+ | 70 |-+ | | | 60 |-+ | 50 |-+ | 40 |-+ | | | 30 |-+ | 20 |-+ | | | 10 |-+ | 0 +---------------------------------------------------------------------+ fio.latency_250ms_ 100 +---------------------------------------------------------------------+ 90 |-+ O | | | 80 |-+ | 70 |-+ | | | 60 |-+ | 50 |-+ | 40 |-+ | | | 30 |-+ | 20 |-+ | | | 10 |-+ | 0 +---------------------------------------------------------------------+ fio.workload 4.6e+06 +-----------------------------------------------------------------+ 4.4e+06 |..+.+..+.+..+.+..+.+..+.+.. .+.. .+.+..+.+..+ | | + +. | 4.2e+06 |-+ | 4e+06 |-+ | 3.8e+06 |-+ | 3.6e+06 |-+ | | | 3.4e+06 |-+ | 3.2e+06 |-+ | 3e+06 |-+ | 2.8e+06 |-+ | | | 2.6e+06 |-+O O O O O O O O O O O O O O O O O O O O O O O O O | 2.4e+06 +-----------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-csl-2sp6: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory ========================================================================================= bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/mount_option/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode: 2M/gcc-9/performance/2pmem/xfs/sync/x86_64-rhel-8.3/dax/50%/debian-10.4-x86_64-20200603.cgz/200s/read/lkp-csl-2sp6/200G/fio-basic/tb/0x5002f01 commit: 7476b91d4d ("x86, powerpc: Rename memcpy_mcsafe() to copy_mc_to_{user, kernel}()") a0ac629ebe ("x86/copy_mc: Introduce copy_mc_generic()") 7476b91d4db369d8 a0ac629ebe7b3d248cb93807782 ---------------- --------------------------- %stddev %change %stddev \ | \ 0.61 ± 15% -0.4 0.22 ± 94% fio.latency_1000us% 0.01 ± 11% +1.3 1.27 ± 25% fio.latency_10ms% 96.06 -95.5 0.60 ± 80% fio.latency_2ms% 1.27 ± 33% +96.2 97.48 fio.latency_4ms% 1.29 ± 55% -1.2 0.05 ± 54% fio.latency_500us% 75143 -55.6% 33381 fio.read_bw_MBps 1372160 +118.5% 2998272 fio.read_clat_90%_us 1409024 +116.9% 3055616 fio.read_clat_95%_us 2142208 ± 19% +120.3% 4718592 ± 17% fio.read_clat_99%_us 1272849 +125.4% 2869293 fio.read_clat_mean_us 228201 ± 15% +103.6% 464620 ± 14% fio.read_clat_stddev 37571 -55.6% 16690 fio.read_iops 69.28 ± 2% -40.3% 41.38 ± 3% fio.time.user_time 7514438 -55.6% 3338252 fio.workload 0.11 ± 3% +0.0 0.14 ± 5% mpstat.cpu.all.soft% 0.43 ± 3% -0.1 0.28 ± 2% mpstat.cpu.all.usr% 115069 -2.3% 112454 proc-vmstat.nr_shmem 20846 ± 6% -27.8% 15052 ± 3% proc-vmstat.pgactivate 967.50 ± 27% -50.0% 483.75 ± 78% slabinfo.xfs_buf_item.active_objs 967.50 ± 27% -50.0% 483.75 ± 78% slabinfo.xfs_buf_item.num_objs 100.00 -2.0% 98.00 vmstat.io.bo 1672 -3.3% 1616 vmstat.system.cs 9.059e+09 ± 6% -32.3% 6.131e+09 ± 54% cpuidle.C1E.time 19004364 ± 3% -22.4% 14741281 ± 34% cpuidle.C1E.usage 4.034e+08 ±133% +713.0% 3.28e+09 ±100% cpuidle.C6.time 570211 ±122% +571.6% 3829822 ± 86% cpuidle.C6.usage 61.80 ± 9% -17.6 44.19 perf-profile.calltrace.cycles-pp.copy_mc_fragile.copy_mc_to_user.copyout_mc._copy_mc_to_iter.dax_iomap_actor 0.00 +7.8 7.81 ± 6% perf-profile.calltrace.cycles-pp.copy_mc_generic.copy_mc_to_user.copyout_mc._copy_mc_to_iter.dax_iomap_actor 0.00 +29.2 29.21 ± 5% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.copy_mc_generic.copy_mc_to_user.copyout_mc._copy_mc_to_iter 61.92 ± 9% -17.7 44.25 perf-profile.children.cycles-pp.copy_mc_fragile 3.47 ±132% +11.7 15.21 ± 5% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 0.00 +22.3 22.32 perf-profile.children.cycles-pp.copy_mc_generic 61.16 ± 9% -17.4 43.78 perf-profile.self.cycles-pp.copy_mc_fragile 0.00 +22.1 22.09 perf-profile.self.cycles-pp.copy_mc_generic 212.00 ± 38% +288.6% 823.90 ± 67% sched_debug.cfs_rq:/.exec_clock.min 34013 ± 3% -17.1% 28181 ± 2% sched_debug.cfs_rq:/.exec_clock.stddev 36118 ± 5% -15.0% 30710 ± 2% sched_debug.cfs_rq:/.min_vruntime.stddev 36118 ± 5% -15.0% 30707 ± 2% sched_debug.cfs_rq:/.spread0.stddev 9.52 ± 11% +33.8% 12.73 ± 9% sched_debug.cpu.clock.stddev 17832 ± 13% -47.5% 9368 ± 17% sched_debug.cpu.sched_count.max 2475 ± 9% -34.4% 1624 ± 8% sched_debug.cpu.sched_count.stddev 8858 ± 13% -48.3% 4577 ± 18% sched_debug.cpu.sched_goidle.max 1260 ± 9% -33.2% 841.68 ± 8% sched_debug.cpu.sched_goidle.stddev 8285 ± 16% -32.1% 5622 ± 7% sched_debug.cpu.ttwu_count.max 1169 ± 9% -24.9% 878.40 ± 4% sched_debug.cpu.ttwu_count.stddev 26587 ± 8% -21.9% 20773 ± 22% softirqs.CPU1.SCHED 19906 ± 37% -55.7% 8824 ± 96% softirqs.CPU10.SCHED 21997 ± 34% -82.2% 3910 ± 55% softirqs.CPU20.SCHED 5126 ± 70% +166.6% 13666 ± 15% softirqs.CPU30.SCHED 5567 ± 56% +165.3% 14772 ± 29% softirqs.CPU31.SCHED 10027 ± 35% +101.3% 20182 ± 18% softirqs.CPU33.SCHED 4868 ± 50% +112.6% 10349 ± 14% softirqs.CPU44.SCHED 6304 ± 60% +154.5% 16043 ± 22% softirqs.CPU46.SCHED 4127 ± 76% +198.6% 12326 ± 32% softirqs.CPU49.SCHED 6313 ± 62% +98.5% 12530 ± 19% softirqs.CPU51.SCHED 8249 ± 58% +148.7% 20515 ± 31% softirqs.CPU57.SCHED 6971 ±109% +268.6% 25698 ± 8% softirqs.CPU68.SCHED 25116 ± 15% -32.4% 16974 ± 12% softirqs.CPU78.SCHED 24757 ± 12% -36.8% 15657 ± 27% softirqs.CPU79.SCHED 20231 ± 14% -45.5% 11024 ± 24% softirqs.CPU81.SCHED 21830 ± 23% -55.4% 9733 ± 67% softirqs.CPU9.SCHED 24043 ± 16% -39.9% 14449 ± 23% softirqs.CPU94.SCHED 42.31 +68.3% 71.22 perf-stat.i.MPKI 9.958e+09 -54.7% 4.511e+09 perf-stat.i.branch-instructions 0.05 ± 2% +0.0 0.08 ± 4% perf-stat.i.branch-miss-rate% 3682118 ± 2% -8.2% 3381534 perf-stat.i.branch-misses 67.34 +10.4 77.74 perf-stat.i.cache-miss-rate% 1.709e+09 -12.2% 1.501e+09 perf-stat.i.cache-misses 2.531e+09 -24.0% 1.923e+09 perf-stat.i.cache-references 1639 -4.1% 1571 perf-stat.i.context-switches 2.25 +121.4% 4.98 perf-stat.i.cpi 99.03 -1.8% 97.24 perf-stat.i.cpu-migrations 85.60 +14.2% 97.78 perf-stat.i.cycles-between-cache-misses 0.00 ± 18% +0.0 0.00 ± 44% perf-stat.i.dTLB-load-miss-rate% 9.996e+09 -54.5% 4.549e+09 perf-stat.i.dTLB-loads 0.00 ± 7% +0.0 0.00 ± 6% perf-stat.i.dTLB-store-miss-rate% 9.904e+09 -54.9% 4.466e+09 perf-stat.i.dTLB-stores 44.79 +4.2 48.99 perf-stat.i.iTLB-load-miss-rate% 2535885 -13.8% 2185118 perf-stat.i.iTLB-load-misses 3134177 -27.3% 2278467 perf-stat.i.iTLB-loads 5.952e+10 -54.9% 2.687e+10 perf-stat.i.instructions 23480 -47.6% 12304 perf-stat.i.instructions-per-iTLB-miss 0.45 -54.6% 0.20 perf-stat.i.ipc 342.39 -51.0% 167.90 perf-stat.i.metric.M/sec 1.165e+08 ± 30% +72.8% 2.013e+08 ± 9% perf-stat.i.node-load-misses 1.257e+08 ± 26% +41.1% 1.773e+08 ± 9% perf-stat.i.node-loads 2.42e+08 +19.6% 2.895e+08 perf-stat.i.node-stores 42.53 +68.3% 71.58 perf-stat.overall.MPKI 0.04 ± 2% +0.0 0.07 perf-stat.overall.branch-miss-rate% 67.52 +10.5 78.07 perf-stat.overall.cache-miss-rate% 2.24 +122.4% 4.99 perf-stat.overall.cpi 78.17 +14.3% 89.34 perf-stat.overall.cycles-between-cache-misses 0.00 ± 25% +0.0 0.00 ± 12% perf-stat.overall.dTLB-load-miss-rate% 0.00 ± 13% +0.0 0.00 ± 10% perf-stat.overall.dTLB-store-miss-rate% 44.72 +4.2 48.96 perf-stat.overall.iTLB-load-miss-rate% 23499 -47.6% 12306 perf-stat.overall.instructions-per-iTLB-miss 0.45 -55.0% 0.20 perf-stat.overall.ipc 1587395 +1.5% 1611895 perf-stat.overall.path-length 9.912e+09 -54.7% 4.489e+09 perf-stat.ps.branch-instructions 3650903 ± 2% -8.4% 3345674 perf-stat.ps.branch-misses 1.701e+09 -12.2% 1.494e+09 perf-stat.ps.cache-misses 2.52e+09 -24.1% 1.914e+09 perf-stat.ps.cache-references 1616 -3.7% 1556 perf-stat.ps.context-switches 9.95e+09 -54.5% 4.526e+09 perf-stat.ps.dTLB-loads 9.859e+09 -54.9% 4.445e+09 perf-stat.ps.dTLB-stores 2521574 -13.8% 2172342 perf-stat.ps.iTLB-load-misses 3116655 -27.3% 2264894 perf-stat.ps.iTLB-loads 5.925e+10 -54.9% 2.673e+10 perf-stat.ps.instructions 1.159e+08 ± 30% +72.8% 2.003e+08 ± 9% perf-stat.ps.node-load-misses 1.25e+08 ± 26% +41.1% 1.764e+08 ± 9% perf-stat.ps.node-loads 2.407e+08 +19.6% 2.878e+08 perf-stat.ps.node-stores 1.193e+13 -54.9% 5.381e+12 perf-stat.total.instructions 0.00 +2.7e+105% 2689 ±171% interrupts.115:PCI-MSI.31981648-edge.i40e-eth0-TxRx-79 62.75 ± 27% +51.8% 95.25 ± 21% interrupts.CPU1.RES:Rescheduling_interrupts 6530 ± 17% -44.1% 3647 ± 35% interrupts.CPU17.NMI:Non-maskable_interrupts 6530 ± 17% -44.1% 3647 ± 35% interrupts.CPU17.PMI:Performance_monitoring_interrupts 62.00 ± 74% +187.9% 178.50 ± 5% interrupts.CPU20.RES:Rescheduling_interrupts 365.00 ± 78% -76.0% 87.50 ± 53% interrupts.CPU25.TLB:TLB_shootdowns 170.50 ± 15% -26.8% 124.75 ± 10% interrupts.CPU30.RES:Rescheduling_interrupts 7605 -43.3% 4316 ± 32% interrupts.CPU31.NMI:Non-maskable_interrupts 7605 -43.3% 4316 ± 32% interrupts.CPU31.PMI:Performance_monitoring_interrupts 169.00 ± 12% -37.1% 106.25 ± 23% interrupts.CPU31.RES:Rescheduling_interrupts 7145 ± 11% -33.0% 4786 ± 18% interrupts.CPU36.NMI:Non-maskable_interrupts 7145 ± 11% -33.0% 4786 ± 18% interrupts.CPU36.PMI:Performance_monitoring_interrupts 136.50 ± 27% -44.7% 75.50 ± 60% interrupts.CPU39.TLB:TLB_shootdowns 149.25 ± 24% -24.6% 112.50 ± 30% interrupts.CPU4.RES:Rescheduling_interrupts 7599 -46.6% 4061 ± 35% interrupts.CPU41.NMI:Non-maskable_interrupts 7599 -46.6% 4061 ± 35% interrupts.CPU41.PMI:Performance_monitoring_interrupts 6661 ± 24% -52.1% 3191 ± 51% interrupts.CPU44.NMI:Non-maskable_interrupts 6661 ± 24% -52.1% 3191 ± 51% interrupts.CPU44.PMI:Performance_monitoring_interrupts 7622 -43.5% 4307 ± 33% interrupts.CPU46.NMI:Non-maskable_interrupts 7622 -43.5% 4307 ± 33% interrupts.CPU46.PMI:Performance_monitoring_interrupts 7613 -43.1% 4331 ± 31% interrupts.CPU47.NMI:Non-maskable_interrupts 7613 -43.1% 4331 ± 31% interrupts.CPU47.PMI:Performance_monitoring_interrupts 5823 ± 32% -36.4% 3703 ± 34% interrupts.CPU5.NMI:Non-maskable_interrupts 5823 ± 32% -36.4% 3703 ± 34% interrupts.CPU5.PMI:Performance_monitoring_interrupts 89.25 ± 48% -61.1% 34.75 ± 31% interrupts.CPU53.TLB:TLB_shootdowns 5698 ± 33% -42.5% 3277 ± 49% interrupts.CPU55.NMI:Non-maskable_interrupts 5698 ± 33% -42.5% 3277 ± 49% interrupts.CPU55.PMI:Performance_monitoring_interrupts 172.00 ± 14% -35.2% 111.50 ± 41% interrupts.CPU56.RES:Rescheduling_interrupts 64.00 ± 42% -39.5% 38.75 ± 29% interrupts.CPU56.TLB:TLB_shootdowns 156.00 ± 17% -36.2% 99.50 ± 21% interrupts.CPU57.RES:Rescheduling_interrupts 146.25 ± 28% -48.9% 74.75 ± 67% interrupts.CPU58.RES:Rescheduling_interrupts 7627 -47.0% 4043 ± 31% interrupts.CPU62.NMI:Non-maskable_interrupts 7627 -47.0% 4043 ± 31% interrupts.CPU62.PMI:Performance_monitoring_interrupts 174.75 ± 12% -29.9% 122.50 ± 30% interrupts.CPU62.RES:Rescheduling_interrupts 76.00 ± 29% -48.4% 39.25 ± 29% interrupts.CPU62.TLB:TLB_shootdowns 7159 ± 11% -50.2% 3564 ± 32% interrupts.CPU63.NMI:Non-maskable_interrupts 7159 ± 11% -50.2% 3564 ± 32% interrupts.CPU63.PMI:Performance_monitoring_interrupts 7628 -62.9% 2831 interrupts.CPU66.NMI:Non-maskable_interrupts 7628 -62.9% 2831 interrupts.CPU66.PMI:Performance_monitoring_interrupts 174.50 ± 10% -36.4% 111.00 ± 50% interrupts.CPU66.RES:Rescheduling_interrupts 4370 ± 18% -34.7% 2853 interrupts.CPU69.NMI:Non-maskable_interrupts 4370 ± 18% -34.7% 2853 interrupts.CPU69.PMI:Performance_monitoring_interrupts 6885 ± 18% -45.8% 3731 ± 28% interrupts.CPU74.NMI:Non-maskable_interrupts 6885 ± 18% -45.8% 3731 ± 28% interrupts.CPU74.PMI:Performance_monitoring_interrupts 5900 ± 18% -57.5% 2510 ± 24% interrupts.CPU77.NMI:Non-maskable_interrupts 5900 ± 18% -57.5% 2510 ± 24% interrupts.CPU77.PMI:Performance_monitoring_interrupts 62.00 ± 41% +58.9% 98.50 ± 14% interrupts.CPU78.RES:Rescheduling_interrupts 0.00 +2.7e+105% 2689 ±171% interrupts.CPU79.115:PCI-MSI.31981648-edge.i40e-eth0-TxRx-79 49.75 ± 47% +119.6% 109.25 ± 28% interrupts.CPU79.RES:Rescheduling_interrupts 61.50 ± 54% +115.0% 132.25 ± 31% interrupts.CPU8.RES:Rescheduling_interrupts 5871 ± 19% -38.8% 3594 ± 32% interrupts.CPU80.NMI:Non-maskable_interrupts 5871 ± 19% -38.8% 3594 ± 32% interrupts.CPU80.PMI:Performance_monitoring_interrupts 60.50 ± 19% +120.2% 133.25 ± 14% interrupts.CPU81.RES:Rescheduling_interrupts 36.00 ± 79% +179.2% 100.50 ± 35% interrupts.CPU86.RES:Rescheduling_interrupts 6322 ± 21% -60.6% 2490 ± 25% interrupts.CPU88.NMI:Non-maskable_interrupts 6322 ± 21% -60.6% 2490 ± 25% interrupts.CPU88.PMI:Performance_monitoring_interrupts 32.50 ± 40% +150.0% 81.25 ± 41% interrupts.CPU92.RES:Rescheduling_interrupts 124.00 ± 11% -22.8% 95.75 ± 4% interrupts.IWI:IRQ_work_interrupts 538989 ± 8% -28.0% 387910 ± 2% interrupts.NMI:Non-maskable_interrupts 538989 ± 8% -28.0% 387910 ± 2% interrupts.PMI:Performance_monitoring_interrupts Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

3 months, 3 weeks

3
5
0 / 0

Re: [IB/srpt] c804af2c1d: last_state.test.blktests.exit_code.143

by Bart Van Assche

On 2020-08-03 00:27, Sagi Grimberg wrote: > >>>> Greeting, >>>> >>>> FYI, we noticed the following commit (built with gcc-9): >>>> >>>> commit: c804af2c1d3152c0cf877eeb50d60c2d49ac0cf0 ("IB/srpt: use new shared CQ mechanism") >>>> https://git.kernel.org/cgit/linux/kernel/git/rdma/rdma.git for-next >>>> >>>> >>>> in testcase: blktests >>>> with following parameters: >>>> >>>> test: srp-group1 >>>> ucode: 0x21 >>>> >>>> >>>> >>>> on test machine: 4 threads Intel(R) Core(TM) i3-3220 CPU @ 3.30GHz with 4G memory >>>> >>>> caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): >>>> >>>> >>>> >>>> >>>> If you fix the issue, kindly add following tag >>>> Reported-by: kernel test robot <rong.a.chen(a)intel.com> >>>> >>>> >>>> user :notice: [ 44.688140] 2020-08-01 16:10:22 ./check srp/001 srp/002 srp/003 srp/004 srp/005 srp/006 srp/007 srp/008 srp/009 srp/010 srp/011 srp/012 srp/013 srp/015 >>>> user :notice: [ 44.706657] srp/001 (Create and remove LUNs) >>>> user :notice: [ 44.718405] srp/001 (Create and remove LUNs) [passed] >>>> user :notice: [ 44.729902] runtime ... 1.972s >>>> user :notice: [ 99.038748] IPMI BMC is not supported on this machine, skip bmc-watchdog setup! >>>> user :notice: [ 3699.039790] Sat Aug 1 17:11:22 UTC 2020 detected soft_timeout >>>> user :notice: [ 3699.060341] kill 960 /usr/bin/time -v -o /tmp/lkp/blktests.time /lkp/lkp/src/tests/blktests >>> Yamin and Max, can you take a look at this? The SRP tests from the >>> blktests repository pass reliably with kernel version v5.7 and before. >>> With label next-20200731 from linux-next however that test triggers the >>> following hang: >> >> I will look into it. > > FWIW, I ran into this as well with nvme-rdma, but it also reproduces > when I revert the shared CQ patch from nvme-rdma. Another data point > is that my tests passes with siw. Hi Jason, The patch below is sufficient to unbreak blktests. I think that the deadlock while unloading rdma_rxe happens because the RDMA core waits for all ib_dev references to be dropped before dealloc_driver is called. The rdma_rxe dealloc_driver implementation drops an ib_dev reference. The dealloc_driver callback was introduced by commit d0899892edd0 ("RDMA/device: Provide APIs from the core code to help unregistration"). Do you agree that this regression has been introduced by commits d0899892edd0 and c367074b6c37 ("RDMA/rxe: Use driver_unregister and new unregistration API")? Thanks, Bart. --- drivers/infiniband/core/device.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c index dca2842a7872..5192f305b253 100644 --- a/drivers/infiniband/core/device.c +++ b/drivers/infiniband/core/device.c @@ -1287,13 +1287,8 @@ static void disable_device(struct ib_device *device) /* Pairs with refcount_set in enable_device */ ib_device_put(device); - wait_for_completion(&device->unreg_completion); - /* - * compat devices must be removed after device refcount drops to zero. - * Otherwise init_net() may add more compatdevs after removing compat - * devices and before device is disabled. - */ + /* To do: prevent init_net() to add more compat_devs. */ remove_compat_devs(device); }

4 months

3
6
0 / 0

[PCI] 3233e41d3e: WARNING:at_drivers/pci/pci.c:#pci_reset_hotplug_slot

by kernel test robot

Greeting, FYI, we noticed the following commit (built with gcc-9): commit: 3233e41d3e8ebcd44e92da47ffed97fd49b84278 ("[PATCH] PCI: pciehp: Fix AB-BA deadlock between reset_lock and device_lock") url: https://github.com/0day-ci/linux/commits/Lukas-Wunner/PCI-pciehp-Fix-AB-B... base: https://git.kernel.org/cgit/linux/kernel/git/helgaas/pci.git next in testcase: boot on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 8 -m 16G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): +------------------------------------------------------+------------+------------+ | | 8a445afd71 | 3233e41d3e | +------------------------------------------------------+------------+------------+ | boot_successes | 4 | 0 | | boot_failures | 0 | 4 | | WARNING:at_drivers/pci/pci.c:#pci_reset_hotplug_slot | 0 | 4 | | RIP:pci_reset_hotplug_slot | 0 | 4 | +------------------------------------------------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <lkp(a)intel.com> [ 0.971752] WARNING: CPU: 0 PID: 1 at drivers/pci/pci.c:4905 pci_reset_hotplug_slot+0x70/0x80 [ 0.971753] Modules linked in: [ 0.971755] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.8.0-rc1-00053-g3233e41d3e8eb #1 [ 0.971756] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [ 0.971757] RIP: 0010:pci_reset_hotplug_slot+0x70/0x80 [ 0.971759] Code: 41 89 c4 48 8b 7b 20 e8 4e 3a c1 ff 44 89 e0 5b 5d 41 5c c3 48 8b 43 18 31 f6 48 8d b8 90 00 00 00 e8 e4 80 70 00 85 c0 75 ba <0f> 0b eb b6 41 bc e7 ff ff ff eb d6 0f 1f 40 00 66 66 66 66 90 48 [ 0.971759] RSP: 0000:ffffbd73c0013ab0 EFLAGS: 00010246 [ 0.971761] RAX: 0000000000000000 RBX: ffff9700475024c0 RCX: ffff9701048c9900 [ 0.971762] RDX: ffff970047e40000 RSI: ffff9701048c9990 RDI: 0000000000000246 [ 0.971763] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000 [ 0.971763] R10: 0000000000000001 R11: 0000000000000000 R12: ffff9700477320b0 [ 0.971764] R13: ffff970047732000 R14: ffff97004769c238 R15: 0000000000000000 [ 0.971765] FS: 0000000000000000(0000) GS:ffff97036fc00000(0000) knlGS:0000000000000000 [ 0.971766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.971767] CR2: 0000000000000000 CR3: 00000001c2c10000 CR4: 00000000000006f0 [ 0.971767] Call Trace: [ 0.971768] pci_probe_reset_function+0xc4/0xe0 [ 0.971769] pci_device_add+0x13f/0x2a0 [ 0.971769] pci_scan_single_device+0xa4/0xc0 [ 0.971770] pci_scan_slot+0x52/0x110 [ 0.971771] pci_scan_child_bus_extend+0x3a/0x2a0 [ 0.971771] acpi_pci_root_create+0x1f7/0x250 [ 0.971772] pci_acpi_scan_root+0x182/0x1b0 [ 0.971773] acpi_pci_root_add.cold+0x59/0x1b0 [ 0.971773] ? acpi_device_always_present+0x20/0x90 [ 0.971774] acpi_bus_attach+0xf6/0x200 [ 0.971775] acpi_bus_attach+0x6b/0x200 [ 0.971776] acpi_bus_scan+0x43/0x90 [ 0.971776] ? acpi_sleep_proc_init+0x24/0x24 [ 0.971777] acpi_scan_init+0x102/0x24b [ 0.971778] acpi_init+0x2c7/0x329 [ 0.971778] do_one_initcall+0x5d/0x330 [ 0.971779] ? rcu_read_lock_sched_held+0x52/0x90 [ 0.971780] kernel_init_freeable+0x248/0x2c9 [ 0.971780] ? rest_init+0x23e/0x23e [ 0.971781] kernel_init+0xa/0x112 [ 0.971782] ret_from_fork+0x22/0x30 [ 0.971782] irq event stamp: 109115 [ 0.971783] hardirqs last enabled at (109115): [<ffffffffaa71a514>] _raw_spin_unlock_irqrestore+0x54/0x70 [ 0.971784] hardirqs last disabled at (109114): [<ffffffffaa71ab01>] _raw_spin_lock_irqsave+0x21/0x60 [ 0.971785] softirqs last enabled at (109060): [<ffffffffaaa003aa>] __do_softirq+0x3aa/0x4af [ 0.971786] softirqs last disabled at (109053): [<ffffffffaa8010b2>] asm_call_on_stack+0x12/0x20 [ 0.971787] ---[ end trace c3e4ce92dee5df5f ]--- To reproduce: # build kernel cd linux cp config-5.8.0-rc1-00053-g3233e41d3e8eb .config make HOSTCC=gcc-9 CC=gcc-9 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email Thanks, lkp

4 months, 1 week

3
2
0 / 0

[ptrace] 201766a20e: kernel_selftests.seccomp.make_fail

by kernel test robot

FYI, we noticed the following commit (built with gcc-7): commit: 201766a20e30f982ccfe36bebfad9602c3ff574a ("ptrace: add PTRACE_GET_SYSCALL_INFO request") https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux.git master in testcase: kernel_selftests with following parameters: group: kselftests-02 test-description: The kernel contains a set of "self tests" under the tools/testing/selftests/ directory. These are intended to be small unit tests to exercise individual code paths in the kernel. test-url: https://www.kernel.org/doc/Documentation/kselftest.txt on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 8G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> 2019-07-26 16:52:03 make run_tests -C seccomp make: Entering directory '/usr/src/perf_selftests-x86_64-rhel-7.6-201766a20e30f982ccfe36bebfad9602c3ff574a/tools/testing/selftests/seccomp' gcc -Wl,-no-as-needed -Wall seccomp_bpf.c -lpthread -o seccomp_bpf In file included from seccomp_bpf.c:51:0: seccomp_bpf.c: In function ‘tracer_ptrace’: seccomp_bpf.c:1787:20: error: ‘PTRACE_EVENTMSG_SYSCALL_ENTRY’ undeclared (first use in this function) EXPECT_EQ(entry ? PTRACE_EVENTMSG_SYSCALL_ENTRY ^ ../kselftest_harness.h:608:13: note: in definition of macro ‘__EXPECT’ __typeof__(_expected) __exp = (_expected); \ ^~~~~~~~~ seccomp_bpf.c:1787:2: note: in expansion of macro ‘EXPECT_EQ’ EXPECT_EQ(entry ? PTRACE_EVENTMSG_SYSCALL_ENTRY ^~~~~~~~~ seccomp_bpf.c:1787:20: note: each undeclared identifier is reported only once for each function it appears in EXPECT_EQ(entry ? PTRACE_EVENTMSG_SYSCALL_ENTRY ^ ../kselftest_harness.h:608:13: note: in definition of macro ‘__EXPECT’ __typeof__(_expected) __exp = (_expected); \ ^~~~~~~~~ seccomp_bpf.c:1787:2: note: in expansion of macro ‘EXPECT_EQ’ EXPECT_EQ(entry ? PTRACE_EVENTMSG_SYSCALL_ENTRY ^~~~~~~~~ seccomp_bpf.c:1788:6: error: ‘PTRACE_EVENTMSG_SYSCALL_EXIT’ undeclared (first use in this function) : PTRACE_EVENTMSG_SYSCALL_EXIT, msg); ^ ../kselftest_harness.h:608:13: note: in definition of macro ‘__EXPECT’ __typeof__(_expected) __exp = (_expected); \ ^~~~~~~~~ seccomp_bpf.c:1787:2: note: in expansion of macro ‘EXPECT_EQ’ EXPECT_EQ(entry ? PTRACE_EVENTMSG_SYSCALL_ENTRY ^~~~~~~~~ Makefile:12: recipe for target 'seccomp_bpf' failed make: *** [seccomp_bpf] Error 1 make: Leaving directory '/usr/src/perf_selftests-x86_64-rhel-7.6-201766a20e30f982ccfe36bebfad9602c3ff574a/tools/testing/selftests/seccomp' To reproduce: # build kernel cd linux cp config-5.2.0-10889-g201766a20e30f9 .config make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email Thanks, Rong Chen

4 months, 1 week

3
3
0 / 0

[rcuperf] 4e88ec4a9e: UBSAN:division-overflow_in_arch/x86/include/asm/div64.h

by kernel test robot

Greeting, FYI, we noticed the following commit (built with gcc-9): commit: 4e88ec4a9eb17527e640b063f79e5b875733eb53 ("rcuperf: Change rcuperf to rcuscale") https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master in testcase: trinity with following parameters: runtime: 300s test-description: Trinity is a linux system call fuzz tester. test-url: http://codemonkey.org.uk/projects/trinity/ on test machine: qemu-system-i386 -enable-kvm -cpu SandyBridge -smp 2 -m 8G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): +---------------------------------------------------------+------------+------------+ | | 65bd77f554 | 4e88ec4a9e | +---------------------------------------------------------+------------+------------+ | boot_successes | 13 | 0 | | boot_failures | 0 | 14 | | UBSAN:division-overflow_in_arch/x86/include/asm/div64.h | 0 | 14 | | error:#[##] | 0 | 14 | | EIP:main_func.cold | 0 | 14 | | Kernel_panic-not_syncing:Fatal_exception | 0 | 14 | +---------------------------------------------------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <lkp(a)intel.com> [ 24.821408] UBSAN: division-overflow in arch/x86/include/asm/div64.h:54:20 [ 24.823429] division by zero [ 24.824123] CPU: 0 PID: 64 Comm: main_func Not tainted 5.9.0-rc1-00019-g4e88ec4a9eb17 #1 [ 24.825684] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [ 24.827073] Call Trace: [ 24.827523] dump_stack+0x45/0x66 [ 24.828139] ubsan_epilogue+0x16/0x75 [ 24.828843] __ubsan_handle_divrem_overflow.cold+0x10f/0x1b0 [ 24.834661] ? destroy_timer_on_stack+0x40/0x40 [ 24.835417] ? __might_sleep+0x88/0x140 [ 24.836116] main_func.cold+0xb3/0x36d [ 24.836878] ? __lock_release+0x62/0x340 [ 24.837565] ? __kthread_parkme+0xd2/0x140 [ 24.838337] kthread+0x2c5/0x320 [ 24.838970] ? kzalloc+0x40/0x40 [ 24.839806] ? kthread_create_worker+0x40/0x40 [ 24.840709] ret_from_fork+0x1c/0x28 [ 24.841427] ================================================================================ [ 24.842989] divide_error: 0000 [#1] [ 24.843812] CPU: 0 PID: 64 Comm: main_func Not tainted 5.9.0-rc1-00019-g4e88ec4a9eb17 #1 [ 24.845435] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [ 24.847007] EIP: main_func.cold+0xb7/0x36d [ 24.847763] Code: 15 14 4b c5 c2 00 5e e9 f0 f7 ff ff 83 05 d0 3e c5 c2 01 b8 30 c6 5e c2 83 15 d4 3e c5 c2 00 31 c9 e8 6c 03 4c 00 89 d8 31 d2 <f7> f6 83 05 d8 3e c5 c2 01 89 da 83 15 dc 3e c5 c2 00 31 c9 89 c7 [ 24.850835] EAX: 00000000 EBX: 00000000 ECX: 00000000 EDX: 00000000 [ 24.851857] ESI: 00000000 EDI: 00000000 EBP: e8b91f84 ESP: e8b91ef8 [ 24.852969] DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068 EFLAGS: 00010246 [ 24.854183] CR0: 80050033 CR2: 096e3000 CR3: 29832160 CR4: 000406b0 [ 24.855270] Call Trace: [ 24.855766] ? __lock_release+0x62/0x340 [ 24.856530] ? __kthread_parkme+0xd2/0x140 [ 24.857245] kthread+0x2c5/0x320 [ 24.857759] ? kzalloc+0x40/0x40 [ 24.858547] ? kthread_create_worker+0x40/0x40 [ 24.859387] ret_from_fork+0x1c/0x28 [ 24.860081] Modules linked in: serio_raw i2c_piix4 i2c_core [ 24.861149] ---[ end trace 59d328920e1c098a ]--- To reproduce: # build kernel cd linux cp config-5.9.0-rc1-00019-g4e88ec4a9eb17 .config make HOSTCC=gcc-9 CC=gcc-9 ARCH=i386 olddefconfig prepare modules_prepare bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email Thanks, lkp

4 months, 2 weeks

4
4
0 / 0

2021

2020

2019

2018

2017

2016

2015

2014

2013

LKP August 2020