October 2020 - LKP - Ml01.01.Org

[ext4] 244adf6426: phoronix-test-suite.fio.SequentialWrite.Sync.Yes.No.4KB.DefaultTestDirectory.mb_s -36.1% regression

by kernel test robot

Greeting, FYI, we noticed a -36.1% regression of phoronix-test-suite.fio.SequentialWrite.Sync.Yes.No.4KB.DefaultTestDirectory.mb_s due to commit: commit: 244adf6426ee31a83f397b700d964cff12a247d3 ("ext4: make dioread_nolock the default") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: phoronix-test-suite on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 192G memory with following parameters: test: fio-1.11.5 option_a: Sequential Write option_b: Sync option_c: Yes option_d: No option_e: 4KB option_f: Default Test Directory cpufreq_governor: performance ucode: 0x5002f01 test-description: The Phoronix Test Suite is the most comprehensive testing and benchmarking platform available that provides an extensible framework for which new tests can be easily added. test-url: http://www.phoronix-test-suite.com/ In addition to that, the commit also has significant impact on the following tests: +------------------+-----------------------------------------------------------------------+ | testcase: change | fio-basic: fio.write_iops -58.8% regression | | test machine | 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 128G memory | | test parameters | bs=4k | | | cpufreq_governor=performance | | | disk=2pmem | | | fs=ext4 | | | ioengine=sync | | | nr_task=50% | | | runtime=200s | | | rw=randwrite | | | test_size=200G | | | time_based=tb | | | ucode=0x43 | +------------------+-----------------------------------------------------------------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/option_a/option_b/option_c/option_d/option_e/option_f/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-8.3/Sequential Write/Sync/Yes/No/4KB/Default Test Directory/debian-x86_64-phoronix/lkp-csl-2sp8/fio-1.11.5/phoronix-test-suite/0x5002f01 commit: 4068664e3c ("ext4: fix extent_status fragmentation for plain files") 244adf6426 ("ext4: make dioread_nolock the default") 4068664e3cd23126 244adf6426ee31a83f397b700d9 ---------------- --------------------------- %stddev %change %stddev \ | \ 235329 -36.2% 150250 phoronix-test-suite.fio.SequentialWrite.Sync.Yes.No.4KB.DefaultTestDirectory.iops 919.25 -36.1% 587.25 phoronix-test-suite.fio.SequentialWrite.Sync.Yes.No.4KB.DefaultTestDirectory.mb_s phoronix-test-suite.fio.SequentialWrite.Sync.Yes.No.4KB.DefaultTestDirectory.mb_ 950 +---------------------------------------------------------------------+ |.. .+.+..+ +.. .+..+.+..+.+..+.+..+ +.+..+.+. + .+.+..+.| 900 |-++.+. + +. | 850 |-+ | | | 800 |-+ | | | 750 |-+ | | | 700 |-+ | 650 |-+ | | | 600 |-+ O O O O O O O O O | | O O O O O O O O O O O O O | 550 +---------------------------------------------------------------------+ phoronix-test-suite.fio.SequentialWrite.Sync.Yes.No.4KB.DefaultTestDirectory.iop 250000 +------------------------------------------------------------------+ 240000 |-+ .+..+. .+. .+. .+. .+.. | |. .+.+..+.+ +..+.+.+. +..+.+.+..+ +..+.+..+ +..+ +.| 230000 |-+. | 220000 |-+ | 210000 |-+ | 200000 |-+ | | | 190000 |-+ | 180000 |-+ | 170000 |-+ | 160000 |-+ | | O O O O O O O O | 150000 |-O O O O O O O O O O O O O O | 140000 +------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-hsw-ep4: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 128G memory ========================================================================================= bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode: 4k/gcc-9/performance/2pmem/ext4/sync/x86_64-rhel-7.6/50%/debian-x86_64-20191114.cgz/200s/randwrite/lkp-hsw-ep4/200G/fio-basic/tb/0x43 commit: 4068664e3c ("ext4: fix extent_status fragmentation for plain files") 244adf6426 ("ext4: make dioread_nolock the default") 4068664e3cd23126 244adf6426ee31a83f397b700d9 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 -25% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x %stddev %change %stddev \ | \ 0.04 ± 14% -0.0 0.01 ± 27% fio.latency_100us% 2.69 ± 15% +1.4 4.14 ± 25% fio.latency_4us% 0.13 +0.2 0.35 ± 17% fio.latency_50ms% 0.97 ± 10% +0.5 1.44 ± 16% fio.latency_50us% 1.205e+09 -58.8% 4.967e+08 ± 10% fio.time.file_system_outputs 19468 ± 28% -87.0% 2523 ± 35% fio.time.involuntary_context_switches 115842 -1.9% 113673 fio.time.minor_page_faults 746.75 ± 3% -60.6% 294.00 ± 12% fio.time.percent_of_cpu_this_job_got 1281 ± 3% -61.0% 500.17 ± 13% fio.time.system_time 218.06 ± 3% -58.5% 90.56 ± 11% fio.time.user_time 317685 ± 4% -17.5% 261941 ± 3% fio.time.voluntary_context_switches 1.506e+08 -58.8% 62083647 ± 10% fio.workload 2940 -58.8% 1212 ± 10% fio.write_bw_MBps 13024 +6.9% 13920 fio.write_clat_95%_us 20736 ± 2% +20.1% 24896 ± 5% fio.write_clat_99%_us 46251 +150.3% 115787 ± 11% fio.write_clat_mean_us 1027511 +77.2% 1821210 ± 3% fio.write_clat_stddev 752793 -58.8% 310367 ± 10% fio.write_iops 6165539 ± 58% -66.5% 2067351 ± 85% cpuidle.C1E.usage 528517 +60.1% 846095 meminfo.Active 223502 +151.9% 562933 meminfo.Active(file) 209593 +162.1% 549446 meminfo.Buffers 37.32 +5.0 42.32 ± 2% mpstat.cpu.all.iowait% 12.03 -6.0 6.06 ± 8% mpstat.cpu.all.sys% 1.47 ± 2% -0.9 0.58 ± 14% mpstat.cpu.all.usr% 49.64 +3.5% 51.38 iostat.cpu.idle 36.95 +13.5% 41.96 ± 2% iostat.cpu.iowait 11.95 -49.1% 6.08 ± 8% iostat.cpu.system 1.46 ± 2% -60.3% 0.58 ± 14% iostat.cpu.user 48.75 +4.1% 50.75 vmstat.cpu.id 36.50 +13.7% 41.50 ± 2% vmstat.cpu.wa 3576950 -64.2% 1281274 ± 10% vmstat.io.bo 209727 +162.2% 549884 vmstat.memory.buff 27.50 +15.5% 31.75 ± 2% vmstat.procs.b 9.50 ± 5% -50.0% 4.75 ± 9% vmstat.procs.r 5870 ± 5% -21.3% 4618 ± 3% vmstat.system.cs 145897 -10.9% 129945 ± 19% vmstat.system.in 62626351 ± 7% -57.9% 26383573 ± 9% numa-numastat.node0.local_node 14010771 ± 33% -65.2% 4882491 ± 19% numa-numastat.node0.numa_foreign 62634553 ± 7% -57.9% 26397605 ± 9% numa-numastat.node0.numa_hit 11906400 ± 26% -62.4% 4479564 ± 27% numa-numastat.node0.numa_miss 11914608 ± 26% -62.3% 4493601 ± 27% numa-numastat.node0.other_node 64372103 ± 7% -56.0% 28345038 ± 12% numa-numastat.node1.local_node 11906400 ± 26% -62.4% 4479564 ± 27% numa-numastat.node1.numa_foreign 64387420 ± 7% -56.0% 28354510 ± 12% numa-numastat.node1.numa_hit 14010771 ± 33% -65.2% 4882491 ± 19% numa-numastat.node1.numa_miss 14026091 ± 33% -65.1% 4891963 ± 19% numa-numastat.node1.other_node 9757 ± 17% -43.4% 5519 ± 18% sched_debug.cfs_rq:/.exec_clock.avg 1747 ± 30% -82.7% 302.81 ± 55% sched_debug.cfs_rq:/.exec_clock.min 174.64 ± 5% -48.1% 90.60 ± 6% sched_debug.cfs_rq:/.load_avg.avg 28268 ± 10% -19.3% 22819 ± 17% sched_debug.cfs_rq:/.min_vruntime.avg 68.07 ± 32% -40.8% 40.28 ± 14% sched_debug.cfs_rq:/.runnable_load_avg.avg 248.98 ± 2% -29.2% 176.25 ± 9% sched_debug.cfs_rq:/.util_avg.avg 69.69 ± 29% -37.9% 43.28 ± 13% sched_debug.cfs_rq:/.util_est_enqueued.avg 502292 +18.5% 595427 ± 7% sched_debug.cpu.max_idle_balance_cost.max 268.33 ±173% +4062.8% 11170 ± 46% sched_debug.cpu.max_idle_balance_cost.stddev 40.23 ± 4% -15.8% 33.88 ± 16% sched_debug.cpu.nr_uninterruptible.max 10.41 ± 7% -15.1% 8.83 ± 7% sched_debug.cpu.nr_uninterruptible.stddev 998.12 ± 4% +43.8% 1434 ± 20% sched_debug.cpu.ttwu_local.stddev 31126962 ± 2% -55.5% 13857067 ± 5% numa-vmstat.node0.nr_dirtied 30262548 ± 2% -57.1% 12975737 ± 6% numa-vmstat.node0.nr_written 7172620 ± 16% -63.9% 2585751 ± 36% numa-vmstat.node0.numa_foreign 27887317 ± 4% -56.2% 12216788 ± 9% numa-vmstat.node0.numa_hit 27877920 ± 4% -56.2% 12202178 ± 9% numa-vmstat.node0.numa_local 4299742 ± 11% -42.7% 2463318 ± 50% numa-vmstat.node0.numa_miss 4309147 ± 11% -42.5% 2477931 ± 50% numa-vmstat.node0.numa_other 31511758 ± 3% -52.9% 14840322 ± 4% numa-vmstat.node1.nr_dirtied 30618693 ± 3% -54.6% 13902619 ± 4% numa-vmstat.node1.nr_written 4299873 ± 11% -42.7% 2463365 ± 50% numa-vmstat.node1.numa_foreign 25420521 ± 6% -48.5% 13090767 ± 11% numa-vmstat.node1.numa_hit 25237813 ± 6% -48.8% 12913429 ± 12% numa-vmstat.node1.numa_local 7172742 ± 16% -63.9% 2585812 ± 36% numa-vmstat.node1.numa_miss 7355456 ± 16% -62.4% 2763153 ± 34% numa-vmstat.node1.numa_other 101.25 ±141% +257.5% 362.00 ± 82% numa-vmstat.node1.workingset_nodes 149.25 ± 3% +24274.2% 36378 ±172% softirqs.CPU12.NET_RX 73193 ± 12% +16.8% 85481 ± 7% softirqs.CPU14.TIMER 92723 ± 5% +8.3% 100391 ± 3% softirqs.CPU16.RCU 91076 ± 8% +11.0% 101106 ± 3% softirqs.CPU17.RCU 71601 ± 11% +21.9% 87292 ± 8% softirqs.CPU17.TIMER 75826 ± 17% +28.6% 97538 ± 7% softirqs.CPU30.TIMER 21336 ± 18% +26.1% 26904 ± 2% softirqs.CPU34.SCHED 66547 ± 26% +33.0% 88538 ± 12% softirqs.CPU36.RCU 69872 ± 11% +24.4% 86887 ± 9% softirqs.CPU37.TIMER 74595 ± 20% +28.5% 95843 ± 7% softirqs.CPU44.RCU 71518 ± 12% +20.8% 86399 ± 9% softirqs.CPU44.TIMER 72709 ± 12% +19.3% 86749 ± 9% softirqs.CPU50.TIMER 71700 ± 12% +22.1% 87578 ± 8% softirqs.CPU6.TIMER 84070 ± 5% +15.8% 97365 ± 8% softirqs.CPU7.RCU 71608 ± 10% +35.8% 97268 ± 15% softirqs.CPU7.TIMER 71680 ± 13% +20.9% 86648 ± 10% softirqs.CPU8.TIMER 355.25 ± 4% -39.5% 215.00 ± 9% slabinfo.biovec-max.active_objs 355.25 ± 4% -39.5% 215.00 ± 9% slabinfo.biovec-max.num_objs 1203 ± 6% -9.2% 1092 ± 2% slabinfo.blkdev_ioc.active_objs 1243 ± 5% -10.1% 1117 ± 4% slabinfo.blkdev_ioc.num_objs 1119 ± 7% -16.6% 933.00 ± 8% slabinfo.dquot.active_objs 1119 ± 7% -16.6% 933.00 ± 8% slabinfo.dquot.num_objs 2958956 ± 2% +77.1% 5238960 slabinfo.ext4_extent_status.active_objs 49004 ± 3% +80.5% 88440 ± 2% slabinfo.ext4_extent_status.active_slabs 4998512 ± 3% +80.5% 9021005 ± 2% slabinfo.ext4_extent_status.num_objs 49004 ± 3% +80.5% 88440 ± 2% slabinfo.ext4_extent_status.num_slabs 1946 ± 6% -64.9% 683.75 ± 26% slabinfo.ext4_io_end.active_objs 1946 ± 6% -64.9% 683.75 ± 26% slabinfo.ext4_io_end.num_objs 4839 ± 6% -32.2% 3279 ± 9% slabinfo.fsnotify_mark_connector.active_objs 4839 ± 6% -32.2% 3279 ± 9% slabinfo.fsnotify_mark_connector.num_objs 31847 ± 3% +304.4% 128797 slabinfo.jbd2_journal_head.active_objs 976.75 ± 2% +300.0% 3907 slabinfo.jbd2_journal_head.active_slabs 33227 ± 2% +299.8% 132847 slabinfo.jbd2_journal_head.num_objs 976.75 ± 2% +300.0% 3907 slabinfo.jbd2_journal_head.num_slabs 6136 ± 62% -95.8% 257.50 ±127% proc-vmstat.allocstall_movable 2149 ± 47% -97.7% 49.75 ±154% proc-vmstat.allocstall_normal 12950089 ± 40% -96.6% 435795 ± 33% proc-vmstat.compact_daemon_free_scanned 2354 ± 15% -48.3% 1217 ± 12% proc-vmstat.compact_daemon_wake 13066988 ± 39% -96.7% 435795 ± 33% proc-vmstat.compact_free_scanned 545508 ± 40% -86.4% 74068 ± 88% proc-vmstat.compact_isolated 148.50 ± 14% +394.6% 734.50 ± 24% proc-vmstat.kswapd_high_wmark_hit_quickly 4562 -37.2% 2865 ± 6% proc-vmstat.kswapd_low_wmark_hit_quickly 76258 -7.2% 70787 proc-vmstat.nr_active_anon 55881 +151.9% 140753 proc-vmstat.nr_active_file 1.511e+08 -58.0% 63509486 ± 10% proc-vmstat.nr_dirtied 1755193 ± 2% +3.5% 1817148 proc-vmstat.nr_dirty 114286 -5.2% 108393 proc-vmstat.nr_shmem 424044 +9.5% 464234 proc-vmstat.nr_slab_reclaimable 32785 -1.0% 32473 proc-vmstat.nr_slab_unreclaimable 1.495e+08 -58.3% 62292440 ± 11% proc-vmstat.nr_written 76268 -7.2% 70798 proc-vmstat.nr_zone_active_anon 55882 +151.9% 140771 proc-vmstat.nr_zone_active_file 1757596 ± 2% +3.5% 1819265 proc-vmstat.nr_zone_write_pending 25870477 ± 6% -63.9% 9327553 ± 15% proc-vmstat.numa_foreign 4277 ± 50% -53.0% 2010 ± 52% proc-vmstat.numa_hint_faults_local 1.268e+08 -56.9% 54628951 ± 10% proc-vmstat.numa_hit 1.268e+08 -56.9% 54605440 ± 10% proc-vmstat.numa_local 25870477 ± 6% -63.9% 9327553 ± 15% proc-vmstat.numa_miss 25894008 ± 6% -63.9% 9351063 ± 15% proc-vmstat.numa_other 4781 -19.7% 3841 ± 2% proc-vmstat.pageoutrun 14089 ± 6% -58.4% 5854 ± 2% proc-vmstat.pgactivate 5717469 ± 5% -60.7% 2248190 ± 13% proc-vmstat.pgalloc_dma32 1.474e+08 -57.9% 62033715 ± 10% proc-vmstat.pgalloc_normal 1.428e+08 -62.0% 54275602 ± 12% proc-vmstat.pgfree 271840 ± 39% -83.2% 45689 ± 74% proc-vmstat.pgmigrate_success 7.259e+08 -64.1% 2.605e+08 ± 10% proc-vmstat.pgpgout 20728683 ± 40% -96.3% 759921 ±110% proc-vmstat.pgscan_direct 92248882 ± 9% -52.9% 43485504 ± 4% proc-vmstat.pgscan_kswapd 20728682 ± 40% -96.3% 759919 ±110% proc-vmstat.pgsteal_direct 92248829 ± 9% -52.9% 43485456 ± 4% proc-vmstat.pgsteal_kswapd 1.16e+08 -25.8% 86033473 ± 5% proc-vmstat.slabs_scanned 9.44 -10.6% 8.44 ± 6% perf-stat.i.MPKI 3.496e+09 -31.7% 2.389e+09 ± 7% perf-stat.i.branch-instructions 1.80 -0.1 1.70 ± 2% perf-stat.i.branch-miss-rate% 62817951 -35.2% 40720451 ± 9% perf-stat.i.branch-misses 32.34 -7.6 24.75 ± 7% perf-stat.i.cache-miss-rate% 58491266 ± 2% -54.1% 26863081 ± 10% perf-stat.i.cache-misses 1.72e+08 -39.5% 1.041e+08 ± 2% perf-stat.i.cache-references 5886 ± 5% -21.7% 4611 ± 3% perf-stat.i.context-switches 1.75 -13.0% 1.52 ± 2% perf-stat.i.cpi 3.23e+10 -40.9% 1.909e+10 ± 4% perf-stat.i.cpu-cycles 98.25 ± 2% -13.9% 84.61 perf-stat.i.cpu-migrations 662.38 +23.4% 817.14 ± 4% perf-stat.i.cycles-between-cache-misses 56505808 ± 18% -35.5% 36423798 ± 8% perf-stat.i.dTLB-load-misses 5.223e+09 -29.3% 3.695e+09 ± 7% perf-stat.i.dTLB-loads 4846212 ± 10% -45.7% 2629596 perf-stat.i.dTLB-store-misses 3.218e+09 -36.3% 2.05e+09 ± 5% perf-stat.i.dTLB-stores 6579800 ± 27% -55.2% 2948423 ± 21% perf-stat.i.iTLB-load-misses 2347506 ± 7% -36.2% 1496584 ± 18% perf-stat.i.iTLB-loads 1.813e+10 -31.3% 1.245e+10 ± 6% perf-stat.i.instructions 3209 ± 21% +44.6% 4639 ± 18% perf-stat.i.instructions-per-iTLB-miss 0.58 +15.0% 0.66 ± 2% perf-stat.i.ipc 0.45 -40.9% 0.27 ± 4% perf-stat.i.metric.GHz 169.13 -32.1% 114.86 ± 6% perf-stat.i.metric.M/sec 3174 -1.1% 3138 perf-stat.i.minor-faults 23143101 ± 2% -50.2% 11521113 ± 11% perf-stat.i.node-load-misses 27856680 ± 2% -55.0% 12547737 ± 17% perf-stat.i.node-loads 3524499 ± 2% -57.9% 1485166 ± 14% perf-stat.i.node-store-misses 3096777 ± 3% -59.1% 1267360 ± 11% perf-stat.i.node-stores 3174 -1.1% 3138 perf-stat.i.page-faults 9.49 -11.6% 8.39 ± 5% perf-stat.overall.MPKI 1.80 -0.1 1.70 ± 2% perf-stat.overall.branch-miss-rate% 34.01 ± 2% -8.2 25.78 ± 8% perf-stat.overall.cache-miss-rate% 1.78 -13.8% 1.54 ± 2% perf-stat.overall.cpi 552.56 ± 2% +29.4% 714.76 ± 6% perf-stat.overall.cycles-between-cache-misses 0.15 ± 12% -0.0 0.13 ± 3% perf-stat.overall.dTLB-store-miss-rate% 72.61 ± 8% -6.4 66.21 perf-stat.overall.iTLB-load-miss-rate% 2940 ± 23% +48.9% 4378 ± 18% perf-stat.overall.instructions-per-iTLB-miss 0.56 +16.1% 0.65 ± 2% perf-stat.overall.ipc 24110 +67.9% 40488 ± 5% perf-stat.overall.path-length 3.479e+09 -31.7% 2.378e+09 ± 7% perf-stat.ps.branch-instructions 62524565 -35.2% 40535951 ± 9% perf-stat.ps.branch-misses 58206685 ± 2% -54.1% 26734388 ± 10% perf-stat.ps.cache-misses 1.711e+08 -39.5% 1.036e+08 ± 2% perf-stat.ps.cache-references 5859 ± 5% -21.7% 4589 ± 3% perf-stat.ps.context-switches 3.214e+10 -40.9% 1.9e+10 ± 4% perf-stat.ps.cpu-cycles 97.78 ± 2% -13.9% 84.21 perf-stat.ps.cpu-migrations 56233980 ± 18% -35.5% 36253818 ± 8% perf-stat.ps.dTLB-load-misses 5.197e+09 -29.3% 3.677e+09 ± 7% perf-stat.ps.dTLB-loads 4822652 ± 10% -45.7% 2616966 perf-stat.ps.dTLB-store-misses 3.203e+09 -36.3% 2.04e+09 ± 5% perf-stat.ps.dTLB-stores 6548345 ± 27% -55.2% 2934364 ± 21% perf-stat.ps.iTLB-load-misses 2335855 ± 7% -36.2% 1489224 ± 18% perf-stat.ps.iTLB-loads 1.804e+10 -31.3% 1.239e+10 ± 6% perf-stat.ps.instructions 3166 -1.3% 3126 perf-stat.ps.minor-faults 23029576 ± 2% -50.2% 11466112 ± 11% perf-stat.ps.node-load-misses 27721032 ± 2% -55.0% 12486426 ± 17% perf-stat.ps.node-loads 3507651 ± 2% -57.8% 1478603 ± 14% perf-stat.ps.node-store-misses 3082410 ± 3% -59.1% 1261681 ± 11% perf-stat.ps.node-stores 3166 -1.3% 3126 perf-stat.ps.page-faults 3.631e+12 -31.1% 2.501e+12 ± 7% perf-stat.total.instructions 130.25 ± 14% +237.8% 440.00 ± 50% interrupts.49:IR-PCI-MSI.1572878-edge.eth0-TxRx-14 1960 ± 36% -68.6% 615.50 ± 22% interrupts.CPU0.NMI:Non-maskable_interrupts 1960 ± 36% -68.6% 615.50 ± 22% interrupts.CPU0.PMI:Performance_monitoring_interrupts 532.00 ± 90% +97.2% 1049 ± 42% interrupts.CPU0.RES:Rescheduling_interrupts 1089 ± 38% -43.2% 619.00 ± 41% interrupts.CPU1.NMI:Non-maskable_interrupts 1089 ± 38% -43.2% 619.00 ± 41% interrupts.CPU1.PMI:Performance_monitoring_interrupts 446.00 ± 55% -83.6% 73.00 ± 48% interrupts.CPU10.RES:Rescheduling_interrupts 1500 ± 40% -67.7% 484.50 ± 31% interrupts.CPU13.NMI:Non-maskable_interrupts 1500 ± 40% -67.7% 484.50 ± 31% interrupts.CPU13.PMI:Performance_monitoring_interrupts 130.25 ± 14% +237.8% 440.00 ± 50% interrupts.CPU14.49:IR-PCI-MSI.1572878-edge.eth0-TxRx-14 259.75 ± 96% -81.1% 49.00 ± 31% interrupts.CPU15.RES:Rescheduling_interrupts 84.25 ± 31% -56.7% 36.50 ± 32% interrupts.CPU16.RES:Rescheduling_interrupts 1518 ± 49% -72.5% 417.75 ± 29% interrupts.CPU2.NMI:Non-maskable_interrupts 1518 ± 49% -72.5% 417.75 ± 29% interrupts.CPU2.PMI:Performance_monitoring_interrupts 1792 ± 27% -59.8% 720.00 ± 44% interrupts.CPU20.NMI:Non-maskable_interrupts 1792 ± 27% -59.8% 720.00 ± 44% interrupts.CPU20.PMI:Performance_monitoring_interrupts 1021 ± 25% -40.6% 606.00 ± 27% interrupts.CPU22.NMI:Non-maskable_interrupts 1021 ± 25% -40.6% 606.00 ± 27% interrupts.CPU22.PMI:Performance_monitoring_interrupts 1426 ± 13% -33.7% 945.25 ± 18% interrupts.CPU23.NMI:Non-maskable_interrupts 1426 ± 13% -33.7% 945.25 ± 18% interrupts.CPU23.PMI:Performance_monitoring_interrupts 1746 ± 18% -56.8% 754.50 ± 38% interrupts.CPU24.NMI:Non-maskable_interrupts 1746 ± 18% -56.8% 754.50 ± 38% interrupts.CPU24.PMI:Performance_monitoring_interrupts 1778 ± 25% -65.1% 620.25 ± 43% interrupts.CPU25.NMI:Non-maskable_interrupts 1778 ± 25% -65.1% 620.25 ± 43% interrupts.CPU25.PMI:Performance_monitoring_interrupts 1748 ± 55% -80.6% 338.50 ± 10% interrupts.CPU27.NMI:Non-maskable_interrupts 1748 ± 55% -80.6% 338.50 ± 10% interrupts.CPU27.PMI:Performance_monitoring_interrupts 554.50 ± 67% -91.2% 49.00 ± 96% interrupts.CPU27.RES:Rescheduling_interrupts 1607 ± 30% -66.3% 542.25 ± 41% interrupts.CPU28.NMI:Non-maskable_interrupts 1607 ± 30% -66.3% 542.25 ± 41% interrupts.CPU28.PMI:Performance_monitoring_interrupts 2257 ± 44% -81.8% 410.75 ± 21% interrupts.CPU29.NMI:Non-maskable_interrupts 2257 ± 44% -81.8% 410.75 ± 21% interrupts.CPU29.PMI:Performance_monitoring_interrupts 203.25 ± 54% -82.8% 35.00 ± 69% interrupts.CPU29.RES:Rescheduling_interrupts 101.50 ± 31% -63.1% 37.50 ± 42% interrupts.CPU30.RES:Rescheduling_interrupts 1482 ± 14% -77.5% 333.50 ± 31% interrupts.CPU31.NMI:Non-maskable_interrupts 1482 ± 14% -77.5% 333.50 ± 31% interrupts.CPU31.PMI:Performance_monitoring_interrupts 3501 ±140% -99.0% 33.75 ± 37% interrupts.CPU31.RES:Rescheduling_interrupts 1944 ± 13% -70.0% 584.25 ± 47% interrupts.CPU32.NMI:Non-maskable_interrupts 1944 ± 13% -70.0% 584.25 ± 47% interrupts.CPU32.PMI:Performance_monitoring_interrupts 2061 ±136% -97.0% 61.50 ± 47% interrupts.CPU33.RES:Rescheduling_interrupts 161.00 ± 58% -56.4% 70.25 ± 44% interrupts.CPU34.RES:Rescheduling_interrupts 1061 ± 12% -40.2% 634.00 ± 38% interrupts.CPU35.NMI:Non-maskable_interrupts 1061 ± 12% -40.2% 634.00 ± 38% interrupts.CPU35.PMI:Performance_monitoring_interrupts 313.75 ± 79% -89.1% 34.25 ± 43% interrupts.CPU35.RES:Rescheduling_interrupts 2638 ± 55% -82.0% 473.75 ± 37% interrupts.CPU36.NMI:Non-maskable_interrupts 2638 ± 55% -82.0% 473.75 ± 37% interrupts.CPU36.PMI:Performance_monitoring_interrupts 684.00 ±104% -92.7% 50.25 ±123% interrupts.CPU36.RES:Rescheduling_interrupts 363.00 ± 12% +176.0% 1001 ± 77% interrupts.CPU37.CAL:Function_call_interrupts 1505 ± 29% -53.2% 704.25 ± 36% interrupts.CPU37.NMI:Non-maskable_interrupts 1505 ± 29% -53.2% 704.25 ± 36% interrupts.CPU37.PMI:Performance_monitoring_interrupts 1503 ± 21% -57.0% 646.50 ± 16% interrupts.CPU38.NMI:Non-maskable_interrupts 1503 ± 21% -57.0% 646.50 ± 16% interrupts.CPU38.PMI:Performance_monitoring_interrupts 371.00 ± 17% +128.6% 848.00 ± 70% interrupts.CPU39.CAL:Function_call_interrupts 1838 ± 21% -51.9% 883.50 ± 43% interrupts.CPU4.NMI:Non-maskable_interrupts 1838 ± 21% -51.9% 883.50 ± 43% interrupts.CPU4.PMI:Performance_monitoring_interrupts 1775 ± 41% -66.8% 588.75 ± 27% interrupts.CPU40.NMI:Non-maskable_interrupts 1775 ± 41% -66.8% 588.75 ± 27% interrupts.CPU40.PMI:Performance_monitoring_interrupts 305.75 ± 32% -90.6% 28.75 ± 27% interrupts.CPU40.RES:Rescheduling_interrupts 1657 ± 32% -65.8% 567.00 ± 30% interrupts.CPU43.NMI:Non-maskable_interrupts 1657 ± 32% -65.8% 567.00 ± 30% interrupts.CPU43.PMI:Performance_monitoring_interrupts 179.25 ±108% -80.1% 35.75 ± 28% interrupts.CPU43.RES:Rescheduling_interrupts 1956 ± 60% -63.2% 719.50 ± 26% interrupts.CPU44.NMI:Non-maskable_interrupts 1956 ± 60% -63.2% 719.50 ± 26% interrupts.CPU44.PMI:Performance_monitoring_interrupts 1959 ± 55% -71.8% 551.75 ± 29% interrupts.CPU46.NMI:Non-maskable_interrupts 1959 ± 55% -71.8% 551.75 ± 29% interrupts.CPU46.PMI:Performance_monitoring_interrupts 1222 ± 21% -40.2% 730.75 ± 31% interrupts.CPU51.NMI:Non-maskable_interrupts 1222 ± 21% -40.2% 730.75 ± 31% interrupts.CPU51.PMI:Performance_monitoring_interrupts 1495 ± 14% -57.1% 641.75 ± 39% interrupts.CPU54.NMI:Non-maskable_interrupts 1495 ± 14% -57.1% 641.75 ± 39% interrupts.CPU54.PMI:Performance_monitoring_interrupts 363.25 ± 4% +121.5% 804.75 ± 84% interrupts.CPU55.CAL:Function_call_interrupts 1859 ± 24% -68.5% 585.25 ± 21% interrupts.CPU56.NMI:Non-maskable_interrupts 1859 ± 24% -68.5% 585.25 ± 21% interrupts.CPU56.PMI:Performance_monitoring_interrupts 1448 ± 20% -50.8% 712.25 ± 37% interrupts.CPU57.NMI:Non-maskable_interrupts 1448 ± 20% -50.8% 712.25 ± 37% interrupts.CPU57.PMI:Performance_monitoring_interrupts 1876 ± 29% -55.1% 842.25 ± 56% interrupts.CPU58.NMI:Non-maskable_interrupts 1876 ± 29% -55.1% 842.25 ± 56% interrupts.CPU58.PMI:Performance_monitoring_interrupts 365.25 ± 4% +178.7% 1018 ± 81% interrupts.CPU59.CAL:Function_call_interrupts 2317 ± 79% -63.8% 839.25 ± 21% interrupts.CPU60.NMI:Non-maskable_interrupts 2317 ± 79% -63.8% 839.25 ± 21% interrupts.CPU60.PMI:Performance_monitoring_interrupts 203.50 ± 71% -80.7% 39.25 ± 34% interrupts.CPU61.RES:Rescheduling_interrupts 64.50 ± 52% -54.7% 29.25 ± 38% interrupts.CPU62.RES:Rescheduling_interrupts 3477 ± 67% -83.4% 576.50 ± 35% interrupts.CPU63.NMI:Non-maskable_interrupts 3477 ± 67% -83.4% 576.50 ± 35% interrupts.CPU63.PMI:Performance_monitoring_interrupts 2162 ± 30% -70.5% 638.00 ± 57% interrupts.CPU66.NMI:Non-maskable_interrupts 2162 ± 30% -70.5% 638.00 ± 57% interrupts.CPU66.PMI:Performance_monitoring_interrupts 118.25 ± 46% -59.6% 47.75 ± 44% interrupts.CPU66.RES:Rescheduling_interrupts 1610 ± 23% -68.6% 506.25 ± 46% interrupts.CPU67.NMI:Non-maskable_interrupts 1610 ± 23% -68.6% 506.25 ± 46% interrupts.CPU67.PMI:Performance_monitoring_interrupts 152.00 ± 35% -72.4% 42.00 ± 42% interrupts.CPU67.RES:Rescheduling_interrupts 1458 ± 15% -58.4% 607.00 ± 37% interrupts.CPU68.NMI:Non-maskable_interrupts 1458 ± 15% -58.4% 607.00 ± 37% interrupts.CPU68.PMI:Performance_monitoring_interrupts 139.00 ± 78% -80.0% 27.75 ± 57% interrupts.CPU68.RES:Rescheduling_interrupts 1873 ± 40% -52.4% 892.75 ± 38% interrupts.CPU70.NMI:Non-maskable_interrupts 1873 ± 40% -52.4% 892.75 ± 38% interrupts.CPU70.PMI:Performance_monitoring_interrupts 23.00 ± 94% +143.5% 56.00 ± 51% interrupts.CPU70.TLB:TLB_shootdowns 1631 ± 26% -49.3% 828.00 ± 58% interrupts.CPU71.NMI:Non-maskable_interrupts 1631 ± 26% -49.3% 828.00 ± 58% interrupts.CPU71.PMI:Performance_monitoring_interrupts 427.00 ± 36% +116.3% 923.75 ± 55% interrupts.CPU8.CAL:Function_call_interrupts 1098 ± 49% -48.4% 566.25 ± 24% interrupts.CPU8.NMI:Non-maskable_interrupts 1098 ± 49% -48.4% 566.25 ± 24% interrupts.CPU8.PMI:Performance_monitoring_interrupts 104.25 ± 6% -81.3% 19.50 ± 17% interrupts.IWI:IRQ_work_interrupts 117583 ± 8% -47.3% 61984 ± 8% interrupts.NMI:Non-maskable_interrupts 117583 ± 8% -47.3% 61984 ± 8% interrupts.PMI:Performance_monitoring_interrupts 47.58 ± 6% -25.5 22.04 ± 14% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 47.41 ± 6% -25.5 21.95 ± 14% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 37.72 ± 7% -17.4 20.34 ± 14% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 37.52 ± 7% -17.3 20.23 ± 14% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 36.49 ± 7% -16.8 19.70 ± 14% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 36.31 ± 7% -16.7 19.60 ± 14% perf-profile.calltrace.cycles-pp.ext4_buffered_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64 35.63 ± 7% -16.5 19.13 ± 14% perf-profile.calltrace.cycles-pp.generic_perform_write.ext4_buffered_write_iter.new_sync_write.vfs_write.ksys_write 27.09 ± 8% -12.2 14.85 ± 14% perf-profile.calltrace.cycles-pp.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter.new_sync_write.vfs_write 8.88 ± 27% -8.9 0.00 perf-profile.calltrace.cycles-pp.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.__writeback_single_inode 8.29 ± 27% -8.3 0.00 perf-profile.calltrace.cycles-pp.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages.do_writepages 9.79 ± 26% -8.0 1.76 ± 14% perf-profile.calltrace.cycles-pp.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes 7.82 ± 27% -7.8 0.00 perf-profile.calltrace.cycles-pp.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_writepages 6.46 ± 63% -6.5 0.00 perf-profile.calltrace.cycles-pp.__x64_sys_fadvise64.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.46 ± 63% -6.5 0.00 perf-profile.calltrace.cycles-pp.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.46 ± 63% -6.5 0.00 perf-profile.calltrace.cycles-pp.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64.entry_SYSCALL_64_after_hwframe 17.01 ± 5% -6.3 10.73 ± 14% perf-profile.calltrace.cycles-pp.ext4_block_write_begin.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter.new_sync_write 6.26 ± 28% -6.3 0.00 perf-profile.calltrace.cycles-pp.ext4_io_submit.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs.mpage_prepare_extent_to_map 6.25 ± 28% -6.2 0.00 perf-profile.calltrace.cycles-pp.submit_bio.ext4_io_submit.ext4_bio_write_page.mpage_submit_page.mpage_process_page_bufs 6.23 ± 28% -6.2 0.00 perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.ext4_io_submit.ext4_bio_write_page.mpage_submit_page 6.09 ± 65% -6.1 0.00 perf-profile.calltrace.cycles-pp.invalidate_mapping_pages.generic_fadvise.ksys_fadvise64_64.__x64_sys_fadvise64.do_syscall_64 5.87 ± 28% -5.9 0.00 perf-profile.calltrace.cycles-pp.pmem_make_request.generic_make_request.submit_bio.ext4_io_submit.ext4_bio_write_page 15.75 ± 5% -5.5 10.21 ± 14% perf-profile.calltrace.cycles-pp.ext4_da_get_block_prep.ext4_block_write_begin.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter 15.59 ± 5% -5.5 10.08 ± 14% perf-profile.calltrace.cycles-pp.ext4_da_map_blocks.ext4_da_get_block_prep.ext4_block_write_begin.ext4_da_write_begin.generic_perform_write 6.11 ± 16% -3.4 2.75 ± 17% perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter.new_sync_write 6.05 ± 16% -3.3 2.72 ± 17% perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter 6.39 ± 9% -3.0 3.40 ± 14% perf-profile.calltrace.cycles-pp.ext4_es_lookup_extent.ext4_da_map_blocks.ext4_da_get_block_prep.ext4_block_write_begin.ext4_da_write_begin 3.60 ± 12% -2.4 1.18 ± 15% perf-profile.calltrace.cycles-pp.__ext4_journal_start_sb.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter.new_sync_write 4.58 ± 8% -2.4 2.19 ± 15% perf-profile.calltrace.cycles-pp.ext4_da_write_end.generic_perform_write.ext4_buffered_write_iter.new_sync_write.vfs_write 2.96 ± 12% -2.0 0.92 ± 18% perf-profile.calltrace.cycles-pp.jbd2__journal_start.__ext4_journal_start_sb.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter 2.79 ± 12% -2.0 0.81 ± 18% perf-profile.calltrace.cycles-pp.start_this_handle.jbd2__journal_start.__ext4_journal_start_sb.ext4_da_write_begin.generic_perform_write 2.21 ± 30% -1.9 0.29 ±100% perf-profile.calltrace.cycles-pp.ext4_end_bio.pmem_make_request.generic_make_request.submit_bio.ext4_io_submit 3.48 ± 27% -1.8 1.66 ± 20% perf-profile.calltrace.cycles-pp.pmem_do_bvec.pmem_make_request.generic_make_request.submit_bio.ext4_io_submit 3.46 ± 27% -1.8 1.65 ± 20% perf-profile.calltrace.cycles-pp.write_pmem.pmem_do_bvec.pmem_make_request.generic_make_request.submit_bio 3.43 ± 27% -1.8 1.63 ± 19% perf-profile.calltrace.cycles-pp.__memcpy_flushcache.write_pmem.pmem_do_bvec.pmem_make_request.generic_make_request 4.08 ± 2% -1.8 2.33 ± 18% perf-profile.calltrace.cycles-pp.__read_extent_tree_block.ext4_find_extent.ext4_ext_map_blocks.ext4_da_map_blocks.ext4_da_get_block_prep 4.00 ± 2% -1.7 2.27 ± 18% perf-profile.calltrace.cycles-pp.__getblk_gfp.__read_extent_tree_block.ext4_find_extent.ext4_ext_map_blocks.ext4_da_map_blocks 3.47 ± 8% -1.7 1.82 ± 13% perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.generic_perform_write.ext4_buffered_write_iter.new_sync_write.vfs_write 3.83 ± 2% -1.6 2.20 ± 18% perf-profile.calltrace.cycles-pp.__find_get_block.__getblk_gfp.__read_extent_tree_block.ext4_find_extent.ext4_ext_map_blocks 3.42 ± 8% -1.6 1.79 ± 13% perf-profile.calltrace.cycles-pp.copyin.iov_iter_copy_from_user_atomic.generic_perform_write.ext4_buffered_write_iter.new_sync_write 3.40 ± 8% -1.6 1.78 ± 13% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.iov_iter_copy_from_user_atomic.generic_perform_write.ext4_buffered_write_iter 3.09 ± 8% -1.5 1.64 ± 15% perf-profile.calltrace.cycles-pp.generic_write_end.ext4_da_write_end.generic_perform_write.ext4_buffered_write_iter.new_sync_write 5.84 ± 2% -1.4 4.44 ± 18% perf-profile.calltrace.cycles-pp.ext4_ext_map_blocks.ext4_da_map_blocks.ext4_da_get_block_prep.ext4_block_write_begin.ext4_da_write_begin 5.61 -1.3 4.30 ± 19% perf-profile.calltrace.cycles-pp.ext4_find_extent.ext4_ext_map_blocks.ext4_da_map_blocks.ext4_da_get_block_prep.ext4_block_write_begin 2.71 ± 8% -1.3 1.45 ± 14% perf-profile.calltrace.cycles-pp.block_write_end.generic_write_end.ext4_da_write_end.generic_perform_write.ext4_buffered_write_iter 2.67 ± 9% -1.2 1.43 ± 14% perf-profile.calltrace.cycles-pp.__block_commit_write.block_write_end.generic_write_end.ext4_da_write_end.generic_perform_write 1.62 ± 55% -1.2 0.40 ± 57% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin.generic_perform_write 3.06 ± 8% -1.1 1.94 ± 4% perf-profile.calltrace.cycles-pp.ext4_es_insert_extent.ext4_da_map_blocks.ext4_da_get_block_prep.ext4_block_write_begin.ext4_da_write_begin 1.98 ± 4% -1.1 0.87 ± 16% perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin.generic_perform_write 2.36 ± 7% -1.0 1.32 ± 18% perf-profile.calltrace.cycles-pp.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin.generic_perform_write 2.33 ± 7% -1.0 1.31 ± 18% perf-profile.calltrace.cycles-pp.xas_load.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin 1.16 ± 11% -0.9 0.27 ±100% perf-profile.calltrace.cycles-pp.create_empty_buffers.ext4_block_write_begin.ext4_da_write_begin.generic_perform_write.ext4_buffered_write_iter 1.57 ± 9% -0.7 0.83 ± 11% perf-profile.calltrace.cycles-pp.mark_buffer_dirty.__block_commit_write.block_write_end.generic_write_end.ext4_da_write_end 1.11 ± 7% -0.7 0.41 ± 57% perf-profile.calltrace.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin 0.93 ± 6% -0.7 0.26 ±100% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64 1.35 ± 11% -0.6 0.70 ± 11% perf-profile.calltrace.cycles-pp.__set_page_dirty.mark_buffer_dirty.__block_commit_write.block_write_end.generic_write_end 0.88 ± 16% -0.6 0.26 ±100% perf-profile.calltrace.cycles-pp.__xa_set_mark.__set_page_dirty.mark_buffer_dirty.__block_commit_write.block_write_end 0.88 ± 5% -0.6 0.26 ±100% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret 1.92 ± 7% -0.6 1.36 ± 2% perf-profile.calltrace.cycles-pp.__es_insert_extent.ext4_es_insert_extent.ext4_da_map_blocks.ext4_da_get_block_prep.ext4_block_write_begin 1.67 ± 3% -0.3 1.33 ± 18% perf-profile.calltrace.cycles-pp.pagecache_get_page.__find_get_block.__getblk_gfp.__read_extent_tree_block.ext4_find_extent 1.63 ± 3% -0.3 1.31 ± 18% perf-profile.calltrace.cycles-pp.find_get_entry.pagecache_get_page.__find_get_block.__getblk_gfp.__read_extent_tree_block 0.71 ± 25% +0.4 1.11 ± 12% perf-profile.calltrace.cycles-pp.find_get_pages_range_tag.pagevec_lookup_range_tag.mpage_prepare_extent_to_map.ext4_writepages.do_writepages 0.71 ± 25% +0.4 1.12 ± 12% perf-profile.calltrace.cycles-pp.pagevec_lookup_range_tag.mpage_prepare_extent_to_map.ext4_writepages.do_writepages.__writeback_single_inode 0.00 +0.6 0.63 ± 11% perf-profile.calltrace.cycles-pp.__es_insert_extent.ext4_es_insert_extent.ext4_map_blocks.mpage_map_and_submit_extent.ext4_writepages 0.00 +0.7 0.71 ± 16% perf-profile.calltrace.cycles-pp.ext4_bio_write_page.mpage_submit_page.mpage_map_and_submit_buffers.mpage_map_and_submit_extent.ext4_writepages 0.00 +0.7 0.74 ± 16% perf-profile.calltrace.cycles-pp.ext4_release_io_end.ext4_put_io_end.ext4_writepages.do_writepages.__writeback_single_inode 0.00 +0.8 0.75 ± 13% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 0.00 +0.8 0.78 ± 14% perf-profile.calltrace.cycles-pp.ext4_es_insert_extent.ext4_map_blocks.ext4_convert_unwritten_extents.ext4_convert_unwritten_io_end_vec.ext4_put_io_end 0.00 +0.8 0.80 ± 2% perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry 0.00 +0.8 0.80 ± 22% perf-profile.calltrace.cycles-pp.es_do_reclaim_extents.es_reclaim_extents.__es_shrink.ext4_es_scan.do_shrink_slab 0.00 +0.8 0.81 ± 22% perf-profile.calltrace.cycles-pp.es_reclaim_extents.__es_shrink.ext4_es_scan.do_shrink_slab.shrink_slab 0.00 +0.8 0.81 ± 13% perf-profile.calltrace.cycles-pp.ext4_find_extent.ext4_ext_map_blocks.ext4_map_blocks.mpage_map_and_submit_extent.ext4_writepages 0.00 +0.8 0.81 ± 22% perf-profile.calltrace.cycles-pp.ext4_es_scan.do_shrink_slab.shrink_slab.shrink_node.balance_pgdat 0.00 +0.8 0.81 ± 22% perf-profile.calltrace.cycles-pp.__es_shrink.ext4_es_scan.do_shrink_slab.shrink_slab.shrink_node 0.00 +0.8 0.82 ± 22% perf-profile.calltrace.cycles-pp.shrink_slab.shrink_node.balance_pgdat.kswapd.kthread 0.00 +0.8 0.82 ± 22% perf-profile.calltrace.cycles-pp.do_shrink_slab.shrink_slab.shrink_node.balance_pgdat.kswapd 0.00 +0.9 0.85 ± 16% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt 0.14 ±173% +0.9 1.04 ± 14% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 0.00 +0.9 0.91 ± 15% perf-profile.calltrace.cycles-pp.mpage_submit_page.mpage_map_and_submit_buffers.mpage_map_and_submit_extent.ext4_writepages.do_writepages 0.00 +1.0 0.98 ± 18% perf-profile.calltrace.cycles-pp.ext4_ext_map_blocks.ext4_map_blocks.ext4_convert_unwritten_extents.ext4_convert_unwritten_io_end_vec.ext4_put_io_end 0.00 +1.0 1.02 ± 2% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary 0.71 ± 12% +1.1 1.76 ± 12% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 0.00 +1.1 1.08 ± 12% perf-profile.calltrace.cycles-pp.ext4_split_extent.ext4_split_convert_extents.ext4_ext_handle_unwritten_extents.ext4_ext_map_blocks.ext4_map_blocks 0.00 +1.1 1.10 ± 12% perf-profile.calltrace.cycles-pp.ext4_split_convert_extents.ext4_ext_handle_unwritten_extents.ext4_ext_map_blocks.ext4_map_blocks.mpage_map_and_submit_extent 0.00 +1.1 1.12 ± 11% perf-profile.calltrace.cycles-pp.ext4_ext_handle_unwritten_extents.ext4_ext_map_blocks.ext4_map_blocks.mpage_map_and_submit_extent.ext4_writepages 0.00 +1.1 1.12 ± 15% perf-profile.calltrace.cycles-pp.mpage_map_and_submit_buffers.mpage_map_and_submit_extent.ext4_writepages.do_writepages.__writeback_single_inode 0.86 ± 36% +1.1 2.00 ± 27% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 0.00 +1.2 1.21 ± 11% perf-profile.calltrace.cycles-pp.ext4_es_insert_extent.ext4_map_blocks.mpage_map_and_submit_extent.ext4_writepages.do_writepages 0.82 ± 10% +1.3 2.15 ± 6% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.43 ±113% +1.3 1.76 ± 27% perf-profile.calltrace.cycles-pp.__remove_mapping.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node 0.00 +1.4 1.35 ± 28% perf-profile.calltrace.cycles-pp.xas_create.xas_store.__delete_from_page_cache.__remove_mapping.shrink_page_list 1.04 ± 33% +1.4 2.42 ± 25% perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 0.00 +1.4 1.38 ± 28% perf-profile.calltrace.cycles-pp.xas_store.__delete_from_page_cache.__remove_mapping.shrink_page_list.shrink_inactive_list 0.00 +1.6 1.56 ± 28% perf-profile.calltrace.cycles-pp.__delete_from_page_cache.__remove_mapping.shrink_page_list.shrink_inactive_list.shrink_lruvec 1.26 ± 16% +1.6 2.88 ± 8% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 0.00 +2.0 2.00 ± 16% perf-profile.calltrace.cycles-pp.ext4_map_blocks.ext4_convert_unwritten_extents.ext4_convert_unwritten_io_end_vec.ext4_put_io_end.ext4_writepages 0.00 +2.1 2.07 ± 12% perf-profile.calltrace.cycles-pp.ext4_ext_map_blocks.ext4_map_blocks.mpage_map_and_submit_extent.ext4_writepages.do_writepages 0.00 +2.3 2.28 ± 17% perf-profile.calltrace.cycles-pp.pmem_make_request.generic_make_request.submit_bio.ext4_io_submit.ext4_writepages 0.22 ±173% +2.5 2.71 ± 27% perf-profile.calltrace.cycles-pp.shrink_page_list.shrink_inactive_list.shrink_lruvec.shrink_node.balance_pgdat 0.00 +2.5 2.49 ± 18% perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.ext4_io_submit.ext4_writepages.do_writepages 0.00 +2.5 2.50 ± 18% perf-profile.calltrace.cycles-pp.submit_bio.ext4_io_submit.ext4_writepages.do_writepages.__writeback_single_inode 0.00 +2.5 2.51 ± 17% perf-profile.calltrace.cycles-pp.ext4_io_submit.ext4_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes 0.00 +2.6 2.58 ± 15% perf-profile.calltrace.cycles-pp.ext4_convert_unwritten_extents.ext4_convert_unwritten_io_end_vec.ext4_put_io_end.ext4_writepages.do_writepages 0.24 ±173% +2.7 2.89 ± 26% perf-profile.calltrace.cycles-pp.shrink_inactive_list.shrink_lruvec.shrink_node.balance_pgdat.kswapd 0.24 ±173% +2.7 2.90 ± 26% perf-profile.calltrace.cycles-pp.shrink_lruvec.shrink_node.balance_pgdat.kswapd.kthread 0.00 +2.9 2.94 ± 14% perf-profile.calltrace.cycles-pp.ext4_convert_unwritten_io_end_vec.ext4_put_io_end.ext4_writepages.do_writepages.__writeback_single_inode 0.34 ±173% +3.4 3.72 ± 25% perf-profile.calltrace.cycles-pp.shrink_node.balance_pgdat.kswapd.kthread.ret_from_fork 0.34 ±173% +3.4 3.73 ± 25% perf-profile.calltrace.cycles-pp.balance_pgdat.kswapd.kthread.ret_from_fork 0.34 ±173% +3.4 3.73 ± 25% perf-profile.calltrace.cycles-pp.kswapd.kthread.ret_from_fork 2.54 ± 20% +3.4 5.97 ± 12% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle 2.73 ± 19% +3.7 6.42 ± 11% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 0.00 +4.0 3.99 ± 15% perf-profile.calltrace.cycles-pp.ext4_put_io_end.ext4_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes 0.00 +4.6 4.57 ± 12% perf-profile.calltrace.cycles-pp.ext4_map_blocks.mpage_map_and_submit_extent.ext4_writepages.do_writepages.__writeback_single_inode 10.05 ± 24% +4.8 14.88 ± 14% perf-profile.calltrace.cycles-pp.wb_workfn.process_one_work.worker_thread.kthread.ret_from_fork 10.05 ± 24% +4.8 14.88 ± 14% perf-profile.calltrace.cycles-pp.wb_writeback.wb_workfn.process_one_work.worker_thread.kthread 10.05 ± 24% +4.8 14.88 ± 14% perf-profile.calltrace.cycles-pp.__writeback_inodes_wb.wb_writeback.wb_workfn.process_one_work.worker_thread 10.05 ± 24% +4.8 14.88 ± 14% perf-profile.calltrace.cycles-pp.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_workfn.process_one_work 10.05 ± 24% +4.8 14.88 ± 14% perf-profile.calltrace.cycles-pp.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_workfn 10.05 ± 24% +4.8 14.88 ± 14% perf-profile.calltrace.cycles-pp.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback 10.01 ± 24% +4.8 14.85 ± 14% perf-profile.calltrace.cycles-pp.ext4_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb 10.13 ± 24% +4.9 14.99 ± 14% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork 10.13 ± 24% +4.9 15.00 ± 14% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork 0.00 +5.8 5.84 ± 12% perf-profile.calltrace.cycles-pp.mpage_map_and_submit_extent.ext4_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes 10.93 ± 21% +8.0 18.96 ± 16% perf-profile.calltrace.cycles-pp.ret_from_fork 10.93 ± 21% +8.0 18.96 ± 16% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork 34.76 ± 5% +9.6 44.39 ± 12% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 36.24 ± 6% +15.2 51.42 ± 9% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 36.41 ± 6% +15.4 51.83 ± 9% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 38.99 ± 5% +16.5 55.50 ± 9% perf-profile.calltrace.cycles-pp.secondary_startup_64 37.52 ± 6% +17.2 54.72 ± 9% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 37.52 ± 6% +17.2 54.74 ± 9% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 37.52 ± 6% +17.2 54.74 ± 9% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 47.75 ± 6% -25.4 22.31 ± 13% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 47.59 ± 6% -25.4 22.22 ± 13% perf-profile.children.cycles-pp.do_syscall_64 37.76 ± 7% -17.4 20.36 ± 14% perf-profile.children.cycles-pp.ksys_write 37.56 ± 7% -17.3 20.25 ± 14% perf-profile.children.cycles-pp.vfs_write 36.53 ± 7% -16.8 19.72 ± 14% perf-profile.children.cycles-pp.new_sync_write 36.31 ± 7% -16.7 19.60 ± 14% perf-profile.children.cycles-pp.ext4_buffered_write_iter 35.67 ± 7% -16.5 19.16 ± 14% perf-profile.children.cycles-pp.generic_perform_write 27.10 ± 8% -12.2 14.86 ± 14% perf-profile.children.cycles-pp.ext4_da_write_begin 9.09 ± 24% -8.8 0.28 ± 23% perf-profile.children.cycles-pp.mpage_process_page_bufs 10.03 ± 24% -8.3 1.76 ± 14% perf-profile.children.cycles-pp.mpage_prepare_extent_to_map 8.57 ± 24% -7.7 0.91 ± 15% perf-profile.children.cycles-pp.mpage_submit_page 8.09 ± 24% -7.4 0.71 ± 16% perf-profile.children.cycles-pp.ext4_bio_write_page 6.49 ± 61% -6.5 0.00 perf-profile.children.cycles-pp.__x64_sys_fadvise64 6.49 ± 61% -6.5 0.00 perf-profile.children.cycles-pp.ksys_fadvise64_64 6.49 ± 61% -6.5 0.00 perf-profile.children.cycles-pp.generic_fadvise 17.01 ± 5% -6.3 10.73 ± 14% perf-profile.children.cycles-pp.ext4_block_write_begin 6.14 ± 64% -6.1 0.00 perf-profile.children.cycles-pp.invalidate_mapping_pages 15.75 ± 5% -5.5 10.21 ± 14% perf-profile.children.cycles-pp.ext4_da_get_block_prep 15.59 ± 5% -5.5 10.09 ± 14% perf-profile.children.cycles-pp.ext4_da_map_blocks 6.57 ± 23% -4.1 2.51 ± 17% perf-profile.children.cycles-pp.ext4_io_submit 6.59 ± 23% -4.0 2.60 ± 17% perf-profile.children.cycles-pp.submit_bio 6.58 ± 23% -4.0 2.59 ± 17% perf-profile.children.cycles-pp.generic_make_request 6.20 ± 23% -3.8 2.37 ± 17% perf-profile.children.cycles-pp.pmem_make_request 7.76 ± 13% -3.7 4.10 ± 17% perf-profile.children.cycles-pp.pagecache_get_page 4.23 ± 54% -3.4 0.86 ± 27% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 6.12 ± 16% -3.4 2.75 ± 17% perf-profile.children.cycles-pp.grab_cache_page_write_begin 3.75 ± 57% -2.8 0.94 ± 26% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 4.60 ± 8% -2.4 2.19 ± 14% perf-profile.children.cycles-pp.ext4_da_write_end 6.42 ± 9% -2.3 4.11 ± 13% perf-profile.children.cycles-pp.ext4_es_lookup_extent 3.67 ± 22% -2.0 1.70 ± 20% perf-profile.children.cycles-pp.pmem_do_bvec 3.65 ± 22% -2.0 1.70 ± 20% perf-profile.children.cycles-pp.write_pmem 3.62 ± 22% -1.9 1.68 ± 20% perf-profile.children.cycles-pp.__memcpy_flushcache 3.68 ± 12% -1.9 1.74 ± 10% perf-profile.children.cycles-pp.__ext4_journal_start_sb 2.31 ± 25% -1.8 0.50 ± 16% perf-profile.children.cycles-pp.ext4_end_bio 3.48 ± 8% -1.7 1.83 ± 13% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic 3.44 ± 8% -1.6 1.80 ± 13% perf-profile.children.cycles-pp.copyin 3.42 ± 8% -1.6 1.79 ± 13% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 3.77 ± 6% -1.6 2.16 ± 15% perf-profile.children.cycles-pp.xas_load 3.02 ± 12% -1.6 1.44 ± 11% perf-profile.children.cycles-pp.jbd2__journal_start 3.11 ± 8% -1.5 1.65 ± 15% perf-profile.children.cycles-pp.generic_write_end 2.84 ± 12% -1.4 1.45 ± 11% perf-profile.children.cycles-pp.start_this_handle 4.02 ± 5% -1.4 2.67 ± 18% perf-profile.children.cycles-pp.find_get_entry 2.72 ± 8% -1.3 1.46 ± 14% perf-profile.children.cycles-pp.block_write_end 2.67 ± 9% -1.2 1.44 ± 14% perf-profile.children.cycles-pp.__block_commit_write 1.73 ± 53% -1.2 0.53 ± 10% perf-profile.children.cycles-pp.__alloc_pages_nodemask 3.94 ± 2% -1.1 2.81 ± 16% perf-profile.children.cycles-pp.__find_get_block 1.98 ± 4% -1.1 0.87 ± 16% perf-profile.children.cycles-pp.add_to_page_cache_lru 4.07 ± 2% -1.0 3.06 ± 16% perf-profile.children.cycles-pp.__getblk_gfp 4.10 ± 2% -1.0 3.12 ± 15% perf-profile.children.cycles-pp.__read_extent_tree_block 1.88 ± 11% -0.8 1.03 ± 11% perf-profile.children.cycles-pp._raw_spin_lock 1.58 ± 9% -0.7 0.84 ± 11% perf-profile.children.cycles-pp.mark_buffer_dirty 1.17 ± 10% -0.7 0.47 ± 18% perf-profile.children.cycles-pp.create_empty_buffers 1.36 ± 11% -0.6 0.71 ± 10% perf-profile.children.cycles-pp.__set_page_dirty 0.82 ± 29% -0.6 0.20 ± 20% perf-profile.children.cycles-pp.__list_del_entry_valid 1.12 ± 7% -0.6 0.51 ± 15% perf-profile.children.cycles-pp.__add_to_page_cache_locked 1.02 ± 6% -0.5 0.47 ± 11% perf-profile.children.cycles-pp.get_page_from_freelist 1.10 ± 18% -0.5 0.56 ± 15% perf-profile.children.cycles-pp.ext4_finish_bio 1.27 ± 10% -0.5 0.73 ± 9% perf-profile.children.cycles-pp.jbd2_journal_stop 1.32 ± 11% -0.5 0.83 ± 8% perf-profile.children.cycles-pp.__ext4_journal_stop 1.02 ± 5% -0.5 0.54 ± 11% perf-profile.children.cycles-pp.syscall_return_via_sysret 0.82 ± 7% -0.5 0.35 ± 17% perf-profile.children.cycles-pp.__lru_cache_add 0.94 ± 6% -0.5 0.48 ± 16% perf-profile.children.cycles-pp.entry_SYSCALL_64 0.76 ± 8% -0.4 0.31 ± 6% perf-profile.children.cycles-pp.rmqueue 0.74 ± 8% -0.4 0.30 ± 18% perf-profile.children.cycles-pp.pagevec_lru_move_fn 0.87 ± 18% -0.4 0.43 ± 16% perf-profile.children.cycles-pp.end_page_writeback 0.86 ± 18% -0.4 0.43 ± 17% perf-profile.children.cycles-pp.test_clear_page_writeback 0.72 ± 11% -0.4 0.30 ± 18% perf-profile.children.cycles-pp.alloc_page_buffers 0.72 ± 19% -0.4 0.31 ± 9% perf-profile.children.cycles-pp.__test_set_page_writeback 0.95 ± 12% -0.4 0.54 ± 18% perf-profile.children.cycles-pp._raw_read_lock 0.88 ± 16% -0.4 0.48 ± 10% perf-profile.children.cycles-pp.__xa_set_mark 0.66 ± 13% -0.4 0.27 ± 17% perf-profile.children.cycles-pp.alloc_buffer_head 0.51 ± 9% -0.4 0.14 ± 18% perf-profile.children.cycles-pp.rmqueue_bulk 0.73 ± 4% -0.4 0.37 ± 14% perf-profile.children.cycles-pp.security_file_permission 0.72 ± 17% -0.3 0.39 ± 10% perf-profile.children.cycles-pp.xas_set_mark 0.68 ± 5% -0.3 0.34 ± 16% perf-profile.children.cycles-pp.common_file_perm 0.64 ± 13% -0.3 0.33 ± 4% perf-profile.children.cycles-pp.ext4_journal_check_start 0.67 ± 10% -0.3 0.38 ± 15% perf-profile.children.cycles-pp.add_transaction_credits 0.48 ± 18% -0.3 0.21 ± 12% perf-profile.children.cycles-pp.clear_page_dirty_for_io 0.53 ± 6% -0.3 0.26 ± 18% perf-profile.children.cycles-pp.aa_file_perm 0.40 ± 15% -0.3 0.14 ± 23% perf-profile.children.cycles-pp.workingset_update_node 0.45 ± 6% -0.2 0.21 ± 12% perf-profile.children.cycles-pp.__mod_lruvec_state 0.43 ± 25% -0.2 0.20 ± 23% perf-profile.children.cycles-pp.bio_alloc_bioset 0.42 ± 13% -0.2 0.21 ± 19% perf-profile.children.cycles-pp.ksys_lseek 0.43 ± 3% -0.2 0.23 ± 19% perf-profile.children.cycles-pp.__pagevec_lru_add_fn 0.37 ± 4% -0.2 0.17 ± 15% perf-profile.children.cycles-pp.account_page_dirtied 0.33 ± 26% -0.2 0.14 ± 20% perf-profile.children.cycles-pp.new_slab 0.32 ± 24% -0.2 0.17 ± 25% perf-profile.children.cycles-pp.mempool_alloc 0.27 ± 9% -0.1 0.14 ± 20% perf-profile.children.cycles-pp.__fdget_pos 0.20 ± 8% -0.1 0.08 ± 24% perf-profile.children.cycles-pp.xas_start 0.31 ± 22% -0.1 0.19 ± 19% perf-profile.children.cycles-pp.generic_make_request_checks 0.24 ± 22% -0.1 0.12 ± 15% perf-profile.children.cycles-pp._raw_spin_lock_irq 0.23 ± 11% -0.1 0.12 ± 20% perf-profile.children.cycles-pp.__fget_light 0.44 ± 8% -0.1 0.33 ± 9% perf-profile.children.cycles-pp.stop_this_handle 0.20 ± 5% -0.1 0.10 ± 9% perf-profile.children.cycles-pp.__mod_node_page_state 0.18 ± 12% -0.1 0.08 ± 8% perf-profile.children.cycles-pp.mem_cgroup_commit_charge 0.18 ± 9% -0.1 0.08 ± 15% perf-profile.children.cycles-pp.mem_cgroup_try_charge 0.20 ± 7% -0.1 0.10 ± 22% perf-profile.children.cycles-pp.page_mapping 0.19 ± 17% -0.1 0.10 ± 13% perf-profile.children.cycles-pp.ext4_llseek 0.18 ± 25% -0.1 0.09 ± 20% perf-profile.children.cycles-pp.bvec_alloc 0.12 ± 21% -0.1 0.04 ± 59% perf-profile.children.cycles-pp.dec_zone_page_state 0.18 ± 11% -0.1 0.11 ± 13% perf-profile.children.cycles-pp.unlock_page 0.16 ± 5% -0.1 0.08 ± 15% perf-profile.children.cycles-pp.__mod_memcg_state 0.19 ± 14% -0.1 0.11 ± 25% perf-profile.children.cycles-pp.node_dirty_ok 0.12 ± 19% -0.1 0.04 ± 57% perf-profile.children.cycles-pp.get_mem_cgroup_from_mm 0.16 ± 21% -0.1 0.09 ± 16% perf-profile.children.cycles-pp.__xa_clear_mark 0.14 ± 3% -0.1 0.07 ± 12% perf-profile.children.cycles-pp.iov_iter_fault_in_readable 0.12 ± 16% -0.1 0.06 ± 9% perf-profile.children.cycles-pp.mem_cgroup_charge_statistics 0.09 ± 20% -0.1 0.03 ±100% perf-profile.children.cycles-pp.blk_throtl_bio 0.12 ± 9% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.down_read 0.10 ± 5% -0.0 0.07 ± 17% perf-profile.children.cycles-pp.ext4_file_write_iter 0.18 ± 3% +0.0 0.20 ± 4% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.06 ± 13% +0.0 0.10 ± 15% perf-profile.children.cycles-pp.update_load_avg 0.07 ± 17% +0.0 0.11 ± 10% perf-profile.children.cycles-pp.mark_page_accessed 0.03 ±100% +0.0 0.07 ± 5% perf-profile.children.cycles-pp.kfree 0.21 ± 7% +0.0 0.25 ± 14% perf-profile.children.cycles-pp.__kmalloc 0.13 ± 8% +0.0 0.17 ± 15% perf-profile.children.cycles-pp.memset_erms 0.19 ± 6% +0.0 0.24 ± 11% perf-profile.children.cycles-pp.__might_sleep 0.00 +0.1 0.05 perf-profile.children.cycles-pp.ext4_init_io_end 0.00 +0.1 0.06 ± 9% perf-profile.children.cycles-pp.schedule 0.00 +0.1 0.06 ± 22% perf-profile.children.cycles-pp.account_process_tick 0.00 +0.1 0.06 ± 7% perf-profile.children.cycles-pp.rcu_eqs_enter 0.00 +0.1 0.06 ± 14% perf-profile.children.cycles-pp.call_cpuidle 0.01 ±173% +0.1 0.07 ± 26% perf-profile.children.cycles-pp.timekeeping_max_deferment 0.00 +0.1 0.06 ± 20% perf-profile.children.cycles-pp.__isolate_lru_page 0.02 ±173% +0.1 0.08 ± 37% perf-profile.children.cycles-pp.delay_tsc 0.00 +0.1 0.06 ± 13% perf-profile.children.cycles-pp.rcu_irq_exit 0.28 ± 5% +0.1 0.34 ± 2% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.07 ± 15% +0.1 0.14 ± 28% perf-profile.children.cycles-pp.native_apic_msr_eoi_write 0.00 +0.1 0.07 ± 17% perf-profile.children.cycles-pp.schedule_idle 0.00 +0.1 0.07 ± 23% perf-profile.children.cycles-pp.__hrtimer_get_next_event 0.21 ± 4% +0.1 0.28 ± 12% perf-profile.children.cycles-pp.__brelse 0.03 ±100% +0.1 0.09 ± 15% perf-profile.children.cycles-pp.timerqueue_add 0.00 +0.1 0.07 ± 16% perf-profile.children.cycles-pp.ext4_data_block_valid 0.00 +0.1 0.07 ± 33% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.03 ±100% +0.1 0.10 ± 11% perf-profile.children.cycles-pp.__intel_pmu_enable_all 0.00 +0.1 0.07 ± 10% perf-profile.children.cycles-pp._raw_write_lock 0.05 ± 58% +0.1 0.12 ± 5% perf-profile.children.cycles-pp.__sched_text_start 0.05 ± 8% +0.1 0.12 ± 8% perf-profile.children.cycles-pp.__remove_hrtimer 0.00 +0.1 0.07 ± 17% perf-profile.children.cycles-pp.idle_cpu 0.01 ±173% +0.1 0.09 ± 31% perf-profile.children.cycles-pp.update_rq_clock 0.03 ±100% +0.1 0.10 ± 16% perf-profile.children.cycles-pp.rcu_idle_exit 0.00 +0.1 0.07 ± 30% perf-profile.children.cycles-pp.crypto_shash_update 0.00 +0.1 0.07 ± 33% perf-profile.children.cycles-pp.pm_qos_request 0.07 ± 17% +0.1 0.14 ± 22% perf-profile.children.cycles-pp.__hrtimer_next_event_base 0.00 +0.1 0.08 ± 19% perf-profile.children.cycles-pp.hrtimer_get_next_event 0.00 +0.1 0.08 ± 14% perf-profile.children.cycles-pp.rcu_dynticks_eqs_exit 0.00 +0.1 0.08 ± 17% perf-profile.children.cycles-pp.__check_block_validity 0.00 +0.1 0.08 ± 17% perf-profile.children.cycles-pp.ext4_inode_csum 0.00 +0.1 0.08 ± 8% perf-profile.children.cycles-pp.ext4_alloc_io_end_vec 0.03 ±100% +0.1 0.11 ± 21% perf-profile.children.cycles-pp.submit_bh_wbc 0.03 ±100% +0.1 0.11 ± 22% perf-profile.children.cycles-pp.enqueue_hrtimer 0.00 +0.1 0.09 ± 26% perf-profile.children.cycles-pp.tsc_verify_tsc_adjust 0.03 ±100% +0.1 0.11 ± 22% perf-profile.children.cycles-pp.ktime_get_update_offsets_now 0.01 ±173% +0.1 0.10 ± 24% perf-profile.children.cycles-pp.interrupt_entry 0.00 +0.1 0.09 ± 30% perf-profile.children.cycles-pp.run_posix_cpu_timers 0.00 +0.1 0.09 ± 24% perf-profile.children.cycles-pp.ksoftirqd_running 0.00 +0.1 0.09 ± 17% perf-profile.children.cycles-pp.timerqueue_del 0.01 ±173% +0.1 0.10 ± 58% perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu 0.00 +0.1 0.09 ± 19% perf-profile.children.cycles-pp.ext4_inode_csum_set 0.00 +0.1 0.09 ± 24% perf-profile.children.cycles-pp.arch_cpu_idle_enter 0.14 ± 6% +0.1 0.23 ± 7% perf-profile.children.cycles-pp.ext4_ext_drop_refs 0.00 +0.1 0.10 ± 15% perf-profile.children.cycles-pp.rwsem_down_read_slowpath 0.00 +0.1 0.10 ± 27% perf-profile.children.cycles-pp.ext4_extent_block_csum_set 0.00 +0.1 0.10 ± 12% perf-profile.children.cycles-pp.find_get_pages_range 0.05 ± 9% +0.1 0.16 ± 19% perf-profile.children.cycles-pp.native_sched_clock 0.00 +0.1 0.10 ± 17% perf-profile.children.cycles-pp.jbd2_write_access_granted 0.00 +0.1 0.10 ± 14% perf-profile.children.cycles-pp.pagevec_lookup_range 0.00 +0.1 0.10 ± 22% perf-profile.children.cycles-pp.nr_iowait_cpu 0.00 +0.1 0.10 ± 24% perf-profile.children.cycles-pp.update_ts_time_stats 0.00 +0.1 0.10 ± 35% perf-profile.children.cycles-pp.pm_qos_read_value 0.07 ± 31% +0.1 0.18 ± 14% perf-profile.children.cycles-pp.note_gp_changes 0.06 ± 14% +0.1 0.16 ± 11% perf-profile.children.cycles-pp.hrtimer_next_event_without 0.06 ± 13% +0.1 0.17 ± 16% perf-profile.children.cycles-pp.__ext4_get_inode_loc 0.00 +0.1 0.11 ± 10% perf-profile.children.cycles-pp.rwsem_spin_on_owner 0.00 +0.1 0.11 ± 24% perf-profile.children.cycles-pp.rwsem_down_write_slowpath 0.08 ± 19% +0.1 0.20 ± 20% perf-profile.children.cycles-pp.rcu_sched_clock_irq 0.06 ± 11% +0.1 0.17 ± 20% perf-profile.children.cycles-pp.sched_clock 0.04 ±101% +0.1 0.16 ± 18% perf-profile.children.cycles-pp.isolate_lru_pages 0.01 ±173% +0.1 0.13 ± 12% perf-profile.children.cycles-pp.ext4_es_free_extent 0.23 ± 11% +0.1 0.35 ± 3% perf-profile.children.cycles-pp.read_tsc 0.00 +0.1 0.12 ± 12% perf-profile.children.cycles-pp.crc_1 0.00 +0.1 0.12 ± 37% perf-profile.children.cycles-pp.osq_lock 0.00 +0.1 0.12 ± 27% perf-profile.children.cycles-pp.__ext4_ext_dirty 0.10 ± 30% +0.1 0.22 ± 42% perf-profile.children.cycles-pp.tick_nohz_irq_exit 0.24 ± 18% +0.1 0.36 ± 11% perf-profile.children.cycles-pp.xas_find_marked 0.00 +0.1 0.13 ± 38% perf-profile.children.cycles-pp.mutex_lock_io 0.00 +0.1 0.13 ± 38% perf-profile.children.cycles-pp.__mutex_lock 0.14 ± 6% +0.1 0.27 ± 10% perf-profile.children.cycles-pp.native_irq_return_iret 0.07 ± 11% +0.1 0.21 ± 20% perf-profile.children.cycles-pp.sched_clock_cpu 0.09 ± 7% +0.1 0.23 ± 8% perf-profile.children.cycles-pp.find_next_bit 0.08 ± 19% +0.1 0.23 ± 14% perf-profile.children.cycles-pp.io_serial_in 0.00 +0.1 0.14 ± 31% perf-profile.children.cycles-pp.__jbd2_log_wait_for_space 0.00 +0.1 0.14 ± 31% perf-profile.children.cycles-pp.jbd2_log_do_checkpoint 0.06 ± 16% +0.1 0.21 ± 27% perf-profile.children.cycles-pp.jbd2_journal_commit_transaction 0.06 ± 16% +0.1 0.21 ± 27% perf-profile.children.cycles-pp.kjournald2 0.00 +0.2 0.15 ± 19% perf-profile.children.cycles-pp.jbd2_journal_get_write_access 0.00 +0.2 0.15 ± 17% perf-profile.children.cycles-pp.ext4_ext_try_to_merge_right 0.07 ± 34% +0.2 0.23 ± 30% perf-profile.children.cycles-pp.update_sd_lb_stats 0.17 ± 5% +0.2 0.33 ± 10% perf-profile.children.cycles-pp.lapic_next_deadline 0.07 ± 10% +0.2 0.23 ± 14% perf-profile.children.cycles-pp.ext4_reserve_inode_write 0.00 +0.2 0.16 ± 16% perf-profile.children.cycles-pp.ext4_ext_try_to_merge 0.00 +0.2 0.17 ± 13% perf-profile.children.cycles-pp.jbd2_journal_dirty_metadata 0.08 ± 37% +0.2 0.25 ± 31% perf-profile.children.cycles-pp.find_busiest_group 0.08 ± 62% +0.2 0.26 ± 23% perf-profile.children.cycles-pp.wait_for_xmitr 0.09 ± 59% +0.2 0.26 ± 24% perf-profile.children.cycles-pp.uart_console_write 0.22 ± 12% +0.2 0.40 ± 9% perf-profile.children.cycles-pp.scheduler_tick 0.08 ± 63% +0.2 0.26 ± 23% perf-profile.children.cycles-pp.serial8250_console_putchar 0.09 ± 60% +0.2 0.27 ± 21% perf-profile.children.cycles-pp.serial8250_console_write 0.09 ± 60% +0.2 0.27 ± 23% perf-profile.children.cycles-pp.console_unlock 0.18 ± 4% +0.2 0.36 ± 8% perf-profile.children.cycles-pp.native_write_msr 0.10 ± 62% +0.2 0.28 ± 22% perf-profile.children.cycles-pp.irq_work_run_list 0.00 +0.2 0.19 ± 16% perf-profile.children.cycles-pp.rwsem_optimistic_spin 0.09 ± 10% +0.2 0.28 ± 27% perf-profile.children.cycles-pp.cpuidle_governor_latency_req 0.20 ± 15% +0.2 0.40 ± 16% perf-profile.children.cycles-pp.rb_next 0.14 ± 28% +0.2 0.34 ± 33% perf-profile.children.cycles-pp.update_blocked_averages 0.15 ± 33% +0.2 0.35 ± 31% perf-profile.children.cycles-pp.run_timer_softirq 0.14 ± 36% +0.2 0.35 ± 32% perf-profile.children.cycles-pp.__run_timers 0.17 ± 6% +0.2 0.38 ± 15% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 0.15 ± 31% +0.2 0.36 ± 33% perf-profile.children.cycles-pp.run_rebalance_domains 0.14 ± 34% +0.2 0.37 ± 20% perf-profile.children.cycles-pp.rb_erase 0.13 ± 5% +0.2 0.36 ± 11% perf-profile.children.cycles-pp.__next_timer_interrupt 0.00 +0.2 0.23 ± 16% perf-profile.children.cycles-pp.memmove 0.11 ± 25% +0.2 0.35 ± 23% perf-profile.children.cycles-pp.tick_irq_enter 0.13 ± 31% +0.2 0.38 ± 29% perf-profile.children.cycles-pp.load_balance 0.00 +0.2 0.25 ± 12% perf-profile.children.cycles-pp.__ext4_handle_dirty_metadata 0.00 +0.2 0.25 ± 16% perf-profile.children.cycles-pp.__ext4_journal_get_write_access 0.00 +0.3 0.25 ± 8% perf-profile.children.cycles-pp.__wake_up_common_lock 0.79 ± 12% +0.3 1.05 ± 10% perf-profile.children.cycles-pp.__es_tree_search 0.00 +0.3 0.27 ± 18% perf-profile.children.cycles-pp.ext4_ext_insert_extent 0.14 ± 20% +0.3 0.42 ± 24% perf-profile.children.cycles-pp.irq_enter 0.00 +0.3 0.28 ± 12% perf-profile.children.cycles-pp.jbd2_journal_start_reserved 0.20 ± 38% +0.3 0.50 ± 30% perf-profile.children.cycles-pp.rebalance_domains 0.18 ± 46% +0.3 0.49 ± 30% perf-profile.children.cycles-pp.get_partial_node 0.03 ±100% +0.3 0.35 ± 11% perf-profile.children.cycles-pp.ext4_do_update_inode 0.95 ± 11% +0.3 1.27 ± 9% perf-profile.children.cycles-pp.__es_remove_extent 0.20 ± 3% +0.3 0.53 ± 10% perf-profile.children.cycles-pp.get_next_timer_interrupt 0.00 +0.3 0.35 ± 9% perf-profile.children.cycles-pp.__ext4_journal_start_reserved 0.03 ±100% +0.4 0.38 ± 13% perf-profile.children.cycles-pp.ext4_mark_iloc_dirty 0.00 +0.4 0.36 ± 14% perf-profile.children.cycles-pp.__es_find_extent_range 0.00 +0.4 0.36 ± 11% perf-profile.children.cycles-pp.crc32c_pcl_intel_update 0.00 +0.4 0.37 ± 15% perf-profile.children.cycles-pp.__es_scan_range 0.74 ± 21% +0.4 1.11 ± 12% perf-profile.children.cycles-pp.find_get_pages_range_tag 0.74 ± 21% +0.4 1.12 ± 12% perf-profile.children.cycles-pp.pagevec_lookup_range_tag 0.00 +0.4 0.42 ± 15% perf-profile.children.cycles-pp.ext4_es_scan_range 2.03 ± 7% +0.5 2.48 ± 6% perf-profile.children.cycles-pp.__es_insert_extent 0.43 ± 15% +0.5 0.90 ± 10% perf-profile.children.cycles-pp.update_process_times 0.29 ± 58% +0.5 0.81 ± 22% perf-profile.children.cycles-pp.es_reclaim_extents 0.29 ± 58% +0.5 0.81 ± 22% perf-profile.children.cycles-pp.es_do_reclaim_extents 0.30 ± 58% +0.5 0.81 ± 22% perf-profile.children.cycles-pp.ext4_es_scan 0.30 ± 58% +0.5 0.81 ± 22% perf-profile.children.cycles-pp.__es_shrink 0.30 ± 59% +0.5 0.82 ± 22% perf-profile.children.cycles-pp.shrink_slab 0.30 ± 59% +0.5 0.82 ± 22% perf-profile.children.cycles-pp.do_shrink_slab 0.12 ± 18% +0.5 0.65 ± 14% perf-profile.children.cycles-pp.ext4_mark_inode_dirty 0.30 ± 12% +0.5 0.83 ± 2% perf-profile.children.cycles-pp.tick_nohz_next_event 0.46 ± 14% +0.5 0.99 ± 12% perf-profile.children.cycles-pp.tick_sched_handle 0.00 +0.6 0.59 ± 15% perf-profile.children.cycles-pp.ext4_split_extent_at 0.56 ± 17% +0.6 1.21 ± 12% perf-profile.children.cycles-pp.tick_sched_timer 0.38 ± 9% +0.7 1.04 ± 2% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 0.00 +0.7 0.74 ± 16% perf-profile.children.cycles-pp.ext4_release_io_end 3.09 ± 8% +0.8 3.94 ± 6% perf-profile.children.cycles-pp.ext4_es_insert_extent 0.11 ±100% +1.0 1.13 ± 14% perf-profile.children.cycles-pp.mpage_map_and_submit_buffers 0.53 ± 33% +1.0 1.56 ± 28% perf-profile.children.cycles-pp.__delete_from_page_cache 0.01 ±173% +1.1 1.08 ± 12% perf-profile.children.cycles-pp.ext4_split_extent 0.99 ± 30% +1.1 2.08 ± 26% perf-profile.children.cycles-pp.__softirqentry_text_start 0.27 ± 54% +1.1 1.36 ± 28% perf-profile.children.cycles-pp.xas_create 0.89 ± 9% +1.1 1.99 ± 10% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.00 +1.1 1.10 ± 12% perf-profile.children.cycles-pp.ext4_split_convert_extents 0.83 ± 11% +1.4 2.21 ± 6% perf-profile.children.cycles-pp.menu_select 1.13 ± 32% +1.4 2.53 ± 25% perf-profile.children.cycles-pp.irq_exit 0.02 ±173% +1.5 1.47 ± 12% perf-profile.children.cycles-pp.ext4_ext_handle_unwritten_extents 5.89 +1.6 7.50 ± 13% perf-profile.children.cycles-pp.ext4_ext_map_blocks 1.52 ± 14% +1.7 3.17 ± 8% perf-profile.children.cycles-pp.hrtimer_interrupt 0.76 ± 94% +2.0 2.72 ± 27% perf-profile.children.cycles-pp.shrink_page_list 0.87 ± 96% +2.0 2.89 ± 26% perf-profile.children.cycles-pp.shrink_inactive_list 0.87 ± 96% +2.0 2.90 ± 26% perf-profile.children.cycles-pp.shrink_lruvec 1.18 ± 86% +2.5 3.72 ± 25% perf-profile.children.cycles-pp.shrink_node 0.00 +2.6 2.58 ± 15% perf-profile.children.cycles-pp.ext4_convert_unwritten_extents 0.00 +2.9 2.94 ± 14% perf-profile.children.cycles-pp.ext4_convert_unwritten_io_end_vec 0.58 ± 79% +3.1 3.73 ± 25% perf-profile.children.cycles-pp.balance_pgdat 0.58 ± 79% +3.1 3.73 ± 25% perf-profile.children.cycles-pp.kswapd 2.90 ± 18% +3.5 6.38 ± 12% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 3.13 ± 16% +3.8 6.90 ± 11% perf-profile.children.cycles-pp.apic_timer_interrupt 0.00 +4.0 3.99 ± 15% perf-profile.children.cycles-pp.ext4_put_io_end 10.41 ± 21% +4.5 14.88 ± 14% perf-profile.children.cycles-pp.do_writepages 10.37 ± 21% +4.5 14.86 ± 14% perf-profile.children.cycles-pp.ext4_writepages 10.05 ± 24% +4.8 14.88 ± 14% perf-profile.children.cycles-pp.wb_workfn 10.05 ± 24% +4.8 14.88 ± 14% perf-profile.children.cycles-pp.wb_writeback 10.05 ± 24% +4.8 14.88 ± 14% perf-profile.children.cycles-pp.__writeback_inodes_wb 10.05 ± 24% +4.8 14.88 ± 14% perf-profile.children.cycles-pp.writeback_sb_inodes 10.05 ± 24% +4.8 14.88 ± 14% perf-profile.children.cycles-pp.__writeback_single_inode 10.13 ± 24% +4.9 14.99 ± 14% perf-profile.children.cycles-pp.process_one_work 10.13 ± 24% +4.9 15.00 ± 14% perf-profile.children.cycles-pp.worker_thread 0.22 ±102% +5.6 5.84 ± 12% perf-profile.children.cycles-pp.mpage_map_and_submit_extent 0.11 ±103% +6.5 6.58 ± 13% perf-profile.children.cycles-pp.ext4_map_blocks 10.93 ± 21% +8.0 18.96 ± 16% perf-profile.children.cycles-pp.kthread 10.93 ± 21% +8.0 18.97 ± 16% perf-profile.children.cycles-pp.ret_from_fork 34.79 ± 5% +9.7 44.47 ± 12% perf-profile.children.cycles-pp.intel_idle 37.87 ± 6% +14.7 52.53 ± 9% perf-profile.children.cycles-pp.cpuidle_enter_state 37.87 ± 6% +14.7 52.54 ± 9% perf-profile.children.cycles-pp.cpuidle_enter 38.99 ± 5% +16.5 55.50 ± 9% perf-profile.children.cycles-pp.secondary_startup_64 38.99 ± 5% +16.5 55.50 ± 9% perf-profile.children.cycles-pp.cpu_startup_entry 39.00 ± 5% +16.5 55.51 ± 9% perf-profile.children.cycles-pp.do_idle 37.52 ± 6% +17.2 54.74 ± 9% perf-profile.children.cycles-pp.start_secondary 4.23 ± 54% -3.4 0.86 ± 27% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 6.15 ± 9% -2.4 3.76 ± 14% perf-profile.self.cycles-pp.ext4_es_lookup_extent 3.60 ± 22% -2.0 1.64 ± 17% perf-profile.self.cycles-pp.__memcpy_flushcache 3.40 ± 8% -1.6 1.79 ± 13% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 3.55 ± 6% -1.5 2.07 ± 15% perf-profile.self.cycles-pp.xas_load 2.61 ± 6% -1.3 1.30 ± 15% perf-profile.self.cycles-pp.do_syscall_64 1.28 ± 15% -0.7 0.56 ± 20% perf-profile.self.cycles-pp.start_this_handle 0.82 ± 29% -0.6 0.19 ± 23% perf-profile.self.cycles-pp.__list_del_entry_valid 1.01 ± 5% -0.5 0.53 ± 12% perf-profile.self.cycles-pp.syscall_return_via_sysret 0.99 ± 9% -0.5 0.52 ± 19% perf-profile.self.cycles-pp.__block_commit_write 0.94 ± 6% -0.5 0.48 ± 17% perf-profile.self.cycles-pp.entry_SYSCALL_64 0.78 ± 13% -0.4 0.35 ± 11% perf-profile.self.cycles-pp.jbd2_journal_stop 0.66 ± 10% -0.4 0.23 ± 15% perf-profile.self.cycles-pp.add_transaction_credits 0.94 ± 12% -0.4 0.53 ± 17% perf-profile.self.cycles-pp._raw_read_lock 0.62 ± 19% -0.3 0.27 ± 21% perf-profile.self.cycles-pp.mpage_process_page_bufs 0.80 ± 11% -0.3 0.45 ± 17% perf-profile.self.cycles-pp._raw_spin_lock 0.72 ± 17% -0.3 0.39 ± 10% perf-profile.self.cycles-pp.xas_set_mark 0.59 ± 14% -0.3 0.29 ± 5% perf-profile.self.cycles-pp.ext4_journal_check_start 0.52 ± 7% -0.3 0.26 ± 18% perf-profile.self.cycles-pp.aa_file_perm 1.01 ± 6% -0.3 0.75 ± 20% perf-profile.self.cycles-pp.find_get_entry 0.42 ± 8% -0.2 0.22 ± 8% perf-profile.self.cycles-pp.stop_this_handle 0.34 ± 9% -0.2 0.14 ± 23% perf-profile.self.cycles-pp.workingset_update_node 0.33 ± 19% -0.2 0.14 ± 21% perf-profile.self.cycles-pp.ext4_bio_write_page 0.22 ± 39% -0.2 0.03 ±102% perf-profile.self.cycles-pp.xas_store 0.34 ± 8% -0.2 0.16 ± 25% perf-profile.self.cycles-pp.generic_write_end 0.24 ± 21% -0.2 0.09 ± 22% perf-profile.self.cycles-pp.test_clear_page_writeback 0.20 ± 20% -0.1 0.07 ± 17% perf-profile.self.cycles-pp.__test_set_page_writeback 0.24 ± 17% -0.1 0.11 ± 12% perf-profile.self.cycles-pp.clear_page_dirty_for_io 0.19 ± 10% -0.1 0.07 ± 24% perf-profile.self.cycles-pp.xas_start 0.23 ± 11% -0.1 0.11 ± 21% perf-profile.self.cycles-pp.__fget_light 0.25 ± 2% -0.1 0.14 ± 18% perf-profile.self.cycles-pp.__pagevec_lru_add_fn 0.20 ± 11% -0.1 0.10 ± 15% perf-profile.self.cycles-pp.ext4_da_write_begin 0.23 ± 22% -0.1 0.13 ± 11% perf-profile.self.cycles-pp.ext4_finish_bio 0.19 ± 5% -0.1 0.09 ± 11% perf-profile.self.cycles-pp.__mod_node_page_state 0.19 ± 6% -0.1 0.09 ± 24% perf-profile.self.cycles-pp.page_mapping 0.17 ± 2% -0.1 0.08 ± 16% perf-profile.self.cycles-pp.__mod_lruvec_state 0.21 ± 24% -0.1 0.12 ± 19% perf-profile.self.cycles-pp.new_slab 0.20 ± 9% -0.1 0.11 ± 9% perf-profile.self.cycles-pp.ext4_da_write_end 0.18 ± 18% -0.1 0.10 ± 15% perf-profile.self.cycles-pp.ext4_llseek 0.19 ± 10% -0.1 0.11 ± 21% perf-profile.self.cycles-pp.rmqueue_bulk 0.18 ± 11% -0.1 0.11 ± 13% perf-profile.self.cycles-pp.unlock_page 0.15 ± 4% -0.1 0.07 ± 20% perf-profile.self.cycles-pp.__add_to_page_cache_locked 0.15 ± 4% -0.1 0.08 ± 14% perf-profile.self.cycles-pp.__mod_memcg_state 0.16 ± 2% -0.1 0.09 ± 30% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.12 ± 19% -0.1 0.04 ± 57% perf-profile.self.cycles-pp.get_mem_cgroup_from_mm 0.10 ± 13% -0.1 0.03 ±100% perf-profile.self.cycles-pp.mpage_prepare_extent_to_map 0.12 ± 10% -0.1 0.05 ± 9% perf-profile.self.cycles-pp.account_page_dirtied 0.10 ± 21% -0.1 0.04 ± 58% perf-profile.self.cycles-pp.dec_zone_page_state 0.11 ± 11% -0.1 0.04 ± 59% perf-profile.self.cycles-pp.create_empty_buffers 0.15 ± 7% -0.1 0.08 ± 17% perf-profile.self.cycles-pp.common_file_perm 0.14 ± 10% -0.1 0.08 ± 23% perf-profile.self.cycles-pp.generic_perform_write 0.13 ± 28% -0.1 0.07 ± 19% perf-profile.self.cycles-pp.pmem_make_request 0.12 ± 4% -0.1 0.06 ± 13% perf-profile.self.cycles-pp.iov_iter_fault_in_readable 0.12 ± 14% -0.1 0.07 ± 59% perf-profile.self.cycles-pp.node_dirty_ok 0.10 ± 23% -0.1 0.04 ± 63% perf-profile.self.cycles-pp.___slab_alloc 0.11 ± 4% -0.1 0.05 ± 63% perf-profile.self.cycles-pp.mark_buffer_dirty 0.12 ± 10% -0.1 0.07 ± 19% perf-profile.self.cycles-pp.rmqueue 0.09 ± 11% -0.0 0.04 ± 58% perf-profile.self.cycles-pp.__lru_cache_add 0.07 ± 22% -0.0 0.03 ±100% perf-profile.self.cycles-pp.generic_make_request_checks 0.09 ± 4% -0.0 0.04 ± 57% perf-profile.self.cycles-pp.ext4_block_write_begin 0.09 ± 7% -0.0 0.05 ± 58% perf-profile.self.cycles-pp.__mark_inode_dirty 0.08 ± 5% -0.0 0.07 ± 17% perf-profile.self.cycles-pp.ext4_file_write_iter 0.07 ± 7% +0.0 0.09 ± 13% perf-profile.self.cycles-pp.ext4_es_insert_extent 0.15 ± 3% +0.0 0.18 ± 7% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax 0.07 ± 16% +0.0 0.10 ± 10% perf-profile.self.cycles-pp.mark_page_accessed 0.03 ±100% +0.0 0.07 ± 7% perf-profile.self.cycles-pp.jbd2__journal_start 0.03 ±100% +0.0 0.07 ± 5% perf-profile.self.cycles-pp.kfree 0.17 ± 4% +0.0 0.21 ± 12% perf-profile.self.cycles-pp.__might_sleep 0.06 ± 11% +0.0 0.10 ± 14% perf-profile.self.cycles-pp.ext4_ext_map_blocks 0.05 ± 8% +0.0 0.10 ± 21% perf-profile.self.cycles-pp.__getblk_gfp 0.08 ± 11% +0.0 0.12 ± 11% perf-profile.self.cycles-pp.__read_extent_tree_block 0.01 ±173% +0.1 0.07 ± 31% perf-profile.self.cycles-pp.rebalance_domains 0.01 ±173% +0.1 0.07 ± 25% perf-profile.self.cycles-pp.timekeeping_max_deferment 0.06 ± 11% +0.1 0.12 ± 15% perf-profile.self.cycles-pp.hrtimer_interrupt 0.00 +0.1 0.06 ± 22% perf-profile.self.cycles-pp.account_process_tick 0.00 +0.1 0.06 ± 14% perf-profile.self.cycles-pp.ext4_map_blocks 0.00 +0.1 0.06 ± 7% perf-profile.self.cycles-pp.rcu_eqs_enter 0.00 +0.1 0.06 ± 14% perf-profile.self.cycles-pp.call_cpuidle 0.07 ± 15% +0.1 0.13 ± 29% perf-profile.self.cycles-pp.native_apic_msr_eoi_write 0.00 +0.1 0.06 ± 20% perf-profile.self.cycles-pp.__isolate_lru_page 0.00 +0.1 0.06 ± 16% perf-profile.self.cycles-pp.lapic_next_deadline 0.02 ±173% +0.1 0.08 ± 37% perf-profile.self.cycles-pp.delay_tsc 0.00 +0.1 0.06 ± 26% perf-profile.self.cycles-pp.get_next_timer_interrupt 0.03 ±100% +0.1 0.09 ± 11% perf-profile.self.cycles-pp.__ext4_journal_stop 0.00 +0.1 0.06 ± 13% perf-profile.self.cycles-pp.timerqueue_add 0.00 +0.1 0.06 ± 6% perf-profile.self.cycles-pp._raw_write_lock 0.00 +0.1 0.06 ± 13% perf-profile.self.cycles-pp.rcu_irq_exit 0.21 ± 4% +0.1 0.28 ± 11% perf-profile.self.cycles-pp.__brelse 0.01 ±173% +0.1 0.08 ± 14% perf-profile.self.cycles-pp.free_unref_page_list 0.00 +0.1 0.07 ± 19% perf-profile.self.cycles-pp.rwsem_optimistic_spin 0.00 +0.1 0.07 ± 12% perf-profile.self.cycles-pp.ext4_ext_handle_unwritten_extents 0.00 +0.1 0.07 ± 22% perf-profile.self.cycles-pp.idle_cpu 0.06 ± 16% +0.1 0.13 ± 19% perf-profile.self.cycles-pp.__hrtimer_next_event_base 0.01 ±173% +0.1 0.08 ± 15% perf-profile.self.cycles-pp.scheduler_tick 0.00 +0.1 0.07 ± 20% perf-profile.self.cycles-pp.ktime_get_update_offsets_now 0.00 +0.1 0.07 ± 30% perf-profile.self.cycles-pp.crypto_shash_update 0.00 +0.1 0.07 ± 33% perf-profile.self.cycles-pp.pm_qos_request 0.00 +0.1 0.07 ± 14% perf-profile.self.cycles-pp.__run_timers 0.24 +0.1 0.31 ± 4% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.00 +0.1 0.08 ± 5% perf-profile.self.cycles-pp.perf_mux_hrtimer_handler 0.00 +0.1 0.08 ± 14% perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit 0.03 ±100% +0.1 0.10 ± 20% perf-profile.self.cycles-pp.__hrtimer_run_queues 0.03 ±100% +0.1 0.11 ± 30% perf-profile.self.cycles-pp.update_blocked_averages 0.00 +0.1 0.08 ± 30% perf-profile.self.cycles-pp.load_balance 0.00 +0.1 0.08 ± 17% perf-profile.self.cycles-pp.__es_remove_extent 0.00 +0.1 0.08 ± 21% perf-profile.self.cycles-pp.rcu_core 0.04 ±101% +0.1 0.12 ± 23% perf-profile.self.cycles-pp.rcu_sched_clock_irq 0.00 +0.1 0.09 ± 30% perf-profile.self.cycles-pp.run_posix_cpu_timers 0.00 +0.1 0.09 ± 24% perf-profile.self.cycles-pp.ksoftirqd_running 0.00 +0.1 0.09 ± 21% perf-profile.self.cycles-pp.smp_apic_timer_interrupt 0.01 ±173% +0.1 0.10 ± 24% perf-profile.self.cycles-pp.interrupt_entry 0.01 ±173% +0.1 0.10 ± 58% perf-profile.self.cycles-pp.tick_check_oneshot_broadcast_this_cpu 0.22 ± 24% +0.1 0.32 ± 21% perf-profile.self.cycles-pp.release_pages 0.00 +0.1 0.10 ± 11% perf-profile.self.cycles-pp.rwsem_spin_on_owner 0.07 ± 28% +0.1 0.17 ± 11% perf-profile.self.cycles-pp.note_gp_changes 0.05 ± 9% +0.1 0.15 ± 18% perf-profile.self.cycles-pp.native_sched_clock 0.07 ± 10% +0.1 0.17 ± 18% perf-profile.self.cycles-pp.io_serial_in 0.03 ±100% +0.1 0.14 ± 30% perf-profile.self.cycles-pp.__softirqentry_text_start 0.00 +0.1 0.10 ± 17% perf-profile.self.cycles-pp.jbd2_write_access_granted 0.00 +0.1 0.10 ± 22% perf-profile.self.cycles-pp.nr_iowait_cpu 0.00 +0.1 0.10 ± 35% perf-profile.self.cycles-pp.pm_qos_read_value 0.22 ± 11% +0.1 0.33 ± 3% perf-profile.self.cycles-pp.read_tsc 0.00 +0.1 0.11 ± 11% perf-profile.self.cycles-pp.crc_1 0.00 +0.1 0.12 ± 40% perf-profile.self.cycles-pp.osq_lock 0.05 ± 58% +0.1 0.17 ± 13% perf-profile.self.cycles-pp.tick_nohz_next_event 0.04 ± 57% +0.1 0.16 ± 13% perf-profile.self.cycles-pp.__next_timer_interrupt 0.08 ± 6% +0.1 0.19 ± 10% perf-profile.self.cycles-pp.find_next_bit 0.23 ± 20% +0.1 0.36 ± 11% perf-profile.self.cycles-pp.xas_find_marked 0.03 ±105% +0.1 0.16 ± 33% perf-profile.self.cycles-pp.update_sd_lb_stats 0.01 ±173% +0.1 0.15 ± 19% perf-profile.self.cycles-pp.shrink_page_list 0.14 ± 6% +0.1 0.27 ± 10% perf-profile.self.cycles-pp.native_irq_return_iret 0.00 +0.1 0.14 ± 3% perf-profile.self.cycles-pp.ext4_do_update_inode 0.07 +0.1 0.21 ± 14% perf-profile.self.cycles-pp.do_idle 0.00 +0.2 0.15 ± 11% perf-profile.self.cycles-pp.jbd2_journal_dirty_metadata 0.18 ± 4% +0.2 0.36 ± 8% perf-profile.self.cycles-pp.native_write_msr 0.20 ± 14% +0.2 0.39 ± 17% perf-profile.self.cycles-pp.rb_next 0.14 ± 34% +0.2 0.34 ± 16% perf-profile.self.cycles-pp.rb_erase 0.00 +0.2 0.21 ± 8% perf-profile.self.cycles-pp.memmove 0.49 ± 23% +0.2 0.73 ± 13% perf-profile.self.cycles-pp.find_get_pages_range_tag 0.79 ± 12% +0.3 1.04 ± 10% perf-profile.self.cycles-pp.__es_tree_search 0.00 +0.3 0.27 ± 12% perf-profile.self.cycles-pp.crc32c_pcl_intel_update 0.06 ± 20% +0.4 0.48 ± 17% perf-profile.self.cycles-pp.ext4_end_bio 0.25 ± 9% +0.4 0.69 ± 14% perf-profile.self.cycles-pp.cpuidle_enter_state 0.35 ± 15% +0.5 0.84 ± 19% perf-profile.self.cycles-pp.menu_select 0.23 ± 66% +1.1 1.36 ± 28% perf-profile.self.cycles-pp.xas_create 1.32 ± 5% +1.4 2.68 ± 15% perf-profile.self.cycles-pp.ext4_find_extent 34.76 ± 5% +9.7 44.41 ± 12% perf-profile.self.cycles-pp.intel_idle Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

4 months, 2 weeks

1
0
0 / 0

[printk] 9126610153: BUG:scheduling_while_atomic

by kernel test robot

Greeting, FYI, we noticed the following commit (built with gcc-9): commit: 912661015315cefbd60e7a0f7fdada88835cf04d ("printk: add pr_flush()") https://git.kernel.org/cgit/linux/kernel/git/rt/linux-rt-devel.git linux-5.9.y-rt-rebase in testcase: boot on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 8G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): +-----------------------------+------------+------------+ | | d932e1b1aa | 9126610153 | +-----------------------------+------------+------------+ | boot_successes | 4 | 0 | | boot_failures | 0 | 6 | | BUG:scheduling_while_atomic | 0 | 5 | | Kernel_panic-not_syncing | 0 | 1 | +-----------------------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> [ 18.469688] BUG: scheduling while atomic: lkp-setup-rootf/360/0x00000002 [ 18.470528] Modules linked in: sr_mod cdrom ata_generic intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul bochs_drm crc32c_intel ghash_clmulni_intel drm_vram_helper ppdev ata_piix drm_ttm_helper ttm joydev parport_pc drm_kms_helper libata serio_raw syscopyarea sysfillrect parport sysimgblt fb_sys_fops drm i2c_piix4 [ 18.473964] CPU: 0 PID: 360 Comm: lkp-setup-rootf Not tainted 5.9.1-00064-g912661015315ce #1 [ 18.475002] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [ 18.476019] Call Trace: [ 18.477021] dump_stack+0x57/0x80 [ 18.478096] __schedule_bug.cold+0x47/0x54 [ 18.478609] __schedule+0x6e5/0x8a0 [ 18.479035] ? vprintk_store+0x1f6/0x540 [ 18.479510] schedule+0x6e/0x100 [ 18.479903] schedule_timeout+0x1a0/0x300 [ 18.480392] ? __next_timer_interrupt+0x100/0x100 [ 18.480965] ? msleep+0x25/0x60 [ 18.481350] msleep+0x39/0x60 [ 18.481717] kmsg_dump+0xa5/0x180 [ 18.482143] emergency_restart+0xf/0x20 [ 18.482609] __handle_sysrq.cold+0x43/0x108 [ 18.483119] write_sysrq_trigger+0x3f/0x4a [ 18.483603] proc_reg_write+0x53/0x80 [ 18.484069] vfs_write+0xe9/0x240 [ 18.484490] ksys_write+0x5f/0xe0 [ 18.484900] do_syscall_64+0x33/0x40 [ 18.485350] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 18.485976] RIP: 0033:0x7f51bf57eab4 [ 18.486461] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 89 07 0d 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53 [ 18.488716] RSP: 002b:00007fff36856fd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 18.489634] RAX: ffffffffffffffda RBX: 00007f51bf4936c0 RCX: 00007f51bf57eab4 [ 18.490475] RDX: 0000000000000002 RSI: 000055a7f6c6ae40 RDI: 0000000000000001 [ 18.491341] RBP: 0000000000000002 R08: 000055a7f6467d20 R09: 000055a7f6c7c040 [ 18.492188] R10: 0000000000000000 R11: 0000000000000246 R12: 000055a7f6c6ae40 [ 18.493039] R13: 0000000000000001 R14: 0000000000000001 R15: 00007fff36857040 [ 18.500239] printk: console [ttyS0]: printing thread stopped To reproduce: # build kernel cd linux cp config-5.9.1-00064-g912661015315ce .config make HOSTCC=gcc-9 CC=gcc-9 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email Thanks, Rong Chen

4 months, 2 weeks

1
0
0 / 0

[mm, shmem] dcdf11ee14: will-it-scale.per_process_ops -17.9% regression

by kernel test robot

Greeting, FYI, we noticed a -17.9% regression of will-it-scale.per_process_ops due to commit: commit: dcdf11ee144133328664d90836e712d840d047d9 ("mm, shmem: add vmstat for hugepage fallback") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: will-it-scale on test machine: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory with following parameters: nr_task: 100% mode: process test: page_fault3 cpufreq_governor: performance ucode: 0x5002f01 test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two. test-url: https://github.com/antonblanchard/will-it-scale In addition to that, the commit also has significant impact on the following tests: +------------------+--------------------------------------------------+ | testcase: change | vm-scalability: boot-time.dhcp -1.0% improvement | | test machine | 104 threads Skylake with 192G memory | | test parameters | cpufreq_governor=performance | | | runtime=300s | | | size=1T | | | test=lru-shm | | | ucode=0x2006906 | +------------------+--------------------------------------------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-8.3/process/100%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2ap4/page_fault3/will-it-scale/0x5002f01 commit: 6aeff241fe ("mm/migrate.c: migrate PG_readahead flag") dcdf11ee14 ("mm, shmem: add vmstat for hugepage fallback") 6aeff241fe6c4561 dcdf11ee144133328664d90836e ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 74:16 -79% 61:16 perf-profile.calltrace.cycles-pp.error_entry.testcase 65:16 -71% 54:16 perf-profile.calltrace.cycles-pp.sync_regs.error_entry.testcase 78:16 -84% 64:16 perf-profile.children.cycles-pp.error_entry 1:16 -2% 0:16 perf-profile.children.cycles-pp.error_exit 8:16 -10% 7:16 perf-profile.self.cycles-pp.error_entry 0:16 -2% 0:16 perf-profile.self.cycles-pp.error_exit %stddev %change %stddev \ | \ 491236 -17.9% 403513 ± 2% will-it-scale.per_process_ops 94317579 -17.9% 77474716 ± 2% will-it-scale.workload 1.22 -0.1 1.08 ± 2% mpstat.cpu.all.irq% 11.72 +4.4 16.12 ± 3% mpstat.cpu.all.usr% 86.94 -5.2% 82.38 vmstat.cpu.sy 11.00 +40.9% 15.50 ± 4% vmstat.cpu.us 941.60 -10.4% 843.27 sched_debug.cpu.sched_count.min 394.51 -12.6% 344.65 sched_debug.cpu.ttwu_count.min 388.73 -12.6% 339.66 sched_debug.cpu.ttwu_local.min 63421776 -15.7% 53463068 proc-vmstat.numa_hit 63328346 -15.7% 53369647 proc-vmstat.numa_local 63583145 -15.7% 53592409 proc-vmstat.pgalloc_normal 2.833e+10 -17.8% 2.329e+10 ± 2% proc-vmstat.pgfault 60231689 ± 5% -14.1% 51752243 ± 5% proc-vmstat.pgfree 8881812 -10.2% 7972540 ± 3% numa-vmstat.node0.numa_hit 8813894 -10.2% 7915641 ± 3% numa-vmstat.node0.numa_local 9101400 -9.7% 8219165 numa-vmstat.node1.numa_local 9135335 ± 2% -8.2% 8388489 ± 2% numa-vmstat.node2.numa_hit 9027011 ± 2% -8.3% 8281210 ± 2% numa-vmstat.node2.numa_local 9130741 -26.6% 6705906 ± 2% numa-vmstat.node3.numa_hit 9014304 -26.9% 6588488 ± 2% numa-vmstat.node3.numa_local 15463298 -12.0% 13607940 ± 4% numa-numastat.node0.local_node 15487319 -12.0% 13634289 ± 4% numa-numastat.node0.numa_hit 15940614 -10.8% 14211661 ± 2% numa-numastat.node1.local_node 15961220 -10.8% 14230733 ± 2% numa-numastat.node1.numa_hit 15994648 -10.0% 14397886 numa-numastat.node2.local_node 16017192 -10.0% 14418935 numa-numastat.node2.numa_hit 15995400 -30.2% 11161473 ± 2% numa-numastat.node3.local_node 16021436 -30.2% 11188323 ± 2% numa-numastat.node3.numa_hit 13258 ± 2% +9.2% 14485 ± 3% softirqs.CPU108.RCU 12789 ± 3% +10.0% 14069 ± 2% softirqs.CPU112.RCU 12501 ± 7% +12.5% 14057 softirqs.CPU114.RCU 12783 ± 3% +9.3% 13968 ± 4% softirqs.CPU125.RCU 13816 ± 3% +9.3% 15102 ± 3% softirqs.CPU14.RCU 12998 ± 3% +11.1% 14440 ± 2% softirqs.CPU16.RCU 13004 ± 7% +12.6% 14637 ± 2% softirqs.CPU17.RCU 13081 ± 3% +10.1% 14396 ± 2% softirqs.CPU19.RCU 12848 ± 3% +12.3% 14425 ± 4% softirqs.CPU21.RCU 12853 ± 3% +11.6% 14347 ± 3% softirqs.CPU22.RCU 12908 ± 3% +10.9% 14321 ± 2% softirqs.CPU23.RCU 13142 ± 3% +15.8% 15223 ± 9% softirqs.CPU27.RCU 2.98 +12.2% 3.35 perf-stat.i.MPKI 4.712e+10 -17.7% 3.876e+10 ± 2% perf-stat.i.branch-instructions 0.29 +0.0 0.30 perf-stat.i.branch-miss-rate% 1.311e+08 -15.1% 1.113e+08 perf-stat.i.branch-misses 57.06 -2.2 54.86 perf-stat.i.cache-miss-rate% 3.9e+08 -10.9% 3.474e+08 perf-stat.i.cache-misses 6.817e+08 -7.3% 6.317e+08 perf-stat.i.cache-references 2.56 +21.4% 3.10 ± 2% perf-stat.i.cpi 1511 +12.3% 1697 perf-stat.i.cycles-between-cache-misses 6.549e+10 -17.8% 5.385e+10 ± 2% perf-stat.i.dTLB-loads 1.571e+09 -17.9% 1.29e+09 ± 2% perf-stat.i.dTLB-store-misses 3.375e+10 -17.8% 2.775e+10 ± 2% perf-stat.i.dTLB-stores 83231523 -12.9% 72514211 perf-stat.i.iTLB-load-misses 244449 ± 4% -42.7% 140075 ± 4% perf-stat.i.iTLB-loads 2.309e+11 -17.7% 1.899e+11 ± 2% perf-stat.i.instructions 2776 -5.6% 2622 perf-stat.i.instructions-per-iTLB-miss 0.39 -17.7% 0.32 ± 2% perf-stat.i.ipc 0.02 ±118% +2130.1% 0.54 ± 2% perf-stat.i.metric.K/sec 775.66 -17.7% 638.38 ± 2% perf-stat.i.metric.M/sec 93608966 -17.9% 76881713 ± 2% perf-stat.i.minor-faults 4012755 +41.4% 5674418 ± 2% perf-stat.i.node-load-misses 22945074 ± 17% +42.0% 32578532 ± 2% perf-stat.i.node-loads 22.22 +0.5 22.70 perf-stat.i.node-store-miss-rate% 26917782 -15.5% 22742426 perf-stat.i.node-store-misses 95169694 -17.8% 78245919 ± 2% perf-stat.i.node-stores 93608966 -17.9% 76881713 ± 2% perf-stat.i.page-faults 2.95 +12.7% 3.33 perf-stat.overall.MPKI 0.28 +0.0 0.29 perf-stat.overall.branch-miss-rate% 57.23 -2.2 55.01 perf-stat.overall.cache-miss-rate% 2.55 +21.7% 3.10 ± 2% perf-stat.overall.cpi 1506 +12.4% 1693 perf-stat.overall.cycles-between-cache-misses 2773 -5.6% 2618 perf-stat.overall.instructions-per-iTLB-miss 0.39 -17.8% 0.32 ± 2% perf-stat.overall.ipc 22.05 +0.5 22.52 perf-stat.overall.node-store-miss-rate% 4.697e+10 -17.7% 3.865e+10 ± 2% perf-stat.ps.branch-instructions 1.307e+08 -15.1% 1.109e+08 perf-stat.ps.branch-misses 3.889e+08 -10.9% 3.464e+08 perf-stat.ps.cache-misses 6.795e+08 -7.3% 6.298e+08 perf-stat.ps.cache-references 6.528e+10 -17.7% 5.37e+10 ± 2% perf-stat.ps.dTLB-loads 1.566e+09 -17.9% 1.287e+09 ± 2% perf-stat.ps.dTLB-store-misses 3.364e+10 -17.7% 2.767e+10 ± 2% perf-stat.ps.dTLB-stores 82972636 -12.9% 72308513 perf-stat.ps.iTLB-load-misses 242363 ± 4% -42.8% 138589 ± 4% perf-stat.ps.iTLB-loads 2.301e+11 -17.7% 1.894e+11 ± 2% perf-stat.ps.instructions 93314496 -17.8% 76668000 ± 2% perf-stat.ps.minor-faults 3997637 +41.4% 5652936 ± 2% perf-stat.ps.node-load-misses 22900278 ± 17% +42.0% 32514566 ± 2% perf-stat.ps.node-loads 26832423 -15.5% 22676967 perf-stat.ps.node-store-misses 94867767 -17.8% 78025979 ± 2% perf-stat.ps.node-stores 93314496 -17.8% 76668001 ± 2% perf-stat.ps.page-faults 6.967e+13 -17.6% 5.738e+13 ± 2% perf-stat.total.instructions 87.22 -6.8 80.39 perf-profile.calltrace.cycles-pp.page_fault.testcase 10.71 ± 2% -2.3 8.37 ± 2% perf-profile.calltrace.cycles-pp.__do_fault.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault 10.27 ± 2% -2.2 8.02 ± 2% perf-profile.calltrace.cycles-pp.shmem_fault.__do_fault.do_fault.__handle_mm_fault.handle_mm_fault 7.33 ± 2% -2.2 5.09 ± 3% perf-profile.calltrace.cycles-pp.__mod_lruvec_state.page_remove_rmap.zap_pte_range.unmap_page_range.unmap_vmas 9.52 ± 2% -2.1 7.43 ± 3% perf-profile.calltrace.cycles-pp.shmem_getpage_gfp.shmem_fault.__do_fault.do_fault.__handle_mm_fault 6.29 ± 3% -2.0 4.25 ± 4% perf-profile.calltrace.cycles-pp.__mod_memcg_state.__mod_lruvec_state.page_remove_rmap.zap_pte_range.unmap_page_range 8.77 ± 3% -1.9 6.85 ± 2% perf-profile.calltrace.cycles-pp.find_lock_entry.shmem_getpage_gfp.shmem_fault.__do_fault.do_fault 7.51 ± 2% -1.7 5.83 ± 2% perf-profile.calltrace.cycles-pp.__mod_lruvec_state.page_add_file_rmap.alloc_set_pte.finish_fault.do_fault 6.43 ± 3% -1.4 5.00 ± 2% perf-profile.calltrace.cycles-pp.__mod_memcg_state.__mod_lruvec_state.page_add_file_rmap.alloc_set_pte.finish_fault 7.33 ± 2% -1.1 6.23 ± 3% perf-profile.calltrace.cycles-pp.__count_memcg_events.handle_mm_fault.do_user_addr_fault.page_fault.testcase 3.22 -0.8 2.39 ± 2% perf-profile.calltrace.cycles-pp.fault_dirty_shared_page.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault 1.89 ± 3% -0.8 1.09 ± 5% perf-profile.calltrace.cycles-pp._raw_spin_lock.alloc_set_pte.finish_fault.do_fault.__handle_mm_fault 0.98 ± 2% -0.4 0.61 ± 4% perf-profile.calltrace.cycles-pp.unlock_page.fault_dirty_shared_page.do_fault.__handle_mm_fault.handle_mm_fault 1.32 ± 2% -0.4 0.96 ± 3% perf-profile.calltrace.cycles-pp.up_read.do_user_addr_fault.page_fault.testcase 3.30 ± 2% -0.3 2.99 ± 2% perf-profile.calltrace.cycles-pp.lock_page_memcg.page_add_file_rmap.alloc_set_pte.finish_fault.do_fault 0.96 ± 2% -0.3 0.67 ± 3% perf-profile.calltrace.cycles-pp.down_read_trylock.do_user_addr_fault.page_fault.testcase 1.20 -0.3 0.94 ± 3% perf-profile.calltrace.cycles-pp.__perf_sw_event.do_user_addr_fault.page_fault.testcase 1.22 -0.2 1.01 ± 2% perf-profile.calltrace.cycles-pp.__perf_sw_event.page_fault.testcase 1.03 -0.2 0.83 ± 2% perf-profile.calltrace.cycles-pp.file_update_time.fault_dirty_shared_page.do_fault.__handle_mm_fault.handle_mm_fault 0.69 -0.2 0.53 ± 2% perf-profile.calltrace.cycles-pp.tlb_flush_mmu.zap_pte_range.unmap_page_range.unmap_vmas.unmap_region 0.91 -0.2 0.76 ± 2% perf-profile.calltrace.cycles-pp.swapgs_restore_regs_and_return_to_usermode.testcase 0.72 ± 3% -0.1 0.58 ± 3% perf-profile.calltrace.cycles-pp.___perf_sw_event.__perf_sw_event.do_user_addr_fault.page_fault.testcase 0.75 ± 2% -0.1 0.61 ± 2% perf-profile.calltrace.cycles-pp.___perf_sw_event.__perf_sw_event.page_fault.testcase 0.99 ± 2% -0.0 0.94 ± 2% perf-profile.calltrace.cycles-pp.lock_page_memcg.page_remove_rmap.zap_pte_range.unmap_page_range.unmap_vmas 1.32 ± 3% +0.3 1.62 ± 2% perf-profile.calltrace.cycles-pp.xas_load.find_get_entry.find_lock_entry.shmem_getpage_gfp.shmem_fault 9.98 ± 2% +0.9 10.88 ± 2% perf-profile.calltrace.cycles-pp.page_remove_rmap.zap_pte_range.unmap_page_range.unmap_vmas.unmap_region 42.89 +1.6 44.49 perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.page_fault.testcase 33.71 +3.0 36.76 perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.page_fault.testcase 31.29 +3.5 34.83 perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.page_fault 16.85 +6.8 23.66 ± 2% perf-profile.calltrace.cycles-pp.finish_fault.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault 16.62 +6.9 23.49 ± 2% perf-profile.calltrace.cycles-pp.alloc_set_pte.finish_fault.do_fault.__handle_mm_fault.handle_mm_fault 13.38 +8.0 21.37 ± 2% perf-profile.calltrace.cycles-pp.page_add_file_rmap.alloc_set_pte.finish_fault.do_fault.__handle_mm_fault 76.90 +8.0 84.93 perf-profile.calltrace.cycles-pp.testcase 14.88 ± 2% -3.9 10.95 ± 3% perf-profile.children.cycles-pp.__mod_lruvec_state 69.38 -3.6 65.83 perf-profile.children.cycles-pp.page_fault 12.73 ± 3% -3.5 9.26 ± 3% perf-profile.children.cycles-pp.__mod_memcg_state 10.72 ± 2% -2.3 8.38 ± 2% perf-profile.children.cycles-pp.__do_fault 10.29 ± 2% -2.3 8.04 ± 2% perf-profile.children.cycles-pp.shmem_fault 9.55 ± 2% -2.1 7.45 ± 3% perf-profile.children.cycles-pp.shmem_getpage_gfp 8.83 ± 3% -1.9 6.90 ± 2% perf-profile.children.cycles-pp.find_lock_entry 7.33 ± 2% -1.1 6.24 ± 3% perf-profile.children.cycles-pp.__count_memcg_events 3.29 -0.8 2.44 ± 2% perf-profile.children.cycles-pp.fault_dirty_shared_page 1.93 ± 3% -0.8 1.12 ± 5% perf-profile.children.cycles-pp._raw_spin_lock 4.25 -0.7 3.52 perf-profile.children.cycles-pp.sync_regs 2.42 -0.5 1.95 ± 2% perf-profile.children.cycles-pp.__perf_sw_event 4.30 ± 2% -0.4 3.93 ± 2% perf-profile.children.cycles-pp.lock_page_memcg 0.98 ± 2% -0.4 0.61 ± 4% perf-profile.children.cycles-pp.unlock_page 1.32 ± 2% -0.4 0.96 ± 3% perf-profile.children.cycles-pp.up_read 1.55 ± 2% -0.3 1.25 ± 2% perf-profile.children.cycles-pp.___perf_sw_event 0.96 ± 2% -0.3 0.67 ± 3% perf-profile.children.cycles-pp.down_read_trylock 1.04 -0.3 0.79 ± 2% perf-profile.children.cycles-pp.set_page_dirty 1.09 -0.2 0.87 ± 2% perf-profile.children.cycles-pp.page_mapping 1.05 -0.2 0.86 ± 2% perf-profile.children.cycles-pp.file_update_time 0.71 -0.2 0.54 ± 2% perf-profile.children.cycles-pp.tlb_flush_mmu 0.54 ± 2% -0.2 0.39 ± 2% perf-profile.children.cycles-pp.release_pages 0.91 -0.2 0.76 ± 2% perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode 0.59 -0.1 0.45 ± 3% perf-profile.children.cycles-pp.find_vma 0.75 -0.1 0.62 ± 2% perf-profile.children.cycles-pp.__mod_node_page_state 0.66 ± 2% -0.1 0.53 ± 4% perf-profile.children.cycles-pp.current_time 0.50 ± 2% -0.1 0.38 ± 4% perf-profile.children.cycles-pp.vmacache_find 0.70 ± 6% -0.1 0.60 ± 6% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 0.65 ± 6% -0.1 0.55 ± 6% perf-profile.children.cycles-pp.hrtimer_interrupt 0.75 ± 5% -0.1 0.66 ± 5% perf-profile.children.cycles-pp.apic_timer_interrupt 0.50 ± 2% -0.1 0.41 ± 2% perf-profile.children.cycles-pp.do_page_fault 0.52 -0.1 0.43 ± 2% perf-profile.children.cycles-pp.___might_sleep 0.35 -0.1 0.27 ± 3% perf-profile.children.cycles-pp.__set_page_dirty_no_writeback 0.37 ± 5% -0.1 0.30 ± 8% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.32 ± 6% -0.1 0.25 ± 9% perf-profile.children.cycles-pp.tick_sched_timer 0.41 ± 2% -0.1 0.35 ± 2% perf-profile.children.cycles-pp.prepare_exit_to_usermode 0.32 -0.1 0.26 ± 2% perf-profile.children.cycles-pp.mark_page_accessed 0.29 ± 3% -0.1 0.23 ± 4% perf-profile.children.cycles-pp.__might_sleep 0.49 ± 3% -0.0 0.45 ± 3% perf-profile.children.cycles-pp.__unlock_page_memcg 0.16 ± 2% -0.0 0.12 ± 4% perf-profile.children.cycles-pp.page_rmapping 0.20 ± 3% -0.0 0.16 ± 5% perf-profile.children.cycles-pp.perf_exclude_event 0.20 ± 3% -0.0 0.17 ± 3% perf-profile.children.cycles-pp.__alloc_pages_nodemask 0.20 ± 2% -0.0 0.17 ± 3% perf-profile.children.cycles-pp.pte_alloc_one 0.16 ± 2% -0.0 0.12 ± 3% perf-profile.children.cycles-pp.PageHuge 0.23 ± 2% -0.0 0.19 ± 2% perf-profile.children.cycles-pp._cond_resched 0.13 ± 3% -0.0 0.10 ± 4% perf-profile.children.cycles-pp.__memcg_kmem_charge_page 0.15 ± 4% -0.0 0.12 ± 5% perf-profile.children.cycles-pp.perf_swevent_event 0.14 ± 3% -0.0 0.12 ± 4% perf-profile.children.cycles-pp.vm_normal_page 0.08 ± 4% -0.0 0.06 ± 6% perf-profile.children.cycles-pp.page_counter_try_charge 0.12 ± 3% -0.0 0.10 ± 4% perf-profile.children.cycles-pp.rcu_all_qs 0.09 ± 2% -0.0 0.07 ± 5% perf-profile.children.cycles-pp.__memcg_kmem_charge 1.34 ± 3% +0.3 1.64 ± 2% perf-profile.children.cycles-pp.xas_load 10.05 ± 2% +0.9 10.94 ± 2% perf-profile.children.cycles-pp.page_remove_rmap 42.95 +1.6 44.54 perf-profile.children.cycles-pp.handle_mm_fault 33.75 +3.0 36.79 perf-profile.children.cycles-pp.__handle_mm_fault 31.37 +3.5 34.89 perf-profile.children.cycles-pp.do_fault 9.24 ± 2% +4.7 13.94 ± 3% perf-profile.children.cycles-pp.native_irq_return_iret 16.89 +6.8 23.68 ± 2% perf-profile.children.cycles-pp.finish_fault 16.67 +6.9 23.52 ± 2% perf-profile.children.cycles-pp.alloc_set_pte 13.42 +8.0 21.41 ± 2% perf-profile.children.cycles-pp.page_add_file_rmap 19.39 -3.5 15.91 ± 2% perf-profile.self.cycles-pp.testcase 12.63 ± 3% -3.4 9.19 ± 3% perf-profile.self.cycles-pp.__mod_memcg_state 7.32 ± 2% -1.1 6.22 ± 3% perf-profile.self.cycles-pp.__count_memcg_events 2.50 ± 2% -1.1 1.40 ± 6% perf-profile.self.cycles-pp.find_lock_entry 1.90 ± 3% -0.8 1.10 ± 5% perf-profile.self.cycles-pp._raw_spin_lock 4.24 -0.7 3.51 perf-profile.self.cycles-pp.sync_regs 2.00 ± 2% -0.5 1.51 ± 2% perf-profile.self.cycles-pp.zap_pte_range 2.34 -0.5 1.87 ± 2% perf-profile.self.cycles-pp.__handle_mm_fault 0.95 ± 2% -0.4 0.59 ± 4% perf-profile.self.cycles-pp.unlock_page 4.24 ± 2% -0.4 3.88 ± 2% perf-profile.self.cycles-pp.lock_page_memcg 1.30 ± 2% -0.3 0.95 ± 3% perf-profile.self.cycles-pp.up_read 1.37 ± 3% -0.3 1.06 ± 5% perf-profile.self.cycles-pp.__mod_lruvec_state 1.06 ± 2% -0.3 0.77 ± 2% perf-profile.self.cycles-pp.alloc_set_pte 0.95 ± 2% -0.3 0.66 ± 3% perf-profile.self.cycles-pp.down_read_trylock 1.60 -0.3 1.33 ± 2% perf-profile.self.cycles-pp.handle_mm_fault 1.15 -0.3 0.90 ± 2% perf-profile.self.cycles-pp.do_user_addr_fault 1.26 ± 2% -0.2 1.01 ± 3% perf-profile.self.cycles-pp.___perf_sw_event 1.05 -0.2 0.84 ± 2% perf-profile.self.cycles-pp.page_mapping 0.86 -0.2 0.70 ± 2% perf-profile.self.cycles-pp.__perf_sw_event 0.70 ± 5% -0.2 0.54 ± 5% perf-profile.self.cycles-pp.shmem_getpage_gfp 0.74 ± 2% -0.2 0.58 perf-profile.self.cycles-pp.shmem_fault 0.52 ± 2% -0.1 0.38 ± 2% perf-profile.self.cycles-pp.release_pages 0.73 -0.1 0.60 ± 2% perf-profile.self.cycles-pp.__mod_node_page_state 0.75 -0.1 0.63 ± 2% perf-profile.self.cycles-pp.page_fault 0.47 ± 2% -0.1 0.36 ± 3% perf-profile.self.cycles-pp.vmacache_find 0.33 -0.1 0.23 ± 2% perf-profile.self.cycles-pp.set_page_dirty 0.49 ± 2% -0.1 0.41 ± 2% perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode 0.49 -0.1 0.41 ± 2% perf-profile.self.cycles-pp.___might_sleep 0.48 -0.1 0.40 ± 2% perf-profile.self.cycles-pp.do_page_fault 0.40 ± 3% -0.1 0.32 ± 3% perf-profile.self.cycles-pp.file_update_time 0.32 -0.1 0.25 ± 4% perf-profile.self.cycles-pp.fault_dirty_shared_page 0.31 -0.1 0.25 ± 3% perf-profile.self.cycles-pp.__set_page_dirty_no_writeback 0.23 ± 4% -0.1 0.16 ± 3% perf-profile.self.cycles-pp.finish_fault 0.37 ± 3% -0.1 0.30 ± 2% perf-profile.self.cycles-pp.prepare_exit_to_usermode 0.31 -0.1 0.25 ± 2% perf-profile.self.cycles-pp.mark_page_accessed 0.26 ± 2% -0.1 0.21 ± 3% perf-profile.self.cycles-pp.__might_sleep 0.21 ± 3% -0.1 0.16 ± 2% perf-profile.self.cycles-pp.__do_fault 0.23 ± 4% -0.0 0.19 ± 5% perf-profile.self.cycles-pp.current_time 0.13 ± 3% -0.0 0.10 ± 4% perf-profile.self.cycles-pp.page_rmapping 0.17 ± 4% -0.0 0.13 ± 5% perf-profile.self.cycles-pp.perf_exclude_event 0.10 ± 5% -0.0 0.08 ± 4% perf-profile.self.cycles-pp.find_vma 0.12 ± 5% -0.0 0.09 ± 4% perf-profile.self.cycles-pp.perf_swevent_event 0.11 ± 2% -0.0 0.09 ± 4% perf-profile.self.cycles-pp.PageHuge 0.12 ± 3% -0.0 0.10 ± 4% perf-profile.self.cycles-pp.vm_normal_page 0.11 ± 3% -0.0 0.09 ± 4% perf-profile.self.cycles-pp._cond_resched 0.07 ± 5% -0.0 0.06 ± 8% perf-profile.self.cycles-pp.page_counter_try_charge 0.08 ± 4% -0.0 0.07 perf-profile.self.cycles-pp.rcu_all_qs 1.04 ± 2% +0.4 1.40 ± 2% perf-profile.self.cycles-pp.xas_load 1.66 ± 4% +3.2 4.84 ± 5% perf-profile.self.cycles-pp.page_remove_rmap 9.22 ± 2% +4.7 13.93 ± 3% perf-profile.self.cycles-pp.native_irq_return_iret 2.58 ± 2% +9.9 12.51 ± 4% perf-profile.self.cycles-pp.page_add_file_rmap will-it-scale.per_process_ops 520000 +------------------------------------------------------------------+ 500000 |-+ + + + + | |+++ ++ +++++++++++++++++++ +++++ ++++++ ++ +++++++++++ | 480000 |++ +++: ++ + ++ + ++ + ++ + + | 460000 |-+ + :: + + | | + | 440000 |-+ | 420000 |-+ O O | 400000 |-+ O O OOOO OOOOOOOOOO OOOOO OOO O OO OOOO| | O OOOO OOOO O OO OOO O OOOOOOOOOOOO O OOO OOO O O| 380000 |OOOO O O O OO O O | 360000 |OO O O OO | | O | 340000 |-+ O | 320000 +------------------------------------------------------------------+ will-it-scale.workload 1e+08 +-----------------------------------------------------------------+ | + + + + | 9.5e+07 |+++ + + +++++++++++++++ ++ + +++ ++++++++ +++ +++++++ | 9e+07 |++ +++:+++ + ++ + ++ + + + + | | : | 8.5e+07 |-+ + | | | 8e+07 |-+ OO O O O | | O O O OOOOOOOOOOOOOOOOO O OOOOOOOOO O OOOOOOOOO| 7.5e+07 |-+ O OOOO OOOO O OOOOO O O OOOOOO OO| 7e+07 |OOO OO OO O | |OO O O OO | 6.5e+07 |-+ O | | | 6e+07 +-----------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-skl-fpga01: 104 threads Skylake with 192G memory ========================================================================================= compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-8.3/debian-10.4-x86_64-20200603.cgz/300s/1T/lkp-skl-fpga01/lru-shm/vm-scalability/0x2006906 commit: 6aeff241fe ("mm/migrate.c: migrate PG_readahead flag") dcdf11ee14 ("mm, shmem: add vmstat for hugepage fallback") 6aeff241fe6c4561 dcdf11ee144133328664d90836e ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :2 50% 1:20 dmesg.WARNING:at#for_ip_interrupt_entry/0x :2 50% 1:20 last_state.booting :2 50% 1:20 last_state.is_incomplete_run 1:2 408% 10:20 perf-profile.calltrace.cycles-pp.sync_regs.error_entry :2 420% 8:20 perf-profile.calltrace.cycles-pp.sync_regs.error_entry.do_access 3:2 875% 21:20 perf-profile.calltrace.cycles-pp.error_entry 0:2 914% 19:20 perf-profile.calltrace.cycles-pp.error_entry.do_access 5:2 1972% 44:20 perf-profile.children.cycles-pp.error_entry 2:2 956% 21:20 perf-profile.self.cycles-pp.error_entry %stddev %change %stddev \ | \ 0.54 ± 13% -0.1 0.45 ± 20% perf-profile.children.cycles-pp.find_get_entries 0.60 ± 17% -0.1 0.51 ± 20% perf-profile.children.cycles-pp.clockevents_program_event 0.29 -0.1 0.21 ± 17% perf-profile.children.cycles-pp.find_vma 0.28 ± 5% -0.1 0.19 ± 17% perf-profile.children.cycles-pp.vmacache_find 0.18 ± 8% -0.1 0.13 ± 24% perf-profile.children.cycles-pp.truncate_cleanup_page 0.12 ± 28% -0.0 0.09 ± 32% perf-profile.children.cycles-pp.tick_irq_enter 0.08 ± 5% -0.0 0.07 ± 12% perf-profile.children.cycles-pp.perf_exclude_event 0.76 ± 7% -0.1 0.64 ± 20% perf-profile.self.cycles-pp.release_pages 0.74 ± 8% -0.1 0.62 ± 13% perf-profile.self.cycles-pp.__handle_mm_fault 0.43 ± 12% -0.1 0.32 ± 41% perf-profile.self.cycles-pp.zap_pte_range 0.70 ± 12% -0.1 0.61 ± 13% perf-profile.self.cycles-pp.__pagevec_lru_add_fn 0.48 ± 14% -0.1 0.39 ± 20% perf-profile.self.cycles-pp.find_get_entries 0.27 ± 3% -0.1 0.19 ± 17% perf-profile.self.cycles-pp.vmacache_find 0.43 ± 8% -0.1 0.37 ± 11% perf-profile.self.cycles-pp.rmqueue 0.17 ± 11% -0.1 0.12 ± 24% perf-profile.self.cycles-pp.truncate_cleanup_page 0.26 ± 5% -0.0 0.22 ± 12% perf-profile.self.cycles-pp.do_user_addr_fault 0.21 ± 11% -0.0 0.18 ± 14% perf-profile.self.cycles-pp.do_fault 0.18 ± 11% -0.0 0.15 ± 15% perf-profile.self.cycles-pp.___might_sleep 0.08 ± 6% -0.0 0.06 ± 12% perf-profile.self.cycles-pp.perf_exclude_event 19.96 -1.0% 19.75 boot-time.dhcp 8812 ± 4% -5.0% 8370 ± 5% numa-meminfo.node1.KernelStack 8800 ± 4% -4.8% 8373 ± 5% numa-vmstat.node1.nr_kernel_stack 53.50 ± 5% +67.3% 89.51 ± 18% sched_debug.cfs_rq:/.nr_spread_over.max 828.33 ± 9% -15.6% 699.50 ± 14% sched_debug.cfs_rq:/.util_est_enqueued.max 1361213 -6.5% 1273045 ± 4% perf-stat.i.node-loads 59.25 +2.9 62.17 ± 3% perf-stat.overall.node-load-miss-rate% 1359101 -6.4% 1271647 ± 3% perf-stat.ps.node-loads 6903 ± 5% -9.2% 6270 ± 11% slabinfo.eventpoll_pwq.active_objs 6903 ± 5% -9.2% 6270 ± 11% slabinfo.eventpoll_pwq.num_objs 1726 ± 3% -7.3% 1600 ± 6% slabinfo.pool_workqueue.num_objs 105593 ± 5% -9.0% 96123 ± 7% softirqs.CPU30.TIMER 27563 ± 5% -10.4% 24703 ± 6% softirqs.CPU35.SCHED 102935 ± 3% -7.5% 95173 ± 5% softirqs.CPU35.TIMER 26724 ± 7% -11.1% 23757 ± 5% softirqs.CPU38.SCHED 101664 ± 3% -6.5% 95094 ± 6% softirqs.CPU48.TIMER 25948 ± 3% -6.0% 24399 ± 5% softirqs.CPU49.SCHED 24817 ± 6% -7.7% 22911 ± 3% softirqs.CPU53.SCHED 106024 ± 13% -20.5% 84307 ± 15% softirqs.CPU62.TIMER 28303 ± 11% -13.7% 24432 ± 6% softirqs.CPU80.SCHED 15005 ± 16% -19.9% 12017 ± 6% softirqs.CPU82.RCU 26967 ± 8% -11.5% 23853 ± 6% softirqs.CPU92.SCHED 25547 ± 4% -6.1% 23979 ± 2% softirqs.CPU97.SCHED 25487 ± 3% -7.1% 23687 ± 4% softirqs.CPU98.SCHED 6613 ± 40% -77.3% 1500 ±144% interrupts.CPU35.RES:Rescheduling_interrupts 3834 ± 91% -92.0% 307.00 ± 83% interrupts.CPU38.RES:Rescheduling_interrupts 2074 ± 28% -23.3% 1590 ± 22% interrupts.CPU4.NMI:Non-maskable_interrupts 2074 ± 28% -23.3% 1590 ± 22% interrupts.CPU4.PMI:Performance_monitoring_interrupts 4380 ± 68% -69.9% 1320 ± 18% interrupts.CPU46.NMI:Non-maskable_interrupts 4380 ± 68% -69.9% 1320 ± 18% interrupts.CPU46.PMI:Performance_monitoring_interrupts 4634 ± 95% -84.8% 704.00 ±147% interrupts.CPU53.RES:Rescheduling_interrupts 2037 ± 26% -32.4% 1376 ± 20% interrupts.CPU64.NMI:Non-maskable_interrupts 2037 ± 26% -32.4% 1376 ± 20% interrupts.CPU64.PMI:Performance_monitoring_interrupts 3381 ± 94% -94.6% 184.05 ± 50% interrupts.CPU72.RES:Rescheduling_interrupts 5218 ± 95% -86.6% 700.84 ±174% interrupts.CPU88.RES:Rescheduling_interrupts 5814 ± 80% -75.4% 1429 ±170% interrupts.CPU92.RES:Rescheduling_interrupts 1957 ± 67% -87.6% 242.00 ± 83% interrupts.CPU97.RES:Rescheduling_interrupts 1496 ± 53% -82.2% 266.58 ± 80% interrupts.CPU98.RES:Rescheduling_interrupts Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

4 months, 2 weeks

1
0
0 / 0

[fsnotify] 9b93f33105: WARNING:missing_R10_value_at__fsnotify_parent/0x

by kernel test robot

Greeting, FYI, we noticed the following commit (built with gcc-9): commit: 9b93f33105f5f9bd3d016ff870eb6000c9d89eff ("fsnotify: send event with parent/name info to sb/mount/non-dir marks") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: will-it-scale version: will-it-scale-x86_64-b695a1b-1_20201007 with following parameters: nr_task: 192 mode: process test: read2 cpufreq_governor: performance ucode: 0x11 test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two. test-url: https://github.com/antonblanchard/will-it-scale on test machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): If you fix the issue, kindly add following tag Reported-by: kernel test robot <lkp(a)intel.com> [ 117.731012] 2020-10-21 19:10:16 /lkp/benchmarks/python3/bin/python3 ./runtest.py read2 295 process 192 [ 117.731023] [ 117.770939] BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled [ 117.770950] [ 117.801209] perf version 5.9.rc8.gc85fb28b6f99 [ 117.801219] [ 182.458747] perf: interrupt took too long (2668 > 2500),perf: interrupt took too long (4516 > 4352), lowering kernel.perf_event_max_sample_rate to 44000 lowering kernel.perf_event_max_sample_rperf: interrupt took too long (5950 > 5645), lowering kernel.perf_event_max_sample_rate to 33000 ate to 74000 perf: interrupt took too long (7658 > 7437), lowering kernel.perf_event_max_sample_rate to 26000 [ 182.502968] perf: interrupt took too long (7658 > 7437), lowering kernel.perf_event_max_sample_rate to 26000 [ 182.525840] perf: interrupt took too long (9889 > 9572), lowering kernel.perf_event_max_sample_rate to 20000 f_event_max_sample_rate to 12000 [ 182.560905] perf: interrupt took too long (21827 > 20585), lowering kernel.perf_event_max_sample_rate to 9000 [ 182.609949] perf: interrupt took too long (30149 > 27283), lowering kernel.perf_event_max_sample_rate to 6000 [ 182.622872] perf: interrupt took too long (38986 > 37686), lowering kernel.perf_event_max_sample_rate to 5000 [ 182.672850] perf: interrupt took too long (50874 > 48732), lowering kernel.perf_event_max_sample_rate to 3000 [ 182.732849] perf: interrupt took too long (63885 > 63592), lowering kernel.perf_event_max_sample_rate to 3000 [ 182.751847] perf: interrupt took too long (80901 > 79856), lowering kernel.perf_event_max_sample_rate to 2000 [ 188.527603] WARNING: missing R10 value at __fsnotify_parent+0x25/0x280 To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml Thanks, lkp

4 months, 2 weeks

2
1
0 / 0

[sched] bdfcae1140: will-it-scale.per_thread_ops -37.0% regression

by kernel test robot

Greeting, FYI, we noticed a -37.0% regression of will-it-scale.per_thread_ops due to commit: commit: bdfcae11403e5099769a7c8dc3262e3c4193edef ("[RFC PATCH 2/3] sched: membarrier: cover kthread_use_mm (v3)") url: https://github.com/0day-ci/linux/commits/Mathieu-Desnoyers/Membarrier-upd... base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git 848785df48835eefebe0c4eb5da7690690b0a8b7 in testcase: will-it-scale on test machine: 104 threads Skylake with 192G memory with following parameters: nr_task: 50% mode: thread test: context_switch1 cpufreq_governor: performance ucode: 0x2006906 test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two. test-url: https://github.com/antonblanchard/will-it-scale If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-8.3/thread/50%/debian-10.4-x86_64-20200603.cgz/lkp-skl-fpga01/context_switch1/will-it-scale/0x2006906 commit: c2daff748f ("sched: fix exit_mm vs membarrier (v3)") bdfcae1140 ("sched: membarrier: cover kthread_use_mm (v3)") c2daff748f0ea954 bdfcae11403e5099769a7c8dc32 ---------------- --------------------------- %stddev %change %stddev \ | \ 161714 ± 2% -37.0% 101819 ± 2% will-it-scale.per_thread_ops 8409170 ± 2% -37.0% 5294646 ± 2% will-it-scale.workload 56783 ± 89% -81.2% 10674 ± 58% numa-vmstat.node0.numa_other 153233 ± 33% +30.3% 199692 ± 3% numa-vmstat.node1.numa_other 1099 ± 6% -5.4% 1039 ± 4% slabinfo.file_lock_cache.active_objs 1099 ± 6% -5.4% 1039 ± 4% slabinfo.file_lock_cache.num_objs 7082 -8.4% 6490 ± 4% slabinfo.kmalloc-rcl-64.active_objs 7082 -8.4% 6490 ± 4% slabinfo.kmalloc-rcl-64.num_objs 20.00 +55.0% 31.00 ± 5% vmstat.cpu.id 69.75 -12.2% 61.25 ± 2% vmstat.cpu.sy 16629437 ± 2% -37.1% 10466452 ± 2% vmstat.system.cs 335444 -39.7% 202300 ± 20% vmstat.system.in 20.06 +11.3 31.32 ± 4% mpstat.cpu.all.idle% 1.29 -0.2 1.07 mpstat.cpu.all.irq% 0.13 -0.0 0.11 ± 2% mpstat.cpu.all.soft% 69.11 -8.4 60.67 ± 2% mpstat.cpu.all.sys% 9.40 -2.6 6.83 ± 3% mpstat.cpu.all.usr% 2556574 ± 18% +1.6e+05% 4.162e+09 ± 18% cpuidle.C1.time 988160 ± 21% +94377.8% 9.336e+08 ± 12% cpuidle.C1.usage 1.167e+08 ± 22% -69.3% 35782884 ± 70% cpuidle.C1E.time 26016930 ± 90% +318.9% 1.09e+08 ± 25% cpuidle.C6.time 73042 ± 62% +140.7% 175848 ± 4% cpuidle.C6.usage 2.507e+09 ± 2% -73.8% 6.571e+08 ± 23% cpuidle.POLL.usage 16809 -4.4% 16075 proc-vmstat.nr_active_anon 20021 -3.9% 19247 proc-vmstat.nr_shmem 16809 -4.4% 16075 proc-vmstat.nr_zone_active_anon 908.50 ±104% +311.8% 3741 ± 54% proc-vmstat.numa_pages_migrated 75555 ± 8% +18.9% 89863 ± 11% proc-vmstat.numa_pte_updates 25016 -5.2% 23727 proc-vmstat.pgactivate 908.50 ±104% +311.8% 3741 ± 54% proc-vmstat.pgmigrate_success 436827 ± 9% -42.9% 249596 ± 14% sched_debug.cfs_rq:/.MIN_vruntime.avg 1339696 ± 4% -26.6% 983616 ± 6% sched_debug.cfs_rq:/.MIN_vruntime.stddev 14427 ± 3% -11.8% 12725 ± 4% sched_debug.cfs_rq:/.load.avg 7845 ± 6% +16.9% 9167 sched_debug.cfs_rq:/.load.stddev 436827 ± 9% -42.9% 249596 ± 14% sched_debug.cfs_rq:/.max_vruntime.avg 1339696 ± 4% -26.6% 983616 ± 6% sched_debug.cfs_rq:/.max_vruntime.stddev 0.74 ± 2% -16.4% 0.62 ± 3% sched_debug.cfs_rq:/.nr_running.avg 0.34 ± 6% +25.6% 0.42 ± 2% sched_debug.cfs_rq:/.nr_running.stddev 315.71 ± 2% -11.8% 278.53 ± 3% sched_debug.cfs_rq:/.util_est_enqueued.avg 3086 ± 5% +101.4% 6218 ± 35% sched_debug.cpu.avg_idle.min 5.18 ± 2% +10.9% 5.75 ± 3% sched_debug.cpu.clock.stddev 24097711 ± 2% -37.0% 15171198 ± 2% sched_debug.cpu.nr_switches.avg 27008905 ± 4% -33.8% 17870323 ± 4% sched_debug.cpu.nr_switches.max 20920055 ± 6% -38.3% 12906743 ± 4% sched_debug.cpu.nr_switches.min 1.733e+10 ± 2% -36.0% 1.11e+10 ± 2% perf-stat.i.branch-instructions 2.838e+08 -36.0% 1.817e+08 ± 2% perf-stat.i.branch-misses 16740394 ± 2% -37.1% 10534103 ± 2% perf-stat.i.context-switches 3.47 ± 2% +48.2% 5.15 perf-stat.i.cpi 2.882e+11 -5.5% 2.724e+11 perf-stat.i.cpu-cycles 659.32 ± 3% +26.2% 832.24 ± 3% perf-stat.i.cpu-migrations 16889909 ± 2% -36.8% 10671898 ± 2% perf-stat.i.dTLB-load-misses 2.424e+10 ± 2% -36.5% 1.539e+10 ± 2% perf-stat.i.dTLB-loads 0.00 ± 3% +0.0 0.00 ± 13% perf-stat.i.dTLB-store-miss-rate% 27601 -12.7% 24088 ± 10% perf-stat.i.dTLB-store-misses 1.533e+10 ± 2% -36.9% 9.667e+09 ± 2% perf-stat.i.dTLB-stores 92.12 -75.8 16.35 ± 10% perf-stat.i.iTLB-load-miss-rate% 17765547 -26.0% 13139994 ± 2% perf-stat.i.iTLB-load-misses 1483996 ± 10% +4486.1% 68058032 ± 10% perf-stat.i.iTLB-loads 8.286e+10 ± 2% -36.2% 5.287e+10 ± 2% perf-stat.i.instructions 4739 -13.8% 4086 perf-stat.i.instructions-per-iTLB-miss 0.29 -32.2% 0.20 perf-stat.i.ipc 0.53 +37.7% 0.73 ± 27% perf-stat.i.major-faults 2.77 -5.5% 2.62 perf-stat.i.metric.GHz 548.46 ± 2% -36.2% 349.65 ± 2% perf-stat.i.metric.M/sec 1818026 ± 2% -12.4% 1591936 ± 3% perf-stat.i.node-store-misses 7101 ± 5% +21.4% 8618 ± 11% perf-stat.i.node-stores 3.48 ± 2% +48.1% 5.15 perf-stat.overall.cpi 0.00 +0.0 0.00 ± 10% perf-stat.overall.dTLB-store-miss-rate% 92.30 -76.0 16.34 ± 10% perf-stat.overall.iTLB-load-miss-rate% 4663 -13.7% 4024 perf-stat.overall.instructions-per-iTLB-miss 0.29 ± 2% -32.5% 0.19 perf-stat.overall.ipc 2962542 +1.4% 3005468 perf-stat.overall.path-length 1.727e+10 ± 2% -35.9% 1.106e+10 ± 2% perf-stat.ps.branch-instructions 2.828e+08 -36.0% 1.811e+08 ± 2% perf-stat.ps.branch-misses 16681900 ± 2% -37.1% 10499012 ± 2% perf-stat.ps.context-switches 2.872e+11 -5.5% 2.715e+11 perf-stat.ps.cpu-cycles 657.84 ± 3% +26.1% 829.45 ± 3% perf-stat.ps.cpu-migrations 16831485 ± 2% -36.8% 10637252 ± 2% perf-stat.ps.dTLB-load-misses 2.416e+10 ± 2% -36.5% 1.534e+10 ± 2% perf-stat.ps.dTLB-loads 27550 -12.7% 24040 ± 10% perf-stat.ps.dTLB-store-misses 1.527e+10 ± 2% -36.9% 9.636e+09 ± 2% perf-stat.ps.dTLB-stores 17705408 -26.0% 13096731 ± 2% perf-stat.ps.iTLB-load-misses 1479549 ± 10% +4483.7% 67818252 ± 10% perf-stat.ps.iTLB-loads 8.257e+10 ± 2% -36.2% 5.269e+10 ± 2% perf-stat.ps.instructions 0.53 +37.4% 0.72 ± 27% perf-stat.ps.major-faults 1811676 ± 2% -12.4% 1586474 ± 3% perf-stat.ps.node-store-misses 7102 ± 5% +21.3% 8615 ± 11% perf-stat.ps.node-stores 2.491e+13 ± 2% -36.1% 1.591e+13 ± 2% perf-stat.total.instructions 27105 ± 11% +20.8% 32744 ± 8% softirqs.CPU1.RCU 26920 ± 12% +19.1% 32067 ± 6% softirqs.CPU10.RCU 26751 ± 12% +17.4% 31413 ± 6% softirqs.CPU100.RCU 26471 ± 12% +17.3% 31045 ± 7% softirqs.CPU101.RCU 26980 ± 12% +15.3% 31101 ± 7% softirqs.CPU102.RCU 27057 ± 13% +17.1% 31695 ± 8% softirqs.CPU103.RCU 26830 ± 11% +20.6% 32356 ± 7% softirqs.CPU11.RCU 26679 ± 11% +27.6% 34039 ± 16% softirqs.CPU12.RCU 26651 ± 11% +19.3% 31806 ± 7% softirqs.CPU13.RCU 26549 ± 11% +20.7% 32044 ± 6% softirqs.CPU14.RCU 28907 ± 9% +18.7% 34299 ± 6% softirqs.CPU15.RCU 28775 ± 11% +18.6% 34116 ± 7% softirqs.CPU16.RCU 28749 ± 13% +17.5% 33768 ± 7% softirqs.CPU17.RCU 28518 ± 10% +17.8% 33600 ± 7% softirqs.CPU18.RCU 28382 ± 10% +18.7% 33700 ± 7% softirqs.CPU19.RCU 28682 ± 10% +17.7% 33759 ± 7% softirqs.CPU20.RCU 28510 ± 10% +20.3% 34310 ± 7% softirqs.CPU21.RCU 28609 ± 10% +18.4% 33874 ± 6% softirqs.CPU22.RCU 28591 ± 11% +19.0% 34025 ± 7% softirqs.CPU23.RCU 27572 ± 7% +21.3% 33449 ± 7% softirqs.CPU24.RCU 28439 ± 11% +18.3% 33639 ± 8% softirqs.CPU25.RCU 27063 ± 10% +18.6% 32089 ± 8% softirqs.CPU27.RCU 27056 ± 11% +16.6% 31547 ± 7% softirqs.CPU28.RCU 27183 ± 10% +18.0% 32078 ± 8% softirqs.CPU3.RCU 27508 ± 11% +16.6% 32073 ± 7% softirqs.CPU30.RCU 27613 ± 12% +15.1% 31776 ± 9% softirqs.CPU32.RCU 27570 ± 13% +16.0% 31990 ± 7% softirqs.CPU33.RCU 27317 ± 11% +17.6% 32121 ± 8% softirqs.CPU34.RCU 27119 ± 11% +18.0% 32010 ± 8% softirqs.CPU35.RCU 27230 ± 11% +18.6% 32306 ± 7% softirqs.CPU37.RCU 27458 ± 12% +16.4% 31963 ± 8% softirqs.CPU38.RCU 27380 ± 11% +18.4% 32431 ± 8% softirqs.CPU39.RCU 27095 ± 11% +18.8% 32190 ± 7% softirqs.CPU4.RCU 27520 ± 12% +19.8% 32962 ± 8% softirqs.CPU40.RCU 27596 ± 12% +16.0% 32021 ± 8% softirqs.CPU41.RCU 27542 ± 11% +17.0% 32218 ± 7% softirqs.CPU42.RCU 27511 ± 12% +19.5% 32878 ± 6% softirqs.CPU45.RCU 27653 ± 13% +15.7% 31998 ± 8% softirqs.CPU46.RCU 27508 ± 12% +16.9% 32156 ± 7% softirqs.CPU47.RCU 27521 ± 12% +17.9% 32444 ± 6% softirqs.CPU48.RCU 27251 ± 10% +17.3% 31971 ± 7% softirqs.CPU5.RCU 27739 ± 12% +15.9% 32140 ± 8% softirqs.CPU51.RCU 27818 ± 10% +17.1% 32564 ± 7% softirqs.CPU52.RCU 28329 ± 11% +19.7% 33910 ± 8% softirqs.CPU53.RCU 27982 ± 12% +19.0% 33291 ± 6% softirqs.CPU54.RCU 28193 ± 11% +18.6% 33433 ± 7% softirqs.CPU55.RCU 28119 ± 11% +19.3% 33544 ± 7% softirqs.CPU56.RCU 28493 ± 10% +16.5% 33182 ± 7% softirqs.CPU57.RCU 28030 ± 11% +17.0% 32782 ± 6% softirqs.CPU58.RCU 27924 ± 11% +19.8% 33461 ± 7% softirqs.CPU59.RCU 26752 ± 11% +17.8% 31504 ± 6% softirqs.CPU6.RCU 26528 ± 13% +19.8% 31789 ± 6% softirqs.CPU60.RCU 26420 ± 12% +20.2% 31760 ± 8% softirqs.CPU61.RCU 27217 ± 12% +16.3% 31656 ± 7% softirqs.CPU62.RCU 26954 ± 11% +17.8% 31765 ± 7% softirqs.CPU63.RCU 26995 ± 11% +18.8% 32066 ± 7% softirqs.CPU64.RCU 27085 ± 11% +17.6% 31848 ± 7% softirqs.CPU65.RCU 26962 ± 10% +18.9% 32056 ± 7% softirqs.CPU66.RCU 27161 ± 10% +17.4% 31895 ± 7% softirqs.CPU67.RCU 27133 ± 11% +17.9% 31994 ± 6% softirqs.CPU68.RCU 27356 ± 13% +17.3% 32078 ± 7% softirqs.CPU69.RCU 27438 ± 10% +16.7% 32025 ± 5% softirqs.CPU7.RCU 26932 ± 11% +18.5% 31904 ± 6% softirqs.CPU70.RCU 26961 ± 11% +17.9% 31784 ± 7% softirqs.CPU71.RCU 27060 ± 11% +17.7% 31850 ± 6% softirqs.CPU72.RCU 26950 ± 11% +18.6% 31968 ± 7% softirqs.CPU73.RCU 27074 ± 12% +17.7% 31871 ± 7% softirqs.CPU74.RCU 26269 ± 10% +18.6% 31163 ± 7% softirqs.CPU75.RCU 25904 ± 10% +20.3% 31161 ± 7% softirqs.CPU76.RCU 26007 ± 11% +28.0% 33287 ± 10% softirqs.CPU77.RCU 27643 ± 12% +17.5% 32490 ± 9% softirqs.CPU79.RCU 26424 ± 12% +20.5% 31852 ± 6% softirqs.CPU8.RCU 27861 ± 11% +17.9% 32839 ± 8% softirqs.CPU81.RCU 28220 ± 10% +14.7% 32355 ± 8% softirqs.CPU82.RCU 27666 ± 11% +15.7% 32006 ± 9% softirqs.CPU84.RCU 27277 ± 11% +17.6% 32086 ± 8% softirqs.CPU85.RCU 27530 ± 11% +17.9% 32453 ± 8% softirqs.CPU87.RCU 27395 ± 11% +17.9% 32286 ± 7% softirqs.CPU89.RCU 26430 ± 11% +18.7% 31362 ± 9% softirqs.CPU90.RCU 26569 ± 12% +17.4% 31202 ± 8% softirqs.CPU91.RCU 26618 ± 12% +18.2% 31472 ± 8% softirqs.CPU92.RCU 26591 ± 10% +17.3% 31180 ± 8% softirqs.CPU93.RCU 26376 ± 12% +18.3% 31204 ± 8% softirqs.CPU94.RCU 26339 ± 12% +17.1% 30837 ± 8% softirqs.CPU96.RCU 26506 ± 12% +17.2% 31057 ± 7% softirqs.CPU97.RCU 26608 ± 12% +16.6% 31034 ± 8% softirqs.CPU98.RCU 2853153 ± 11% +17.8% 3360022 ± 7% softirqs.RCU 67.56 -19.1 48.43 ± 4% perf-profile.calltrace.cycles-pp.__libc_read 64.22 -17.9 46.31 ± 4% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_read 62.09 -17.3 44.74 ± 4% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read 61.78 -17.2 44.56 ± 4% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read 61.17 -16.9 44.23 ± 4% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read 60.19 -16.6 43.63 ± 4% perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 59.84 -16.5 43.39 ± 4% perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64 54.14 -12.8 41.30 ± 4% perf-profile.calltrace.cycles-pp.schedule.pipe_read.new_sync_read.vfs_read.ksys_read 54.07 -12.8 41.26 ± 4% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.pipe_read.new_sync_read.vfs_read 25.92 -7.7 18.25 ± 2% perf-profile.calltrace.cycles-pp.finish_task_switch.__sched_text_start.schedule.pipe_read.new_sync_read 11.52 ± 2% -3.0 8.49 perf-profile.calltrace.cycles-pp.__libc_write 7.82 ± 2% -2.7 5.14 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write 7.58 ± 2% -2.6 5.01 ± 2% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write 6.98 ± 3% -2.3 4.66 ± 2% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write 9.47 ± 2% -2.3 7.16 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write 6.13 ± 3% -1.9 4.22 ± 2% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.88 ± 2% -1.8 4.08 ± 2% perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64 1.49 ± 20% -1.2 0.28 ±100% perf-profile.calltrace.cycles-pp.prepare_to_wait_event.pipe_read.new_sync_read.vfs_read.ksys_read 3.65 -1.2 2.46 ± 6% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 4.19 ± 3% -1.2 3.04 ± 2% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write 3.87 ± 3% -1.0 2.83 ± 2% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write 3.79 ± 3% -1.0 2.78 ± 3% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write 1.29 -1.0 0.29 ±100% perf-profile.calltrace.cycles-pp.tick_nohz_idle_exit.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 3.59 ± 3% -0.9 2.67 ± 3% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write 1.93 -0.5 1.41 ± 3% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__libc_read 2.02 ± 3% -0.5 1.50 ± 6% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 1.95 ± 4% -0.5 1.47 ± 6% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common 1.31 ± 3% -0.4 0.88 perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry 0.98 ± 2% -0.4 0.60 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__libc_read 1.56 ± 4% -0.4 1.20 ± 6% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function 0.97 ± 2% -0.3 0.63 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__libc_write 0.85 ± 2% -0.3 0.54 ± 3% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__libc_read 0.84 -0.3 0.55 ± 2% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__libc_write 2.24 ± 7% -0.3 1.96 ± 7% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.pipe_read.new_sync_read 0.82 ± 4% -0.2 0.59 ± 3% perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__sched_text_start.schedule_idle.do_idle 0.72 +0.3 0.98 ± 8% perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry 1.57 +0.4 1.97 ± 3% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.__libc_write 0.00 +0.8 0.82 ± 5% perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle 0.00 +3.8 3.84 ± 39% perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry 7.42 ± 3% +12.2 19.66 ± 7% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 7.39 ± 3% +12.2 19.63 ± 7% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 3.68 ± 6% +12.4 16.11 ± 7% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 3.57 ± 6% +12.5 16.04 ± 7% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary 0.00 +13.1 13.14 ± 18% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 19.21 +22.5 41.73 ± 4% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 19.18 +22.5 41.72 ± 4% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 19.05 +22.6 41.65 ± 4% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 19.39 +22.8 42.18 ± 4% perf-profile.calltrace.cycles-pp.secondary_startup_64 73.73 -20.2 53.51 ± 3% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 69.92 -20.0 49.91 ± 4% perf-profile.children.cycles-pp.do_syscall_64 67.70 -19.2 48.52 ± 4% perf-profile.children.cycles-pp.__libc_read 61.82 -17.2 44.59 ± 4% perf-profile.children.cycles-pp.ksys_read 61.20 -17.0 44.24 ± 4% perf-profile.children.cycles-pp.vfs_read 60.20 -16.6 43.63 ± 4% perf-profile.children.cycles-pp.new_sync_read 59.88 -16.5 43.41 ± 4% perf-profile.children.cycles-pp.pipe_read 54.14 -12.8 41.30 ± 4% perf-profile.children.cycles-pp.schedule 26.24 -7.8 18.48 ± 2% perf-profile.children.cycles-pp.finish_task_switch 11.66 ± 2% -3.1 8.57 perf-profile.children.cycles-pp.__libc_write 7.61 ± 2% -2.6 5.03 ± 2% perf-profile.children.cycles-pp.ksys_write 7.00 ± 3% -2.3 4.67 ± 2% perf-profile.children.cycles-pp.vfs_write 6.13 ± 2% -1.9 4.23 ± 2% perf-profile.children.cycles-pp.new_sync_write 5.90 ± 2% -1.8 4.09 ± 2% perf-profile.children.cycles-pp.pipe_write 3.70 -1.2 2.49 ± 6% perf-profile.children.cycles-pp.menu_select 2.03 ± 14% -1.2 0.83 ± 8% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 4.20 ± 3% -1.2 3.04 ± 2% perf-profile.children.cycles-pp.__wake_up_common_lock 3.88 ± 3% -1.0 2.84 ± 2% perf-profile.children.cycles-pp.__wake_up_common 3.80 ± 3% -1.0 2.79 ± 3% perf-profile.children.cycles-pp.autoremove_wake_function 1.17 ± 5% -1.0 0.18 ± 7% perf-profile.children.cycles-pp._raw_spin_lock_irq 1.51 ± 20% -1.0 0.53 ± 7% perf-profile.children.cycles-pp.prepare_to_wait_event 3.61 ± 3% -0.9 2.69 ± 3% perf-profile.children.cycles-pp.try_to_wake_up 1.33 -0.8 0.52 ± 13% perf-profile.children.cycles-pp.tick_nohz_idle_exit 1.96 ± 2% -0.7 1.24 ± 2% perf-profile.children.cycles-pp.entry_SYSCALL_64 1.95 -0.7 1.25 ± 2% perf-profile.children.cycles-pp.syscall_return_via_sysret 1.29 -0.6 0.67 ± 12% perf-profile.children.cycles-pp.ktime_get 1.60 ± 3% -0.6 1.05 ± 2% perf-profile.children.cycles-pp.pick_next_task_fair 0.68 ± 3% -0.5 0.13 ± 6% perf-profile.children.cycles-pp.finish_wait 2.04 ± 3% -0.5 1.51 ± 6% perf-profile.children.cycles-pp.ttwu_do_activate 1.97 ± 4% -0.5 1.47 ± 6% perf-profile.children.cycles-pp.enqueue_task_fair 0.93 -0.5 0.43 ± 12% perf-profile.children.cycles-pp.read_tsc 0.92 ± 2% -0.4 0.48 ± 3% perf-profile.children.cycles-pp.security_file_permission 1.63 ± 4% -0.4 1.23 ± 6% perf-profile.children.cycles-pp.enqueue_entity 0.72 ± 3% -0.4 0.36 ± 5% perf-profile.children.cycles-pp.mutex_lock 0.91 -0.4 0.55 ± 2% perf-profile.children.cycles-pp.__switch_to_asm 0.77 ± 2% -0.4 0.41 ± 5% perf-profile.children.cycles-pp.sched_clock_cpu 0.71 ± 2% -0.3 0.38 ± 5% perf-profile.children.cycles-pp.sched_clock 0.67 ± 2% -0.3 0.35 ± 4% perf-profile.children.cycles-pp.native_sched_clock 0.60 ± 3% -0.3 0.29 ± 5% perf-profile.children.cycles-pp.copy_page_to_iter 0.64 -0.3 0.33 ± 14% perf-profile.children.cycles-pp.rcu_idle_exit 1.50 ± 4% -0.3 1.21 ± 7% perf-profile.children.cycles-pp.update_load_avg 0.54 ± 14% -0.3 0.26 ± 4% perf-profile.children.cycles-pp.cpuidle_governor_latency_req 2.25 ± 7% -0.3 1.96 ± 7% perf-profile.children.cycles-pp.dequeue_task_fair 0.74 ± 2% -0.3 0.46 ± 4% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.40 ± 3% -0.3 0.15 ± 3% perf-profile.children.cycles-pp.mutex_unlock 0.86 ± 3% -0.2 0.62 ± 3% perf-profile.children.cycles-pp.set_next_entity 0.65 ± 2% -0.2 0.41 ± 2% perf-profile.children.cycles-pp.select_task_rq_fair 0.65 ± 5% -0.2 0.41 ± 5% perf-profile.children.cycles-pp.__fdget_pos 0.33 ± 7% -0.2 0.10 ± 4% perf-profile.children.cycles-pp.__pthread_enable_asynccancel 0.40 ± 2% -0.2 0.17 ± 7% perf-profile.children.cycles-pp.exit_to_user_mode_prepare 1.93 ± 7% -0.2 1.70 ± 7% perf-profile.children.cycles-pp.dequeue_entity 0.61 ± 5% -0.2 0.39 ± 6% perf-profile.children.cycles-pp.__fget_light 0.43 ± 2% -0.2 0.22 ± 4% perf-profile.children.cycles-pp.__x86_retpoline_rax 0.50 ± 2% -0.2 0.29 ± 2% perf-profile.children.cycles-pp.copy_page_from_iter 0.47 ± 2% -0.2 0.27 ± 4% perf-profile.children.cycles-pp.fsnotify 0.40 ± 3% -0.2 0.21 ± 5% perf-profile.children.cycles-pp.___might_sleep 0.43 -0.2 0.25 ± 4% perf-profile.children.cycles-pp.common_file_perm 0.28 ± 5% -0.2 0.10 ± 5% perf-profile.children.cycles-pp.fput_many 0.46 ± 7% -0.2 0.30 ± 6% perf-profile.children.cycles-pp.__fget_files 0.24 ± 5% -0.2 0.08 ± 5% perf-profile.children.cycles-pp.switch_fpu_return 0.29 ± 2% -0.1 0.15 ± 3% perf-profile.children.cycles-pp.put_prev_task_fair 0.17 ± 30% -0.1 0.03 ±100% perf-profile.children.cycles-pp.pm_qos_read_value 0.40 ± 3% -0.1 0.25 ± 2% perf-profile.children.cycles-pp.copy_fpregs_to_fpstate 0.21 ± 4% -0.1 0.07 ± 5% perf-profile.children.cycles-pp.update_ts_time_stats 0.48 -0.1 0.34 ± 10% perf-profile.children.cycles-pp.tick_nohz_idle_enter 0.40 ± 2% -0.1 0.26 perf-profile.children.cycles-pp.update_rq_clock 0.46 -0.1 0.33 ± 3% perf-profile.children.cycles-pp.__update_load_avg_se 0.24 ± 4% -0.1 0.12 ± 3% perf-profile.children.cycles-pp.copyout 0.24 ± 2% -0.1 0.13 ± 3% perf-profile.children.cycles-pp.__might_sleep 0.34 ± 4% -0.1 0.23 ± 3% perf-profile.children.cycles-pp.reweight_entity 0.19 ± 3% -0.1 0.08 ± 5% perf-profile.children.cycles-pp.__might_fault 0.26 ± 4% -0.1 0.16 ± 4% perf-profile.children.cycles-pp.__wrgsbase_inactive 0.27 ± 3% -0.1 0.17 ± 14% perf-profile.children.cycles-pp.rcu_eqs_exit 0.44 ± 7% -0.1 0.34 ± 5% perf-profile.children.cycles-pp.update_cfs_group 0.40 ± 3% -0.1 0.30 ± 5% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.33 ± 2% -0.1 0.23 ± 4% perf-profile.children.cycles-pp.___perf_sw_event 0.15 ± 2% -0.1 0.06 ± 9% perf-profile.children.cycles-pp.put_prev_entity 0.18 ± 2% -0.1 0.09 ± 4% perf-profile.children.cycles-pp.newidle_balance 0.23 ± 2% -0.1 0.15 ± 5% perf-profile.children.cycles-pp.copy_user_generic_unrolled 0.19 ± 2% -0.1 0.11 ± 4% perf-profile.children.cycles-pp.anon_pipe_buf_release 0.24 -0.1 0.15 ± 5% perf-profile.children.cycles-pp.menu_reflect 0.25 ± 4% -0.1 0.17 ± 8% perf-profile.children.cycles-pp.current_time 0.13 ± 3% -0.1 0.05 ± 8% perf-profile.children.cycles-pp.can_stop_idle_tick 0.10 ± 4% -0.1 0.03 ±100% perf-profile.children.cycles-pp.__x64_sys_write 0.24 ± 2% -0.1 0.17 ± 7% perf-profile.children.cycles-pp.touch_atime 0.14 ± 5% -0.1 0.07 ± 5% perf-profile.children.cycles-pp.apparmor_file_permission 0.20 ± 2% -0.1 0.13 ± 5% perf-profile.children.cycles-pp.copyin 0.18 ± 2% -0.1 0.12 ± 3% perf-profile.children.cycles-pp.select_idle_sibling 0.27 ± 3% -0.1 0.21 ± 3% perf-profile.children.cycles-pp.file_update_time 0.12 ± 7% -0.1 0.06 ± 9% perf-profile.children.cycles-pp.nr_iowait_cpu 0.17 ± 4% -0.1 0.11 perf-profile.children.cycles-pp.syscall_enter_from_user_mode 0.18 ± 4% -0.1 0.12 ± 6% perf-profile.children.cycles-pp.atime_needs_update 0.14 ± 3% -0.1 0.08 perf-profile.children.cycles-pp._cond_resched 0.16 ± 2% -0.1 0.10 perf-profile.children.cycles-pp.__rdgsbase_inactive 0.20 ± 2% -0.1 0.14 ± 3% perf-profile.children.cycles-pp.__pthread_disable_asynccancel 0.12 ± 4% -0.1 0.07 ± 5% perf-profile.children.cycles-pp.put_prev_task_idle 0.09 -0.1 0.04 ± 57% perf-profile.children.cycles-pp.rb_erase 0.08 ± 14% -0.1 0.03 ±100% perf-profile.children.cycles-pp.aa_file_perm 0.48 ± 5% -0.0 0.44 ± 5% perf-profile.children.cycles-pp.asm_call_on_stack 0.10 ± 7% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.06 ± 6% -0.0 0.03 ±100% perf-profile.children.cycles-pp.available_idle_cpu 0.11 ± 4% -0.0 0.07 perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 0.12 ± 5% -0.0 0.08 ± 13% perf-profile.children.cycles-pp.rcu_dynticks_eqs_exit 0.08 ± 8% -0.0 0.05 perf-profile.children.cycles-pp.__cgroup_account_cputime 0.12 ± 5% -0.0 0.09 ± 4% perf-profile.children.cycles-pp.cpuidle_not_available 0.08 ± 5% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.timestamp_truncate 0.10 ± 4% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.update_min_vruntime 0.08 ± 10% -0.0 0.06 ± 17% perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64 0.08 ± 5% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.tick_nohz_idle_got_tick 0.15 ± 4% -0.0 0.13 perf-profile.children.cycles-pp.pick_next_entity 0.07 ± 7% +0.0 0.09 ± 4% perf-profile.children.cycles-pp.place_entity 0.60 +0.0 0.65 ± 2% perf-profile.children.cycles-pp._raw_spin_lock 0.26 ± 4% +0.1 0.35 ± 3% perf-profile.children.cycles-pp.ttwu_do_wakeup 0.23 ± 4% +0.1 0.33 ± 4% perf-profile.children.cycles-pp.check_preempt_curr 0.13 ± 6% +0.1 0.26 ± 6% perf-profile.children.cycles-pp.resched_curr 0.21 ± 3% +0.1 0.35 ± 12% perf-profile.children.cycles-pp.rcu_eqs_enter 0.14 ± 8% +0.2 0.32 ± 14% perf-profile.children.cycles-pp.rcu_dynticks_eqs_enter 0.74 +0.3 1.00 ± 7% perf-profile.children.cycles-pp.tick_nohz_next_event 0.18 ± 22% +0.3 0.45 ± 4% perf-profile.children.cycles-pp.start_kernel 0.46 +0.4 0.83 ± 5% perf-profile.children.cycles-pp.get_next_timer_interrupt 0.35 ± 32% +3.5 3.89 ± 39% perf-profile.children.cycles-pp.switch_mm_irqs_off 7.50 ± 3% +12.4 19.87 ± 7% perf-profile.children.cycles-pp.cpuidle_enter 7.48 ± 3% +12.4 19.86 ± 7% perf-profile.children.cycles-pp.cpuidle_enter_state 3.72 ± 5% +12.6 16.29 ± 7% perf-profile.children.cycles-pp.schedule_idle 0.00 +13.3 13.30 ± 18% perf-profile.children.cycles-pp.intel_idle 19.21 +22.5 41.73 ± 4% perf-profile.children.cycles-pp.start_secondary 19.39 +22.8 42.18 ± 4% perf-profile.children.cycles-pp.secondary_startup_64 19.39 +22.8 42.18 ± 4% perf-profile.children.cycles-pp.cpu_startup_entry 19.31 +22.8 42.15 ± 4% perf-profile.children.cycles-pp.do_idle 25.88 -7.7 18.14 ± 2% perf-profile.self.cycles-pp.finish_task_switch 2.01 ± 14% -1.2 0.82 ± 8% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 1.16 ± 5% -1.0 0.18 ± 5% perf-profile.self.cycles-pp._raw_spin_lock_irq 1.65 ± 2% -0.9 0.75 ± 6% perf-profile.self.cycles-pp.menu_select 1.94 -0.7 1.24 ± 2% perf-profile.self.cycles-pp.syscall_return_via_sysret 1.70 ± 2% -0.6 1.08 ± 2% perf-profile.self.cycles-pp.entry_SYSCALL_64 0.90 ± 2% -0.5 0.41 ± 13% perf-profile.self.cycles-pp.read_tsc 0.91 -0.4 0.54 ± 2% perf-profile.self.cycles-pp.__switch_to_asm 0.65 ± 2% -0.3 0.34 ± 4% perf-profile.self.cycles-pp.native_sched_clock 0.69 -0.3 0.43 ± 3% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.39 ± 2% -0.2 0.14 ± 3% perf-profile.self.cycles-pp.mutex_unlock 0.43 ± 6% -0.2 0.19 ± 6% perf-profile.self.cycles-pp.mutex_lock 0.33 ± 7% -0.2 0.10 ± 7% perf-profile.self.cycles-pp.__pthread_enable_asynccancel 0.38 -0.2 0.17 ± 13% perf-profile.self.cycles-pp.rcu_idle_exit 0.36 -0.2 0.17 ± 3% perf-profile.self.cycles-pp.prepare_to_wait_event 0.45 ± 3% -0.2 0.25 ± 5% perf-profile.self.cycles-pp.fsnotify 0.37 ± 2% -0.2 0.18 ± 3% perf-profile.self.cycles-pp.__x86_retpoline_rax 0.39 ± 4% -0.2 0.21 ± 6% perf-profile.self.cycles-pp.___might_sleep 0.27 ± 5% -0.2 0.09 ± 8% perf-profile.self.cycles-pp.fput_many 0.45 ± 8% -0.2 0.29 ± 6% perf-profile.self.cycles-pp.__fget_files 0.44 ± 3% -0.2 0.28 ± 2% perf-profile.self.cycles-pp.select_task_rq_fair 0.26 ± 11% -0.2 0.11 ± 4% perf-profile.self.cycles-pp.cpuidle_governor_latency_req 0.36 ± 5% -0.2 0.21 ± 7% perf-profile.self.cycles-pp.common_file_perm 0.39 ± 3% -0.2 0.24 ± 3% perf-profile.self.cycles-pp.copy_fpregs_to_fpstate 0.23 ± 4% -0.1 0.08 ± 10% perf-profile.self.cycles-pp.switch_fpu_return 0.17 ± 29% -0.1 0.03 ±100% perf-profile.self.cycles-pp.pm_qos_read_value 0.44 -0.1 0.31 ± 2% perf-profile.self.cycles-pp.__update_load_avg_se 0.41 ± 2% -0.1 0.29 ± 3% perf-profile.self.cycles-pp.pipe_read 0.30 ± 2% -0.1 0.18 ± 4% perf-profile.self.cycles-pp.vfs_read 0.19 ± 2% -0.1 0.07 ± 14% perf-profile.self.cycles-pp.tick_nohz_idle_exit 0.20 ± 14% -0.1 0.08 ± 5% perf-profile.self.cycles-pp.security_file_permission 0.30 -0.1 0.18 ± 3% perf-profile.self.cycles-pp.pipe_write 0.42 ± 3% -0.1 0.31 ± 7% perf-profile.self.cycles-pp.enqueue_entity 0.14 ± 5% -0.1 0.03 ±100% perf-profile.self.cycles-pp.update_ts_time_stats 0.34 ± 3% -0.1 0.23 ± 4% perf-profile.self.cycles-pp.reweight_entity 0.29 ± 6% -0.1 0.18 ± 3% perf-profile.self.cycles-pp.vfs_write 0.22 ± 3% -0.1 0.11 ± 3% perf-profile.self.cycles-pp.__might_sleep 0.34 ± 4% -0.1 0.23 ± 9% perf-profile.self.cycles-pp.enqueue_task_fair 0.26 ± 4% -0.1 0.16 ± 4% perf-profile.self.cycles-pp.__wrgsbase_inactive 0.32 ± 2% -0.1 0.22 ± 17% perf-profile.self.cycles-pp.ktime_get 0.38 ± 2% -0.1 0.28 ± 5% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.24 ± 3% -0.1 0.15 ± 2% perf-profile.self.cycles-pp.pick_next_task_fair 0.13 ± 3% -0.1 0.04 ± 57% perf-profile.self.cycles-pp.put_prev_entity 0.42 ± 8% -0.1 0.33 ± 6% perf-profile.self.cycles-pp.update_cfs_group 0.23 ± 2% -0.1 0.14 ± 6% perf-profile.self.cycles-pp.copy_user_generic_unrolled 0.17 ± 3% -0.1 0.08 perf-profile.self.cycles-pp.newidle_balance 0.28 ± 2% -0.1 0.20 ± 6% perf-profile.self.cycles-pp.___perf_sw_event 0.27 -0.1 0.19 ± 3% perf-profile.self.cycles-pp.new_sync_read 0.29 ± 2% -0.1 0.21 ± 7% perf-profile.self.cycles-pp.dequeue_entity 0.18 ± 2% -0.1 0.10 ± 4% perf-profile.self.cycles-pp.anon_pipe_buf_release 0.13 ± 5% -0.1 0.05 ± 8% perf-profile.self.cycles-pp.can_stop_idle_tick 0.19 -0.1 0.11 ± 4% perf-profile.self.cycles-pp.ksys_read 0.19 ± 2% -0.1 0.12 perf-profile.self.cycles-pp.__libc_write 0.28 ± 4% -0.1 0.21 ± 3% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.19 ± 4% -0.1 0.12 ± 4% perf-profile.self.cycles-pp.try_to_wake_up 0.11 ± 8% -0.1 0.04 ± 57% perf-profile.self.cycles-pp.nr_iowait_cpu 0.15 ± 4% -0.1 0.08 ± 5% perf-profile.self.cycles-pp.autoremove_wake_function 0.15 ± 3% -0.1 0.08 ± 8% perf-profile.self.cycles-pp.copy_page_to_iter 0.15 -0.1 0.08 ± 5% perf-profile.self.cycles-pp.exit_to_user_mode_prepare 0.13 -0.1 0.07 perf-profile.self.cycles-pp.copy_page_from_iter 0.14 ± 5% -0.1 0.08 ± 5% perf-profile.self.cycles-pp.put_prev_task_fair 0.16 ± 4% -0.1 0.10 ± 4% perf-profile.self.cycles-pp.syscall_enter_from_user_mode 0.12 ± 4% -0.1 0.07 ± 12% perf-profile.self.cycles-pp.apparmor_file_permission 0.13 ± 3% -0.1 0.07 ± 17% perf-profile.self.cycles-pp.rcu_eqs_exit 0.08 ± 8% -0.1 0.03 ±100% perf-profile.self.cycles-pp.get_next_timer_interrupt 0.15 ± 2% -0.1 0.09 ± 4% perf-profile.self.cycles-pp.menu_reflect 0.18 ± 4% -0.0 0.13 ± 3% perf-profile.self.cycles-pp.__pthread_disable_asynccancel 0.15 ± 4% -0.0 0.10 perf-profile.self.cycles-pp.__rdgsbase_inactive 0.20 -0.0 0.15 ± 2% perf-profile.self.cycles-pp.__libc_read 0.14 ± 11% -0.0 0.09 ± 9% perf-profile.self.cycles-pp.__fget_light 0.07 ± 5% -0.0 0.03 ±100% perf-profile.self.cycles-pp.__wake_up_common 0.12 ± 3% -0.0 0.08 ± 6% perf-profile.self.cycles-pp.do_syscall_64 0.14 ± 9% -0.0 0.10 ± 11% perf-profile.self.cycles-pp.new_sync_write 0.13 ± 3% -0.0 0.09 ± 4% perf-profile.self.cycles-pp.update_rq_clock 0.11 ± 4% -0.0 0.06 perf-profile.self.cycles-pp.cpu_startup_entry 0.12 ± 8% -0.0 0.08 ± 15% perf-profile.self.cycles-pp.ksys_write 0.11 ± 12% -0.0 0.07 ± 16% perf-profile.self.cycles-pp.__hrtimer_next_event_base 0.11 ± 4% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 0.27 ± 8% -0.0 0.23 ± 8% perf-profile.self.cycles-pp.dequeue_task_fair 0.14 ± 3% -0.0 0.10 ± 4% perf-profile.self.cycles-pp.file_update_time 0.11 ± 4% -0.0 0.07 ± 6% perf-profile.self.cycles-pp.put_prev_task_idle 0.06 ± 6% -0.0 0.03 ±100% perf-profile.self.cycles-pp.available_idle_cpu 0.09 -0.0 0.05 ± 8% perf-profile.self.cycles-pp.schedule_idle 0.07 ± 5% -0.0 0.04 ± 57% perf-profile.self.cycles-pp.select_idle_sibling 0.21 ± 5% -0.0 0.18 ± 4% perf-profile.self.cycles-pp.set_next_entity 0.12 ± 4% -0.0 0.10 ± 5% perf-profile.self.cycles-pp.cpuidle_enter_state 0.12 ± 7% -0.0 0.09 ± 4% perf-profile.self.cycles-pp.cpuidle_not_available 0.11 ± 4% -0.0 0.08 ± 13% perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit 0.10 ± 5% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.__wake_up_common_lock 0.10 ± 4% -0.0 0.07 ± 6% perf-profile.self.cycles-pp.check_preempt_curr 0.08 ± 10% -0.0 0.06 ± 15% perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64 0.08 ± 8% -0.0 0.06 ± 9% perf-profile.self.cycles-pp.current_time 0.08 ± 6% -0.0 0.05 perf-profile.self.cycles-pp.timestamp_truncate 0.11 ± 3% -0.0 0.09 perf-profile.self.cycles-pp.pick_next_entity 0.08 ± 5% -0.0 0.06 ± 7% perf-profile.self.cycles-pp.tick_nohz_idle_got_tick 0.10 ± 4% -0.0 0.08 ± 5% perf-profile.self.cycles-pp.update_min_vruntime 0.05 ± 8% +0.0 0.07 ± 6% perf-profile.self.cycles-pp.__enqueue_entity 0.06 ± 7% +0.0 0.08 perf-profile.self.cycles-pp.place_entity 0.58 ± 2% +0.1 0.64 ± 2% perf-profile.self.cycles-pp._raw_spin_lock 0.12 ± 6% +0.1 0.26 ± 6% perf-profile.self.cycles-pp.resched_curr 0.13 ± 6% +0.2 0.32 ± 14% perf-profile.self.cycles-pp.rcu_dynticks_eqs_enter 0.34 ± 32% +3.5 3.89 ± 39% perf-profile.self.cycles-pp.switch_mm_irqs_off 24.89 +3.6 28.49 ± 2% perf-profile.self.cycles-pp.__sched_text_start 0.00 +13.3 13.30 ± 18% perf-profile.self.cycles-pp.intel_idle 659.50 ±108% +688.7% 5201 ± 71% interrupts.40:PCI-MSI.67633155-edge.eth0-TxRx-2 23689902 ± 8% -91.3% 2063004 ± 5% interrupts.CAL:Function_call_interrupts 195589 ± 29% -92.1% 15544 ± 34% interrupts.CPU0.CAL:Function_call_interrupts 91255 ± 35% -74.1% 23629 ± 11% interrupts.CPU0.RES:Rescheduling_interrupts 920.75 ± 5% -17.2% 762.25 ± 6% interrupts.CPU0.TLB:TLB_shootdowns 211340 ± 33% -91.7% 17475 ± 22% interrupts.CPU1.CAL:Function_call_interrupts 86057 ± 41% -75.1% 21460 ± 21% interrupts.CPU1.RES:Rescheduling_interrupts 975.25 ± 6% -23.7% 743.75 ± 7% interrupts.CPU1.TLB:TLB_shootdowns 254911 ± 55% -90.7% 23715 ± 43% interrupts.CPU10.CAL:Function_call_interrupts 105373 ± 30% -80.7% 20353 ± 14% interrupts.CPU10.RES:Rescheduling_interrupts 955.25 ± 4% -22.0% 744.75 ± 4% interrupts.CPU10.TLB:TLB_shootdowns 249461 ± 11% -93.8% 15439 ± 13% interrupts.CPU100.CAL:Function_call_interrupts 204149 ± 27% -91.0% 18319 ± 13% interrupts.CPU100.RES:Rescheduling_interrupts 930.00 ± 4% -18.6% 756.75 ± 9% interrupts.CPU100.TLB:TLB_shootdowns 259504 ± 9% -93.4% 17229 ± 17% interrupts.CPU101.CAL:Function_call_interrupts 204261 ± 24% -91.2% 18064 ± 6% interrupts.CPU101.RES:Rescheduling_interrupts 954.50 ± 5% -23.8% 727.00 ± 10% interrupts.CPU101.TLB:TLB_shootdowns 258375 ± 24% -94.2% 15030 ± 14% interrupts.CPU102.CAL:Function_call_interrupts 182795 ± 16% -90.0% 18358 ± 15% interrupts.CPU102.RES:Rescheduling_interrupts 916.00 ± 3% -20.4% 729.50 ± 6% interrupts.CPU102.TLB:TLB_shootdowns 291365 ± 33% -94.3% 16647 ± 26% interrupts.CPU103.CAL:Function_call_interrupts 173485 ± 10% -89.9% 17467 ± 19% interrupts.CPU103.RES:Rescheduling_interrupts 958.50 -26.6% 703.25 ± 7% interrupts.CPU103.TLB:TLB_shootdowns 179350 ± 33% -90.4% 17303 ± 21% interrupts.CPU11.CAL:Function_call_interrupts 114563 ± 29% -82.5% 20100 ± 23% interrupts.CPU11.RES:Rescheduling_interrupts 956.75 ± 2% -17.4% 790.75 ± 6% interrupts.CPU11.TLB:TLB_shootdowns 262516 ± 28% -92.6% 19306 ± 29% interrupts.CPU12.CAL:Function_call_interrupts 89238 ± 24% -76.0% 21412 ± 23% interrupts.CPU12.RES:Rescheduling_interrupts 938.75 ± 2% -19.8% 752.50 ± 3% interrupts.CPU12.TLB:TLB_shootdowns 243783 ± 17% -89.7% 25212 ± 55% interrupts.CPU13.CAL:Function_call_interrupts 96811 ± 24% -78.4% 20953 ± 14% interrupts.CPU13.RES:Rescheduling_interrupts 965.00 ± 3% -22.7% 746.00 ± 5% interrupts.CPU13.TLB:TLB_shootdowns 223957 ± 47% -90.6% 21081 ± 51% interrupts.CPU14.CAL:Function_call_interrupts 92346 ± 16% -76.2% 22002 ± 17% interrupts.CPU14.RES:Rescheduling_interrupts 916.75 ± 7% -17.8% 753.75 ± 5% interrupts.CPU14.TLB:TLB_shootdowns 238289 ± 23% -93.8% 14785 ± 36% interrupts.CPU15.CAL:Function_call_interrupts 94092 ± 23% -80.7% 18181 ± 11% interrupts.CPU15.RES:Rescheduling_interrupts 975.75 ± 2% -25.0% 732.25 ± 5% interrupts.CPU15.TLB:TLB_shootdowns 232503 ± 46% -92.2% 18043 ± 37% interrupts.CPU16.CAL:Function_call_interrupts 92258 ± 39% -77.3% 20901 ± 14% interrupts.CPU16.RES:Rescheduling_interrupts 970.75 -21.4% 762.75 ± 2% interrupts.CPU16.TLB:TLB_shootdowns 212775 ± 51% -93.4% 13990 ± 15% interrupts.CPU17.CAL:Function_call_interrupts 89631 ± 50% -77.5% 20194 ± 17% interrupts.CPU17.RES:Rescheduling_interrupts 980.50 ± 3% -22.4% 761.25 ± 5% interrupts.CPU17.TLB:TLB_shootdowns 402401 ± 33% -96.3% 15049 ± 8% interrupts.CPU18.CAL:Function_call_interrupts 94841 ± 34% -78.2% 20633 ± 14% interrupts.CPU18.RES:Rescheduling_interrupts 1008 ± 3% -24.8% 758.00 ± 5% interrupts.CPU18.TLB:TLB_shootdowns 214995 ± 21% -93.5% 14000 ± 15% interrupts.CPU19.CAL:Function_call_interrupts 98312 ± 27% -79.7% 19918 ± 13% interrupts.CPU19.RES:Rescheduling_interrupts 971.75 ± 2% -23.5% 743.75 ± 4% interrupts.CPU19.TLB:TLB_shootdowns 215367 ± 43% -93.4% 14320 ± 17% interrupts.CPU2.CAL:Function_call_interrupts 86227 ± 36% -79.8% 17380 ± 17% interrupts.CPU2.RES:Rescheduling_interrupts 1001 ± 4% -26.6% 734.75 ± 5% interrupts.CPU2.TLB:TLB_shootdowns 201442 ± 23% -92.4% 15238 ± 27% interrupts.CPU20.CAL:Function_call_interrupts 107574 ± 25% -81.5% 19865 ± 18% interrupts.CPU20.RES:Rescheduling_interrupts 968.75 ± 2% -23.8% 738.50 interrupts.CPU20.TLB:TLB_shootdowns 229077 ± 36% -93.9% 13909 ± 18% interrupts.CPU21.CAL:Function_call_interrupts 84901 ± 27% -78.5% 18250 ± 7% interrupts.CPU21.RES:Rescheduling_interrupts 990.50 ± 4% -25.6% 737.25 ± 5% interrupts.CPU21.TLB:TLB_shootdowns 290748 ± 28% -94.6% 15580 ± 30% interrupts.CPU22.CAL:Function_call_interrupts 95230 ± 22% -78.4% 20559 ± 14% interrupts.CPU22.RES:Rescheduling_interrupts 958.75 ± 3% -22.1% 746.75 ± 3% interrupts.CPU22.TLB:TLB_shootdowns 237541 ± 25% -92.2% 18579 ± 48% interrupts.CPU23.CAL:Function_call_interrupts 102695 ± 43% -80.5% 20071 ± 24% interrupts.CPU23.RES:Rescheduling_interrupts 971.25 ± 3% -22.7% 751.00 ± 3% interrupts.CPU23.TLB:TLB_shootdowns 297244 ± 50% -94.1% 17649 ± 24% interrupts.CPU24.CAL:Function_call_interrupts 94620 ± 32% -80.1% 18865 ± 12% interrupts.CPU24.RES:Rescheduling_interrupts 978.75 ± 3% -26.0% 724.50 ± 4% interrupts.CPU24.TLB:TLB_shootdowns 253790 ± 22% -94.1% 15076 ± 38% interrupts.CPU25.CAL:Function_call_interrupts 94250 ± 20% -79.8% 19007 ± 16% interrupts.CPU25.RES:Rescheduling_interrupts 992.50 ± 2% -22.8% 765.75 ± 4% interrupts.CPU25.TLB:TLB_shootdowns 205152 ± 5% -90.1% 20339 ± 7% interrupts.CPU26.CAL:Function_call_interrupts 170083 ± 10% -87.8% 20741 ± 18% interrupts.CPU26.RES:Rescheduling_interrupts 1002 ± 4% -23.7% 765.25 ± 6% interrupts.CPU26.TLB:TLB_shootdowns 230776 ± 9% -90.7% 21504 ± 22% interrupts.CPU27.CAL:Function_call_interrupts 173305 ± 10% -87.7% 21271 ± 23% interrupts.CPU27.RES:Rescheduling_interrupts 978.00 ± 3% -19.8% 784.00 ± 8% interrupts.CPU27.TLB:TLB_shootdowns 262539 ± 42% -92.4% 19830 ± 19% interrupts.CPU28.CAL:Function_call_interrupts 147713 ± 21% -86.3% 20248 ± 18% interrupts.CPU28.RES:Rescheduling_interrupts 1013 -24.6% 764.75 ± 8% interrupts.CPU28.TLB:TLB_shootdowns 286525 ± 46% -93.3% 19228 ± 20% interrupts.CPU29.CAL:Function_call_interrupts 165484 ± 35% -87.5% 20618 ± 18% interrupts.CPU29.RES:Rescheduling_interrupts 981.50 ± 3% -19.7% 787.75 ± 10% interrupts.CPU29.TLB:TLB_shootdowns 180881 ± 33% -90.6% 16928 ± 20% interrupts.CPU3.CAL:Function_call_interrupts 103722 ± 25% -83.2% 17437 ± 14% interrupts.CPU3.RES:Rescheduling_interrupts 974.25 ± 3% -19.8% 781.25 ± 6% interrupts.CPU3.TLB:TLB_shootdowns 283317 ± 38% -94.2% 16329 ± 13% interrupts.CPU30.CAL:Function_call_interrupts 148002 ± 14% -87.7% 18221 ± 13% interrupts.CPU30.RES:Rescheduling_interrupts 973.00 ± 2% -22.0% 758.75 ± 5% interrupts.CPU30.TLB:TLB_shootdowns 241331 ± 11% -91.3% 21053 ± 25% interrupts.CPU31.CAL:Function_call_interrupts 180308 ± 6% -88.8% 20187 ± 10% interrupts.CPU31.RES:Rescheduling_interrupts 996.25 ± 3% -22.1% 775.75 ± 9% interrupts.CPU31.TLB:TLB_shootdowns 659.50 ±108% +688.7% 5201 ± 71% interrupts.CPU32.40:PCI-MSI.67633155-edge.eth0-TxRx-2 244508 ± 18% -91.3% 21297 ± 26% interrupts.CPU32.CAL:Function_call_interrupts 161645 ± 15% -88.4% 18762 ± 20% interrupts.CPU32.RES:Rescheduling_interrupts 972.25 ± 3% -17.7% 800.25 ± 6% interrupts.CPU32.TLB:TLB_shootdowns 270008 ± 31% -92.3% 20768 ± 25% interrupts.CPU33.CAL:Function_call_interrupts 154188 ± 12% -86.1% 21419 ± 19% interrupts.CPU33.RES:Rescheduling_interrupts 987.50 ± 3% -21.1% 779.00 ± 6% interrupts.CPU33.TLB:TLB_shootdowns 231024 ± 31% -91.2% 20252 ± 33% interrupts.CPU34.CAL:Function_call_interrupts 168487 ± 16% -88.7% 18989 ± 20% interrupts.CPU34.RES:Rescheduling_interrupts 997.50 ± 4% -23.4% 764.50 ± 3% interrupts.CPU34.TLB:TLB_shootdowns 233104 ± 24% -92.6% 17182 ± 18% interrupts.CPU35.CAL:Function_call_interrupts 183698 ± 18% -89.8% 18677 ± 18% interrupts.CPU35.RES:Rescheduling_interrupts 999.50 ± 4% -21.8% 781.50 ± 9% interrupts.CPU35.TLB:TLB_shootdowns 215073 ± 11% -91.4% 18523 ± 16% interrupts.CPU36.CAL:Function_call_interrupts 153732 ± 18% -89.2% 16659 ± 5% interrupts.CPU36.RES:Rescheduling_interrupts 979.50 ± 2% -19.7% 786.50 ± 10% interrupts.CPU36.TLB:TLB_shootdowns 217389 ± 25% -90.9% 19719 ± 30% interrupts.CPU37.CAL:Function_call_interrupts 174628 ± 20% -89.0% 19148 ± 18% interrupts.CPU37.RES:Rescheduling_interrupts 990.75 ± 3% -20.7% 786.00 ± 7% interrupts.CPU37.TLB:TLB_shootdowns 311522 ± 55% -91.9% 25347 ± 45% interrupts.CPU38.CAL:Function_call_interrupts 161443 ± 7% -87.5% 20216 ± 22% interrupts.CPU38.RES:Rescheduling_interrupts 975.25 ± 2% -19.6% 784.50 ± 8% interrupts.CPU38.TLB:TLB_shootdowns 314470 ± 35% -94.1% 18708 ± 20% interrupts.CPU39.CAL:Function_call_interrupts 139101 ± 26% -87.0% 18130 ± 12% interrupts.CPU39.RES:Rescheduling_interrupts 991.25 ± 2% -18.5% 808.25 ± 9% interrupts.CPU39.TLB:TLB_shootdowns 175190 ± 34% -90.3% 16958 ± 19% interrupts.CPU4.CAL:Function_call_interrupts 89945 ± 33% -81.8% 16380 ± 22% interrupts.CPU4.RES:Rescheduling_interrupts 964.75 ± 3% -12.5% 844.50 ± 5% interrupts.CPU4.TLB:TLB_shootdowns 297916 ± 25% -93.5% 19294 ± 29% interrupts.CPU40.CAL:Function_call_interrupts 179694 ± 7% -89.3% 19231 ± 14% interrupts.CPU40.RES:Rescheduling_interrupts 964.75 ± 2% -21.0% 762.50 ± 6% interrupts.CPU40.TLB:TLB_shootdowns 274482 ± 32% -91.0% 24643 ± 12% interrupts.CPU41.CAL:Function_call_interrupts 196924 ± 3% -88.7% 22200 ± 14% interrupts.CPU41.RES:Rescheduling_interrupts 983.50 ± 2% -22.0% 767.50 ± 8% interrupts.CPU41.TLB:TLB_shootdowns 229609 ± 25% -88.8% 25708 ± 33% interrupts.CPU42.CAL:Function_call_interrupts 154570 ± 11% -86.5% 20869 ± 19% interrupts.CPU42.RES:Rescheduling_interrupts 979.00 ± 6% -22.0% 763.50 ± 8% interrupts.CPU42.TLB:TLB_shootdowns 202307 ± 8% -88.5% 23288 ± 32% interrupts.CPU43.CAL:Function_call_interrupts 175707 ± 19% -87.8% 21465 ± 13% interrupts.CPU43.RES:Rescheduling_interrupts 995.75 ± 3% -21.3% 783.25 ± 3% interrupts.CPU43.TLB:TLB_shootdowns 234634 ± 22% -89.8% 23836 ± 30% interrupts.CPU44.CAL:Function_call_interrupts 167024 ± 7% -87.8% 20416 ± 13% interrupts.CPU44.RES:Rescheduling_interrupts 975.00 ± 2% -20.9% 771.50 ± 8% interrupts.CPU44.TLB:TLB_shootdowns 261681 ± 15% -91.2% 23004 ± 16% interrupts.CPU45.CAL:Function_call_interrupts 160884 ± 14% -87.1% 20679 ± 13% interrupts.CPU45.RES:Rescheduling_interrupts 963.50 ± 4% -18.6% 784.25 ± 10% interrupts.CPU45.TLB:TLB_shootdowns 241381 ± 23% -89.2% 25977 ± 36% interrupts.CPU46.CAL:Function_call_interrupts 187691 ± 24% -87.6% 23222 ± 9% interrupts.CPU46.RES:Rescheduling_interrupts 998.25 -23.5% 763.25 ± 4% interrupts.CPU46.TLB:TLB_shootdowns 196129 ± 9% -86.3% 26786 ± 44% interrupts.CPU47.CAL:Function_call_interrupts 171732 ± 18% -88.1% 20448 ± 15% interrupts.CPU47.RES:Rescheduling_interrupts 952.75 -17.6% 785.25 ± 4% interrupts.CPU47.TLB:TLB_shootdowns 220500 ± 5% -88.2% 25940 ± 44% interrupts.CPU48.CAL:Function_call_interrupts 184654 ± 22% -88.1% 21930 ± 20% interrupts.CPU48.RES:Rescheduling_interrupts 992.25 ± 4% -21.7% 777.00 ± 6% interrupts.CPU48.TLB:TLB_shootdowns 218929 ± 12% -89.5% 23004 ± 17% interrupts.CPU49.CAL:Function_call_interrupts 177347 ± 22% -88.1% 21118 ± 10% interrupts.CPU49.RES:Rescheduling_interrupts 951.75 ± 5% -17.5% 785.00 ± 6% interrupts.CPU49.TLB:TLB_shootdowns 194819 ± 40% -92.4% 14902 ± 18% interrupts.CPU5.CAL:Function_call_interrupts 79044 ± 32% -75.2% 19640 ± 19% interrupts.CPU5.RES:Rescheduling_interrupts 980.75 ± 5% -24.7% 738.50 ± 7% interrupts.CPU5.TLB:TLB_shootdowns 223002 ± 20% -88.2% 26348 ± 45% interrupts.CPU50.CAL:Function_call_interrupts 160668 ± 23% -86.2% 22177 ± 19% interrupts.CPU50.RES:Rescheduling_interrupts 1004 ± 5% -23.9% 764.50 ± 6% interrupts.CPU50.TLB:TLB_shootdowns 309989 ± 32% -91.0% 27839 ± 29% interrupts.CPU51.CAL:Function_call_interrupts 170499 ± 19% -87.4% 21434 ± 8% interrupts.CPU51.RES:Rescheduling_interrupts 988.00 ± 4% -21.2% 778.75 ± 6% interrupts.CPU51.TLB:TLB_shootdowns 203769 ± 18% -91.2% 17893 ± 29% interrupts.CPU52.CAL:Function_call_interrupts 100629 ± 43% -77.5% 22610 ± 52% interrupts.CPU52.RES:Rescheduling_interrupts 994.50 ± 5% -24.5% 750.50 ± 3% interrupts.CPU52.TLB:TLB_shootdowns 268555 ± 46% -93.6% 17244 ± 20% interrupts.CPU53.CAL:Function_call_interrupts 84390 ± 27% -73.6% 22265 ± 25% interrupts.CPU53.RES:Rescheduling_interrupts 948.50 ± 2% -22.7% 732.75 ± 4% interrupts.CPU53.TLB:TLB_shootdowns 243003 ± 32% -92.3% 18730 ± 33% interrupts.CPU54.CAL:Function_call_interrupts 88915 ± 40% -79.0% 18628 ± 14% interrupts.CPU54.RES:Rescheduling_interrupts 939.25 ± 3% -17.2% 777.25 ± 6% interrupts.CPU54.TLB:TLB_shootdowns 202800 ± 23% -90.7% 18836 ± 14% interrupts.CPU55.CAL:Function_call_interrupts 109024 ± 22% -81.4% 20276 ± 16% interrupts.CPU55.RES:Rescheduling_interrupts 947.25 ± 3% -19.7% 761.00 ± 4% interrupts.CPU55.TLB:TLB_shootdowns 224205 ± 41% -87.8% 27356 ± 53% interrupts.CPU56.CAL:Function_call_interrupts 103035 ± 34% -79.4% 21240 ± 13% interrupts.CPU56.RES:Rescheduling_interrupts 937.50 ± 3% -19.7% 753.25 ± 6% interrupts.CPU56.TLB:TLB_shootdowns 239213 ± 26% -92.9% 16972 ± 21% interrupts.CPU57.CAL:Function_call_interrupts 87777 ± 25% -77.0% 20174 ± 21% interrupts.CPU57.RES:Rescheduling_interrupts 978.00 ± 2% -22.9% 754.00 ± 2% interrupts.CPU57.TLB:TLB_shootdowns 297549 ± 46% -93.1% 20527 ± 29% interrupts.CPU58.CAL:Function_call_interrupts 85880 ± 29% -65.3% 29818 ± 42% interrupts.CPU58.RES:Rescheduling_interrupts 956.50 ± 4% -24.0% 726.50 ± 5% interrupts.CPU58.TLB:TLB_shootdowns 174057 ± 18% -85.9% 24531 ± 44% interrupts.CPU59.CAL:Function_call_interrupts 115811 ± 43% -81.8% 21024 ± 23% interrupts.CPU59.RES:Rescheduling_interrupts 960.25 ± 2% -21.3% 755.75 ± 9% interrupts.CPU59.TLB:TLB_shootdowns 224175 ± 41% -94.2% 13090 ± 20% interrupts.CPU6.CAL:Function_call_interrupts 88182 ± 37% -79.2% 18317 ± 34% interrupts.CPU6.RES:Rescheduling_interrupts 939.75 ± 5% -18.6% 764.75 ± 6% interrupts.CPU6.TLB:TLB_shootdowns 181880 ± 33% -87.2% 23212 ± 45% interrupts.CPU60.CAL:Function_call_interrupts 92656 ± 33% -78.0% 20396 ± 13% interrupts.CPU60.RES:Rescheduling_interrupts 947.50 ± 3% -19.2% 765.75 ± 8% interrupts.CPU60.TLB:TLB_shootdowns 204760 ± 19% -91.4% 17653 ± 8% interrupts.CPU61.CAL:Function_call_interrupts 92688 ± 21% -79.5% 19011 ± 11% interrupts.CPU61.RES:Rescheduling_interrupts 913.75 ± 3% -16.4% 763.50 interrupts.CPU61.TLB:TLB_shootdowns 220484 ± 44% -91.7% 18307 ± 29% interrupts.CPU62.CAL:Function_call_interrupts 100766 ± 38% -79.3% 20865 ± 15% interrupts.CPU62.RES:Rescheduling_interrupts 951.00 ± 4% -19.7% 763.50 ± 4% interrupts.CPU62.TLB:TLB_shootdowns 157447 ± 21% -89.1% 17150 ± 23% interrupts.CPU63.CAL:Function_call_interrupts 111514 ± 27% -82.8% 19217 ± 24% interrupts.CPU63.RES:Rescheduling_interrupts 946.50 ± 4% -24.4% 715.25 ± 4% interrupts.CPU63.TLB:TLB_shootdowns 211518 ± 9% -92.6% 15749 ± 17% interrupts.CPU64.CAL:Function_call_interrupts 91890 ± 26% -78.2% 19991 ± 20% interrupts.CPU64.RES:Rescheduling_interrupts 972.25 -23.8% 740.75 ± 4% interrupts.CPU64.TLB:TLB_shootdowns 191458 ± 36% -89.0% 21148 ± 29% interrupts.CPU65.CAL:Function_call_interrupts 93824 ± 28% -78.6% 20106 ± 18% interrupts.CPU65.RES:Rescheduling_interrupts 940.25 ± 5% -21.7% 736.00 ± 6% interrupts.CPU65.TLB:TLB_shootdowns 175909 ± 28% -88.4% 20457 ± 32% interrupts.CPU66.CAL:Function_call_interrupts 97276 ± 26% -78.4% 21043 ± 29% interrupts.CPU66.RES:Rescheduling_interrupts 955.25 ± 4% -20.5% 759.50 ± 4% interrupts.CPU66.TLB:TLB_shootdowns 147273 ± 16% -85.2% 21858 ± 57% interrupts.CPU67.CAL:Function_call_interrupts 95512 ± 25% -80.5% 18580 ± 14% interrupts.CPU67.RES:Rescheduling_interrupts 929.00 ± 3% -17.6% 765.50 ± 5% interrupts.CPU67.TLB:TLB_shootdowns 145708 ± 41% -87.0% 18998 ± 33% interrupts.CPU68.CAL:Function_call_interrupts 99859 ± 44% -77.6% 22326 ± 25% interrupts.CPU68.RES:Rescheduling_interrupts 934.75 ± 3% -19.7% 750.75 ± 2% interrupts.CPU68.TLB:TLB_shootdowns 199682 ± 39% -91.4% 17253 ± 27% interrupts.CPU69.CAL:Function_call_interrupts 91410 ± 44% -77.2% 20819 ± 23% interrupts.CPU69.RES:Rescheduling_interrupts 925.50 ± 2% -20.4% 737.00 ± 5% interrupts.CPU69.TLB:TLB_shootdowns 222715 ± 50% -92.1% 17636 ± 32% interrupts.CPU7.CAL:Function_call_interrupts 95265 ± 33% -79.6% 19454 ± 19% interrupts.CPU7.RES:Rescheduling_interrupts 952.75 ± 4% -19.7% 765.00 ± 3% interrupts.CPU7.TLB:TLB_shootdowns 142175 ± 20% -87.9% 17144 ± 2% interrupts.CPU70.CAL:Function_call_interrupts 94122 ± 38% -79.2% 19614 ± 14% interrupts.CPU70.RES:Rescheduling_interrupts 913.50 ± 5% -18.2% 747.25 ± 6% interrupts.CPU70.TLB:TLB_shootdowns 183802 ± 17% -87.6% 22799 ± 48% interrupts.CPU71.CAL:Function_call_interrupts 94544 ± 20% -78.1% 20725 ± 22% interrupts.CPU71.RES:Rescheduling_interrupts 226205 ± 48% -88.4% 26142 ± 52% interrupts.CPU72.CAL:Function_call_interrupts 103772 ± 24% -80.4% 20331 ± 27% interrupts.CPU72.RES:Rescheduling_interrupts 926.00 -19.0% 750.00 ± 7% interrupts.CPU72.TLB:TLB_shootdowns 201882 ± 40% -89.9% 20366 ± 38% interrupts.CPU73.CAL:Function_call_interrupts 88094 ± 30% -77.4% 19940 ± 20% interrupts.CPU73.RES:Rescheduling_interrupts 915.50 ± 6% -14.9% 779.00 ± 4% interrupts.CPU73.TLB:TLB_shootdowns 143309 ± 25% -87.9% 17359 ± 36% interrupts.CPU74.CAL:Function_call_interrupts 109788 ± 26% -82.2% 19581 ± 19% interrupts.CPU74.RES:Rescheduling_interrupts 935.00 ± 3% -19.0% 757.00 ± 4% interrupts.CPU74.TLB:TLB_shootdowns 150353 ± 34% -90.0% 15069 ± 31% interrupts.CPU75.CAL:Function_call_interrupts 92423 ± 32% -78.9% 19478 ± 23% interrupts.CPU75.RES:Rescheduling_interrupts 924.50 ± 2% -19.2% 746.75 ± 4% interrupts.CPU75.TLB:TLB_shootdowns 121045 ± 21% -84.9% 18221 ± 36% interrupts.CPU76.CAL:Function_call_interrupts 94876 ± 25% -79.3% 19631 ± 18% interrupts.CPU76.RES:Rescheduling_interrupts 954.50 ± 2% -19.1% 771.75 ± 3% interrupts.CPU76.TLB:TLB_shootdowns 137345 ± 30% -86.8% 18197 ± 63% interrupts.CPU77.CAL:Function_call_interrupts 104707 ± 35% -81.9% 19003 ± 22% interrupts.CPU77.RES:Rescheduling_interrupts 917.00 ± 4% -18.6% 746.75 ± 2% interrupts.CPU77.TLB:TLB_shootdowns 193947 ± 16% -87.7% 23840 ± 22% interrupts.CPU78.CAL:Function_call_interrupts 173906 ± 14% -88.4% 20248 ± 18% interrupts.CPU78.RES:Rescheduling_interrupts 961.00 ± 2% -21.3% 756.00 ± 7% interrupts.CPU78.TLB:TLB_shootdowns 209991 ± 19% -88.3% 24613 ± 29% interrupts.CPU79.CAL:Function_call_interrupts 168981 ± 12% -88.1% 20098 ± 18% interrupts.CPU79.RES:Rescheduling_interrupts 966.75 ± 3% -21.7% 757.25 ± 9% interrupts.CPU79.TLB:TLB_shootdowns 156899 ± 32% -88.7% 17697 ± 37% interrupts.CPU8.CAL:Function_call_interrupts 102075 ± 31% -81.5% 18874 ± 5% interrupts.CPU8.RES:Rescheduling_interrupts 968.00 ± 5% -22.7% 748.25 ± 2% interrupts.CPU8.TLB:TLB_shootdowns 220471 ± 22% -85.9% 31011 ± 45% interrupts.CPU80.CAL:Function_call_interrupts 144946 ± 16% -85.6% 20909 ± 23% interrupts.CPU80.RES:Rescheduling_interrupts 928.50 ± 4% -15.5% 784.25 ± 7% interrupts.CPU80.TLB:TLB_shootdowns 216800 ± 13% -90.5% 20532 ± 10% interrupts.CPU81.CAL:Function_call_interrupts 143282 ± 20% -86.1% 19864 ± 14% interrupts.CPU81.RES:Rescheduling_interrupts 961.50 ± 4% -21.8% 751.50 ± 8% interrupts.CPU81.TLB:TLB_shootdowns 202116 ± 25% -89.6% 20966 ± 27% interrupts.CPU82.CAL:Function_call_interrupts 146866 ± 12% -87.6% 18262 ± 14% interrupts.CPU82.RES:Rescheduling_interrupts 941.00 -19.8% 754.25 ± 6% interrupts.CPU82.TLB:TLB_shootdowns 234326 ± 7% -92.2% 18201 ± 16% interrupts.CPU83.CAL:Function_call_interrupts 182247 ± 9% -89.8% 18610 ± 23% interrupts.CPU83.RES:Rescheduling_interrupts 944.50 ± 4% -20.8% 748.25 ± 8% interrupts.CPU83.TLB:TLB_shootdowns 249155 ± 14% -92.0% 19999 ± 23% interrupts.CPU84.CAL:Function_call_interrupts 170031 ± 12% -83.5% 28060 ± 58% interrupts.CPU84.RES:Rescheduling_interrupts 939.00 ± 2% -22.4% 728.25 ± 8% interrupts.CPU84.TLB:TLB_shootdowns 204078 ± 27% -87.6% 25298 ± 20% interrupts.CPU85.CAL:Function_call_interrupts 155195 ± 21% -87.1% 19997 ± 6% interrupts.CPU85.RES:Rescheduling_interrupts 926.25 ± 4% -19.4% 746.25 ± 7% interrupts.CPU85.TLB:TLB_shootdowns 269649 ± 21% -92.2% 21025 ± 24% interrupts.CPU86.CAL:Function_call_interrupts 172678 ± 20% -87.9% 20899 ± 19% interrupts.CPU86.RES:Rescheduling_interrupts 945.50 ± 3% -19.4% 762.25 ± 9% interrupts.CPU86.TLB:TLB_shootdowns 261448 ± 40% -92.4% 19996 ± 22% interrupts.CPU87.CAL:Function_call_interrupts 186423 ± 22% -90.2% 18237 ± 16% interrupts.CPU87.RES:Rescheduling_interrupts 933.50 ± 2% -18.1% 764.25 ± 8% interrupts.CPU87.TLB:TLB_shootdowns 347606 ± 48% -94.2% 20143 ± 23% interrupts.CPU88.CAL:Function_call_interrupts 170646 ± 11% -86.0% 23921 ± 47% interrupts.CPU88.RES:Rescheduling_interrupts 969.50 ± 6% -21.8% 757.75 ± 11% interrupts.CPU88.TLB:TLB_shootdowns 256935 ± 31% -93.5% 16819 ± 22% interrupts.CPU89.CAL:Function_call_interrupts 177665 ± 17% -89.0% 19552 ± 26% interrupts.CPU89.RES:Rescheduling_interrupts 963.00 -22.6% 745.00 ± 5% interrupts.CPU89.TLB:TLB_shootdowns 181023 ± 32% -89.9% 18246 ± 28% interrupts.CPU9.CAL:Function_call_interrupts 109042 ± 36% -82.2% 19398 ± 5% interrupts.CPU9.RES:Rescheduling_interrupts 968.50 ± 4% -23.1% 744.50 ± 7% interrupts.CPU9.TLB:TLB_shootdowns 264579 ± 24% -90.7% 24508 ± 24% interrupts.CPU90.CAL:Function_call_interrupts 183053 ± 12% -88.8% 20547 ± 17% interrupts.CPU90.RES:Rescheduling_interrupts 977.50 ± 3% -23.4% 749.25 ± 8% interrupts.CPU90.TLB:TLB_shootdowns 223839 ± 35% -90.8% 20586 ± 17% interrupts.CPU91.CAL:Function_call_interrupts 144548 ± 14% -87.1% 18586 ± 13% interrupts.CPU91.RES:Rescheduling_interrupts 961.00 ± 5% -24.5% 726.00 ± 8% interrupts.CPU91.TLB:TLB_shootdowns 227207 ± 29% -90.4% 21839 ± 21% interrupts.CPU92.CAL:Function_call_interrupts 175733 ± 14% -88.6% 20048 ± 17% interrupts.CPU92.RES:Rescheduling_interrupts 944.00 ± 3% -20.9% 746.25 ± 8% interrupts.CPU92.TLB:TLB_shootdowns 240500 ± 10% -91.8% 19716 ± 23% interrupts.CPU93.CAL:Function_call_interrupts 190796 ± 12% -89.9% 19239 ± 21% interrupts.CPU93.RES:Rescheduling_interrupts 944.75 ± 2% -20.0% 755.75 ± 7% interrupts.CPU93.TLB:TLB_shootdowns 259259 ± 55% -92.8% 18649 ± 42% interrupts.CPU94.CAL:Function_call_interrupts 154375 ± 15% -87.4% 19425 ± 25% interrupts.CPU94.RES:Rescheduling_interrupts 944.25 ± 3% -16.6% 787.25 ± 7% interrupts.CPU94.TLB:TLB_shootdowns 254216 ± 16% -91.5% 21545 ± 24% interrupts.CPU95.CAL:Function_call_interrupts 186091 ± 17% -89.2% 20135 ± 20% interrupts.CPU95.RES:Rescheduling_interrupts 938.75 ± 2% -19.4% 756.75 ± 10% interrupts.CPU95.TLB:TLB_shootdowns 266996 ± 10% -93.0% 18642 ± 19% interrupts.CPU96.CAL:Function_call_interrupts 172889 ± 16% -89.7% 17801 ± 14% interrupts.CPU96.RES:Rescheduling_interrupts 951.00 ± 2% -20.4% 757.25 ± 14% interrupts.CPU96.TLB:TLB_shootdowns 193323 ± 21% -89.9% 19611 ± 27% interrupts.CPU97.CAL:Function_call_interrupts 160260 ± 15% -86.7% 21327 ± 17% interrupts.CPU97.RES:Rescheduling_interrupts 947.00 ± 4% -19.1% 766.50 ± 5% interrupts.CPU97.TLB:TLB_shootdowns 261735 ± 26% -90.3% 25274 ± 31% interrupts.CPU98.CAL:Function_call_interrupts 196196 ± 25% -89.8% 19959 ± 15% interrupts.CPU98.RES:Rescheduling_interrupts 958.50 ± 2% -21.6% 751.75 ± 10% interrupts.CPU98.TLB:TLB_shootdowns 186979 ± 12% -89.5% 19580 ± 22% interrupts.CPU99.CAL:Function_call_interrupts 171722 ± 21% -88.3% 20089 ± 18% interrupts.CPU99.RES:Rescheduling_interrupts 961.75 ± 4% -24.8% 723.25 ± 7% interrupts.CPU99.TLB:TLB_shootdowns 13865203 ± 5% -84.9% 2096456 ± 6% interrupts.RES:Rescheduling_interrupts 99890 ± 2% -20.9% 79026 ± 5% interrupts.TLB:TLB_shootdowns will-it-scale.per_thread_ops 170000 +------------------------------------------------------------------+ | .+.+. .+..+.+.+.+.. .+. .+. .+. +.+.+..+ + .+.+.| 160000 |.+.+. + +.+ +..+ + +.. + +.+. | 150000 |-+ + | | | 140000 |-+ | | | 130000 |-+ | | | 120000 |-+ | 110000 |-+ | | O O | 100000 |-O O O O O O O O O O O O O O O O O O O O O | | O | 90000 +------------------------------------------------------------------+ will-it-scale.workload 9e+06 +-----------------------------------------------------------------+ | .+. .+. .+ .+. .+. .+..+. .+. .+. | 8.5e+06 |.+ .+.+.+.+.+. + +.+. + .+ +. +. .+ + +.+. +.| 8e+06 |-+.+. + + | | | 7.5e+06 |-+ | 7e+06 |-+ | | | 6.5e+06 |-+ | 6e+06 |-+ | | | 5.5e+06 |-+ O O O O O O O O O O | 5e+06 |-O O O O O O O O O O O O O | | O | 4.5e+06 +-----------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

4 months, 2 weeks

3
7
0 / 0

[drm/i915] 55ea1cb178: phoronix-test-suite.gluxmark.Fullscreen.1280x1024.Fragment.total_score 44.1% improvement

by kernel test robot

Greeting, FYI, we noticed a 44.1% improvement of phoronix-test-suite.gluxmark.Fullscreen.1280x1024.Fragment.total_score due to commit: commit: 55ea1cb178ef85677b35126685fbf5d8d649b62e ("drm/i915: Enable async flips in i915") git://anongit.freedesktop.org/drm-intel drm-intel-next-queued in testcase: phoronix-test-suite on test machine: 12 threads Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz with 8G memory with following parameters: need_x: true test: gluxmark-1.1.1 option_a: Fullscreen option_b: Fragment cpufreq_governor: performance ucode: 0xd6 test-description: The Phoronix Test Suite is the most comprehensive testing and benchmarking platform available that provides an extensible framework for which new tests can be easily added. test-url: http://www.phoronix-test-suite.com/ In addition to that, the commit also has significant impact on the following tests: +------------------+-------------------------------------------------------------------------------------------------------------------------------------------+ | testcase: change | phoronix-test-suite: phoronix-test-suite.supertuxkart.1280x1024.Fullscreen.Ultimate.1.CandelaCity.frames_per_second 2.4% improvement | | test machine | 12 threads Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz with 8G memory | | test parameters | cpufreq_governor=performance | | | need_x=true | | | option_a=Fullscreen | | | option_b=Ultimate | | | option_c=1 | | | option_d=Candela City [Approximately 400k triangles; advanced graphics] | | | test=supertuxkart-1.5.2 | | | ucode=0xd6 | +------------------+-------------------------------------------------------------------------------------------------------------------------------------------+ | testcase: change | phoronix-test-suite: phoronix-test-suite.gputest.Plot3D.1280x1024.Windowed.points 5.9% regression | | test machine | 12 threads Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz with 8G memory | | test parameters | cpufreq_governor=performance | | | need_x=true | | | option_a=Plot3D | | | option_b=Windowed | | | test=gputest-1.3.2 | | | ucode=0xd6 | +------------------+-------------------------------------------------------------------------------------------------------------------------------------------+ | testcase: change | phoronix-test-suite: phoronix-test-suite.supertuxkart.1280x1024.Fullscreen.Basic.1.GranParadisoIsland.frames_per_second -12.7% regression | | test machine | 12 threads Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz with 8G memory | | test parameters | cpufreq_governor=performance | | | need_x=true | | | option_a=Fullscreen | | | option_b=Basic | | | option_c=1 | | | option_d=Gran Paradiso Island [Approximately 275k triangles; advanced graphics] | | | test=supertuxkart-1.5.2 | | | ucode=0xd6 | +------------------+-------------------------------------------------------------------------------------------------------------------------------------------+ Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/need_x/option_a/option_b/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-8.3/true/Fullscreen/Fragment/debian-x86_64-phoronix/lkp-cfl-d1/gluxmark-1.1.1/phoronix-test-suite/0xd6 commit: 6914c9685e ("Documentation/gpu: Add asynchronous flip documentation for i915") 55ea1cb178 ("drm/i915: Enable async flips in i915") 6914c9685e0a2376 55ea1cb178ef85677b35126685f ---------------- --------------------------- %stddev %change %stddev \ | \ 497.24 +44.1% 716.57 phoronix-test-suite.gluxmark.Fullscreen.1280x1024.Fragment.total_score 1487 ± 3% +1916.9% 29997 ± 6% phoronix-test-suite.time.involuntary_context_switches 29229 -15.6% 24658 phoronix-test-suite.time.major_page_faults 28.00 -25.0% 21.00 phoronix-test-suite.time.percent_of_cpu_this_job_got 25.70 ± 3% -4.9% 24.45 phoronix-test-suite.time.system_time 72.73 -33.7% 48.20 phoronix-test-suite.time.user_time 186297 +131.1% 430577 phoronix-test-suite.time.voluntary_context_switches phoronix-test-suite.gluxmark.Fullscreen.1280x1024.Fragment.total_score 800 +---------------------------------------------------------------------+ | O O O O O O OO O O OO O O OO O O OO O O OO O O O O OO O O | 700 |-+ O O O O O OO | 600 |-+ | | | 500 |.+.+ +.+.++.+.+ +.+.+.+ +.+.++.+.+.++.+.+.++.+.+.++.+.+.++.+ | | : : : : : : | 400 |-+ : : : : : : | | : : : : : : | 300 |-+ : : : : : : | 200 |-+ :: :: :: | | :: :: :: | 100 |-+ : : : | | : : : | 0 +---------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-cfl-d1: 12 threads Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz with 8G memory ========================================================================================= compiler/cpufreq_governor/kconfig/need_x/option_a/option_b/option_c/option_d/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-8.3/true/Fullscreen/Ultimate/1/Candela City [Approximately 400k triangles; advanced graphics]/debian-x86_64-phoronix/lkp-cfl-d1/supertuxkart-1.5.2/phoronix-test-suite/0xd6 commit: 6914c9685e ("Documentation/gpu: Add asynchronous flip documentation for i915") 55ea1cb178 ("drm/i915: Enable async flips in i915") 6914c9685e0a2376 55ea1cb178ef85677b35126685f ---------------- --------------------------- %stddev %change %stddev \ | \ 16.54 +2.4% 16.93 phoronix-test-suite.supertuxkart.1280x1024.Fullscreen.Ultimate.1.CandelaCity.frames_per_second 2734 ± 3% -6.0% 2569 phoronix-test-suite.time.involuntary_context_switches 43.97 -4.2% 42.13 phoronix-test-suite.time.user_time 296118 -1.1% 292950 phoronix-test-suite.time.voluntary_context_switches *************************************************************************************************** lkp-cfl-d1: 12 threads Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz with 8G memory ========================================================================================= compiler/cpufreq_governor/kconfig/need_x/option_a/option_b/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-8.3/true/Plot3D/Windowed/debian-x86_64-phoronix/lkp-cfl-d1/gputest-1.3.2/phoronix-test-suite/0xd6 commit: 6914c9685e ("Documentation/gpu: Add asynchronous flip documentation for i915") 55ea1cb178 ("drm/i915: Enable async flips in i915") 6914c9685e0a2376 55ea1cb178ef85677b35126685f ---------------- --------------------------- %stddev %change %stddev \ | \ 8937 +5.9% 9466 phoronix-test-suite.gputest.Plot3D.1280x1024.Windowed.points 750.25 ± 14% +25.7% 943.00 ± 5% phoronix-test-suite.time.involuntary_context_switches 17310 -3.6% 16684 phoronix-test-suite.time.major_page_faults 10.00 +20.0% 12.00 ± 5% phoronix-test-suite.time.percent_of_cpu_this_job_got 13.36 +24.2% 16.59 ± 5% phoronix-test-suite.time.user_time 62842 +89.3% 118984 phoronix-test-suite.time.voluntary_context_switches *************************************************************************************************** lkp-cfl-d1: 12 threads Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz with 8G memory ========================================================================================= compiler/cpufreq_governor/kconfig/need_x/option_a/option_b/option_c/option_d/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-8.3/true/Fullscreen/Basic/1/Gran Paradiso Island [Approximately 275k triangles; advanced graphics]/debian-x86_64-phoronix/lkp-cfl-d1/supertuxkart-1.5.2/phoronix-test-suite/0xd6 commit: 6914c9685e ("Documentation/gpu: Add asynchronous flip documentation for i915") 55ea1cb178 ("drm/i915: Enable async flips in i915") 6914c9685e0a2376 55ea1cb178ef85677b35126685f ---------------- --------------------------- %stddev %change %stddev \ | \ 252.40 -12.7% 220.41 phoronix-test-suite.supertuxkart.1280x1024.Fullscreen.Basic.1.GranParadisoIsland.frames_per_second 25367 ± 68% +183.7% 71956 ± 28% phoronix-test-suite.time.involuntary_context_switches 92.25 +2.2% 94.25 phoronix-test-suite.time.percent_of_cpu_this_job_got Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

4 months, 2 weeks

1
0
0 / 0

[iomap] 11b5156248: xfstests.xfs.310.fail

by kernel test robot

Greeting, FYI, we noticed the following commit (built with gcc-9): commit: 11b5156248eb1aab1d6f8bb7fdf2bb6e19c73a3f ("[PATCH 2/2] iomap: zero cached pages over unwritten extents on zero range") url: https://github.com/0day-ci/linux/commits/Brian-Foster/iomap-zero-dirty-pa... base: https://git.kernel.org/cgit/fs/xfs/xfs-linux.git for-next in testcase: xfstests version: xfstests-x86_64-d41dcbd-1_20201007 with following parameters: disk: 4HDD fs: xfs test: xfs-rmapbt ucode: 0x28 test-description: xfstests is a regression test suite for xfs and other files ystems. test-url: git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git on test machine: 8 threads Intel(R) Core(TM) i7-4790 v3 @ 3.60GHz with 6G memory caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> 2020-10-22 13:11:24 export TEST_DIR=/fs/sdb1 2020-10-22 13:11:24 export TEST_DEV=/dev/sdb1 2020-10-22 13:11:24 export FSTYP=xfs 2020-10-22 13:11:24 export SCRATCH_MNT=/fs/scratch 2020-10-22 13:11:24 mkdir /fs/scratch -p 2020-10-22 13:11:24 export SCRATCH_DEV=/dev/sdb4 2020-10-22 13:11:24 export SCRATCH_LOGDEV=/dev/sdb2 2020-10-22 13:11:24 export SCRATCH_XFS_LIST_METADATA_FIELDS=u3.sfdir3.hdr.parent.i4 2020-10-22 13:11:24 export SCRATCH_XFS_LIST_FUZZ_VERBS=random 2020-10-22 13:11:24 export MKFS_OPTIONS=-mrmapbt=1 2020-10-22 13:11:24 sed "s:^:xfs/:" //lkp/benchmarks/xfstests/tests/xfs-rmapbt | grep -F -f merged_ignored_files 2020-10-22 13:11:24 sed "s:^:xfs/:" //lkp/benchmarks/xfstests/tests/xfs-rmapbt | grep -v -F -f merged_ignored_files 2020-10-22 13:11:24 ./check xfs/114 xfs/233 xfs/234 xfs/235 xfs/236 xfs/271 xfs/272 xfs/277 xfs/310 xfs/317 xfs/332 xfs/333 xfs/368 xfs/369 xfs/370 xfs/371 xfs/422 xfs/450 xfs/462 xfs/463 xfs/517 FSTYP -- xfs (debug) PLATFORM -- Linux/x86_64 lkp-hsw-d02 5.9.0-rc4-00087-g11b5156248eb1 #1 SMP Tue Oct 13 20:22:06 CST 2020 MKFS_OPTIONS -- -f -mrmapbt=1 /dev/sdb4 MOUNT_OPTIONS -- /dev/sdb4 /fs/scratch xfs/114 3s xfs/233 1059s xfs/234 348s xfs/235 7s xfs/236 13s xfs/271 - output mismatch (see /lkp/benchmarks/xfstests/results//xfs/271.out.bad) --- tests/xfs/271.out 2020-10-07 10:39:37.000000000 +0000 +++ /lkp/benchmarks/xfstests/results//xfs/271.out.bad 2020-10-22 13:35:19.723880557 +0000 @@ -4,7 +4,8 @@ Check AG header AG header count is in range Check freesp/rmap btrees -freesp extent count is in range +freesp extent count has value of 9 +freesp extent count is NOT in range 12 .. 1000011 Check inode btrees ... (Run 'diff -u /lkp/benchmarks/xfstests/tests/xfs/271.out /lkp/benchmarks/xfstests/results//xfs/271.out.bad' to see the entire diff) xfs/272 3s xfs/277 3s xfs/310 - output mismatch (see /lkp/benchmarks/xfstests/results//xfs/310.out.bad) --- tests/xfs/310.out 2020-10-07 10:39:37.000000000 +0000 +++ /lkp/benchmarks/xfstests/results//xfs/310.out.bad 2020-10-22 13:35:28.850879806 +0000 @@ -3,11 +3,21 @@ Format huge device Create the original file blocks Check extent count +/fs/scratch/test-310/file1: + EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS + 0: [0..8388599]: 192..8388791 0 (192..8388791) 8388600 010000 + 1: [8388600..8388607]: 8388792..8388799 0 (8388792..8388799) 8 000000 ... (Run 'diff -u /lkp/benchmarks/xfstests/tests/xfs/310.out /lkp/benchmarks/xfstests/results//xfs/310.out.bad' to see the entire diff) xfs/317 4s xfs/332 3s xfs/333 [not run] realtime rmapbt not supported xfs/368 81s xfs/369 7s xfs/370 7s xfs/371 7s xfs/422 - output mismatch (see /lkp/benchmarks/xfstests/results//xfs/422.out.bad) --- tests/xfs/422.out 2020-10-07 10:39:37.000000000 +0000 +++ /lkp/benchmarks/xfstests/results//xfs/422.out.bad 2020-10-22 13:37:54.937867791 +0000 @@ -1,4 +1,196 @@ QA output created by 422 Format and populate Concurrent repair +scrub: Operation not supported +scrub: Operation not supported +scrub: Operation not supported +scrub: Operation not supported ... (Run 'diff -u /lkp/benchmarks/xfstests/tests/xfs/422.out /lkp/benchmarks/xfstests/results//xfs/422.out.bad' to see the entire diff) xfs/450 4s xfs/462 6s xfs/463 7s xfs/517 36s Ran: xfs/114 xfs/233 xfs/234 xfs/235 xfs/236 xfs/271 xfs/272 xfs/277 xfs/310 xfs/317 xfs/332 xfs/333 xfs/368 xfs/369 xfs/370 xfs/371 xfs/422 xfs/450 xfs/462 xfs/463 xfs/517 Not run: xfs/333 Failures: xfs/271 xfs/310 xfs/422 Failed 3 of 21 tests To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml Thanks, Rong Chen

4 months, 2 weeks

1
0
0 / 0

[NFS] cd2ed9bdc0: netperf.Throughput_tps 18.6% improvement

by kernel test robot

Greeting, FYI, we noticed a 18.6% improvement of netperf.Throughput_tps due to commit: commit: cd2ed9bdc0794464d57a5ae2d979b1543a75694b ("NFS: Add a tracepoint in nfs_set_pgio_error()") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: netperf on test machine: 144 threads Intel(R) Xeon(R) Gold 5318H CPU @ 2.50GHz with 128G memory with following parameters: ip: ipv4 runtime: 300s nr_threads: 16 cluster: cs-localhost test: TCP_CRR cpufreq_governor: performance ucode: 0x700001c test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance. test-url: http://www.netperf.org/netperf/ Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode: cs-localhost/gcc-9/performance/ipv4/x86_64-rhel-8.3/16/debian-10.4-x86_64-20200603.cgz/300s/lkp-cpl-4sp1/TCP_CRR/netperf/0x700001c commit: fd2b612141 ("NFS: Trace short NFS READs") cd2ed9bdc0 ("NFS: Add a tracepoint in nfs_set_pgio_error()") fd2b6121415539c3 cd2ed9bdc0794464d57a5ae2d97 ---------------- --------------------------- %stddev %change %stddev \ | \ 344552 +18.6% 408472 netperf.Throughput_total_tps 21534 +18.6% 25529 netperf.Throughput_tps 696891 +2.5% 714654 netperf.time.involuntary_context_switches 11571 +5.6% 12225 ± 2% netperf.time.minor_page_faults 846.25 +2.4% 866.25 netperf.time.percent_of_cpu_this_job_got 2531 +1.9% 2580 netperf.time.system_time 51.72 +23.0% 63.62 netperf.time.user_time 1.903e+08 +24.0% 2.36e+08 netperf.time.voluntary_context_switches 1.034e+08 +18.6% 1.225e+08 netperf.workload 35611 ± 6% -13.0% 30991 ± 7% numa-vmstat.node0.nr_slab_unreclaimable 0.68 +0.1 0.77 mpstat.cpu.all.irq% 0.24 +0.0 0.27 ± 2% mpstat.cpu.all.usr% 142446 ± 6% -13.0% 123962 ± 7% numa-meminfo.node0.SUnreclaim 54368 ± 64% -70.7% 15945 ±111% numa-meminfo.node2.AnonHugePages 85.00 -1.2% 84.00 vmstat.cpu.id 2260239 +10.1% 2488445 vmstat.system.cs 266.03 ±134% -88.7% 30.01 ± 14% sched_debug.cfs_rq:/.load_avg.stddev 50625 ± 57% +84.5% 93412 ± 5% sched_debug.cfs_rq:/.spread0.max 112.94 ± 4% -8.0% 103.95 sched_debug.cfs_rq:/.util_est_enqueued.avg 1.912e+08 -11.3% 1.696e+08 cpuidle.C1.usage 74059064 -36.3% 47207216 ± 30% cpuidle.C1E.usage 3.701e+08 +13.6% 4.204e+08 cpuidle.POLL.time 1.55e+08 +38.6% 2.148e+08 cpuidle.POLL.usage 77888 -12.0% 68513 slabinfo.sock_inode_cache.active_objs 2000 -12.0% 1760 slabinfo.sock_inode_cache.active_slabs 78049 -12.0% 68681 slabinfo.sock_inode_cache.num_objs 2000 -12.0% 1760 slabinfo.sock_inode_cache.num_slabs 82549 -0.9% 81783 proc-vmstat.nr_active_anon 11275 -4.5% 10767 ± 2% proc-vmstat.nr_shmem 46843 -4.9% 44558 proc-vmstat.nr_slab_reclaimable 119991 -0.8% 119086 proc-vmstat.nr_slab_unreclaimable 82549 -0.9% 81783 proc-vmstat.nr_zone_active_anon 1054811 +10.2% 1162809 proc-vmstat.numa_hit 984545 +11.0% 1092545 proc-vmstat.numa_local 1637946 +55.6% 2548597 ± 3% proc-vmstat.pgalloc_normal 1645287 +55.3% 2554964 ± 3% proc-vmstat.pgfree 14.04 -7.5% 12.98 ± 2% perf-stat.i.MPKI 8.41e+09 +16.2% 9.771e+09 perf-stat.i.branch-instructions 1.729e+08 +15.1% 1.99e+08 perf-stat.i.branch-misses 13.68 -3.6 10.11 ± 4% perf-stat.i.cache-miss-rate% 79625575 -23.5% 60928499 ± 4% perf-stat.i.cache-misses 5.928e+08 +1.9% 6.044e+08 perf-stat.i.cache-references 2282102 +10.1% 2512440 perf-stat.i.context-switches 1.96 -13.4% 1.70 perf-stat.i.cpi 8.137e+10 -1.2% 8.042e+10 perf-stat.i.cpu-cycles 1025 +39.0% 1426 ± 3% perf-stat.i.cycles-between-cache-misses 1.276e+10 +16.5% 1.486e+10 perf-stat.i.dTLB-loads 0.02 ± 3% -0.0 0.01 ± 11% perf-stat.i.dTLB-store-miss-rate% 1452729 ± 3% -48.1% 754050 ± 2% perf-stat.i.dTLB-store-misses 7.504e+09 +16.5% 8.74e+09 perf-stat.i.dTLB-stores 88.85 +2.1 90.96 perf-stat.i.iTLB-load-miss-rate% 1.311e+08 +17.4% 1.54e+08 perf-stat.i.iTLB-load-misses 15307315 -7.9% 14092472 perf-stat.i.iTLB-loads 4.213e+10 +16.3% 4.9e+10 perf-stat.i.instructions 0.52 +17.4% 0.61 perf-stat.i.ipc 0.57 -1.2% 0.56 perf-stat.i.metric.GHz 0.07 -6.2% 0.07 perf-stat.i.metric.K/sec 203.53 +16.1% 236.27 perf-stat.i.metric.M/sec 90.88 -0.9 89.96 perf-stat.i.node-load-miss-rate% 16351518 ± 2% -20.2% 13044449 ± 2% perf-stat.i.node-load-misses 1638072 -12.5% 1432598 ± 2% perf-stat.i.node-loads 7126496 +19.8% 8540461 perf-stat.i.node-store-misses 505279 +17.4% 593125 ± 3% perf-stat.i.node-stores 14.07 -12.3% 12.34 perf-stat.overall.MPKI 13.43 -3.4 10.08 ± 3% perf-stat.overall.cache-miss-rate% 1.93 -15.0% 1.64 perf-stat.overall.cpi 1022 +29.4% 1322 ± 4% perf-stat.overall.cycles-between-cache-misses 0.01 ± 2% -0.0 0.01 ± 5% perf-stat.overall.dTLB-load-miss-rate% 0.02 ± 3% -0.0 0.01 ± 2% perf-stat.overall.dTLB-store-miss-rate% 89.55 +2.1 91.62 perf-stat.overall.iTLB-load-miss-rate% 0.52 +17.7% 0.61 perf-stat.overall.ipc 124335 -1.9% 121971 perf-stat.overall.path-length 8.383e+09 +16.2% 9.739e+09 perf-stat.ps.branch-instructions 1.723e+08 +15.1% 1.984e+08 perf-stat.ps.branch-misses 79364332 -23.5% 60722285 ± 4% perf-stat.ps.cache-misses 5.909e+08 +1.9% 6.023e+08 perf-stat.ps.cache-references 2274506 +10.1% 2503968 perf-stat.ps.context-switches 8.11e+10 -1.2% 8.015e+10 perf-stat.ps.cpu-cycles 1.272e+10 +16.5% 1.481e+10 perf-stat.ps.dTLB-loads 1447936 ± 3% -48.1% 751551 ± 2% perf-stat.ps.dTLB-store-misses 7.479e+09 +16.5% 8.711e+09 perf-stat.ps.dTLB-stores 1.307e+08 +17.4% 1.535e+08 perf-stat.ps.iTLB-load-misses 15256427 -7.9% 14044651 perf-stat.ps.iTLB-loads 4.199e+10 +16.3% 4.883e+10 perf-stat.ps.instructions 16297319 ± 2% -20.2% 12999740 ± 2% perf-stat.ps.node-load-misses 1632987 -12.5% 1428325 ± 2% perf-stat.ps.node-loads 7103017 +19.8% 8511899 perf-stat.ps.node-store-misses 504382 +17.4% 592137 ± 3% perf-stat.ps.node-stores 1.285e+13 +16.3% 1.495e+13 perf-stat.total.instructions 1596 ±122% +312.9% 6591 ± 71% interrupts.32:PCI-MSI.524290-edge.eth0-TxRx-1 1596 ±122% +312.9% 6591 ± 71% interrupts.CPU10.32:PCI-MSI.524290-edge.eth0-TxRx-1 4202 ± 12% -31.8% 2865 ± 14% interrupts.CPU100.RES:Rescheduling_interrupts 3472 -18.5% 2829 ± 12% interrupts.CPU101.RES:Rescheduling_interrupts 3584 ± 11% -24.2% 2716 ± 6% interrupts.CPU102.RES:Rescheduling_interrupts 2279 ± 15% -32.2% 1545 ± 20% interrupts.CPU103.NMI:Non-maskable_interrupts 2279 ± 15% -32.2% 1545 ± 20% interrupts.CPU103.PMI:Performance_monitoring_interrupts 4094 ± 13% -30.5% 2845 ± 6% interrupts.CPU103.RES:Rescheduling_interrupts 4082 ± 9% -27.4% 2962 ± 10% interrupts.CPU104.RES:Rescheduling_interrupts 3178 ± 13% -57.1% 1362 ± 20% interrupts.CPU105.NMI:Non-maskable_interrupts 3178 ± 13% -57.1% 1362 ± 20% interrupts.CPU105.PMI:Performance_monitoring_interrupts 3966 ± 2% -27.2% 2889 ± 6% interrupts.CPU105.RES:Rescheduling_interrupts 3843 ± 7% -25.9% 2847 ± 4% interrupts.CPU106.RES:Rescheduling_interrupts 3867 ± 14% -25.3% 2890 ± 8% interrupts.CPU107.RES:Rescheduling_interrupts 3498 ± 2% -16.9% 2907 ± 5% interrupts.CPU110.RES:Rescheduling_interrupts 3425 ± 15% -35.4% 2212 ± 27% interrupts.CPU112.NMI:Non-maskable_interrupts 3425 ± 15% -35.4% 2212 ± 27% interrupts.CPU112.PMI:Performance_monitoring_interrupts 2627 ± 27% -44.5% 1459 ± 56% interrupts.CPU114.NMI:Non-maskable_interrupts 2627 ± 27% -44.5% 1459 ± 56% interrupts.CPU114.PMI:Performance_monitoring_interrupts 3487 ± 11% -21.6% 2735 ± 13% interrupts.CPU115.RES:Rescheduling_interrupts 3597 ± 10% -20.9% 2846 ± 10% interrupts.CPU116.RES:Rescheduling_interrupts 3498 ± 6% -25.1% 2621 ± 10% interrupts.CPU117.RES:Rescheduling_interrupts 4029 ± 10% -30.1% 2814 ± 15% interrupts.CPU118.RES:Rescheduling_interrupts 1879 ± 23% -43.4% 1063 ± 9% interrupts.CPU119.NMI:Non-maskable_interrupts 1879 ± 23% -43.4% 1063 ± 9% interrupts.CPU119.PMI:Performance_monitoring_interrupts 3516 ± 6% -22.0% 2742 ± 10% interrupts.CPU119.RES:Rescheduling_interrupts 3639 ± 6% -32.0% 2476 ± 14% interrupts.CPU120.RES:Rescheduling_interrupts 3655 ± 15% -30.5% 2541 ± 10% interrupts.CPU121.RES:Rescheduling_interrupts 3819 ± 14% -38.3% 2355 ± 13% interrupts.CPU122.RES:Rescheduling_interrupts 3551 ± 3% -28.8% 2527 ± 23% interrupts.CPU123.RES:Rescheduling_interrupts 3513 ± 12% -35.2% 2277 ± 11% interrupts.CPU124.RES:Rescheduling_interrupts 159.25 ±102% -88.4% 18.50 ±139% interrupts.CPU124.TLB:TLB_shootdowns 3330 ± 5% -35.3% 2155 ± 14% interrupts.CPU125.RES:Rescheduling_interrupts 1504 ± 2% -18.7% 1222 ± 4% interrupts.CPU128.CAL:Function_call_interrupts 7814 ± 6% -16.0% 6564 ± 8% interrupts.CPU19.RES:Rescheduling_interrupts 6584 ± 7% -10.4% 5901 interrupts.CPU2.RES:Rescheduling_interrupts 6750 ± 5% -13.0% 5871 ± 9% interrupts.CPU20.RES:Rescheduling_interrupts 6671 ± 7% -14.5% 5706 ± 8% interrupts.CPU21.RES:Rescheduling_interrupts 3685 ± 7% -28.5% 2634 ± 20% interrupts.CPU36.NMI:Non-maskable_interrupts 3685 ± 7% -28.5% 2634 ± 20% interrupts.CPU36.PMI:Performance_monitoring_interrupts 6860 -14.9% 5835 ± 5% interrupts.CPU36.RES:Rescheduling_interrupts 8275 ± 3% -13.9% 7127 interrupts.CPU37.RES:Rescheduling_interrupts 7063 ± 7% -14.2% 6062 ± 8% interrupts.CPU38.RES:Rescheduling_interrupts 7154 ± 9% -13.7% 6175 ± 6% interrupts.CPU39.RES:Rescheduling_interrupts 3750 ± 11% -39.1% 2285 ± 27% interrupts.CPU40.NMI:Non-maskable_interrupts 3750 ± 11% -39.1% 2285 ± 27% interrupts.CPU40.PMI:Performance_monitoring_interrupts 6273 ± 7% -13.4% 5430 interrupts.CPU41.RES:Rescheduling_interrupts 6081 ± 3% -11.0% 5411 ± 3% interrupts.CPU42.RES:Rescheduling_interrupts 5887 ± 8% -16.3% 4928 ± 5% interrupts.CPU43.RES:Rescheduling_interrupts 3884 ± 10% -14.8% 3310 ± 3% interrupts.CPU51.RES:Rescheduling_interrupts 4014 ± 8% -33.4% 2672 ± 43% interrupts.CPU54.NMI:Non-maskable_interrupts 4014 ± 8% -33.4% 2672 ± 43% interrupts.CPU54.PMI:Performance_monitoring_interrupts 7221 ± 6% -12.5% 6317 ± 7% interrupts.CPU56.RES:Rescheduling_interrupts 86.00 ± 34% -64.8% 30.25 ± 52% interrupts.CPU60.TLB:TLB_shootdowns 59.00 ± 76% -74.2% 15.25 ±112% interrupts.CPU64.TLB:TLB_shootdowns 4485 ± 12% -13.2% 3891 ± 7% interrupts.CPU69.RES:Rescheduling_interrupts 2828 ± 9% -35.4% 1828 ± 11% interrupts.CPU72.NMI:Non-maskable_interrupts 2828 ± 9% -35.4% 1828 ± 11% interrupts.CPU72.PMI:Performance_monitoring_interrupts 3405 ± 8% -17.4% 2811 ± 8% interrupts.CPU73.RES:Rescheduling_interrupts 4192 ± 8% -17.3% 3466 ± 14% interrupts.CPU76.RES:Rescheduling_interrupts 66.75 ± 65% -68.9% 20.75 ± 95% interrupts.CPU76.TLB:TLB_shootdowns 4103 ± 7% -19.4% 3307 ± 5% interrupts.CPU78.RES:Rescheduling_interrupts 3990 ± 8% -16.6% 3328 ± 11% interrupts.CPU79.RES:Rescheduling_interrupts 3940 ± 10% -20.4% 3135 ± 12% interrupts.CPU80.RES:Rescheduling_interrupts 1588 ± 31% +32.4% 2103 ± 11% interrupts.CPU81.NMI:Non-maskable_interrupts 1588 ± 31% +32.4% 2103 ± 11% interrupts.CPU81.PMI:Performance_monitoring_interrupts 3756 ± 6% -23.1% 2890 ± 13% interrupts.CPU82.RES:Rescheduling_interrupts 2664 ± 24% -38.1% 1650 ± 23% interrupts.CPU84.NMI:Non-maskable_interrupts 2664 ± 24% -38.1% 1650 ± 23% interrupts.CPU84.PMI:Performance_monitoring_interrupts 3933 ± 12% -25.3% 2937 ± 19% interrupts.CPU84.RES:Rescheduling_interrupts 2883 ± 23% -37.3% 1808 ± 19% interrupts.CPU85.NMI:Non-maskable_interrupts 2883 ± 23% -37.3% 1808 ± 19% interrupts.CPU85.PMI:Performance_monitoring_interrupts 4577 ± 7% -28.6% 3269 ± 10% interrupts.CPU85.RES:Rescheduling_interrupts 2533 ± 10% -33.6% 1683 ± 27% interrupts.CPU86.NMI:Non-maskable_interrupts 2533 ± 10% -33.6% 1683 ± 27% interrupts.CPU86.PMI:Performance_monitoring_interrupts 4210 ± 5% -36.5% 2674 ± 10% interrupts.CPU87.RES:Rescheduling_interrupts 4207 ± 4% -33.3% 2807 ± 17% interrupts.CPU88.RES:Rescheduling_interrupts 4244 ± 14% -38.3% 2617 ± 12% interrupts.CPU89.RES:Rescheduling_interrupts 1813 ± 35% +50.4% 2728 ± 6% interrupts.CPU90.NMI:Non-maskable_interrupts 1813 ± 35% +50.4% 2728 ± 6% interrupts.CPU90.PMI:Performance_monitoring_interrupts 1747 ± 41% +73.6% 3034 ± 18% interrupts.CPU92.NMI:Non-maskable_interrupts 1747 ± 41% +73.6% 3034 ± 18% interrupts.CPU92.PMI:Performance_monitoring_interrupts 3482 ± 5% -16.3% 2914 ± 9% interrupts.CPU93.RES:Rescheduling_interrupts 3743 ± 8% -17.1% 3102 ± 4% interrupts.CPU98.RES:Rescheduling_interrupts 3843 ± 10% -21.3% 3025 ± 7% interrupts.CPU99.RES:Rescheduling_interrupts 321364 ± 3% -9.5% 290702 ± 5% interrupts.NMI:Non-maskable_interrupts 321364 ± 3% -9.5% 290702 ± 5% interrupts.PMI:Performance_monitoring_interrupts 620411 ± 2% -11.1% 551544 ± 3% interrupts.RES:Rescheduling_interrupts 3593372 ± 3% +24.1% 4458722 ± 5% softirqs.CPU1.NET_RX 226535 ± 2% +21.9% 276124 ± 3% softirqs.CPU1.RCU 1986693 ± 8% +37.1% 2723958 ± 11% softirqs.CPU10.NET_RX 158646 ± 4% +25.5% 199166 ± 5% softirqs.CPU10.RCU 119014 -9.4% 107770 ± 2% softirqs.CPU100.TIMER 118480 ± 2% -9.1% 107703 softirqs.CPU101.TIMER 2207946 ± 4% -14.1% 1897640 ± 7% softirqs.CPU104.NET_RX 120528 -10.0% 108443 ± 3% softirqs.CPU104.TIMER 120600 -9.0% 109800 ± 2% softirqs.CPU106.TIMER 120545 -10.1% 108427 ± 2% softirqs.CPU107.TIMER 1582511 ± 7% +23.0% 1946344 ± 7% softirqs.CPU109.NET_RX 141528 ± 3% +15.6% 163591 ± 4% softirqs.CPU109.RCU 118009 ± 2% -8.9% 107517 softirqs.CPU109.TIMER 1912511 ± 18% +42.7% 2728897 ± 7% softirqs.CPU11.NET_RX 155760 ± 10% +28.7% 200424 ± 4% softirqs.CPU11.RCU 146133 ± 7% +12.1% 163827 ± 4% softirqs.CPU110.RCU 1626185 ± 15% +23.6% 2009224 ± 4% softirqs.CPU111.NET_RX 142074 ± 7% +16.6% 165660 ± 2% softirqs.CPU111.RCU 1747237 ± 8% +16.6% 2038148 ± 5% softirqs.CPU114.NET_RX 153419 ± 4% +15.9% 177751 ± 2% softirqs.CPU114.RCU 2110126 ± 7% +28.4% 2708515 ± 4% softirqs.CPU12.NET_RX 166607 ± 4% +20.6% 200889 ± 2% softirqs.CPU12.RCU 2113615 ± 5% -13.5% 1827266 ± 8% softirqs.CPU120.NET_RX 120534 ± 2% -7.8% 111115 ± 2% softirqs.CPU124.TIMER 1909929 ± 7% -16.4% 1596274 ± 10% softirqs.CPU125.NET_RX 1636286 ± 11% +28.3% 2099950 ± 5% softirqs.CPU126.NET_RX 137556 ± 6% +18.6% 163127 ± 3% softirqs.CPU126.RCU 119192 -10.1% 107168 ± 4% softirqs.CPU126.TIMER 1475844 ± 4% +24.9% 1842610 softirqs.CPU127.NET_RX 129985 +16.3% 151196 softirqs.CPU127.RCU 1452187 ± 7% +33.3% 1935909 ± 15% softirqs.CPU128.NET_RX 132133 ± 3% +22.4% 161687 ± 8% softirqs.CPU128.RCU 135284 ± 8% +20.4% 162872 ± 7% softirqs.CPU129.RCU 2011051 ± 11% +32.6% 2666473 ± 5% softirqs.CPU13.NET_RX 162143 ± 6% +23.0% 199371 ± 3% softirqs.CPU13.RCU 121124 -9.3% 109854 ± 4% softirqs.CPU130.TIMER 131083 ± 2% +17.7% 154299 ± 5% softirqs.CPU131.RCU 119764 -11.4% 106134 ± 4% softirqs.CPU131.TIMER 1452793 ± 7% +18.9% 1727905 ± 6% softirqs.CPU132.NET_RX 132592 ± 3% +16.1% 153885 ± 3% softirqs.CPU132.RCU 121779 -10.9% 108527 ± 4% softirqs.CPU132.TIMER 1392612 ± 8% +31.3% 1828076 ± 11% softirqs.CPU134.NET_RX 129678 ± 4% +21.3% 157332 ± 5% softirqs.CPU134.RCU 134110 +11.5% 149577 ± 5% softirqs.CPU135.RCU 119023 -11.2% 105731 ± 4% softirqs.CPU135.TIMER 125517 ± 6% +15.7% 145278 ± 5% softirqs.CPU136.RCU 119262 -11.0% 106099 ± 3% softirqs.CPU136.TIMER 121415 -13.1% 105464 ± 4% softirqs.CPU137.TIMER 122181 -11.3% 108337 ± 3% softirqs.CPU138.TIMER 122335 -11.3% 108451 ± 3% softirqs.CPU139.TIMER 1955730 ± 7% +45.8% 2851594 ± 3% softirqs.CPU14.NET_RX 158244 ± 4% +28.8% 203771 ± 2% softirqs.CPU14.RCU 117804 ± 3% -10.7% 105216 ± 2% softirqs.CPU140.TIMER 1436375 ± 12% +21.0% 1738245 ± 6% softirqs.CPU142.NET_RX 130378 ± 5% +15.8% 151042 ± 2% softirqs.CPU142.RCU 122696 -12.5% 107390 softirqs.CPU142.TIMER 127172 ± 6% +13.1% 143830 ± 3% softirqs.CPU143.RCU 120429 -10.6% 107672 ± 4% softirqs.CPU143.TIMER 1934365 +42.6% 2758014 ± 6% softirqs.CPU15.NET_RX 156614 +27.7% 199936 ± 3% softirqs.CPU15.RCU 1682969 ± 7% +53.2% 2577586 ± 6% softirqs.CPU16.NET_RX 139353 ± 3% +30.7% 182084 ± 3% softirqs.CPU16.RCU 1667289 ± 2% +47.7% 2461831 ± 6% softirqs.CPU17.NET_RX 137886 +28.3% 176873 ± 4% softirqs.CPU17.RCU 243373 ± 2% +15.8% 281889 ± 6% softirqs.CPU18.RCU 3597851 ± 4% +15.6% 4159558 ± 5% softirqs.CPU19.NET_RX 230245 ± 3% +15.6% 266243 ± 4% softirqs.CPU19.RCU 3260027 ± 6% +20.1% 3915431 ± 5% softirqs.CPU2.NET_RX 214070 ± 3% +18.0% 252670 ± 4% softirqs.CPU2.RCU 3336728 ± 6% +15.6% 3857806 ± 7% softirqs.CPU20.NET_RX 219501 ± 3% +15.3% 253004 ± 4% softirqs.CPU20.RCU 3158811 ± 3% +23.2% 3892880 ± 10% softirqs.CPU21.NET_RX 212206 ± 2% +20.5% 255774 ± 7% softirqs.CPU21.RCU 210621 ± 2% +18.5% 249495 ± 8% softirqs.CPU22.RCU 2884709 ± 8% +27.9% 3690802 ± 13% softirqs.CPU23.NET_RX 202022 ± 5% +22.4% 247237 ± 9% softirqs.CPU23.RCU 2664674 ± 4% +28.4% 3422601 ± 11% softirqs.CPU24.NET_RX 191442 ± 3% +21.9% 233288 ± 8% softirqs.CPU24.RCU 2490550 ± 4% +38.3% 3443438 ± 14% softirqs.CPU25.NET_RX 184538 ± 2% +27.7% 235614 ± 10% softirqs.CPU25.RCU 2337283 ± 3% +40.3% 3278081 ± 15% softirqs.CPU26.NET_RX 178781 +27.6% 228079 ± 10% softirqs.CPU26.RCU 2220201 ± 2% +47.3% 3270829 ± 12% softirqs.CPU27.NET_RX 172858 ± 2% +31.6% 227541 ± 8% softirqs.CPU27.RCU 2321206 ± 5% +35.2% 3137549 ± 12% softirqs.CPU28.NET_RX 177082 ± 2% +25.0% 221425 ± 8% softirqs.CPU28.RCU 2246678 ± 6% +31.9% 2964377 ± 12% softirqs.CPU29.NET_RX 171083 +24.1% 212231 ± 8% softirqs.CPU29.RCU 127186 ± 16% -14.2% 109173 softirqs.CPU29.TIMER 3226488 ± 6% +19.3% 3848958 ± 8% softirqs.CPU3.NET_RX 213112 ± 4% +16.8% 248948 ± 5% softirqs.CPU3.RCU 2113065 ± 6% +31.8% 2785406 ± 10% softirqs.CPU30.NET_RX 168449 ± 2% +22.6% 206573 ± 7% softirqs.CPU30.RCU 2115777 ± 3% +36.7% 2891924 ± 7% softirqs.CPU31.NET_RX 168113 ± 2% +25.3% 210584 ± 4% softirqs.CPU31.RCU 1866022 ± 3% +45.6% 2716794 ± 13% softirqs.CPU32.NET_RX 153261 +27.6% 195634 ± 9% softirqs.CPU32.RCU 1914777 ± 8% +36.7% 2618195 ± 13% softirqs.CPU33.NET_RX 155525 ± 4% +23.0% 191256 ± 9% softirqs.CPU33.RCU 1918176 ± 7% +32.2% 2536083 ± 18% softirqs.CPU34.NET_RX 156200 ± 4% +20.8% 188697 ± 11% softirqs.CPU34.RCU 1859446 ± 11% +43.6% 2670610 ± 10% softirqs.CPU35.NET_RX 153154 ± 5% +27.7% 195597 ± 6% softirqs.CPU35.RCU 4028702 ± 5% +17.3% 4725668 ± 5% softirqs.CPU36.NET_RX 249892 ± 4% +15.9% 289642 ± 4% softirqs.CPU36.RCU 3901006 ± 4% +18.5% 4621206 ± 5% softirqs.CPU37.NET_RX 246085 ± 3% +17.5% 289260 ± 4% softirqs.CPU37.RCU 3453472 ± 4% +17.5% 4059413 ± 5% softirqs.CPU38.NET_RX 227481 ± 3% +16.1% 264060 ± 3% softirqs.CPU38.RCU 3486864 ± 8% +16.7% 4070191 ± 8% softirqs.CPU39.NET_RX 228240 ± 5% +16.9% 266796 ± 5% softirqs.CPU39.RCU 2847467 ± 9% +27.0% 3616029 ± 8% softirqs.CPU4.NET_RX 196920 ± 6% +22.9% 242024 ± 5% softirqs.CPU4.RCU 3341551 ± 3% +16.2% 3881977 ± 4% softirqs.CPU40.NET_RX 223409 ± 2% +15.5% 257937 ± 3% softirqs.CPU40.RCU 3023399 ± 6% +23.2% 3726154 ± 6% softirqs.CPU41.NET_RX 208766 ± 4% +19.9% 250363 ± 4% softirqs.CPU41.RCU 2957016 ± 6% +18.3% 3497870 ± 3% softirqs.CPU42.NET_RX 205603 ± 3% +16.4% 239352 ± 2% softirqs.CPU42.RCU 2734344 ± 5% +23.6% 3379733 ± 6% softirqs.CPU43.NET_RX 196564 ± 4% +19.3% 234447 ± 4% softirqs.CPU43.RCU 2439722 ± 4% +26.0% 3073133 ± 13% softirqs.CPU44.NET_RX 183663 ± 3% +20.3% 221001 ± 8% softirqs.CPU44.RCU 2347126 ± 2% +28.6% 3018639 ± 7% softirqs.CPU45.NET_RX 179921 +20.9% 217552 ± 4% softirqs.CPU45.RCU 2117973 ± 4% +47.1% 3115648 ± 8% softirqs.CPU46.NET_RX 169760 ± 2% +30.4% 221452 ± 5% softirqs.CPU46.RCU 2197927 ± 7% +32.2% 2905245 ± 4% softirqs.CPU47.NET_RX 174027 ± 4% +22.1% 212537 ± 3% softirqs.CPU47.RCU 2209516 ± 13% +22.3% 2702620 ± 5% softirqs.CPU48.NET_RX 171322 ± 7% +17.4% 201079 ± 3% softirqs.CPU48.RCU 2132649 ± 4% +28.1% 2731631 ± 2% softirqs.CPU49.NET_RX 167794 ± 4% +21.0% 203072 softirqs.CPU49.RCU 2790329 ± 10% +26.4% 3526292 softirqs.CPU5.NET_RX 194162 ± 6% +22.0% 236849 softirqs.CPU5.RCU 2085514 ± 3% +27.6% 2661937 ± 6% softirqs.CPU50.NET_RX 167584 +18.0% 197794 ± 4% softirqs.CPU50.RCU 1991718 ± 8% +25.2% 2494153 ± 6% softirqs.CPU51.NET_RX 163095 ± 4% +17.6% 191788 ± 3% softirqs.CPU51.RCU 2133613 ± 8% +22.2% 2606830 ± 7% softirqs.CPU52.NET_RX 169663 ± 5% +17.2% 198894 ± 4% softirqs.CPU52.RCU 2066577 ± 7% +28.5% 2656488 ± 8% softirqs.CPU53.NET_RX 166297 ± 3% +20.1% 199660 ± 4% softirqs.CPU53.RCU 4000568 ± 2% +24.1% 4965013 ± 2% softirqs.CPU54.NET_RX 243490 +21.6% 295986 ± 2% softirqs.CPU54.RCU 3805713 ± 8% +27.1% 4838877 ± 3% softirqs.CPU55.NET_RX 235071 ± 5% +24.1% 291795 ± 3% softirqs.CPU55.RCU 3335036 ± 3% +30.0% 4336664 ± 4% softirqs.CPU56.NET_RX 215881 ± 2% +24.1% 268008 ± 3% softirqs.CPU56.RCU 3306944 ± 3% +21.7% 4024256 ± 3% softirqs.CPU57.NET_RX 214474 ± 2% +19.3% 255972 ± 2% softirqs.CPU57.RCU 2864592 ± 2% +30.4% 3734762 ± 8% softirqs.CPU58.NET_RX 195973 +24.3% 243647 ± 5% softirqs.CPU58.RCU 2683800 ± 2% +40.3% 3766474 ± 10% softirqs.CPU59.NET_RX 188865 +29.8% 245156 ± 7% softirqs.CPU59.RCU 2519426 ± 5% +35.5% 3414021 ± 4% softirqs.CPU6.NET_RX 180725 ± 2% +26.6% 228828 ± 2% softirqs.CPU6.RCU 2536942 ± 2% +38.2% 3506452 ± 6% softirqs.CPU60.NET_RX 182554 +28.1% 233925 ± 4% softirqs.CPU60.RCU 2409532 ± 9% +38.7% 3342882 ± 8% softirqs.CPU61.NET_RX 176350 ± 5% +28.5% 226650 ± 5% softirqs.CPU61.RCU 2252649 ± 11% +28.7% 2899665 ± 9% softirqs.CPU62.NET_RX 170252 ± 5% +19.0% 202671 ± 5% softirqs.CPU62.RCU 2106398 ± 7% +34.2% 2825938 ± 13% softirqs.CPU63.NET_RX 164218 ± 3% +23.3% 202447 ± 8% softirqs.CPU63.RCU 2170137 ± 9% +24.5% 2701698 ± 8% softirqs.CPU64.NET_RX 167624 ± 5% +18.0% 197837 ± 5% softirqs.CPU64.RCU 2129554 ± 10% +32.8% 2827714 ± 8% softirqs.CPU65.NET_RX 165952 ± 6% +22.1% 202621 ± 5% softirqs.CPU65.RCU 2031092 ± 5% +31.8% 2677262 ± 7% softirqs.CPU66.NET_RX 161584 ± 2% +22.6% 198169 ± 4% softirqs.CPU66.RCU 2106872 ± 4% +29.9% 2737450 ± 3% softirqs.CPU67.NET_RX 164853 ± 2% +21.7% 200598 ± 2% softirqs.CPU67.RCU 2004926 ± 4% +35.1% 2709009 ± 12% softirqs.CPU68.NET_RX 160577 ± 2% +23.1% 197654 ± 7% softirqs.CPU68.RCU 2107869 ± 6% +33.1% 2805708 ± 11% softirqs.CPU69.NET_RX 164080 ± 2% +23.0% 201767 ± 7% softirqs.CPU69.RCU 2618567 ± 6% +21.3% 3176081 ± 6% softirqs.CPU7.NET_RX 186722 ± 4% +18.6% 221432 ± 4% softirqs.CPU7.RCU 1995860 ± 4% +33.7% 2668106 ± 9% softirqs.CPU70.NET_RX 160498 ± 2% +22.7% 197002 ± 6% softirqs.CPU70.RCU 2086691 ± 7% +24.0% 2588389 ± 14% softirqs.CPU71.NET_RX 164539 ± 3% +17.7% 193735 ± 8% softirqs.CPU71.RCU 1583296 ± 5% +25.5% 1986784 ± 10% softirqs.CPU72.NET_RX 146349 ± 3% +17.8% 172393 ± 5% softirqs.CPU72.RCU 1723540 ± 3% +19.3% 2056671 ± 9% softirqs.CPU73.NET_RX 154770 +13.9% 176248 ± 4% softirqs.CPU73.RCU 2132400 ± 5% +13.4% 2418332 ± 6% softirqs.CPU76.NET_RX 170748 +13.3% 193404 ± 4% softirqs.CPU76.RCU 169841 +9.8% 186537 ± 6% softirqs.CPU77.RCU 172572 ± 4% +12.9% 194754 ± 4% softirqs.CPU79.RCU 2165682 ± 10% +33.0% 2880495 ± 3% softirqs.CPU8.NET_RX 168076 ± 5% +23.6% 207768 ± 2% softirqs.CPU8.RCU 173735 +9.6% 190394 ± 6% softirqs.CPU80.RCU 170255 ± 5% +6.9% 182041 ± 6% softirqs.CPU86.RCU 120389 -9.3% 109250 ± 2% softirqs.CPU88.TIMER 2341253 ± 12% -17.7% 1927807 ± 13% softirqs.CPU89.NET_RX 51644 ± 3% -7.1% 47971 ± 3% softirqs.CPU89.SCHED 2224210 ± 9% +30.2% 2896872 ± 9% softirqs.CPU9.NET_RX 170317 ± 5% +21.6% 207023 ± 5% softirqs.CPU9.RCU 1680399 ± 10% +32.9% 2233023 ± 13% softirqs.CPU91.NET_RX 145790 ± 4% +21.7% 177408 ± 7% softirqs.CPU91.RCU 152008 ± 6% +13.5% 172492 ± 3% softirqs.CPU93.RCU 119850 ± 2% -8.7% 109385 ± 2% softirqs.CPU93.TIMER 1769949 ± 9% +20.5% 2133146 ± 7% softirqs.CPU94.NET_RX 151556 ± 3% +17.1% 177407 ± 5% softirqs.CPU94.RCU 119053 ± 2% -8.4% 109067 ± 3% softirqs.CPU94.TIMER 120111 -9.3% 108987 ± 2% softirqs.CPU96.TIMER 119393 -9.6% 107881 ± 2% softirqs.CPU99.TIMER 3.185e+08 +18.7% 3.78e+08 softirqs.NET_RX 24554353 +15.9% 28462115 softirqs.RCU 39.93 -6.5 33.45 ± 10% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 50.65 -6.0 44.69 ± 6% perf-profile.calltrace.cycles-pp.secondary_startup_64 50.31 -6.0 44.35 ± 6% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 50.33 -5.9 44.38 ± 6% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 50.34 -5.9 44.40 ± 6% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 45.56 -5.9 39.62 ± 7% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 45.16 -5.7 39.47 ± 7% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 5.86 -0.8 5.06 ± 6% perf-profile.calltrace.cycles-pp.tcp_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect 7.05 -0.7 6.31 ± 6% perf-profile.calltrace.cycles-pp.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect 4.76 -0.6 4.17 ± 6% perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect 1.75 ± 10% -0.5 1.22 perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 4.01 -0.5 3.53 ± 6% perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect.tcp_v4_connect 3.98 -0.5 3.51 ± 6% perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect 4.34 -0.4 3.93 ± 6% perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect.tcp_v4_connect.__inet_stream_connect 2.21 -0.4 1.81 ± 6% perf-profile.calltrace.cycles-pp.tcp_conn_request.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu 1.32 -0.4 0.94 ± 6% perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg 1.36 -0.4 0.99 ± 5% perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg 1.41 -0.4 1.05 ± 5% perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto 1.05 ± 2% -0.4 0.69 ± 5% perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock 1.46 -0.4 1.11 ± 5% perf-profile.calltrace.cycles-pp.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 0.56 -0.2 0.41 ± 57% perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu 0.59 ± 2% +0.1 0.65 ± 5% perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv 0.65 ± 2% +0.1 0.73 ± 4% perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule_idle.do_idle 0.55 ± 4% +0.1 0.64 ± 5% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout 0.61 ± 3% +0.1 0.71 ± 5% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.wait_woken 0.60 ± 2% +0.1 0.70 ± 5% perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish 0.86 +0.1 0.99 ± 4% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule_idle.do_idle.cpu_startup_entry 0.93 ± 2% +0.1 1.07 ± 7% perf-profile.calltrace.cycles-pp.__x64_sys_bind.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.93 +0.1 1.06 ± 6% perf-profile.calltrace.cycles-pp.__sys_bind.__x64_sys_bind.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.79 +0.1 0.93 ± 5% perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu 0.71 ± 2% +0.2 0.87 ± 4% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 1.32 ± 3% +0.2 1.48 ± 4% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle 0.51 +0.2 0.69 ± 5% perf-profile.calltrace.cycles-pp.sock_alloc_file.__sys_accept4_file.__sys_accept4.__x64_sys_accept.do_syscall_64 0.76 +0.2 0.94 ± 6% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_child_process 0.75 ± 2% +0.2 0.93 ± 6% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable 0.51 +0.2 0.69 ± 5% perf-profile.calltrace.cycles-pp.alloc_file_pseudo.sock_alloc_file.__sys_accept4_file.__sys_accept4.__x64_sys_accept 0.77 ± 2% +0.2 0.95 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_child_process.tcp_v4_rcv 0.52 ± 4% +0.2 0.70 ± 5% perf-profile.calltrace.cycles-pp.__sk_destruct.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver 0.52 +0.2 0.70 ± 5% perf-profile.calltrace.cycles-pp.__dentry_kill.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64 0.53 ± 3% +0.2 0.72 ± 5% perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.inet_csk_accept.inet_accept.__sys_accept4_file 0.57 ± 3% +0.2 0.77 ± 6% perf-profile.calltrace.cycles-pp.schedule_timeout.inet_csk_accept.inet_accept.__sys_accept4_file.__sys_accept4 0.53 +0.2 0.73 ± 8% perf-profile.calltrace.cycles-pp.sock_alloc_file.__sys_socket.__x64_sys_socket.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.83 +0.2 1.03 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu 0.65 ± 2% +0.2 0.85 ± 5% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_wakeup.tcp_fin 0.52 +0.2 0.73 ± 8% perf-profile.calltrace.cycles-pp.alloc_file_pseudo.sock_alloc_file.__sys_socket.__x64_sys_socket.do_syscall_64 1.25 ± 2% +0.2 1.46 ± 5% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.wait_woken.sk_wait_data 0.90 +0.2 1.11 ± 6% perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish 0.68 ± 2% +0.2 0.90 ± 5% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_wakeup.tcp_fin.tcp_data_queue 1.38 ± 2% +0.2 1.60 ± 5% perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg.inet_recvmsg 1.31 +0.2 1.53 ± 5% perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg 0.58 ± 2% +0.2 0.81 ± 4% perf-profile.calltrace.cycles-pp.tcp_send_fin.tcp_close.inet_release.__sock_release.sock_close 0.73 ± 2% +0.2 0.97 ± 5% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_wakeup.tcp_fin.tcp_data_queue.tcp_rcv_established 1.49 +0.2 1.73 ± 6% perf-profile.calltrace.cycles-pp.__sock_create.__sys_socket.__x64_sys_socket.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.38 ± 3% +0.2 1.62 ± 6% perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.tcp_close.inet_release.__sock_release 1.37 ± 3% +0.2 1.61 ± 6% perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.tcp_close.inet_release 0.76 ± 2% +0.2 1.00 ± 6% perf-profile.calltrace.cycles-pp.sock_def_wakeup.tcp_fin.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv 1.41 ± 3% +0.2 1.65 ± 6% perf-profile.calltrace.cycles-pp.__release_sock.tcp_close.inet_release.__sock_release.sock_close 0.81 ± 3% +0.2 1.06 ± 6% perf-profile.calltrace.cycles-pp.tcp_fin.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv 0.75 ± 3% +0.2 1.00 ± 6% perf-profile.calltrace.cycles-pp.sk_clone_lock.inet_csk_clone_lock.tcp_create_openreq_child.tcp_v4_syn_recv_sock.tcp_check_req 0.77 ± 2% +0.3 1.02 ± 6% perf-profile.calltrace.cycles-pp.inet_csk_clone_lock.tcp_create_openreq_child.tcp_v4_syn_recv_sock.tcp_check_req.tcp_v4_rcv 0.68 +0.3 0.94 ± 5% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established 0.89 ± 2% +0.3 1.15 ± 6% perf-profile.calltrace.cycles-pp.tcp_create_openreq_child.tcp_v4_syn_recv_sock.tcp_check_req.tcp_v4_rcv.ip_protocol_deliver_rcu 1.60 ± 2% +0.3 1.87 ± 5% perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg.inet_recvmsg.__sys_recvfrom 0.72 +0.3 1.00 ± 5% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv 1.69 +0.3 1.98 ± 4% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary 1.73 +0.3 2.02 ± 4% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 1.61 ± 2% +0.3 1.91 ± 5% perf-profile.calltrace.cycles-pp.tcp_v4_syn_recv_sock.tcp_check_req.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish 0.40 ± 57% +0.3 0.70 ± 5% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.inet_csk_accept.inet_accept 0.97 ± 2% +0.3 1.28 ± 6% perf-profile.calltrace.cycles-pp.inet_csk_accept.inet_accept.__sys_accept4_file.__sys_accept4.__x64_sys_accept 0.78 +0.3 1.08 ± 5% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv 0.83 +0.3 1.15 ± 5% perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu 1.54 +0.3 1.88 ± 5% perf-profile.calltrace.cycles-pp.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver 0.25 ±100% +0.4 0.65 ± 6% perf-profile.calltrace.cycles-pp.__sk_destruct.inet_release.__sock_release.sock_close.__fput 2.07 +0.5 2.53 ± 7% perf-profile.calltrace.cycles-pp.__x64_sys_socket.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.06 +0.5 2.52 ± 7% perf-profile.calltrace.cycles-pp.__sys_socket.__x64_sys_socket.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.13 ±173% +0.5 0.63 ± 6% perf-profile.calltrace.cycles-pp.security_file_free.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64 0.13 ±173% +0.5 0.63 ± 6% perf-profile.calltrace.cycles-pp.apparmor_file_free_security.security_file_free.__fput.task_work_run.exit_to_usermode_loop 4.47 +0.5 5.00 ± 5% perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_synsent_state_process.tcp_rcv_state_process.tcp_v4_do_rcv 0.00 +0.5 0.54 ± 4% perf-profile.calltrace.cycles-pp.tcp_time_wait.tcp_fin.tcp_data_queue.tcp_rcv_state_process.tcp_v4_do_rcv 4.25 +0.5 4.79 ± 5% perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_synsent_state_process 0.00 +0.6 0.55 ± 5% perf-profile.calltrace.cycles-pp.__sk_mem_reduce_allocated.tcp_close.inet_release.__sock_release.sock_close 0.00 +0.6 0.55 ± 4% perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 4.59 +0.6 5.15 ± 5% perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_rcv_synsent_state_process.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock 0.00 +0.6 0.56 ± 4% perf-profile.calltrace.cycles-pp.lock_sock_nested.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 4.33 +0.6 4.89 ± 5% perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_rcv_synsent_state_process.tcp_rcv_state_process 0.00 +0.6 0.58 ± 6% perf-profile.calltrace.cycles-pp.inet_csk_destroy_sock.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.tcp_close 0.00 +0.6 0.58 ± 9% perf-profile.calltrace.cycles-pp.tcp_done.tcp_fin.tcp_data_queue.tcp_rcv_state_process.tcp_v4_do_rcv 0.00 +0.6 0.64 ± 5% perf-profile.calltrace.cycles-pp.inet_csk_destroy_sock.tcp_fin.tcp_data_queue.tcp_rcv_state_process.tcp_v4_do_rcv 1.71 +0.6 2.35 ± 5% perf-profile.calltrace.cycles-pp.tcp_fin.tcp_data_queue.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv 1.74 +0.7 2.40 ± 5% perf-profile.calltrace.cycles-pp.tcp_data_queue.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu 0.00 +0.7 0.69 ± 32% perf-profile.calltrace.cycles-pp.__inet_lookup_established.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver 4.01 +0.8 4.79 ± 5% perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish 3.21 +0.8 4.04 ± 5% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 4.40 ± 2% +0.9 5.26 ± 4% perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64 4.42 ± 2% +0.9 5.29 ± 4% perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.31 ± 2% +0.9 2.25 ± 4% perf-profile.calltrace.cycles-pp.inet_accept.__sys_accept4_file.__sys_accept4.__x64_sys_accept.do_syscall_64 4.91 +1.0 5.91 ± 4% perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.89 +1.0 5.89 ± 4% perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 8.90 +1.1 9.95 ± 5% perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver 2.15 +1.2 3.37 ± 4% perf-profile.calltrace.cycles-pp.__sys_accept4_file.__sys_accept4.__x64_sys_accept.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.19 ± 2% +1.2 3.42 ± 4% perf-profile.calltrace.cycles-pp.__x64_sys_accept.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.18 ± 2% +1.2 3.41 ± 4% perf-profile.calltrace.cycles-pp.__sys_accept4.__x64_sys_accept.do_syscall_64.entry_SYSCALL_64_after_hwframe 7.99 +1.3 9.31 ± 4% perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_close.inet_release.__sock_release 6.61 +1.3 7.94 ± 5% perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_close 8.00 +1.3 9.33 ± 4% perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_close.inet_release.__sock_release.sock_close 6.86 +1.4 8.27 ± 4% perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_close.inet_release 11.56 +1.7 13.28 ± 4% perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit 11.70 +1.7 13.44 ± 4% perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames 10.78 +2.1 12.86 ± 4% perf-profile.calltrace.cycles-pp.tcp_close.inet_release.__sock_release.sock_close.__fput 11.34 +2.3 13.63 ± 5% perf-profile.calltrace.cycles-pp.inet_release.__sock_release.sock_close.__fput.task_work_run 11.45 +2.3 13.78 ± 5% perf-profile.calltrace.cycles-pp.__sock_release.sock_close.__fput.task_work_run.exit_to_usermode_loop 11.46 +2.3 13.80 ± 5% perf-profile.calltrace.cycles-pp.sock_close.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64 15.90 +2.4 18.32 ± 5% perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core 15.93 +2.4 18.36 ± 5% perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog 15.50 +2.5 17.96 ± 5% perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv 15.97 +2.5 18.45 ± 5% perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action 12.72 +2.7 15.47 ± 5% perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 12.89 +2.8 15.68 ± 5% perf-profile.calltrace.cycles-pp.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 13.00 +2.8 15.84 ± 5% perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 47.98 +5.6 53.61 ± 5% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 48.15 +5.7 53.81 ± 5% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 40.11 -6.5 33.62 ± 10% perf-profile.children.cycles-pp.intel_idle 50.63 -6.0 44.67 ± 6% perf-profile.children.cycles-pp.do_idle 50.65 -6.0 44.69 ± 6% perf-profile.children.cycles-pp.secondary_startup_64 50.65 -6.0 44.69 ± 6% perf-profile.children.cycles-pp.cpu_startup_entry 45.83 -6.0 39.88 ± 7% perf-profile.children.cycles-pp.cpuidle_enter 45.82 -6.0 39.87 ± 7% perf-profile.children.cycles-pp.cpuidle_enter_state 50.34 -5.9 44.40 ± 6% perf-profile.children.cycles-pp.start_secondary 1.83 -1.6 0.22 ± 6% perf-profile.children.cycles-pp.tcp_schedule_loss_probe 1.57 -1.3 0.30 ± 7% perf-profile.children.cycles-pp.ip_rcv_finish_core 1.59 -1.3 0.34 ± 6% perf-profile.children.cycles-pp.ip_rcv_finish 5.86 -0.8 5.06 ± 6% perf-profile.children.cycles-pp.tcp_connect 7.05 -0.7 6.31 ± 6% perf-profile.children.cycles-pp.tcp_v4_connect 2.92 -0.7 2.21 ± 5% perf-profile.children.cycles-pp.tcp_ack 0.68 -0.6 0.13 ± 5% perf-profile.children.cycles-pp.tcp_parse_options 1.77 ± 10% -0.5 1.22 perf-profile.children.cycles-pp.menu_select 0.56 ± 3% -0.4 0.14 ± 6% perf-profile.children.cycles-pp.tcp_sync_mss 0.45 -0.4 0.06 perf-profile.children.cycles-pp.tcp_stream_memory_free 2.21 -0.4 1.82 ± 6% perf-profile.children.cycles-pp.tcp_conn_request 0.46 ± 2% -0.4 0.08 perf-profile.children.cycles-pp.secure_tcp_ts_off 0.55 ± 3% -0.3 0.21 ± 8% perf-profile.children.cycles-pp.tcp_connect_init 0.33 ± 2% -0.3 0.07 ± 10% perf-profile.children.cycles-pp.tcp_mtup_init 0.17 -0.1 0.07 ± 17% perf-profile.children.cycles-pp.inet_lhash2_lookup 0.19 -0.1 0.11 ± 15% perf-profile.children.cycles-pp.__inet_lookup_listener 0.17 ± 7% -0.0 0.13 ± 9% perf-profile.children.cycles-pp.irq_enter 0.08 ± 10% -0.0 0.04 ± 58% perf-profile.children.cycles-pp.ip_build_and_send_pkt 0.14 ± 9% -0.0 0.10 ± 8% perf-profile.children.cycles-pp.tick_irq_enter 0.05 +0.0 0.06 perf-profile.children.cycles-pp.bictcp_acked 0.07 +0.0 0.08 ± 5% perf-profile.children.cycles-pp._atomic_dec_and_lock 0.05 ± 8% +0.0 0.07 ± 6% perf-profile.children.cycles-pp.__calc_delta 0.06 ± 11% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.tcp_release_cb 0.05 ± 8% +0.0 0.07 ± 10% perf-profile.children.cycles-pp.__cgroup_bpf_run_filter_sock_ops 0.05 ± 8% +0.0 0.07 perf-profile.children.cycles-pp.move_addr_to_kernel 0.05 +0.0 0.07 ± 6% perf-profile.children.cycles-pp.__list_del_entry_valid 0.05 ± 9% +0.0 0.07 ± 5% perf-profile.children.cycles-pp.account_entity_dequeue 0.06 ± 7% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.kfree_skbmem 0.11 ± 4% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.cgroup_rstat_updated 0.10 ± 5% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.tcp_current_mss 0.07 ± 11% +0.0 0.09 ± 4% perf-profile.children.cycles-pp._copy_to_user 0.11 +0.0 0.13 ± 5% perf-profile.children.cycles-pp._raw_spin_trylock 0.07 ± 5% +0.0 0.09 ± 8% perf-profile.children.cycles-pp.tcp_init_buffer_space 0.08 +0.0 0.10 perf-profile.children.cycles-pp.tcp_cleanup_rbuf 0.07 ± 5% +0.0 0.09 ± 8% perf-profile.children.cycles-pp.security_socket_bind 0.07 ± 5% +0.0 0.09 ± 8% perf-profile.children.cycles-pp.tcp_send_mss 0.07 ± 15% +0.0 0.09 ± 8% perf-profile.children.cycles-pp.tcp_openreq_init_rwin 0.06 ± 6% +0.0 0.08 ± 10% perf-profile.children.cycles-pp.skb_page_frag_refill 0.05 ± 9% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.tcp_check_space 0.08 ± 5% +0.0 0.10 ± 7% perf-profile.children.cycles-pp.rcu_segcblist_enqueue 0.06 ± 6% +0.0 0.09 ± 13% perf-profile.children.cycles-pp.sk_page_frag_refill 0.13 ± 5% +0.0 0.15 ± 7% perf-profile.children.cycles-pp.__cgroup_account_cputime 0.12 ± 5% +0.0 0.14 ± 5% perf-profile.children.cycles-pp.rcu_dynticks_eqs_exit 0.09 +0.0 0.11 ± 7% perf-profile.children.cycles-pp.inode_init_always 0.08 ± 6% +0.0 0.10 ± 7% perf-profile.children.cycles-pp.iput 0.04 ± 57% +0.0 0.06 ± 6% perf-profile.children.cycles-pp.put_prev_task_fair 0.06 ± 9% +0.0 0.08 ± 8% perf-profile.children.cycles-pp._raw_spin_lock_irq 0.06 ± 9% +0.0 0.08 ± 8% perf-profile.children.cycles-pp.fsnotify_destroy_marks 0.08 ± 5% +0.0 0.10 ± 4% perf-profile.children.cycles-pp.netif_skb_features 0.06 +0.0 0.09 ± 10% perf-profile.children.cycles-pp.rcu_all_qs 0.16 ± 4% +0.0 0.19 ± 6% perf-profile.children.cycles-pp.__update_load_avg_se 0.08 ± 5% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.memset_erms 0.04 ± 57% +0.0 0.07 ± 7% perf-profile.children.cycles-pp.kfree_skb 0.06 ± 9% +0.0 0.08 ± 5% perf-profile.children.cycles-pp.tcp_update_skb_after_send 0.04 ± 58% +0.0 0.07 ± 12% perf-profile.children.cycles-pp.del_timer_sync 0.06 ± 7% +0.0 0.09 ± 10% perf-profile.children.cycles-pp.move_addr_to_user 0.11 ± 7% +0.0 0.14 ± 3% perf-profile.children.cycles-pp.simple_copy_to_iter 0.18 ± 2% +0.0 0.21 ± 2% perf-profile.children.cycles-pp.__check_object_size 0.14 ± 3% +0.0 0.17 ± 5% perf-profile.children.cycles-pp.finish_task_switch 0.08 ± 5% +0.0 0.11 ± 9% perf-profile.children.cycles-pp.dput 0.07 ± 7% +0.0 0.10 ± 9% perf-profile.children.cycles-pp.kmem_cache_alloc_trace 0.07 ± 7% +0.0 0.10 ± 11% perf-profile.children.cycles-pp.tcp_queue_rcv 0.07 ± 12% +0.0 0.10 ± 8% perf-profile.children.cycles-pp.sock_setsockopt 0.08 ± 5% +0.0 0.12 ± 13% perf-profile.children.cycles-pp.raw_local_deliver 0.18 +0.0 0.21 ± 3% perf-profile.children.cycles-pp.pick_next_entity 0.15 ± 3% +0.0 0.18 ± 4% perf-profile.children.cycles-pp.account_entity_enqueue 0.14 ± 3% +0.0 0.17 ± 2% perf-profile.children.cycles-pp.tcp_rcv_space_adjust 0.14 ± 3% +0.0 0.17 ± 5% perf-profile.children.cycles-pp.scheduler_tick 0.13 ± 5% +0.0 0.16 ± 9% perf-profile.children.cycles-pp.sk_setup_caps 0.07 ± 5% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.tcp_init_xmit_timers 0.07 +0.0 0.10 ± 12% perf-profile.children.cycles-pp.inet_twsk_alloc 0.07 +0.0 0.10 ± 4% perf-profile.children.cycles-pp.nf_hook_slow 0.06 +0.0 0.09 ± 8% perf-profile.children.cycles-pp.__get_user_4 0.08 ± 6% +0.0 0.11 ± 6% perf-profile.children.cycles-pp.__destroy_inode 0.21 ± 6% +0.0 0.24 ± 3% perf-profile.children.cycles-pp.__x64_sys_close 0.03 ±100% +0.0 0.06 ± 6% perf-profile.children.cycles-pp.minmax_running_min 0.18 ± 2% +0.0 0.22 ± 3% perf-profile.children.cycles-pp.__netif_receive_skb_core 0.17 ± 2% +0.0 0.21 ± 5% perf-profile.children.cycles-pp.update_ts_time_stats 0.11 ± 4% +0.0 0.15 ± 5% perf-profile.children.cycles-pp.resched_curr 0.09 ± 4% +0.0 0.13 perf-profile.children.cycles-pp.sock_put 0.09 ± 7% +0.0 0.13 ± 6% perf-profile.children.cycles-pp.apparmor_sk_alloc_security 0.06 ± 7% +0.0 0.10 ± 5% perf-profile.children.cycles-pp.inet_sock_destruct 0.04 ± 57% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.rcu_eqs_enter 0.11 ± 6% +0.0 0.15 ± 8% perf-profile.children.cycles-pp.d_alloc_pseudo 0.18 ± 3% +0.0 0.22 ± 7% perf-profile.children.cycles-pp.reweight_entity 0.55 ± 4% +0.0 0.59 ± 3% perf-profile.children.cycles-pp.irq_exit 0.19 ± 2% +0.0 0.23 ± 4% perf-profile.children.cycles-pp.switch_mm_irqs_off 0.03 ±100% +0.0 0.07 ± 7% perf-profile.children.cycles-pp.__hrtimer_init 0.22 ± 3% +0.0 0.26 ± 5% perf-profile.children.cycles-pp.__kmalloc_reserve 0.15 ± 5% +0.0 0.19 ± 8% perf-profile.children.cycles-pp.fib_table_lookup 0.14 ± 7% +0.0 0.18 ± 5% perf-profile.children.cycles-pp.kfree 0.12 ± 3% +0.0 0.16 ± 8% perf-profile.children.cycles-pp.__fget_light 0.11 ± 4% +0.0 0.15 ± 7% perf-profile.children.cycles-pp.copyout 0.09 ± 4% +0.0 0.13 ± 3% perf-profile.children.cycles-pp.security_sk_alloc 0.11 ± 7% +0.0 0.15 ± 8% perf-profile.children.cycles-pp.__d_alloc 0.19 ± 3% +0.0 0.23 ± 7% perf-profile.children.cycles-pp.__kmalloc_node_track_caller 0.15 ± 7% +0.0 0.20 ± 2% perf-profile.children.cycles-pp.security_socket_getsockopt 0.08 ± 6% +0.0 0.12 ± 9% perf-profile.children.cycles-pp.destroy_inode 0.01 ±173% +0.0 0.05 ± 9% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 0.01 ±173% +0.0 0.05 ± 9% perf-profile.children.cycles-pp.tcp_v4_inbound_md5_hash 0.18 ± 2% +0.0 0.22 ± 5% perf-profile.children.cycles-pp.nr_iowait_cpu 0.15 ± 5% +0.0 0.19 ± 6% perf-profile.children.cycles-pp.sock_getsockopt 0.16 ± 4% +0.0 0.20 ± 7% perf-profile.children.cycles-pp.memcg_kmem_get_cache 0.12 ± 3% +0.0 0.17 ± 12% perf-profile.children.cycles-pp._cond_resched 0.14 ± 3% +0.0 0.18 ± 6% perf-profile.children.cycles-pp.__indirect_thunk_start 0.13 +0.0 0.17 ± 2% perf-profile.children.cycles-pp.__might_fault 0.13 ± 7% +0.0 0.17 ± 7% perf-profile.children.cycles-pp.sock_alloc_inode 0.10 ± 8% +0.0 0.15 ± 3% perf-profile.children.cycles-pp.d_instantiate 0.27 ± 2% +0.0 0.32 ± 4% perf-profile.children.cycles-pp.tick_nohz_idle_exit 0.04 ± 57% +0.0 0.08 ± 10% perf-profile.children.cycles-pp.tcp_rearm_rto 0.17 ± 2% +0.0 0.21 ± 9% perf-profile.children.cycles-pp.copy_user_generic_unrolled 0.16 ± 2% +0.0 0.20 ± 5% perf-profile.children.cycles-pp.validate_xmit_skb 0.14 ± 5% +0.0 0.19 ± 6% perf-profile.children.cycles-pp._copy_to_iter 0.13 ± 3% +0.0 0.17 ± 4% perf-profile.children.cycles-pp.sockfd_lookup_light 0.01 ±173% +0.0 0.06 perf-profile.children.cycles-pp.menu_reflect 0.16 +0.0 0.21 ± 5% perf-profile.children.cycles-pp.ttwu_do_wakeup 0.12 +0.0 0.17 ± 4% perf-profile.children.cycles-pp.ip_send_check 0.18 ± 2% +0.0 0.23 ± 8% perf-profile.children.cycles-pp.sock_rfree 0.20 ± 2% +0.0 0.24 ± 6% perf-profile.children.cycles-pp.__ksize 0.18 ± 6% +0.0 0.23 ± 8% perf-profile.children.cycles-pp.rcu_do_batch 0.10 ± 4% +0.0 0.15 ± 11% perf-profile.children.cycles-pp.run_ksoftirqd 0.01 ±173% +0.1 0.06 ± 6% perf-profile.children.cycles-pp.security_sock_rcv_skb 0.00 +0.1 0.05 perf-profile.children.cycles-pp.tcp_select_initial_window 0.03 ±100% +0.1 0.08 ± 11% perf-profile.children.cycles-pp.fsnotify_grab_connector 0.15 +0.1 0.20 ± 3% perf-profile.children.cycles-pp.check_preempt_curr 0.21 ± 3% +0.1 0.26 ± 4% perf-profile.children.cycles-pp.call_rcu 0.13 ± 3% +0.1 0.18 ± 2% perf-profile.children.cycles-pp.ip_finish_output 0.00 +0.1 0.05 ± 8% perf-profile.children.cycles-pp.ip_queue_xmit 0.17 ± 3% +0.1 0.22 ± 5% perf-profile.children.cycles-pp.__x64_sys_setsockopt 0.11 ± 3% +0.1 0.17 ± 10% perf-profile.children.cycles-pp.smpboot_thread_fn 0.23 ± 5% +0.1 0.28 ± 6% perf-profile.children.cycles-pp.rcu_eqs_exit 0.28 ± 2% +0.1 0.33 ± 6% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.26 ± 2% +0.1 0.32 ± 6% perf-profile.children.cycles-pp.__switch_to_asm 0.19 ± 2% +0.1 0.25 ± 5% perf-profile.children.cycles-pp.sk_filter_trim_cap 0.00 +0.1 0.05 ± 9% perf-profile.children.cycles-pp.inet_sk_rx_dst_set 0.00 +0.1 0.05 ± 9% perf-profile.children.cycles-pp.try_to_del_timer_sync 0.03 ±100% +0.1 0.08 ± 8% perf-profile.children.cycles-pp.tcp_synack_rtt_meas 0.01 ±173% +0.1 0.07 ± 6% perf-profile.children.cycles-pp.__d_instantiate 0.13 ± 5% +0.1 0.19 ± 11% perf-profile.children.cycles-pp.ret_from_fork 0.13 ± 5% +0.1 0.19 ± 11% perf-profile.children.cycles-pp.kthread 0.20 ± 5% +0.1 0.26 ± 8% perf-profile.children.cycles-pp.__inet_inherit_port 0.19 ± 3% +0.1 0.25 ± 8% perf-profile.children.cycles-pp.entry_SYSCALL_64 0.15 ± 3% +0.1 0.20 ± 6% perf-profile.children.cycles-pp.ipv4_mtu 0.14 ± 6% +0.1 0.19 ± 4% perf-profile.children.cycles-pp.sk_stream_kill_queues 0.00 +0.1 0.06 ± 7% perf-profile.children.cycles-pp.__inet_twsk_schedule 0.15 ± 2% +0.1 0.21 ± 3% perf-profile.children.cycles-pp.__ip_local_out 0.16 ± 5% +0.1 0.22 ± 6% perf-profile.children.cycles-pp.del_timer 0.27 ± 4% +0.1 0.33 ± 5% perf-profile.children.cycles-pp.__tcp_send_ack 0.16 ± 2% +0.1 0.22 ± 5% perf-profile.children.cycles-pp.__sys_setsockopt 0.27 +0.1 0.33 ± 2% perf-profile.children.cycles-pp.tick_sched_handle 0.26 ± 5% +0.1 0.32 ± 7% perf-profile.children.cycles-pp.rcu_idle_exit 0.16 ± 4% +0.1 0.22 ± 3% perf-profile.children.cycles-pp.__might_sleep 0.00 +0.1 0.06 ± 6% perf-profile.children.cycles-pp.__list_add_valid 0.00 +0.1 0.06 ± 13% perf-profile.children.cycles-pp.calc_global_load_tick 0.12 ± 3% +0.1 0.18 ± 8% perf-profile.children.cycles-pp.evict 0.22 +0.1 0.29 ± 6% perf-profile.children.cycles-pp.security_socket_sendmsg 0.17 ± 3% +0.1 0.23 ± 3% perf-profile.children.cycles-pp.ip_local_out 0.14 ± 3% +0.1 0.21 ± 8% perf-profile.children.cycles-pp.kmem_cache_alloc_node 0.00 +0.1 0.07 ± 7% perf-profile.children.cycles-pp.inet_ehash_nolisten 0.00 +0.1 0.07 ± 7% perf-profile.children.cycles-pp.tcp_rack_update_reo_wnd 0.20 ± 4% +0.1 0.27 ± 6% perf-profile.children.cycles-pp.irqtime_account_irq 0.18 ± 2% +0.1 0.25 ± 2% perf-profile.children.cycles-pp.__cgroup_bpf_run_filter_skb 0.00 +0.1 0.07 ± 12% perf-profile.children.cycles-pp.skb_entail 0.00 +0.1 0.07 ± 6% perf-profile.children.cycles-pp.ip_copy_addrs 0.21 ± 2% +0.1 0.28 ± 20% perf-profile.children.cycles-pp.ktime_get_with_offset 0.31 ± 2% +0.1 0.38 ± 5% perf-profile.children.cycles-pp.native_sched_clock 0.26 ± 2% +0.1 0.33 ± 2% perf-profile.children.cycles-pp.update_process_times 0.20 ± 2% +0.1 0.27 ± 4% perf-profile.children.cycles-pp.__skb_clone 0.00 +0.1 0.07 ± 10% perf-profile.children.cycles-pp.ipv4_default_advmss 0.26 ± 4% +0.1 0.33 ± 7% perf-profile.children.cycles-pp.rcu_core 0.23 ± 2% +0.1 0.30 ± 6% perf-profile.children.cycles-pp.enqueue_to_backlog 0.22 ± 3% +0.1 0.29 ± 7% perf-profile.children.cycles-pp.alloc_inode 0.33 ± 3% +0.1 0.40 ± 6% perf-profile.children.cycles-pp.sched_clock 0.00 +0.1 0.07 ± 5% perf-profile.children.cycles-pp.tcp_assign_congestion_control 0.34 +0.1 0.42 ± 5% perf-profile.children.cycles-pp.tcp_mstamp_refresh 0.31 ± 6% +0.1 0.39 ± 6% perf-profile.children.cycles-pp.security_sk_clone 0.08 ± 5% +0.1 0.16 ± 4% perf-profile.children.cycles-pp.inet_sk_rebuild_header 0.19 ± 4% +0.1 0.27 ± 9% perf-profile.children.cycles-pp.inet_csk_clear_xmit_timers 0.31 ± 6% +0.1 0.39 ± 6% perf-profile.children.cycles-pp.apparmor_sk_clone_security 0.01 ±173% +0.1 0.09 ± 11% perf-profile.children.cycles-pp.inet_reqsk_alloc 0.26 ± 2% +0.1 0.34 ± 3% perf-profile.children.cycles-pp.syscall_return_via_sysret 0.07 ± 6% +0.1 0.15 ± 5% perf-profile.children.cycles-pp.tcp_update_pacing_rate 0.38 ± 4% +0.1 0.46 ± 3% perf-profile.children.cycles-pp.update_curr 0.04 ± 57% +0.1 0.12 ± 18% perf-profile.children.cycles-pp.inet_twsk_hashdance 0.36 ± 2% +0.1 0.44 ± 6% perf-profile.children.cycles-pp.sched_clock_cpu 0.35 ± 3% +0.1 0.44 ± 2% perf-profile.children.cycles-pp.tick_sched_timer 0.23 ± 7% +0.1 0.32 ± 7% perf-profile.children.cycles-pp.inet_unhash 0.19 +0.1 0.27 ± 6% perf-profile.children.cycles-pp.inet_ehash_insert 0.22 ± 5% +0.1 0.30 ± 7% perf-profile.children.cycles-pp.sk_stop_timer 0.20 ± 4% +0.1 0.28 ± 5% perf-profile.children.cycles-pp.sk_prot_alloc 0.28 ± 4% +0.1 0.36 ± 6% perf-profile.children.cycles-pp.apparmor_sk_free_security 0.46 ± 3% +0.1 0.55 ± 2% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.28 +0.1 0.37 ± 5% perf-profile.children.cycles-pp.inet_csk_reqsk_queue_hash_add 0.06 ± 9% +0.1 0.15 ± 7% perf-profile.children.cycles-pp.inet_lookup_ifaddr_rcu 0.66 +0.1 0.75 ± 4% perf-profile.children.cycles-pp.set_next_entity 0.54 ± 2% +0.1 0.63 ± 6% perf-profile.children.cycles-pp.ip_route_output_flow 0.36 ± 2% +0.1 0.45 ± 3% perf-profile.children.cycles-pp.security_socket_recvmsg 0.22 ± 5% +0.1 0.31 ± 6% perf-profile.children.cycles-pp.__slab_free 0.07 ± 7% +0.1 0.16 ± 6% perf-profile.children.cycles-pp.__ip_dev_find 0.00 +0.1 0.09 ± 4% perf-profile.children.cycles-pp.sock_prot_inuse_add 0.37 ± 2% +0.1 0.46 ± 6% perf-profile.children.cycles-pp.sk_alloc 0.40 ± 3% +0.1 0.49 ± 5% perf-profile.children.cycles-pp.__skb_datagram_iter 0.37 +0.1 0.46 ± 7% perf-profile.children.cycles-pp.___might_sleep 0.27 ± 3% +0.1 0.36 ± 7% perf-profile.children.cycles-pp.tcp_write_queue_purge 0.34 ± 3% +0.1 0.44 ± 6% perf-profile.children.cycles-pp.apparmor_socket_post_create 0.32 ± 2% +0.1 0.42 ± 5% perf-profile.children.cycles-pp.skb_release_head_state 0.36 +0.1 0.46 ± 2% perf-profile.children.cycles-pp.sock_recvmsg 0.34 ± 3% +0.1 0.44 ± 6% perf-profile.children.cycles-pp.security_socket_post_create 0.29 ± 5% +0.1 0.38 ± 8% perf-profile.children.cycles-pp.__inet_check_established 0.00 +0.1 0.10 ± 4% perf-profile.children.cycles-pp.__sk_free 0.11 ± 4% +0.1 0.21 ± 10% perf-profile.children.cycles-pp.inet_twsk_kill 0.41 ± 2% +0.1 0.50 ± 5% perf-profile.children.cycles-pp.skb_copy_datagram_iter 0.37 ± 5% +0.1 0.47 ± 5% perf-profile.children.cycles-pp.clockevents_program_event 0.79 +0.1 0.89 ± 6% perf-profile.children.cycles-pp.ip_route_output_key_hash 0.29 ± 3% +0.1 0.39 ± 6% perf-profile.children.cycles-pp.security_sk_free 0.34 +0.1 0.44 ± 5% perf-profile.children.cycles-pp.skb_release_all 0.46 +0.1 0.56 ± 2% perf-profile.children.cycles-pp.dst_release 0.11 ± 4% +0.1 0.21 ± 5% perf-profile.children.cycles-pp.tcp_ack_update_rtt 0.36 ± 2% +0.1 0.46 ± 4% perf-profile.children.cycles-pp.skb_release_data 0.58 +0.1 0.68 ± 7% perf-profile.children.cycles-pp.update_rq_clock 0.29 ± 3% +0.1 0.40 ± 7% perf-profile.children.cycles-pp.new_inode_pseudo 0.12 +0.1 0.23 ± 7% perf-profile.children.cycles-pp.inet_twsk_deschedule_put 0.75 +0.1 0.86 ± 6% perf-profile.children.cycles-pp.update_load_avg 0.30 ± 3% +0.1 0.41 ± 6% perf-profile.children.cycles-pp.sock_alloc 0.23 +0.1 0.34 ± 13% perf-profile.children.cycles-pp.inet_put_port 0.40 ± 4% +0.1 0.52 ± 5% perf-profile.children.cycles-pp.__sys_getsockopt 0.31 ± 2% +0.1 0.43 ± 5% perf-profile.children.cycles-pp.tcp_finish_connect 0.00 +0.1 0.12 ± 5% perf-profile.children.cycles-pp.__tcp_select_window 0.50 ± 2% +0.1 0.63 ± 4% perf-profile.children.cycles-pp.tcp_event_new_data_sent 0.40 ± 4% +0.1 0.53 ± 5% perf-profile.children.cycles-pp.__x64_sys_getsockopt 0.34 ± 2% +0.1 0.46 ± 3% perf-profile.children.cycles-pp.lock_timer_base 0.42 ± 3% +0.1 0.55 ± 11% perf-profile.children.cycles-pp.netif_rx_internal 0.36 ± 3% +0.1 0.50 ± 6% perf-profile.children.cycles-pp.kmem_cache_free 0.42 ± 3% +0.1 0.56 ± 11% perf-profile.children.cycles-pp.netif_rx 0.38 ± 4% +0.1 0.51 ± 8% perf-profile.children.cycles-pp.inet_csk_bind_conflict 0.42 ± 3% +0.1 0.55 ± 7% perf-profile.children.cycles-pp.kmem_cache_alloc 0.93 ± 2% +0.1 1.07 ± 7% perf-profile.children.cycles-pp.__x64_sys_bind 0.49 ± 2% +0.1 0.63 ± 6% perf-profile.children.cycles-pp.security_file_free 0.93 +0.1 1.06 ± 6% perf-profile.children.cycles-pp.__sys_bind 0.32 ± 5% +0.1 0.46 ± 7% perf-profile.children.cycles-pp.__inet_hash_connect 0.49 ± 3% +0.1 0.63 ± 6% perf-profile.children.cycles-pp.apparmor_file_free_security 0.95 +0.1 1.10 ± 3% perf-profile.children.cycles-pp.pick_next_task_fair 1.27 +0.1 1.42 ± 4% perf-profile.children.cycles-pp.ktime_get 0.50 +0.2 0.65 ± 6% perf-profile.children.cycles-pp.read_tsc 0.59 ± 3% +0.2 0.75 ± 9% perf-profile.children.cycles-pp.tcp_set_state 0.88 ± 2% +0.2 1.04 ± 6% perf-profile.children.cycles-pp.enqueue_entity 0.52 ± 3% +0.2 0.68 ± 2% perf-profile.children.cycles-pp.mod_timer 0.42 ± 4% +0.2 0.58 ± 6% perf-profile.children.cycles-pp.tcp_v4_destroy_sock 0.59 ± 3% +0.2 0.76 ± 2% perf-profile.children.cycles-pp.sk_reset_timer 0.47 ± 2% +0.2 0.65 ± 6% perf-profile.children.cycles-pp.sk_stream_alloc_skb 0.21 ± 2% +0.2 0.38 ± 4% perf-profile.children.cycles-pp.__sk_dst_check 0.80 ± 3% +0.2 0.98 ± 4% perf-profile.children.cycles-pp.dequeue_entity 0.76 +0.2 0.94 ± 6% perf-profile.children.cycles-pp.autoremove_wake_function 0.62 ± 2% +0.2 0.80 ± 6% perf-profile.children.cycles-pp.tcp_init_transfer 0.52 ± 2% +0.2 0.70 ± 6% perf-profile.children.cycles-pp.__dentry_kill 0.54 ± 3% +0.2 0.73 ± 7% perf-profile.children.cycles-pp.apparmor_file_alloc_security 0.21 ± 2% +0.2 0.41 ± 5% perf-profile.children.cycles-pp.ipv4_dst_check 0.94 ± 2% +0.2 1.14 ± 3% perf-profile.children.cycles-pp.hrtimer_interrupt 1.10 +0.2 1.29 ± 6% perf-profile.children.cycles-pp.enqueue_task_fair 0.88 ± 3% +0.2 1.07 ± 4% perf-profile.children.cycles-pp.dequeue_task_fair 1.11 +0.2 1.30 ± 6% perf-profile.children.cycles-pp.activate_task 1.12 +0.2 1.31 ± 6% perf-profile.children.cycles-pp.ttwu_do_activate 1.68 ± 2% +0.2 1.88 ± 3% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 0.34 ± 2% +0.2 0.54 ± 4% perf-profile.children.cycles-pp.tcp_update_metrics 0.60 ± 2% +0.2 0.81 ± 7% perf-profile.children.cycles-pp.security_file_alloc 0.33 ± 2% +0.2 0.54 ± 4% perf-profile.children.cycles-pp.tcp_time_wait 0.72 ± 2% +0.2 0.94 ± 6% perf-profile.children.cycles-pp.__alloc_skb 0.47 +0.2 0.69 ± 8% perf-profile.children.cycles-pp.__tcp_get_metrics 0.97 +0.2 1.19 ± 4% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.58 ± 2% +0.2 0.81 ± 4% perf-profile.children.cycles-pp.tcp_send_fin 0.91 ± 2% +0.2 1.14 ± 3% perf-profile.children.cycles-pp.aa_sk_perm 0.28 ± 3% +0.2 0.52 ± 7% perf-profile.children.cycles-pp.tcp_init_metrics 1.49 +0.2 1.73 ± 6% perf-profile.children.cycles-pp.__sock_create 0.81 +0.2 1.05 ± 9% perf-profile.children.cycles-pp.loopback_xmit 0.75 ± 3% +0.2 1.00 ± 6% perf-profile.children.cycles-pp.sk_clone_lock 0.84 +0.3 1.10 ± 4% perf-profile.children.cycles-pp.__kfree_skb 0.77 ± 2% +0.3 1.02 ± 6% perf-profile.children.cycles-pp.inet_csk_clone_lock 0.85 ± 2% +0.3 1.10 ± 5% perf-profile.children.cycles-pp.sock_def_wakeup 0.67 ± 4% +0.3 0.93 ± 9% perf-profile.children.cycles-pp.tcp_done 0.89 ± 2% +0.3 1.15 ± 6% perf-profile.children.cycles-pp.tcp_create_openreq_child 0.69 ± 2% +0.3 0.96 ± 5% perf-profile.children.cycles-pp.__sk_mem_reduce_allocated 0.86 +0.3 1.14 ± 9% perf-profile.children.cycles-pp.dev_hard_start_xmit 1.62 ± 2% +0.3 1.90 ± 4% perf-profile.children.cycles-pp.wait_woken 0.79 +0.3 1.07 ± 7% perf-profile.children.cycles-pp.alloc_empty_file 0.80 +0.3 1.08 ± 7% perf-profile.children.cycles-pp.alloc_file 0.78 +0.3 1.06 ± 7% perf-profile.children.cycles-pp.__alloc_file 0.62 +0.3 0.91 ± 5% perf-profile.children.cycles-pp.percpu_counter_add_batch 1.75 +0.3 2.04 ± 4% perf-profile.children.cycles-pp.schedule_idle 0.98 ± 2% +0.3 1.29 ± 6% perf-profile.children.cycles-pp.inet_csk_accept 1.61 ± 2% +0.3 1.92 ± 5% perf-profile.children.cycles-pp.tcp_v4_syn_recv_sock 0.80 ± 2% +0.3 1.12 ± 3% perf-profile.children.cycles-pp.tcp_clean_rtx_queue 2.27 +0.3 2.59 ± 5% perf-profile.children.cycles-pp.sock_def_readable 1.03 ± 2% +0.3 1.36 ± 5% perf-profile.children.cycles-pp.__sk_destruct 0.61 ± 4% +0.3 0.94 ± 5% perf-profile.children.cycles-pp.sk_forced_mem_schedule 1.54 +0.3 1.88 ± 5% perf-profile.children.cycles-pp.tcp_child_process 0.53 +0.4 0.88 ± 7% perf-profile.children.cycles-pp.tcp_get_metrics 0.87 ± 2% +0.4 1.23 ± 5% perf-profile.children.cycles-pp.inet_csk_destroy_sock 1.04 +0.4 1.42 ± 7% perf-profile.children.cycles-pp.alloc_file_pseudo 1.04 +0.4 1.43 ± 7% perf-profile.children.cycles-pp.sock_alloc_file 0.19 ± 4% +0.4 0.57 ± 7% perf-profile.children.cycles-pp.inet_ehashfn 1.86 ± 2% +0.4 2.28 ± 5% perf-profile.children.cycles-pp.schedule 1.96 ± 2% +0.4 2.38 ± 5% perf-profile.children.cycles-pp.schedule_timeout 2.06 +0.5 2.52 ± 7% perf-profile.children.cycles-pp.__sys_socket 2.07 +0.5 2.53 ± 7% perf-profile.children.cycles-pp.__x64_sys_socket 2.49 +0.5 2.96 ± 6% perf-profile.children.cycles-pp.try_to_wake_up 1.38 +0.5 1.86 ± 7% perf-profile.children.cycles-pp.__dev_queue_xmit 2.63 +0.5 3.12 ± 5% perf-profile.children.cycles-pp.__wake_up_common 0.83 ± 4% +0.5 1.34 ± 3% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 1.77 +0.5 2.29 ± 5% perf-profile.children.cycles-pp._raw_spin_lock 2.90 +0.5 3.43 ± 5% perf-profile.children.cycles-pp.__wake_up_common_lock 1.02 +0.6 1.57 ± 6% perf-profile.children.cycles-pp.__inet_lookup_established 1.06 ± 2% +0.6 1.61 ± 4% perf-profile.children.cycles-pp.lock_sock_nested 1.33 ± 3% +0.7 2.00 ± 4% perf-profile.children.cycles-pp._raw_spin_lock_bh 3.53 +0.7 4.22 ± 4% perf-profile.children.cycles-pp.__schedule 3.26 ± 2% +0.8 4.11 ± 5% perf-profile.children.cycles-pp.poll_idle 4.41 ± 2% +0.9 5.27 ± 4% perf-profile.children.cycles-pp.tcp_recvmsg 4.42 ± 2% +0.9 5.29 ± 4% perf-profile.children.cycles-pp.inet_recvmsg 3.57 +0.9 4.44 ± 5% perf-profile.children.cycles-pp.tcp_data_queue 2.54 +0.9 3.42 ± 5% perf-profile.children.cycles-pp.tcp_fin 1.31 ± 2% +0.9 2.25 ± 4% perf-profile.children.cycles-pp.inet_accept 4.89 +1.0 5.90 ± 4% perf-profile.children.cycles-pp.__sys_recvfrom 4.91 +1.0 5.92 ± 4% perf-profile.children.cycles-pp.__x64_sys_recvfrom 2.16 +1.2 3.38 ± 4% perf-profile.children.cycles-pp.__sys_accept4_file 2.18 ± 2% +1.2 3.41 ± 4% perf-profile.children.cycles-pp.__sys_accept4 2.19 ± 2% +1.2 3.42 ± 4% perf-profile.children.cycles-pp.__x64_sys_accept 15.09 +1.7 16.81 ± 4% perf-profile.children.cycles-pp.tcp_write_xmit 15.11 +1.7 16.84 ± 4% perf-profile.children.cycles-pp.__tcp_push_pending_frames 10.78 +2.1 12.88 ± 4% perf-profile.children.cycles-pp.tcp_close 11.34 +2.3 13.64 ± 5% perf-profile.children.cycles-pp.inet_release 11.45 +2.3 13.78 ± 5% perf-profile.children.cycles-pp.__sock_release 11.46 +2.3 13.81 ± 5% perf-profile.children.cycles-pp.sock_close 16.08 +2.5 18.54 ± 5% perf-profile.children.cycles-pp.ip_protocol_deliver_rcu 16.10 +2.5 18.57 ± 5% perf-profile.children.cycles-pp.ip_local_deliver_finish 15.69 +2.5 18.18 ± 5% perf-profile.children.cycles-pp.tcp_v4_rcv 16.15 +2.5 18.67 ± 5% perf-profile.children.cycles-pp.ip_local_deliver 12.72 +2.8 15.48 ± 5% perf-profile.children.cycles-pp.__fput 12.89 +2.8 15.69 ± 5% perf-profile.children.cycles-pp.task_work_run 13.02 +2.8 15.86 ± 5% perf-profile.children.cycles-pp.exit_to_usermode_loop 48.04 +5.6 53.68 ± 5% perf-profile.children.cycles-pp.do_syscall_64 48.20 +5.7 53.88 ± 5% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 40.11 -6.5 33.61 ± 10% perf-profile.self.cycles-pp.intel_idle 1.79 -1.6 0.19 ± 6% perf-profile.self.cycles-pp.tcp_schedule_loss_probe 1.56 -1.3 0.30 ± 6% perf-profile.self.cycles-pp.ip_rcv_finish_core 1.36 ± 10% -0.6 0.79 ± 3% perf-profile.self.cycles-pp.menu_select 0.68 ± 2% -0.6 0.13 ± 5% perf-profile.self.cycles-pp.tcp_parse_options 0.94 ± 7% -0.5 0.46 ± 4% perf-profile.self.cycles-pp.cpuidle_enter_state 0.64 ± 2% -0.5 0.18 ± 10% perf-profile.self.cycles-pp.tcp_conn_request 0.55 ± 3% -0.4 0.13 ± 5% perf-profile.self.cycles-pp.tcp_sync_mss 0.44 ± 2% -0.4 0.03 ±100% perf-profile.self.cycles-pp.secure_tcp_ts_off 0.45 ± 2% -0.4 0.06 ± 7% perf-profile.self.cycles-pp.tcp_stream_memory_free 0.35 ± 2% -0.3 0.06 ± 13% perf-profile.self.cycles-pp.tcp_connect_init 0.33 ± 2% -0.3 0.07 ± 12% perf-profile.self.cycles-pp.tcp_mtup_init 0.77 -0.2 0.53 ± 8% perf-profile.self.cycles-pp.__ip_queue_xmit 1.08 -0.2 0.90 ± 5% perf-profile.self.cycles-pp.__tcp_transmit_skb 0.23 ± 3% -0.2 0.05 ± 9% perf-profile.self.cycles-pp.tcp_init_sock 0.17 -0.1 0.06 ± 20% perf-profile.self.cycles-pp.inet_lhash2_lookup 0.19 ± 3% -0.1 0.10 ± 15% perf-profile.self.cycles-pp.inet_csk_get_port 0.31 ± 2% -0.1 0.24 ± 7% perf-profile.self.cycles-pp.ip_protocol_deliver_rcu 0.61 ± 2% -0.0 0.57 ± 5% perf-profile.self.cycles-pp.tcp_ack 0.08 ± 8% -0.0 0.04 ± 58% perf-profile.self.cycles-pp.ip_build_and_send_pkt 0.15 -0.0 0.12 ± 7% perf-profile.self.cycles-pp.tcp_connect 0.20 ± 4% -0.0 0.17 ± 4% perf-profile.self.cycles-pp.__ip_finish_output 0.15 ± 2% -0.0 0.13 ± 10% perf-profile.self.cycles-pp.tcp_newly_delivered 0.13 ± 3% -0.0 0.11 ± 7% perf-profile.self.cycles-pp.tcp_set_state 0.11 ± 4% -0.0 0.09 ± 5% perf-profile.self.cycles-pp.__inet_bind 0.05 ± 8% +0.0 0.07 ± 6% perf-profile.self.cycles-pp.ip_rcv 0.05 ± 8% +0.0 0.07 ± 6% perf-profile.self.cycles-pp.__calc_delta 0.05 ± 8% +0.0 0.07 ± 6% perf-profile.self.cycles-pp.__sock_wfree 0.13 ± 5% +0.0 0.15 ± 3% perf-profile.self.cycles-pp.account_entity_enqueue 0.06 ± 7% +0.0 0.07 ± 5% perf-profile.self.cycles-pp.schedule 0.06 +0.0 0.08 ± 5% perf-profile.self.cycles-pp.__netif_receive_skb_one_core 0.06 ± 6% +0.0 0.08 ± 8% perf-profile.self.cycles-pp.skb_page_frag_refill 0.05 +0.0 0.07 ± 12% perf-profile.self.cycles-pp.lock_sock_nested 0.05 ± 8% +0.0 0.07 perf-profile.self.cycles-pp.__might_fault 0.06 ± 9% +0.0 0.07 ± 5% perf-profile.self.cycles-pp.sock_def_wakeup 0.13 ± 6% +0.0 0.15 ± 2% perf-profile.self.cycles-pp.update_curr 0.11 ± 4% +0.0 0.12 ± 4% perf-profile.self.cycles-pp.cgroup_rstat_updated 0.08 ± 5% +0.0 0.10 ± 4% perf-profile.self.cycles-pp.tcp_cleanup_rbuf 0.11 +0.0 0.13 ± 5% perf-profile.self.cycles-pp._raw_spin_trylock 0.08 ± 5% +0.0 0.10 ± 8% perf-profile.self.cycles-pp.rcu_segcblist_enqueue 0.07 ± 7% +0.0 0.09 ± 5% perf-profile.self.cycles-pp.sk_filter_trim_cap 0.05 ± 8% +0.0 0.07 ± 5% perf-profile.self.cycles-pp.tcp_check_space 0.05 ± 8% +0.0 0.07 ± 5% perf-profile.self.cycles-pp.tcp_release_cb 0.30 ± 3% +0.0 0.33 perf-profile.self.cycles-pp.ip_output 0.10 +0.0 0.12 ± 6% perf-profile.self.cycles-pp.finish_task_switch 0.08 ± 6% +0.0 0.10 ± 4% perf-profile.self.cycles-pp.validate_xmit_skb 0.08 ± 6% +0.0 0.10 ± 4% perf-profile.self.cycles-pp.tcp_rcv_space_adjust 0.04 ± 57% +0.0 0.06 perf-profile.self.cycles-pp.update_min_vruntime 0.07 ± 6% +0.0 0.09 ± 7% perf-profile.self.cycles-pp.tcp_v4_do_rcv 0.16 ± 4% +0.0 0.18 ± 4% perf-profile.self.cycles-pp.__update_load_avg_se 0.12 ± 5% +0.0 0.14 ± 5% perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit 0.08 ± 5% +0.0 0.10 ± 8% perf-profile.self.cycles-pp.memset_erms 0.05 ± 8% +0.0 0.08 ± 5% perf-profile.self.cycles-pp.tcp_data_queue 0.08 ± 5% +0.0 0.10 ± 4% perf-profile.self.cycles-pp.release_sock 0.06 ± 7% +0.0 0.08 ± 10% perf-profile.self.cycles-pp.inode_init_always 0.14 ± 6% +0.0 0.17 ± 3% perf-profile.self.cycles-pp.try_to_wake_up 0.10 ± 4% +0.0 0.12 ± 4% perf-profile.self.cycles-pp.__indirect_thunk_start 0.05 ± 8% +0.0 0.08 ± 8% perf-profile.self.cycles-pp._raw_spin_lock_irq 0.08 ± 5% +0.0 0.11 ± 11% perf-profile.self.cycles-pp.raw_local_deliver 0.07 +0.0 0.10 ± 11% perf-profile.self.cycles-pp.inet_csk_accept 0.07 ± 6% +0.0 0.10 ± 9% perf-profile.self.cycles-pp.inet_twsk_alloc 0.06 +0.0 0.09 ± 4% perf-profile.self.cycles-pp.__get_user_4 0.07 ± 7% +0.0 0.09 ± 8% perf-profile.self.cycles-pp.tcp_queue_rcv 0.11 ± 4% +0.0 0.14 ± 3% perf-profile.self.cycles-pp.sk_reset_timer 0.11 ± 4% +0.0 0.14 ± 7% perf-profile.self.cycles-pp.rcu_eqs_exit 0.11 ± 7% +0.0 0.14 ± 3% perf-profile.self.cycles-pp.__check_object_size 0.13 ± 3% +0.0 0.16 ± 6% perf-profile.self.cycles-pp.sk_setup_caps 0.11 ± 7% +0.0 0.15 ± 7% perf-profile.self.cycles-pp.__kmalloc_node_track_caller 0.17 +0.0 0.20 ± 4% perf-profile.self.cycles-pp.pick_next_entity 0.17 ± 6% +0.0 0.20 ± 5% perf-profile.self.cycles-pp.sock_def_readable 0.11 ± 4% +0.0 0.14 ± 3% perf-profile.self.cycles-pp.call_rcu 0.08 ± 10% +0.0 0.11 ± 11% perf-profile.self.cycles-pp.__alloc_file 0.03 ±100% +0.0 0.06 perf-profile.self.cycles-pp.__list_del_entry_valid 0.22 ± 3% +0.0 0.26 ± 3% perf-profile.self.cycles-pp.tcp_rcv_established 0.15 ± 5% +0.0 0.18 ± 6% perf-profile.self.cycles-pp.fib_table_lookup 0.14 ± 6% +0.0 0.17 ± 6% perf-profile.self.cycles-pp.kfree 0.12 ± 5% +0.0 0.15 ± 7% perf-profile.self.cycles-pp.do_softirq 0.09 ± 4% +0.0 0.13 ± 3% perf-profile.self.cycles-pp.sock_put 0.04 ± 57% +0.0 0.07 ± 5% perf-profile.self.cycles-pp.rcu_eqs_enter 0.03 ±100% +0.0 0.06 ± 6% perf-profile.self.cycles-pp.minmax_running_min 0.17 ± 3% +0.0 0.20 ± 4% perf-profile.self.cycles-pp.select_task_rq_fair 0.15 ± 3% +0.0 0.18 ± 7% perf-profile.self.cycles-pp.enqueue_to_backlog 0.12 ± 6% +0.0 0.16 ± 2% perf-profile.self.cycles-pp.net_rx_action 0.11 ± 3% +0.0 0.15 ± 8% perf-profile.self.cycles-pp.__fget_light 0.11 ± 4% +0.0 0.15 ± 5% perf-profile.self.cycles-pp.resched_curr 0.11 ± 3% +0.0 0.15 ± 8% perf-profile.self.cycles-pp.mod_timer 0.18 ± 2% +0.0 0.22 ± 5% perf-profile.self.cycles-pp.__netif_receive_skb_core 0.03 ±100% +0.0 0.07 ± 7% perf-profile.self.cycles-pp.__sys_recvfrom 0.03 ±100% +0.0 0.07 ± 7% perf-profile.self.cycles-pp.__hrtimer_init 0.22 +0.0 0.26 ± 7% perf-profile.self.cycles-pp.enqueue_entity 0.16 ± 4% +0.0 0.20 ± 7% perf-profile.self.cycles-pp.memcg_kmem_get_cache 0.18 ± 4% +0.0 0.23 ± 4% perf-profile.self.cycles-pp.switch_mm_irqs_off 0.01 ±173% +0.0 0.05 ± 9% perf-profile.self.cycles-pp.tcp_v4_fill_cb 0.01 ±173% +0.0 0.05 ± 9% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 0.23 ± 2% +0.0 0.27 ± 6% perf-profile.self.cycles-pp.sk_alloc 0.18 ± 2% +0.0 0.22 ± 5% perf-profile.self.cycles-pp.nr_iowait_cpu 0.17 ± 3% +0.0 0.21 ± 3% perf-profile.self.cycles-pp.process_backlog 0.13 ± 3% +0.0 0.18 ± 6% perf-profile.self.cycles-pp.loopback_xmit 0.10 ± 7% +0.0 0.14 ± 7% perf-profile.self.cycles-pp.kmem_cache_alloc_node 0.03 ±100% +0.0 0.07 ± 10% perf-profile.self.cycles-pp.__cgroup_bpf_run_filter_sock_ops 0.04 ± 57% +0.0 0.08 ± 10% perf-profile.self.cycles-pp.dev_hard_start_xmit 0.01 ±173% +0.0 0.06 ± 14% perf-profile.self.cycles-pp.tcp_child_process 0.16 +0.0 0.21 ± 2% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.18 +0.0 0.23 ± 7% perf-profile.self.cycles-pp.sock_rfree 0.01 ±173% +0.0 0.06 ± 11% perf-profile.self.cycles-pp.rcu_all_qs 0.01 ±173% +0.0 0.06 ± 11% perf-profile.self.cycles-pp.tcp_v4_connect 0.03 ±100% +0.0 0.07 ± 5% perf-profile.self.cycles-pp.irqtime_account_irq 0.16 ± 4% +0.0 0.21 ± 9% perf-profile.self.cycles-pp.copy_user_generic_unrolled 0.14 ± 3% +0.0 0.18 ± 7% perf-profile.self.cycles-pp.kmem_cache_alloc 0.14 ± 3% +0.0 0.18 ± 7% perf-profile.self.cycles-pp.ipv4_mtu 0.36 ± 3% +0.0 0.41 ± 5% perf-profile.self.cycles-pp.update_load_avg 0.25 ± 3% +0.0 0.30 ± 5% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.01 ±173% +0.1 0.06 ± 13% perf-profile.self.cycles-pp.tcp_rcv_synsent_state_process 0.01 ±173% +0.1 0.06 ± 6% perf-profile.self.cycles-pp.lock_timer_base 0.12 ± 5% +0.1 0.17 ± 7% perf-profile.self.cycles-pp.sk_clone_lock 0.17 ± 3% +0.1 0.22 ± 6% perf-profile.self.cycles-pp.__alloc_skb 0.19 ± 2% +0.1 0.24 ± 6% perf-profile.self.cycles-pp.__ksize 0.11 ± 4% +0.1 0.17 ± 4% perf-profile.self.cycles-pp.ip_send_check 0.24 ± 6% +0.1 0.29 ± 2% perf-profile.self.cycles-pp.wait_woken 0.19 ± 3% +0.1 0.24 ± 8% perf-profile.self.cycles-pp.entry_SYSCALL_64 0.06 ± 7% +0.1 0.11 ± 3% perf-profile.self.cycles-pp.tcp_update_metrics 0.00 +0.1 0.05 ± 9% perf-profile.self.cycles-pp.inet_sk_rx_dst_set 0.23 ± 2% +0.1 0.28 ± 4% perf-profile.self.cycles-pp.__softirqentry_text_start 0.23 ± 2% +0.1 0.29 ± 2% perf-profile.self.cycles-pp.do_syscall_64 0.14 ± 5% +0.1 0.20 ± 4% perf-profile.self.cycles-pp.__might_sleep 0.00 +0.1 0.06 ± 7% perf-profile.self.cycles-pp.tcp_mstamp_refresh 0.16 ± 2% +0.1 0.22 ± 4% perf-profile.self.cycles-pp.__skb_clone 0.26 ± 3% +0.1 0.32 ± 6% perf-profile.self.cycles-pp.__switch_to_asm 0.00 +0.1 0.06 ± 11% perf-profile.self.cycles-pp.__list_add_valid 0.00 +0.1 0.06 perf-profile.self.cycles-pp.__inet_hash_connect 0.00 +0.1 0.06 perf-profile.self.cycles-pp.kmem_cache_alloc_trace 0.00 +0.1 0.06 ± 11% perf-profile.self.cycles-pp.calc_global_load_tick 0.33 ± 4% +0.1 0.39 ± 2% perf-profile.self.cycles-pp.set_next_entity 0.16 ± 5% +0.1 0.22 ± 6% perf-profile.self.cycles-pp.__local_bh_enable_ip 0.35 ± 3% +0.1 0.42 ± 4% perf-profile.self.cycles-pp.do_idle 0.18 ± 3% +0.1 0.24 perf-profile.self.cycles-pp.__cgroup_bpf_run_filter_skb 0.00 +0.1 0.06 ± 6% perf-profile.self.cycles-pp.skb_entail 0.00 +0.1 0.06 ± 13% perf-profile.self.cycles-pp.rb_first 0.00 +0.1 0.07 ± 7% perf-profile.self.cycles-pp.tcp_rack_update_reo_wnd 0.00 +0.1 0.07 ± 6% perf-profile.self.cycles-pp.tcp_init_metrics 0.00 +0.1 0.07 ± 6% perf-profile.self.cycles-pp.inet_put_port 0.00 +0.1 0.07 ± 6% perf-profile.self.cycles-pp.tcp_rearm_rto 0.00 +0.1 0.07 ± 6% perf-profile.self.cycles-pp.tcp_assign_congestion_control 0.00 +0.1 0.07 ± 6% perf-profile.self.cycles-pp.ip_copy_addrs 0.30 ± 2% +0.1 0.37 ± 5% perf-profile.self.cycles-pp.native_sched_clock 0.22 +0.1 0.29 ± 8% perf-profile.self.cycles-pp.kmem_cache_free 0.18 ± 4% +0.1 0.26 ± 11% perf-profile.self.cycles-pp.tcp_close 0.00 +0.1 0.07 ± 5% perf-profile.self.cycles-pp.inet_twsk_kill 0.19 ± 6% +0.1 0.27 ± 6% perf-profile.self.cycles-pp.tcp_sendmsg_locked 0.30 ± 5% +0.1 0.38 ± 5% perf-profile.self.cycles-pp.apparmor_sk_clone_security 0.26 ± 4% +0.1 0.34 ± 3% perf-profile.self.cycles-pp.syscall_return_via_sysret 0.00 +0.1 0.08 ± 8% perf-profile.self.cycles-pp.inet_reqsk_alloc 0.00 +0.1 0.08 ± 5% perf-profile.self.cycles-pp.__sk_free 0.07 ± 7% +0.1 0.15 ± 5% perf-profile.self.cycles-pp.tcp_update_pacing_rate 0.01 ±173% +0.1 0.10 ± 9% perf-profile.self.cycles-pp.ip_local_deliver 0.23 ± 4% +0.1 0.32 ± 11% perf-profile.self.cycles-pp.ip_finish_output2 0.45 ± 2% +0.1 0.54 ± 6% perf-profile.self.cycles-pp.update_rq_clock 0.20 ± 2% +0.1 0.28 ± 3% perf-profile.self.cycles-pp.tcp_clean_rtx_queue 0.05 +0.1 0.14 ± 3% perf-profile.self.cycles-pp.tcp_ack_update_rtt 0.36 +0.1 0.45 ± 6% perf-profile.self.cycles-pp.___might_sleep 0.28 ± 4% +0.1 0.36 ± 6% perf-profile.self.cycles-pp.apparmor_sk_free_security 0.21 ± 5% +0.1 0.30 ± 8% perf-profile.self.cycles-pp.__slab_free 0.00 +0.1 0.09 ± 7% perf-profile.self.cycles-pp.sock_prot_inuse_add 0.34 ± 4% +0.1 0.43 ± 7% perf-profile.self.cycles-pp.apparmor_socket_post_create 0.04 ± 58% +0.1 0.14 ± 7% perf-profile.self.cycles-pp.inet_lookup_ifaddr_rcu 0.35 ± 3% +0.1 0.46 ± 4% perf-profile.self.cycles-pp.skb_release_data 0.45 ± 2% +0.1 0.56 ± 2% perf-profile.self.cycles-pp.dst_release 0.00 +0.1 0.12 ± 5% perf-profile.self.cycles-pp.__tcp_select_window 0.38 ± 4% +0.1 0.50 ± 8% perf-profile.self.cycles-pp.inet_csk_bind_conflict 0.06 ± 11% +0.1 0.19 ± 7% perf-profile.self.cycles-pp.tcp_get_metrics 0.30 ± 2% +0.1 0.43 ± 5% perf-profile.self.cycles-pp.__dev_queue_xmit 0.48 ± 2% +0.1 0.62 ± 5% perf-profile.self.cycles-pp.apparmor_file_free_security 0.43 ± 5% +0.1 0.57 ± 4% perf-profile.self.cycles-pp.tcp_recvmsg 0.49 +0.1 0.63 ± 6% perf-profile.self.cycles-pp.read_tsc 0.15 ± 4% +0.2 0.31 ± 5% perf-profile.self.cycles-pp.tcp_write_xmit 0.62 ± 2% +0.2 0.78 ± 6% perf-profile.self.cycles-pp.__sk_destruct 0.71 ± 2% +0.2 0.89 ± 7% perf-profile.self.cycles-pp.__schedule 0.52 ± 3% +0.2 0.71 ± 6% perf-profile.self.cycles-pp.apparmor_file_alloc_security 0.21 ± 3% +0.2 0.40 ± 4% perf-profile.self.cycles-pp.ipv4_dst_check 0.85 ± 2% +0.2 1.06 ± 3% perf-profile.self.cycles-pp.aa_sk_perm 0.95 +0.2 1.16 ± 4% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.47 +0.2 0.69 ± 7% perf-profile.self.cycles-pp.__tcp_get_metrics 0.86 ± 2% +0.2 1.08 ± 6% perf-profile.self.cycles-pp.__inet_lookup_established 0.68 ± 2% +0.3 0.96 ± 5% perf-profile.self.cycles-pp.__sk_mem_reduce_allocated 0.89 ± 3% +0.3 1.18 ± 5% perf-profile.self.cycles-pp._raw_spin_lock_bh 0.61 ± 2% +0.3 0.90 ± 6% perf-profile.self.cycles-pp.percpu_counter_add_batch 0.61 ± 4% +0.3 0.93 ± 5% perf-profile.self.cycles-pp.sk_forced_mem_schedule 0.19 ± 4% +0.4 0.57 ± 7% perf-profile.self.cycles-pp.inet_ehashfn 1.35 +0.4 1.74 ± 5% perf-profile.self.cycles-pp._raw_spin_lock 0.83 ± 4% +0.5 1.33 ± 3% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 3.20 ± 2% +0.8 4.03 ± 5% perf-profile.self.cycles-pp.poll_idle netperf.Throughput_tps 28000 +-------------------------------------------------------------------+ | O | 27000 |-O O O O O O O O O O O | | O O O O | 26000 |-+ | | O O O O O O O O O O O O O | 25000 |-+ | | | 24000 |-+ | | + | 23000 |-+ + + .+.+.. .+. .+. | | .+.+. + + +.+ +. + | 22000 |.+..+ + + | | +..+.+.+.+..+ | 21000 +-------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

4 months, 2 weeks

1
0
0 / 0

[sched] c3a340f7e7: invalid_opcode:#[##]

by kernel test robot

Greeting, FYI, we noticed the following commit (built with gcc-4.9): commit: c3a340f7e7eadac7662ab104ceb16432e5a4c6b2 ("sched: Have sched_class_highest define by vmlinux.lds.h") https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/core in testcase: trinity with following parameters: runtime: 300s test-description: Trinity is a linux system call fuzz tester. test-url: http://codemonkey.org.uk/projects/trinity/ on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): +------------------------------------------+------------+------------+ | | 590d697963 | c3a340f7e7 | +------------------------------------------+------------+------------+ | boot_successes | 4 | 0 | | boot_failures | 0 | 4 | | invalid_opcode:#[##] | 0 | 4 | | EIP:sched_init | 0 | 4 | | Kernel_panic-not_syncing:Fatal_exception | 0 | 4 | | kernel_BUG_at_kernel/sched/core.c | 0 | 4 | +------------------------------------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <lkp(a)intel.com> [ 1.813939] ** If you see this message and you are not debugging ** [ 1.814781] ** the kernel, report this immediately to your vendor! ** [ 1.815661] ** ** [ 1.816509] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 1.817362] ********************************************************** [ 1.820914] invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC [ 1.821634] CPU: 0 PID: 0 Comm: swapper Not tainted 5.8.0-rc1-00024-gc3a340f7e7ead #2 [ 1.822630] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014 [ 1.823710] EIP: sched_init+0x3d/0x2fb [ 1.824191] Code: 20 a6 e5 c1 3d 40 a6 e5 c1 0f 95 c0 08 c2 75 1a b8 a0 a6 e5 c1 3d c0 a6 e5 c1 75 0e b8 20 a7 e5 c1 3d 40 a7 e5 c1 74 04 0f 0b <0f> 0b 55 89 e5 57 56 53 83 ec 08 e8 67 04 00 00 8b 3d 80 ad ec c1 [ 1.826607] EAX: c1e5a601 EBX: c1dac484 ECX: 00000000 EDX: 00000001 [ 1.827434] ESI: 00000000 EDI: 000003f8 EBP: c1e93f9c ESP: c1e93f78 [ 1.828241] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00210002 [ 1.829112] CR0: 80050033 CR2: ffc70000 CR3: 021ce000 CR4: 00000690 [ 1.829919] Call Trace: [ 1.830240] ? start_kernel+0x43f/0x677 [ 1.830738] i386_start_kernel+0x48/0x4a [ 1.831261] startup_32_smp+0x15f/0x164 [ 1.831757] Modules linked in: [ 1.832165] ---[ end trace db9d28bea02c9e7f ]--- [ 1.832761] EIP: sched_init+0x3d/0x2fb [ 1.833243] Code: 20 a6 e5 c1 3d 40 a6 e5 c1 0f 95 c0 08 c2 75 1a b8 a0 a6 e5 c1 3d c0 a6 e5 c1 75 0e b8 20 a7 e5 c1 3d 40 a7 e5 c1 74 04 0f 0b <0f> 0b 55 89 e5 57 56 53 83 ec 08 e8 67 04 00 00 8b 3d 80 ad ec c1 [ 1.835656] EAX: c1e5a601 EBX: c1dac484 ECX: 00000000 EDX: 00000001 [ 1.836463] ESI: 00000000 EDI: 000003f8 EBP: c1e93f9c ESP: c1e93f78 [ 1.838603] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00210002 [ 1.839496] CR0: 80050033 CR2: ffc70000 CR3: 021ce000 CR4: 00000690 [ 1.840300] Kernel panic - not syncing: Fatal exception [ 1.840969] ------------[ cut here ]------------ [ 1.840970] kernel BUG at kernel/sched/core.c:6652! Elapsed time: 60 qemu-img create -f qcow2 disk-vm-snb-94-0 256G qemu-img create -f qcow2 disk-vm-snb-94-1 256G To reproduce: # build kernel cd linux cp config-5.8.0-rc1-00024-gc3a340f7e7ead .config make HOSTCC=gcc-4.9 CC=gcc-4.9 ARCH=i386 olddefconfig prepare modules_prepare bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email Thanks, lkp

4 months, 2 weeks

6
12
0 / 0

Re: GCC section alignment, and GCC-4.9 being a weird one

by Joe Perches

(adding cc's of kernel-mentees and a few checkpatch contributors) On Wed, 2020-10-21 at 19:54 +0200, Miguel Ojeda wrote: > On Wed, Oct 21, 2020 at 7:42 PM Nick Desaulniers > <ndesaulniers(a)google.com> wrote: > > If you used some of the macros from > > include/linux/compiler_attributes.h like __section and __aligned, I > > would prefer it. Please consider spelling out __attribute__(()) an > > antipattern. > > +1, the shorthands should be used unless there is a reason not to (and > please write the reason in a comment in that case). Perhaps something to add as a generic test in checkpatch. Right now it checks separately for each __attribute__ use with any of aligned, section, printf and scanf. Likely it should include more of the #defines from include/linux/compiler_attributes.h. include/linux/compiler_attributes.h:#define __alias(symbol) __attribute__((__alias__(#symbol))) include/linux/compiler_attributes.h:#define __aligned(x) __attribute__((__aligned__(x))) include/linux/compiler_attributes.h:#define __aligned_largest __attribute__((__aligned__)) include/linux/compiler_attributes.h:#define __always_inline inline __attribute__((__always_inline__)) include/linux/compiler_attributes.h:# define __assume_aligned(a, ...) __attribute__((__assume_aligned__(a, ## __VA_ARGS__))) include/linux/compiler_attributes.h:#define __cold __attribute__((__cold__)) include/linux/compiler_attributes.h:#define __attribute_const__ __attribute__((__const__)) include/linux/compiler_attributes.h:# define __copy(symbol) __attribute__((__copy__(symbol))) include/linux/compiler_attributes.h:# define __designated_init __attribute__((__designated_init__)) include/linux/compiler_attributes.h:# define __visible __attribute__((__externally_visible__)) include/linux/compiler_attributes.h:#define __printf(a, b) __attribute__((__format__(printf, a, b))) include/linux/compiler_attributes.h:#define __scanf(a, b) __attribute__((__format__(scanf, a, b))) include/linux/compiler_attributes.h:#define __gnu_inline __attribute__((__gnu_inline__)) include/linux/compiler_attributes.h:#define __malloc __attribute__((__malloc__)) include/linux/compiler_attributes.h:#define __mode(x) __attribute__((__mode__(x))) include/linux/compiler_attributes.h:# define __no_caller_saved_registers __attribute__((__no_caller_saved_registers__)) include/linux/compiler_attributes.h:# define __noclone __attribute__((__noclone__)) include/linux/compiler_attributes.h:# define fallthrough __attribute__((__fallthrough__)) include/linux/compiler_attributes.h:#define noinline __attribute__((__noinline__)) include/linux/compiler_attributes.h:# define __nonstring __attribute__((__nonstring__)) include/linux/compiler_attributes.h:#define __noreturn __attribute__((__noreturn__)) include/linux/compiler_attributes.h:#define __packed __attribute__((__packed__)) include/linux/compiler_attributes.h:#define __pure __attribute__((__pure__)) include/linux/compiler_attributes.h:#define __section(S) __attribute__((__section__(#S))) include/linux/compiler_attributes.h:#define __always_unused __attribute__((__unused__)) include/linux/compiler_attributes.h:#define __maybe_unused __attribute__((__unused__)) include/linux/compiler_attributes.h:#define __used __attribute__((__used__)) include/linux/compiler_attributes.h:#define __weak __attribute__((__weak__))

4 months, 2 weeks

1
0
0 / 0

2021

2020

2019

2018

2017

2016

2015

2014

2013

LKP October 2020