You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-26 - 21:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot4s.osadl.org (updated Tue Aug 26, 2025 12:44:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
318490121060,3chrt75-21kworker/3:1+events11:05:293
30900232880,1chrt1-21systemd09:30:001
32746632800,5chrt0-21swapper/312:35:273
30721052690,4sleep0201rcuc/009:10:380
2948250996458,5cyclictest3004776-21kworker/u8:0+flush-8:008:30:381
2948248995852,4cyclictest3166826-21kworker/u8:0+flush-8:011:05:010
2948250995751,5cyclictest3030295-21kworker/u8:3+flush-8:008:50:411
2948250995651,4cyclictest3004776-21kworker/u8:0+events_unbound08:10:011
2948248995647,6cyclictest2985086-21latency_hist07:45:010
2948248995647,6cyclictest2985086-21latency_hist07:45:000
2948250995549,5cyclictest3189307-21kworker/u8:3+events_unbound11:30:351
2948250995549,4cyclictest3166826-21kworker/u8:0+events_unbound11:40:431
2948250995450,3cyclictest3227146-21kworker/u8:3+events_unbound12:35:281
2948250995450,3cyclictest2802081-21kworker/u8:0+flush-8:007:15:051
2948250995445,6cyclictest3080086-21latency_hist09:20:011
2948248995448,4cyclictest3184008-21kworker/u8:2+flush-8:011:15:300
30907022530,5sleep00-21swapper/009:30:350
2948250995347,5cyclictest3004776-21kworker/u8:0+events_unbound09:05:011
2948253995241,8cyclictest276-21systemd-journal12:00:012
2948250995246,5cyclictest3065215-21kworker/u8:1+flush-8:010:15:461
2948253995142,6cyclictest276-21systemd-journal10:45:002
2948250995046,3cyclictest3166826-21kworker/u8:0+flush-8:010:55:451
2948250995045,4cyclictest3119044-21kworker/u8:2+events_unbound10:20:391
2948248995044,5cyclictest2995419-21kworker/u8:1+flush-8:008:00:000
2948250994944,4cyclictest3227146-21kworker/u8:3+events_unbound12:10:461
2948250994944,4cyclictest3166826-21kworker/u8:0+events_unbound10:45:441
2948250994943,4cyclictest2947985-21kworker/u8:2+flush-8:007:35:511
2948248994941,6cyclictest3209116-21latency_hist11:30:000
2948250994844,3cyclictest3004776-21kworker/u8:0+flush-8:008:10:421
2948250994843,4cyclictest3149980-21kworker/u8:2+events_unbound10:45:011
2948250994843,4cyclictest2802081-21kworker/u8:0+events_unbound07:45:401
2948250994843,4cyclictest1724259-21kworker/u8:1+flush-8:007:20:321
2948250994843,3cyclictest3030295-21kworker/u8:3+events_unbound09:05:361
2948250994842,4cyclictest2947985-21kworker/u8:2+events_unbound08:25:391
2948250994841,6cyclictest3227146-21kworker/u8:3+flush-8:012:15:451
2948250994841,5cyclictest2995419-21kworker/u8:1+events_unbound08:15:301
2948248994838,7cyclictest3085014-21latency_hist09:25:010
2948250994742,4cyclictest3227146-21kworker/u8:3+events_unbound12:05:011
2948250994742,4cyclictest3065215-21kworker/u8:1+flush-8:010:25:401
2948250994742,4cyclictest3065215-21kworker/u8:1+flush-8:010:25:401
2948248994743,3cyclictest3065215-21kworker/u8:1+flush-8:009:15:410
2948248994743,3cyclictest3030295-21kworker/u8:3+flush-8:008:55:310
2948248994737,7cyclictest3193431-21ntpq11:10:420
2948250994642,3cyclictest1724259-21kworker/u8:1+events_unbound07:10:421
2948250994641,4cyclictest3094903-21kworker/u8:2+events_unbound09:40:301
2948250994641,3cyclictest3119044-21kworker/u8:2+events_unbound10:00:381
2948250994641,3cyclictest2969933-21kworker/u8:3+events_unbound07:30:291
2948250994640,5cyclictest3119044-21kworker/u8:2+events_unbound10:05:291
2948250994640,5cyclictest2802081-21kworker/u8:0+flush-8:007:40:401
2948250994640,5cyclictest2802081-21kworker/u8:0+flush-8:007:40:401
2948250994640,4cyclictest3197026-21kworker/u8:1+flush-8:011:15:371
2948250994639,5cyclictest2969933-21kworker/u8:3+events_unbound08:00:301
2948250994637,7cyclictest3273957-21latency_hist12:35:011
2948248994637,7cyclictest3224251-21apt-get11:44:590
2948248994635,8cyclictest3160158-21diskstats10:40:300
30544652450,6sleep20-21swapper/208:50:412
2948250994540,3cyclictest1724259-21kworker/u8:1+events_unbound07:25:291
2948248994539,4cyclictest3065215-21kworker/u8:1+flush-8:009:55:360
2948250994440,3cyclictest3166826-21kworker/u8:0+events_unbound11:20:291
2948250994437,5cyclictest3197026-21kworker/u8:1+flush-8:011:25:411
2948250994437,5cyclictest3105634-21kworker/u8:3+events_unbound11:00:441
2948250994437,5cyclictest2995419-21kworker/u8:1+flush-8:007:55:421
2948248994440,3cyclictest3227146-21kworker/u8:3+flush-8:012:35:290
31943352430,5sleep13194334-21latency_hist11:15:001
2948250994339,3cyclictest3065215-21kworker/u8:1+flush-8:009:20:411
2948250994338,4cyclictest3065215-21kworker/u8:1+flush-8:010:10:431
2948250994338,3cyclictest3166826-21kworker/u8:0+events_unbound11:35:421
2948250994338,3cyclictest3166826-21kworker/u8:0+events_unbound11:35:421
2948250994337,5cyclictest3065215-21kworker/u8:1+events_unbound10:30:361
2948250994337,4cyclictest3219498-21kworker/u8:1+events_unbound12:09:461
2948250994337,4cyclictest3065215-21kworker/u8:1+flush-8:009:55:461
2948250994337,4cyclictest2995419-21kworker/u8:1+events_unbound08:40:001
2948248994335,3cyclictest3004776-21kworker/u8:0+events_unbound08:05:330
2948248994333,8cyclictest2959987-21latency_hist07:20:000
2948250994238,3cyclictest2995419-21kworker/u8:1+flush-8:008:50:011
2948250994237,4cyclictest3065215-21kworker/u8:1+flush-8:010:35:441
2948248994238,3cyclictest3004776-21kworker/u8:0+flush-8:008:10:420
2948250994137,3cyclictest3214334-21kworker/u8:2+events_unbound12:25:431
2948250994136,4cyclictest2995419-21kworker/u8:1+flush-8:008:40:401
2948250994134,5cyclictest3219498-21kworker/u8:1+flush-8:011:50:391
2948248994137,3cyclictest3004776-21kworker/u8:0+flush-8:009:45:290
2948248994132,7cyclictest3130152-21/usr/sbin/munin10:10:370
29550292400,6sleep375-21kworker/3:1+mm_percpu_wq07:15:053
2948250994036,3cyclictest3219498-21kworker/u8:1+flush-8:011:55:291
2948250994035,4cyclictest3166826-21kworker/u8:0+events_unbound10:50:441
2948250994035,4cyclictest3065215-21kworker/u8:1+flush-8:009:10:501
2948250994031,7cyclictest3059337-21ntpq08:55:401
2948248994036,3cyclictest3105634-21kworker/u8:3+events_unbound09:50:410
2948250993934,4cyclictest3094903-21kworker/u8:2+events_unbound09:45:291
2948250993933,4cyclictest2802081-21kworker/u8:0+events_unbound07:50:271
2948248993935,3cyclictest2802081-21kworker/u8:0+flush-8:007:50:300
2948250993834,3cyclictest3065215-21kworker/u8:1+flush-8:009:30:451
2948248993833,4cyclictest3004776-21kworker/u8:0+flush-8:008:35:420
2948248993833,3cyclictest2947985-21kworker/u8:2+events_unbound07:30:430
2948250993732,3cyclictest3094903-21kworker/u8:2+events_unbound09:35:271
2948248993733,3cyclictest3249457-21kworker/u8:0+flush-8:012:30:410
2948248993733,3cyclictest3004776-21kworker/u8:0+flush-8:009:25:410
2948248993732,4cyclictest3119044-21kworker/u8:2+flush-8:010:05:420
2948248993727,7cyclictest3005112-21latency_hist08:05:010
2948250993631,3cyclictest2995419-21kworker/u8:1+events_unbound08:20:381
2948248993628,6cyclictest3065215-21kworker/u8:1+flush-8:010:35:450
2948248993628,6cyclictest276-21systemd-journal09:45:000
2948250993529,5cyclictest3214334-21kworker/u8:2+events_unbound12:21:321
2948248993531,3cyclictest2947985-21kworker/u8:2+events_unbound08:25:300
2948248993529,4cyclictest3184008-21kworker/u8:2+flush-8:011:05:280
2948253993425,7cyclictest276-21systemd-journal08:18:482
2948250993430,3cyclictest3065215-21kworker/u8:1+flush-8:009:50:281
2948248993430,3cyclictest3166826-21kworker/u8:0+flush-8:011:35:440
2948248993430,3cyclictest3166826-21kworker/u8:0+flush-8:011:35:440
2948248993430,3cyclictest3065215-21kworker/u8:1+events_unbound10:25:430
2948248993430,3cyclictest3065215-21kworker/u8:1+events_unbound10:25:420
2948248993430,3cyclictest2969933-21kworker/u8:3+flush-8:007:35:460
2948248993429,3cyclictest3004776-21kworker/u8:0+flush-8:008:20:440
2948248993425,7cyclictest276-21systemd-journal11:45:000
2948250993328,3cyclictest3184008-21kworker/u8:2+flush-8:011:05:361
2948248993329,3cyclictest3189307-21kworker/u8:3+events_unbound11:30:400
2948248993329,3cyclictest3004776-21kworker/u8:0+flush-8:009:35:410
2948248993328,4cyclictest1724259-21kworker/u8:1+flush-8:007:20:400
2948248993323,8cyclictest3045273-21cp08:45:010
29478712334,10sleep20-21swapper/207:07:422
2948248993226,5cyclictest2802081-21kworker/u8:0+flush-8:007:11:200
2948248993226,4cyclictest3264992-21irqcore12:25:310
2948248993224,6cyclictest3065454-21/usr/sbin/munin09:05:400
2948248993223,7cyclictest2965723-21diskstats07:25:290
2948248993127,3cyclictest3004776-21kworker/u8:0+events_unbound08:15:450
2948248993126,4cyclictest3219498-21kworker/u8:1+events_unbound11:55:270
2948248993126,4cyclictest3030295-21kworker/u8:3+events_unbound08:30:280
2948248993124,5cyclictest3065215-21kworker/u8:1+flush-8:010:15:420
294824899311,11cyclictest2988933-21/usr/sbin/munin07:45:350
2948253993026,2cyclictest3159249-21unixbench_singl10:35:492
294825399300,26cyclictest3194994-21awk11:15:322
2948250993026,3cyclictest3214334-21kworker/u8:2+flush-8:011:45:021
2948248993025,4cyclictest3227146-21kworker/u8:3+events_unbound12:05:380
2948248993022,6cyclictest3170320-21irqcore10:50:310
2948248993019,9cyclictest3178498-21ntpq10:55:380
2948253992923,4cyclictest3229031-21unixbench_singl11:45:492
2948248992924,3cyclictest2995419-21kworker/u8:1+events_unbound08:45:390
2948248992923,3cyclictest3197026-21kworker/u8:1+flush-8:011:23:010
32213562280,2sleep20-21swapper/211:40:332
2948253992824,3cyclictest3203761-21unixbench_multi11:20:442
2948253992824,2cyclictest3264667-21fschecks_count12:25:282
2948248992824,3cyclictest3219498-21kworker/u8:1+flush-8:012:00:310
2948248992823,4cyclictest3149980-21kworker/u8:2+flush-8:010:45:020
2948248992820,6cyclictest3249457-21kworker/u8:0+flush-8:012:20:270
2948253992724,2cyclictest3273570-21unixbench_singl12:30:492
2948253992723,2cyclictest3065829-21fschecks_time09:05:332
2948253992722,3cyclictest2984095-21needreboot07:40:362
2948253992722,3cyclictest2984095-21needreboot07:40:362
2948248992721,4cyclictest3124508-21unixbench_multi10:00:450
2948248992720,5cyclictest3219498-21kworker/u8:1+events_unbound11:50:380
294825399263,21cyclictest0-21swapper/210:33:102
2948253992622,2cyclictest3259698-21tr12:20:282
2948253992622,2cyclictest3249825-21fschecks_time12:10:332
2948253992622,2cyclictest3074700-21unixbench_multi09:10:482
2948253992621,3cyclictest3000650-21cpuspeed_turbos08:00:282
2948253992620,4cyclictest3094527-21unixbench_multi09:30:502
2948253992617,5cyclictest3179688-21apt-get11:00:282
2948253992616,8cyclictest3039963-21apt-get08:40:002
294825399261,22cyclictest3204616-21cpu11:25:272
294825399260,24cyclictest3249145-21taskset12:09:062
2948248992620,4cyclictest3030295-21kworker/u8:3+flush-8:008:51:530
294824899260,24cyclictest3065222-21munin-run09:05:010
29478642263,18sleep00-21swapper/007:07:360
2948253992521,3cyclictest0-21swapper/207:14:182
2948253992521,2cyclictest3100248-21aten2_r5power_c09:40:262
2948248992516,7cyclictest3150227-21diskstats10:30:340
294824899251,22cyclictest3144398-21users10:20:460
2948253992421,2cyclictest0-21swapper/212:17:002
2948253992421,2cyclictest0-21swapper/212:00:122
2948253992421,2cyclictest0-21swapper/211:53:522
2948253992421,2cyclictest0-21swapper/211:37:052
2948253992421,2cyclictest0-21swapper/211:37:052
2948253992421,2cyclictest0-21swapper/211:33:482
2948253992421,2cyclictest0-21swapper/211:07:382
2948253992421,2cyclictest0-21swapper/210:51:052
2948253992421,2cyclictest0-21swapper/210:47:492
2948253992421,2cyclictest0-21swapper/210:01:492
2948253992421,2cyclictest0-21swapper/209:58:322
2948253992421,2cyclictest0-21swapper/209:48:412
2948253992421,2cyclictest0-21swapper/209:38:422
2948253992421,2cyclictest0-21swapper/209:28:512
2948253992421,2cyclictest0-21swapper/209:22:172
2948253992421,2cyclictest0-21swapper/209:02:572
2948253992421,2cyclictest0-21swapper/208:56:082
2948253992421,2cyclictest0-21swapper/208:46:092
2948253992421,2cyclictest0-21swapper/208:42:522
2948253992421,2cyclictest0-21swapper/208:33:012
2948253992421,2cyclictest0-21swapper/208:29:582
2948253992421,2cyclictest0-21swapper/208:07:062
2948253992421,2cyclictest0-21swapper/207:56:522
2948253992421,2cyclictest0-21swapper/207:50:182
2948253992421,2cyclictest0-21swapper/207:47:012
2948253992421,2cyclictest0-21swapper/207:23:552
2948253992421,2cyclictest0-21swapper/207:17:222
2948248992420,3cyclictest3249457-21kworker/u8:0+events_unbound12:13:480
2948248992419,3cyclictest3258346-21ntp_states12:15:400
294825399232,2cyclictest0-21swapper/210:25:432
294825399232,2cyclictest0-21swapper/210:25:432
294825399232,2cyclictest0-21swapper/210:20:242
294825399232,2cyclictest0-21swapper/209:15:572
294825399232,20cyclictest0-21swapper/210:58:002
294825399232,20cyclictest0-21swapper/208:21:352
2948253992320,2cyclictest0-21swapper/212:35:572
2948253992320,2cyclictest0-21swapper/211:10:412
2948253992320,2cyclictest0-21swapper/210:17:532
2948253992320,2cyclictest0-21swapper/210:11:332
2948253992320,2cyclictest0-21swapper/209:51:432
2948253992320,2cyclictest0-21swapper/208:12:562
2948253992320,2cyclictest0-21swapper/207:30:152
2948253992320,2cyclictest0-21swapper/207:26:572
294825399222,2cyclictest0-21swapper/210:05:132
294825399222,2cyclictest0-21swapper/207:35:212
29435002225,10sleep10-21swapper/107:05:281
31045712202,14sleep30-21swapper/309:41:153
31643642192,13sleep30-21swapper/310:42:333
30255952190,3chrt75-21kworker/3:1+events08:25:263
29434342183,7sleep30-21swapper/307:05:023
31367922140,5sleep30-21swapper/310:15:363
29709152145,5sleep30-21swapper/307:30:313
32392992132,6sleep30-21swapper/312:00:013
32173872133,6sleep30-21swapper/311:35:393
32173872133,6sleep30-21swapper/311:35:383
30149862132,7sleep30-21swapper/308:15:003
32739412122,6sleep30-21swapper/312:35:003
32641142122,5sleep30-21swapper/312:25:013
32594752122,6sleep30-21swapper/312:20:283
32543542122,5sleep30-21swapper/312:15:023
32346312122,6sleep30-21swapper/311:55:243
31945472122,6sleep30-21swapper/311:15:283
31936012122,6sleep30-21swapper/311:10:463
31744652122,5sleep30-21swapper/310:55:013
31650992122,6sleep30-21swapper/310:45:303
31573802122,6sleep30-21swapper/310:35:393
31538042122,6sleep30-21swapper/310:30:463
31457582122,6sleep30-21swapper/310:25:323
31457582122,6sleep30-21swapper/310:25:323
31413912123,6sleep30-21swapper/310:20:353
31306142122,6sleep30-21swapper/310:10:323
31295722122,6sleep30-21swapper/310:05:583
31107242122,6sleep30-21swapper/309:50:313
31051082122,6sleep30-21swapper/309:45:083
30991002122,6sleep30-21swapper/309:35:423
30949232122,5sleep30-21swapper/309:35:013
30851082122,6sleep30-21swapper/309:25:023
30746822122,6sleep30-21swapper/309:10:493
30608842122,6sleep30-21swapper/309:00:303
30442862122,6sleep30-21swapper/308:40:403
30390512123,5sleep30-21swapper/308:35:393
30300502122,5sleep30-21swapper/308:30:013
30204732122,6sleep30-21swapper/308:20:263
30192592123,6sleep30-21swapper/308:15:413
29942642122,6sleep30-21swapper/307:50:373
29802782123,6sleep30-21swapper/307:40:013
29802782123,6sleep30-21swapper/307:40:013
29610482122,6sleep30-21swapper/307:20:353
32479602112,5sleep30-21swapper/312:05:353
32334762112,6sleep30-21swapper/311:50:383
32288822112,6sleep30-21swapper/311:45:493
32239662113,5sleep30-21swapper/311:41:283
32134942112,6sleep30-21swapper/311:30:443
32081932112,5sleep30-21swapper/311:25:393
31992642111,5sleep30-21swapper/311:20:023
31801762111,6sleep30-21swapper/311:00:353
31697642112,6sleep30-21swapper/310:50:243
31242912111,6sleep30-21swapper/310:00:453
31162142112,5sleep30-21swapper/309:55:393
30804122112,5sleep30-21swapper/309:20:283
30800672111,6sleep30-21swapper/309:20:003
30654312112,6sleep30-21swapper/309:05:123
30506032112,5sleep30-21swapper/308:50:273
30461232112,5sleep30-21swapper/308:45:303
30054192112,5sleep30-21swapper/308:05:293
30004312112,5sleep30-21swapper/308:00:293
29952812112,5sleep30-21swapper/307:55:023
29858082112,5sleep30-21swapper/307:45:313
29756242112,5sleep30-21swapper/307:35:363
29652112112,5sleep30-21swapper/307:25:013
29483622112,6sleep30-21swapper/307:10:083
32494692102,5sleep30-21swapper/312:10:163
30560282102,5sleep30-21swapper/308:55:333
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional