You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-16 - 06:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot4s.osadl.org (updated Sun Nov 16, 2025 00:44:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
302995622050,3sleep3586-21kworker/3:2-events22:56:253
308863921740,4sleep30-21swapper/323:55:003
29508462850,5sleep30-21swapper/321:36:233
30570582830,5sleep30-21swapper/323:21:253
2803210996963,4cyclictest2994057-21kworker/u8:3+events_unbound22:36:331
2803210996762,4cyclictest2934244-21kworker/u8:2+events_unbound21:51:301
2803210996151,8cyclictest3054374-21/usr/sbin/munin23:21:261
2803210995853,4cyclictest2449338-21kworker/u8:0+events_unbound23:16:291
2803210995852,5cyclictest2934244-21kworker/u8:2+events_unbound21:56:351
2803204995851,5cyclictest2449338-21kworker/u8:0+flush-8:020:06:330
2803210995751,5cyclictest2449338-21kworker/u8:0+flush-8:020:41:311
2803210995652,3cyclictest3053767-21kworker/u8:1+events_unbound23:56:341
2803210995651,4cyclictest2840178-21kworker/u8:1+events_unbound20:46:351
2803210995651,4cyclictest2840178-21kworker/u8:1+events_unbound20:46:341
2803210995648,5cyclictest2954001-21latency_hist21:40:011
2803216995544,8cyclictest2834984-21gpgv19:40:002
2803210995549,4cyclictest3093608-21kworker/u8:2+events_unbound00:16:201
2803210995549,4cyclictest2449338-21kworker/u8:0+events_unbound20:31:391
2803210995450,3cyclictest2829718-21kworker/u8:3+flush-8:019:51:351
2803210995449,4cyclictest2840178-21kworker/u8:1+flush-8:020:56:321
2803210995449,4cyclictest2840178-21kworker/u8:1+flush-8:020:56:321
2803210995348,4cyclictest2934244-21kworker/u8:2+events_unbound22:31:321
2803204995342,8cyclictest282-21systemd-journal19:15:010
2803210995248,3cyclictest2934244-21kworker/u8:2+flush-8:021:21:321
2803210995247,4cyclictest3113135-21kworker/u8:1+flush-8:000:31:371
2803210995245,5cyclictest2449338-21kworker/u8:0+flush-8:021:41:321
2803210995146,4cyclictest2994057-21kworker/u8:3+events_unbound23:11:341
2803210995145,5cyclictest2696769-21kworker/u8:2+flush-8:019:31:301
2803210995144,5cyclictest2449338-21kworker/u8:0+flush-8:000:11:341
2803204995045,3cyclictest2840178-21kworker/u8:1+flush-8:020:21:360
2803210994944,4cyclictest3053767-21kworker/u8:1+flush-8:023:41:301
2803210994944,4cyclictest2449338-21kworker/u8:0+events_unbound22:51:341
2803204994938,8cyclictest2959548-21diskstats21:46:200
2803210994844,3cyclictest2449338-21kworker/u8:0+events_unbound23:31:331
2803204994842,4cyclictest3004052-21apt-get22:31:150
2803210994742,4cyclictest2934244-21kworker/u8:2+events_unbound22:11:281
2803210994742,4cyclictest2449338-21kworker/u8:0+events_unbound23:36:321
2803210994741,5cyclictest2449338-21kworker/u8:0+flush-8:020:26:311
2803210994741,4cyclictest2948194-21ntpq21:31:291
2803210994741,4cyclictest2934244-21kworker/u8:2+flush-8:022:26:221
2803210994741,4cyclictest2449338-21kworker/u8:0+events_unbound20:06:331
2803210994642,3cyclictest3018279-21kworker/u8:1+events_unbound22:46:351
2803216994535,8cyclictest3028729-21cp22:55:012
2803210994541,3cyclictest3118625-21kworker/u8:0+events_unbound00:36:251
2803210994540,3cyclictest2829718-21kworker/u8:3+events_unbound21:16:251
2803210994538,5cyclictest2449338-21kworker/u8:0+flush-8:023:06:281
2803204994536,7cyclictest2999069-21/usr/sbin/munin22:26:290
2803210994440,3cyclictest3053767-21kworker/u8:1+flush-8:023:46:361
2803210994439,4cyclictest2840178-21kworker/u8:1+events_unbound20:21:351
2803210994438,5cyclictest3118625-21kworker/u8:0+events_unbound00:26:361
2803210994438,5cyclictest2840178-21kworker/u8:1+flush-8:020:11:301
2803210994438,4cyclictest2934244-21kworker/u8:2+events_unbound22:16:291
2803210994438,4cyclictest2934244-21kworker/u8:2+events_unbound21:46:371
2803210994437,5cyclictest3053767-21kworker/u8:1+flush-8:023:26:331
2803210994337,4cyclictest2449338-21kworker/u8:0+events_unbound19:46:241
2803210994336,6cyclictest2696769-21kworker/u8:2+events_unbound19:26:351
2803204994336,5cyclictest2803077-21kworker/u8:3+events_unbound19:15:040
2803210994236,4cyclictest2994057-21kworker/u8:3+events_unbound00:06:321
2803210994236,4cyclictest2449338-21kworker/u8:0+events_unbound19:36:321
28310992410,2sleep3471rcuc/319:36:243
2803216994130,8cyclictest3034198-21cat23:00:012
2803210994136,3cyclictest2840178-21kworker/u8:1+events_unbound20:16:211
2803210994135,5cyclictest2994057-21kworker/u8:3+flush-8:023:01:301
2803210994134,5cyclictest2994057-21kworker/u8:3+flush-8:000:21:311
2803204994136,3cyclictest3118625-21kworker/u8:0+flush-8:000:36:360
2803204994135,4cyclictest2449338-21kworker/u8:0+flush-8:020:01:270
2803204994135,4cyclictest2449338-21kworker/u8:0+events_unbound20:00:000
2803204994134,5cyclictest282-21systemd-journal23:39:110
2803210994036,3cyclictest2696769-21kworker/u8:2+flush-8:019:21:241
2803210994035,4cyclictest2840178-21kworker/u8:1+events_unbound20:36:331
2803210994035,3cyclictest2994057-21kworker/u8:3+events_unbound22:21:261
2803210994034,4cyclictest2449338-21kworker/u8:0+flush-8:019:56:311
2803204994034,4cyclictest2449338-21kworker/u8:0+flush-8:021:36:320
2803204993933,4cyclictest3093608-21kworker/u8:2+flush-8:000:16:350
2803204993932,5cyclictest282-21systemd-journal23:50:500
2803204993929,8cyclictest3064540-21diskstats23:31:190
2803216993827,8cyclictest3054051-21awk23:20:002
2803210993833,4cyclictest2449338-21kworker/u8:0+events_unbound21:26:181
2803210993833,3cyclictest2449338-21kworker/u8:0+flush-8:019:41:321
2803210993832,5cyclictest2803077-21kworker/u8:3+flush-8:019:15:051
2803210993831,5cyclictest2983081-21ntpq22:06:291
2803210993831,5cyclictest2983081-21ntpq22:06:291
2803210993831,5cyclictest2919705-21apt-get21:06:151
2803210993831,5cyclictest2449338-21kworker/u8:0+flush-8:020:01:281
2803204993834,3cyclictest2449338-21kworker/u8:0+events_unbound22:56:370
2803204993833,3cyclictest2449338-21kworker/u8:0+events_unbound21:06:190
2803204993828,8cyclictest282-21systemd-journal20:55:000
2803204993828,8cyclictest282-21systemd-journal20:55:000
2803210993733,3cyclictest2449338-21kworker/u8:0+events_unbound00:01:231
2803204993729,6cyclictest282-21systemd-journal23:45:010
2803204993728,7cyclictest2974355-21/usr/sbin/munin22:01:300
2803204993728,7cyclictest2974355-21/usr/sbin/munin22:01:300
2803204993726,10cyclictest2840047-21turbostat.cron19:45:000
27982942374,10sleep20-21swapper/219:05:022
2803210993632,3cyclictest2840178-21kworker/u8:1+events_unbound21:11:181
2803210993632,3cyclictest2840178-21kworker/u8:1+events_unbound21:01:361
2803204993631,4cyclictest2449338-21kworker/u8:0+flush-8:023:16:340
2803210993529,4cyclictest2803077-21kworker/u8:3+events_unbound19:11:251
2803204993530,4cyclictest2449338-21kworker/u8:0+events_unbound23:06:260
2803204993529,4cyclictest2994057-21kworker/u8:3+events_unbound22:41:350
2803204993528,5cyclictest2696769-21kworker/u8:2+events_unbound19:21:360
28027992353,8sleep10-21swapper/119:07:101
2803216993427,5cyclictest3128410-21gpgv00:35:002
2803210993429,4cyclictest2449338-21kworker/u8:0+events_unbound23:51:201
2803210993427,5cyclictest2449338-21kworker/u8:0+events_unbound22:56:351
2803210993423,9cyclictest2908674-21ntpq20:51:301
2803210993423,9cyclictest2908674-21ntpq20:51:291
2803204993430,3cyclictest2934244-21kworker/u8:2+events_unbound22:06:270
2803204993430,3cyclictest2934244-21kworker/u8:2+events_unbound22:06:270
2803204993430,3cyclictest2829718-21kworker/u8:3+events_unbound21:16:340
2803204993430,2cyclictest2948163-21needreboot21:31:280
2803204993429,4cyclictest2449338-21kworker/u8:0+flush-8:021:45:010
2803204993429,4cyclictest2449338-21kworker/u8:0+flush-8:020:26:350
2803204993429,4cyclictest2449338-21kworker/u8:0+events_unbound00:01:330
2803204993429,3cyclictest3053767-21kworker/u8:1+events_unbound23:26:360
2803204993428,4cyclictest3018279-21kworker/u8:1+flush-8:022:46:350
2803204993425,7cyclictest2820328-21/usr/sbin/munin19:26:310
29350992330,3sleep20-21swapper/221:21:212
2803204993329,3cyclictest2994057-21kworker/u8:3+flush-8:023:01:340
2803204993329,3cyclictest2840178-21kworker/u8:1+events_unbound20:36:290
2803204993329,3cyclictest2829718-21kworker/u8:3+events_unbound19:51:210
2803216993226,4cyclictest3008321-21unixbench_singl22:31:362
2803210993225,5cyclictest2994057-21kworker/u8:3+flush-8:022:41:501
2803204993228,3cyclictest2449338-21kworker/u8:0+events_unbound21:26:320
2803204993227,4cyclictest2449338-21kworker/u8:0+flush-8:000:11:210
2803204993226,4cyclictest3053767-21kworker/u8:1+flush-8:023:46:250
2803204993226,4cyclictest2840178-21kworker/u8:1+flush-8:021:11:180
28243422310,1sleep3471rcuc/319:26:373
2803210993128,2cyclictest2934244-21kworker/u8:2+flush-8:022:01:361
2803210993128,2cyclictest2934244-21kworker/u8:2+flush-8:022:01:361
2803204993126,3cyclictest2449338-21kworker/u8:0+events_unbound21:01:320
2803204993123,6cyclictest3128382-21latency_hist00:35:010
28315122300,3sleep0201rcuc/019:36:240
2803216993026,2cyclictest3122969-21unixbench_multi00:26:372
2803204993026,3cyclictest3053767-21kworker/u8:1+events_unbound23:56:300
2803204993025,4cyclictest2449338-21kworker/u8:0+flush-8:020:16:350
2803204993024,5cyclictest2994057-21kworker/u8:3+flush-8:023:11:200
2803204993024,5cyclictest2994057-21kworker/u8:3+events_unbound00:21:290
2803204993024,4cyclictest3053767-21kworker/u8:1+events_unbound23:21:320
280320499301,24cyclictest2963916-21kworker/0:2+events22:50:500
2803204992926,2cyclictest2904550-21systemd-run20:50:000
2803204992926,2cyclictest2904550-21systemd-run20:50:000
2803204992924,4cyclictest2934244-21kworker/u8:2+events_unbound21:21:340
2803216992823,3cyclictest2855099-21munin-plugin-st20:00:002
2803204992823,4cyclictest2994057-21kworker/u8:3+flush-8:000:06:350
2803216992723,2cyclictest2924860-21cpuspeed_turbos21:11:172
2803216992722,4cyclictest2904228-21unixbench_singl20:46:392
2803216992722,4cyclictest2904228-21unixbench_singl20:46:392
2803204992723,3cyclictest2934244-21kworker/u8:2+events_unbound22:11:180
2803204992722,4cyclictest2449338-21kworker/u8:0+flush-8:020:41:350
2803204992722,3cyclictest2934244-21kworker/u8:2+events_unbound22:16:210
2803204992722,3cyclictest2840178-21kworker/u8:1+events_unbound20:56:230
2803204992722,3cyclictest2840178-21kworker/u8:1+events_unbound20:56:230
2803204992722,3cyclictest2449338-21kworker/u8:0+events_unbound21:56:190
2803216992622,2cyclictest2874552-21unixbench_singl20:16:382
2803216992621,3cyclictest2810607-21cpuspeed_turbos19:16:212
280321699261,13cyclictest2889920-21cat20:35:012
280320499262,23cyclictest0-21swapper/022:36:340
2803204992621,4cyclictest2696769-21kworker/u8:2+flush-8:019:31:320
2803204992621,4cyclictest2449338-21kworker/u8:0+flush-8:019:46:350
280320499260,25cyclictest3093617-21kworker/0:0+events00:29:180
280321699258,3cyclictest39-21ksoftirqd/219:51:242
280321699253,4cyclictest0-21swapper/219:32:202
2803216992522,2cyclictest3113017-21unixbench_multi00:16:372
2803216992521,2cyclictest2919510-21latency_hist21:05:002
2803216992518,5cyclictest2924308-21awk21:10:012
280320499255,18cyclictest2840195-21kworker/0:1+events20:10:380
2803216992420,2cyclictest2944510-21aten2_r5power_e21:31:182
2803216992420,2cyclictest0-21swapper/223:10:012
2803216992418,1cyclictest421rcuog/223:46:192
2803216992314,7cyclictest2914395-21cat21:00:002
2803216992314,7cyclictest2914395-21cat21:00:002
280321699230,9cyclictest2816433-21cut19:21:242
280321699230,6cyclictest2862725-21cut20:06:252
280321699230,18cyclictest2996414-21cut22:21:262
280321699230,13cyclictest2993863-21sed22:20:002
280321699230,12cyclictest2865443-21wget20:11:192
280320499232,20cyclictest2963916-21kworker/0:2+events22:23:140
280320499230,9cyclictest29677711kthreadcore21:51:270
280321699229,2cyclictest161rcu_preempt23:21:202
280321699227,10cyclictest2984244-21apt-get22:11:172
280321699225,7cyclictest0-21swapper/222:37:502
280321699223,13cyclictest3034407-21apt-get23:01:172
280321699222,9cyclictest0-21swapper/221:25:082
2803216992219,2cyclictest371rcuc/220:20:552
280321699221,10cyclictest0-21swapper/220:44:512
280321699220,12cyclictest2949875-21awk21:36:212
280321699220,10cyclictest28418992grep19:46:262
2803204992218,3cyclictest2889727-21sh20:35:000
280321699212,9cyclictest0-21swapper/221:54:132
2803216992118,2cyclictest371rcuc/221:44:382
2803216992118,2cyclictest3044592-21aten2_r5power_p23:11:172
2803216992118,2cyclictest2840472-21apt-key19:45:012
2803216992117,3cyclictest371rcuc/222:41:362
2803216992116,3cyclictest3088641-21idleruntime-cro23:55:002
280321699210,9cyclictest2959764-21awk21:46:232
280321699210,8cyclictest2855799-21fschecks_time20:01:182
280321699210,5cyclictest2821590-21cut19:26:272
280321699205,2cyclictest161rcu_preempt19:11:202
280321699202,9cyclictest0-21swapper/222:07:542
280321699202,9cyclictest0-21swapper/222:07:542
280321699202,6cyclictest39-21ksoftirqd/223:26:292
280321699202,5cyclictest0-21swapper/200:21:252
280321699202,4cyclictest0-21swapper/222:26:242
280321699202,4cyclictest0-21swapper/220:51:262
280321699202,4cyclictest0-21swapper/220:51:262
280321699201,9cyclictest0-21swapper/220:35:472
280321699200,6cyclictest39-21ksoftirqd/223:31:252
280321699200,6cyclictest3092751-21ntpq23:56:302
280321699200,11cyclictest2973566-21sed21:56:362
280321699194,5cyclictest0-21swapper/223:39:042
280321699194,4cyclictest0-21swapper/200:05:212
280321699194,2cyclictest161rcu_preempt22:46:202
280321699193,4cyclictest0-21swapper/223:41:302
280321699192,4cyclictest0-21swapper/222:00:172
280321699192,4cyclictest0-21swapper/222:00:172
2803216991916,2cyclictest371rcuc/200:10:132
280321699190,10cyclictest2931087-21cut21:16:262
2803204991914,3cyclictest0-21swapper/019:10:010
280321699182,2cyclictest161rcu_preempt00:35:372
2803216991815,2cyclictest371rcuc/200:00:542
280321699180,9cyclictest2880944-21cut20:26:242
27984082184,8sleep30-21swapper/319:05:033
30907202143,8sleep30-21swapper/323:56:263
30379572142,8sleep30-21swapper/323:01:283
30750982134,5sleep30-21swapper/323:41:253
30738032132,7sleep30-21swapper/323:40:003
30663722132,6sleep30-21swapper/323:31:263
30502992132,7sleep30-21swapper/323:16:253
29979152133,6sleep30-21swapper/322:21:313
29266492132,6sleep30-21swapper/321:11:243
28103912134,5sleep30-21swapper/319:16:193
31237832122,6sleep30-21swapper/300:31:213
31134312122,6sleep30-21swapper/300:20:013
31074332122,6sleep30-21swapper/300:11:313
30991552122,6sleep30-21swapper/300:06:213
30976572122,6sleep30-21swapper/300:01:273
30816262122,6sleep30-21swapper/323:46:273
30190812122,6sleep30-21swapper/322:46:153
30040672122,6sleep30-21swapper/322:31:183
29990682122,6sleep30-21swapper/322:26:173
29888582122,6sleep30-21swapper/322:15:013
29733472122,6sleep30-21swapper/321:56:343
29687732122,6sleep30-21swapper/321:51:373
29540312122,6sleep30-21swapper/321:40:013
29444572122,6sleep30-21swapper/321:31:193
29399122122,6sleep30-21swapper/321:26:213
29345002123,5sleep30-21swapper/321:20:513
29221512122,6sleep30-21swapper/321:06:273
29151312122,6sleep30-21swapper/321:01:213
29136182122,6sleep30-21swapper/320:56:353
29136182122,6sleep30-21swapper/320:56:353
29045722122,6sleep30-21swapper/320:50:013
29045722122,6sleep30-21swapper/320:50:013
29043972122,6sleep30-21swapper/320:48:333
29043972122,6sleep30-21swapper/320:48:323
28994332122,6sleep30-21swapper/320:42:513
28798382122,6sleep30-21swapper/320:25:013
28693372122,6sleep30-21swapper/320:11:373
28603712122,6sleep30-21swapper/320:06:153
28554352122,6sleep30-21swapper/320:01:123
28353942122,6sleep30-21swapper/319:40:513
28067082122,6sleep30-21swapper/319:11:283
31309392112,5sleep30-21swapper/300:36:273
31186882112,6sleep30-21swapper/300:25:553
31088932112,5sleep30-21swapper/300:16:193
30632072112,5sleep30-21swapper/323:26:353
30446912112,5sleep30-21swapper/323:11:193
30433472113,5sleep30-21swapper/323:06:343
30241792112,5sleep30-21swapper/322:51:213
30138542112,5sleep30-21swapper/322:40:293
30097282112,5sleep30-21swapper/322:36:243
29878302112,5sleep30-21swapper/322:11:283
29820132112,5sleep30-21swapper/322:06:263
29820132112,5sleep30-21swapper/322:06:263
29766932112,5sleep30-21swapper/322:01:283
29766932112,5sleep30-21swapper/322:01:283
29303012112,5sleep30-21swapper/321:16:263
28927762112,5sleep30-21swapper/320:36:293
28891152112,5sleep30-21swapper/320:31:393
28752192112,5sleep30-21swapper/320:21:163
28711542112,5sleep30-21swapper/320:16:263
28504202112,5sleep30-21swapper/319:55:073
28495952112,5sleep30-21swapper/319:51:343
28440172112,5sleep30-21swapper/319:46:293
28289942112,5sleep30-21swapper/319:31:283
28191262112,6sleep30-21swapper/319:21:283
29591842102,5sleep30-21swapper/321:45:073
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional