You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-29 - 07:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Wed Apr 29, 2026 00:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5216993333,0cyclictest12852-21lspci19:23:111
5212992723,3cyclictest0-21swapper/000:03:230
521299270,26cyclictest31961-21awk21:10:010
521299260,26cyclictest0-21swapper/022:57:110
521299260,25cyclictest29128-21ssh21:37:540
521299260,25cyclictest22163-21ssh00:10:010
521299260,25cyclictest18171-21awk19:35:010
521299260,25cyclictest14869-21latency_hist21:25:010
521299260,25cyclictest0-21swapper/021:19:270
521299260,25cyclictest0-21swapper/021:00:490
521299260,25cyclictest0-21swapper/000:32:470
522299250,24cyclictest31925-21diskmemload23:56:433
522299250,24cyclictest31925-21diskmemload22:19:593
522299250,24cyclictest31925-21diskmemload21:16:113
5220992523,1cyclictest15782-21ssh23:31:322
522099250,24cyclictest0-21swapper/222:45:472
521699259,15cyclictest31925-21diskmemload22:41:421
5216992523,1cyclictest0-21swapper/122:21:571
5216992522,2cyclictest31965-21cut21:10:011
521699250,24cyclictest31925-21diskmemload23:50:081
521699250,24cyclictest31925-21diskmemload22:56:531
521699250,24cyclictest31925-21diskmemload21:15:331
521699250,24cyclictest0-21swapper/122:54:521
521699250,24cyclictest0-21swapper/122:25:461
5212992523,1cyclictest0-21swapper/023:31:390
5212992521,3cyclictest0-21swapper/000:28:190
5212992520,4cyclictest1308-21ssh21:42:060
521299251,23cyclictest62550irq/127-eno100:19:100
521299250,25cyclictest31925-21diskmemload23:37:360
521299250,25cyclictest0-21swapper/022:00:360
521299250,25cyclictest0-21swapper/021:14:050
521299250,24cyclictest31925-21diskmemload21:34:230
521299250,24cyclictest0-21swapper/023:14:280
521299250,24cyclictest0-21swapper/022:45:300
521299250,24cyclictest0-21swapper/022:25:290
521299250,24cyclictest0-21swapper/022:15:570
521299250,24cyclictest0-21swapper/021:55:580
521299250,24cyclictest0-21swapper/021:50:490
521299250,24cyclictest0-21swapper/000:20:380
5222992423,1cyclictest0-21swapper/319:23:113
5222992422,1cyclictest0-21swapper/323:25:563
5222992422,1cyclictest0-21swapper/321:12:593
5222992422,1cyclictest0-21swapper/300:00:013
522299240,24cyclictest31925-21diskmemload21:59:183
522299240,24cyclictest0-21swapper/321:24:173
522299240,23cyclictest31925-21diskmemload23:49:013
522299240,23cyclictest31925-21diskmemload22:45:473
5220992423,1cyclictest21791-21ssh23:38:002
5220992423,0cyclictest301-21kworker/2:123:51:392
5220992420,3cyclictest0-21swapper/200:31:462
5220992420,3cyclictest0-21swapper/200:22:502
522099241,1cyclictest5246-21ssh22:50:402
522099240,24cyclictest31925-21diskmemload23:12:572
522099240,24cyclictest0-21swapper/223:56:432
522099240,24cyclictest0-21swapper/222:19:592
522099240,23cyclictest29256-21apt-get22:10:502
522099240,23cyclictest2293-21ssh21:43:432
5216992424,0cyclictest0-21swapper/121:34:001
5216992423,1cyclictest0-21swapper/121:52:371
521699242,22cyclictest22-21ksoftirqd/121:14:581
5216992422,1cyclictest0-21swapper/122:05:541
5216992421,2cyclictest18174-21sh19:35:011
521699240,24cyclictest0-21swapper/122:36:581
521699240,23cyclictest0-21swapper/122:10:371
521699240,23cyclictest0-21swapper/100:30:271
521699240,1cyclictest75-21kswapd000:00:001
521699240,0cyclictest0-21swapper/123:28:281
5212992420,3cyclictest0-21swapper/022:11:110
521299240,24cyclictest31925-21diskmemload22:20:200
521299240,24cyclictest0-21swapper/022:50:010
521299240,24cyclictest0-21swapper/021:26:380
521299240,23cyclictest31925-21diskmemload23:29:430
521299240,23cyclictest20770-21diskmemload22:33:550
521299240,23cyclictest0-21swapper/023:40:390
521299240,23cyclictest0-21swapper/023:20:530
521299240,23cyclictest0-21swapper/023:15:260
521299240,23cyclictest0-21swapper/023:05:420
5222992323,0cyclictest7946-21apt-get19:15:483
5222992323,0cyclictest31925-21diskmemload23:30:303
5222992323,0cyclictest31925-21diskmemload21:36:223
5222992323,0cyclictest0-21swapper/323:38:003
5222992323,0cyclictest0-21swapper/322:10:493
5222992323,0cyclictest0-21swapper/322:09:093
5222992323,0cyclictest0-21swapper/300:36:273
5222992322,1cyclictest6144-21ssh23:22:253
5222992322,1cyclictest31925-21diskmemload21:54:283
5222992321,1cyclictest26104-21df23:10:503
5222992320,2cyclictest24945-21runrttasks22:37:303
522299231,21cyclictest31925-21diskmemload00:06:483
522299230,0cyclictest0-21swapper/322:00:163
5220992323,0cyclictest31925-21diskmemload21:20:522
5220992323,0cyclictest0-21swapper/222:21:122
5220992323,0cyclictest0-21swapper/221:45:322
5220992321,2cyclictest13939-21sh21:55:392
5220992321,1cyclictest0-21swapper/223:25:562
5220992321,1cyclictest0-21swapper/200:08:092
5220992320,2cyclictest29767-21ssh22:42:012
522099231,21cyclictest14994-1kworker/u9:100:16:322
522099230,23cyclictest31925-21diskmemload22:01:102
522099230,23cyclictest0-21swapper/223:23:182
522099230,22cyclictest22135-21ssh21:30:532
522099230,22cyclictest10314-21sh22:55:302
522099230,22cyclictest0-21swapper/219:25:482
522099230,22cyclictest0-21swapper/200:00:062
5216992323,0cyclictest31925-21diskmemload23:41:361
5216992323,0cyclictest0-21swapper/100:26:551
5216992321,1cyclictest0-21swapper/123:20:541
5216992321,1cyclictest0-21swapper/123:06:411
5216992321,1cyclictest0-21swapper/123:01:461
5216992321,1cyclictest0-21swapper/122:48:201
5216992321,1cyclictest0-21swapper/100:35:261
5216992320,2cyclictest0-21swapper/100:10:011
521699231,22cyclictest22-21ksoftirqd/123:48:401
521699230,23cyclictest0-21swapper/100:22:021
521699230,22cyclictest6517-21sh21:47:221
521699230,22cyclictest31925-21diskmemload21:35:101
521699230,22cyclictest29607-21rm23:14:271
521699230,22cyclictest1623-21ssh22:15:331
521699230,22cyclictest14333-21sh00:01:381
521699230,22cyclictest0-21swapper/122:32:421
521699230,0cyclictest0-21swapper/121:58:031
521699230,0cyclictest0-21swapper/121:21:081
5212992321,1cyclictest0-21swapper/023:45:120
5212992320,3cyclictest0-21swapper/000:05:430
521299230,23cyclictest0-21swapper/000:35:040
521299230,22cyclictest0-21swapper/023:56:100
521299230,1cyclictest0-21swapper/021:47:580
5222992222,0cyclictest31925-21diskmemload00:12:073
5222992222,0cyclictest0-21swapper/322:30:213
5222992222,0cyclictest0-21swapper/321:47:033
5222992222,0cyclictest0-21swapper/300:31:063
5222992221,0cyclictest0-21swapper/322:50:073
5222992220,1cyclictest7588-21munin-plugin-st19:15:003
5222992220,1cyclictest4111-21ssh23:52:233
5222992220,1cyclictest26840-21ssh22:40:063
5222992220,1cyclictest15487-21ssh21:25:363
5222992219,2cyclictest31925-21diskmemload21:30:523
5222992219,2cyclictest27164-21perf21:00:003
522299220,2cyclictest677-21ssh21:41:103
522299220,2cyclictest27249-21sh23:43:393
522299220,22cyclictest0-21swapper/300:19:043
522299220,21cyclictest6461-21ssh22:20:013
522299220,21cyclictest5280-21ssh00:25:353
522299220,21cyclictest31937-21ssh21:10:013
522299220,21cyclictest31925-21diskmemload23:09:133
522299220,21cyclictest31925-21diskmemload23:01:493
522299220,21cyclictest13710-21apt-get19:25:483
522299220,21cyclictest0-21swapper/300:22:373
522299220,1cyclictest2325-21sh23:19:283
522299220,0cyclictest0-21swapper/322:25:363
5220992222,0cyclictest31925-21diskmemload21:15:222
5220992222,0cyclictest0-21swapper/222:30:212
5220992222,0cyclictest0-21swapper/221:36:222
5220992222,0cyclictest0-21swapper/219:15:482
5220992222,0cyclictest0-21swapper/200:36:272
5220992221,1cyclictest17876-21apt-get20:40:492
522099220,22cyclictest0-21swapper/222:07:162
522099220,21cyclictest8275-21apt-get20:20:492
522099220,21cyclictest8275-21apt-get20:20:492
522099220,21cyclictest5120-21rm00:25:142
522099220,21cyclictest23359-21ssh23:40:062
522099220,21cyclictest0-21swapper/219:10:482
522099220,21cyclictest0-21swapper/200:10:022
522099220,1cyclictest28723-21ssh23:45:372
522099220,1cyclictest19806-21sh23:04:502
522099220,0cyclictest31925-21diskmemload21:52:292
522099220,0cyclictest0-21swapper/223:18:562
522099220,0cyclictest0-21swapper/222:25:362
522099220,0cyclictest0-21swapper/221:10:202
5216992222,0cyclictest0-21swapper/122:01:171
5216992222,0cyclictest0-21swapper/121:00:491
5216992220,1cyclictest0-21swapper/123:30:031
521699220,21cyclictest31925-21diskmemload23:35:541
521699220,21cyclictest19137-21ssh21:28:471
521699220,21cyclictest0-21swapper/121:40:481
521699220,21cyclictest0-21swapper/119:15:481
521699220,1cyclictest25427-21dump-pmu-power19:50:011
521699220,1cyclictest0-21swapper/100:19:561
5212992220,2cyclictest0-21swapper/023:00:270
5212992220,1cyclictest15158-21munin-run20:35:010
521299220,21cyclictest31925-21diskmemload22:35:060
521299220,21cyclictest13402-21dump-pmu-power19:25:000
521299220,21cyclictest0-21swapper/022:40:290
521299220,1cyclictest22998-21latency_hist19:45:010
521299220,1cyclictest1110-21ssh23:50:190
522299210,1cyclictest20921-21apt-get19:40:473
522299210,0cyclictest0-21swapper/322:55:533
522299210,0cyclictest0-21swapper/320:35:503
5220992121,0cyclictest0-21swapper/223:05:242
5220992120,1cyclictest15130-21ssh21:25:042
5220992119,1cyclictest27160-21awk21:00:002
522099210,0cyclictest23316-21apt-get19:45:492
522099210,0cyclictest10367-21apt-get19:20:492
522099210,0cyclictest0-21swapper/222:36:032
521699210,0cyclictest0-21swapper/120:40:491
5212992119,1cyclictest24731-21cron20:55:000
5212992119,1cyclictest0-21swapper/020:25:480
5212992119,1cyclictest0-21swapper/019:50:480
521299210,20cyclictest0-21swapper/020:55:480
521299210,20cyclictest0-21swapper/020:00:570
521299210,20cyclictest0-21swapper/019:55:480
521299210,0cyclictest0-21swapper/020:35:490
5222992020,0cyclictest13063-21apt-get20:30:483
5222992020,0cyclictest0-21swapper/320:50:503
5220992019,0cyclictest22664-21apt-get20:50:502
5220992018,1cyclictest27839-21munin-run19:55:002
522099200,0cyclictest0-21swapper/220:15:492
522099200,0cyclictest0-21swapper/219:40:472
5216992020,0cyclictest0-21swapper/120:30:481
5216992020,0cyclictest0-21swapper/120:20:491
5216992020,0cyclictest0-21swapper/120:20:491
5216992020,0cyclictest0-21swapper/120:05:491
5216992020,0cyclictest0-21swapper/119:40:471
5216992020,0cyclictest0-21swapper/119:25:481
5216992019,0cyclictest0-21swapper/119:50:491
521699200,0cyclictest0-21swapper/120:25:491
5212992018,1cyclictest10033-21latency_hist19:20:000
521299200,20cyclictest0-21swapper/019:45:490
5222991919,0cyclictest16109-21apt-get19:30:503
5222991919,0cyclictest0-21swapper/319:55:003
5222991918,1cyclictest30988-21apt-get20:00:483
5220991919,0cyclictest0-21swapper/220:30:482
5220991917,1cyclictest28177-21apt-get19:55:482
5216991919,0cyclictest0-21swapper/119:55:481
5212991918,0cyclictest0-21swapper/020:15:490
5212991916,2cyclictest15799-21idleruntime-cro19:30:010
521299190,18cyclictest0-21swapper/020:45:000
521299190,18cyclictest0-21swapper/020:20:490
521299190,18cyclictest0-21swapper/020:20:490
521299190,18cyclictest0-21swapper/020:10:000
5222991818,0cyclictest0-21swapper/320:45:483
5222991818,0cyclictest0-21swapper/320:20:493
5222991818,0cyclictest0-21swapper/320:20:493
5220991818,0cyclictest20268-21apt-get20:45:482
5220991818,0cyclictest0-21swapper/219:30:502
5216991818,0cyclictest0-21swapper/120:35:491
521699180,17cyclictest0-21swapper/119:10:481
521299180,17cyclictest0-21swapper/020:45:490
521299180,13cyclictest3840-21fschecks_count20:10:490
5222991717,0cyclictest0-21swapper/319:55:483
5222991716,1cyclictest7929-21tr20:20:013
5220991717,0cyclictest0-21swapper/220:00:482
522099171,11cyclictest0-21swapper/220:10:472
5216991716,0cyclictest0-21swapper/120:55:481
5216991715,1cyclictest24732-21latency_hist20:55:001
5212991717,0cyclictest0-21swapper/019:10:480
5222991616,0cyclictest0-21swapper/320:40:493
5222991616,0cyclictest0-21swapper/320:10:473
5220991615,1cyclictest27477-21apt-get21:00:492
5216991616,0cyclictest0-21swapper/120:10:471
5216991615,1cyclictest0-21swapper/119:35:521
5216991614,1cyclictest0-21swapper/120:15:491
5216991613,2cyclictest0-21swapper/120:00:491
5222991515,0cyclictest0-21swapper/321:00:493
5222991515,0cyclictest0-21swapper/320:25:483
5222991515,0cyclictest0-21swapper/320:06:273
5220991514,1cyclictest10667-21apt-get20:25:482
5220991513,1cyclictest31963-21munin-run21:10:012
522099150,14cyclictest0-21swapper/219:36:482
521299150,15cyclictest0-21swapper/019:35:570
5222991414,0cyclictest0-21swapper/319:35:493
522299140,14cyclictest0-21swapper/319:45:463
5220991414,0cyclictest0-21swapper/220:06:202
522099140,14cyclictest0-21swapper/220:35:172
521699140,0cyclictest0-21swapper/120:45:181
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional