You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-20 - 00:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Thu Mar 19, 2026 12:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
305922670,2sleep013-21rcuc/007:05:340
283922210,3sleep20-21swapper/207:05:322
2011721720,5sleep0369999cyclictest11:12:330
100842910,4sleep39240-21/usr/sbin/munin07:25:323
100842910,4sleep39240-21/usr/sbin/munin07:25:323
70962880,2sleep20-21swapper/211:35:302
3701998726,12cyclictest0-21swapper/211:45:172
3699998340,37cyclictest0-21swapper/010:35:120
301252830,2sleep30-21swapper/309:27:253
215492790,5sleep2370199cyclictest10:35:232
370099787,13cyclictest7850irq/56-dwc_otg_09:55:161
369999761,23cyclictest0-21swapper/009:05:200
118652750,5sleep3370299cyclictest10:24:053
3699997313,12cyclictest0-21swapper/008:30:160
369999725,35cyclictest0-21swapper/008:00:170
369999725,27cyclictest0-21swapper/008:35:190
369999725,27cyclictest0-21swapper/008:35:190
3702996728,36cyclictest17769-21ssh09:51:123
207612660,1sleep20-21swapper/207:55:262
304132650,1sleep20-21swapper/209:27:562
194332650,4sleep37950irq/56-dwc_otg_10:32:473
189252640,2sleep30-21swapper/309:14:533
189252640,2sleep30-21swapper/309:14:533
370299633,45cyclictest13937-21ssh11:44:433
369999632,32cyclictest0-21swapper/008:10:170
370299624,5cyclictest30055-21taskset10:45:273
370299624,5cyclictest30055-21taskset10:45:273
369999621,27cyclictest8073-21fschecks_count08:50:220
370299612,44cyclictest7850irq/56-dwc_otg_12:19:133
3699996130,16cyclictest7750irq/56-dwc_otg09:55:330
370299607,42cyclictest7850irq/56-dwc_otg_10:05:083
370299601,4cyclictest7950irq/56-dwc_otg_11:20:223
370299601,10cyclictest0-21swapper/309:18:343
370199605,9cyclictest7850irq/56-dwc_otg_11:40:162
369999605,31cyclictest0-21swapper/012:30:180
3699996028,10cyclictest0-21swapper/008:20:160
370199591,8cyclictest7750irq/56-dwc_otg10:15:172
369999596,31cyclictest0-21swapper/012:20:290
3699995924,9cyclictest11178-21aten_rbpower_cu07:30:180
370299581,6cyclictest32306-21fschecks_time09:30:223
370299581,5cyclictest18421-21latency_hist11:50:073
370299581,11cyclictest7950irq/56-dwc_otg_11:37:423
3699995844,6cyclictest7750irq/56-dwc_otg11:20:120
369999584,17cyclictest28340-21switchtime08:15:310
369999581,31cyclictest15585-21uniq12:25:220
370299573,5cyclictest7950irq/56-dwc_otg_12:05:153
370299572,4cyclictest0-21swapper/311:12:523
370299572,11cyclictest0-21swapper/312:25:163
370299571,4cyclictest0-21swapper/312:20:233
370299571,13cyclictest31105-21ssh11:25:313
3700995716,20cyclictest30241-21apt-get11:25:171
3699995729,3cyclictest0-21swapper/011:55:190
369999572,31cyclictest0-21swapper/010:30:300
3699995720,12cyclictest16569-21kworker/u8:011:45:170
369999571,32cyclictest23335-21latency_hist09:20:060
369999571,31cyclictest10033-21latency_hist11:40:060
370299561,8cyclictest7950irq/56-dwc_otg_09:40:073
370299561,5cyclictest24403-21/usr/sbin/munin08:05:253
370299561,3cyclictest0-21swapper/312:32:443
369999563,32cyclictest13051-21wget07:35:200
3699995631,7cyclictest7850irq/56-dwc_otg_10:55:330
3699995616,25cyclictest8172-21timerwakeupswit12:15:320
370299554,4cyclictest0-21swapper/310:56:463
370299553,5cyclictest24435-21aten_rbpower_en10:00:213
3700995514,22cyclictest32109-21/usr/sbin/munin12:05:301
369999552,22cyclictest0-21swapper/011:30:260
369999551,30cyclictest13085-21fschecks_time10:25:220
370299542,4cyclictest0-21swapper/307:50:163
370299542,4cyclictest0-21swapper/307:50:163
3702995416,35cyclictest22184-21cut11:15:233
370299541,4cyclictest7950irq/56-dwc_otg_10:40:033
370299541,4cyclictest13556-21ssh11:05:173
3702995411,39cyclictest149-21systemd-journal10:15:063
370299541,10cyclictest9644-21ssh11:00:203
370199545,18cyclictest7950irq/56-dwc_otg_08:30:192
370099544,21cyclictest0-21swapper/111:30:161
3699995433,11cyclictest0-21swapper/011:25:190
3699995427,3cyclictest7950irq/56-dwc_otg_09:40:170
3699995427,3cyclictest7950irq/56-dwc_otg_09:40:170
3699995426,13cyclictest7750irq/56-dwc_otg09:30:260
3699995426,11cyclictest908-21runrttasks10:00:180
3699995425,6cyclictest4329-21latency_hist10:15:070
369999541,33cyclictest12644-21cat09:45:280
369999541,28cyclictest25661-21cron11:20:060
370299533,8cyclictest481-21wget08:30:193
370299532,3cyclictest7950irq/56-dwc_otg_11:58:453
370299531,4cyclictest30996-21latency_hist12:05:073
370299531,4cyclictest11949-21ssh09:45:173
370299531,3cyclictest7950irq/56-dwc_otg_11:30:313
370299531,3cyclictest7950irq/56-dwc_otg_07:40:063
370299531,13cyclictest0-21swapper/308:40:073
369999535,6cyclictest7950irq/56-dwc_otg_12:00:180
3699995322,16cyclictest0-21swapper/009:50:310
370299528,4cyclictest7950irq/56-dwc_otg_08:00:163
370299523,8cyclictest0-21swapper/309:45:073
370299523,8cyclictest0-21swapper/309:45:073
370299522,10cyclictest0-21swapper/311:50:333
370299522,10cyclictest0-21swapper/311:50:333
370299521,9cyclictest15413-21sh10:28:003
370299521,6cyclictest6347-21ssh10:16:353
370299521,3cyclictest0-21swapper/310:51:323
3700995215,21cyclictest7390-21ssh09:39:581
369999526,27cyclictest5283-21apt07:15:060
3699995236,4cyclictest7850irq/56-dwc_otg_12:05:260
3699995231,5cyclictest7950irq/56-dwc_otg_09:05:060
369999522,29cyclictest13-21rcuc/009:25:230
369999522,25cyclictest917-21snmpd12:10:280
369999522,25cyclictest917-21snmpd12:10:280
369999522,25cyclictest0-21swapper/008:25:170
3699995222,13cyclictest7750irq/56-dwc_otg10:05:230
369999521,31cyclictest24252-21missed_timers12:35:250
370299515,7cyclictest21888-21latency_hist08:00:063
370299513,6cyclictest0-21swapper/308:25:333
370299511,7cyclictest27436-21latency_hist08:15:073
370299511,5cyclictest7750irq/56-dwc_otg12:35:463
370299511,3cyclictest7850irq/56-dwc_otg_08:50:333
3700995116,7cyclictest6884-21ls11:35:261
3700995110,23cyclictest21695-21ssh09:17:151
3699995130,5cyclictest7950irq/56-dwc_otg_10:20:170
369999512,7cyclictest0-21swapper/010:45:170
369999512,7cyclictest0-21swapper/010:45:170
3699995123,8cyclictest0-21swapper/009:20:200
3699995123,13cyclictest7750irq/56-dwc_otg10:45:060
370299503,7cyclictest24029-21modprobe10:00:063
370299501,6cyclictest24247-21sed09:20:273
370299501,4cyclictest16562-21/usr/sbin/munin07:45:303
370099509,28cyclictest18292-21latency_hist07:50:071
370099509,23cyclictest20465-21cut07:55:221
370099506,26cyclictest26562-21ssh10:02:091
3700995013,20cyclictest25350-21ssh12:36:191
3700995011,22cyclictest30855-21chrt09:28:431
3700995011,22cyclictest13098-21sed09:45:341
3699995025,13cyclictest0-21swapper/007:45:320
369999501,29cyclictest0-21swapper/007:55:170
369999501,27cyclictest13913-21fschecks_time11:05:220
370299491,9cyclictest14075-21switchtime09:05:323
370299491,3cyclictest7950irq/56-dwc_otg_08:45:073
370299491,3cyclictest7950irq/56-dwc_otg_07:22:213
3700994929,11cyclictest7750irq/56-dwc_otg10:54:021
369999493,43cyclictest101ktimersoftd/009:36:560
3699994918,10cyclictest0-21swapper/007:40:170
370299482,4cyclictest0-21swapper/308:15:173
370299482,3cyclictest7950irq/56-dwc_otg_10:41:143
370299481,4cyclictest0-21swapper/312:10:493
370299481,4cyclictest0-21swapper/312:10:493
370299481,3cyclictest0-21swapper/308:55:373
370299481,3cyclictest0-21swapper/307:42:363
370099488,26cyclictest0-21swapper/108:00:181
370099487,31cyclictest0-21swapper/107:30:131
3700994815,11cyclictest25681-21apt-get11:20:111
3700994813,18cyclictest8961-21runrttasks12:16:391
369999482,25cyclictest0-21swapper/008:45:270
369999482,22cyclictest0-21swapper/009:10:300
369999482,22cyclictest0-21swapper/009:10:300
3699994818,17cyclictest7850irq/56-dwc_otg_11:00:330
178962480,1sleep10-21swapper/109:13:051
178962480,1sleep10-21swapper/109:13:051
370299478,5cyclictest11921-21latency09:00:273
370099479,21cyclictest15117-21memory11:45:271
370099476,27cyclictest10411-21/usr/sbin/munin08:55:321
370099476,12cyclictest23925-21/usr/sbin/munin09:20:231
3700994719,18cyclictest8243-21if_wlan008:50:241
3700994715,13cyclictest4431-21ssh12:11:011
3700994715,13cyclictest4431-21ssh12:11:011
3700994714,12cyclictest1393-21ssh09:31:351
3700994714,11cyclictest13388-21memory10:25:261
3700994711,21cyclictest7750irq/56-dwc_otg10:45:231
3700994711,21cyclictest7750irq/56-dwc_otg10:45:231
3700994710,24cyclictest18544-21sh10:31:091
3699994718,15cyclictest0-21swapper/010:50:200
369999471,3cyclictest0-21swapper/010:15:160
370299462,8cyclictest11429-21fschecks_time07:30:213
370299461,10cyclictest7950irq/56-dwc_otg_08:23:473
3700994617,16cyclictest28921-21fschecks_count10:05:231
3700994616,11cyclictest24781-21ssh10:39:491
3700994613,20cyclictest29483-21ssh12:02:401
3700994613,12cyclictest19385-21ssh12:30:091
3700994611,25cyclictest8072-21ssh10:19:431
3699994618,14cyclictest0-21swapper/007:30:060
3699994618,14cyclictest0-21swapper/007:30:060
369999461,21cyclictest91750snmpd07:52:420
369999461,21cyclictest91750snmpd07:52:420
370299452,2cyclictest0-21swapper/307:10:213
370299451,7cyclictest6642-21grep07:15:343
370099459,23cyclictest0-21swapper/110:57:141
370099457,23cyclictest0-21swapper/107:20:111
370099457,22cyclictest26786-21tail08:10:341
370099457,13cyclictest24175-21fschecks_time08:05:231
370099456,25cyclictest0-21swapper/108:30:191
3700994517,12cyclictest26524-21ssh11:59:501
3700994512,20cyclictest17828-21sh12:27:441
3700994511,18cyclictest16648-21chrt11:08:481
369999455,21cyclictest19309-21ls11:50:270
369999455,21cyclictest19309-21ls11:50:270
369999452,15cyclictest4464-21fschecks_count08:40:240
3699994521,3cyclictest0-21swapper/007:20:120
3699994518,7cyclictest10345-21ssh11:40:170
370299443,6cyclictest7950irq/56-dwc_otg_08:45:193
3701994423,6cyclictest31-21ksoftirqd/211:25:232
370199440,4cyclictest7950irq/56-dwc_otg_09:30:162
3700994410,23cyclictest21203-21sh11:14:331
369999446,25cyclictest0-21swapper/007:15:200
3699994414,21cyclictest0-21swapper/008:08:320
3699994413,18cyclictest0-21swapper/008:55:190
370299431,5cyclictest917-21snmpd08:37:493
370199437,24cyclictest0-21swapper/208:00:182
3701994328,5cyclictest2958-21aten_rbpower_po12:10:192
3701994328,5cyclictest2958-21aten_rbpower_po12:10:192
370199431,27cyclictest5766-21apt-get11:35:072
3700994318,5cyclictest3984-21latency_hist08:40:071
3700994318,5cyclictest3984-21latency_hist08:40:071
3700994313,10cyclictest1937-21ssh10:11:071
370199428,25cyclictest7950irq/56-dwc_otg_08:35:192
370199428,25cyclictest7950irq/56-dwc_otg_08:35:192
370199427,23cyclictest0-21swapper/207:20:112
370199426,21cyclictest0-21swapper/208:05:112
370199423,4cyclictest0-21swapper/210:10:152
370199422,4cyclictest0-21swapper/210:00:202
3701994216,6cyclictest19896-21sh09:55:062
370099427,10cyclictest0-21swapper/110:42:261
370099426,17cyclictest24174-21ssh11:17:241
370099425,24cyclictest7750irq/56-dwc_otg11:00:161
370099425,11cyclictest12413-21ssh10:25:061
370099422,8cyclictest7750irq/56-dwc_otg07:25:181
370099422,8cyclictest7750irq/56-dwc_otg07:25:181
370199417,21cyclictest0-21swapper/210:05:122
3701994115,7cyclictest31-21ksoftirqd/212:35:332
370099418,22cyclictest12961-21ssh11:42:561
370099417,20cyclictest11402-21/usr/sbin/munin09:00:281
3700994119,19cyclictest3859-21apt-get08:40:131
3700994110,18cyclictest7707-21latency_hist08:50:061
193812410,2sleep2301ktimersoftd/212:30:092
370199405,4cyclictest7850irq/56-dwc_otg_07:50:172
370199405,4cyclictest7850irq/56-dwc_otg_07:50:172
370199401,4cyclictest7950irq/56-dwc_otg_12:30:062
370099408,19cyclictest7850irq/56-dwc_otg_08:18:471
370099408,19cyclictest21922-21ssh11:54:111
370099408,19cyclictest21922-21ssh11:54:111
370099407,12cyclictest16469-21latency_hist07:45:071
3700994020,14cyclictest7750irq/56-dwc_otg09:53:501
370199397,19cyclictest0-21swapper/209:58:022
370199393,24cyclictest7950irq/56-dwc_otg_09:10:072
3701993911,4cyclictest9177-21latency10:20:262
370099396,19cyclictest7736-21ssh09:40:111
370099396,19cyclictest7736-21ssh09:40:111
3700993923,10cyclictest7750irq/56-dwc_otg12:24:561
3700993913,12cyclictest400-21latency_hist08:30:071
370199381,27cyclictest0-21swapper/212:00:182
370199377,4cyclictest0-21swapper/210:33:242
370199377,25cyclictest0-21swapper/207:45:202
370199377,18cyclictest7950irq/56-dwc_otg_08:10:192
3701993715,12cyclictest7950irq/56-dwc_otg_08:55:352
370199371,25cyclictest20552-21ssh09:15:332
370099377,17cyclictest18372-21seq07:50:161
370099377,17cyclictest18372-21seq07:50:161
370099373,12cyclictest0-21swapper/107:39:541
370199369,16cyclictest4914-21unixbench_multi09:35:332
3701993618,6cyclictest31-21ksoftirqd/210:25:172
370199360,17cyclictest7850irq/56-dwc_otg_10:57:262
370099361,5cyclictest0-21swapper/107:10:231
370199356,6cyclictest31-21ksoftirqd/209:45:062
370199356,6cyclictest31-21ksoftirqd/209:45:062
370199356,20cyclictest29063-21kworker/u8:008:40:132
370199355,4cyclictest2715-21latency_hist12:10:072
370199355,20cyclictest18654-21ssh11:50:192
370199355,20cyclictest18654-21ssh11:50:192
3701993515,9cyclictest301ktimersoftd/210:45:232
3701993515,9cyclictest301ktimersoftd/210:45:232
3701993515,4cyclictest29392-21aten_rbpower_po08:20:192
370199351,24cyclictest0-21swapper/209:20:262
370199351,18cyclictest7950irq/56-dwc_otg_11:05:322
370099359,12cyclictest7056-21chrt07:19:421
370099355,15cyclictest7950irq/56-dwc_otg_08:20:291
370099353,15cyclictest14477-21taskset09:06:271
370199345,7cyclictest149-21systemd-journal09:50:052
370199343,18cyclictest19692-21ssh11:11:482
370199342,19cyclictest10911-21latency_hist12:20:062
3701993420,6cyclictest7950irq/56-dwc_otg_11:15:342
3701993417,6cyclictest27822-21sh10:42:482
370199341,4cyclictest16652-21rm09:10:552
370199341,4cyclictest16652-21rm09:10:552
3701993411,5cyclictest27615-21sh11:20:542
3701993410,11cyclictest0-21swapper/207:10:342
370199334,5cyclictest0-21swapper/207:40:332
3701993316,7cyclictest7103-21unixbench_multi08:45:342
3701993314,6cyclictest0-21swapper/211:55:092
370199327,6cyclictest31-21ksoftirqd/208:50:192
370199326,4cyclictest9540-21aten_rbpower_vo11:00:192
3701993211,5cyclictest6321-21uname07:15:312
370199321,15cyclictest11-21rcu_preempt10:50:372
370199317,10cyclictest7950irq/56-dwc_otg_07:35:292
370199312,6cyclictest7850irq/56-dwc_otg_12:25:062
370199312,6cyclictest11724-21missed_timers07:30:252
370199311,15cyclictest7950irq/56-dwc_otg_09:00:242
3701993110,4cyclictest9478-21forks07:25:232
3701993110,4cyclictest9478-21forks07:25:232
370199292,21cyclictest0-21swapper/208:17:342
29332296,9sleep30-21swapper/307:05:343
370199285,21cyclictest0-21swapper/208:25:192
32932273,14sleep10-21swapper/107:08:271
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional