You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-05 - 02:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sun Jan 04, 2026 12:45:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3121599656,39cyclictest27470-21ssh11:25:121
3122499643,33cyclictest23714-21ssh12:00:142
3121599625,9cyclictest131rcu_preempt10:20:141
3121599575,37cyclictest2595-21ssh12:25:121
3121599561,42cyclictest29197-21ssh09:35:151
3121599551,39cyclictest20210-21ssh12:35:131
3121599532,38cyclictest24822-21ssh11:05:131
31224995114,27cyclictest27-21ksoftirqd/210:35:142
3121599505,38cyclictest1170-21ssh11:10:131
3121599494,36cyclictest31806-21ssh09:55:131
31224994830,8cyclictest27-21ksoftirqd/211:15:122
3122499468,25cyclictest27-21ksoftirqd/209:35:142
3121599443,29cyclictest9958-21ssh11:15:121
3121599441,37cyclictest3804-21ssh11:30:131
3122499421,23cyclictest6190-21ssh11:50:132
3122499412,26cyclictest25695-21ssh10:10:122
3122499412,26cyclictest25695-21ssh10:10:122
31236993914,13cyclictest28366irq/58-eth007:15:153
3123699391,27cyclictest28366irq/58-eth012:05:133
3122499382,19cyclictest21471-21ssh10:25:202
3120499383,20cyclictest29881-21ssh11:45:130
3120499381,27cyclictest17983-21ssh09:10:130
3123699374,18cyclictest28366irq/58-eth011:45:133
3123699371,20cyclictest28366irq/58-eth010:40:133
3122499372,18cyclictest15338-21ssh10:03:332
3121599373,30cyclictest17515-21ssh12:15:171
3121599373,30cyclictest17515-21ssh12:15:171
31215993712,11cyclictest21-21ksoftirqd/110:45:121
3120499373,24cyclictest7221-21ssh10:55:120
3123699364,23cyclictest28366irq/58-eth009:45:143
3122499352,17cyclictest20704-21ssh09:47:332
3120499351,27cyclictest19448-21ssh10:25:110
3121599344,26cyclictest5539-21ssh09:40:131
3121599341,30cyclictest19482-21ssh10:25:131
31204993417,10cyclictest10757-21apt-get10:20:150
3123699332,19cyclictest28366irq/58-eth012:10:133
3122499330,16cyclictest22994-21ssh09:50:132
3121599334,22cyclictest4572-21ssh10:35:141
3121599332,28cyclictest28263-21ssh10:30:141
3120499334,19cyclictest25704-21ssh10:10:130
3120499334,19cyclictest25704-21ssh10:10:130
31204993320,7cyclictest4546-21apt-get10:35:140
3123699322,15cyclictest28366irq/58-eth009:40:143
31224993221,5cyclictest27-21ksoftirqd/210:55:132
31224993218,8cyclictest27-21ksoftirqd/208:00:142
3121599329,13cyclictest131rcu_preempt08:30:111
3121599324,22cyclictest23730-21ssh12:00:151
31204993220,10cyclictest3793-21apt-get11:30:140
31204993212,12cyclictest0-21swapper/008:35:120
3122499312,23cyclictest0-21swapper/210:18:332
3121599312,22cyclictest18737-21ssh11:20:141
3121599312,21cyclictest21120-21ssh11:40:141
31204993117,8cyclictest6205-21ssh11:50:140
3123699303,19cyclictest28366irq/58-eth010:25:123
31236993010,16cyclictest28366irq/58-eth010:05:133
3122499304,7cyclictest0-21swapper/208:30:112
3122499303,9cyclictest26713-21ssh09:15:132
3122499301,22cyclictest1186-21ssh11:10:142
31224993010,9cyclictest27-21ksoftirqd/210:50:142
31215993015,10cyclictest21-21ksoftirqd/109:10:141
3120499302,22cyclictest32059-21cat11:25:310
3122499292,11cyclictest20332-21ssh09:30:112
3121599294,20cyclictest11342-21ssh12:30:151
3121599292,23cyclictest16475-21cat09:25:331
3120499292,21cyclictest3008-21ssh09:20:140
3120499292,10cyclictest32583-21ps09:55:210
3123699282,13cyclictest28366irq/58-eth011:35:133
3123699282,13cyclictest28366irq/58-eth011:35:133
3122499282,20cyclictest0-21swapper/208:35:132
3121599284,15cyclictest17582-21diskmemload12:20:121
3121599283,16cyclictest8745-21ssh12:10:131
3121599282,17cyclictest0-21swapper/109:15:131
3120499288,16cyclictest0-21swapper/011:35:140
3120499288,16cyclictest0-21swapper/011:35:140
3120499285,5cyclictest12-21ksoftirqd/012:31:260
3120499284,7cyclictest13201-21latency_hist07:25:010
3120499284,7cyclictest13201-21latency_hist07:25:010
31204992817,7cyclictest30803-21apt-get10:50:150
3123699273,15cyclictest0-21swapper/310:03:333
3123699271,14cyclictest28366irq/58-eth010:50:133
3122499274,18cyclictest0-21swapper/208:20:132
3122499272,19cyclictest0-21swapper/207:45:152
3122499271,19cyclictest22346irq/40-dwc2_hso11:40:132
3121599271,4cyclictest131rcu_preempt10:10:141
3121599271,4cyclictest131rcu_preempt10:10:141
3120499276,16cyclictest9224-21ssh10:18:330
3120499272,19cyclictest31931-21ssh12:21:330
31204992719,4cyclictest28100-21apt-get07:40:140
3120499271,15cyclictest14290-21ssh09:45:140
3120499271,10cyclictest2949-21ssh09:37:140
3123699262,14cyclictest28366irq/58-eth009:50:133
3122499268,9cyclictest2610-21ssh12:25:142
3122499263,15cyclictest22346irq/40-dwc2_hso07:40:132
3122499262,19cyclictest0-21swapper/209:25:162
3122499262,18cyclictest0-21swapper/207:15:132
3122499262,14cyclictest0-21swapper/208:50:142
3121599262,19cyclictest21264-21irqstats09:30:331
3121599261,20cyclictest2066-21ssh10:15:151
3120499267,5cyclictest0-21swapper/009:53:260
3120499262,18cyclictest24764-21ssh12:18:330
3120499262,18cyclictest24764-21ssh12:18:330
31204992614,8cyclictest1158-21apt-get11:10:140
31204992611,8cyclictest531-21usb-storage08:00:050
3123699252,17cyclictest0-21swapper/312:18:333
3123699252,17cyclictest0-21swapper/312:18:333
3122499252,8cyclictest5526-21apt-get09:40:132
3122499252,8cyclictest25515-21ssh12:19:332
3122499252,8cyclictest25515-21ssh12:19:332
3122499252,19cyclictest0-21swapper/208:45:152
3121599256,11cyclictest0-21swapper/107:50:111
3120499258,4cyclictest0-21swapper/009:31:580
3120499257,4cyclictest0-21swapper/012:09:040
3120499254,5cyclictest12-21ksoftirqd/010:05:040
3120499254,15cyclictest0-21swapper/008:25:130
3120499251,7cyclictest13662-21ssh11:15:270
3120499250,10cyclictest20682-21ssh11:56:380
3123699243,13cyclictest0-21swapper/309:05:133
3122499243,6cyclictest0-21swapper/212:05:142
3122499242,16cyclictest20983-21users11:00:352
3122499242,16cyclictest0-21swapper/209:59:332
31224992411,10cyclictest3033-21sshd11:29:332
3122499240,18cyclictest924-21snmpd07:20:152
3122499240,18cyclictest924-21snmpd07:20:152
3121599246,9cyclictest131rcu_preempt11:50:131
31215992411,3cyclictest131rcu_preempt11:35:281
31215992411,3cyclictest131rcu_preempt11:35:281
3120499245,6cyclictest12-21ksoftirqd/012:35:190
3120499244,5cyclictest12-21ksoftirqd/011:43:060
3120499244,5cyclictest12-21ksoftirqd/010:47:280
3120499244,4cyclictest151rcuc/012:12:330
3120499244,4cyclictest151rcuc/009:42:220
3120499244,4cyclictest12-21ksoftirqd/009:15:360
3120499244,14cyclictest0-21swapper/009:05:150
3120499243,5cyclictest17582-21diskmemload10:03:330
3120499243,5cyclictest12-21ksoftirqd/012:02:570
3120499243,5cyclictest12-21ksoftirqd/011:20:250
3120499243,12cyclictest12-21ksoftirqd/010:41:220
31204992418,3cyclictest11616-21apt-get08:30:120
31204992412,8cyclictest24812-21apt-get11:05:130
3120499241,19cyclictest23664-21ls08:05:290
3123699239,7cyclictest28366irq/58-eth009:35:153
3123699237,9cyclictest0-21swapper/310:45:143
3123699233,13cyclictest0-21swapper/311:25:313
31236992315,5cyclictest28366irq/58-eth010:55:143
3122499234,8cyclictest18727-21kthreadcore10:05:212
3122499233,17cyclictest0-21swapper/211:30:142
3122499233,11cyclictest26476-21ssh12:20:162
3122499232,5cyclictest0-21swapper/210:43:332
31224992312,9cyclictest0-21swapper/208:10:272
31215992313,6cyclictest29565-21apt-get08:15:141
31215992313,6cyclictest29565-21apt-get08:15:141
3120499232,5cyclictest11666-21apt-get09:25:150
3120499232,16cyclictest839-21munin-node07:25:130
3120499231,8cyclictest0-21swapper/011:01:370
3120499231,19cyclictest29747-21cut08:15:180
3120499231,19cyclictest29747-21cut08:15:180
3123699225,6cyclictest28366irq/58-eth007:40:133
3123699222,16cyclictest0-21swapper/309:10:133
3122499228,6cyclictest26440-21cat09:15:022
3122499226,12cyclictest25690-21cut11:05:202
3122499222,16cyclictest0-21swapper/211:47:332
3122499222,15cyclictest22346irq/40-dwc2_hso10:20:152
3122499222,15cyclictest12490-21ssh11:35:142
3122499222,15cyclictest12490-21ssh11:35:142
3122499222,14cyclictest22346irq/40-dwc2_hso09:05:142
3122499221,10cyclictest9127-21irqcore12:10:172
3121599223,13cyclictest32448-21ssh12:05:141
3121599222,15cyclictest8162-21ssh10:00:151
3121599222,14cyclictest13284-21rm10:40:151
3121599221,18cyclictest23511-21ls07:35:171
3121599221,15cyclictest6598-21ssh09:20:261
3120499226,4cyclictest0-21swapper/010:30:150
3120499225,14cyclictest0-21swapper/008:10:110
3120499224,6cyclictest19583-21tr07:30:180
3120499224,5cyclictest27099-21unixbench_multi07:35:290
3120499224,5cyclictest12-21ksoftirqd/012:25:370
3120499224,5cyclictest12-21ksoftirqd/009:00:150
3120499224,5cyclictest12-21ksoftirqd/009:00:150
31204992212,6cyclictest30921-21apt-get08:50:120
3123699212,8cyclictest0-21swapper/308:55:123
3123699212,11cyclictest0-21swapper/308:10:123
3123699212,10cyclictest0-21swapper/310:15:133
31236992116,3cyclictest321rcuc/312:26:003
31236992116,3cyclictest321rcuc/309:29:503
31236992114,4cyclictest321rcuc/309:22:583
31236992110,8cyclictest0-21swapper/310:38:343
31236992110,8cyclictest0-21swapper/309:57:193
3122499213,15cyclictest14482-21kthreadcore07:25:182
3122499212,13cyclictest0-21swapper/208:05:132
3122499211,4cyclictest31683-21tr07:10:182
3122499211,16cyclictest14994-21rm11:55:172
3122499211,11cyclictest3316-21if_err_eth109:20:172
3122499211,11cyclictest29028-21kthreadcore10:30:222
3122499211,11cyclictest22099-21ssh10:45:142
31224992110,7cyclictest27-21ksoftirqd/209:00:142
31224992110,7cyclictest27-21ksoftirqd/209:00:142
3121599219,5cyclictest21-21ksoftirqd/111:55:151
3121599211,5cyclictest1133-21ntp_states11:45:261
3121599211,15cyclictest30826-21sh10:50:151
3121599210,15cyclictest16925-21ssh10:05:121
3120499216,12cyclictest26299-21fschecks_time08:45:170
3120499214,9cyclictest0-21swapper/007:55:130
3123699203,12cyclictest0-21swapper/312:20:443
31236992015,3cyclictest321rcuc/310:22:343
31236992014,4cyclictest0-21swapper/310:12:053
31236992014,4cyclictest0-21swapper/310:12:053
3122499203,14cyclictest18874-21fschecks_count11:20:162
3122499202,10cyclictest23684-21irqcore07:35:182
3122499201,9cyclictest14917-21missed_timers12:30:262
3122499201,15cyclictest10327-21cut07:55:152
31224992011,3cyclictest0-21swapper/212:35:142
3122499201,10cyclictest22963-21grep07:35:012
3122499201,10cyclictest10648-21unixbench_multi08:25:292
3121599202,15cyclictest29503-21latency_hist08:15:021
31215992011,7cyclictest24202-1kworker/u9:211:05:021
3121599201,16cyclictest31347-21kthreadcore07:40:231
31215992010,6cyclictest7724-21sed10:55:181
3120499203,11cyclictest839-21munin-node08:55:130
3120499202,4cyclictest12-21ksoftirqd/007:45:200
3120499201,4cyclictest0-21swapper/008:21:330
3120499201,15cyclictest528-21kthreadcore07:10:200
3123699192,11cyclictest0-21swapper/311:00:163
31236991914,3cyclictest321rcuc/311:58:023
31236991914,3cyclictest321rcuc/311:50:483
31236991911,6cyclictest0-21swapper/309:30:373
31236991910,6cyclictest0-21swapper/312:38:273
31236991910,6cyclictest0-21swapper/312:30:233
31236991910,6cyclictest0-21swapper/311:22:003
31236991910,6cyclictest0-21swapper/311:15:283
31236991910,6cyclictest0-21swapper/311:11:073
3122499192,9cyclictest29547-21/usr/sbin/munin08:15:192
3122499192,9cyclictest29547-21/usr/sbin/munin08:15:192
3122499191,9cyclictest21465-21cut08:40:152
3122499191,15cyclictest5720-21irqcore07:50:162
3122499191,14cyclictest31268-21latency_hist07:10:022
3122499191,10cyclictest3326-21wc08:55:132
3121599196,10cyclictest3176-21cut07:45:221
31215991912,4cyclictest0-21swapper/108:35:261
3121599191,15cyclictest21525-21cut08:05:211
31215991911,4cyclictest18122-21switchtime09:45:291
3121599191,11cyclictest14756-21latency_hist08:00:021
3121599191,10cyclictest26153-21ssh09:50:221
3120499191,13cyclictest4072-21grep07:15:190
3120499191,13cyclictest25073-21timerwakeupswit08:40:280
3123699182,10cyclictest0-21swapper/308:45:153
3123699182,10cyclictest0-21swapper/308:00:053
31236991813,3cyclictest0-21swapper/307:30:133
31236991812,4cyclictest0-21swapper/312:01:203
31236991812,4cyclictest0-21swapper/311:41:333
31236991812,4cyclictest0-21swapper/310:30:443
31236991812,4cyclictest0-21swapper/309:16:303
3123699181,13cyclictest0-21swapper/311:05:293
3123699181,11cyclictest0-21swapper/308:25:283
31236991810,6cyclictest321rcuc/307:55:133
31236991810,6cyclictest0-21swapper/311:30:313
3121599183,12cyclictest29823-21timerandwakeup08:45:301
3121599181,9cyclictest10268-21cut09:00:221
3121599181,9cyclictest10268-21cut09:00:221
31215991812,4cyclictest17581-21sh09:10:011
3121599181,13cyclictest16738-21cat07:25:211
3121599180,11cyclictest15087-21apt-get08:00:131
3120499185,10cyclictest6016-21kthreadcore07:50:170
3123699172,10cyclictest0-21swapper/308:35:133
3123699171,12cyclictest0-21swapper/308:40:283
3121599179,5cyclictest0-21swapper/108:55:311
3121599176,2cyclictest131rcu_preempt07:10:191
3121599175,10cyclictest2486-21irqrtprio08:20:181
3121599173,9cyclictest0-21swapper/108:50:131
3121599173,11cyclictest21702-21irqcore08:40:171
3121599172,12cyclictest11297-21tr07:20:231
3121599172,12cyclictest11297-21tr07:20:231
3121599171,9cyclictest6911-21cut08:25:161
3123699168,5cyclictest33-21ksoftirqd/308:50:193
3123699162,9cyclictest0-21swapper/307:35:043
3123699162,10cyclictest0-21swapper/307:25:133
3123699161,12cyclictest0-21swapper/308:15:183
3123699161,12cyclictest0-21swapper/308:15:183
3121599161,2cyclictest0-21swapper/107:30:211
31215991611,3cyclictest4250-21irqrtprio07:15:191
3123699159,4cyclictest0-21swapper/308:05:273
3123699159,4cyclictest0-21swapper/307:50:153
3123699152,8cyclictest0-21swapper/309:00:153
3123699152,8cyclictest0-21swapper/309:00:153
3123699152,10cyclictest0-21swapper/307:20:153
3123699152,10cyclictest0-21swapper/307:20:153
3120499151,10cyclictest31267-21latency_hist07:10:020
3123699149,3cyclictest0-21swapper/308:20:153
3123699148,4cyclictest0-21swapper/307:10:313
3123699141,7cyclictest0-21swapper/308:35:013
3121599141,10cyclictest31269-21latency_hist07:10:021
3123699127,3cyclictest0-21swapper/307:45:173
3123699111,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional