You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-13 - 16:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Mon Apr 13, 2026 00:45:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2311899716,47cyclictest23690-21ssh00:00:131
2311899616,8cyclictest131rcu_preempt00:30:141
2312699603,46cyclictest14700-21ssh23:55:122
2311099597,45cyclictest29431-21ssh22:15:130
2311099545,40cyclictest21343-21ssh00:35:120
2311099536,37cyclictest2423-21ssh22:00:110
23131995219,18cyclictest28366irq/58-eth021:35:143
2312699521,4cyclictest0-21swapper/221:35:142
2312699521,4cyclictest0-21swapper/219:15:132
2312699521,4cyclictest0-21swapper/219:15:132
2311899513,38cyclictest27679-21ssh22:50:141
2312699506,41cyclictest2391-21ssh23:30:142
2312699492,35cyclictest0-21swapper/222:00:122
2312699482,4cyclictest0-21swapper/219:20:112
2311099473,35cyclictest5993-21ssh22:20:130
2311899463,29cyclictest29268-21ssh23:45:131
2312699441,25cyclictest4029-21ssh21:25:132
2311899443,38cyclictest32690-21ssh00:05:131
2311099431,31cyclictest4293-21ssh22:55:140
2311899422,8cyclictest131rcu_preempt21:55:131
2311899422,8cyclictest131rcu_preempt21:55:131
23131994019,14cyclictest28366irq/58-eth023:20:123
2312699392,19cyclictest11608-21cut19:30:182
2312699392,17cyclictest23919-21kernelversion19:10:182
2312699391,8cyclictest26976-21latency_hist20:55:022
2311899392,26cyclictest22253-21ssh21:35:141
23118993920,14cyclictest0-21swapper/120:40:121
2311099391,26cyclictest9654-21ssh22:40:140
2312699377,2cyclictest0-21swapper/220:15:162
2312699376,5cyclictest22467-21run-parts22:45:282
2312699371,9cyclictest7425-21sh21:45:022
2312699371,9cyclictest26548-21kthreadcore19:45:222
2312699371,9cyclictest16202-21mailstats20:40:272
2312699371,8cyclictest8498-21cut20:35:202
2312699371,14cyclictest16393-21cron23:20:012
2311899371,30cyclictest20251-21rm23:40:141
2311899371,30cyclictest20251-21rm23:40:141
23131993616,12cyclictest33-21ksoftirqd/300:30:143
2312699361,9cyclictest17017-21grep21:50:172
2312699361,8cyclictest12388-21/usr/sbin/munin00:30:152
2312699361,12cyclictest9976-21ssh00:10:192
2311899361,2cyclictest131rcu_preempt22:55:131
2312699354,28cyclictest0-21swapper/222:20:282
2312699353,23cyclictest0-21swapper/222:30:192
2312699352,17cyclictest29701-21irqcore19:50:182
2312699351,8cyclictest902-21grep22:35:172
2312699351,8cyclictest902-21grep22:35:172
2312699351,6cyclictest4264-21apt-get22:55:132
2312699351,10cyclictest12346-21kthreadcore23:35:182
2312699350,9cyclictest4174-21timerandwakeup00:05:282
2312699350,7cyclictest20244-21ssh23:40:132
2312699350,7cyclictest20244-21ssh23:40:122
2311099352,24cyclictest18118-21ssh00:15:130
2311099351,10cyclictest12-21ksoftirqd/022:45:120
2312699343,8cyclictest0-21swapper/221:31:192
2312699342,28cyclictest23706-21sh00:00:152
2312699342,10cyclictest22346irq/40-dwc2_hso21:05:142
2312699342,10cyclictest22346irq/40-dwc2_hso21:05:132
2312699341,9cyclictest9837-21sed23:50:312
2312699341,8cyclictest30658-21kthreadcore22:50:242
2312699341,8cyclictest24588-21kthreadcore23:05:222
2312699341,8cyclictest20628-21awk23:20:282
2312699341,8cyclictest17152-21cut20:10:192
2312699341,8cyclictest16042-21latency_hist20:10:012
2312699341,3cyclictest0-21swapper/222:15:092
23126993410,21cyclictest10210-21find20:00:282
23126993410,21cyclictest10210-21find20:00:272
2311899344,20cyclictest7777-21ssh21:45:141
2311899342,23cyclictest28131-21ssh21:20:191
2311899341,24cyclictest0-21swapper/119:10:141
2311099342,23cyclictest28138-21uname21:20:200
2311099341,27cyclictest18319-21sh21:15:150
2313199339,16cyclictest28366irq/58-eth000:25:133
2312699334,6cyclictest11642-21fschecks_time22:05:162
2312699331,9cyclictest27442-21wc21:20:152
2312699331,9cyclictest22774-21kthreadcore22:10:212
2312699331,9cyclictest12984-21kthreadcore22:40:242
2312699331,9cyclictest0-21swapper/219:35:172
2312699331,13cyclictest4391-21kthreadcore20:30:222
2312699331,10cyclictest8819-21diskmemload00:20:272
2312699331,10cyclictest8819-21diskmemload00:20:272
2311899332,24cyclictest11436-21ssh22:05:141
2312699324,7cyclictest31293-21sh23:10:142
2312699322,9cyclictest18273-21kthreadcore20:45:192
2312699321,9cyclictest0-21swapper/200:28:412
2312699321,7cyclictest9007-21xargs19:25:262
2311899322,23cyclictest3108-21grep19:55:191
23110993212,10cyclictest23098-21cyclictest23:25:120
2312699311,9cyclictest1987-21idleruntime-cro23:30:012
2312699311,9cyclictest12778-21ssh21:45:402
2312699311,8cyclictest19796-21kthreadcore21:15:212
2312699310,9cyclictest30841-21grep20:55:282
2312699310,10cyclictest24058-21ssh00:16:512
2311899312,21cyclictest22331-21cut19:40:191
2311899312,20cyclictest7879-21ssh23:15:161
2311099312,20cyclictest22332-21tr19:40:190
2313199301,12cyclictest28366irq/58-eth022:10:123
2312699301,9cyclictest5377-21ssh23:50:002
2312699301,9cyclictest0-21swapper/220:25:182
2312699301,9cyclictest0-21swapper/220:25:172
2312699301,8cyclictest29843-21usage_Anybus20:20:302
2312699301,8cyclictest0-21swapper/222:25:112
2312699300,9cyclictest17108-21rm21:14:042
2312699300,7cyclictest1335-21ssh21:59:122
2312699300,7cyclictest1335-21ssh21:59:122
2311099302,7cyclictest12-21ksoftirqd/023:05:140
23131992916,10cyclictest28366irq/58-eth000:15:143
2313199291,16cyclictest28366irq/58-eth000:35:133
2311899293,19cyclictest18640-21ssh21:31:181
2311899292,21cyclictest9607-21ssh22:03:191
2311899292,19cyclictest17620-21ssh23:37:191
2311899291,19cyclictest2386-21ssh23:30:121
23118992911,11cyclictest0-21swapper/120:20:131
2311099299,11cyclictest11416-21ssh22:05:130
2311099293,18cyclictest3541-21ssh00:25:150
2311099292,19cyclictest25069-21cut22:30:180
2311099292,19cyclictest24930-21sed19:45:200
2311099292,19cyclictest23918-21kernelversion19:10:190
23110992916,8cyclictest8819-21diskmemload21:55:140
23110992916,8cyclictest8819-21diskmemload21:55:130
2313199283,17cyclictest0-21swapper/319:35:183
2313199281,16cyclictest28366irq/58-eth019:50:133
2312699281,9cyclictest15037-21kthreadcore23:00:212
2312699281,8cyclictest5305-21mailstats19:55:262
2312699280,9cyclictest2649-21kthreadcore21:00:222
2312699280,9cyclictest2649-21kthreadcore21:00:212
2312699280,10cyclictest21192-21ssh00:35:032
2311899282,19cyclictest20463-21ssh22:10:141
23110992820,5cyclictest0-21swapper/023:10:140
2313199273,13cyclictest28366irq/58-eth023:45:123
23131992716,7cyclictest28366irq/58-eth000:05:143
2312699272,8cyclictest19749-21/usr/sbin/munin19:40:162
2312699271,23cyclictest23031-21latency_hist19:10:022
2311899272,19cyclictest7563-21kthreadcore19:25:181
2311899272,19cyclictest24729-21ssh21:17:191
2311899271,21cyclictest31264-21ssh21:40:131
2311099272,19cyclictest28294-21ssh23:44:190
2311099272,19cyclictest28294-21ssh23:44:190
23110992715,8cyclictest17960-21users21:30:310
23110992713,10cyclictest7378-21cron21:45:020
2313199264,14cyclictest28366irq/58-eth022:40:143
2313199261,16cyclictest0-21swapper/319:10:193
2311899263,16cyclictest13374-21ssh23:00:141
2311899261,18cyclictest17247-21cut21:50:181
23131992511,7cyclictest28366irq/58-eth022:20:153
2313199251,12cyclictest28366irq/58-eth023:10:133
2311899257,12cyclictest633-21Async12:11:531
2311899257,12cyclictest633-21Async12:11:531
2311899253,15cyclictest16826-21ssh23:20:131
2311099256,11cyclictest12-21ksoftirqd/022:51:510
2313199245,10cyclictest28366irq/58-eth023:40:143
2313199245,10cyclictest28366irq/58-eth023:40:143
2313199243,17cyclictest28366irq/58-eth019:25:123
2311899241,15cyclictest27057-21sh00:20:141
2311899241,15cyclictest27057-21sh00:20:141
2311099249,8cyclictest12-21ksoftirqd/000:23:590
2311099249,8cyclictest12-21ksoftirqd/000:23:590
2311099249,5cyclictest0-21swapper/022:26:450
2311099248,7cyclictest12-21ksoftirqd/000:01:510
2311099244,14cyclictest9307-21ssh21:10:130
2311099242,3cyclictest131rcu_preempt22:35:090
2311099242,3cyclictest131rcu_preempt22:35:080
2311099242,19cyclictest32706-21ssh00:05:140
2313199232,14cyclictest0-21swapper/323:35:143
2311899234,9cyclictest31269-21ssh23:10:111
2311899232,16cyclictest9198-21ssh00:10:131
2311899231,18cyclictest571-21find22:35:121
2311899231,18cyclictest571-21find22:35:121
23118992311,3cyclictest8819-21diskmemload22:40:481
2311099239,5cyclictest12-21ksoftirqd/020:20:160
2311099237,5cyclictest12-21ksoftirqd/023:56:550
2311099237,3cyclictest131rcu_preempt21:50:140
2311099236,7cyclictest12-21ksoftirqd/023:17:250
2311099236,4cyclictest131rcu_preempt19:55:170
2311099231,5cyclictest12-21ksoftirqd/023:33:460
23110992312,3cyclictest0-21swapper/000:34:190
2313199223,16cyclictest0-21swapper/321:50:193
23118992211,7cyclictest29887-21sendmail00:40:011
2311899221,16cyclictest6000-21rm22:20:141
23118992210,8cyclictest8331-21cat00:25:321
2311099229,4cyclictest12-21ksoftirqd/021:48:010
2311099226,9cyclictest12-21ksoftirqd/023:45:410
2311099225,3cyclictest131rcu_preempt23:24:000
2311099223,4cyclictest12-21ksoftirqd/023:04:090
23110992211,3cyclictest0-21swapper/021:37:040
23110992211,3cyclictest0-21swapper/021:29:130
23110992210,4cyclictest12-21ksoftirqd/000:10:120
2313199219,8cyclictest0-21swapper/321:15:163
2313199218,7cyclictest28366irq/58-eth021:30:133
2313199214,11cyclictest0-21swapper/322:00:133
2313199213,13cyclictest0-21swapper/321:58:213
2313199213,13cyclictest0-21swapper/321:58:213
2313199212,12cyclictest28366irq/58-eth019:45:193
2311899216,10cyclictest22373-21diskmemload23:05:151
2311899215,4cyclictest131rcu_preempt22:45:121
23118992114,4cyclictest18712-21open_inodes22:25:261
23118992112,6cyclictest17410-21apt-get20:45:131
2311899211,17cyclictest27888-21ls22:30:281
2311099216,5cyclictest12-21ksoftirqd/023:39:380
2311099215,3cyclictest131rcu_preempt19:25:110
23110992110,5cyclictest151rcuc/023:50:150
2311099210,18cyclictest20601-21uniq22:10:160
2313199209,6cyclictest0-21swapper/323:05:473
2313199205,9cyclictest28366irq/58-eth021:20:153
2313199203,9cyclictest0-21swapper/319:30:193
2313199203,13cyclictest0-21swapper/319:55:193
23131992012,5cyclictest28366irq/58-eth022:50:153
2313199201,14cyclictest0-21swapper/321:45:143
23131992010,8cyclictest0-21swapper/321:10:383
2311899209,8cyclictest16733-21cat19:35:191
2311899208,3cyclictest131rcu_preempt21:10:261
2311899207,10cyclictest8911-21readlink21:10:011
2311899207,10cyclictest8911-21readlink21:10:001
2311899201,9cyclictest3557-21kthreadcore19:20:201
23118992011,6cyclictest20086-21kthreadcore00:15:221
2311899201,15cyclictest25803-21ssh23:25:131
23118992011,5cyclictest14007-21unixbench_singl19:30:271
23118992010,6cyclictest28772-21sshd19:15:191
23118992010,6cyclictest28772-21sshd19:15:191
2311099207,4cyclictest12-21ksoftirqd/019:30:160
2311099202,4cyclictest12-21ksoftirqd/020:10:010
2311099201,12cyclictest28164-21uniq19:15:160
2311099201,12cyclictest28164-21uniq19:15:160
2313199195,9cyclictest28366irq/58-eth023:50:163
2313199195,8cyclictest28366irq/58-eth023:15:163
2313199193,13cyclictest28366irq/58-eth022:55:133
2313199193,10cyclictest0-21swapper/319:40:193
23131991911,5cyclictest0-21swapper/300:20:233
23131991911,5cyclictest0-21swapper/300:20:223
2311899198,7cyclictest633-21Async12:11:531
2311899198,7cyclictest633-21Async12:11:531
2311899194,11cyclictest5773-21kthreadcore20:30:231
2311899193,6cyclictest131rcu_preempt20:55:161
2311899192,13cyclictest0-21swapper/121:00:131
2311899192,13cyclictest0-21swapper/121:00:121
2311899191,8cyclictest9289-21memory23:50:241
2311899191,7cyclictest18397-21ntp_states23:55:261
23118991910,6cyclictest0-21swapper/119:45:191
2311099194,3cyclictest131rcu_preempt19:50:270
23110991913,4cyclictest0-21swapper/020:45:120
2313199189,6cyclictest28366irq/58-eth000:00:133
2313199184,8cyclictest28366irq/58-eth023:30:133
2313199183,9cyclictest0-21swapper/323:25:293
2313199183,9cyclictest0-21swapper/319:15:193
2313199183,9cyclictest0-21swapper/319:15:183
2313199183,10cyclictest28366irq/58-eth021:40:143
2313199182,11cyclictest0-21swapper/321:25:193
2313199182,11cyclictest0-21swapper/319:20:193
2313199182,10cyclictest0-21swapper/320:40:053
2313199182,10cyclictest0-21swapper/320:25:143
2313199182,10cyclictest0-21swapper/320:25:133
2313199182,10cyclictest0-21swapper/320:20:133
2313199182,10cyclictest0-21swapper/320:15:143
2313199182,10cyclictest0-21swapper/320:00:143
2313199182,10cyclictest0-21swapper/320:00:133
2313199181,11cyclictest28366irq/58-eth022:30:133
23131991810,6cyclictest0-21swapper/323:55:313
23131991810,6cyclictest0-21swapper/322:19:133
23131991810,5cyclictest0-21swapper/322:50:023
2311899188,5cyclictest633-21Async12:11:531
2311899187,2cyclictest21-21ksoftirqd/121:25:181
2311899185,10cyclictest22358-21kthreadcore20:15:201
2311899183,10cyclictest0-21swapper/120:50:151
2311099189,6cyclictest0-21swapper/021:05:230
2311099189,6cyclictest0-21swapper/021:05:230
2311099186,3cyclictest131rcu_preempt19:35:170
2311099184,4cyclictest12-21ksoftirqd/020:15:270
2311099184,3cyclictest131rcu_preempt20:35:180
2311099183,2cyclictest131rcu_preempt20:40:120
2311099182,5cyclictest12-21ksoftirqd/020:25:170
2311099182,5cyclictest12-21ksoftirqd/020:25:170
23110991812,4cyclictest0-21swapper/019:20:230
2313199179,5cyclictest28366irq/58-eth023:00:143
2313199179,3cyclictest0-21swapper/300:13:313
2313199171,11cyclictest0-21swapper/322:25:173
2313199171,10cyclictest0-21swapper/320:55:143
2313199171,10cyclictest0-21swapper/320:45:133
2313199171,10cyclictest0-21swapper/320:05:133
23131991710,5cyclictest28366irq/58-eth022:05:133
23131991710,4cyclictest0-21swapper/322:35:133
23131991710,4cyclictest0-21swapper/322:35:133
2311899177,7cyclictest17988-21cut20:10:201
2311899171,5cyclictest29408-21apt-get22:15:121
2311899171,12cyclictest29804-21irqcore19:50:191
2311899170,9cyclictest12980-21kthreadcore20:05:201
2311099178,3cyclictest12-21ksoftirqd/020:50:190
2311099176,3cyclictest131rcu_preempt20:00:230
2311099176,3cyclictest131rcu_preempt20:00:230
2311099172,10cyclictest0-21swapper/021:00:140
2311099172,10cyclictest0-21swapper/021:00:140
2311099172,10cyclictest0-21swapper/020:30:160
2311099167,6cyclictest17718-21kthreadcore20:10:200
23110991610,4cyclictest0-21swapper/020:55:160
2313199149,3cyclictest0-21swapper/320:37:183
2313199142,7cyclictest0-21swapper/320:10:123
2313199142,10cyclictest0-21swapper/320:50:163
2313199141,7cyclictest0-21swapper/321:00:133
2313199141,7cyclictest0-21swapper/321:00:133
2311899141,10cyclictest23137-21latency_hist19:10:021
2311099141,10cyclictest23203-21latency_hist19:10:020
2313199125,5cyclictest0-21swapper/321:05:223
2313199125,5cyclictest0-21swapper/321:05:223
2313199122,6cyclictest0-21swapper/320:30:163
2313199111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional