You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-08 - 13:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Wed Jul 08, 2026 00:45:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6170997940,34cyclictest35966irq/58-eth022:15:133
616299713,41cyclictest9423-21ssh00:00:142
6162996713,8cyclictest27-21ksoftirqd/223:00:132
614699612,40cyclictest13645-21ssh21:39:400
614699571,44cyclictest24721-21ssh21:10:150
6170995625,21cyclictest35966irq/58-eth023:45:133
614699541,47cyclictest0-21swapper/023:45:130
617099533,35cyclictest35966irq/58-eth023:50:153
615499538,8cyclictest131rcu_preempt20:55:131
614699531,41cyclictest5542-21ssh23:40:130
614699512,44cyclictest0-21swapper/020:35:130
615499493,9cyclictest131rcu_preempt20:45:131
617099485,27cyclictest35966irq/58-eth019:50:123
617099485,27cyclictest35966irq/58-eth019:50:113
614699482,39cyclictest4272-21apt-get22:10:130
614699481,41cyclictest0-21swapper/020:40:120
614699462,39cyclictest0-21swapper/021:45:130
614699462,39cyclictest0-21swapper/021:45:130
614699461,38cyclictest25594-21apt-get19:30:130
616299455,7cyclictest27346irq/40-dwc2_hso21:25:142
614699451,41cyclictest0-21swapper/023:05:140
616299442,31cyclictest13422-21ssh22:15:132
615499433,6cyclictest131rcu_preempt23:00:131
615499432,36cyclictest27494-21ssh00:10:131
6146994313,15cyclictest42-21kauditd20:30:000
6170994226,11cyclictest35966irq/58-eth021:10:133
614699427,24cyclictest17248-21sh22:35:120
614699423,34cyclictest23736-21ssh23:50:120
614699422,2cyclictest445-21apt-get20:45:120
614699421,5cyclictest0-21swapper/021:00:120
614699421,36cyclictest0-21swapper/023:15:150
614699421,36cyclictest0-21swapper/000:00:130
614699421,34cyclictest13332-21ssh00:20:120
614699421,34cyclictest13332-21ssh00:20:120
617099415,25cyclictest35966irq/58-eth023:00:123
617099414,23cyclictest35966irq/58-eth021:55:123
615499411,32cyclictest32739-21ssh23:55:131
614699411,35cyclictest1318-21apt-get21:15:140
616299407,22cyclictest27-21ksoftirqd/223:15:152
616299402,32cyclictest6042-21ssh23:56:312
614699404,33cyclictest4223-21ssh00:15:130
614699401,36cyclictest0-21swapper/021:05:110
614699401,35cyclictest0-21swapper/022:20:130
617099393,27cyclictest35966irq/58-eth021:30:143
614699391,3cyclictest0-21swapper/020:20:140
614699391,2cyclictest0-21swapper/000:05:120
614699391,25cyclictest8672-21sh23:23:050
614699390,10cyclictest4231-21find21:50:290
617099384,22cyclictest35966irq/58-eth023:55:113
617099384,19cyclictest35966irq/58-eth022:00:133
617099384,19cyclictest35966irq/58-eth022:00:133
617099381,22cyclictest35966irq/58-eth020:35:143
616299381,32cyclictest0-21swapper/222:08:402
616299381,32cyclictest0-21swapper/222:08:392
6154993818,13cyclictest21-21ksoftirqd/122:50:131
614699381,33cyclictest19706-21apt-get23:30:100
614699381,2cyclictest28605-21apt-get21:30:150
614699380,5cyclictest14180-21ssh21:40:020
6170993715,13cyclictest35966irq/58-eth022:50:123
614699371,31cyclictest30219-21apt-get23:00:120
617099364,19cyclictest35966irq/58-eth022:45:143
617099362,20cyclictest35966irq/58-eth021:45:133
617099362,20cyclictest35966irq/58-eth021:45:123
614699363,29cyclictest13411-21apt-get22:15:130
614699361,3cyclictest0-21swapper/020:05:120
614699361,31cyclictest7825-21grep19:45:180
614699361,31cyclictest7825-21grep19:45:180
614699361,31cyclictest20642-21tr20:30:230
614699361,31cyclictest20642-21tr20:30:220
614699361,30cyclictest12116-21apt-get22:50:120
614699361,23cyclictest27233-21ssh21:28:520
614699360,7cyclictest3248-21cut19:40:180
614699351,3cyclictest31669-21apt-get20:10:140
614699351,30cyclictest25464-21ntp_192.168.11420:00:230
614699351,30cyclictest25464-21ntp_192.168.11420:00:230
614699351,30cyclictest18740-21fschecks_time22:00:170
614699351,30cyclictest18740-21fschecks_time22:00:160
614699351,2cyclictest21170-21apt-get22:55:130
617099342,21cyclictest35966irq/58-eth021:35:133
617099342,18cyclictest35966irq/58-eth000:05:113
616299342,19cyclictest27-21ksoftirqd/220:20:132
615499343,18cyclictest13409-21ssh22:15:131
615499341,25cyclictest19723-21ssh23:30:111
614699341,8cyclictest2465-21kthreadcore23:55:200
614699341,2cyclictest0-21swapper/023:10:130
614699341,29cyclictest20836-21fschecks_count19:25:160
614699341,11cyclictest22491-21ssh00:25:160
616299331,20cyclictest5544-21rm23:40:132
614699331,9cyclictest11454-21ssh23:25:190
614699331,7cyclictest6707-21ntp_192.168.11422:45:270
614699331,2cyclictest0-21swapper/020:50:280
614699331,29cyclictest30909-21kthreadcore23:35:210
614699331,28cyclictest28290-21sed22:05:170
614699331,28cyclictest28290-21sed22:05:170
614699331,28cyclictest15896-21latency_hist19:20:020
614699331,28cyclictest13245-21kthreadcore19:50:180
614699331,28cyclictest13245-21kthreadcore19:50:180
614699331,20cyclictest0-21swapper/000:33:420
614699330,11cyclictest20721-21ntp_states19:55:250
617099323,18cyclictest35966irq/58-eth023:35:133
617099321,19cyclictest35966irq/58-eth022:10:123
616299324,15cyclictest27346irq/40-dwc2_hso19:30:132
616299323,19cyclictest27346irq/40-dwc2_hso22:50:152
615499321,19cyclictest21179-21ssh22:55:131
614699321,9cyclictest4664-21irqstats20:15:180
614699321,9cyclictest19391-21/usr/sbin/munin19:20:260
614699321,2cyclictest0-21swapper/022:40:120
614699321,27cyclictest27459-21/usr/sbin/munin00:10:290
6146993212,16cyclictest0-21swapper/021:20:150
614699320,8cyclictest12211-21cat22:30:290
614699320,3cyclictest10142-21apt-get20:55:130
616299312,22cyclictest32541-21sh21:50:052
616299312,18cyclictest23453-21ssh21:45:142
616299312,18cyclictest23453-21ssh21:45:132
614699311,8cyclictest6591-21/usr/sbin/munin19:10:160
614699311,8cyclictest2140-21cat19:35:290
614699311,2cyclictest0-21swapper/021:55:120
614699311,2cyclictest0-21swapper/000:35:210
614699311,26cyclictest2835-21awk22:25:250
6170993013,11cyclictest35966irq/58-eth000:10:143
615499302,21cyclictest28859-21ssh23:35:141
615499302,19cyclictest21-21ksoftirqd/119:30:131
617099298,16cyclictest35966irq/58-eth022:05:103
617099298,16cyclictest35966irq/58-eth022:05:103
617099294,17cyclictest35966irq/58-eth021:15:143
616299295,10cyclictest15859-1kworker/u9:023:30:102
616299292,21cyclictest24864-21ssh22:38:402
616299292,19cyclictest27346irq/40-dwc2_hso00:05:122
615499291,22cyclictest16942-21ssh00:20:221
615499291,22cyclictest16942-21ssh00:20:221
617099284,13cyclictest35966irq/58-eth023:30:093
6170992812,12cyclictest35966irq/58-eth023:10:133
615499282,19cyclictest27566-21ssh22:05:121
615499282,19cyclictest27566-21ssh22:05:111
615499281,21cyclictest23448-21ssh21:45:141
615499281,21cyclictest23448-21ssh21:45:141
6154992811,12cyclictest21-21ksoftirqd/122:46:411
616299272,20cyclictest8544-21ssh22:46:152
616299271,19cyclictest0-21swapper/223:45:142
615499274,6cyclictest131rcu_preempt23:05:141
615499272,19cyclictest1344-21ssh21:15:151
617099267,10cyclictest35966irq/58-eth000:35:133
617099263,17cyclictest0-21swapper/323:25:423
617099263,17cyclictest0-21swapper/323:21:243
6162992618,5cyclictest15296-21ssh21:40:202
616299261,20cyclictest15741-21diskmemload22:33:422
616299261,10cyclictest4080-21ssh00:15:052
6154992618,4cyclictest131rcu_preempt21:55:151
617099259,9cyclictest35966irq/58-eth022:30:133
617099254,13cyclictest35966irq/58-eth021:40:113
617099253,18cyclictest35966irq/58-eth023:05:153
617099253,15cyclictest0-21swapper/300:27:453
6162992513,5cyclictest2103-21ssh00:13:052
615499253,7cyclictest131rcu_preempt22:38:411
6154992514,3cyclictest131rcu_preempt00:20:011
614699251,5cyclictest6152-21date19:10:020
617099242,13cyclictest0-21swapper/321:25:143
616299242,18cyclictest2359-21ssh22:44:412
616299241,5cyclictest2886-21ls19:40:142
6162992412,4cyclictest0-21swapper/219:50:282
6162992412,4cyclictest0-21swapper/219:50:282
616299241,20cyclictest31650-21ls22:25:142
6162992411,9cyclictest28613-21ssh21:30:142
615499242,18cyclictest4288-21ssh22:10:111
617099233,13cyclictest0-21swapper/321:50:053
617099232,10cyclictest0-21swapper/323:40:403
616299237,13cyclictest1-21systemd23:06:212
616299232,6cyclictest0-21swapper/223:12:412
616299232,6cyclictest0-21swapper/200:25:462
616299232,16cyclictest0-21swapper/223:54:402
6162992312,9cyclictest17440-21ssh21:23:072
6162992312,5cyclictest8341-21ssh21:18:052
616299230,20cyclictest19549-21ls19:20:272
6154992310,8cyclictest8524-21ls22:30:161
616299227,8cyclictest930-21snmpd19:45:142
616299227,8cyclictest930-21snmpd19:45:132
616299226,7cyclictest1-21systemd20:40:512
616299225,4cyclictest25531-21cat21:10:202
616299222,16cyclictest0-21swapper/223:21:242
616299221,9cyclictest890666ssh21:35:272
6162992217,3cyclictest27-21ksoftirqd/222:10:122
616299221,6cyclictest0-21swapper/222:57:412
6162992211,9cyclictest3614-21ssh23:38:132
616299220,17cyclictest22496-21ssh22:20:142
615499221,4cyclictest131rcu_preempt20:20:131
6154992214,5cyclictest0-21swapper/120:15:291
615499221,17cyclictest31552-21ssh22:25:131
615499221,17cyclictest24302-21diskmemload21:25:141
615499221,15cyclictest630-21Async09:36:311
617099219,9cyclictest0-21swapper/321:24:213
617099214,11cyclictest0-21swapper/321:00:123
617099212,13cyclictest36066irq/59-eth000:20:103
617099212,13cyclictest36066irq/59-eth000:20:093
616299219,5cyclictest5441-21ssh00:32:192
615499211,15cyclictest9470-21grep19:45:221
615499211,15cyclictest9470-21grep19:45:221
615499211,11cyclictest0-21swapper/120:40:211
6154992110,8cyclictest4257-21sed21:50:291
615499210,12cyclictest16207-21kthreadcore20:25:191
6170992010,8cyclictest0-21swapper/322:40:553
616299208,3cyclictest0-21swapper/220:35:222
616299207,10cyclictest0-21swapper/220:05:172
616299206,11cyclictest27-21ksoftirqd/200:40:012
616299205,13cyclictest18678-21fschecks_count22:00:172
616299205,13cyclictest18678-21fschecks_count22:00:172
616299201,9cyclictest10957-21grep19:15:012
616299201,8cyclictest13278-21/usr/sbin/munin00:20:122
616299201,8cyclictest13278-21/usr/sbin/munin00:20:122
616299201,16cyclictest17354-21grep19:55:162
616299201,10cyclictest22649-21cut21:05:192
616299201,10cyclictest22225-21fschecks_count20:00:152
616299201,10cyclictest22225-21fschecks_count20:00:142
616299201,10cyclictest11770-21cut21:55:222
615499209,8cyclictest18535-21cpuspeed00:05:141
6154992014,4cyclictest31498-21sh00:30:141
6154992011,6cyclictest18465-21kthreadcore23:10:211
6154992011,3cyclictest24302-21diskmemload00:28:551
615499201,10cyclictest5801-21irqrtprio21:35:191
617099198,6cyclictest0-21swapper/300:00:173
617099195,11cyclictest0-21swapper/322:20:233
617099193,11cyclictest0-21swapper/323:16:233
617099193,11cyclictest0-21swapper/322:26:523
617099192,12cyclictest0-21swapper/322:57:413
617099192,11cyclictest35966irq/58-eth022:35:113
617099192,11cyclictest0-21swapper/320:15:143
617099192,10cyclictest0-21swapper/320:05:133
6170991911,4cyclictest0-21swapper/300:16:383
617099191,13cyclictest0-21swapper/319:45:173
617099191,13cyclictest0-21swapper/319:45:163
617099191,12cyclictest0-21swapper/319:20:223
616299195,6cyclictest14695-21awk21:00:012
616299193,4cyclictest0-21swapper/220:10:132
616299192,4cyclictest131rcu_preempt20:15:202
616299191,9cyclictest9017-21grep20:50:282
616299191,9cyclictest24020-21missed_timers19:25:232
616299191,9cyclictest11319-21sed23:25:172
616299191,8cyclictest3352-21kthreadcore20:45:222
616299191,15cyclictest2717-21latency_hist19:40:022
616299191,15cyclictest20405-21kthreadcore20:30:222
616299191,15cyclictest20405-21kthreadcore20:30:222
616299191,10cyclictest15128-21fschecks_time21:00:162
615499198,2cyclictest0-21swapper/123:43:031
615499192,7cyclictest131rcu_preempt19:10:111
615499192,13cyclictest22468-21apt-get22:20:111
615499191,2cyclictest131rcu_preempt00:00:141
6154991912,4cyclictest29495-21usage_EthcTask23:15:321
615499191,16cyclictest20400-21kthreadcore19:55:231
615499191,15cyclictest20702-21apt-get19:25:131
615499191,15cyclictest19212-21grep20:30:211
615499191,15cyclictest19212-21grep20:30:211
615499190,4cyclictest28608-21ssh21:30:121
617099188,8cyclictest0-21swapper/300:31:383
617099183,6cyclictest0-21swapper/320:25:093
617099182,10cyclictest0-21swapper/320:55:143
617099182,10cyclictest0-21swapper/320:20:053
617099182,10cyclictest0-21swapper/320:10:143
617099182,10cyclictest0-21swapper/320:00:133
617099182,10cyclictest0-21swapper/320:00:123
616299181,8cyclictest14063-21grep20:25:162
616299181,14cyclictest11950-21kthreadcore19:15:242
615499188,8cyclictest5386-21df_inode20:50:131
615499185,3cyclictest131rcu_preempt22:41:541
615499185,3cyclictest10180-21ssh23:24:511
615499184,10cyclictest2758-21grep19:40:021
615499182,12cyclictest31763-21diskstats20:10:161
615499181,8cyclictest131rcu_preempt23:25:121
6154991812,4cyclictest0-21swapper/120:00:271
6154991812,4cyclictest0-21swapper/120:00:261
6154991812,3cyclictest22324-21sendmail_mailqu22:00:291
6154991812,3cyclictest22324-21sendmail_mailqu22:00:291
615499181,14cyclictest24302-21diskmemload00:35:141
6154991810,6cyclictest0-21swapper/119:10:021
615499180,6cyclictest14533-21/usr/sbin/munin21:40:161
617099172,10cyclictest0-21swapper/321:05:133
617099172,10cyclictest0-21swapper/320:50:133
617099172,10cyclictest0-21swapper/320:45:133
617099172,10cyclictest0-21swapper/320:40:143
617099172,10cyclictest0-21swapper/319:55:153
617099172,10cyclictest0-21swapper/319:35:133
617099171,9cyclictest0-21swapper/319:25:123
616299171,13cyclictest6255-21latency_hist19:10:022
615499171,8cyclictest10963-21ps21:20:191
615499171,6cyclictest4035-21kthreadcore19:40:201
6154991711,4cyclictest21-21ksoftirqd/123:50:131
615499171,13cyclictest17858-21ssh23:45:221
615499171,13cyclictest17057-21kthreadcore19:20:191
617099162,10cyclictest0-21swapper/319:15:183
615499162,11cyclictest23966-21kernelversion20:35:181
615499162,11cyclictest12875-21kthreadcore19:15:241
615499161,9cyclictest19798-21cpuspeed21:05:131
6154991611,3cyclictest15581-21kernelversion21:00:181
615499161,12cyclictest26859-21apt-get20:05:131
615499161,12cyclictest12409-21apt-get19:50:131
615499161,12cyclictest12409-21apt-get19:50:131
617099151,9cyclictest0-21swapper/319:30:143
617099141,10cyclictest0-21swapper/319:40:203
617099123,7cyclictest0-21swapper/320:30:153
617099123,7cyclictest0-21swapper/320:30:153
617099121,6cyclictest0-21swapper/319:10:183
617099101,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional