You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-14 - 19:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri Aug 14, 2026 12:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2724299814,45cyclictest18863-21ssh10:25:131
2722199576,30cyclictest31413-21ssh09:20:120
2724299541,42cyclictest17118-21ssh09:30:131
2724299533,40cyclictest13483-21ssh10:40:121
27221995120,23cyclictest12-21ksoftirqd/009:40:130
2722199509,26cyclictest5920-21ssh11:30:130
2725899491,30cyclictest11680-21ssh09:45:142
2722199486,7cyclictest131rcu_preempt07:15:130
2724299473,5cyclictest131rcu_preempt10:45:121
27221994612,16cyclictest131rcu_preempt10:50:110
2727399452,27cyclictest35966irq/58-eth009:25:163
2724299452,5cyclictest131rcu_preempt10:15:121
27258994416,22cyclictest27-21ksoftirqd/211:35:122
27221994429,8cyclictest0-21swapper/009:45:130
2724299422,30cyclictest492-21ssh11:45:141
2725899414,23cyclictest27-21ksoftirqd/210:55:142
2727399405,22cyclictest35966irq/58-eth009:45:133
2727399404,24cyclictest35966irq/58-eth011:25:133
2725899392,29cyclictest2305-21sh11:10:042
2727399371,23cyclictest35966irq/58-eth011:10:123
2725899362,28cyclictest13502-21ssh10:40:142
2725899362,18cyclictest20323-21cat11:55:192
2725899361,8cyclictest28982-21kworker/2:111:45:132
2724299360,28cyclictest21967-21ssh12:15:141
2727399352,20cyclictest35966irq/58-eth011:30:123
2725899340,26cyclictest9458-21ssh11:50:132
2724299341,23cyclictest17063-21ssh11:00:141
2722199344,26cyclictest16430-21ssh12:30:120
2722199344,22cyclictest12963-21diskmemload11:35:120
2722199342,22cyclictest12-21ksoftirqd/008:05:140
27221993420,7cyclictest12963-21diskmemload10:20:150
27221993420,7cyclictest12963-21diskmemload10:20:150
2722199341,22cyclictest20607-21ssh09:50:130
2724299331,23cyclictest13452-21ssh09:10:131
2727399324,21cyclictest35966irq/58-eth011:05:133
2725899322,15cyclictest24155-21ssh10:10:132
2727399314,13cyclictest35966irq/58-eth011:40:143
27258993010,13cyclictest0-21swapper/211:05:132
2724299300,23cyclictest7411-21ssh12:25:121
2722199302,22cyclictest18401-21ssh11:55:120
2725899293,21cyclictest27346irq/40-dwc2_hso12:30:132
2725899293,18cyclictest18816-21/usr/sbin/munin10:25:242
2725899293,15cyclictest27346irq/40-dwc2_hso08:00:142
2725899292,19cyclictest31499-21ssh10:50:132
2725899292,17cyclictest27346irq/40-dwc2_hso07:40:122
2725899292,11cyclictest1155-21ssh10:31:342
27258992911,5cyclictest27-21ksoftirqd/210:15:272
2724299293,18cyclictest628-21tcf-agent08:35:141
27242992919,6cyclictest29326-21kworker/1:210:50:121
2722199292,22cyclictest24170-21ssh10:10:140
2722199292,18cyclictest25916-21sh11:05:130
27221992913,10cyclictest6203-21diskmemload10:00:130
27221992912,14cyclictest12-21ksoftirqd/011:25:120
2727399282,16cyclictest35966irq/58-eth010:35:123
2725899283,17cyclictest27346irq/40-dwc2_hso07:50:112
2725899282,13cyclictest22415-21ssh09:15:112
2724299283,15cyclictest4417-21ssh10:35:121
2724299282,19cyclictest27876-21ssh10:30:131
2724299282,18cyclictest15205-21sh10:05:141
2727399271,17cyclictest35966irq/58-eth010:30:133
2725899279,9cyclictest30519-21sshd11:42:332
2725899277,16cyclictest536-21usb-storage08:45:052
2725899273,9cyclictest0-21swapper/208:05:142
2725899272,19cyclictest0-21swapper/209:40:132
2725899271,6cyclictest6340-21ssh10:00:152
27242992712,10cyclictest21-21ksoftirqd/109:55:131
27242992712,10cyclictest21-21ksoftirqd/109:55:131
27242992711,10cyclictest5820-21sshd11:30:071
2722199276,12cyclictest0-21swapper/008:20:140
2722199272,19cyclictest19215-21ssh10:41:340
2727399263,15cyclictest0-21swapper/310:05:143
2725899264,17cyclictest930-21snmpd09:20:132
2725899262,18cyclictest20392-21apt-get11:20:122
2722199262,17cyclictest12-21ksoftirqd/011:45:130
27273992510,9cyclictest35966irq/58-eth010:40:143
2725899252,17cyclictest27346irq/40-dwc2_hso07:15:132
2725899252,15cyclictest0-21swapper/207:10:132
2725899251,18cyclictest11303-21ssh11:15:042
2725899251,14cyclictest31268-21ssh12:20:182
2725899251,13cyclictest4608-21fschecks_time10:35:162
27258992511,11cyclictest0-21swapper/207:35:282
2724299255,14cyclictest201rcuc/109:15:121
2722199258,12cyclictest12-21ksoftirqd/010:34:580
2722199250,2cyclictest131rcu_preempt10:15:270
27273992412,9cyclictest0-21swapper/311:55:153
2725899242,19cyclictest12691-21latency_hist12:10:012
2725899241,8cyclictest14151-21ssh11:34:332
27258992411,6cyclictest29905-21ssh12:19:152
2725899241,14cyclictest15196-21ssh10:05:132
27258992411,10cyclictest20616-21diskmemload09:50:142
27258992410,7cyclictest12944-21ssh12:26:172
2727399233,13cyclictest0-21swapper/310:55:043
27273992317,4cyclictest36066irq/59-eth012:35:153
27273992311,6cyclictest0-21swapper/307:25:163
2725899237,10cyclictest131rcu_preempt12:35:242
2725899232,6cyclictest0-21swapper/210:45:342
2725899232,5cyclictest30007-21sshd12:00:212
2725899231,9cyclictest13039-21latency_hist09:10:012
27258992312,8cyclictest0-21swapper/209:25:042
2725899231,18cyclictest3892-21tail09:00:182
2725899231,17cyclictest19877-21grep08:40:232
27258992310,11cyclictest19797-21ssh12:12:542
2724299232,4cyclictest131rcu_preempt10:20:291
2724299232,4cyclictest131rcu_preempt10:20:291
2724299232,16cyclictest8154-21ssh09:25:151
27242992311,9cyclictest73-21kswapd012:00:131
2724299231,18cyclictest15692-21ls08:05:181
2722199232,19cyclictest12989-21ssh12:10:130
2722199232,17cyclictest4112-21diskstats12:05:170
27221992314,6cyclictest0-21swapper/008:00:290
27221992310,3cyclictest0-21swapper/009:00:290
2725899222,6cyclictest25543-21grep08:15:192
2725899222,12cyclictest29384-21ssh11:25:122
2725899222,10cyclictest6497-21kthreadcore07:55:202
2725899221,9cyclictest32543-21grep07:45:292
2725899221,11cyclictest307-21kthreadcore09:55:242
2725899221,11cyclictest307-21kthreadcore09:55:242
2725899221,11cyclictest23858-21switchtime08:10:272
2725899221,10cyclictest30039-21switchtime09:35:302
2725899220,14cyclictest15490-21ls08:35:292
2724299221,18cyclictest27648-21ls07:40:271
2724299221,18cyclictest14020-21ls07:30:021
2724299221,18cyclictest1192-21cut07:50:151
2724299221,17cyclictest3998-21ssh12:05:151
2724299221,17cyclictest32646-21ls11:45:011
2724299221,16cyclictest10645-21apt-get08:00:131
2724299221,12cyclictest6820-21grep09:40:301
2722199222,2cyclictest131rcu_preempt12:25:330
2722199222,17cyclictest20404-21ssh11:20:120
2722199222,15cyclictest26113-21ssh09:35:150
2722199221,7cyclictest12973-21apt-get09:10:030
27221992212,6cyclictest6330-21sh11:10:270
2722199221,18cyclictest6065-21ls07:55:180
2722199221,18cyclictest25034-21find08:15:130
2722199220,3cyclictest131rcu_preempt12:35:040
2722199220,2cyclictest131rcu_preempt12:15:190
2722199220,18cyclictest22497-21cpuspeed09:15:130
2727399214,9cyclictest35966irq/58-eth008:20:123
2727399212,13cyclictest35966irq/58-eth010:00:123
27273992110,8cyclictest0-21swapper/312:20:293
2725899219,9cyclictest16880-21ssh11:00:022
2725899212,3cyclictest0-21swapper/208:20:132
27258992114,4cyclictest0-21swapper/207:25:192
2725899211,11cyclictest17827-21threads07:30:312
2725899211,11cyclictest17252-21awk09:30:162
2725899211,10cyclictest8662-21kthreadcore08:30:212
2725899211,10cyclictest15570-21kthreadcore09:10:222
2725899211,10cyclictest12360-21ssh10:20:222
2725899211,10cyclictest12360-21ssh10:20:222
2725899210,14cyclictest26316-21sort08:50:122
2724299213,11cyclictest25890-21ssh11:05:101
27242992113,5cyclictest20369-21/usr/sbin/munin11:20:241
27242992112,7cyclictest622-21sh11:25:261
2724299211,17cyclictest15117-21cut11:35:151
2724299211,16cyclictest7029-21latency_hist12:25:021
2722199219,7cyclictest12-21ksoftirqd/011:00:150
2722199212,4cyclictest131rcu_preempt08:50:280
2722199212,3cyclictest131rcu_preempt10:06:530
27221992114,4cyclictest0-21swapper/009:25:170
27221992113,6cyclictest14339-21sshd11:50:330
27221992113,5cyclictest0-21swapper/008:45:170
2722199211,2cyclictest131rcu_preempt11:43:570
2722199211,11cyclictest5102-21tr07:20:180
2722199211,10cyclictest17340-21fschecks_time09:30:170
2722199210,18cyclictest12637-21ls07:25:250
2727399209,8cyclictest0-21swapper/312:31:393
2727399203,11cyclictest0-21swapper/311:15:043
27273992011,6cyclictest35966irq/58-eth010:10:143
2725899201,9cyclictest4778-21fschecks_time07:20:162
2725899201,9cyclictest434-21kthreadcore08:55:242
2725899200,2cyclictest0-21swapper/208:25:172
2724299203,5cyclictest21-21ksoftirqd/109:20:291
2724299201,17cyclictest24307-21cut10:10:161
2724299201,16cyclictest14177-21find07:30:141
2724299201,15cyclictest6499-21tail07:55:201
2724299201,14cyclictest30057-21timerandwakeup08:50:281
2722199202,15cyclictest22487-21ssh10:45:140
2722199202,14cyclictest27465-21ssh12:00:130
27221992013,4cyclictest0-21swapper/009:05:140
27221992013,3cyclictest12-21ksoftirqd/007:45:120
27221992011,6cyclictest1-21systemd07:34:130
27221992011,5cyclictest5145-21kernelversion10:35:190
27221992010,7cyclictest2030-21memory12:20:240
2722199200,17cyclictest0-21swapper/010:25:120
2727399199,8cyclictest0-21swapper/311:22:563
2727399198,5cyclictest0-21swapper/309:15:133
2727399196,10cyclictest0-21swapper/312:05:173
2727399192,10cyclictest0-21swapper/308:50:133
27273991913,4cyclictest35966irq/58-eth012:25:143
2727399191,12cyclictest0-21swapper/308:00:173
2727399191,12cyclictest0-21swapper/307:35:293
27273991910,7cyclictest0-21swapper/309:11:483
27273991910,6cyclictest35966irq/58-eth011:50:133
27273991910,6cyclictest35966irq/58-eth010:15:123
27273991910,6cyclictest35966irq/58-eth009:20:133
27273991910,6cyclictest0-21swapper/312:10:233
2724299196,4cyclictest8716-21grep09:05:171
2724299193,9cyclictest0-21swapper/111:10:041
2724299192,12cyclictest26103-21ssh09:35:141
27242991913,4cyclictest0-21swapper/107:10:171
2724299191,12cyclictest32368-21cut07:15:161
2722199192,15cyclictest20286-21switchtime08:40:290
2722199192,11cyclictest7942-21ssh10:55:040
27221991913,4cyclictest536-21usb-storage07:40:110
27221991910,7cyclictest12-21ksoftirqd/009:55:140
27221991910,7cyclictest12-21ksoftirqd/009:55:140
2727399189,6cyclictest35966irq/58-eth011:35:123
2727399189,6cyclictest0-21swapper/309:34:053
2727399188,7cyclictest35966irq/58-eth012:15:133
2727399186,9cyclictest0-21swapper/311:45:343
2727399183,10cyclictest0-21swapper/310:25:343
2727399182,10cyclictest0-21swapper/308:25:133
2727399182,10cyclictest0-21swapper/308:05:143
2727399182,10cyclictest0-21swapper/307:45:143
2727399182,10cyclictest0-21swapper/307:30:163
2727399181,13cyclictest0-21swapper/309:35:243
27273991810,6cyclictest0-21swapper/312:00:443
2724299188,5cyclictest18452-21cpuspeed11:55:131
2724299185,10cyclictest20425-21fschecks_time08:10:161
2724299181,7cyclictest12955-21memory11:50:231
2724299181,15cyclictest9634-21latency10:00:221
2724299181,13cyclictest20054-21ntp_192.168.11412:30:231
2724299180,8cyclictest14594-21grep11:15:211
2722199182,3cyclictest131rcu_preempt07:35:300
2722199181,9cyclictest12600-21kthreadcore11:15:190
2722199181,4cyclictest131rcu_preempt08:30:240
2727399179,6cyclictest0-21swapper/310:20:113
2727399179,6cyclictest0-21swapper/310:20:113
2727399179,6cyclictest0-21swapper/309:57:073
2727399179,6cyclictest0-21swapper/309:57:073
2727399179,5cyclictest35966irq/58-eth009:50:133
2727399176,6cyclictest0-21swapper/309:40:353
2727399172,10cyclictest0-21swapper/311:00:153
2727399172,10cyclictest0-21swapper/310:45:343
2727399172,10cyclictest0-21swapper/307:50:133
27273991711,4cyclictest0-21swapper/310:50:203
2727399171,10cyclictest0-21swapper/308:45:283
2727399171,10cyclictest0-21swapper/307:40:043
2724299179,6cyclictest1994-21sshd08:21:341
2724299178,6cyclictest18456-21cut08:40:211
2724299175,9cyclictest0-21swapper/108:55:171
2724299174,10cyclictest22125-21kthreadcore09:50:201
2724299173,3cyclictest131rcu_preempt07:25:011
2724299171,9cyclictest12946-21/usr/sbin/munin12:10:121
2724299171,6cyclictest7604-21apt-get10:55:021
2724299171,13cyclictest29084-21ntp_states12:35:261
2724299171,12cyclictest15365-21ntp_states09:45:271
2722199171,14cyclictest21096-21kthreadcore08:10:200
2722199171,11cyclictest3353-21idleruntime-cro09:00:010
27221991710,4cyclictest0-21swapper/008:35:190
2727399162,10cyclictest0-21swapper/307:15:143
2724299169,4cyclictest29606-21vmstat08:15:331
2724299166,8cyclictest3706-21fschecks_count09:00:161
2724299162,10cyclictest0-21swapper/107:35:141
2724299161,12cyclictest7150-21diskstats08:30:151
27242991610,4cyclictest0-21swapper/108:25:151
2724299160,3cyclictest53646usb-storage08:45:101
2722199162,11cyclictest1379-21irqcore07:50:160
2722199162,10cyclictest0-21swapper/007:10:130
2722199160,13cyclictest3362-21kthreadcore08:25:190
27273991510,3cyclictest33-21ksoftirqd/308:15:173
2725899154,8cyclictest0-21swapper/207:10:022
27242991510,3cyclictest28659-21/usr/sbin/munin07:45:151
2727399149,3cyclictest33-21ksoftirqd/307:55:263
2727399146,6cyclictest0-21swapper/308:55:263
2727399141,11cyclictest0-21swapper/309:10:013
2722199141,9cyclictest27224-21grep07:10:020
2727399138,3cyclictest0-21swapper/308:30:553
2727399138,3cyclictest0-21swapper/307:20:293
2727399136,5cyclictest0-21swapper/308:35:213
2727399133,8cyclictest0-21swapper/307:10:273
2724299131,3cyclictest131rcu_preempt07:10:021
2727399127,3cyclictest0-21swapper/308:40:123
2727399125,5cyclictest0-21swapper/308:10:203
2727399111,7cyclictest0-21swapper/309:05:013
2727399101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional