You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-19 - 18:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sun Jul 19, 2026 12:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1942099490,40cyclictest310-21ssh10:55:141
19429994530,9cyclictest11303-21ssh10:25:162
1942999452,8cyclictest17553-21ssh09:50:312
1942099451,2cyclictest25578-21apt-get07:50:131
1942099451,2cyclictest20797-21apt-get07:45:141
1942099443,33cyclictest2002-21ssh10:20:121
1942099441,28cyclictest9908-21ssh12:30:151
1941099445,8cyclictest131rcu_preempt07:45:130
1944499432,23cyclictest35966irq/58-eth010:00:133
1944499432,23cyclictest35966irq/58-eth010:00:133
1942099433,34cyclictest9087-21ssh09:30:121
1942099431,3cyclictest0-21swapper/108:30:131
1942099425,30cyclictest11593-21ssh11:55:121
1942999412,34cyclictest0-21swapper/209:48:312
1942999412,21cyclictest24746-21usage_EthcTask10:30:312
1942999412,19cyclictest21686-21/usr/sbin/munin09:55:152
1942099413,30cyclictest0-21swapper/110:25:311
1942099412,2cyclictest6421-21apt-get07:30:141
1942099411,3cyclictest0-21swapper/108:40:131
1941099412,35cyclictest23425-21ssh09:20:130
1941099411,34cyclictest6218-21rm12:10:120
1942099402,2cyclictest0-21swapper/107:55:131
1942099401,6cyclictest0-21swapper/107:10:141
1942099401,3cyclictest16985-21apt-get11:40:131
1942099401,3cyclictest0-21swapper/111:20:131
1942099401,13cyclictest21834-21apt-get11:25:031
19444993929,3cyclictest35966irq/58-eth009:15:143
1942999392,10cyclictest5086-21kthreadcore11:15:192
1942999391,9cyclictest1771-21kthreadcore10:00:222
1942999391,9cyclictest1771-21kthreadcore10:00:222
1942099392,24cyclictest15215-21ssh12:15:131
1942099391,2cyclictest0-21swapper/111:15:121
1942099390,3cyclictest0-21swapper/107:25:121
1941099393,31cyclictest17011-21ssh11:40:140
1941099392,27cyclictest11077-21ssh10:25:120
19444993831,3cyclictest35966irq/58-eth009:45:133
1942999381,9cyclictest18460-21latency_hist12:35:012
1942999381,10cyclictest8546-21cut10:05:182
1942999380,21cyclictest14792-21ssh10:45:142
1942099381,6cyclictest0-21swapper/108:00:131
1942099381,4cyclictest27983-21apt-get09:00:121
1942099381,3cyclictest0-21swapper/109:25:141
1942099381,31cyclictest0-21swapper/110:00:131
1942099381,31cyclictest0-21swapper/110:00:121
1944499378,11cyclictest0-21swapper/310:23:313
1942999371,6cyclictest15468-21grep12:15:182
1942999370,9cyclictest25540-21grep11:25:232
1942099372,3cyclictest2587-21apt-get11:50:141
1942099372,2cyclictest24236-21apt-get12:20:121
1942099371,3cyclictest0-21swapper/108:35:121
1942099371,3cyclictest0-21swapper/107:20:141
1942099370,5cyclictest6199-21apt-get12:10:131
1942999362,17cyclictest10244-21ssh10:24:302
1942999361,9cyclictest334-21sendmail_mailqu07:20:272
1942999361,9cyclictest30941-21irqrtprio07:55:192
1942099361,3cyclictest0-21swapper/112:25:151
1942099361,3cyclictest0-21swapper/111:10:111
1942099361,12cyclictest20141-21ssh10:30:141
19410993615,15cyclictest12-21ksoftirqd/007:30:150
1942999351,8cyclictest24647-21unixbench_singl07:45:292
1942999351,8cyclictest20340-21latency_hist12:00:012
1942999351,7cyclictest7384-21latency_hist08:05:012
1942999351,13cyclictest6306-21latency_hist07:30:012
1942999351,13cyclictest5814-21cron12:10:002
1942099353,5cyclictest0-21swapper/108:10:121
1942099351,3cyclictest0-21swapper/112:35:131
1942099351,3cyclictest0-21swapper/108:05:151
1942099351,3cyclictest0-21swapper/107:35:141
1942099351,2cyclictest16382-21apt-get10:10:131
1942099350,3cyclictest25391-21apt-get10:15:121
1944499341,22cyclictest35966irq/58-eth012:15:133
1942999341,9cyclictest16362-21acpi10:10:112
1942999341,14cyclictest21936-21/usr/sbin/munin08:20:282
1942999341,14cyclictest16669-21grep11:20:242
1942999341,13cyclictest11183-21latency_hist07:35:022
1942999341,10cyclictest20011-21tr11:40:212
19429993410,9cyclictest0-21swapper/208:40:242
1942099342,2cyclictest26765-21apt-get08:25:131
1942099341,30cyclictest22072-21timerandwakeup08:50:261
1942999331,9cyclictest1818-21kthreadcore08:30:222
1942999331,9cyclictest11998-21cut09:30:222
19429993313,4cyclictest0-21swapper/211:15:012
1942999331,14cyclictest27146-21grep09:20:272
1942099331,8cyclictest28238-21/usr/sbin/munin07:15:291
1942099331,7cyclictest20588-21/usr/sbin/munin12:00:271
1942099331,27cyclictest14899-21diskstats10:45:161
1942099331,11cyclictest26985-21netstat09:20:241
1942099331,10cyclictest16007-21munin-run10:10:011
1942099330,6cyclictest13367-21find11:00:281
1942099330,3cyclictest26035-21apt-get11:45:151
19410993311,9cyclictest131rcu_preempt12:25:140
1944499323,14cyclictest0-21swapper/309:55:313
1942999327,7cyclictest27-21ksoftirqd/209:40:012
1942999321,10cyclictest30360-21usage_Anybus11:45:312
1942999320,9cyclictest31499-21uname09:00:282
1942099321,9cyclictest9160-21grep11:35:181
1942099321,9cyclictest29786-21fschecks_count12:05:161
1942099321,8cyclictest24680-21cut08:55:231
1942099321,7cyclictest7421-21tr09:10:201
1942099321,7cyclictest15986-21sort07:40:121
1942099321,3cyclictest0-21swapper/108:45:141
1942099321,3cyclictest0-21swapper/108:15:121
1942099321,13cyclictest29142-21apt-get10:35:121
1942099320,9cyclictest3845-21/usr/sbin/munin09:05:271
1942099320,9cyclictest3845-21/usr/sbin/munin09:05:271
1942099320,9cyclictest18013-21ntpq09:15:261
1941099322,22cyclictest7401-21ssh10:05:130
19410993218,9cyclictest630-21Async09:36:310
19410993218,9cyclictest630-21Async09:36:310
1944499314,13cyclictest0-21swapper/310:25:313
1942999319,7cyclictest4922-21diskmemload09:28:512
1942999313,22cyclictest0-21swapper/209:41:322
1942999311,13cyclictest22193-21unixbench_multi08:50:272
1942099311,9cyclictest6454-21ssh10:40:191
1942099311,8cyclictest22590-21irqstats08:20:191
1942099311,8cyclictest18111-21/usr/sbin/munin09:35:331
1942099311,7cyclictest3643-21/usr/sbin/munin09:45:161
1942099311,10cyclictest27477-21ssh09:56:371
1942099310,8cyclictest27563-21sendmail_mailqu10:50:281
1941099314,6cyclictest131rcu_preempt10:30:140
1941099314,20cyclictest16381-21ssh10:10:120
1941099312,23cyclictest3691-21ssh09:45:140
1941099311,17cyclictest4922-21diskmemload12:20:130
1942999303,20cyclictest27346irq/40-dwc2_hso09:05:142
1942999303,20cyclictest27346irq/40-dwc2_hso09:05:132
1942999302,21cyclictest25677-21ssh10:15:162
1942099301,7cyclictest32112-21cut11:30:211
1942099301,7cyclictest32112-21cut11:30:201
1942099301,3cyclictest18416-21apt-get11:05:131
1942999291,8cyclictest18419-21ssh11:05:122
1942099291,6cyclictest27073-21/usr/sbin/munin09:40:161
19420992914,12cyclictest12696-21ssh09:50:121
1941099293,23cyclictest27482-21ssh11:10:110
1941099293,23cyclictest18782-21ssh12:35:120
1941099292,19cyclictest12-21ksoftirqd/011:50:140
1941099292,15cyclictest20620-21ssh12:00:120
19410992919,5cyclictest12-21ksoftirqd/010:55:130
1941099282,19cyclictest12812-21diskstats09:50:130
1941099282,18cyclictest22163-21ssh11:25:060
19410992810,9cyclictest0-21swapper/008:25:130
19444992722,2cyclictest0-21swapper/310:10:123
1944499272,18cyclictest0-21swapper/310:45:143
1942999273,18cyclictest27346irq/40-dwc2_hso07:15:132
1942999272,16cyclictest2610-21ssh11:50:152
1941099277,15cyclictest0-21swapper/009:15:140
1941099275,15cyclictest25396-21ssh10:15:110
19410992714,7cyclictest0-21swapper/011:00:110
1944499263,17cyclictest0-21swapper/311:37:313
19444992617,6cyclictest35966irq/58-eth011:20:273
1942999262,20cyclictest0-21swapper/208:35:132
19429992621,3cyclictest27-21ksoftirqd/212:10:122
1942999261,8cyclictest930-21snmpd07:35:122
19429992613,10cyclictest19681-21grep07:10:162
1941099262,17cyclictest844-21munin-node10:35:130
1941099261,20cyclictest7910-21ssh11:35:110
1941099261,20cyclictest23769-21ssh10:50:140
1944499251,17cyclictest35966irq/58-eth007:55:133
19429992517,6cyclictest0-21swapper/212:20:122
1942999251,17cyclictest27346irq/40-dwc2_hso08:15:112
1942099251,5cyclictest19482-21latency_hist07:10:021
1941099253,20cyclictest12-21ksoftirqd/007:40:140
1941099253,18cyclictest5764-21ssh10:40:150
1941099252,15cyclictest19397-21cyclictest07:25:120
19410992518,3cyclictest0-21swapper/009:10:120
19410992512,5cyclictest0-21swapper/008:55:150
1941099243,16cyclictest0-21swapper/009:55:140
1941099243,14cyclictest630-21Async09:36:310
1941099242,19cyclictest30759-21/usr/sbin/munin10:00:160
1941099242,19cyclictest30759-21/usr/sbin/munin10:00:150
1941099242,15cyclictest18146-21ssh09:35:140
1941099242,13cyclictest26055-21ssh11:45:140
1944499234,16cyclictest0-21swapper/309:20:133
1942999231,3cyclictest131rcu_preempt12:00:132
19429992311,5cyclictest0-21swapper/208:05:242
1942999222,15cyclictest0-21swapper/211:39:312
1942999222,15cyclictest0-21swapper/210:55:312
1942999221,11cyclictest7638-21latency_hist11:35:012
1942999221,11cyclictest7638-21latency_hist11:35:012
19429992210,9cyclictest0-21swapper/211:01:022
1941099222,8cyclictest18840-21/usr/sbin/munin12:15:250
1941099221,18cyclictest7675-21ls08:35:270
19410992211,5cyclictest0-21swapper/010:45:270
1944499219,9cyclictest35966irq/58-eth012:20:133
1944499216,11cyclictest35966irq/58-eth011:30:143
1944499216,11cyclictest35966irq/58-eth011:30:143
1944499215,11cyclictest0-21swapper/312:10:123
1944499213,12cyclictest0-21swapper/309:41:313
1944499211,14cyclictest36066irq/59-eth010:15:113
1944499211,13cyclictest35966irq/58-eth011:15:143
1944499211,12cyclictest0-21swapper/307:30:143
19444992110,6cyclictest0-21swapper/311:51:293
1942999212,11cyclictest25750-21tune2fs07:50:162
1942999212,10cyclictest27405-21grep08:25:202
1942999211,10cyclictest7614-21ssh09:10:212
1942999211,10cyclictest16338-21kthreadcore09:15:212
1941099218,10cyclictest16903-21cron08:15:010
1941099213,15cyclictest1682-21cat07:55:290
1941099211,12cyclictest7729-21uniq08:05:170
1944499209,9cyclictest0-21swapper/310:08:153
1944499209,9cyclictest0-21swapper/309:10:143
1944499208,9cyclictest0-21swapper/312:35:293
19444992013,5cyclictest35966irq/58-eth009:35:163
19444992011,5cyclictest35966irq/58-eth008:35:123
19444992010,6cyclictest0-21swapper/310:50:303
1942999209,9cyclictest19026-21fschecks_time12:35:162
1942999208,5cyclictest12097-21cron08:10:022
1942999207,4cyclictest0-21swapper/208:45:202
1942999202,15cyclictest17063-21kthreadcore07:40:202
1942999201,13cyclictest5709-21/usr/sbin/munin10:40:152
1942999201,12cyclictest23208-21cpuspeed08:55:152
1942999201,10cyclictest32377-21awk10:55:012
1941099203,5cyclictest21726-21sendmail08:20:010
1941099202,11cyclictest29635-21ssh12:05:140
1941099201,7cyclictest9058-21/usr/sbin/munin09:30:260
1941099201,15cyclictest24364-21apt-get07:15:130
19410992010,7cyclictest17864-21cat11:20:320
1944499198,6cyclictest0-21swapper/310:55:483
1944499192,11cyclictest0-21swapper/307:20:153
1944499191,12cyclictest0-21swapper/311:05:183
19444991910,6cyclictest0-21swapper/312:06:313
1942999191,9cyclictest1023-21fschecks_count12:25:172
1942999191,15cyclictest29371-21grep10:35:162
1941099196,9cyclictest15593-21grep11:55:290
1941099193,13cyclictest27271-21grep09:40:140
1941099193,13cyclictest15756-21ls07:40:010
1941099193,13cyclictest1509-21latency_hist07:25:010
1941099191,9cyclictest22472-21irqrtprio08:20:200
1941099191,7cyclictest32561-21munin-run09:05:010
1941099191,14cyclictest3371-21ssh10:20:190
1944499189,7cyclictest0-21swapper/311:43:003
1944499189,7cyclictest0-21swapper/309:33:283
1944499189,6cyclictest35966irq/58-eth012:25:153
1944499189,6cyclictest0-21swapper/311:55:133
1944499187,9cyclictest0-21swapper/312:03:163
1944499184,11cyclictest35966irq/58-eth011:45:133
1944499183,10cyclictest0-21swapper/309:50:313
1944499182,12cyclictest35966irq/58-eth011:00:113
1944499182,10cyclictest0-21swapper/309:25:053
1944499182,10cyclictest0-21swapper/308:50:123
1944499182,10cyclictest0-21swapper/308:30:143
1944499181,11cyclictest0-21swapper/310:30:163
19444991810,5cyclictest35966irq/58-eth011:25:143
19444991810,5cyclictest35966irq/58-eth010:40:133
1942999182,7cyclictest19397-21cyclictest07:10:012
1941099188,7cyclictest24127-21cron07:15:020
1941099181,7cyclictest2714-21kthreadcore09:05:210
1941099181,7cyclictest2714-21kthreadcore09:05:210
1941099181,6cyclictest2605-21cut09:25:210
1941099181,14cyclictest10089-21kthreadcore08:40:200
1941099181,12cyclictest13384-21ssh12:30:240
19410991810,5cyclictest18719-21idleruntime11:05:170
1941099180,10cyclictest5246-21kthreadcore11:15:190
1944499179,6cyclictest0-21swapper/310:38:343
1944499179,5cyclictest35966irq/58-eth012:30:143
1944499172,10cyclictest0-21swapper/309:05:143
1944499172,10cyclictest0-21swapper/309:05:143
1944499172,10cyclictest0-21swapper/308:40:143
1944499172,10cyclictest0-21swapper/307:35:143
1944499171,11cyclictest0-21swapper/308:15:173
1941099178,7cyclictest6631-21grep08:00:290
1941099172,10cyclictest31572-21apt-get08:30:140
1941099172,10cyclictest0-21swapper/008:45:140
1941099171,12cyclictest18365-21apt-get08:50:110
1944499168,6cyclictest0-21swapper/311:12:203
1944499164,4cyclictest0-21swapper/307:25:123
1944499162,10cyclictest0-21swapper/309:00:143
1944499156,7cyclictest0-21swapper/308:55:143
1944499152,10cyclictest0-21swapper/307:45:153
1944499151,9cyclictest0-21swapper/308:25:143
19444991510,3cyclictest0-21swapper/308:08:313
1944499149,3cyclictest0-21swapper/308:24:313
1944499144,8cyclictest0-21swapper/307:50:133
1944499141,9cyclictest0-21swapper/307:10:143
1944499141,6cyclictest0-21swapper/307:15:193
1941099146,5cyclictest19441-21latency_hist07:10:010
1944499133,8cyclictest0-21swapper/307:40:153
1944499122,5cyclictest0-21swapper/308:10:143
1944499111,7cyclictest0-21swapper/308:45:193
1944499111,6cyclictest0-21swapper/308:00:133
194449991,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional