You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-03 - 01:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Wed Sep 02, 2026 12:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29764996936,25cyclictest35966irq/58-eth010:30:133
2975699607,28cyclictest29095-21ssh11:05:112
2975699603,41cyclictest2943-21ssh12:20:132
29764995739,6cyclictest35966irq/58-eth010:50:123
2975699522,39cyclictest13400-21ssh11:50:132
2973399512,29cyclictest0-21swapper/011:05:120
2976499506,24cyclictest35966irq/58-eth009:45:133
2976499502,30cyclictest35966irq/58-eth008:05:133
2975699484,23cyclictest27-21ksoftirqd/208:30:122
29746994814,23cyclictest21-21ksoftirqd/108:20:131
2976499474,27cyclictest35966irq/58-eth010:20:133
2973399472,3cyclictest131rcu_preempt12:10:120
2976499466,24cyclictest35966irq/58-eth011:55:133
2976499441,26cyclictest35966irq/58-eth012:30:143
2976499434,24cyclictest35966irq/58-eth009:25:133
2973399435,28cyclictest14177-21ssh09:45:130
2973399432,31cyclictest26936-21ssh10:10:120
2975699421,30cyclictest25449-21ssh10:45:132
2974699422,29cyclictest20091-21ssh11:00:131
2974699422,29cyclictest20091-21ssh11:00:131
2973399428,21cyclictest12-21ksoftirqd/008:25:140
2976499413,25cyclictest35966irq/58-eth010:55:143
2976499412,26cyclictest35966irq/58-eth009:50:123
2974699411,33cyclictest9536-21ssh11:30:121
2974699401,33cyclictest13415-21ssh11:50:141
2973399405,8cyclictest131rcu_preempt10:25:140
2976499394,23cyclictest35966irq/58-eth011:10:153
2976499394,21cyclictest35966irq/58-eth010:15:143
29764993919,12cyclictest35966irq/58-eth011:40:123
2975699396,26cyclictest12726-21cat12:05:292
2974699393,32cyclictest8831-21ssh10:00:121
2975699382,20cyclictest930-21snmpd08:50:132
2975699381,25cyclictest31514-21ssh12:00:132
29756993811,20cyclictest27-21ksoftirqd/209:10:122
2973399387,19cyclictest15376-21diskmemload10:45:130
2975699379,19cyclictest27-21ksoftirqd/210:05:132
2976499361,23cyclictest35966irq/58-eth010:40:133
2974699361,26cyclictest14769-21ssh11:15:151
29733993626,5cyclictest12-21ksoftirqd/008:45:120
2973399354,26cyclictest31523-21ssh12:00:130
2973399351,25cyclictest412-21ssh11:25:130
29746993421,8cyclictest0-21swapper/110:50:151
2973399343,22cyclictest24814-21ssh09:15:120
2973399342,24cyclictest12-21ksoftirqd/011:55:120
2976499332,17cyclictest35966irq/58-eth008:35:143
2975699336,15cyclictest27346irq/40-dwc2_hso07:10:132
2975699334,19cyclictest27346irq/40-dwc2_hso09:45:132
2975699333,17cyclictest15817-1kworker/u9:211:35:122
2974699336,20cyclictest5730-21ssh11:10:131
2974699333,18cyclictest21-21ksoftirqd/112:05:131
29746993313,13cyclictest21-21ksoftirqd/107:50:131
29746993313,13cyclictest21-21ksoftirqd/107:50:131
2974699330,27cyclictest0-21swapper/108:05:131
2976499324,18cyclictest35966irq/58-eth011:25:133
2976499322,18cyclictest35966irq/58-eth007:10:123
2973399322,23cyclictest16847-21ssh12:25:310
29733993218,6cyclictest12-21ksoftirqd/007:40:130
2976499313,19cyclictest35966irq/58-eth011:30:113
2975699312,16cyclictest24796-21ssh09:15:112
2975699312,14cyclictest9543-21sh11:30:132
2974699312,22cyclictest32366-21cat11:40:301
2973399313,23cyclictest15787-21ssh09:10:140
2976499302,19cyclictest35966irq/58-eth010:00:133
2975699302,20cyclictest30744-21ssh10:30:132
2975699302,16cyclictest20095-21ssh11:00:132
2975699302,16cyclictest20095-21ssh11:00:122
29756993015,12cyclictest0-21swapper/211:45:012
2975699301,18cyclictest5257-21ssh09:40:132
2975699301,18cyclictest5257-21ssh09:40:122
2975699301,15cyclictest28676-21ssh09:35:132
2974699302,22cyclictest30551-21ssh11:59:221
29746993014,12cyclictest0-21swapper/107:30:131
2975699296,15cyclictest27346irq/40-dwc2_hso07:20:142
2975699294,16cyclictest27346irq/40-dwc2_hso08:45:122
2974699292,20cyclictest18568-21ssh11:35:131
2974699291,8cyclictest21-21ksoftirqd/110:25:121
2973399292,19cyclictest12520-21ssh10:20:040
29733992911,7cyclictest29729-21cyclictest11:40:130
2976499285,14cyclictest35966irq/58-eth009:30:143
2975699282,21cyclictest0-21swapper/212:17:292
2975699281,16cyclictest27346irq/40-dwc2_hso08:25:142
2974699285,6cyclictest21-21ksoftirqd/110:12:261
2974699282,18cyclictest31245-21ssh12:15:361
29733992811,12cyclictest12-21ksoftirqd/012:22:300
2976499273,15cyclictest0-21swapper/312:25:313
2976499272,14cyclictest35966irq/58-eth008:10:133
2975699273,19cyclictest0-21swapper/212:14:262
2975699273,16cyclictest12639-21ssh10:20:112
29746992711,8cyclictest0-21swapper/107:15:121
2973399273,11cyclictest16421-21ssh10:40:110
2973399272,19cyclictest10542-21sh09:25:150
2976499266,13cyclictest35966irq/58-eth012:10:123
2976499261,15cyclictest35966irq/58-eth012:20:133
2975699265,12cyclictest27346irq/40-dwc2_hso09:05:122
2975699263,15cyclictest27346irq/40-dwc2_hso11:10:142
2975699262,9cyclictest0-21swapper/207:50:142
2975699262,9cyclictest0-21swapper/207:50:142
2975699262,10cyclictest0-21swapper/207:15:132
29756992615,8cyclictest261rcuc/210:25:142
2975699261,20cyclictest7336-21apt-get10:35:122
2974699264,13cyclictest21-21ksoftirqd/109:05:131
2974699261,19cyclictest1445-21ssh09:20:141
2973399262,18cyclictest15376-21diskmemload11:38:300
29733992612,6cyclictest12-21ksoftirqd/007:25:140
29733992612,6cyclictest12-21ksoftirqd/007:25:140
2976499252,18cyclictest35966irq/58-eth007:50:133
2976499252,18cyclictest35966irq/58-eth007:50:123
2975699252,15cyclictest27346irq/40-dwc2_hso10:55:152
2975699251,17cyclictest5997-21find09:00:132
2975699251,17cyclictest27346irq/40-dwc2_hso08:55:152
29746992515,6cyclictest21-21ksoftirqd/111:45:101
29746992515,6cyclictest21-21ksoftirqd/111:45:101
2974699251,18cyclictest29947-21apt-get07:10:121
2976499242,12cyclictest35966irq/58-eth010:25:133
2975699242,9cyclictest12061-21timerandwakeup07:55:292
2975699242,15cyclictest0-21swapper/209:30:052
2975699242,14cyclictest0-21swapper/208:00:122
2975699241,7cyclictest15056-21ls11:15:182
2975699241,18cyclictest29835-21ssh11:58:312
2974699242,16cyclictest32275-21ssh09:55:161
2974699242,15cyclictest16844-21users12:25:311
2974699241,18cyclictest3594-21ssh10:15:141
29746992411,3cyclictest21-21ksoftirqd/110:41:081
2973399242,13cyclictest12-21ksoftirqd/009:40:120
2973399242,13cyclictest12-21ksoftirqd/009:40:110
29764992314,6cyclictest35966irq/58-eth012:15:143
29764992310,7cyclictest35966irq/58-eth012:00:133
2975699239,5cyclictest29729-21cyclictest11:20:132
2975699232,6cyclictest0-21swapper/211:45:052
2975699232,6cyclictest0-21swapper/211:45:052
29756992312,9cyclictest5646-21ssh09:56:402
2975699230,9cyclictest27669-21kernelversion10:10:182
2974699232,17cyclictest11065-21ssh10:55:151
29746992311,3cyclictest21-21ksoftirqd/109:14:181
2973399236,10cyclictest26281-21ssh12:15:130
2973399234,14cyclictest19734-21kworker/u8:111:45:110
2973399234,14cyclictest19734-21kworker/u8:111:45:100
2973399232,15cyclictest8847-21ssh10:00:120
2973399231,9cyclictest21143-21diskstats12:30:160
2973399231,12cyclictest24012-21fschecks_count11:20:160
2976499223,14cyclictest35966irq/58-eth009:20:133
2976499223,13cyclictest0-21swapper/310:05:143
2976499223,12cyclictest0-21swapper/309:10:133
2975699223,13cyclictest0-21swapper/211:25:122
2975699222,15cyclictest0-21swapper/212:37:302
29756992212,8cyclictest0-21swapper/207:35:252
29756992212,8cyclictest0-21swapper/207:30:192
29756992211,9cyclictest25988-21ssh12:30:392
29756992211,7cyclictest3584-21ssh10:15:142
2975699221,17cyclictest15308-21kthreadcore12:25:212
2975699221,12cyclictest23114-21sed08:10:182
2974699229,3cyclictest21-21ksoftirqd/109:44:151
2974699229,3cyclictest21-21ksoftirqd/109:44:141
2974699228,3cyclictest0-21swapper/112:37:391
2974699225,6cyclictest21-21ksoftirqd/109:15:321
2974699223,14cyclictest13054-1kworker/1:1H11:20:121
2974699222,5cyclictest0-21swapper/112:03:291
2974699221,15cyclictest9251-21ls11:30:021
2973399227,10cyclictest12-21ksoftirqd/008:55:120
2973399221,17cyclictest5735-21ssh11:10:140
2973399221,16cyclictest2150-21switchtime10:30:280
29733992210,6cyclictest121ksoftirqd/010:50:130
2975699219,5cyclictest1906-21ssh10:50:082
2975699213,15cyclictest11480-21cut10:00:212
2975699211,8cyclictest26896-21ntp_states09:50:272
29756992116,3cyclictest18617-21ssh10:40:222
2975699211,17cyclictest26490-21ls07:40:142
2974699219,3cyclictest21-21ksoftirqd/110:32:581
2974699219,3cyclictest21-21ksoftirqd/109:46:011
2974699219,3cyclictest21-21ksoftirqd/108:00:241
2974699217,4cyclictest21-21ksoftirqd/110:36:581
29746992110,2cyclictest21-21ksoftirqd/109:35:191
2973399212,15cyclictest14757-21apt-get11:15:140
2973399211,5cyclictest0-21swapper/011:00:260
2973399211,5cyclictest0-21swapper/011:00:250
2976499202,14cyclictest0-21swapper/311:15:163
2976499201,12cyclictest0-21swapper/308:00:163
2975699207,5cyclictest21034-21kthreadcore08:40:202
2975699204,8cyclictest614-21dbus-daemon08:08:372
2975699203,9cyclictest18920-21latency_hist08:40:022
2975699202,9cyclictest31363-21fschecks_count07:45:162
2975699201,8cyclictest1653-21fschecks_time09:20:172
2975699201,15cyclictest3247-21wc08:20:232
2975699201,11cyclictest31401-21grep08:15:292
2975699201,11cyclictest10638-21df_inode09:25:152
2975699201,10cyclictest13731-21kthreadcore07:25:202
2975699201,10cyclictest13731-21kthreadcore07:25:192
2974699209,8cyclictest20620-21cat12:10:221
2974699208,3cyclictest21-21ksoftirqd/109:50:141
2974699207,3cyclictest131rcu_preempt12:21:101
2974699203,9cyclictest19689-21ssh09:30:141
2974699202,16cyclictest15376-21diskmemload11:05:121
2974699201,9cyclictest21295-21grep12:30:161
2974699201,15cyclictest26332-21latency_hist07:40:021
2973399201,4cyclictest0-21swapper/007:10:120
29733992012,5cyclictest15376-21diskmemload12:05:130
29733992011,6cyclictest2307-21usage_EthcTask12:35:310
2973399201,15cyclictest7334-21ssh10:35:120
2973399201,14cyclictest19201-21apt-get09:30:020
2976499198,9cyclictest0-21swapper/312:06:153
2976499198,6cyclictest0-21swapper/311:54:463
2976499194,9cyclictest35966irq/58-eth011:00:133
2976499194,9cyclictest35966irq/58-eth011:00:123
2976499192,12cyclictest0-21swapper/311:05:123
2976499192,10cyclictest35966irq/58-eth011:35:133
2976499191,13cyclictest0-21swapper/307:45:233
2974699199,7cyclictest28073-21kthreadcore10:45:221
2974699197,3cyclictest21-21ksoftirqd/107:20:221
2974699196,10cyclictest28579-21latency_hist08:50:011
2974699192,13cyclictest10649-21diskstats09:25:151
29746991913,4cyclictest0-21swapper/108:40:261
29746991910,7cyclictest17865-21apt-get10:05:131
2973399197,10cyclictest26169-21cron07:40:020
2973399196,10cyclictest19340-21fschecks_time08:40:150
2973399195,4cyclictest131rcu_preempt11:50:220
2973399195,11cyclictest12394-21gpgv10:20:020
2973399193,10cyclictest0-21swapper/010:05:140
2973399191,6cyclictest3073-21kthreadcore09:55:240
29733991913,4cyclictest0-21swapper/007:25:010
2973399191,14cyclictest14764-21ssh10:55:290
2973399191,13cyclictest23214-21ssh09:50:140
2973399191,11cyclictest1439-21ssh09:20:120
2973399191,11cyclictest13343-21/usr/sbin/munin11:30:280
2973399190,17cyclictest16224-21tr08:35:220
2976499189,6cyclictest0-21swapper/310:46:123
2976499187,8cyclictest0-21swapper/309:40:063
2976499187,8cyclictest0-21swapper/309:40:063
2976499182,9cyclictest35966irq/58-eth009:55:143
2976499182,11cyclictest0-21swapper/312:38:303
2976499182,10cyclictest0-21swapper/309:05:143
2976499182,10cyclictest0-21swapper/307:40:133
2976499181,14cyclictest35966irq/58-eth011:20:133
2976499181,14cyclictest0-21swapper/308:40:233
2976499181,11cyclictest0-21swapper/309:35:173
29764991810,6cyclictest0-21swapper/309:15:263
2975699182,8cyclictest29856-21latency_hist07:10:022
2974699187,9cyclictest0-21swapper/108:35:341
2974699185,6cyclictest131rcu_preempt08:25:251
29746991813,3cyclictest0-21swapper/107:25:161
29746991813,3cyclictest0-21swapper/107:25:151
2974699181,15cyclictest12798-21kthreadcore08:30:211
29746991811,4cyclictest16073-21kthreadcore10:20:231
2974699181,13cyclictest32147-21idleruntime-cro08:20:011
2974699181,10cyclictest7269-21kthreadcore09:00:191
29733991811,5cyclictest1-21systemd08:18:380
29733991811,4cyclictest945-21usage_EthcTask09:35:330
2973399181,14cyclictest2326-21apt-get07:15:110
29733991810,5cyclictest18891-21kthreadcore07:30:220
2973399180,12cyclictest9752-21timerwakeupswit09:00:290
2976499179,6cyclictest0-21swapper/310:36:093
2976499178,6cyclictest0-21swapper/310:13:003
2976499172,10cyclictest0-21swapper/311:45:053
2976499172,10cyclictest0-21swapper/311:45:043
2976499172,10cyclictest0-21swapper/308:45:133
2976499171,12cyclictest0-21swapper/308:25:143
2974699177,2cyclictest21-21ksoftirqd/107:45:311
2974699176,3cyclictest21-21ksoftirqd/108:50:111
2974699172,12cyclictest1351-21fschecks_count08:55:171
2974699171,13cyclictest22986-21fschecks_time08:10:151
2974699171,12cyclictest29778-21latency_hist07:10:021
2973399177,7cyclictest13205-21apt-get08:00:130
2973399176,8cyclictest12-21ksoftirqd/008:10:200
2973399171,5cyclictest7138-21sendmail_mailqu07:50:280
2973399171,5cyclictest7138-21sendmail_mailqu07:50:280
2973399171,5cyclictest11927-21find07:55:270
2976499162,9cyclictest0-21swapper/308:50:133
2976499162,9cyclictest0-21swapper/308:30:123
2976499162,9cyclictest0-21swapper/307:55:133
2976499162,9cyclictest0-21swapper/307:25:053
2976499162,9cyclictest0-21swapper/307:25:053
2976499162,11cyclictest0-21swapper/307:35:143
2974699161,13cyclictest26415-21apt-get07:40:131
29746991611,3cyclictest0-21swapper/107:55:151
2973399167,6cyclictest11720-21kthreadcore09:05:190
2973399162,4cyclictest131rcu_preempt08:20:230
2973399161,3cyclictest131rcu_preempt07:45:210
2973399161,12cyclictest31680-21tr08:50:210
2973399160,11cyclictest21687-21timerandwakeup08:05:280
2976499151,8cyclictest0-21swapper/307:30:153
2976499151,12cyclictest0-21swapper/309:00:163
2976499151,11cyclictest0-21swapper/308:55:183
2973399151,10cyclictest12811-21latency08:30:220
2976499141,10cyclictest0-21swapper/308:15:213
2976499131,9cyclictest0-21swapper/307:25:023
2973399131,10cyclictest29729-21cyclictest07:10:010
2976499122,6cyclictest0-21swapper/307:15:133
2976499122,5cyclictest0-21swapper/308:20:143
297649991,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional