You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-13 - 06:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Wed May 13, 2026 00:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
549399524,26cyclictest27-21ksoftirqd/221:15:142
547199513,37cyclictest5856-21ssh00:35:140
547199511,42cyclictest1329-21ssh22:45:140
5505994725,12cyclictest35966irq/58-eth023:40:123
548199453,29cyclictest20883-21ssh22:20:131
548199401,6cyclictest21-21ksoftirqd/121:05:111
549399388,22cyclictest27-21ksoftirqd/222:35:132
548199382,9cyclictest21-21ksoftirqd/123:30:131
547199370,6cyclictest131rcu_preempt19:25:130
549399361,17cyclictest20162-21ssh22:19:382
5481993611,4cyclictest0-21swapper/119:40:101
547199362,7cyclictest12-21ksoftirqd/023:55:120
548199352,8cyclictest131rcu_preempt23:20:131
550599341,23cyclictest35966irq/58-eth022:35:133
550599341,17cyclictest35966irq/58-eth022:45:133
549399342,16cyclictest17767-21ssh22:53:392
548199344,26cyclictest31566-21ssh21:50:131
547199340,27cyclictest26245-21ssh22:05:120
549399330,18cyclictest29145-21ssh00:30:152
547199338,4cyclictest131rcu_preempt23:35:140
547199334,22cyclictest12-21ksoftirqd/020:00:130
550599321,15cyclictest35966irq/58-eth019:15:133
550599321,15cyclictest35966irq/58-eth019:15:133
550599311,15cyclictest35966irq/58-eth019:20:133
548199313,25cyclictest29139-21ssh00:30:131
548199312,26cyclictest24781-21ssh22:40:131
5481993112,16cyclictest14036-21ssh23:10:141
547199311,23cyclictest23890-21ssh22:02:380
5471993110,16cyclictest15777-21diskmemload22:35:140
5493993018,5cyclictest27-21ksoftirqd/223:45:282
5481993017,4cyclictest21-21ksoftirqd/123:35:291
547199302,20cyclictest6590-21ssh22:30:130
547199302,20cyclictest6590-21ssh22:30:120
5471993012,12cyclictest12-21ksoftirqd/023:10:160
549399293,20cyclictest0-21swapper/222:12:412
549399292,21cyclictest0-21swapper/219:25:152
549399292,20cyclictest0-21swapper/223:10:152
549399292,11cyclictest27571-21ssh22:22:422
548199292,22cyclictest23677-21diskmemload23:50:131
5481992921,5cyclictest0-21swapper/121:45:141
5481992920,7cyclictest0-21swapper/100:20:131
5481992920,7cyclictest0-21swapper/100:20:121
548199290,23cyclictest0-21swapper/100:00:121
550599284,22cyclictest35966irq/58-eth023:10:153
549399282,20cyclictest0-21swapper/222:45:142
549399282,11cyclictest28247-21ssh23:52:402
548199283,17cyclictest24187-21ssh21:10:151
548199282,23cyclictest3579-21sh23:40:131
548199282,16cyclictest8840-21ssh23:25:141
547199282,20cyclictest15688-21ssh00:04:400
547199281,5cyclictest131rcu_preempt00:20:120
547199281,5cyclictest131rcu_preempt00:20:120
5505992719,5cyclictest35966irq/58-eth023:50:133
550599271,14cyclictest35966irq/58-eth023:55:143
549399273,16cyclictest261rcuc/219:15:132
549399273,16cyclictest261rcuc/219:15:132
549399272,9cyclictest31201-21ssh22:05:392
549399272,10cyclictest0-21swapper/221:55:132
549399271,16cyclictest13121-21munin-run21:40:002
5493992710,11cyclictest27-21ksoftirqd/222:40:242
547199272,19cyclictest9488-21ssh22:12:400
547199270,9cyclictest131rcu_preempt19:30:130
547199270,20cyclictest503-21ssh00:13:390
5505992611,6cyclictest35966irq/58-eth022:05:123
5493992615,9cyclictest27-21ksoftirqd/223:20:142
5493992615,5cyclictest27-21ksoftirqd/220:55:222
5493992613,10cyclictest0-21swapper/220:05:162
549399261,14cyclictest28369-21ssh23:00:152
5493992611,12cyclictest0-21swapper/221:00:272
549399260,18cyclictest12791-21kthreadcore21:20:192
548199263,7cyclictest21-21ksoftirqd/122:45:141
548199263,19cyclictest10417-21ssh22:50:151
548199262,7cyclictest21-21ksoftirqd/100:35:121
547199261,8cyclictest1966966ssh23:48:130
547199261,3cyclictest131rcu_preempt20:40:160
547199261,18cyclictest24797-21sh22:40:150
547199260,17cyclictest9752-21rm21:20:120
549399259,7cyclictest16133-21sed00:05:022
549399257,2cyclictest28722-21grep19:30:292
549399256,15cyclictest131rcu_preempt20:40:132
549399255,13cyclictest131rcu_preempt23:40:202
549399254,14cyclictest9839-21ssh23:05:312
549399252,9cyclictest0-21swapper/220:00:122
549399252,15cyclictest27346irq/40-dwc2_hso19:50:142
5493992512,10cyclictest0-21swapper/223:35:122
549399251,16cyclictest20408-21grep00:25:172
549399251,10cyclictest13302-21kthreadcore00:20:212
549399251,10cyclictest13302-21kthreadcore00:20:202
548199257,3cyclictest131rcu_preempt23:45:291
548199254,3cyclictest131rcu_preempt21:25:011
548199253,4cyclictest131rcu_preempt19:45:121
548199252,19cyclictest23938-21ssh00:08:411
5481992520,3cyclictest8722-21ssh00:17:401
5481992513,9cyclictest16310-21cron19:55:001
548199251,15cyclictest6580-21ssh22:30:121
548199251,15cyclictest6580-21ssh22:30:111
5505992414,6cyclictest35966irq/58-eth022:00:133
549399245,12cyclictest30128-21fschecks_time22:25:152
549399245,11cyclictest22439-21latency_hist20:35:012
5493992417,3cyclictest17840-21ssh23:30:142
5493992413,8cyclictest0-21swapper/219:20:262
5493992413,7cyclictest27346irq/40-dwc2_hso23:15:042
549399241,20cyclictest2084-21ssh00:15:152
549399241,20cyclictest0-21swapper/221:25:152
548199246,4cyclictest131rcu_preempt22:10:211
548199244,13cyclictest27897-21ssh21:30:131
548199243,17cyclictest23120-21rm23:15:151
5481992413,4cyclictest21-21ksoftirqd/123:05:271
548199240,21cyclictest13515-21ssh21:40:131
548199240,21cyclictest13515-21ssh21:40:121
547199246,3cyclictest131rcu_preempt23:40:120
550599231,15cyclictest0-21swapper/322:12:403
549399239,11cyclictest0-21swapper/219:10:212
549399235,4cyclictest131rcu_preempt21:10:212
549399233,16cyclictest0-21swapper/222:32:442
549399233,16cyclictest0-21swapper/222:32:432
549399231,8cyclictest27-21ksoftirqd/219:45:262
549399231,14cyclictest0-21swapper/221:40:122
549399231,14cyclictest0-21swapper/221:40:112
5493992311,10cyclictest0-21swapper/223:58:562
5493992310,11cyclictest14900-21ssh23:26:582
548199239,11cyclictest0-21swapper/123:01:401
548199235,4cyclictest21-21ksoftirqd/122:05:171
548199231,2cyclictest131rcu_preempt21:38:521
5481992312,3cyclictest131rcu_preempt20:25:271
548199231,18cyclictest16728-21cut19:55:151
5481992310,5cyclictest21-21ksoftirqd/100:25:311
547199233,16cyclictest29127-21apt-get00:30:130
547199231,9cyclictest131rcu_preempt21:15:200
5471992316,3cyclictest13488-21ssh21:40:120
5471992316,3cyclictest13488-21ssh21:40:110
547199231,11cyclictest131rcu_preempt21:45:240
547199230,7cyclictest131rcu_preempt00:09:400
547199230,18cyclictest4181-21ssh22:27:400
5505992211,8cyclictest0-21swapper/322:40:233
5505992211,7cyclictest35966irq/58-eth021:45:153
5505992210,6cyclictest0-21swapper/322:55:323
549399226,9cyclictest131rcu_preempt20:50:032
549399226,9cyclictest131rcu_preempt20:50:022
549399225,4cyclictest131rcu_preempt00:10:182
549399222,3cyclictest5841-21ssh00:35:122
549399222,16cyclictest0-21swapper/222:59:392
5493992216,3cyclictest0-21swapper/220:45:122
549399221,4cyclictest2669-21cut19:40:152
5493992212,7cyclictest22561-21ssh21:45:142
549399221,18cyclictest575-21needreboot19:35:232
549399221,15cyclictest131rcu_preempt21:30:312
5493992210,5cyclictest7883-21sshd20:15:402
548199228,4cyclictest21-21ksoftirqd/119:10:291
548199226,3cyclictest131rcu_preempt22:25:111
548199224,3cyclictest131rcu_preempt20:55:211
548199223,4cyclictest131rcu_preempt20:15:121
548199222,12cyclictest844-21munin-node21:15:141
5481992215,5cyclictest0-21swapper/122:15:141
547199228,7cyclictest5455-21cyclictest21:35:130
547199221,9cyclictest23688-21/usr/sbin/munin21:25:330
547199221,9cyclictest0-21swapper/019:40:260
547199221,7cyclictest25083-21/usr/sbin/munin23:50:210
547199221,3cyclictest131rcu_preempt20:10:310
547199221,3cyclictest131rcu_preempt00:25:170
547199221,2cyclictest131rcu_preempt23:26:160
5471992211,8cyclictest23660-21cron21:10:010
547199221,11cyclictest18893-21cat19:20:280
5471992210,3cyclictest12-21ksoftirqd/021:30:300
550599219,9cyclictest0-21swapper/323:00:533
550599215,14cyclictest33-21ksoftirqd/323:07:473
550599215,14cyclictest33-21ksoftirqd/321:22:223
550599213,12cyclictest0-21swapper/322:22:413
5505992113,6cyclictest0-21swapper/322:50:183
5505992112,6cyclictest35966irq/58-eth000:00:133
5505992111,6cyclictest35966irq/58-eth023:20:143
549399219,3cyclictest131rcu_preempt21:50:132
549399214,5cyclictest13187-21kworker/2:222:00:322
549399212,13cyclictest27346irq/40-dwc2_hso20:20:122
549399211,9cyclictest18075-21cut00:05:202
549399211,17cyclictest14575-21kthreadcore20:25:222
549399211,10cyclictest24517-21cut20:35:192
548199217,5cyclictest21-21ksoftirqd/120:05:011
548199215,4cyclictest21-21ksoftirqd/120:05:271
548199214,3cyclictest131rcu_preempt00:15:021
548199214,13cyclictest18811-21ssh21:25:131
548199211,16cyclictest27605-21ls20:40:171
547199213,2cyclictest131rcu_preempt20:05:200
547199213,2cyclictest131rcu_preempt19:50:050
547199212,3cyclictest131rcu_preempt22:50:170
547199211,8cyclictest131rcu_preempt23:15:230
547199211,2cyclictest131rcu_preempt23:08:000
5471992111,8cyclictest0-21swapper/020:20:140
547199211,17cyclictest29117-21ps23:00:220
547199211,16cyclictest32168-21find23:20:130
547199210,2cyclictest131rcu_preempt23:30:270
550599205,13cyclictest33-21ksoftirqd/323:16:273
550599205,13cyclictest33-21ksoftirqd/322:25:103
550599204,14cyclictest33-21ksoftirqd/322:00:003
550599204,14cyclictest33-21ksoftirqd/321:52:473
5505992014,4cyclictest0-21swapper/323:35:143
5505992013,4cyclictest35966irq/58-eth023:25:143
5505992011,6cyclictest0-21swapper/322:30:123
5505992011,6cyclictest0-21swapper/322:30:113
549399201,5cyclictest21917-21kthreadcore21:05:212
549399201,16cyclictest5543-21latency_hist19:10:022
549399201,10cyclictest16718-21fschecks_time19:55:162
548199209,8cyclictest4706-21apt-get20:50:131
548199209,8cyclictest4706-21apt-get20:50:121
548199207,4cyclictest21-21ksoftirqd/119:30:171
548199206,6cyclictest21-21ksoftirqd/121:55:281
548199205,5cyclictest0-21swapper/122:02:321
548199204,3cyclictest131rcu_preempt22:55:201
548199203,8cyclictest131rcu_preempt19:20:261
548199203,5cyclictest21-21ksoftirqd/120:35:181
548199202,5cyclictest21-21ksoftirqd/123:55:131
5481992014,4cyclictest0-21swapper/119:35:141
5481992013,4cyclictest0-21swapper/122:35:031
547199205,9cyclictest11050-21irqstats19:15:190
547199205,9cyclictest11050-21irqstats19:15:180
547199204,11cyclictest20012-21cut22:55:190
547199202,3cyclictest131rcu_preempt22:15:210
547199202,3cyclictest131rcu_preempt20:25:200
547199202,2cyclictest131rcu_preempt21:05:020
5471992013,5cyclictest24203-21ssh21:10:170
5471992011,6cyclictest0-21swapper/020:30:160
550599199,8cyclictest0-21swapper/321:31:053
550599193,11cyclictest0-21swapper/322:15:393
550599193,11cyclictest0-21swapper/300:24:413
550599193,11cyclictest0-21swapper/300:24:403
5505991912,5cyclictest321rcuc/320:40:163
5505991911,5cyclictest0-21swapper/323:45:293
550599191,13cyclictest0-21swapper/321:25:133
5505991910,6cyclictest35966irq/58-eth000:15:123
5505991910,6cyclictest0-21swapper/321:10:393
549399192,14cyclictest30962-21apt-get20:10:142
548199197,4cyclictest131rcu_preempt21:00:231
548199194,9cyclictest0-21swapper/120:30:111
548199192,5cyclictest21-21ksoftirqd/120:10:181
5481991912,5cyclictest201rcuc/119:15:131
5481991912,5cyclictest201rcuc/119:15:121
547199192,8cyclictest0-21swapper/000:16:120
547199192,2cyclictest131rcu_preempt19:10:190
547199191,9cyclictest16876-21irqcore19:55:170
547199191,8cyclictest22572-21kthreadcore22:20:200
547199191,8cyclictest131rcu_preempt19:45:170
547199191,5cyclictest0-21swapper/021:50:200
547199191,2cyclictest131rcu_preempt21:55:580
547199191,10cyclictest0-21swapper/020:35:100
550599189,6cyclictest35966irq/58-eth000:30:143
550599189,5cyclictest35966irq/58-eth000:05:223
5505991810,5cyclictest35966irq/58-eth023:30:133
5505991810,5cyclictest35966irq/58-eth000:10:163
548199181,2cyclictest131rcu_preempt20:45:241
5481991810,5cyclictest23934-21unixbench_singl19:25:291
547199182,4cyclictest4146-21kthreadcore20:15:190
547199181,3cyclictest131rcu_preempt20:45:210
547199181,2cyclictest131rcu_preempt20:55:150
5471991811,5cyclictest9255-21sshd20:50:400
5471991811,5cyclictest9255-21sshd20:50:390
550599179,6cyclictest0-21swapper/321:42:033
550599179,6cyclictest0-21swapper/321:42:033
550599179,6cyclictest0-21swapper/321:35:423
550599179,6cyclictest0-21swapper/300:36:053
550599171,10cyclictest0-21swapper/320:05:123
550599171,10cyclictest0-21swapper/320:00:133
5505991710,4cyclictest35966irq/58-eth021:16:123
548199171,13cyclictest8654-21grep20:20:181
547199172,2cyclictest131rcu_preempt19:35:120
550599169,5cyclictest35966irq/58-eth000:25:143
550599161,9cyclictest0-21swapper/319:35:273
550599161,9cyclictest0-21swapper/319:25:143
550599161,10cyclictest0-21swapper/320:50:053
550599161,10cyclictest0-21swapper/320:50:043
550599161,10cyclictest0-21swapper/319:10:143
5505991610,4cyclictest0-21swapper/319:55:163
547199161,4cyclictest131rcu_preempt19:10:010
550599151,9cyclictest0-21swapper/320:45:043
550599149,3cyclictest0-21swapper/320:20:183
550599148,3cyclictest321rcuc/319:30:133
550599145,6cyclictest0-21swapper/320:10:203
550599141,9cyclictest0-21swapper/320:15:133
550599141,9cyclictest0-21swapper/319:10:013
548199141,9cyclictest5544-21latency_hist19:10:011
550599138,3cyclictest0-21swapper/319:45:143
550599134,4cyclictest0-21swapper/321:05:123
550599132,9cyclictest0-21swapper/321:00:153
550599132,9cyclictest0-21swapper/319:50:053
550599132,9cyclictest0-21swapper/319:40:123
550599132,7cyclictest0-21swapper/320:30:123
550599131,7cyclictest0-21swapper/320:25:163
550599122,5cyclictest0-21swapper/320:35:113
550599111,7cyclictest0-21swapper/320:55:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional