You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-29 - 09:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Wed Jul 29, 2026 00:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32523997513,7cyclictest131rcu_preempt20:20:120
3252399705,48cyclictest19168-21ssh22:40:130
3252399705,48cyclictest19168-21ssh22:40:120
32544996546,12cyclictest35966irq/58-eth023:40:123
3252399623,48cyclictest20594-21ssh23:35:130
32544995732,10cyclictest35966irq/58-eth022:45:153
3252499513,10cyclictest131rcu_preempt22:20:131
3254499492,27cyclictest35966irq/58-eth023:45:153
3252499485,35cyclictest15057-21ssh23:50:151
3252499476,3cyclictest131rcu_preempt00:30:111
3252399462,32cyclictest2928-21ssh21:55:140
3252399462,32cyclictest2928-21ssh21:55:130
32544994524,12cyclictest35966irq/58-eth022:20:143
3252499453,26cyclictest4116-21ssh00:20:141
3253599422,25cyclictest27-21ksoftirqd/223:35:142
3252499418,18cyclictest21-21ksoftirqd/119:10:151
3252499393,31cyclictest18988-21ssh21:10:151
3254499383,19cyclictest35966irq/58-eth022:55:143
3254499368,18cyclictest0-21swapper/321:10:143
3254499361,19cyclictest35966irq/58-eth021:20:143
3252499363,19cyclictest9628-21ssh00:05:151
3252399369,19cyclictest12-21ksoftirqd/023:25:130
32523993621,9cyclictest22675-21rm23:00:140
3252399361,28cyclictest27526-21ssh00:15:130
3253599345,20cyclictest27-21ksoftirqd/221:55:142
3253599345,20cyclictest27-21ksoftirqd/221:55:132
32535993412,7cyclictest27-21ksoftirqd/223:15:142
32524993318,7cyclictest0-21swapper/123:45:161
32524993313,10cyclictest32522-21cyclictest20:30:141
3254499325,20cyclictest35966irq/58-eth021:35:133
3253599322,25cyclictest5476-21ssh00:37:482
32524993218,4cyclictest131rcu_preempt21:40:131
3252499311,19cyclictest30056-21ssh22:10:141
3252399311,22cyclictest28064-21ssh21:15:170
3253599303,18cyclictest27346irq/40-dwc2_hso19:30:132
3252499308,14cyclictest0-21swapper/122:50:121
3252499304,21cyclictest20588-21ssh23:35:141
3252399301,4cyclictest131rcu_preempt00:05:170
3253599295,17cyclictest27346irq/40-dwc2_hso00:10:112
3253599292,17cyclictest27346irq/40-dwc2_hso19:50:142
3252499292,18cyclictest8195-21ssh23:10:141
3252499291,25cyclictest6658-21ssh22:15:151
3254499283,18cyclictest0-21swapper/300:30:503
32544992816,4cyclictest0-21swapper/322:10:133
3253599282,20cyclictest495-21ssh00:00:042
3253599281,7cyclictest27-21ksoftirqd/223:10:132
3252399283,18cyclictest10130-21ssh22:35:120
3254499273,17cyclictest0-21swapper/300:23:493
3253599274,11cyclictest27346irq/40-dwc2_hso20:00:112
3253599272,19cyclictest0-21swapper/220:20:132
3253599271,8cyclictest2627-21ssh23:25:132
3252399272,20cyclictest12-21ksoftirqd/021:00:110
3252399272,17cyclictest22103-21ssh00:30:140
3252399272,16cyclictest6639-21ssh22:15:150
3252399271,18cyclictest17192-21ssh21:45:050
3253599264,17cyclictest27346irq/40-dwc2_hso22:25:122
3253599263,17cyclictest27346irq/40-dwc2_hso23:00:142
3253599262,19cyclictest0-21swapper/220:15:152
3253599262,16cyclictest27346irq/40-dwc2_hso22:35:112
32535992613,6cyclictest667-21ssh21:15:472
3252499262,17cyclictest17347-21ssh21:45:141
32524992614,6cyclictest630-21Async09:36:311
3252399267,3cyclictest131rcu_preempt20:15:140
3254499252,14cyclictest35966irq/58-eth021:45:143
32544992512,9cyclictest35966irq/58-eth022:05:133
3254499251,16cyclictest35966irq/58-eth000:15:143
3253599252,16cyclictest0-21swapper/221:10:052
3252499252,20cyclictest13703-21ssh22:55:141
3252499251,19cyclictest13554-21sh21:25:131
3252399252,17cyclictest18480-21diskmemload23:30:150
3252399251,3cyclictest131rcu_preempt20:45:340
32523992511,5cyclictest0-21swapper/022:55:300
3252399251,11cyclictest131rcu_preempt23:15:140
3254499249,10cyclictest0-21swapper/300:29:493
3253599249,4cyclictest27-21ksoftirqd/221:00:122
3253599242,7cyclictest0-21swapper/200:18:492
3253599242,6cyclictest0-21swapper/200:26:502
32535992412,9cyclictest24842-21cut21:30:222
32535992412,9cyclictest24842-21cut21:30:212
32535992410,6cyclictest10970-21cat22:15:332
3253599240,13cyclictest4612-21cut20:55:192
3252499242,18cyclictest24608-21ssh22:25:121
3254499233,13cyclictest0-21swapper/321:25:133
3254499231,15cyclictest0-21swapper/321:55:143
3254499231,15cyclictest0-21swapper/321:55:143
3253599235,15cyclictest30308-21rm22:10:162
32535992312,9cyclictest4100-21ssh21:35:342
32535992312,9cyclictest0-21swapper/200:09:342
3253599231,13cyclictest4627-21ssh21:20:132
3253599231,12cyclictest12169-21irqrtprio23:30:192
3252499232,18cyclictest9932-21find19:20:111
3252499232,17cyclictest2947-21ssh21:55:151
3252499232,17cyclictest2947-21ssh21:55:141
3252499231,18cyclictest26138-21ssh23:20:161
3252399233,4cyclictest131rcu_preempt23:55:130
3252399233,4cyclictest131rcu_preempt23:55:120
32523992314,4cyclictest0-21swapper/022:45:280
3252399231,18cyclictest4752-21ssh22:50:110
3254499223,11cyclictest35966irq/58-eth000:00:143
3254499222,13cyclictest0-21swapper/322:50:143
3254499222,12cyclictest36066irq/59-eth023:10:143
32544992213,7cyclictest0-21swapper/323:25:173
3253599229,11cyclictest24779-21ssh22:05:282
3253599227,5cyclictest0-21swapper/200:30:212
3253599224,8cyclictest21403-21irqrtprio20:05:182
3253599224,8cyclictest21403-21irqrtprio20:05:182
3253599223,13cyclictest27346irq/40-dwc2_hso19:40:132
3253599222,5cyclictest0-21swapper/200:20:482
32535992212,8cyclictest0-21swapper/223:45:012
32535992212,8cyclictest0-21swapper/221:54:562
32535992211,9cyclictest2445-21ssh22:47:442
32535992211,9cyclictest24154-21ssh22:40:412
32535992211,9cyclictest24154-21ssh22:40:412
3253599221,16cyclictest17125-21ls21:45:032
32535992211,3cyclictest27-21ksoftirqd/222:55:252
3253599221,12cyclictest12771-21grep20:30:172
3253599221,11cyclictest6605-21cut22:50:202
3253599221,11cyclictest5794-21uname19:15:212
3253599221,11cyclictest11942-21/usr/sbin/munin22:00:172
3253599221,11cyclictest11942-21/usr/sbin/munin22:00:162
3252499228,8cyclictest0-21swapper/122:45:141
3252499223,14cyclictest21036-21ssh22:05:131
3252499221,9cyclictest30101-21kthreadcore21:15:231
3252499221,3cyclictest0-21swapper/122:35:271
32524992211,8cyclictest17206-21awk23:15:151
32524992210,9cyclictest30287-21cron20:15:001
3252399223,12cyclictest0-21swapper/019:30:140
3252399222,5cyclictest0-21swapper/000:35:480
3252399222,4cyclictest131rcu_preempt19:45:140
3254499216,10cyclictest35966irq/58-eth023:20:163
3254499214,11cyclictest35966irq/58-eth022:15:143
3254499212,14cyclictest35966irq/58-eth023:15:153
3254499211,3cyclictest0-21swapper/321:30:153
3254499211,3cyclictest0-21swapper/321:30:143
3253599218,10cyclictest18485-21grep21:10:012
3253599212,16cyclictest1148-21/usr/sbin/munin22:30:222
32535992111,9cyclictest27-21ksoftirqd/223:50:152
32535992111,8cyclictest0-21swapper/223:20:552
3253599211,12cyclictest19062-21wc22:20:222
3253599211,10cyclictest30681-21threads20:45:282
3253599211,10cyclictest26249-21kthreadcore23:55:222
3253599211,10cyclictest26249-21kthreadcore23:55:222
3253599211,10cyclictest131rcu_preempt21:45:112
32535992110,5cyclictest19158-21ssh21:26:252
32535992110,4cyclictest25707-21open_inodes20:40:272
3252499214,9cyclictest19203-21kthreadcore20:00:201
3252499213,13cyclictest0-21swapper/120:05:141
3252499213,13cyclictest0-21swapper/120:05:131
3252499212,14cyclictest26324-21ssh21:50:151
3252499211,17cyclictest8106-21tail20:55:311
3252499211,13cyclictest7783-21latency_hist21:40:021
3252499211,13cyclictest5252-21uniq19:15:171
3252499211,13cyclictest18705-21cut00:10:151
3252399219,3cyclictest4485-21ssh22:12:550
3252399212,8cyclictest0-21swapper/021:39:050
3252399212,14cyclictest655-21sh00:00:140
3252399211,9cyclictest131rcu_preempt21:44:270
3252399211,6cyclictest12058-21ssh21:23:410
3252399211,3cyclictest131rcu_preempt00:10:170
32523992112,7cyclictest7877-21date23:10:020
32523992111,7cyclictest0-21swapper/019:55:020
3252399211,16cyclictest22650-21ssh21:30:130
3252399211,16cyclictest22650-21ssh21:30:120
3252399211,15cyclictest18707-21ssh23:50:290
3252399210,18cyclictest8197-21diskmemload23:10:140
3254499209,8cyclictest0-21swapper/322:33:173
3253599206,6cyclictest0-21swapper/219:35:182
3253599202,9cyclictest390-21fschecks_count19:10:192
3253599202,10cyclictest8433-21kthreadcore20:25:202
3253599201,15cyclictest14543-21latency_hist19:25:022
3253599201,12cyclictest29524-21ssh23:40:132
3253599201,11cyclictest27190-21kthreadcore20:10:202
3253599201,10cyclictest32292-21cut23:05:212
3253599201,10cyclictest32246-21irqcore20:50:202
3253599201,10cyclictest17665-21grep20:35:182
3253599201,10cyclictest15432-21irqrtprio19:25:192
3253599201,10cyclictest13390-21kthreadcore19:55:202
3252499207,3cyclictest131rcu_preempt23:40:171
3252499204,9cyclictest32522-21cyclictest23:55:141
3252499204,9cyclictest32522-21cyclictest23:55:131
3252499201,9cyclictest6129-21memory23:25:231
32524992014,4cyclictest32522-21cyclictest19:35:131
32524992013,5cyclictest0-21swapper/120:40:181
32524992011,7cyclictest31006-21ssh21:34:461
32524992011,7cyclictest31006-21ssh21:34:451
3252499201,14cyclictest11993-21sh22:00:151
3252499201,14cyclictest11993-21sh22:00:141
32524992010,3cyclictest131rcu_preempt00:40:021
3252399201,8cyclictest0-21swapper/021:27:130
3252399201,7cyclictest26886-21grep23:20:200
3252399201,5cyclictest15784-21awk22:00:290
3252399201,5cyclictest15784-21awk22:00:280
3252399201,2cyclictest131rcu_preempt19:10:000
32523992012,6cyclictest28922-21cut21:50:230
3252399201,14cyclictest18480-21diskmemload00:25:140
3252399200,3cyclictest131rcu_preempt22:05:130
3254499199,6cyclictest35966irq/58-eth021:40:133
3254499196,10cyclictest0-21swapper/300:10:273
3254499193,11cyclictest35966irq/58-eth021:15:163
3254499192,13cyclictest35966irq/58-eth023:00:153
3254499192,11cyclictest0-21swapper/320:05:143
3254499192,11cyclictest0-21swapper/320:05:133
32544991914,2cyclictest0-21swapper/319:50:133
3254499191,12cyclictest0-21swapper/321:05:013
32544991910,6cyclictest0-21swapper/300:05:293
32544991910,4cyclictest0-21swapper/300:35:263
3253599192,8cyclictest2348-21kthreadcore19:45:202
3252499199,7cyclictest11576-21apt-get23:30:131
3252499193,13cyclictest18005-21ssh00:25:341
3252499192,13cyclictest0-21swapper/119:55:141
3252499191,7cyclictest5833-21cut21:20:201
32524991912,5cyclictest0-21swapper/121:00:161
32524991911,6cyclictest12308-21sshd20:27:471
3252499191,11cyclictest0-21swapper/122:30:251
3252399193,4cyclictest12-21ksoftirqd/023:42:390
3252399191,5cyclictest0-21swapper/023:47:210
3252399191,4cyclictest131rcu_preempt22:26:130
3252399191,3cyclictest0-21swapper/000:23:060
3252399190,3cyclictest131rcu_preempt22:22:120
3252399190,16cyclictest18480-21diskmemload22:30:120
3254499187,6cyclictest0-21swapper/323:50:063
3254499184,11cyclictest0-21swapper/322:25:103
3254499182,12cyclictest35966irq/58-eth022:35:123
3254499182,10cyclictest0-21swapper/319:25:053
3254499182,10cyclictest0-21swapper/319:10:173
32544991810,6cyclictest0-21swapper/322:01:373
32544991810,6cyclictest0-21swapper/322:01:373
3253599181,14cyclictest32610-21latency_hist19:10:012
3252499189,7cyclictest3971-21cron20:55:011
3252499189,7cyclictest21-21ksoftirqd/120:20:161
3252499183,12cyclictest11267-21latency_hist19:55:011
3252499182,12cyclictest27663-21df_inode00:15:171
32524991811,5cyclictest30807-21ssh23:04:281
32524991811,4cyclictest2653-21kthreadcore00:00:231
3252499181,14cyclictest19647-21irqcore22:40:181
3252499181,14cyclictest19647-21irqcore22:40:171
3252399189,6cyclictest13841-21cut19:55:210
3252399188,8cyclictest13242-21tr20:30:180
3252399188,7cyclictest0-21swapper/019:25:020
3252399181,4cyclictest131rcu_preempt19:10:220
3252399180,12cyclictest18096-21ls19:25:230
3254499179,6cyclictest0-21swapper/323:39:073
3254499179,6cyclictest0-21swapper/323:33:243
3254499179,6cyclictest0-21swapper/322:40:073
3254499179,6cyclictest0-21swapper/322:40:063
3254499178,6cyclictest0-21swapper/323:55:233
3254499178,6cyclictest0-21swapper/323:55:223
3254499171,12cyclictest0-21swapper/320:35:233
3254499171,12cyclictest0-21swapper/320:00:133
3254499171,11cyclictest0-21swapper/320:25:243
3254499171,10cyclictest0-21swapper/320:10:283
3254499171,10cyclictest0-21swapper/319:40:143
3254499171,10cyclictest0-21swapper/319:20:233
3252499178,6cyclictest18468-21cron21:10:001
3252499175,9cyclictest29798-21kthreadcore20:45:231
32524991711,4cyclictest0-21swapper/119:30:251
3252499170,14cyclictest15004-21fschecks_count19:25:161
3252399171,9cyclictest0-21swapper/020:00:110
3252399171,5cyclictest8434-21memory19:15:250
3252399171,4cyclictest131rcu_preempt20:10:250
3252399171,3cyclictest131rcu_preempt20:40:310
32523991711,4cyclictest329-21sendmail_mailst19:40:270
3252399171,13cyclictest5385-21kthreadcore20:55:220
3252399171,13cyclictest19099-21diskstats21:10:160
3252399170,2cyclictest131rcu_preempt20:25:130
3254499168,6cyclictest0-21swapper/323:07:083
3254499168,6cyclictest0-21swapper/321:50:383
3254499162,9cyclictest0-21swapper/319:35:133
3254499161,12cyclictest0-21swapper/319:30:193
3254499161,10cyclictest0-21swapper/320:30:123
3252499164,9cyclictest30677-21fschecks_time20:15:181
3252499162,12cyclictest20374-21kthreadcore20:35:231
3252499162,11cyclictest4146-21kthreadcore19:45:241
32524991610,4cyclictest29590-21sed19:40:171
3252399168,6cyclictest21318-21cat20:35:300
3252399168,6cyclictest17311-21ntpq21:05:250
3252399161,4cyclictest0-21swapper/020:06:380
3252399161,4cyclictest0-21swapper/020:06:370
3252399161,13cyclictest31808-21apt-get20:50:150
3254499151,9cyclictest0-21swapper/321:05:143
3252399152,2cyclictest131rcu_preempt19:35:130
3254499149,3cyclictest0-21swapper/320:20:233
3254499142,8cyclictest0-21swapper/319:45:163
3252499141,10cyclictest32560-21latency_hist19:10:011
3254499134,6cyclictest35966irq/58-eth020:45:143
3254499132,7cyclictest35966irq/58-eth020:55:143
3254499131,9cyclictest0-21swapper/319:15:193
3254499131,10cyclictest0-21swapper/320:15:233
3254499122,8cyclictest0-21swapper/320:40:123
3254499122,7cyclictest0-21swapper/320:50:153
3254499122,6cyclictest0-21swapper/319:55:043
3254499111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional