You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-16 - 07:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sat May 16, 2026 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
15015997831,17cyclictest131rcu_preempt23:25:142
15030995925,23cyclictest35966irq/58-eth021:35:133
1500599591,46cyclictest0-21swapper/120:10:111
1499999507,26cyclictest0-21swapper/022:45:140
1499999501,41cyclictest24519-21ssh22:35:130
1500599474,34cyclictest19496-21ssh21:20:141
1500599470,43cyclictest15532-21apt-get22:30:141
14999994728,12cyclictest151rcuc/023:25:130
15005994615,26cyclictest21-21ksoftirqd/121:15:131
1503099444,22cyclictest35966irq/58-eth021:40:123
15030994419,14cyclictest35966irq/58-eth021:20:143
15005994426,12cyclictest0-21swapper/123:05:111
15005994426,12cyclictest0-21swapper/123:05:101
1500599441,33cyclictest0-21swapper/122:50:121
1500599440,39cyclictest28450-21apt-get21:25:131
1500599431,36cyclictest30065-21apt-get22:20:141
1500599421,37cyclictest0-21swapper/122:05:141
1500599421,36cyclictest0-21swapper/123:50:111
1500599421,25cyclictest0-21swapper/121:05:271
1500599411,37cyclictest11370-21apt-get23:40:111
1500599411,36cyclictest0-21swapper/100:05:141
1500599411,32cyclictest17591-21ssh21:55:141
1500599411,32cyclictest17591-21ssh21:55:131
1500599402,29cyclictest5010-21ssh21:30:121
1500599401,33cyclictest8552-21apt-get21:50:121
1500599401,24cyclictest17920-21wc20:50:281
15005993922,14cyclictest0-21swapper/123:35:121
1500599391,34cyclictest0-21swapper/119:30:121
1499999392,20cyclictest12-21ksoftirqd/020:10:120
1500599382,31cyclictest9122-21ssh00:15:131
1500599381,33cyclictest0-21swapper/121:00:131
1500599381,31cyclictest0-21swapper/122:25:121
1500599381,23cyclictest23700-21users20:55:291
1500599380,33cyclictest24509-21apt-get22:35:131
1499999382,31cyclictest18957-21ssh22:50:110
15030993725,9cyclictest35966irq/58-eth000:25:133
1500599373,30cyclictest0-21swapper/120:25:121
1500599372,24cyclictest2806-21ssh22:40:231
1500599372,23cyclictest22965-21ssh21:40:131
1500599371,32cyclictest9971-21apt-get22:45:141
1500599371,27cyclictest1015-21diskmemload22:14:021
1500599371,24cyclictest29802-21df_inode19:25:141
1500599370,23cyclictest4317-21cat21:45:301
15030993629,4cyclictest0-21swapper/300:20:123
1500599361,32cyclictest0-21swapper/119:10:121
1500599361,29cyclictest0-21swapper/121:10:131
1500599361,23cyclictest12359-21sh23:57:451
1500599360,24cyclictest1015-21diskmemload23:33:401
1499999364,9cyclictest12-21ksoftirqd/000:35:130
1499999364,19cyclictest12-21ksoftirqd/023:35:110
1500599351,30cyclictest20106-21diskstats19:15:151
1500599351,20cyclictest21822-21apt-get23:10:031
1500599351,19cyclictest17696-21ntp_states21:35:261
1500599350,24cyclictest4399-21grep23:00:141
1500599350,24cyclictest1015-21diskmemload00:01:141
1500599350,23cyclictest1691-21ssh00:28:131
1500599350,22cyclictest8459-21ssh00:30:421
1500599350,19cyclictest25815-21ssh00:24:031
1501599341,17cyclictest27081-21ssh00:25:142
1500599342,27cyclictest0-21swapper/100:35:131
1500599341,30cyclictest0-21swapper/122:55:111
1500599341,30cyclictest0-21swapper/122:55:111
1500599341,29cyclictest0-21swapper/100:10:111
1500599341,28cyclictest0-21swapper/120:15:131
1500599341,21cyclictest15102-21memory19:40:211
15030993326,5cyclictest35966irq/58-eth022:05:133
1501599334,20cyclictest27-21ksoftirqd/219:15:132
1500599332,28cyclictest16817-21apt-get23:25:121
1500599331,28cyclictest20238-21ssh23:45:121
1500599331,18cyclictest27002-21ssh22:16:481
1500599331,18cyclictest0-21swapper/123:20:021
1499999335,11cyclictest12-21ksoftirqd/022:25:130
1499999331,24cyclictest18110-21ssh00:20:120
1503099322,20cyclictest35966irq/58-eth000:05:133
1500599320,29cyclictest0-21swapper/120:05:141
1499999324,4cyclictest131rcu_preempt21:10:120
1503099312,17cyclictest35966irq/58-eth021:15:123
1503099312,16cyclictest35966irq/58-eth000:35:153
15030993118,10cyclictest35966irq/58-eth023:10:143
15015993111,16cyclictest27-21ksoftirqd/222:10:142
15005993118,9cyclictest0-21swapper/120:40:121
1500599311,27cyclictest31732-21irqstats20:00:181
1500599311,27cyclictest16362-21awk23:25:001
1500599311,27cyclictest0-21swapper/119:50:121
1500599310,28cyclictest27624-21df_inode20:30:151
1500599301,24cyclictest0-21swapper/119:35:091
1500599300,26cyclictest14098-21users20:45:291
1500599300,17cyclictest32665-21ssh22:02:021
14999993015,2cyclictest131rcu_preempt19:20:140
1503099293,15cyclictest35966irq/58-eth021:25:133
1501599293,18cyclictest30076-21ssh22:20:132
15005992919,6cyclictest0-21swapper/119:55:131
1500599290,26cyclictest19906-21memory19:45:231
1500599290,18cyclictest0-21swapper/119:20:131
14999992917,8cyclictest151rcuc/021:00:130
14999992914,9cyclictest1015-21diskmemload21:25:130
1503099282,20cyclictest35966irq/58-eth023:45:133
1501599283,16cyclictest27346irq/40-dwc2_hso21:50:112
1501599282,20cyclictest0-21swapper/223:50:142
1501599282,20cyclictest0-21swapper/222:30:142
1500599282,23cyclictest32333-21/usr/sbin/munin20:35:111
1503099277,14cyclictest0-21swapper/322:20:123
1501599272,15cyclictest20648-21irqcore23:45:172
1500599270,24cyclictest22509-21sshd20:23:441
1499999274,4cyclictest131rcu_preempt21:40:130
1499999272,21cyclictest21101-21ssh22:15:140
14999992720,4cyclictest12-21ksoftirqd/022:10:170
14999992712,12cyclictest12-21ksoftirqd/000:25:400
15030992613,8cyclictest35966irq/58-eth023:35:133
1501599264,7cyclictest31995-21ssh21:45:122
1501599262,9cyclictest18120-21ssh00:20:132
1501599262,20cyclictest3517-21ssh00:30:122
15015992617,4cyclictest0-21swapper/219:55:142
1500599260,24cyclictest14994-21cyclictest19:10:011
1499999267,4cyclictest131rcu_preempt21:05:120
1499999262,2cyclictest131rcu_preempt20:15:160
1499999262,20cyclictest5677-21ssh23:55:150
14999992613,3cyclictest12-21ksoftirqd/022:20:230
14999992612,5cyclictest131rcu_preempt21:50:130
1499999261,11cyclictest22267-21ssh23:10:120
14999992610,2cyclictest131rcu_preempt00:15:300
1503099252,10cyclictest35966irq/58-eth020:10:123
15030992515,7cyclictest35966irq/58-eth023:00:113
1501599252,16cyclictest0-21swapper/222:05:122
1503099245,9cyclictest0-21swapper/322:10:113
1503099245,13cyclictest0-21swapper/322:35:133
1501599242,15cyclictest27346irq/40-dwc2_hso23:40:122
1501599242,11cyclictest22290-21rm23:10:132
15015992413,9cyclictest9295-21ssh21:14:092
1501599241,18cyclictest12824-21latency_hist20:15:032
1501599241,14cyclictest27346irq/40-dwc2_hso00:15:142
1501599241,10cyclictest14381-21ps21:35:172
1501599240,9cyclictest19512-21ssh21:20:142
1499999244,15cyclictest12-21ksoftirqd/021:45:120
1499999243,18cyclictest20247-21ssh23:45:120
1503099234,14cyclictest0-21swapper/323:25:133
1503099234,14cyclictest0-21swapper/300:30:133
1503099233,17cyclictest35966irq/58-eth023:15:113
1503099232,10cyclictest35966irq/58-eth000:15:123
15030992312,8cyclictest0-21swapper/323:50:133
15030992310,9cyclictest35966irq/58-eth023:40:123
1501599235,8cyclictest9788-21unixbench_multi23:55:302
1501599232,6cyclictest0-21swapper/200:38:542
1501599232,16cyclictest0-21swapper/222:35:132
1501599231,6cyclictest2070-21latency_hist23:35:012
15015992312,9cyclictest6248-21ssh22:40:562
15015992312,9cyclictest10380-21ssh23:01:542
1501599231,10cyclictest1453566ssh22:45:322
1499999232,18cyclictest4366-21ssh23:00:120
1499999232,18cyclictest12-21ksoftirqd/020:45:120
1499999231,18cyclictest930-21snmpd19:25:130
1499999231,18cyclictest2102-21apt-get19:30:130
1499999231,16cyclictest14649-21ssh00:00:120
1501599229,9cyclictest0-21swapper/221:25:142
1501599222,16cyclictest32142-21ls21:05:212
1501599221,9cyclictest0-21swapper/223:39:122
1501599221,12cyclictest31276-21fschecks_time20:00:152
1501599221,11cyclictest23731-21kthreadcore22:15:222
1499999225,4cyclictest12-21ksoftirqd/021:55:260
1499999225,4cyclictest12-21ksoftirqd/021:55:250
14999992214,6cyclictest27833-21kthreadcore23:30:220
14999992212,8cyclictest12-21ksoftirqd/022:40:130
1499999221,18cyclictest14000-21ssh21:35:140
1499999221,16cyclictest6996-21apt-get19:35:090
1499999220,19cyclictest26714-21cut22:00:130
1503099214,11cyclictest35966irq/58-eth023:05:123
1503099214,11cyclictest35966irq/58-eth023:05:113
1503099213,12cyclictest0-21swapper/321:10:143
15030992110,9cyclictest0-21swapper/322:50:583
1501599216,5cyclictest17536-21/usr/sbin/munin21:55:322
1501599216,5cyclictest17536-21/usr/sbin/munin21:55:312
1501599213,2cyclictest131rcu_preempt00:00:232
1501599212,10cyclictest8688-21ntp_states21:30:242
1501599211,4cyclictest0-21swapper/220:55:162
1501599211,18cyclictest19109-21fschecks_count22:50:152
15015992111,8cyclictest0-21swapper/221:43:212
1501599211,17cyclictest20097-21ps19:45:272
1501599211,16cyclictest16358-21munin-run23:25:002
1501599211,11cyclictest9968-21kthreadcore22:25:222
1501599211,11cyclictest8257-21grep20:40:272
1501599211,11cyclictest21838-21idleruntime-cro23:10:022
1501599211,11cyclictest21838-21idleruntime-cro23:10:012
1501599211,11cyclictest1971-21latency_hist19:30:022
15015992110,9cyclictest30095-21ssh00:07:352
15015992110,5cyclictest3847-21ssh22:59:532
15015992110,5cyclictest3847-21ssh22:59:522
1499999215,3cyclictest131rcu_preempt22:34:410
1499999214,14cyclictest1015-21diskmemload00:10:120
1499999212,14cyclictest32344-21apt-get20:35:110
14999992115,4cyclictest0-21swapper/023:20:120
14999992115,3cyclictest12-21ksoftirqd/000:05:150
14999992114,5cyclictest0-21swapper/021:15:050
14999992112,3cyclictest0-21swapper/021:32:370
1499999211,16cyclictest151rcuc/023:05:130
1499999211,16cyclictest151rcuc/023:05:120
14999992110,8cyclictest0-21swapper/023:50:250
1499999210,2cyclictest131rcu_preempt20:20:170
1503099209,9cyclictest0-21swapper/321:56:203
1503099209,9cyclictest0-21swapper/321:56:193
1503099202,11cyclictest0-21swapper/319:35:043
15030992012,6cyclictest0-21swapper/322:40:153
1501599205,12cyclictest2094-21sort19:30:122
1501599205,12cyclictest0-21swapper/220:45:142
1501599202,3cyclictest131rcu_preempt23:15:182
1501599201,9cyclictest6785-21wc20:05:222
15015992013,4cyclictest0-21swapper/222:00:132
15015992011,7cyclictest10457-21acpi21:15:132
1501599201,15cyclictest25163-21grep19:20:182
1501599201,10cyclictest27555-21wc21:00:262
1501599201,10cyclictest27324-21latency_hist20:30:012
15015992010,8cyclictest21400-1kworker/u9:000:10:292
1501599200,17cyclictest10313-21ls19:35:212
1499999207,3cyclictest131rcu_preempt21:25:010
1499999203,14cyclictest630-21Async09:36:310
14999992013,3cyclictest12-21ksoftirqd/000:30:280
14999992010,4cyclictest12-21ksoftirqd/022:59:310
14999992010,4cyclictest12-21ksoftirqd/022:59:300
14999992010,2cyclictest12-21ksoftirqd/019:45:010
1503099192,11cyclictest0-21swapper/322:55:143
1503099192,11cyclictest0-21swapper/322:55:133
1503099192,11cyclictest0-21swapper/321:05:133
1503099192,11cyclictest0-21swapper/321:00:043
1503099191,8cyclictest0-21swapper/322:25:143
1503099191,15cyclictest35966irq/58-eth022:34:443
1503099191,13cyclictest0-21swapper/322:45:173
1503099191,13cyclictest0-21swapper/319:45:153
1503099191,12cyclictest0-21swapper/319:55:123
1503099191,12cyclictest0-21swapper/319:10:153
15030991910,6cyclictest35966irq/58-eth020:05:143
15030991910,6cyclictest0-21swapper/323:33:353
15030991910,6cyclictest0-21swapper/322:02:333
1501599197,5cyclictest30080-21tr20:30:222
1501599196,11cyclictest18960-21kthreadcore20:20:202
1501599192,4cyclictest0-21swapper/220:50:132
1501599191,9cyclictest16961-21kthreadcore19:10:202
1501599191,9cyclictest13471-21irqcore20:15:182
1501599191,9cyclictest11994-21fschecks_time19:40:152
1501599191,8cyclictest22480-21kthreadcore19:50:202
1501599191,15cyclictest835-21kthreadcore20:35:172
1499999199,5cyclictest12-21ksoftirqd/023:15:110
1499999198,3cyclictest12-21ksoftirqd/019:50:010
1499999198,2cyclictest12-21ksoftirqd/022:05:070
1499999192,2cyclictest131rcu_preempt23:40:370
1499999192,2cyclictest131rcu_preempt20:05:270
14999991914,3cyclictest12-21ksoftirqd/019:50:190
14999991912,4cyclictest12-21ksoftirqd/019:10:290
1503099189,6cyclictest35966irq/58-eth021:30:123
1503099187,8cyclictest0-21swapper/323:22:133
1503099183,11cyclictest0-21swapper/323:55:153
1503099182,10cyclictest0-21swapper/319:30:143
1503099182,10cyclictest0-21swapper/319:25:043
1503099181,13cyclictest0-21swapper/321:50:223
15030991810,6cyclictest0-21swapper/321:45:253
1501599181,8cyclictest15042-21apt-key19:10:022
1499999189,7cyclictest12-21ksoftirqd/020:40:170
1499999188,3cyclictest12-21ksoftirqd/020:30:130
1499999185,4cyclictest131rcu_preempt20:50:160
14999991812,4cyclictest0-21swapper/020:00:160
1503099173,11cyclictest0-21swapper/322:15:153
1503099172,9cyclictest0-21swapper/320:50:143
1503099172,8cyclictest0-21swapper/320:35:123
1503099172,10cyclictest0-21swapper/300:00:133
15030991710,5cyclictest0-21swapper/300:10:213
1499999173,11cyclictest23152-21kthreadcore19:15:220
1499999172,12cyclictest23745-21kthreadcore20:25:190
14999991710,4cyclictest12-21ksoftirqd/019:55:200
1503099168,6cyclictest0-21swapper/320:15:383
14999991610,4cyclictest0-21swapper/019:10:010
1503099158,3cyclictest0-21swapper/320:45:263
1503099149,3cyclictest0-21swapper/319:40:243
1503099149,3cyclictest0-21swapper/319:15:143
1503099142,8cyclictest0-21swapper/320:25:173
1503099142,8cyclictest0-21swapper/320:00:053
1503099141,8cyclictest35966irq/58-eth019:20:153
1503099141,7cyclictest0-21swapper/320:20:213
1503099141,6cyclictest0-21swapper/320:45:013
1503099141,10cyclictest0-21swapper/319:10:013
1503099133,8cyclictest0-21swapper/320:55:273
1503099131,10cyclictest0-21swapper/319:50:293
1503099122,6cyclictest0-21swapper/320:30:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional