You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-13 - 03:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sat Jun 13, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
306052174102,29sleep30-21swapper/319:09:343
304702169101,27sleep20-21swapper/219:07:492
305972167106,45sleep00-21swapper/019:09:260
303382162103,28sleep10-21swapper/119:06:071
1679321030,1sleep00-21swapper/023:33:470
234472910,1sleep30-21swapper/321:37:163
162202650,2chrt0-21swapper/100:21:231
79412620,1sleep00-21swapper/021:25:210
184552570,5sleep317970-21diskmemload23:10:183
140342570,1sleep30-21swapper/322:19:103
116752560,2sleep20-21swapper/223:54:242
165172550,1sleep10-21swapper/122:20:201
108082530,2sleep20-21swapper/221:28:002
315292520,4sleep231527-21ssh23:45:012
213682520,2sleep00-21swapper/022:48:360
9802510,2sleep10-21swapper/123:21:161
301892490,3sleep334-21ksoftirqd/322:55:133
232602490,2sleep00-21swapper/022:50:130
232602490,2sleep00-21swapper/022:50:130
170412480,1sleep10-21swapper/119:50:201
110812480,5sleep012-21ksoftirqd/021:52:220
30682460,2sleep10-21swapper/122:10:211
131922450,1sleep10-21swapper/123:06:071
258752440,2chrt0-21swapper/123:15:261
79842420,1sleep00-21swapper/023:26:370
317452400,2sleep131710-21ssh21:20:001
3091199400,7cyclictest12-21ksoftirqd/021:16:510
3091499391,2cyclictest0-21swapper/300:04:563
3091199387,8cyclictest18393-21sh21:10:010
3091199387,28cyclictest10704-21sh22:15:470
30911993830,5cyclictest24119-21/usr/sbin/munin23:15:100
3091199377,9cyclictest9390-21sendmail-msp22:40:000
30911993728,6cyclictest7074-21sh22:13:510
30914993622,12cyclictest9015-21ssh23:03:383
30914993622,12cyclictest0-21swapper/322:21:313
30914993621,12cyclictest22537-21ssh00:25:493
3091499361,4cyclictest16281-21ssh21:31:463
3091499361,33cyclictest0-21swapper/321:29:533
3091499361,2cyclictest0-21swapper/322:44:373
3091499361,23cyclictest0-21swapper/323:47:333
3091199368,10cyclictest7615-21sed23:50:240
30914993519,14cyclictest26107-21ssh22:04:033
30914993519,14cyclictest26107-21ssh22:04:033
3091499351,32cyclictest0-21swapper/322:13:413
3091299351,31cyclictest15999-21ssh22:44:561
3091199356,10cyclictest8895-21ssh23:03:300
30914993412,20cyclictest13813-21expr00:20:133
3091499340,7cyclictest13453-21ssh23:55:173
3091399341,30cyclictest21138-21ssh22:48:222
3091299341,30cyclictest10700-21ssh21:51:581
3091199348,11cyclictest2418-21ssh00:10:540
3091199347,9cyclictest22619-21ssh23:37:550
3091199347,9cyclictest22619-21ssh23:37:550
3091499331,28cyclictest12882-21awk23:30:203
3091499331,23cyclictest0-21swapper/321:50:513
3091499330,6cyclictest8654-21ssh00:15:293
3091499330,23cyclictest27549-21ssh22:29:093
3091499330,23cyclictest0-21swapper/323:51:353
3091299331,29cyclictest9237-21sh23:51:541
3091299331,29cyclictest32126-21ssh22:32:011
3091299331,29cyclictest21875-21sh23:37:111
3091299331,29cyclictest21875-21sh23:37:101
3091199338,15cyclictest29535-21sh00:07:190
3091199336,10cyclictest27854-21ssh23:41:270
3091199335,9cyclictest23933-21ssh22:25:280
3091199335,9cyclictest17970-21diskmemload00:19:150
3091199331,29cyclictest25096-21ssh00:04:350
282822330,2chrt0-21swapper/122:53:511
282822330,2chrt0-21swapper/122:53:511
3091499321,29cyclictest0-21swapper/321:24:103
3091499321,28cyclictest560-21dbus-daemon23:40:003
3091499321,28cyclictest560-21dbus-daemon23:40:003
3091499321,1cyclictest0-21swapper/323:26:453
3091499321,1cyclictest0-21swapper/322:48:183
3091499321,1cyclictest0-21swapper/321:18:433
3091499320,29cyclictest0-21swapper/322:06:073
3091399325,25cyclictest17970-21diskmemload00:26:322
30912993227,3cyclictest13299-21sh22:18:251
3091199327,9cyclictest1782-21ssh22:58:050
3091199327,10cyclictest25400-21ssh22:03:200
3091199327,10cyclictest25400-21ssh22:03:200
3091199326,9cyclictest28638-21sed00:30:240
3091199325,9cyclictest0-21swapper/021:22:440
3091199325,11cyclictest14660-21ssh23:55:400
3091199322,13cyclictest0-21swapper/022:00:000
3091499317,6cyclictest4789-21date19:25:003
30914993122,7cyclictest0-21swapper/321:48:483
30914993122,7cyclictest0-21swapper/300:37:263
30914993112,16cyclictest17467-21sh21:57:043
3091499310,7cyclictest32345-21chrt00:34:023
3091499310,23cyclictest0-21swapper/322:32:593
3091399311,27cyclictest0-21swapper/221:10:202
3091299311,29cyclictest30219-21ssh00:31:511
3091299311,22cyclictest17970-21diskmemload00:35:411
3091299310,7cyclictest8886-21sh21:26:031
3091299310,28cyclictest24477-21ssh22:26:011
3091199316,9cyclictest15942-21ssh21:31:260
3091199316,11cyclictest8623-21sed20:45:230
3091199315,11cyclictest17970-21diskmemload00:20:140
3091199314,18cyclictest12561-21taskset22:41:290
3091199314,10cyclictest18306-21grep19:55:130
3091199314,10cyclictest17400-21sh22:20:360
30914993022,7cyclictest0-21swapper/323:24:493
30914993012,15cyclictest30014-21sh00:07:473
30914993010,17cyclictest2999-21sh00:11:313
3091499300,6cyclictest24355-21ssh22:50:223
3091499300,6cyclictest24355-21ssh22:50:223
3091399301,25cyclictest17970-21diskmemload23:36:142
3091399301,25cyclictest17970-21diskmemload23:36:142
3091399300,6cyclictest19767-21ssh00:25:002
30912993023,5cyclictest28853-21ssh21:40:581
30912993022,6cyclictest27263-21ssh23:40:521
3091299301,27cyclictest17970-21diskmemload21:23:461
3091299301,27cyclictest13922-21ssh23:30:531
3091299301,27cyclictest0-21swapper/123:56:271
3091299301,27cyclictest0-21swapper/121:35:411
3091199307,15cyclictest636-21runrttasks20:17:150
3091199306,9cyclictest2666-21rm23:47:000
3091199306,9cyclictest17970-21diskmemload21:45:170
3091199305,9cyclictest22253-21ssh00:25:320
3091199305,9cyclictest1678-21missed_timers20:30:190
3091199305,9cyclictest1678-21missed_timers20:30:190
3091199305,8cyclictest29403-21sh21:41:310
3091199305,8cyclictest16092-21/usr/sbin/munin19:50:180
3091199305,10cyclictest32221-21sed19:10:220
3091199304,9cyclictest1073-21sh23:21:220
30911993024,4cyclictest0-21swapper/021:35:320
3091199300,20cyclictest0-21swapper/023:10:100
3091499296,5cyclictest16782-21cat21:05:183
3091499294,5cyclictest1371-21cat19:15:173
3091499292,3cyclictest0-21swapper/320:40:113
30914992918,9cyclictest0-21swapper/322:36:393
3091499291,26cyclictest22385-21ssh21:12:303
3091499291,26cyclictest17970-21diskmemload23:17:253
30914992912,14cyclictest27777-21ps21:40:223
30914992911,15cyclictest27967-21sh23:41:343
3091399291,4cyclictest640-21snmpd23:14:492
30912992923,5cyclictest0-21swapper/122:03:471
30912992923,5cyclictest0-21swapper/122:03:471
30912992923,4cyclictest0-21swapper/100:18:381
30912992921,6cyclictest17838-21ssh21:57:261
30912992917,9cyclictest28252-21ssh00:06:001
3091299291,3cyclictest25601-21ntp_states20:10:201
3091299291,26cyclictest0-21swapper/123:11:461
3091299290,26cyclictest6711-21ssh22:37:211
3091199294,10cyclictest16834-21taskset21:05:190
3091199293,9cyclictest29562-21sh22:05:340
3091199293,9cyclictest28782-21apt-get22:30:080
3091199293,9cyclictest11545-21/usr/sbin/munin20:55:150
3091199293,24cyclictest0-21swapper/023:09:230
3091199291,3cyclictest4549-21ssh00:36:570
3091499286,5cyclictest24851-21cat20:10:123
30914992823,3cyclictest15463-21ssh23:08:253
3091499281,24cyclictest8145-21netstat20:45:183
3091499281,24cyclictest23530-21grep20:05:223
3091499280,6cyclictest9796-21df19:35:143
30913992824,2cyclictest0-21swapper/222:29:402
30913992822,4cyclictest11720-21ssh22:16:492
3091399281,25cyclictest25458-21ssh21:39:182
3091399280,25cyclictest2167-21sensors21:45:222
30912992823,3cyclictest23591-21ssh00:26:531
3091299282,2cyclictest0-21swapper/123:01:511
3091299281,25cyclictest0-21swapper/122:57:591
3091299281,25cyclictest0-21swapper/121:32:181
3091199284,9cyclictest29787-21cat20:20:160
3091499278,16cyclictest0-21swapper/320:25:133
3091499278,16cyclictest0-21swapper/320:20:223
3091499277,5cyclictest16241-21cut19:50:123
3091499271,9cyclictest391-21munin-run19:15:013
3091499271,8cyclictest14906-21ntpq19:45:193
3091499271,8cyclictest11646-21/usr/sbin/munin20:55:133
3091499271,5cyclictest27444-21cat20:15:163
30914992711,13cyclictest2748-21idleruntime-cro20:35:003
30914992711,13cyclictest2748-21idleruntime-cro20:35:003
30913992722,3cyclictest17903-21ssh23:34:542
3091399271,3cyclictest0-21swapper/223:02:332
3091399271,2cyclictest0-21swapper/223:24:562
3091399271,25cyclictest0-21swapper/222:44:432
3091399271,24cyclictest0-21swapper/223:55:332
3091399271,24cyclictest0-21swapper/221:18:132
3091399271,24cyclictest0-21swapper/200:38:442
3091399271,1cyclictest0-21swapper/200:00:072
30912992723,3cyclictest19641-21ssh21:10:201
30912992722,3cyclictest30261-21sh22:06:161
30912992722,3cyclictest15951-21apt-config21:05:001
3091299271,25cyclictest0-21swapper/123:29:451
3091299271,24cyclictest0-21swapper/123:45:021
3091299271,24cyclictest0-21swapper/121:45:091
3091299271,24cyclictest0-21swapper/120:45:001
3091299271,24cyclictest0-21swapper/120:35:231
3091299270,3cyclictest3033-21ssh00:11:331
3091299270,25cyclictest18712-21ssh22:45:551
3091199274,9cyclictest4880-21latency_hist20:40:000
3091199274,9cyclictest0-21swapper/019:45:020
3091499267,5cyclictest9990-21smartctl20:50:153
3091499267,3cyclictest14249-21gltestperf21:00:153
3091499261,8cyclictest19384-21systemctl19:55:233
30913992622,2cyclictest17970-21diskmemload22:20:152
30913992622,2cyclictest14064-21ssh23:07:002
30913992622,2cyclictest0-21swapper/221:50:132
3091399261,2cyclictest0-21swapper/222:10:532
3091399261,24cyclictest0-21swapper/222:38:112
3091399261,24cyclictest0-21swapper/222:01:192
3091399261,24cyclictest0-21swapper/222:01:192
3091399261,24cyclictest0-21swapper/221:57:072
3091399261,24cyclictest0-21swapper/221:41:112
3091399261,23cyclictest0-21swapper/223:46:552
3091399261,23cyclictest0-21swapper/222:50:092
3091399261,23cyclictest0-21swapper/222:50:092
3091399261,23cyclictest0-21swapper/222:30:372
3091399261,23cyclictest0-21swapper/220:10:202
3091399261,23cyclictest0-21swapper/200:05:032
3091399260,24cyclictest3099-21ssh00:11:362
30912992624,1cyclictest0-21swapper/100:00:401
30912992623,2cyclictest0-21swapper/120:05:131
3091299261,24cyclictest0-21swapper/120:50:181
30911992624,1cyclictest0-21swapper/020:50:130
3091499252,4cyclictest0-21swapper/319:30:153
3091499251,6cyclictest6888-21idleruntime-cro19:30:003
3091499251,6cyclictest13555-21latency_hist19:45:003
3091499251,6cyclictest13555-21latency_hist19:45:003
3091499251,5cyclictest21755-21cat20:00:243
3091499251,4cyclictest3644-21irqstats20:35:173
30913992522,2cyclictest0-21swapper/223:25:512
30913992522,2cyclictest0-21swapper/221:31:262
3091399251,23cyclictest0-21swapper/222:55:052
3091399251,22cyclictest0-21swapper/223:15:022
3091399251,1cyclictest0-21swapper/222:07:502
3091399251,1cyclictest0-21swapper/200:18:022
3091399250,23cyclictest0-21swapper/221:20:312
3091399250,23cyclictest0-21swapper/200:31:142
3091199254,6cyclictest13731-21apt-get21:00:110
30913992422,1cyclictest0-21swapper/220:35:232
30913992422,1cyclictest0-21swapper/219:45:232
30913992419,3cyclictest19081-21/usr/sbin/munin19:55:192
3091199244,7cyclictest10118-21irqstats19:35:170
30911992422,1cyclictest0-21swapper/020:10:250
30913992321,1cyclictest0-21swapper/220:20:002
30912992320,2cyclictest0-21swapper/120:20:111
3091199232,8cyclictest0-21swapper/020:25:160
3091199230,8cyclictest0-21swapper/019:20:220
3091399222,14cyclictest9473-21chrt19:35:122
30913992217,3cyclictest7044-21munin-run20:45:002
30912992213,7cyclictest9932-21fw_packets19:35:151
30912992213,7cyclictest13466-21cron21:00:011
3091199221,8cyclictest12-21ksoftirqd/019:29:280
127122220,1sleep00-21swapper/019:40:200
127122220,1sleep00-21swapper/019:40:200
3091399212,13cyclictest16080-21apt-get21:05:122
30913992112,7cyclictest2357-21users20:30:252
30913992112,7cyclictest2357-21users20:30:252
30912992119,1cyclictest0-21swapper/119:45:191
30912992113,6cyclictest27283-21perl20:15:141
3091199213,4cyclictest0-21swapper/019:15:150
30911992114,3cyclictest20413-21apt-get20:00:110
30911992112,7cyclictest5318-21cat20:40:130
3091199210,8cyclictest0-21swapper/019:30:130
209912210,2sleep20-21swapper/220:00:162
3091399202,13cyclictest640-21snmpd19:25:122
30913992018,1cyclictest0-21swapper/220:45:132
30913992012,6cyclictest32487-21sed20:25:222
30913992011,6cyclictest3430-21idleruntime19:20:162
3091299200,4cyclictest0-21swapper/120:41:311
30911992013,5cyclictest22525-21/usr/sbin/munin20:05:150
30913991914,3cyclictest23310-21missed_timers20:05:192
30913991911,6cyclictest18203-21date19:55:012
3091399190,8cyclictest11322-21idleruntime-cro20:55:002
30912991913,4cyclictest6866-21sh19:30:001
30912991913,3cyclictest12282-21kernelversion19:40:161
30912991913,3cyclictest12282-21kernelversion19:40:161
30912991912,5cyclictest32686-21seq20:25:541
30912991911,7cyclictest32550-21sshd19:11:241
30912991911,6cyclictest21304-21ntpq20:00:201
30912991910,7cyclictest8214-21which19:30:221
3091299190,6cyclictest642-21/usr/sbin/munin19:15:101
3091399189,6cyclictest11817-21cat19:40:112
3091399189,6cyclictest11817-21cat19:40:112
3091399188,7cyclictest278-21systemd-journal20:25:002
3091399181,12cyclictest7119-21apt-get19:30:122
3091299182,11cyclictest2842-21apt-get19:20:101
3091299181,15cyclictest2751-21ls20:35:001
3091299181,15cyclictest2751-21ls20:35:001
30912991811,5cyclictest14304-21cat21:00:161
30912991810,6cyclictest19003-21ls19:55:181
3091399171,13cyclictest820-21tr19:15:122
30913991710,5cyclictest14382-21seq21:00:172
30913991611,3cyclictest384-21latency_hist19:15:012
3091399161,11cyclictest11572-21apt-get20:55:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional