You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-17 - 04:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Wed Jun 17, 2026 00:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
261462183106,28sleep10-21swapper/119:08:571
260432181108,29sleep00-21swapper/019:07:340
259222176102,29sleep30-21swapper/319:06:023
243222174120,47sleep20-21swapper/219:05:052
922021670,1sleep20-21swapper/223:55:202
228342660,3sleep20-21swapper/223:41:082
254212630,2sleep20-21swapper/221:18:262
84042600,3sleep3341ksoftirqd/323:55:133
203692600,1sleep30-21swapper/321:39:053
139102560,1sleep30-21swapper/322:45:593
324552550,9sleep122-21ksoftirqd/123:48:531
86992520,2sleep10-21swapper/120:55:251
325442520,3sleep10-21swapper/121:23:541
117462520,3sleep30-21swapper/321:32:073
16462510,2sleep10-21swapper/121:49:231
100592510,2sleep30-21swapper/323:07:543
276442480,2sleep30-21swapper/322:08:353
103412470,2chrt0-21swapper/221:30:422
33502460,2sleep00-21swapper/023:50:330
76552450,1sleep20-21swapper/220:55:142
212352450,2sleep10-21swapper/122:27:391
301252440,1sleep30-21swapper/322:58:513
26500994314,18cyclictest25400-21sh21:42:220
212202400,1sleep10-21swapper/123:15:391
19352400,1sleep10-21swapper/100:38:541
2650099386,19cyclictest13559-21diskmemload00:18:020
259562380,1sleep30-21swapper/323:44:183
26500993511,12cyclictest13559-21diskmemload21:20:390
26500993422,10cyclictest0-21swapper/000:14:240
2650099340,3cyclictest31357-21grep19:20:140
121982340,2sleep10-21swapper/119:50:171
2650099331,1cyclictest0-21swapper/023:27:500
2650099330,6cyclictest3863-21grep20:45:220
2650099320,5cyclictest30001-21gltestperf22:10:130
2650099320,4cyclictest25423-21ssh21:18:260
2650099320,3cyclictest8477-21cpuspeed_turbos21:30:140
26500993110,19cyclictest19584-21seq23:39:380
26500993110,19cyclictest19584-21seq23:39:370
2650099310,3cyclictest11502-21ssh00:21:020
2650099310,3cyclictest0-21swapper/000:27:290
2650099308,19cyclictest29816-21pmu-power20:30:220
2650099308,19cyclictest20740-21expr23:40:130
2650099302,23cyclictest0-21swapper/000:30:120
26500993014,14cyclictest21281-21sh23:15:430
26500993014,13cyclictest23514-21sh22:29:570
26500993013,14cyclictest21814-21sh22:52:130
2650099301,2cyclictest0-21swapper/022:00:110
26500993010,17cyclictest278-21systemd-journal20:10:000
2650099300,5cyclictest31711-21ssh00:36:130
2650099300,5cyclictest24906-21df20:20:140
2650099300,4cyclictest8408-21cat23:55:130
2650099300,4cyclictest5577-21ssh22:40:170
2650099300,3cyclictest4299-21ssh21:26:220
2650099300,3cyclictest26013-21ssh23:20:010
2650099300,3cyclictest15187-21chrt19:55:540
2650099300,3cyclictest12907-21cstates23:10:130
2650099300,3cyclictest10009-21head21:00:140
2650099300,2cyclictest20449-21ssh21:39:100
2650299290,26cyclictest2905-21ssh00:15:152
2650199291,26cyclictest4469-21ssh00:15:401
2650099298,19cyclictest19180-21seq21:13:540
2650099294,22cyclictest0-21swapper/023:30:100
2650099292,23cyclictest30689-21apt-get22:35:140
2650099292,21cyclictest25009-21apt-get22:55:130
2650099291,2cyclictest0-21swapper/000:00:100
2650099291,25cyclictest6656-21sort23:05:190
2650099290,6cyclictest15067-21rm22:47:090
2650099290,4cyclictest25526-21ssh00:07:410
2650099290,3cyclictest26648-21tr19:10:140
2650099290,3cyclictest13559-21diskmemload22:07:210
2650099290,2cyclictest4419-21fschecks_count22:15:170
2650099290,2cyclictest2070-21ssh23:01:290
2650399281,24cyclictest0-21swapper/300:15:373
2650199289,7cyclictest688-21cut19:25:141
2650099285,20cyclictest27415-21cut23:45:130
2650099281,9cyclictest9354-21/usr/sbin/munin19:45:120
2650099281,9cyclictest9354-21/usr/sbin/munin19:45:120
2650099281,6cyclictest21585-21awk20:10:280
2650099280,3cyclictest15053-21ssh22:23:090
2650099280,3cyclictest13438-21sh21:57:450
78162270,2sleep10-21swapper/122:17:361
26503992713,11cyclictest3031-21idleruntime-cro00:40:003
2650199272,7cyclictest26187-21apt-get23:20:121
2650199271,2cyclictest13559-21diskmemload23:53:571
2650199271,24cyclictest0-21swapper/123:06:591
2650099279,15cyclictest7293-21cpuspeed_turbos19:40:130
2650099278,16cyclictest2664-21munin-node21:50:110
2650099276,18cyclictest12129-21gltestperf21:05:160
2650099275,19cyclictest2044-21cat20:40:260
2650099271,10cyclictest30190-21/usr/sbin/munin19:15:260
2650099270,24cyclictest28276-21ssh21:45:040
193622270,2sleep30-21swapper/321:14:053
2650299268,12cyclictest28-21ksoftirqd/200:31:402
26501992624,1cyclictest0-21swapper/123:30:241
2650099262,2cyclictest0-21swapper/019:35:120
2650099261,9cyclictest3263-21/usr/sbin/munin19:30:150
2650099261,8cyclictest16396-21/usr/sbin/munin20:00:170
2650099261,10cyclictest11475-21/usr/sbin/munin19:50:100
199612260,2chrt0-21swapper/300:03:493
2650399251,23cyclictest0-21swapper/323:53:543
2650299251,23cyclictest0-21swapper/223:27:452
2650299251,23cyclictest0-21swapper/200:14:082
26501992512,10cyclictest8254-21cpuspeed_turbos23:55:131
2650199251,11cyclictest145402ssh23:10:391
2650099251,9cyclictest7691-21df20:55:140
2650099251,9cyclictest2751-21sed19:30:000
2650099251,7cyclictest26422-21ssh22:31:080
26503992413,8cyclictest8939-21cron21:55:003
26502992422,1cyclictest0-21swapper/223:53:572
2650299241,1cyclictest0-21swapper/223:06:582
2650099241,7cyclictest6374-21sed20:50:260
2650099241,7cyclictest22411-21/usr/sbin/munin20:15:100
2650099241,5cyclictest26862-21grep20:25:130
2650099241,4cyclictest31583-21ls20:35:170
202152240,10sleep228-21ksoftirqd/221:14:562
153812240,1sleep10-21swapper/121:35:141
26503992321,1cyclictest0-21swapper/300:14:263
2650399230,1cyclictest0-21swapper/323:30:363
2650399230,1cyclictest0-21swapper/323:27:503
2650299232,16cyclictest26056-21ssh23:20:042
2650299230,1cyclictest9019-21ssh23:30:362
2650399223,12cyclictest13559-21diskmemload00:25:133
2650399222,14cyclictest27390-21ssh21:20:123
2650399222,14cyclictest17334-21runrttasks22:25:133
26502992214,6cyclictest24387-21diskstats22:30:152
26502992213,7cyclictest9700-21cat19:45:162
26502992213,7cyclictest9700-21cat19:45:162
26502992213,6cyclictest4231-21ls22:15:162
2650199229,8cyclictest52950irq/141-enp3s0-23:00:141
26501992220,1cyclictest0-21swapper/100:14:031
2650199221,8cyclictest16739-21sed22:00:171
26501992213,7cyclictest1-21systemd22:55:001
237342220,2sleep30-21swapper/300:05:533
2650399218,3cyclictest19980-21latency_hist23:40:003
2650399218,3cyclictest19980-21latency_hist23:40:003
2650399212,14cyclictest27148-21ssh23:45:033
2650399211,4cyclictest0-21swapper/300:31:403
26503992114,4cyclictest25965-21grep20:20:253
26503992112,6cyclictest18495-21sh22:01:053
26503992112,6cyclictest17664-21cron20:05:003
2650399210,12cyclictest307562ssh23:23:183
26502992113,6cyclictest8126-21cat21:30:012
26502992113,6cyclictest27803-21sort21:20:152
26502992113,6cyclictest23223-21ls21:40:232
26502992112,7cyclictest29839-21date21:45:252
26502992112,7cyclictest2870-21cat20:45:102
26502992112,6cyclictest9868-21forks21:00:132
2650299211,14cyclictest29186-21ssh00:35:112
26502992110,3cyclictest141rcu_preempt23:48:132
2650199219,8cyclictest13979-21apt-get21:10:111
2650199219,10cyclictest31948-21ssh22:11:121
2650199212,17cyclictest28021-21sh21:45:001
26501992113,6cyclictest23327-21seq23:41:381
26501992113,6cyclictest11214-21seq19:49:521
26501992113,6cyclictest11214-21seq19:49:521
26501992112,6cyclictest2661-21ntpq21:25:211
26501992112,6cyclictest23785-21date20:15:241
145382210,2sleep30-21swapper/323:10:393
2650399201,7cyclictest19257-21apt-get23:15:113
2650399201,5cyclictest0-21swapper/300:24:213
26503992012,6cyclictest3440-21smartctl20:45:163
26503992012,5cyclictest5559-21basename23:05:003
26503992012,5cyclictest3699-21sed19:30:193
26503992010,7cyclictest6933-21cron19:40:003
26503992010,7cyclictest27864-21tr20:25:233
26503992010,6cyclictest278-21systemd-journal22:15:033
2650299209,9cyclictest0-21swapper/222:21:562
2650299209,6cyclictest0-21swapper/221:55:122
26502992013,5cyclictest29642-21ls22:10:112
26502992013,4cyclictest17569-21sh21:36:142
26502992012,6cyclictest4368-21grep21:50:252
26502992012,6cyclictest26664-21seq22:07:362
26502992012,6cyclictest18643-21expr22:50:152
26502992012,6cyclictest15724-21cat00:00:172
26502992012,6cyclictest13970-21diskstats19:55:152
26502992012,5cyclictest17388-21grep00:25:262
26502992011,7cyclictest25474-21tune2fs22:55:162
26502992011,6cyclictest18689-21memory20:05:192
26501992016,3cyclictest52950irq/141-enp3s0-22:57:541
26501992016,3cyclictest52950irq/141-enp3s0-22:46:061
26501992012,6cyclictest8932-21seq19:42:571
26501992012,6cyclictest885-21tune2fs23:25:161
26501992012,6cyclictest5263-21expr19:35:131
26501992012,6cyclictest30323-21seq19:16:571
26501992012,6cyclictest14381-21ls19:55:181
26501992011,7cyclictest18362-21seq00:26:131
26501992010,7cyclictest24907-21df_inode20:20:141
297622190,3chrt0-21swapper/321:45:233
2650399199,6cyclictest30018-21python320:30:233
2650399191,8cyclictest11208-21perf22:45:003
26503991913,4cyclictest14144-21sh22:22:143
26503991912,5cyclictest31606-21sed22:35:213
26503991912,5cyclictest24969-21netstat22:30:203
26503991912,5cyclictest1845-21expr21:25:143
26503991912,4cyclictest13559-21diskmemload22:10:103
26503991911,6cyclictest32189-21sed20:35:243
26503991911,6cyclictest32132-21cat19:20:223
26503991911,6cyclictest12602-21awk21:05:223
26503991911,6cyclictest10605-21sendmail_mailqu21:00:203
26503991911,5cyclictest13559-21diskmemload21:16:593
26502991913,4cyclictest2029-21sh22:37:302
26502991913,4cyclictest19237-21sh23:39:172
26502991913,4cyclictest19237-21sh23:39:172
26502991913,4cyclictest10271-21sh00:20:222
26502991912,5cyclictest53050irq/142-enp3s0-23:15:122
26502991912,5cyclictest3668-21ls19:30:182
26502991912,5cyclictest28905-21expr19:15:132
26502991912,5cyclictest18333-21ntp_kernel_err22:25:212
26502991912,4cyclictest12684-21cpuspeed_turbos23:10:132
26502991911,6cyclictest5754-21cut23:05:012
26502991911,6cyclictest16448-21expr22:00:142
26502991911,6cyclictest1630-21ntpq20:40:212
26502991910,7cyclictest5715-21cut22:40:192
26502991910,7cyclictest28617-21pluginstate20:30:002
26502991910,6cyclictest12003-21df_inode22:45:152
2650199199,7cyclictest30988-21expr19:20:111
2650199192,12cyclictest26544-21apt-get19:10:131
26501991914,3cyclictest27271-21sh22:08:121
26501991914,3cyclictest24657-21hwlatdetect22:30:171
26501991912,5cyclictest6590-21awk22:40:261
26501991911,6cyclictest9086-21awk21:30:181
26501991911,6cyclictest4188-21cat21:50:241
26501991910,7cyclictest16337-21cat00:00:241
26501991910,6cyclictest22318-21if_err_enp3s000:05:201
26501991910,6cyclictest10434-21sh21:55:201
2650399189,7cyclictest27242-21sed19:10:193
2650399189,6cyclictest22382-21sed21:40:163
2650399189,6cyclictest18601-21cut20:05:193
26503991813,3cyclictest9095-21cron19:45:003
26503991811,5cyclictest8055-21wc20:55:183
26503991811,5cyclictest19613-21tr22:50:243
26503991811,4cyclictest5568-21grep20:50:173
2650399181,12cyclictest11491-21apt-get19:50:123
2650399180,16cyclictest417-21cron20:44:553
2650299189,7cyclictest31992-21grep20:35:222
2650299189,6cyclictest20725-21uname20:10:192
2650299182,11cyclictest5020-21apt-get20:50:122
2650299181,3cyclictest29778-21ntp_states20:30:212
26502991813,3cyclictest8341-21sendmail_mailtr19:40:222
26502991813,3cyclictest2072-21uptime19:25:272
26502991811,5cyclictest27813-21cat19:10:262
26502991811,5cyclictest24837-21expr20:20:132
26502991811,5cyclictest21810-21seq00:05:162
26502991811,5cyclictest12451-21seq21:05:192
26502991811,5cyclictest11735-21expr19:50:122
26502991810,6cyclictest5223-21cat19:35:132
26502991810,6cyclictest24110-21seq20:17:402
2650199189,6cyclictest52950irq/141-enp3s0-22:20:121
2650199188,7cyclictest0-21swapper/119:30:161
2650199181,8cyclictest16806-21ssh23:36:481
2650199181,8cyclictest16806-21ssh23:36:481
2650199181,15cyclictest28266-21ssh00:34:291
26501991811,4cyclictest13559-21diskmemload00:20:311
26501991811,4cyclictest0-21swapper/120:35:201
26501991810,6cyclictest560-21dbus-daemon20:35:001
26501991810,6cyclictest4621-21cat22:40:001
26501991810,6cyclictest4228-21timerwakeupswit20:45:241
26501991810,6cyclictest21631-21seq20:11:021
26501991810,6cyclictest19690-21seq20:08:341
26501991810,6cyclictest17735-21sed20:05:011
2650399172,8cyclictest9356-21apt-get19:45:123
2650399172,8cyclictest9356-21apt-get19:45:123
2650399172,10cyclictest9377-21apt-get21:55:123
2650399172,10cyclictest640-21snmpd19:55:123
26503991713,2cyclictest23329-21ntp_states20:15:203
26503991712,3cyclictest29601-21ntp_states19:15:213
26503991710,5cyclictest20949-21ntp_kernel_err20:10:203
26502991710,5cyclictest31108-21cat19:20:112
26502991710,5cyclictest15979-21expr20:00:142
2650199179,6cyclictest2333-21seq20:41:201
2650199178,6cyclictest9513-21basename21:00:091
26501991712,3cyclictest5977-21ls20:50:231
26501991712,3cyclictest11638-21/usr/sbin/munin21:05:161
26501991710,5cyclictest21459-21cat21:15:191
2650399169,5cyclictest278-21systemd-journal19:30:003
2650199168,6cyclictest26812-21expr20:25:131
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional