You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-14 - 04:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Thu May 14, 2026 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
220572178102,26sleep30-21swapper/319:08:063
282221660,1sleep10-21swapper/123:54:061
21997216697,62sleep00-21swapper/019:07:200
22018216498,55sleep10-21swapper/119:07:361
220092139104,24sleep20-21swapper/219:07:302
1241421190,6sleep228-21ksoftirqd/222:48:102
2643421130,1sleep20-21swapper/223:22:542
42412950,3sleep20-21swapper/223:55:122
223142700,10sleep228-21ksoftirqd/200:32:252
79702620,2sleep20-21swapper/221:05:132
161172600,1sleep30-21swapper/300:27:563
195082540,2sleep10-21swapper/122:29:361
47212530,1sleep30-21swapper/321:54:293
280412530,1sleep00-21swapper/021:23:160
52522500,2sleep00-21swapper/022:42:390
130942500,2sleep313029-21latency_hist22:25:003
18612480,2sleep30-21swapper/323:29:093
138262480,1sleep10-21swapper/100:01:421
138262480,1sleep10-21swapper/100:01:411
271872470,1sleep10-21swapper/100:35:331
162612470,2sleep00-21swapper/000:04:100
162612470,2sleep00-21swapper/000:04:090
163592460,1sleep20-21swapper/200:28:112
24652450,1sleep20-21swapper/223:05:122
134822450,1sleep30-21swapper/321:36:003
17512430,2sleep30-21swapper/300:17:013
139612410,1chrt13960-21/usr/sbin/munin20:05:130
139612410,1chrt13960-21/usr/sbin/munin20:05:120
150052340,11sleep228-21ksoftirqd/222:25:242
75072320,3chrt0-21swapper/123:09:051
199552310,8sleep2281ksoftirqd/223:18:052
199552310,8sleep2281ksoftirqd/223:18:052
296752290,2sleep00-21swapper/022:36:210
292302290,9sleep228-21ksoftirqd/223:49:462
100432280,2sleep10-21swapper/123:59:411
45552260,2sleep00-21swapper/000:19:510
22479992615,7cyclictest29561-21sort23:25:152
2247799261,24cyclictest0-21swapper/019:12:260
319032250,2sleep10-21swapper/123:26:431
22480992514,8cyclictest18282-21ls21:15:273
2248099251,23cyclictest0-21swapper/319:12:453
2247999253,16cyclictest16480-21ssh23:40:112
2247899251,1cyclictest0-21swapper/119:12:191
2247799251,10cyclictest326772grep20:45:260
62972240,1chrt0-21swapper/021:30:310
22480992415,7cyclictest5042-21ls23:55:193
2247999243,14cyclictest5176-21ssh00:20:122
2247999243,13cyclictest9531-21diskmemload22:20:122
22479992422,1cyclictest0-21swapper/219:12:052
2247999242,16cyclictest8891-21ssh23:10:042
2247999242,16cyclictest21387-21ssh22:55:122
22479992411,9cyclictest18373-21ssh21:15:272
22479992411,12cyclictest28-21ksoftirqd/221:25:012
2247899243,15cyclictest5648-21ssh21:55:111
2247899242,15cyclictest31339-21ssh21:50:101
22478992414,7cyclictest24956-21df_inode20:30:131
2248099236,4cyclictest141rcu_preempt21:25:153
22478992317,4cyclictest5494-21rm22:42:521
22478992310,8cyclictest53050irq/142-enp3s0-23:15:131
22478992310,8cyclictest53050irq/142-enp3s0-23:15:121
22477992313,7cyclictest30235-21python323:25:220
163162230,2sleep00-21swapper/022:26:210
318642220,2sleep10-21swapper/120:45:171
2248099221,6cyclictest3729-21hddtemp_smartct23:30:143
22480992213,7cyclictest24796-21python321:20:233
22480992212,8cyclictest5592-21tune2fs00:20:143
2247999222,14cyclictest32474-21taskset22:15:112
22479992212,7cyclictest1095-21ls22:40:002
2247899221,14cyclictest16728-21ssh21:15:131
22478992210,8cyclictest5334-21apt-get19:45:121
2247899220,19cyclictest13923-21unixbench-2d22:00:231
22477992214,6cyclictest26745-21ls00:35:230
48882210,3chrt0-21swapper/222:42:172
281592210,1sleep10-21swapper/100:12:421
2248099217,3cyclictest138252sh00:01:423
2248099217,3cyclictest138252sh00:01:413
2248099214,3cyclictest9531-21diskmemload21:57:103
2248099214,3cyclictest31080-21processes23:50:213
2248099213,16cyclictest5240-21cat21:00:003
2248099212,14cyclictest17891-21ssh21:40:103
2248099211,4cyclictest0-21swapper/300:07:093
2247999212,3cyclictest141rcu_preempt21:40:122
22479992110,9cyclictest560-21dbus-daemon23:40:002
22478992112,7cyclictest32154-21tr00:15:221
22478992111,7cyclictest595-21seq19:33:031
22478992111,7cyclictest17397-21cat20:10:241
2247799212,13cyclictest20343-21apt-get20:20:110
22477992113,6cyclictest3876-21date23:55:010
22477992112,6cyclictest27152-21df_inode19:20:140
2247799211,14cyclictest640-21snmpd20:35:020
2247799210,6cyclictest12860-21cut00:25:200
159842210,1sleep30-21swapper/322:02:213
2248099205,3cyclictest65602sh22:43:583
2248099202,12cyclictest29136-21apt-get20:40:123
22480992013,4cyclictest26639-21missed_timers22:10:183
22480992012,6cyclictest28736-21gpgv19:25:013
22480992012,5cyclictest18950irq/143-ahci[0023:00:013
22480992011,7cyclictest25766-21tune2fs00:35:153
22480992011,7cyclictest16504-21perl20:10:153
22480992011,6cyclictest3036-21load23:05:163
22480992011,6cyclictest20413-21cat20:20:133
22480992011,6cyclictest10900-21cat21:10:203
22480992011,5cyclictest52950irq/141-enp3s0-23:10:193
2248099201,13cyclictest19054-21ssh22:05:123
22480992010,7cyclictest31276-21apt-get20:45:143
22480992010,7cyclictest22825-21diskstats20:25:143
2247999209,8cyclictest560-21dbus-daemon19:40:012
2247999202,10cyclictest26828-21taskset19:20:122
22479992012,5cyclictest9531-21diskmemload00:16:282
22479992011,6cyclictest52850irq/140-enp3s0-00:00:112
22479992011,6cyclictest52850irq/140-enp3s0-00:00:112
22479992011,5cyclictest5600-21/usr/sbin/munin21:55:172
2247899209,7cyclictest12098-21df_inode20:00:151
2247899203,12cyclictest9029-21apt-get23:10:101
2247899203,11cyclictest20343-21apt-get22:30:121
2247899203,11cyclictest20343-21apt-get22:30:111
22478992012,6cyclictest9680-21seq21:33:571
22478992012,6cyclictest560-21dbus-daemon19:20:001
22478992012,6cyclictest560-21dbus-daemon19:19:591
22478992012,6cyclictest4912-21seq20:56:041
22478992012,6cyclictest12824-21grep21:35:241
22478992012,5cyclictest19617-21tr22:05:161
22478992011,6cyclictest19482-21sed00:30:181
22478992011,6cyclictest10715-21tr19:55:211
2247799209,5cyclictest18563-21sh00:05:200
2247799208,7cyclictest52750irq/139-enp3s0-22:10:120
2247799202,5cyclictest0-21swapper/021:15:160
2247799202,12cyclictest9608-21apt-get19:55:130
22477992016,2cyclictest52750irq/139-enp3s0-23:07:310
22477992014,4cyclictest7215-21latency_hist19:50:000
22477992014,4cyclictest16631-21idleruntime20:10:160
22477992012,6cyclictest3064-21sed19:40:000
22477992012,6cyclictest27690-21cat23:00:000
22477992011,6cyclictest16043-21cron23:40:010
22477992010,6cyclictest52750irq/139-enp3s0-23:45:140
71002190,2sleep00-21swapper/021:55:230
265882190,6sleep226568-21perf19:20:012
265882190,6sleep226568-21perf19:20:002
2248099192,12cyclictest13907-21apt-get20:05:123
2248099192,12cyclictest13907-21apt-get20:05:113
22480991913,4cyclictest6153-21irqstats21:00:173
22480991913,4cyclictest19218-21sh23:17:183
22480991913,4cyclictest19218-21sh23:17:173
22480991913,4cyclictest14134-21sh23:38:053
22480991912,5cyclictest29506-21sh00:14:033
22480991912,5cyclictest13282-21sh22:49:043
22480991912,4cyclictest18453-21sh00:30:033
22480991911,6cyclictest3933-21cat19:40:183
22480991911,6cyclictest32679-21cat22:15:123
22480991911,6cyclictest26010-21cat19:15:243
22480991911,6cyclictest26010-21cat19:15:233
22480991911,6cyclictest1030-21apt-get19:35:113
22480991911,5cyclictest31293-21basename19:30:113
22480991910,7cyclictest25225-21cat20:30:153
22480991910,6cyclictest52950irq/141-enp3s0-21:30:113
22480991910,6cyclictest28120-21grep23:00:133
2247999191,9cyclictest636-21runrttasks00:09:562
2247999191,9cyclictest20707-21sh22:30:142
2247999191,9cyclictest20707-21sh22:30:132
22479991914,3cyclictest7389-21sh21:05:002
22479991912,5cyclictest3396-21awk20:55:122
22479991912,5cyclictest14129-21expr20:05:132
22479991912,5cyclictest14129-21expr20:05:122
22479991912,4cyclictest636-21runrttasks21:50:132
22479991911,6cyclictest624-21gdbus23:55:002
22479991911,6cyclictest32202-21ntpq20:45:222
22479991911,6cyclictest30331-21sed20:40:232
22479991911,6cyclictest25690-21ls00:10:242
22479991911,6cyclictest18932-21ls20:15:212
22479991911,5cyclictest30265-21grep21:25:112
22479991911,5cyclictest28690-21sh23:00:182
2247999190,7cyclictest159162ssh21:38:262
2247899199,7cyclictest8412-21cat21:05:171
2247899199,7cyclictest23044-21hddtemp_smartct20:25:151
2247899198,8cyclictest7188-21kworker/u8:100:25:121
2247899198,7cyclictest22764-21grep22:55:251
2247899192,12cyclictest9970-21apt-get21:10:141
2247899192,12cyclictest25742-21apt-get22:10:111
2247899192,12cyclictest24586-21/usr/sbin/munin21:45:271
2247899191,8cyclictest27012-21/usr/sbin/munin22:35:231
22478991913,3cyclictest10392-21head23:35:161
22478991912,5cyclictest582-21grep22:15:161
22478991912,4cyclictest32721-21sh23:03:301
22478991911,5cyclictest9652-21grep22:45:261
2247899191,15cyclictest24571-21/usr/sbin/munin23:45:251
22478991910,7cyclictest278-21systemd-journal20:25:011
22478991910,6cyclictest29924-21seq20:40:201
22478991910,6cyclictest17041-21gltestperf23:40:151
2247899190,17cyclictest6203-21sed21:00:181
2247799192,12cyclictest3345-21apt-get20:55:120
2247799192,12cyclictest31321-21apt-get21:50:100
2247799192,12cyclictest24661-21apt-get19:15:120
2247799192,12cyclictest24661-21apt-get19:15:110
2247799192,12cyclictest24171-21apt-get00:10:110
2247799191,7cyclictest636-21runrttasks22:00:190
22477991914,3cyclictest9531-21diskmemload00:30:120
22477991913,4cyclictest28703-21kernelversion23:00:180
22477991913,4cyclictest20259-21sh21:41:030
22477991913,4cyclictest16723-21sh21:39:170
22477991912,5cyclictest26634-21sed20:35:010
22477991912,4cyclictest9531-21diskmemload23:22:560
22477991911,6cyclictest8038-21hostname19:50:140
22477991911,6cyclictest20992-21cat22:30:160
22477991911,6cyclictest20992-21cat22:30:150
22477991911,6cyclictest20090-21basename20:20:000
22477991911,6cyclictest16977-21perl23:40:140
22477991911,6cyclictest11261-21sh22:46:590
22477991910,6cyclictest32602-21cut22:15:120
22477991910,6cyclictest13808-21cut20:05:000
207142190,9sleep012-21ksoftirqd/023:18:510
207142190,9sleep012-21ksoftirqd/023:18:500
2248099189,7cyclictest22872-21cat23:45:003
2248099189,6cyclictest12665-21ntpq20:00:213
22480991813,3cyclictest27223-21sh21:46:183
22480991812,4cyclictest29360-21sh22:36:023
22480991812,4cyclictest17335-21sh22:51:253
22480991812,4cyclictest16476-21sh22:26:313
22480991811,5cyclictest9681-21cat19:55:133
22480991811,5cyclictest23445-21cat23:45:153
22480991811,5cyclictest23310-21ntpq23:20:203
22480991811,5cyclictest1471-21expr20:50:123
2248099181,14cyclictest29170-21apt-get19:25:113
22480991810,6cyclictest7387-21cat19:50:013
22480991810,5cyclictest7743-21grep21:05:123
2247999189,7cyclictest23142-21kernelversion20:25:182
2247999184,10cyclictest7222-21munin-run19:50:002
2247999182,11cyclictest25751-21fschecks_count00:35:142
22479991813,3cyclictest636-21runrttasks21:46:152
22479991812,4cyclictest17862-21sh22:04:152
22479991811,5cyclictest24894-21ls20:30:112
2247999181,15cyclictest16784-21sed20:10:172
22479991811,4cyclictest9531-21diskmemload21:31:372
22479991810,5cyclictest26028-21ls22:10:132
2247899189,7cyclictest16085-21sed22:50:221
2247899182,12cyclictest31481-21python321:25:211
22478991812,3cyclictest7740-21sh00:21:171
22478991812,3cyclictest26120-21sh21:21:221
22478991811,5cyclictest5179-21date19:45:011
22478991811,5cyclictest30757-21seq19:26:211
22478991811,5cyclictest18598-21cat21:40:161
22478991811,5cyclictest1136-21expr19:35:121
2247899181,12cyclictest18184-21apt-get20:15:141
2247799189,6cyclictest2412-21switchtime20:50:220
2247799182,11cyclictest24629-21ssh21:45:120
2247799182,11cyclictest22491-21apt-get20:25:110
2247799181,4cyclictest141rcu_preempt22:50:110
22477991813,3cyclictest31766-21unixbench_multi21:25:240
22477991813,3cyclictest12232-21sh23:11:550
22477991812,4cyclictest30258-21sendmail_mailtr20:40:230
22477991812,4cyclictest22403-21sh22:07:040
22477991812,4cyclictest16073-21sh21:14:490
22477991811,5cyclictest6077-21sed21:00:160
22477991810,6cyclictest7717-21seq00:21:150
22477991810,6cyclictest7591-21memory22:20:180
22477991810,6cyclictest4596-21tune2fs23:55:150
22477991810,6cyclictest29417-21expr19:25:130
22477991810,5cyclictest7651-21grep21:05:120
2248099179,6cyclictest9548-21cat19:55:003
22480991711,4cyclictest18258-21cpuspeed_turbos20:15:143
22480991710,5cyclictest20512-21grep22:30:133
22480991710,5cyclictest20512-21grep22:30:123
2247999177,6cyclictest52850irq/140-enp3s0-22:05:112
22479991712,3cyclictest32232-21ntp_states19:30:202
22479991712,3cyclictest15790-21processes22:50:212
22479991711,4cyclictest4701-21vmstat19:40:252
22479991710,5cyclictest22423-21cut20:25:002
22479991710,5cyclictest12894-21which20:00:232
22479991710,5cyclictest10285-21cut19:55:182
22479991710,4cyclictest9531-21diskmemload23:32:092
2247899179,6cyclictest14081-21ls20:05:131
2247899179,6cyclictest14081-21ls20:05:121
2247899179,5cyclictest278-21systemd-journal19:25:001
2247899175,10cyclictest22984-21if_enp3s023:20:181
22478991712,3cyclictest9531-21diskmemload23:30:361
22478991711,4cyclictest8866-21sh22:20:491
22478991711,4cyclictest22620-21sh00:08:501
2247799178,6cyclictest4793-21sed23:30:240
2248099169,5cyclictest27049-21cat20:35:123
2247999169,5cyclictest10332-21expr21:10:142
2247999167,7cyclictest9156-21seq19:52:362
2247999167,6cyclictest29940-21sort19:25:192
2247999161,11cyclictest746-21apt-get20:50:022
2247899169,5cyclictest27515-21expr20:35:161
22478991611,4cyclictest8680-21sed19:50:221
2247899161,11cyclictest1199-21apt-get20:50:121
2247799161,11cyclictest31312-21apt-get19:30:130
2247799161,10cyclictest3195-21apt-get19:40:120
2247999155,7cyclictest27080-21tr20:35:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional