You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-13 - 12:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Jul 13, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
73662244168,24sleep20-21swapper/219:09:002
74442226153,24sleep10-21swapper/119:10:001
71522205166,26sleep30-21swapper/319:06:123
72042201162,27sleep00-21swapper/019:06:530
2453021610,5sleep2753899cyclictest19:45:162
1915721590,4sleep3754599cyclictest23:15:133
7538996966,2cyclictest0-21swapper/223:10:182
7538995754,2cyclictest0-21swapper/221:45:152
226082570,3sleep230-21rcuc/223:20:262
66802480,2sleep20-21swapper/221:30:252
7533994110,13cyclictest29236-21sort22:25:011
7533994110,13cyclictest29236-21sort22:25:011
753399379,19cyclictest31639-21uptime23:40:271
753399377,21cyclictest14668-21gltestperf21:50:161
7533993712,22cyclictest16100-21fschecks_count20:40:161
7533993611,12cyclictest9231-21cut00:05:151
753399359,22cyclictest19050-21taskset19:33:241
753399359,18cyclictest28591-21diskmemload00:00:151
753399355,19cyclictest31260-21kworker/u8:222:54:001
753399348,13cyclictest11854-21/usr/sbin/munin21:45:191
7533993411,17cyclictest26788-21kworker/u8:022:37:381
753399339,11cyclictest16877-21/usr/sbin/munin23:10:241
753399338,11cyclictest13969-21expr00:15:191
753399337,16cyclictest31130-21tr21:15:121
753399336,13cyclictest6518-21smart_sda21:30:251
753399336,11cyclictest87550irq/128-enp1s0-23:09:201
753399333,19cyclictest6280-21kworker/u8:100:12:351
7533993310,15cyclictest30349-21memory22:25:171
7533993310,13cyclictest10483-21expr22:55:131
752899335,13cyclictest4065-21munin-run23:55:010
753399325,12cyclictest5122-21cat21:30:011
753399325,11cyclictest0-21swapper/123:20:151
7533993212,17cyclictest23881-21awk00:40:011
7533993211,8cyclictest16248-21/usr/sbin/munin21:55:261
7533993210,13cyclictest27620-21latency_hist23:35:011
753399318,7cyclictest25341-21taskset23:29:031
753399318,12cyclictest19939-21fschecks_time19:35:151
753399317,12cyclictest19665-21cut00:30:011
753399314,18cyclictest0-21swapper/123:15:151
7533993110,18cyclictest23597-21awk19:45:011
752899311,14cyclictest533-21munin-run21:20:010
753899303,16cyclictest19770-21kworker/u8:119:21:132
753399308,14cyclictest827-21runrttasks20:07:451
753399308,13cyclictest26915-21load19:50:191
753399307,9cyclictest11282-21kworker/1:120:15:011
753399306,10cyclictest9658-21/usr/sbin/munin21:40:261
753399305,18cyclictest26788-21kworker/u8:022:44:551
753399305,16cyclictest87550irq/128-enp1s0-23:55:141
7533993011,11cyclictest22292-21/usr/sbin/munin20:55:141
752899304,11cyclictest6275-21perf00:00:010
752899300,14cyclictest10058-21awk22:55:010
753399298,18cyclictest3270-21proc_pri23:50:231
753399298,14cyclictest808-21snmpd21:20:071
753399298,12cyclictest8972-21fgrep19:10:241
753399297,10cyclictest20359-21sed00:30:171
753399296,12cyclictest30788-21latency_hist21:15:001
7533992910,5cyclictest25040-21/usr/sbin/munin22:15:241
7533992910,15cyclictest28721-21cat21:10:001
752899292,13cyclictest6607-21munin-run20:20:010
752899291,13cyclictest3460-21awk22:40:010
7528992910,16cyclictest5646-21munin-run22:45:000
753399289,11cyclictest15398-21latency_hist20:40:021
753399289,10cyclictest19510-21ntp_states22:00:201
753399287,13cyclictest26189-21chrt21:02:281
753399285,8cyclictest6838-21/usr/sbin/munin20:20:261
753399284,20cyclictest25774-21latency_hist19:50:011
753399284,17cyclictest827-21runrttasks23:37:591
753399284,17cyclictest827-21runrttasks23:37:591
7533992820,5cyclictest10423-21smart_sda19:15:041
7533992819,5cyclictest9212-40dnf21:37:231
7533992810,10cyclictest18245-1kworker/u9:222:32:501
752899282,21cyclictest5947-21kworker/0:000:11:450
753399278,11cyclictest13132-21chrt20:34:151
753399276,8cyclictest28591-21diskmemload00:20:161
753399276,8cyclictest15018-21/usr/sbin/munin19:25:121
753399276,12cyclictest13543-21ntpq23:00:221
753399276,10cyclictest0-21swapper/123:45:271
753399275,15cyclictest17014-21kworker/u8:222:05:051
7533992720,4cyclictest6680-21irqstats22:45:171
752899271,8cyclictest0-21swapper/023:17:030
752899271,14cyclictest16649-21munin-run23:10:010
7528992710,14cyclictest28591-21diskmemload21:15:010
754599261,10cyclictest0-21swapper/320:45:473
754599260,11cyclictest0-21swapper/321:10:233
753399265,5cyclictest19346-21unixbench_multi20:45:281
753399265,10cyclictest12050irq/123-ahci[0019:25:011
753399264,12cyclictest0-21swapper/119:55:251
7533992618,5cyclictest21098-21ntp_states20:50:221
752899262,12cyclictest12596-21munin-run19:20:010
752899262,12cyclictest10696-21munin-run00:10:020
754599255,3cyclictest1866-21munin-run23:50:013
754599251,12cyclictest0-21swapper/322:20:473
754599251,12cyclictest0-21swapper/322:20:473
753399257,5cyclictest579-21systemd-journal20:26:121
753399251,18cyclictest0-21swapper/122:12:391
752899251,4cyclictest0-21swapper/023:25:260
752899251,22cyclictest11991-21kworker/0:022:58:050
753899241,9cyclictest0-21swapper/219:17:592
753899241,4cyclictest0-21swapper/223:03:292
7533992415,6cyclictest5322-21/usr/sbin/munin20:15:171
7533992415,6cyclictest30399-21/usr/sbin/munin20:00:101
7533992415,6cyclictest30399-21/usr/sbin/munin20:00:101
752899242,9cyclictest10674-21kworker/0:200:16:170
752899242,8cyclictest21604-21kworker/0:119:42:090
752899241,9cyclictest28340-21kworker/0:321:20:550
752899241,12cyclictest7853-21munin-run22:50:000
270782241,19sleep327164-21proc_pri19:50:233
134182240,4sleep30-21swapper/319:20:163
754599238,8cyclictest0-21swapper/300:10:233
753899239,3cyclictest0-21swapper/223:50:012
753899231,5cyclictest0-21swapper/200:28:092
753899231,5cyclictest0-21swapper/200:07:362
752899232,7cyclictest9784-21kworker/0:223:00:150
752899231,6cyclictest0-21swapper/023:30:570
752899231,6cyclictest0-21swapper/022:16:370
752899231,6cyclictest0-21swapper/021:45:230
752899231,6cyclictest0-21swapper/021:38:260
754599222,11cyclictest2091-21/usr/sbin/munin23:50:143
754599221,6cyclictest0-21swapper/321:57:213
7545992214,3cyclictest0-21swapper/321:32:393
754599221,12cyclictest1813-21df22:35:143
754599221,10cyclictest6921-21ntpq22:45:203
754599221,10cyclictest0-21swapper/321:15:193
7545992210,3cyclictest0-21swapper/322:19:183
753899223,5cyclictest31637-21/usr/sbin/munin22:30:212
753899221,6cyclictest26009-21/usr/sbin/munin19:50:142
753899221,4cyclictest0-21swapper/200:02:242
753899221,12cyclictest0-21swapper/220:31:292
753899221,11cyclictest0-21swapper/222:16:212
752899223,11cyclictest32607-21/usr/sbin/munin20:05:280
752899222,9cyclictest29606-21kworker/0:223:42:070
752899221,8cyclictest13102-21kworker/0:100:26:490
752899221,6cyclictest0-21swapper/022:06:350
752899221,6cyclictest0-21swapper/021:58:310
752899221,5cyclictest0-21swapper/023:20:320
752899221,5cyclictest0-21swapper/021:41:570
752899221,5cyclictest0-21swapper/000:35:030
6102221,3sleep30-21swapper/322:30:283
754599218,3cyclictest1775-21memory21:20:193
754599216,7cyclictest0-21swapper/323:34:423
754599211,6cyclictest0-21swapper/323:36:183
754599211,6cyclictest0-21swapper/323:36:183
7545992116,3cyclictest9146-21ntp_states22:50:213
7545992114,2cyclictest0-21swapper/300:30:173
7545992112,3cyclictest0-21swapper/321:36:003
754599211,17cyclictest11864-21cat21:45:123
7545992110,3cyclictest12104-21sed19:15:263
754599210,9cyclictest0-21swapper/323:25:163
754599210,10cyclictest0-21swapper/320:39:093
753899211,9cyclictest24682-21runrttasks22:13:252
753899211,9cyclictest0-21swapper/223:15:012
753899211,6cyclictest29801-21/usr/sbin/munin22:25:132
753899211,6cyclictest18448-21/usr/sbin/munin22:00:162
753899211,6cyclictest0-21swapper/222:09:022
753899211,5cyclictest0-21swapper/223:59:012
753899211,5cyclictest0-21swapper/223:09:192
753899211,5cyclictest0-21swapper/200:35:202
753899211,4cyclictest0-21swapper/222:23:532
753899211,4cyclictest0-21swapper/222:23:532
752899218,4cyclictest808-21snmpd22:20:190
752899218,4cyclictest808-21snmpd22:20:190
752899217,3cyclictest13532-21kworker/0:221:53:030
752899212,5cyclictest18245-1kworker/u9:200:34:430
752899211,7cyclictest0-21swapper/021:26:090
752899211,5cyclictest0-21swapper/023:45:100
752899211,5cyclictest0-21swapper/023:13:250
752899211,5cyclictest0-21swapper/022:28:390
752899211,5cyclictest0-21swapper/021:33:040
37642210,3sleep3754599cyclictest21:25:173
754599209,3cyclictest0-21swapper/321:00:243
754599204,3cyclictest0-21swapper/322:42:353
754599204,11cyclictest15628-21kworker/3:122:13:203
754599201,8cyclictest0-21swapper/323:00:103
754599201,5cyclictest0-21swapper/323:20:103
7545992014,4cyclictest11858-21seq20:30:163
7545992014,3cyclictest10598-21grep20:25:263
7545992011,3cyclictest16216-21/usr/sbin/munin23:05:263
754599201,13cyclictest0-21swapper/300:27:343
754599201,12cyclictest0-21swapper/322:05:203
754599201,11cyclictest28591-21diskmemload23:11:083
7545992010,3cyclictest0-21swapper/323:57:263
7545992010,1cyclictest0-21swapper/321:40:143
754599200,9cyclictest0-21swapper/323:40:263
754599200,9cyclictest0-21swapper/320:10:273
753899201,7cyclictest0-21swapper/221:59:252
753899201,6cyclictest12239-21latency_hist23:00:022
753899201,6cyclictest0-21swapper/221:13:102
753899201,4cyclictest0-21swapper/223:27:012
752899205,8cyclictest755-21rngd00:21:330
752899202,10cyclictest14257-21kworker/0:123:36:090
752899202,10cyclictest14257-21kworker/0:123:36:090
752899201,6cyclictest0-21swapper/022:31:110
752899201,3cyclictest0-21swapper/000:01:410
752899201,11cyclictest808-21snmpd19:29:450
752899201,11cyclictest26628-21/usr/sbin/munin21:05:250
752899201,11cyclictest24416-21/usr/sbin/munin21:00:140
752899201,11cyclictest22853-21/usr/sbin/munin22:10:180
752899201,10cyclictest26009-21/usr/sbin/munin19:50:170
754599199,3cyclictest6299-21sshd20:16:113
754599199,3cyclictest19259-21irqstats22:00:163
754599198,3cyclictest9710-21awk19:15:013
754599196,3cyclictest0-21swapper/319:35:193
754599193,10cyclictest87350irq/126-enp1s0-00:15:093
754599191,8cyclictest909-21sssd_nss22:25:133
7545991913,3cyclictest7412-21memory00:00:213
7545991912,2cyclictest9-21rcu_preempt22:55:263
7545991911,2cyclictest22298-21smartctl19:40:173
7545991911,2cyclictest16898-21users00:20:263
754599191,10cyclictest15235-21processes21:50:223
754599190,9cyclictest0-21swapper/319:30:253
753899195,3cyclictest30-21rcuc/222:38:032
753899193,5cyclictest28594-21munin-run21:10:012
753899191,8cyclictest87350irq/126-enp1s0-21:21:092
753899191,6cyclictest0-21swapper/223:50:462
753899191,6cyclictest0-21swapper/220:06:112
753899191,5cyclictest5871-21/usr/sbin/munin22:45:182
753899191,5cyclictest11730-21irqstats00:10:192
753899191,4cyclictest9553-21cat21:40:022
753899191,4cyclictest0-21swapper/221:42:572
753899191,3cyclictest0-21swapper/220:49:332
7538991913,4cyclictest8410-21df22:50:132
752899191,9cyclictest13175-21kworker/0:020:40:140
752899191,7cyclictest0-21swapper/019:20:120
752899191,6cyclictest0-21swapper/022:04:160
752899190,8cyclictest6836-21kworker/0:020:24:190
7545991813,3cyclictest401ktimersoftd/319:55:213
7545991810,3cyclictest0-21swapper/300:05:113
753899182,4cyclictest20031-21/usr/sbin/munin20:50:122
753899182,14cyclictest17599-21grep20:45:022
753899181,7cyclictest0-21swapper/221:15:192
753899181,6cyclictest0-21swapper/200:31:392
753899181,5cyclictest6838-21/usr/sbin/munin20:20:212
753899181,5cyclictest32212-21NetworkManager20:03:032
753899181,5cyclictest32212-21NetworkManager20:03:032
753899181,5cyclictest14224-21ls21:50:132
753899181,5cyclictest0-21swapper/221:29:342
753899181,4cyclictest25659-21/usr/sbin/munin23:30:222
753899181,4cyclictest2446-21/usr/sbin/munin20:10:132
753899181,4cyclictest0-21swapper/200:18:302
753899181,3cyclictest28052-21cstates23:35:142
753899181,3cyclictest28052-21cstates23:35:132
753899181,11cyclictest0-21swapper/222:44:252
752899187,3cyclictest20031-21/usr/sbin/munin20:50:170
752899181,5cyclictest15409-21perf20:40:000
752899181,10cyclictest18952-21sshd19:32:090
752899180,10cyclictest0-21swapper/020:55:090
319432180,2sleep20-21swapper/223:44:102
219522181,13sleep221965-21diskstats19:40:152
128262181,3sleep00-21swapper/020:30:280
754599177,3cyclictest22005-21awk20:55:013
754599174,3cyclictest1835-21users20:05:283
754599171,4cyclictest30478-21cpuspeed_turbos20:00:113
754599171,4cyclictest30478-21cpuspeed_turbos20:00:103
754599171,13cyclictest15633-21/usr/sbin/munin20:40:123
754599171,10cyclictest0-21swapper/300:35:023
7545991710,2cyclictest41-21ksoftirqd/320:57:373
753899171,8cyclictest0-21swapper/200:23:062
753899171,6cyclictest16081-21ntp_states19:25:212
753899171,6cyclictest0-21swapper/219:35:332
753899171,6cyclictest0-21swapper/219:14:282
753899171,3cyclictest0-21swapper/220:59:352
7538991713,2cyclictest29118-21memory19:55:202
7538991711,4cyclictest18393-21processes19:30:212
7538991710,4cyclictest5617-21/usr/sbin/munin20:15:202
752899176,2cyclictest30087-21kworker/0:220:01:110
752899176,2cyclictest30087-21kworker/0:220:01:110
7528991712,3cyclictest17828-21/usr/sbin/munin20:45:270
752899170,6cyclictest0-21swapper/019:10:250
257032170,2sleep00-21swapper/019:49:120
150442170,2sleep20-21swapper/220:35:372
754599168,4cyclictest24459-21gltestperf19:45:153
754599161,12cyclictest15931-21memory19:25:193
7545991610,3cyclictest26961-21df21:05:143
7545991610,2cyclictest39-21rcuc/320:20:563
753899161,5cyclictest0-21swapper/220:25:162
753899161,14cyclictest0-21swapper/221:00:252
752899161,6cyclictest2446-21/usr/sbin/munin20:10:180
7528991611,3cyclictest9039-21/usr/sbin/munin20:25:240
752899161,12cyclictest20975-21head19:35:260
752899151,5cyclictest28998-21cat19:55:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional