You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-10 - 17:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Thu Sep 10, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
62922238163,24sleep20-21swapper/207:08:522
61942234161,23sleep00-21swapper/007:07:360
63612203166,24sleep30-21swapper/307:09:453
60362201166,24sleep10-21swapper/107:05:331
1982221610,4sleep2648299cyclictest10:05:122
223522550,2sleep20-21swapper/212:35:262
92502520,4sleep19254-21gltestperf09:40:151
39052480,2sleep20-21swapper/211:55:172
646999432,10cyclictest21494-40dnf10:09:250
648699421,25cyclictest9206-21/usr/sbin/munin10:55:143
648699411,25cyclictest18621-21/usr/sbin/munin12:30:133
648699389,15cyclictest25563-21/usr/sbin/munin09:05:243
648699381,24cyclictest14226-21/usr/sbin/munin12:20:113
648699380,23cyclictest0-21swapper/311:20:173
646999389,13cyclictest15584-21munin-run11:10:010
646999380,12cyclictest1144-21awk08:10:010
648699376,3cyclictest87350irq/126-enp1s0-12:10:153
646999379,13cyclictest9954-21munin-run08:30:000
648699361,7cyclictest1357-21/usr/sbin/munin08:10:203
648699356,3cyclictest15810-21/usr/sbin/munin11:10:243
648699356,3cyclictest15810-21/usr/sbin/munin11:10:243
648699351,6cyclictest19549-21grep11:15:273
6482993532,2cyclictest0-21swapper/209:31:252
6482993526,8cyclictest0-21swapper/208:03:112
648699345,3cyclictest22739-21/usr/sbin/munin07:45:133
648699343,3cyclictest87650irq/129-enp1s0-09:50:203
648699341,3cyclictest29711-21cat10:25:233
6486993410,21cyclictest7008-21/usr/sbin/munin10:50:233
6486993410,21cyclictest22114-21grep07:40:253
6486993410,21cyclictest22114-21grep07:40:253
648699339,20cyclictest1057-21sshd08:09:163
648699338,3cyclictest9379-21sshd12:07:163
648699332,6cyclictest18346-21/usr/sbin/munin07:35:173
646999335,14cyclictest0-21swapper/010:00:020
648699329,20cyclictest13956-21/usr/sbin/munin07:25:133
648699325,3cyclictest87650irq/129-enp1s0-08:31:163
648699324,3cyclictest87650irq/129-enp1s0-08:00:273
648699323,5cyclictest4538-21diskstats09:30:133
648699323,3cyclictest87650irq/129-enp1s0-10:05:123
648699322,5cyclictest26526-21diskstats10:20:133
648699322,3cyclictest12377-21users09:45:253
648699322,19cyclictest0-21swapper/310:18:053
648699321,5cyclictest4802-21/usr/sbin/munin10:45:193
648699321,4cyclictest32536-21sh10:35:013
648699321,4cyclictest26346-21wc07:50:243
646999328,11cyclictest14355-21awk08:40:000
648699319,19cyclictest21165-21acpi08:55:123
648699319,19cyclictest18979-21cpu08:50:143
648699319,18cyclictest827-21runrttasks12:02:523
648699315,6cyclictest0-21swapper/312:20:013
648699313,3cyclictest0-21swapper/311:50:203
648699312,26cyclictest27428-21cut11:35:153
648699312,25cyclictest15162-21grep11:05:253
648699311,6cyclictest0-21swapper/309:42:213
6486993112,16cyclictest16423-21/usr/sbin/munin12:25:193
6486993111,17cyclictest827-21runrttasks10:10:393
6486993110,18cyclictest3575-21/usr/sbin/munin08:15:203
6486993110,18cyclictest1128-21timerandwakeup09:20:233
648699309,18cyclictest827-21runrttasks08:27:173
648699309,18cyclictest18659-21switchtime10:00:243
648699309,18cyclictest18659-21switchtime10:00:243
648699302,4cyclictest7018-21diskstats07:10:143
648699302,3cyclictest28873-21open_inodes09:10:253
648699301,4cyclictest30048-21/usr/sbin/munin09:15:123
648699301,4cyclictest13183-21awk08:35:173
6486993012,15cyclictest22064-21mailstats12:35:243
6486993011,16cyclictest24452-21open_inodes09:00:233
6486993010,17cyclictest808-21snmpd08:22:473
647599300,9cyclictest0-21swapper/110:13:571
646999306,11cyclictest19989-21idleruntime-cro11:20:010
648699299,17cyclictest6383-21/usr/sbin/munin09:35:183
648699299,17cyclictest16349-21expr07:30:113
648699298,18cyclictest5197-21turbostat12:00:013
648699296,4cyclictest31839-21head11:45:153
648699296,4cyclictest12680-21/usr/sbin/munin07:20:193
648699295,3cyclictest30529-21unixbench_singl11:40:253
648699292,5cyclictest10656-21/usr/sbin/munin07:15:213
648699292,4cyclictest23311-21memory11:25:183
648699292,3cyclictest25416-21runrttasks11:30:223
648699292,2cyclictest87650irq/129-enp1s0-10:35:153
648699292,2cyclictest87550irq/128-enp1s0-09:55:193
648699291,4cyclictest12583-21processes11:00:233
6486992910,16cyclictest2846-21wc09:25:173
6486992910,16cyclictest14647-21cat08:40:133
646999292,12cyclictest24726-21idleruntime-cro07:50:010
648699289,16cyclictest17301-21cut08:45:153
648699282,4cyclictest27811-21hddtemp_smartct07:55:153
648699281,6cyclictest0-21swapper/310:41:123
646999285,11cyclictest18133-21awk07:35:010
6482992710,4cyclictest581-21wc09:20:172
647599271,24cyclictest0-21swapper/109:56:531
647599271,17cyclictest14059-21ntp_states09:50:201
646999272,10cyclictest20917-21kworker/0:208:59:500
6469992712,3cyclictest1673-21kworker/0:009:35:110
646999271,13cyclictest16209-21awk12:25:020
647599262,4cyclictest28765-21munin-run11:40:011
646999261,13cyclictest14960-21munin-run09:55:020
646999261,11cyclictest16411-21kworker/0:212:25:190
647599251,18cyclictest0-21swapper/111:26:021
6475992510,2cyclictest0-21swapper/110:15:241
647599250,18cyclictest0-21swapper/112:33:161
647599250,18cyclictest0-21swapper/108:00:121
646999252,10cyclictest30936-21kworker/0:111:53:310
646999251,10cyclictest9795-21kworker/0:208:32:100
6475992421,2cyclictest0-21swapper/110:09:331
6475992419,2cyclictest161692sleep112:24:491
647599241,16cyclictest0-21swapper/110:35:241
647599241,16cyclictest0-21swapper/110:00:271
647599241,16cyclictest0-21swapper/110:00:261
647599241,16cyclictest0-21swapper/109:33:591
647599241,11cyclictest9952-21awk08:30:001
647599241,11cyclictest29105-21munin-run08:00:011
647599241,11cyclictest17470-21/usr/sbin/munin12:25:201
6475992410,3cyclictest22788-21munin-run12:40:021
646999249,3cyclictest27216-21kworker/0:109:07:380
646999246,4cyclictest923-21kworker/0:308:24:380
646999242,10cyclictest23348-21kworker/0:309:03:100
646999241,4cyclictest0-21swapper/012:10:130
646999241,10cyclictest22081-21kworker/0:111:37:420
6482992316,4cyclictest13745-21tr11:05:112
6475992320,2cyclictest0-21swapper/108:08:371
647599231,4cyclictest24662-21/usr/sbin/munin11:30:161
647599231,19cyclictest22520-21perf07:45:021
647599231,19cyclictest22520-21perf07:45:011
647599231,11cyclictest8115-21awk10:50:221
647599231,11cyclictest2089-21cpuspeed_turbos09:25:111
647599231,11cyclictest10345-21cstates08:30:111
647599231,10cyclictest7459-21sh09:35:181
647599231,10cyclictest20548-21diskstats11:20:131
647599231,10cyclictest10895-21ntp_states12:10:191
647599231,10cyclictest0-21swapper/111:08:171
647599230,2cyclictest0-21swapper/107:15:031
646999239,2cyclictest18988-21kworker/0:010:13:100
646999238,3cyclictest4311-21kworker/0:210:52:580
646999238,3cyclictest1673-21kworker/0:009:30:190
646999236,14cyclictest22185-21awk11:25:010
646999232,9cyclictest3935-21kworker/0:207:11:220
646999231,9cyclictest20258-21kworker/0:112:36:220
648299221,8cyclictest18963-21/usr/sbin/munin08:50:202
648299221,6cyclictest12990-21gzip07:20:232
647599228,3cyclictest30956-21sh11:45:011
647599228,3cyclictest26205-21smartctl07:50:231
647599222,8cyclictest18354-21diskstats11:15:141
647599222,10cyclictest755-21rngd10:41:291
647599221,9cyclictest0-21swapper/112:06:271
6475992219,2cyclictest0-21swapper/108:57:411
647599221,8cyclictest20030-21ntp_states08:50:211
6475992218,3cyclictest0-21swapper/107:13:231
647599221,7cyclictest1701-21hddtemp_smartct11:50:151
647599221,19cyclictest0-21swapper/110:30:221
647599221,18cyclictest27534-21diskmemload09:24:181
647599221,15cyclictest0-21swapper/108:24:571
647599221,10cyclictest2443-21ntp_states08:10:211
647599221,10cyclictest12825-21ntpq07:20:211
647599221,10cyclictest0-21swapper/109:45:151
647599220,16cyclictest0-21swapper/111:46:351
646999227,4cyclictest26851-21kworker/0:108:02:020
646999226,4cyclictest2284-21kworker/0:010:40:260
646999225,14cyclictest16550-21awk08:45:010
646999221,11cyclictest763-21dbus-daemon11:40:160
646999220,10cyclictest13600-21kworker/0:107:23:350
648299215,3cyclictest12050irq/123-ahci[0010:30:012
648299211,6cyclictest23759-21munin-run10:15:012
648299211,6cyclictest21247-21awk11:20:192
648299210,15cyclictest0-21swapper/208:25:462
647599212,17cyclictest0-21swapper/107:25:091
647599211,8cyclictest0-21swapper/110:59:431
647599211,8cyclictest0-21swapper/110:20:261
647599211,7cyclictest0-21swapper/109:15:231
647599211,19cyclictest0-21swapper/109:02:551
647599211,17cyclictest0-21swapper/111:10:031
647599211,17cyclictest0-21swapper/111:10:031
647599211,11cyclictest20297-21cat07:39:571
647599211,11cyclictest0-21swapper/111:57:241
647599211,10cyclictest4829-21cpu10:45:111
647599211,10cyclictest24707-21latency_hist07:50:011
647599211,10cyclictest0-21swapper/111:00:181
647599210,9cyclictest14102-21latency_hist12:20:011
646999217,2cyclictest20087-21kworker/0:107:42:540
646999217,2cyclictest20087-21kworker/0:107:42:540
646999216,4cyclictest29509-21kworker/0:209:14:500
646999213,15cyclictest32555-21awk10:35:010
646999212,10cyclictest24939-21/usr/sbin/munin07:50:110
646999211,9cyclictest6736-21kworker/0:109:49:030
646999211,9cyclictest4311-21kworker/0:210:48:460
646999211,8cyclictest3935-21kworker/0:207:17:500
646999211,8cyclictest10231-21kworker/0:209:43:110
646999211,11cyclictest16764-21/usr/sbin/munin08:45:210
6469992110,3cyclictest24379-21awk11:30:010
648299206,3cyclictest8564-1kworker/u9:108:32:502
648299205,4cyclictest5494-21hwlatdetect10:45:172
648299205,3cyclictest30837-21expr10:30:122
648299201,8cyclictest0-21swapper/212:31:442
648299201,7cyclictest0-21swapper/212:25:312
6482992017,2cyclictest0-21swapper/211:02:222
648299201,6cyclictest29904-21memory11:40:172
648299201,6cyclictest14633-21cpuspeed_turbos08:40:122
648299201,5cyclictest0-21swapper/209:40:142
648299201,11cyclictest0-21swapper/207:18:042
647599201,7cyclictest28160-21cut09:10:171
6475992017,2cyclictest0-21swapper/108:35:031
647599201,6cyclictest5419-21/usr/sbin/munin12:00:231
647599201,3cyclictest0-21swapper/108:49:031
647599201,10cyclictest0-21swapper/107:30:261
647599200,18cyclictest0-21swapper/110:26:591
646999206,5cyclictest0-21swapper/010:37:170
646999206,3cyclictest26851-21kworker/0:107:55:340
646999206,2cyclictest27978-21kworker/0:210:23:380
646999201,9cyclictest20258-21kworker/0:112:33:300
646999201,7cyclictest18988-21kworker/0:010:29:420
648299195,3cyclictest87650irq/129-enp1s0-11:50:122
648299195,3cyclictest87650irq/129-enp1s0-10:35:202
648299195,3cyclictest87650irq/129-enp1s0-09:55:132
648299195,3cyclictest87350irq/126-enp1s0-08:23:022
648299193,5cyclictest27534-21diskmemload12:21:052
648299192,5cyclictest27465-40dnf09:09:162
648299191,7cyclictest0-21swapper/212:11:182
648299191,6cyclictest9379-21expr10:55:112
648299191,6cyclictest808-21snmpd07:42:392
648299191,6cyclictest808-21snmpd07:42:392
648299191,6cyclictest7564-21ps09:35:202
648299191,6cyclictest2606-21/usr/sbin/munin10:40:142
648299191,6cyclictest23086-21diskstats07:45:152
648299191,6cyclictest15433-21sed07:25:232
648299191,6cyclictest0-21swapper/209:16:392
648299191,5cyclictest0-21swapper/212:02:042
648299191,5cyclictest0-21swapper/210:00:322
648299191,5cyclictest0-21swapper/210:00:322
6482991914,3cyclictest23880-21grep11:25:242
6482991913,3cyclictest7649-21tail10:50:162
6482991913,3cyclictest13582-21sed12:15:232
6482991912,4cyclictest29235-21grep09:10:272
648299191,15cyclictest31703-21tr08:05:142
648299191,10cyclictest0-21swapper/208:18:342
648299190,7cyclictest3166-21ps09:25:212
648299190,7cyclictest14265-21/usr/sbin/munin09:50:232
648299190,14cyclictest0-21swapper/207:35:072
647599191,6cyclictest25563-21/usr/sbin/munin09:05:151
6475991916,2cyclictest0-21swapper/108:40:291
647599190,17cyclictest0-21swapper/108:15:031
646999198,3cyclictest25048-21ntp_states10:15:200
646999198,3cyclictest11405-21/usr/sbin/munin11:00:250
646999197,9cyclictest827-21runrttasks12:06:570
646999197,3cyclictest0-21swapper/009:23:440
646999196,2cyclictest30936-21kworker/0:112:03:220
646999196,2cyclictest11347-21kworker/0:007:38:460
646999195,6cyclictest0-21swapper/011:31:020
646999194,5cyclictest17380-21/usr/sbin/munin10:00:210
646999194,5cyclictest17380-21/usr/sbin/munin10:00:200
646999191,3cyclictest0-21swapper/011:59:280
648299181,7cyclictest0-21swapper/212:05:232
648299181,7cyclictest0-21swapper/209:47:092
648299181,6cyclictest27534-21diskmemload10:15:152
648299181,5cyclictest87350irq/126-enp1s0-07:10:232
648299181,5cyclictest31525-21df_inode11:45:122
648299181,5cyclictest0-21swapper/210:20:112
6482991813,3cyclictest18005-21/usr/sbin/munin11:15:182
6482991813,3cyclictest16887-21sh11:10:242
6482991813,3cyclictest16887-21sh11:10:242
648299181,15cyclictest0-21swapper/211:36:562
648299180,9cyclictest0-21swapper/207:59:372
646999186,3cyclictest0-21swapper/012:15:330
646999181,4cyclictest31184-21/usr/sbin/munin11:45:170
646999181,15cyclictest0-21swapper/008:53:270
646999181,11cyclictest0-21swapper/011:11:060
646999181,11cyclictest0-21swapper/011:11:060
646999180,10cyclictest30993-21/usr/sbin/munin09:15:200
648299172,3cyclictest25022-21cpuspeed_turbos07:50:132
648299171,5cyclictest0-21swapper/211:30:152
6482991712,3cyclictest17191-21ntp_kernel_pll_07:30:192
648299171,13cyclictest21264-21tr08:55:112
6482991711,3cyclictest1818-21fschecks_count08:10:162
648299171,13cyclictest13598-21grep08:35:232
646999175,3cyclictest0-21swapper/010:59:560
646999172,12cyclictest1993-21/usr/sbin/munin09:25:220
648299161,13cyclictest24031-21hddtemp_smartct09:00:162
646999161,9cyclictest0-21swapper/008:15:210
646999161,6cyclictest0-21swapper/007:25:130
6469991611,3cyclictest1357-21/usr/sbin/munin08:10:180
6482991510,3cyclictest17996-21find08:45:222
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional