You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-17 - 13:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue Mar 17, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
129192238164,23sleep10-21swapper/119:09:231
129452234156,24sleep00-21swapper/019:09:420
127122230155,24sleep20-21swapper/219:06:422
128132202163,27sleep30-21swapper/319:07:593
2645621610,4sleep11306599cyclictest00:30:231
13075999592,2cyclictest0-21swapper/322:29:013
13075995450,3cyclictest0-21swapper/319:42:273
309092520,4sleep123-21ksoftirqd/121:00:171
287192500,4sleep20-21swapper/222:10:142
265562500,1sleep20-21swapper/220:50:202
6482490,2sleep10-21swapper/122:20:121
189132440,1sleep10-21swapper/120:35:021
1305999448,33cyclictest27202-21munin-run00:35:010
1305999440,13cyclictest1773-21awk21:10:010
1305999411,14cyclictest28145-21munin-run22:10:010
13059993910,13cyclictest24385-21idleruntime-cro23:15:020
13075993834,3cyclictest0-21swapper/321:13:213
13059993510,4cyclictest0-21swapper/022:20:110
1305999348,3cyclictest5768-21/usr/sbin/munin20:05:150
1305999345,9cyclictest31929-21awk21:05:010
1305999343,14cyclictest21556-21perf21:55:020
53262330,4sleep339-21rcuc/321:15:213
185272330,6sleep011-21rcuc/019:20:120
1305999339,11cyclictest23744-21munin-run22:00:020
1305999338,3cyclictest29327-21/usr/sbin/munin19:45:230
1305999337,12cyclictest29405-21awk00:40:010
1305999332,8cyclictest0-21swapper/000:10:240
13059993312,3cyclictest22665-21sed20:40:250
13059993311,3cyclictest9807-21/usr/sbin/munin23:55:210
1305999325,3cyclictest12050irq/123-ahci[0022:45:020
1305999311,15cyclictest0-21swapper/019:30:350
13059993111,3cyclictest23488-21cut00:25:150
13572300,4sleep339-21rcuc/322:20:173
13075993012,4cyclictest13190-21kworker/3:221:30:213
1305999308,3cyclictest12013-21/usr/sbin/munin00:00:230
1305999301,14cyclictest26909-21munin-run19:40:020
13059993011,3cyclictest12443-21ntpq22:45:190
13059992912,3cyclictest16747-21cut20:30:020
13059992911,3cyclictest13318-21sh21:35:130
1305999291,10cyclictest15282-21sshd22:51:330
13059992910,3cyclictest7450-21sh21:20:210
13059992910,3cyclictest7450-21sh21:20:210
56442280,3sleep20-21swapper/221:15:242
152062280,4sleep230-21rcuc/221:40:122
1307599282,8cyclictest13190-21kworker/3:200:15:453
1305999281,5cyclictest17376-21/usr/sbin/munin21:45:160
1305999281,4cyclictest7966-21/usr/sbin/munin20:10:180
1305999281,4cyclictest7966-21/usr/sbin/munin20:10:180
13059992811,3cyclictest31653-21grep23:30:160
161382270,4sleep123-21ksoftirqd/121:40:191
13075992712,2cyclictest13190-21kworker/3:220:55:253
1307599271,10cyclictest13190-21kworker/3:220:42:413
1306599271,4cyclictest24374-21munin-run23:15:021
1306599271,18cyclictest4808-21users23:40:261
1305999271,5cyclictest13259-21/usr/sbin/munin19:10:190
1305999271,4cyclictest16139-21diskstats22:55:150
1305999271,4cyclictest0-21swapper/021:40:050
1305999271,12cyclictest0-21swapper/020:35:100
1305999270,12cyclictest0-21swapper/000:20:370
1307599262,12cyclictest13190-21kworker/3:220:53:313
1307099262,2cyclictest86550irq/126-enp1s0-00:40:012
1305999261,5cyclictest22125-21users19:25:250
1305999260,12cyclictest0-21swapper/023:50:470
33242250,2sleep30-21swapper/319:59:593
20352250,2sleep10-21swapper/123:35:211
13075992511,2cyclictest13190-21kworker/3:219:20:153
1307099250,5cyclictest32-21ksoftirqd/223:59:032
1306599254,18cyclictest10673-21cat21:30:011
1305999258,3cyclictest28604-21taskset23:23:100
1305999253,3cyclictest0-21swapper/023:49:190
1305999253,3cyclictest0-21swapper/023:49:190
1305999251,6cyclictest812-21snmpd19:19:460
1305999251,12cyclictest0-21swapper/023:16:460
13059992511,2cyclictest0-21swapper/022:25:230
1305999251,11cyclictest0-21swapper/021:33:510
13059992510,4cyclictest0-21swapper/023:05:230
1305999250,13cyclictest0-21swapper/021:14:230
1307599245,10cyclictest1764-21diskmemload22:07:173
1307599242,11cyclictest13190-21kworker/3:200:20:273
1307599242,10cyclictest13190-21kworker/3:223:08:073
1307599241,5cyclictest0-21swapper/321:35:023
1307599241,11cyclictest13190-21kworker/3:219:15:593
1306599249,4cyclictest0-21swapper/123:26:191
1306599249,13cyclictest23-21ksoftirqd/100:13:091
1306599241,5cyclictest27544-21grep00:35:131
1306599241,17cyclictest0-21swapper/122:56:081
1306599241,17cyclictest0-21swapper/100:01:231
1306599241,14cyclictest0-21swapper/122:30:461
1305999241,11cyclictest0-21swapper/020:48:380
1305999241,10cyclictest0-21swapper/020:04:470
1305999241,10cyclictest0-21swapper/019:40:110
1305999240,10cyclictest0-21swapper/020:22:230
54572231,2sleep14184-21/usr/sbin/munin21:15:241
1307599232,9cyclictest13190-21kworker/3:223:15:113
1307599232,9cyclictest13190-21kworker/3:221:00:233
1307599232,10cyclictest13190-21kworker/3:200:09:313
1307599231,9cyclictest13190-21kworker/3:220:16:393
13075992316,4cyclictest30829-21grep19:45:243
13075992310,2cyclictest13190-21kworker/3:221:46:223
1307099233,6cyclictest32-21ksoftirqd/220:00:012
1306599238,3cyclictest0-21swapper/123:50:261
13065992315,3cyclictest0-21swapper/119:52:491
1306599231,11cyclictest15373-21munin-node19:15:021
1306599230,16cyclictest0-21swapper/121:20:051
1306599230,16cyclictest0-21swapper/121:20:051
1306599230,16cyclictest0-21swapper/100:17:511
1305999239,2cyclictest0-21swapper/020:50:140
1305999237,3cyclictest1764-21diskmemload00:15:150
1305999237,3cyclictest0-21swapper/022:35:310
1305999231,12cyclictest0-21swapper/022:10:110
1305999231,10cyclictest0-21swapper/020:30:220
1305999230,11cyclictest0-21swapper/022:02:470
1307599225,7cyclictest0-21swapper/322:40:103
1307599225,11cyclictest812-21snmpd19:10:363
1307599222,10cyclictest13190-21kworker/3:223:14:073
1307599222,10cyclictest13190-21kworker/3:200:10:513
13075992216,3cyclictest27924-21awk23:20:223
1307599221,10cyclictest13190-21kworker/3:221:54:303
1307599221,10cyclictest13190-21kworker/3:221:42:233
1307099223,6cyclictest12050irq/123-ahci[0021:55:022
1307099221,6cyclictest30-21rcuc/219:55:012
13070992215,4cyclictest7724-21grep23:50:132
13065992215,2cyclictest0-21swapper/119:56:591
1306599221,3cyclictest26910-21perf19:40:021
1306599221,19cyclictest0-21swapper/123:45:181
1306599221,19cyclictest0-21swapper/123:45:181
1306599221,19cyclictest0-21swapper/122:00:141
1306599221,16cyclictest0-21swapper/119:41:181
1306599221,15cyclictest0-21swapper/120:10:031
1306599221,15cyclictest0-21swapper/120:10:031
1306599221,14cyclictest0-21swapper/121:12:371
1306599220,16cyclictest0-21swapper/120:05:061
1305999228,3cyclictest0-21swapper/019:53:230
1305999225,3cyclictest0-21swapper/020:56:270
317882210,3sleep231792-21gltestperf23:30:162
19002210,4sleep121-21rcuc/121:10:001
1307599219,3cyclictest13190-21kworker/3:220:12:553
1307599219,3cyclictest13190-21kworker/3:220:12:553
1307599213,3cyclictest0-21swapper/321:22:053
1307599213,3cyclictest0-21swapper/321:22:053
1307599212,9cyclictest13190-21kworker/3:223:02:193
1307599212,12cyclictest13190-21kworker/3:220:45:133
1307599212,10cyclictest7615-21/usr/sbin/munin23:50:143
1307599212,10cyclictest3222-21/usr/sbin/munin23:40:153
1307599211,8cyclictest0-21swapper/323:59:033
1307599211,10cyclictest13190-21kworker/3:219:53:023
13075992110,2cyclictest14013-21awk00:05:013
1307599210,10cyclictest0-21swapper/323:37:053
1307099219,3cyclictest12050irq/123-ahci[0020:50:022
1307099213,6cyclictest86850irq/129-enp1s0-19:30:212
1307099212,5cyclictest22728-21taskset00:24:142
1307099211,11cyclictest0-21swapper/219:11:392
1306599219,3cyclictest13971-21strings21:35:201
1306599211,5cyclictest0-21swapper/122:06:201
1306599211,3cyclictest30330-21crond22:15:011
1306599211,3cyclictest0-21swapper/119:22:511
13065992113,3cyclictest0-21swapper/120:04:001
1306599211,15cyclictest0-21swapper/122:53:361
1306599211,15cyclictest0-21swapper/120:18:421
1306599211,14cyclictest0-21swapper/119:25:271
1306599211,11cyclictest27920-21pmu-power23:20:221
1306599211,11cyclictest0-21swapper/122:46:581
1306599211,11cyclictest0-21swapper/121:57:131
13065992110,3cyclictest21493-21df20:40:141
1306599210,19cyclictest0-21swapper/121:50:221
1306599210,16cyclictest0-21swapper/123:17:521
1306599210,15cyclictest0-21swapper/123:08:511
1306599210,15cyclictest0-21swapper/120:25:191
1305999217,3cyclictest0-21swapper/023:36:270
1305999217,3cyclictest0-21swapper/023:25:030
1305999217,3cyclictest0-21swapper/000:07:390
1305999214,8cyclictest1894-21fschecks_time19:55:160
1305999211,11cyclictest0-21swapper/023:01:550
1305999211,11cyclictest0-21swapper/022:34:230
176112200,2sleep20-21swapper/219:15:262
1307599204,3cyclictest0-21swapper/323:31:103
1307599204,3cyclictest0-21swapper/300:28:533
1307599201,8cyclictest13190-21kworker/3:220:04:423
13075992015,3cyclictest25860-21cut00:30:153
1307599201,3cyclictest6187-21cat22:30:253
1307599201,10cyclictest1764-21diskmemload22:15:173
1307599201,10cyclictest0-21swapper/321:27:573
1307099203,6cyclictest86550irq/126-enp1s0-21:25:172
1307099203,10cyclictest0-21swapper/221:35:132
13070992011,2cyclictest86550irq/126-enp1s0-23:00:162
1307099201,11cyclictest311ktimersoftd/219:20:112
13070992010,2cyclictest86550irq/126-enp1s0-20:01:482
13065992016,3cyclictest0-21swapper/122:17:381
13065992016,2cyclictest0-21swapper/122:37:111
1306599201,18cyclictest0-21swapper/120:52:081
1306599201,17cyclictest1764-21diskmemload22:29:031
1306599201,15cyclictest0-21swapper/100:08:381
1306599201,14cyclictest0-21swapper/123:04:311
1306599201,14cyclictest0-21swapper/120:20:081
1306599201,14cyclictest0-21swapper/119:46:261
1306599201,14cyclictest0-21swapper/119:15:251
1306599201,11cyclictest302-21users23:30:261
1306599201,11cyclictest24396-21chrt00:25:251
1306599201,11cyclictest23499-21cpuspeed_turbos20:45:131
1306599201,10cyclictest27849-21cpuspeed_turbos20:55:121
1306599201,10cyclictest0-21swapper/121:49:101
1306599200,15cyclictest0-21swapper/120:36:401
1306599200,15cyclictest0-21swapper/119:31:451
1305999201,6cyclictest0-21swapper/023:41:130
1305999200,8cyclictest0-21swapper/022:17:470
1305999200,8cyclictest0-21swapper/021:25:390
283562191,3sleep30-21swapper/322:10:043
1307599198,3cyclictest20540-21/usr/sbin/munin19:25:173
1307599196,3cyclictest0-21swapper/322:37:523
1307599194,3cyclictest0-21swapper/300:38:053
1307599192,5cyclictest20930-21munin-run20:40:013
1307599191,10cyclictest30594-21sshd23:25:343
1307599191,10cyclictest1764-21diskmemload22:50:033
1307599191,10cyclictest0-21swapper/323:48:523
1307599191,10cyclictest0-21swapper/323:48:523
1307599190,8cyclictest0-21swapper/321:58:463
1307099198,8cyclictest20845-21gltestperf23:05:162
1307099198,3cyclictest23146-21if_enp2s023:10:172
1307099195,2cyclictest12050irq/123-ahci[0020:15:012
1307099195,2cyclictest12050irq/123-ahci[0020:15:012
1307099191,8cyclictest0-21swapper/222:02:562
1307099191,7cyclictest10979-21irqstats20:15:192
1307099191,6cyclictest0-21swapper/223:36:322
13070991914,3cyclictest20637-21tr19:25:102
1307099191,2cyclictest86550irq/126-enp1s0-22:40:202
1307099191,10cyclictest5907-21/usr/sbin/munin22:30:222
13065991915,3cyclictest0-21swapper/121:31:561
1306599191,2cyclictest772-21avahi-daemon00:20:241
1306599190,12cyclictest7373-21kworker/u8:223:59:101
267542181,2sleep20-21swapper/219:38:172
1307599187,3cyclictest24935-21/usr/sbin/munin19:35:133
1307599185,5cyclictest782-21in:imjournal20:29:333
1307599184,3cyclictest0-21swapper/322:45:403
13075991812,3cyclictest17378-21users22:55:263
1307599180,7cyclictest0-21swapper/322:03:493
1307099187,3cyclictest86550irq/126-enp1s0-20:30:022
1307099186,3cyclictest5202-21awk23:45:012
1307099185,4cyclictest15516-21sshd22:54:332
1307099185,3cyclictest7384-21crond23:50:012
1307099185,3cyclictest7384-21crond23:50:012
1307099182,7cyclictest12013-21/usr/sbin/munin00:00:212
1307099181,9cyclictest0-21swapper/220:30:132
1307099181,8cyclictest21156-21/usr/sbin/munin20:40:172
1307099181,8cyclictest0-21swapper/222:56:532
1307099181,7cyclictest0-21swapper/223:25:222
1307099181,7cyclictest0-21swapper/222:39:532
1307099181,7cyclictest0-21swapper/222:25:242
1307099181,6cyclictest0-21swapper/221:30:112
1307099180,2cyclictest86750irq/128-enp1s0-21:00:212
1306599181,16cyclictest0-21swapper/122:40:111
1307599176,2cyclictest1377-21sshd21:05:333
1307599171,2cyclictest86850irq/129-enp1s0-19:30:233
1307099178,3cyclictest0-21swapper/220:39:092
1307099176,4cyclictest30-21rcuc/200:09:442
1307099175,2cyclictest86850irq/129-enp1s0-00:30:172
1307099174,3cyclictest86550irq/126-enp1s0-19:40:142
1307099174,2cyclictest86550irq/126-enp1s0-21:55:162
1307099171,8cyclictest0-21swapper/223:19:472
1307099171,7cyclictest28799-21cat23:25:012
1307099171,7cyclictest12362-21/usr/sbin/munin20:20:252
1307099171,6cyclictest0-21swapper/222:20:152
1307099171,3cyclictest0-21swapper/221:21:222
1307099171,3cyclictest0-21swapper/221:21:212
13070991713,3cyclictest0-21swapper/221:07:322
13070991712,3cyclictest27168-21seq22:05:182
13070991712,3cyclictest18051-21hddtemp_smartct21:45:162
13070991712,3cyclictest17246-21irqstats00:10:172
1307099171,14cyclictest0-21swapper/221:13:002
13070991711,3cyclictest7757-21idleruntime-cro20:10:012
1305999171,2cyclictest0-21swapper/021:17:190
1307599161,6cyclictest12362-21/usr/sbin/munin20:20:123
13075991611,3cyclictest5768-21/usr/sbin/munin20:05:103
1307599160,7cyclictest0-21swapper/320:35:023
1307099167,2cyclictest86850irq/129-enp1s0-19:45:252
1307099161,7cyclictest0-21swapper/222:15:142
1307099161,6cyclictest23027-21/usr/sbin/munin00:25:182
1307099161,3cyclictest0-21swapper/200:15:022
13070991611,3cyclictest11750-21forks22:45:142
1305999163,3cyclictest0-21swapper/020:15:070
1307099152,4cyclictest27741-21/usr/sbin/munin20:55:182
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional