You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-16 - 13:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat May 16, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10082233165,56sleep20-21swapper/219:09:502
9782228161,55sleep30-21swapper/319:09:273
10062202165,25sleep00-21swapper/019:09:470
6042201166,23sleep10-21swapper/119:05:331
2851421490,4sleep0111199cyclictest23:50:140
11249910399,3cyclictest0-21swapper/221:25:542
2266921000,4sleep122667-21latency19:55:181
214752690,4sleep2112499cyclictest21:05:222
214752690,4sleep2112499cyclictest21:05:222
207352520,1sleep30-21swapper/319:50:223
111199377,14cyclictest753-21munin-run21:35:010
112899341,11cyclictest0-21swapper/320:17:073
112499331,10cyclictest0-21swapper/219:25:262
112499321,22cyclictest0-21swapper/220:06:112
111199326,12cyclictest9-21rcu_preempt21:05:020
112499310,23cyclictest0-21swapper/221:50:272
111199316,12cyclictest3320-21perf19:15:010
111199314,12cyclictest32235-21awk00:00:010
111199311,14cyclictest27219-21munin-run22:35:020
10212310,4sleep339-21rcuc/300:00:223
39232300,5sleep3112899cyclictest22:55:113
112499301,13cyclictest30704-21ntp_states22:40:222
111199307,11cyclictest8410-21perf19:25:020
111199306,12cyclictest18429-21munin-run22:15:010
1111993011,16cyclictest28807-21awk21:25:010
94532290,4sleep09-21rcu_preempt20:40:120
112899291,9cyclictest22202-21diskmemload00:35:263
112899291,18cyclictest0-21swapper/319:42:463
112899291,10cyclictest0-21swapper/320:33:293
112899290,12cyclictest0-21swapper/322:53:263
112499290,20cyclictest0-21swapper/220:39:062
112899281,10cyclictest0-21swapper/300:15:353
112899280,20cyclictest0-21swapper/322:41:123
112499281,13cyclictest0-21swapper/221:31:232
112499281,13cyclictest0-21swapper/219:55:232
112499281,12cyclictest0-21swapper/221:00:182
112499280,9cyclictest0-21swapper/223:26:512
111799282,12cyclictest7530-21irqstats00:15:181
111199284,12cyclictest15293-21perf00:35:020
112899271,9cyclictest0-21swapper/322:31:143
112899271,17cyclictest0-21swapper/319:39:463
112899271,11cyclictest0-21swapper/323:58:103
112899271,11cyclictest0-21swapper/319:26:303
112899270,16cyclictest0-21swapper/322:06:053
112899270,11cyclictest0-21swapper/323:37:223
112899270,11cyclictest0-21swapper/322:24:303
1124992715,6cyclictest0-21swapper/222:29:022
112499271,16cyclictest0-21swapper/200:08:502
112499270,12cyclictest0-21swapper/220:03:072
112899261,12cyclictest0-21swapper/319:23:063
112899261,11cyclictest0-21swapper/320:50:053
112899260,5cyclictest0-21swapper/321:11:303
112899260,10cyclictest0-21swapper/300:14:553
112499266,2cyclictest0-21swapper/223:48:552
112499264,10cyclictest88950irq/132-enp2s0-23:43:172
112499261,7cyclictest0-21swapper/220:33:342
112499261,7cyclictest0-21swapper/200:15:102
1124992616,2cyclictest108562sleep220:40:422
1124992615,3cyclictest0-21swapper/223:04:142
112499261,11cyclictest0-21swapper/222:23:182
112499261,10cyclictest0-21swapper/220:59:222
112499260,18cyclictest0-21swapper/222:56:462
112899251,11cyclictest0-21swapper/321:06:483
112899251,11cyclictest0-21swapper/321:06:483
112899251,11cyclictest0-21swapper/320:29:213
112899251,11cyclictest0-21swapper/320:22:353
112899251,10cyclictest0-21swapper/322:38:403
112899251,10cyclictest0-21swapper/319:16:363
112899251,10cyclictest0-21swapper/319:12:043
112899250,9cyclictest0-21swapper/300:27:153
112499251,9cyclictest16199-21strings19:40:192
112499251,20cyclictest12405-21users19:30:272
112499251,12cyclictest32276-21chrt20:16:232
112499250,9cyclictest0-21swapper/223:37:072
112499250,12cyclictest0-21swapper/222:11:262
112499250,11cyclictest0-21swapper/221:47:032
112499250,11cyclictest0-21swapper/200:14:142
112499250,10cyclictest8860-21fw_packets23:05:142
112499250,10cyclictest0-21swapper/222:19:582
112499250,10cyclictest0-21swapper/222:19:582
111199258,14cyclictest8059-21munin-run23:05:010
35412240,3sleep0111199cyclictest00:05:230
112899249,3cyclictest0-21swapper/321:37:323
112899241,9cyclictest0-21swapper/322:15:053
112899241,9cyclictest0-21swapper/322:15:053
112899241,9cyclictest0-21swapper/321:21:003
112899241,9cyclictest0-21swapper/320:56:443
112899241,9cyclictest0-21swapper/320:07:083
112899241,9cyclictest0-21swapper/319:31:163
112899241,7cyclictest0-21swapper/300:20:393
112899241,12cyclictest12715-21hddtemp_smartct22:00:153
112899241,11cyclictest23813-21if_err_enp1s022:25:173
112899241,10cyclictest0-21swapper/323:50:343
112899241,10cyclictest0-21swapper/319:47:283
112899240,8cyclictest0-21swapper/321:58:193
112899240,13cyclictest0-21swapper/323:30:283
112899240,13cyclictest0-21swapper/323:30:283
112899240,11cyclictest0-21swapper/323:13:583
112499249,3cyclictest0-21swapper/219:54:032
112499245,10cyclictest0-21swapper/220:10:212
112499244,12cyclictest89150irq/134-enp2s0-00:37:082
1124992415,2cyclictest87450irq/127-enp1s0-22:50:462
112499241,14cyclictest89150irq/134-enp2s0-21:21:482
112499241,13cyclictest0-21swapper/220:53:182
1124992411,2cyclictest87450irq/127-enp1s0-22:05:582
112499241,11cyclictest26828-21users22:30:262
112499240,9cyclictest0-21swapper/222:35:022
112499240,8cyclictest0-21swapper/222:03:022
112499240,11cyclictest0-21swapper/219:49:512
112499240,10cyclictest0-21swapper/221:37:192
112499240,10cyclictest0-21swapper/221:12:422
111799241,17cyclictest0-21swapper/120:07:191
111799241,15cyclictest0-21swapper/122:00:001
111799241,13cyclictest0-21swapper/123:19:161
111199241,12cyclictest6495-21munin-run00:15:000
112899231,9cyclictest0-21swapper/322:46:163
112899231,9cyclictest0-21swapper/319:55:243
112899231,10cyclictest0-21swapper/320:41:013
112899231,10cyclictest0-21swapper/320:14:053
112899230,9cyclictest0-21swapper/323:08:443
112899230,9cyclictest0-21swapper/321:28:163
112899230,12cyclictest0-21swapper/300:06:453
112899230,11cyclictest0-21swapper/300:30:083
112899230,10cyclictest0-21swapper/323:44:223
112899230,10cyclictest0-21swapper/320:35:153
112499238,3cyclictest0-21swapper/221:55:032
112499233,11cyclictest87350irq/126-enp1s0-23:23:422
112499231,9cyclictest0-21swapper/220:49:302
112499230,8cyclictest0-21swapper/221:40:032
112499230,12cyclictest0-21swapper/200:23:262
112499230,10cyclictest0-21swapper/219:22:142
112499230,10cyclictest0-21swapper/219:12:262
111799237,3cyclictest4250-21wc19:15:121
111799235,10cyclictest22202-21diskmemload21:11:191
111799231,5cyclictest23045-21/usr/sbin/munin22:25:231
111799231,4cyclictest981-21memory00:00:201
111799231,16cyclictest0-21swapper/123:30:471
111799231,16cyclictest0-21swapper/123:30:461
111799231,16cyclictest0-21swapper/120:53:361
111799231,12cyclictest13406-21perf20:50:001
111199235,15cyclictest25995-21awk20:05:010
111199230,5cyclictest18256-21kworker/u8:023:11:030
112899221,9cyclictest0-21swapper/322:10:093
112899221,9cyclictest0-21swapper/321:32:363
112899221,11cyclictest14467-21sshd23:17:533
112899221,11cyclictest0-21swapper/323:01:323
112899221,10cyclictest0-21swapper/320:01:203
112899220,8cyclictest0-21swapper/321:45:523
112899220,8cyclictest0-21swapper/321:41:543
112499222,3cyclictest0-21swapper/219:39:192
1124992210,3cyclictest0-21swapper/222:46:142
112499220,9cyclictest0-21swapper/220:27:542
111799223,11cyclictest87550irq/128-enp1s0-22:45:541
1117992218,3cyclictest0-21swapper/123:13:521
111799221,4cyclictest15122-21kworker/u8:100:38:251
111799221,4cyclictest0-21swapper/122:53:071
1117992214,2cyclictest0-21swapper/100:07:211
111799221,16cyclictest0-21swapper/123:53:231
111799221,16cyclictest0-21swapper/121:04:231
111799221,11cyclictest0-21swapper/123:28:191
111799221,11cyclictest0-21swapper/123:20:141
111799221,11cyclictest0-21swapper/121:36:461
111799221,11cyclictest0-21swapper/100:33:471
111799221,11cyclictest0-21swapper/100:28:201
111799221,11cyclictest0-21swapper/100:24:581
111799221,10cyclictest0-21swapper/121:45:311
112899216,3cyclictest0-21swapper/323:46:003
112899211,8cyclictest0-21swapper/321:05:003
112899210,8cyclictest0-21swapper/321:52:483
112899210,10cyclictest0-21swapper/321:17:523
112499216,2cyclictest0-21swapper/200:04:322
112499213,3cyclictest0-21swapper/219:19:062
112499213,11cyclictest89150irq/134-enp2s0-23:53:532
112499212,13cyclictest87650irq/129-enp1s0-00:31:382
1124992113,2cyclictest87450irq/127-enp1s0-23:10:022
112499210,8cyclictest0-21swapper/223:33:152
112499210,8cyclictest0-21swapper/223:33:152
111799216,9cyclictest0-21swapper/122:37:221
111799212,9cyclictest21874-21tune2fs23:35:131
1117992116,3cyclictest10300-21ntp_states20:40:211
1117992114,5cyclictest2406-21systemd-journal19:10:221
1117992113,3cyclictest0-21swapper/122:14:411
1117992113,3cyclictest0-21swapper/119:28:271
111799211,2cyclictest22202-21diskmemload23:56:241
111799211,15cyclictest0-21swapper/121:34:171
111799211,15cyclictest0-21swapper/120:32:031
111799211,15cyclictest0-21swapper/120:29:381
111799211,15cyclictest0-21swapper/120:24:541
111799211,15cyclictest0-21swapper/119:54:531
111799211,15cyclictest0-21swapper/100:13:131
111799211,14cyclictest0-21swapper/121:40:071
111799211,11cyclictest8513-21cat19:25:021
111799211,11cyclictest0-21swapper/123:09:131
111799211,10cyclictest0-21swapper/121:19:151
1117992110,3cyclictest4788-21memory22:55:181
111799210,6cyclictest0-21swapper/122:44:231
111799210,5cyclictest0-21swapper/122:30:071
111799210,1cyclictest0-21swapper/119:30:131
111799210,15cyclictest0-21swapper/123:46:391
111199211,7cyclictest0-21swapper/021:18:550
111199211,5cyclictest0-21swapper/023:24:470
1111992114,4cyclictest19753-21tr19:50:120
111199211,11cyclictest12673-21/usr/sbin/munin23:15:240
1111992110,3cyclictest4247-21strings21:40:190
112899201,7cyclictest0-21swapper/323:23:363
112499207,2cyclictest0-21swapper/223:59:152
112499206,4cyclictest26056-21taskset21:15:252
112499204,6cyclictest0-21swapper/220:20:552
112499202,11cyclictest87450irq/127-enp1s0-00:26:062
1124992011,2cyclictest87450irq/127-enp1s0-23:15:382
111799208,6cyclictest13572-21kworker/u8:120:15:271
111799201,7cyclictest1-21systemd21:20:211
1117992013,2cyclictest0-21swapper/120:56:261
1117992012,3cyclictest0-21swapper/121:50:171
1117992012,3cyclictest0-21swapper/119:37:411
111799201,14cyclictest0-21swapper/122:16:531
111799201,14cyclictest0-21swapper/122:16:531
111799201,14cyclictest0-21swapper/121:28:511
111199201,8cyclictest808-21snmpd22:21:280
111199201,11cyclictest8278-21/usr/sbin/munin23:05:140
111199201,11cyclictest18652-21/usr/sbin/munin22:15:150
111199201,11cyclictest18652-21/usr/sbin/munin22:15:150
111199201,10cyclictest23658-21/usr/sbin/munin23:40:200
292372191,2sleep00-21swapper/022:38:000
16702190,2sleep00-21swapper/022:50:080
112899195,3cyclictest0-21swapper/323:25:183
112899195,3cyclictest0-21swapper/320:45:353
111799198,3cyclictest20851-21acpi22:20:111
111799198,3cyclictest0-21swapper/122:07:041
111799196,4cyclictest19358-21sshd19:49:541
111799195,4cyclictest0-21swapper/123:42:581
1117991916,2cyclictest0-21swapper/123:04:141
111799191,2cyclictest22202-21diskmemload22:02:591
111799191,16cyclictest0-21swapper/120:00:071
111799191,14cyclictest0-21swapper/120:37:291
111799191,10cyclictest21815-21users21:05:251
111799191,10cyclictest21815-21users21:05:251
111799190,2cyclictest0-21swapper/120:12:021
111199198,3cyclictest32240-21sshd20:15:530
111199196,4cyclictest9066-1kworker/u9:000:15:070
111199194,5cyclictest11860-21sa119:30:200
111199191,6cyclictest0-21swapper/023:37:400
1111991912,4cyclictest26328-21tr20:05:120
1111991912,4cyclictest15919-21grep20:55:120
111199191,15cyclictest9689-21ntp_states19:25:210
1111991911,2cyclictest87550irq/128-enp1s0-23:28:310
111199191,10cyclictest7033-21/usr/sbin/munin20:35:130
19212180,2sleep00-21swapper/000:03:030
111799181,16cyclictest0-21swapper/119:42:001
111199188,2cyclictest13827-21taskset19:35:150
111199187,3cyclictest28404-21/usr/sbin/munin20:10:190
111199181,5cyclictest87350irq/126-enp1s0-21:36:570
111199181,5cyclictest0-21swapper/022:57:130
111199181,5cyclictest0-21swapper/022:05:080
111199181,11cyclictest0-21swapper/023:45:150
111199181,11cyclictest0-21swapper/023:34:210
111199181,11cyclictest0-21swapper/023:34:210
106122181,2sleep00-21swapper/000:21:420
246922171,2sleep00-21swapper/022:26:000
111199177,7cyclictest10484-21gltestperf21:55:150
111199176,3cyclictest13630-21/usr/sbin/munin20:50:170
111199174,3cyclictest1413-21grep20:20:190
111199173,11cyclictest31972-21tr22:45:140
111199171,7cyclictest0-21swapper/000:35:190
111199171,6cyclictest11488-21fschecks_count00:25:160
111199171,6cyclictest0-21swapper/021:10:080
111199171,4cyclictest0-21swapper/021:49:330
1111991711,3cyclictest6813-21idleruntime-cro20:35:010
111199171,13cyclictest21359-21sed21:05:220
111199171,13cyclictest21359-21sed21:05:220
111199170,11cyclictest0-21swapper/022:00:180
111199165,3cyclictest4226-21/usr/sbin/munin19:15:180
111199161,6cyclictest22939-21pmu-power19:55:220
111199161,13cyclictest30254-21grep22:40:170
1111991611,3cyclictest2719-21cat20:25:120
111199161,13cyclictest0-21swapper/019:45:190
1111991610,3cyclictest15209-21/usr/sbin/munin19:40:140
1111991610,3cyclictest11427-21/usr/sbin/munin20:45:170
111199151,5cyclictest29081-21grep21:25:110
111199151,11cyclictest7652-21/usr/sbin/munin21:50:110
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional