You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-22 - 01:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu May 21, 2026 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
199722228157,59sleep00-21swapper/007:07:360
200862204168,24sleep20-21swapper/207:09:072
199002203167,24sleep10-21swapper/107:06:411
199852201165,24sleep30-21swapper/307:07:473
2112521400,4sleep12024899cyclictest12:04:271
130921250,4sleep22025599cyclictest07:35:222
20255991111,108cyclictest0-21swapper/208:10:452
20255991091,106cyclictest0-21swapper/212:38:552
20255998077,2cyclictest0-21swapper/211:39:482
309272530,1sleep00-21swapper/007:30:200
267262530,4sleep339-21rcuc/312:15:193
201042520,2sleep10-21swapper/110:45:251
20242994111,14cyclictest13954-21munin-run10:35:010
20242993812,12cyclictest25313-21munin-run07:20:000
20242993810,13cyclictest1745-21idleruntime-cro07:40:020
2026099361,23cyclictest0-21swapper/309:25:023
2026099360,25cyclictest0-21swapper/310:10:193
273842350,4sleep232-21ksoftirqd/208:35:232
2024299355,14cyclictest10177-21munin-run11:40:010
2026099341,24cyclictest0-21swapper/308:56:233
2026099341,24cyclictest0-21swapper/308:43:113
2026099340,23cyclictest0-21swapper/310:47:563
20260993313,7cyclictest8943-21diskmemload11:08:113
2026099331,24cyclictest0-21swapper/308:00:063
2026099331,23cyclictest0-21swapper/309:03:433
2026099330,26cyclictest0-21swapper/308:24:153
2026099330,26cyclictest0-21swapper/308:24:153
2024299335,13cyclictest12372-21munin-run11:45:010
2026099321,23cyclictest0-21swapper/308:11:553
2024299321,15cyclictest6748-21munin-run09:05:020
2026099311,2cyclictest0-21swapper/310:42:443
2026099311,21cyclictest0-21swapper/308:35:033
2026099310,24cyclictest0-21swapper/311:46:243
2026099310,24cyclictest0-21swapper/308:25:433
2024299316,11cyclictest25931-21idleruntime-cro08:35:000
2024299314,12cyclictest16762-21idleruntime-cro11:55:010
2024299314,12cyclictest16762-21idleruntime-cro11:55:010
216702300,3sleep22025599cyclictest12:05:122
2026099301,21cyclictest8943-21diskmemload09:57:153
2026099301,18cyclictest0-21swapper/307:17:593
2026099300,23cyclictest0-21swapper/308:47:273
2026099300,22cyclictest0-21swapper/312:05:153
2024299306,11cyclictest23732-21idleruntime-cro08:30:000
2026099291,19cyclictest0-21swapper/307:42:503
2026099291,17cyclictest0-21swapper/312:25:523
2026099291,13cyclictest0-21swapper/310:53:353
2026099291,11cyclictest0-21swapper/308:06:063
2026099290,22cyclictest0-21swapper/309:08:153
2026099290,21cyclictest0-21swapper/311:52:563
2026099290,21cyclictest0-21swapper/311:52:553
2026099290,13cyclictest0-21swapper/311:02:243
2026099290,13cyclictest0-21swapper/310:56:523
2026099290,13cyclictest0-21swapper/307:57:103
2024899291,21cyclictest0-21swapper/109:45:201
2026099281,20cyclictest0-21swapper/309:23:073
2026099281,18cyclictest0-21swapper/307:30:353
2026099281,12cyclictest0-21swapper/307:47:143
2026099280,2cyclictest0-21swapper/310:16:353
2026099280,12cyclictest0-21swapper/310:09:553
2026099280,11cyclictest0-21swapper/307:52:143
2026099277,3cyclictest0-21swapper/309:35:103
2026099271,10cyclictest0-21swapper/311:14:523
2026099270,16cyclictest0-21swapper/309:17:503
2026099270,13cyclictest0-21swapper/309:13:343
2026099270,12cyclictest0-21swapper/312:30:523
2026099270,12cyclictest0-21swapper/310:37:553
2026099270,11cyclictest0-21swapper/311:41:363
2026099270,11cyclictest0-21swapper/308:51:193
20248992711,4cyclictest0-21swapper/110:20:151
2024899270,22cyclictest0-21swapper/111:51:301
2024899270,22cyclictest0-21swapper/111:51:301
2026099264,7cyclictest25584-21awk12:15:013
2026099261,9cyclictest0-21swapper/310:00:433
2026099261,8cyclictest0-21swapper/309:43:583
2026099261,11cyclictest0-21swapper/311:30:043
2026099261,11cyclictest0-21swapper/309:46:273
2026099261,10cyclictest0-21swapper/309:31:073
2026099261,10cyclictest0-21swapper/308:18:193
2026099260,12cyclictest0-21swapper/312:39:123
2026099260,12cyclictest0-21swapper/312:02:553
2026099260,12cyclictest0-21swapper/310:27:023
2026099260,10cyclictest0-21swapper/310:21:423
2024899262,4cyclictest31240-21taskset12:25:201
2024899262,12cyclictest0-21swapper/110:25:231
20248992617,2cyclictest26454-21kworker/u8:111:15:161
2024899261,18cyclictest0-21swapper/111:25:461
2024899261,18cyclictest0-21swapper/109:52:221
20248992610,3cyclictest32065-21grep08:45:261
316882250,2sleep00-21swapper/012:25:240
2026099251,13cyclictest0-21swapper/307:14:073
2026099250,11cyclictest0-21swapper/312:24:553
2026099250,11cyclictest0-21swapper/311:38:003
2026099250,10cyclictest0-21swapper/311:59:353
2026099250,10cyclictest0-21swapper/309:51:153
2024899253,11cyclictest0-21swapper/110:00:311
2024899252,15cyclictest0-21swapper/112:16:261
2024899251,4cyclictest28128-21runrttasks08:40:001
2024899251,17cyclictest0-21swapper/112:20:471
2024899251,17cyclictest0-21swapper/109:05:221
2024899251,16cyclictest0-21swapper/111:35:241
141082250,5sleep014112-21unixbench_multi11:45:270
2026099241,9cyclictest0-21swapper/311:25:483
2026099241,8cyclictest0-21swapper/310:30:383
2026099241,11cyclictest0-21swapper/311:16:493
2024899241,17cyclictest87550irq/128-enp1s0-11:45:051
2024899241,16cyclictest0-21swapper/111:32:331
2024899241,16cyclictest0-21swapper/110:50:261
2024899241,12cyclictest0-21swapper/110:07:371
2024899240,16cyclictest0-21swapper/110:35:261
2024299240,11cyclictest27516-21basename07:25:010
14442240,3sleep21448-21fschecks_time10:05:162
2026099231,7cyclictest0-21swapper/307:23:293
2026099231,10cyclictest0-21swapper/308:33:213
2026099230,9cyclictest0-21swapper/307:25:113
2026099230,11cyclictest0-21swapper/307:37:083
2025599238,7cyclictest87350irq/126-enp1s0-10:50:082
2025599236,2cyclictest0-21swapper/208:30:592
2024899239,9cyclictest0-21swapper/107:44:191
2024899237,9cyclictest0-21swapper/107:35:121
2024899231,9cyclictest8943-21diskmemload10:10:261
2024899231,9cyclictest0-21swapper/111:43:441
20248992319,3cyclictest0-21swapper/112:32:561
2024899231,19cyclictest827-21runrttasks08:45:001
2024899231,16cyclictest0-21swapper/110:42:271
2024899231,16cyclictest0-21swapper/109:16:451
2024899231,16cyclictest0-21swapper/107:15:211
2024899231,14cyclictest0-21swapper/109:33:131
2024899231,11cyclictest22975-21gltestperf09:40:141
2024899231,10cyclictest29074-21grep09:55:111
2024899230,6cyclictest0-21swapper/111:12:561
2024899230,15cyclictest0-21swapper/112:05:191
2024899230,15cyclictest0-21swapper/111:08:571
2024899230,15cyclictest0-21swapper/111:01:031
2024899230,15cyclictest0-21swapper/110:31:181
2024899230,15cyclictest0-21swapper/110:15:041
20242992310,4cyclictest0-21swapper/009:55:140
2024299230,11cyclictest6140-21perf07:50:020
2025599226,8cyclictest12050irq/123-ahci[0009:05:012
2025599222,8cyclictest32394-21/usr/sbin/munin12:30:152
2025599221,15cyclictest0-21swapper/211:50:292
2025599221,15cyclictest0-21swapper/211:50:292
2024899221,3cyclictest17590-21expr08:15:141
2024899221,18cyclictest9972-21sed07:55:261
2024899221,16cyclictest0-21swapper/108:00:261
2024899221,15cyclictest0-21swapper/107:50:171
2024899221,11cyclictest0-21swapper/108:52:501
2024899221,10cyclictest25206-21sensors_temp08:30:221
2024899220,6cyclictest0-21swapper/111:23:531
2024899220,5cyclictest0-21swapper/109:20:041
2024899220,16cyclictest0-21swapper/110:55:191
2024899220,16cyclictest0-21swapper/108:25:511
2024899220,15cyclictest0-21swapper/109:37:021
2024299222,8cyclictest21122-21df_inode10:50:130
2026099210,9cyclictest0-21swapper/311:23:363
2025599216,4cyclictest0-21swapper/212:13:352
2025599211,7cyclictest22664-21memory08:25:192
2025599211,6cyclictest0-21swapper/210:13:482
2025599211,6cyclictest0-21swapper/207:30:252
2025599211,5cyclictest0-21swapper/211:02:532
20255992115,4cyclictest14821-21mailstats09:20:232
2025599211,13cyclictest0-21swapper/208:15:182
2025599210,6cyclictest0-21swapper/212:15:262
2024899214,4cyclictest8943-21diskmemload09:10:241
2024899212,11cyclictest3352-21if_err_enp2s008:55:151
20248992118,2cyclictest0-21swapper/108:08:121
20248992118,2cyclictest0-21swapper/107:30:031
20248992118,2cyclictest0-21swapper/107:26:341
20248992118,2cyclictest0-21swapper/107:22:131
20248992116,2cyclictest25735-21cut12:15:021
2024899211,18cyclictest0-21swapper/108:20:201
2024899211,18cyclictest0-21swapper/108:20:191
2024899211,18cyclictest0-21swapper/107:10:231
2024899211,15cyclictest0-21swapper/112:39:311
2024899210,19cyclictest0-21swapper/109:25:361
2024899210,14cyclictest0-21swapper/111:55:171
2024299214,15cyclictest7994-21idleruntime-cro11:35:010
2024299212,8cyclictest4375-21kworker/0:109:15:250
2024299212,11cyclictest19544-21/usr/sbin/munin08:20:220
2024299212,11cyclictest19544-21/usr/sbin/munin08:20:220
20242992115,3cyclictest17256-21grep09:25:250
2024299211,12cyclictest14179-21/usr/sbin/munin10:35:220
40572201,3sleep00-21swapper/012:38:210
2025599203,5cyclictest909-21sssd_nss07:15:172
2025599203,5cyclictest683-21/usr/sbin/munin10:05:022
2025599201,9cyclictest0-21swapper/211:15:222
2025599201,8cyclictest32-21ksoftirqd/210:40:182
2025599201,7cyclictest7679-21idleruntime09:05:162
2025599201,7cyclictest13271-21cat10:30:252
2025599201,7cyclictest0-21swapper/211:09:502
2025599201,6cyclictest0-21swapper/210:15:412
2025599200,7cyclictest20766-21gltestperf09:35:162
2024899201,7cyclictest4767-21/usr/sbin/munin09:00:241
2024899200,18cyclictest0-21swapper/108:12:461
2024299204,14cyclictest0-21swapper/011:20:020
2024299201,10cyclictest0-21swapper/009:40:240
2025599194,3cyclictest22375-21cpu09:40:102
2025599193,4cyclictest9552-21latency_hist10:25:022
2025599193,3cyclictest87350irq/126-enp1s0-12:00:252
2025599192,6cyclictest1604-21systemd11:20:082
2025599192,5cyclictest17955-21sshd09:30:082
2025599192,16cyclictest0-21swapper/207:48:152
2025599191,7cyclictest12600-21/usr/sbin/munin11:45:142
2025599191,7cyclictest0-21swapper/212:25:052
2025599191,7cyclictest0-21swapper/211:55:302
2025599191,6cyclictest19544-21/usr/sbin/munin08:20:132
2025599191,6cyclictest19544-21/usr/sbin/munin08:20:132
2025599191,6cyclictest14179-21/usr/sbin/munin10:35:202
2025599191,5cyclictest579-21systemd-journal08:05:012
2025599191,5cyclictest0-21swapper/209:55:152
2025599191,5cyclictest0-21swapper/208:05:252
2025599191,2cyclictest0-21swapper/209:10:252
2025599191,10cyclictest0-21swapper/211:31:522
2025599191,10cyclictest0-21swapper/211:14:452
2024899190,14cyclictest0-21swapper/107:46:111
2024299198,3cyclictest27381-21/usr/sbin/munin11:05:210
2024299196,4cyclictest10319-21pmu-power09:10:230
2024299191,9cyclictest18344-21needreboot08:15:200
2024299191,5cyclictest0-21swapper/009:31:080
20242991912,4cyclictest22027-21grep12:05:150
2024299191,11cyclictest5392-21/usr/sbin/munin10:15:260
2024299191,10cyclictest9204-21sshd10:20:350
299782180,2sleep00-21swapper/008:40:560
2025599185,3cyclictest87350irq/126-enp1s0-07:25:132
2025599184,3cyclictest87350irq/126-enp1s0-11:25:202
2025599182,15cyclictest0-21swapper/208:44:362
2025599181,7cyclictest0-21swapper/210:25:172
2025599181,5cyclictest808-21snmpd10:58:102
2025599181,5cyclictest2767-21irqstats07:40:172
2025599181,5cyclictest27478-21basename09:50:182
2025599181,5cyclictest26629-21latency_hist09:50:022
2025599181,5cyclictest2568-21/usr/sbin/munin08:55:102
2025599181,5cyclictest0-21swapper/211:40:092
2025599181,3cyclictest87350irq/126-enp1s0-12:24:162
20255991813,3cyclictest22438-21service07:15:012
20255991813,3cyclictest19641-21ntpq10:45:212
20255991812,3cyclictest1347-21cut08:50:202
20255991811,4cyclictest15889-21grep09:25:102
2024299189,2cyclictest18185-21sshd10:43:070
2024299188,3cyclictest25173-21/usr/sbin/munin11:00:160
2024299185,4cyclictest26567-21if_enp2s012:15:180
2024299181,7cyclictest30571-21head11:10:190
20242991813,3cyclictest13304-21diskstats08:05:130
20242991812,4cyclictest27093-21df09:50:150
2024299181,15cyclictest0-21swapper/008:57:440
2024299180,8cyclictest19291-21grep12:00:120
37142170,2sleep00-21swapper/007:42:290
264032171,2sleep00-21swapper/009:48:340
2025599173,6cyclictest12096-21if_enp1s009:15:172
2025599171,5cyclictest0-21swapper/208:45:212
2025599171,3cyclictest87350irq/126-enp1s0-07:20:252
20255991712,3cyclictest7942-21users07:50:262
2024299177,3cyclictest87550irq/128-enp1s0-11:22:360
2024299173,8cyclictest0-21swapper/012:13:310
2024299173,2cyclictest87350irq/126-enp1s0-10:10:240
2024299172,4cyclictest0-21swapper/012:30:180
2024299171,6cyclictest0-21swapper/011:56:350
2024299171,3cyclictest28003-21/usr/sbin/munin12:20:130
20242991713,3cyclictest0-21swapper/010:48:210
20242991712,3cyclictest20447-21acpi07:10:110
2024299171,14cyclictest8943-21diskmemload11:27:330
2024299171,14cyclictest0-21swapper/008:50:120
20242991711,3cyclictest7124-21if_err_enp1s007:50:170
2024299171,13cyclictest10104-21grep07:55:260
2025599161,4cyclictest8551-21/usr/sbin/munin07:55:272
2024299164,3cyclictest11098-21diskstats08:00:140
2024299161,7cyclictest8943-21diskmemload10:25:550
2024299161,7cyclictest0-21swapper/010:05:100
2024299161,6cyclictest0-21swapper/007:25:240
20242991612,3cyclictest0-21swapper/009:05:170
20242991612,3cyclictest0-21swapper/008:10:060
2024299161,14cyclictest0-21swapper/008:38:030
20242991611,3cyclictest30536-21/usr/sbin/munin08:45:200
20242991611,3cyclictest22973-21/usr/sbin/munin10:55:190
2024299161,12cyclictest20154-21/usr/sbin/munin09:35:120
20242991610,5cyclictest7-21ksoftirqd/009:20:170
20242991510,3cyclictest31234-21cpuspeed_turbos10:00:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional