You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-09 - 23:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Apr 09, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
253192239164,24sleep10-21swapper/107:06:311
254602233162,58sleep30-21swapper/307:08:233
252672203165,25sleep00-21swapper/007:05:510
253102201164,24sleep20-21swapper/207:06:252
1288721710,4sleep32568699cyclictest10:20:023
1288721710,4sleep32568699cyclictest10:20:023
1028421310,3sleep12567599cyclictest07:45:161
25681999693,2cyclictest0-21swapper/209:17:222
23952660,1sleep10-21swapper/108:40:191
25681995343,9cyclictest0-21swapper/211:06:022
2568699388,4cyclictest0-21swapper/311:30:143
2568699388,4cyclictest0-21swapper/311:30:143
2568699383,6cyclictest20624-40dnf09:20:453
2568699381,11cyclictest26214-21/usr/sbin/munin10:50:243
2566999379,13cyclictest20367-21awk08:10:010
25686993612,3cyclictest4227-21/usr/sbin/munin10:00:263
2568699361,11cyclictest29039-21/usr/sbin/munin12:10:243
2568699361,11cyclictest2659-21/usr/sbin/munin11:10:223
2567599366,18cyclictest0-21swapper/109:40:591
2567599362,22cyclictest0-21swapper/112:21:521
2568699345,3cyclictest87350irq/126-enp1s0-07:40:143
25686993412,3cyclictest29375-21/usr/sbin/munin08:30:223
25686993411,3cyclictest13642-21/usr/sbin/munin11:35:183
2568699340,9cyclictest8385-21kworker/u8:210:10:243
2568699332,9cyclictest7247-21irqstats10:05:183
25686993314,16cyclictest26293-21fschecks_count07:10:143
2568699331,10cyclictest15363-21kworker/u8:111:50:263
2566999338,12cyclictest8423-21idleruntime-cro10:10:010
2568699329,3cyclictest15458-21grep07:55:323
2568699329,3cyclictest0-21swapper/309:45:233
2568699326,6cyclictest0-21swapper/311:26:073
2568699326,3cyclictest87350irq/126-enp1s0-10:45:163
2568699325,3cyclictest1391-21/usr/sbin/munin08:40:213
2568699324,8cyclictest808-21snmpd12:21:533
2568699321,3cyclictest19388-21latency_hist10:35:023
25686993212,2cyclictest7818-21if_err_enp1s011:20:153
25686993210,3cyclictest26535-21sed08:20:253
2568699319,3cyclictest23147-21grep08:15:153
2568699319,2cyclictest827-21runrttasks08:11:273
2568699318,4cyclictest3761-21cpuspeed_turbos08:45:113
2568699318,3cyclictest25598-21/usr/sbin/munin09:35:223
2568699312,5cyclictest29998-21users10:55:233
2568699311,7cyclictest491-21sshd12:16:513
25686993112,3cyclictest29793-21nfs4_client07:15:183
25686993111,3cyclictest6529-21ntp_offset12:30:213
2568699311,12cyclictest5195-21/usr/sbin/munin07:35:213
2568699311,11cyclictest31647-21kworker/u8:010:20:113
2568699311,11cyclictest18038-21/usr/sbin/munin11:45:143
25686993110,3cyclictest827-21runrttasks09:53:213
25686993110,3cyclictest20545-21cat08:10:023
2568699309,3cyclictest827-21runrttasks10:44:093
2568699309,3cyclictest27796-21/usr/sbin/munin09:40:223
2568699309,3cyclictest19888-21expr10:35:133
2568699301,4cyclictest18614-21seq09:18:053
25686993010,3cyclictest15448-21cat10:25:133
2568199309,3cyclictest0-21swapper/207:56:372
2566999306,12cyclictest22564-21awk08:15:020
2568699299,3cyclictest25971-21threads12:00:233
2568699298,4cyclictest15837-21sendmail_mailqu09:10:233
2568699294,4cyclictest15363-21kworker/u8:112:05:143
2568699293,3cyclictest0-21swapper/311:57:243
2568699292,10cyclictest18156-21latency_hist08:05:023
2568699291,4cyclictest810-21sshd12:27:503
2568699291,11cyclictest2624-21head09:55:153
25686992910,3cyclictest931-21expr07:25:113
25686992910,3cyclictest7672-21/usr/sbin/munin12:35:123
25686992910,3cyclictest5921-21ntpq11:15:203
25686992910,3cyclictest32276-21basename08:35:153
2567599292,20cyclictest0-21swapper/110:39:421
2567599292,20cyclictest0-21swapper/110:05:301
2567599291,22cyclictest0-21swapper/108:19:431
56672280,4sleep011-21rcuc/012:30:130
2568699289,2cyclictest4443-21sed07:30:243
2568699288,3cyclictest827-21runrttasks09:26:243
2568699288,3cyclictest11878-21cpuspeed_turbos07:50:123
2568699288,3cyclictest11878-21cpuspeed_turbos07:50:123
2568699287,4cyclictest31425-21tune2fs07:20:143
2568699285,4cyclictest790-21in:imjournal08:50:493
2568699281,14cyclictest0-21swapper/307:45:103
2568699281,10cyclictest32076-21/usr/sbin/munin11:00:253
25675992824,3cyclictest0-21swapper/112:31:141
2567599281,21cyclictest0-21swapper/111:33:161
2567599281,21cyclictest0-21swapper/111:33:161
2567599281,21cyclictest0-21swapper/108:57:491
2567599281,21cyclictest0-21swapper/108:39:311
2567599281,21cyclictest0-21swapper/108:12:021
2567599281,21cyclictest0-21swapper/107:13:341
2567599281,20cyclictest0-21swapper/112:17:551
25686992711,3cyclictest12547-21cpuspeed_turbos09:05:123
25686992710,3cyclictest23482-21cpuspeed_turbos09:30:113
2568699270,2cyclictest87350irq/126-enp1s0-11:40:143
2568699270,2cyclictest0-21swapper/309:03:503
25675992723,3cyclictest0-21swapper/111:54:251
2567599272,11cyclictest2286-21expr09:55:111
2567599271,22cyclictest23500-21tr09:30:111
2567599271,20cyclictest0-21swapper/110:15:261
2567599271,20cyclictest0-21swapper/110:15:261
2567599271,20cyclictest0-21swapper/108:49:451
2567599270,21cyclictest0-21swapper/110:27:551
2568699262,9cyclictest1597-21pmu-power11:05:223
25686992610,3cyclictest0-21swapper/308:25:143
25681992623,2cyclictest0-21swapper/209:45:322
25675992622,3cyclictest0-21swapper/111:40:111
2567599262,22cyclictest0-21swapper/108:20:161
2567599261,21cyclictest23891-21/usr/sbin/munin11:55:241
2567599261,17cyclictest23432-21chrt10:40:381
2567599260,20cyclictest0-21swapper/110:21:251
2566999264,10cyclictest1822-21idleruntime-cro09:55:010
314132250,4sleep230-21rcuc/207:20:142
25675992522,2cyclictest0-21swapper/112:38:301
25675992522,2cyclictest0-21swapper/109:48:381
2567599251,21cyclictest5922-21ntpq11:15:211
2567599251,19cyclictest0-21swapper/110:32:341
2567599251,19cyclictest0-21swapper/109:53:171
2567599251,19cyclictest0-21swapper/109:10:271
2567599251,17cyclictest0-21swapper/110:49:021
25675992511,3cyclictest27758-21memory12:05:161
2567599250,20cyclictest0-21swapper/107:54:291
2567599250,20cyclictest0-21swapper/107:54:291
323902240,4sleep230-21rcuc/212:15:222
2567599249,3cyclictest29835-21ntp_states07:15:211
2567599249,2cyclictest8732-40dnf11:21:231
2567599248,3cyclictest14379-21diskmemload11:36:361
2567599242,20cyclictest0-21swapper/107:40:151
2567599241,3cyclictest11232-21awk11:30:011
25675992412,7cyclictest0-21swapper/108:51:231
2567599241,21cyclictest0-21swapper/108:29:511
2567599241,21cyclictest0-21swapper/107:59:151
2567599241,21cyclictest0-21swapper/107:20:211
2567599241,20cyclictest4541-21smart_sda12:25:241
2567599241,12cyclictest0-21swapper/110:50:131
2567599241,11cyclictest18317-21cat08:05:011
2567599240,22cyclictest0-21swapper/109:10:021
2566999241,12cyclictest31971-21munin-run09:50:000
2568199235,9cyclictest15964-21cat09:10:252
2568199235,4cyclictest579-21systemd-journal12:05:022
2568199234,10cyclictest22460-21smart_sda09:25:232
2567599238,3cyclictest30101-21ntp_states12:10:191
2567599231,20cyclictest0-21swapper/111:02:301
2567599231,20cyclictest0-21swapper/110:59:151
2567599231,20cyclictest0-21swapper/109:24:331
2567599231,20cyclictest0-21swapper/107:29:311
2567599231,19cyclictest11384-21processes09:00:221
2567599231,14cyclictest0-21swapper/110:10:311
2567599231,11cyclictest1509-21ntp_states11:05:221
2567599230,21cyclictest0-21swapper/111:48:211
303012221,3sleep030434-21timerandwakeup12:10:240
2568699220,9cyclictest0-21swapper/308:56:093
2567599228,3cyclictest4684-21cat11:15:011
2567599227,4cyclictest0-21swapper/108:05:171
25675992214,6cyclictest4188-21processes07:30:211
25675992212,3cyclictest0-21swapper/110:01:501
25675992211,3cyclictest0-21swapper/108:33:411
2567599221,11cyclictest26741-21ls09:35:291
2567599221,11cyclictest0-21swapper/109:27:461
2567599221,10cyclictest25541-21missed_timers12:00:161
2567599220,20cyclictest0-21swapper/109:15:501
2566999227,4cyclictest0-21swapper/011:42:520
2566999223,12cyclictest2792-40dnf12:21:530
2566999222,11cyclictest6411-21head11:15:250
2568199215,8cyclictest87350irq/126-enp1s0-11:45:182
2568199215,5cyclictest755-21rngd07:31:022
2568199214,4cyclictest25595-21expr08:20:142
2568199214,3cyclictest87650irq/129-enp1s0-08:45:212
2568199211,7cyclictest26968-21cat09:35:322
2568199211,7cyclictest19972-21users08:05:262
2568199211,7cyclictest19762-21if_enp2s009:20:182
2568199211,7cyclictest0-21swapper/212:38:102
2568199211,7cyclictest0-21swapper/210:35:322
2568199211,11cyclictest3281-21/usr/sbin/munin12:25:152
2567599211,19cyclictest0-21swapper/107:38:321
2566999212,11cyclictest24971-21/usr/sbin/munin08:20:210
2566999211,13cyclictest0-21swapper/009:29:090
2568199205,8cyclictest87350irq/126-enp1s0-08:35:112
2568199202,6cyclictest28811-21latency_hist12:10:022
2568199201,7cyclictest0-21swapper/212:23:302
2568199201,6cyclictest22710-21cat08:15:012
2568199201,6cyclictest0-21swapper/210:55:342
2568199201,11cyclictest14538-21sed10:20:232
2568199201,10cyclictest9589-21/usr/sbin/munin07:45:162
2568199201,10cyclictest28351-21ps08:25:232
2568199201,10cyclictest27026-21irqstats10:50:162
2568199201,10cyclictest24563-21sshd08:17:482
2568199201,10cyclictest11343-21awk09:00:212
2566999202,11cyclictest12410-21/usr/sbin/munin09:05:200
2566999202,10cyclictest22432-21/usr/sbin/munin11:55:230
2566999201,8cyclictest0-21swapper/007:35:120
2566999201,11cyclictest31345-21fschecks_count07:20:120
2566999201,11cyclictest27796-21/usr/sbin/munin09:40:140
2566999200,3cyclictest28073-21kworker/0:110:54:090
2568199195,3cyclictest30331-21cat12:10:232
2568199194,8cyclictest0-21swapper/209:41:052
2568199194,3cyclictest16564-21threads10:25:242
2568199193,4cyclictest17103-21sensors_temp11:40:252
2568199192,9cyclictest9252-21/usr/sbin/munin11:25:152
2568199192,2cyclictest87550irq/128-enp1s0-07:19:402
2568199192,10cyclictest0-21swapper/211:30:062
2568199192,10cyclictest0-21swapper/211:30:062
2568199191,7cyclictest0-21swapper/211:54:142
2568199191,7cyclictest0-21swapper/211:00:402
2568199191,6cyclictest7859-21irqstats11:20:162
2568199191,6cyclictest27364-21grep07:10:252
2568199191,6cyclictest23781-21latency_hist10:45:012
2568199191,6cyclictest18484-21ntp_states10:30:192
2568199191,6cyclictest0-21swapper/210:15:172
2568199191,6cyclictest0-21swapper/210:15:172
2568199191,5cyclictest5816-21users10:00:262
2568199191,5cyclictest11783-21/usr/sbin/munin07:50:112
2568199191,5cyclictest11783-21/usr/sbin/munin07:50:112
2568199191,5cyclictest0-21swapper/211:16:412
2568199191,5cyclictest0-21swapper/211:10:212
2568199191,14cyclictest1199-21grep07:25:132
2568199191,10cyclictest2032-21/usr/sbin/munin09:55:182
2568199190,10cyclictest1785-21fschecks_count08:40:142
2566999199,3cyclictest87550irq/128-enp1s0-12:37:000
2566999192,7cyclictest87350irq/126-enp1s0-07:25:100
2566999191,5cyclictest0-21swapper/011:48:460
25669991912,4cyclictest0-21swapper/011:35:110
2566999191,15cyclictest3526-21unixbench_multi09:55:240
2566999191,10cyclictest3002-21/usr/sbin/munin07:30:140
2566999191,10cyclictest1437-21cut11:05:200
2568199185,2cyclictest87350irq/126-enp1s0-10:45:112
2568199184,6cyclictest9558-21sed08:55:272
2568199184,3cyclictest87350irq/126-enp1s0-08:30:162
2568199181,6cyclictest23578-21awk11:55:222
2568199181,6cyclictest0-21swapper/209:05:132
2568199181,6cyclictest0-21swapper/207:36:422
2568199181,5cyclictest808-21snmpd08:02:402
2568199181,5cyclictest24368-21sed09:30:182
2568199181,5cyclictest0-21swapper/210:10:222
2568199181,4cyclictest0-21swapper/210:05:212
2568199181,3cyclictest87350irq/126-enp1s0-09:50:072
25681991812,3cyclictest8021-21gltestperf07:40:172
2566999188,3cyclictest87550irq/128-enp1s0-12:01:240
2566999187,3cyclictest17421-21/usr/sbin/munin10:30:190
2566999182,14cyclictest22221-21perf11:55:010
2566999181,7cyclictest0-21swapper/012:29:260
2566999181,6cyclictest0-21swapper/010:41:230
2566999181,5cyclictest0-21swapper/011:32:470
2566999181,5cyclictest0-21swapper/011:32:470
2566999181,4cyclictest0-21swapper/009:33:210
2566999181,3cyclictest27551-21sort12:05:150
25669991813,3cyclictest9547-21grep08:55:270
25669991812,5cyclictest21551-21kworker/u8:110:48:200
25669991812,4cyclictest17672-21unixbench_multi08:00:260
25669991812,3cyclictest8308-21sensors_temp11:20:200
25669991812,3cyclictest32062-21kernelversion12:15:180
2566999181,10cyclictest0-21swapper/010:20:180
2566999180,16cyclictest30398-21awk11:00:020
313352171,2sleep00-21swapper/008:35:010
2568199174,3cyclictest87350irq/126-enp1s0-08:50:252
25681991713,2cyclictest6531-21ntpq12:30:202
25681991712,3cyclictest14172-21grep11:35:152
2566999178,2cyclictest0-21swapper/011:25:110
2566999177,3cyclictest87550irq/128-enp1s0-09:21:020
2566999176,2cyclictest14992-21fschecks_count09:10:140
2566999172,12cyclictest12186-21grep09:05:010
2566999171,9cyclictest0-21swapper/010:16:220
2566999171,9cyclictest0-21swapper/010:16:220
2566999171,6cyclictest0-21swapper/009:37:400
2566999171,4cyclictest11783-21/usr/sbin/munin07:50:160
2566999171,4cyclictest11783-21/usr/sbin/munin07:50:160
2566999171,4cyclictest0-21swapper/010:25:100
25669991712,3cyclictest20687-21ntp_states10:35:220
25669991711,3cyclictest9862-21expr07:45:130
25669991711,3cyclictest30612-21/usr/sbin/munin11:00:160
25669991711,3cyclictest1574-21cat08:40:120
2566999171,13cyclictest14980-21strings07:55:260
2566999171,10cyclictest0-21swapper/007:15:250
245972171,2sleep00-21swapper/008:18:160
2566999165,3cyclictest3225-21fschecks_time11:10:160
2566999161,6cyclictest0-21swapper/007:10:270
2566999161,5cyclictest0-21swapper/009:18:350
25669991611,3cyclictest8631-21/usr/sbin/munin10:10:190
2566999161,10cyclictest0-21swapper/008:50:200
25669991610,3cyclictest3880-21expr08:45:120
2566999156,3cyclictest87350irq/126-enp1s0-10:00:180
2566999152,5cyclictest31572-21/usr/sbin/munin08:35:140
25669991510,3cyclictest7398-21acpi07:40:130
25669991510,3cyclictest27181-21/usr/sbin/munin08:25:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional