You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-08 - 18:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Wed Jul 08, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
321092241165,23sleep30-21swapper/307:08:053
321612238165,24sleep10-21swapper/107:08:441
321752235162,60sleep00-21swapper/007:08:550
319842232201,21sleep20-21swapper/207:06:272
718521770,4sleep03234399cyclictest09:50:150
445321460,4sleep13234999cyclictest07:15:261
195892520,5sleep019590-21unixbench-2d11:30:240
272202510,3sleep027222-21unixbench_singl09:20:250
32343993910,14cyclictest24488-21munin-run11:45:010
32343993810,13cyclictest18249-21awk07:50:000
3235599331,22cyclictest0-21swapper/208:38:172
3234399336,13cyclictest6282-21munin-run09:50:010
3235599321,11cyclictest0-21swapper/210:11:142
3234399329,11cyclictest29245-21basename08:15:010
3234399321,18cyclictest15951-21kworker/0:107:56:110
3235599311,11cyclictest5372-21ntp_states09:45:222
3235599310,12cyclictest0-21swapper/211:45:492
3234399301,22cyclictest32344-21kworker/0:310:48:160
3235599290,12cyclictest0-21swapper/207:40:332
3234399296,11cyclictest23272-21perf09:15:010
3234399292,18cyclictest15951-21kworker/0:012:36:390
3234399292,11cyclictest27964-21kworker/0:210:37:430
3234399291,13cyclictest2509-21munin-run10:55:020
3235599281,9cyclictest0-21swapper/207:24:222
3235599281,21cyclictest0-21swapper/211:32:452
3235599280,11cyclictest0-21swapper/209:25:222
3235599280,11cyclictest0-21swapper/209:24:452
3235599280,11cyclictest0-21swapper/208:44:452
3234399282,11cyclictest11005-21kworker/0:311:13:160
3234399281,20cyclictest14924-21kworker/0:010:16:030
3234399281,13cyclictest32055-21perf09:35:010
3234399281,11cyclictest10428-21kworker/0:210:03:190
32355992711,5cyclictest9046-21df_inode09:55:142
3235599271,11cyclictest7068-21unixbench_multi12:15:262
3235599271,11cyclictest0-21swapper/211:19:052
3235599271,11cyclictest0-21swapper/209:02:412
3235599271,11cyclictest0-21swapper/208:16:492
3235599271,11cyclictest0-21swapper/207:31:132
3235599271,10cyclictest0-21swapper/207:29:572
3235599270,13cyclictest0-21swapper/210:26:012
3235599270,13cyclictest0-21swapper/210:26:012
3234999271,21cyclictest0-21swapper/112:34:401
3234399279,8cyclictest16584-21kworker/0:008:59:470
3234399275,16cyclictest17875-21kworker/u8:011:48:350
3234399272,16cyclictest28677-21kworker/0:012:02:070
3234399272,10cyclictest2459-21kworker/0:111:06:360
3234399272,10cyclictest14924-21kworker/0:010:11:510
3234399271,10cyclictest9674-21kworker/0:212:34:150
3235599261,7cyclictest0-21swapper/208:07:292
3235599261,7cyclictest0-21swapper/208:07:292
32355992611,4cyclictest11239-21df10:00:142
3235599260,9cyclictest0-21swapper/212:11:422
3235599260,11cyclictest0-21swapper/208:27:292
3234399261,9cyclictest11005-21kworker/0:311:17:480
3234399261,21cyclictest27298-21kworker/0:009:27:310
3234399261,10cyclictest2459-21kworker/0:110:56:360
3234399261,10cyclictest11005-21kworker/0:311:26:390
58282250,3sleep121-21rcuc/107:20:151
3236099254,7cyclictest32648-21tr07:10:123
32360992511,3cyclictest18195-21ls09:00:253
3235599258,3cyclictest0-21swapper/207:14:192
3235599256,14cyclictest0-21swapper/211:44:552
3235599251,9cyclictest20159-21sh09:05:212
3235599251,8cyclictest0-21swapper/209:38:542
3235599251,15cyclictest0-21swapper/210:41:012
3235599251,12cyclictest20706-21netstat10:20:192
3235599251,11cyclictest0-21swapper/209:40:382
3235599250,9cyclictest0-21swapper/211:35:372
3235599250,10cyclictest0-21swapper/207:53:372
3234999251,3cyclictest32545-21/usr/sbin/munin07:10:171
3234399259,3cyclictest28951-21kworker/0:208:15:560
3234399252,8cyclictest12682-21kworker/0:110:08:590
3234399251,9cyclictest9674-21kworker/0:212:25:310
3234399251,9cyclictest29516-21kworker/0:109:38:230
3234399251,11cyclictest22943-21kworker/0:209:15:430
3234399251,11cyclictest22943-21kworker/0:209:15:430
3234399251,10cyclictest23505-21kworker/0:010:26:190
3234399251,10cyclictest23505-21kworker/0:010:26:190
79762240,15sleep00-21swapper/012:20:120
3236099241,8cyclictest0-21swapper/311:25:123
3236099240,2cyclictest87350irq/126-enp1s0-07:49:133
3235599248,2cyclictest0-21swapper/207:55:212
3235599245,3cyclictest0-21swapper/211:00:012
3235599241,9cyclictest0-21swapper/208:32:052
3235599241,6cyclictest0-21swapper/209:00:012
3235599241,11cyclictest0-21swapper/208:12:372
3235599241,11cyclictest0-21swapper/208:03:432
3235599241,10cyclictest0-21swapper/209:15:372
3235599241,10cyclictest0-21swapper/209:15:372
3235599240,9cyclictest0-21swapper/212:40:022
3235599240,10cyclictest0-21swapper/212:24:462
3235599240,10cyclictest0-21swapper/211:25:052
3235599240,10cyclictest0-21swapper/210:49:012
32349992410,4cyclictest16038-21munin-run07:45:001
32349992410,11cyclictest27402-21cpuspeed_turbos08:10:121
3234399247,2cyclictest2467-21kworker/0:209:40:470
3234399242,10cyclictest31318-21kworker/0:308:20:480
3234399241,9cyclictest6532-21kworker/0:011:03:520
3234399241,9cyclictest23505-21kworker/0:010:40:110
3234399241,12cyclictest16048-21idleruntime-cro07:45:000
3234399241,10cyclictest17099-21kworker/0:310:24:230
3236099233,17cyclictest29452-21df_inode11:55:143
3236099233,10cyclictest0-21swapper/310:10:193
3236099232,9cyclictest24095-21/usr/sbin/munin10:30:153
3236099231,13cyclictest0-21swapper/311:01:193
3236099230,14cyclictest87650irq/129-enp1s0-09:50:183
3235599234,5cyclictest0-21swapper/209:31:092
3235599231,9cyclictest0-21swapper/210:07:062
3235599231,9cyclictest0-21swapper/208:51:392
3235599231,7cyclictest0-21swapper/207:17:192
3235599231,11cyclictest0-21swapper/208:47:332
3235599231,11cyclictest0-21swapper/207:46:352
3235599230,9cyclictest0-21swapper/209:52:222
3235599230,10cyclictest0-21swapper/210:31:012
3234399232,9cyclictest718-21kworker/0:112:08:110
3234399232,8cyclictest3009-21kworker/0:012:19:590
3234399231,9cyclictest20410-21kworker/0:208:01:050
32343992311,3cyclictest23505-21kworker/0:010:33:430
3234399231,10cyclictest5973-21kworker/0:109:55:270
284202230,2sleep2311ktimersoftd/210:40:012
3236099225,8cyclictest6276-21latency_hist09:50:023
3236099225,3cyclictest21056-21diskmemload11:42:393
3236099224,8cyclictest23532-21smartctl08:00:163
3236099224,2cyclictest8667-21memory12:20:183
3236099223,4cyclictest87350irq/126-enp1s0-09:10:143
3236099222,12cyclictest23485-21/usr/sbin/munin09:15:183
3236099222,12cyclictest23485-21/usr/sbin/munin09:15:183
3236099221,12cyclictest0-21swapper/312:35:173
3235599227,3cyclictest0-21swapper/211:51:442
3235599221,10cyclictest0-21swapper/212:32:522
3235599221,10cyclictest0-21swapper/211:05:452
3235599221,10cyclictest0-21swapper/209:12:012
3235599221,10cyclictest0-21swapper/207:36:392
3235599220,9cyclictest0-21swapper/210:50:132
3235599220,9cyclictest0-21swapper/210:18:172
3235599220,10cyclictest0-21swapper/211:11:152
3234999222,10cyclictest14463-21perf08:55:001
32349992216,3cyclictest532-21cut10:50:111
3234999221,5cyclictest10376-21kworker/u8:110:03:411
3234999221,19cyclictest0-21swapper/109:15:171
3234999221,19cyclictest0-21swapper/109:15:171
3234999221,18cyclictest32688-21grep09:35:151
3234999221,18cyclictest32601-40dnf10:49:221
3234999221,15cyclictest0-21swapper/110:59:151
3234999221,13cyclictest0-21swapper/112:22:171
3234999220,5cyclictest0-21swapper/108:39:301
3234399228,3cyclictest22852-21kworker/0:008:05:590
3234399228,3cyclictest22852-21kworker/0:008:05:590
3234399227,7cyclictest11942-21kworker/0:108:51:280
3234399222,9cyclictest31940-21kworker/0:207:21:270
3234399222,9cyclictest28677-21kworker/0:011:57:270
3234399221,9cyclictest3140-21kworker/0:108:28:040
3234399221,12cyclictest15951-21kworker/0:107:51:390
3234399220,10cyclictest31247-21kworker/0:008:30:120
3236099215,4cyclictest0-21swapper/311:10:233
3236099214,8cyclictest3698-21acpi08:30:093
3236099213,6cyclictest26632-21df_inode10:35:123
3236099213,3cyclictest20453-21grep11:35:103
3236099213,2cyclictest401ktimersoftd/310:30:023
3236099213,2cyclictest401ktimersoftd/310:30:023
3236099212,7cyclictest29619-21grep10:40:223
3236099212,7cyclictest1473-21/usr/sbin/munin08:25:133
3236099212,6cyclictest41-21ksoftirqd/310:04:533
3236099212,2cyclictest0-21swapper/311:15:193
3236099211,8cyclictest0-21swapper/307:43:003
32360992115,4cyclictest1161-21sed10:50:153
3236099211,10cyclictest24713-21/usr/sbin/munin11:45:133
3236099210,9cyclictest0-21swapper/312:29:373
3236099210,12cyclictest0-21swapper/307:51:223
3235599219,3cyclictest0-21swapper/211:20:032
3235599211,10cyclictest0-21swapper/208:20:332
3235599210,10cyclictest0-21swapper/211:59:342
3234999212,11cyclictest6515-21users11:00:251
32349992114,4cyclictest4513-21pmu-power12:10:211
3234999211,3cyclictest25183-21fschecks_time11:45:151
3234999211,3cyclictest20158-21ntp_states09:05:221
3234999211,18cyclictest0-21swapper/110:25:241
3234999211,18cyclictest0-21swapper/110:25:241
3234999211,17cyclictest29467-21needreboot10:40:191
3234999211,17cyclictest17848-21cat09:00:201
3234999211,15cyclictest0-21swapper/108:59:331
3234999211,14cyclictest0-21swapper/109:48:401
3234999211,11cyclictest9621-21memory09:55:191
3234999211,11cyclictest0-21swapper/111:15:141
3234399216,2cyclictest31940-21kworker/0:207:12:160
3234399213,16cyclictest5330-21munin-run12:15:010
3234399212,9cyclictest15566-21kworker/0:211:24:360
3234399212,11cyclictest5891-21/usr/sbin/munin08:35:150
3234399212,10cyclictest28677-21kworker/0:011:52:510
32343992114,2cyclictest8083-21kworker/0:308:45:280
3234399211,11cyclictest16584-21kworker/0:009:06:270
3234399210,8cyclictest15566-21kworker/0:211:38:270
3236099208,4cyclictest21056-21diskmemload11:54:303
3236099206,7cyclictest87350irq/126-enp1s0-07:25:243
3236099205,8cyclictest401ktimersoftd/310:15:093
3236099205,7cyclictest7517-21latency_hist12:20:023
3236099203,2cyclictest30078-21seq09:30:123
3236099201,6cyclictest0-21swapper/312:07:213
3236099201,4cyclictest0-21swapper/309:56:423
3236099200,7cyclictest14408-21taskset11:20:163
3236099200,13cyclictest0-21swapper/308:50:073
3236099200,12cyclictest87650irq/129-enp1s0-09:25:263
3235599206,3cyclictest0-21swapper/212:00:342
3235599205,3cyclictest0-21swapper/212:29:202
3235599201,10cyclictest0-21swapper/211:04:252
3234999206,4cyclictest11304-21munin-run11:15:001
3234999206,12cyclictest4032-21kworker/u8:212:15:221
3234999204,8cyclictest0-21swapper/111:55:141
3234999201,8cyclictest0-21swapper/111:39:361
32349992017,2cyclictest0-21swapper/107:32:051
32349992016,3cyclictest0-21swapper/110:20:591
3234999201,2cyclictest0-21swapper/110:05:251
3234999201,18cyclictest0-21swapper/108:20:221
3234999201,14cyclictest0-21swapper/112:37:351
3234999201,13cyclictest0-21swapper/111:31:301
3234999201,11cyclictest4764-21ntp_states08:30:191
3234999201,11cyclictest26743-21ntp_states09:20:211
3234999201,11cyclictest14403-21hddtemp_smartct11:20:151
3234999201,10cyclictest24496-21perf11:45:021
3234999201,10cyclictest24444-21diskstats10:30:121
3234999200,18cyclictest0-21swapper/108:49:071
3234399202,9cyclictest7145-21kworker/0:007:25:550
3234399201,8cyclictest9351-21kworker/0:107:35:350
3234399201,8cyclictest14373-21kworker/0:309:01:110
3236099195,8cyclictest41-21ksoftirqd/308:05:153
3236099195,8cyclictest41-21ksoftirqd/308:05:153
3236099194,8cyclictest20054-21entropy10:20:143
3236099194,12cyclictest8554-21fschecks_time08:40:143
3236099193,7cyclictest20662-21/usr/sbin/munin07:55:133
3236099191,7cyclictest0-21swapper/310:55:023
3236099191,6cyclictest0-21swapper/310:49:193
3236099191,15cyclictest11636-21latency_hist07:35:023
3236099191,15cyclictest0-21swapper/309:40:353
3236099191,11cyclictest0-21swapper/308:20:243
3234999198,3cyclictest1436-21expr12:05:121
32349991915,3cyclictest0-21swapper/112:28:321
3234999191,17cyclictest0-21swapper/109:52:031
3234999191,17cyclictest0-21swapper/108:16:491
3234999191,13cyclictest0-21swapper/111:54:321
32349991911,3cyclictest0-21swapper/109:10:511
3234999191,10cyclictest9437-21latency_hist07:30:011
3234999191,10cyclictest898-21latency_hist12:05:021
3234999191,10cyclictest7947-21irqstats11:05:181
3234999191,10cyclictest4075-21latency_hist09:45:011
3234999191,10cyclictest30401-21cstates09:30:131
3234999191,10cyclictest28789-21cut09:25:191
3234999191,10cyclictest2846-21expr08:25:231
3234999191,10cyclictest25297-21cstates08:05:121
3234999191,10cyclictest25297-21cstates08:05:121
3234999191,10cyclictest16603-21df_inode07:45:141
3234999190,5cyclictest0-21swapper/111:26:091
3234999190,11cyclictest0-21swapper/110:15:081
3234399198,3cyclictest8084-21/usr/sbin/munin08:40:190
3234399191,7cyclictest4684-21runrttasks07:15:280
3236099187,3cyclictest31305-21/usr/sbin/munin12:00:193
3236099186,3cyclictest87650irq/129-enp1s0-07:15:223
3236099185,7cyclictest1831-21seq09:39:273
3236099184,11cyclictest6884-21cut08:35:193
3236099182,13cyclictest5339-21ls07:20:133
3236099181,6cyclictest0-21swapper/311:30:113
3236099181,6cyclictest0-21swapper/310:05:183
3236099181,5cyclictest808-21snmpd12:12:493
32360991812,4cyclictest28072-21irqstats08:10:183
32360991810,6cyclictest13539-21cat12:30:243
3235599185,3cyclictest0-21swapper/212:06:302
32349991814,3cyclictest0-21swapper/110:37:591
3234999181,3cyclictest0-21swapper/107:53:551
3234999181,2cyclictest0-21swapper/108:44:391
3234999181,16cyclictest0-21swapper/108:00:441
3234999181,16cyclictest0-21swapper/107:55:031
3234999181,13cyclictest87650irq/129-enp1s0-10:10:371
3234999180,16cyclictest0-21swapper/107:35:141
3234399181,10cyclictest31940-21kworker/0:207:30:030
197132180,2sleep30-21swapper/309:05:153
3236099179,5cyclictest26748-21ntp_states09:20:213
3236099173,6cyclictest29454-21/usr/sbin/munin08:15:223
3236099172,8cyclictest87550irq/128-enp1s0-11:05:223
3236099172,2cyclictest87650irq/129-enp1s0-08:55:233
3236099172,2cyclictest11866-21/usr/sbin/munin07:35:293
3236099171,13cyclictest0-21swapper/308:46:133
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional