You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-03 - 16:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Sun May 03, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
185172233157,24sleep00-21swapper/007:09:100
183712203166,24sleep10-21swapper/107:07:141
182432203166,24sleep30-21swapper/307:05:353
164002202169,21sleep20-21swapper/207:05:022
2768521220,3sleep31868399cyclictest08:40:193
18676999490,3cyclictest0-21swapper/212:12:492
205872450,1sleep10-21swapper/112:05:161
202472450,2sleep30-21swapper/308:25:143
18683994515,11cyclictest17397-21latency_hist12:00:023
18683994229,7cyclictest16784-40dnf10:45:053
1868399378,20cyclictest7376-21diskmemload10:00:173
18683993712,11cyclictest22416-21uniq12:10:153
18683993612,13cyclictest1925-21expr10:10:123
1866399367,14cyclictest25596-21munin-run11:05:010
18683993517,9cyclictest11388-21/usr/sbin/munin11:45:143
1866399358,12cyclictest24981-21perf09:50:010
18683993427,4cyclictest24102-21/usr/sbin/munin09:45:223
18683993413,9cyclictest19705-21cat12:05:033
18683993413,12cyclictest8225-21sshd10:25:113
18683993410,12cyclictest13293-21hddtemp_smartct10:35:183
1868399339,15cyclictest30817-21tail11:15:153
1868399338,10cyclictest31139-21cstates12:30:123
18683993315,4cyclictest29433-21interrupts12:25:173
18683993314,12cyclictest20980-21/usr/sbin/munin09:40:123
18683993310,5cyclictest1377-21latency_hist10:10:013
18683993310,14cyclictest21738-21sshd12:09:123
1866399332,24cyclictest31436-21kworker/0:110:04:540
318722320,1sleep10-21swapper/107:35:231
18683993215,10cyclictest15265-21ntp_states09:25:223
18683993211,6cyclictest19877-21/usr/sbin/munin09:35:253
1866399326,12cyclictest22787-21basename09:45:010
1866399321,24cyclictest10067-21kworker/0:210:44:500
1868399318,13cyclictest0-21swapper/310:40:183
1868399313,17cyclictest0-21swapper/307:20:203
18683993114,9cyclictest12351-21diskstats09:20:133
18683993113,10cyclictest21877-1kworker/3:2H09:54:163
18683993110,13cyclictest4446-21/usr/sbin/munin11:30:263
1868399307,16cyclictest0-21swapper/311:35:253
1868399307,14cyclictest0-21swapper/311:25:133
1868399307,14cyclictest0-21swapper/311:25:133
18683993018,7cyclictest808-21snmpd07:44:243
18683993014,4cyclictest6112-21/usr/sbin/munin10:20:123
18683993014,4cyclictest17460-21ntp_states09:30:213
18683993011,16cyclictest23616-21/usr/sbin/munin11:00:263
1866399304,12cyclictest17762-21basename08:20:010
1866399303,12cyclictest0-21swapper/007:30:000
18663993011,16cyclictest13016-21munin-run11:50:010
1868399299,8cyclictest14686-21cat11:50:253
1868399299,8cyclictest10420-21/usr/sbin/munin10:30:123
1868399299,12cyclictest24577-21/usr/sbin/munin08:35:193
1868399298,11cyclictest0-21swapper/310:15:233
1868399298,11cyclictest0-21swapper/310:15:233
1868399296,16cyclictest0-21swapper/309:10:123
18683992913,8cyclictest15782-21/usr/sbin/munin08:15:283
18683992913,5cyclictest10505-21taskset11:41:143
1866399296,11cyclictest8626-21perf11:40:010
1866399292,13cyclictest21194-21munin-run10:55:010
1866399292,11cyclictest9934-21kworker/0:010:30:500
1866399291,13cyclictest28410-21munin-run12:25:010
1868399288,17cyclictest22490-21ntp_states10:55:223
1868399287,12cyclictest0-21swapper/307:25:273
1868399286,8cyclictest21186-21latency_hist10:55:023
18683992821,4cyclictest7319-21/usr/sbin/munin07:55:143
18683992819,5cyclictest32239-21ntp_states08:50:213
18683992819,5cyclictest10875-21ntp_states09:15:213
18683992810,5cyclictest30166-21sshd07:33:103
18683992810,3cyclictest87350irq/126-enp1s0-12:35:143
18683992810,3cyclictest87350irq/126-enp1s0-12:35:143
1866399283,12cyclictest26208-21munin-run12:20:010
1866399282,11cyclictest27619-21kworker/0:311:11:340
1866399281,22cyclictest12044-21kworker/0:009:33:100
1868399279,15cyclictest21765-21/usr/sbin/munin07:15:213
1868399278,7cyclictest5413-21/usr/sbin/munin09:05:183
1868399278,11cyclictest11387-21/usr/sbin/munin08:05:253
1868399276,10cyclictest0-21swapper/310:00:013
18683992718,6cyclictest10115-21sed08:00:193
18683992717,7cyclictest26586-21/usr/sbin/munin11:05:173
1866399279,15cyclictest2637-21munin-run12:40:010
1866399279,15cyclictest2637-21munin-run12:40:010
1866399272,17cyclictest8469-21kworker/0:211:42:180
1866399271,14cyclictest26545-21munin-run08:40:000
1866399271,11cyclictest9934-21kworker/0:010:35:300
1866399271,11cyclictest29093-21kworker/0:009:57:260
1866399271,11cyclictest16105-21kworker/0:309:36:100
1866399271,10cyclictest15072-21kworker/0:311:55:220
1866399270,12cyclictest6425-21awk11:35:010
1868399269,14cyclictest32548-21cpuspeed_turbos11:20:113
1868399266,18cyclictest0-21swapper/312:15:123
18683992619,4cyclictest19049-21ntp_states08:20:213
18683992618,5cyclictest19223-21diskstats07:10:143
18683992617,6cyclictest30548-21/usr/sbin/munin07:35:243
18683992617,6cyclictest29596-21/usr/sbin/munin08:45:163
18683992617,6cyclictest26433-21/usr/sbin/munin12:20:153
18683992611,12cyclictest28167-21cpuspeed_turbos11:10:123
18663992617,4cyclictest7673-21kworker/0:110:22:340
18683992518,4cyclictest5369-21/usr/sbin/munin07:50:163
18683992518,4cyclictest5369-21/usr/sbin/munin07:50:153
18683992518,4cyclictest2932-21df_inode07:45:143
18683992518,4cyclictest13932-21/usr/sbin/munin08:10:143
18683992518,4cyclictest1338-21/usr/sbin/munin08:55:143
18683992517,5cyclictest22720-21df08:30:123
18683992516,6cyclictest3205-21/usr/sbin/munin09:00:123
18676992510,5cyclictest0-21swapper/211:25:112
18676992510,5cyclictest0-21swapper/211:25:112
1867199258,4cyclictest0-21swapper/109:18:051
18671992516,2cyclictest23190-21interrupts08:30:171
1867199251,4cyclictest17751-21munin-run08:20:011
1867199251,16cyclictest7792-21cstates09:10:121
1867199250,18cyclictest0-21swapper/109:25:141
1866399252,11cyclictest27619-21kworker/0:311:18:300
1866399252,11cyclictest24366-21perf08:35:010
1866399251,9cyclictest24260-21kworker/0:208:40:230
18663992511,2cyclictest4471-21kworker/0:107:55:580
1866399251,12cyclictest1795-21kworker/0:111:27:140
1866399251,12cyclictest1795-21kworker/0:111:27:140
1866399251,10cyclictest31893-21kworker/0:011:21:500
18663992510,8cyclictest9544-21kworker/0:309:19:580
1867699249,5cyclictest579-21systemd-journal08:37:112
1867699241,5cyclictest0-21swapper/209:40:112
1867699241,10cyclictest0-21swapper/212:05:022
1867199244,14cyclictest0-21swapper/109:35:281
1867199242,16cyclictest23-21ksoftirqd/112:30:121
1867199240,16cyclictest0-21swapper/108:05:201
1866399242,11cyclictest20972-21kworker/0:110:57:230
1866399242,11cyclictest13230-21kworker/0:212:02:180
1866399241,9cyclictest3577-21kworker/0:010:18:220
1866399241,9cyclictest3577-21kworker/0:010:18:220
1866399241,10cyclictest1234-21kworker/0:310:08:380
1866399241,10cyclictest12044-21kworker/0:009:26:340
1867699231,5cyclictest0-21swapper/211:31:122
1867699231,5cyclictest0-21swapper/210:21:242
1867199238,3cyclictest21610-21irqstats09:40:191
1867199237,9cyclictest0-21swapper/111:42:221
1867199231,16cyclictest0-21swapper/112:16:121
1867199231,16cyclictest0-21swapper/110:01:551
1867199231,11cyclictest0-21swapper/109:47:511
1867199231,10cyclictest0-21swapper/112:25:011
1866399237,9cyclictest30210-21kworker/0:107:39:060
1866399236,2cyclictest374-21kworker/0:308:54:150
1866399232,11cyclictest5068-21kworker/0:209:13:100
1866399232,11cyclictest27960-21kworker/0:007:30:220
1866399231,9cyclictest28341-21kworker/0:012:31:340
1866399230,10cyclictest9544-21kworker/0:309:20:020
1867699220,7cyclictest0-21swapper/209:30:182
1867199229,10cyclictest9050-21kernelversion10:25:191
1867199228,4cyclictest0-21swapper/110:46:241
1867199228,4cyclictest0-21swapper/109:30:061
18671992214,3cyclictest17485-21kworker/u8:208:36:511
1867199221,3cyclictest25320-21cpuspeed_turbos09:50:111
1867199221,3cyclictest12629-21users11:45:271
1867199221,18cyclictest21596-21cut07:15:041
1867199221,18cyclictest20136-21smart_sda07:10:241
1867199221,16cyclictest87550irq/128-enp1s0-12:27:021
1867199221,16cyclictest0-21swapper/111:08:131
1867199221,15cyclictest0-21swapper/110:42:251
1867199221,15cyclictest0-21swapper/107:56:571
1867199221,11cyclictest0-21swapper/110:16:281
1867199221,11cyclictest0-21swapper/110:16:281
1866399228,3cyclictest28341-21kworker/0:012:27:140
1866399222,8cyclictest2233-21kworker/0:307:50:420
1866399222,8cyclictest2233-21kworker/0:307:50:420
1866399221,9cyclictest5068-21kworker/0:209:09:050
1866399221,10cyclictest10067-21kworker/0:210:29:380
1867699215,6cyclictest20809-21users10:50:262
1867199219,4cyclictest7433-21cat09:10:011
1867199217,4cyclictest0-21swapper/111:01:481
1867199217,3cyclictest0-21swapper/111:12:031
1867199212,11cyclictest22857-21irqstats12:10:171
1867199211,3cyclictest29271-21memory07:30:201
1867199211,3cyclictest0-21swapper/108:55:011
1867199211,18cyclictest0-21swapper/107:54:331
1867199211,18cyclictest0-21swapper/107:54:331
1867199211,17cyclictest31343-21pmu-power11:15:231
1867199211,15cyclictest0-21swapper/107:45:241
1867199211,14cyclictest0-21swapper/108:54:291
18671992111,3cyclictest0-21swapper/111:31:121
18671992111,3cyclictest0-21swapper/110:21:231
1867199211,12cyclictest30441-21kworker/u8:212:36:501
1867199211,12cyclictest30441-21kworker/u8:212:36:501
1867199211,11cyclictest4220-21awk11:30:011
1867199211,11cyclictest4220-21awk11:30:011
1867199211,10cyclictest0-21swapper/111:53:371
1867199211,10cyclictest0-21swapper/110:54:331
18671992110,2cyclictest18418-21if_err_enp2s012:00:171
1867199210,16cyclictest0-21swapper/108:26:441
1867199210,16cyclictest0-21swapper/107:41:031
1866399219,7cyclictest87350irq/126-enp1s0-09:03:250
1866399217,2cyclictest1234-21kworker/0:310:11:180
1866399212,9cyclictest23659-21kworker/0:012:12:060
1866399211,8cyclictest374-21kworker/0:308:55:110
1866399211,14cyclictest18661-21kworker/0:010:46:140
1867699201,6cyclictest0-21swapper/211:18:102
1867699201,10cyclictest0-21swapper/211:50:422
18676992010,2cyclictest21825-21awk12:10:012
1867199207,8cyclictest0-21swapper/110:57:481
1867199207,3cyclictest0-21swapper/111:37:511
1867199201,9cyclictest0-21swapper/110:13:171
18671992016,3cyclictest0-21swapper/111:55:471
18671992012,3cyclictest0-21swapper/108:41:141
1867199201,17cyclictest0-21swapper/108:22:141
1867199201,14cyclictest0-21swapper/109:20:071
1867199201,14cyclictest0-21swapper/109:00:541
1867199201,14cyclictest0-21swapper/108:03:241
1867199201,13cyclictest0-21swapper/110:38:321
1867199201,10cyclictest2145-21cat11:25:021
1867199201,10cyclictest0-21swapper/109:59:491
18671992010,2cyclictest0-21swapper/110:33:201
1866399209,3cyclictest4471-21kworker/0:108:08:260
1866399207,2cyclictest4471-21kworker/0:108:13:020
1866399206,2cyclictest32475-21kworker/0:007:44:300
1866399201,8cyclictest23418-21kworker/0:007:20:540
1866399201,6cyclictest13230-21kworker/0:212:07:380
18663992010,3cyclictest20500-21kworker/0:207:15:220
1867699192,5cyclictest26259-21ntp_states09:50:202
1867699192,4cyclictest12147-21grep11:45:242
18676991912,4cyclictest21167-21sed08:25:212
1867699191,11cyclictest0-21swapper/210:02:532
18671991915,3cyclictest0-21swapper/107:23:101
1867199191,16cyclictest0-21swapper/110:08:451
1867199191,10cyclictest15633-21cat08:15:011
1866399197,3cyclictest4471-21kworker/0:108:27:360
1866399196,2cyclictest20972-21kworker/0:111:05:580
1866399192,11cyclictest32475-21kworker/0:007:45:140
1866399191,8cyclictest10721-21kworker/0:011:50:500
1866399191,6cyclictest30599-21kworker/0:008:48:550
1866399191,10cyclictest9181-21/usr/sbin/munin08:00:180
1866399191,10cyclictest0-21swapper/007:10:150
26082181,2sleep20-21swapper/208:55:272
1867699188,3cyclictest30-21rcuc/209:05:192
1867699182,6cyclictest20881-21munin-node07:15:012
1867699181,8cyclictest8410-21cstates10:25:142
1867699181,5cyclictest0-21swapper/210:31:192
1867699181,3cyclictest0-21swapper/209:45:072
18676991813,3cyclictest4774-21sed09:00:262
18676991812,4cyclictest2720-21grep07:45:132
18676991811,4cyclictest22540-21tr08:30:122
18676991811,4cyclictest18320-21df_inode08:20:132
1867699181,11cyclictest0-21swapper/211:55:472
1867699181,11cyclictest0-21swapper/211:06:222
1867199181,4cyclictest26769-21gltestperf07:25:161
1866399182,10cyclictest19791-21kworker/0:308:24:140
67072171,2sleep20-21swapper/207:54:362
67072171,2sleep20-21swapper/207:54:362
1867699174,4cyclictest4957-21pmu-power10:15:242
1867699174,4cyclictest4957-21pmu-power10:15:242
1867699171,7cyclictest3598-21latency_hist10:15:022
1867699171,7cyclictest1550-21sed12:35:192
1867699171,7cyclictest1550-21sed12:35:192
1867699171,7cyclictest11998-21/usr/sbin/munin09:20:142
1867699171,6cyclictest0-21swapper/211:35:152
1867699171,6cyclictest0-21swapper/211:01:312
1867699171,6cyclictest0-21swapper/210:46:202
1867699171,6cyclictest0-21swapper/210:40:182
1867699171,6cyclictest0-21swapper/209:25:412
1867699171,6cyclictest0-21swapper/208:17:152
1867699171,6cyclictest0-21swapper/207:29:112
1867699171,4cyclictest30912-21kworker/u8:109:15:472
1867699171,3cyclictest0-21swapper/212:34:262
1867699171,3cyclictest0-21swapper/210:35:122
18676991711,3cyclictest31798-21/usr/sbin/munin10:05:242
18676991711,3cyclictest24359-21cpuspeed_turbos12:15:122
1867699171,11cyclictest0-21swapper/211:43:392
1867699171,10cyclictest0-21swapper/209:59:092
1867199176,3cyclictest29357-21fschecks_count08:45:151
1867699165,3cyclictest28038-21grep11:10:122
1867699161,9cyclictest0-21swapper/210:55:122
1867699161,6cyclictest7610-21/usr/sbin/munin09:10:162
1867699161,6cyclictest29889-21sensors_temp12:25:242
1867699161,6cyclictest0-21swapper/212:21:552
1867699161,5cyclictest0-21swapper/208:05:142
1867699161,3cyclictest303312cat08:45:252
1867699161,3cyclictest0-21swapper/211:20:142
18676991613,2cyclictest0-21swapper/207:37:042
1867699161,14cyclictest0-21swapper/208:52:382
1867699161,14cyclictest0-21swapper/207:43:252
18676991611,3cyclictest27700-21sort08:40:192
18676991611,3cyclictest24367-21cut07:20:142
18676991611,3cyclictest22449-21hwlatdetect07:15:172
18676991611,3cyclictest18946-21df_inode09:35:142
1866399161,12cyclictest25192-21/usr/sbin/munin09:50:110
1867699159,3cyclictest29231-21latency07:30:202
1867699151,5cyclictest13579-21/usr/sbin/munin08:10:122
18676991510,3cyclictest7333-21entropy07:55:152
18676991410,2cyclictest10183-21netstat08:00:192
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional