You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-25 - 20:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Thu Jun 25, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
276982220145,24sleep10-21swapper/107:07:181
278772204165,26sleep00-21swapper/007:09:390
276722203167,24sleep20-21swapper/207:06:592
276882202166,24sleep30-21swapper/307:07:123
1467821780,4sleep32801099cyclictest11:30:193
182421440,5sleep21823-21gltestperf12:15:162
219721370,2sleep12799999cyclictest08:35:151
245912590,4sleep081ktimersoftd/009:25:220
307812510,2sleep00-21swapper/007:15:040
205722440,1sleep30-21swapper/311:45:113
27993993712,12cyclictest28313-21munin-run10:50:010
27993993712,12cyclictest28313-21munin-run10:50:000
2800599341,18cyclictest0-21swapper/212:09:372
2799999333,16cyclictest13555-40dnf11:30:031
2800599321,2cyclictest27983-21cyclictest09:36:442
28005993119,3cyclictest0-21swapper/211:25:172
2800599311,20cyclictest18522-21diskstats11:40:132
2800599310,24cyclictest0-21swapper/210:48:532
2800599310,24cyclictest0-21swapper/210:48:532
2800599310,24cyclictest0-21swapper/210:41:252
2800599301,3cyclictest0-21swapper/212:22:012
2800599301,18cyclictest0-21swapper/210:14:122
2800599301,13cyclictest0-21swapper/211:01:212
2799999306,11cyclictest27981-21cpuspeed_turbos09:35:121
2799999301,27cyclictest0-21swapper/107:33:521
27993993011,16cyclictest17947-21munin-run11:40:010
75612290,1sleep20-21swapper/210:00:192
2800599298,10cyclictest0-21swapper/207:56:212
2800599291,11cyclictest22852-21awk09:20:252
2800599290,21cyclictest0-21swapper/208:10:212
2799999299,8cyclictest0-21swapper/112:00:121
2800599284,9cyclictest0-21swapper/212:01:242
2800599281,11cyclictest2971-21irqstats09:50:162
2800599281,11cyclictest0-21swapper/208:18:172
2800599280,9cyclictest0-21swapper/208:25:212
2800599280,22cyclictest0-21swapper/210:37:292
2800599280,20cyclictest0-21swapper/209:34:012
2800599280,20cyclictest0-21swapper/209:34:012
2800599280,17cyclictest0-21swapper/210:55:372
2799999281,25cyclictest0-21swapper/109:43:341
2799999281,25cyclictest0-21swapper/108:47:091
2799999281,25cyclictest0-21swapper/108:27:141
2799399281,22cyclictest11396-21kworker/0:007:10:280
2799399281,19cyclictest18633-21kworker/0:009:46:110
2801099274,12cyclictest41-21ksoftirqd/311:15:263
2800599271,7cyclictest8095-21perl07:35:132
2800599271,17cyclictest8769-21users11:15:252
28005992711,3cyclictest0-21swapper/208:04:252
2800599271,12cyclictest0-21swapper/211:47:452
2800599271,10cyclictest0-21swapper/208:05:252
28005992710,7cyclictest0-21swapper/211:05:292
2800599270,16cyclictest0-21swapper/211:55:242
2800599270,16cyclictest0-21swapper/211:10:252
2800599270,15cyclictest0-21swapper/209:47:402
2800599270,12cyclictest23872-21/usr/sbin/munin11:50:252
2800599270,11cyclictest0-21swapper/210:25:012
2799999271,24cyclictest0-21swapper/110:24:021
2799999271,24cyclictest0-21swapper/108:51:251
2799999271,24cyclictest0-21swapper/107:58:031
2799399272,12cyclictest1309-21idleruntime-cro08:35:010
2800599261,10cyclictest0-21swapper/207:45:452
2800599260,9cyclictest0-21swapper/210:05:572
2800599260,12cyclictest0-21swapper/211:37:272
2800599260,11cyclictest0-21swapper/210:31:252
2800599260,11cyclictest0-21swapper/210:24:052
2799999265,9cyclictest0-21swapper/111:40:141
2799999261,23cyclictest0-21swapper/107:14:231
2799999261,12cyclictest0-21swapper/112:30:131
2799999260,24cyclictest0-21swapper/109:01:031
2799999260,24cyclictest0-21swapper/108:41:071
2799399261,12cyclictest3180-21munin-run12:20:010
2800599251,9cyclictest4663-21missed_timers08:40:192
2800599251,12cyclictest0-21swapper/207:42:412
2800599251,11cyclictest0-21swapper/209:29:332
2800599251,10cyclictest0-21swapper/212:35:092
2800599251,10cyclictest0-21swapper/207:22:522
2800599250,9cyclictest0-21swapper/209:45:002
2800599250,18cyclictest12050irq/123-ahci[0008:45:292
2800599250,11cyclictest0-21swapper/212:26:372
2800599250,10cyclictest0-21swapper/212:34:292
2799999251,15cyclictest0-21swapper/111:07:201
2799999251,15cyclictest0-21swapper/110:45:101
2799999251,15cyclictest0-21swapper/110:45:101
2799999251,11cyclictest8775-21egrep10:05:111
2799999250,16cyclictest0-21swapper/111:19:131
2799999250,10cyclictest0-21swapper/110:36:291
2799399256,11cyclictest31728-21kworker/0:109:44:110
2801099241,4cyclictest0-21swapper/311:14:213
2801099241,15cyclictest0-21swapper/307:10:283
2800599241,9cyclictest0-21swapper/209:58:322
2800599241,12cyclictest8767-21gltestperf08:50:162
2800599241,10cyclictest0-21swapper/208:35:452
2800599240,9cyclictest0-21swapper/207:50:252
2800599240,11cyclictest0-21swapper/211:20:452
2800599240,10cyclictest0-21swapper/212:10:532
2800599240,10cyclictest0-21swapper/211:31:252
2799999242,2cyclictest87650irq/129-enp1s0-12:23:131
2799999241,21cyclictest0-21swapper/108:32:221
2799999241,15cyclictest0-21swapper/112:08:451
2799999241,15cyclictest0-21swapper/111:04:141
2799999241,15cyclictest0-21swapper/110:42:061
2799999241,11cyclictest5419-21ntp_states09:55:221
2799999241,11cyclictest22635-21cpuspeed_turbos11:50:131
2799999241,11cyclictest12604-21awk11:25:211
2799399242,8cyclictest6878-21kworker/0:011:46:510
2799399242,8cyclictest10391-21kworker/0:210:07:330
2799399242,10cyclictest2782-21kworker/0:107:42:490
2799399240,8cyclictest11534-21kworker/0:207:45:130
2801099239,5cyclictest0-21swapper/309:20:013
2801099231,16cyclictest0-21swapper/310:34:093
2801099231,11cyclictest0-21swapper/308:01:093
2801099230,16cyclictest0-21swapper/310:44:373
2800599236,3cyclictest0-21swapper/209:00:132
2800599231,10cyclictest0-21swapper/210:50:132
2800599231,10cyclictest0-21swapper/209:10:052
2800599230,11cyclictest0-21swapper/210:19:012
2800599230,11cyclictest0-21swapper/207:25:372
2799999238,3cyclictest7469-21memory10:00:191
2799999232,4cyclictest16710-21munin-run09:10:001
2799999232,10cyclictest0-21swapper/111:46:121
27999992313,2cyclictest0-21swapper/112:19:501
27999992312,3cyclictest0-21swapper/111:13:441
27999992312,3cyclictest0-21swapper/110:59:111
27999992312,3cyclictest0-21swapper/110:31:251
2799999231,16cyclictest0-21swapper/111:23:071
2799999231,15cyclictest0-21swapper/112:11:381
2799999231,11cyclictest18523-21grep08:00:121
2799999231,11cyclictest14554-21timerwakeupswit10:15:241
2799999231,11cyclictest1-21systemd09:30:011
2799999231,10cyclictest593-21latency_hist07:20:001
2799999230,8cyclictest0-21swapper/110:14:211
2799999230,21cyclictest0-21swapper/110:26:201
2799399236,15cyclictest17326-21munin-run10:25:010
2799399231,9cyclictest22338-21kworker/0:208:10:370
2799399231,7cyclictest21533-21kworker/0:110:37:330
2801099225,4cyclictest0-21swapper/310:35:223
28010992213,6cyclictest22313-21systemd-journal08:05:293
2801099220,10cyclictest0-21swapper/312:37:113
2800599228,3cyclictest0-21swapper/207:10:372
2800599227,2cyclictest0-21swapper/207:31:552
2800599221,10cyclictest0-21swapper/209:15:492
2800599220,10cyclictest0-21swapper/208:30:092
2799999221,9cyclictest0-21swapper/112:38:111
2799999221,4cyclictest22022-21idleruntime09:20:151
27999992211,3cyclictest0-21swapper/112:27:521
2799999221,11cyclictest87650irq/129-enp1s0-09:15:131
2799999220,14cyclictest0-21swapper/110:50:281
2799399228,3cyclictest5930-21kworker/0:309:04:190
2799399228,2cyclictest5930-21kworker/0:309:08:470
2799399225,7cyclictest0-21swapper/010:03:540
2799399222,9cyclictest31460-21kworker/0:108:41:500
2799399222,9cyclictest24770-21kworker/0:308:15:060
2799399222,9cyclictest16371-21kworker/0:109:11:030
2799399222,8cyclictest2782-21kworker/0:107:39:370
2799399221,9cyclictest9511-21kworker/0:112:35:590
2799399221,9cyclictest35932interrupts11:05:150
2799399221,9cyclictest23849-21kworker/0:210:55:140
2799399221,9cyclictest23796-21kworker/0:010:40:030
2799399221,9cyclictest10379-21kworker/0:111:21:310
110722220,4sleep339-21rcuc/310:10:123
28010992114,4cyclictest8529-21dump-pmu-power10:05:013
2801099211,11cyclictest16163-21irqstats10:20:183
2801099211,11cyclictest12650-21ntp_states11:25:233
28005992115,3cyclictest30784-21fschecks_count07:15:042
2799999219,2cyclictest28564-21sensors08:20:231
2799999219,2cyclictest28564-21sensors08:20:231
2799999218,2cyclictest12295-21latency_hist09:00:011
2799999211,9cyclictest24011-21pmu-power08:10:221
2799999211,3cyclictest0-21swapper/107:42:261
2799999210,19cyclictest0-21swapper/108:15:141
2799399214,2cyclictest6878-21kworker/0:012:25:250
2799399213,12cyclictest87650irq/129-enp1s0-09:50:450
2799399212,9cyclictest32499-21kworker/0:111:02:190
2799399212,8cyclictest20780-21kworker/0:209:20:270
2799399211,9cyclictest9511-21kworker/0:112:33:230
2799399211,9cyclictest6878-21kworker/0:011:56:070
2799399211,9cyclictest667-21kworker/0:312:12:310
2799399211,8cyclictest17886-21kworker/0:211:43:230
27993992112,2cyclictest6878-21kworker/0:012:22:010
27993992111,6cyclictest827-21runrttasks08:28:250
2799399211,12cyclictest4353-21/usr/sbin/munin09:55:190
2799399210,9cyclictest23849-21kworker/0:210:53:500
2801099202,2cyclictest25343-21kworker/u8:009:33:323
2801099202,2cyclictest25343-21kworker/u8:009:33:323
2801099201,8cyclictest13884-21perf07:50:013
2801099201,6cyclictest0-21swapper/309:13:213
2801099201,5cyclictest0-21swapper/311:02:093
28010992015,3cyclictest4253-21grep11:05:233
28010992014,3cyclictest1690-21tr08:35:123
2801099201,13cyclictest0-21swapper/311:50:203
2801099201,12cyclictest0-21swapper/309:29:433
2801099201,11cyclictest808-21snmpd09:43:073
2801099201,11cyclictest2155-21/usr/sbin/munin09:50:193
2801099201,10cyclictest0-21swapper/311:35:093
2800599201,5cyclictest27592-21cstates08:20:122
2800599201,5cyclictest27592-21cstates08:20:122
2799999201,18cyclictest0-21swapper/109:50:551
2799999201,18cyclictest0-21swapper/107:51:451
2799999201,17cyclictest0-21swapper/109:31:491
2799999201,17cyclictest0-21swapper/109:31:491
2799999200,5cyclictest0-21swapper/111:37:491
2799399208,2cyclictest6878-21kworker/0:011:19:430
2799399208,2cyclictest11534-21kworker/0:207:50:050
2799399207,3cyclictest26989-21kworker/0:208:23:540
2799399207,3cyclictest26989-21kworker/0:208:23:540
2799399207,2cyclictest20780-21kworker/0:209:17:390
2799399206,3cyclictest14823-21kworker/0:310:34:350
2799399202,8cyclictest6878-21kworker/0:012:02:270
2799399201,9cyclictest26560-21kworker/0:112:06:070
2799399201,9cyclictest18633-21kworker/0:009:37:270
2799399201,9cyclictest14823-21kworker/0:310:27:070
2799399201,9cyclictest13409-21kworker/0:211:29:530
2799399201,8cyclictest4658-21kworker/0:211:10:510
2799399201,6cyclictest0-21swapper/007:35:010
27993992013,2cyclictest87350irq/126-enp1s0-07:57:090
311022190,3sleep30-21swapper/312:09:403
2801099193,6cyclictest0-21swapper/309:48:433
2801099191,9cyclictest0-21swapper/307:30:263
2801099191,9cyclictest0-21swapper/307:27:103
2801099191,8cyclictest14849-21grep09:05:123
2801099191,4cyclictest0-21swapper/311:41:033
28010991913,5cyclictest22974-21kworker/u8:109:23:433
2801099191,2cyclictest0-21swapper/308:49:163
2801099191,16cyclictest0-21swapper/309:00:223
2801099191,10cyclictest839-21lldpd08:15:213
2801099191,10cyclictest7800-21/usr/sbin/munin12:30:193
2799999197,3cyclictest1739-21irqstats07:20:171
27999991915,3cyclictest0-21swapper/107:25:161
2799999191,3cyclictest481-21tail09:45:151
2799999191,2cyclictest0-21swapper/109:12:111
2799999191,17cyclictest0-21swapper/108:07:541
2799999191,16cyclictest0-21swapper/107:35:321
2799999191,11cyclictest0-21swapper/107:46:121
2799399198,2cyclictest27585-21kworker/0:207:29:430
2799399198,2cyclictest22338-21kworker/0:208:06:430
2799399197,2cyclictest18633-21kworker/0:009:33:110
2799399197,2cyclictest18633-21kworker/0:009:33:110
2799399191,9cyclictest10391-21kworker/0:210:10:590
2799399191,5cyclictest0-21swapper/011:54:020
27993991911,2cyclictest87650irq/129-enp1s0-10:19:090
2799399191,10cyclictest10331-21/usr/sbin/munin08:55:180
2799399190,9cyclictest6878-21kworker/0:011:31:590
2801099181,9cyclictest18605-21ntpq10:25:193
2801099181,6cyclictest0-21swapper/310:50:413
28010991813,3cyclictest31341-21grep10:55:173
28010991812,3cyclictest8133-21/usr/sbin/munin08:50:203
28010991811,4cyclictest4383-21gltestperf08:40:163
2801099181,10cyclictest0-21swapper/312:21:483
2801099181,10cyclictest0-21swapper/312:15:253
2800599181,5cyclictest10534-21expr08:55:122
2800599181,13cyclictest0-21swapper/209:07:572
27999991813,3cyclictest24867-21tr11:55:121
2799399185,2cyclictest15979-21kworker/0:108:01:210
2799399181,7cyclictest1192-21kworker/0:208:35:200
27993991811,2cyclictest87550irq/128-enp1s0-08:47:410
2801099178,3cyclictest10353-21vmstat10:05:253
2801099176,1cyclictest0-21swapper/308:57:093
2801099172,7cyclictest0-21swapper/312:28:123
2801099171,7cyclictest26421-21cpuspeed_turbos10:45:123
2801099171,7cyclictest26421-21cpuspeed_turbos10:45:123
2801099171,6cyclictest0-21swapper/310:15:163
2801099171,6cyclictest0-21swapper/309:55:423
2801099171,6cyclictest0-21swapper/307:50:273
2801099171,4cyclictest0-21swapper/311:59:503
28010991713,3cyclictest0-21swapper/308:27:393
2801099171,15cyclictest0-21swapper/307:17:123
2801099171,14cyclictest16700-21diskmemload09:37:113
2801099171,11cyclictest0-21swapper/312:11:283
2799399175,2cyclictest5930-21kworker/0:308:51:050
27993991712,3cyclictest884-21/usr/sbin/munin07:20:120
2801099167,7cyclictest87350irq/126-enp1s0-07:55:133
2801099161,7cyclictest0-21swapper/307:40:203
2801099161,6cyclictest0-21swapper/311:20:343
28010991612,3cyclictest0-21swapper/308:10:143
28010991611,3cyclictest884-21/usr/sbin/munin07:20:113
28010991611,3cyclictest26958-21/usr/sbin/munin12:00:163
2801099161,13cyclictest0-21swapper/308:20:263
2801099161,13cyclictest0-21swapper/308:20:263
2801099151,12cyclictest430-21processes08:30:223
28010991510,3cyclictest8218-21basename07:35:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional