You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-24 - 00:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Sep 23, 2026 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
233152238163,24sleep30-21swapper/307:09:113
232362238163,24sleep20-21swapper/207:08:092
231802203164,26sleep00-21swapper/007:07:270
230462202164,25sleep10-21swapper/107:05:421
1702121820,4sleep02347099cyclictest09:20:140
143612990,3sleep02347099cyclictest09:14:550
124562960,2sleep20-21swapper/207:55:162
80022600,4sleep12347599cyclictest09:00:121
80022600,4sleep12347599cyclictest09:00:121
17572510,5sleep22348099cyclictest07:30:202
23480994533,3cyclictest0-21swapper/211:44:252
228692450,1sleep30-21swapper/310:45:213
2347599411,25cyclictest13744-21sh08:00:011
227682410,1sleep00-21swapper/012:00:140
2347599401,29cyclictest13413-21munin-run11:40:011
23487993926,8cyclictest0-21swapper/309:00:393
23487993926,8cyclictest0-21swapper/309:00:393
23475993912,18cyclictest0-21swapper/108:30:231
2347599381,2cyclictest0-21swapper/110:00:001
2347599381,27cyclictest0-21swapper/107:25:121
2347099388,14cyclictest7438-21munin-run12:40:010
2347099388,14cyclictest7438-21munin-run12:40:010
2348799371,19cyclictest0-21swapper/310:15:273
2347599374,29cyclictest23-21ksoftirqd/107:12:541
2347099379,12cyclictest3395-21basename08:50:010
23470993711,12cyclictest16581-21munin-run09:20:010
2348799361,19cyclictest0-21swapper/309:11:233
2348099368,4cyclictest23880-21cat12:00:282
2347599361,26cyclictest28502-21sort09:45:151
2348099350,26cyclictest0-21swapper/210:10:092
2348099350,26cyclictest0-21swapper/210:10:092
2347599357,13cyclictest19735-21tr10:40:111
2347599353,29cyclictest9616-21fschecks_count11:30:181
2347599351,3cyclictest30924-21cat07:25:011
2347599351,22cyclictest8339-21sed11:25:241
2347599351,22cyclictest8339-21sed11:25:241
23475993512,17cyclictest20350-21latency_hist08:15:021
2347599350,23cyclictest21507-21/usr/sbin/munin11:55:241
2348799341,24cyclictest0-21swapper/312:16:493
2348799341,24cyclictest0-21swapper/307:59:393
2348799341,14cyclictest0-21swapper/311:42:053
2347599343,18cyclictest9809-21processes10:15:221
2347599341,30cyclictest27389-21fschecks_time12:10:181
2347599341,2cyclictest0-21swapper/112:30:241
2347599341,19cyclictest18043-21/usr/sbin/munin11:50:221
2347599340,2cyclictest0-21swapper/107:45:231
2347599340,2cyclictest0-21swapper/107:45:231
2347599340,23cyclictest0-21swapper/108:15:261
2348799331,20cyclictest0-21swapper/308:01:073
2348799331,19cyclictest0-21swapper/309:39:433
2348799331,14cyclictest0-21swapper/311:14:413
2348099331,15cyclictest17669-21taskset11:48:232
2347599332,27cyclictest18681-21smart_sda10:35:251
2347599331,3cyclictest5439-21gltestperf11:20:151
2347599331,3cyclictest28182-21latency_hist11:00:011
2347599331,3cyclictest0-21swapper/110:05:091
2347599331,25cyclictest26148-21cat10:55:021
2347599331,17cyclictest0-21swapper/109:40:241
23487993216,8cyclictest0-21swapper/311:56:133
2348799321,23cyclictest0-21swapper/311:22:493
2348799321,21cyclictest0-21swapper/310:06:593
2348799321,20cyclictest0-21swapper/308:21:593
2348799321,19cyclictest0-21swapper/312:39:293
2348799321,19cyclictest0-21swapper/312:39:293
2348799321,19cyclictest0-21swapper/310:43:093
2348799321,18cyclictest0-21swapper/309:45:553
2348799321,11cyclictest8625-21uptime11:25:253
2348799321,11cyclictest8625-21uptime11:25:253
2348099321,25cyclictest0-21swapper/208:44:252
2348099321,14cyclictest0-21swapper/210:23:122
2348099321,14cyclictest0-21swapper/207:19:372
2348099321,13cyclictest0-21swapper/209:05:452
2347599322,26cyclictest14904-21smart_sda11:40:251
2347599321,6cyclictest24642-21/usr/sbin/munin12:05:171
2347599321,4cyclictest11274-21ntp_states09:05:221
2347599321,3cyclictest15956-21turbostat.cron08:05:011
2347599321,2cyclictest87350irq/126-enp1s0-09:15:021
2347599321,17cyclictest17067-21cstates09:20:141
2347099321,15cyclictest31007-21munin-run12:20:010
2348799311,20cyclictest0-21swapper/307:22:273
2348799311,11cyclictest6872-21ntp_states08:55:233
2348799311,11cyclictest0-21swapper/310:52:253
2348099311,20cyclictest0-21swapper/209:22:252
2348099311,15cyclictest0-21swapper/207:38:532
2348099311,14cyclictest0-21swapper/208:28:412
2348099311,13cyclictest0-21swapper/211:08:152
2348099311,13cyclictest0-21swapper/208:53:332
2348099311,13cyclictest0-21swapper/208:35:092
2348099311,12cyclictest23459-21cyclictest09:00:292
2348099311,12cyclictest23459-21cyclictest09:00:292
2348099310,20cyclictest0-21swapper/212:15:102
2347599317,21cyclictest1160-21latency_hist08:45:011
2347599311,4cyclictest351-21/usr/sbin/munin11:10:241
2347599311,3cyclictest87650irq/129-enp1s0-08:45:151
2347599311,24cyclictest766-21ntpd10:46:151
2347599311,18cyclictest0-21swapper/110:01:241
2347599311,18cyclictest0-21swapper/109:15:201
23487993016,9cyclictest0-21swapper/308:11:193
2348799301,20cyclictest0-21swapper/309:08:353
2348799301,19cyclictest0-21swapper/310:02:393
2348799301,19cyclictest0-21swapper/309:56:393
2348799301,19cyclictest0-21swapper/309:16:193
2348799301,18cyclictest0-21swapper/307:35:393
2348799301,14cyclictest31942-21strings07:25:193
2348799301,14cyclictest0-21swapper/312:30:333
2348799301,14cyclictest0-21swapper/312:06:173
2348799301,13cyclictest0-21swapper/311:31:053
2348799301,13cyclictest0-21swapper/310:28:313
2348799301,13cyclictest0-21swapper/310:13:113
2348799301,13cyclictest0-21swapper/310:13:113
2348799301,12cyclictest0-21swapper/312:00:053
2348799301,12cyclictest0-21swapper/309:41:313
2348799301,11cyclictest0-21swapper/310:39:133
23480993012,3cyclictest0-21swapper/208:02:412
2348099301,13cyclictest0-21swapper/210:55:112
2348099301,11cyclictest0-21swapper/212:33:542
2348099301,11cyclictest0-21swapper/209:53:412
2348099301,11cyclictest0-21swapper/208:10:442
2348099300,20cyclictest0-21swapper/211:53:472
2347599306,17cyclictest23107-21hddtemp_smartct12:00:161
2347599303,19cyclictest0-21swapper/111:05:491
2347599302,4cyclictest31578-21diskstats12:20:141
2347599301,5cyclictest29677-21/usr/sbin/munin11:00:241
2347599301,4cyclictest702-21/usr/sbin/munin07:30:201
2347599301,3cyclictest8494-21cat10:15:011
2347599301,3cyclictest8494-21cat10:15:001
2347599301,3cyclictest3674-21cpuspeed_turbos08:50:131
2347599301,3cyclictest22324-21pmu-power09:30:241
2347599301,26cyclictest0-21swapper/110:30:201
2348799292,7cyclictest0-21swapper/309:31:503
2348799291,23cyclictest0-21swapper/307:42:353
2348799291,20cyclictest0-21swapper/312:12:573
2348799291,19cyclictest0-21swapper/308:45:223
2348799290,20cyclictest0-21swapper/310:55:283
2348099291,14cyclictest0-21swapper/209:56:492
2348099291,14cyclictest0-21swapper/209:49:252
2348099291,14cyclictest0-21swapper/207:50:372
2348099291,13cyclictest0-21swapper/210:54:072
2348099291,13cyclictest0-21swapper/210:02:572
2348099291,13cyclictest0-21swapper/209:32:132
2348099291,13cyclictest0-21swapper/208:22:532
2348099291,13cyclictest0-21swapper/207:45:492
2348099291,13cyclictest0-21swapper/207:45:492
2348099291,12cyclictest0-21swapper/210:41:262
23480992910,10cyclictest0-21swapper/209:25:132
2348099290,19cyclictest0-21swapper/211:01:032
2347599298,18cyclictest3981-21grep07:35:201
2347599291,5cyclictest0-21swapper/112:39:361
2347599291,5cyclictest0-21swapper/112:39:361
2347599291,3cyclictest31502-21cat08:40:011
2347599291,3cyclictest17054-21ps11:45:221
2347599291,2cyclictest0-21swapper/108:20:381
2347599291,25cyclictest9977-21fschecks_count07:50:131
2347599291,25cyclictest0-21swapper/109:35:151
2348799281,19cyclictest0-21swapper/312:22:053
2348799281,19cyclictest0-21swapper/311:15:573
2348799281,12cyclictest0-21swapper/311:04:283
2348799281,12cyclictest0-21swapper/310:30:113
2348799281,10cyclictest0-21swapper/307:47:433
2348799281,10cyclictest0-21swapper/307:47:433
2348799280,13cyclictest0-21swapper/311:45:233
2348099288,4cyclictest0-21swapper/211:15:192
23480992811,3cyclictest22368-21perl10:45:152
2348099281,13cyclictest0-21swapper/212:21:262
2348099281,13cyclictest0-21swapper/210:31:402
2348099281,12cyclictest0-21swapper/212:29:022
2348099281,12cyclictest0-21swapper/210:06:332
2348099281,12cyclictest0-21swapper/208:32:572
2348099281,11cyclictest0-21swapper/211:13:112
2348099281,11cyclictest0-21swapper/208:19:202
23480992810,3cyclictest0-21swapper/207:29:012
2348099280,12cyclictest0-21swapper/210:27:082
2348099280,11cyclictest0-21swapper/211:56:072
2348099280,11cyclictest0-21swapper/208:59:132
2347599286,19cyclictest0-21swapper/107:40:111
2347599282,3cyclictest27714-21/usr/sbin/munin07:15:221
2347599281,3cyclictest2928-21cat11:15:141
2347599281,3cyclictest20260-21smart_sda09:25:231
2347599281,3cyclictest1469-21sort12:25:151
2347599281,2cyclictest29124-21cpuspeed_turbos12:15:131
2347599281,24cyclictest579-21systemd-journal10:20:211
2347599281,18cyclictest13024-21/usr/sbin/munin10:25:201
23475992811,14cyclictest827-21runrttasks08:25:571
2347599280,17cyclictest0-21swapper/109:50:151
2347099281,12cyclictest13763-21awk08:00:010
2348799279,4cyclictest1119-21diskstats07:30:153
23487992716,3cyclictest0-21swapper/312:25:093
2348799271,19cyclictest0-21swapper/308:27:113
2348799271,13cyclictest0-21swapper/308:34:303
2348799271,12cyclictest0-21swapper/311:09:123
2348799271,10cyclictest0-21swapper/310:24:433
23480992711,3cyclictest0-21swapper/208:47:292
2348099271,12cyclictest495-21jbd2/dm-0-809:35:172
2348099271,12cyclictest0-21swapper/211:20:112
2348099271,12cyclictest0-21swapper/209:16:012
2348099271,12cyclictest0-21swapper/207:14:402
2348099271,11cyclictest0-21swapper/212:37:302
2348099271,11cyclictest0-21swapper/212:37:302
2348099271,11cyclictest0-21swapper/210:35:412
23480992710,9cyclictest0-21swapper/212:06:232
2348099270,13cyclictest0-21swapper/211:31:432
2348099270,11cyclictest0-21swapper/209:10:572
2347599279,15cyclictest7957-21sort09:00:021
2347599271,2cyclictest16190-21/usr/sbin/munin08:05:171
2347099272,12cyclictest18161-21munin-run08:10:010
2347099271,13cyclictest31966-21munin-run09:55:010
23487992616,3cyclictest0-21swapper/308:51:273
2348799261,14cyclictest0-21swapper/309:53:553
2348799261,12cyclictest0-21swapper/309:27:073
2348799261,11cyclictest0-21swapper/308:42:103
2348799261,11cyclictest0-21swapper/307:50:073
2348099261,13cyclictest0-21swapper/211:29:112
2348099261,13cyclictest0-21swapper/211:29:112
2348099260,12cyclictest0-21swapper/210:18:012
2347099261,12cyclictest4010-21munin-run10:05:010
2348799251,13cyclictest0-21swapper/311:36:213
2348799251,11cyclictest0-21swapper/309:20:433
2348799251,11cyclictest0-21swapper/308:17:473
2348799251,11cyclictest0-21swapper/307:10:313
2348799251,10cyclictest0-21swapper/308:36:343
2348099258,3cyclictest0-21swapper/207:22:132
2348099251,12cyclictest0-21swapper/209:44:052
2348099251,11cyclictest0-21swapper/208:06:362
23480992510,3cyclictest0-21swapper/211:36:512
23480992510,3cyclictest0-21swapper/207:44:452
2348799241,14cyclictest0-21swapper/308:06:193
2348799241,11cyclictest0-21swapper/311:53:493
2348799241,10cyclictest0-21swapper/307:18:553
2348099241,10cyclictest0-21swapper/212:11:392
2347099226,7cyclictest0-21swapper/011:09:120
2347099221,5cyclictest0-21swapper/009:55:060
2347099220,4cyclictest0-21swapper/011:26:250
2347099220,4cyclictest0-21swapper/011:26:250
2347099219,11cyclictest0-21swapper/011:05:010
2347099213,3cyclictest87350irq/126-enp1s0-11:37:460
2347099213,10cyclictest20235-21/usr/sbin/munin11:55:170
2347099211,4cyclictest0-21swapper/012:13:220
2347099211,10cyclictest25972-21head08:25:190
2347099206,4cyclictest0-21swapper/009:31:470
2347099205,6cyclictest0-21swapper/011:13:520
2347099201,9cyclictest0-21swapper/008:40:070
2347099201,6cyclictest17497-21cpuspeed_turbos10:35:130
2347099201,6cyclictest0-21swapper/009:49:210
23470992014,3cyclictest11441-21sendmail_mailqu09:05:230
2347099201,11cyclictest6619-21irqstats08:55:200
2347099201,11cyclictest0-21swapper/012:25:160
2347099201,10cyclictest16784-21head10:30:300
2347099201,10cyclictest0-21swapper/010:20:100
2347099200,9cyclictest24762-21kworker/u8:111:18:380
36092191,2sleep03776-21cstates08:50:130
2347099198,3cyclictest23744-21df_inode09:35:140
2347099191,6cyclictest0-21swapper/012:24:530
2347099191,5cyclictest0-21swapper/010:06:290
2347099191,4cyclictest0-21swapper/010:51:060
23470991913,4cyclictest7932-21taskset10:10:280
23470991913,4cyclictest7932-21taskset10:10:280
2347099191,11cyclictest27207-21strings10:55:200
2347099191,10cyclictest21814-21/usr/sbin/munin10:45:250
2347099191,10cyclictest15736-21latency_hist11:45:020
2347099191,10cyclictest13856-21kernelversion10:25:190
2347099190,6cyclictest0-21swapper/012:06:550
2347099190,2cyclictest87350irq/126-enp1s0-11:49:200
2347099190,10cyclictest827-21runrttasks08:00:100
110782190,4sleep011-21rcuc/007:50:250
77922180,4sleep011-21rcuc/007:45:150
77922180,4sleep011-21rcuc/007:45:150
2347099186,6cyclictest0-21swapper/011:30:500
2347099186,3cyclictest0-21swapper/011:21:380
2347099184,12cyclictest9-21rcu_preempt10:40:150
2347099182,4cyclictest0-21swapper/011:51:260
2347099181,9cyclictest2086-21cat07:30:250
2347099181,9cyclictest0-21swapper/007:12:060
2347099181,6cyclictest0-21swapper/010:15:560
2347099181,3cyclictest0-21swapper/009:44:530
2347099181,2cyclictest11-21rcuc/008:20:220
2347099181,16cyclictest0-21swapper/007:41:480
2347099181,14cyclictest0-21swapper/009:25:460
2347099181,10cyclictest0-21swapper/012:30:030
2347099180,10cyclictest0-21swapper/007:15:140
2347099174,7cyclictest0-21swapper/009:02:110
2347099174,7cyclictest0-21swapper/009:02:110
2347099171,7cyclictest31340-21fschecks_count07:25:140
2347099171,7cyclictest2992-21/usr/sbin/munin07:35:140
2347099171,15cyclictest0-21swapper/008:18:050
293662161,2sleep00-21swapper/008:35:040
2347099161,3cyclictest20165-21sshd08:12:480
2347099161,14cyclictest0-21swapper/008:33:100
2347099160,14cyclictest0-21swapper/007:22:540
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional