You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-17 - 15:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun May 17, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
290022238164,24sleep20-21swapper/207:08:142
288892237159,25sleep00-21swapper/007:06:460
290232235162,24sleep30-21swapper/307:08:323
289682203168,23sleep10-21swapper/107:07:491
901421600,5sleep02923099cyclictest11:15:160
52321130,3sleep30-21swapper/307:15:163
29241999390,2cyclictest0-21swapper/209:27:342
29241999086,3cyclictest0-21swapper/208:19:252
205472570,9sleep011-21rcuc/008:00:210
13402490,2sleep00-21swapper/007:15:260
244072470,3sleep2311ktimersoftd/208:10:132
32072460,2sleep20-21swapper/208:35:132
2924199331,12cyclictest0-21swapper/210:41:292
2924199330,24cyclictest0-21swapper/212:17:242
2924199310,21cyclictest0-21swapper/208:32:562
2924199301,10cyclictest0-21swapper/208:41:082
29241992925,3cyclictest0-21swapper/209:23:112
2924199291,13cyclictest0-21swapper/209:31:132
2924199291,12cyclictest0-21swapper/211:46:562
2924199291,12cyclictest0-21swapper/210:16:082
2924199290,13cyclictest0-21swapper/212:11:122
2924199290,12cyclictest0-21swapper/211:15:252
2924199290,11cyclictest0-21swapper/208:00:202
2923099293,12cyclictest14997-21/usr/sbin/munin11:30:270
2924199281,17cyclictest0-21swapper/212:23:162
2924199281,17cyclictest0-21swapper/212:23:162
2924199281,12cyclictest0-21swapper/209:39:012
2924199281,11cyclictest0-21swapper/212:35:372
2924199281,11cyclictest0-21swapper/211:53:082
2924199281,10cyclictest0-21swapper/209:45:212
2924199280,18cyclictest0-21swapper/210:49:492
2924199280,17cyclictest0-21swapper/211:14:012
2924199280,17cyclictest0-21swapper/211:14:012
2924199280,17cyclictest0-21swapper/209:59:242
2924199280,10cyclictest0-21swapper/211:36:562
2923099289,16cyclictest13184-21munin-run12:40:020
59652270,1sleep10-21swapper/109:55:151
2924199271,17cyclictest0-21swapper/207:26:452
2924199270,13cyclictest0-21swapper/210:33:172
2924199270,11cyclictest0-21swapper/212:06:322
2924199270,11cyclictest0-21swapper/210:36:052
2924199270,10cyclictest0-21swapper/211:01:452
2924199270,10cyclictest0-21swapper/209:15:212
2923599271,17cyclictest12394-21sshd11:22:531
119182270,8sleep341-21ksoftirqd/311:20:263
292582261,3sleep229340-21/usr/sbin/munin12:00:222
2924199261,9cyclictest0-21swapper/210:06:162
2924199261,11cyclictest0-21swapper/210:01:522
2924199261,11cyclictest0-21swapper/210:01:522
2924199260,9cyclictest0-21swapper/211:09:492
2924199260,11cyclictest0-21swapper/212:30:212
2924199260,11cyclictest0-21swapper/211:22:532
2924199260,11cyclictest0-21swapper/207:41:212
2924199260,11cyclictest0-21swapper/207:30:132
2924199260,10cyclictest0-21swapper/211:40:482
2923599261,3cyclictest3800-21awk11:05:011
2923599261,22cyclictest24810-21hddtemp_smartct08:10:161
29230992615,7cyclictest20094-21sort10:25:250
2924199259,3cyclictest0-21swapper/211:31:092
2924199251,11cyclictest0-21swapper/210:58:292
2924199251,11cyclictest0-21swapper/210:10:342
2924199251,11cyclictest0-21swapper/208:45:202
2924199251,11cyclictest0-21swapper/207:59:012
2924199251,10cyclictest22965-21taskset08:05:202
2924199251,10cyclictest0-21swapper/207:15:442
2924199250,9cyclictest0-21swapper/210:24:482
2924199250,9cyclictest0-21swapper/209:10:092
2924199250,11cyclictest0-21swapper/212:29:172
2924199250,10cyclictest0-21swapper/210:27:242
2923599259,3cyclictest13183-21idleruntime-cro12:40:021
2923599251,8cyclictest0-21swapper/109:14:541
2923599251,3cyclictest22380-21fschecks_count08:05:151
2923599251,21cyclictest30580-21irqstats10:50:171
2923599251,21cyclictest28633-21ntp_states10:45:221
2923599251,21cyclictest16311-21diskstats09:05:151
29235992511,12cyclictest19211-21kworker/u8:209:04:391
29235992510,3cyclictest21366-21munin-run11:45:011
2923599250,23cyclictest0-21swapper/109:33:461
2924799241,12cyclictest0-21swapper/310:24:383
2924199249,3cyclictest0-21swapper/208:26:262
2924199241,9cyclictest0-21swapper/207:24:172
2924199240,10cyclictest0-21swapper/209:41:292
2923599242,3cyclictest0-21swapper/112:07:361
2923599241,6cyclictest0-21swapper/111:33:401
2923599241,21cyclictest0-21swapper/109:40:251
2923599241,20cyclictest8755-21diskstats11:15:141
2923599241,20cyclictest6732-21cstates07:30:121
2923599241,20cyclictest32225-21cstates10:55:141
2923599241,20cyclictest25709-21df_inode10:40:151
2923599241,20cyclictest21060-21cpuspeed_turbos10:30:121
2923599241,20cyclictest0-21swapper/108:50:151
2923599241,20cyclictest0-21swapper/108:50:151
2923599241,19cyclictest13488-21diskstats07:45:141
2923599241,18cyclictest0-21swapper/112:02:551
2923099248,13cyclictest8525-21munin-run07:35:000
2924799234,12cyclictest20314-21unixbench_singl10:25:253
2924799231,16cyclictest87350irq/126-enp1s0-09:45:553
2924799231,11cyclictest22116-21runrttasks09:17:253
2924199235,3cyclictest0-21swapper/209:09:172
2924199231,9cyclictest0-21swapper/208:20:042
2924199230,9cyclictest0-21swapper/211:57:042
2924199230,9cyclictest0-21swapper/209:50:242
2924199230,9cyclictest0-21swapper/207:10:162
2924199230,10cyclictest0-21swapper/211:25:532
2923599233,12cyclictest24762-21sed11:50:181
29235992315,5cyclictest20209-21timerwakeupswit10:25:261
2923599231,20cyclictest0-21swapper/111:48:251
2923599231,20cyclictest0-21swapper/111:35:311
2923599231,20cyclictest0-21swapper/109:26:251
2923599231,1cyclictest5352-21crond09:55:001
2923599231,13cyclictest14017-21/usr/sbin/munin11:25:231
2923599230,18cyclictest0-21swapper/109:16:201
2923099239,12cyclictest26736-21awk09:30:010
2923099236,4cyclictest0-21swapper/011:54:430
2923099236,4cyclictest0-21swapper/009:11:190
2923099231,12cyclictest8211-21fschecks_count10:00:170
2923099231,12cyclictest8211-21fschecks_count10:00:170
2923099231,11cyclictest26123-21idleruntime-cro08:15:020
34942221,17sleep03725-21df09:50:140
2924799222,6cyclictest28293-21timerandwakeup09:30:253
2924799222,10cyclictest3227-21interrupts12:15:173
2924799221,16cyclictest87350irq/126-enp1s0-08:18:303
2924799221,12cyclictest23805-21sshd08:08:543
2924199223,2cyclictest12050irq/123-ahci[0008:59:032
2924199221,8cyclictest0-21swapper/207:36:572
2924199221,12cyclictest0-21swapper/207:46:172
2924199220,9cyclictest0-21swapper/209:00:092
2923599228,3cyclictest2685-21expr12:15:131
2923599225,7cyclictest0-21swapper/109:23:461
2923599221,19cyclictest0-21swapper/108:45:171
2923599221,19cyclictest0-21swapper/108:24:461
2923599221,18cyclictest1936-21awk07:20:011
2923599221,13cyclictest5300-21hddtemp_smartct12:20:171
2923599221,13cyclictest5300-21hddtemp_smartct12:20:171
2923599220,20cyclictest0-21swapper/109:49:101
2923599220,16cyclictest0-21swapper/110:36:511
2923099224,2cyclictest87350irq/126-enp1s0-12:26:050
2923099222,11cyclictest17191-21/usr/sbin/munin11:35:200
2923099221,8cyclictest59682cut09:55:160
123912221,17sleep112534-21users07:40:271
2924799215,8cyclictest87350irq/126-enp1s0-09:00:223
2924799215,4cyclictest20654-21smart_sda11:40:253
2924799214,9cyclictest12050irq/123-ahci[0010:45:013
2924799213,9cyclictest0-21swapper/311:00:163
2924799212,6cyclictest31402-21munin-run07:15:013
2924799211,4cyclictest0-21swapper/309:24:283
2924799211,13cyclictest0-21swapper/311:11:153
2924799211,13cyclictest0-21swapper/311:11:153
2924799211,13cyclictest0-21swapper/310:53:093
2924799211,12cyclictest29149-21/usr/sbin/munin09:35:173
2924799211,11cyclictest6818-21/usr/sbin/munin12:25:183
2924199210,9cyclictest0-21swapper/208:51:592
2924199210,9cyclictest0-21swapper/208:51:592
2923599215,4cyclictest14936-21perl10:15:151
2923599214,8cyclictest87650irq/129-enp1s0-10:00:271
2923599214,8cyclictest87650irq/129-enp1s0-10:00:271
2923599211,2cyclictest0-21swapper/108:31:271
2923599211,19cyclictest0-21swapper/107:23:551
2923599211,18cyclictest0-21swapper/108:57:311
2923599211,17cyclictest30693-21smartctl07:10:241
2923599211,13cyclictest0-21swapper/112:25:311
2923599211,12cyclictest0-21swapper/112:33:261
2923599211,10cyclictest11049-21ntpq10:05:191
2923599210,12cyclictest0-21swapper/110:11:521
2923099219,3cyclictest30728-21df12:05:130
2923099219,3cyclictest20361-21/usr/sbin/munin09:15:120
2923099219,2cyclictest24161-21needreboot10:35:190
2923099218,3cyclictest10530-21sed12:30:240
2923099217,3cyclictest87550irq/128-enp1s0-07:45:150
2923099216,4cyclictest0-21swapper/011:09:470
2923099216,14cyclictest0-21swapper/012:05:010
2923099215,5cyclictest0-21swapper/011:02:210
2923099214,14cyclictest11-21rcuc/009:05:160
2923099212,10cyclictest4354-21/usr/sbin/munin07:25:220
2923099211,12cyclictest0-21swapper/011:10:010
2923099211,12cyclictest0-21swapper/011:10:010
2923099211,11cyclictest0-21swapper/009:40:120
2923099211,10cyclictest15527-21head10:15:230
62512200,2sleep10-21swapper/108:40:231
2924799205,8cyclictest87650irq/129-enp1s0-08:35:193
2924799205,8cyclictest87350irq/126-enp1s0-11:55:203
2924799205,3cyclictest7105-21perl07:30:133
2924799205,3cyclictest12050irq/123-ahci[0011:40:023
2924799203,10cyclictest19211-21kworker/u8:209:56:063
2924799202,10cyclictest2152-21/usr/sbin/munin07:20:113
2924799202,10cyclictest10945-21/usr/sbin/munin07:40:193
2924799201,7cyclictest24960-21grep11:50:233
2924799201,7cyclictest18791-21sensors07:55:233
2924799201,6cyclictest23597-21cat11:50:023
2924799201,5cyclictest28911-21if_enp1s012:00:173
29247992012,4cyclictest26102-21dump-pmu-power08:15:023
2924799201,12cyclictest87350irq/126-enp1s0-09:25:213
2924799201,12cyclictest0-21swapper/312:23:253
2924799201,12cyclictest0-21swapper/312:23:253
2924799201,11cyclictest28528-21/usr/sbin/munin08:20:233
2924799201,11cyclictest13141-21/usr/sbin/munin07:45:263
2924799201,10cyclictest8746-21/usr/sbin/munin07:35:183
2924799201,10cyclictest7783-21/usr/sbin/munin10:00:183
2924799201,10cyclictest7783-21/usr/sbin/munin10:00:183
2923599205,4cyclictest6212-21/usr/sbin/munin11:10:191
2923599205,4cyclictest6212-21/usr/sbin/munin11:10:191
2923599201,9cyclictest1204-21head12:10:201
2923599201,4cyclictest29170-21grep09:35:111
2923599201,2cyclictest0-21swapper/111:05:471
2923599201,2cyclictest0-21swapper/108:00:121
2923599201,18cyclictest0-21swapper/111:55:241
2923599201,18cyclictest0-21swapper/107:53:151
2923599200,7cyclictest0-21swapper/110:22:571
2923599200,18cyclictest0-21swapper/108:25:241
2923099208,2cyclictest0-21swapper/012:20:160
2923099208,2cyclictest0-21swapper/012:20:160
2923099207,3cyclictest0-21swapper/010:41:540
2923099206,5cyclictest0-21swapper/009:24:280
2923099206,4cyclictest0-21swapper/011:59:130
2923099206,3cyclictest3489-21idleruntime08:35:160
2923099205,12cyclictest11-21rcuc/009:35:220
2923099203,14cyclictest11-21rcuc/008:05:270
2923099201,6cyclictest2332-21grep09:45:210
2923099201,6cyclictest21895-21missed_timers10:30:190
2923099201,11cyclictest29429-21/usr/sbin/munin07:10:240
2923099201,10cyclictest16577-21/usr/sbin/munin10:20:150
2923099201,10cyclictest11702-21sshd10:06:540
123942200,4sleep339-21rcuc/312:35:223
2924799196,3cyclictest87350irq/126-enp1s0-09:50:143
2924799194,8cyclictest12050irq/123-ahci[0011:30:243
2924799193,3cyclictest87350irq/126-enp1s0-09:10:013
2924799191,6cyclictest7166-21/usr/sbin/munin08:45:253
2924799191,6cyclictest15783-21timerwakeupswit10:15:243
29247991916,2cyclictest0-21swapper/308:50:123
29247991916,2cyclictest0-21swapper/308:50:123
2924799191,5cyclictest32710-21sed09:40:263
2924799191,5cyclictest12007-21cat10:10:023
2924799191,4cyclictest2547-21munin-run08:35:003
29247991913,3cyclictest32593-21/usr/sbin/munin12:10:253
2924799191,2cyclictest0-21swapper/310:47:433
2924799191,11cyclictest4951-21missed_timers11:05:203
2924799191,10cyclictest30735-21/usr/sbin/munin08:25:263
2924799191,10cyclictest0-21swapper/310:33:313
2924799191,10cyclictest0-21swapper/310:11:093
2924799191,10cyclictest0-21swapper/309:10:173
29235991916,2cyclictest0-21swapper/107:57:341
2923599191,16cyclictest0-21swapper/108:16:101
2923599191,15cyclictest5268-21memory07:25:191
2923599190,2cyclictest0-21swapper/108:37:381
2923099198,3cyclictest26951-21/usr/sbin/munin09:30:140
2923099198,3cyclictest10603-21/usr/sbin/munin11:20:250
2923099198,2cyclictest0-21swapper/011:25:130
2923099197,11cyclictest0-21swapper/008:20:020
2923099196,3cyclictest0-21swapper/011:49:060
2923099193,6cyclictest0-21swapper/012:11:120
2923099193,13cyclictest7-21ksoftirqd/011:40:200
2923099193,13cyclictest7-21ksoftirqd/007:50:240
2923099193,10cyclictest4031-21taskset07:23:570
2923099191,5cyclictest30504-21latency_hist08:25:010
100672191,2sleep30-21swapper/311:16:133
2924799183,8cyclictest0-21swapper/308:55:213
2924799181,7cyclictest0-21swapper/312:34:383
2924799181,5cyclictest808-21snmpd12:08:103
2924799181,5cyclictest24068-21sed10:35:173
2924799181,5cyclictest21073-21grep08:00:283
2924799181,5cyclictest14762-21latency_hist11:30:023
2924799181,4cyclictest824-21smart_sda10:55:253
2924799181,4cyclictest15683-21diskstats07:50:143
2924199184,3cyclictest0-21swapper/210:51:572
2924199183,3cyclictest0-21swapper/207:50:492
2923599181,16cyclictest0-21swapper/107:35:101
2923099186,5cyclictest827-21runrttasks10:51:060
2923099185,2cyclictest87350irq/126-enp1s0-12:16:070
2923099182,3cyclictest27912-21diskstats10:45:140
2923099182,15cyclictest0-21swapper/007:59:190
2923099181,9cyclictest0-21swapper/010:59:000
2923099181,6cyclictest0-21swapper/008:50:230
2923099181,6cyclictest0-21swapper/008:50:230
2923099181,3cyclictest30735-21/usr/sbin/munin08:25:160
29230991810,4cyclictest12923-21idleruntime-cro07:45:000
2923099180,9cyclictest0-21swapper/010:10:300
29247991711,3cyclictest5358-21fschecks_count08:40:133
29247991710,5cyclictest4992-21sed07:25:153
2923099174,3cyclictest933-21fschecks_count08:30:140
2923099173,12cyclictest11-21rcuc/007:35:170
2923099171,4cyclictest11560-21/usr/sbin/munin08:55:120
2923099171,13cyclictest8674-21tail08:45:260
2923099170,5cyclictest0-21swapper/009:00:270
29230991612,3cyclictest0-21swapper/008:40:090
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional