You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-05 - 04:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Jul 05, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
12922236163,25sleep10-21swapper/119:07:491
12982235163,21sleep20-21swapper/219:07:542
12222212163,37sleep30-21swapper/319:06:563
12182202163,27sleep00-21swapper/019:06:520
839021740,3sleep2156999cyclictest00:15:242
2451521510,4sleep1156599cyclictest23:40:151
2451521510,4sleep1156599cyclictest23:40:151
1569999182,8cyclictest0-21swapper/223:34:332
212122640,11sleep2311ktimersoftd/222:20:012
47542580,4sleep24743-21munin-run00:10:022
165672580,3sleep00-21swapper/023:20:230
142272570,9sleep123-21ksoftirqd/123:15:231
1569994239,2cyclictest0-21swapper/222:51:022
1547994111,14cyclictest9137-21munin-run00:20:010
157599361,18cyclictest0-21swapper/323:50:493
156599364,9cyclictest22646-21munin-run21:10:011
154799367,13cyclictest15092-21munin-run23:20:000
157599341,16cyclictest0-21swapper/322:07:503
154799335,13cyclictest3743-21munin-run19:15:020
78792320,3sleep07882-21perf21:50:010
154799327,12cyclictest14472-21munin-run22:05:000
157599311,13cyclictest0-21swapper/300:10:113
157599310,12cyclictest0-21swapper/300:18:413
157599301,15cyclictest21109-21ntp_states19:50:223
154799293,13cyclictest31430-21munin-run21:30:010
157599288,8cyclictest827-21runrttasks23:36:073
157599281,12cyclictest0-21swapper/322:15:503
156599281,19cyclictest0-21swapper/121:51:461
154799289,16cyclictest8502-21munin-run23:05:010
157599271,13cyclictest0-21swapper/323:01:133
157599271,12cyclictest0-21swapper/300:34:283
157599271,11cyclictest16855-21/usr/sbin/munin23:20:253
157599270,12cyclictest0-21swapper/320:29:193
157599270,11cyclictest0-21swapper/322:10:243
157599270,11cyclictest0-21swapper/320:20:273
156599277,3cyclictest8016-21proc_pri19:20:221
154799272,12cyclictest2864-21awk20:25:020
154799272,12cyclictest19818-21munin-run19:50:000
239632260,2sleep30-21swapper/321:10:233
157599268,8cyclictest29643-21ls21:25:113
157599261,9cyclictest0-21swapper/321:21:473
157599261,12cyclictest0-21swapper/319:44:323
157599261,10cyclictest14544-21fschecks_time20:50:153
157599261,10cyclictest0-21swapper/322:38:313
156599261,17cyclictest0-21swapper/119:15:471
154799263,11cyclictest18877-21awk22:15:010
182972250,7sleep2311ktimersoftd/219:45:142
157599251,9cyclictest16945-21smartctl20:55:173
157599251,9cyclictest0-21swapper/320:36:233
157599251,9cyclictest0-21swapper/319:11:153
157599251,12cyclictest4221-21ntpq19:15:033
157599251,11cyclictest0-21swapper/321:41:023
157599251,11cyclictest0-21swapper/320:16:113
157599250,9cyclictest0-21swapper/319:55:163
157599250,13cyclictest0-21swapper/320:47:353
156599258,15cyclictest12006-21kworker/u8:022:30:171
156599251,19cyclictest0-21swapper/122:00:461
156599251,12cyclictest0-21swapper/123:12:381
154799251,12cyclictest21694-21taskset23:35:010
154799250,12cyclictest7265-21idleruntime-cro20:35:020
157599241,9cyclictest0-21swapper/321:15:513
157599241,14cyclictest0-21swapper/321:58:343
157599241,13cyclictest0-21swapper/323:46:593
157599241,13cyclictest0-21swapper/322:03:383
157599241,11cyclictest0-21swapper/300:24:543
157599241,10cyclictest0-21swapper/321:00:113
157599240,7cyclictest0-21swapper/321:05:193
156999243,7cyclictest6288-40dnf19:15:472
156599242,7cyclictest87650irq/129-enp1s0-22:25:141
156599242,7cyclictest87650irq/129-enp1s0-22:25:141
1565992412,10cyclictest13359-21kworker/u8:020:29:171
157599238,3cyclictest6380-21fschecks_time21:45:163
157599238,3cyclictest0-21swapper/321:30:193
157599231,9cyclictest0-21swapper/300:37:143
157599231,7cyclictest0-21swapper/320:42:093
1575992316,4cyclictest21406-21cut22:20:123
1575992311,2cyclictest0-21swapper/323:57:173
157599230,9cyclictest0-21swapper/323:05:173
157599230,11cyclictest0-21swapper/319:27:523
156599239,8cyclictest0-21swapper/119:42:001
156599231,9cyclictest83922grep00:15:231
156599231,16cyclictest0-21swapper/100:34:141
156599231,15cyclictest21-21rcuc/120:15:161
156599231,15cyclictest12549-21strings00:25:191
157599225,3cyclictest0-21swapper/322:44:073
157599221,9cyclictest0-21swapper/320:13:493
157599221,9cyclictest0-21swapper/319:21:593
157599221,12cyclictest0-21swapper/321:50:183
157599220,9cyclictest0-21swapper/320:07:413
156999225,4cyclictest0-21swapper/223:45:172
156599228,12cyclictest13359-21kworker/u8:019:59:171
156599223,16cyclictest11968-21grep20:45:121
156599223,11cyclictest19848-21tr23:30:131
156599222,9cyclictest32245-21head21:30:151
156599221,3cyclictest28518-21sensors_temp21:20:241
156599221,3cyclictest14230-21cat19:35:161
156599220,4cyclictest0-21swapper/122:22:211
156599220,3cyclictest0-21swapper/122:51:581
154799224,15cyclictest15742-21perf00:35:000
1547992215,4cyclictest8138-21gltestperf20:35:150
157599216,3cyclictest0-21swapper/319:49:543
157599212,8cyclictest0-21swapper/320:34:473
157599211,3cyclictest87550irq/128-enp1s0-22:30:153
157599210,9cyclictest0-21swapper/320:01:023
157599210,9cyclictest0-21swapper/320:01:023
157599210,9cyclictest0-21swapper/319:36:043
156999211,4cyclictest0-21swapper/222:55:382
156999211,12cyclictest0-21swapper/222:49:572
156999211,12cyclictest0-21swapper/220:21:412
156999210,4cyclictest29670-21kworker/u8:222:41:202
156599217,2cyclictest10699-21perf23:10:021
156599213,8cyclictest19683-21tail22:15:161
156599212,3cyclictest0-21swapper/120:33:161
156599212,14cyclictest0-21swapper/120:03:231
156599212,14cyclictest0-21swapper/120:03:231
1565992115,3cyclictest30474-21munin-run23:55:021
1565992113,3cyclictest0-21swapper/119:27:111
156599211,15cyclictest0-21swapper/122:48:251
156599211,15cyclictest0-21swapper/100:13:081
156599211,15cyclictest0-21swapper/100:05:071
156599211,14cyclictest0-21swapper/120:23:551
156599211,14cyclictest0-21swapper/119:10:411
156599211,11cyclictest16153-21cat20:55:021
1565992110,4cyclictest17569-21kworker/u8:122:37:161
156599210,15cyclictest0-21swapper/120:12:521
157599206,3cyclictest2308-21irqstats21:35:173
157599205,3cyclictest0-21swapper/300:25:283
157599204,3cyclictest0-21swapper/323:10:393
157599202,10cyclictest11254-21/usr/sbin/munin19:30:203
157599201,11cyclictest4320-21/usr/sbin/munin22:55:243
156999209,3cyclictest6854-21df23:00:142
156999203,4cyclictest0-21swapper/220:25:192
156999202,7cyclictest11975-21/usr/sbin/munin19:30:182
156999201,12cyclictest0-21swapper/200:00:222
156599209,3cyclictest25308-21expr21:15:141
156599208,3cyclictest21-21rcuc/121:36:181
156599206,2cyclictest24831-21turbostat.cron21:15:021
156599203,9cyclictest0-21swapper/121:00:201
156599202,10cyclictest18322-21irqstats23:25:181
1565992016,3cyclictest0-21swapper/121:29:241
1565992016,3cyclictest0-21swapper/119:47:291
1565992016,3cyclictest0-21swapper/100:00:251
156599201,17cyclictest30066-21taskset22:40:021
156599201,17cyclictest0-21swapper/123:35:291
156599201,16cyclictest12971-21taskset19:31:551
156599201,14cyclictest0-21swapper/119:54:161
156599201,12cyclictest0-21swapper/121:57:141
156599201,11cyclictest10478-21irqstats20:40:161
156599201,10cyclictest5213-21latency22:55:201
156599200,15cyclictest0-21swapper/120:08:111
156599200,11cyclictest0-21swapper/100:24:281
154799209,3cyclictest14695-21/usr/sbin/munin22:05:250
154799202,5cyclictest18245-1kworker/u9:222:47:260
154799201,6cyclictest0-21swapper/023:57:220
154799201,6cyclictest0-21swapper/021:57:200
154799201,6cyclictest0-21swapper/000:28:380
154799201,4cyclictest0-21swapper/000:21:300
154799201,11cyclictest22241-21/usr/sbin/munin19:55:150
59362190,2sleep00-21swapper/022:55:410
157599195,3cyclictest0-21swapper/322:25:583
157599195,3cyclictest0-21swapper/322:25:573
157599193,6cyclictest0-21swapper/300:08:203
157599193,3cyclictest0-21swapper/322:50:513
157599191,4cyclictest0-21swapper/323:40:523
157599191,4cyclictest0-21swapper/323:40:523
157599191,10cyclictest0-21swapper/323:27:463
156999195,3cyclictest87650irq/129-enp1s0-22:40:012
156999191,5cyclictest30-21rcuc/221:35:192
156999191,2cyclictest0-21swapper/221:55:112
156999191,11cyclictest0-21swapper/220:45:232
156599192,3cyclictest0-21swapper/121:48:501
156599191,16cyclictest0-21swapper/123:55:441
156599191,16cyclictest0-21swapper/123:21:511
156599191,16cyclictest0-21swapper/121:41:291
156599191,16cyclictest0-21swapper/100:36:281
156599191,10cyclictest6851-21df23:00:151
156599190,17cyclictest0-21swapper/122:07:481
154799199,3cyclictest4662-21/usr/sbin/munin19:15:150
154799198,3cyclictest4743-21munin-run00:10:010
154799197,4cyclictest30076-21find20:10:240
154799197,4cyclictest0-21swapper/022:36:170
154799196,4cyclictest0-21swapper/021:42:000
154799195,5cyclictest0-21swapper/022:34:260
154799191,8cyclictest4518-21kworker/0:000:36:310
154799191,7cyclictest185352grep21:00:120
154799191,6cyclictest0-21swapper/021:35:100
154799191,5cyclictest0-21swapper/021:22:040
154799191,15cyclictest31577-21unixbench_multi22:40:260
154799191,15cyclictest23699-21expr22:25:120
154799191,15cyclictest23699-21expr22:25:120
154799191,12cyclictest0-21swapper/021:54:210
154799191,10cyclictest26300-21/usr/sbin/munin23:45:160
154799190,2cyclictest87350irq/126-enp1s0-23:26:470
157599182,2cyclictest87650irq/129-enp1s0-00:03:553
157599181,6cyclictest20275-21cut23:30:153
157599181,11cyclictest0-21swapper/322:46:473
156999183,5cyclictest0-21swapper/223:17:182
156999182,6cyclictest0-21swapper/222:05:182
156999181,7cyclictest18024-21pmu-power22:10:222
156999181,7cyclictest12475-21memory00:25:192
156999181,6cyclictest0-21swapper/223:26:062
156999181,3cyclictest0-21swapper/223:10:302
1569991813,3cyclictest6478-21unixbench_multi00:10:262
156999181,11cyclictest0-21swapper/220:03:502
156999181,11cyclictest0-21swapper/220:03:502
156999181,11cyclictest0-21swapper/200:30:412
1565991815,2cyclictest0-21swapper/120:56:121
1565991814,3cyclictest0-21swapper/123:45:101
1565991812,2cyclictest0-21swapper/120:38:231
156599181,16cyclictest0-21swapper/122:14:111
154799188,2cyclictest19091-21/usr/sbin/munin22:15:260
154799185,7cyclictest19523-21kworker/u8:221:15:590
154799183,6cyclictest0-21swapper/023:13:330
154799182,3cyclictest0-21swapper/021:32:270
154799181,6cyclictest827-21runrttasks00:14:450
154799181,6cyclictest0-21swapper/023:37:320
154799181,3cyclictest0-21swapper/021:10:360
1547991811,4cyclictest4487-21sort20:25:240
154799181,11cyclictest0-21swapper/023:42:160
154799181,11cyclictest0-21swapper/023:42:160
154799181,10cyclictest0-21swapper/023:08:130
150762180,2sleep00-21swapper/019:35:460
86522171,2sleep00-21swapper/019:22:500
184472171,2sleep20-21swapper/221:00:042
156999171,9cyclictest87350irq/126-enp1s0-22:25:142
156999171,9cyclictest87350irq/126-enp1s0-22:25:142
156999171,7cyclictest32663-21latency_hist00:00:022
156999171,7cyclictest0-21swapper/200:35:122
156999171,6cyclictest0-21swapper/223:38:082
156999171,6cyclictest0-21swapper/221:46:042
156999171,6cyclictest0-21swapper/221:08:072
1569991712,3cyclictest24181-21grep21:10:262
1569991712,3cyclictest22408-21cstates19:55:132
156999171,15cyclictest0-21swapper/220:16:052
156999171,14cyclictest0-21swapper/220:50:062
1569991711,3cyclictest4758-21ntp_states21:40:222
1569991711,3cyclictest21577-21cat22:20:112
156999171,10cyclictest0-21swapper/200:20:172
1569991710,4cyclictest18227-21latency_hist21:00:002
156782171,2sleep00-21swapper/020:50:360
154799175,3cyclictest531-21cat20:19:570
154799173,8cyclictest0-21swapper/021:08:450
154799173,3cyclictest87550irq/128-enp1s0-19:40:190
154799171,9cyclictest0-21swapper/022:51:140
1547991712,3cyclictest21758-21grep22:20:150
1547991711,4cyclictest29422-21wc23:50:180
1547991710,4cyclictest0-21swapper/000:00:130
157599161,6cyclictest13803-21/usr/sbin/munin23:15:163
156999166,3cyclictest87650irq/129-enp1s0-21:20:152
156999165,3cyclictest22637-21diskmemload21:25:172
156999162,5cyclictest26222-21fschecks_time22:30:172
156999161,6cyclictest9577-21cat20:40:012
156999161,6cyclictest24451-21diskstats23:40:142
156999161,6cyclictest24451-21diskstats23:40:142
156999161,6cyclictest20957-21memory19:50:182
156999161,6cyclictest0-21swapper/223:20:152
156999161,6cyclictest0-21swapper/219:36:222
156999161,5cyclictest0-21swapper/220:05:242
156999161,3cyclictest0-21swapper/219:40:262
1569991612,3cyclictest0-21swapper/223:05:342
1569991612,3cyclictest0-21swapper/221:19:422
156999161,14cyclictest0-21swapper/223:53:042
156999161,14cyclictest0-21swapper/222:02:342
156999161,14cyclictest0-21swapper/219:14:342
1569991611,3cyclictest8257-21cstates21:50:112
1569991611,3cyclictest634-21wc21:30:262
1569991611,3cyclictest30990-21latency_hist20:15:012
1569991610,3cyclictest6964-21cut19:20:132
154799163,10cyclictest12771-21tail19:30:260
154799161,6cyclictest17045-21if_err_enp1s020:55:190
154799161,6cyclictest0-21swapper/020:05:190
1547991612,2cyclictest21606-21unixbench_singl19:50:250
1547991611,3cyclictest9066-21acpi19:25:110
1547991611,3cyclictest24679-21ls20:00:130
1547991611,3cyclictest24679-21ls20:00:130
108852161,2sleep20-21swapper/219:28:232
156999159,3cyclictest10668-21netstat20:40:182
156999151,5cyclictest6348-21ntp_states20:30:232
1547991510,3cyclictest10730-21cat20:40:200
154799150,13cyclictest0-21swapper/020:45:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional