You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-13 - 02:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Wed May 13, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22662233164,56sleep20-21swapper/219:07:172
21332220145,23sleep10-21swapper/119:05:351
22672206168,25sleep30-21swapper/319:07:173
22212203166,25sleep00-21swapper/019:06:430
3166821740,4sleep0255099cyclictest21:25:240
3166821740,4sleep0255099cyclictest21:25:240
3248721730,5sleep1255899cyclictest22:40:261
2577721270,4sleep1255899cyclictest20:00:131
117032610,4sleep121-21rcuc/121:55:151
67572590,2sleep00-21swapper/022:55:250
224082540,1sleep222411-21tr22:20:112
27302530,1sleep20-21swapper/200:00:252
122612430,1sleep10-21swapper/123:10:161
258452420,3sleep00-21swapper/021:15:000
65972370,4sleep2256699cyclictest19:15:202
255099357,13cyclictest19859-21munin-run22:15:010
255099335,13cyclictest586-21munin-run22:45:010
2566993228,3cyclictest0-21swapper/200:33:132
256699311,21cyclictest0-21swapper/220:47:172
257099301,13cyclictest0-21swapper/323:02:233
256699301,10cyclictest0-21swapper/223:18:462
257099293,10cyclictest10075-21strings21:50:193
2550992911,15cyclictest25204-21munin-run20:00:010
257099281,11cyclictest0-21swapper/320:11:383
255099289,16cyclictest28647-21munin-run22:35:010
255099289,16cyclictest28647-21munin-run22:35:010
147782280,4sleep011-21rcuc/000:30:130
257099271,18cyclictest0-21swapper/323:37:363
257099271,11cyclictest0-21swapper/321:57:163
257099271,10cyclictest0-21swapper/322:11:003
257099270,13cyclictest0-21swapper/320:40:273
257099270,12cyclictest0-21swapper/323:55:203
257099270,12cyclictest0-21swapper/300:22:163
257099270,11cyclictest0-21swapper/322:50:223
257099261,9cyclictest0-21swapper/323:07:033
257099261,11cyclictest0-21swapper/319:50:183
77122250,3sleep07715-21kernelversion21:45:190
257099251,11cyclictest0-21swapper/323:49:403
257099251,10cyclictest0-21swapper/323:33:403
257099250,14cyclictest0-21swapper/323:25:163
257099250,12cyclictest0-21swapper/322:42:353
257099250,11cyclictest0-21swapper/323:40:043
257099250,11cyclictest0-21swapper/323:40:043
257099250,10cyclictest0-21swapper/320:35:153
255899255,9cyclictest0-21swapper/122:18:391
255899252,12cyclictest15468-21munin-run22:05:021
255899251,17cyclictest21350-21cstates19:50:131
255099251,9cyclictest3290-21kworker/0:000:15:370
255099251,12cyclictest26453-21munin-run22:30:010
257099241,9cyclictest0-21swapper/321:48:113
257099241,9cyclictest0-21swapper/300:05:123
257099241,11cyclictest0-21swapper/322:49:103
257099241,10cyclictest0-21swapper/300:11:363
257099240,11cyclictest0-21swapper/322:59:033
257099240,11cyclictest0-21swapper/321:33:233
257099240,10cyclictest0-21swapper/323:19:003
257099240,10cyclictest0-21swapper/319:55:223
255899241,9cyclictest0-21swapper/123:55:491
255899241,3cyclictest13583-21ntp_kernel_pll_00:25:221
255099242,8cyclictest7861-21kworker/0:200:14:520
255099241,12cyclictest8248-21munin-run20:35:000
257099231,9cyclictest0-21swapper/320:19:383
257099231,9cyclictest0-21swapper/320:19:383
257099231,9cyclictest0-21swapper/319:34:193
257099231,10cyclictest0-21swapper/323:55:003
257099231,10cyclictest0-21swapper/322:16:523
257099230,16cyclictest0-21swapper/320:25:283
257099230,10cyclictest0-21swapper/322:21:473
257099230,10cyclictest0-21swapper/321:27:033
257099230,10cyclictest0-21swapper/321:27:033
257099230,10cyclictest0-21swapper/300:39:313
2558992314,2cyclictest4902-21/usr/sbin/munin00:05:231
255899231,17cyclictest0-21swapper/123:02:151
255899231,16cyclictest0-21swapper/123:40:141
255899231,16cyclictest0-21swapper/123:40:141
255899231,16cyclictest0-21swapper/120:36:201
255899231,16cyclictest0-21swapper/119:49:191
255099232,9cyclictest24509-21kworker/0:123:54:250
2550992310,3cyclictest18418-21kworker/0:120:11:310
223572230,5sleep2256699cyclictest21:05:162
257099221,10cyclictest0-21swapper/319:25:133
257099220,11cyclictest0-21swapper/319:35:263
257099220,10cyclictest0-21swapper/321:22:193
256699229,5cyclictest87350irq/126-enp1s0-23:50:132
256699223,9cyclictest311ktimersoftd/223:20:162
256699221,8cyclictest10048-21/usr/sbin/munin19:25:142
256699221,7cyclictest12050irq/123-ahci[0021:05:012
256699221,7cyclictest0-21swapper/222:05:152
255899224,9cyclictest0-21swapper/100:35:141
255899223,3cyclictest0-21swapper/121:26:041
255899223,3cyclictest0-21swapper/121:26:041
2558992216,1cyclictest0-21swapper/100:33:301
255899221,4cyclictest14637-21grep23:15:161
255899221,18cyclictest5036-21df_inode21:40:131
255899221,10cyclictest12631-21latency_hist20:45:021
255899220,5cyclictest0-21swapper/100:18:051
255099229,2cyclictest7298-21kworker/0:019:20:070
255099227,6cyclictest0-21swapper/022:52:440
255099226,5cyclictest0-21swapper/023:10:180
255099222,8cyclictest30642-21kworker/0:222:38:090
255099221,9cyclictest16001-21kworker/0:323:30:220
257099217,8cyclictest0-21swapper/300:18:563
257099211,9cyclictest0-21swapper/322:39:343
257099210,9cyclictest0-21swapper/321:14:473
257099210,8cyclictest0-21swapper/323:20:233
257099210,13cyclictest0-21swapper/300:00:243
256699216,7cyclictest12050irq/123-ahci[0022:19:552
256699214,8cyclictest311ktimersoftd/223:55:252
256699214,8cyclictest12050irq/123-ahci[0023:00:022
2566992110,3cyclictest87350irq/126-enp1s0-21:20:212
256699210,12cyclictest32-21ksoftirqd/222:10:242
256699210,10cyclictest311ktimersoftd/222:30:112
256699210,10cyclictest311ktimersoftd/222:30:112
2558992113,3cyclictest0-21swapper/120:17:101
2558992113,3cyclictest0-21swapper/120:17:101
2558992113,3cyclictest0-21swapper/120:07:591
2558992113,2cyclictest0-21swapper/119:55:261
255899211,2cyclictest28033-21munin-run21:20:001
255899211,18cyclictest0-21swapper/122:35:121
255899211,15cyclictest0-21swapper/119:16:461
255899211,15cyclictest0-21swapper/119:12:431
255899211,14cyclictest0-21swapper/120:50:281
255899211,12cyclictest0-21swapper/122:54:461
255899211,11cyclictest9690-21gltestperf21:50:161
255899211,10cyclictest0-21swapper/122:10:461
2558992110,3cyclictest17660-21munin-run22:10:011
255899210,16cyclictest0-21swapper/120:13:241
255899210,16cyclictest0-21swapper/119:27:391
255099219,2cyclictest11141-21kworker/0:219:19:100
255099218,2cyclictest14105-21kworker/0:219:40:570
255099217,2cyclictest11756-21kworker/0:019:28:450
255099214,3cyclictest1298-21kworker/0:220:26:280
255099212,9cyclictest10112-21kworker/0:100:24:420
255099212,8cyclictest3537-21kworker/0:120:50:380
255099212,16cyclictest16951-21cut23:20:150
255099212,11cyclictest21661-21/usr/sbin/munin21:05:180
255099211,9cyclictest18935-21kworker/0:321:03:200
255099211,9cyclictest14105-21kworker/0:219:34:210
255099211,8cyclictest19748-21kworker/0:022:16:410
255099211,8cyclictest1298-21kworker/0:220:21:090
255099211,12cyclictest0-21swapper/023:19:270
255099211,11cyclictest14552-21ntp_states22:00:230
255099211,11cyclictest12816-21sed21:55:260
252642210,6sleep011-21rcuc/023:40:130
252642210,6sleep011-21rcuc/023:40:130
92632201,15sleep09347-21ls20:35:160
257099201,5cyclictest27052-21cut21:15:203
257099201,11cyclictest11913-21/usr/sbin/munin23:10:163
257099201,10cyclictest0-21swapper/322:08:523
257099200,9cyclictest0-21swapper/322:26:223
257099200,11cyclictest0-21swapper/322:31:153
257099200,11cyclictest0-21swapper/322:31:153
256699206,3cyclictest87650irq/129-enp1s0-21:18:592
256699205,8cyclictest87350irq/126-enp1s0-22:00:122
256699203,5cyclictest0-21swapper/221:40:162
256699203,2cyclictest87550irq/128-enp1s0-00:05:142
256699201,7cyclictest0-21swapper/222:35:222
255899209,3cyclictest14832-21munin-run20:50:021
255899207,11cyclictest32095-21kworker/u8:222:45:161
255899206,4cyclictest0-21swapper/123:53:501
255899205,4cyclictest0-21swapper/100:11:021
255899201,5cyclictest0-21swapper/100:21:411
2558992012,3cyclictest0-21swapper/120:31:061
255899201,18cyclictest0-21swapper/123:22:531
255899201,15cyclictest0-21swapper/123:47:311
255899201,14cyclictest0-21swapper/119:43:321
255899201,14cyclictest0-21swapper/119:38:041
255899201,14cyclictest0-21swapper/119:31:251
255899201,14cyclictest0-21swapper/119:24:471
255899201,14cyclictest0-21swapper/100:01:541
255899201,10cyclictest2377-21date21:35:011
255099208,2cyclictest27326-21kworker/0:220:05:050
255099207,2cyclictest25003-21kworker/0:020:01:130
255099206,2cyclictest29560-21kworker/0:020:18:570
255099206,2cyclictest29560-21kworker/0:020:18:570
255099206,2cyclictest18418-21kworker/0:119:53:250
255099205,2cyclictest87350irq/126-enp1s0-21:36:090
255099202,10cyclictest2761-21/usr/sbin/munin19:10:130
255099202,10cyclictest18502-21/usr/sbin/munin23:25:250
255099201,9cyclictest24702-21kworker/0:223:39:020
255099201,8cyclictest26049-21kworker/0:321:23:250
255099201,8cyclictest14105-21kworker/0:219:35:130
255099201,11cyclictest27544-21cstates23:45:130
255099201,11cyclictest26060-21/usr/sbin/munin21:15:120
255099201,11cyclictest1515-21/usr/sbin/munin00:00:260
255099200,9cyclictest30556-21kworker/0:223:55:130
92572191,2sleep00-21swapper/023:02:120
73372190,5sleep2256699cyclictest20:30:202
257099195,3cyclictest0-21swapper/320:09:363
257099191,8cyclictest3485-21strings21:35:193
257099191,15cyclictest19749-21memory19:45:173
256699199,2cyclictest311ktimersoftd/219:55:122
256699198,3cyclictest87650irq/129-enp1s0-23:45:012
256699198,3cyclictest87650irq/129-enp1s0-23:45:012
256699194,3cyclictest3997-21sendmail_mailqu19:10:232
256699193,4cyclictest32-21ksoftirqd/221:10:152
256699193,4cyclictest27552-21expr23:45:132
256699193,10cyclictest18618-21chrt00:36:252
256699191,11cyclictest0-21swapper/221:55:172
256699191,10cyclictest0-21swapper/222:40:192
255899198,3cyclictest18234-21needreboot20:55:191
255899191,8cyclictest0-21swapper/122:31:051
255899191,8cyclictest0-21swapper/122:31:051
255899191,6cyclictest0-21swapper/122:29:191
2558991913,3cyclictest2604-21cpuspeed_turbos21:35:121
255899191,2cyclictest23632-21diskmemload23:28:251
255899191,17cyclictest0-21swapper/122:58:051
255899191,17cyclictest0-21swapper/121:20:141
255899191,16cyclictest0-21swapper/123:05:411
255899191,16cyclictest0-21swapper/121:06:381
255899191,14cyclictest0-21swapper/120:24:311
255899191,10cyclictest6122-21cat20:30:021
2558991910,6cyclictest20701-21/usr/sbin/munin23:30:241
255899190,17cyclictest0-21swapper/123:35:111
255899190,17cyclictest0-21swapper/122:22:091
255099198,4cyclictest808-21snmpd20:43:390
255099196,2cyclictest3537-21kworker/0:120:48:400
255099196,2cyclictest3290-21kworker/0:000:06:200
255099196,2cyclictest16704-21kworker/0:220:56:020
255099193,3cyclictest87350irq/126-enp1s0-21:52:530
255099192,8cyclictest8532-21kworker/0:122:06:010
255099192,5cyclictest2221-40dnf21:34:420
255099191,8cyclictest4228-21kworker/0:221:43:390
255099191,12cyclictest0-21swapper/022:20:280
255099191,11cyclictest0-21swapper/000:25:200
255099191,10cyclictest10047-21df23:05:120
255099190,6cyclictest0-21swapper/000:36:250
188212190,2sleep20-21swapper/223:25:142
98102180,2sleep20-21swapper/200:16:112
26782181,3sleep00-21swapper/022:47:320
257099183,5cyclictest19572-21grep21:00:133
257099182,5cyclictest16755-21sshd20:51:403
257099181,11cyclictest0-21swapper/320:02:343
256699189,2cyclictest87650irq/129-enp1s0-20:00:232
256699189,2cyclictest87350irq/126-enp1s0-00:10:172
256699188,3cyclictest3854-21awk20:25:012
256699187,2cyclictest23632-21diskmemload23:35:192
256699183,7cyclictest0-21swapper/221:50:022
256699183,3cyclictest22913-21runrttasks19:53:592
256699182,7cyclictest87350irq/126-enp1s0-20:25:132
256699182,6cyclictest0-21swapper/223:05:062
256699182,13cyclictest8987-21head19:20:222
256699181,9cyclictest30-21rcuc/221:31:182
256699181,6cyclictest0-21swapper/223:30:272
256699181,4cyclictest0-21swapper/222:27:232
256699181,15cyclictest0-21swapper/200:25:562
256699181,10cyclictest0-21swapper/223:00:202
256699181,10cyclictest0-21swapper/221:25:542
256699181,10cyclictest0-21swapper/221:25:532
256699180,9cyclictest2491-21/usr/sbin/munin21:35:222
256699180,5cyclictest30-21rcuc/220:55:192
255899186,3cyclictest8061-21ps21:45:241
2558991815,2cyclictest0-21swapper/121:11:341
255899181,16cyclictest0-21swapper/121:02:351
255099181,10cyclictest0-21swapper/019:46:030
257099179,5cyclictest763-21dbus-daemon20:34:223
257099176,3cyclictest14992-21tail22:00:273
257099171,7cyclictest147802cpu00:30:123
257099171,2cyclictest87350irq/126-enp1s0-20:22:393
2570991712,4cyclictest20127-1kworker/3:2H00:25:063
2570991712,3cyclictest5654-21/usr/sbin/munin19:15:133
257099171,10cyclictest0-21swapper/320:45:153
256699175,4cyclictest0-21swapper/220:15:392
256699175,4cyclictest0-21swapper/220:15:392
256699173,6cyclictest30-21rcuc/222:45:142
256699173,11cyclictest7444-21tune2fs21:45:152
256699172,9cyclictest0-21swapper/222:50:212
256699172,6cyclictest827-21runrttasks19:40:172
256699172,3cyclictest0-21swapper/220:51:132
256699171,9cyclictest30-21rcuc/220:05:122
256699171,7cyclictest0-21swapper/223:13:152
256699171,4cyclictest0-21swapper/219:35:232
256699171,3cyclictest0-21swapper/220:36:432
2566991712,3cyclictest30465-21sed20:10:182
233812170,2sleep30-21swapper/321:06:583
257099161,5cyclictest16981-21diskstats19:40:113
257099161,5cyclictest0-21swapper/319:10:133
2570991611,3cyclictest4692-21/usr/sbin/munin21:40:243
2570991611,3cyclictest17619-21awk20:55:143
256699164,3cyclictest12251-21cat19:30:112
256699163,2cyclictest87550irq/128-enp1s0-19:45:172
2566991610,4cyclictest11011-21cut00:20:152
2570991510,3cyclictest8058-21expr19:20:133
256699150,14cyclictest0-21swapper/220:41:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional