You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-30 - 18:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Aug 30, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
285762234161,23sleep20-21swapper/207:07:122
285032212162,37sleep10-21swapper/107:06:151
287532204128,25sleep00-21swapper/007:09:280
287932203167,24sleep30-21swapper/307:09:593
28888998177,3cyclictest0-21swapper/212:26:232
28888995753,3cyclictest0-21swapper/210:55:462
28888995250,1cyclictest0-21swapper/208:55:212
242442440,2sleep20-21swapper/209:25:012
28894993913,12cyclictest21243-21/usr/sbin/munin11:45:193
28894993723,8cyclictest3222-40dnf11:02:263
28894993712,12cyclictest19384-21/usr/sbin/munin08:00:133
2887999379,13cyclictest24177-21munin-run09:25:010
28894993514,11cyclictest17578-21latency_hist09:10:013
28894993510,7cyclictest827-21runrttasks11:08:123
28894993510,11cyclictest13518-21ntp_states11:25:223
2887999359,12cyclictest0-21swapper/011:00:010
28894993412,5cyclictest31281-21diskstats07:15:023
28894993412,12cyclictest28796-21/usr/sbin/munin09:35:173
2889499338,12cyclictest0-21swapper/311:55:203
28894993314,7cyclictest9185-40dnf10:02:163
28894993314,6cyclictest8786-21munin-run08:50:013
28894993311,12cyclictest23573-21cat09:20:273
28894993310,10cyclictest2385-21cat12:15:133
2889499325,16cyclictest304672cat08:25:133
28894993212,9cyclictest0-21swapper/309:30:013
2889499317,13cyclictest0-21swapper/308:15:183
2889499317,10cyclictest32765-21tr10:55:233
2889499316,14cyclictest17812-21/usr/sbin/munin09:10:243
28894993112,7cyclictest17636-21if_err_enp2s011:35:163
28894993111,11cyclictest29814-21sh12:05:013
28894993111,10cyclictest32272-21expr07:15:133
28894993110,18cyclictest16334-21grep10:20:113
28894993110,12cyclictest15157-21pmu-power10:15:223
28894993110,12cyclictest15157-21pmu-power10:15:213
2889499309,9cyclictest12883-21latency_hist12:40:013
2889499307,14cyclictest22719-21cut10:35:013
2889499307,11cyclictest6921-21ntpq11:10:213
28894993021,5cyclictest1165-21diskstats09:45:153
28894993013,8cyclictest8398-21/usr/sbin/munin07:35:243
28894993012,15cyclictest30487-21ntp_states10:50:213
28894993012,15cyclictest13132-21df07:45:143
28894993010,16cyclictest5883-21tail09:55:173
2889499299,8cyclictest632-21kernelversion12:10:183
2889499298,9cyclictest8677-21/usr/sbin/munin12:30:293
2889499298,8cyclictest20350-21df09:15:133
2889499298,13cyclictest27609-21netstat09:30:203
2889499298,13cyclictest23648-21kernelversion10:35:173
2889499296,14cyclictest0-21swapper/312:20:263
2889499295,10cyclictest0-21swapper/311:40:233
2889499291,8cyclictest0-21swapper/309:52:203
28894992918,7cyclictest31310-21smart_sda12:05:243
28894992911,15cyclictest13185-21munin-run09:00:013
304662280,2sleep10-21swapper/108:25:131
2889499289,16cyclictest6204-21/usr/sbin/munin07:30:163
2889499287,12cyclictest87350irq/126-enp1s0-11:30:143
2889499287,12cyclictest87350irq/126-enp1s0-11:30:143
2889499286,12cyclictest0-21swapper/311:20:013
2889499285,10cyclictest0-21swapper/311:20:223
28894992811,4cyclictest2191-21latency_hist08:35:023
2887999288,16cyclictest26996-21munin-run10:45:010
28894992719,5cyclictest16060-21ntp_states07:50:193
28894992716,8cyclictest3683-21smart_sda08:35:243
28894992716,8cyclictest15073-21taskset09:01:103
28894992711,5cyclictest87350irq/126-enp1s0-10:10:223
28894992710,14cyclictest810-21sshd11:53:163
2888499279,9cyclictest4948-21kworker/u8:210:01:321
2888499275,3cyclictest23-21ksoftirqd/111:37:241
2888499271,17cyclictest19405-21users09:10:271
2889499265,12cyclictest12050irq/123-ahci[0008:40:083
2889499265,11cyclictest87350irq/126-enp1s0-10:45:143
2889499262,12cyclictest10940-21df_inode07:40:153
28894992619,4cyclictest4696-21/usr/sbin/munin07:25:173
28894992619,4cyclictest24127-21df_inode08:10:133
28894992618,5cyclictest10899-21smart_sda10:05:243
28894992618,5cyclictest10274-21smart_sda08:50:243
28894992617,7cyclictest7157-21/usr/sbin/munin12:25:183
28894992617,7cyclictest28181-21/usr/sbin/munin08:20:113
28894992615,8cyclictest31352-21diskstats09:40:153
2887999263,3cyclictest24409-21users10:35:270
28894992517,5cyclictest2143-21df07:20:123
28894992517,5cyclictest19343-21/usr/sbin/munin10:25:183
28894992517,5cyclictest17534-21df_inode07:55:143
28894992517,5cyclictest17534-21df_inode07:55:143
28894992515,7cyclictest21593-21/usr/sbin/munin08:05:113
28894992512,10cyclictest25572-21tune2fs10:40:163
28884992514,3cyclictest9569-21perl08:50:151
28884992513,6cyclictest12378-21kworker/u8:008:36:451
2888899240,10cyclictest0-21swapper/209:40:502
2888499245,10cyclictest27598-21latency_hist12:00:021
2888899234,7cyclictest21071-40dnf08:01:252
28884992319,3cyclictest0-21swapper/107:12:581
2888499231,16cyclictest0-21swapper/112:32:351
2888499231,16cyclictest0-21swapper/108:47:011
2888499231,16cyclictest0-21swapper/108:44:191
2888499231,15cyclictest0-21swapper/109:05:181
28884992311,4cyclictest87650irq/129-enp1s0-09:35:001
2887999237,4cyclictest5253-21mailstats07:25:240
2887999235,15cyclictest18832-21basename11:40:010
2887999235,15cyclictest10376-21awk07:40:010
2887999233,16cyclictest12840-21munin-run12:40:010
2887999231,9cyclictest0-21swapper/009:35:200
2887999231,11cyclictest2086-21/usr/sbin/munin12:15:160
2888899222,11cyclictest4008-21df_inode11:05:132
2888899220,6cyclictest0-21swapper/212:35:212
2888499222,12cyclictest6554-21diskstats07:30:151
2888499221,4cyclictest1399-21perl09:45:151
2888499221,15cyclictest0-21swapper/111:35:011
2888499221,15cyclictest0-21swapper/111:35:011
2888499221,15cyclictest0-21swapper/108:22:441
2888499221,15cyclictest0-21swapper/108:01:011
2888499221,15cyclictest0-21swapper/107:51:331
2888499221,11cyclictest25313-21grep08:10:261
2888499221,11cyclictest21976-21sh09:20:011
2888499221,10cyclictest0-21swapper/107:45:131
2888499220,9cyclictest0-21swapper/107:27:141
2888499220,16cyclictest0-21swapper/109:00:181
2887999227,7cyclictest0-21swapper/011:24:230
2887999226,6cyclictest0-21swapper/010:50:500
2887999221,6cyclictest0-21swapper/010:27:230
2887999221,5cyclictest0-21swapper/009:46:430
2888899215,4cyclictest1005-21kworker/u8:111:30:222
2888899215,4cyclictest1005-21kworker/u8:111:30:222
2888899211,7cyclictest0-21swapper/211:50:192
2888899211,5cyclictest0-21swapper/210:07:332
2888899211,11cyclictest4008-21/usr/sbin/munin07:25:252
2888899211,11cyclictest17812-21/usr/sbin/munin09:10:192
2888899211,11cyclictest10042-21basename11:20:012
2888899211,10cyclictest29207-21grep07:10:132
2888899211,10cyclictest18429-21/usr/sbin/munin10:25:182
2888499219,6cyclictest1804-21kworker/u8:109:29:441
2888499219,3cyclictest87550irq/128-enp1s0-12:20:251
2888499218,4cyclictest127612sleep112:39:121
2888499218,11cyclictest3282-21kworker/u8:211:40:471
2888499217,5cyclictest18518-21wc10:25:131
2888499212,11cyclictest0-21swapper/112:15:241
2888499211,9cyclictest0-21swapper/111:06:301
28884992116,3cyclictest7840-21perf11:15:011
28884992115,3cyclictest316-21irqstats07:15:181
2888499211,18cyclictest0-21swapper/111:49:221
2888499211,15cyclictest0-21swapper/109:38:121
2888499211,14cyclictest0-21swapper/112:26:231
2888499211,11cyclictest28647-21irqstats12:00:181
28884992110,3cyclictest25260-21sshd11:53:161
2888499210,19cyclictest0-21swapper/111:00:371
2887999216,6cyclictest0-21swapper/011:01:110
2887999211,6cyclictest0-21swapper/011:06:090
2887999211,5cyclictest0-21swapper/010:20:510
2887999211,3cyclictest0-21swapper/011:13:250
2887999211,3cyclictest0-21swapper/009:12:460
2887999210,6cyclictest0-21swapper/012:26:400
2888899201,8cyclictest0-21swapper/210:45:422
2888899201,7cyclictest26383-21awk09:30:012
2888899201,7cyclictest2086-21/usr/sbin/munin12:15:252
2888899201,10cyclictest4466-21timerwakeupswit09:50:242
2888499204,14cyclictest4948-21kworker/u8:209:55:061
2888499201,9cyclictest0-21swapper/109:21:211
2888499201,6cyclictest0-21swapper/110:56:241
28884992016,3cyclictest0-21swapper/107:41:181
28884992015,2cyclictest221ktimersoftd/112:10:021
2888499201,3cyclictest0-21swapper/111:15:351
2888499201,3cyclictest0-21swapper/110:52:591
28884992013,2cyclictest0-21swapper/108:05:251
2888499201,17cyclictest0-21swapper/110:23:461
2888499201,15cyclictest0-21swapper/110:08:181
2888499201,14cyclictest0-21swapper/108:57:581
2888499201,11cyclictest26999-21awk10:45:011
2888499200,15cyclictest0-21swapper/109:40:171
2888499200,12cyclictest87550irq/128-enp1s0-08:33:371
2887999209,3cyclictest20012-21/usr/sbin/munin09:15:280
2887999202,3cyclictest0-21swapper/009:34:580
2887999202,15cyclictest6300-21ntp_states09:55:230
2887999201,11cyclictest0-21swapper/010:13:590
2887999201,10cyclictest0-21swapper/012:00:520
2887999201,10cyclictest0-21swapper/011:54:250
2887999200,5cyclictest0-21swapper/011:16:210
80752191,14sleep08195-21timerandwakeup08:45:220
77162190,4sleep230-21rcuc/210:00:122
316042190,3sleep011-21rcuc/009:40:160
2888899195,3cyclictest87650irq/129-enp1s0-10:36:152
2888899195,3cyclictest87650irq/129-enp1s0-08:25:202
2888899192,6cyclictest26764-21cpuspeed_turbos09:30:132
2888899191,9cyclictest12900-21ntp_states10:10:202
2888899191,9cyclictest0-21swapper/211:48:092
2888899191,8cyclictest13008-21perl11:25:152
2888899191,6cyclictest28440-21gltestperf12:00:162
2888899191,2cyclictest0-21swapper/211:58:022
2888899191,10cyclictest3222-40dnf11:02:242
2888899191,10cyclictest0-21swapper/209:16:592
2888499197,3cyclictest18619-21timerwakeupswit07:55:261
2888499197,3cyclictest18619-21timerwakeupswit07:55:261
2888499192,7cyclictest87350irq/126-enp1s0-11:20:141
28884991915,3cyclictest0-21swapper/110:45:101
2888499191,2cyclictest0-21swapper/107:21:061
28884991912,3cyclictest0-21swapper/108:19:021
2888499191,16cyclictest0-21swapper/112:11:131
2888499191,16cyclictest0-21swapper/110:37:531
2888499191,16cyclictest0-21swapper/107:36:291
2888499191,11cyclictest0-21swapper/111:25:241
2888499191,10cyclictest20959-21df10:30:141
2887999197,4cyclictest25215-21irqstats09:25:180
2887999195,3cyclictest0-21swapper/011:45:360
2887999191,8cyclictest0-21swapper/008:30:150
2887999191,4cyclictest17582-21diskmemload12:05:240
2887999191,10cyclictest17582-21diskmemload11:30:380
2887999191,10cyclictest17582-21diskmemload11:30:370
2887999191,10cyclictest14030-21/usr/sbin/munin10:15:220
2887999191,10cyclictest14030-21/usr/sbin/munin10:15:220
2887999191,10cyclictest0-21swapper/011:56:340
2887999191,10cyclictest0-21swapper/011:40:530
267962191,2sleep20-21swapper/210:42:442
2888899183,7cyclictest755-21rngd08:20:172
2888899183,4cyclictest17507-21sshd09:09:142
2888899181,9cyclictest20966-21df_inode10:30:142
2888899181,7cyclictest3778-21latency_hist07:25:022
2888899181,7cyclictest0-21swapper/212:12:292
2888899181,6cyclictest0-21swapper/211:20:312
2888899181,5cyclictest909-21sssd_nss09:35:202
2888899181,5cyclictest2580-21sshd09:47:152
2888899181,5cyclictest23787-21/usr/sbin/munin08:10:162
2888899181,5cyclictest14030-21/usr/sbin/munin10:15:142
2888899181,5cyclictest14030-21/usr/sbin/munin10:15:142
28888991812,3cyclictest19688-21gltestperf11:40:162
28888991811,4cyclictest0-21swapper/211:35:112
2888899181,11cyclictest0-21swapper/207:55:212
2888899181,11cyclictest0-21swapper/207:55:212
2888899181,10cyclictest0-21swapper/210:50:362
28884991812,2cyclictest0-21swapper/110:13:081
28884991812,2cyclictest0-21swapper/109:50:341
2888499181,12cyclictest0-21swapper/110:17:081
2888499181,12cyclictest0-21swapper/110:17:081
2887999187,8cyclictest20883-21unixbench_multi08:00:250
2887999186,3cyclictest0-21swapper/010:00:180
2887999184,6cyclictest32244-21/usr/sbin/munin12:10:160
2887999183,12cyclictest32367-21awk08:30:010
2887999182,6cyclictest0-21swapper/008:07:190
2887999182,3cyclictest0-21swapper/009:51:120
28879991812,3cyclictest4281-21/usr/sbin/munin12:20:120
28879991811,4cyclictest25755-21idleruntime-cro08:15:010
28879991811,4cyclictest17588-21munin-run09:10:010
2887999181,14cyclictest12794-21/usr/sbin/munin07:45:270
2887999181,10cyclictest0-21swapper/011:29:350
2888899176,3cyclictest87650irq/129-enp1s0-07:50:132
2888899172,4cyclictest31584-21sed12:05:272
2888899171,7cyclictest0-21swapper/210:20:222
2888899171,5cyclictest0-21swapper/212:20:462
2888899171,4cyclictest0-21swapper/209:55:162
28888991712,3cyclictest9904-21tail07:35:262
2888899171,15cyclictest0-21swapper/207:47:432
2888899171,14cyclictest9668-21strings12:30:202
2888899171,14cyclictest0-21swapper/209:00:122
2887999175,3cyclictest0-21swapper/010:05:320
2887999171,6cyclictest0-21swapper/008:42:140
2887999171,4cyclictest20625-21/usr/sbin/munin10:30:160
2887999171,3cyclictest27214-21/usr/sbin/munin10:45:140
28879991711,3cyclictest655-21ls07:15:220
2887999171,13cyclictest11557-21sshd08:55:140
2887999171,10cyclictest0-21swapper/012:31:130
2887999170,15cyclictest16974-21basename07:55:010
2888899161,6cyclictest27960-21ls08:20:012
2888899161,5cyclictest32320-21diskstats07:15:142
2888899161,2cyclictest0-21swapper/208:50:082
28888991611,3cyclictest6565-21forks07:30:152
28888991611,3cyclictest5863-21/usr/sbin/munin11:10:132
28888991611,3cyclictest5429-21interrupts08:40:192
28888991611,3cyclictest4396-21awk08:40:022
28888991611,3cyclictest11600-21netstat07:40:192
2887999168,5cyclictest909-21sssd_nss08:20:140
28879991612,3cyclictest0-21swapper/008:38:020
28879991612,3cyclictest0-21swapper/007:55:090
28879991612,3cyclictest0-21swapper/007:55:090
28879991611,3cyclictest29732-21sed07:10:170
28879991611,3cyclictest27245-21sensors_temp08:15:230
28879991610,3cyclictest9317-21expr08:50:120
28879991610,3cyclictest827-21runrttasks09:00:290
122032160,2sleep00-21swapper/007:40:300
2888899159,3cyclictest8401-21vmstat08:45:252
2888899152,10cyclictest32697-21grep08:30:132
28888991510,3cyclictest21609-21cpu08:05:122
2887999159,3cyclictest1940-21cpuspeed_turbos07:20:120
2887999151,5cyclictest6386-21cstates07:30:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional