You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-30 - 02:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Sun Aug 30, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
147372244169,24sleep20-21swapper/219:06:272
150092229154,24sleep30-21swapper/319:09:573
146872203167,24sleep00-21swapper/019:05:480
147412202165,24sleep10-21swapper/119:06:291
327921800,5sleep21510699cyclictest22:20:182
313421640,4sleep01509699cyclictest22:20:160
85321400,5sleep21510699cyclictest21:00:242
85321400,5sleep21510699cyclictest21:00:242
2893821220,1sleep20-21swapper/222:05:172
151069910299,2cyclictest0-21swapper/221:37:172
147882540,2sleep00-21swapper/000:00:170
253542490,4sleep011-21rcuc/000:25:130
297962460,1sleep10-21swapper/100:35:131
15096993913,12cyclictest28956-21munin-run19:40:010
1509699386,15cyclictest15417-21munin-run22:50:010
1511399350,21cyclictest0-21swapper/322:56:223
1509699356,13cyclictest14797-21munin-run21:35:020
1509699356,13cyclictest13830-40updatedb00:00:010
1511399341,26cyclictest0-21swapper/323:28:583
1510199328,4cyclictest827-21runrttasks00:06:481
15101993215,3cyclictest9284-21grep20:05:261
15113993119,3cyclictest29038-21irqstats22:05:183
1511399311,19cyclictest0-21swapper/319:52:523
1511399311,12cyclictest7149-21memory21:15:183
1509699313,13cyclictest8824-21munin-run22:35:010
190502300,5sleep11510199cyclictest00:10:141
15113993015,8cyclictest0-21swapper/322:13:503
1511399301,12cyclictest0-21swapper/322:03:223
15113993011,11cyclictest0-21swapper/323:10:343
15101993025,2cyclictest261982sleep120:45:181
15101993013,4cyclictest5716-21ntp_states22:25:221
1510199301,17cyclictest29922-21if_enp1s019:40:171
1509699302,13cyclictest23595-21munin-run21:55:010
311992290,1sleep00-21swapper/019:45:010
15113992916,7cyclictest0-21swapper/322:15:303
1511399291,13cyclictest0-21swapper/323:33:103
1511399291,10cyclictest0-21swapper/323:51:263
1510699292,14cyclictest5066-40dnf19:56:102
1509699296,11cyclictest26425-21idleruntime-cro23:15:010
15113992813,8cyclictest0-21swapper/322:34:463
1511399281,14cyclictest0-21swapper/322:41:023
1511399281,13cyclictest0-21swapper/323:04:543
1511399281,11cyclictest0-21swapper/319:45:163
1511399280,11cyclictest0-21swapper/321:52:063
1510199282,10cyclictest13829-21munin-run00:00:011
106832280,2sleep10-21swapper/123:50:191
15113992716,3cyclictest0-21swapper/300:27:383
1511399271,18cyclictest0-21swapper/319:36:183
1511399270,13cyclictest0-21swapper/323:05:463
1510199274,20cyclictest2149-21switchtime23:30:221
1510199272,21cyclictest2236-21munin-run22:20:011
1510199272,15cyclictest0-21swapper/100:28:271
1509699274,11cyclictest563-21perf23:30:010
1509699271,13cyclictest11632-21munin-run23:55:010
15113992614,3cyclictest0-21swapper/322:37:583
1510699261,3cyclictest0-21swapper/222:29:532
1510199262,21cyclictest16636-21/usr/sbin/munin19:10:241
1510199261,22cyclictest579-21systemd-journal23:25:011
1510199261,16cyclictest0-21swapper/121:56:471
1511399259,8cyclictest0-21swapper/300:08:423
1511399259,3cyclictest0-21swapper/320:43:503
1511399258,11cyclictest0-21swapper/322:45:343
1511399251,13cyclictest0-21swapper/300:04:343
1511399251,11cyclictest0-21swapper/323:22:023
1511399251,10cyclictest0-21swapper/319:34:463
1511399250,12cyclictest0-21swapper/320:50:223
1511399250,12cyclictest0-21swapper/320:14:583
1510199252,2cyclictest538-21latency_hist23:30:011
1510199251,21cyclictest28727-21cat23:20:011
1510199251,21cyclictest15350-21expr21:35:131
1510199251,21cyclictest14414-21users21:30:261
1510199251,21cyclictest11545-21missed_timers21:25:191
15101992510,3cyclictest9246-21irqstats21:20:181
15101992510,3cyclictest28411-21irqstats20:50:181
101182251,3sleep2808-21snmpd21:21:322
1511399249,10cyclictest0-21swapper/323:55:103
1511399248,3cyclictest0-21swapper/320:49:273
15113992413,3cyclictest0-21swapper/322:23:343
1511399241,17cyclictest0-21swapper/323:40:543
1511399241,11cyclictest0-21swapper/321:10:023
1511399240,12cyclictest0-21swapper/300:17:463
1510199245,6cyclictest19589-40dnf22:57:251
15101992418,3cyclictest23500-21memory19:25:181
1510199241,3cyclictest19592-21timerandwakeup19:15:261
1510199241,20cyclictest26394-21cat23:14:561
1510199241,20cyclictest25701-21memory19:30:201
15101992411,3cyclictest0-21swapper/122:20:211
1510199241,11cyclictest6335-21ntp_states23:40:211
1510199241,11cyclictest31933-21perl19:45:151
1510199241,11cyclictest20558-21tune2fs23:00:141
1510199241,11cyclictest1-21systemd21:00:011
1509699248,14cyclictest24864-21perf00:25:010
1509699241,11cyclictest20473-21perf00:15:010
1511399238,8cyclictest0-21swapper/300:36:103
1511399238,3cyclictest0-21swapper/322:25:463
1511399238,10cyclictest0-21swapper/321:56:063
1511399237,11cyclictest0-21swapper/323:37:223
1511399236,3cyclictest0-21swapper/321:25:503
1511399236,11cyclictest0-21swapper/321:21:223
15113992313,3cyclictest0-21swapper/300:33:503
15113992313,3cyclictest0-21swapper/300:33:503
1511399231,11cyclictest0-21swapper/322:52:183
1511399231,10cyclictest0-21swapper/321:42:023
15113992310,8cyclictest0-21swapper/319:24:543
1510699236,3cyclictest0-21swapper/221:58:522
1510199239,3cyclictest28045-21ntp_states19:35:201
1510199232,19cyclictest0-21swapper/120:30:101
1510199231,3cyclictest16377-21crond20:25:011
15101992312,3cyclictest0-21swapper/122:06:491
1510199231,19cyclictest12473-21sendmail_mailqu22:40:251
1510199231,11cyclictest8158-21hwlatdetect23:45:171
1510199231,11cyclictest22593-21diskstats23:05:141
1510199231,11cyclictest20526-21open_inodes21:45:231
1510199231,10cyclictest4044-21munin-node21:10:121
1510199231,10cyclictest31589-21ps22:10:231
1510199230,21cyclictest0-21swapper/121:54:351
1510199230,20cyclictest15501-21sed00:00:271
1510199230,12cyclictest0-21swapper/122:01:331
1509699231,5cyclictest0-21swapper/022:57:250
1509699230,8cyclictest8517-21ntpq23:45:210
1511399227,3cyclictest0-21swapper/320:15:343
1511399226,3cyclictest0-21swapper/321:35:103
1511399221,12cyclictest0-21swapper/323:45:263
1510699223,9cyclictest2675550kworker/u8:200:39:332
1510199228,3cyclictest22777-21sed00:20:011
1510199226,5cyclictest18753-21sed21:40:261
1510199222,3cyclictest0-21swapper/120:29:151
1510199221,19cyclictest0-21swapper/122:51:011
1510199221,19cyclictest0-21swapper/122:45:531
1510199221,18cyclictest10722-21idleruntime20:10:161
1510199221,10cyclictest9812-21if_err_enp1s022:35:171
1510199221,10cyclictest1742-21cat21:05:011
1510199221,10cyclictest1742-21cat21:05:011
1509699227,6cyclictest0-21swapper/023:00:360
1509699221,12cyclictest15302-21/usr/sbin/munin19:10:170
24262211,16sleep12540-21wc21:05:171
1511399219,3cyclictest0-21swapper/319:27:103
1511399217,8cyclictest0-21swapper/320:01:583
1511399216,3cyclictest0-21swapper/320:27:583
1511399214,9cyclictest0-21swapper/319:44:583
1511399214,8cyclictest0-21swapper/323:17:223
1511399211,14cyclictest0-21swapper/320:06:463
1511399211,13cyclictest0-21swapper/321:05:143
1511399211,11cyclictest18878-21fschecks_count00:10:133
1510699211,6cyclictest22658-21munin-run00:20:012
1510699211,12cyclictest0-21swapper/200:21:082
1510699211,11cyclictest1198-21/usr/sbin/munin19:50:202
1510199218,3cyclictest2443-21gzip19:50:231
1510199211,7cyclictest0-21swapper/120:38:341
15101992113,5cyclictest24118-21fgrep00:20:221
1510199211,18cyclictest0-21swapper/122:33:101
1510199211,18cyclictest0-21swapper/120:16:441
1510199210,19cyclictest0-21swapper/120:43:501
1510199210,19cyclictest0-21swapper/119:55:441
1509699215,3cyclictest0-21swapper/020:57:210
1509699214,15cyclictest19207-21perf21:45:010
1509699211,12cyclictest11244-21/usr/sbin/munin22:40:160
139602210,4sleep339-21rcuc/321:30:233
1511399207,3cyclictest0-21swapper/321:00:113
1511399207,3cyclictest0-21swapper/321:00:103
1511399206,3cyclictest0-21swapper/321:46:543
1511399205,8cyclictest0-21swapper/300:20:583
1511399203,3cyclictest0-21swapper/320:33:423
1511399201,3cyclictest19665-21sed19:15:263
1511399201,12cyclictest0-21swapper/320:37:103
1511399200,13cyclictest0-21swapper/319:12:423
1510699209,3cyclictest737-21taskset19:48:002
1510699205,7cyclictest87650irq/129-enp1s0-23:10:172
1510699204,4cyclictest29981-21if_err_enp2s019:40:172
1510699203,8cyclictest22511-21kworker/2:221:50:252
1510699201,6cyclictest31770-21pluginstate21:00:012
1510699201,6cyclictest31706-21hddtemp_smartct23:25:172
1510699201,6cyclictest25814-21if_enp1s000:25:172
1510699201,6cyclictest18829-21grep22:55:202
1510699201,6cyclictest15389-21ls19:10:122
1510699201,5cyclictest1037-21expr23:30:112
15106992013,4cyclictest7468-21find21:15:232
1510699201,12cyclictest0-21swapper/223:16:462
1510699201,12cyclictest0-21swapper/221:49:242
1510699201,12cyclictest0-21swapper/200:00:452
1510699201,11cyclictest18799-21grep19:15:162
1510699201,11cyclictest0-21swapper/222:18:322
1510699201,11cyclictest0-21swapper/200:13:372
1510699201,10cyclictest8452-21strings23:45:202
1510199201,3cyclictest0-21swapper/123:36:351
1510199201,18cyclictest0-21swapper/121:15:371
1510199201,10cyclictest0-21swapper/119:20:141
1510199200,19cyclictest0-21swapper/100:31:051
1510199200,19cyclictest0-21swapper/100:31:051
1509699209,3cyclictest18196-21/usr/sbin/munin19:15:130
1509699207,4cyclictest0-21swapper/023:41:060
1509699207,4cyclictest0-21swapper/023:16:480
1509699201,11cyclictest866-21sshd19:49:080
1509699201,11cyclictest4654-21/usr/sbin/munin22:25:260
1509699201,11cyclictest15365-21df21:35:140
41512190,3sleep04155-21cpuspeed_turbos21:10:130
1511399195,6cyclictest29787-21sshd20:55:073
1511399195,3cyclictest0-21swapper/320:22:143
1511399192,4cyclictest0-21swapper/319:56:103
1510699195,3cyclictest87350irq/126-enp1s0-23:35:252
1510699193,9cyclictest0-21swapper/221:13:562
1510699192,9cyclictest30808-21awk23:25:012
1510699192,10cyclictest30-21rcuc/222:00:212
1510699191,9cyclictest0-21swapper/223:40:132
1510699191,7cyclictest18220-21grep21:40:202
1510699191,6cyclictest7493-21gltestperf22:30:162
1510699191,6cyclictest23132-21sed23:05:182
1510699191,6cyclictest13974-21awk21:30:232
1510699191,5cyclictest21382-21strings19:20:192
1510699191,11cyclictest0-21swapper/222:35:192
1510699191,11cyclictest0-21swapper/220:50:202
1510699191,10cyclictest21002-21/usr/sbin/munin20:35:182
1510699190,10cyclictest13566-21cpuspeed_turbos22:45:142
1510199190,2cyclictest0-21swapper/120:01:341
1509699198,3cyclictest25398-21/usr/sbin/munin20:45:250
1509699196,6cyclictest0-21swapper/023:20:150
1509699192,3cyclictest87350irq/126-enp1s0-20:22:220
1509699191,9cyclictest0-21swapper/000:18:020
1509699191,10cyclictest3810-21diskmemload21:45:200
1509699191,10cyclictest2532-21sshd23:31:120
1509699191,10cyclictest23202-21/usr/sbin/munin20:40:240
1509699191,10cyclictest0-21swapper/023:06:390
1509699191,10cyclictest0-21swapper/000:33:070
1509699191,10cyclictest0-21swapper/000:33:070
1509699190,6cyclictest0-21swapper/022:52:410
1510699185,3cyclictest87350irq/126-enp1s0-21:25:142
1510699185,2cyclictest23434-21cstates20:40:112
1510699181,9cyclictest10298-21grep23:50:142
1510699181,9cyclictest0-21swapper/219:25:452
1510699181,8cyclictest0-21swapper/200:05:032
1510699181,7cyclictest0-21swapper/219:30:152
1510699181,6cyclictest16403-21if_enp2s022:50:172
1510699181,6cyclictest0-21swapper/223:03:122
1510699181,6cyclictest0-21swapper/222:41:192
1510699181,5cyclictest32385-21latency_hist22:15:012
1510699181,5cyclictest2759-21memory21:05:192
1510699181,5cyclictest0-21swapper/219:35:222
1510699181,4cyclictest0-21swapper/220:22:232
1510699181,15cyclictest0-21swapper/220:45:422
1510699181,15cyclictest0-21swapper/220:30:112
1510699181,11cyclictest0-21swapper/223:55:212
1510699181,10cyclictest32-21ksoftirqd/200:30:252
1510699181,10cyclictest32-21ksoftirqd/200:30:252
1509699188,2cyclictest0-21swapper/020:19:080
1509699187,3cyclictest3407-21sshd19:55:080
1509699187,3cyclictest31997-21/usr/sbin/munin21:00:120
1509699187,3cyclictest31997-21/usr/sbin/munin21:00:120
1509699187,3cyclictest16609-21/usr/sbin/munin20:25:200
1509699185,3cyclictest0-21swapper/023:35:570
1509699183,8cyclictest0-21swapper/022:05:240
1509699183,3cyclictest12050irq/123-ahci[0020:35:000
1509699181,5cyclictest0-21swapper/021:25:340
1509699181,15cyclictest0-21swapper/020:11:160
1509699181,14cyclictest8404-21latency_hist21:20:020
1509699181,11cyclictest0-21swapper/021:58:150
109002181,3sleep00-21swapper/022:38:430
220272171,2sleep00-21swapper/019:20:540
1510699175,2cyclictest87550irq/128-enp1s0-20:10:252
1510699171,7cyclictest16609-21/usr/sbin/munin20:25:172
1510699171,3cyclictest6012-21fschecks_count20:00:132
15106991712,3cyclictest8049-21cstates20:05:132
1509699176,3cyclictest87550irq/128-enp1s0-20:01:260
1509699171,9cyclictest0-21swapper/000:07:110
1509699171,6cyclictest0-21swapper/022:15:260
1509699171,4cyclictest26018-21/usr/sbin/munin22:00:220
1509699171,4cyclictest1198-21/usr/sbin/munin19:50:200
15096991713,2cyclictest8781-21diskstats21:20:140
15096991711,3cyclictest2192-21diskstats21:05:150
1509699171,11cyclictest0-21swapper/000:36:100
1509699170,2cyclictest0-21swapper/022:10:560
1510699163,3cyclictest12050irq/123-ahci[0020:20:022
1509699165,3cyclictest87350irq/126-enp1s0-20:35:110
1509699163,6cyclictest0-21swapper/020:50:160
1509699161,6cyclictest0-21swapper/019:30:160
1509699161,14cyclictest0-21swapper/020:09:210
15096991611,3cyclictest22588-21/usr/sbin/munin19:25:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional