You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-03 - 07:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Sat Jan 03, 2026 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
191770,0ktimersoftd/111665-21perf19:07:531
304450650,0irq/25-eth10-21swapper/319:05:523
1787623723,0sleep00-21swapper/019:03:410
1814423526,0sleep20-21swapper/219:06:482
183209999,0cyclictest29440-21awk00:28:211
29715070,0irq/26-eth0-rx-29293-21ssh22:30:373
339960,0migration/39412-21cpuspeed_turbos20:38:133
339960,0migration/34725-21sh22:34:203
339960,0migration/331509-21sh23:11:033
339960,0migration/322947-21sh23:05:313
339960,0migration/319190-21sh00:21:213
339960,0migration/318037-21kthreadcore20:43:193
339960,0migration/312399-21sh22:20:003
29715060,0irq/26-eth0-rx-5772-21irqstats00:33:183
29715060,0irq/26-eth0-rx-28147-21python20:48:253
29715060,0irq/26-eth0-rx-15835-21ssh23:20:543
29715060,0irq/26-eth0-rx-10425-21diskstats21:23:163
29715060,0irq/26-eth0-rx-0-21swapper/300:13:203
183209966,0cyclictest16615-21diskmemload22:15:441
149960,0migration/029718-21aten_r1power_po22:13:130
339950,0migration/36622-21aten_r1power_en22:38:143
339950,0migration/36165-21munin-node19:43:263
339950,0migration/35906-21kthreadcore21:58:183
339950,0migration/330137-21munin-node20:53:193
339950,0migration/329218-21kthreadcore00:08:203
339950,0migration/328790-21fschecks_time20:28:183
339950,0migration/328524-21kthreadcore23:28:193
339950,0migration/327704-21fschecks_count21:33:163
339950,0migration/326457-21munin-node19:13:193
339950,0migration/324455-21taskset23:44:103
339950,0migration/323374-21runrttasks21:09:413
339950,0migration/319442-21latency_hist19:33:033
339950,0migration/316650-21kthreadcore19:53:193
339950,0migration/316452-21taskset21:06:173
339950,0migration/315109-21missed_timers23:38:193
339950,0migration/31437-21aten_r1power_fr21:18:143
339950,0migration/31384-21aten_r1power_vo23:33:143
339950,0migration/31384-21aten_r1power_vo23:33:143
339950,0migration/313554-21missed_timers20:13:243
339950,0migration/312837-21sendmail_mailqu19:48:253
339950,0migration/311937-21switchtime19:23:233
29725050,0irq/27-eth0-tx-24057-21basename22:48:162
29725050,0irq/27-eth0-tx-23395-21ssh21:09:462
29725050,0irq/27-eth0-tx-21120-21ssh21:48:182
29725050,0irq/27-eth0-tx-15904-21munin-node23:03:172
29725050,0irq/27-eth0-tx-14563-21ssh22:42:562
29725050,0irq/27-eth0-tx-11583-21sort22:18:262
29725050,0irq/27-eth0-tx-0-21swapper/223:43:192
29725050,0irq/27-eth0-tx-0-21swapper/223:11:122
29725050,0irq/27-eth0-tx-0-21swapper/222:28:192
29725050,0irq/27-eth0-tx-0-21swapper/222:06:432
29725050,0irq/27-eth0-tx-0-21swapper/221:33:222
29725050,0irq/27-eth0-tx-0-21swapper/221:18:042
29725050,0irq/27-eth0-tx-0-21swapper/200:24:312
29715050,0irq/26-eth0-rx-6559-21runrttasks23:02:093
29715050,0irq/26-eth0-rx-4676-21kthreadcore21:38:173
29715050,0irq/26-eth0-rx-31559-21munin-node22:53:183
29715050,0irq/26-eth0-rx-31006-21kthreadcore00:28:233
29715050,0irq/26-eth0-rx-29011-21python22:48:243
29715050,0irq/26-eth0-rx-27680-21ssh22:09:063
29715050,0irq/26-eth0-rx-26338-21ssh21:48:353
29715050,0irq/26-eth0-rx-19904-21hwlatdetect00:03:153
29715050,0irq/26-eth0-rx-0-21swapper/323:58:183
29715050,0irq/26-eth0-rx-0-21swapper/323:53:143
29715050,0irq/26-eth0-rx-0-21swapper/323:49:463
29715050,0irq/26-eth0-rx-0-21swapper/323:24:033
29715050,0irq/26-eth0-rx-0-21swapper/322:43:423
29715050,0irq/26-eth0-rx-0-21swapper/322:17:113
29715050,0irq/26-eth0-rx-0-21swapper/321:54:143
29715050,0irq/26-eth0-rx-0-21swapper/321:43:163
29715050,0irq/26-eth0-rx-0-21swapper/321:28:203
29715050,0irq/26-eth0-rx-0-21swapper/319:38:143
29715050,0irq/26-eth0-rx-0-21swapper/319:18:213
259950,0migration/27258-21kthreadcore21:58:192
259950,0migration/25006-21kthreadcore21:38:172
259950,0migration/228326-21kthreadcore23:48:182
259950,0migration/216462-21fschecks_time20:43:162
259950,0migration/214514-21fschecks_time19:53:162
249950,0watchdog/28430-21if_err_eth022:58:162
183219955,0cyclictest0-21swapper/200:08:182
183219950,0cyclictest0-21swapper/223:18:202
183209955,0cyclictest0-21swapper/123:14:331
183209955,0cyclictest0-21swapper/100:18:431
183209950,0cyclictest0-21swapper/123:53:391
183209950,0cyclictest0-21swapper/123:47:301
183209950,0cyclictest0-21swapper/123:32:311
183209950,0cyclictest0-21swapper/122:54:501
183209950,0cyclictest0-21swapper/122:21:251
183209950,0cyclictest0-21swapper/121:51:181
183209950,0cyclictest0-21swapper/121:09:031
183209950,0cyclictest0-21swapper/119:29:151
183209950,0cyclictest0-21swapper/119:19:191
183209950,0cyclictest0-21swapper/100:01:501
183199954,0cyclictest869-21sh21:33:260
179950,0migration/131232-21kthreadcore21:33:201
179950,0migration/129209-21kthreadcore20:03:191
179950,0migration/12859-21sh21:54:581
179950,0migration/117520-21taskset21:26:221
179950,0migration/114341-21sh22:23:141
149950,0migration/07925-21taskset21:19:120
149950,0migration/06046-21sh22:37:320
149950,0migration/04106-21sh00:32:520
149950,0migration/03075-21gltestperf23:53:150
149950,0migration/027796-21sh22:09:220
149950,0migration/027369-21kthreadcore21:13:200
149950,0migration/02688-21irqrtprio20:08:200
149950,0migration/024759-21kthreadcore22:48:180
149950,0migration/019704-21sendmail_mailqu22:23:230
149950,0migration/019671-21sh00:22:320
149950,0migration/013848-21munin-node19:28:180
149950,0migration/012971-21taskset22:21:230
149950,0migration/011995-21kthreadcore23:38:180
149950,0migration/01091-21unixbench_singl00:08:270
149950,0migration/01009-21kthreadcore20:53:210
339940,0migration/314891-21kthreadcore19:28:193
29725040,0irq/27-eth0-tx-4645-21ssh22:34:082
29725040,0irq/27-eth0-tx-32705-21tail23:13:142
29725040,0irq/27-eth0-tx-29512-21ssh21:53:172
29725040,0irq/27-eth0-tx-29441-21ssh00:28:212
29725040,0irq/27-eth0-tx-21532-21munin-node20:23:192
29725040,0irq/27-eth0-tx-16615-21diskmemload23:23:222
29725040,0irq/27-eth0-tx-1244-21munin-node23:33:232
29725040,0irq/27-eth0-tx-1244-21munin-node23:33:232
29725040,0irq/27-eth0-tx-10738-21ssh23:58:132
29725040,0irq/27-eth0-tx-0-21swapper/223:53:072
29725040,0irq/27-eth0-tx-0-21swapper/223:38:572
29725040,0irq/27-eth0-tx-0-21swapper/222:44:272
29725040,0irq/27-eth0-tx-0-21swapper/220:03:162
29725040,0irq/27-eth0-tx-0-21swapper/219:28:142
29725040,0irq/27-eth0-tx-0-21swapper/200:33:032
29715040,0irq/26-eth0-rx-27337-21aten_r1power_po20:03:143
29715040,0irq/26-eth0-rx-0-21swapper/323:13:113
29715040,0irq/26-eth0-rx-0-21swapper/322:23:193
29715040,0irq/26-eth0-rx-0-21swapper/322:03:113
29715040,0irq/26-eth0-rx-0-21swapper/321:13:203
29715040,0irq/26-eth0-rx-0-21swapper/300:23:383
259940,0migration/26173-21runrttasks19:23:102
259940,0migration/229592-21sh22:13:122
259940,0migration/22851-21fschecks_count20:33:172
259940,0migration/228228-21kthreadcore23:28:192
259940,0migration/226690-21kthreadcore21:13:192
259940,0migration/225132-21rt-features19:33:242
259940,0migration/223673-21kthreadcore22:08:162
259940,0migration/222346-21kthreadcore19:08:212
259940,0migration/221558-21kthreadcore21:28:202
259940,0migration/215348-21kthreadcore22:23:192
259940,0migration/214319-21kthreadcore21:23:202
249940,0watchdog/27775-21tune2fs19:48:172
249940,0watchdog/26559-21runrttasks19:41:092
249940,0watchdog/24125-21tail00:13:162
249940,0watchdog/2372-21munin-node19:43:172
249940,0watchdog/226486-21cut20:48:212
249940,0watchdog/217959-21kthreadcore20:18:212
249940,0watchdog/217945-21ssh21:43:332
249940,0watchdog/20-21swapper/220:58:212
183219943,0cyclictest3556-21lldpd22:53:152
183219943,0cyclictest3556-21lldpd21:03:452
183219943,0cyclictest3556-21lldpd20:40:452
183219943,0cyclictest3556-21lldpd19:14:452
183219943,0cyclictest3556-21lldpd00:18:142
183219943,0cyclictest29140-21awk20:28:202
183219943,0cyclictest2383-21awk20:08:192
183219943,0cyclictest21426-21awk19:58:182
183219943,0cyclictest14635-21sadc20:18:022
183219940,0cyclictest19111-21munin-node00:03:232
183209944,0cyclictest0-21swapper/121:58:051
183209943,0cyclictest9109-21awk20:13:201
183209943,0cyclictest3556-21lldpd19:58:141
183209943,0cyclictest3514-21snmpd23:34:161
183209943,0cyclictest3514-21snmpd23:34:161
183209943,0cyclictest3514-21snmpd22:52:461
183209943,0cyclictest3514-21snmpd21:45:161
183209943,0cyclictest3241-21irqbalance22:46:301
183209943,0cyclictest29028-21awk20:28:191
183209943,0cyclictest27336-21awk23:48:171
183209943,0cyclictest22246-21sadc20:48:021
183209943,0cyclictest20349-21awk19:33:181
183199943,0cyclictest8114-21awk19:48:190
183199943,0cyclictest7453-21awk22:38:170
183199943,0cyclictest7132-21awk19:23:160
183199943,0cyclictest5447-21awk21:58:170
183199943,0cyclictest4111-21tail21:38:140
183199943,0cyclictest3556-21lldpd23:50:150
183199943,0cyclictest3556-21lldpd21:00:450
183199943,0cyclictest3556-21lldpd20:41:450
183199943,0cyclictest3514-21snmpd23:04:460
183199943,0cyclictest3514-21snmpd22:55:310
183199943,0cyclictest3241-21irqbalance23:43:500
183199943,0cyclictest3241-21irqbalance23:13:200
183199943,0cyclictest3241-21irqbalance22:59:100
183199943,0cyclictest3241-21irqbalance22:43:200
183199943,0cyclictest3241-21irqbalance22:03:310
183199943,0cyclictest3241-21irqbalance21:54:010
183199943,0cyclictest3241-21irqbalance21:48:310
183199943,0cyclictest3241-21irqbalance21:25:210
183199943,0cyclictest3241-21irqbalance20:49:210
183199943,0cyclictest3241-21irqbalance19:43:110
183199943,0cyclictest26277-21awk23:28:170
183199943,0cyclictest23196-21awk22:28:160
183199943,0cyclictest21387-21awk19:58:180
183199943,0cyclictest20442-21awk19:33:180
183199943,0cyclictest20065-21awk00:03:170
183199943,0cyclictest14695-21awk19:53:170
183199943,0cyclictest12608-21awk21:43:170
179940,0migration/19398-21kthreadcore22:38:201
179940,0migration/18294-21gltestperf22:58:151
179940,0migration/17063-21kthreadcore00:33:181
179940,0migration/125643-21kthreadcore20:48:201
179940,0migration/123349-21kthreadcore00:23:171
179940,0migration/122891-21runrttasks23:05:221
179940,0migration/119663-21munin-node00:03:151
179940,0migration/119468-21kthreadcore23:23:191
179940,0migration/114629-21hddtemp_smartct19:53:161
179940,0migration/11458-21kthreadcore20:53:211
179940,0migration/112319-21sshd19:24:401
169940,0watchdog/17777-21fschecks_time19:48:171
169940,0watchdog/16509-21tr20:33:211
169940,0watchdog/15689-21kthreadcore20:58:211
169940,0watchdog/13294-21rm00:13:041
169940,0watchdog/129583-21ssh22:31:201
169940,0watchdog/127803-21df_inode00:08:161
169940,0watchdog/123787-21cut22:08:161
169940,0watchdog/115394-21kthreadcore22:03:201
169940,0watchdog/115341-21fschecks_count20:18:171
169940,0watchdog/114885-21ssh23:18:321
169940,0watchdog/10-21swapper/123:40:401
169940,0watchdog/10-21swapper/121:28:211
169940,0watchdog/10-21swapper/120:08:211
169940,0watchdog/10-21swapper/119:43:211
149940,0migration/026164-21taskset19:38:130
149940,0migration/024107-21sh23:25:480
149940,0migration/020482-21kthreadcore21:28:200
149940,0migration/01905-21kthreadcore19:18:210
149940,0migration/013963-21kthreadcore23:18:210
149940,0migration/013153-21kthreadcore21:03:220
149940,0migration/012000-21irqrtprio23:58:180
149940,0migration/011572-21sh00:35:380
29715030,0irq/26-eth0-rx-0-21swapper/320:58:563
29715030,0irq/26-eth0-rx-0-21swapper/320:33:153
29715030,0irq/26-eth0-rx-0-21swapper/320:23:193
29715030,0irq/26-eth0-rx-0-21swapper/320:18:183
29715030,0irq/26-eth0-rx-0-21swapper/320:08:153
29715030,0irq/26-eth0-rx-0-21swapper/319:13:003
183219930,0cyclictest31842-21munin-node19:18:172
183219930,0cyclictest0-21swapper/220:53:162
183209932,0cyclictest3556-21lldpd20:38:451
183209932,0cyclictest31840-21sensors19:38:241
183209932,0cyclictest31155-21cut22:33:161
183209932,0cyclictest27059-21tr23:08:181
183209932,0cyclictest26094-21awk19:13:181
183209932,0cyclictest24711-21kthreadcore20:23:201
183209932,0cyclictest11178-21awk21:03:201
183209930,0cyclictest4812-21grep21:18:201
183199933,0cyclictest0-21swapper/023:33:090
183199933,0cyclictest0-21swapper/023:33:090
183199932,0cyclictest3556-21lldpd21:08:150
183199932,0cyclictest3556-21lldpd20:35:150
183199932,0cyclictest3241-21irqbalance00:13:100
183199932,0cyclictest30055-21sensors23:08:220
183199932,0cyclictest28162-21munin-node20:28:290
183199932,0cyclictest27564-21df_abs20:03:150
183199932,0cyclictest19287-21grep19:08:180
183199932,0cyclictest18546-21kthreadcore20:18:210
183199932,0cyclictest13666-21grep20:13:250
183199931,0cyclictest23825-21kthreadcore20:23:200
183199930,0cyclictest25822-21perl19:13:160
183199930,0cyclictest20874-21gltestperf00:23:140
179930,0migration/126220-21if_eth1.321:13:181
169930,0watchdog/18967-21ntp_states21:38:211
169930,0watchdog/121431-21kthreadcore19:08:211
29715020,0irq/26-eth0-rx-0-21swapper/319:58:193
183199921,0cyclictest15827-21aten_r1power_cu20:43:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional