You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-18 - 04:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2s.osadl.org (updated Thu Sep 18, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1501126160,0sleep00-21swapper/018:38:410
1523024435,0sleep20-21swapper/218:41:152
291350350,0irq/25-eth10-21swapper/318:37:263
1514823324,0sleep10-21swapper/118:40:181
28415070,0irq/27-eth0-tx-0-21swapper/020:42:110
8225060,0irq/24-0000:00:29630-21latency_hist19:17:012
35160,0ktimersoftd/330974-21smartctl22:37:253
339960,0migration/3663-21munin-node20:57:183
339960,0migration/331413-21runrttasks21:15:013
339960,0migration/31043-21taskset22:01:593
28415060,0irq/27-eth0-tx-0-21swapper/223:47:062
28415060,0irq/27-eth0-tx-0-21swapper/223:46:012
28415060,0irq/27-eth0-tx-0-21swapper/223:31:102
28415060,0irq/27-eth0-tx-0-21swapper/223:22:272
28415060,0irq/27-eth0-tx-0-21swapper/223:15:562
28415060,0irq/27-eth0-tx-0-21swapper/222:29:432
28415060,0irq/27-eth0-tx-0-21swapper/221:37:472
28415060,0irq/27-eth0-tx-0-21swapper/221:28:152
28405060,0irq/26-eth0-rx-27150-21iostat_ios21:37:193
249960,0watchdog/20-21swapper/221:08:322
179960,0migration/17714-21idleruntime-cro21:02:011
179960,0migration/122694-21ntp_offset22:32:231
153709966,0cyclictest7535-21diskmemload22:23:311
153709960,0cyclictest13570-21kthreadcore23:07:211
149960,0migration/06469-21sh21:18:100
149960,0migration/023220-21sh21:09:580
149960,0migration/021640-21timerwakeupswit20:47:250
149960,0migration/01333-21sh22:02:120
35150,0ktimersoftd/323351-21chrt22:53:473
339950,0migration/38926-21taskset21:24:383
339950,0migration/38272-21latency_hist23:07:013
339950,0migration/3801-21munin-node23:22:183
339950,0migration/34909-21kthreadcore22:22:193
339950,0migration/34681-21munin-node23:42:283
339950,0migration/33752-21kthreadcore20:32:183
339950,0migration/33374-21sh00:03:073
339950,0migration/331729-21latency_hist20:07:013
339950,0migration/330218-21kthreadcore20:02:183
339950,0migration/329890-21timerwakeupswit20:52:253
339950,0migration/328252-21kthreadcore19:12:193
339950,0migration/32824-21taskset18:56:183
339950,0migration/323572-21aten_r1power_vo23:37:163
339950,0migration/323405-21kthreadcore23:57:183
339950,0migration/322790-21fschecks_time18:47:173
339950,0migration/322197-21kthreadcore22:12:203
339950,0migration/318770-21kernelversion19:32:163
339950,0migration/318344-21kthreadcore19:07:183
339950,0migration/318255-21kthreadcore21:07:213
339950,0migration/317955-21latency_hist21:32:013
339950,0migration/317019-21kthreadcore23:52:203
339950,0migration/315828-21kthreadcore19:27:183
339950,0migration/31508-21kthreadcore21:17:163
339950,0migration/312298-21kthreadcore22:47:183
339950,0migration/311177-21taskset00:08:343
28415050,0irq/27-eth0-tx-5276-21needreboot20:57:212
28415050,0irq/27-eth0-tx-18109-21perl22:12:162
28415050,0irq/27-eth0-tx-15666-21ssh21:06:352
28415050,0irq/27-eth0-tx-0-21swapper/223:55:412
28415050,0irq/27-eth0-tx-0-21swapper/223:09:102
28415050,0irq/27-eth0-tx-0-21swapper/222:57:502
28415050,0irq/27-eth0-tx-0-21swapper/222:53:202
28415050,0irq/27-eth0-tx-0-21swapper/222:47:552
28415050,0irq/27-eth0-tx-0-21swapper/222:43:392
28415050,0irq/27-eth0-tx-0-21swapper/222:38:112
28415050,0irq/27-eth0-tx-0-21swapper/222:18:482
28415050,0irq/27-eth0-tx-0-21swapper/222:07:452
28415050,0irq/27-eth0-tx-0-21swapper/222:05:032
28415050,0irq/27-eth0-tx-0-21swapper/221:59:212
28415050,0irq/27-eth0-tx-0-21swapper/221:52:172
28415050,0irq/27-eth0-tx-0-21swapper/221:32:292
28415050,0irq/27-eth0-tx-0-21swapper/220:53:052
28415050,0irq/27-eth0-tx-0-21swapper/220:44:202
28415050,0irq/27-eth0-tx-0-21swapper/200:07:142
28415050,0irq/27-eth0-tx-0-21swapper/019:51:110
28415050,0irq/27-eth0-tx-0-21swapper/019:22:190
28405050,0irq/26-eth0-rx-9130-21kernelversion23:27:203
28405050,0irq/26-eth0-rx-6547-21runrttasks23:12:063
28405050,0irq/26-eth0-rx-30252-21ssh22:57:213
28405050,0irq/26-eth0-rx-24717-21fschecks_count23:17:163
28405050,0irq/26-eth0-rx-23535-21mailstats21:52:263
28405050,0irq/26-eth0-rx-10505-21grep21:47:153
28405050,0irq/26-eth0-rx-0-21swapper/323:35:373
28405050,0irq/26-eth0-rx-0-21swapper/323:02:153
28405050,0irq/26-eth0-rx-0-21swapper/322:27:173
28405050,0irq/26-eth0-rx-0-21swapper/322:07:173
28405050,0irq/26-eth0-rx-0-21swapper/322:05:403
28405050,0irq/26-eth0-rx-0-21swapper/321:45:263
28405050,0irq/26-eth0-rx-0-21swapper/321:02:473
28405050,0irq/26-eth0-rx-0-21swapper/320:47:343
28405050,0irq/26-eth0-rx-0-21swapper/320:42:203
28405050,0irq/26-eth0-rx-0-21swapper/320:22:233
259950,0migration/28485-21kthreadcore19:47:192
249950,0watchdog/25835-21tr21:42:202
249950,0watchdog/227987-21cut21:12:202
249950,0watchdog/20-21swapper/223:41:442
249950,0watchdog/20-21swapper/223:32:082
249950,0watchdog/20-21swapper/223:18:122
249950,0watchdog/20-21swapper/221:49:362
249950,0watchdog/20-21swapper/221:22:202
249950,0watchdog/20-21swapper/221:17:122
249950,0watchdog/20-21swapper/220:47:202
249950,0watchdog/20-21swapper/220:32:402
249950,0watchdog/20-21swapper/220:26:482
249950,0watchdog/20-21swapper/219:57:482
249950,0watchdog/20-21swapper/219:43:402
249950,0watchdog/20-21swapper/219:37:162
249950,0watchdog/20-21swapper/219:34:162
249950,0watchdog/20-21swapper/219:27:402
249950,0watchdog/20-21swapper/219:23:002
249950,0watchdog/20-21swapper/219:12:002
249950,0watchdog/20-21swapper/218:58:002
249950,0watchdog/20-21swapper/218:56:202
249950,0watchdog/20-21swapper/200:00:552
19150,0ktimersoftd/10-21swapper/122:18:011
19150,0ktimersoftd/10-21swapper/120:32:221
179950,0migration/1715-21aten_r1power_cu22:42:141
179950,0migration/131375-21sshd22:37:351
179950,0migration/120254-21kthreadcore19:07:191
179950,0migration/118266-21idleruntime-cro19:57:011
179950,0migration/114793-21taskset20:45:231
179950,0migration/110577-21kthreadcore23:27:201
169950,0watchdog/124807-21perl19:37:161
169950,0watchdog/116167-21ssh21:48:121
169950,0watchdog/110527-21perl19:02:161
153749955,0cyclictest0-21swapper/222:22:582
153749955,0cyclictest0-21swapper/220:27:242
153749950,0cyclictest0-21swapper/223:02:042
153709955,0cyclictest0-21swapper/122:58:311
153709955,0cyclictest0-21swapper/121:34:391
153709950,0cyclictest0-21swapper/123:19:391
153709950,0cyclictest0-21swapper/122:02:051
153699950,0cyclictest0-21swapper/023:32:420
153699950,0cyclictest0-21swapper/023:03:410
153699950,0cyclictest0-21swapper/022:48:050
153699950,0cyclictest0-21swapper/022:43:400
153699950,0cyclictest0-21swapper/022:19:520
153699950,0cyclictest0-21swapper/000:06:260
149950,0migration/07278-21taskset20:39:150
149950,0migration/03267-21aten_r1power_po18:57:100
149950,0migration/031373-21sh22:58:580
149950,0migration/030432-21fschecks_time19:17:150
149950,0migration/029639-21kthreadcore20:02:180
149950,0migration/029539-21kthreadcore21:57:180
149950,0migration/027740-21kthreadcore21:37:200
149950,0migration/025011-21munin-node19:37:170
149950,0migration/023820-21taskset20:52:030
149950,0migration/022990-21kthreadcore21:32:190
149950,0migration/022935-21taskset22:52:300
149950,0migration/02160-21switchtime18:52:250
149950,0migration/020489-21kthreadcore21:52:210
149950,0migration/018398-21munin-node22:12:180
149950,0migration/017117-21idleruntime23:12:170
149950,0migration/017006-21sh21:50:390
149950,0migration/014998-21kthreadcore19:27:180
149950,0migration/014363-21taskset21:03:050
149950,0migration/014340-21kthreadcore20:17:180
149950,0migration/011971-21aten_r1power_vo19:52:130
149950,0migration/011291-21kthreadcore19:02:190
149950,0migration/010478-21fschecks_count21:27:120
339940,0migration/39095-21kthreadcore19:47:203
339940,0migration/37674-21kthreadcore20:12:193
339940,0migration/331721-21sensors_fan20:27:223
339940,0migration/324744-21runrttasks20:00:123
339940,0migration/319023-21kthreadcore18:42:193
339940,0migration/313540-21gltestperf20:17:153
339940,0migration/311744-21kthreadcore19:02:193
28415040,0irq/27-eth0-tx-0-21swapper/020:27:160
28415040,0irq/27-eth0-tx-0-21swapper/020:22:230
28415040,0irq/27-eth0-tx-0-21swapper/019:57:220
28405040,0irq/26-eth0-rx-0-21swapper/323:47:083
28405040,0irq/26-eth0-rx-0-21swapper/322:44:583
28405040,0irq/26-eth0-rx-0-21swapper/322:32:253
28405040,0irq/26-eth0-rx-0-21swapper/322:17:423
28405040,0irq/26-eth0-rx-0-21swapper/321:29:433
259940,0migration/27304-21irqrtprio20:12:192
259940,0migration/231248-21kthreadcore19:17:182
259940,0migration/220842-21kthreadcore22:32:202
249940,0watchdog/232625-21if_err_eth020:07:162
249940,0watchdog/20-21swapper/220:38:002
249940,0watchdog/20-21swapper/220:21:442
249940,0watchdog/20-21swapper/220:06:202
249940,0watchdog/20-21swapper/219:52:162
249940,0watchdog/20-21swapper/219:04:082
249940,0watchdog/20-21swapper/218:42:202
249940,0watchdog/20-21swapper/200:02:152
179940,0migration/19885-21kthreadcore21:02:201
179940,0migration/19499-21kthreadcore20:12:201
179940,0migration/15317-21fschecks_time00:07:151
179940,0migration/13689-21fschecks_count18:57:121
179940,0migration/132020-21taskset21:57:491
179940,0migration/130028-21fschecks_time00:02:171
179940,0migration/128670-21kthreadcore21:37:201
179940,0migration/124252-21kthreadcore23:57:181
179940,0migration/121930-21kthreadcore19:57:191
179940,0migration/119029-21runrttasks20:20:111
179940,0migration/114816-21proc_pri22:27:241
179940,0migration/113619-21kthreadcore21:27:161
179940,0migration/113483-21kthreadcore19:27:181
179940,0migration/111489-21kthreadcore22:47:181
169940,0watchdog/18608-21ssh21:23:441
169940,0watchdog/16888-21fschecks_time23:47:201
169940,0watchdog/16608-21kthreadcore19:22:201
169940,0watchdog/13207-21timerandwakeup19:17:241
169940,0watchdog/130874-21ssh23:42:041
169940,0watchdog/128580-21needreboot19:12:201
169940,0watchdog/120361-21kthreadcore22:12:201
169940,0watchdog/118953-21kthreadcore20:47:201
169940,0watchdog/118821-21cut19:32:161
169940,0watchdog/117853-21kthreadcore23:32:161
169940,0watchdog/114443-21diskstats23:52:161
169940,0watchdog/10-21swapper/123:12:201
169940,0watchdog/10-21swapper/122:07:201
169940,0watchdog/10-21swapper/120:22:241
153749943,0cyclictest3425-21lldpd18:48:152
153709944,0cyclictest0-21swapper/123:25:481
153709944,0cyclictest0-21swapper/121:53:341
153709944,0cyclictest0-21swapper/120:52:431
153709943,0cyclictest6445-21snmpd21:15:131
153709943,0cyclictest6445-21snmpd20:31:571
153709943,0cyclictest6445-21snmpd18:52:111
153709943,0cyclictest6445-21snmpd18:49:411
153709943,0cyclictest6008-21awk19:47:171
153709943,0cyclictest4854-21kworker/1:223:39:511
153709943,0cyclictest4854-21kworker/1:221:44:011
153709943,0cyclictest4854-21kworker/1:221:18:191
153709943,0cyclictest4854-21kworker/1:221:08:231
153709943,0cyclictest3425-21lldpd22:54:221
153709943,0cyclictest32753-21awk20:07:171
153699944,0cyclictest0-21swapper/022:32:470
153699943,0cyclictest7050-21awk20:12:170
153699943,0cyclictest6445-21snmpd19:13:110
153699943,0cyclictest616-21awk23:22:170
153699943,0cyclictest26100-21awk22:37:180
153699943,0cyclictest24395-21awk23:37:200
153699943,0cyclictest23116-21awk18:47:190
153699943,0cyclictest1998-21awk22:22:170
149940,0migration/05825-21diskmemload20:57:250
149940,0migration/023616-21kthreadcore23:57:180
149940,0migration/021501-21latency_hist23:57:010
149940,0migration/013638-21switchtime23:27:260
28415030,0irq/27-eth0-tx-0-21swapper/019:42:130
28405030,0irq/26-eth0-rx-0-21swapper/320:37:443
28405030,0irq/26-eth0-rx-0-21swapper/319:52:133
28405030,0irq/26-eth0-rx-0-21swapper/319:42:163
28405030,0irq/26-eth0-rx-0-21swapper/319:37:143
28405030,0irq/26-eth0-rx-0-21swapper/319:22:153
28405030,0irq/26-eth0-rx-0-21swapper/319:17:133
153709933,0cyclictest4854-21kworker/1:223:02:091
153709933,0cyclictest4854-21kworker/1:220:41:211
153709933,0cyclictest4854-21kworker/1:220:02:131
153709933,0cyclictest4854-21kworker/1:219:42:151
153709932,0cyclictest4854-21kworker/1:218:42:271
153699932,0cyclictest9339-21awk23:07:180
153699932,0cyclictest6445-21snmpd00:08:290
153699932,0cyclictest24502-21tr23:17:150
153699932,0cyclictest17135-21grep19:07:150
153699932,0cyclictest16140-21grep18:42:150
153699932,0cyclictest1219-21awk20:32:160
153699932,0cyclictest12088-21sensors23:47:280
153699930,0cyclictest32433-21perl20:07:150
153699930,0cyclictest18156-21perl19:32:130
153699930,0cyclictest0-21swapper/021:12:160
149930,0migration/09132-21sh21:45:570
149930,0migration/016141-21sh22:09:110
149930,0migration/015637-21diskmemload22:28:280
28405020,0irq/26-eth0-rx-0-21swapper/318:57:093
153699921,0cyclictest7960-21latency_hist21:22:010
153699920,0cyclictest0-21swapper/023:42:100
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional