You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-29 - 23:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Sat Nov 29, 2025 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350810,0irq/25-eth10-21swapper/306:34:463
135925756,0sleep20-21swapper/206:38:182
111123624,0sleep10-21swapper/106:35:251
112923523,0sleep00-21swapper/006:35:360
28415070,0irq/27-eth0-tx-0-21swapper/211:29:022
339960,0migration/38200-21sh11:06:213
339960,0migration/35095-21sh09:43:393
339960,0migration/328840-21kthreadcore11:59:403
339960,0migration/323448-21sh11:58:033
339960,0migration/319032-21idleruntime10:54:383
339960,0migration/315001-21sh11:51:473
28415060,0irq/27-eth0-tx-22815-21ssh11:56:192
28415060,0irq/27-eth0-tx-0-21swapper/210:48:242
28415060,0irq/27-eth0-tx-0-21swapper/210:27:092
28415060,0irq/27-eth0-tx-0-21swapper/210:06:452
28405060,0irq/26-eth0-rx-26470-21ssh10:14:503
28405060,0irq/26-eth0-rx-25642-21perl11:19:343
28405060,0irq/26-eth0-rx-17560-21df_abs11:14:363
28405060,0irq/26-eth0-rx-16330-21interrupts09:09:373
249960,0watchdog/20-21swapper/211:34:352
15249965,0cyclictest2890-21sh10:22:552
15219966,0cyclictest32047-21diskmemload11:01:590
8225050,0irq/24-0000:00:25414-21latency_hist08:34:212
35150,0ktimersoftd/30-21swapper/308:48:223
339950,0migration/38084-21runrttasks11:28:183
339950,0migration/34700-21irqcore10:24:383
339950,0migration/34574-21timerwakeupswit09:19:463
339950,0migration/33557-21munin-node07:04:393
339950,0migration/332453-21aten_r1power_vo08:39:333
339950,0migration/330544-21timerwakeupswit11:39:473
339950,0migration/322604-21taskset08:51:383
339950,0migration/318221-21kthreadcore09:29:393
339950,0migration/317362-21sh11:34:373
339950,0migration/314649-21kthreadcore11:09:413
339950,0migration/312839-21kthreadcore10:49:393
339950,0migration/31253-21irqrtprio12:04:383
339950,0migration/312406-21idleruntime10:29:363
339950,0migration/310370-21kthreadcore08:19:403
28415050,0irq/27-eth0-tx-1734-21munin-node06:39:340
28415050,0irq/27-eth0-tx-0-21swapper/211:41:032
28415050,0irq/27-eth0-tx-0-21swapper/211:30:202
28415050,0irq/27-eth0-tx-0-21swapper/211:19:212
28415050,0irq/27-eth0-tx-0-21swapper/211:12:002
28415050,0irq/27-eth0-tx-0-21swapper/211:07:322
28415050,0irq/27-eth0-tx-0-21swapper/210:57:112
28415050,0irq/27-eth0-tx-0-21swapper/210:43:082
28415050,0irq/27-eth0-tx-0-21swapper/210:14:352
28415050,0irq/27-eth0-tx-0-21swapper/210:11:332
28415050,0irq/27-eth0-tx-0-21swapper/210:01:202
28415050,0irq/27-eth0-tx-0-21swapper/209:57:042
28415050,0irq/27-eth0-tx-0-21swapper/209:31:402
28415050,0irq/27-eth0-tx-0-21swapper/209:25:102
28415050,0irq/27-eth0-tx-0-21swapper/209:20:472
28415050,0irq/27-eth0-tx-0-21swapper/209:19:082
28415050,0irq/27-eth0-tx-0-21swapper/209:06:262
28415050,0irq/27-eth0-tx-0-21swapper/209:02:422
28415050,0irq/27-eth0-tx-0-21swapper/208:54:362
28415050,0irq/27-eth0-tx-0-21swapper/208:49:502
28415050,0irq/27-eth0-tx-0-21swapper/208:45:332
28415050,0irq/27-eth0-tx-0-21swapper/208:41:442
28415050,0irq/27-eth0-tx-0-21swapper/008:23:530
28415050,0irq/27-eth0-tx-0-21swapper/008:09:390
28415050,0irq/27-eth0-tx-0-21swapper/007:39:390
28415050,0irq/27-eth0-tx-0-21swapper/007:17:080
28405050,0irq/26-eth0-rx-9157-21sh10:46:043
28405050,0irq/26-eth0-rx-779-21cut10:19:403
28405050,0irq/26-eth0-rx-3779-21ssh10:02:303
28405050,0irq/26-eth0-rx-28282-21port_sshpriv07:19:423
28405050,0irq/26-eth0-rx-23102-21kthreadcore06:54:383
28405050,0irq/26-eth0-rx-12999-21kthreadcore11:29:403
28405050,0irq/26-eth0-rx-12098-21munin-node07:59:343
28405050,0irq/26-eth0-rx-10769-21sshd07:34:353
28405050,0irq/26-eth0-rx-10166-21cut09:04:393
28405050,0irq/26-eth0-rx-0-21swapper/311:48:363
28405050,0irq/26-eth0-rx-0-21swapper/310:59:593
28405050,0irq/26-eth0-rx-0-21swapper/309:46:233
28405050,0irq/26-eth0-rx-0-21swapper/309:36:263
28405050,0irq/26-eth0-rx-0-21swapper/308:59:443
28405050,0irq/26-eth0-rx-0-21swapper/308:24:343
28405050,0irq/26-eth0-rx-0-21swapper/308:09:333
27150,0ktimersoftd/232047-21diskmemload09:46:492
249950,0watchdog/20-21swapper/210:39:152
249950,0watchdog/20-21swapper/206:44:392
179950,0migration/19249-21taskset10:46:371
179950,0migration/14951-21munin-node10:24:381
179950,0migration/131803-21munin-node10:59:431
179950,0migration/11586-21kthreadcore07:49:381
179950,0migration/115033-21fschecks_time09:29:351
179950,0migration/114487-21kthreadcore08:24:381
179950,0migration/113836-21timerandwakeup08:44:441
179950,0migration/112551-21kthreadcore06:44:381
179950,0migration/110248-21hwlatdetect07:09:361
169950,0watchdog/126936-21kthreadcore11:59:391
169950,0watchdog/125732-21munin-node11:39:391
169950,0watchdog/12095-21grep11:24:391
169950,0watchdog/120437-21kthreadcore08:29:391
169950,0watchdog/11715-21ssh10:42:471
169950,0watchdog/114941-21kthreadcore10:09:391
169950,0watchdog/10-21swapper/111:58:231
169950,0watchdog/10-21swapper/110:24:031
169950,0watchdog/10-21swapper/108:49:391
169950,0watchdog/10-21swapper/106:50:271
15249950,0cyclictest0-21swapper/212:05:332
15249950,0cyclictest0-21swapper/212:01:552
15249950,0cyclictest0-21swapper/211:44:252
15249950,0cyclictest0-21swapper/211:16:502
15249950,0cyclictest0-21swapper/211:00:262
15249950,0cyclictest0-21swapper/210:32:042
15239950,0cyclictest0-21swapper/111:10:171
15239950,0cyclictest0-21swapper/110:08:571
15219955,0cyclictest0-21swapper/011:09:520
15219955,0cyclictest0-21swapper/011:06:150
15219954,0cyclictest324-21grep08:39:350
15219954,0cyclictest17118-21awk11:54:380
15219950,0cyclictest0-21swapper/011:54:200
15219950,0cyclictest0-21swapper/009:13:030
15219950,0cyclictest0-21swapper/008:44:210
149950,0migration/09461-21kthreadcore09:24:400
149950,0migration/05308-21munin-node08:59:480
149950,0migration/04087-21sh10:03:300
149950,0migration/032260-21kernelversion09:19:390
149950,0migration/031807-21sh12:03:300
149950,0migration/0315-21sh11:44:350
149950,0migration/030655-21unixbench_singl11:39:470
149950,0migration/022675-21missed_timers11:14:420
149950,0migration/022562-21irqrtprio09:54:390
149950,0migration/022159-21munin-node11:34:430
149950,0migration/019335-21aten_r1power_cu10:34:320
149950,0migration/016819-21idleruntime-cro07:39:210
149950,0migration/016524-21taskset10:49:480
149950,0migration/011983-21munin-node10:29:350
8225040,0irq/24-0000:00:0-21swapper/208:15:162
8225040,0irq/24-0000:00:0-21swapper/206:49:412
35140,0ktimersoftd/33067-17auditd08:09:213
339940,0migration/37655-21irqcore09:24:393
339940,0migration/324163-21fschecks_count07:44:353
339940,0migration/323097-21taskset07:40:173
339940,0migration/322530-21kthreadcore10:34:393
339940,0migration/321235-21taskset06:50:363
339940,0migration/319458-21sh10:12:293
339940,0migration/31818-21kthreadcore08:14:393
339940,0migration/316548-21fschecks_count07:14:343
28415040,0irq/27-eth0-tx-6372-21iceccd08:34:230
28405040,0irq/26-eth0-rx-27215-21wc09:54:463
28405040,0irq/26-eth0-rx-0-21swapper/310:39:273
28405040,0irq/26-eth0-rx-0-21swapper/310:08:363
28405040,0irq/26-eth0-rx-0-21swapper/309:49:413
28405040,0irq/26-eth0-rx-0-21swapper/309:16:003
28405040,0irq/26-eth0-rx-0-21swapper/308:54:403
28405040,0irq/26-eth0-rx-0-21swapper/307:57:163
259940,0migration/226248-21kthreadcore08:09:392
259940,0migration/223020-21kthreadcore08:29:402
249940,0watchdog/29071-21kthreadcore08:19:392
249940,0watchdog/288422sleep207:04:552
249940,0watchdog/231908-21kthreadcore09:39:392
249940,0watchdog/216109-21hddtemp_smartct09:09:352
249940,0watchdog/214970-21kthreadcore09:49:392
249940,0watchdog/20-21swapper/210:49:392
249940,0watchdog/20-21swapper/209:34:472
179940,0migration/18918-21kthreadcore07:54:401
179940,0migration/15973-21kthreadcore06:39:381
179940,0migration/132036-21sh08:14:351
179940,0migration/13150-21kthreadcore12:04:391
179940,0migration/128163-21taskset09:57:041
179940,0migration/127555-21kthreadcore11:19:381
179940,0migration/122350-21taskset09:12:271
179940,0migration/12042-21kthreadcore11:44:391
179940,0migration/116581-21fschecks_count07:14:341
169940,0watchdog/1884-21kthreadcore08:59:431
169940,0watchdog/17913-21df_inode09:04:351
169940,0watchdog/16479-21ssh09:24:311
169940,0watchdog/15914-21kthreadcore11:04:391
169940,0watchdog/1553-21cut09:19:391
169940,0watchdog/124340-21ssh10:54:471
169940,0watchdog/122214-21chrt09:34:311
169940,0watchdog/118599-21fschecks_time08:04:351
169940,0watchdog/110772-21diskstats07:34:351
169940,0watchdog/10-21swapper/111:37:071
169940,0watchdog/10-21swapper/110:36:351
169940,0watchdog/10-21swapper/110:34:151
169940,0watchdog/10-21swapper/110:14:231
169940,0watchdog/10-21swapper/109:52:111
169940,0watchdog/10-21swapper/109:14:311
169940,0watchdog/10-21swapper/107:29:351
169940,0watchdog/10-21swapper/106:54:391
15249943,0cyclictest6445-21snmpd07:59:082
15249943,0cyclictest6445-21snmpd07:16:382
15249943,0cyclictest6445-21snmpd07:12:382
15249943,0cyclictest6445-21snmpd07:02:372
15249943,0cyclictest3425-21lldpd11:51:162
15249943,0cyclictest31297-21awk07:49:362
15249943,0cyclictest30312-21awk07:24:372
15249943,0cyclictest2617-21awk06:39:362
15249943,0cyclictest24618-21awk07:44:372
15249943,0cyclictest23631-21awk07:19:372
15249943,0cyclictest18812-21awk08:04:362
15239944,0cyclictest0-21swapper/109:39:471
15239943,0cyclictest6445-21snmpd11:31:251
15239943,0cyclictest6445-21snmpd08:34:231
15239943,0cyclictest6445-21snmpd08:19:381
15239943,0cyclictest6445-21snmpd08:12:381
15239943,0cyclictest6445-21snmpd07:24:231
15239943,0cyclictest6445-21snmpd07:19:231
15239943,0cyclictest3705-21awk07:04:401
15239943,0cyclictest31529-21sadc08:59:201
15239943,0cyclictest29987-21awk09:59:361
15239943,0cyclictest29292-21awk06:59:351
15239943,0cyclictest18197-21awk11:14:391
15239943,0cyclictest17962-21awk07:39:381
15239943,0cyclictest12229-21awk07:59:351
15239943,0cyclictest11305-21awk10:49:371
15239940,0cyclictest0-21swapper/108:40:021
15219944,0cyclictest0-21swapper/009:42:220
15219943,0cyclictest9417-21awk06:44:360
15219943,0cyclictest928-21awk12:04:370
15219943,0cyclictest8365-21awk09:04:370
15219943,0cyclictest6558-21awk09:44:380
15219943,0cyclictest6445-21snmpd09:30:390
15219943,0cyclictest6445-21snmpd07:52:230
15219943,0cyclictest3425-21lldpd07:26:420
15219943,0cyclictest3110-21irqbalance11:20:350
15219943,0cyclictest3110-21irqbalance10:24:550
15219943,0cyclictest24724-21awk08:54:380
15219943,0cyclictest16056-21awk06:49:390
15219943,0cyclictest1512-21awk11:24:370
15219943,0cyclictest13401-21awk10:09:370
149940,0migration/05465-21idleruntime10:04:360
149940,0migration/05405-21kthreadcore07:04:420
149940,0migration/04266-21fschecks_time07:29:360
149940,0migration/031553-21kthreadcore06:59:370
149940,0migration/030230-21kthreadcore10:19:390
149940,0migration/026671-21taskset10:15:280
149940,0migration/024937-21kthreadcore09:14:390
149940,0migration/013728-21cpuspeed_turbos09:49:350
4130,0ktimersoftd/00-21swapper/008:06:220
339930,0migration/313919-21taskset06:44:413
28415030,0irq/27-eth0-tx-0-21swapper/007:44:320
28415030,0irq/27-eth0-tx-0-21swapper/007:19:340
28405030,0irq/26-eth0-rx-0-21swapper/308:34:423
28405030,0irq/26-eth0-rx-0-21swapper/308:29:243
28405030,0irq/26-eth0-rx-0-21swapper/307:49:323
28405030,0irq/26-eth0-rx-0-21swapper/307:29:443
28405030,0irq/26-eth0-rx-0-21swapper/307:24:343
28405030,0irq/26-eth0-rx-0-21swapper/306:59:323
28405030,0irq/26-eth0-rx-0-21swapper/306:40:353
259930,0migration/213331-21kthreadcore07:59:372
249930,0watchdog/20-21swapper/207:39:352
169930,0watchdog/124399-21gltestperf07:44:351
169930,0watchdog/10-21swapper/109:44:351
15249933,0cyclictest3425-21lldpd08:26:192
15249932,0cyclictest3761-21basename07:29:342
15249932,0cyclictest22171-21sed06:54:342
15249930,0cyclictest0-21swapper/207:34:402
15239933,0cyclictest32607-21kworker/1:111:49:301
15219932,0cyclictest9717-21awk11:29:380
15219932,0cyclictest927-21sh10:40:110
15219932,0cyclictest8680-21grep10:44:460
15219932,0cyclictest4709-21basename07:54:340
15219932,0cyclictest29818-21sadc09:39:200
15219932,0cyclictest13049-21expr08:24:350
15219932,0cyclictest10313-21awk07:09:360
15219931,0cyclictest20102-21cat10:54:400
15219930,0cyclictest31261-21munin-node08:14:380
15219930,0cyclictest24087-21grep06:54:380
15219930,0cyclictest11321-21munin-node07:59:330
15219930,0cyclictest0-21swapper/008:29:340
329920,0watchdog/30-21swapper/307:09:393
15219921,0cyclictest3425-21lldpd08:49:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional