You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 22:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Sat Nov 22, 2025 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3045928985,0sleep20-21swapper/206:39:192
291350480,0irq/25-eth10-21swapper/306:35:133
3025023726,0sleep00-21swapper/006:36:560
3039323127,0sleep10-21swapper/106:38:331
4160,0ktimersoftd/00-21swapper/010:55:000
339960,0migration/328396-21sh09:00:073
339960,0migration/327812-21sh12:08:363
339960,0migration/322642-21sh11:03:103
305599960,0cyclictest0-21swapper/210:09:382
305559966,0cyclictest0-21swapper/108:09:511
305559960,0cyclictest0-21swapper/111:31:301
28415060,0irq/27-eth0-tx-32584-21ssh10:49:552
28415060,0irq/27-eth0-tx-23378-21munin-node11:04:492
28415060,0irq/27-eth0-tx-15760-21munin-node08:29:550
28415060,0irq/27-eth0-tx-11088-21ssh11:55:512
28415060,0irq/27-eth0-tx-0-21swapper/211:49:572
28415060,0irq/27-eth0-tx-0-21swapper/211:46:422
28415060,0irq/27-eth0-tx-0-21swapper/210:59:092
28415060,0irq/27-eth0-tx-0-21swapper/210:31:382
28415060,0irq/27-eth0-tx-0-21swapper/210:14:592
28415060,0irq/27-eth0-tx-0-21swapper/210:05:052
28415060,0irq/27-eth0-tx-0-21swapper/209:50:472
28415060,0irq/27-eth0-tx-0-21swapper/209:30:112
28415060,0irq/27-eth0-tx-0-21swapper/209:24:482
28415060,0irq/27-eth0-tx-0-21swapper/209:10:372
28415060,0irq/27-eth0-tx-0-21swapper/007:42:170
28405060,0irq/26-eth0-rx-31406-21awk10:24:593
28405060,0irq/26-eth0-rx-14546-21tr11:59:563
19160,0ktimersoftd/10-21swapper/109:49:221
169960,0watchdog/158292chrt07:09:031
149960,0migration/0488-21sh10:06:030
8225050,0irq/24-0000:00:31611-21munin-run07:04:362
8225050,0irq/24-0000:00:14606-21munin-run08:04:362
8225050,0irq/24-0000:00:11561-21latency_hist06:49:362
4150,0ktimersoftd/00-21swapper/010:41:450
339950,0migration/39032-21kthreadcore11:54:543
339950,0migration/36236-21aten_r1power_fr07:09:503
339950,0migration/35698-21fschecks_time06:44:513
339950,0migration/35089-21kthreadcore09:49:533
339950,0migration/33873-21sshd09:05:213
339950,0migration/33562-21kthreadcore07:29:543
339950,0migration/332403-21kthreadcore06:39:543
339950,0migration/331635-21munin-node07:24:573
339950,0migration/324510-21taskset06:55:143
339950,0migration/324447-21kthreadcore07:44:543
339950,0migration/324249-21kthreadcore09:19:573
339950,0migration/322163-21munin-node11:44:533
339950,0migration/32024-21kthreadcore11:29:533
339950,0migration/31951-21kthreadcore08:44:563
339950,0migration/318081-21kthreadcore10:39:533
339950,0migration/316521-21munin-node08:29:513
339950,0migration/31612-21kthreadcore07:04:543
339950,0migration/315243-21kthreadcore07:39:543
339950,0migration/314307-21kthreadcore09:34:553
339950,0migration/313944-21munin-node09:54:523
339950,0migration/312664-21munin-node07:34:573
339950,0migration/311837-21kthreadcore10:34:553
339950,0migration/31166-21kthreadcore11:09:553
339950,0migration/310832-21latency09:09:563
305599955,0cyclictest0-21swapper/210:01:542
305599955,0cyclictest0-21swapper/209:54:482
305599955,0cyclictest0-21swapper/209:36:402
305559955,0cyclictest12330-21kworker/1:006:46:161
305559955,0cyclictest0-21swapper/111:40:481
305559955,0cyclictest0-21swapper/111:14:471
305559955,0cyclictest0-21swapper/109:04:431
305559950,0cyclictest0-21swapper/111:22:501
305539955,0cyclictest0-21swapper/011:27:480
305539955,0cyclictest0-21swapper/011:27:480
305539950,0cyclictest0-21swapper/009:10:380
28415050,0irq/27-eth0-tx-8151-21munin-node07:59:580
28415050,0irq/27-eth0-tx-6372-21iceccd07:11:550
28415050,0irq/27-eth0-tx-26980-21ssh09:20:062
28415050,0irq/27-eth0-tx-26296-21cut08:59:542
28415050,0irq/27-eth0-tx-22949-21diskmemload10:45:112
28415050,0irq/27-eth0-tx-11794-21munin-node06:49:560
28415050,0irq/27-eth0-tx-0-21swapper/211:38:232
28415050,0irq/27-eth0-tx-0-21swapper/211:33:102
28415050,0irq/27-eth0-tx-0-21swapper/211:26:462
28415050,0irq/27-eth0-tx-0-21swapper/211:26:462
28415050,0irq/27-eth0-tx-0-21swapper/211:22:212
28415050,0irq/27-eth0-tx-0-21swapper/211:11:312
28415050,0irq/27-eth0-tx-0-21swapper/211:00:552
28415050,0irq/27-eth0-tx-0-21swapper/210:37:112
28415050,0irq/27-eth0-tx-0-21swapper/210:24:572
28415050,0irq/27-eth0-tx-0-21swapper/210:22:582
28415050,0irq/27-eth0-tx-0-21swapper/209:40:442
28415050,0irq/27-eth0-tx-0-21swapper/209:14:542
28415050,0irq/27-eth0-tx-0-21swapper/209:05:312
28415050,0irq/27-eth0-tx-0-21swapper/208:58:522
28415050,0irq/27-eth0-tx-0-21swapper/208:53:232
28415050,0irq/27-eth0-tx-0-21swapper/208:44:582
28415050,0irq/27-eth0-tx-0-21swapper/008:39:550
28405050,0irq/26-eth0-rx-9914-21proc_pri09:29:583
28405050,0irq/26-eth0-rx-7329-21smartctl08:49:543
28405050,0irq/26-eth0-rx-22188-21cat07:20:003
28405050,0irq/26-eth0-rx-0-21swapper/311:49:523
28405050,0irq/26-eth0-rx-0-21swapper/311:35:223
28405050,0irq/26-eth0-rx-0-21swapper/311:28:253
28405050,0irq/26-eth0-rx-0-21swapper/311:28:253
28405050,0irq/26-eth0-rx-0-21swapper/311:21:263
28405050,0irq/26-eth0-rx-0-21swapper/310:54:553
28405050,0irq/26-eth0-rx-0-21swapper/310:19:483
28405050,0irq/26-eth0-rx-0-21swapper/309:44:553
28405050,0irq/26-eth0-rx-0-21swapper/309:24:443
28405050,0irq/26-eth0-rx-0-21swapper/306:59:533
27150,0ktimersoftd/20-21swapper/210:41:452
27150,0ktimersoftd/20-21swapper/209:49:222
27150,0ktimersoftd/20-21swapper/208:41:252
259950,0migration/215844-21kthreadcore11:39:552
249950,0watchdog/221876-21grep12:04:542
249950,0watchdog/213216-21diskstats07:14:552
19150,0ktimersoftd/10-21swapper/110:41:451
179950,0migration/17667-21kthreadcore09:49:541
179950,0migration/126975-21fschecks_count10:04:511
179950,0migration/115793-21switchtime10:15:011
179950,0migration/111522-21munin-node07:09:561
169950,0watchdog/120406-21perl09:39:541
169950,0watchdog/114482-21perl07:39:511
169950,0watchdog/11263-21cut10:49:581
169950,0watchdog/10-21swapper/111:55:301
169950,0watchdog/10-21swapper/111:36:381
169950,0watchdog/10-21swapper/110:24:461
169950,0watchdog/10-21swapper/109:32:141
169950,0watchdog/10-21swapper/108:04:191
149950,0migration/08654-21kthreadcore09:29:550
149950,0migration/06743-21kthreadcore08:19:540
149950,0migration/04085-21kthreadcore07:54:550
149950,0migration/03368-21runrttasks09:28:040
149950,0migration/032321-21kthreadcore11:29:530
149950,0migration/032172-21irqcore08:44:550
149950,0migration/032122-21cpuspeed_turbos07:04:490
149950,0migration/031208-21irqrtprio09:04:560
149950,0migration/030181-21sh11:07:080
149950,0migration/027812-21aten_r1power_vo09:44:490
149950,0migration/02551-21fschecks_time10:09:500
149950,0migration/025296-21kthreadcore09:19:580
149950,0migration/023364-21kthreadcore08:09:540
149950,0migration/022608-21kthreadcore06:54:540
149950,0migration/022506-21cpuspeed_turbos08:59:490
149950,0migration/022035-21taskset11:01:130
149950,0migration/021617-21irqcore07:44:520
149950,0migration/021031-21taskset08:07:240
149950,0migration/020927-21kthreadcore10:00:030
149950,0migration/02009-21munin-node10:29:520
149950,0migration/017888-21kthreadcore08:54:550
149950,0migration/01621-21irqrtprio07:29:530
149950,0migration/015977-21kthreadcore09:54:530
149950,0migration/014294-21kthreadcore10:34:550
149950,0migration/013237-21sh08:51:340
339940,0migration/39360-21irqrtprio07:59:533
339940,0migration/35486-21kthreadcore10:29:543
339940,0migration/330615-21kthreadcore07:49:543
339940,0migration/330533-21diskmemload11:08:173
339940,0migration/326078-21kthreadcore08:09:553
339940,0migration/319851-21taskset09:39:513
339940,0migration/319766-21irqrtprio10:00:033
339940,0migration/319092-21kthreadcore08:54:563
339940,0migration/317571-21kthreadcore08:04:563
339940,0migration/312931-21aten_r1power_po07:14:523
305599943,0cyclictest6445-21snmpd08:20:092
305599943,0cyclictest6445-21snmpd08:18:242
305599943,0cyclictest6445-21snmpd07:46:542
305599943,0cyclictest3425-21lldpd06:57:492
305599943,0cyclictest14714-21awk07:39:522
305599943,0cyclictest1279-21awk07:29:512
305599942,0cyclictest7919-21sadc07:59:362
305559944,0cyclictest0-21swapper/111:50:061
305559943,0cyclictest6445-21snmpd12:00:101
305559943,0cyclictest3425-21lldpd11:24:371
305559943,0cyclictest3425-21lldpd11:24:371
305559943,0cyclictest3425-21lldpd08:45:021
305559943,0cyclictest32112-21awk11:09:521
305559943,0cyclictest3110-21irqbalance10:44:501
305559943,0cyclictest3110-21irqbalance10:11:001
305559943,0cyclictest3110-21irqbalance09:59:401
305559943,0cyclictest3110-21irqbalance08:39:401
305559943,0cyclictest3110-21irqbalance08:36:101
305559943,0cyclictest3110-21irqbalance08:24:501
305559943,0cyclictest3110-21irqbalance08:20:101
305559943,0cyclictest3110-21irqbalance08:15:101
305559943,0cyclictest3110-21irqbalance08:04:401
305559943,0cyclictest3110-21irqbalance07:54:501
305559943,0cyclictest3110-21irqbalance07:31:301
305559943,0cyclictest3110-21irqbalance07:00:101
305559943,0cyclictest3110-21irqbalance06:54:401
305559943,0cyclictest3110-21irqbalance06:49:501
305559943,0cyclictest28020-21awk07:49:521
305559943,0cyclictest23088-21awk08:59:521
305559943,0cyclictest21639-21awk09:19:551
305559943,0cyclictest20457-21awk07:19:581
305539944,0cyclictest0-21swapper/011:11:050
305539943,0cyclictest6445-21snmpd11:35:550
305539943,0cyclictest6445-21snmpd08:37:540
305539943,0cyclictest6445-21snmpd06:59:380
305539943,0cyclictest6445-21snmpd06:42:230
305539943,0cyclictest595-21awk10:49:570
305539943,0cyclictest3425-21lldpd11:20:330
305539943,0cyclictest3425-21lldpd07:18:520
305539943,0cyclictest29833-21awk11:49:510
305539943,0cyclictest21933-21awk11:44:520
305539943,0cyclictest20740-21awk09:39:560
28415040,0irq/27-eth0-tx-12594-21munin-node11:59:552
28415040,0irq/27-eth0-tx-0-21swapper/211:14:482
28415040,0irq/27-eth0-tx-0-21swapper/007:35:000
28415040,0irq/27-eth0-tx-0-21swapper/007:22:140
28405040,0irq/26-eth0-rx-0-21swapper/311:40:113
28405040,0irq/26-eth0-rx-0-21swapper/311:14:523
28405040,0irq/26-eth0-rx-0-21swapper/310:50:303
28405040,0irq/26-eth0-rx-0-21swapper/310:46:473
28405040,0irq/26-eth0-rx-0-21swapper/310:14:363
28405040,0irq/26-eth0-rx-0-21swapper/310:09:443
28405040,0irq/26-eth0-rx-0-21swapper/310:08:013
28405040,0irq/26-eth0-rx-0-21swapper/309:14:493
28405040,0irq/26-eth0-rx-0-21swapper/308:41:393
259940,0migration/22744-21kthreadcore06:39:552
259940,0migration/222383-21latency_hist08:34:362
259940,0migration/219913-21fschecks_count07:19:562
259940,0migration/218073-21runrttasks06:52:482
259940,0migration/216156-21munin-node08:29:512
259940,0migration/21098-21runrttasks07:54:012
249940,0watchdog/226616-21fschecks_count07:24:512
249940,0watchdog/221984-21fschecks_count08:09:512
249940,0watchdog/215831-21iostat08:04:552
249940,0watchdog/20-21swapper/208:24:592
179940,0migration/19566-21fschecks_time10:34:511
179940,0migration/129140-21fschecks_count09:24:511
179940,0migration/128249-21kthreadcore07:24:541
179940,0migration/126272-21kthreadcore11:04:531
179940,0migration/123246-21kthreadcore07:44:531
179940,0migration/12289-21sh10:29:531
179940,0migration/121433-21kthreadcore10:19:541
179940,0migration/11376-21kthreadcore06:39:541
179940,0migration/113512-21munin-node09:14:531
179940,0migration/112732-21fschecks_time09:34:511
179940,0migration/110159-21kthreadcore09:09:551
169940,0watchdog/124121-21kthreadcore11:44:541
169940,0watchdog/121874-21ps12:04:541
169940,0watchdog/120637-21kthreadcore10:59:541
169940,0watchdog/115500-21cut07:14:591
169940,0watchdog/112853-21ssh08:50:261
169940,0watchdog/10-21swapper/109:54:581
169940,0watchdog/10-21swapper/108:29:551
169940,0watchdog/10-21swapper/107:34:511
149940,0migration/09515-21kthreadcore11:14:580
149940,0migration/08604-21ntp_kernel_err09:49:550
149940,0migration/029642-21kthreadcore07:49:540
149940,0migration/017787-21kthreadcore11:39:550
149940,0migration/011866-21kthreadcore08:24:530
8225030,0irq/24-0000:00:0-21swapper/207:09:472
305599932,0cyclictest32704-21awk07:04:522
305559932,0cyclictest8549-21awk10:54:521
305559930,0cyclictest14927-21diskstats08:54:511
305539933,0cyclictest0-21swapper/011:56:390
305539932,0cyclictest6445-21snmpd08:14:390
305539932,0cyclictest27621-21kthreadcore07:24:540
305539932,0cyclictest26606-21awk10:24:520
305539932,0cyclictest25421-21awk10:44:540
305539932,0cyclictest20538-21munin-node12:04:580
305539932,0cyclictest18387-21sensors12:00:010
305539932,0cyclictest17814-21sensors09:35:000
305539932,0cyclictest14244-21awk09:14:560
305539932,0cyclictest10699-21tail10:14:520
305539930,0cyclictest0-21swapper/010:23:000
28405030,0irq/26-eth0-rx-0-21swapper/308:34:543
28405030,0irq/26-eth0-rx-0-21swapper/308:19:483
28405030,0irq/26-eth0-rx-0-21swapper/307:58:133
28405030,0irq/26-eth0-rx-0-21swapper/306:49:573
249930,0watchdog/20-21swapper/207:34:512
305539920,0cyclictest4891-21latency_hist06:44:360
28405020,0irq/26-eth0-rx-0-21swapper/308:24:443
28405020,0irq/26-eth0-rx-0-21swapper/308:14:523
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional