You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-17 - 19:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Mon Nov 17, 2025 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
507124948,0sleep20-21swapper/206:36:562
351450,0ktimersoftd/331097-21perf06:39:373
520523727,0sleep10-21swapper/106:38:271
505923624,0sleep00-21swapper/006:36:460
249970,0watchdog/230410-40yum-updatesd-he11:22:572
53909960,0cyclictest0-21swapper/210:01:212
53889960,0cyclictest0-21swapper/012:01:460
339960,0migration/330697-21sh11:23:483
339960,0migration/315057-21aten_r1power_fr11:35:023
339960,0migration/310157-21sh09:01:063
28415060,0irq/27-eth0-tx-25348-21ssh09:31:202
28415060,0irq/27-eth0-tx-0-21swapper/212:01:212
28415060,0irq/27-eth0-tx-0-21swapper/211:50:082
28415060,0irq/27-eth0-tx-0-21swapper/211:26:112
28415060,0irq/27-eth0-tx-0-21swapper/210:34:462
28415060,0irq/27-eth0-tx-0-21swapper/210:26:482
28415060,0irq/27-eth0-tx-0-21swapper/210:12:402
28415060,0irq/27-eth0-tx-0-21swapper/209:37:352
28415060,0irq/27-eth0-tx-0-21swapper/209:27:182
28415060,0irq/27-eth0-tx-0-21swapper/209:19:412
28415060,0irq/27-eth0-tx-0-21swapper/208:43:002
28415060,0irq/27-eth0-tx-0-21swapper/007:55:010
28415060,0irq/27-eth0-tx-0-21swapper/007:00:000
28405060,0irq/26-eth0-rx-9953-21perl10:30:033
28405060,0irq/26-eth0-rx-32390-21irqstats11:45:053
28405060,0irq/26-eth0-rx-21183-21python11:15:103
28405060,0irq/26-eth0-rx-16739-21diskstats10:55:023
28405060,0irq/26-eth0-rx-0-21swapper/310:16:593
169960,0watchdog/131335-21grep07:45:101
169960,0watchdog/124517-21cpuspeed_turbos11:00:011
169960,0watchdog/119615-21fschecks_time09:30:051
169960,0watchdog/10-21swapper/111:49:411
149960,0migration/013260-21sh11:50:480
53909955,0cyclictest0-21swapper/211:45:152
53909955,0cyclictest0-21swapper/207:30:092
53899955,0cyclictest0-21swapper/107:25:051
53899954,0cyclictest28524-21cut09:15:041
53899950,0cyclictest0-21swapper/107:30:071
53889955,0cyclictest6445-21snmpd09:13:010
53889950,0cyclictest0-21swapper/010:45:460
53889950,0cyclictest0-21swapper/009:22:410
4150,0ktimersoftd/00-21swapper/010:02:060
339950,0migration/38241-21taskset07:53:113
339950,0migration/34218-21kthreadcore09:40:073
339950,0migration/328364-21kthreadcore11:00:063
339950,0migration/326381-21kthreadcore10:40:043
339950,0migration/326249-21munin-node10:20:013
339950,0migration/322458-21taskset08:26:213
339950,0migration/321083-21irqcore07:15:063
339950,0migration/320102-21munin-node09:10:013
339950,0migration/319579-21latency_hist09:09:473
339950,0migration/319274-21kthreadcore11:55:063
339950,0migration/319222-21cpuspeed_turbos09:30:043
339950,0migration/318140-21kthreadcore07:10:083
339950,0migration/316157-21kthreadcore08:45:063
28415050,0irq/27-eth0-tx-9680-21munin-node08:20:000
28415050,0irq/27-eth0-tx-3660-21munin-node09:00:102
28415050,0irq/27-eth0-tx-31997-21munin-node11:05:032
28415050,0irq/27-eth0-tx-2940-21munin-node08:15:010
28415050,0irq/27-eth0-tx-23123-21telnet08:30:020
28415050,0irq/27-eth0-tx-21452-21ssh11:56:042
28415050,0irq/27-eth0-tx-0-21swapper/212:05:472
28415050,0irq/27-eth0-tx-0-21swapper/211:15:022
28415050,0irq/27-eth0-tx-0-21swapper/211:11:502
28415050,0irq/27-eth0-tx-0-21swapper/211:02:022
28415050,0irq/27-eth0-tx-0-21swapper/210:57:352
28415050,0irq/27-eth0-tx-0-21swapper/210:51:092
28415050,0irq/27-eth0-tx-0-21swapper/210:47:592
28415050,0irq/27-eth0-tx-0-21swapper/210:44:332
28415050,0irq/27-eth0-tx-0-21swapper/210:36:242
28415050,0irq/27-eth0-tx-0-21swapper/210:21:292
28415050,0irq/27-eth0-tx-0-21swapper/210:04:482
28415050,0irq/27-eth0-tx-0-21swapper/209:46:132
28415050,0irq/27-eth0-tx-0-21swapper/209:40:302
28415050,0irq/27-eth0-tx-0-21swapper/209:10:112
28415050,0irq/27-eth0-tx-0-21swapper/209:07:082
28415050,0irq/27-eth0-tx-0-21swapper/208:57:252
28415050,0irq/27-eth0-tx-0-21swapper/208:57:252
28415050,0irq/27-eth0-tx-0-21swapper/208:53:242
28415050,0irq/27-eth0-tx-0-21swapper/208:45:362
28415050,0irq/27-eth0-tx-0-21swapper/007:50:150
28415050,0irq/27-eth0-tx-0-21swapper/007:46:260
28415050,0irq/27-eth0-tx-0-21swapper/007:15:100
28415050,0irq/27-eth0-tx-0-21swapper/007:12:370
28405050,0irq/26-eth0-rx-8825-21nfs4_client09:20:073
28405050,0irq/26-eth0-rx-7758-21cpu11:09:593
28405050,0irq/26-eth0-rx-7538-21ssh10:47:563
28405050,0irq/26-eth0-rx-6821-21ssh11:08:123
28405050,0irq/26-eth0-rx-113012sleep308:43:493
28405050,0irq/26-eth0-rx-0-21swapper/312:00:553
28405050,0irq/26-eth0-rx-0-21swapper/311:50:043
28405050,0irq/26-eth0-rx-0-21swapper/311:40:003
28405050,0irq/26-eth0-rx-0-21swapper/311:25:003
28405050,0irq/26-eth0-rx-0-21swapper/310:53:043
28405050,0irq/26-eth0-rx-0-21swapper/310:29:253
28405050,0irq/26-eth0-rx-0-21swapper/310:00:013
28405050,0irq/26-eth0-rx-0-21swapper/309:50:173
28405050,0irq/26-eth0-rx-0-21swapper/308:50:083
259950,0migration/219826-21munin-node07:35:092
249950,0watchdog/224801-21ssh10:16:452
249950,0watchdog/222357-21kthreadcore09:55:052
249950,0watchdog/216704-21kthreadcore09:50:052
179950,0migration/17275-21timerandwakeup10:25:121
179950,0migration/14300-21kthreadcore09:40:071
179950,0migration/132061-21munin-node12:05:061
179950,0migration/125178-21kthreadcore11:20:051
179950,0migration/123732-21gltestperf08:30:041
179950,0migration/123159-21kthreadcore06:50:051
179950,0migration/121829-21needreboot10:55:071
179950,0migration/117754-21kthreadcore11:55:051
179950,0migration/112442-21kthreadcore10:10:051
179950,0migration/110993-21missed_timers06:40:071
179950,0migration/11093-21sh08:55:081
179950,0migration/11093-21sh08:55:081
179950,0migration/110716-21irqcore08:20:041
169950,0watchdog/17916-21irqcore11:50:051
169950,0watchdog/121429-21kthreadcore08:25:051
169950,0watchdog/110937-21ssh09:03:211
169950,0watchdog/110530-21ssh09:23:491
169950,0watchdog/10-21swapper/111:15:411
169950,0watchdog/10-21swapper/109:25:491
169950,0watchdog/10-21swapper/109:05:251
169950,0watchdog/10-21swapper/108:47:131
169950,0watchdog/10-21swapper/108:40:051
149950,0migration/08978-21turbostat10:29:460
149950,0migration/04049-21aten_r1power_fr08:40:000
149950,0migration/032024-21missed_timers07:20:070
149950,0migration/031496-21munin-node11:45:020
149950,0migration/031470-21ksmtuned11:04:030
149950,0migration/031165-21taskset10:40:180
149950,0migration/02933-21aten2.4-expect10:04:590
149950,0migration/029107-21kthreadcore09:35:070
149950,0migration/02773-21kthreadcore07:25:070
149950,0migration/027571-21kthreadcore10:20:050
149950,0migration/024881-21sh10:17:000
149950,0migration/024195-21sh10:38:030
149950,0migration/023314-21taskset11:19:500
149950,0migration/023171-21kthreadcore07:40:050
149950,0migration/022420-21kthreadcore09:55:050
149950,0migration/02083-21taskset08:56:220
149950,0migration/02083-21taskset08:56:220
149950,0migration/02074-21timerandwakeup08:10:260
149950,0migration/019677-21sh08:49:440
149950,0migration/019340-21kthreadcore08:00:060
149950,0migration/015907-21sh10:31:380
149950,0migration/015633-21fschecks_time11:55:020
149950,0migration/013949-21sh11:32:350
149950,0migration/011623-21latency_hist09:04:470
8225040,0irq/24-0000:00:0-21swapper/207:04:562
53909944,0cyclictest3721-21diskmemload11:35:002
53909943,0cyclictest6445-21snmpd06:53:512
53909943,0cyclictest3425-21lldpd09:21:462
53909943,0cyclictest3425-21lldpd08:21:352
53909943,0cyclictest3425-21lldpd08:18:052
53909943,0cyclictest3425-21lldpd07:19:572
53909943,0cyclictest3425-21lldpd06:55:572
53909943,0cyclictest3425-21lldpd06:40:252
53909943,0cyclictest30579-21awk08:35:042
53909943,0cyclictest28628-21awk07:45:082
53909943,0cyclictest20929-21awk07:15:062
53909943,0cyclictest17217-21awk08:25:022
53909943,0cyclictest13245-21awk06:45:022
53899944,0cyclictest0-21swapper/111:33:421
53899943,0cyclictest7570-21awk07:05:051
53899943,0cyclictest3425-21lldpd11:04:471
53899943,0cyclictest3425-21lldpd10:05:001
53899943,0cyclictest21915-21awk07:40:031
53899943,0cyclictest20838-21kworker/1:209:56:261
53899943,0cyclictest20838-21kworker/1:209:35:011
53899943,0cyclictest20838-21kworker/1:208:51:031
53899943,0cyclictest20838-21kworker/1:208:35:051
53899943,0cyclictest16258-21awk08:00:041
53899943,0cyclictest15769-21kworker/1:111:37:261
53899940,0cyclictest0-21swapper/110:01:001
53889943,0cyclictest3425-21lldpd11:13:470
53889943,0cyclictest3425-21lldpd09:00:170
53889943,0cyclictest3425-21lldpd06:57:570
53889943,0cyclictest3110-21irqbalance11:15:410
53889943,0cyclictest3110-21irqbalance10:49:510
53889943,0cyclictest3110-21irqbalance09:31:510
53889943,0cyclictest3110-21irqbalance09:26:110
53889943,0cyclictest3110-21irqbalance08:49:510
53889943,0cyclictest30628-21awk08:35:050
53889943,0cyclictest2968-21awk09:40:050
53889943,0cyclictest23835-21awk11:40:030
53889943,0cyclictest22844-21awk08:05:060
53889943,0cyclictest19878-21awk06:50:030
53889943,0cyclictest17208-21awk10:55:030
53889943,0cyclictest13269-21awk06:45:030
53889943,0cyclictest13050-21sadc07:09:470
53889943,0cyclictest12561-21awk09:50:020
339940,0migration/39405-21hwlatdetect07:55:033
339940,0migration/39062-21kthreadcore07:05:063
339940,0migration/33949-21irqcore10:05:033
339940,0migration/331848-21fschecks_time12:05:053
339940,0migration/331173-21kthreadcore08:35:063
339940,0migration/31737-21latency_hist07:49:473
339940,0migration/31614-21missed_timers08:10:103
339940,0migration/312561-21kthreadcore09:25:083
339940,0migration/31173-21irqstats07:00:053
28415040,0irq/27-eth0-tx-7800-21aten2.4-expect07:30:000
28415040,0irq/27-eth0-tx-0-21swapper/211:40:022
28415040,0irq/27-eth0-tx-0-21swapper/008:25:040
28415040,0irq/27-eth0-tx-0-21swapper/006:39:590
28405040,0irq/26-eth0-rx-0-21swapper/311:29:473
28405040,0irq/26-eth0-rx-0-21swapper/310:34:593
28405040,0irq/26-eth0-rx-0-21swapper/310:10:003
28405040,0irq/26-eth0-rx-0-21swapper/309:56:113
28405040,0irq/26-eth0-rx-0-21swapper/309:45:463
28405040,0irq/26-eth0-rx-0-21swapper/309:34:543
28405040,0irq/26-eth0-rx-0-21swapper/309:15:053
28405040,0irq/26-eth0-rx-0-21swapper/308:56:123
28405040,0irq/26-eth0-rx-0-21swapper/308:56:123
28405040,0irq/26-eth0-rx-0-21swapper/308:15:063
259940,0migration/27640-21fschecks_time11:30:012
259940,0migration/228670-21switchtime08:30:122
259940,0migration/228110-21runrttasks08:06:482
259940,0migration/217330-21kthreadcore07:10:072
259940,0migration/213019-21kthreadcore07:55:062
249940,0watchdog/22758-21kthreadcore07:00:062
249940,0watchdog/20-21swapper/208:00:052
249940,0watchdog/20-21swapper/207:52:412
249940,0watchdog/20-21swapper/207:40:022
179940,0migration/18153-21taskset07:52:101
179940,0migration/17511-21latency_hist11:09:461
179940,0migration/1384-21taskset07:00:021
179940,0migration/129898-21kthreadcore10:40:051
179940,0migration/128881-21cpuspeed_turbos08:10:031
179940,0migration/127328-21kernelversion10:20:051
179940,0migration/119302-21kthreadcore10:35:051
179940,0migration/114364-21irqcore07:10:061
169940,0watchdog/119767-21ssh09:09:571
169940,0watchdog/118552-21diskstats10:15:011
169940,0watchdog/116079-21ssh10:32:131
169940,0watchdog/10-21swapper/112:01:251
169940,0watchdog/10-21swapper/111:39:531
169940,0watchdog/10-21swapper/111:24:571
169940,0watchdog/10-21swapper/110:44:571
169940,0watchdog/10-21swapper/109:50:051
169940,0watchdog/10-21swapper/108:05:091
169940,0watchdog/10-21swapper/107:55:051
169940,0watchdog/10-21swapper/107:20:061
169940,0watchdog/10-21swapper/106:55:061
149940,0migration/032049-21kthreadcore09:15:080
149940,0migration/0317-21kthreadcore12:05:080
149940,0migration/017037-21kthreadcore07:35:060
53909933,0cyclictest0-21swapper/207:24:482
53909932,0cyclictest413-21kthreadcore08:10:092
53899933,0cyclictest20838-21kworker/1:210:49:501
53899933,0cyclictest20838-21kworker/1:209:44:461
53899933,0cyclictest20838-21kworker/1:206:45:051
53889933,0cyclictest3425-21lldpd11:36:170
53889932,0cyclictest6445-21snmpd11:05:520
53889932,0cyclictest2455-21kthreadcore11:25:050
53889932,0cyclictest11427-21sadc09:49:460
53889932,0cyclictest10612-21tail10:09:580
339930,0migration/326644-21munin-node07:40:103
339930,0migration/322983-21taskset08:29:533
339930,0migration/321708-21runrttasks08:04:463
339930,0migration/320571-21runrttasks07:37:363
28405030,0irq/26-eth0-rx-0-21swapper/308:22:503
28405030,0irq/26-eth0-rx-0-21swapper/308:05:073
28405030,0irq/26-eth0-rx-0-21swapper/307:30:013
28405030,0irq/26-eth0-rx-0-21swapper/307:25:193
28405030,0irq/26-eth0-rx-0-21swapper/306:55:013
28405030,0irq/26-eth0-rx-0-21swapper/306:49:593
28405030,0irq/26-eth0-rx-0-21swapper/306:44:583
28405030,0irq/26-eth0-rx-0-21swapper/306:40:083
169930,0watchdog/10-21swapper/108:15:051
169930,0watchdog/10-21swapper/107:35:021
169930,0watchdog/10-21swapper/107:15:061
329920,0watchdog/329613-21cut07:20:053
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional