You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-16 - 11:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Sun Nov 16, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1023228281,0sleep00-21swapper/018:36:540
291350790,0irq/25-eth10-21swapper/318:39:053
1037923425,0sleep10-21swapper/118:38:391
1040123228,0sleep20-21swapper/218:38:522
28415070,0irq/27-eth0-tx-0-21swapper/223:06:572
249970,0watchdog/227813-21yum23:50:162
339960,0migration/33650-21sh23:56:363
339960,0migration/331071-21switchtime21:10:163
339960,0migration/316339-21munin-node22:05:033
339960,0migration/315082-21sh21:20:273
339960,0migration/313561-21sh23:44:453
28415060,0irq/27-eth0-tx-7849-21munin-node22:20:162
28415060,0irq/27-eth0-tx-14104-21ssh23:03:002
28415060,0irq/27-eth0-tx-12323-21cut21:20:092
28415060,0irq/27-eth0-tx-0-21swapper/223:58:492
28415060,0irq/27-eth0-tx-0-21swapper/223:41:322
28415060,0irq/27-eth0-tx-0-21swapper/223:32:252
28415060,0irq/27-eth0-tx-0-21swapper/223:26:152
28415060,0irq/27-eth0-tx-0-21swapper/223:20:002
28415060,0irq/27-eth0-tx-0-21swapper/223:15:492
28415060,0irq/27-eth0-tx-0-21swapper/222:55:532
28415060,0irq/27-eth0-tx-0-21swapper/222:46:362
28415060,0irq/27-eth0-tx-0-21swapper/222:36:492
28415060,0irq/27-eth0-tx-0-21swapper/222:17:432
28415060,0irq/27-eth0-tx-0-21swapper/222:05:142
28415060,0irq/27-eth0-tx-0-21swapper/222:00:092
28415060,0irq/27-eth0-tx-0-21swapper/221:54:182
28415060,0irq/27-eth0-tx-0-21swapper/221:48:492
28415060,0irq/27-eth0-tx-0-21swapper/221:31:332
28415060,0irq/27-eth0-tx-0-21swapper/221:15:232
28415060,0irq/27-eth0-tx-0-21swapper/221:06:102
28415060,0irq/27-eth0-tx-0-21swapper/221:02:362
28415060,0irq/27-eth0-tx-0-21swapper/220:51:162
28415060,0irq/27-eth0-tx-0-21swapper/220:40:152
28415060,0irq/27-eth0-tx-0-21swapper/200:05:502
28415060,0irq/27-eth0-tx-0-21swapper/020:09:530
28415060,0irq/27-eth0-tx-0-21swapper/019:24:450
28405060,0irq/26-eth0-rx-5870-21ssh21:55:133
28405060,0irq/26-eth0-rx-5870-21ssh21:55:133
28405060,0irq/26-eth0-rx-4803-21unixbench_singl23:15:163
28405060,0irq/26-eth0-rx-30102-21sh22:51:403
249960,0watchdog/28998-21iostat20:15:092
169960,0watchdog/1913-21chrt20:08:531
105579966,0cyclictest0-21swapper/221:44:422
8225050,0irq/24-0000:00:4842-21latency_hist18:59:502
4150,0ktimersoftd/00-21swapper/021:57:170
4150,0ktimersoftd/00-21swapper/021:57:170
339950,0migration/37218-21aten_r1power_vo23:00:023
339950,0migration/36651-21taskset19:48:383
339950,0migration/36158-21kthreadcore20:35:093
339950,0migration/35704-21idleruntime00:00:103
339950,0migration/33900-21kthreadcore21:15:103
339950,0migration/33116-21kthreadcore18:55:123
339950,0migration/326930-21if_eth1.219:40:083
339950,0migration/326052-21switchtime20:00:193
339950,0migration/323677-21taskset21:27:173
339950,0migration/323446-21munin-node23:10:053
339950,0migration/32344-21kthreadcore22:35:063
339950,0migration/32143-21kernelversion20:55:073
339950,0migration/320038-21taskset23:25:103
339950,0migration/319310-21missed_timers23:45:113
339950,0migration/319295-21kthreadcore21:45:073
339950,0migration/317748-21munin-node18:45:063
339950,0migration/315119-21aten_r1power_fr23:05:033
339950,0migration/313260-21kthreadcore19:05:093
28415050,0irq/27-eth0-tx-6846-21ssh21:56:302
28415050,0irq/27-eth0-tx-6846-21ssh21:56:302
28415050,0irq/27-eth0-tx-29548-21ssh23:11:162
28415050,0irq/27-eth0-tx-25096-21munin-node19:15:160
28415050,0irq/27-eth0-tx-2252-21munin-node20:35:110
28415050,0irq/27-eth0-tx-20342-21munin-node20:00:130
28415050,0irq/27-eth0-tx-0-21swapper/223:36:342
28415050,0irq/27-eth0-tx-0-21swapper/222:51:152
28415050,0irq/27-eth0-tx-0-21swapper/222:42:192
28415050,0irq/27-eth0-tx-0-21swapper/222:30:012
28415050,0irq/27-eth0-tx-0-21swapper/222:25:272
28415050,0irq/27-eth0-tx-0-21swapper/222:10:202
28415050,0irq/27-eth0-tx-0-21swapper/221:35:322
28415050,0irq/27-eth0-tx-0-21swapper/221:14:172
28415050,0irq/27-eth0-tx-0-21swapper/220:57:322
28415050,0irq/27-eth0-tx-0-21swapper/220:46:062
28405050,0irq/26-eth0-rx-32418-21grep23:35:093
28405050,0irq/26-eth0-rx-30724-21munin-node21:30:173
28405050,0irq/26-eth0-rx-29226-21ssh23:31:393
28405050,0irq/26-eth0-rx-15634-21munin-node00:05:093
28405050,0irq/26-eth0-rx-0-21swapper/323:50:063
28405050,0irq/26-eth0-rx-0-21swapper/323:20:093
28405050,0irq/26-eth0-rx-0-21swapper/322:56:533
28405050,0irq/26-eth0-rx-0-21swapper/322:40:043
28405050,0irq/26-eth0-rx-0-21swapper/322:26:423
28405050,0irq/26-eth0-rx-0-21swapper/322:15:083
28405050,0irq/26-eth0-rx-0-21swapper/322:10:553
28405050,0irq/26-eth0-rx-0-21swapper/320:50:033
28405050,0irq/26-eth0-rx-0-21swapper/320:44:183
259950,0migration/224876-21munin-node18:50:052
259950,0migration/224115-21munin-node19:10:162
249950,0watchdog/223269-21timerwakeupswit18:45:172
21656250,0sleep221834-21grep20:25:082
19150,0ktimersoftd/10-21swapper/122:18:071
179950,0migration/1773-21kthreadcore19:20:071
179950,0migration/130730-21munin-node21:50:161
179950,0migration/130686-21kthreadcore19:40:111
179950,0migration/129974-21munin-node22:30:161
179950,0migration/125182-21aten_r1power_fr20:50:011
179950,0migration/124117-21munin-node19:10:161
179950,0migration/123972-21fschecks_count22:50:051
179950,0migration/116719-21sh21:45:011
179950,0migration/115778-21sh21:43:081
179950,0migration/113779-21latency_hist23:44:501
169950,0watchdog/14256-21kthreadcore21:35:091
169950,0watchdog/118750-21ssh00:05:441
169950,0watchdog/118643-21kthreadcore22:45:091
169950,0watchdog/10-21swapper/122:38:411
169950,0watchdog/10-21swapper/122:11:011
169950,0watchdog/10-21swapper/122:03:011
169950,0watchdog/10-21swapper/121:30:411
169950,0watchdog/10-21swapper/121:01:171
169950,0watchdog/10-21swapper/120:00:131
149950,0migration/08404-21munin-node20:15:060
149950,0migration/05717-21munin-node00:00:100
149950,0migration/032625-21rt-features20:05:140
149950,0migration/029470-21sh23:32:250
149950,0migration/023110-21sh21:46:140
149950,0migration/02090-21kthreadcore22:15:080
149950,0migration/019447-21kthreadcore21:25:090
149950,0migration/016783-21vmstat18:40:170
149950,0migration/014406-21sh20:40:070
149950,0migration/014021-21sh22:41:270
149950,0migration/012442-21kthreadcore22:20:100
105579950,0cyclictest0-21swapper/223:47:502
105569955,0cyclictest0-21swapper/121:27:591
105569954,0cyclictest8569-21sadc21:19:501
105559955,0cyclictest0-21swapper/000:08:280
105559950,0cyclictest0-21swapper/022:12:350
105559950,0cyclictest0-21swapper/020:55:200
339940,0migration/39340-21kthreadcore19:50:093
339940,0migration/3521-21irqrtprio19:20:073
339940,0migration/3396-21sh21:34:523
339940,0migration/325779-21fschecks_time19:15:063
339940,0migration/325297-21fschecks_time21:50:063
339940,0migration/320491-21kthreadcore20:45:133
339940,0migration/315958-21fschecks_count22:45:053
339940,0migration/315486-21kthreadcore19:30:063
339940,0migration/313633-21kthreadcore18:40:103
28415040,0irq/27-eth0-tx-0-21swapper/200:00:182
28415040,0irq/27-eth0-tx-0-21swapper/019:40:050
28415040,0irq/27-eth0-tx-0-21swapper/019:10:060
28415040,0irq/27-eth0-tx-0-21swapper/019:09:420
28405040,0irq/26-eth0-rx-13946-21munin-node22:20:163
28405040,0irq/26-eth0-rx-10307-21kthreadcore20:15:103
28405040,0irq/26-eth0-rx-0-21swapper/322:30:003
28405040,0irq/26-eth0-rx-0-21swapper/321:59:573
28405040,0irq/26-eth0-rx-0-21swapper/321:40:553
28405040,0irq/26-eth0-rx-0-21swapper/321:06:513
28405040,0irq/26-eth0-rx-0-21swapper/321:01:163
28405040,0irq/26-eth0-rx-0-21swapper/320:30:073
28405040,0irq/26-eth0-rx-0-21swapper/320:09:533
28405040,0irq/26-eth0-rx-0-21swapper/319:10:043
259940,0migration/26280-21kthreadcore20:35:092
259940,0migration/231653-21kthreadcore20:30:082
259940,0migration/219999-21kthreadcore21:25:092
249940,0watchdog/26547-21runrttasks19:55:052
249940,0watchdog/26547-21runrttasks18:59:532
249940,0watchdog/232222-21sshd19:42:372
249940,0watchdog/225295-21cpu19:15:052
249940,0watchdog/223980-21kthreadcore20:00:132
249940,0watchdog/212695-21munin-node19:30:012
179940,0migration/19700-21kthreadcore19:50:091
179940,0migration/17239-21fschecks_count23:20:041
179940,0migration/15616-21kthreadcore20:55:081
179940,0migration/1524-21kthreadcore23:15:091
179940,0migration/128965-21if_eth1.320:30:061
179940,0migration/123612-21kthreadcore23:50:081
179940,0migration/120433-21smart_sda23:25:141
179940,0migration/119146-21kthreadcore21:05:071
179940,0migration/116355-21kernelversion23:05:081
179940,0migration/116322-21proc_pri18:40:141
179940,0migration/115732-21latency_hist22:24:501
179940,0migration/112129-21taskset19:26:041
179940,0migration/111937-21aten_r1power_vo19:05:051
179940,0migration/110203-21kthreadcore22:20:091
169940,0watchdog/18085-21perl22:40:051
169940,0watchdog/17968-21ssh21:59:451
169940,0watchdog/17968-21ssh21:59:451
169940,0watchdog/17053-21kthreadcore20:35:091
169940,0watchdog/16180-21irqstats00:00:121
169940,0watchdog/12562-21kthreadcore20:10:091
169940,0watchdog/112326-21cut21:20:091
169940,0watchdog/10-21swapper/123:55:041
169940,0watchdog/10-21swapper/123:35:041
169940,0watchdog/10-21swapper/123:33:001
169940,0watchdog/10-21swapper/123:10:051
169940,0watchdog/10-21swapper/123:02:371
169940,0watchdog/10-21swapper/119:55:131
169940,0watchdog/10-21swapper/118:45:171
149940,0migration/09389-21kthreadcore23:00:070
149940,0migration/09257-21if_eth1.222:00:060
149940,0migration/05422-21diskmemload23:16:570
149940,0migration/04790-21taskset18:59:290
149940,0migration/031460-21taskset22:55:050
149940,0migration/028787-21kthreadcore18:50:090
149940,0migration/028545-21kthreadcore21:10:110
149940,0migration/027889-21kthreadcore23:10:090
149940,0migration/025204-21kthreadcore22:30:070
149940,0migration/016569-21kthreadcore20:20:070
149940,0migration/014928-21taskset21:40:390
149940,0migration/012188-21taskset19:26:430
149940,0migration/010813-21kthreadcore19:50:090
105579943,0cyclictest7872-21awk19:50:072
105579943,0cyclictest6445-21snmpd20:23:532
105579943,0cyclictest6445-21snmpd20:09:532
105579943,0cyclictest6445-21snmpd20:04:532
105579943,0cyclictest6445-21snmpd19:36:082
105579943,0cyclictest6445-21snmpd19:06:532
105579943,0cyclictest1157-21awk19:45:072
105569944,0cyclictest3425-21lldpd20:26:411
105569943,0cyclictest6445-21snmpd23:40:101
105569943,0cyclictest6445-21snmpd22:55:091
105569943,0cyclictest6445-21snmpd20:20:381
105569943,0cyclictest6445-21snmpd19:35:381
105569943,0cyclictest3425-21lldpd19:31:101
105569943,0cyclictest26132-21awk21:10:081
105569943,0cyclictest26053-21awk19:15:081
105569943,0cyclictest24947-21awk18:50:061
105569943,0cyclictest17900-21awk20:45:111
105569943,0cyclictest17267-21awk22:05:071
105569943,0cyclictest1118-21awk19:45:071
105559944,0cyclictest0-21swapper/023:53:570
105559944,0cyclictest0-21swapper/021:08:530
105559943,0cyclictest7595-21awk23:20:060
105559943,0cyclictest3425-21lldpd20:47:110
105559943,0cyclictest3110-21irqbalance23:35:340
105559943,0cyclictest3110-21irqbalance23:08:440
105559943,0cyclictest3110-21irqbalance21:21:140
105559943,0cyclictest30384-21awk23:55:070
105559943,0cyclictest20178-21awk19:35:060
105559943,0cyclictest18306-21awk18:45:090
105559943,0cyclictest17241-21awk22:05:070
105559943,0cyclictest16376-21awk22:45:060
105559943,0cyclictest15642-21awk23:25:060
105559943,0cyclictest1381-21awk21:35:060
105559943,0cyclictest1205-21awk19:45:080
105559940,0cyclictest0-21swapper/019:34:500
28405030,0irq/26-eth0-rx-0-21swapper/320:05:053
28405030,0irq/26-eth0-rx-0-21swapper/319:57:383
28405030,0irq/26-eth0-rx-0-21swapper/319:35:033
28405030,0irq/26-eth0-rx-0-21swapper/319:25:063
28405030,0irq/26-eth0-rx-0-21swapper/319:00:023
28405030,0irq/26-eth0-rx-0-21swapper/318:50:023
249930,0watchdog/20-21swapper/219:20:092
169930,0watchdog/10-21swapper/118:59:531
149930,0migration/017154-21kthreadcore19:55:090
105579930,0cyclictest6670-21perl19:25:072
105569933,0cyclictest30044-21kworker/1:120:14:521
105569932,0cyclictest31697-21awk18:55:101
105569930,0cyclictest9608-21perl20:40:031
105559933,0cyclictest0-21swapper/021:15:020
105559932,0cyclictest9939-21awk21:00:070
105559932,0cyclictest6951-21awk23:40:090
105559932,0cyclictest6355-21ssh22:37:280
105559932,0cyclictest30913-21sensors20:50:120
105559932,0cyclictest29529-21grep21:50:100
105559932,0cyclictest28871-21awk20:30:060
105559932,0cyclictest25786-21awk21:30:080
105559932,0cyclictest24460-21awk22:50:070
105559932,0cyclictest22173-21awk20:25:090
105559932,0cyclictest16744-21awk22:25:090
105559932,0cyclictest13858-21ssh23:44:590
105559930,0cyclictest5070-21munin-node19:00:030
28405020,0irq/26-eth0-rx-0-21swapper/320:24:513
28405020,0irq/26-eth0-rx-0-21swapper/320:19:503
105579920,0cyclictest10710-21latency_hist18:39:502
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional