You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-19 - 02:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Sun Oct 19, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350800,0irq/25-eth10-21swapper/318:40:273
2904124738,0sleep10-21swapper/118:35:561
302923733,0sleep00-21swapper/018:40:510
268323223,0sleep20-21swapper/218:36:572
31069983,0cyclictest2519-21awk21:41:071
31079970,0cyclictest0-21swapper/222:21:182
31069973,0cyclictest6196-21awk19:31:071
31069973,0cyclictest32273-21awk00:01:091
28415070,0irq/27-eth0-tx-0-21swapper/019:16:080
339960,0migration/39039-21fschecks_time23:06:103
339960,0migration/332346-21sh21:36:533
339960,0migration/331628-21munin-node00:01:063
339960,0migration/331091-21sh22:57:103
339960,0migration/324336-21munin-node19:41:153
339960,0migration/323997-21sh23:15:273
339960,0migration/322544-21munin-node22:31:113
339960,0migration/318845-21timerwakeupswit20:01:173
339960,0migration/315837-21sh22:49:243
31079966,0cyclictest0-21swapper/221:11:462
31079960,0cyclictest0-21swapper/222:12:372
31069966,0cyclictest1341-21diskmemload23:40:211
31069966,0cyclictest1336-21sadc20:40:521
31069963,0cyclictest7239-21awk19:56:071
28415060,0irq/27-eth0-tx-0-21swapper/223:54:292
28415060,0irq/27-eth0-tx-0-21swapper/223:31:552
28415060,0irq/27-eth0-tx-0-21swapper/223:23:352
28415060,0irq/27-eth0-tx-0-21swapper/223:15:022
28415060,0irq/27-eth0-tx-0-21swapper/222:34:012
28415060,0irq/27-eth0-tx-0-21swapper/222:30:062
28415060,0irq/27-eth0-tx-0-21swapper/222:10:372
28415060,0irq/27-eth0-tx-0-21swapper/221:51:432
28415060,0irq/27-eth0-tx-0-21swapper/019:46:360
28415060,0irq/27-eth0-tx-0-21swapper/018:45:540
28405060,0irq/26-eth0-rx-30216-21python23:16:203
28405060,0irq/26-eth0-rx-27112-21iostat_ios20:56:103
28405060,0irq/26-eth0-rx-10714-21irqcore21:46:093
249960,0watchdog/20-21swapper/222:44:152
249960,0watchdog/20-21swapper/220:42:432
249960,0watchdog/20-21swapper/220:18:072
20671260,0sleep021698-21kthreadcore20:06:090
149960,0migration/09569-21cpuspeed_turbos22:06:080
339950,0migration/39341-21runrttasks21:04:373
339950,0migration/38622-21taskset21:43:353
339950,0migration/38479-21runrttasks22:04:163
339950,0migration/37997-21fschecks_time20:21:103
339950,0migration/36919-21runrttasks23:02:273
339950,0migration/36441-21kthreadcore19:06:123
339950,0migration/35739-21munin-node19:31:063
339950,0migration/34664-21kthreadcore22:41:133
339950,0migration/331837-21idleruntime19:26:053
339950,0migration/331035-21runrttasks22:36:163
339950,0migration/330923-21kthreadcore20:11:103
339950,0migration/326073-21munin-node22:16:063
339950,0migration/325676-21munin-node20:06:163
339950,0migration/31827-21kthreadcore19:51:113
339950,0migration/317130-21munin-node23:31:093
339950,0migration/313093-21irqcore19:36:093
339950,0migration/311395-21df_abs19:11:063
339950,0migration/310645-21kernelversion22:06:123
339950,0migration/310525-21munin-node20:46:103
31079955,0cyclictest0-21swapper/222:35:572
31079955,0cyclictest0-21swapper/222:01:162
31079955,0cyclictest0-21swapper/220:51:452
31079954,0cyclictest6445-21snmpd18:48:252
31079954,0cyclictest30316-21port_sshpriv22:56:182
31069955,0cyclictest0-21swapper/123:49:391
31069955,0cyclictest0-21swapper/121:03:471
31069955,0cyclictest0-21swapper/120:48:111
31069954,0cyclictest0-21swapper/121:21:081
31069950,0cyclictest0-21swapper/123:21:261
31069950,0cyclictest0-21swapper/121:30:261
31059954,0cyclictest14334-21kthreadcore20:01:110
31059950,0cyclictest0-21swapper/023:46:470
31059950,0cyclictest0-21swapper/023:43:290
31059950,0cyclictest0-21swapper/023:40:100
31059950,0cyclictest0-21swapper/022:48:460
31059950,0cyclictest0-21swapper/021:39:150
28415050,0irq/27-eth0-tx-8643-21munin-node22:46:152
28415050,0irq/27-eth0-tx-6885-21ssh23:02:222
28415050,0irq/27-eth0-tx-6372-21iceccd19:51:090
28415050,0irq/27-eth0-tx-31340-21ssh23:18:362
28415050,0irq/27-eth0-tx-2433-21ssh20:41:080
28415050,0irq/27-eth0-tx-16221-21ssh20:47:242
28415050,0irq/27-eth0-tx-11013-21munin-node19:11:110
28415050,0irq/27-eth0-tx-0-21swapper/223:50:112
28415050,0irq/27-eth0-tx-0-21swapper/223:41:072
28415050,0irq/27-eth0-tx-0-21swapper/223:36:052
28415050,0irq/27-eth0-tx-0-21swapper/223:26:052
28415050,0irq/27-eth0-tx-0-21swapper/222:54:032
28415050,0irq/27-eth0-tx-0-21swapper/221:57:512
28415050,0irq/27-eth0-tx-0-21swapper/221:46:122
28415050,0irq/27-eth0-tx-0-21swapper/221:42:162
28415050,0irq/27-eth0-tx-0-21swapper/221:36:132
28415050,0irq/27-eth0-tx-0-21swapper/221:31:182
28415050,0irq/27-eth0-tx-0-21swapper/221:21:322
28415050,0irq/27-eth0-tx-0-21swapper/221:15:552
28415050,0irq/27-eth0-tx-0-21swapper/020:16:100
28415050,0irq/27-eth0-tx-0-21swapper/018:51:070
28405050,0irq/26-eth0-rx-9694-21ssh21:25:443
28405050,0irq/26-eth0-rx-9484-21grep22:26:073
28405050,0irq/26-eth0-rx-8260-21sh22:24:223
28405050,0irq/26-eth0-rx-7310-21proc_pri20:41:153
28405050,0irq/26-eth0-rx-4042-21kthreadcore23:41:123
28405050,0irq/26-eth0-rx-26721-21diskstats21:16:063
28405050,0irq/26-eth0-rx-23820-21munin-node21:31:153
28405050,0irq/26-eth0-rx-19668-21tr20:51:113
28405050,0irq/26-eth0-rx-11383-21kthreadcore23:46:133
28405050,0irq/26-eth0-rx-0-21swapper/323:51:193
28405050,0irq/26-eth0-rx-0-21swapper/323:22:233
28405050,0irq/26-eth0-rx-0-21swapper/322:53:333
28405050,0irq/26-eth0-rx-0-21swapper/322:13:223
28405050,0irq/26-eth0-rx-0-21swapper/318:46:113
27150,0ktimersoftd/20-21swapper/221:04:212
259950,0migration/232382-21kthreadcore19:26:072
249950,0watchdog/21322-21kthreadcore00:01:102
249950,0watchdog/20-21swapper/223:57:302
249950,0watchdog/20-21swapper/223:07:032
249950,0watchdog/20-21swapper/222:16:512
249950,0watchdog/20-21swapper/221:26:112
249950,0watchdog/20-21swapper/221:06:112
249950,0watchdog/20-21swapper/220:58:272
249950,0watchdog/20-21swapper/220:26:032
249950,0watchdog/20-21swapper/219:57:072
249950,0watchdog/20-21swapper/219:36:032
249950,0watchdog/20-21swapper/219:25:352
249950,0watchdog/20-21swapper/219:19:152
249950,0watchdog/20-21swapper/219:06:072
249950,0watchdog/20-21swapper/218:56:112
249950,0watchdog/20-21swapper/200:10:182
149950,0migration/07247-21kthreadcore21:21:120
149950,0migration/06345-21sh00:05:130
149950,0migration/06226-21switchtime23:01:180
149950,0migration/031710-21taskset22:38:020
149950,0migration/028251-21kthreadcore20:56:110
149950,0migration/027774-21kthreadcore23:56:110
149950,0migration/027630-21cpuspeed_turbos20:36:070
149950,0migration/026765-21irqcore21:56:090
149950,0migration/026739-21kthreadcore22:16:090
149950,0migration/026271-21kthreadcore22:56:140
149950,0migration/024423-21sh21:53:010
149950,0migration/02427-21munin-node22:01:110
149950,0migration/023798-21sh23:35:500
149950,0migration/023729-21sh22:32:590
149950,0migration/022068-21kthreadcore21:31:110
149950,0migration/020073-21kthreadcore22:11:100
149950,0migration/019354-21needreboot20:26:110
149950,0migration/018521-21kthreadcore23:11:070
149950,0migration/016459-21idleruntime23:51:090
149950,0migration/015883-21switchtime21:26:160
149950,0migration/012703-21kthreadcore22:26:110
149950,0migration/011445-21kthreadcore20:46:120
149950,0migration/011251-21kthreadcore20:21:140
149950,0migration/01102-21aten_r1power_vo22:21:050
149950,0migration/010690-21fschecks_time21:06:090
339940,0migration/323938-21kthreadcore20:31:123
339940,0migration/321420-21kthreadcore21:51:103
339940,0migration/318638-21kthreadcore20:26:103
31079944,0cyclictest0-21swapper/218:41:122
31079943,0cyclictest3425-21lldpd20:36:152
31079943,0cyclictest31780-21sadc19:50:522
31079943,0cyclictest27200-21awk20:11:072
31079943,0cyclictest13982-21awk20:01:102
31079943,0cyclictest11853-21awk19:11:082
31069944,0cyclictest0-21swapper/122:42:101
31069944,0cyclictest0-21swapper/121:57:141
31069944,0cyclictest0-21swapper/121:48:531
31069944,0cyclictest0-21swapper/121:38:101
31069944,0cyclictest0-21swapper/121:18:081
31069944,0cyclictest0-21swapper/119:41:111
31069943,0cyclictest7691-21awk00:06:091
31069943,0cyclictest6445-21snmpd20:26:561
31069943,0cyclictest5120-21sadc19:30:521
31069943,0cyclictest3425-21lldpd23:57:321
31069943,0cyclictest3425-21lldpd22:59:321
31069943,0cyclictest3425-21lldpd22:49:021
31069943,0cyclictest3425-21lldpd21:51:161
31069943,0cyclictest3425-21lldpd21:06:151
31069943,0cyclictest3425-21lldpd20:57:151
31069943,0cyclictest3425-21lldpd20:55:151
31069943,0cyclictest3425-21lldpd20:22:441
31069943,0cyclictest3425-21lldpd19:04:131
31069943,0cyclictest27185-21awk20:11:071
31069943,0cyclictest2599-21awk20:41:101
31069943,0cyclictest17061-21sadc22:10:521
31069943,0cyclictest16964-21awk23:31:081
31069943,0cyclictest16109-21sadc23:10:521
31069943,0cyclictest10030-21awk22:26:091
31069940,0cyclictest0-21swapper/118:43:101
31059944,0cyclictest0-21swapper/021:47:360
31059943,0cyclictest9396-21awk23:06:120
31059943,0cyclictest6445-21snmpd22:51:120
31059943,0cyclictest4327-21munin-node19:06:180
31059943,0cyclictest32007-21awk19:26:060
31059943,0cyclictest21462-21awk20:31:100
31059940,0cyclictest29989-21munin-node19:01:170
31059940,0cyclictest0-21swapper/023:22:530
28415040,0irq/27-eth0-tx-0-21swapper/019:41:050
28405040,0irq/26-eth0-rx-9997-21kthreadcore19:56:103
28405040,0irq/26-eth0-rx-0-21swapper/323:56:103
28405040,0irq/26-eth0-rx-0-21swapper/323:38:043
28405040,0irq/26-eth0-rx-0-21swapper/323:26:143
28405040,0irq/26-eth0-rx-0-21swapper/321:55:533
28405040,0irq/26-eth0-rx-0-21swapper/321:27:243
28405040,0irq/26-eth0-rx-0-21swapper/321:11:053
28405040,0irq/26-eth0-rx-0-21swapper/321:06:493
28405040,0irq/26-eth0-rx-0-21swapper/319:05:363
28405040,0irq/26-eth0-rx-0-21swapper/318:51:083
28405040,0irq/26-eth0-rx-0-21swapper/300:06:183
259940,0migration/23909-21runrttasks19:03:522
249940,0watchdog/20-21swapper/220:31:112
249940,0watchdog/20-21swapper/220:21:072
249940,0watchdog/20-21swapper/220:05:552
249940,0watchdog/20-21swapper/219:51:232
249940,0watchdog/20-21swapper/219:41:072
249940,0watchdog/20-21swapper/219:31:112
249940,0watchdog/20-21swapper/218:51:232
169940,0watchdog/10-21swapper/122:51:191
169940,0watchdog/10-21swapper/122:11:111
149940,0migration/09246-21kthreadcore19:56:100
149940,0migration/07674-21idleruntime00:06:090
149940,0migration/05029-21kthreadcore18:41:110
149940,0migration/04844-21kthreadcore21:41:100
31069933,0cyclictest3425-21lldpd23:41:021
31069933,0cyclictest3425-21lldpd20:19:141
31069933,0cyclictest3425-21lldpd19:50:441
31069933,0cyclictest3425-21lldpd19:25:131
31069932,0cyclictest6445-21snmpd19:05:581
31069932,0cyclictest3425-21lldpd21:35:161
31069932,0cyclictest3425-21lldpd20:07:451
31069932,0cyclictest32240-21cpuspeed_turbos19:51:071
31069932,0cyclictest27804-21grep22:36:101
31069932,0cyclictest25310-21awk23:16:121
31069932,0cyclictest18945-21tune2fs21:11:081
31069932,0cyclictest17616-21awk18:51:101
31069932,0cyclictest13951-21awk20:01:101
31069932,0cyclictest12930-21awk19:36:081
31069932,0cyclictest12191-21ps19:11:101
31069932,0cyclictest11224-21irqrtprio18:46:111
31069930,0cyclictest1956-21ssh22:21:101
31059933,0cyclictest0-21swapper/021:03:180
31059932,0cyclictest647-21ssh21:18:070
31059932,0cyclictest6445-21snmpd18:55:560
31059932,0cyclictest30198-21sensors23:16:200
31059932,0cyclictest30031-21sensors19:21:140
31059932,0cyclictest20414-21grep21:11:110
31059932,0cyclictest18757-21awk20:51:090
31059932,0cyclictest1575-21awk22:41:110
31059932,0cyclictest13904-21sensors23:26:160
31059930,0cyclictest0-21swapper/020:11:110
31059930,0cyclictest0-21swapper/019:31:060
28415030,0irq/27-eth0-tx-0-21swapper/019:36:050
28405030,0irq/26-eth0-rx-0-21swapper/320:36:073
28405030,0irq/26-eth0-rx-0-21swapper/320:16:043
28405030,0irq/26-eth0-rx-0-21swapper/319:50:353
28405030,0irq/26-eth0-rx-0-21swapper/319:21:013
28405030,0irq/26-eth0-rx-0-21swapper/319:17:123
28405030,0irq/26-eth0-rx-0-21swapper/318:59:213
28405030,0irq/26-eth0-rx-0-21swapper/318:41:063
169930,0watchdog/113770-21ssh23:26:151
169930,0watchdog/10-21swapper/123:54:021
169930,0watchdog/10-21swapper/123:10:551
169930,0watchdog/10-21swapper/123:01:031
169930,0watchdog/10-21swapper/122:31:031
169930,0watchdog/10-21swapper/122:15:551
169930,0watchdog/10-21swapper/122:01:071
169930,0watchdog/10-21swapper/120:31:591
169930,0watchdog/10-21swapper/118:56:071
169920,0watchdog/10-21swapper/119:15:551
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional