You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-03 - 09:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Thu Jul 03, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350970,0irq/25-eth10-21swapper/318:43:213
291350970,0irq/25-eth10-21swapper/318:43:213
1363025251,0sleep10-21swapper/118:42:101
1363025251,0sleep10-21swapper/118:42:101
1362423622,0sleep00-21swapper/018:42:050
1362423622,0sleep00-21swapper/018:42:050
1371223324,0sleep20-21swapper/218:43:072
1371223324,0sleep20-21swapper/218:43:072
35160,0ktimersoftd/39527-21sh00:09:303
35160,0ktimersoftd/30-21swapper/321:07:433
339960,0migration/36150-21sh21:04:113
339960,0migration/327579-21sh22:35:543
339960,0migration/326872-21sh23:16:223
339960,0migration/321072-21sh20:51:193
339960,0migration/317813-21sh23:51:563
339960,0migration/314119-21sh21:29:243
28415060,0irq/27-eth0-tx-4321-21munin-node19:55:160
28415060,0irq/27-eth0-tx-10016-21munin-node19:35:100
28415060,0irq/27-eth0-tx-0-21swapper/223:41:402
28415060,0irq/27-eth0-tx-0-21swapper/223:22:442
28415060,0irq/27-eth0-tx-0-21swapper/223:05:232
28415060,0irq/27-eth0-tx-0-21swapper/222:40:582
28415060,0irq/27-eth0-tx-0-21swapper/222:18:202
28415060,0irq/27-eth0-tx-0-21swapper/222:11:432
28415060,0irq/27-eth0-tx-0-21swapper/221:59:552
28415060,0irq/27-eth0-tx-0-21swapper/221:50:502
28415060,0irq/27-eth0-tx-0-21swapper/221:47:202
28415060,0irq/27-eth0-tx-0-21swapper/221:41:122
28415060,0irq/27-eth0-tx-0-21swapper/221:13:422
28415060,0irq/27-eth0-tx-0-21swapper/220:49:572
28405060,0irq/26-eth0-rx-31095-21munin-node22:00:113
28405060,0irq/26-eth0-rx-0-21swapper/323:40:493
249960,0watchdog/2116842sleep220:23:372
249960,0watchdog/20-21swapper/222:55:202
249960,0watchdog/20-21swapper/222:53:002
249960,0watchdog/20-21swapper/222:25:492
249960,0watchdog/20-21swapper/221:33:132
249960,0watchdog/20-21swapper/221:25:132
149960,0migration/031079-21aten_r1power_en20:15:050
149960,0migration/03089-21sh23:23:200
149960,0migration/029035-21latency_hist22:39:540
149960,0migration/028485-21sh22:17:040
149960,0migration/016257-21irqrtprio19:10:100
149960,0migration/011914-21sh22:26:290
4150,0ktimersoftd/00-21swapper/021:55:340
339950,0migration/38493-21kthreadcore20:45:143
339950,0migration/38441-21aten_r1power_cu19:05:063
339950,0migration/34404-21runrttasks21:40:233
339950,0migration/3437-21munin-node20:40:113
339950,0migration/33573-21kthreadcore21:20:123
339950,0migration/33487-21aten_r1power_vo19:30:023
339950,0migration/3338-21kthreadcore20:15:113
339950,0migration/331790-21taskset20:39:443
339950,0migration/331425-21kthreadcore22:20:153
339950,0migration/330557-21kthreadcore22:40:113
339950,0migration/323761-21kthreadcore23:55:123
339950,0migration/32328-21timerwakeupswit23:00:183
339950,0migration/322798-21irqcore22:15:153
339950,0migration/321105-21taskset19:10:353
339950,0migration/320765-21switchtime21:10:193
339950,0migration/320497-21taskset18:48:523
339950,0migration/319855-21kthreadcore21:30:133
339950,0migration/318032-21kthreadcore21:50:133
339950,0migration/315950-21unixbench_multi00:10:213
339950,0migration/315039-21kthreadcore20:25:113
339950,0migration/313694-21kthreadcore23:30:133
339950,0migration/312210-21kthreadcore19:35:123
339950,0migration/310401-21unixbench-2d23:05:173
28415050,0irq/27-eth0-tx-3642-21ssh22:20:462
28415050,0irq/27-eth0-tx-3347-21ssh23:45:082
28415050,0irq/27-eth0-tx-2204-21ssh00:04:572
28415050,0irq/27-eth0-tx-21719-21munin-node19:15:180
28415050,0irq/27-eth0-tx-18502-21ssh23:31:292
28415050,0irq/27-eth0-tx-14150-21munin-node18:45:090
28415050,0irq/27-eth0-tx-0-21swapper/223:28:222
28415050,0irq/27-eth0-tx-0-21swapper/223:15:252
28415050,0irq/27-eth0-tx-0-21swapper/222:38:382
28415050,0irq/27-eth0-tx-0-21swapper/222:31:522
28415050,0irq/27-eth0-tx-0-21swapper/222:05:472
28415050,0irq/27-eth0-tx-0-21swapper/221:55:522
28415050,0irq/27-eth0-tx-0-21swapper/221:16:222
28415050,0irq/27-eth0-tx-0-21swapper/220:56:282
28415050,0irq/27-eth0-tx-0-21swapper/220:45:112
28415050,0irq/27-eth0-tx-0-21swapper/020:05:180
28415050,0irq/27-eth0-tx-0-21swapper/019:45:040
28405050,0irq/26-eth0-rx-9439-21kthreadcore22:45:153
28405050,0irq/26-eth0-rx-8038-21grep23:25:143
28405050,0irq/26-eth0-rx-30010-21latency_hist19:49:543
28405050,0irq/26-eth0-rx-26599-21tr21:35:143
28405050,0irq/26-eth0-rx-25701-21uname23:35:183
28405050,0irq/26-eth0-rx-22995-21irqstats19:15:123
28405050,0irq/26-eth0-rx-0-21swapper/323:12:193
28405050,0irq/26-eth0-rx-0-21swapper/322:55:393
28405050,0irq/26-eth0-rx-0-21swapper/322:53:293
28405050,0irq/26-eth0-rx-0-21swapper/322:10:073
28405050,0irq/26-eth0-rx-0-21swapper/321:55:163
28405050,0irq/26-eth0-rx-0-21swapper/321:47:183
28405050,0irq/26-eth0-rx-0-21swapper/320:55:333
28405050,0irq/26-eth0-rx-0-21swapper/300:01:363
27150,0ktimersoftd/20-21swapper/223:01:292
259950,0migration/232143-21irqrtprio21:20:112
259950,0migration/226567-21kthreadcore21:35:142
249950,0watchdog/20-21swapper/223:55:202
249950,0watchdog/20-21swapper/223:51:522
249950,0watchdog/20-21swapper/223:35:482
249950,0watchdog/20-21swapper/223:10:122
249950,0watchdog/20-21swapper/221:05:252
249950,0watchdog/20-21swapper/221:04:532
249950,0watchdog/20-21swapper/220:25:292
249950,0watchdog/20-21swapper/220:14:132
249950,0watchdog/20-21swapper/219:55:372
249950,0watchdog/20-21swapper/219:25:132
249950,0watchdog/20-21swapper/219:21:132
249950,0watchdog/20-21swapper/219:14:572
249950,0watchdog/20-21swapper/219:10:492
249950,0watchdog/20-21swapper/219:08:252
249950,0watchdog/20-21swapper/218:54:572
249950,0watchdog/20-21swapper/218:50:132
249950,0watchdog/20-21swapper/218:44:572
249950,0watchdog/20-21swapper/200:12:002
249950,0watchdog/20-21swapper/200:01:322
19150,0ktimersoftd/10-21swapper/123:01:291
179950,0migration/15901-21kthreadcore19:55:111
179950,0migration/132205-21kthreadcore21:40:121
179950,0migration/119818-21latency_hist23:34:541
179950,0migration/113566-21latency_hist22:09:541
169950,0watchdog/125092-21kthreadcore22:55:121
169950,0watchdog/112420-21yum21:45:211
149950,0migration/04860-21aten_r1power_cu22:45:080
149950,0migration/0362-21hwlatdetect20:40:110
149950,0migration/031176-21irqcore19:50:100
149950,0migration/029991-21kthreadcore18:55:110
149950,0migration/027823-21kthreadcore21:15:120
149950,0migration/027297-21sh23:17:350
149950,0migration/021966-21munin-node22:55:110
149950,0migration/020379-21sh21:50:250
149950,0migration/02018-21kthreadcore21:00:130
149950,0migration/016637-21kthreadcore23:10:130
149950,0migration/016044-21kthreadcore23:50:120
149950,0migration/016044-21kthreadcore22:50:120
149950,0migration/014893-21kthreadcore22:30:140
149950,0migration/01437-21ntp_offset19:20:150
149950,0migration/011876-21unixbench_multi22:05:180
149950,0migration/010322-21kthreadcore21:25:130
149950,0migration/010017-21aten_r1power_fr00:10:090
139419953,0cyclictest3425-21lldpd22:45:232
139419950,0cyclictest0-21swapper/219:30:052
139399955,0cyclictest0-21swapper/122:48:441
139399954,0cyclictest22895-21ps19:15:121
139399950,0cyclictest0-21swapper/121:35:061
139399950,0cyclictest0-21swapper/121:28:191
139399950,0cyclictest0-21swapper/121:07:591
139399950,0cyclictest0-21swapper/121:03:151
139389955,0cyclictest0-21swapper/023:06:220
139389955,0cyclictest0-21swapper/021:07:200
139389955,0cyclictest0-21swapper/020:45:060
139389950,0cyclictest0-21swapper/023:01:470
339940,0migration/39134-21munin-node23:45:193
339940,0migration/329058-21kthreadcore18:55:113
339940,0migration/326107-21kthreadcore20:10:113
339940,0migration/324794-21kthreadcore21:15:113
339940,0migration/322942-21runrttasks19:43:143
339940,0migration/321413-21fschecks_time18:50:063
28415040,0irq/27-eth0-tx-0-21swapper/019:00:120
28415040,0irq/27-eth0-tx-0-21swapper/018:54:310
28405040,0irq/26-eth0-rx-0-21swapper/323:21:323
28405040,0irq/26-eth0-rx-0-21swapper/322:29:593
28405040,0irq/26-eth0-rx-0-21swapper/322:25:023
28405040,0irq/26-eth0-rx-0-21swapper/322:05:023
259940,0migration/231077-21munin-node20:15:052
259940,0migration/22066-21kthreadcore20:40:132
249940,0watchdog/20-21swapper/220:35:012
249940,0watchdog/20-21swapper/220:05:172
249940,0watchdog/20-21swapper/220:00:012
249940,0watchdog/20-21swapper/219:49:572
179940,0migration/19824-21kthreadcore19:05:121
179940,0migration/19430-21runrttasks19:30:551
179940,0migration/18742-21kthreadcore20:45:141
179940,0migration/17978-21runrttasks19:02:291
179940,0migration/126107-21taskset23:58:181
179940,0migration/124351-21kthreadcore21:15:111
179940,0migration/124057-21missed_timers20:30:151
179940,0migration/123035-21munin-node20:05:161
179940,0migration/11949-21kthreadcore21:20:121
179940,0migration/118308-21kthreadcore22:10:181
179940,0migration/115529-21fschecks_time21:10:101
179940,0migration/110716-21runrttasks23:05:491
169940,0watchdog/15730-21diskstats22:25:091
169940,0watchdog/13882-21ssh22:21:291
169940,0watchdog/132231-21cpu20:40:091
169940,0watchdog/130521-21diskstats22:00:091
169940,0watchdog/128516-21ssh21:56:011
169940,0watchdog/125723-21kthreadcore22:15:171
169940,0watchdog/122239-21processes19:40:171
169940,0watchdog/121233-21ssh22:34:571
169940,0watchdog/120168-21uptime18:45:171
169940,0watchdog/119857-21port_sshpriv21:50:171
169940,0watchdog/113735-21kthreadcore23:10:121
149940,0migration/09550-21sh23:45:310
149940,0migration/09039-21runrttasks20:20:130
149940,0migration/025404-21missed_timers23:35:150
149940,0migration/021291-21sh21:11:020
149940,0migration/014658-21kthreadcore20:25:110
139419943,0cyclictest3425-21lldpd19:35:072
139419943,0cyclictest3110-21irqbalance19:44:592
139419943,0cyclictest3110-21irqbalance19:40:092
139419943,0cyclictest3110-21irqbalance19:00:092
139419943,0cyclictest19657-21awk20:30:122
139399944,0cyclictest0-21swapper/123:49:071
139399943,0cyclictest7227-21kworker/1:119:10:241
139399943,0cyclictest6445-21snmpd20:35:351
139399943,0cyclictest6445-21snmpd20:15:051
139399943,0cyclictest6445-21snmpd00:02:361
139399943,0cyclictest6276-21awk20:20:111
139399943,0cyclictest3425-21lldpd23:29:551
139399943,0cyclictest3425-21lldpd20:54:091
139399943,0cyclictest30802-21kworker/1:223:25:181
139399943,0cyclictest30802-21kworker/1:200:06:381
139399943,0cyclictest28737-21awk23:40:101
139399943,0cyclictest25225-21awk20:10:091
139399943,0cyclictest21373-21awk23:15:131
139399943,0cyclictest15473-21awk21:30:101
139399943,0cyclictest10853-21awk19:35:101
139389944,0cyclictest0-21swapper/022:20:390
139389944,0cyclictest0-21swapper/021:45:100
139389944,0cyclictest0-21swapper/021:31:370
139389943,0cyclictest9357-21awk19:05:100
139389943,0cyclictest3425-21lldpd20:50:390
139389943,0cyclictest2969-21sadc23:44:540
139389943,0cyclictest27860-21awk00:00:100
139389943,0cyclictest25240-21awk20:10:090
139389943,0cyclictest19633-21awk20:30:120
139389943,0cyclictest17567-21awk19:40:110
28415030,0irq/27-eth0-tx-0-21swapper/020:00:060
28405030,0irq/26-eth0-rx-0-21swapper/320:30:083
28405030,0irq/26-eth0-rx-0-21swapper/320:21:053
28405030,0irq/26-eth0-rx-0-21swapper/319:55:053
28405030,0irq/26-eth0-rx-0-21swapper/319:20:073
28405030,0irq/26-eth0-rx-0-21swapper/319:00:073
169930,0watchdog/10-21swapper/122:30:091
169930,0watchdog/10-21swapper/120:55:091
139399933,0cyclictest7227-21kworker/1:119:49:561
139399933,0cyclictest7227-21kworker/1:119:44:561
139399933,0cyclictest7227-21kworker/1:119:25:001
139399933,0cyclictest7227-21kworker/1:119:19:561
139399933,0cyclictest7227-21kworker/1:118:54:561
139399933,0cyclictest7227-21kworker/1:118:49:561
139399933,0cyclictest3425-21lldpd23:19:551
139399933,0cyclictest30802-21kworker/1:222:49:581
139399933,0cyclictest30802-21kworker/1:220:24:561
139399933,0cyclictest30802-21kworker/1:219:59:561
139399933,0cyclictest30802-21kworker/1:200:10:141
139399932,0cyclictest313-21kthreadcore22:40:121
139399932,0cyclictest16943-21sensors23:50:161
139389933,0cyclictest19027-21munin-node23:55:130
139389933,0cyclictest0-21swapper/021:38:230
139389932,0cyclictest6445-21snmpd22:00:050
139389932,0cyclictest6445-21snmpd19:30:010
139389932,0cyclictest6429-21kthreadcore23:25:140
139389932,0cyclictest31843-21awk21:20:100
139389932,0cyclictest30091-21awk22:40:090
139389932,0cyclictest23582-21awk20:55:090
139389932,0cyclictest15856-21cut23:30:140
139389932,0cyclictest0-21swapper/020:35:150
139389930,0cyclictest0-21swapper/021:40:190
329920,0watchdog/323055-21missed_timers20:05:173
329920,0watchdog/30-21swapper/319:45:133
28405020,0irq/26-eth0-rx-0-21swapper/319:59:563
28405020,0irq/26-eth0-rx-0-21swapper/319:25:393
139389922,0cyclictest0-21swapper/022:10:070
139389921,0cyclictest6547-21runrttasks19:24:590
139389920,0cyclictest1968-21crond00:04:540
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional