You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-01 - 08:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2s.osadl.org (updated Tue Jul 01, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2871127473,0sleep00-21swapper/018:44:300
2870427473,0sleep30-21swapper/318:44:253
2840926765,0sleep20-21swapper/218:40:562
2858623324,0sleep10-21swapper/118:42:591
288279977,0cyclictest29037-21sensors19:05:230
28415070,0irq/27-eth0-tx-0-21swapper/223:38:562
339960,0migration/36955-21sh22:11:243
339960,0migration/324384-21sh22:45:163
339960,0migration/31501-21sh20:46:273
339960,0migration/310626-21idleruntime-cro21:14:593
288299960,0cyclictest0-21swapper/219:35:212
288289960,0cyclictest8943-21timerwakeupswit21:10:261
28415060,0irq/27-eth0-tx-0-21swapper/223:43:502
28415060,0irq/27-eth0-tx-0-21swapper/223:26:222
28415060,0irq/27-eth0-tx-0-21swapper/223:18:252
28415060,0irq/27-eth0-tx-0-21swapper/223:09:092
28415060,0irq/27-eth0-tx-0-21swapper/223:01:242
28415060,0irq/27-eth0-tx-0-21swapper/222:44:582
28415060,0irq/27-eth0-tx-0-21swapper/222:30:472
28415060,0irq/27-eth0-tx-0-21swapper/222:29:282
28415060,0irq/27-eth0-tx-0-21swapper/221:56:502
28415060,0irq/27-eth0-tx-0-21swapper/221:46:432
28415060,0irq/27-eth0-tx-0-21swapper/221:44:552
28415060,0irq/27-eth0-tx-0-21swapper/221:39:362
28415060,0irq/27-eth0-tx-0-21swapper/221:25:112
28415060,0irq/27-eth0-tx-0-21swapper/221:22:192
28415060,0irq/27-eth0-tx-0-21swapper/221:22:192
28415060,0irq/27-eth0-tx-0-21swapper/220:55:022
28415060,0irq/27-eth0-tx-0-21swapper/200:01:392
28405060,0irq/26-eth0-rx-27706-21irqcore21:25:163
249960,0watchdog/20-21swapper/222:45:212
249960,0watchdog/20-21swapper/222:06:452
249960,0watchdog/20-21swapper/221:00:092
249960,0watchdog/20-21swapper/220:53:412
8225050,0irq/24-0000:00:19609-21cat20:14:592
4150,0ktimersoftd/00-21swapper/022:47:540
339950,0migration/37258-21latency_hist20:29:593
339950,0migration/36857-21kthreadcore21:30:203
339950,0migration/35835-21taskset20:02:193
339950,0migration/35103-21munin-node19:15:183
339950,0migration/331633-21sh22:29:303
339950,0migration/326492-21kthreadcore19:05:183
339950,0migration/322594-21taskset23:48:503
339950,0migration/320436-21kthreadcore22:20:193
339950,0migration/31911-21kthreadcore19:10:233
339950,0migration/317758-21latency_hist21:39:593
339950,0migration/315580-21kthreadcore19:20:163
339950,0migration/315014-21kthreadcore20:55:203
339950,0migration/314114-21munin-node23:20:123
339950,0migration/313619-21kthreadcore22:15:193
339950,0migration/313413-21latency_hist23:39:583
339950,0migration/312366-21missed_timers22:55:213
339950,0migration/311791-21munin-node23:15:203
288299955,0cyclictest0-21swapper/223:12:552
288299955,0cyclictest0-21swapper/220:40:182
288289955,0cyclictest0-21swapper/121:51:441
288289950,0cyclictest0-21swapper/123:46:311
288289950,0cyclictest0-21swapper/122:42:401
288289950,0cyclictest0-21swapper/121:09:011
288289950,0cyclictest0-21swapper/100:13:001
288289950,0cyclictest0-21swapper/100:08:071
288279955,0cyclictest0-21swapper/023:57:060
288279955,0cyclictest0-21swapper/022:06:350
288279950,0cyclictest0-21swapper/023:12:010
28415050,0irq/27-eth0-tx-5332-21munin-node19:40:162
28415050,0irq/27-eth0-tx-4389-21ssh22:50:182
28415050,0irq/27-eth0-tx-25156-21kthreadcore23:50:152
28415050,0irq/27-eth0-tx-23374-21munin-node19:05:152
28415050,0irq/27-eth0-tx-22612-21ssh22:00:232
28415050,0irq/27-eth0-tx-0-21swapper/223:55:242
28415050,0irq/27-eth0-tx-0-21swapper/223:33:012
28415050,0irq/27-eth0-tx-0-21swapper/222:20:002
28415050,0irq/27-eth0-tx-0-21swapper/221:11:342
28415050,0irq/27-eth0-tx-0-21swapper/221:05:422
28415050,0irq/27-eth0-tx-0-21swapper/220:45:342
28415050,0irq/27-eth0-tx-0-21swapper/200:10:432
28415050,0irq/27-eth0-tx-0-21swapper/200:05:032
28405050,0irq/26-eth0-rx-6481-21sh22:32:183
28405050,0irq/26-eth0-rx-30826-21if_err_eth1.323:30:153
28405050,0irq/26-eth0-rx-23353-21kthreadcore21:00:183
28405050,0irq/26-eth0-rx-21124-21port_sshpriv22:40:233
28405050,0irq/26-eth0-rx-20325-21munin-node23:00:193
28405050,0irq/26-eth0-rx-12397-21diskstats19:45:143
28405050,0irq/26-eth0-rx-1051-21cat23:10:163
28405050,0irq/26-eth0-rx-0-21swapper/323:55:103
28405050,0irq/26-eth0-rx-0-21swapper/323:50:163
28405050,0irq/26-eth0-rx-0-21swapper/321:55:113
28405050,0irq/26-eth0-rx-0-21swapper/321:51:273
28405050,0irq/26-eth0-rx-0-21swapper/321:46:513
28405050,0irq/26-eth0-rx-0-21swapper/321:36:333
28405050,0irq/26-eth0-rx-0-21swapper/321:20:283
28405050,0irq/26-eth0-rx-0-21swapper/321:20:283
28405050,0irq/26-eth0-rx-0-21swapper/320:15:173
27150,0ktimersoftd/20-21swapper/221:51:152
259950,0migration/25660-21kthreadcore22:10:202
259950,0migration/24319-21if_eth1.318:50:132
259950,0migration/22257-21kthreadcore20:00:182
259950,0migration/222005-21kthreadcore19:50:202
249950,0watchdog/284122chrt21:31:092
249950,0watchdog/217954-21kthreadcore20:10:172
249950,0watchdog/215646-21kthreadcore20:35:172
249950,0watchdog/20-21swapper/223:45:172
249950,0watchdog/20-21swapper/223:20:132
249950,0watchdog/20-21swapper/222:56:332
249950,0watchdog/20-21swapper/222:35:172
249950,0watchdog/20-21swapper/222:19:172
249950,0watchdog/20-21swapper/221:19:452
249950,0watchdog/20-21swapper/220:06:172
249950,0watchdog/20-21swapper/219:32:532
249950,0watchdog/20-21swapper/219:25:412
249950,0watchdog/20-21swapper/219:21:572
249950,0watchdog/20-21swapper/219:15:212
19150,0ktimersoftd/10-21swapper/120:53:431
179950,0migration/15889-21latency20:25:161
179950,0migration/13305-21kthreadcore19:35:161
179950,0migration/131873-21kthreadcore23:30:161
179950,0migration/127503-21kthreadcore19:55:201
179950,0migration/124330-21aten_r1power_po22:25:121
179950,0migration/119580-21idleruntime21:20:191
179950,0migration/119580-21idleruntime21:20:191
179950,0migration/119103-21fschecks_count19:50:161
179950,0migration/117254-21if_eth122:20:171
179950,0migration/110114-21sh21:55:181
169950,0watchdog/116414-21needreboot19:20:171
169950,0watchdog/112922-21grep22:55:251
149950,0migration/07962-21kthreadcore20:05:180
149950,0migration/07179-21sh22:34:260
149950,0migration/04084-21netstat19:35:170
149950,0migration/032498-21kthreadcore18:45:180
149950,0migration/030148-21kthreadcore20:20:170
149950,0migration/029535-21kthreadcore20:45:180
149950,0migration/02884-21latency23:30:180
149950,0migration/027849-21kthreadcore23:05:180
149950,0migration/027183-21kthreadcore23:25:190
149950,0migration/025688-21cpuspeed_turbos19:55:150
149950,0migration/025017-21idleruntime22:25:150
149950,0migration/024183-21kthreadcore20:40:190
149950,0migration/023396-21kthreadcore21:20:220
149950,0migration/023396-21kthreadcore21:20:220
149950,0migration/023324-21aten_r1power_fr23:50:100
149950,0migration/023299-21runrttasks22:00:510
149950,0migration/023263-21munin-node21:40:200
149950,0migration/022923-21taskset19:02:370
149950,0migration/020621-21kthreadcore19:50:200
149950,0migration/019801-21aten_r1power_po20:15:110
149950,0migration/019644-21df_abs21:00:130
149950,0migration/019448-21kthreadcore22:40:200
149950,0migration/01920-21sh21:30:020
149950,0migration/016394-21taskset21:58:560
149950,0migration/016372-21needreboot21:15:180
149950,0migration/015241-21irqrtprio23:20:170
149950,0migration/013650-21taskset20:34:030
149950,0migration/013494-21kthreadcore19:45:170
149950,0migration/011642-21kthreadcore21:35:160
149950,0migration/01057-21fschecks_time22:10:160
339940,0migration/36933-21taskset20:28:013
339940,0migration/36344-21kthreadcore18:50:153
339940,0migration/35888-21df_abs23:35:113
339940,0migration/330497-21kthreadcore20:20:173
339940,0migration/328970-21kthreadcore19:55:203
339940,0migration/327097-21taskset20:44:513
339940,0migration/320299-21kthreadcore22:00:203
339940,0migration/319035-21kthreadcore19:25:163
339940,0migration/314151-21kthreadcore21:15:173
288299943,0cyclictest3110-21irqbalance20:31:032
288299943,0cyclictest3110-21irqbalance19:10:432
288299943,0cyclictest27203-21awk20:20:142
288299943,0cyclictest26219-21awk19:55:182
288289943,0cyclictest8223-21awk20:30:151
288289943,0cyclictest6445-21snmpd23:11:401
288289943,0cyclictest6445-21snmpd22:32:251
288289943,0cyclictest6445-21snmpd21:37:241
288289943,0cyclictest6445-21snmpd21:30:241
288289943,0cyclictest6445-21snmpd20:20:541
288289943,0cyclictest6445-21snmpd20:17:241
288289943,0cyclictest16546-21kworker/1:023:44:001
288289943,0cyclictest16546-21kworker/1:023:27:181
288289943,0cyclictest16546-21kworker/1:020:57:041
288289940,0cyclictest5406-21sh23:56:451
288279944,0cyclictest0-21swapper/021:45:180
288279944,0cyclictest0-21swapper/000:00:150
288279943,0cyclictest6445-21snmpd22:19:400
288279943,0cyclictest6445-21snmpd21:10:090
288279943,0cyclictest5195-21awk19:15:190
288279943,0cyclictest3425-21lldpd18:50:580
288279943,0cyclictest23264-21awk00:10:130
288279943,0cyclictest17179-21awk22:20:170
288279943,0cyclictest15745-21awk23:00:160
28405040,0irq/26-eth0-rx-22488-21fschecks_count23:25:143
28405040,0irq/26-eth0-rx-11507-21taskset19:42:053
28405040,0irq/26-eth0-rx-0-21swapper/323:06:343
28405040,0irq/26-eth0-rx-0-21swapper/322:50:333
28405040,0irq/26-eth0-rx-0-21swapper/322:35:133
28405040,0irq/26-eth0-rx-0-21swapper/322:09:143
28405040,0irq/26-eth0-rx-0-21swapper/321:05:113
28405040,0irq/26-eth0-rx-0-21swapper/320:50:473
28405040,0irq/26-eth0-rx-0-21swapper/319:00:173
28405040,0irq/26-eth0-rx-0-21swapper/300:10:183
28405040,0irq/26-eth0-rx-0-21swapper/300:05:213
28405040,0irq/26-eth0-rx-0-21swapper/300:02:313
259940,0migration/2990-21fschecks_count20:25:122
259940,0migration/222174-21ntp_states19:00:202
259940,0migration/210150-21munin-node18:55:162
249940,0watchdog/20-21swapper/219:45:212
249940,0watchdog/20-21swapper/218:45:172
179940,0migration/18666-21interrupts22:35:161
179940,0migration/18221-21kthreadcore23:15:161
179940,0migration/17457-21kthreadcore19:15:211
179940,0migration/13343-21kthreadcore22:10:191
179940,0migration/131096-21kthreadcore20:45:181
179940,0migration/130568-21fschecks_count19:10:191
179940,0migration/129528-21kthreadcore22:05:191
179940,0migration/126510-21idleruntime21:45:181
179940,0migration/123166-21fschecks_count23:05:131
179940,0migration/120636-21kthreadcore19:25:161
179940,0migration/120171-21if_eth021:00:151
179940,0migration/116895-21kthreadcore23:00:171
179940,0migration/115741-21kthreadcore21:15:171
169940,0watchdog/16547-21runrttasks00:02:531
169940,0watchdog/130024-21ssh23:52:171
169940,0watchdog/129730-21cut18:45:131
169940,0watchdog/128821-21grep19:30:171
169940,0watchdog/128406-21kthreadcore21:25:171
169940,0watchdog/118812-21if_eth1.221:40:171
169940,0watchdog/115035-21ssh22:17:011
169940,0watchdog/114406-21ssh23:20:131
169940,0watchdog/114235-21kthreadcore18:55:211
169940,0watchdog/10-21swapper/122:45:171
169940,0watchdog/10-21swapper/122:00:411
169940,0watchdog/10-21swapper/120:35:171
169940,0watchdog/10-21swapper/120:05:211
169940,0watchdog/10-21swapper/119:00:171
149940,0migration/03256-21kthreadcore20:00:180
149940,0migration/029425-21kthreadcore19:30:180
149940,0migration/028250-21kthreadcore21:25:170
149940,0migration/027447-21aten_r1power_fr21:05:120
149940,0migration/023611-21unixbench_multi19:25:220
149940,0migration/021933-21runrttasks23:46:490
149940,0migration/018727-21sh20:59:220
149940,0migration/018052-21kthreadcore00:05:180
149940,0migration/012489-21diskmemload23:37:470
149940,0migration/012442-21kthreadcore18:55:210
149940,0migration/011258-21cpuspeed_turbos19:20:110
35130,0ktimersoftd/30-21swapper/319:37:573
288289933,0cyclictest16546-21kworker/1:023:35:401
288289933,0cyclictest16546-21kworker/1:022:50:141
288289933,0cyclictest16546-21kworker/1:020:40:061
288289933,0cyclictest16546-21kworker/1:020:00:071
288289933,0cyclictest16546-21kworker/1:019:45:051
288289933,0cyclictest16546-21kworker/1:019:40:011
288289933,0cyclictest16546-21kworker/1:019:05:071
288289930,0cyclictest13146-21sh20:10:111
288279933,0cyclictest6445-21snmpd20:39:390
288279932,0cyclictest7770-21awk22:55:170
288279932,0cyclictest5063-21grep20:50:170
288279932,0cyclictest30880-21awk19:10:200
288279932,0cyclictest1900-21grep21:50:200
288279932,0cyclictest17805-21kthreadcore20:10:170
288279932,0cyclictest13681-21sh23:40:490
288279932,0cyclictest12347-21ksmtuned23:15:450
288279931,0cyclictest1664-21kthreadcore22:50:170
288279930,0cyclictest6199-21port_sshpriv20:25:190
288279930,0cyclictest0-21swapper/019:43:160
28405030,0irq/26-eth0-rx-0-21swapper/320:35:223
28405030,0irq/26-eth0-rx-0-21swapper/320:09:533
28405030,0irq/26-eth0-rx-0-21swapper/319:50:013
28405030,0irq/26-eth0-rx-0-21swapper/319:30:113
28405030,0irq/26-eth0-rx-0-21swapper/318:45:113
169930,0watchdog/10-21swapper/118:50:091
339920,0migration/311047-21iostat_ios18:55:203
288279921,0cyclictest7508-21latency_hist22:34:580
28405020,0irq/26-eth0-rx-0-21swapper/320:10:223
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional