You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-05 - 01:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack1slot2s.osadl.org (updated Sun May 04, 2025 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350450,0irq/25-eth10-21swapper/306:42:343
1942923423,0sleep00-21swapper/006:44:450
1941223323,0sleep10-21swapper/106:44:341
1947722925,0sleep20-21swapper/206:45:232
249970,0watchdog/222061-21irqrtprio07:12:252
339960,0migration/323832-21kthreadcore06:47:263
339960,0migration/321655-21sh10:56:113
339960,0migration/320871-21sh11:13:193
339960,0migration/311381-21dump-pmu-power08:42:063
28415060,0irq/27-eth0-tx-8155-21ssh09:19:452
28415060,0irq/27-eth0-tx-5423-21ssh12:03:102
28415060,0irq/27-eth0-tx-5321-21ssh10:44:012
28415060,0irq/27-eth0-tx-28537-21munin-node07:42:250
28415060,0irq/27-eth0-tx-0-21swapper/212:09:342
28415060,0irq/27-eth0-tx-0-21swapper/211:54:042
28415060,0irq/27-eth0-tx-0-21swapper/211:42:092
28415060,0irq/27-eth0-tx-0-21swapper/211:41:582
28415060,0irq/27-eth0-tx-0-21swapper/211:32:262
28415060,0irq/27-eth0-tx-0-21swapper/211:29:102
28415060,0irq/27-eth0-tx-0-21swapper/211:19:222
28415060,0irq/27-eth0-tx-0-21swapper/211:04:412
28415060,0irq/27-eth0-tx-0-21swapper/210:54:512
28415060,0irq/27-eth0-tx-0-21swapper/210:32:462
28415060,0irq/27-eth0-tx-0-21swapper/209:58:202
28415060,0irq/27-eth0-tx-0-21swapper/209:47:272
28415060,0irq/27-eth0-tx-0-21swapper/209:38:042
28415060,0irq/27-eth0-tx-0-21swapper/209:13:372
28415060,0irq/27-eth0-tx-0-21swapper/209:07:252
28415060,0irq/27-eth0-tx-0-21swapper/209:04:372
28415060,0irq/27-eth0-tx-0-21swapper/208:52:112
28415060,0irq/27-eth0-tx-0-21swapper/208:50:442
28405060,0irq/26-eth0-rx-5357-21ssh11:43:443
28405060,0irq/26-eth0-rx-21134-21sh09:07:263
249960,0watchdog/28781-21ssh09:00:172
249960,0watchdog/20-21swapper/209:25:492
196989960,0cyclictest0-21swapper/107:12:311
196979960,0cyclictest0-21swapper/011:58:000
149960,0migration/08572-21sh09:21:150
8225050,0irq/24-0000:00:0-21swapper/211:07:102
4150,0ktimersoftd/00-21swapper/012:16:360
4150,0ktimersoftd/00-21swapper/009:13:260
4150,0ktimersoftd/00-21swapper/009:02:220
339950,0migration/38450-21sh09:20:523
339950,0migration/37611-21munin-node11:27:223
339950,0migration/36898-21kthreadcore08:57:263
339950,0migration/36519-21kthreadcore07:47:253
339950,0migration/34302-21taskset11:02:433
339950,0migration/331924-21fschecks_time09:57:203
339950,0migration/331847-21munin-node08:52:303
339950,0migration/331589-21sh10:42:223
339950,0migration/331415-21idleruntime08:32:213
339950,0migration/329970-21ksmtuned09:53:243
339950,0migration/328227-21gltestperf07:17:263
339950,0migration/327165-21kthreadcore10:12:253
339950,0migration/325926-21kthreadcore08:02:243
339950,0migration/325439-21kthreadcore10:37:253
339950,0migration/324914-21kthreadcore07:37:263
339950,0migration/323240-21fschecks_time11:17:203
339950,0migration/323187-21kthreadcore08:47:283
339950,0migration/321698-21taskset10:33:213
339950,0migration/316138-21kthreadcore11:32:233
339950,0migration/313173-21diskmemload10:03:163
339950,0migration/311150-21fschecks_time08:17:203
339950,0migration/311150-21fschecks_time08:17:203
28415050,0irq/27-eth0-tx-9002-21cat10:27:242
28415050,0irq/27-eth0-tx-2770-21munin-node07:47:210
28415050,0irq/27-eth0-tx-23549-21munin-node09:52:242
28415050,0irq/27-eth0-tx-0-21swapper/211:48:232
28415050,0irq/27-eth0-tx-0-21swapper/211:22:122
28415050,0irq/27-eth0-tx-0-21swapper/211:12:342
28415050,0irq/27-eth0-tx-0-21swapper/210:57:132
28415050,0irq/27-eth0-tx-0-21swapper/210:37:112
28415050,0irq/27-eth0-tx-0-21swapper/210:24:362
28415050,0irq/27-eth0-tx-0-21swapper/210:12:532
28415050,0irq/27-eth0-tx-0-21swapper/210:02:512
28415050,0irq/27-eth0-tx-0-21swapper/008:05:460
28405050,0irq/26-eth0-rx-6445-21snmpd09:47:203
28405050,0irq/26-eth0-rx-27729-21kthreadcore09:12:243
28405050,0irq/26-eth0-rx-26625-21kthreadcore09:32:283
28405050,0irq/26-eth0-rx-21405-21grep12:12:303
28405050,0irq/26-eth0-rx-18360-21kthreadcore10:07:243
28405050,0irq/26-eth0-rx-15262-21ntp_kernel_err09:02:263
28405050,0irq/26-eth0-rx-14252-21ssh11:50:353
28405050,0irq/26-eth0-rx-11791-21kthreadcore09:42:243
28405050,0irq/26-eth0-rx-0-21swapper/312:07:063
28405050,0irq/26-eth0-rx-0-21swapper/312:04:463
28405050,0irq/26-eth0-rx-0-21swapper/311:58:063
28405050,0irq/26-eth0-rx-0-21swapper/311:22:233
28405050,0irq/26-eth0-rx-0-21swapper/311:07:273
28405050,0irq/26-eth0-rx-0-21swapper/311:00:593
28405050,0irq/26-eth0-rx-0-21swapper/309:40:373
28405050,0irq/26-eth0-rx-0-21swapper/309:22:253
28405050,0irq/26-eth0-rx-0-21swapper/308:22:243
28405050,0irq/26-eth0-rx-0-21swapper/308:12:233
259950,0migration/28822-21kthreadcore08:12:232
259950,0migration/27278-21fschecks_time10:47:202
259950,0migration/230823-21unixbench_singl09:32:352
259950,0migration/230680-21kthreadcore07:42:242
259950,0migration/224089-21munin-node08:27:192
259950,0migration/217971-21taskset08:46:372
249950,0watchdog/27603-21kthreadcore08:37:252
249950,0watchdog/2631-21proc_pri07:17:372
249950,0watchdog/217843-21if_eth1.209:27:252
249950,0watchdog/20-21swapper/212:12:202
249950,0watchdog/20-21swapper/211:57:442
249950,0watchdog/20-21swapper/210:17:442
249950,0watchdog/20-21swapper/210:08:562
249950,0watchdog/20-21swapper/209:42:122
249950,0watchdog/20-21swapper/208:17:492
249950,0watchdog/20-21swapper/208:17:492
249950,0watchdog/20-21swapper/207:59:252
249950,0watchdog/20-21swapper/207:07:292
196989955,0cyclictest19703-21kworker/1:010:10:001
196989955,0cyclictest0-21swapper/108:59:431
196989954,0cyclictest29032-21kworker/1:110:53:201
196989954,0cyclictest29032-21kworker/1:110:43:401
196989954,0cyclictest22032-21kworker/1:012:02:451
196989954,0cyclictest22032-21kworker/1:011:01:501
196989954,0cyclictest19703-21kworker/1:010:37:201
196989954,0cyclictest19703-21kworker/1:010:00:101
196989954,0cyclictest19703-21kworker/1:008:42:421
196989954,0cyclictest19703-21kworker/1:008:32:301
196989950,0cyclictest0-21swapper/109:53:001
196979955,0cyclictest0-21swapper/011:46:100
19150,0ktimersoftd/10-21swapper/112:10:381
19150,0ktimersoftd/10-21swapper/109:13:261
179950,0migration/17249-21cpuspeed_turbos11:47:191
179950,0migration/16804-21munin-node06:57:311
179950,0migration/129297-21missed_timers08:27:261
179950,0migration/120180-21cpuspeed_turbos06:47:211
169950,0watchdog/126495-21fschecks_count08:52:211
149950,0migration/08048-21kthreadcore08:12:230
149950,0migration/07894-21fschecks_time12:07:210
149950,0migration/0708-21fschecks_time09:37:230
149950,0migration/06194-21latency06:57:260
149950,0migration/04749-21sh11:03:580
149950,0migration/03414-21kthreadcore07:22:270
149950,0migration/032444-21kthreadcore08:07:240
149950,0migration/029836-21kthreadcore09:32:290
149950,0migration/029324-21sh11:38:130
149950,0migration/029161-21kthreadcore08:52:240
149950,0migration/028215-21kthreadcore08:27:250
149950,0migration/026088-21taskset06:49:220
149950,0migration/024722-21kthreadcore07:37:260
149950,0migration/023479-21irqcore10:57:230
149950,0migration/021847-21proc_pri07:57:450
149950,0migration/021526-21fschecks_time07:12:220
149950,0migration/017559-21kthreadcore10:07:230
149950,0migration/016761-21latency_hist09:27:060
149950,0migration/015104-21sh09:45:370
149950,0migration/015002-21cpuspeed_turbos11:32:190
149950,0migration/013836-21sh10:51:230
149950,0migration/012533-21if_eth108:42:220
149950,0migration/011149-21kthreadcore07:52:280
149950,0migration/010067-21kthreadcore11:27:230
339940,0migration/323687-21kthreadcore07:12:263
339940,0migration/321089-21kthreadcore07:57:433
339940,0migration/31393-21load07:42:253
28415040,0irq/27-eth0-tx-0-21swapper/008:17:180
28415040,0irq/27-eth0-tx-0-21swapper/008:17:180
28415040,0irq/27-eth0-tx-0-21swapper/006:52:210
28405040,0irq/26-eth0-rx-30860-21ps08:07:233
28405040,0irq/26-eth0-rx-0-21swapper/311:52:243
28405040,0irq/26-eth0-rx-0-21swapper/311:37:113
28405040,0irq/26-eth0-rx-0-21swapper/310:47:583
28405040,0irq/26-eth0-rx-0-21swapper/310:27:173
28405040,0irq/26-eth0-rx-0-21swapper/310:22:133
28405040,0irq/26-eth0-rx-0-21swapper/310:18:093
28405040,0irq/26-eth0-rx-0-21swapper/309:27:063
28405040,0irq/26-eth0-rx-0-21swapper/306:52:203
259940,0migration/26929-21kthreadcore07:47:252
259940,0migration/23043-21timerwakeupswit08:07:302
259940,0migration/211389-21kthreadcore07:52:282
249940,0watchdog/20-21swapper/207:32:212
249940,0watchdog/20-21swapper/207:24:212
249940,0watchdog/20-21swapper/206:52:252
196999944,0cyclictest5641-21kworker/2:008:32:202
196999944,0cyclictest5641-21kworker/2:008:03:182
196999944,0cyclictest5641-21kworker/2:007:27:202
196999944,0cyclictest5641-21kworker/2:007:06:082
196999943,0cyclictest22785-21awk07:37:242
196999943,0cyclictest18122-21awk08:22:222
196999943,0cyclictest1699-21awk06:57:232
196989944,0cyclictest29032-21kworker/1:110:47:081
196989944,0cyclictest22032-21kworker/1:012:12:131
196989944,0cyclictest22032-21kworker/1:011:57:151
196989944,0cyclictest22032-21kworker/1:011:52:131
196989944,0cyclictest22032-21kworker/1:011:42:071
196989944,0cyclictest22032-21kworker/1:011:22:181
196989944,0cyclictest22032-21kworker/1:011:12:081
196989944,0cyclictest19703-21kworker/1:010:33:001
196989944,0cyclictest19703-21kworker/1:010:27:281
196989944,0cyclictest19703-21kworker/1:010:17:181
196989944,0cyclictest19703-21kworker/1:010:02:141
196989944,0cyclictest19703-21kworker/1:009:47:061
196989944,0cyclictest19703-21kworker/1:009:37:101
196989944,0cyclictest19703-21kworker/1:009:32:081
196989944,0cyclictest19703-21kworker/1:009:22:181
196989944,0cyclictest19703-21kworker/1:009:17:181
196989944,0cyclictest19703-21kworker/1:009:02:221
196989944,0cyclictest19703-21kworker/1:008:47:321
196989944,0cyclictest19703-21kworker/1:008:37:081
196989944,0cyclictest19703-21kworker/1:008:22:121
196989944,0cyclictest19703-21kworker/1:008:07:381
196989944,0cyclictest19703-21kworker/1:008:02:241
196989944,0cyclictest19703-21kworker/1:007:42:101
196989944,0cyclictest19703-21kworker/1:007:37:101
196989944,0cyclictest19703-21kworker/1:007:32:101
196989944,0cyclictest19703-21kworker/1:007:27:161
196989944,0cyclictest19703-21kworker/1:007:22:161
196989944,0cyclictest19703-21kworker/1:007:17:121
196989944,0cyclictest19703-21kworker/1:007:03:041
196989944,0cyclictest19703-21kworker/1:006:52:161
196989943,0cyclictest353-21awk10:22:221
196989943,0cyclictest3110-21irqbalance11:32:201
196989943,0cyclictest3110-21irqbalance10:12:101
196989943,0cyclictest3110-21irqbalance08:17:201
196989943,0cyclictest3110-21irqbalance08:17:201
196989943,0cyclictest3110-21irqbalance08:12:101
196989943,0cyclictest3110-21irqbalance07:52:101
196989943,0cyclictest3110-21irqbalance07:48:001
196989943,0cyclictest23778-21awk11:37:241
196989943,0cyclictest23464-21awk11:17:221
196989943,0cyclictest19703-21kworker/1:009:07:141
196989943,0cyclictest15216-21awk07:07:241
196979944,0cyclictest0-21swapper/010:29:230
196979944,0cyclictest0-21swapper/009:59:450
196979944,0cyclictest0-21swapper/008:49:450
196979944,0cyclictest0-21swapper/008:32:240
196979943,0cyclictest7834-21awk10:02:220
196979943,0cyclictest6445-21snmpd11:08:350
196979943,0cyclictest6445-21snmpd10:38:500
196979943,0cyclictest6445-21snmpd09:22:500
196979943,0cyclictest6445-21snmpd07:31:350
196979943,0cyclictest5862-21awk08:37:230
196979943,0cyclictest3425-21lldpd11:14:050
196979943,0cyclictest23504-21awk10:12:230
196979943,0cyclictest22008-21sadc10:57:060
196979943,0cyclictest14231-21munin-node07:07:220
196979943,0cyclictest13998-21sadc07:07:060
196979940,0cyclictest0-21swapper/007:32:210
179940,0migration/18731-21kthreadcore11:27:231
179940,0migration/1705-21kthreadcore11:02:231
179940,0migration/16200-21sh11:07:121
169940,0watchdog/10-21swapper/109:42:201
169940,0watchdog/10-21swapper/109:27:251
169940,0watchdog/10-21swapper/107:57:411
149940,0migration/04915-21timerandwakeup12:02:290
149940,0migration/029706-21kthreadcore07:17:290
149940,0migration/018687-21sh09:07:230
28415030,0irq/27-eth0-tx-0-21swapper/008:22:210
28405030,0irq/26-eth0-rx-0-21swapper/308:42:283
28405030,0irq/26-eth0-rx-0-21swapper/307:52:263
28405030,0irq/26-eth0-rx-0-21swapper/307:33:323
28405030,0irq/26-eth0-rx-0-21swapper/307:25:133
28405030,0irq/26-eth0-rx-0-21swapper/307:08:363
28405030,0irq/26-eth0-rx-0-21swapper/307:02:063
28405030,0irq/26-eth0-rx-0-21swapper/306:58:283
196999933,0cyclictest5641-21kworker/2:006:47:122
196979933,0cyclictest3425-21lldpd09:54:030
196979932,0cyclictest7831-21awk11:47:220
196979932,0cyclictest6445-21snmpd11:54:060
196979932,0cyclictest5330-21sensors10:22:290
196979932,0cyclictest31078-21cut11:22:180
196979930,0cyclictest31745-21perl10:42:220
196979930,0cyclictest21085-21python10:32:280
149930,0migration/08148-21sh08:58:300
149930,0migration/022314-21runrttasks09:48:340
329920,0watchdog/30-21swapper/308:27:403
28405020,0irq/26-eth0-rx-0-21swapper/307:27:203
196979921,0cyclictest22664-21aten_r1power_po11:17:180
196979920,0cyclictest30077-21latency_hist10:17:060
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional