You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-09-12 - 07:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2s.osadl.org (updated Thu Sep 12, 2024 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350450,0irq/25-eth10-21swapper/318:54:023
2570424533,0sleep10-21swapper/118:50:561
2561223824,0sleep00-21swapper/018:50:550
2635623524,0sleep20-21swapper/218:52:572
266489970,0cyclictest6436-21cut19:50:502
339960,0migration/38783-21sh23:25:543
339960,0migration/34143-21sh22:01:503
339960,0migration/321524-21cpuspeed_turbos19:15:463
339960,0migration/316838-21sh23:52:273
28415060,0irq/27-eth0-tx-7977-21tail19:05:460
28415060,0irq/27-eth0-tx-7934-21ssh00:08:362
28415060,0irq/27-eth0-tx-24872-21ls20:55:340
28415060,0irq/27-eth0-tx-0-21swapper/223:28:082
28415060,0irq/27-eth0-tx-0-21swapper/223:14:332
28415060,0irq/27-eth0-tx-0-21swapper/223:01:122
28415060,0irq/27-eth0-tx-0-21swapper/222:42:402
28415060,0irq/27-eth0-tx-0-21swapper/222:39:362
28415060,0irq/27-eth0-tx-0-21swapper/222:27:422
28415060,0irq/27-eth0-tx-0-21swapper/222:19:342
28415060,0irq/27-eth0-tx-0-21swapper/221:33:362
28415060,0irq/27-eth0-tx-0-21swapper/221:04:282
28405060,0irq/26-eth0-rx-1735-21sh23:22:403
28405060,0irq/26-eth0-rx-14908-21ssh21:30:333
28405060,0irq/26-eth0-rx-0-21swapper/322:45:483
27160,0ktimersoftd/224874-21diskmemload23:55:122
266469966,0cyclictest24874-21diskmemload23:14:010
249960,0watchdog/211077-21ssh22:47:362
249960,0watchdog/20-21swapper/220:56:372
249960,0watchdog/20-21swapper/200:13:362
249960,0watchdog/20-21swapper/200:13:362
149960,0migration/029999-21latency_hist22:00:340
149960,0migration/019497-21latency_hist23:15:340
8225050,0irq/24-0000:00:28614-21sadc19:45:342
339950,0migration/39960-21dump-pmu-power23:50:293
339950,0migration/39799-21munin-node20:15:553
339950,0migration/39479-21munin-node23:05:553
339950,0migration/37673-21kthreadcore22:05:493
339950,0migration/33817-21kthreadcore19:25:513
339950,0migration/33794-21kthreadcore19:00:503
339950,0migration/331292-21taskset20:56:033
339950,0migration/328947-21timerandwakeup20:05:563
339950,0migration/328597-21taskset21:56:503
339950,0migration/327958-21sh22:39:003
339950,0migration/325839-21pmu-power22:55:563
339950,0migration/323788-21taskset20:29:463
339950,0migration/322268-21timerandwakeup20:00:563
339950,0migration/318727-21munin-node23:55:483
339950,0migration/31865-21munin-node19:45:533
339950,0migration/317382-21munin-node20:45:563
339950,0migration/316794-21kthreadcore22:10:523
339950,0migration/313755-21idleruntime22:30:483
339950,0migration/312938-21fschecks_time22:50:473
339950,0migration/312527-21taskset21:46:133
28415050,0irq/27-eth0-tx-8855-21munin-node19:30:530
28415050,0irq/27-eth0-tx-5705-21ssh21:21:162
28415050,0irq/27-eth0-tx-31306-21fschecks_count21:40:482
28415050,0irq/27-eth0-tx-30492-21cut23:40:502
28415050,0irq/27-eth0-tx-25636-21munin-node00:00:502
28415050,0irq/27-eth0-tx-24095-21munin-node20:30:550
28415050,0irq/27-eth0-tx-23536-21proc_pri23:55:562
28415050,0irq/27-eth0-tx-21304-21cut00:15:522
28415050,0irq/27-eth0-tx-0-21swapper/223:37:512
28415050,0irq/27-eth0-tx-0-21swapper/223:15:502
28415050,0irq/27-eth0-tx-0-21swapper/223:06:382
28415050,0irq/27-eth0-tx-0-21swapper/222:55:542
28415050,0irq/27-eth0-tx-0-21swapper/222:51:442
28415050,0irq/27-eth0-tx-0-21swapper/222:32:542
28415050,0irq/27-eth0-tx-0-21swapper/222:22:322
28415050,0irq/27-eth0-tx-0-21swapper/222:05:412
28415050,0irq/27-eth0-tx-0-21swapper/221:51:512
28415050,0irq/27-eth0-tx-0-21swapper/221:37:022
28415050,0irq/27-eth0-tx-0-21swapper/221:17:282
28415050,0irq/27-eth0-tx-0-21swapper/221:13:492
28415050,0irq/27-eth0-tx-0-21swapper/020:55:460
28415050,0irq/27-eth0-tx-0-21swapper/020:25:510
28415050,0irq/27-eth0-tx-0-21swapper/020:20:550
28415050,0irq/27-eth0-tx-0-21swapper/019:29:100
28405050,0irq/26-eth0-rx-8074-21ssh00:09:043
28405050,0irq/26-eth0-rx-32265-21kthreadcore19:20:513
28405050,0irq/26-eth0-rx-31104-21ssh21:19:093
28405050,0irq/26-eth0-rx-23519-21kthreadcore23:35:503
28405050,0irq/26-eth0-rx-0-21swapper/323:30:523
28405050,0irq/26-eth0-rx-0-21swapper/323:10:513
28405050,0irq/26-eth0-rx-0-21swapper/322:40:453
28405050,0irq/26-eth0-rx-0-21swapper/322:21:133
28405050,0irq/26-eth0-rx-0-21swapper/321:42:303
28405050,0irq/26-eth0-rx-0-21swapper/321:30:583
28405050,0irq/26-eth0-rx-0-21swapper/321:07:503
28405050,0irq/26-eth0-rx-0-21swapper/321:00:503
27150,0ktimersoftd/20-21swapper/221:46:522
266479955,0cyclictest0-21swapper/123:26:341
266479955,0cyclictest0-21swapper/123:21:321
266479955,0cyclictest0-21swapper/123:16:291
266479955,0cyclictest0-21swapper/123:01:211
266479955,0cyclictest0-21swapper/122:17:501
266479955,0cyclictest0-21swapper/121:45:591
266479955,0cyclictest0-21swapper/121:26:361
266479955,0cyclictest0-21swapper/121:10:121
266479955,0cyclictest0-21swapper/100:13:051
266479955,0cyclictest0-21swapper/100:13:051
266479954,0cyclictest20966-21kworker/1:222:28:201
266479950,0cyclictest0-21swapper/123:59:221
266479950,0cyclictest0-21swapper/122:56:181
266479950,0cyclictest0-21swapper/121:59:331
266479950,0cyclictest0-21swapper/120:25:351
266469955,0cyclictest0-21swapper/023:24:060
266469955,0cyclictest0-21swapper/022:10:380
266469955,0cyclictest0-21swapper/021:52:210
266469955,0cyclictest0-21swapper/021:11:500
266469955,0cyclictest0-21swapper/000:24:200
266469954,0cyclictest0-21swapper/020:00:530
266469950,0cyclictest0-21swapper/022:58:530
266469950,0cyclictest0-21swapper/021:20:020
259950,0migration/214475-21latency_hist19:10:342
259950,0migration/213447-21kthreadcore21:05:552
249950,0watchdog/20-21swapper/223:35:002
249950,0watchdog/20-21swapper/223:20:402
249950,0watchdog/20-21swapper/222:11:402
249950,0watchdog/20-21swapper/222:00:522
249950,0watchdog/20-21swapper/221:57:322
249950,0watchdog/20-21swapper/220:51:052
249950,0watchdog/20-21swapper/220:45:532
249950,0watchdog/20-21swapper/220:43:532
249950,0watchdog/20-21swapper/220:29:172
249950,0watchdog/20-21swapper/220:21:532
249950,0watchdog/20-21swapper/220:10:412
249950,0watchdog/20-21swapper/220:01:572
249950,0watchdog/20-21swapper/219:35:532
249950,0watchdog/20-21swapper/219:08:092
249950,0watchdog/20-21swapper/219:02:412
249950,0watchdog/20-21swapper/218:57:532
179950,0migration/19930-21munin-node22:45:521
179950,0migration/18302-21kthreadcore23:05:511
179950,0migration/130251-21taskset20:32:481
179950,0migration/115581-21kthreadcore20:20:511
149950,0migration/08250-21kthreadcore20:40:490
149950,0migration/04641-21fschecks_time20:15:470
149950,0migration/030449-21fschecks_time20:10:470
149950,0migration/030032-21kthreadcore23:00:500
149950,0migration/02812-21kthreadcore00:05:490
149950,0migration/025998-21kthreadcore22:15:520
149950,0migration/023082-21latency_hist20:05:340
149950,0migration/019681-21munin-node23:35:460
149950,0migration/018976-21irqcore23:55:500
149950,0migration/013186-21kthreadcore19:55:500
149950,0migration/010739-21kthreadcore00:10:510
149950,0migration/010739-21kthreadcore00:10:510
149950,0migration/01024-21latency_hist19:00:340
339940,0migration/36330-21kthreadcore19:50:503
339940,0migration/32831-21kthreadcore20:35:513
339940,0migration/316597-21timerandwakeup20:20:563
339940,0migration/310772-21kthreadcore00:10:513
339940,0migration/310772-21kthreadcore00:10:513
28415040,0irq/27-eth0-tx-0-21swapper/020:45:540
28405040,0irq/26-eth0-rx-0-21swapper/323:40:453
28405040,0irq/26-eth0-rx-0-21swapper/323:01:193
28405040,0irq/26-eth0-rx-0-21swapper/322:27:013
28405040,0irq/26-eth0-rx-0-21swapper/322:15:543
28405040,0irq/26-eth0-rx-0-21swapper/321:50:363
28405040,0irq/26-eth0-rx-0-21swapper/321:35:493
28405040,0irq/26-eth0-rx-0-21swapper/321:21:503
28405040,0irq/26-eth0-rx-0-21swapper/321:10:523
28405040,0irq/26-eth0-rx-0-21swapper/319:59:093
28405040,0irq/26-eth0-rx-0-21swapper/319:05:503
28405040,0irq/26-eth0-rx-0-21swapper/300:21:123
28405040,0irq/26-eth0-rx-0-21swapper/300:15:543
28405040,0irq/26-eth0-rx-0-21swapper/300:04:073
266489943,0cyclictest3425-21lldpd00:21:002
266489943,0cyclictest29811-21awk19:45:502
266479944,0cyclictest20966-21kworker/1:223:40:381
266479944,0cyclictest20966-21kworker/1:223:36:261
266479944,0cyclictest20966-21kworker/1:222:51:241
266479944,0cyclictest20966-21kworker/1:222:40:401
266479944,0cyclictest20966-21kworker/1:222:30:541
266479944,0cyclictest20966-21kworker/1:222:25:001
266479944,0cyclictest20966-21kworker/1:222:05:421
266479944,0cyclictest20966-21kworker/1:221:42:401
266479944,0cyclictest20966-21kworker/1:221:00:521
266479944,0cyclictest20966-21kworker/1:220:50:521
266479944,0cyclictest20966-21kworker/1:220:45:441
266479944,0cyclictest20966-21kworker/1:220:35:401
266479944,0cyclictest20966-21kworker/1:220:15:421
266479944,0cyclictest20966-21kworker/1:220:10:461
266479944,0cyclictest20966-21kworker/1:220:05:421
266479944,0cyclictest20966-21kworker/1:220:00:441
266479944,0cyclictest20966-21kworker/1:219:55:481
266479944,0cyclictest20966-21kworker/1:219:41:001
266479944,0cyclictest20966-21kworker/1:219:36:201
266479944,0cyclictest20966-21kworker/1:219:25:361
266479944,0cyclictest20966-21kworker/1:219:20:381
266479944,0cyclictest20966-21kworker/1:200:21:231
266479944,0cyclictest20966-21kworker/1:200:18:291
266479944,0cyclictest20966-21kworker/1:200:06:271
266479944,0cyclictest0-21swapper/119:50:441
266479943,0cyclictest9693-21awk19:30:471
266479943,0cyclictest5996-21awk20:40:471
266479943,0cyclictest3251-21awk23:45:481
266479943,0cyclictest20966-21kworker/1:223:50:471
266479943,0cyclictest20966-21kworker/1:222:00:401
266479943,0cyclictest1967-21awk19:00:481
266479943,0cyclictest15060-21awk21:50:481
266469944,0cyclictest0-21swapper/023:50:540
266469944,0cyclictest0-21swapper/021:34:410
266469944,0cyclictest0-21swapper/021:22:050
266469943,0cyclictest8238-21awk21:25:490
266469943,0cyclictest6445-21snmpd23:34:060
266469943,0cyclictest5200-21awk22:45:460
266469943,0cyclictest30322-21awk22:20:470
266469940,0cyclictest0-21swapper/019:21:310
259940,0migration/22631-21kthreadcore20:35:512
259940,0migration/211858-21kthreadcore19:55:502
249940,0watchdog/27881-21python19:25:572
249940,0watchdog/20-21swapper/223:46:002
249940,0watchdog/20-21swapper/221:25:372
249940,0watchdog/20-21swapper/220:31:252
249940,0watchdog/20-21swapper/220:17:332
249940,0watchdog/20-21swapper/220:06:132
249940,0watchdog/20-21swapper/219:32:532
249940,0watchdog/20-21swapper/219:20:492
179940,0migration/129509-21kthreadcore20:55:511
179940,0migration/125045-21irqcore21:15:501
179940,0migration/121785-21fschecks_time19:15:471
179940,0migration/121261-21kthreadcore21:10:531
179940,0migration/12001-21proc_pri19:45:551
179940,0migration/113276-21needreboot19:05:541
179940,0migration/110835-21latency_hist23:30:341
169940,0watchdog/128261-21kthreadcore18:55:491
169940,0watchdog/123047-21ssh21:35:371
169940,0watchdog/120874-21aten_r1power_en22:35:441
169940,0watchdog/117857-21ssh23:11:161
169940,0watchdog/115146-21ssh21:30:371
169940,0watchdog/10-21swapper/122:10:521
169940,0watchdog/10-21swapper/121:20:531
169940,0watchdog/10-21swapper/119:10:531
169940,0watchdog/10-21swapper/100:00:521
149940,0migration/09646-21kthreadcore22:05:500
149940,0migration/09525-21kthreadcore21:45:520
149940,0migration/07903-21kthreadcore23:25:510
149940,0migration/032502-21kthreadcore19:45:520
149940,0migration/026916-21kthreadcore21:35:510
149940,0migration/023497-21kthreadcore22:35:510
149940,0migration/022347-21sh19:40:450
149940,0migration/018290-21latency22:30:520
149940,0migration/012465-21sh22:50:460
149940,0migration/012233-21sh22:29:130
339930,0migration/315178-21taskset19:34:103
28415030,0irq/27-eth0-tx-3292-21aten2.4-expect19:50:440
28415030,0irq/27-eth0-tx-21230-21aten2.4-expect19:15:440
28415030,0irq/27-eth0-tx-0-21swapper/018:55:430
28405030,0irq/26-eth0-rx-0-21swapper/323:15:353
28405030,0irq/26-eth0-rx-0-21swapper/320:50:453
28405030,0irq/26-eth0-rx-0-21swapper/320:42:303
28405030,0irq/26-eth0-rx-0-21swapper/320:32:293
28405030,0irq/26-eth0-rx-0-21swapper/320:10:463
28405030,0irq/26-eth0-rx-0-21swapper/319:40:553
28405030,0irq/26-eth0-rx-0-21swapper/319:35:513
28405030,0irq/26-eth0-rx-0-21swapper/319:10:463
28405030,0irq/26-eth0-rx-0-21swapper/318:56:593
266489930,0cyclictest0-21swapper/219:15:482
266469933,0cyclictest0-21swapper/023:42:330
266469932,0cyclictest6260-21sensors21:00:550
266469932,0cyclictest6042-21sadc21:45:340
266469932,0cyclictest4644-21awk23:05:490
266469932,0cyclictest29612-21awk22:40:470
266469932,0cyclictest22302-21sensors00:15:560
266469932,0cyclictest15445-21if_eth019:10:500
266469930,0cyclictest20840-21memory19:35:510
149930,0migration/09569-21sh23:49:130
266469922,0cyclictest0-21swapper/021:05:410
266469920,0cyclictest25543-21latency_hist00:00:340
266469920,0cyclictest0-21swapper/021:55:380
266469920,0cyclictest0-21swapper/020:35:340
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional