You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-23 - 17:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack1slot2s.osadl.org (updated Wed Apr 23, 2025 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2299126867,0sleep10-21swapper/106:42:331
2933524938,0sleep30-21swapper/306:47:033
2901223627,0sleep20-21swapper/206:43:182
2900023622,0sleep00-21swapper/006:43:080
28415070,0irq/27-eth0-tx-0-21swapper/211:37:502
8225060,0irq/24-0000:00:24213-21latency_hist11:52:302
4160,0ktimersoftd/00-21swapper/012:07:470
35160,0ktimersoftd/30-21swapper/310:33:283
339960,0migration/38930-21sh09:54:593
339960,0migration/37590-21aten_r1power_vo08:12:463
339960,0migration/331308-21sh09:20:413
339960,0migration/312053-21sh10:49:443
294469961,0cyclictest0-21swapper/208:42:302
294469960,0cyclictest0-21swapper/212:05:492
294429966,0cyclictest0-21swapper/111:29:031
28415060,0irq/27-eth0-tx-29889-21ssh08:53:252
28415060,0irq/27-eth0-tx-0-21swapper/212:14:472
28415060,0irq/27-eth0-tx-0-21swapper/212:10:182
28415060,0irq/27-eth0-tx-0-21swapper/212:01:222
28415060,0irq/27-eth0-tx-0-21swapper/211:46:132
28415060,0irq/27-eth0-tx-0-21swapper/211:31:242
28415060,0irq/27-eth0-tx-0-21swapper/211:17:222
28415060,0irq/27-eth0-tx-0-21swapper/211:08:592
28415060,0irq/27-eth0-tx-0-21swapper/210:54:402
28415060,0irq/27-eth0-tx-0-21swapper/210:25:122
28415060,0irq/27-eth0-tx-0-21swapper/210:18:442
28415060,0irq/27-eth0-tx-0-21swapper/210:14:072
28415060,0irq/27-eth0-tx-0-21swapper/210:10:532
28415060,0irq/27-eth0-tx-0-21swapper/209:49:342
28415060,0irq/27-eth0-tx-0-21swapper/209:09:492
28415060,0irq/27-eth0-tx-0-21swapper/209:05:322
28415060,0irq/27-eth0-tx-0-21swapper/208:58:372
28415060,0irq/27-eth0-tx-0-21swapper/208:52:132
28415060,0irq/27-eth0-tx-0-21swapper/008:47:520
249960,0watchdog/20-21swapper/211:27:202
249960,0watchdog/20-21swapper/209:54:252
19160,0ktimersoftd/10-21swapper/109:48:001
149960,0migration/029317-21taskset11:05:040
149960,0migration/026554-21aten_r1power_en08:02:400
4150,0ktimersoftd/00-21swapper/012:06:130
339950,0migration/39344-21munin-node09:07:403
339950,0migration/36566-21irqcore07:22:453
339950,0migration/35997-21kthreadcore10:17:503
339950,0migration/35697-21latency_hist11:12:303
339950,0migration/332746-21taskset09:24:523
339950,0migration/329894-21taskset11:06:593
339950,0migration/328248-21kthreadcore11:52:473
339950,0migration/327842-21cpuspeed_turbos10:12:453
339950,0migration/326992-21fschecks_count08:02:423
339950,0migration/325948-21taskset09:46:263
339950,0migration/325518-21kthreadcore08:52:473
339950,0migration/324706-21taskset11:27:433
339950,0migration/323896-21runrttasks09:17:013
339950,0migration/322749-21sensors_fan09:37:523
339950,0migration/318696-21munin-node07:07:403
339950,0migration/315288-21kthreadcore07:27:513
339950,0migration/315185-21kthreadcore10:52:473
339950,0migration/315181-21irqcore08:17:483
339950,0migration/314766-21runrttasks11:42:303
339950,0migration/313985-21kthreadcore12:07:433
339950,0migration/313769-21kthreadcore09:57:483
339950,0migration/313466-21idleruntime10:27:523
339950,0migration/311630-21irqcore06:57:473
339950,0migration/311300-21taskset10:23:543
294469955,0cyclictest0-21swapper/210:48:432
294469953,0cyclictest26500-21kworker/2:209:32:502
294429955,0cyclictest0-21swapper/111:06:311
294429955,0cyclictest0-21swapper/109:30:111
294429954,0cyclictest8233-21kworker/1:212:05:441
294429954,0cyclictest8233-21kworker/1:210:20:381
294429953,0cyclictest8233-21kworker/1:210:38:441
294429950,0cyclictest0-21swapper/109:23:051
294419955,0cyclictest0-21swapper/009:05:420
294419955,0cyclictest0-21swapper/008:52:460
294419954,0cyclictest6445-21snmpd12:13:440
294419954,0cyclictest0-21swapper/009:35:580
294419950,0cyclictest0-21swapper/011:35:380
294419950,0cyclictest0-21swapper/009:26:020
28415050,0irq/27-eth0-tx-0-21swapper/211:02:502
28415050,0irq/27-eth0-tx-0-21swapper/210:57:392
28415050,0irq/27-eth0-tx-0-21swapper/210:28:052
28415050,0irq/27-eth0-tx-0-21swapper/210:04:032
28415050,0irq/27-eth0-tx-0-21swapper/209:42:562
28415050,0irq/27-eth0-tx-0-21swapper/209:41:112
28415050,0irq/27-eth0-tx-0-21swapper/209:28:162
28415050,0irq/27-eth0-tx-0-21swapper/209:23:282
28405050,0irq/26-eth0-rx-7539-21ssh09:05:033
28405050,0irq/26-eth0-rx-6547-21runrttasks09:49:213
28405050,0irq/26-eth0-rx-4765-21python11:32:533
28405050,0irq/26-eth0-rx-2746-21cat11:57:453
28405050,0irq/26-eth0-rx-0-21swapper/312:03:543
28405050,0irq/26-eth0-rx-0-21swapper/311:24:543
28405050,0irq/26-eth0-rx-0-21swapper/310:45:503
28405050,0irq/26-eth0-rx-0-21swapper/310:38:293
28405050,0irq/26-eth0-rx-0-21swapper/309:32:533
259950,0migration/23803-21sh10:43:272
259950,0migration/227483-21aten_r1power_cu08:27:432
259950,0migration/227023-21kthreadcore11:52:462
259950,0migration/216765-21cpuspeed_turbos09:12:412
259950,0migration/213986-21latency_hist08:17:302
259950,0migration/210586-21latency_hist06:57:312
249950,0watchdog/20-21swapper/211:34:162
249950,0watchdog/20-21swapper/211:20:402
249950,0watchdog/20-21swapper/210:39:402
249950,0watchdog/20-21swapper/210:33:442
249950,0watchdog/20-21swapper/209:18:012
249950,0watchdog/20-21swapper/208:07:132
249950,0watchdog/20-21swapper/207:51:492
249950,0watchdog/20-21swapper/207:37:332
249950,0watchdog/20-21swapper/207:03:532
19150,0ktimersoftd/121607-21diskmemload11:12:301
179950,0migration/16137-21timerwakeupswit11:57:511
179950,0migration/15758-21kthreadcore06:52:481
179950,0migration/121311-21taskset08:44:101
179950,0migration/119892-21kthreadcore11:47:431
179950,0migration/113869-21latency09:07:441
169950,0watchdog/14302-21grep11:32:481
149950,0migration/09029-21taskset09:55:170
149950,0migration/08478-21latency_hist09:07:300
149950,0migration/07955-21kthreadcore11:12:490
149950,0migration/06059-21taskset07:45:170
149950,0migration/03797-21munin-node10:17:450
149950,0migration/032114-21kthreadcore10:12:510
149950,0migration/030926-21sh09:19:270
149950,0migration/02893-21kthreadcore11:57:450
149950,0migration/028090-21kthreadcore07:12:480
149950,0migration/027023-21sh10:11:510
149950,0migration/025458-21latency_hist07:37:310
149950,0migration/023703-21cpuspeed_turbos11:27:390
149950,0migration/019565-21kthreadcore09:12:460
149950,0migration/018824-21taskset10:05:450
149950,0migration/018732-21idleruntime07:07:410
149950,0migration/015470-21kthreadcore11:17:450
149950,0migration/01192-21taskset08:29:080
149950,0migration/011619-21kthreadcore09:57:480
149950,0migration/011464-21kthreadcore08:37:500
149950,0migration/011125-21munin-node10:47:550
339940,0migration/38476-21kthreadcore11:37:473
339940,0migration/37737-21kthreadcore06:52:493
339940,0migration/34381-21diskmemload11:08:583
339940,0migration/325069-21kthreadcore10:57:483
339940,0migration/31858-21kthreadcore07:42:483
339940,0migration/317662-21taskset07:07:183
339940,0migration/316377-21kthreadcore11:17:453
339940,0migration/313625-21fschecks_count07:52:433
294469944,0cyclictest26500-21kworker/2:208:42:542
294469944,0cyclictest26500-21kworker/2:207:32:462
294469944,0cyclictest26500-21kworker/2:207:27:522
294469944,0cyclictest26500-21kworker/2:207:07:322
294469943,0cyclictest30584-21awk06:47:472
294469943,0cyclictest21631-21awk08:22:432
294429944,0cyclictest8233-21kworker/1:212:13:421
294429944,0cyclictest8233-21kworker/1:211:53:041
294429944,0cyclictest8233-21kworker/1:211:42:381
294429944,0cyclictest8233-21kworker/1:211:22:421
294429944,0cyclictest8233-21kworker/1:210:57:541
294429944,0cyclictest8233-21kworker/1:210:53:081
294429944,0cyclictest8233-21kworker/1:210:47:501
294429944,0cyclictest8233-21kworker/1:210:43:121
294429944,0cyclictest8233-21kworker/1:210:22:541
294429944,0cyclictest8233-21kworker/1:210:12:461
294429944,0cyclictest8233-21kworker/1:210:03:141
294429944,0cyclictest8233-21kworker/1:209:52:341
294429944,0cyclictest8233-21kworker/1:209:43:141
294429944,0cyclictest8233-21kworker/1:209:04:341
294429944,0cyclictest8233-21kworker/1:208:57:461
294429944,0cyclictest8233-21kworker/1:208:52:541
294429944,0cyclictest5375-21kworker/1:007:39:581
294429944,0cyclictest5375-21kworker/1:007:27:501
294429944,0cyclictest5375-21kworker/1:007:08:521
294429944,0cyclictest0-21swapper/107:52:541
294429943,0cyclictest8233-21kworker/1:211:21:081
294429943,0cyclictest8233-21kworker/1:210:07:361
294429943,0cyclictest8233-21kworker/1:209:40:141
294429943,0cyclictest8233-21kworker/1:208:22:441
294429943,0cyclictest7721-21awk11:37:441
294429943,0cyclictest6445-21snmpd08:37:271
294429943,0cyclictest6445-21snmpd08:09:271
294429943,0cyclictest6445-21snmpd08:04:421
294429943,0cyclictest6445-21snmpd07:49:121
294429943,0cyclictest6445-21snmpd07:34:421
294429943,0cyclictest6445-21snmpd07:24:421
294429943,0cyclictest6445-21snmpd07:05:271
294429943,0cyclictest3425-21lldpd06:47:531
294429943,0cyclictest15012-21awk08:17:471
294429943,0cyclictest10065-21awk09:32:431
294419944,0cyclictest0-21swapper/010:31:280
294419944,0cyclictest0-21swapper/008:12:530
294419943,0cyclictest6504-21munin-node07:47:450
294419943,0cyclictest6445-21snmpd11:46:440
294419943,0cyclictest6445-21snmpd10:58:140
294419943,0cyclictest6445-21snmpd10:40:430
294419943,0cyclictest6445-21snmpd09:46:130
294419943,0cyclictest6445-21snmpd08:45:270
294419943,0cyclictest3425-21lldpd11:09:430
294419943,0cyclictest25435-21awk11:52:440
294419943,0cyclictest21449-21awk10:32:480
294419943,0cyclictest17504-21awk11:47:410
294419943,0cyclictest14051-21awk10:52:450
28415040,0irq/27-eth0-tx-20795-21munin-node08:22:430
28415040,0irq/27-eth0-tx-0-21swapper/209:57:422
28415040,0irq/27-eth0-tx-0-21swapper/008:17:450
28415040,0irq/27-eth0-tx-0-21swapper/008:07:530
28415040,0irq/27-eth0-tx-0-21swapper/007:27:450
28415040,0irq/27-eth0-tx-0-21swapper/007:22:400
28415040,0irq/27-eth0-tx-0-21swapper/006:57:430
28415040,0irq/27-eth0-tx-0-21swapper/006:52:430
28405040,0irq/26-eth0-rx-6445-21snmpd08:29:273
28405040,0irq/26-eth0-rx-0-21swapper/312:12:413
28405040,0irq/26-eth0-rx-0-21swapper/311:48:303
28405040,0irq/26-eth0-rx-0-21swapper/310:07:433
28405040,0irq/26-eth0-rx-0-21swapper/310:03:043
28405040,0irq/26-eth0-rx-0-21swapper/308:57:473
28405040,0irq/26-eth0-rx-0-21swapper/308:47:533
28405040,0irq/26-eth0-rx-0-21swapper/308:47:303
28405040,0irq/26-eth0-rx-0-21swapper/308:32:473
28405040,0irq/26-eth0-rx-0-21swapper/307:47:503
28405040,0irq/26-eth0-rx-0-21swapper/307:12:433
259940,0migration/29906-21unixbench_multi06:52:542
259940,0migration/227589-21kthreadcore07:12:482
259940,0migration/220239-21forks07:57:452
249940,0watchdog/20-21swapper/208:33:532
249940,0watchdog/20-21swapper/208:17:492
249940,0watchdog/20-21swapper/208:07:492
249940,0watchdog/20-21swapper/207:55:052
249940,0watchdog/20-21swapper/207:23:052
249940,0watchdog/20-21swapper/207:17:492
179940,0migration/130590-21cpuspeed_turbos11:07:381
179940,0migration/128898-21kthreadcore09:18:001
179940,0migration/128052-21kthreadcore07:12:481
179940,0migration/122838-21kthreadcore10:32:491
179940,0migration/121955-21aten_r1power_fr08:47:531
179940,0migration/118889-21runrttasks10:28:151
179940,0migration/116950-21runrttasks09:12:421
179940,0migration/114986-21kthreadcore06:57:491
179940,0migration/110952-21kthreadcore08:12:511
179940,0migration/110667-21kthreadcore08:37:501
169940,0watchdog/110727-21perl09:57:451
169940,0watchdog/10-21swapper/112:07:441
169940,0watchdog/10-21swapper/108:27:491
169940,0watchdog/10-21swapper/107:42:491
169940,0watchdog/10-21swapper/107:17:491
294469933,0cyclictest26500-21kworker/2:207:42:322
294429933,0cyclictest5375-21kworker/1:007:57:461
294419932,0cyclictest6445-21snmpd07:06:120
294419932,0cyclictest23142-21idleruntime-cro11:27:300
294419930,0cyclictest31452-21aten_r1power_en08:57:410
294419930,0cyclictest29654-21munin-node06:47:560
294419930,0cyclictest22538-21ntpq09:37:500
294419930,0cyclictest2023-21perl09:27:400
294419930,0cyclictest12226-21latency11:37:480
294419930,0cyclictest0-21swapper/008:32:480
294419930,0cyclictest0-21swapper/007:52:490
294419930,0cyclictest0-21swapper/007:17:440
28415030,0irq/27-eth0-tx-0-21swapper/007:57:430
28415030,0irq/27-eth0-tx-0-21swapper/007:32:420
28405030,0irq/26-eth0-rx-0-21swapper/309:27:363
28405030,0irq/26-eth0-rx-0-21swapper/308:23:433
28405030,0irq/26-eth0-rx-0-21swapper/307:37:443
28405030,0irq/26-eth0-rx-0-21swapper/307:21:203
28405030,0irq/26-eth0-rx-0-21swapper/306:47:443
294419921,0cyclictest3429-21latency_hist10:42:300
294419920,0cyclictest10774-21cat10:22:300
294419920,0cyclictest0-21swapper/009:47:380
28405020,0irq/26-eth0-rx-0-21swapper/308:39:573
28405020,0irq/26-eth0-rx-0-21swapper/308:09:303
28405020,0irq/26-eth0-rx-0-21swapper/307:57:413
28405020,0irq/26-eth0-rx-0-21swapper/307:32:443
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional