You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-07 - 11:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Mon Jul 07, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2496323723,0sleep00-21swapper/018:39:480
3125323626,0sleep30-21swapper/318:43:593
3121623625,0sleep20-21swapper/218:43:302
3101723430,0sleep10-21swapper/118:41:091
249970,0watchdog/25505-21kthreadcore22:30:042
8225060,0irq/24-0000:00:12342-21latency_hist18:54:452
35160,0ktimersoftd/30-21swapper/320:46:313
35160,0ktimersoftd/30-21swapper/300:11:193
339960,0migration/38923-21sh22:30:203
339960,0migration/34635-21sh21:07:503
339960,0migration/327264-21sh21:40:523
313949960,0cyclictest29535-21diskmemload20:53:432
313939965,0cyclictest7874-21kworker/1:023:53:411
313939965,0cyclictest7874-21kworker/1:023:40:471
313939965,0cyclictest7874-21kworker/1:000:04:591
313939965,0cyclictest17464-21kworker/1:222:45:591
313939965,0cyclictest17464-21kworker/1:222:30:031
313939965,0cyclictest17464-21kworker/1:221:00:071
313939965,0cyclictest17464-21kworker/1:220:05:051
28415060,0irq/27-eth0-tx-21831-21ssh23:40:082
28415060,0irq/27-eth0-tx-20776-21ssh21:18:122
28415060,0irq/27-eth0-tx-18473-21munin-node22:40:092
28415060,0irq/27-eth0-tx-0-21swapper/223:57:182
28415060,0irq/27-eth0-tx-0-21swapper/223:49:112
28415060,0irq/27-eth0-tx-0-21swapper/223:36:422
28415060,0irq/27-eth0-tx-0-21swapper/223:34:352
28415060,0irq/27-eth0-tx-0-21swapper/223:28:252
28415060,0irq/27-eth0-tx-0-21swapper/223:23:052
28415060,0irq/27-eth0-tx-0-21swapper/223:14:452
28415060,0irq/27-eth0-tx-0-21swapper/223:11:012
28415060,0irq/27-eth0-tx-0-21swapper/222:50:002
28415060,0irq/27-eth0-tx-0-21swapper/222:17:092
28415060,0irq/27-eth0-tx-0-21swapper/222:08:102
28415060,0irq/27-eth0-tx-0-21swapper/222:04:092
28415060,0irq/27-eth0-tx-0-21swapper/221:43:252
28415060,0irq/27-eth0-tx-0-21swapper/221:37:412
28415060,0irq/27-eth0-tx-0-21swapper/221:00:112
28415060,0irq/27-eth0-tx-0-21swapper/200:10:252
28415060,0irq/27-eth0-tx-0-21swapper/200:05:592
28405060,0irq/26-eth0-rx-0-21swapper/323:04:593
249960,0watchdog/20-21swapper/222:29:282
149960,0migration/06401-21fschecks_count18:50:000
149960,0migration/032367-21sh23:07:350
149960,0migration/024148-21cpuspeed_turbos00:10:020
149960,0migration/020085-21kthreadcore22:40:060
8225050,0irq/24-0000:00:29623-21latency_hist21:04:452
339950,0migration/39064-21pmu-power22:10:083
339950,0migration/38624-21turbostat23:14:453
339950,0migration/37013-21munin-node19:15:003
339950,0migration/36218-21timerwakeupswit23:30:113
339950,0migration/330934-21kthreadcore20:20:053
339950,0migration/328904-21taskset21:24:003
339950,0migration/328400-21kthreadcore19:05:043
339950,0migration/32414-21munin-node23:50:043
339950,0migration/322850-21kthreadcore19:00:033
339950,0migration/321894-21munin-node20:35:093
339950,0migration/318550-21runrttasks22:40:013
339950,0migration/318368-21munin-node23:00:003
339950,0migration/317791-21kthreadcore23:40:033
339950,0migration/317703-21kthreadcore00:05:043
339950,0migration/317188-21kthreadcore20:55:103
339950,0migration/317151-21munin-node23:19:563
339950,0migration/315321-21sh00:02:103
339950,0migration/314922-21runrttasks19:44:593
339950,0migration/31405-21kthreadcore18:45:033
339950,0migration/312799-21sh21:13:013
339950,0migration/312226-21munin-node22:15:013
339950,0migration/311513-21runrttasks21:53:283
339950,0migration/311513-21kthreadcore19:40:103
339950,0migration/311425-21munin-node22:35:053
313949955,0cyclictest0-21swapper/223:07:342
313949950,0cyclictest0-21swapper/223:02:222
313939955,0cyclictest7874-21kworker/1:023:54:491
313939955,0cyclictest7874-21kworker/1:023:45:571
313939955,0cyclictest7874-21kworker/1:023:35:251
313939955,0cyclictest7874-21kworker/1:000:10:171
313939955,0cyclictest7874-21kworker/1:000:01:471
313939955,0cyclictest17464-21kworker/1:223:29:531
313939955,0cyclictest17464-21kworker/1:223:25:051
313939955,0cyclictest17464-21kworker/1:223:19:451
313939955,0cyclictest17464-21kworker/1:223:14:531
313939955,0cyclictest17464-21kworker/1:223:09:491
313939955,0cyclictest17464-21kworker/1:223:07:151
313939955,0cyclictest17464-21kworker/1:222:59:551
313939955,0cyclictest17464-21kworker/1:222:55:011
313939955,0cyclictest17464-21kworker/1:222:49:451
313939955,0cyclictest17464-21kworker/1:222:44:211
313939955,0cyclictest17464-21kworker/1:222:34:551
313939955,0cyclictest17464-21kworker/1:222:25:011
313939955,0cyclictest17464-21kworker/1:222:20:391
313939955,0cyclictest17464-21kworker/1:222:15:571
313939955,0cyclictest17464-21kworker/1:222:09:451
313939955,0cyclictest17464-21kworker/1:222:04:571
313939955,0cyclictest17464-21kworker/1:221:59:451
313939955,0cyclictest17464-21kworker/1:221:55:211
313939955,0cyclictest17464-21kworker/1:221:47:111
313939955,0cyclictest17464-21kworker/1:221:39:511
313939955,0cyclictest17464-21kworker/1:221:34:571
313939955,0cyclictest17464-21kworker/1:221:29:591
313939955,0cyclictest17464-21kworker/1:221:20:571
313939955,0cyclictest17464-21kworker/1:221:15:551
313939955,0cyclictest17464-21kworker/1:221:09:471
313939955,0cyclictest17464-21kworker/1:221:05:051
313939955,0cyclictest17464-21kworker/1:220:54:471
313939955,0cyclictest17464-21kworker/1:220:40:071
313939955,0cyclictest17464-21kworker/1:220:35:191
313939955,0cyclictest17464-21kworker/1:220:29:591
313939955,0cyclictest17464-21kworker/1:220:20:031
313939955,0cyclictest17464-21kworker/1:220:16:071
313939955,0cyclictest17464-21kworker/1:220:14:091
313939955,0cyclictest17464-21kworker/1:220:02:071
313939955,0cyclictest17464-21kworker/1:219:54:571
313939955,0cyclictest17464-21kworker/1:219:46:051
313939955,0cyclictest17464-21kworker/1:219:40:071
313939955,0cyclictest17464-21kworker/1:219:35:011
313939955,0cyclictest17464-21kworker/1:219:30:051
313939955,0cyclictest17464-21kworker/1:219:23:091
313939955,0cyclictest17464-21kworker/1:219:10:171
313939955,0cyclictest17464-21kworker/1:218:49:591
313939954,0cyclictest17464-21kworker/1:221:49:571
313939954,0cyclictest17464-21kworker/1:220:51:431
313939954,0cyclictest17464-21kworker/1:220:44:471
313929955,0cyclictest0-21swapper/000:06:220
313929950,0cyclictest0-21swapper/023:15:450
313929950,0cyclictest0-21swapper/022:56:310
313929950,0cyclictest0-21swapper/021:17:210
28415050,0irq/27-eth0-tx-5620-21munin-node21:10:002
28415050,0irq/27-eth0-tx-4334-21munin-node21:50:082
28415050,0irq/27-eth0-tx-0-21swapper/222:55:122
28415050,0irq/27-eth0-tx-0-21swapper/222:47:442
28415050,0irq/27-eth0-tx-0-21swapper/222:36:572
28415050,0irq/27-eth0-tx-0-21swapper/222:11:472
28415050,0irq/27-eth0-tx-0-21swapper/221:46:292
28415050,0irq/27-eth0-tx-0-21swapper/221:24:172
28415050,0irq/27-eth0-tx-0-21swapper/200:00:182
28405050,0irq/26-eth0-rx-9363-21ssh22:54:383
28405050,0irq/26-eth0-rx-6547-21runrttasks22:01:033
28405050,0irq/26-eth0-rx-32342-21kthreadcore21:45:023
28405050,0irq/26-eth0-rx-24715-21munin-node23:25:003
28405050,0irq/26-eth0-rx-0-21swapper/323:35:043
28405050,0irq/26-eth0-rx-0-21swapper/323:16:553
28405050,0irq/26-eth0-rx-0-21swapper/322:55:063
28405050,0irq/26-eth0-rx-0-21swapper/322:20:083
28405050,0irq/26-eth0-rx-0-21swapper/321:34:453
28405050,0irq/26-eth0-rx-0-21swapper/321:17:103
28405050,0irq/26-eth0-rx-0-21swapper/320:49:533
259950,0migration/223332-21kthreadcore19:00:032
259950,0migration/222508-21run-parts20:35:452
259950,0migration/220766-21sh20:57:422
249950,0watchdog/20-21swapper/223:50:352
249950,0watchdog/20-21swapper/222:20:002
249950,0watchdog/20-21swapper/221:31:482
249950,0watchdog/20-21swapper/221:24:482
249950,0watchdog/20-21swapper/220:48:322
249950,0watchdog/20-21swapper/220:40:082
249950,0watchdog/20-21swapper/220:25:042
249950,0watchdog/20-21swapper/220:10:042
249950,0watchdog/20-21swapper/219:55:082
179950,0migration/13047-21timerwakeupswit21:25:101
179950,0migration/127097-21irqrtprio19:05:031
149950,0migration/09738-21kthreadcore21:30:040
149950,0migration/09075-21kthreadcore21:50:050
149950,0migration/08623-21gltestperf19:40:070
149950,0migration/07475-21kthreadcore22:10:040
149950,0migration/07416-21cpu23:55:000
149950,0migration/06641-21kthreadcore22:50:060
149950,0migration/06213-21kthreadcore19:35:040
149950,0migration/05898-21kthreadcore22:30:040
149950,0migration/04892-21kthreadcore20:00:030
149950,0migration/04601-21kthreadcore20:25:030
149950,0migration/030416-21diskmemload21:05:000
149950,0migration/029409-21kthreadcore22:05:030
149950,0migration/028752-21kthreadcore19:05:040
149950,0migration/028367-21kthreadcore23:25:040
149950,0migration/027462-21fschecks_time19:30:010
149950,0migration/027065-21taskset22:02:590
149950,0migration/025380-21kthreadcore21:20:020
149950,0migration/02349-21kthreadcore23:30:050
149950,0migration/023278-21sh23:43:420
149950,0migration/020160-21kthreadcore20:35:060
149950,0migration/019728-21sh21:59:400
149950,0migration/019635-21kthreadcore19:45:040
149950,0migration/019363-21kthreadcore23:20:010
149950,0migration/014021-21kthreadcore20:30:040
149950,0migration/012954-21kthreadcore23:35:050
149950,0migration/011991-21fschecks_time22:15:000
149950,0migration/011277-21pmu-power21:10:070
149950,0migration/01094-21kthreadcore21:25:050
149950,0migration/010784-21kthreadcore19:15:060
8225040,0irq/24-0000:00:0-21swapper/219:04:452
339940,0migration/36519-21missed_timers19:35:053
339940,0migration/36467-21kthreadcore20:00:033
339940,0migration/35324-21kthreadcore20:25:033
339940,0migration/331171-21kthreadcore21:25:043
339940,0migration/329490-21taskset20:44:293
339940,0migration/329249-21kthreadcore19:55:033
339940,0migration/328001-21kthreadcore22:45:023
339940,0migration/327908-21taskset22:04:543
339940,0migration/327138-21aten_r1power_cu22:24:553
339940,0migration/323838-21sh23:44:493
339940,0migration/323057-21kthreadcore19:50:023
339940,0migration/319820-21kthreadcore20:10:053
339940,0migration/313568-21kthreadcore20:30:043
339940,0migration/310087-21kthreadcore18:50:043
313949944,0cyclictest30974-21kworker/2:121:54:452
313949944,0cyclictest30974-21kworker/2:120:29:472
313949944,0cyclictest30974-21kworker/2:120:19:532
313949944,0cyclictest30974-21kworker/2:120:04:512
313949944,0cyclictest30974-21kworker/2:119:59:472
313949944,0cyclictest30974-21kworker/2:119:39:492
313949944,0cyclictest30974-21kworker/2:119:24:532
313949944,0cyclictest30974-21kworker/2:119:20:152
313949944,0cyclictest30974-21kworker/2:119:15:052
313949944,0cyclictest30974-21kworker/2:119:09:512
313949942,0cyclictest22101-21munin-node20:15:042
313939944,0cyclictest17464-21kworker/1:219:24:471
313939944,0cyclictest17464-21kworker/1:219:15:011
313939944,0cyclictest17464-21kworker/1:218:54:591
313939944,0cyclictest17464-21kworker/1:218:44:531
313939943,0cyclictest3110-21irqbalance20:24:501
313939943,0cyclictest22018-21awk19:50:001
313939943,0cyclictest20179-21awk19:00:011
313929944,0cyclictest0-21swapper/023:52:580
313929944,0cyclictest0-21swapper/019:00:110
313929944,0cyclictest0-21swapper/018:54:590
313929943,0cyclictest6445-21snmpd23:48:240
313929943,0cyclictest6445-21snmpd21:37:380
313929943,0cyclictest6445-21snmpd20:19:070
313929943,0cyclictest27218-21awk22:45:010
313929943,0cyclictest25186-21sadc23:04:440
313929943,0cyclictest22768-21awk21:00:030
313929943,0cyclictest22050-21awk19:50:000
313929943,0cyclictest13774-21tail22:35:080
28415040,0irq/27-eth0-tx-0-21swapper/218:44:592
28405040,0irq/26-eth0-rx-0-21swapper/323:55:243
28405040,0irq/26-eth0-rx-0-21swapper/321:55:063
28405040,0irq/26-eth0-rx-0-21swapper/321:35:043
28405040,0irq/26-eth0-rx-0-21swapper/321:00:453
28405040,0irq/26-eth0-rx-0-21swapper/318:55:033
259940,0migration/227095-21cpuspeed_turbos19:29:592
259940,0migration/215205-21fschecks_time19:45:002
249940,0watchdog/20-21swapper/219:49:522
249940,0watchdog/20-21swapper/219:35:002
249940,0watchdog/20-21swapper/218:54:482
149940,0migration/02929-21kthreadcore18:45:030
149940,0migration/02196-21ksmtuned21:45:070
149940,0migration/017000-21kthreadcore20:55:100
149940,0migration/015885-21runrttasks00:04:280
313929933,0cyclictest0-21swapper/020:46:080
313929932,0cyclictest6445-21snmpd22:19:550
313929932,0cyclictest29646-21awk20:20:030
313929932,0cyclictest26667-21sensors21:40:070
313929932,0cyclictest23950-21awk20:40:010
313929932,0cyclictest21387-21grep20:10:100
313929930,0cyclictest776-21perl19:10:000
313929930,0cyclictest3365-21grep23:10:010
313929930,0cyclictest27751-21perl22:24:580
313929930,0cyclictest1881-21sed19:59:450
313929930,0cyclictest10996-21mdstat20:50:080
313929930,0cyclictest0-21swapper/019:24:590
313929930,0cyclictest0-21swapper/019:20:020
28405030,0irq/26-eth0-rx-0-21swapper/320:17:273
28405030,0irq/26-eth0-rx-0-21swapper/320:06:233
28405030,0irq/26-eth0-rx-0-21swapper/319:33:503
28405030,0irq/26-eth0-rx-0-21swapper/319:12:363
339920,0migration/313949-21diskstats19:19:583
313929921,0cyclictest8897-21cut20:04:570
28405020,0irq/26-eth0-rx-0-21swapper/319:25:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional