You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-13 - 12:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Tue May 13, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3034826462,0sleep20-21swapper/218:45:212
291350450,0irq/25-eth10-21swapper/318:46:193
3034123624,0sleep00-21swapper/018:45:150
3013523525,0sleep10-21swapper/118:42:481
339960,0migration/37553-21sh00:16:183
339960,0migration/327015-21sh22:24:543
339960,0migration/326614-21sh23:05:543
339960,0migration/319738-21sh21:39:413
339960,0migration/314713-21kthreadcore22:17:063
339960,0migration/311084-21sh22:14:553
305529966,0cyclictest28606-21diskmemload22:06:492
305529965,0cyclictest1235-21sed00:12:082
28415060,0irq/27-eth0-tx-30761-21munin-node18:47:120
28415060,0irq/27-eth0-tx-17048-21munin-node00:02:022
28415060,0irq/27-eth0-tx-0-21swapper/223:30:532
28415060,0irq/27-eth0-tx-0-21swapper/223:19:322
28415060,0irq/27-eth0-tx-0-21swapper/223:12:082
28415060,0irq/27-eth0-tx-0-21swapper/223:02:172
28415060,0irq/27-eth0-tx-0-21swapper/222:58:552
28415060,0irq/27-eth0-tx-0-21swapper/222:54:112
28415060,0irq/27-eth0-tx-0-21swapper/222:48:262
28415060,0irq/27-eth0-tx-0-21swapper/222:37:572
28415060,0irq/27-eth0-tx-0-21swapper/222:34:242
28415060,0irq/27-eth0-tx-0-21swapper/222:19:432
28415060,0irq/27-eth0-tx-0-21swapper/222:12:312
28415060,0irq/27-eth0-tx-0-21swapper/221:47:342
28415060,0irq/27-eth0-tx-0-21swapper/221:44:302
28415060,0irq/27-eth0-tx-0-21swapper/221:34:422
28415060,0irq/27-eth0-tx-0-21swapper/221:30:012
28405060,0irq/26-eth0-rx-0-21swapper/321:19:383
249960,0watchdog/20-21swapper/223:59:502
249960,0watchdog/20-21swapper/223:37:062
249960,0watchdog/20-21swapper/222:44:342
249960,0watchdog/20-21swapper/222:22:422
249960,0watchdog/20-21swapper/221:57:062
249960,0watchdog/20-21swapper/221:18:502
149960,0migration/02459-21latency_hist23:31:480
8225050,0irq/24-0000:00:12446-21latency_hist19:21:472
339950,0migration/39866-21irqcore20:32:033
339950,0migration/332102-21kthreadcore21:27:043
339950,0migration/331318-21needreboot19:32:063
339950,0migration/330190-21kthreadcore19:57:063
339950,0migration/32591-21kthreadcore19:12:053
339950,0migration/324268-21taskset00:06:183
339950,0migration/319846-21missed_timers20:12:093
339950,0migration/317887-21diskmemload23:19:263
339950,0migration/315899-21munin-node20:37:003
339950,0migration/313701-21kthreadcore18:57:083
339950,0migration/313674-21sh19:22:043
339950,0migration/312158-21taskset19:18:303
305489954,0cyclictest7666-21kworker/1:223:37:431
305489954,0cyclictest7666-21kworker/1:222:50:291
305489954,0cyclictest7666-21kworker/1:221:15:551
305489954,0cyclictest7666-21kworker/1:219:32:331
305469950,0cyclictest0-21swapper/023:37:550
305469950,0cyclictest0-21swapper/023:32:240
305469950,0cyclictest0-21swapper/022:46:310
305469950,0cyclictest0-21swapper/022:41:190
305469950,0cyclictest0-21swapper/020:53:280
28415050,0irq/27-eth0-tx-6372-21iceccd19:24:250
28415050,0irq/27-eth0-tx-27943-21munin-node22:27:032
28415050,0irq/27-eth0-tx-21734-21tune2fs21:02:012
28415050,0irq/27-eth0-tx-18349-21munin-node19:02:020
28415050,0irq/27-eth0-tx-15504-21munin-node20:37:030
28415050,0irq/27-eth0-tx-13131-21cpu20:57:012
28415050,0irq/27-eth0-tx-0-21swapper/223:53:382
28415050,0irq/27-eth0-tx-0-21swapper/223:44:392
28415050,0irq/27-eth0-tx-0-21swapper/223:32:142
28415050,0irq/27-eth0-tx-0-21swapper/223:22:002
28415050,0irq/27-eth0-tx-0-21swapper/223:06:552
28415050,0irq/27-eth0-tx-0-21swapper/221:52:002
28415050,0irq/27-eth0-tx-0-21swapper/221:13:022
28415050,0irq/27-eth0-tx-0-21swapper/220:52:572
28405050,0irq/26-eth0-rx-3653-21rm22:52:053
28405050,0irq/26-eth0-rx-3265-21entropy23:12:043
28405050,0irq/26-eth0-rx-30598-21kthreadcore22:47:083
28405050,0irq/26-eth0-rx-27056-21ssh22:03:533
28405050,0irq/26-eth0-rx-26204-21ssh21:42:083
28405050,0irq/26-eth0-rx-2483-21yum21:47:153
28405050,0irq/26-eth0-rx-24175-21kthreadcore21:02:053
28405050,0irq/26-eth0-rx-20850-21cat21:01:473
28405050,0irq/26-eth0-rx-1910-21unixbench-2d22:07:123
28405050,0irq/26-eth0-rx-17250-21kthreadcore21:57:063
28405050,0irq/26-eth0-rx-17164-21pmu-power22:37:123
28405050,0irq/26-eth0-rx-0-21swapper/323:52:023
28405050,0irq/26-eth0-rx-0-21swapper/323:42:113
28405050,0irq/26-eth0-rx-0-21swapper/323:26:563
28405050,0irq/26-eth0-rx-0-21swapper/323:23:063
28405050,0irq/26-eth0-rx-0-21swapper/323:07:003
28405050,0irq/26-eth0-rx-0-21swapper/322:35:543
28405050,0irq/26-eth0-rx-0-21swapper/321:11:023
28405050,0irq/26-eth0-rx-0-21swapper/320:56:363
28405050,0irq/26-eth0-rx-0-21swapper/320:47:053
28405050,0irq/26-eth0-rx-0-21swapper/300:07:233
259950,0migration/25999-21aten_r1power_cu19:17:002
259950,0migration/230181-21missed_timers00:07:092
259950,0migration/229340-21aten_r1power_po21:07:002
259950,0migration/229210-21kthreadcore19:07:072
259950,0migration/215046-21kthreadcore19:47:072
259950,0migration/210369-21kthreadcore19:42:062
249950,0watchdog/221401-21cpuspeed_turbos21:22:022
249950,0watchdog/20-21swapper/223:46:542
249950,0watchdog/20-21swapper/222:01:542
249950,0watchdog/20-21swapper/221:38:102
249950,0watchdog/20-21swapper/220:47:022
249950,0watchdog/20-21swapper/220:32:582
249950,0watchdog/20-21swapper/220:25:222
249950,0watchdog/20-21swapper/220:17:222
249950,0watchdog/20-21swapper/220:06:582
249950,0watchdog/20-21swapper/220:00:022
249950,0watchdog/20-21swapper/219:35:262
249950,0watchdog/20-21swapper/219:13:462
249950,0watchdog/20-21swapper/218:52:062
249950,0watchdog/20-21swapper/218:48:182
19150,0ktimersoftd/10-21swapper/122:36:051
179950,0migration/17159-21kthreadcore23:32:061
179950,0migration/16194-21kthreadcore20:27:081
179950,0migration/15174-21munin-node22:12:071
179950,0migration/123058-21kthreadcore21:02:051
179950,0migration/122708-21kthreadcore23:02:071
179950,0migration/115179-21kthreadcore21:37:041
179950,0migration/115156-21kthreadcore22:57:071
179950,0migration/114612-21kthreadcore20:57:071
179950,0migration/112641-21kthreadcore20:07:081
179950,0migration/112226-21cpuspeed_turbos22:17:011
169950,0watchdog/131821-21cut22:07:061
149950,0migration/07871-21munin-node20:27:120
149950,0migration/07719-21kthreadcore21:32:070
149950,0migration/07220-21cpuspeed_turbos19:42:010
149950,0migration/05917-21latency_hist19:16:470
149950,0migration/04728-21kthreadcore22:52:090
149950,0migration/04487-21kthreadcore23:12:070
149950,0migration/032563-21kthreadcore22:27:060
149950,0migration/032220-21kthreadcore22:47:080
149950,0migration/030645-21kthreadcore19:57:060
149950,0migration/030412-21kthreadcore19:32:050
149950,0migration/02757-21kthreadcore19:12:050
149950,0migration/027230-21kthreadcore20:42:070
149950,0migration/024587-21kthreadcore21:42:070
149950,0migration/022439-21kthreadcore23:22:060
149950,0migration/02163-21kthreadcore00:12:090
149950,0migration/020938-21kthreadcore23:02:060
149950,0migration/019501-21sh21:39:010
149950,0migration/018394-21sh23:41:500
149950,0migration/017405-21kthreadcore20:12:080
149950,0migration/015434-21kthreadcore20:57:070
149950,0migration/014731-21needreboot23:57:080
149950,0migration/014389-21munin-node19:47:040
149950,0migration/014182-21kthreadcore22:57:070
149950,0migration/012848-21kthreadcore20:32:040
149950,0migration/012327-21kthreadcore23:17:060
149950,0migration/010152-21kthreadcore20:07:070
8225040,0irq/24-0000:00:0-21swapper/218:56:562
339940,0migration/37277-21kthreadcore21:32:073
339940,0migration/36083-21kthreadcore20:27:083
339940,0migration/34963-21kthreadcore19:37:073
339940,0migration/314309-21kthreadcore23:37:073
339940,0migration/31323-21unixbench-2d20:22:143
339940,0migration/312371-21kthreadcore23:57:073
305529944,0cyclictest2470-21kworker/2:119:26:592
305529943,0cyclictest2470-21kworker/2:120:26:472
305529943,0cyclictest2470-21kworker/2:120:01:472
305529943,0cyclictest19187-21awk19:02:022
305529943,0cyclictest16413-21awk20:37:032
305489944,0cyclictest7666-21kworker/1:223:57:451
305489944,0cyclictest7666-21kworker/1:223:51:491
305489944,0cyclictest7666-21kworker/1:223:46:571
305489944,0cyclictest7666-21kworker/1:223:43:291
305489944,0cyclictest7666-21kworker/1:223:11:511
305489944,0cyclictest7666-21kworker/1:223:07:031
305489944,0cyclictest7666-21kworker/1:222:42:291
305489944,0cyclictest7666-21kworker/1:222:37:351
305489944,0cyclictest7666-21kworker/1:222:22:011
305489944,0cyclictest7666-21kworker/1:221:56:571
305489944,0cyclictest7666-21kworker/1:221:51:511
305489944,0cyclictest7666-21kworker/1:221:46:551
305489944,0cyclictest7666-21kworker/1:221:31:551
305489944,0cyclictest7666-21kworker/1:221:21:511
305489944,0cyclictest7666-21kworker/1:221:16:531
305489944,0cyclictest7666-21kworker/1:220:41:571
305489944,0cyclictest7666-21kworker/1:220:22:411
305489944,0cyclictest7666-21kworker/1:220:16:591
305489944,0cyclictest7666-21kworker/1:220:12:191
305489944,0cyclictest7666-21kworker/1:220:01:531
305489944,0cyclictest7666-21kworker/1:219:56:471
305489944,0cyclictest7666-21kworker/1:219:51:471
305489944,0cyclictest7666-21kworker/1:219:47:191
305489944,0cyclictest7666-21kworker/1:219:41:511
305489944,0cyclictest7666-21kworker/1:219:21:471
305489944,0cyclictest7666-21kworker/1:219:16:591
305489944,0cyclictest7666-21kworker/1:219:12:331
305489944,0cyclictest7666-21kworker/1:219:07:191
305489944,0cyclictest7666-21kworker/1:219:02:091
305489944,0cyclictest7666-21kworker/1:218:56:531
305489944,0cyclictest7666-21kworker/1:218:52:091
305489944,0cyclictest7666-21kworker/1:218:47:151
305489944,0cyclictest7666-21kworker/1:200:02:191
305489943,0cyclictest7666-21kworker/1:222:51:511
305489943,0cyclictest7666-21kworker/1:221:42:051
305489943,0cyclictest7666-21kworker/1:200:11:591
305489943,0cyclictest6445-21snmpd23:27:021
305489943,0cyclictest6445-21snmpd19:37:001
305489943,0cyclictest3425-21lldpd20:52:461
305489943,0cyclictest30068-21awk21:27:021
305489943,0cyclictest21344-21awk22:02:061
305489943,0cyclictest16389-21awk20:37:021
305469944,0cyclictest0-21swapper/022:13:050
305469944,0cyclictest0-21swapper/021:47:050
305469943,0cyclictest6445-21snmpd22:34:160
305469943,0cyclictest6445-21snmpd22:05:160
305469943,0cyclictest30120-21awk21:07:040
305469943,0cyclictest28154-21awk23:07:040
305469943,0cyclictest22021-21awk20:17:040
305469943,0cyclictest2136-21awk20:02:060
305469943,0cyclictest13214-21awk21:57:030
305469941,0cyclictest0-21swapper/020:22:250
305469941,0cyclictest0-21swapper/019:07:000
305469940,0cyclictest22041-21grep19:52:020
305469940,0cyclictest0-21swapper/021:28:380
28415040,0irq/27-eth0-tx-32595-21munin-node19:37:010
28415040,0irq/27-eth0-tx-0-21swapper/020:46:530
28405040,0irq/26-eth0-rx-0-21swapper/323:47:183
28405040,0irq/26-eth0-rx-0-21swapper/323:33:143
28405040,0irq/26-eth0-rx-0-21swapper/322:42:433
28405040,0irq/26-eth0-rx-0-21swapper/322:26:473
28405040,0irq/26-eth0-rx-0-21swapper/321:54:393
28405040,0irq/26-eth0-rx-0-21swapper/321:24:443
28405040,0irq/26-eth0-rx-0-21swapper/320:16:473
28405040,0irq/26-eth0-rx-0-21swapper/319:52:003
259940,0migration/220962-21fschecks_time19:51:592
249940,0watchdog/224178-21grep20:42:062
249940,0watchdog/20-21swapper/220:14:222
249940,0watchdog/20-21swapper/219:36:542
179940,0migration/18913-21aten_r1power_en20:31:581
179940,0migration/125241-21fschecks_count00:07:041
179940,0migration/122619-21kthreadcore23:22:061
179940,0migration/121061-21kthreadcore19:27:071
179940,0migration/112639-21kthreadcore23:17:061
179940,0migration/11113-21kthreadcore21:07:061
169940,0watchdog/127895-21ssh22:26:501
169940,0watchdog/10-21swapper/120:47:021
149940,0migration/09069-21kthreadcore21:52:070
149940,0migration/06606-21kthreadcore23:52:070
149940,0migration/023389-21kthreadcore22:22:060
149940,0migration/022427-21irqstats21:22:060
149940,0migration/021911-21idleruntime21:02:020
149940,0migration/019429-21aten_r1power_po19:27:020
305469932,0cyclictest6033-21awk21:12:060
305469932,0cyclictest3425-21lldpd23:49:570
305469932,0cyclictest27841-21kthreadcore00:07:080
305469932,0cyclictest17141-21awk18:57:110
305469932,0cyclictest14058-21awk21:17:040
28405030,0irq/26-eth0-rx-0-21swapper/322:56:483
28405030,0irq/26-eth0-rx-0-21swapper/321:12:013
28405030,0irq/26-eth0-rx-0-21swapper/320:42:013
28405030,0irq/26-eth0-rx-0-21swapper/320:11:213
28405030,0irq/26-eth0-rx-0-21swapper/320:02:023
28405030,0irq/26-eth0-rx-0-21swapper/319:41:593
28405030,0irq/26-eth0-rx-0-21swapper/319:07:043
28405030,0irq/26-eth0-rx-0-21swapper/319:01:593
28405030,0irq/26-eth0-rx-0-21swapper/318:52:103
28405030,0irq/26-eth0-rx-0-21swapper/318:47:083
149930,0migration/019142-21sh22:20:140
305469921,0cyclictest28394-21aten_r1power_fr22:07:000
305469920,0cyclictest0-21swapper/018:51:500
305469920,0cyclictest0-21swapper/000:01:480
28405020,0irq/26-eth0-rx-0-21swapper/319:46:563
28405020,0irq/26-eth0-rx-0-21swapper/319:27:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional