You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 09:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Sun Nov 23, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2455927473,0sleep00-21swapper/018:36:310
2447423928,0sleep10-21swapper/118:35:321
2447123828,0sleep30-21swapper/318:35:303
2465423526,0sleep20-21swapper/218:37:372
248939988,0cyclictest6445-21snmpd23:24:120
4160,0ktimersoftd/00-21swapper/023:34:400
339960,0migration/36377-21sh21:27:093
339960,0migration/36101-21latency_hist23:09:353
339960,0migration/321895-21sh20:55:333
339960,0migration/320690-21sh23:37:193
339960,0migration/320669-21sh23:15:543
28415060,0irq/27-eth0-tx-20136-21ssh23:35:312
28415060,0irq/27-eth0-tx-0-21swapper/223:49:192
28415060,0irq/27-eth0-tx-0-21swapper/223:31:232
28415060,0irq/27-eth0-tx-0-21swapper/223:14:592
28415060,0irq/27-eth0-tx-0-21swapper/223:12:562
28415060,0irq/27-eth0-tx-0-21swapper/222:57:182
28415060,0irq/27-eth0-tx-0-21swapper/222:45:062
28415060,0irq/27-eth0-tx-0-21swapper/222:41:572
28415060,0irq/27-eth0-tx-0-21swapper/222:35:252
28415060,0irq/27-eth0-tx-0-21swapper/222:25:102
28415060,0irq/27-eth0-tx-0-21swapper/222:02:332
28415060,0irq/27-eth0-tx-0-21swapper/221:50:252
28415060,0irq/27-eth0-tx-0-21swapper/221:44:542
28415060,0irq/27-eth0-tx-0-21swapper/221:27:142
28415060,0irq/27-eth0-tx-0-21swapper/221:21:552
28415060,0irq/27-eth0-tx-0-21swapper/220:55:532
28415060,0irq/27-eth0-tx-0-21swapper/220:51:292
28415060,0irq/27-eth0-tx-0-21swapper/220:49:022
28415060,0irq/27-eth0-tx-0-21swapper/200:05:232
28415060,0irq/27-eth0-tx-0-21swapper/020:29:470
248959966,0cyclictest0-21swapper/220:41:322
248959960,0cyclictest0-21swapper/222:32:412
248959960,0cyclictest0-21swapper/220:24:492
249960,0watchdog/224502-21if_err_eth121:39:492
149960,0migration/021686-21sh23:18:490
8225050,0irq/24-0000:00:3082-21munin-run20:24:352
339950,0migration/39384-21kthreadcore19:39:533
339950,0migration/38544-21kthreadcore23:29:553
339950,0migration/37728-21kthreadcore23:49:543
339950,0migration/35679-21sh22:06:313
339950,0migration/35128-21kthreadcore19:59:523
339950,0migration/33964-21unixbench_multi23:25:013
339950,0migration/332727-21munin-node21:04:523
339950,0migration/331340-21latency_hist20:44:353
339950,0migration/329775-21aten_r1power_vo23:44:473
339950,0migration/327585-21kthreadcore18:39:543
339950,0migration/32703-21kthreadcore22:24:533
339950,0migration/326479-21df_abs19:04:483
339950,0migration/324526-21idleruntime20:59:493
339950,0migration/323738-21munin-node22:39:513
339950,0migration/320863-21taskset22:35:053
339950,0migration/319374-21latency_hist18:59:353
339950,0migration/319127-21needreboot19:19:543
339950,0migration/31872-21munin-node00:04:543
339950,0migration/315866-21gltestperf22:14:493
339950,0migration/314426-21sh22:54:443
339950,0migration/312884-21kthreadcore21:09:543
28415050,0irq/27-eth0-tx-7062-21munin-node22:09:512
28415050,0irq/27-eth0-tx-3561-21proc_pri23:24:582
28415050,0irq/27-eth0-tx-14701-21munin-node19:44:530
28415050,0irq/27-eth0-tx-0-21swapper/223:59:432
28415050,0irq/27-eth0-tx-0-21swapper/223:56:502
28415050,0irq/27-eth0-tx-0-21swapper/223:42:152
28415050,0irq/27-eth0-tx-0-21swapper/223:04:432
28415050,0irq/27-eth0-tx-0-21swapper/223:04:432
28415050,0irq/27-eth0-tx-0-21swapper/223:02:412
28415050,0irq/27-eth0-tx-0-21swapper/222:15:432
28415050,0irq/27-eth0-tx-0-21swapper/222:07:532
28415050,0irq/27-eth0-tx-0-21swapper/221:55:082
28415050,0irq/27-eth0-tx-0-21swapper/221:35:102
28415050,0irq/27-eth0-tx-0-21swapper/221:29:462
28415050,0irq/27-eth0-tx-0-21swapper/221:15:162
28415050,0irq/27-eth0-tx-0-21swapper/221:09:482
28415050,0irq/27-eth0-tx-0-21swapper/221:07:332
28415050,0irq/27-eth0-tx-0-21swapper/221:02:502
28415050,0irq/27-eth0-tx-0-21swapper/019:54:250
28405050,0irq/26-eth0-rx-32743-21kthreadcore22:44:523
28405050,0irq/26-eth0-rx-27905-21runrttasks00:02:243
28405050,0irq/26-eth0-rx-27466-21chrt19:50:233
28405050,0irq/26-eth0-rx-23519-21iostat_ios22:59:503
28405050,0irq/26-eth0-rx-21151-21ssh21:34:543
28405050,0irq/26-eth0-rx-20295-21ssh23:58:203
28405050,0irq/26-eth0-rx-16675-21kthreadcore21:54:523
28405050,0irq/26-eth0-rx-16173-21df_inode21:14:493
28405050,0irq/26-eth0-rx-13874-21ssh21:30:233
28405050,0irq/26-eth0-rx-10460-21cut22:49:533
28405050,0irq/26-eth0-rx-0-21swapper/323:41:563
28405050,0irq/26-eth0-rx-0-21swapper/322:29:523
28405050,0irq/26-eth0-rx-0-21swapper/322:09:513
28405050,0irq/26-eth0-rx-0-21swapper/321:59:533
28405050,0irq/26-eth0-rx-0-21swapper/321:47:593
28405050,0irq/26-eth0-rx-0-21swapper/321:43:483
28405050,0irq/26-eth0-rx-0-21swapper/321:19:533
28405050,0irq/26-eth0-rx-0-21swapper/320:41:223
28405050,0irq/26-eth0-rx-0-21swapper/319:24:483
259950,0migration/232085-21kthreadcore19:29:542
259950,0migration/210147-21kthreadcore22:49:532
248959955,0cyclictest0-21swapper/223:51:402
248949950,0cyclictest0-21swapper/118:49:551
248939955,0cyclictest0-21swapper/023:46:290
248939955,0cyclictest0-21swapper/022:16:560
248939950,0cyclictest0-21swapper/021:17:210
249950,0watchdog/228648-21sh23:20:452
249950,0watchdog/21722-21forks19:34:502
179950,0migration/16915-21latency_hist22:09:351
179950,0migration/132358-21fschecks_time18:44:491
179950,0migration/129902-21sh20:41:141
179950,0migration/128061-21taskset19:54:431
179950,0migration/125741-21missed_timers19:24:561
179950,0migration/123809-21kthreadcore23:19:511
179950,0migration/117773-21kthreadcore22:14:521
179950,0migration/113018-21pmu-power20:49:561
179950,0migration/112237-21switchtime22:49:581
169950,0watchdog/185612chrt20:01:171
169950,0watchdog/1321922sleep119:06:101
169950,0watchdog/130234-21ssh22:23:011
169950,0watchdog/118354-21processes18:54:581
169950,0watchdog/112540-21rm23:10:211
169950,0watchdog/10-21swapper/122:55:451
169950,0watchdog/10-21swapper/122:43:011
169950,0watchdog/10-21swapper/121:22:091
169950,0watchdog/10-21swapper/120:44:531
149950,0migration/08627-21irqrtprio20:49:520
149950,0migration/08085-21irqcore19:14:510
149950,0migration/04812-21kthreadcore20:24:520
149950,0migration/03430-21kthreadcore21:04:560
149950,0migration/031555-21taskset21:04:230
149950,0migration/029949-21kthreadcore19:05:000
149950,0migration/029411-21df_abs20:19:480
149950,0migration/028833-21latency_hist20:19:350
149950,0migration/027790-21gltestperf19:29:500
149950,0migration/027744-21sh00:01:540
149950,0migration/024362-21gltestperf21:39:480
149950,0migration/023441-21latency_hist21:39:350
149950,0migration/023302-21latency_hist20:39:350
149950,0migration/023006-21taskset20:38:330
149950,0migration/020075-21runrttasks19:23:260
149950,0migration/014878-21cpu22:34:460
149950,0migration/013562-21timerwakeupswit21:29:580
149950,0migration/012866-21taskset22:50:530
149950,0migration/01259-21kthreadcore21:44:520
149950,0migration/012001-21kthreadcore19:39:530
339940,0migration/38094-21irqcore19:14:513
339940,0migration/35818-21nfs_client19:09:563
339940,0migration/323625-21fschecks_count22:19:503
339940,0migration/313297-21fschecks_count18:54:503
339940,0migration/310615-21kthreadcore18:49:533
28415040,0irq/27-eth0-tx-0-21swapper/018:39:510
28405040,0irq/26-eth0-rx-0-21swapper/323:21:263
28405040,0irq/26-eth0-rx-0-21swapper/323:06:553
28405040,0irq/26-eth0-rx-0-21swapper/323:06:553
28405040,0irq/26-eth0-rx-0-21swapper/321:52:413
28405040,0irq/26-eth0-rx-0-21swapper/320:51:053
28405040,0irq/26-eth0-rx-0-21swapper/320:04:463
259940,0migration/29443-21kthreadcore19:14:522
259940,0migration/23000-21fschecks_time19:59:492
259940,0migration/225199-21kthreadcore22:19:542
259940,0migration/221512-21kthreadcore20:34:532
259940,0migration/212899-21kthreadcore19:39:542
259940,0migration/212730-21kthreadcore20:04:542
248959944,0cyclictest0-21swapper/219:23:012
248959944,0cyclictest0-21swapper/219:09:562
248959943,0cyclictest6445-21snmpd20:14:562
248959943,0cyclictest3425-21lldpd19:06:522
248959943,0cyclictest32621-21awk18:44:502
248959943,0cyclictest3110-21irqbalance20:30:492
248959943,0cyclictest3110-21irqbalance19:54:392
248959943,0cyclictest3110-21irqbalance19:49:592
248959943,0cyclictest20368-21awk18:59:522
248959943,0cyclictest16647-21awk20:09:512
248959943,0cyclictest13762-21awk18:54:522
248949943,0cyclictest6445-21snmpd21:00:121
248949943,0cyclictest6445-21snmpd20:09:411
248949943,0cyclictest6437-21sadc22:29:351
248949943,0cyclictest6144-21sadc22:49:351
248949943,0cyclictest5272-21kworker/1:121:49:521
248949943,0cyclictest5272-21kworker/1:121:39:401
248949943,0cyclictest5272-21kworker/1:121:29:471
248949943,0cyclictest463-21awk21:04:541
248949943,0cyclictest4179-21awk20:24:501
248949943,0cyclictest3425-21lldpd22:02:311
248949943,0cyclictest3425-21lldpd20:07:231
248949943,0cyclictest32677-21awk21:44:511
248949943,0cyclictest30490-21awk23:44:501
248949943,0cyclictest29728-21awk00:04:501
248949943,0cyclictest22243-21awk19:49:521
248949943,0cyclictest21130-21kworker/1:223:58:201
248949943,0cyclictest15180-21awk23:14:511
248949942,0cyclictest5272-21kworker/1:121:34:411
248939944,0cyclictest0-21swapper/023:58:000
248939944,0cyclictest0-21swapper/023:26:000
248939944,0cyclictest0-21swapper/023:00:370
248939944,0cyclictest0-21swapper/022:40:450
248939944,0cyclictest0-21swapper/022:09:510
248939944,0cyclictest0-21swapper/021:50:270
248939943,0cyclictest8629-21awk21:09:510
248939943,0cyclictest6808-21awk23:29:530
248939943,0cyclictest32652-21awk18:44:500
248939943,0cyclictest2248-21awk19:34:520
248939943,0cyclictest21298-21awk19:24:520
248939943,0cyclictest15396-21awk22:54:520
249940,0watchdog/222555-21kthreadcore19:24:542
249940,0watchdog/20-21swapper/219:44:532
249940,0watchdog/20-21swapper/218:49:582
179940,0migration/17161-21aten_r1power_fr19:14:461
179940,0migration/126070-21kthreadcore23:59:531
179940,0migration/122077-21df_abs23:39:481
179940,0migration/116616-21runrttasks20:54:531
179940,0migration/115957-21fschecks_time21:54:491
179940,0migration/112028-21unixbench_singl23:30:021
179940,0migration/111215-21timerwakeupswit23:50:001
179940,0migration/110695-21fschecks_time20:29:491
179940,0migration/110145-21kthreadcore22:09:521
179940,0migration/110037-21kthreadcore19:39:531
169940,0watchdog/13202-21kthreadcore21:24:531
169940,0watchdog/128488-21timerwakeupswit22:59:571
169940,0watchdog/120165-21ssh23:35:371
169940,0watchdog/116178-21diskstats21:14:491
169940,0watchdog/10-21swapper/122:34:491
169940,0watchdog/10-21swapper/121:09:491
169940,0watchdog/10-21swapper/119:34:541
169940,0watchdog/10-21swapper/119:29:501
169940,0watchdog/10-21swapper/118:59:541
149940,0migration/04268-21kthreadcore19:09:540
149940,0migration/023201-21aten_r1power_po22:19:480
149940,0migration/015777-21aten_r1power_en20:09:470
149940,0migration/013347-21kthreadcore20:04:540
149940,0migration/011962-21kthreadcore22:29:520
28405030,0irq/26-eth0-rx-0-21swapper/320:34:513
28405030,0irq/26-eth0-rx-0-21swapper/320:30:013
28405030,0irq/26-eth0-rx-0-21swapper/320:24:473
28405030,0irq/26-eth0-rx-0-21swapper/320:19:473
28405030,0irq/26-eth0-rx-0-21swapper/320:11:013
28405030,0irq/26-eth0-rx-0-21swapper/319:54:493
28405030,0irq/26-eth0-rx-0-21swapper/319:35:033
28405030,0irq/26-eth0-rx-0-21swapper/319:34:023
248959932,0cyclictest3425-21lldpd18:39:522
248949933,0cyclictest2765-21kworker/1:020:34:411
248949933,0cyclictest2765-21kworker/1:020:14:471
248949933,0cyclictest2765-21kworker/1:019:09:491
248949933,0cyclictest2765-21kworker/1:018:39:351
248949932,0cyclictest14542-21idleruntime19:19:501
248949932,0cyclictest14468-21latency_hist19:44:351
248939933,0cyclictest3425-21lldpd23:53:380
248939933,0cyclictest0-21swapper/022:24:400
248939932,0cyclictest6445-21snmpd22:01:570
248939932,0cyclictest6445-21snmpd20:57:120
248939932,0cyclictest6445-21snmpd20:44:350
248939932,0cyclictest6247-21aten2.4-expect23:09:450
248939932,0cyclictest3334-21awk19:59:510
248939932,0cyclictest30680-21stty23:04:460
248939932,0cyclictest30680-21stty23:04:460
248939932,0cyclictest29026-21awk19:54:530
248939932,0cyclictest2886-21kthreadcore22:04:540
248939932,0cyclictest24951-21ps21:19:520
248939932,0cyclictest20418-21awk18:59:520
248939932,0cyclictest16774-21tr21:54:520
248939932,0cyclictest12898-21tail18:54:480
248939931,0cyclictest0-21swapper/021:24:530
248939930,0cyclictest1226-21grep22:44:530
248939930,0cyclictest0-21swapper/018:49:480
169930,0watchdog/129806-21smartctl20:19:491
169930,0watchdog/10-21swapper/123:24:371
169930,0watchdog/10-21swapper/123:04:491
169930,0watchdog/10-21swapper/123:04:491
169930,0watchdog/10-21swapper/122:29:451
339920,0migration/315704-21iostat_ios19:44:533
329920,0watchdog/328045-21python20:14:573
28405020,0irq/26-eth0-rx-0-21swapper/318:45:283
248939921,0cyclictest21778-21aten_r1power_fr23:39:470
248939920,0cyclictest28772-21cat00:04:340
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional