You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 05:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Mon Nov 24, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3050829590,0sleep00-21swapper/018:38:450
2962327874,0sleep10-21swapper/118:34:531
3041423632,0sleep20-21swapper/218:37:392
3029723627,0sleep30-21swapper/318:36:203
30650991313,0cyclictest18065-21sensors23:09:552
35160,0ktimersoftd/30-21swapper/323:23:553
339960,0migration/35280-21fschecks_time20:44:463
339960,0migration/33877-21sh21:21:343
339960,0migration/327064-21sh22:15:083
339960,0migration/320171-21sh22:13:163
339960,0migration/313929-21sh20:49:493
339960,0migration/312235-21sh21:48:003
339960,0migration/312235-21sh21:48:003
339960,0migration/311251-21sh23:27:513
339960,0migration/310172-21sh00:07:173
306499965,0cyclictest30632-21kworker/1:122:01:261
306499965,0cyclictest30632-21kworker/1:121:48:041
306499965,0cyclictest30632-21kworker/1:121:48:041
306499965,0cyclictest30632-21kworker/1:121:38:321
306499965,0cyclictest30632-21kworker/1:121:10:361
306499960,0cyclictest0-21swapper/123:40:421
28415060,0irq/27-eth0-tx-13254-21munin-node21:29:572
28415060,0irq/27-eth0-tx-10685-21sed20:44:562
28415060,0irq/27-eth0-tx-0-21swapper/223:58:442
28415060,0irq/27-eth0-tx-0-21swapper/223:52:292
28415060,0irq/27-eth0-tx-0-21swapper/223:49:012
28415060,0irq/27-eth0-tx-0-21swapper/223:24:392
28415060,0irq/27-eth0-tx-0-21swapper/223:21:362
28415060,0irq/27-eth0-tx-0-21swapper/223:15:132
28415060,0irq/27-eth0-tx-0-21swapper/223:02:592
28415060,0irq/27-eth0-tx-0-21swapper/222:34:562
28415060,0irq/27-eth0-tx-0-21swapper/222:25:082
28415060,0irq/27-eth0-tx-0-21swapper/222:13:562
28415060,0irq/27-eth0-tx-0-21swapper/222:02:362
28415060,0irq/27-eth0-tx-0-21swapper/221:52:262
28415060,0irq/27-eth0-tx-0-21swapper/221:39:452
28415060,0irq/27-eth0-tx-0-21swapper/221:17:542
28415060,0irq/27-eth0-tx-0-21swapper/221:02:372
28415060,0irq/27-eth0-tx-0-21swapper/200:09:302
28415060,0irq/27-eth0-tx-0-21swapper/020:39:500
28415060,0irq/27-eth0-tx-0-21swapper/019:24:450
28405060,0irq/26-eth0-rx-19223-21head18:54:483
28405060,0irq/26-eth0-rx-14905-21munin-node20:04:463
28405060,0irq/26-eth0-rx-12782-21python19:34:553
249960,0watchdog/230801-21cut22:39:512
249960,0watchdog/226041-21munin-node19:44:552
169960,0watchdog/111534-21ssh20:46:511
8225050,0irq/24-0000:00:6906-21latency_hist19:34:332
8225050,0irq/24-0000:00:20838-21latency_hist21:54:332
8225050,0irq/24-0000:00:20185-21latency_hist22:54:322
8225050,0irq/24-0000:00:14739-21munin-node20:04:532
8225050,0irq/24-0000:00:13712-21cat19:39:332
4150,0ktimersoftd/00-21swapper/020:51:330
339950,0migration/38017-21kthreadcore21:04:513
339950,0migration/37999-21kthreadcore18:44:493
339950,0migration/33917-21munin-node18:39:543
339950,0migration/329141-21aten_r1power_en21:59:433
339950,0migration/328371-21munin-node19:49:513
339950,0migration/328322-21aten_r1power_en23:39:483
339950,0migration/328009-21smart_sdb20:34:553
339950,0migration/322674-21kernelversion20:09:493
339950,0migration/322126-21fschecks_time21:14:463
339950,0migration/321337-21fschecks_time22:34:463
339950,0migration/320450-21munin-node19:44:463
339950,0migration/320297-21taskset23:14:393
339950,0migration/318729-21runrttasks23:31:053
339950,0migration/31864-21taskset00:01:093
339950,0migration/318037-21munin-node23:49:563
339950,0migration/317870-21diskmemload21:09:503
339950,0migration/317867-21kthreadcore22:29:503
339950,0migration/317202-21kthreadcore19:14:523
339950,0migration/316171-21munin-node20:29:483
339950,0migration/313557-21munin-node19:59:523
306499955,0cyclictest5352-21kworker/1:223:54:341
306499955,0cyclictest5352-21kworker/1:223:49:381
306499955,0cyclictest5352-21kworker/1:223:44:461
306499955,0cyclictest5352-21kworker/1:223:35:141
306499955,0cyclictest5352-21kworker/1:223:25:041
306499955,0cyclictest5352-21kworker/1:223:19:541
306499955,0cyclictest5352-21kworker/1:223:14:461
306499955,0cyclictest5352-21kworker/1:223:09:441
306499955,0cyclictest5352-21kworker/1:223:04:321
306499955,0cyclictest5352-21kworker/1:222:59:441
306499955,0cyclictest5352-21kworker/1:222:49:481
306499955,0cyclictest5352-21kworker/1:222:45:021
306499955,0cyclictest5352-21kworker/1:222:40:021
306499955,0cyclictest5352-21kworker/1:222:29:541
306499955,0cyclictest5352-21kworker/1:222:24:501
306499955,0cyclictest5352-21kworker/1:200:02:501
306499955,0cyclictest30632-21kworker/1:122:14:401
306499955,0cyclictest30632-21kworker/1:122:09:401
306499955,0cyclictest30632-21kworker/1:122:05:001
306499955,0cyclictest30632-21kworker/1:121:58:221
306499955,0cyclictest30632-21kworker/1:121:49:561
306499955,0cyclictest30632-21kworker/1:121:40:001
306499955,0cyclictest30632-21kworker/1:121:31:401
306499955,0cyclictest30632-21kworker/1:121:24:421
306499955,0cyclictest30632-21kworker/1:121:19:341
306499955,0cyclictest30632-21kworker/1:121:14:461
306499955,0cyclictest30632-21kworker/1:121:04:561
306499955,0cyclictest30632-21kworker/1:120:59:361
306499955,0cyclictest30632-21kworker/1:120:54:461
306499955,0cyclictest30632-21kworker/1:120:51:401
306499955,0cyclictest30632-21kworker/1:120:39:361
306499955,0cyclictest30632-21kworker/1:120:29:401
306499955,0cyclictest30632-21kworker/1:120:24:461
306499955,0cyclictest30632-21kworker/1:120:14:521
306499955,0cyclictest30632-21kworker/1:120:09:441
306499955,0cyclictest30632-21kworker/1:119:54:481
306499955,0cyclictest30632-21kworker/1:119:44:541
306499955,0cyclictest30632-21kworker/1:119:26:331
306499955,0cyclictest30632-21kworker/1:119:23:471
306499955,0cyclictest30632-21kworker/1:119:04:551
306499955,0cyclictest30632-21kworker/1:119:02:071
306499955,0cyclictest30632-21kworker/1:118:57:571
306499954,0cyclictest5352-21kworker/1:222:55:421
306499954,0cyclictest5352-21kworker/1:222:36:561
306499954,0cyclictest30632-21kworker/1:119:34:511
306489955,0cyclictest0-21swapper/023:01:280
306489955,0cyclictest0-21swapper/021:29:430
306489954,0cyclictest8189-21kthreadcore00:04:510
306489950,0cyclictest0-21swapper/023:47:490
306489950,0cyclictest0-21swapper/022:37:400
306489950,0cyclictest0-21swapper/021:56:120
28415050,0irq/27-eth0-tx-8653-40yum-updatesd-he20:22:590
28415050,0irq/27-eth0-tx-5228-21munin-node21:04:472
28415050,0irq/27-eth0-tx-21475-21cpu21:34:432
28415050,0irq/27-eth0-tx-19232-21ssh22:31:142
28415050,0irq/27-eth0-tx-14739-21munin-node20:04:500
28415050,0irq/27-eth0-tx-11694-21ssh21:46:272
28415050,0irq/27-eth0-tx-11694-21ssh21:46:272
28415050,0irq/27-eth0-tx-10340-21netstat22:04:532
28415050,0irq/27-eth0-tx-0-21swapper/223:40:062
28415050,0irq/27-eth0-tx-0-21swapper/223:36:162
28415050,0irq/27-eth0-tx-0-21swapper/223:34:122
28415050,0irq/27-eth0-tx-0-21swapper/223:06:152
28415050,0irq/27-eth0-tx-0-21swapper/222:44:422
28415050,0irq/27-eth0-tx-0-21swapper/222:19:482
28415050,0irq/27-eth0-tx-0-21swapper/222:14:372
28415050,0irq/27-eth0-tx-0-21swapper/221:25:172
28415050,0irq/27-eth0-tx-0-21swapper/221:19:552
28415050,0irq/27-eth0-tx-0-21swapper/221:10:542
28415050,0irq/27-eth0-tx-0-21swapper/220:54:572
28415050,0irq/27-eth0-tx-0-21swapper/220:51:342
28415050,0irq/27-eth0-tx-0-21swapper/220:40:482
28415050,0irq/27-eth0-tx-0-21swapper/200:00:242
28405050,0irq/26-eth0-rx-8352-21kthreadcore21:24:513
28405050,0irq/26-eth0-rx-6547-21runrttasks23:10:013
28405050,0irq/26-eth0-rx-32465-21kthreadcore21:39:513
28405050,0irq/26-eth0-rx-32044-21cut22:19:503
28405050,0irq/26-eth0-rx-21359-21hostname21:34:423
28405050,0irq/26-eth0-rx-21087-21fschecks_time22:54:473
28405050,0irq/26-eth0-rx-10912-21ssh23:06:283
28405050,0irq/26-eth0-rx-0-21swapper/323:54:523
28405050,0irq/26-eth0-rx-0-21swapper/323:47:563
28405050,0irq/26-eth0-rx-0-21swapper/323:34:523
28405050,0irq/26-eth0-rx-0-21swapper/323:02:373
28405050,0irq/26-eth0-rx-0-21swapper/322:51:493
28405050,0irq/26-eth0-rx-0-21swapper/322:49:153
28405050,0irq/26-eth0-rx-0-21swapper/322:24:503
28405050,0irq/26-eth0-rx-0-21swapper/321:50:423
28405050,0irq/26-eth0-rx-0-21swapper/320:54:473
28405050,0irq/26-eth0-rx-0-21swapper/320:39:503
28405050,0irq/26-eth0-rx-0-21swapper/319:54:443
259950,0migration/25057-21latency_hist18:44:332
259950,0migration/228852-21kthreadcore18:59:512
259950,0migration/218368-21latency_hist18:54:332
249950,0watchdog/24022-21port_sshpriv18:39:552
179950,0migration/15555-21kthreadcore00:04:501
179950,0migration/15540-21munin-node18:44:441
179950,0migration/129880-21munin-node22:19:491
149950,0migration/09615-21kthreadcore20:44:510
149950,0migration/03650-21kthreadcore19:04:490
149950,0migration/032134-21kthreadcore21:39:510
149950,0migration/027845-21sh22:17:240
149950,0migration/027248-21timerwakeupswit21:34:540
149950,0migration/026306-21kthreadcore21:14:510
149950,0migration/022817-21fschecks_count20:34:470
149950,0migration/019981-21taskset21:11:050
149950,0migration/019707-21latency_hist23:54:320
149950,0migration/017755-21kthreadcore21:49:510
149950,0migration/016737-21kthreadcore22:29:490
149950,0migration/013866-21kthreadcore22:49:520
149950,0migration/011051-21munin-node21:04:560
339940,0migration/36850-21kthreadcore22:04:513
339940,0migration/326518-21kthreadcore18:59:503
339940,0migration/324603-21kthreadcore19:19:513
339940,0migration/32053-21kthreadcore19:04:493
306509943,0cyclictest8969-21awk19:59:472
306509943,0cyclictest6445-21snmpd19:25:302
306509943,0cyclictest3425-21lldpd20:18:152
306509943,0cyclictest3197-21awk20:19:482
306499944,0cyclictest5352-21kworker/1:223:30:401
306499944,0cyclictest30632-21kworker/1:120:34:461
306499944,0cyclictest30632-21kworker/1:120:19:461
306499944,0cyclictest30632-21kworker/1:120:04:461
306499944,0cyclictest30632-21kworker/1:119:40:211
306499944,0cyclictest30632-21kworker/1:119:29:471
306499944,0cyclictest30632-21kworker/1:119:14:451
306499944,0cyclictest30632-21kworker/1:119:09:511
306499944,0cyclictest30632-21kworker/1:118:49:491
306499944,0cyclictest30632-21kworker/1:118:39:411
306499943,0cyclictest8938-21awk19:59:461
306499943,0cyclictest3110-21irqbalance19:49:471
306489944,0cyclictest0-21swapper/022:28:210
306489943,0cyclictest7086-21awk19:09:500
306489943,0cyclictest6445-21snmpd23:37:310
306489943,0cyclictest6445-21snmpd23:09:460
306489943,0cyclictest6445-21snmpd22:56:450
306489943,0cyclictest6445-21snmpd21:27:450
306489943,0cyclictest6445-21snmpd20:10:150
306489943,0cyclictest5161-21awk23:04:470
306489943,0cyclictest3425-21lldpd19:41:160
306489943,0cyclictest3110-21irqbalance23:27:270
306489943,0cyclictest3110-21irqbalance23:16:270
306489943,0cyclictest30389-21awk21:19:480
306489943,0cyclictest30028-21awk21:59:470
306489943,0cyclictest2341-21awk19:54:480
306489943,0cyclictest12857-21awk23:49:470
306489943,0cyclictest1233-21awk19:29:480
306489940,0cyclictest0-21swapper/019:49:540
28415040,0irq/27-eth0-tx-18514-21munin-node18:54:460
28415040,0irq/27-eth0-tx-0-21swapper/222:50:062
28415040,0irq/27-eth0-tx-0-21swapper/020:14:490
28415040,0irq/27-eth0-tx-0-21swapper/019:14:450
28405040,0irq/26-eth0-rx-20287-21ssh21:32:273
28405040,0irq/26-eth0-rx-0-21swapper/322:40:043
28405040,0irq/26-eth0-rx-0-21swapper/321:54:553
28405040,0irq/26-eth0-rx-0-21swapper/320:59:583
28405040,0irq/26-eth0-rx-0-21swapper/318:49:523
259940,0migration/2994-21kthreadcore19:04:482
259940,0migration/28613-21kthreadcore19:09:522
259940,0migration/27602-21taskset19:56:452
259940,0migration/227780-21fschecks_time19:49:482
259940,0migration/227281-21kthreadcore20:34:512
259940,0migration/221591-21taskset20:09:452
259940,0migration/218026-21kthreadcore19:14:522
259940,0migration/213180-21ksmtuned19:35:322
249940,0watchdog/20-21swapper/218:54:512
149940,0migration/07388-21kthreadcore18:44:480
149940,0migration/04542-21aten2.4-expect22:44:430
149940,0migration/04166-21latency_hist22:44:320
149940,0migration/032132-21kthreadcore23:19:510
149940,0migration/031470-21runrttasks19:03:090
149940,0migration/027418-21sh20:55:170
149940,0migration/021035-21fschecks_time19:44:480
149940,0migration/013643-21sh22:09:460
149940,0migration/013457-21kthreadcore18:49:510
149940,0migration/01336-21kthreadcore22:19:510
149940,0migration/010454-21kthreadcore20:24:480
306509932,0cyclictest19943-21sort19:19:462
306509932,0cyclictest10010-21awk20:24:472
306509930,0cyclictest0-21swapper/220:32:492
306489932,0cyclictest31821-21awk18:39:500
306489932,0cyclictest30281-21awk20:59:470
306489932,0cyclictest1120-21sensors23:59:560
306489930,0cyclictest19857-21diskstats19:19:460
306489930,0cyclictest12550-21df_inode23:29:450
306489930,0cyclictest0-21swapper/019:35:060
28415030,0irq/27-eth0-tx-0-21swapper/020:29:460
28405030,0irq/26-eth0-rx-0-21swapper/320:19:543
28405030,0irq/26-eth0-rx-0-21swapper/320:18:553
28405030,0irq/26-eth0-rx-0-21swapper/319:24:473
28405030,0irq/26-eth0-rx-0-21swapper/319:09:463
149930,0migration/028225-21diskmemload23:39:400
149930,0migration/010915-21kthreadcore19:59:490
306489921,0cyclictest0-21swapper/021:44:330
306489921,0cyclictest0-21swapper/021:44:330
306489920,0cyclictest0-21swapper/022:04:380
28405020,0irq/26-eth0-rx-0-21swapper/320:24:473
28405020,0irq/26-eth0-rx-0-21swapper/319:41:313
28405020,0irq/26-eth0-rx-0-21swapper/319:29:373
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional