You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 19:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Fri Nov 21, 2025 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2815823724,0sleep00-21swapper/006:39:120
2781323526,0sleep30-21swapper/306:35:103
2798623425,0sleep10-21swapper/106:37:121
2181822622,0sleep20-21swapper/206:34:452
8225080,0irq/24-0000:00:0-21swapper/207:59:382
282649988,0cyclictest6445-21snmpd10:17:250
28415070,0irq/27-eth0-tx-0-21swapper/211:28:432
8225060,0irq/24-0000:00:11266-21latency_hist07:39:382
339960,0migration/34604-21munin-node10:30:013
339960,0migration/331344-21munin-node09:44:573
339960,0migration/328411-21munin-node06:59:573
339960,0migration/327018-21sh11:48:023
339960,0migration/32479-21sh11:31:223
339960,0migration/324330-21sh09:40:183
339960,0migration/318375-21sh11:41:063
339960,0migration/313466-21missed_timers10:14:573
28415060,0irq/27-eth0-tx-26806-21munin-node08:39:520
28415060,0irq/27-eth0-tx-11044-21cat09:14:382
28415060,0irq/27-eth0-tx-0-21swapper/212:08:542
28415060,0irq/27-eth0-tx-0-21swapper/211:55:342
28415060,0irq/27-eth0-tx-0-21swapper/211:31:092
28415060,0irq/27-eth0-tx-0-21swapper/211:14:522
28415060,0irq/27-eth0-tx-0-21swapper/211:12:502
28415060,0irq/27-eth0-tx-0-21swapper/211:06:112
28415060,0irq/27-eth0-tx-0-21swapper/210:46:032
28415060,0irq/27-eth0-tx-0-21swapper/210:44:252
28415060,0irq/27-eth0-tx-0-21swapper/210:17:552
28415060,0irq/27-eth0-tx-0-21swapper/210:07:082
28415060,0irq/27-eth0-tx-0-21swapper/209:35:412
28415060,0irq/27-eth0-tx-0-21swapper/209:26:482
28415060,0irq/27-eth0-tx-0-21swapper/209:14:092
28415060,0irq/27-eth0-tx-0-21swapper/209:04:352
28415060,0irq/27-eth0-tx-0-21swapper/208:46:332
28415060,0irq/27-eth0-tx-0-21swapper/006:51:460
28405060,0irq/26-eth0-rx-5125-21irqrtprio11:34:563
28405060,0irq/26-eth0-rx-21677-21kthreadcore08:09:553
282659960,0cyclictest26568-21diskmemload12:08:431
282649966,0cyclictest0-21swapper/009:29:580
149960,0migration/031392-21missed_timers09:44:570
149960,0migration/026529-21sh11:25:230
149960,0migration/022756-21kthreadcore08:34:590
8225050,0irq/24-0000:00:22838-21python07:20:122
8225050,0irq/24-0000:00:10225-21perf07:14:382
35150,0ktimersoftd/30-21swapper/311:54:303
339950,0migration/39339-21aten_r1power_fr09:54:483
339950,0migration/38678-21kthreadcore07:59:563
339950,0migration/38434-21ksmtuned09:52:593
339950,0migration/35409-21taskset07:57:483
339950,0migration/34915-21taskset10:53:063
339950,0migration/34618-21kthreadcore08:19:553
339950,0migration/329287-21kthreadcore09:24:543
339950,0migration/326769-21kthreadcore07:49:553
339950,0migration/325604-21kthreadcore07:24:543
339950,0migration/32397-21kthreadcore10:09:573
339950,0migration/32333-21taskset06:42:563
339950,0migration/322708-21taskset10:22:143
339950,0migration/322499-21munin-node07:20:083
339950,0migration/320830-21kthreadcore09:19:553
339950,0migration/318405-21kthreadcore09:59:573
339950,0migration/317108-21munin-node06:54:523
339950,0migration/310023-21kthreadcore10:54:553
28415050,0irq/27-eth0-tx-8669-21ssh09:30:312
28415050,0irq/27-eth0-tx-3936-21munin-node11:34:592
28415050,0irq/27-eth0-tx-32421-21ssh09:46:052
28415050,0irq/27-eth0-tx-0-21swapper/212:01:152
28415050,0irq/27-eth0-tx-0-21swapper/211:45:062
28415050,0irq/27-eth0-tx-0-21swapper/211:42:282
28415050,0irq/27-eth0-tx-0-21swapper/211:01:532
28415050,0irq/27-eth0-tx-0-21swapper/210:57:522
28415050,0irq/27-eth0-tx-0-21swapper/210:51:382
28415050,0irq/27-eth0-tx-0-21swapper/210:36:552
28415050,0irq/27-eth0-tx-0-21swapper/210:34:302
28415050,0irq/27-eth0-tx-0-21swapper/210:27:272
28415050,0irq/27-eth0-tx-0-21swapper/210:00:032
28415050,0irq/27-eth0-tx-0-21swapper/209:59:292
28415050,0irq/27-eth0-tx-0-21swapper/209:49:452
28415050,0irq/27-eth0-tx-0-21swapper/209:24:002
28415050,0irq/27-eth0-tx-0-21swapper/208:56:022
28415050,0irq/27-eth0-tx-0-21swapper/208:49:482
28415050,0irq/27-eth0-tx-0-21swapper/008:14:510
28415050,0irq/27-eth0-tx-0-21swapper/007:04:500
28405050,0irq/26-eth0-rx-27998-21irqrtprio08:39:523
28405050,0irq/26-eth0-rx-25233-21python12:04:583
28405050,0irq/26-eth0-rx-24521-21ssh08:55:593
28405050,0irq/26-eth0-rx-22696-21diskstats10:44:513
28405050,0irq/26-eth0-rx-21408-21cut11:04:503
28405050,0irq/26-eth0-rx-18480-21ssh12:02:463
28405050,0irq/26-eth0-rx-12490-21basename11:19:493
28405050,0irq/26-eth0-rx-0-21swapper/311:57:003
28405050,0irq/26-eth0-rx-0-21swapper/311:15:333
28405050,0irq/26-eth0-rx-0-21swapper/309:04:553
28405050,0irq/26-eth0-rx-0-21swapper/309:00:003
28405050,0irq/26-eth0-rx-0-21swapper/308:50:163
28405050,0irq/26-eth0-rx-0-21swapper/308:34:583
282659955,0cyclictest0-21swapper/111:09:071
282659955,0cyclictest0-21swapper/110:22:181
282659955,0cyclictest0-21swapper/110:05:161
282659950,0cyclictest0-21swapper/110:31:171
282659950,0cyclictest0-21swapper/110:13:381
282659950,0cyclictest0-21swapper/109:29:101
282659950,0cyclictest0-21swapper/108:56:221
282659950,0cyclictest0-21swapper/108:40:271
282649955,0cyclictest0-21swapper/010:36:210
282649950,0cyclictest0-21swapper/009:38:010
282649950,0cyclictest0-21swapper/008:45:490
259950,0migration/228383-21fschecks_time09:04:522
259950,0migration/22762-21switchtime07:05:022
259950,0migration/215773-21latency_hist10:19:382
259950,0migration/213318-21if_eth111:19:532
249950,0watchdog/2936-21awk08:19:532
249950,0watchdog/26459-21ssh10:10:132
249950,0watchdog/228157-21cpuspeed_turbos11:49:522
249950,0watchdog/224308-21ssh09:40:132
169950,0watchdog/123834-21cut06:59:531
169950,0watchdog/10-21swapper/110:34:561
169950,0watchdog/10-21swapper/109:50:331
169950,0watchdog/10-21swapper/109:39:571
169950,0watchdog/10-21swapper/109:01:571
169950,0watchdog/10-21swapper/107:52:491
149950,0migration/09504-21switchtime07:10:040
149950,0migration/08475-21kthreadcore10:54:540
149950,0migration/0811-21munin-node08:19:520
149950,0migration/06851-21aten_r1power_en08:24:500
149950,0migration/03951-21munin-node06:44:550
149950,0migration/032578-21munin-node09:24:590
149950,0migration/029841-21kthreadcore07:49:560
149950,0migration/027770-21kthreadcore10:04:550
149950,0migration/021716-21processes06:54:560
149950,0migration/020262-21rt-features10:39:580
149950,0migration/019294-21switchtime11:00:000
149950,0migration/017913-21latency_hist08:54:380
149950,0migration/017166-21kthreadcore11:39:560
149950,0migration/015433-21kthreadcore08:29:540
149950,0migration/010544-21aten_r1power_po07:14:480
339940,0migration/37143-21kthreadcore07:09:593
339940,0migration/35282-21kthreadcore06:44:553
339940,0migration/34807-21kthreadcore08:44:553
339940,0migration/33445-21cpuspeed_turbos09:09:513
339940,0migration/320311-21ksmtuned10:39:593
339940,0migration/313766-21kthreadcore07:39:533
28415040,0irq/27-eth0-tx-0-21swapper/008:10:010
28415040,0irq/27-eth0-tx-0-21swapper/006:59:490
28405040,0irq/26-eth0-rx-29118-21diskmemload11:09:433
28405040,0irq/26-eth0-rx-0-21swapper/311:26:143
28405040,0irq/26-eth0-rx-0-21swapper/311:00:533
28405040,0irq/26-eth0-rx-0-21swapper/310:34:543
28405040,0irq/26-eth0-rx-0-21swapper/310:24:503
28405040,0irq/26-eth0-rx-0-21swapper/310:05:493
28405040,0irq/26-eth0-rx-0-21swapper/309:34:553
28405040,0irq/26-eth0-rx-0-21swapper/309:30:333
28405040,0irq/26-eth0-rx-0-21swapper/309:16:363
28405040,0irq/26-eth0-rx-0-21swapper/307:46:043
282669943,0cyclictest6445-21snmpd08:30:582
282669943,0cyclictest6445-21snmpd07:48:282
282669943,0cyclictest32315-21awk07:54:522
282669943,0cyclictest29344-21awk06:39:542
282669943,0cyclictest26698-21awk08:14:542
282669943,0cyclictest25683-21awk07:49:542
282669943,0cyclictest21049-21awk08:34:572
282669943,0cyclictest11372-21awk07:14:512
282669940,0cyclictest0-21swapper/208:04:502
282659944,0cyclictest0-21swapper/111:37:491
282659944,0cyclictest0-21swapper/110:40:351
282659944,0cyclictest0-21swapper/109:57:331
282659944,0cyclictest0-21swapper/109:35:381
282659943,0cyclictest6754-21awk07:59:541
282659943,0cyclictest6445-21snmpd08:26:281
282659943,0cyclictest6445-21snmpd06:43:271
282659943,0cyclictest4593-21awk07:09:571
282659943,0cyclictest4079-21awk09:09:541
282659943,0cyclictest3425-21lldpd09:21:361
282659943,0cyclictest3425-21lldpd08:09:391
282659943,0cyclictest3425-21lldpd07:31:081
282659943,0cyclictest32360-21awk07:54:531
282659943,0cyclictest21047-21irqrtprio11:44:551
282659943,0cyclictest14316-21awk10:59:521
282659943,0cyclictest14311-21awk08:29:521
282659943,0cyclictest13166-21sadc10:59:381
282659940,0cyclictest0-21swapper/112:00:401
282659940,0cyclictest0-21swapper/111:52:381
282649944,0cyclictest0-21swapper/011:53:550
282649944,0cyclictest0-21swapper/011:10:140
282649944,0cyclictest0-21swapper/010:46:350
282649944,0cyclictest0-21swapper/010:25:090
282649944,0cyclictest0-21swapper/007:20:130
282649943,0cyclictest6700-21awk07:59:530
282649943,0cyclictest32385-21awk07:54:530
282649943,0cyclictest32303-21awk10:29:560
282649943,0cyclictest28874-21sadc11:09:380
282649943,0cyclictest28654-21awk09:04:540
282649943,0cyclictest20437-21awk12:04:520
282649943,0cyclictest19838-21awk09:19:530
282649943,0cyclictest13426-21awk08:04:530
282649943,0cyclictest12455-21awk07:39:520
282649943,0cyclictest12080-21awk09:14:530
282649940,0cyclictest18156-21munin-node07:44:540
259940,0migration/25214-21kthreadcore06:44:552
259940,0migration/231788-21kthreadcore08:39:532
259940,0migration/22143-21kthreadcore07:29:562
259940,0migration/215509-21runrttasks06:50:272
259940,0migration/215427-21kthreadcore07:39:542
249940,0watchdog/20-21swapper/207:24:492
249940,0watchdog/20-21swapper/206:54:532
179940,0migration/132434-21latency_hist08:19:381
179940,0migration/124350-21fschecks_time10:24:521
179940,0migration/123173-21munin-node10:44:531
179940,0migration/118092-21munin-node07:20:051
169940,0watchdog/16547-21runrttasks09:30:011
169940,0watchdog/16547-21runrttasks08:45:411
169940,0watchdog/129028-21smartctl11:29:521
169940,0watchdog/123932-21aten_r1power_vo07:24:491
169940,0watchdog/118386-21unixbench-2d11:20:001
169940,0watchdog/117069-21unixbench-2d09:15:011
169940,0watchdog/11211-21grep07:04:571
169940,0watchdog/10-21swapper/111:55:001
169940,0watchdog/10-21swapper/111:40:281
169940,0watchdog/10-21swapper/111:11:361
169940,0watchdog/10-21swapper/110:52:521
169940,0watchdog/10-21swapper/110:14:531
169940,0watchdog/10-21swapper/110:02:291
169940,0watchdog/10-21swapper/109:49:051
169940,0watchdog/10-21swapper/108:49:411
169940,0watchdog/10-21swapper/108:14:571
169940,0watchdog/10-21swapper/107:14:531
169940,0watchdog/10-21swapper/106:57:051
149940,0migration/08072-21kthreadcore07:34:530
149940,0migration/05107-21kthreadcore09:09:560
149940,0migration/020529-21kthreadcore09:39:540
149940,0migration/018389-21kthreadcore09:59:570
149940,0migration/011333-21runrttasks11:17:140
4130,0ktimersoftd/026847-21sh11:47:320
28405030,0irq/26-eth0-rx-0-21swapper/308:29:493
28405030,0irq/26-eth0-rx-0-21swapper/308:04:503
28405030,0irq/26-eth0-rx-0-21swapper/307:37:293
28405030,0irq/26-eth0-rx-0-21swapper/307:17:493
28405030,0irq/26-eth0-rx-0-21swapper/307:04:503
28405030,0irq/26-eth0-rx-0-21swapper/306:49:483
282669932,0cyclictest6445-21snmpd08:25:132
282669932,0cyclictest24835-21sensors08:09:592
282669932,0cyclictest22536-21latency_hist06:59:382
282659932,0cyclictest20436-21cpuspeed_turbos08:34:541
282659930,0cyclictest18481-21df07:44:521
282649933,0cyclictest0-21swapper/011:19:510
282649932,0cyclictest6445-21snmpd11:38:440
282649932,0cyclictest6445-21snmpd06:44:120
282649932,0cyclictest31178-21grep10:49:570
282649932,0cyclictest24781-21awk07:24:520
282649930,0cyclictest15843-21grep12:00:000
282649930,0cyclictest0-21swapper/011:54:490
179930,0migration/15487-21fschecks_time07:34:501
169930,0watchdog/10-21swapper/111:24:441
169930,0watchdog/10-21swapper/111:14:401
169930,0watchdog/10-21swapper/109:04:451
169930,0watchdog/10-21swapper/108:04:531
169930,0watchdog/10-21swapper/107:39:411
169930,0watchdog/10-21swapper/106:49:411
169930,0watchdog/10-21swapper/106:44:491
149930,0migration/07554-21sh09:50:080
149930,0migration/022552-21sh10:21:460
35120,0ktimersoftd/30-21swapper/308:14:583
28405020,0irq/26-eth0-rx-0-21swapper/308:24:523
28405020,0irq/26-eth0-rx-0-21swapper/307:29:503
282649921,0cyclictest30412-21date07:29:380
282649921,0cyclictest26568-21diskmemload09:54:380
282649921,0cyclictest10168-21ssh08:49:400
282649921,0cyclictest0-21swapper/010:09:450
282649920,0cyclictest3-21ksoftirqd/008:59:380
282649920,0cyclictest0-21swapper/011:29:500
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional