You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-10 - 10:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2s.osadl.org (updated Thu Jul 10, 2025 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350720,0irq/25-eth10-21swapper/318:40:183
2085125036,0sleep20-21swapper/218:44:292
2082224935,0sleep00-21swapper/018:44:070
1459123332,0sleep10-21swapper/118:39:511
20935991111,0cyclictest14362-21sensors22:59:590
209369988,0cyclictest6445-21snmpd23:31:141
339970,0migration/332071-21sh22:52:503
28415070,0irq/27-eth0-tx-22873-21munin-node00:09:572
209379970,0cyclictest25340-21runrttasks21:47:102
339960,0migration/37838-21sh22:36:423
339960,0migration/310478-21sh21:18:353
28415060,0irq/27-eth0-tx-16954-21ssh21:40:522
28415060,0irq/27-eth0-tx-16399-21munin-node19:29:550
28415060,0irq/27-eth0-tx-0-21swapper/223:23:312
28415060,0irq/27-eth0-tx-0-21swapper/223:07:452
28415060,0irq/27-eth0-tx-0-21swapper/223:00:522
28415060,0irq/27-eth0-tx-0-21swapper/222:46:052
28415060,0irq/27-eth0-tx-0-21swapper/222:41:102
28415060,0irq/27-eth0-tx-0-21swapper/222:25:412
28415060,0irq/27-eth0-tx-0-21swapper/222:19:502
28415060,0irq/27-eth0-tx-0-21swapper/222:16:332
28415060,0irq/27-eth0-tx-0-21swapper/221:57:322
28415060,0irq/27-eth0-tx-0-21swapper/221:21:422
28415060,0irq/27-eth0-tx-0-21swapper/221:03:242
28415060,0irq/27-eth0-tx-0-21swapper/220:55:552
28415060,0irq/27-eth0-tx-0-21swapper/220:47:092
28415060,0irq/27-eth0-tx-0-21swapper/200:05:332
28415060,0irq/27-eth0-tx-0-21swapper/200:02:552
28415060,0irq/27-eth0-tx-0-21swapper/020:23:370
28415060,0irq/27-eth0-tx-0-21swapper/019:24:580
28405060,0irq/26-eth0-rx-26215-21perl22:29:553
249960,0watchdog/20-21swapper/222:50:052
249960,0watchdog/20-21swapper/221:06:492
209379960,0cyclictest0-21swapper/222:06:062
209379960,0cyclictest0-21swapper/221:30:282
209379960,0cyclictest0-21swapper/221:30:282
149960,0migration/032137-21sh22:10:070
149960,0migration/021584-21missed_timers23:24:550
149960,0migration/017923-21sh21:43:460
339950,0migration/39820-21munin-node19:24:503
339950,0migration/38661-21idleruntime23:39:533
339950,0migration/37460-21kthreadcore19:44:553
339950,0migration/36359-21munin-node20:34:543
339950,0migration/330658-21kthreadcore20:49:573
339950,0migration/327777-21kthreadcore21:49:563
339950,0migration/326136-21taskset21:27:403
339950,0migration/325507-21fschecks_count20:24:513
339950,0migration/324458-21kthreadcore00:09:583
339950,0migration/324016-21sh22:26:043
339950,0migration/32333-21runrttasks21:13:173
339950,0migration/322982-21sshd23:27:023
339950,0migration/322491-21kthreadcore18:44:553
339950,0migration/320483-21munin-node00:04:593
339950,0migration/32021-21ksmtuned18:54:473
339950,0migration/32001-21kthreadcore23:34:553
339950,0migration/31784-21munin-node22:14:523
339950,0migration/317735-21kthreadcore20:39:573
339950,0migration/317416-21munin-node23:04:533
339950,0migration/317211-21idleruntime19:29:543
339950,0migration/31527-21munin-node22:54:543
339950,0migration/314575-21timerandwakeup19:00:033
339950,0migration/314122-21diskmemload00:02:583
28415050,0irq/27-eth0-tx-31984-21nfs_client21:49:592
28415050,0irq/27-eth0-tx-22552-21ssh23:25:442
28415050,0irq/27-eth0-tx-19239-21telnet21:24:512
28415050,0irq/27-eth0-tx-19116-21diskmemload22:11:222
28415050,0irq/27-eth0-tx-0-21swapper/223:56:452
28415050,0irq/27-eth0-tx-0-21swapper/223:50:112
28415050,0irq/27-eth0-tx-0-21swapper/223:47:082
28415050,0irq/27-eth0-tx-0-21swapper/223:41:052
28415050,0irq/27-eth0-tx-0-21swapper/223:34:402
28415050,0irq/27-eth0-tx-0-21swapper/223:31:572
28415050,0irq/27-eth0-tx-0-21swapper/221:59:452
28415050,0irq/27-eth0-tx-0-21swapper/221:37:542
28415050,0irq/27-eth0-tx-0-21swapper/020:36:030
28415050,0irq/27-eth0-tx-0-21swapper/020:15:370
28415050,0irq/27-eth0-tx-0-21swapper/020:09:580
28415050,0irq/27-eth0-tx-0-21swapper/020:08:070
28415050,0irq/27-eth0-tx-0-21swapper/019:05:020
28405050,0irq/26-eth0-rx-8714-21open_inodes20:54:593
28405050,0irq/26-eth0-rx-30406-21ssh23:30:253
28405050,0irq/26-eth0-rx-27305-21kthreadcore22:09:553
28405050,0irq/26-eth0-rx-19466-21kthreadcore23:44:573
28405050,0irq/26-eth0-rx-17528-21grep22:44:503
28405050,0irq/26-eth0-rx-17229-21ssh21:41:423
28405050,0irq/26-eth0-rx-15525-21ssh22:40:563
28405050,0irq/26-eth0-rx-13013-21kthreadcore22:59:553
28405050,0irq/26-eth0-rx-0-21swapper/323:49:563
28405050,0irq/26-eth0-rx-0-21swapper/323:15:153
28405050,0irq/26-eth0-rx-0-21swapper/322:24:013
28405050,0irq/26-eth0-rx-0-21swapper/321:47:233
28405050,0irq/26-eth0-rx-0-21swapper/321:38:343
28405050,0irq/26-eth0-rx-0-21swapper/321:19:563
28405050,0irq/26-eth0-rx-0-21swapper/321:07:373
28405050,0irq/26-eth0-rx-0-21swapper/321:00:313
28405050,0irq/26-eth0-rx-0-21swapper/320:44:543
259950,0migration/29683-21latency_hist19:24:392
259950,0migration/230658-21kthreadcore22:29:592
259950,0migration/230082-21vmstat20:00:042
259950,0migration/21484-21kthreadcore23:14:562
249950,0watchdog/20-21swapper/223:11:492
249950,0watchdog/20-21swapper/221:10:452
249950,0watchdog/20-21swapper/220:49:572
249950,0watchdog/20-21swapper/220:05:132
249950,0watchdog/20-21swapper/219:54:572
249950,0watchdog/20-21swapper/219:39:572
249950,0watchdog/20-21swapper/219:34:492
249950,0watchdog/20-21swapper/219:29:452
249950,0watchdog/20-21swapper/219:04:452
249950,0watchdog/20-21swapper/218:51:092
209379950,0cyclictest0-21swapper/221:16:542
209369950,0cyclictest0-21swapper/122:00:541
209369950,0cyclictest0-21swapper/121:29:221
209369950,0cyclictest0-21swapper/121:22:381
209369950,0cyclictest0-21swapper/120:49:101
209369950,0cyclictest0-21swapper/119:50:031
209369950,0cyclictest0-21swapper/100:02:271
209359955,0cyclictest0-21swapper/023:18:130
209359955,0cyclictest0-21swapper/020:59:000
209359950,0cyclictest0-21swapper/021:58:170
209359950,0cyclictest0-21swapper/000:13:140
19150,0ktimersoftd/10-21swapper/123:09:391
179950,0migration/1784-21munin-node20:50:021
179950,0migration/1765-21fschecks_time23:14:531
179950,0migration/15785-21latency_hist20:34:391
179950,0migration/129726-21munin-node20:00:021
179950,0migration/128655-21kthreadcore22:29:591
179950,0migration/124434-21kthreadcore19:34:561
179950,0migration/121845-21munin-node23:45:031
179950,0migration/119503-21kthreadcore23:04:561
179950,0migration/119067-21kthreadcore19:54:581
149950,0migration/07516-21switchtime22:15:030
149950,0migration/05762-21taskset23:56:300
149950,0migration/03482-21idleruntime21:34:570
149950,0migration/029794-21kthreadcore20:24:560
149950,0migration/02959-21munin-node19:40:000
149950,0migration/028732-21kthreadcore22:29:590
149950,0migration/027525-21kthreadcore22:50:000
149950,0migration/027370-21cpuspeed_turbos21:29:540
149950,0migration/027370-21cpuspeed_turbos21:29:540
149950,0migration/024930-21sh21:45:580
149950,0migration/024147-21aten_r1power_cu23:29:500
149950,0migration/023813-21kthreadcore19:09:580
149950,0migration/021782-21kthreadcore21:04:590
149950,0migration/020104-21kthreadcore22:24:570
149950,0migration/018083-21kthreadcore23:04:550
149950,0migration/017632-21taskset21:00:460
149950,0migration/016745-21kthreadcore00:04:570
149950,0migration/016441-21kthreadcore21:19:570
149950,0migration/015537-21kthreadcore19:49:580
149950,0migration/014411-21kthreadcore22:39:580
149950,0migration/010160-21kthreadcore23:19:590
8225040,0irq/24-0000:00:0-21swapper/219:10:032
339940,0migration/3893-21diskmemload21:30:173
339940,0migration/3893-21diskmemload21:30:173
339940,0migration/35029-21kthreadcore21:54:573
339940,0migration/331940-21kthreadcore18:49:573
339940,0migration/326461-21kthreadcore19:59:573
339940,0migration/318441-21irqcore19:54:573
339940,0migration/316980-21kthreadcore19:04:583
339940,0migration/310926-21kthreadcore23:19:593
28415040,0irq/27-eth0-tx-0-21swapper/222:54:382
28415040,0irq/27-eth0-tx-0-21swapper/020:44:570
28415040,0irq/27-eth0-tx-0-21swapper/020:39:590
28415040,0irq/27-eth0-tx-0-21swapper/019:59:540
28415040,0irq/27-eth0-tx-0-21swapper/019:20:040
28415040,0irq/27-eth0-tx-0-21swapper/018:54:580
28405040,0irq/26-eth0-rx-0-21swapper/323:57:313
28405040,0irq/26-eth0-rx-0-21swapper/323:09:473
28405040,0irq/26-eth0-rx-0-21swapper/322:05:103
28405040,0irq/26-eth0-rx-0-21swapper/322:02:523
28405040,0irq/26-eth0-rx-0-21swapper/320:12:333
28405040,0irq/26-eth0-rx-0-21swapper/320:08:483
28405040,0irq/26-eth0-rx-0-21swapper/319:39:513
259940,0migration/26014-21kthreadcore19:19:572
259940,0migration/24174-21aten_r1power_vo19:44:502
259940,0migration/221158-21aten_r1power_cu18:44:502
259940,0migration/211150-21kthreadcore18:59:572
249940,0watchdog/29868-21grep20:09:572
249940,0watchdog/21030-21munin-node22:34:532
249940,0watchdog/20-21swapper/220:19:092
249940,0watchdog/20-21swapper/219:14:532
249940,0watchdog/20-21swapper/218:57:052
209379943,0cyclictest6445-21snmpd20:34:042
209379943,0cyclictest3425-21lldpd20:35:072
209379943,0cyclictest13532-21awk20:39:542
209379943,0cyclictest11658-21awk19:49:562
209369944,0cyclictest0-21swapper/120:55:001
209369943,0cyclictest6445-21snmpd20:40:041
209369943,0cyclictest6445-21snmpd18:56:041
209369943,0cyclictest3961-21awk19:19:551
209369943,0cyclictest26835-21awk22:09:531
209369943,0cyclictest2393-21awk22:14:541
209369943,0cyclictest10724-21awk19:24:541
209369943,0cyclictest10181-21kworker/1:119:29:541
209369943,0cyclictest10181-21kworker/1:100:08:321
209369942,0cyclictest10181-21kworker/1:122:35:301
209369940,0cyclictest0-21swapper/123:21:091
209369940,0cyclictest0-21swapper/121:40:431
209359943,0cyclictest8783-21awk23:39:540
209359943,0cyclictest575-21awk23:34:540
209359943,0cyclictest28730-21munin-node19:14:490
209359943,0cyclictest23543-21sort19:34:530
209359940,0cyclictest0-21swapper/022:08:320
179940,0migration/19683-21kthreadcore23:39:551
179940,0migration/15290-21kthreadcore21:14:561
179940,0migration/129209-21kthreadcore18:49:561
179940,0migration/129189-21kthreadcore21:49:561
179940,0migration/126475-21missed_timers18:44:571
179940,0migration/125974-21if_eth022:49:581
179940,0migration/125753-21kthreadcore00:09:581
179940,0migration/125241-21cpuspeed_turbos20:24:501
179940,0migration/123267-21kthreadcore20:19:561
179940,0migration/122198-21kthreadcore22:44:551
179940,0migration/121504-21kthreadcore21:04:591
179940,0migration/120164-21kthreadcore23:24:541
179940,0migration/11508-21kthreadcore20:04:551
179940,0migration/11154-21kthreadcore23:34:551
179940,0migration/111405-21kthreadcore22:39:571
179940,0migration/11010-21fschecks_count22:54:511
169940,0watchdog/19655-21iostat22:59:531
169940,0watchdog/19095-21ssh21:36:011
169940,0watchdog/16547-21runrttasks23:51:051
169940,0watchdog/16547-21runrttasks21:01:171
169940,0watchdog/13784-21kthreadcore23:54:571
169940,0watchdog/130529-21if_err_eth019:39:531
169940,0watchdog/128525-21irqcore21:09:571
169940,0watchdog/122207-21aten_r1power_po19:09:531
169940,0watchdog/119116-21diskmemload21:29:531
169940,0watchdog/119116-21diskmemload21:29:531
169940,0watchdog/110221-21perl22:19:531
169940,0watchdog/110095-21kthreadcore18:59:571
169940,0watchdog/10-21swapper/122:24:531
169940,0watchdog/10-21swapper/122:04:571
169940,0watchdog/10-21swapper/121:44:531
169940,0watchdog/10-21swapper/119:14:531
169940,0watchdog/10-21swapper/119:04:531
149940,0migration/07877-21kthreadcore21:14:560
149940,0migration/05452-21kthreadcore22:54:560
149940,0migration/04840-21kthreadcore22:34:560
149940,0migration/0354-21kthreadcore18:49:570
149940,0migration/031924-21kthreadcore21:09:590
149940,0migration/029184-21kthreadcore21:49:560
149940,0migration/020224-21kthreadcore23:44:580
149940,0migration/016802-21sh22:01:520
149940,0migration/012165-21kthreadcore23:59:570
149940,0migration/010933-21kthreadcore18:59:570
339930,0migration/329238-21runrttasks19:37:023
339930,0migration/327450-21needreboot19:10:003
339930,0migration/31147-21kthreadcore20:29:563
28415030,0irq/27-eth0-tx-0-21swapper/020:29:510
28405030,0irq/26-eth0-rx-0-21swapper/319:54:013
28405030,0irq/26-eth0-rx-0-21swapper/319:19:513
28405030,0irq/26-eth0-rx-0-21swapper/319:17:523
249930,0watchdog/20-21swapper/220:24:572
209379930,0cyclictest0-21swapper/220:21:342
209369933,0cyclictest3006-21awk21:54:551
209369933,0cyclictest10181-21kworker/1:120:34:581
209369933,0cyclictest10181-21kworker/1:120:14:401
209369933,0cyclictest10181-21kworker/1:120:09:541
209369933,0cyclictest10181-21kworker/1:119:44:401
209359933,0cyclictest6445-21snmpd23:51:350
209359932,0cyclictest6445-21snmpd22:48:200
209359932,0cyclictest6445-21snmpd21:27:190
209359932,0cyclictest10461-21awk22:19:540
209359930,0cyclictest22151-21iostat18:44:540
149930,0migration/031486-21runrttasks23:12:260
329920,0watchdog/30-21swapper/320:19:533
28405020,0irq/26-eth0-rx-0-21swapper/320:16:313
209359920,0cyclictest27431-21ssh20:49:400
209359920,0cyclictest0-21swapper/019:54:540
209359920,0cyclictest0-21swapper/019:44:500
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional