You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 21:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Tue Nov 18, 2025 12:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1664799270,26cyclictest30836-21latency_hist08:45:003
1664799260,25cyclictest0-21swapper/310:21:493
1664799260,21cyclictest30596-21munin-run10:55:003
1664799260,0cyclictest0-21swapper/309:25:033
1664599250,24cyclictest7147-21apt-get07:56:492
1664599250,24cyclictest6731-21awk07:55:012
1664199250,1cyclictest488-21cron11:00:001
1663699250,1cyclictest26605-21awk07:30:000
1664799240,0cyclictest3505-21apt-get12:11:513
16645992422,1cyclictest527-21awk11:00:002
1664199241,22cyclictest20950irq/141-nvme0q208:40:001
1663699240,23cyclictest0-21swapper/008:46:520
16647992320,2cyclictest13644-21cut09:15:003
1664599230,0cyclictest0-21swapper/211:26:492
16641992321,1cyclictest4357-21perf07:50:001
16641992321,1cyclictest23736-21apt-get10:41:511
1664199230,22cyclictest833-21munin-plugin-st08:50:011
1664199230,22cyclictest593-21dbus-daemon12:35:001
1664199230,22cyclictest313-21systemd-journal08:19:591
1664199230,22cyclictest28527-21apt-get10:51:501
1664199230,22cyclictest25982-21dump-pmu-power08:35:001
1664199230,0cyclictest28254-21apt-get11:56:531
16636992323,0cyclictest16987-21apt-get07:11:490
1663699230,2cyclictest18774-21idleruntime-cro08:20:000
1663699230,22cyclictest13946-21dump-pmu-power08:09:590
16647992220,1cyclictest18997-21munin-run07:15:013
1664799220,2cyclictest1854-21gltestperf08:51:523
1664799220,21cyclictest25454-21turbostat11:50:003
1664799220,21cyclictest22927-21sendmail-msp10:39:593
1664799220,21cyclictest0-21swapper/311:06:523
1664799220,21cyclictest0-21swapper/311:05:003
1664799220,21cyclictest0-21swapper/308:55:013
1664799220,1cyclictest21645-21apt-get09:31:513
1664799220,1cyclictest0-21swapper/312:40:003
1664799220,1cyclictest0-21swapper/312:40:003
16645992220,1cyclictest17512-21latency_hist12:40:002
16645992220,1cyclictest17512-21latency_hist12:40:002
1664599220,21cyclictest28794-21apt-get08:41:512
1664599220,21cyclictest24596-21apt-get07:26:492
1664599220,21cyclictest20514-21ls10:35:002
1664599220,21cyclictest0-21swapper/211:51:522
1664599220,21cyclictest0-21swapper/209:35:022
1664599220,1cyclictest834-21munin-run08:50:012
1664599220,1cyclictest1276-21awk09:55:002
1664599220,0cyclictest0-21swapper/209:31:502
16641992221,0cyclictest29257-21apt-get09:46:521
16641992220,1cyclictest0-21swapper/108:05:001
1664199220,21cyclictest9234-21apt-get09:06:511
1664199220,21cyclictest0-21swapper/111:51:521
1664199220,21cyclictest0-21swapper/111:41:501
1664199220,21cyclictest0-21swapper/111:00:021
1664199220,21cyclictest0-21swapper/109:16:501
1664199220,1cyclictest6118-21perf10:05:001
16636992221,1cyclictest0-21swapper/008:59:590
1663699220,21cyclictest29398-21awk07:35:000
1663699220,21cyclictest23652-21grep09:35:010
1663699220,21cyclictest0-21swapper/011:51:510
1663699220,21cyclictest0-21swapper/010:01:510
1663699220,21cyclictest0-21swapper/008:21:480
1663699220,21cyclictest0-21swapper/007:56:490
1663699220,1cyclictest0-21swapper/010:16:520
16647992121,0cyclictest0-21swapper/312:05:023
16647992121,0cyclictest0-21swapper/308:11:483
16647992121,0cyclictest0-21swapper/307:31:503
16647992120,1cyclictest10576-21apt-get11:20:033
16647992119,1cyclictest28866-21munin-run09:44:593
1664799210,2cyclictest0-21swapper/311:56:533
1664799210,20cyclictest0-21swapper/310:11:513
1664799210,20cyclictest0-21swapper/308:21:493
1664799210,20cyclictest0-21swapper/307:51:483
1664799210,1cyclictest22025-21needreboot11:41:533
1664799210,1cyclictest13788-21apt-get11:26:503
1664799210,16cyclictest6513-21apt-get10:06:543
1664799210,0cyclictest0-21swapper/312:31:513
1664799210,0cyclictest0-21swapper/307:36:483
16645992121,0cyclictest0-21swapper/210:36:522
16645992120,1cyclictest0-21swapper/209:56:502
1664599211,18cyclictest0-21swapper/208:06:502
1664599210,21cyclictest0-21swapper/208:55:012
1664599210,20cyclictest21576-21apt-get08:26:482
1664599210,20cyclictest10715-21apt-get12:26:502
1664599210,1cyclictest18183-21cron11:35:002
16641992120,1cyclictest0-21swapper/111:31:511
16641992119,1cyclictest18993-21perf07:15:011
1664199210,0cyclictest3216-21apt-get08:55:011
1664199210,0cyclictest29788-21apt-get07:36:491
1664199210,0cyclictest0-21swapper/112:01:471
1664199210,0cyclictest0-21swapper/111:21:511
1664199210,0cyclictest0-21swapper/110:26:501
16636992121,0cyclictest0-21swapper/009:36:520
16636992119,1cyclictest8180-21apt11:15:010
1663699210,21cyclictest0-21swapper/011:20:030
1663699210,21cyclictest0-21swapper/010:41:510
1663699210,21cyclictest0-21swapper/009:56:490
1663699210,21cyclictest0-21swapper/007:15:020
1663699210,20cyclictest5905-21apt-get12:16:530
1663699210,0cyclictest16450-21apt-get09:21:520
1663699210,0cyclictest0-21swapper/009:41:510
1663699210,0cyclictest0-21swapper/009:26:520
16647992020,0cyclictest0-21swapper/312:21:523
16647992020,0cyclictest0-21swapper/308:36:513
16647992020,0cyclictest0-21swapper/307:15:023
16647992019,1cyclictest26133-21apt10:46:523
16647992018,2cyclictest0-21swapper/310:40:003
1664799200,7cyclictest18728-21cstates10:31:523
1664799200,1cyclictest0-21swapper/311:36:573
1664799200,19cyclictest3662-21cron10:00:003
1664799200,19cyclictest25853-21apt-get11:51:513
1664799200,0cyclictest0-21swapper/310:26:503
1664799200,0cyclictest0-21swapper/307:21:523
16645992020,0cyclictest19178-21apt-get08:21:482
16645992020,0cyclictest0-21swapper/212:05:022
16645992020,0cyclictest0-21swapper/212:01:492
16645992020,0cyclictest0-21swapper/211:21:502
16645992020,0cyclictest0-21swapper/211:01:512
16645992020,0cyclictest0-21swapper/210:26:492
16645992020,0cyclictest0-21swapper/210:16:522
16645992020,0cyclictest0-21swapper/208:51:502
16645992019,1cyclictest4115-21apt-get10:01:522
16645992019,0cyclictest19072-21apt-get09:25:032
16645992018,1cyclictest11568-21perf08:05:002
16645992018,1cyclictest10573-21perf11:19:592
1664599200,0cyclictest14029-21apt-get09:16:512
1664599200,0cyclictest0-21swapper/211:56:532
1664599200,0cyclictest0-21swapper/207:36:482
16641992020,0cyclictest0-21swapper/112:36:531
16641992020,0cyclictest0-21swapper/112:36:531
16641992020,0cyclictest0-21swapper/112:26:511
16641992020,0cyclictest0-21swapper/108:21:491
16641992020,0cyclictest0-21swapper/108:11:471
16641992020,0cyclictest0-21swapper/108:06:501
16641992018,1cyclictest30839-21sh08:45:011
16641992018,1cyclictest10314-21awk12:25:001
1664199200,0cyclictest18525-21apt-get10:31:521
16636992020,0cyclictest0-21swapper/012:21:520
16636992020,0cyclictest0-21swapper/012:06:510
16636992020,0cyclictest0-21swapper/011:56:520
16636992020,0cyclictest0-21swapper/011:41:500
16636992020,0cyclictest0-21swapper/011:36:540
16636992020,0cyclictest0-21swapper/011:00:020
16636992020,0cyclictest0-21swapper/010:06:540
16636992020,0cyclictest0-21swapper/009:51:500
16636992020,0cyclictest0-21swapper/009:06:500
16636992019,1cyclictest13721-21apt-get10:21:500
1663699200,20cyclictest0-21swapper/008:51:490
1663699200,0cyclictest20924-21apt-get10:36:520
1663699200,0cyclictest0-21swapper/009:00:000
16647991919,0cyclictest0-21swapper/312:26:513
16647991919,0cyclictest0-21swapper/312:16:533
16647991919,0cyclictest0-21swapper/309:46:513
16647991917,1cyclictest0-21swapper/308:05:003
1664799190,1cyclictest17503-21modprobe08:16:523
1664799190,1cyclictest17365-21grep11:31:583
1664799190,1cyclictest0-21swapper/308:26:583
1664799190,18cyclictest0-21swapper/309:21:583
16645991919,0cyclictest14354-21apt-get08:11:482
16645991919,0cyclictest0-21swapper/211:36:542
16645991919,0cyclictest0-21swapper/210:51:502
16645991919,0cyclictest0-21swapper/210:41:512
16645991919,0cyclictest0-21swapper/209:41:502
16645991919,0cyclictest0-21swapper/209:06:512
16645991919,0cyclictest0-21swapper/208:16:502
16645991919,0cyclictest0-21swapper/207:31:492
16645991918,1cyclictest10329-21perf12:24:592
16645991918,1cyclictest0-21swapper/210:21:492
16641991919,0cyclictest8511-21perf10:09:591
16641991919,0cyclictest0-21swapper/112:06:501
16641991919,0cyclictest0-21swapper/107:26:491
16641991918,1cyclictest8904-21apt-get10:11:501
16641991918,0cyclictest0-21swapper/107:15:021
1664199190,17cyclictest313-21systemd-journal12:20:001
1664199190,0cyclictest6033-21apt-get09:00:011
16636991919,0cyclictest0-21swapper/012:26:510
16636991919,0cyclictest0-21swapper/012:01:490
16636991919,0cyclictest0-21swapper/011:26:500
16636991917,1cyclictest15090-21idleruntime-cro12:35:000
1664799184,13cyclictest10037-21irqstats09:06:553
16647991818,0cyclictest6835-21apt-get09:01:503
16647991818,0cyclictest0-21swapper/310:16:523
16647991814,3cyclictest0-21swapper/310:56:513
1664799180,3cyclictest0-21swapper/309:35:023
1664799180,1cyclictest12572-21tail08:06:533
1664799180,1cyclictest0-21swapper/308:46:523
1664799180,0cyclictest0-21swapper/307:46:573
16645991818,0cyclictest13115-21apt-get12:31:522
16645991818,0cyclictest0-21swapper/212:16:512
16645991818,0cyclictest0-21swapper/211:06:522
16645991818,0cyclictest0-21swapper/210:06:542
16645991818,0cyclictest0-21swapper/209:11:522
16641991818,0cyclictest0-21swapper/110:46:511
16641991818,0cyclictest0-21swapper/109:56:501
16641991818,0cyclictest0-21swapper/108:51:501
16641991818,0cyclictest0-21swapper/108:26:491
16641991818,0cyclictest0-21swapper/107:56:501
16641991818,0cyclictest0-21swapper/107:51:481
16641991817,1cyclictest16449-21apt09:21:511
16641991816,1cyclictest28848-21perf09:45:001
16641991816,1cyclictest26432-21latency_hist09:40:011
16636991818,0cyclictest0-21swapper/012:11:510
16636991818,0cyclictest0-21swapper/011:16:520
16636991818,0cyclictest0-21swapper/011:06:520
16636991818,0cyclictest0-21swapper/008:31:510
16636991816,1cyclictest31967-21perf07:40:000
16647991717,0cyclictest32369-21apt-get07:41:523
16645991717,0cyclictest26134-21apt-get10:46:522
16645991717,0cyclictest0-21swapper/210:11:512
16645991717,0cyclictest0-21swapper/207:46:532
1664599170,16cyclictest18068-21grep09:22:012
16641991717,0cyclictest0-21swapper/110:25:011
16641991717,0cyclictest0-21swapper/109:26:521
16641991715,1cyclictest22939-21perf10:40:001
16641991714,2cyclictest3702-21cut12:11:511
16636991717,0cyclictest0-21swapper/011:46:500
16636991717,0cyclictest0-21swapper/010:56:490
16636991717,0cyclictest0-21swapper/008:41:510
16636991717,0cyclictest0-21swapper/008:36:520
16636991717,0cyclictest0-21swapper/008:26:490
16636991717,0cyclictest0-21swapper/007:51:490
16636991717,0cyclictest0-21swapper/007:50:000
1664799163,12cyclictest31655-21missed_timers12:01:533
1664799160,3cyclictest0-21swapper/309:20:003
1664799160,15cyclictest0-21swapper/307:56:553
16645991616,0cyclictest0-21swapper/207:41:532
16645991615,1cyclictest21058-21apt-get11:41:502
16645991615,0cyclictest0-21swapper/209:46:522
16645991614,1cyclictest21778-21perf07:20:002
1664599160,15cyclictest22572-21ls07:21:542
16641991616,0cyclictest22204-21apt-get07:21:521
16641991616,0cyclictest0-21swapper/109:51:501
1664199160,0cyclictest0-21swapper/107:30:011
16636991616,0cyclictest0-21swapper/007:41:530
16636991614,1cyclictest20850irq/140-nvme0q109:14:590
16636991614,1cyclictest16117-21apt-get10:26:500
1663699160,15cyclictest29158-21cut10:51:520
1664799150,2cyclictest0-21swapper/308:35:003
1664799150,14cyclictest0-21swapper/311:12:023
16645991515,0cyclictest0-21swapper/212:11:522
16645991515,0cyclictest0-21swapper/208:36:522
16645991513,1cyclictest25467-21munin-run11:50:002
16645991513,1cyclictest0-21swapper/207:15:012
1664599150,7cyclictest0-21swapper/211:11:542
16641991515,0cyclictest0-21swapper/111:26:501
1664199150,1cyclictest0-21swapper/111:46:541
16636991515,0cyclictest0-21swapper/007:21:520
16636991514,1cyclictest0-21swapper/010:32:000
16636991514,0cyclictest0-21swapper/008:11:480
16636991512,2cyclictest29152-21sshd09:45:580
16647991414,0cyclictest0-21swapper/309:50:263
16647991413,1cyclictest0-21swapper/307:25:313
1664799140,14cyclictest0-21swapper/311:15:013
1664799140,0cyclictest0-21swapper/310:01:503
1664599140,14cyclictest0-21swapper/208:30:202
16641991414,0cyclictest0-21swapper/111:09:521
16641991414,0cyclictest0-21swapper/107:41:421
1664199140,14cyclictest0-21swapper/110:15:491
1664199140,14cyclictest0-21swapper/109:31:381
16636991414,0cyclictest0-21swapper/011:32:480
16636991414,0cyclictest0-21swapper/010:46:030
1663699140,14cyclictest0-21swapper/009:15:590
1663699140,0cyclictest0-21swapper/012:35:190
1663699140,0cyclictest0-21swapper/012:35:180
1663699140,0cyclictest0-21swapper/010:10:060
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional