You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-11 - 19:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Wed Mar 11, 2026 12:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
51070899490456,14cyclictest528832-21turbostat08:00:003
51070399474440,20cyclictest527098-21turbostat07:59:542
51070899466457,6cyclictest607382-21turbostat11:49:553
51070899465456,6cyclictest546188-21turbostat08:50:003
51070899463455,5cyclictest562073-21turbostat09:34:593
51070899462455,4cyclictest514575-21turbostat07:24:553
51070899462455,4cyclictest514575-21turbostat07:24:553
51070899461454,4cyclictest622971-21turbostat12:29:593
51070899461454,4cyclictest540997-21turbostat08:35:013
51070899461453,5cyclictest588339-21turbostat10:49:593
51070899461452,6cyclictest574194-21turbostat10:14:563
51070899460453,4cyclictest518085-21turbostat07:34:553
51070899460451,6cyclictest612591-21turbostat12:00:003
51070899458451,4cyclictest519812-21turbostat07:39:553
51070899457448,6cyclictest588339-21turbostat10:54:553
51070899456449,4cyclictest532292-21turbostat08:14:553
51070899456449,4cyclictest532292-21turbostat08:14:543
51069499456448,5cyclictest549649-21turbostat09:04:560
51070899454447,4cyclictest568998-21turbostat09:54:593
51070899454447,4cyclictest537536-21turbostat08:29:553
51070899452445,4cyclictest572459-21turbostat10:05:003
51070899452445,4cyclictest560337-21turbostat09:30:003
51070899451444,4cyclictest556886-21turbostat09:24:563
51070899451442,6cyclictest547917-21turbostat08:55:003
51070899451431,17cyclictest590074-21turbostat10:59:553
51070899450443,4cyclictest621246-21turbostat12:24:593
51070899450443,4cyclictest527098-21turbostat07:55:003
51070899448441,4cyclictest616054-21turbostat12:10:013
51070899444436,5cyclictest605656-21turbostat11:44:553
51070899443435,5cyclictest542730-21turbostat08:40:003
51070899443435,5cyclictest542730-21turbostat08:40:003
51070899441433,5cyclictest595262-21turbostat11:09:593
51070899441432,6cyclictest591800-21turbostat11:04:563
51070899441432,6cyclictest591800-21turbostat11:04:563
51070899440431,6cyclictest614321-21turbostat12:04:593
51070899438430,5cyclictest512358-21turbostat07:15:003
51070899436429,4cyclictest521601-21turbostat07:44:553
51070399436429,4cyclictest577660-21turbostat10:24:552
51070899435426,6cyclictest619515-21turbostat12:20:003
51070899434411,20cyclictest600469-21turbostat11:29:563
51070899433411,19cyclictest602198-21turbostat11:34:543
51070899431418,10cyclictest617779-21turbostat12:15:013
51070899427404,20cyclictest542730-21turbostat08:44:553
51070899424416,5cyclictest565536-21turbostat09:45:013
51070899418410,5cyclictest547917-21turbostat08:59:553
51070899417410,4cyclictest537536-21turbostat08:25:003
51070899415408,4cyclictest556886-21turbostat09:20:013
51070899414405,6cyclictest624703-21turbostat12:35:013
51070899413406,4cyclictest567266-21turbostat09:50:003
51070899413406,4cyclictest567266-21turbostat09:50:003
51070899412405,4cyclictest532292-21turbostat08:10:003
51070899411403,5cyclictest575920-21turbostat10:19:553
51070899406386,17cyclictest584879-21turbostat10:40:003
51070899406386,17cyclictest584879-21turbostat10:40:003
51070899404396,5cyclictest603928-21turbostat11:39:553
51070899403393,7cyclictest553413-21turbostat09:10:003
51070899401394,4cyclictest568998-21turbostat09:59:553
51070899397389,5cyclictest535805-21turbostat08:20:003
51069499389381,5cyclictest534077-21turbostat08:19:550
51070899388381,4cyclictest572459-21turbostat10:09:553
51070899386377,6cyclictest598736-21turbostat11:24:553
51070899385376,6cyclictest596988-21turbostat11:19:553
51070399383376,4cyclictest560337-21turbostat09:34:552
51070899381374,4cyclictest530557-21turbostat08:05:003
51070899380374,4cyclictest512358-21turbostat07:19:553
51070899372365,5cyclictest579384-21turbostat10:24:593
51070899367359,5cyclictest595262-21turbostat11:14:543
51070899366345,18cyclictest584879-21turbostat10:44:553
51070899363356,4cyclictest518085-21turbostat07:30:003
51070899358352,3cyclictest609111-21turbostat11:54:563
51070899358352,3cyclictest609111-21turbostat11:54:563
51069499357350,4cyclictest510574-21turbostat07:14:550
51070899356350,3cyclictest626740-21turbostat12:40:013
51070899346339,4cyclictest555138-21turbostat09:15:003
51070899333326,4cyclictest523338-21turbostat07:49:543
51070399329322,4cyclictest570726-21turbostat10:04:542
51069499326318,5cyclictest581116-21turbostat10:34:550
51070899265247,13cyclictest551387-21turbostat09:04:593
51070899253226,23cyclictest583146-21turbostat10:35:013
51070899198189,6cyclictest563802-21turbostat09:40:003
1973991490,96rtkit-daemon0-21swapper/307:06:253
510520211077,23sleep10-21swapper/107:09:381
510271210774,23sleep20-21swapper/207:06:242
510514210573,22sleep00-21swapper/007:09:330
1973991050,6rtkit-daemon4497-21PK-Backend10:47:090
5107089910188,9cyclictest581116-21turbostat10:30:003
1973991010,6rtkit-daemon611496-21diskstats11:55:140
5277302900,8sleep04497-21PK-Backend07:55:150
197399820,6rtkit-daemon4497-21PK-Backend11:40:461
5265392760,5sleep1526542-21switchtime07:50:301
197399760,5rtkit-daemon4497-21PK-Backend10:42:441
197399720,6rtkit-daemon4497-21PK-Backend11:21:091
197399720,6rtkit-daemon4497-21PK-Backend11:07:241
197399720,6rtkit-daemon4497-21PK-Backend10:48:501
197399720,5rtkit-daemon0-21swapper/112:06:261
197399720,5rtkit-daemon0-21swapper/111:25:541
197399720,5rtkit-daemon0-21swapper/111:13:261
197399710,6rtkit-daemon4497-21PK-Backend11:32:171
197399710,5rtkit-daemon0-21swapper/112:00:421
197399710,5rtkit-daemon0-21swapper/111:00:491
197399710,5rtkit-daemon0-21swapper/111:00:481
197399710,5rtkit-daemon0-21swapper/110:51:241
5823752700,4chrt581610-21/usr/sbin/munin10:30:231
197399700,5rtkit-daemon0-21swapper/112:10:071
197399700,5rtkit-daemon0-21swapper/111:51:291
197399700,5rtkit-daemon0-21swapper/111:51:291
197399700,5rtkit-daemon0-21swapper/110:56:001
197399700,5rtkit-daemon0-21swapper/110:35:451
197399700,5rtkit-daemon0-21swapper/110:35:451
5177322690,14sleep2341ksoftirqd/207:25:462
5812912620,6chrt0-21swapper/010:30:020
5305932560,7chrt4497-21PK-Backend08:05:010
5223122530,6sleep20-21swapper/207:40:202
51070399520,48cyclictest1972-21rtkit-daemon10:26:182
51070399510,47cyclictest1972-21rtkit-daemon09:17:112
6220892460,7sleep04497-21PK-Backend12:25:200
5410292460,5sleep24497-21PK-Backend08:35:022
510694994627,8cyclictest13-21ksoftirqd/011:25:560
51070399450,42cyclictest1972-21rtkit-daemon07:38:352
510699994532,10cyclictest622978-21munin-plugin-st12:30:001
510699994523,6cyclictest141rcu_preempt08:00:001
510694994521,10cyclictest13-21ksoftirqd/012:35:020
6112222440,15sleep234-21ksoftirqd/211:55:122
5469222440,6sleep126-21ksoftirqd/108:50:171
51070399440,41cyclictest1972-21rtkit-daemon10:18:392
51069999435,16cyclictest609270-21grep11:50:021
51069999432,19cyclictest26-21ksoftirqd/109:05:251
510694994320,10cyclictest13-21ksoftirqd/007:50:000
510699994214,8cyclictest26-21ksoftirqd/110:01:071
51069999421,20cyclictest611704-21cut11:55:191
51069999421,17cyclictest438-20systemd-journal08:10:011
510694994219,10cyclictest13-21ksoftirqd/011:15:010
510694994216,8cyclictest13-21ksoftirqd/011:34:590
51069999416,18cyclictest26-21ksoftirqd/108:35:231
51069999416,18cyclictest26-21ksoftirqd/108:35:231
51069999416,16cyclictest520238-21tr07:35:141
510699994114,7cyclictest26-21ksoftirqd/107:27:341
510699994113,8cyclictest26-21ksoftirqd/109:03:091
510699994111,16cyclictest26-21ksoftirqd/110:14:331
510694994123,7cyclictest13-21ksoftirqd/007:25:030
510694994122,6cyclictest13-21ksoftirqd/009:00:010
510694994119,9cyclictest13-21ksoftirqd/011:00:010
510694994114,13cyclictest593223-21sshd11:01:170
510694994114,13cyclictest593223-21sshd11:01:170
510694994113,8cyclictest13-21ksoftirqd/012:00:120
510694994112,16cyclictest13-21ksoftirqd/008:10:160
510694994112,16cyclictest13-21ksoftirqd/008:10:160
510699994017,5cyclictest141rcu_preempt10:20:121
510699994016,6cyclictest141rcu_preempt10:05:221
510699994014,8cyclictest26-21ksoftirqd/107:31:591
510699994012,8cyclictest26-21ksoftirqd/111:35:281
51069999401,21cyclictest567259-21sh09:50:001
51069999401,21cyclictest567259-21sh09:49:591
51069999401,19cyclictest620351-21/usr/sbin/munin12:20:201
51069999401,17cyclictest513479-21taskset07:15:151
51069999401,16cyclictest560360-21taskset09:30:001
51069999401,15cyclictest438-20systemd-journal10:00:001
510699994010,9cyclictest26-21ksoftirqd/109:15:021
510694994023,6cyclictest13-21ksoftirqd/010:35:310
510694994023,6cyclictest13-21ksoftirqd/010:35:310
510694994023,13cyclictest13-21ksoftirqd/008:45:130
510694994022,14cyclictest13-21ksoftirqd/011:55:010
510694994022,14cyclictest13-21ksoftirqd/011:55:010
510694994022,13cyclictest13-21ksoftirqd/010:15:120
51069499401,8cyclictest570720-21perf10:00:000
510694994012,15cyclictest13-21ksoftirqd/012:20:220
510694994010,17cyclictest13-21ksoftirqd/010:10:260
510703993928,7cyclictest573672-21find10:05:292
510703993923,11cyclictest438-20systemd-journal11:25:012
51069999395,13cyclictest26-21ksoftirqd/108:15:011
51069999395,13cyclictest26-21ksoftirqd/108:15:011
51069999393,20cyclictest26-21ksoftirqd/110:18:161
510699993923,11cyclictest523339-21munin-plugin-st07:45:001
51069999391,20cyclictest625205-21/usr/sbin/munin12:35:221
51069999391,20cyclictest597715-21/usr/sbin/munin11:15:221
51069999391,18cyclictest555927-21latency09:15:221
510694993923,5cyclictest13-21ksoftirqd/010:50:170
510694993922,6cyclictest13-21ksoftirqd/009:54:570
510694993922,13cyclictest13-21ksoftirqd/010:20:130
510694993922,13cyclictest13-21ksoftirqd/010:00:140
510694993921,6cyclictest13-21ksoftirqd/009:35:120
510694993921,6cyclictest13-21ksoftirqd/008:40:110
510694993921,5cyclictest13-21ksoftirqd/012:05:140
510694993921,13cyclictest13-21ksoftirqd/007:20:020
510694993920,15cyclictest625627-21/usr/sbin/munin12:35:180
510694993916,10cyclictest13-21ksoftirqd/009:20:010
510694993911,8cyclictest13-21ksoftirqd/007:20:270
510694993910,16cyclictest13-21ksoftirqd/009:45:280
510694993910,16cyclictest13-21ksoftirqd/009:45:280
510694993910,16cyclictest13-21ksoftirqd/008:30:260
510694993910,16cyclictest13-21ksoftirqd/008:25:200
197399390,32rtkit-daemon2481-40tracker-miner-f08:39:572
197399390,32rtkit-daemon2481-40tracker-miner-f08:39:572
197399390,31rtkit-daemon34-21ksoftirqd/207:47:162
510703993824,9cyclictest543387-21hddtemp_smartct08:40:172
510703993815,19cyclictest551369-21apt-get09:05:092
510703993813,5cyclictest537528-21apt08:24:592
51069999387,16cyclictest568570-21sed09:50:291
51069999387,16cyclictest563191-21grep09:35:251
51069999386,18cyclictest0-21swapper/108:45:021
510699993826,9cyclictest26-21ksoftirqd/108:20:001
51069999381,9cyclictest4497-21PK-Backend12:30:591
51069999381,19cyclictest618837-21/usr/sbin/munin12:15:241
51069999381,19cyclictest529677-21/usr/sbin/munin08:00:211
51069999381,15cyclictest564436-21chrt09:40:171
510694993822,5cyclictest13-21ksoftirqd/011:20:210
510694993822,12cyclictest13-21ksoftirqd/012:10:140
510694993822,12cyclictest13-21ksoftirqd/011:20:020
510694993821,6cyclictest13-21ksoftirqd/008:05:230
510694993821,5cyclictest13-21ksoftirqd/007:40:140
510694993821,13cyclictest13-21ksoftirqd/008:21:200
510694993820,6cyclictest13-21ksoftirqd/007:30:280
510694993810,15cyclictest13-21ksoftirqd/009:25:140
51070399372,7cyclictest565558-21munin-run09:45:022
510703993722,11cyclictest588185-21sshd10:48:172
510703993721,12cyclictest529825-21ntpq08:00:242
510703993719,13cyclictest512804-21cat07:15:032
510703993714,12cyclictest526535-40lsb_release07:50:302
51069999377,5cyclictest141rcu_preempt07:48:561
51069999375,15cyclictest26-21ksoftirqd/107:20:301
51069999375,15cyclictest26-21ksoftirqd/107:20:301
51069999372,9cyclictest26-21ksoftirqd/108:20:181
51069999371,18cyclictest544697-21/usr/sbin/munin08:45:281
51069999371,10cyclictest0-21swapper/109:34:441
51069499377,17cyclictest13-21ksoftirqd/009:30:150
510694993722,11cyclictest586594-21cron10:44:590
510694993721,5cyclictest13-21ksoftirqd/009:09:060
510694993721,12cyclictest595255-21cron11:09:590
510694993720,5cyclictest13-21ksoftirqd/007:25:210
5218222360,4chrt0-21swapper/007:40:020
510703993619,12cyclictest609039-21sshd11:49:162
510703993616,15cyclictest601128-21which11:25:182
51069999363,15cyclictest0-21swapper/107:11:491
510699993628,5cyclictest4497-21PK-Backend08:26:171
510694993620,5cyclictest13-21ksoftirqd/008:51:270
510694993619,6cyclictest13-21ksoftirqd/011:36:560
510694993619,13cyclictest13-21ksoftirqd/008:35:140
510694993619,13cyclictest13-21ksoftirqd/008:35:130
510694993613,6cyclictest141rcu_preempt11:40:450
510694993613,10cyclictest13-21ksoftirqd/009:40:260
510694993612,10cyclictest13-21ksoftirqd/007:50:150
197399360,30rtkit-daemon0-21swapper/208:28:592
51070399358,13cyclictest141rcu_preempt11:40:232
510703993519,11cyclictest550291-21diskstats09:00:172
510703993518,12cyclictest567508-21find09:50:102
510703993515,16cyclictest976-21rs:main2
510703993514,8cyclictest141rcu_preempt12:05:202
51069999351,14cyclictest581113-21idleruntime-cro10:30:001
51069999351,14cyclictest4497-21PK-Backend09:20:121
51069999351,14cyclictest4497-21PK-Backend08:30:291
51069499358,13cyclictest557681-21sed09:20:200
510694993519,12cyclictest608033-21sshd11:45:170
510703993427,4cyclictest4497-21PK-Backend08:15:542
510703993421,9cyclictest624793-21rm12:35:012
510703993421,9cyclictest624690-21cron12:39:562
510703993419,11cyclictest557789-21needreboot09:20:212
510703993419,11cyclictest533973-21unattended-upgr08:14:082
510703993419,11cyclictest533973-21unattended-upgr08:14:082
510703993419,10cyclictest546726-21expr08:50:132
510703993418,12cyclictest586158-21sed10:40:292
510703993418,11cyclictest438-20systemd-journal08:58:092
510703993416,7cyclictest618752-21ntp_kernel_err12:15:222
51070399341,30cyclictest0-21swapper/211:15:232
51069499345,16cyclictest13-21ksoftirqd/012:20:010
510703993326,4cyclictest4497-21PK-Backend11:12:172
510703993321,8cyclictest567424-21sort09:50:012
510703993321,8cyclictest567424-21sort09:50:012
510703993319,10cyclictest560340-21grep09:30:002
510703993319,10cyclictest555125-21cron09:14:592
510703993318,11cyclictest622230-21ntp_kernel_pll_12:25:232
510703993318,11cyclictest531111-21expr08:05:142
510703993317,12cyclictest582409-21awk10:30:242
510703993316,12cyclictest1033-21gdbus12:25:012
51070399331,4cyclictest4497-21PK-Backend10:10:592
51070399331,30cyclictest0-21swapper/208:49:222
51070399331,29cyclictest4497-21PK-Backend12:04:492
51070399331,29cyclictest4497-21PK-Backend10:56:192
51070399331,29cyclictest4497-21PK-Backend10:35:232
51070399331,29cyclictest4497-21PK-Backend10:35:232
51070399331,28cyclictest4497-21PK-Backend11:51:032
51070399331,28cyclictest4497-21PK-Backend11:51:032
51069999331,9cyclictest4497-21PK-Backend08:55:261
510694993315,7cyclictest141rcu_preempt10:05:560
197399330,27rtkit-daemon0-21swapper/207:10:162
510703993224,5cyclictest0-21swapper/211:06:012
510703993215,12cyclictest438-20systemd-journal11:40:002
510703993214,13cyclictest602678-21expr11:30:122
6164942310,7sleep24497-21PK-Backend12:10:132
510703993124,4cyclictest4497-21PK-Backend07:22:442
510703993124,4cyclictest4497-21PK-Backend07:22:442
510694993117,10cyclictest553834-21cut09:10:130
51070399303,7cyclictest141rcu_preempt09:55:122
51070399291,9cyclictest924-21NetworkManager11:00:272
51070399291,9cyclictest924-21NetworkManager11:00:272
510703992819,6cyclictest0-21swapper/207:34:192
510703992712,11cyclictest588603-21sh10:50:102
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional