You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-10 - 11:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Tue Mar 10, 2026 00:44:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
384706399479441,18cyclictest3874349-21turbostat20:25:003
384706399474451,20cyclictest3948330-21turbostat23:59:553
384706399465457,5cyclictest3954147-21turbostat00:10:003
384706399465457,5cyclictest3932400-21turbostat23:14:553
384706399463456,4cyclictest3867419-21turbostat20:09:553
384706399461453,5cyclictest3946601-21turbostat23:50:013
384705999457450,4cyclictest3916799-21turbostat22:29:552
384705999457450,4cyclictest3874349-21turbostat20:29:562
384706399456448,4cyclictest3887070-21turbostat21:00:003
384706399456433,20cyclictest3860201-21turbostat19:49:543
384706399455447,5cyclictest3941417-21turbostat23:39:553
384706399455447,5cyclictest3855016-21turbostat19:34:553
384706399455446,6cyclictest3920261-21turbostat22:35:003
384706399454447,4cyclictest3961423-21turbostat00:30:003
384706399454430,7cyclictest3899500-21turbostat21:35:003
384706399449442,4cyclictest3963162-21turbostat00:35:003
384706399448441,4cyclictest3957663-21turbostat00:20:003
384706399447440,4cyclictest3872616-21turbostat20:20:013
384705999443417,23cyclictest3904692-21turbostat21:54:552
384706399442435,4cyclictest3883308-21turbostat20:50:003
384706399441434,4cyclictest3913338-21turbostat22:15:003
384706399438431,4cyclictest3941417-21turbostat23:35:003
384706399436429,4cyclictest3923720-21turbostat22:49:553
384706399436429,4cyclictest3877804-21turbostat20:39:553
384705999435428,4cyclictest3856744-21turbostat19:39:552
384706399434426,5cyclictest3876086-21turbostat20:34:543
384706399433425,5cyclictest3848706-21turbostat19:15:003
384706399432425,4cyclictest3899500-21turbostat21:39:563
384706399432425,4cyclictest3899500-21turbostat21:39:553
384706399431423,5cyclictest3944878-21turbostat23:45:013
384706399430423,4cyclictest3867419-21turbostat20:05:003
384706399430422,5cyclictest3946601-21turbostat23:54:553
384706399428394,18cyclictest3870888-21turbostat20:15:013
384706399423394,14cyclictest3959400-21turbostat00:25:003
384706399422414,5cyclictest3909874-21turbostat22:09:543
384706399418409,6cyclictest3909874-21turbostat22:05:003
384706399418409,6cyclictest3909874-21turbostat22:04:593
384706399418409,6cyclictest3852974-21turbostat19:25:003
384706399417410,4cyclictest3913338-21turbostat22:19:553
384706399417409,5cyclictest3863661-21turbostat19:55:013
384706399415406,6cyclictest3916799-21turbostat22:25:003
384705999412405,4cyclictest3858477-21turbostat19:44:552
384706399407399,5cyclictest3892565-21turbostat21:15:013
384706399405398,4cyclictest3930679-21turbostat23:09:553
384706399405398,4cyclictest3894301-21turbostat21:24:543
384706399405397,5cyclictest3928912-21turbostat23:04:553
384705999403394,6cyclictest3927182-21turbostat22:59:562
384705999401393,5cyclictest3863661-21turbostat19:59:562
384705999401393,5cyclictest3863661-21turbostat19:59:562
384706399398388,7cyclictest3918526-21turbostat22:30:013
384706399396374,19cyclictest3906414-21turbostat21:55:003
384706399396363,18cyclictest3885044-21turbostat20:54:593
384706399393375,15cyclictest3879549-21turbostat20:44:563
384706399393373,17cyclictest3860201-21turbostat19:44:593
384705999393386,4cyclictest3851245-21turbostat19:24:552
384706399391384,4cyclictest3865399-21turbostat20:00:003
384706399391384,4cyclictest3865399-21turbostat20:00:003
384706399389372,14cyclictest3963162-21turbostat00:39:553
384706399388380,5cyclictest3927182-21turbostat22:55:013
384706399384375,6cyclictest3921994-21turbostat22:40:003
384706399380372,5cyclictest3888805-21turbostat21:09:553
384706399379370,6cyclictest3908146-21turbostat22:00:003
384706399374367,4cyclictest3876086-21turbostat20:29:593
384706399372364,5cyclictest3928912-21turbostat23:00:003
384706399369362,4cyclictest3888805-21turbostat21:05:003
384705999367360,4cyclictest3918526-21turbostat22:34:562
384706399363355,5cyclictest3858477-21turbostat19:40:003
384705999362355,4cyclictest3934440-21turbostat23:19:552
384705999358351,4cyclictest3892565-21turbostat21:19:562
384706399356348,5cyclictest3936179-21turbostat23:24:553
384706399355348,4cyclictest3894301-21turbostat21:19:593
384706399349342,4cyclictest3852974-21turbostat19:29:553
384706399348341,4cyclictest3902946-21turbostat21:44:593
384705399348341,4cyclictest3902946-21turbostat21:49:550
384706399344337,4cyclictest3937956-21turbostat23:29:553
384706399343336,4cyclictest3904692-21turbostat21:50:003
384706399341333,5cyclictest3848706-21turbostat19:19:553
384706399340333,4cyclictest3923720-21turbostat22:45:003
384706399337330,4cyclictest3897761-21turbostat21:30:003
384705999335328,4cyclictest3869144-21turbostat20:14:542
384705399335328,4cyclictest3897761-21turbostat21:34:550
384705399332324,5cyclictest3950367-21turbostat00:04:550
384706399331324,4cyclictest3954147-21turbostat00:14:563
384706399327320,4cyclictest3952427-21turbostat00:05:003
384705999326319,4cyclictest3896034-21turbostat21:29:552
384705399324317,4cyclictest3906414-21turbostat21:59:550
384705999315307,5cyclictest3887070-21turbostat21:04:552
3846547212872,46sleep00-21swapper/019:05:280
3846617210774,23sleep20-21swapper/219:06:242
3846666210672,23sleep10-21swapper/119:07:021
3846613210674,22sleep30-21swapper/319:06:203
395186421010,7sleep14497-21PK-Backend00:00:291
3847059999989,7cyclictest3908146-21turbostat22:04:552
3847059999989,7cyclictest3908146-21turbostat22:04:552
3847059999790,4cyclictest3921994-21turbostat22:44:552
3847063999482,9cyclictest3936179-21turbostat23:20:003
197399870,10rtkit-daemon3947230-21df23:50:152
38754812700,4chrt3874587-21/usr/sbin/munin20:25:280
384705999692,45cyclictest3874145-21taskset20:22:372
384705999682,51cyclictest3926964-21taskset22:52:282
384705999671,40cyclictest3867042-21taskset20:00:512
384705999661,39cyclictest3900728-21sed21:35:292
384705999661,39cyclictest3900728-21sed21:35:292
384705999661,38cyclictest3943303-21sed23:40:012
384705999661,37cyclictest3950110-21taskset23:56:592
384705999661,37cyclictest3903003-21cat21:45:022
384705999651,44cyclictest3944451-21timerwakeupswit23:40:312
384705999651,37cyclictest3882998-21taskset20:46:482
384705999641,37cyclictest3936765-21cstates23:20:152
384705999641,37cyclictest3911415-21chrt22:07:482
384705999641,36cyclictest3854957-21taskset19:29:362
384705999632,35cyclictest3961042-21taskset00:25:492
384705999631,37cyclictest3881033-21taskset20:42:082
384705999631,36cyclictest3884775-21taskset20:51:522
384705999631,33cyclictest3863625-21taskset19:54:472
384705999631,31cyclictest3952491-21cat00:05:012
384705999631,30cyclictest0-21swapper/200:35:022
384705999621,38cyclictest3869120-21taskset20:09:482
384705999621,35cyclictest3963769-21expr00:35:142
384705999621,35cyclictest3861819-21taskset19:48:472
384705999621,34cyclictest3850959-21taskset19:17:162
384705999611,36cyclictest3906840-21expr21:55:132
384705999611,35cyclictest3913206-21taskset22:13:372
384705999611,34cyclictest3957234-21sed00:15:322
384705999601,39cyclictest0-21swapper/200:05:372
384705999601,38cyclictest3959157-21taskset00:22:152
384705999601,34cyclictest3939420-21taskset23:26:572
384705999601,34cyclictest3892048-21sed21:10:302
384705999601,34cyclictest0-21swapper/223:30:292
384705999601,32cyclictest3903194-21/usr/sbin/munin21:45:162
384705999591,32cyclictest3921140-21missed_timers22:35:222
384705999591,32cyclictest3915660-21expr22:20:142
384705999581,34cyclictest0-21swapper/220:36:002
384705999571,33cyclictest3930389-21taskset23:02:012
384705999561,34cyclictest3848589-21taskset19:13:462
384705999561,33cyclictest3885915-21cstates20:55:142
384705999561,32cyclictest3899329-21taskset21:33:072
384705999552,31cyclictest3889711-21diskstats21:05:172
384705999551,34cyclictest0-21swapper/223:49:552
384705999551,33cyclictest0-21swapper/220:17:092
384705999541,31cyclictest3856420-21taskset19:31:092
3847059995316,32cyclictest3931357-21idleruntime23:05:192
384705999531,46cyclictest3954702-21expr00:10:142
38944862520,5chrt0-21swapper/021:20:010
3847059995117,29cyclictest3925568-21ls22:50:012
39489922490,7sleep14497-21PK-Backend23:55:131
384705999478,34cyclictest3913577-21cat22:15:112
384705999462,39cyclictest3895242-21needreboot21:20:222
3847059994513,28cyclictest3876552-21cstates20:30:142
38789552440,18chrt13-21ksoftirqd/020:35:310
384705999441,37cyclictest3934668-21cut23:15:012
39548112430,6sleep03847044-21cyclictest00:10:170
38654142430,7sleep013-21ksoftirqd/020:00:000
38654142430,7sleep013-21ksoftirqd/020:00:000
384705399431,17cyclictest3950464-40gzip00:00:010
38830652420,7sleep0161rcuc/020:47:250
3847055994226,12cyclictest3848697-21apt-get19:15:011
3847053994224,14cyclictest13-21ksoftirqd/000:38:320
3847053994219,7cyclictest3871280-21tr20:15:160
3847055994122,15cyclictest3893575-21ntp_states21:15:261
384705599411,11cyclictest3858467-21apt19:40:011
3847055994111,13cyclictest3905095-21tr21:50:151
3847053994122,14cyclictest3933299-21df23:10:170
384705399411,11cyclictest3916783-21cron22:25:020
3847055994010,27cyclictest3920252-21sh22:35:001
384705399404,11cyclictest13-21ksoftirqd/000:17:540
3847053994022,13cyclictest13-21ksoftirqd/019:20:450
384705399402,11cyclictest1-21systemd00:25:000
3847053994010,5cyclictest141rcu_preempt00:35:010
3847055993923,12cyclictest2080-21gdbus23:04:151
384705599391,16cyclictest39015102cpu21:40:131
384705599391,14cyclictest4497-21PK-Backend22:55:271
3847053993917,18cyclictest3885070-21unin-run20:55:000
384705399391,12cyclictest3889037-21apt-key21:05:020
39104732380,7chrt4497-21PK-Backend22:05:141
3847053993825,9cyclictest13-21ksoftirqd/020:30:140
3847053993823,11cyclictest3916781-21cron22:25:010
3847053993823,11cyclictest13-21ksoftirqd/019:16:340
3847053993821,13cyclictest3866286-21df_inode20:00:140
3847053993820,14cyclictest3947227-21df23:50:140
3847053993820,13cyclictest13-21ksoftirqd/023:00:270
3847055993720,13cyclictest3897037-21ntp_states21:25:251
3847055993719,14cyclictest3912239-21df_inode22:10:161
3847055993715,19cyclictest3957016-21processes00:15:281
384705599371,11cyclictest4497-21PK-Backend21:35:001
384705399377,10cyclictest141rcu_preempt23:30:150
384705399374,11cyclictest3862330-21tr19:50:150
3847053993730,4cyclictest4497-21PK-Backend23:35:200
384705399372,9cyclictest3928896-21cron23:00:000
3847053993728,5cyclictest4497-21PK-Backend21:25:240
3847053993722,11cyclictest13-21ksoftirqd/021:21:540
3847053993720,12cyclictest438-20systemd-journal22:15:000
3847053993720,12cyclictest3902268-21pluginstate21:40:260
3847053993719,14cyclictest3936803-21df23:20:150
384705399371,8cyclictest3913759-21cpuspeed_turbos22:15:130
3847053993712,5cyclictest141rcu_preempt19:25:020
384705399371,11cyclictest924-21NetworkManager19:34:490
3847055993626,6cyclictest3909862-21sh22:04:591
3847055993626,6cyclictest3909862-21sh22:04:591
3847055993621,11cyclictest3875395-21pluginstate20:25:271
3847055993620,12cyclictest3864644-21ntp_kernel_pll_19:55:231
3847055993620,12cyclictest3864644-21ntp_kernel_pll_19:55:231
3847055993620,11cyclictest438-20systemd-journal22:20:011
384705399367,6cyclictest141rcu_preempt19:40:000
3847053993628,4cyclictest4497-21PK-Backend21:51:020
384705399361,11cyclictest141rcu_preempt21:05:250
384705399361,10cyclictest13-21ksoftirqd/022:01:110
384705399361,10cyclictest13-21ksoftirqd/022:01:110
39385842350,8chrt4497-21PK-Backend23:25:151
384705599357,5cyclictest141rcu_preempt23:40:311
384705599357,5cyclictest141rcu_preempt21:45:141
384705599356,5cyclictest141rcu_preempt22:20:131
384705599355,6cyclictest141rcu_preempt00:30:291
384705599353,9cyclictest26-21ksoftirqd/123:53:201
3847055993527,4cyclictest4497-21PK-Backend22:27:011
3847055993521,10cyclictest3931437-21cut23:05:221
3847055993520,11cyclictest3923012-21open_files22:40:251
384705599351,9cyclictest0-21swapper/123:23:061
3847055993519,12cyclictest976-21rs:main1
3847055993518,13cyclictest3876927-21memory20:30:221
384705599351,6cyclictest3873031-21tr20:20:131
384705599351,10cyclictest4497-21PK-Backend21:10:141
384705399353,7cyclictest0-21swapper/000:25:210
3847053993527,4cyclictest4497-21PK-Backend23:45:170
3847053993521,10cyclictest3910917-21awk22:05:250
384705399351,9cyclictest4497-21PK-Backend20:20:210
3847053993519,12cyclictest3891882-21awk21:10:270
3847053993517,13cyclictest438-20systemd-journal19:50:010
384705399351,14cyclictest3935416-21netstat23:15:220
384705399351,11cyclictest3922408-21tr22:40:120
3847053993510,5cyclictest141rcu_preempt00:08:490
384705599346,5cyclictest141rcu_preempt19:27:581
3847055993428,4cyclictest0-21swapper/121:24:471
3847055993427,4cyclictest4497-21PK-Backend00:21:281
3847055993427,4cyclictest4497-21PK-Backend00:14:351
3847055993423,7cyclictest3943128-21sh23:40:001
3847055993420,10cyclictest3933880-21sed23:10:301
384705599341,9cyclictest4497-21PK-Backend19:40:111
384705599341,9cyclictest4497-21PK-Backend19:15:221
3847055993417,13cyclictest3963791-21df_inode00:35:151
384705599341,30cyclictest4497-21PK-Backend20:35:221
384705599341,30cyclictest0-21swapper/120:15:221
3847055993410,5cyclictest141rcu_preempt20:52:051
384705399341,9cyclictest0-21swapper/019:11:120
384705399341,8cyclictest3887055-21cron21:00:000
3847053993418,5cyclictest13-21ksoftirqd/023:29:550
3847053993418,12cyclictest3918791-21apt22:30:110
3847053993410,5cyclictest141rcu_preempt23:42:330
384705599337,5cyclictest141rcu_preempt19:45:211
384705599336,5cyclictest141rcu_preempt22:47:551
384705599336,5cyclictest141rcu_preempt21:00:181
384705599335,5cyclictest141rcu_preempt23:35:031
3847055993326,4cyclictest4497-21PK-Backend20:12:331
384705599332,5cyclictest4497-21PK-Backend20:05:181
384705599332,26cyclictest3907049-21df_inode21:55:161
384705599331,9cyclictest4497-21PK-Backend19:24:481
3847055993318,11cyclictest3945538-21sshd23:45:181
384705599331,4cyclictest4497-21PK-Backend22:50:471
384705599331,4cyclictest0-21swapper/119:31:351
384705599331,10cyclictest4497-21PK-Backend20:45:281
384705399332,11cyclictest3870879-21apt20:15:010
3847053993320,9cyclictest3932750-21mktemp23:10:020
3847053993316,12cyclictest3880213-21hddtemp_smartct20:40:200
384705599327,5cyclictest141rcu_preempt23:20:011
384705599326,5cyclictest141rcu_preempt22:35:131
384705599324,5cyclictest141rcu_preempt19:50:141
3847055993225,4cyclictest4497-21PK-Backend21:08:311
384705599321,9cyclictest4497-21PK-Backend20:56:261
384705599321,9cyclictest4497-21PK-Backend20:40:141
384705599321,9cyclictest4497-21PK-Backend00:26:241
384705599321,6cyclictest4497-21PK-Backend21:38:171
384705599321,6cyclictest4497-21PK-Backend21:38:171
384705399325,5cyclictest141rcu_preempt22:36:220
3847053993225,4cyclictest4497-21PK-Backend21:39:100
3847053993225,4cyclictest4497-21PK-Backend21:39:100
3847053993219,9cyclictest3925483-21wc22:50:020
3847053993218,10cyclictest3926281-21sed22:50:210
384705399321,11cyclictest0-21swapper/019:42:160
3847053993117,11cyclictest3868281-21missed_timers20:05:220
384705599294,5cyclictest141rcu_preempt00:09:511
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional