You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-31 - 02:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Sun Aug 31, 2025 00:40:35)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
347698499558,10cyclictest171rcu_preempt21:58:271
347698499558,10cyclictest171rcu_preempt21:58:271
347698399533,45cyclictest151ktimers/021:38:300
3476985995243,6cyclictest3479212-21munin-plugin-st19:55:012
3476984994939,6cyclictest959-21mta-sts-daemon20:53:251
3476985994738,6cyclictest3533842-21munin-run22:00:012
3476985994738,6cyclictest3533842-21munin-run22:00:012
3476985994637,6cyclictest422-21dbus-daemon19:20:012
3476985994536,5cyclictest3619685-21sendmail00:00:012
3476985994428,10cyclictest266-21systemd-journal22:10:582
3476984994436,5cyclictest3525015-21fschecks_count21:48:181
3476985994335,5cyclictest3558951-21systemd-run22:35:022
3476985994331,9cyclictest3587401-21cut23:15:022
3476984994335,5cyclictest3505381-21idleruntime-cro21:20:011
3476984994335,5cyclictest3505381-21idleruntime-cro21:20:011
3476983994336,4cyclictest3512454-21systemd-run21:30:010
3476983994334,7cyclictest422-21dbus-daemon00:00:010
3476983994334,6cyclictest3600738-21sendmail_mailtr23:33:330
3476983994323,10cyclictest151ktimers/023:43:260
3476985994235,4cyclictest0-21swapper/223:00:012
3476985994235,3cyclictest0-21swapper/200:33:132
347698599423,30cyclictest3525425-21ssh21:48:252
3476984994236,3cyclictest0-21swapper/122:58:231
3476984994233,5cyclictest1-21systemd22:40:021
3476985994134,4cyclictest0-21swapper/223:05:022
3476985994133,5cyclictest3605360-21awk23:40:012
3476984994132,5cyclictest3534522-21run-parts22:01:011
3476983994133,5cyclictest422-21dbus-daemon23:20:010
3476983994126,12cyclictest3508288-21head21:23:400
347698399411,31cyclictest3539764-21ssh22:08:280
3476984994033,4cyclictest0-21swapper/122:10:011
3476984994032,3cyclictest0-21swapper/100:04:021
3476984994030,7cyclictest3619805-21latency_hist00:00:031
3476983994034,4cyclictest3562635-21latency_hist22:40:030
3476983994033,4cyclictest3615370-21sed23:53:360
3476983994032,5cyclictest3572562-21sed22:53:390
347698599395,26cyclictest3611224-21ssh23:48:222
3476985993929,6cyclictest3593018-21kernelversion23:23:202
347698599392,29cyclictest3631951-21ssh00:18:272
3476985993831,3cyclictest0-21swapper/221:38:182
3476985993827,9cyclictest3560888-21sh22:38:192
347698499384,26cyclictest3514760-21ssh21:33:291
3476984993828,7cyclictest3598858-21sshd23:31:091
3476984993824,8cyclictest0-21swapper/100:33:251
3476983993826,2cyclictest0-21swapper/022:33:200
3476985993728,6cyclictest421-21avahi-daemon21:31:412
3476985993728,6cyclictest3489522-21fschecks_time20:43:182
3476985993728,6cyclictest1-21systemd00:15:022
3476985993726,8cyclictest3522270-21switchtime21:43:392
347698499373,29cyclictest3604176-21ssh23:38:271
347698499373,28cyclictest3589702-21ssh23:18:251
3476984993730,4cyclictest3489088-21latency_hist20:40:031
3476984993729,4cyclictest0-21swapper/123:23:271
3476984993728,4cyclictest0-21swapper/121:13:301
347698399375,8cyclictest171rcu_preempt22:48:290
3476983993730,4cyclictest3548217-21latency_hist22:20:020
3476983993722,11cyclictest0-21swapper/000:18:400
347698599363,26cyclictest3589801-21ssh23:18:292
3476985993631,2cyclictest0-21swapper/222:30:012
3476985993628,5cyclictest3494955-21munin-run21:00:012
3476985993627,6cyclictest3480749-21true20:15:012
3476985993627,6cyclictest266-21systemd-journal22:45:012
3476985993626,6cyclictest957-21mta-sts-daemon21:13:292
3476985993626,6cyclictest0-21swapper/223:33:252
3476985993624,9cyclictest3530280-21munin-plugin-st21:55:022
3476985993624,9cyclictest3485298-21needreboot20:28:222
3476984993629,3cyclictest0-21swapper/122:18:291
3476984993627,6cyclictest1713136-21cupsd23:45:491
3476984993627,4cyclictest0-21swapper/100:08:421
3476983993628,5cyclictest3555381-21unin-run22:30:020
3476983993624,8cyclictest151ktimers/021:48:270
3476985993528,3cyclictest0-21swapper/223:08:372
347698599352,5cyclictest0-21swapper/222:03:252
347698599352,27cyclictest3547032-21ssh22:18:312
3476985993515,14cyclictest421-21avahi-daemon21:23:252
3476984993529,3cyclictest0-21swapper/123:05:011
3476984993526,6cyclictest421-21avahi-daemon22:51:431
3476984993526,6cyclictest3544583-21cut22:15:011
3476984993523,8cyclictest422-21dbus-daemon21:40:011
3476983993527,5cyclictest3533831-21cron22:00:010
3476983993527,5cyclictest3533831-21cron22:00:010
3476983993521,11cyclictest3537484-21idleruntime-cro22:05:010
347698599343,22cyclictest3643776-21awk00:38:252
3476985993427,4cyclictest957-21mta-sts-daemon20:53:262
3476985993427,3cyclictest0-21swapper/220:33:282
3476985993426,5cyclictest3504011-21needreboot21:18:202
3476985993426,5cyclictest3504011-21needreboot21:18:202
3476985993425,5cyclictest3479457-21cron20:00:012
3476985993424,8cyclictest3550355-21missed_timers22:23:232
3476985993424,7cyclictest421-21avahi-daemon20:18:132
347698599341,25cyclictest3572130-21ssh22:53:332
3476984993425,6cyclictest421-21avahi-daemon22:44:541
3476983993428,4cyclictest3503684-21fschecks_count21:18:160
3476983993428,4cyclictest3503684-21fschecks_count21:18:160
3476983993422,8cyclictest0-21swapper/023:08:140
3476983993420,3cyclictest0-21swapper/021:05:020
347698599335,22cyclictest3622254-21ssh00:03:592
3476985993327,3cyclictest0-21swapper/222:08:222
3476985993327,3cyclictest0-21swapper/220:50:032
3476985993316,9cyclictest171rcu_preempt21:08:282
3476984993327,3cyclictest0-21swapper/123:53:251
3476984993326,4cyclictest3608947-21cron23:45:011
3476983993329,2cyclictest0-21swapper/022:15:010
3476983993328,3cyclictest956-21mta-sts-daemon22:23:350
3476983993325,5cyclictest421-21avahi-daemon21:31:540
3476983993325,5cyclictest3621301-21/usr/sbin/munin00:03:200
3476983993325,5cyclictest1713138-21cups-browsed22:55:240
347698399332,24cyclictest3521940-21ntpq21:43:290
3476983993320,6cyclictest12450irq/32-eth000:23:290
347698599324,22cyclictest3625477-21ssh00:08:352
3476985993226,2cyclictest0-21swapper/223:53:362
3476985993225,4cyclictest0-21swapper/200:23:212
3476985993225,3cyclictest0-21swapper/223:43:272
3476985993224,5cyclictest3496465-21cron21:05:012
3476985993224,4cyclictest3479717-21cron20:05:012
347698499324,21cyclictest3582670-21ssh23:08:271
3476984993224,3cyclictest0-21swapper/121:45:011
3476984993223,6cyclictest1713138-21cups-browsed21:26:451
3476984993223,6cyclictest1713138-21cups-browsed19:20:591
3476984993223,6cyclictest0-21swapper/119:35:011
3476984993223,4cyclictest0-21swapper/122:33:291
3476984993223,4cyclictest0-21swapper/121:53:291
3476983993225,4cyclictest955-21mta-sts-daemon20:53:250
3476985993123,5cyclictest3638352-21mailstats00:28:342
3476985993123,4cyclictest0-21swapper/223:28:202
347698599311,6cyclictest0-21swapper/221:28:312
347698499314,22cyclictest3644126-21ssh00:38:341
347698499313,21cyclictest3632044-21ssh00:18:301
3476984993125,3cyclictest0-21swapper/122:25:011
3476984993124,4cyclictest3477288-21latency_hist19:15:021
3476984993124,3cyclictest0-21swapper/122:48:211
347698399313,23cyclictest3500588-21ssh21:13:290
347698399312,23cyclictest3644126-21ssh00:38:340
3476983993113,12cyclictest3628799-21ssh00:13:370
3476985993023,3cyclictest2998741-21kworker/u8:8-rpciod20:10:012
3476985993022,5cyclictest3544607-21kworker/u8:3-xprtiod22:48:282
3476984993025,2cyclictest0-21swapper/120:20:011
3476984993024,3cyclictest0-21swapper/123:28:231
3476984993022,6cyclictest3476982-21cyclictest21:21:531
3476984993022,5cyclictest1713138-21cups-browsed23:12:041
3476984993022,5cyclictest0-21swapper/119:20:001
347698499302,22cyclictest3628588-21ssh00:13:271
3476983993024,4cyclictest3611316-21ssh23:48:260
3476983993024,4cyclictest3490460-21cron20:45:010
3476983993023,4cyclictest3605361-21basename23:40:010
3476983993012,14cyclictest151ktimers/020:48:240
3476985992922,4cyclictest2775360-21kworker/u8:10-rpciod19:25:012
347698499293,12cyclictest3497964-21diskmemload22:25:521
3476984992924,2cyclictest0-21swapper/120:00:011
3476984992922,4cyclictest957-21mta-sts-daemon20:48:231
3476984992921,5cyclictest3484498-21systemd-run20:25:011
3476984992921,5cyclictest3484498-21systemd-run20:25:011
3476983992923,4cyclictest3638235-21sh00:28:310
3476983992923,4cyclictest3487397-21head20:33:400
3476983992923,3cyclictest453-21systemd-logind19:55:010
347698399292,23cyclictest3564906-21ssh22:43:290
347698399292,21cyclictest3625355-21ssh00:08:300
3476985992820,5cyclictest0-21swapper/219:30:012
3476984992822,3cyclictest0-21swapper/100:23:391
3476984992821,4cyclictest3478731-21munin-plugin-st19:45:011
3476984992820,5cyclictest421-21avahi-daemon20:29:151
347698499281,21cyclictest3638057-21ssh00:28:241
3476983992822,3cyclictest3488991-21kworker/u8:3-xprtiod21:00:020
347698399282,21cyclictest3586357-21awk23:13:330
3476983992821,4cyclictest3478730-21munin-plugin-st19:45:010
3476983992820,5cyclictest3484467-21cron20:25:010
3476983992820,5cyclictest3484467-21cron20:25:010
3476983992820,5cyclictest1713136-21cupsd21:50:120
3476983992818,7cyclictest12450irq/32-eth023:03:240
347698399280,3cyclictest171rcu_preempt21:08:390
3476985992722,2cyclictest0-21swapper/219:15:012
3476985992721,3cyclictest0-21swapper/220:40:022
3476985992719,5cyclictest3478475-21basename19:40:012
3476984992723,2cyclictest0-21swapper/119:40:021
3476984992721,3cyclictest0-21swapper/121:08:221
3476984992718,6cyclictest421-21avahi-daemon19:28:301
3476983992721,3cyclictest0-21swapper/023:23:270
3476983992720,4cyclictest3479509-21latency_hist20:00:020
3476983992719,5cyclictest3473459-21kworker/u8:6-xprtiod20:18:390
3476983992718,7cyclictest3640943-21missed_timers00:33:190
3476983992718,4cyclictest3487989-21fschecks_count20:38:170
3476984992620,3cyclictest0-21swapper/120:33:281
3476984992619,5cyclictest0-21swapper/121:00:011
3476984992617,6cyclictest3480736-21/usr/sbin/munin20:15:011
3476984992516,6cyclictest3496413-21unixbench_singl21:03:331
3476983992519,4cyclictest3596961-21ntpq23:28:280
347698599241,7cyclictest171rcu_preempt20:23:292
347698599241,7cyclictest171rcu_preempt20:23:292
3476983992419,3cyclictest3485853-21unixbench_multi20:28:360
3476985992316,5cyclictest3479063-21cat19:50:022
3476985992314,6cyclictest3478240-21systemd-run19:35:022
347698499234,14cyclictest3490040-21ps20:43:311
3476984992314,6cyclictest491-21gdbus20:02:381
3476983992318,2cyclictest0-21swapper/019:50:010
3476983992317,4cyclictest422-21dbus-daemon19:35:020
3476985992216,3cyclictest3478242-21kworker/u8:3+xprtiod19:45:012
3476984992214,5cyclictest1713138-21cups-browsed19:49:471
3476983992216,4cyclictest1713138-21cups-browsed20:03:110
3476984992114,4cyclictest1713138-21cups-browsed20:08:201
347698699201,9cyclictest0-21swapper/300:04:003
3476983992015,2cyclictest0-21swapper/020:15:010
3476983992013,5cyclictest0-21swapper/019:20:010
3476983992010,7cyclictest0-21swapper/019:40:010
347698699193,9cyclictest0-21swapper/323:43:273
347698699192,8cyclictest0-21swapper/321:43:253
347698699192,10cyclictest0-21swapper/323:53:293
347698699192,10cyclictest0-21swapper/322:08:343
347698699192,10cyclictest0-21swapper/300:33:273
347698699191,11cyclictest0-21swapper/323:58:253
347698699191,10cyclictest0-21swapper/300:18:303
3476985991913,3cyclictest0-21swapper/219:10:012
3476985991913,3cyclictest0-21swapper/219:10:012
3476984991913,3cyclictest0-21swapper/119:55:011
3476984991913,3cyclictest0-21swapper/119:10:031
3476984991913,3cyclictest0-21swapper/119:10:031
347698399197,10cyclictest12550irq/33-eth019:28:430
3476983991914,2cyclictest0-21swapper/019:15:010
3476983991912,5cyclictest3477748-21systemd-run19:25:010
347698699183,8cyclictest0-21swapper/300:23:263
347698699182,9cyclictest0-21swapper/323:38:233
347698699181,9cyclictest0-21swapper/323:03:203
347698699181,10cyclictest0-21swapper/322:58:253
347698699174,7cyclictest0-21swapper/323:48:223
347698699173,9cyclictest0-21swapper/322:18:333
347698699173,8cyclictest0-21swapper/322:03:253
347698699173,8cyclictest0-21swapper/321:48:233
347698699172,9cyclictest0-21swapper/321:13:273
347698699172,10cyclictest0-21swapper/300:08:303
347698699171,9cyclictest0-21swapper/322:23:273
347698699171,8cyclictest0-21swapper/321:33:273
347698699171,7cyclictest0-21swapper/320:28:253
347698699171,10cyclictest0-21swapper/322:53:273
347698399171,8cyclictest3479968-21latency_hist20:10:010
347698399171,13cyclictest3476933-21latency_hist19:10:040
347698399171,13cyclictest3476933-21latency_hist19:10:040
347698699165,7cyclictest0-21swapper/320:48:223
347698699164,5cyclictest0-21swapper/320:58:283
347698699163,7cyclictest0-21swapper/322:28:233
347698699162,9cyclictest0-21swapper/322:13:273
347698699162,7cyclictest0-21swapper/321:38:243
347698699162,7cyclictest0-21swapper/300:38:253
347698699161,9cyclictest0-21swapper/323:33:233
347698699161,9cyclictest0-21swapper/321:23:333
347698699161,8cyclictest0-21swapper/323:13:263
347698699161,7cyclictest0-21swapper/321:08:273
347698699161,11cyclictest0-21swapper/300:13:273
347698699153,7cyclictest0-21swapper/323:18:243
347698699152,8cyclictest0-21swapper/323:08:243
347698699152,8cyclictest0-21swapper/322:43:203
347698699152,8cyclictest0-21swapper/321:57:563
347698699152,8cyclictest0-21swapper/321:57:563
347698699152,8cyclictest0-21swapper/320:33:283
347698699152,7cyclictest0-21swapper/321:28:233
347698699151,9cyclictest0-21swapper/323:23:243
347698699151,8cyclictest0-21swapper/322:38:233
347698699143,6cyclictest0-21swapper/322:33:193
347698699142,6cyclictest0-21swapper/320:23:243
347698699142,6cyclictest0-21swapper/320:23:243
347698699141,8cyclictest0-21swapper/323:28:233
347698699141,8cyclictest0-21swapper/321:53:263
347698699141,8cyclictest0-21swapper/321:18:213
347698699141,8cyclictest0-21swapper/321:18:213
347698699141,7cyclictest0-21swapper/320:43:243
347698699141,7cyclictest0-21swapper/320:15:163
347698699141,6cyclictest0-21swapper/321:03:203
347698699132,5cyclictest0-21swapper/300:26:163
347698699131,9cyclictest0-21swapper/322:48:263
347698699131,6cyclictest0-21swapper/320:38:223
347698699126,3cyclictest0-21swapper/320:53:283
347698699122,6cyclictest0-21swapper/319:20:003
347698699122,5cyclictest0-21swapper/319:55:013
347698699121,7cyclictest0-21swapper/320:10:013
347698699121,6cyclictest0-21swapper/319:30:013
347698699111,6cyclictest0-21swapper/320:05:013
347698699111,6cyclictest0-21swapper/319:40:013
347698699101,5cyclictest0-21swapper/319:10:223
34769869992,4cyclictest0-21swapper/319:50:013
34769869991,6cyclictest0-21swapper/319:10:013
34769869991,6cyclictest0-21swapper/319:10:013
34769869991,5cyclictest0-21swapper/320:13:013
34769869991,5cyclictest0-21swapper/320:00:013
34769869991,5cyclictest0-21swapper/319:45:013
34769869991,5cyclictest0-21swapper/319:35:013
34769869981,5cyclictest0-21swapper/319:20:363
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional