You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 03:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Wed Dec 03, 2025 00:39:40)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3510681996355,5cyclictest3554400-21munin-plugin-st21:20:011
3510682995541,7cyclictest0-21swapper/223:44:172
3510680995343,4cyclictest0-21swapper/022:19:200
3510682995142,5cyclictest0-21swapper/222:54:182
3510681995142,5cyclictest3614714-21ssh22:40:011
351068199511,30cyclictest3594960-21ssh22:14:231
3510682995043,4cyclictest3577240-21kworker/u8:4-xprtiod22:04:302
3510680995035,11cyclictest3629723-21cron23:00:010
3510682994837,7cyclictest959-21mta-sts-daemon23:40:012
3510681994835,9cyclictest3685973-21cron00:15:011
351068099483,38cyclictest3561007-21ssh21:29:230
3510680994833,12cyclictest698-21cron23:35:010
3510680994822,22cyclictest3546724-21systemd-run21:10:010
3510680994822,22cyclictest3546724-21systemd-run21:10:010
3510682994737,5cyclictest0-21swapper/219:35:002
3510682994736,5cyclictest0-21swapper/222:34:232
3510682994736,5cyclictest0-21swapper/222:34:232
351068099474,24cyclictest3663082-21ntpq23:44:200
3510680994738,6cyclictest151ktimers/020:14:250
3510680994738,6cyclictest151ktimers/020:14:250
3510680994720,24cyclictest3640328-21ssh23:14:250
3510682994633,9cyclictest3647717-21netstat23:24:152
3510680994635,7cyclictest0-21swapper/022:34:230
3510680994635,7cyclictest0-21swapper/022:34:230
3510680994631,12cyclictest3652499-21munin-plugin-st23:30:010
3510682994537,5cyclictest3656326-21grep23:35:012
3510681994536,6cyclictest3628243-21aten_r2power_po22:59:091
3510681994535,6cyclictest3545322-21latency_hist21:05:021
3510681994535,6cyclictest3545322-21latency_hist21:05:021
3510680994534,4cyclictest3696859-21latency_hist00:30:030
351068099453,35cyclictest657-21snmpd22:39:240
351068099451,27cyclictest3635140-21ssh23:07:590
3510682994429,11cyclictest453-21systemd-logind23:15:022
3510682994414,9cyclictest171rcu_preempt00:09:222
351068199445,33cyclictest3643976-21ssh23:19:181
351068199445,25cyclictest3549783-21ssh21:14:181
351068199445,25cyclictest3549783-21ssh21:14:181
3510681994437,3cyclictest0-21swapper/121:25:011
3510681994436,5cyclictest3565542-21cron21:35:011
3510681994436,4cyclictest0-21swapper/121:59:251
3510681994419,17cyclictest422-21dbus-daemon20:34:201
351068099449,31cyclictest3538652-21ntpq20:44:220
351068099443,32cyclictest3554174-21ssh21:19:320
351068099440,40cyclictest3568499-21ssh21:39:220
3510682994335,4cyclictest0-21swapper/200:00:032
3510682994334,5cyclictest3513664-21idleruntime-cro19:20:012
3510681994335,5cyclictest3699554-21mailstats00:34:251
3510680994335,5cyclictest3573127-21systemd-run21:45:010
351068099433,22cyclictest3549907-21ntp_states21:14:240
351068099433,22cyclictest3549907-21ntp_states21:14:240
3510680994319,20cyclictest0-21swapper/020:55:010
351068099431,40cyclictest3701954-21ssh00:38:410
351068099431,36cyclictest3520557-21ntpq19:44:180
3510681994234,5cyclictest3652522-21systemd-run23:30:011
3510681994233,5cyclictest0-21swapper/100:39:231
3510680994233,5cyclictest3620938-21sh22:49:110
351068099421,37cyclictest3692298-21ntpq00:24:220
351068099421,37cyclictest3544719-21ntpq21:04:250
351068099421,37cyclictest3544719-21ntpq21:04:250
3510682994133,5cyclictest3622274-21latency_hist22:50:012
3510682994128,5cyclictest3667204-21threads23:49:292
3510681994129,9cyclictest3590851-21latency22:09:131
351068099417,31cyclictest3673431-21sh23:58:380
3510680994132,5cyclictest151ktimers/023:49:180
351068099412,34cyclictest3522171-21ntpq19:49:220
351068099412,34cyclictest3516177-21ntpq19:29:240
351068099412,34cyclictest3516177-21ntpq19:29:240
3510680994119,19cyclictest3576103-21ntpq21:49:220
3510680994115,23cyclictest0-21swapper/023:55:020
351068099411,37cyclictest3605230-21ssh22:28:320
351068099411,37cyclictest3605230-21ssh22:28:320
351068099411,36cyclictest3523587-21ntpq19:54:210
351068099411,35cyclictest3513161-21ntpq19:19:230
351068099411,35cyclictest3511827-21swap19:14:290
3510680994110,29cyclictest3657832-21ssh23:37:460
3510682994032,5cyclictest3531667-21idleruntime-cro20:20:012
3510682994032,5cyclictest0-21swapper/221:00:012
3510682994032,5cyclictest0-21swapper/221:00:012
3510682994030,7cyclictest421-21avahi-daemon22:58:022
3510682994024,3cyclictest171rcu_preempt21:50:012
351068199403,29cyclictest3692255-21ssh00:24:201
3510681994032,5cyclictest3602040-21ssh22:24:121
3510681994032,5cyclictest3602040-21ssh22:24:121
3510681994032,5cyclictest3546676-21head21:09:321
3510681994032,5cyclictest3546676-21head21:09:321
3510681994031,4cyclictest0-21swapper/100:29:261
351068099407,29cyclictest3685143-21ntpq00:14:210
351068099401,37cyclictest3563829-21ssh21:33:540
351068099401,36cyclictest3680868-21/usr/sbin/munin00:09:370
351068099401,35cyclictest3534143-21ntpq20:29:230
351068099401,35cyclictest3531163-21ntpq20:19:240
351068099401,34cyclictest3535622-21ntpq20:34:200
351068099401,34cyclictest3532594-21ntpq20:24:200
351068099401,34cyclictest3528031-21ntpq20:09:160
351068099401,33cyclictest3519062-21ntpq19:39:200
351068099400,36cyclictest3593720-21ssh22:13:140
3510682993931,5cyclictest3633222-21unixbench_multi23:04:322
3510681993931,5cyclictest3576934-21systemd-run21:50:011
3510681993930,6cyclictest1-21systemd20:30:021
351068099392,33cyclictest3557371-21ntpq21:24:260
351068099391,35cyclictest3623448-21ssh22:52:060
351068099391,35cyclictest3588064-21ssh22:04:380
351068099391,35cyclictest3579874-21ntpq21:54:240
351068099391,35cyclictest3540072-21ntpq20:49:200
351068099391,34cyclictest3697649-21ssh00:31:520
351068099391,34cyclictest3617534-21ntpq22:44:170
351068099391,34cyclictest3525190-21ps19:59:260
351068099391,33cyclictest3543135-21ntpq20:59:190
351068099391,33cyclictest3543135-21ntpq20:59:190
3510682993830,5cyclictest3595772-21cron22:15:012
3510682993830,4cyclictest0-21swapper/222:39:242
3510682993829,5cyclictest3644881-21cron23:20:012
3510682993829,4cyclictest0-21swapper/200:39:182
3510682993827,7cyclictest1662271-21cupsd22:26:172
3510682993827,7cyclictest1662271-21cupsd22:26:172
3510682993826,7cyclictest3590999-21needreboot22:09:142
351068299382,31cyclictest1690079-21ntpd21:29:242
3510681993830,5cyclictest3689567-21idleruntime-cro00:20:011
3510681993830,5cyclictest3675314-21cron00:00:011
3510681993830,5cyclictest3617033-21aten2.4-expect22:44:091
351068099382,32cyclictest3602458-21ssh22:24:190
351068099382,32cyclictest3602458-21ssh22:24:190
351068099381,34cyclictest3645748-21ssh23:21:200
351068099381,34cyclictest3644086-21ntpq23:19:220
351068099381,34cyclictest3514571-21ntp_kernel_pll_19:24:200
351068099381,33cyclictest3677430-21ssh00:04:000
351068099381,33cyclictest3630698-21ssh23:01:340
351068099381,33cyclictest3517287-21/usr/sbin/munin19:34:120
351068099381,32cyclictest3688693-21ntpq00:19:200
351068099381,32cyclictest3537092-21ntpq20:39:240
351068099381,32cyclictest3526573-21ntpq20:04:190
351068099380,36cyclictest3582426-21ssh21:58:170
3510682993729,4cyclictest0-21swapper/200:30:012
3510682993727,6cyclictest3554397-21sendmail21:20:012
3510682993727,4cyclictest0-21swapper/219:59:182
351068299371,28cyclictest3699305-21ssh00:34:172
3510681993729,5cyclictest3663572-21unixbench-2d23:44:301
3510681993729,5cyclictest3598173-21cpuspeed22:19:111
3510681993729,5cyclictest3539079-21tail20:44:291
3510681993729,5cyclictest3517212-21fschecks_time19:34:111
3510681993727,6cyclictest1-21systemd19:45:021
351068099371,32cyclictest3591913-21ssh22:09:490
3510682993630,3cyclictest3533406-21kworker/u8:5-rpciod21:25:012
3510682993628,5cyclictest0-21swapper/220:55:012
3510682993627,6cyclictest3519662-21basename19:40:012
3510682993627,6cyclictest0-21swapper/223:54:232
3510682993627,4cyclictest0-21swapper/200:05:012
3510682993626,6cyclictest3530150-21sh20:15:012
3510682993626,6cyclictest3530150-21sh20:15:012
3510682993623,10cyclictest3651932-21switchtime23:29:282
3510681993629,4cyclictest3528730-21latency_hist20:10:021
3510681993628,5cyclictest3561558-21unixbench_singl21:29:331
3510681993628,5cyclictest3510691-21systemd-run19:10:011
3510681993627,5cyclictest3655705-21sendmail_mailtr23:34:291
3510681993627,5cyclictest3624966-21missed_timers22:54:151
3510682993526,6cyclictest421-21avahi-daemon21:51:372
3510681993529,4cyclictest3510679-21cyclictest23:01:401
3510681993528,4cyclictest3640783-21unixbench_multi23:14:311
3510681993527,5cyclictest3541253-21fschecks_time20:54:131
3510681993526,6cyclictest3513699-21systemd-run19:20:011
3510681993524,8cyclictest3659125-21needreboot23:39:181
351068099351,31cyclictest3510692-21munin-run19:10:020
351068299346,21cyclictest3572215-21ssh21:44:202
3510682993426,4cyclictest0-21swapper/222:20:022
3510682993423,7cyclictest1662271-21cupsd21:56:462
3510682993422,8cyclictest266-21systemd-journal21:11:152
3510682993422,8cyclictest266-21systemd-journal21:11:152
351068299342,25cyclictest3602599-21rm22:24:252
351068299342,25cyclictest3602599-21rm22:24:252
351068199343,23cyclictest3572167-21ssh21:44:191
3510681993427,4cyclictest957-21mta-sts-daemon23:24:211
3510681993425,6cyclictest421-21avahi-daemon22:29:021
3510681993425,6cyclictest421-21avahi-daemon22:29:021
3510681993425,6cyclictest3527182-21systemd-run20:05:011
3510681993424,7cyclictest3587099-21latency22:04:141
3510682993326,4cyclictest956-21mta-sts-daemon21:38:482
3510682993326,4cyclictest956-21mta-sts-daemon00:10:152
3510682993323,7cyclictest3521170-21/usr/sbin/munin19:45:022
351068299332,24cyclictest3617707-21sh22:44:232
3510681993327,3cyclictest0-21swapper/123:49:201
3510681993326,4cyclictest0-21swapper/121:39:191
3510681993325,5cyclictest3637011-21unixbench_multi23:09:311
3510682993224,5cyclictest3546695-21cron21:10:002
3510682993224,5cyclictest3546695-21cron21:10:002
3510682993224,5cyclictest3537694-21systemd20:40:022
3510682993223,6cyclictest3522661-21idleruntime-cro19:50:012
351068299321,24cyclictest3564679-21sh21:34:202
3510681993224,5cyclictest3537693-21systemd20:40:021
3510681993224,5cyclictest1662287-21cups-browsed19:26:181
3510681993224,5cyclictest1662287-21cups-browsed19:26:181
3510681993221,7cyclictest1662271-21cupsd21:53:491
3510682993123,5cyclictest3637486-21latency_hist23:10:022
3510682993122,5cyclictest3510693-21systemd19:10:022
351068299312,21cyclictest3546706-21diskmemload00:19:212
3510681993125,3cyclictest0-21swapper/100:05:011
3510682993022,5cyclictest421-21avahi-daemon00:21:182
3510682993022,5cyclictest3540675-21systemd-run20:50:012
3510681993024,3cyclictest0-21swapper/122:49:151
3510681993023,3cyclictest0-21swapper/100:09:151
3510681993022,5cyclictest3543662-21unixbench_singl20:59:281
3510681993022,5cyclictest3543662-21unixbench_singl20:59:281
3510681993020,6cyclictest3671740-21latency_hist23:55:031
3510682992923,3cyclictest0-21swapper/219:30:012
3510682992923,3cyclictest0-21swapper/219:30:012
3510682992922,3cyclictest0-21swapper/221:05:012
3510682992922,3cyclictest0-21swapper/221:05:012
3510682992920,4cyclictest0-21swapper/220:29:212
351068199293,21cyclictest3610096-21ssh22:34:221
351068199293,21cyclictest3610096-21ssh22:34:221
3510681992923,3cyclictest0-21swapper/119:55:011
3510682992822,3cyclictest0-21swapper/220:25:012
3510682992820,5cyclictest1662287-21cups-browsed20:44:142
3510681992822,3cyclictest0-21swapper/120:50:021
3510681992820,5cyclictest3533163-21cron20:25:001
3510681992819,6cyclictest421-21avahi-daemon19:14:271
3510682992720,4cyclictest957-21mta-sts-daemon20:04:192
3510682992720,4cyclictest3516769-21kworker/u8:7-xprtiod19:54:312
3510682992719,5cyclictest3535416-21missed_timers20:34:142
3510682992715,9cyclictest3528085-21ntp_kernel_pll_20:09:192
3510681992719,4cyclictest0-21swapper/119:59:171
3510681992717,7cyclictest956-21mta-sts-daemon19:39:211
3510681992516,6cyclictest421-21avahi-daemon20:11:021
3510681992516,6cyclictest421-21avahi-daemon20:11:021
351068299242,16cyclictest3514555-21ntpq19:24:192
351068199242,16cyclictest3531659-21munin-run20:20:011
3510681992314,6cyclictest3514754-21awk19:24:261
3510681992216,4cyclictest3510679-21cyclictest19:50:031
3510682992115,3cyclictest0-21swapper/219:15:012
351068399184,9cyclictest0-21swapper/300:24:243
351068399183,9cyclictest0-21swapper/323:59:203
351068399173,8cyclictest0-21swapper/300:04:173
351068399172,9cyclictest0-21swapper/323:54:203
351068399172,9cyclictest0-21swapper/323:34:243
351068399172,9cyclictest0-21swapper/300:09:283
351068399172,8cyclictest0-21swapper/322:09:173
351068399171,9cyclictest0-21swapper/322:49:193
351068399171,8cyclictest0-21swapper/323:04:193
351068399163,7cyclictest0-21swapper/323:00:013
351068399163,7cyclictest0-21swapper/322:54:223
351068399163,7cyclictest0-21swapper/322:20:013
351068399163,7cyclictest0-21swapper/321:24:223
351068399163,7cyclictest0-21swapper/300:29:203
351068399162,8cyclictest0-21swapper/322:44:193
351068399162,8cyclictest0-21swapper/322:39:223
351068399162,8cyclictest0-21swapper/321:39:123
351068399162,6cyclictest0-21swapper/323:47:143
351068399161,9cyclictest0-21swapper/300:34:183
351068399161,7cyclictest0-21swapper/320:04:223
351068399153,7cyclictest0-21swapper/321:29:223
351068399152,8cyclictest0-21swapper/322:14:193
351068399152,8cyclictest0-21swapper/321:44:273
351068399152,8cyclictest0-21swapper/320:34:213
351068399152,8cyclictest0-21swapper/319:40:013
351068399152,8cyclictest0-21swapper/300:14:243
351068399152,7cyclictest0-21swapper/321:19:243
351068399151,8cyclictest0-21swapper/321:54:213
351068399151,7cyclictest0-21swapper/320:14:193
351068399151,7cyclictest0-21swapper/320:14:193
351068399145,7cyclictest0-21swapper/321:59:193
351068399142,8cyclictest0-21swapper/323:39:193
351068399142,8cyclictest0-21swapper/323:24:173
351068399142,8cyclictest0-21swapper/300:39:193
351068399142,8cyclictest0-21swapper/300:19:203
351068399142,7cyclictest0-21swapper/322:24:303
351068399142,7cyclictest0-21swapper/322:24:303
351068399142,7cyclictest0-21swapper/320:19:293
351068399142,6cyclictest0-21swapper/323:14:173
351068399142,6cyclictest0-21swapper/321:49:093
351068399141,9cyclictest0-21swapper/323:44:183
351068399141,9cyclictest0-21swapper/323:09:143
351068399141,8cyclictest0-21swapper/322:29:223
351068399141,8cyclictest0-21swapper/322:29:223
351068399141,8cyclictest0-21swapper/322:03:143
351068399141,6cyclictest0-21swapper/320:44:183
351068399133,7cyclictest0-21swapper/321:34:233
351068399132,7cyclictest0-21swapper/323:19:163
351068399132,7cyclictest0-21swapper/319:24:203
351068399132,6cyclictest0-21swapper/322:33:153
351068399132,6cyclictest0-21swapper/322:33:153
351068399132,6cyclictest0-21swapper/320:24:193
351068399132,6cyclictest0-21swapper/319:34:153
351068399132,6cyclictest0-21swapper/319:14:223
351068399132,5cyclictest0-21swapper/319:29:173
351068399132,5cyclictest0-21swapper/319:29:173
351068399131,8cyclictest0-21swapper/323:29:133
351068399131,7cyclictest0-21swapper/320:40:013
351068399131,6cyclictest0-21swapper/320:09:183
351068399131,6cyclictest0-21swapper/319:59:173
351068399131,6cyclictest0-21swapper/319:54:193
351068399122,6cyclictest0-21swapper/320:55:013
351068399122,6cyclictest0-21swapper/319:20:013
351068399121,7cyclictest0-21swapper/321:14:233
351068399121,7cyclictest0-21swapper/321:14:233
351068399121,7cyclictest0-21swapper/321:10:003
351068399121,7cyclictest0-21swapper/321:10:003
351068399121,7cyclictest0-21swapper/321:04:243
351068399121,7cyclictest0-21swapper/321:04:243
351068399121,6cyclictest0-21swapper/320:29:213
351068399111,6cyclictest0-21swapper/320:59:143
351068399111,6cyclictest0-21swapper/320:59:143
351068399111,6cyclictest0-21swapper/320:49:103
351068399101,5cyclictest0-21swapper/319:49:103
351068399101,5cyclictest0-21swapper/319:44:153
35106839981,4cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional