You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 21:54
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Sun Nov 30, 2025 12:44:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30764213917,82sleep030795-21idleruntime-cro07:08:500
309259910184,11cyclictest20042-21apt-get08:14:052
30925999772,19cyclictest23549-21idleruntime-cro08:23:502
30918999667,19cyclictest30399-21latency_hist08:43:531
30925999555,35cyclictest30904-21cyclictest10:38:512
30925999555,35cyclictest30904-21cyclictest10:38:512
30925999446,41cyclictest11250-21sh12:14:242
30910999453,21cyclictest181rcu_preempt10:05:490
30925999343,41cyclictest26729-21modprobe08:33:502
30925999340,45cyclictest0-21swapper/211:54:042
30910999167,17cyclictest1124-21latency_hist08:53:520
30910999167,17cyclictest1124-21latency_hist08:53:520
30910999054,30cyclictest20409-21sh11:34:090
30910999035,14cyclictest181rcu_preempt11:29:120
30910998967,16cyclictest23096-21sh11:39:100
30925998849,29cyclictest5540-21kworker/u19:0+nfsiod12:25:272
30918998872,11cyclictest20529-21acpi08:14:031
30910998761,18cyclictest2148-21latency_hist07:18:530
30910998761,18cyclictest2148-21latency_hist07:18:530
3091099873,48cyclictest28549-21ssh10:49:220
3091099873,48cyclictest28549-21ssh10:49:220
30918998668,12cyclictest292-21dbus-daemon08:13:511
30918998664,16cyclictest15091-21threads09:24:261
30918998661,18cyclictest691-21cron07:18:501
30918998661,18cyclictest691-21cron07:18:501
30910998656,21cyclictest6610-21kernelversion09:09:160
30925998541,38cyclictest9507-21switchtime07:39:272
30910998550,12cyclictest181rcu_preempt08:14:320
30925998465,13cyclictest32113-21open_files10:54:372
30925998465,13cyclictest32113-21open_files10:54:372
30925998464,14cyclictest7316-21apt12:08:492
30925998464,14cyclictest7316-21apt12:08:492
30918998460,13cyclictest0-21swapper/109:59:131
30918998460,13cyclictest0-21swapper/109:59:131
3091899843,44cyclictest28456-21apt-get10:49:091
3091899843,44cyclictest28456-21apt-get10:49:091
3091899843,35cyclictest17715-21cstates11:29:071
30910998448,9cyclictest181rcu_preempt11:23:210
30910998448,9cyclictest181rcu_preempt11:23:210
3091099843,52cyclictest25854-21memory09:44:160
30925998322,51cyclictest6182-21awk09:09:012
30918998361,16cyclictest11835-21apt-key11:18:521
3091899833,60cyclictest7316-21apt-get12:09:041
3091899833,60cyclictest7316-21apt-get12:09:041
3091899832,62cyclictest11015-21ssh12:14:071
30910998361,14cyclictest207-21systemd-journal08:28:020
3092599826,65cyclictest86-21kswapd010:49:182
3092599826,65cyclictest86-21kswapd010:49:182
30918998265,10cyclictest0-21swapper/108:19:131
30918998262,11cyclictest0-21swapper/111:20:061
30918998262,11cyclictest0-21swapper/111:20:061
30918998259,9cyclictest0-21swapper/109:54:141
30918998244,29cyclictest32696-21kworker/u18:2+nfsiod12:19:591
3091899823,49cyclictest31487-21ssh10:54:141
3091899823,49cyclictest31487-21ssh10:54:141
30910998271,6cyclictest0-21swapper/008:44:580
30910998271,6cyclictest0-21swapper/008:44:580
30910998254,11cyclictest181rcu_preempt07:33:510
3092599814,65cyclictest5317-21kworker/u19:1+flush-179:007:59:272
30925998139,37cyclictest0-21swapper/209:19:112
30925998134,40cyclictest0-21swapper/207:54:242
30918998165,8cyclictest0-21swapper/109:23:521
30918998149,12cyclictest58-21kcompactd009:08:511
30910998155,18cyclictest29984-21latency_hist08:43:530
30910998151,15cyclictest181rcu_preempt10:39:570
30910998148,13cyclictest181rcu_preempt11:59:280
30910998131,9cyclictest181rcu_preempt10:15:550
30925998060,11cyclictest0-21swapper/208:43:532
30925998038,37cyclictest0-21swapper/211:38:532
30925998037,33cyclictest0-21swapper/209:48:532
30918998064,9cyclictest3669-21modprobe07:23:491
30918998064,9cyclictest3669-21modprobe07:23:491
30918998059,11cyclictest0-21swapper/108:45:071
30918998059,11cyclictest0-21swapper/108:45:071
3091899802,29cyclictest23243-21cstates10:39:071
30910998053,18cyclictest10430-21latency_hist12:13:540
30910998053,18cyclictest10430-21latency_hist12:13:540
30910998046,12cyclictest181rcu_preempt09:08:520
30925997963,9cyclictest0-21swapper/209:51:482
30925997933,37cyclictest0-21swapper/211:18:532
30925997930,38cyclictest0-21swapper/208:04:342
30918997961,12cyclictest30904-21cyclictest09:14:271
30918997961,12cyclictest30904-21cyclictest09:14:271
30918997959,10cyclictest0-21swapper/109:34:061
30918997951,7cyclictest181rcu_preempt11:54:251
30910997948,13cyclictest181rcu_preempt12:29:110
30910997946,12cyclictest181rcu_preempt12:14:260
30910997946,10cyclictest181rcu_preempt08:29:290
3091099794,41cyclictest355-21runrttasks10:27:250
30910997944,17cyclictest181rcu_preempt07:58:540
3091099793,48cyclictest15-21ksoftirqd/009:52:030
3091099793,44cyclictest20025-21cron08:13:490
30910997931,8cyclictest181rcu_preempt10:44:080
30925997858,14cyclictest0-21swapper/209:18:542
30925997838,33cyclictest0-21swapper/211:04:192
30925997837,33cyclictest0-21swapper/209:34:152
30925997837,32cyclictest0-21swapper/212:09:302
30925997837,32cyclictest0-21swapper/212:09:302
30925997831,39cyclictest0-21swapper/208:34:262
30925997831,39cyclictest0-21swapper/208:34:262
3092599782,67cyclictest0-21swapper/207:09:062
30918997864,7cyclictest0-21swapper/111:39:271
30918997858,11cyclictest0-21swapper/107:09:211
30918997839,25cyclictest28049-21apt-get09:49:011
3091099786,40cyclictest3881-21latency_hist11:03:510
30910997855,8cyclictest181rcu_preempt07:49:190
30910997855,16cyclictest11894-21latency_hist07:48:530
3091099784,61cyclictest22566-21apt-get09:39:000
30910997836,31cyclictest0-21swapper/010:19:250
30910997836,31cyclictest0-21swapper/010:19:250
30910997832,29cyclictest181rcu_preempt07:24:150
30910997832,29cyclictest181rcu_preempt07:24:150
3093099772,61cyclictest12402-21chrt11:19:403
3093099772,61cyclictest12402-21chrt11:19:403
30925997747,21cyclictest924-21modprobe09:58:502
3092599773,65cyclictest0-21swapper/210:59:012
30925997735,33cyclictest0-21swapper/207:44:072
30925997730,41cyclictest0-21swapper/210:29:142
30925997728,43cyclictest2057-21cut07:18:522
30925997728,43cyclictest2057-21cut07:18:522
30918997759,12cyclictest292-21dbus-daemon10:18:491
30918997746,20cyclictest22566-21apt-get09:39:031
30910997757,14cyclictest31412-21snmpd12:38:520
30910997757,14cyclictest31412-21snmpd12:38:520
30910997747,9cyclictest181rcu_preempt10:29:530
3091099773,61cyclictest20051-21fschecks_time09:34:120
3091099773,45cyclictest16423-21awk12:23:490
30910997733,32cyclictest6774-21/usr/sbin/munin11:09:180
30925997653,15cyclictest207-21systemd-journal10:48:022
30925997632,37cyclictest0-21swapper/212:38:512
30925997632,37cyclictest0-21swapper/212:38:512
30918997662,10cyclictest30904-21cyclictest09:29:211
30918997657,8cyclictest0-21swapper/107:29:221
30918997635,31cyclictest30904-21cyclictest07:08:541
3091099763,52cyclictest15472-21apt-get07:59:550
30910997625,20cyclictest181rcu_preempt10:33:590
30910997625,20cyclictest181rcu_preempt10:33:590
3092599753,66cyclictest0-21swapper/210:24:062
30925997526,41cyclictest0-21swapper/208:49:292
30925997526,41cyclictest0-21swapper/208:49:292
30918997558,11cyclictest22016-21latency_hist12:33:511
30918997547,22cyclictest12557-21fschecks_time10:19:151
30918997547,22cyclictest12557-21fschecks_time10:19:151
30918997532,33cyclictest23798-21apt-get08:24:061
30918997521,44cyclictest6203-21taskset12:05:271
30918997521,44cyclictest6203-21taskset12:05:271
30910997561,10cyclictest26412-21kworker/u17:0+xprtiod09:16:470
30910997561,10cyclictest26412-21kworker/u17:0+xprtiod09:16:470
3091099754,35cyclictest16868-21latency_hist09:28:530
30925997452,15cyclictest9125-21fschecks_time09:14:142
3092599743,53cyclictest18226-21/usr/sbin/munin11:29:222
30925997431,32cyclictest0-21swapper/209:43:532
30925997427,40cyclictest2536-21ssh10:00:092
30925997427,40cyclictest2536-21ssh10:00:092
3092599742,54cyclictest14574-21sendmail_mailqu12:19:232
30918997453,15cyclictest316-21rs:main1
30918997453,12cyclictest0-21swapper/108:54:121
30918997438,21cyclictest10394-21apt-get07:44:061
30918997436,22cyclictest32108-21apt-get08:49:061
30918997436,22cyclictest32108-21apt-get08:49:061
30910997453,16cyclictest5871-21diskmemload11:24:400
30910997450,18cyclictest5871-21diskmemload12:05:040
30910997450,18cyclictest5871-21diskmemload12:05:040
3091099744,56cyclictest31382-21ssh09:54:190
30910997445,11cyclictest181rcu_preempt11:04:230
30910997444,8cyclictest181rcu_preempt09:59:140
30910997444,8cyclictest181rcu_preempt09:59:140
3091099744,35cyclictest355-21runrttasks12:27:250
3092599733,65cyclictest0-21swapper/207:24:022
3092599733,65cyclictest0-21swapper/207:24:022
3092599733,64cyclictest0-21swapper/210:09:032
3092599733,63cyclictest0-21swapper/207:29:052
30925997327,38cyclictest0-21swapper/211:44:082
30925997322,44cyclictest19329-21awk08:09:242
30925997313,44cyclictest369-21ntpd10:39:182
30918997346,19cyclictest5871-21diskmemload11:08:531
30910997347,12cyclictest181rcu_preempt09:21:020
30910997342,9cyclictest181rcu_preempt10:54:190
30910997342,9cyclictest181rcu_preempt10:54:190
30910997339,12cyclictest181rcu_preempt07:23:510
30910997339,12cyclictest181rcu_preempt07:23:510
30910997334,16cyclictest181rcu_preempt10:13:530
3091099733,37cyclictest355-21runrttasks09:33:530
3091099733,35cyclictest31848-21idleruntime11:54:140
30925997253,12cyclictest0-21swapper/207:38:502
30925997233,32cyclictest0-21swapper/211:23:082
30925997233,32cyclictest0-21swapper/211:23:082
30925997232,33cyclictest0-21swapper/212:29:252
3092599721,66cyclictest381ktimers/210:19:032
3092599721,66cyclictest381ktimers/210:19:032
30918997257,7cyclictest0-21swapper/111:24:301
30918997250,11cyclictest17094-21apt-get08:04:061
30918997236,24cyclictest28587-21apt-get11:49:041
30910997250,8cyclictest181rcu_preempt11:48:520
30910997246,12cyclictest181rcu_preempt07:34:230
30910997244,20cyclictest9973-21sh11:14:220
30910997237,13cyclictest181rcu_preempt08:34:140
30910997237,13cyclictest181rcu_preempt08:34:140
3091099723,53cyclictest1488-21fschecks_count08:54:120
30910997234,13cyclictest181rcu_preempt11:53:240
30930997114,37cyclictest10997-21chrt12:14:043
30925997131,32cyclictest0-21swapper/208:24:272
30925997125,39cyclictest355-21runrttasks11:59:402
30918997153,12cyclictest31412-21snmpd11:00:451
30918997148,12cyclictest0-21swapper/107:39:281
30918997138,18cyclictest13607-21apt-get07:54:021
30918997125,30cyclictest20257-21grep10:33:521
30910997150,14cyclictest30904-21cyclictest08:08:100
30910997140,11cyclictest181rcu_preempt08:59:220
30925997048,16cyclictest23207-21fschecks_count11:39:142
3092599704,57cyclictest0-21swapper/207:49:052
3092599704,44cyclictest6774-21/usr/sbin/munin11:09:162
3092599703,62cyclictest0-21swapper/209:04:022
30925997031,31cyclictest0-21swapper/209:32:012
3092599702,61cyclictest28778-21runrttasks11:49:082
30918997051,14cyclictest20497-21ssh10:34:081
30918997051,14cyclictest20497-21ssh10:34:081
30918997048,12cyclictest0-21swapper/110:48:521
30918997048,10cyclictest0-21swapper/112:34:191
30918997048,10cyclictest0-21swapper/112:34:191
30918997047,13cyclictest0-21swapper/110:09:121
30918997045,19cyclictest5871-21diskmemload12:27:041
3091899704,43cyclictest7002-21fschecks_count11:09:131
30918997029,33cyclictest0-21swapper/108:59:011
30918997029,28cyclictest25708-21/usr/sbin/munin11:44:301
30918997027,30cyclictest2551-21ntp_states11:59:241
3091099705,42cyclictest8972-21fschecks_count07:39:140
3092599694,54cyclictest5796-21ssh10:06:472
30925996929,33cyclictest0-21swapper/210:17:502
3092599692,56cyclictest355-21runrttasks07:19:042
3092599692,56cyclictest355-21runrttasks07:19:042
30918996951,9cyclictest0-21swapper/108:03:521
30918996950,9cyclictest0-21swapper/110:24:141
30918996936,22cyclictest20235-21apt-get11:34:061
30918996922,33cyclictest5448-21latency_hist07:28:521
30918996922,33cyclictest5448-21latency_hist07:28:521
30918996920,31cyclictest8289-21ssh09:12:401
30910996920,29cyclictest22640-21taskset08:19:170
3092599684,56cyclictest30926-21sed07:08:542
3092599684,51cyclictest2904-21apt-get08:59:012
30925996828,33cyclictest15392-21latency11:24:302
30925996827,33cyclictest0-21swapper/209:24:092
30925996821,38cyclictest31045-21netstat08:45:122
30925996821,38cyclictest31045-21netstat08:45:122
30918996747,10cyclictest0-21swapper/109:44:161
30918996733,26cyclictest0-21swapper/108:34:011
30918996733,26cyclictest0-21swapper/108:34:011
30918996732,26cyclictest7443-21taskset07:34:151
3092599663,34cyclictest1998-21wc08:54:252
30918996637,19cyclictest4022-21apt-get10:04:031
30918996629,25cyclictest26082-21/usr/sbin/munin08:29:241
3091099663,50cyclictest355-21runrttasks07:09:120
3093099657,36cyclictest31668-21chrt10:54:223
3093099657,36cyclictest31668-21chrt10:54:223
30930996425,26cyclictest30832-21taskset10:53:453
30930996425,26cyclictest30832-21taskset10:53:453
30930996330,21cyclictest10183-21chrt07:43:523
30930996328,22cyclictest29292-21taskset09:49:563
30930996328,22cyclictest16208-21chrt08:00:203
30930996326,25cyclictest30979-21taskset09:54:093
30930996231,20cyclictest7740-21chrt07:34:243
30930996225,25cyclictest17542-21taskset08:04:173
30930996124,24cyclictest32731-21taskset11:54:513
30930996123,24cyclictest2553-21taskset10:00:133
30930996123,24cyclictest2553-21taskset10:00:133
30930996122,25cyclictest8961-21taskset10:13:093
3093099603,44cyclictest12924-21taskset10:19:253
3093099603,44cyclictest12924-21taskset10:19:253
30930996028,21cyclictest23760-21chrt10:39:223
30930996027,22cyclictest22484-21taskset09:38:543
30930996027,21cyclictest5685-21chrt11:06:093
30930996023,26cyclictest26001-21taskset10:44:163
30930996023,25cyclictest5341-21taskset12:04:193
30930996023,25cyclictest5341-21taskset12:04:193
30930996023,24cyclictest9420-21taskset11:14:073
3093099595,38cyclictest9170-21chrt09:14:143
3093099595,38cyclictest9170-21chrt09:14:143
3093099595,35cyclictest0-21swapper/308:14:033
3093099593,43cyclictest14936-21taskset10:23:593
30930995929,19cyclictest31896-21chrt07:09:253
30930995927,21cyclictest7882-21chrt12:09:053
30930995927,21cyclictest7882-21chrt12:09:053
30930995926,21cyclictest5237-21chrt07:28:293
30930995926,21cyclictest5237-21chrt07:28:293
30930995925,22cyclictest2180-21taskset07:18:533
30930995925,22cyclictest2180-21taskset07:18:533
3093099584,39cyclictest10182-21taskset10:14:233
30930995837,13cyclictest11690-21chrt09:19:053
30930995827,21cyclictest25366-21chrt09:44:043
30930995826,21cyclictest30524-21chrt08:44:223
30930995826,21cyclictest30524-21chrt08:44:223
30930995826,20cyclictest1813-21chrt10:59:083
30930995826,20cyclictest10842-21chrt07:44:163
30930995824,21cyclictest28697-21chrt08:39:083
30930995823,22cyclictest6802-21chrt11:09:053
30930995822,24cyclictest21321-21taskset12:31:433
30930995726,20cyclictest15429-21chrt07:58:543
30930995724,22cyclictest3050-21taskset11:59:333
30930995724,21cyclictest7753-21chrt09:10:583
30930995724,21cyclictest14475-21taskset12:19:193
30930995724,20cyclictest12157-21chrt07:49:103
30930995723,23cyclictest32117-21chrt08:49:043
30930995723,23cyclictest32117-21chrt08:49:043
30930995723,22cyclictest4674-21chrt09:04:103
30930995723,21cyclictest3656-21chrt08:59:243
30930995723,21cyclictest30207-21chrt11:50:573
30930995722,23cyclictest14263-21taskset09:24:033
30930995722,22cyclictest23874-21chrt12:35:533
30930995722,22cyclictest23874-21chrt12:35:533
30930995721,24cyclictest24791-21taskset08:24:303
3093099563,38cyclictest22649-21taskset11:38:493
30930995624,20cyclictest19165-21chrt11:31:033
30930995624,19cyclictest27491-21chrt08:34:183
30930995624,19cyclictest27491-21chrt08:34:183
3093099562,38cyclictest24025-21chrt11:39:303
30930995623,21cyclictest5106-21chrt10:04:303
30930995623,20cyclictest27680-21chrt11:47:043
30930995524,21cyclictest24170-21chrt09:41:043
30930995524,20cyclictest22397-21chrt10:37:363
30930995524,20cyclictest22397-21chrt10:37:363
30930995524,20cyclictest18773-21chrt09:31:403
30930995523,21cyclictest23704-21chrt08:23:533
30930995523,20cyclictest3545-21chrt07:22:213
30930995523,20cyclictest3545-21chrt07:22:213
30930995523,20cyclictest19748-21chrt08:10:123
30930995523,20cyclictest19213-21chrt10:31:043
30930995523,20cyclictest16576-21chrt11:26:333
30930995423,20cyclictest1280-21chrt08:54:063
30930995421,21cyclictest6551-21chrt07:29:383
30930995421,21cyclictest18485-21chrt12:26:413
30930995321,20cyclictest26477-21chrt08:30:333
308342439,20sleep30-21swapper/307:08:533
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional