You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-30 - 21:01
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Tue Dec 30, 2025 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2652026230,15sleep30-21swapper/307:04:513
27233215214,107sleep20-21swapper/207:06:182
2718421469,91sleep127231-21awk07:06:171
27988991253,88cyclictest19874-21ssh11:29:462
2798899994,68cyclictest14681-21apt-get09:30:052
2798899994,68cyclictest14681-21apt-get09:30:052
2797999997,79cyclictest2345-21apt-get07:29:411
211622940,7sleep30-21swapper/310:35:023
2797999934,78cyclictest14681-21apt-get09:29:431
2797999934,78cyclictest14681-21apt-get09:29:431
2798899924,47cyclictest17682-21ssh10:30:152
2798899913,40cyclictest346-21runrttasks12:00:142
2798899903,11cyclictest0-21swapper/210:24:452
27979999069,14cyclictest15939-21ssh10:26:161
27967998966,14cyclictest14022-21latency_hist08:04:300
2798899884,44cyclictest1954-21head07:25:042
27988998829,37cyclictest0-21swapper/209:55:312
27967998863,16cyclictest26408-21ssh11:40:030
27967998863,16cyclictest26408-21ssh11:40:030
27988998770,10cyclictest14249-21apt-key09:29:302
27988998770,10cyclictest14249-21apt-key09:29:302
27988998746,33cyclictest0-21swapper/210:45:032
27988998745,35cyclictest29146-21/usr/sbin/munin10:50:042
2798899873,35cyclictest20290-21latency_hist09:39:302
27988998646,34cyclictest27960-21cyclictest12:25:262
27988998644,35cyclictest22981-21mailstats12:30:002
2798899864,34cyclictest25532-21/usr/sbin/munin11:40:002
2798899864,34cyclictest25532-21/usr/sbin/munin11:40:002
27988998638,41cyclictest0-21swapper/209:09:262
27988998636,43cyclictest25200-21modprobe11:39:262
2798899863,37cyclictest26634-21uname09:49:542
2798899863,37cyclictest11392-21wc10:19:312
2798899863,36cyclictest20478-21ssh09:39:442
2798899863,36cyclictest20478-21ssh09:39:442
27979998664,16cyclictest14251-21apt-config09:29:291
27979998664,16cyclictest14251-21apt-config09:29:291
27979998661,16cyclictest12882-21ssh11:17:261
27979998661,16cyclictest12882-21ssh11:17:261
2798899853,38cyclictest3468-21runrttasks10:05:492
2798899853,35cyclictest25730-21fschecks_time08:39:502
2798899852,36cyclictest31717-21unin-run08:59:282
2797999854,71cyclictest16097-21apt-get12:19:431
2797999854,71cyclictest16097-21apt-get12:19:431
265172858,33sleep00-21swapper/007:04:490
2798899843,37cyclictest30381-21fschecks_count07:14:512
2798899843,35cyclictest2309-21latency_hist10:59:292
2798899843,35cyclictest13873-21/usr/sbin/munin11:20:062
2798899842,38cyclictest5858-21tune2fs10:09:512
2798899842,38cyclictest28655-21ntpq07:09:562
2798899842,34cyclictest11783-21mailstats11:15:042
2798899842,34cyclictest11783-21mailstats11:15:042
27979998416,53cyclictest1369-21apt-get11:54:411
27979998416,53cyclictest1369-21apt-get11:54:411
2798899833,37cyclictest32731-21grep07:20:022
2798899833,35cyclictest20923-21/usr/sbin/munin08:24:502
2798899833,34cyclictest23586-21cut08:34:292
2798899833,33cyclictest346-21runrttasks11:48:552
2798899833,32cyclictest19113-21cstates08:19:442
2798899832,42cyclictest11690-21ssh10:19:572
2798899832,36cyclictest15630-21cat08:09:302
2798899832,35cyclictest13408-21latency_hist12:14:292
27988998310,30cyclictest24161-21sh10:41:002
27979998363,15cyclictest27669-21latency_hist12:39:301
27979998363,14cyclictest32698-21processes10:00:001
2798899824,31cyclictest10896-21/usr/sbin/munin07:54:502
2798899823,36cyclictest27666-21wc08:45:012
2798899823,35cyclictest5457-21ssh09:14:292
2798899823,35cyclictest24753-21unixbench_multi08:35:052
2798899823,34cyclictest346-21runrttasks12:24:072
2798899823,34cyclictest346-21runrttasks12:24:072
2798899823,33cyclictest8558-21ssh11:09:512
2798899823,32cyclictest31249-21/usr/sbin/munin11:49:522
27988998212,27cyclictest17676-21idleruntime08:14:522
27979998257,19cyclictest14189-21cut11:19:491
2797999823,70cyclictest89-21kswapd011:59:501
27988998162,13cyclictest3280-21unixbench_singl11:56:042
27988998162,13cyclictest3280-21unixbench_singl11:56:042
2798899814,34cyclictest13432-21unixbench_multi08:00:042
2798899814,32cyclictest5667-21/usr/sbin/munin09:14:382
2798899814,32cyclictest13522-21/usr/sbin/munin12:15:002
2798899814,32cyclictest13522-21/usr/sbin/munin12:15:002
2798899813,35cyclictest346-21runrttasks08:14:312
2798899813,33cyclictest3688-21timerandwakeup07:30:532
2798899813,33cyclictest32003-21/usr/sbin/munin10:00:042
2798899813,33cyclictest20318-21/usr/sbin/munin10:34:462
2798899813,31cyclictest25097-21/usr/sbin/munin12:34:522
2798899813,26cyclictest6312-21grep07:39:492
2798899813,26cyclictest6312-21grep07:39:492
2798899812,34cyclictest24029-21grep09:45:002
27979998158,17cyclictest5572-21cut11:04:491
27979998158,17cyclictest5572-21cut11:04:491
27979998114,46cyclictest23461-21apt-get10:40:091
2796799812,58cyclictest19899-21ssh11:29:530
27988998058,16cyclictest19283-21idleruntime-cro11:29:262
2798899804,8cyclictest0-21swapper/210:59:392
2798899803,38cyclictest5483-21df11:04:462
2798899803,38cyclictest5483-21df11:04:462
2798899803,34cyclictest8146-21fschecks_time12:04:512
2798899803,34cyclictest8146-21fschecks_time12:04:512
27988998027,46cyclictest7873-21fschecks_count07:44:492
2798899802,34cyclictest9656-21kernelversion07:49:522
2798899802,34cyclictest28717-21cpu08:49:452
2798899802,34cyclictest11217-21latency_hist09:24:302
2797999805,66cyclictest19825-21apt-get11:29:451
27979998055,18cyclictest2465-21diskmemload11:24:571
2797999804,48cyclictest13972-21latency_hist08:04:291
2798899793,35cyclictest855-21munin-run09:04:262
2798899793,35cyclictest855-21munin-run09:04:262
27967997955,18cyclictest17014-21idleruntime-cro10:29:270
27967997954,16cyclictest320-21dbus-daemon10:39:280
27979997856,16cyclictest3157-21sendmail_mailqu11:00:011
2797999783,66cyclictest17559-21apt-get10:29:421
2797999783,61cyclictest11019-21apt-get10:19:421
27967997863,11cyclictest27960-21cyclictest11:49:550
27967997858,14cyclictest2187-21apt-key10:04:280
27967997852,17cyclictest10829-21latency_hist11:14:290
27967997849,20cyclictest26222-21ssh09:49:390
27979997757,14cyclictest2465-21diskmemload10:54:541
27979997750,16cyclictest26963-21sh10:45:001
2796799779,19cyclictest15-21ksoftirqd/012:34:420
27967997728,31cyclictest181rcu_preempt10:09:510
27988997611,27cyclictest5266-21cat07:35:032
27988997611,27cyclictest5266-21cat07:35:032
27979997656,13cyclictest13971-21kworker/u17:2-writeback10:19:301
27979997646,21cyclictest25174-21cron08:39:271
2797999763,34cyclictest14211-21/usr/sbin/munin12:15:001
2797999763,34cyclictest14211-21/usr/sbin/munin12:15:001
2796799766,40cyclictest15-21ksoftirqd/010:30:190
27967997655,15cyclictest201-21systemd-journal12:19:260
27967997655,15cyclictest201-21systemd-journal12:19:260
27967997652,15cyclictest1137-21ssh10:56:300
2796799763,45cyclictest32732-21grep07:20:020
27979997554,15cyclictest2465-21diskmemload09:09:551
27979997553,13cyclictest0-21swapper/112:14:301
27979997552,13cyclictest0-21swapper/107:55:031
2796799752,60cyclictest346-21runrttasks12:20:090
2796799752,60cyclictest346-21runrttasks12:20:090
27979997450,18cyclictest22994-21sh11:34:501
27979997424,14cyclictest301ktimers/108:14:391
2796799745,38cyclictest23847-21netstat09:44:540
27967997435,17cyclictest5571-21fschecks_count11:04:490
27967997435,17cyclictest5571-21fschecks_count11:04:490
2796799743,43cyclictest11785-21vmstat12:10:070
27967997351,16cyclictest14216-21sh11:19:510
27967997350,14cyclictest2465-21diskmemload09:34:190
27967997350,14cyclictest2465-21diskmemload09:34:190
27967997346,8cyclictest181rcu_preempt10:14:480
27979997256,11cyclictest27960-21cyclictest08:24:431
27979997247,17cyclictest23581-21munin-plugin-st08:34:291
27979997225,20cyclictest31-21ksoftirqd/111:40:261
27979997225,20cyclictest31-21ksoftirqd/111:40:261
27967997244,21cyclictest19035-21cpu08:19:440
2796799724,33cyclictest346-21runrttasks11:00:480
27967997243,23cyclictest24118-21kworker/u17:0+events_unbound12:09:260
27967997243,23cyclictest24118-21kworker/u17:0+events_unbound12:09:260
2796799723,34cyclictest13669-21latency_hist11:19:290
2796799723,34cyclictest13669-21latency_hist11:19:290
2797999715,50cyclictest30496-21idleruntime07:14:531
27979997148,15cyclictest32226-21latency07:19:511
27979997146,17cyclictest6299-21mailstats10:10:021
2797999714,48cyclictest10692-21grep07:54:291
2797999714,35cyclictest31-21ksoftirqd/110:08:171
27979997141,21cyclictest10781-21sh11:14:291
2797999713,46cyclictest15698-21latency_hist08:09:311
2797999712,45cyclictest23199-21gpgv10:39:301
27967997150,13cyclictest0-21swapper/009:19:290
27967997150,12cyclictest20237-21latency_hist09:39:290
2796799714,37cyclictest11569-21/usr/sbin/munin10:20:070
27967997115,20cyclictest24419-21kworker/0:0-events08:14:190
27967997111,39cyclictest2133-21sshd11:55:300
27967997111,39cyclictest2133-21sshd11:55:300
2797999706,26cyclictest31-21ksoftirqd/110:53:061
27979997046,19cyclictest27960-21cyclictest07:44:441
27979997044,9cyclictest181rcu_preempt09:51:431
2797999703,48cyclictest11467-21latency_hist09:24:311
27979997024,11cyclictest301ktimers/112:09:271
27979997024,11cyclictest301ktimers/112:09:271
27979997021,8cyclictest291rcuc/111:54:301
27967997053,10cyclictest1763-21mailstats09:05:010
27967997037,13cyclictest181rcu_preempt10:05:260
27967997037,12cyclictest181rcu_preempt11:35:470
27967997036,25cyclictest1392-21wc07:24:530
27967997027,30cyclictest14351-21fschecks_count08:04:520
27979996948,12cyclictest0-21swapper/107:29:271
27979996946,13cyclictest0-21swapper/109:44:481
2797999694,51cyclictest19558-21cstates12:24:451
27967996950,13cyclictest2465-21diskmemload11:44:490
27967996944,18cyclictest17304-21sh11:24:550
27967996940,10cyclictest181rcu_preempt08:24:420
2796799693,41cyclictest363-21snmpd12:00:140
2796799693,31cyclictest346-21runrttasks08:33:160
27979996846,15cyclictest32040-21cstates08:59:441
27979996846,15cyclictest32040-21cstates08:59:441
27979996841,16cyclictest30981-21mailstats08:54:591
27979996833,20cyclictest31-21ksoftirqd/109:15:221
2796799684,34cyclictest9344-21grep09:20:020
27967996843,11cyclictest181rcu_preempt10:43:030
2796799683,54cyclictest19522-21cstates12:24:440
27967996825,12cyclictest181rcu_preempt10:44:460
2797999674,44cyclictest6035-21latency_hist07:39:301
2797999674,44cyclictest6035-21latency_hist07:39:301
2797999673,45cyclictest29507-21latency_hist07:14:301
27979996731,26cyclictest26810-21modprobe08:44:271
27979996730,23cyclictest31-21ksoftirqd/111:45:571
27979996719,35cyclictest31869-21diskmemload09:59:301
27967996754,9cyclictest60-21kcompactd007:39:450
27967996754,9cyclictest60-21kcompactd007:39:450
27967996749,12cyclictest1-21systemd08:59:270
27967996747,13cyclictest27960-21cyclictest07:35:010
27967996747,13cyclictest27960-21cyclictest07:35:010
27967996739,11cyclictest181rcu_preempt08:49:440
2796799673,35cyclictest22541-21fschecks_time12:29:490
277682672,6sleep00-21swapper/008:45:030
27979996645,12cyclictest17519-21/usr/sbin/munin09:35:101
2797999662,34cyclictest30090-21latency_hist08:54:291
2796799664,41cyclictest30225-21grep10:50:050
27967996628,19cyclictest181rcu_preempt08:40:040
27967996618,35cyclictest30211-21awk09:55:070
2797999654,43cyclictest17275-21latency_hist08:14:301
2796799654,35cyclictest2927-21cpu07:30:340
2796799653,44cyclictest346-21runrttasks09:41:560
2796799653,44cyclictest346-21runrttasks09:41:560
2796799652,34cyclictest25297-21latency_hist08:39:290
27967996514,32cyclictest14251-21apt-config09:29:290
27967996514,32cyclictest14251-21apt-config09:29:290
27979996433,10cyclictest181rcu_preempt09:44:051
27979996433,10cyclictest181rcu_preempt09:44:051
27979996428,16cyclictest0-21swapper/108:44:561
2797999642,45cyclictest6804-21switchtime07:40:011
2797999642,45cyclictest6804-21switchtime07:40:011
2796799644,48cyclictest2760-21/usr/sbin/munin09:10:020
27967996442,12cyclictest0-21swapper/007:14:340
2796799643,47cyclictest346-21runrttasks07:45:000
27967996434,19cyclictest0-21swapper/008:59:430
27967996434,19cyclictest0-21swapper/008:59:430
27967996412,23cyclictest29512-21munin-run07:14:270
2797999633,41cyclictest1379-21fschecks_time09:04:501
27967996341,9cyclictest181rcu_preempt07:54:420
2797999623,42cyclictest19123-21cstates08:19:451
27979996219,8cyclictest301ktimers/112:31:191
27967996244,12cyclictest0-21swapper/008:14:500
2799699604,43cyclictest14813-21chrt10:24:533
27967996021,10cyclictest181rcu_preempt07:49:490
193822600,9sleep30-21swapper/311:29:283
2799699584,39cyclictest15355-21taskset09:30:273
2799699584,39cyclictest15355-21taskset09:30:273
2799699562,40cyclictest11252-21taskset12:09:563
2799699554,44cyclictest0-21swapper/310:29:423
2799699553,41cyclictest26787-21taskset10:44:543
2799699544,39cyclictest20491-21taskset12:25:053
2799699544,39cyclictest0-21swapper/311:55:393
2799699544,39cyclictest0-21swapper/311:55:393
27996995439,9cyclictest451rcuc/309:56:293
2799699543,40cyclictest24401-21taskset10:41:053
2799699543,38cyclictest32128-21chrt07:19:493
2799699543,38cyclictest10086-21taskset10:16:423
2799699542,39cyclictest14427-21chrt12:15:033
2799699542,39cyclictest14427-21chrt12:15:033
2799699535,36cyclictest0-21swapper/307:29:503
2799699533,37cyclictest16618-21taskset12:19:333
2799699533,37cyclictest16618-21taskset12:19:333
2799699533,36cyclictest22188-21chrt08:29:303
2799699533,34cyclictest8042-21chrt11:09:303
2799699533,34cyclictest8042-21chrt11:09:303
27996995321,27cyclictest0-21swapper/310:05:163
27996995317,26cyclictest6227-21chrt10:10:013
27996995238,8cyclictest451rcuc/311:46:353
2799699523,41cyclictest0-21swapper/311:29:423
2799699523,35cyclictest22614-21chrt11:34:313
2799699523,35cyclictest22538-21taskset09:43:033
2799699523,35cyclictest22538-21taskset09:43:033
2799699523,34cyclictest2625-21chrt10:04:313
2799699523,34cyclictest18953-21chrt08:19:303
2799699522,34cyclictest5507-21chrt09:14:303
27996995136,9cyclictest451rcuc/308:06:473
2799699513,37cyclictest2276-21chrt10:59:293
2799699513,34cyclictest10065-21chrt07:50:023
2799699513,33cyclictest28272-21chrt07:09:453
2799699504,33cyclictest25435-21taskset08:39:303
2799699503,36cyclictest11462-21chrt09:24:313
2799699503,35cyclictest9959-21chrt11:12:143
2799699503,34cyclictest6251-21chrt12:00:443
2799699503,34cyclictest28068-21taskset08:45:053
2799699503,31cyclictest24990-21chrt12:34:303
2799699502,36cyclictest26612-21chrt12:36:163
2799699502,35cyclictest18908-21chrt09:35:543
2799699495,28cyclictest0-21swapper/310:19:473
2799699494,34cyclictest7929-21chrt12:04:463
2799699494,34cyclictest7929-21chrt12:04:463
2799699494,32cyclictest24726-21chrt09:45:593
2799699493,36cyclictest11588-21chrt07:55:003
2799699493,33cyclictest963-21chrt09:04:293
2799699493,33cyclictest963-21chrt09:04:293
2799699493,32cyclictest19580-21chrt08:19:593
2799699493,32cyclictest14911-21chrt11:20:043
2799699493,30cyclictest28999-21chrt09:54:303
2799699492,33cyclictest14254-21chrt09:29:293
2799699492,33cyclictest14254-21chrt09:29:293
2799699483,33cyclictest8254-21chrt07:45:003
2799699483,33cyclictest25930-21chrt08:39:553
2799699483,32cyclictest602-21chrt11:52:103
2799699483,31cyclictest31362-21taskset10:53:253
27996994824,15cyclictest451rcuc/307:14:313
2799699473,36cyclictest11566-21chrt11:14:573
2799699473,36cyclictest11566-21chrt11:14:573
27996994731,9cyclictest451rcuc/308:55:303
2799699472,35cyclictest4449-21chrt11:02:493
2799699463,31cyclictest5707-21chrt09:14:423
27996994630,10cyclictest451rcuc/308:03:393
27996994623,17cyclictest451rcuc/311:39:553
27996994623,17cyclictest451rcuc/311:39:553
2799699453,33cyclictest29242-21chrt08:49:593
2799699453,33cyclictest1336-21chrt09:04:493
2799699452,31cyclictest2445-21chrt07:29:293
2799699443,26cyclictest17138-21chrt08:14:283
27996994330,8cyclictest451rcuc/307:34:403
27996994330,8cyclictest451rcuc/307:34:403
27996994328,8cyclictest451rcuc/307:42:453
27996994328,8cyclictest451rcuc/307:42:453
27996994228,8cyclictest451rcuc/308:33:393
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional