You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-24 - 23:25
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Wed Dec 24, 2025 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31455991342,122cyclictest17425-21apt-get12:10:170
31455991312,111cyclictest25996-21kworker/u19:2+ext4-rsv-conversion11:35:050
31455991148,95cyclictest14228-21apt-get12:05:070
31455991148,95cyclictest14228-21apt-get12:05:070
314599911242,12cyclictest0-21swapper/109:30:091
314599911242,12cyclictest0-21swapper/109:30:091
314559911131,39cyclictest23171-21ssh11:25:250
314559911037,47cyclictest21081-21ssh12:15:220
314559910932,28cyclictest8808-21ssh09:14:260
314559910843,30cyclictest32111-21/usr/sbin/munin11:40:170
314559910830,36cyclictest23598-21ssh12:20:120
314559910830,36cyclictest23598-21ssh12:20:120
31459991072,42cyclictest4410-21wc09:04:521
31459991072,42cyclictest4410-21wc09:04:521
31455991076,63cyclictest9517-21apt-get09:15:070
31455991076,63cyclictest9517-21apt-get09:15:070
31455991076,63cyclictest9517-21apt-get09:15:070
314559910749,48cyclictest32141-21kworker/u19:1+ext4-rsv-conversion11:10:150
314649910652,47cyclictest931-21/usr/sbin/munin09:55:182
314559910538,28cyclictest20473-21sed11:20:270
314559910439,23cyclictest27727-21grep09:45:110
314559910439,23cyclictest27727-21grep09:45:110
314559910436,31cyclictest30018-21find12:30:240
31326210411,54sleep231334-21modprobe07:09:512
31326210411,54sleep231334-21modprobe07:09:512
314649910341,7cyclictest0-21swapper/211:10:012
314559910342,21cyclictest21560-21latency_hist10:29:520
314559910340,21cyclictest8328-21grep11:00:300
314559910337,25cyclictest30498-21wc10:44:520
314559910334,29cyclictest17124-21grep11:15:230
314559910334,29cyclictest17124-21grep11:15:230
314599910236,9cyclictest0-21swapper/111:55:261
314599910236,9cyclictest0-21swapper/111:55:261
314559910278,15cyclictest382-21rs:main0
314559910242,25cyclictest12049-21grep12:00:270
314559910242,25cyclictest12049-21grep12:00:270
314559910239,25cyclictest201-21systemd-journal11:46:080
314559910239,21cyclictest25655-21ssh10:35:270
314559910238,24cyclictest4584-21rm10:00:230
314559910232,30cyclictest24066-21sed08:24:540
314559910224,38cyclictest0-21swapper/008:05:270
314599910141,15cyclictest0-21swapper/111:10:311
31459991013,36cyclictest18369-21mailstats12:10:551
314559910142,51cyclictest16474-21cat09:25:270
314559910141,27cyclictest3918-21sed09:00:270
314559910141,27cyclictest3918-21sed09:00:270
314559910138,25cyclictest13096-21cut07:50:110
314559910135,23cyclictest0-21swapper/008:30:400
314559910134,27cyclictest4910-21fschecks_time09:05:110
314559910133,35cyclictest13064-21apt-key11:09:520
314559910035,28cyclictest9772-21ssh11:57:180
314559910035,28cyclictest9772-21ssh11:57:180
314559910031,37cyclictest22903-21grep10:30:290
2544721000,5sleep30-21swapper/311:30:093
31459999942,9cyclictest24128-21apt09:39:501
31455999960,30cyclictest5058-21apt-get11:50:020
31455999941,20cyclictest10433-21ls07:40:250
31455999939,24cyclictest6965-21sh10:05:060
31464999840,9cyclictest0-21swapper/210:20:132
3146499983,38cyclictest8066-21ssh11:55:072
3146499983,38cyclictest8066-21ssh11:55:072
31455999839,25cyclictest32575-21grep07:10:240
31455999839,25cyclictest32575-21grep07:10:240
31455999839,20cyclictest25585-21ssh11:30:140
31455999838,25cyclictest12804-21date07:49:540
31455999833,28cyclictest0-21swapper/012:25:250
31464999773,18cyclictest18384-21latency_hist09:29:542
3145999974,38cyclictest30222-21latency_hist09:49:531
3145999974,38cyclictest30222-21latency_hist09:49:531
3145999973,42cyclictest18952-21cpuspeed10:25:061
31455999740,25cyclictest32058-21latency_hist08:49:530
31455999740,24cyclictest0-21swapper/010:10:030
31455999737,20cyclictest13893-21timerandwakeup09:21:410
31455999732,28cyclictest21321-21date09:34:540
31455999732,28cyclictest21321-21date09:34:540
31455999730,25cyclictest0-21swapper/010:15:260
31464999660,32cyclictest31447-21cyclictest10:37:012
31464999654,37cyclictest31447-21cyclictest11:15:262
31464999654,37cyclictest31447-21cyclictest11:15:262
31464999642,47cyclictest0-21swapper/212:21:412
31464999642,47cyclictest0-21swapper/212:21:412
3145999964,35cyclictest30206-21timerandwakeup12:30:271
31455999662,24cyclictest16091-21apt-get08:00:060
31455999638,24cyclictest19440-21cstates08:10:080
31455999634,22cyclictest0-21swapper/007:30:030
31455999634,22cyclictest0-21swapper/007:30:030
31464999541,9cyclictest0-21swapper/211:20:192
31464999537,7cyclictest0-21swapper/209:05:232
31459999538,10cyclictest0-21swapper/107:30:091
31459999538,10cyclictest0-21swapper/107:30:091
3145999952,41cyclictest14786-21latency12:05:331
3145999952,41cyclictest14786-21latency12:05:331
31455999536,21cyclictest5985-21diskmemload09:38:530
31455999535,23cyclictest5537-21sh10:55:580
31455999533,27cyclictest0-21swapper/010:24:530
31464999449,39cyclictest17472-21unin-run08:04:522
31464999437,49cyclictest0-21swapper/208:20:152
3146499943,35cyclictest6850-21latency_hist10:04:532
31464999432,19cyclictest1-21systemd09:47:092
31464999432,19cyclictest1-21systemd09:47:092
3145999943,39cyclictest2914-21cpu07:20:091
31455999444,37cyclictest32401-21grep08:50:110
31455999437,23cyclictest32375-21cpuspeed12:35:080
31455999437,23cyclictest32375-21cpuspeed12:35:080
31455999436,24cyclictest30799-21cpu10:45:060
31455999433,23cyclictest0-21swapper/007:25:250
31455999430,27cyclictest29925-21sed08:40:260
3146499933,39cyclictest17404-21/usr/sbin/munin12:10:572
3146499933,37cyclictest15286-21latency_hist09:24:532
3146499933,36cyclictest28807-21latency_hist08:39:532
31464999328,14cyclictest0-21swapper/210:30:582
3146499932,45cyclictest25651-21fschecks_time11:30:152
3146499932,36cyclictest29930-21/usr/sbin/munin12:30:212
3145999935,43cyclictest346-21runrttasks08:00:071
31459999332,15cyclictest0-21swapper/108:59:531
31459999331,16cyclictest0-21swapper/107:10:161
31459999331,16cyclictest0-21swapper/107:10:161
31455999333,23cyclictest0-21swapper/007:35:300
3146499923,36cyclictest26291-21ntp_kernel_pll_08:30:402
3146499923,36cyclictest19612-21ssh10:25:242
3146499923,34cyclictest11813-21/usr/sbin/munin12:00:252
3146499923,34cyclictest11813-21/usr/sbin/munin12:00:252
3146499923,31cyclictest874-21/usr/sbin/munin12:35:282
3146499923,31cyclictest874-21/usr/sbin/munin12:35:282
3146499922,37cyclictest25560-21ssh09:40:282
3146499922,36cyclictest2913-21cpu07:20:092
3146499922,32cyclictest15950-21latency_hist07:59:532
31464999218,33cyclictest10685-21expr11:05:152
31459999246,8cyclictest10946-21ssh10:10:281
3145999924,36cyclictest346-21runrttasks12:27:241
31459999231,17cyclictest0-21swapper/109:17:041
31459999231,17cyclictest0-21swapper/109:17:041
31459999231,17cyclictest0-21swapper/109:17:041
31455999238,23cyclictest3013-21ssh10:52:130
31455999234,22cyclictest0-21swapper/007:19:530
3146499914,28cyclictest22504-21ls11:25:082
31464999140,12cyclictest31960-21latency_hist08:49:542
3146499913,38cyclictest28990-21/usr/sbin/munin08:40:102
3146499913,35cyclictest28117-21ssh12:27:242
3146499913,33cyclictest29486-21sshd10:42:092
3146499912,37cyclictest21304-21latency_hist09:34:542
3146499912,37cyclictest21304-21latency_hist09:34:542
31455999159,23cyclictest953-21apt-get09:55:070
31455999132,22cyclictest21858-21grep08:15:270
31464999065,16cyclictest201-21systemd-journal10:58:092
3146499904,38cyclictest14667-21/usr/sbin/munin12:05:302
3146499904,38cyclictest14667-21/usr/sbin/munin12:05:302
3146499903,37cyclictest7195-21grep10:05:112
3146499903,33cyclictest1883-21ntpq08:55:212
3146499903,32cyclictest30506-21cpuspeed09:50:082
3146499903,30cyclictest5817-21awk11:50:232
3146499902,38cyclictest19233-21latency_hist08:09:542
3146499902,36cyclictest8426-21latency07:35:302
31459999075,9cyclictest60-21kcompactd011:51:031
3145999903,38cyclictest346-21runrttasks11:39:221
3145999903,28cyclictest21097-21fschecks_count08:15:131
31455999057,26cyclictest31456-21latency_hist07:09:540
31455999057,26cyclictest31456-21latency_hist07:09:540
31455999038,29cyclictest24885-21/usr/sbin/munin09:40:160
31455999036,24cyclictest25057-21unixbench_multi08:25:260
31455999032,20cyclictest0-21swapper/009:50:280
31464998941,41cyclictest0-21swapper/207:30:262
31464998941,41cyclictest0-21swapper/207:30:262
3146499893,34cyclictest346-21runrttasks07:14:572
3146499893,33cyclictest6411-21ssh09:10:072
3146499893,30cyclictest1275-21/usr/sbin/munin10:50:302
3146499892,38cyclictest346-21runrttasks07:45:162
3146499892,33cyclictest3254-21fschecks_time09:00:132
3146499892,33cyclictest3254-21fschecks_time09:00:132
31459998930,19cyclictest0-21swapper/109:10:541
31455998938,20cyclictest27409-21grep08:35:060
3146499883,39cyclictest8230-21timerwakeupswit11:00:282
3146499883,37cyclictest10622-21timerandwakeup09:16:012
3146499883,37cyclictest10622-21timerandwakeup09:16:012
3146499883,37cyclictest10622-21timerandwakeup09:16:012
3146499883,33cyclictest346-21runrttasks10:49:302
3146499883,33cyclictest32568-21unixbench_multi11:40:262
3146499883,33cyclictest22506-21grep09:35:312
31455998826,25cyclictest0-21swapper/007:24:510
3146499873,33cyclictest20215-21tail08:10:272
3146499873,31cyclictest13089-21fschecks_time07:50:112
31455998753,21cyclictest14457-21apt-get07:55:070
31464998670,10cyclictest10153-21fschecks_count10:10:132
3146499863,38cyclictest13330-21ssh10:15:172
3146499863,34cyclictest11197-21latency_hist07:44:542
3146499863,33cyclictest25663-21latency_hist08:29:542
3146499863,32cyclictest5925-21grep07:29:532
3146499862,36cyclictest22451-21latency_hist08:19:542
3145999864,34cyclictest32208-21/usr/sbin/munin08:50:231
3146499853,36cyclictest31973-21if_err_eth007:10:122
3146499853,36cyclictest31973-21if_err_eth007:10:122
3146499853,31cyclictest2418-21fschecks_time11:45:142
3146499853,31cyclictest1200-21latency_hist08:54:532
3146499853,30cyclictest346-21runrttasks12:15:022
3146499852,35cyclictest29452-21ntp_kernel_err11:36:162
3145999853,34cyclictest346-21runrttasks10:41:121
31459998365,12cyclictest14192-21latency_hist07:54:541
3145999824,33cyclictest346-21runrttasks08:42:131
3145999815,44cyclictest5843-21munin-plugin-st07:29:501
3145999813,36cyclictest7997-21apt-get07:35:091
31459998063,11cyclictest3090-21unixbench_multi11:45:281
31459997964,8cyclictest0-21swapper/108:09:511
31459997862,10cyclictest31447-21cyclictest11:19:541
31459997862,10cyclictest31447-21cyclictest11:19:541
31459997762,10cyclictest25384-21ntp_kernel_pll_10:35:211
31459997652,16cyclictest18703-21sh10:24:531
3145999754,35cyclictest12665-21ls09:20:091
3146899747,24cyclictest0-21swapper/311:36:053
31459997437,13cyclictest16101-21ssh09:25:201
307142747,42sleep130780-21grep07:05:381
307142747,42sleep130780-21grep07:05:381
31468997314,44cyclictest0-21swapper/312:10:053
31459997256,12cyclictest31447-21cyclictest11:04:311
31459997235,13cyclictest368-21lldpd12:04:011
31459997235,13cyclictest368-21lldpd12:04:011
31459997055,8cyclictest0-21swapper/111:25:081
31459997054,11cyclictest0-21swapper/110:30:281
31459997053,13cyclictest31447-21cyclictest10:15:201
31459997053,12cyclictest31447-21cyclictest11:31:131
31459997029,12cyclictest181rcu_preempt07:59:521
31459997027,33cyclictest31952-21modprobe08:49:501
3021927015,30sleep30-21swapper/307:05:193
3021927015,30sleep30-21swapper/307:05:193
31468996916,31cyclictest0-21swapper/312:05:053
31468996916,31cyclictest0-21swapper/312:05:053
31459996952,12cyclictest31447-21cyclictest10:53:351
3145999693,48cyclictest4413-21latency10:00:181
31468996812,37cyclictest135282sleep311:10:163
31468996713,37cyclictest0-21swapper/309:15:373
31468996713,37cyclictest0-21swapper/309:15:373
31468996713,37cyclictest0-21swapper/309:15:373
3145999674,46cyclictest346-21runrttasks08:14:031
31459996734,10cyclictest9566-21kworker/u18:3-rpciod09:51:201
31459996648,8cyclictest0-21swapper/109:59:531
31459996647,13cyclictest2540-21apt-key12:39:511
31459996647,13cyclictest2540-21apt-key12:39:511
3145999663,47cyclictest12888-21latency_hist11:09:541
3145999663,44cyclictest5286-21awk09:05:221
31459996545,14cyclictest28716-21sh08:39:501
3145999654,44cyclictest25492-21unixbench_multi09:40:271
31459996524,30cyclictest1-21systemd10:49:511
31459996417,19cyclictest31560-21apt-get11:40:001
3145999633,46cyclictest20785-21ssh12:15:151
3145999633,43cyclictest23939-21ntp_kernel_err12:20:191
3145999633,43cyclictest23939-21ntp_kernel_err12:20:191
3145999624,39cyclictest4509-21fschecks_count10:55:121
3145999624,37cyclictest2014-21timerandwakeup07:15:271
31459996239,12cyclictest0-21swapper/111:20:191
31459996235,18cyclictest12223-21vmstat07:45:281
3145999623,34cyclictest346-21runrttasks08:32:501
3145999622,41cyclictest9732-21cstates07:40:081
3145999613,44cyclictest23501-21grep08:20:261
31468996018,26cyclictest0-21swapper/307:35:153
31459996039,11cyclictest0-21swapper/110:05:231
3145999593,36cyclictest24406-21fschecks_time08:25:131
3146899582,41cyclictest27615-21taskset10:39:533
3146899563,38cyclictest4406-21chrt09:04:523
3146899563,38cyclictest4406-21chrt09:04:523
3146899562,39cyclictest23334-21taskset12:19:573
3146899562,39cyclictest23334-21taskset12:19:573
3146899553,38cyclictest7581-21chrt11:00:153
3146899543,37cyclictest17889-21chrt10:23:033
3146899543,36cyclictest7521-21chrt11:54:203
3146899543,35cyclictest13696-21chrt10:15:263
3146899542,41cyclictest2531-21taskset12:39:503
3146899542,41cyclictest2531-21taskset12:39:503
3146899534,35cyclictest4105-21taskset10:00:103
3146899533,39cyclictest1320-21taskset11:43:343
3146899533,37cyclictest4769-21chrt11:49:503
3146899533,36cyclictest29217-21chrt10:41:233
3146899533,35cyclictest3004-21chrt07:20:113
3146899533,35cyclictest14049-21chrt12:04:543
3146899533,35cyclictest14049-21chrt12:04:543
3146899524,35cyclictest2171-21chrt09:55:493
3146899524,34cyclictest4526-21taskset10:55:123
3146899523,34cyclictest24314-21chrt11:27:503
3146899522,33cyclictest8474-21chrt11:55:173
3146899522,33cyclictest8474-21chrt11:55:173
3146899522,31cyclictest28941-21chrt08:39:543
3146899513,36cyclictest26263-21taskset08:30:393
3146899513,35cyclictest1711-21chrt08:55:163
3146899512,36cyclictest18085-21chrt11:17:043
3146899512,36cyclictest18085-21chrt11:17:043
3146899512,34cyclictest17874-21taskset09:29:023
31468995037,8cyclictest451rcuc/309:30:503
31468995037,8cyclictest451rcuc/309:30:503
3146899503,35cyclictest14595-21chrt07:55:093
3146899503,34cyclictest24755-21chrt09:40:133
3146899503,33cyclictest5971-21chrt09:09:503
3146899503,32cyclictest31752-21chrt09:51:073
3146899503,29cyclictest23232-21chrt12:19:533
3146899502,37cyclictest6875-21chrt09:10:183
3146899502,32cyclictest25720-21taskset08:29:543
31468994934,9cyclictest451rcuc/311:07:293
3146899493,31cyclictest22508-21chrt08:19:543
3146899492,38cyclictest29827-21taskset12:30:183
3146899492,34cyclictest21908-21chrt10:30:083
31468994911,26cyclictest17951-21chrt08:05:133
3146899483,37cyclictest754-21taskset08:50:273
3146899483,36cyclictest11905-21chrt10:13:033
3146899483,33cyclictest28728-21taskset12:29:093
3146899483,33cyclictest13914-21chrt09:21:413
3146899482,35cyclictest14257-21chrt07:54:533
31468994733,9cyclictest451rcuc/307:43:033
3146899473,34cyclictest5857-21taskset07:29:503
3146899473,32cyclictest27609-21taskset09:45:073
3146899473,32cyclictest27609-21taskset09:45:073
3146899472,36cyclictest31946-21chrt07:10:113
3146899472,36cyclictest31946-21chrt07:10:113
3146899472,32cyclictest3605-21taskset10:53:553
3146899463,32cyclictest21745-21chrt11:23:573
3146899463,31cyclictest7835-21chrt10:05:253
3146899463,26cyclictest32149-21chrt08:49:543
3146899453,30cyclictest12600-21chrt07:49:503
3146899453,28cyclictest2637-21chrt07:19:513
3146899443,29cyclictest6437-21chrt07:30:153
3146899443,29cyclictest6437-21chrt07:30:153
31468994432,8cyclictest451rcuc/308:23:443
31468994330,9cyclictest451rcuc/308:03:323
3146899432,29cyclictest30933-21chrt10:45:083
3146899423,31cyclictest18921-21chrt10:25:063
31468994126,8cyclictest451rcuc/308:40:473
31468994124,12cyclictest451rcuc/308:12:143
31468994014,15cyclictest0-21swapper/309:39:093
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional