You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-29 - 16:50
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Mon Dec 29, 2025 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1724321239,83sleep117264-21df07:04:531
186179911366,37cyclictest0-21swapper/211:00:042
18617991112,79cyclictest26497-21diskstats10:05:302
186089911031,9cyclictest0-21swapper/011:45:070
186089911031,9cyclictest0-21swapper/011:45:070
18608991094,46cyclictest23992-21/usr/sbin/munin10:55:110
18608991094,46cyclictest23992-21/usr/sbin/munin10:55:110
186089910939,62cyclictest0-21swapper/010:20:020
186089910939,62cyclictest0-21swapper/010:20:020
18608991093,35cyclictest20152-21/usr/sbin/munin10:49:500
18608991093,29cyclictest29483-21ps07:40:040
18608991093,29cyclictest29483-21ps07:40:040
18617991072,84cyclictest22917-21apt-get09:59:462
18608991064,34cyclictest26612-21timerwakeupswit11:55:070
18608991063,43cyclictest31826-21latency_hist11:09:340
186089910631,20cyclictest32262-21switchtime12:05:040
186179910582,16cyclictest30676-21ssh11:07:102
18608991053,33cyclictest4995-21fschecks_time08:04:530
18608991043,33cyclictest17037-21fschecks_count09:49:530
18608991037,21cyclictest12213-21grep10:35:070
18608991037,21cyclictest12213-21grep10:35:070
18608991033,37cyclictest26622-21grep07:30:080
186089910243,51cyclictest0-21swapper/010:10:040
186089910243,51cyclictest0-21swapper/010:10:040
186089910242,53cyclictest1-21systemd12:20:040
18608991023,32cyclictest17470-21latency_hist08:44:320
18608991023,32cyclictest17470-21latency_hist08:44:320
18608991014,45cyclictest11642-21latency08:24:570
18608991014,40cyclictest25731-21latency_hist09:09:350
186089910133,61cyclictest0-21swapper/012:34:590
18608991013,28cyclictest89-21kswapd011:49:430
186089910130,9cyclictest0-21swapper/010:29:320
186089910128,10cyclictest0-21swapper/011:09:450
186089910128,10cyclictest0-21swapper/011:09:450
186089910127,13cyclictest0-21swapper/011:35:070
18608991004,33cyclictest8616-21mailstats08:15:060
186089910042,50cyclictest0-21swapper/007:34:520
186089910042,50cyclictest0-21swapper/007:34:520
18608991003,40cyclictest28415-21latency_hist09:14:330
186089910033,7cyclictest0-21swapper/010:40:440
18608991003,34cyclictest22384-21latency_hist08:59:330
18608991003,23cyclictest10944-21ssh11:29:320
186089910026,22cyclictest320-21dbus-daemon11:19:310
18608991002,34cyclictest24310-21grep07:24:540
186089910018,24cyclictest26494-21cstates11:00:540
1860899997,33cyclictest5236-21uname12:14:570
1860899996,30cyclictest25499-21diskmemload10:05:220
18608999939,54cyclictest25499-21diskmemload12:14:320
1860899992,36cyclictest4662-21latency_hist08:04:340
1860899992,36cyclictest4662-21latency_hist08:04:340
1860899992,35cyclictest537-21unixbench_singl10:15:080
3592980,8sleep10-21swapper/110:15:071
18617999834,38cyclictest20515-21open_files11:45:022
18617999834,38cyclictest20515-21open_files11:45:022
1860899985,34cyclictest11665-21ssh09:40:050
1860899983,40cyclictest2434-21fschecks_count09:24:520
1860899983,36cyclictest21170-21kernelversion07:14:580
1860899983,34cyclictest15298-21sort08:35:070
1860899983,33cyclictest19464-21fschecks_count08:49:520
1860899983,33cyclictest19464-21fschecks_count08:49:520
1860899983,31cyclictest5817-21/usr/sbin/munin11:19:560
1860899983,30cyclictest25540-21apt-get10:04:300
1860899983,30cyclictest18616-21latency_hist07:09:350
18608999829,61cyclictest0-21swapper/009:55:070
18608999829,61cyclictest0-21swapper/009:55:070
1860899974,35cyclictest8381-21cstates10:29:510
1860899974,33cyclictest31127-21sendmail_mailqu07:45:040
1860899974,33cyclictest31127-21sendmail_mailqu07:45:040
1860899973,32cyclictest758-21ssh09:21:200
1860899973,32cyclictest758-21ssh09:21:200
18608999725,9cyclictest0-21swapper/007:50:000
18608999722,16cyclictest0-21swapper/009:48:560
18615999669,18cyclictest346-21runrttasks10:50:061
18608999633,7cyclictest0-21swapper/009:34:560
1860899963,34cyclictest16180-21latency_hist12:34:330
1860899963,32cyclictest20329-21munin-setwarn07:14:350
1860899962,37cyclictest5510-21cut09:29:580
18608999614,30cyclictest24031-21latency_hist09:04:340
1860899954,28cyclictest17008-21ssh11:39:450
1860899953,34cyclictest29384-21/usr/sbin/munin09:15:040
1860899953,34cyclictest29384-21/usr/sbin/munin09:15:040
1860899953,34cyclictest17022-21/usr/sbin/munin10:44:520
1802029516,32sleep20-21swapper/207:05:192
1860899944,32cyclictest10607-21/usr/sbin/munin12:24:400
1860899943,34cyclictest3033-21latency_hist07:59:340
1860899943,32cyclictest13317-21sshd08:30:010
18608999430,8cyclictest0-21swapper/008:10:010
1860899942,32cyclictest18645-21grep08:45:070
1860899942,32cyclictest18645-21grep08:45:070
1862399939,46cyclictest0-21swapper/309:59:503
1860899933,35cyclictest29628-21grep12:00:100
1860899933,35cyclictest29628-21grep12:00:100
1860899933,29cyclictest9811-21fschecks_count08:19:520
1860899924,35cyclictest22837-21grep07:19:570
18615999145,40cyclictest5131-21sh11:19:311
1860899913,31cyclictest11302-21runrttasks11:29:480
1861799887,70cyclictest32080-21ssh11:09:502
1861799887,70cyclictest32080-21ssh11:09:502
18617998865,14cyclictest18601-21cyclictest10:51:562
18615998846,35cyclictest16298-21runrttasks11:38:291
18615998747,34cyclictest27035-21kworker/u17:2-xprtiod09:29:541
1861799866,54cyclictest2619-21apt-get07:59:442
1861799866,54cyclictest2619-21apt-get07:59:442
18615998646,33cyclictest0-21swapper/111:29:561
1861599864,34cyclictest346-21runrttasks09:44:521
1861599863,39cyclictest9081-21modprobe08:16:301
1861599862,31cyclictest19436-21idleruntime-cro09:54:311
1861599862,22cyclictest24057-21apt-get07:24:431
1861599854,35cyclictest10637-21munin-run09:39:311
18615998537,41cyclictest5583-21timerandwakeup08:05:071
176702858,19sleep30-21swapper/307:05:123
1861599844,31cyclictest28910-21taskset07:39:451
1861599844,31cyclictest28910-21taskset07:39:451
1861599843,36cyclictest16299-21cat12:34:341
1861599843,33cyclictest28756-21cat11:04:341
1861599843,32cyclictest20497-21ssh11:45:011
1861599843,32cyclictest20497-21ssh11:45:011
1861599842,45cyclictest60-21kcompactd011:09:351
1861599842,45cyclictest60-21kcompactd011:09:351
18615998419,57cyclictest22893-21apt-get10:54:441
18615998419,57cyclictest22893-21apt-get10:54:441
1861599834,36cyclictest31455-21/usr/sbin/munin12:05:031
1861599833,37cyclictest31465-21latency_hist09:19:331
1861599833,37cyclictest31465-21latency_hist09:19:331
1861599833,33cyclictest16985-21/usr/sbin/munin11:40:111
1861599833,26cyclictest11644-21grep09:40:041
1861599823,38cyclictest7750-21unin-run08:14:321
1861599823,37cyclictest30923-21wc07:44:571
1861599823,37cyclictest30923-21wc07:44:571
1861599823,36cyclictest3687-21sshd08:00:171
1861599823,36cyclictest3687-21sshd08:00:171
1861599823,33cyclictest28334-21latency_hist11:59:331
1861599823,26cyclictest20280-21sh09:54:581
1861599823,26cyclictest20280-21sh09:54:581
1861599822,29cyclictest26306-21apt-get10:05:211
1861599822,26cyclictest6044-21open_files11:20:031
18615998214,28cyclictest11812-21processes08:25:041
18615998213,27cyclictest2904-21cat12:10:071
18615998211,29cyclictest28806-21sort11:59:541
18615998211,29cyclictest28806-21sort11:59:541
1861799812,44cyclictest14027-21latency_hist10:39:342
1861799812,44cyclictest14027-21latency_hist10:39:342
18617998116,20cyclictest346-21runrttasks11:18:012
18615998164,11cyclictest11475-21fschecks_time10:34:531
18615998164,11cyclictest11475-21fschecks_time10:34:531
1861599814,35cyclictest22275-21sendmail08:59:311
1861599814,26cyclictest18193-21/usr/sbin/munin08:44:591
1861599814,26cyclictest18193-21/usr/sbin/munin08:44:591
1861599813,37cyclictest29902-21unixbench_multi11:05:071
1861599813,34cyclictest5366-21ssh10:24:481
1861599813,24cyclictest25791-21/usr/sbin/munin09:10:051
1861599812,26cyclictest23108-21grep09:00:041
1861599812,22cyclictest15341-21unixbench_multi08:35:081
1862399809,49cyclictest0-21swapper/310:04:453
1861599806,31cyclictest17739-21rm12:35:481
1861599803,38cyclictest24843-21switchtime09:05:051
1861599803,35cyclictest7866-21cstates12:19:481
1861599803,34cyclictest20794-21/usr/sbin/munin07:14:441
1861599803,26cyclictest32264-21chrt07:49:491
1861599803,26cyclictest20635-21latency_hist08:54:341
1861599803,26cyclictest20635-21latency_hist08:54:341
18615998024,49cyclictest0-21swapper/112:29:331
1861599802,26cyclictest3025-21find09:25:061
1861599802,26cyclictest17286-21fschecks_count10:44:521
1861599802,25cyclictest13332-21/usr/sbin/munin08:30:011
1861599802,24cyclictest9310-21ssh10:30:101
1861599802,24cyclictest8864-21ssh11:25:001
18617997963,11cyclictest18601-21cyclictest12:30:442
18617997921,21cyclictest1834-21rm09:24:292
18617997921,21cyclictest1834-21rm09:24:292
1861599794,32cyclictest27752-21ntpq07:34:581
1861599794,32cyclictest27752-21ntpq07:34:581
1861599793,36cyclictest22346-21apt-get09:59:421
1861599793,28cyclictest28995-21apt-get10:09:421
1861599793,28cyclictest28995-21apt-get10:09:421
1861599793,26cyclictest14118-21/usr/sbin/munin10:40:091
1861599793,23cyclictest19094-21/usr/sbin/munin07:09:551
18615997929,42cyclictest0-21swapper/111:50:001
1861599792,26cyclictest3263-21runrttasks10:20:041
1861599792,26cyclictest3263-21runrttasks10:20:041
18617997853,16cyclictest0-21swapper/208:49:522
18617997853,16cyclictest0-21swapper/208:49:522
1861599784,30cyclictest31670-21/usr/sbin/munin09:20:051
1861599784,30cyclictest31670-21/usr/sbin/munin09:20:051
1861599783,22cyclictest16059-21/usr/sbin/munin08:39:441
1861599783,22cyclictest16059-21/usr/sbin/munin08:39:441
1861599782,29cyclictest26187-21/usr/sbin/munin07:30:001
1861599782,22cyclictest7458-21modprobe12:19:311
1861599782,22cyclictest2347-21seq07:56:071
1861799773,41cyclictest19027-21cut07:09:532
1861599773,32cyclictest22819-21kernelversion07:19:571
1861599763,22cyclictest10001-21/usr/sbin/munin08:19:561
18617997417,19cyclictest7399-21seq08:10:332
18617997354,13cyclictest13775-21latency_hist11:34:332
18623997211,39cyclictest0-21swapper/311:09:513
18623997211,39cyclictest0-21swapper/311:09:513
18617997149,12cyclictest0-21swapper/209:29:542
18617997144,19cyclictest25966-21fschecks_time11:54:532
1861799713,38cyclictest24326-21fschecks_time07:24:542
1861799712,45cyclictest21577-21timerandwakeup07:15:072
18617997116,20cyclictest6088-21seq08:09:282
18617997113,27cyclictest0-21swapper/211:25:062
18617997051,12cyclictest329-21systemd-logind11:54:312
18617997015,23cyclictest346-21runrttasks12:14:512
18617996950,13cyclictest201-21systemd-journal08:25:202
18617996949,11cyclictest11593-21kworker/u17:4-xprtiod09:50:072
18617996946,16cyclictest23800-21cut07:24:322
18617996847,14cyclictest18601-21cyclictest09:39:352
18617996846,13cyclictest6000-21kworker/u17:3-rpciod12:19:482
18617996841,17cyclictest0-21swapper/209:40:082
18617996839,9cyclictest181rcu_preempt07:55:082
189632670,8sleep30-21swapper/312:39:253
1862399679,36cyclictest0-21swapper/308:00:013
1862399679,36cyclictest0-21swapper/308:00:013
18617996748,9cyclictest0-21swapper/209:55:082
18617996748,9cyclictest0-21swapper/209:55:082
18617996748,13cyclictest19795-21ssh10:49:332
18617996742,13cyclictest0-21swapper/208:59:522
18617996736,21cyclictest5416-21apt-get11:19:462
18617996721,32cyclictest29825-21grep07:40:082
18617996721,32cyclictest29825-21grep07:40:082
18617996651,9cyclictest29559-21ntp_kernel_pll_10:09:592
18617996651,9cyclictest29559-21ntp_kernel_pll_10:09:592
18617996650,9cyclictest0-21swapper/210:59:312
18617996650,9cyclictest0-21swapper/210:59:312
18617996645,11cyclictest0-21swapper/210:30:012
18617996643,15cyclictest0-21swapper/210:44:332
18617996641,17cyclictest27780-21sh09:12:422
18617996640,14cyclictest0-21swapper/209:14:542
18617996640,14cyclictest0-21swapper/209:14:542
18617996625,30cyclictest5780-21cat10:24:582
18617996616,37cyclictest4356-21ssh12:14:072
18617996616,36cyclictest17857-21cat11:40:092
1861799661,26cyclictest14363-21fschecks_time11:34:532
1861799654,41cyclictest32200-21fschecks_time10:14:532
1861799653,44cyclictest9605-21/usr/sbin/munin08:20:042
1861799653,40cyclictest16007-21latency_hist08:39:352
18617996445,12cyclictest201-21systemd-journal08:49:302
18617996445,12cyclictest201-21systemd-journal08:49:302
18617996438,16cyclictest0-21swapper/207:45:072
18617996438,16cyclictest0-21swapper/207:45:072
1861799643,44cyclictest346-21runrttasks07:51:132
1861799643,24cyclictest14218-21latency_hist08:34:332
1861799642,47cyclictest27683-21wc07:34:552
1861799642,47cyclictest27683-21wc07:34:552
1861799642,34cyclictest32058-21ntpq12:04:582
1861799639,33cyclictest25715-21latency_hist09:09:352
18617996340,15cyclictest0-21swapper/212:04:332
18617996340,15cyclictest0-21swapper/212:04:332
1861799633,39cyclictest346-21runrttasks08:16:052
1861799633,38cyclictest346-21runrttasks08:55:542
1861799633,38cyclictest13350-21grep12:29:332
18617996235,7cyclictest181rcu_preempt09:44:532
1861799623,38cyclictest16057-21/usr/sbin/munin08:39:562
1861799623,38cyclictest16057-21/usr/sbin/munin08:39:562
1861799622,36cyclictest346-21runrttasks09:28:572
18617996116,32cyclictest17118-21awk12:35:042
1861799604,27cyclictest2411-21apt-get10:19:402
1861799604,27cyclictest2411-21apt-get10:19:402
1861799603,41cyclictest27132-21latency_hist07:34:332
18623995837,14cyclictest451rcuc/310:55:533
18623995837,14cyclictest451rcuc/310:55:533
18623995813,33cyclictest0-21swapper/310:59:513
18623995636,13cyclictest451rcuc/309:55:573
18623995636,13cyclictest451rcuc/309:55:573
18623995543,8cyclictest451rcuc/310:39:573
1862399553,41cyclictest7547-21taskset10:28:253
1862399553,39cyclictest24203-21taskset09:04:473
1862399544,35cyclictest10403-21chrt09:38:543
18623995441,8cyclictest451rcuc/311:22:183
1862399543,36cyclictest31302-21chrt12:04:333
1862399543,36cyclictest31302-21chrt12:04:333
18623995339,9cyclictest451rcuc/310:22:063
18623995339,9cyclictest451rcuc/310:22:063
18623995339,9cyclictest451rcuc/308:39:373
18623995339,9cyclictest451rcuc/308:39:373
1862399533,40cyclictest16209-21taskset09:48:433
18623995331,15cyclictest451rcuc/307:59:113
18623995239,9cyclictest451rcuc/310:53:023
1862399523,37cyclictest27258-21chrt07:34:413
1862399523,37cyclictest27258-21chrt07:34:413
1862399523,36cyclictest31478-21chrt12:04:413
1862399523,35cyclictest7865-21chrt12:19:483
1862399523,35cyclictest13936-21taskset11:34:333
18623995229,15cyclictest451rcuc/308:53:303
18623995229,15cyclictest451rcuc/308:53:303
1862399514,38cyclictest28233-21chrt11:59:303
18623995138,9cyclictest451rcuc/308:34:183
1862399513,37cyclictest9375-21taskset08:19:313
1862399513,36cyclictest9628-21chrt08:19:433
18623995129,15cyclictest451rcuc/311:15:213
18623995128,16cyclictest451rcuc/312:30:393
1862399512,37cyclictest30943-21chrt07:44:573
1862399512,37cyclictest30943-21chrt07:44:573
1862399512,36cyclictest18892-21taskset11:42:323
1862399504,32cyclictest14717-21chrt08:34:543
18623995036,9cyclictest451rcuc/311:37:553
18623995036,10cyclictest451rcuc/310:39:283
18623995036,10cyclictest451rcuc/310:39:283
18623995035,10cyclictest451rcuc/309:31:133
1862399503,38cyclictest2065-21taskset10:19:303
1862399503,36cyclictest11712-21taskset12:25:073
1862399503,34cyclictest11675-21chrt08:24:583
18623994935,9cyclictest451rcuc/309:20:443
18623994935,9cyclictest451rcuc/309:20:443
1862399493,37cyclictest32150-21chrt07:49:473
18623994926,15cyclictest451rcuc/308:06:363
1862399484,31cyclictest11898-21chrt09:40:083
18623994835,8cyclictest451rcuc/309:13:473
1862399483,34cyclictest26244-21chrt07:30:033
1862399483,33cyclictest18168-21chrt08:44:583
1862399483,33cyclictest18168-21chrt08:44:583
1862399483,32cyclictest19205-21chrt10:48:153
1862399483,31cyclictest8559-21chrt10:29:553
1862399483,30cyclictest30485-21chrt11:06:363
18623994829,12cyclictest451rcuc/309:26:033
1862399482,33cyclictest8435-21chrt11:24:493
1862399482,33cyclictest23865-21chrt07:24:333
1862399482,33cyclictest23534-21chrt11:50:063
1862399482,33cyclictest22180-21chrt11:48:443
1862399482,33cyclictest22180-21chrt11:48:443
1862399482,33cyclictest16925-21chrt09:49:483
1862399482,31cyclictest7174-21chrt08:10:083
1862399463,34cyclictest2618-21chrt12:10:043
1862399463,32cyclictest29709-21chrt09:15:083
1862399463,32cyclictest29709-21chrt09:15:083
1862399452,30cyclictest5064-21chrt12:14:543
1862399452,29cyclictest24176-21chrt07:24:493
1862399442,30cyclictest21457-21chrt07:15:063
1862399442,29cyclictest28893-21chrt07:39:433
1862399442,29cyclictest28893-21chrt07:39:433
18623994331,8cyclictest451rcuc/309:01:403
1862399403,26cyclictest28983-21chrt10:09:423
1862399403,26cyclictest28983-21chrt10:09:423
1862399402,26cyclictest19753-21taskset07:10:083
18623993925,10cyclictest451rcuc/308:56:253
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional