You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-09 - 22:15
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Tue Dec 09, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
354221247,17sleep00-21swapper/007:06:430
354221247,17sleep00-21swapper/007:06:430
39369912231,58cyclictest0-21swapper/009:33:380
39369911153,53cyclictest26743-21ssh12:33:350
39369910839,62cyclictest0-21swapper/012:28:150
39369910839,62cyclictest0-21swapper/012:28:150
39369910739,62cyclictest0-21swapper/011:43:270
39369910739,62cyclictest0-21swapper/011:43:270
39369910625,73cyclictest0-21swapper/009:43:160
39499910584,10cyclictest0-21swapper/111:53:561
39369910537,62cyclictest355-21runrttasks09:55:190
3936991053,24cyclictest4294-21/usr/sbin/munin11:53:410
39369910418,21cyclictest20249-21diskmemload12:22:350
39499910345,20cyclictest16859-21kworker/u18:0+nfsiod12:29:301
39369910344,52cyclictest15006-21rm09:13:490
39369910342,55cyclictest0-21swapper/009:04:070
3936991023,27cyclictest24193-21sh09:32:100
39369910219,24cyclictest5730-21latency_hist07:13:170
3970991013,41cyclictest20824-21ssh12:23:312
3970991013,41cyclictest20824-21ssh12:23:312
39369910143,51cyclictest0-21swapper/011:58:590
39369910143,51cyclictest0-21swapper/011:58:590
3936991013,27cyclictest27032-21sort10:38:160
39369910039,55cyclictest0-21swapper/010:48:520
39369910039,55cyclictest0-21swapper/010:48:520
3936991002,24cyclictest14544-21chrt12:12:170
393699993,26cyclictest11102-21/usr/sbin/munin07:28:400
393699993,26cyclictest11102-21/usr/sbin/munin07:28:400
393699993,23cyclictest11922-21/usr/sbin/munin11:08:330
393699993,22cyclictest32489-21sh09:48:140
393699993,22cyclictest32489-21sh09:48:140
393699993,21cyclictest5147-21ssh10:56:520
393699992,30cyclictest17169-21grep07:44:060
393699992,30cyclictest17169-21grep07:44:060
3936999843,48cyclictest0-21swapper/007:28:170
3936999838,53cyclictest0-21swapper/009:28:190
393699983,30cyclictest21698-21awk07:58:500
393699982,26cyclictest29369-21ls11:38:520
393699982,26cyclictest29369-21ls11:38:520
3936999742,48cyclictest0-21swapper/010:13:360
3936999742,48cyclictest0-21swapper/010:13:360
3936999733,57cyclictest0-21swapper/011:03:480
3936999733,57cyclictest0-21swapper/011:03:480
393699973,30cyclictest8861-21grep07:18:510
393699973,26cyclictest8146-21fschecks_count08:58:350
393699973,24cyclictest27950-21ssh11:37:440
393699973,23cyclictest30495-21ssh10:43:410
3970999638,52cyclictest25799-21sh10:34:352
3970999629,11cyclictest0-21swapper/207:18:422
3936999645,43cyclictest0-21swapper/008:28:300
393699963,28cyclictest15622-21/usr/sbin/munin11:13:540
393699963,28cyclictest15622-21/usr/sbin/munin11:13:540
393699963,23cyclictest25015-21grep12:29:440
3936999629,62cyclictest0-21swapper/011:28:420
3936999629,58cyclictest0-21swapper/008:48:160
3936999629,58cyclictest0-21swapper/008:48:160
393699962,23cyclictest7110-21runrttasks10:59:500
393699962,23cyclictest16633-21kernelversion10:18:400
393699962,23cyclictest1-21systemd10:13:150
393699953,30cyclictest22637-21/usr/sbin/munin08:03:330
393699952,42cyclictest19694-21cron11:23:130
393699952,22cyclictest17733-21timerandwakeup09:18:580
393699952,21cyclictest13154-21latency07:33:410
3936999437,50cyclictest0-21swapper/007:53:320
393699943,28cyclictest9033-21timerwakeupswit10:03:520
393699942,35cyclictest29389-21sshd10:42:190
393699942,24cyclictest6471-21fschecks_count07:13:390
393699942,24cyclictest26085-21fschecks_count08:13:360
393699942,21cyclictest3582-21taskset11:51:590
7732930,7sleep30-21swapper/310:48:413
7732930,7sleep30-21swapper/310:48:413
3970999339,47cyclictest0-21swapper/210:53:542
3970999325,18cyclictest0-21swapper/211:23:152
393699933,36cyclictest10588-21mailstats12:03:490
393699933,25cyclictest32742-21/usr/sbin/munin09:48:520
393699933,25cyclictest32742-21/usr/sbin/munin09:48:520
393699933,25cyclictest22311-21ssh10:28:580
393699933,24cyclictest22712-21ssh11:28:140
393699932,39cyclictest17542-21rm12:17:570
3970999233,9cyclictest0-21swapper/212:13:502
393699924,20cyclictest17802-21apt-get07:48:240
393699923,25cyclictest14339-21/usr/sbin/munin07:38:400
3936999227,7cyclictest0-21swapper/008:33:510
3936999225,9cyclictest0-21swapper/008:23:300
393699922,34cyclictest6949-21/usr/sbin/munin08:53:500
3936999223,13cyclictest0-21swapper/008:18:360
3970999166,19cyclictest4093-21latency_hist11:53:162
393699913,25cyclictest12832-21/usr/sbin/munin09:09:500
393699913,23cyclictest19197-21fschecks_count10:23:370
3936999123,9cyclictest0-21swapper/009:58:430
3936999123,9cyclictest0-21swapper/009:58:430
393699912,24cyclictest6066-21modprobe08:53:130
393699912,24cyclictest6066-21modprobe08:53:130
29032918,56sleep22932-21/usr/sbin/munin07:03:562
29032918,56sleep22932-21/usr/sbin/munin07:03:562
3970999033,50cyclictest0-21swapper/207:28:462
3970999033,50cyclictest0-21swapper/207:28:462
397099903,30cyclictest5412-21grep11:53:572
3970999031,51cyclictest0-21swapper/209:23:522
3970999030,8cyclictest0-21swapper/207:13:512
393699903,22cyclictest1494-21/usr/sbin/munin08:38:420
393699893,22cyclictest24523-21grep08:08:380
397099882,27cyclictest11170-21cmp09:08:162
3970998743,37cyclictest12374-21open_files09:09:432
3970998741,39cyclictest20468-21latency_hist12:23:192
3970998739,41cyclictest0-21swapper/210:43:422
3970998738,42cyclictest27677-21forks08:18:352
3970998731,13cyclictest13516-21timerandwakeup12:08:582
397099872,29cyclictest8147-21fschecks_count08:58:352
3970998636,43cyclictest28283-21timerandwakeup09:38:492
3970998636,41cyclictest0-21swapper/211:03:412
3970998636,41cyclictest0-21swapper/211:03:412
3970998635,45cyclictest19787-21latency07:53:402
3970998633,7cyclictest0-21swapper/209:32:102
397099863,27cyclictest8652-21latency_hist11:03:162
3970998630,8cyclictest0-21swapper/209:13:542
3970998628,9cyclictest0-21swapper/208:48:502
3970998628,9cyclictest0-21swapper/208:48:502
3970998627,11cyclictest0-21swapper/207:53:172
3970998626,10cyclictest0-21swapper/208:33:372
397099862,38cyclictest32743-21apt-get10:48:262
397099862,38cyclictest32743-21apt-get10:48:262
397099862,30cyclictest3249-21ssh09:53:462
3970998560,18cyclictest16469-21fschecks_count10:18:362
397099853,29cyclictest8475-21/usr/sbin/munin10:03:392
397099852,8cyclictest181rcu_preempt10:28:362
3970998528,8cyclictest0-21swapper/209:18:572
3970998525,11cyclictest0-21swapper/208:03:462
397099852,33cyclictest26329-21grep12:33:162
397099852,27cyclictest12488-21latency_hist12:08:182
397099852,26cyclictest1351-21head09:48:502
397099852,26cyclictest1351-21head09:48:502
397099852,23cyclictest355-21runrttasks10:11:252
3970998515,28cyclictest14442-21ls07:38:332
397099843,31cyclictest7683-21latency_hist08:58:172
3970998432,8cyclictest0-21swapper/211:08:332
3970998431,7cyclictest0-21swapper/210:23:322
3970998430,8cyclictest0-21swapper/209:58:312
3970998430,8cyclictest0-21swapper/209:58:312
3970998423,11cyclictest0-21swapper/208:01:502
397099842,25cyclictest29097-21ssh10:41:212
3970998417,24cyclictest31893-21seq08:31:162
3970998333,42cyclictest0-21swapper/209:46:392
3970998333,42cyclictest0-21swapper/209:46:392
397099833,25cyclictest14857-21grep11:13:392
397099833,25cyclictest14857-21grep11:13:392
397099833,24cyclictest2858-21munin-plugin-st08:43:142
3970998332,46cyclictest3909-21cyclictest11:38:232
3970998332,46cyclictest3909-21cyclictest11:38:232
3970998330,8cyclictest0-21swapper/212:33:482
397099832,27cyclictest24643-21latency08:08:412
3970998310,20cyclictest14273-21cat10:13:482
3970998310,20cyclictest14273-21cat10:13:482
397099823,26cyclictest26799-21unixbench_singl11:33:512
397099823,25cyclictest22258-21seq11:26:492
397099823,24cyclictest3184-21cpuspeed08:43:322
397099823,24cyclictest3184-21cpuspeed08:43:322
3970998231,7cyclictest0-21swapper/209:35:542
3970998224,8cyclictest0-21swapper/207:23:452
397099813,25cyclictest31383-21apt-get11:43:242
397099813,25cyclictest31383-21apt-get11:43:242
397099812,30cyclictest7400-21fschecks_time11:58:382
397099812,30cyclictest7400-21fschecks_time11:58:382
397099812,25cyclictest13122-21latency07:33:412
3949998161,15cyclictest29855-21sh11:39:541
3949998161,15cyclictest29855-21sh11:39:541
397099804,24cyclictest16414-21cut07:43:512
397099804,24cyclictest16414-21cut07:43:512
397099802,27cyclictest26121-21fschecks_time08:13:372
279328017,34sleep30-21swapper/307:03:493
279328017,34sleep30-21swapper/307:03:493
397099793,29cyclictest25549-21latency_hist11:33:162
3970997923,12cyclictest0-21swapper/208:23:382
3970997917,21cyclictest4215-21cat07:08:252
3949997862,9cyclictest9905-21kworker/u17:3+rpciod10:18:501
3949997849,21cyclictest23130-21cron12:28:131
3949997849,21cyclictest23130-21cron12:28:131
3949997657,11cyclictest292-21dbus-daemon12:23:151
3949997560,10cyclictest3909-21cyclictest10:53:531
3949997555,13cyclictest7707-21sendmail08:58:141
3949997548,20cyclictest9096-21kworker/u18:2+nfsiod10:07:371
3949997459,9cyclictest58-21kcompactd012:09:021
3949997458,10cyclictest25335-21netstat09:33:421
3949997456,13cyclictest9939-21sh11:03:521
3949997456,13cyclictest9939-21sh11:03:521
368827412,41sleep13676-21apt-get07:08:151
368827412,41sleep13676-21apt-get07:08:151
3949997356,11cyclictest5133-21systemd-run09:58:151
3949997356,11cyclictest12665-21/usr/sbin/munin07:33:511
3949997353,13cyclictest2968-21sh09:53:181
3949997353,13cyclictest2968-21sh09:53:181
3949997257,9cyclictest1351-21kworker/u17:3-xprtiod11:02:161
3949997251,15cyclictest7023-21threads07:13:521
3949997151,14cyclictest10933-21diskmemload10:28:541
3949997056,8cyclictest0-21swapper/111:28:181
3949997055,10cyclictest3909-21cyclictest11:28:371
3949997051,13cyclictest8193-21fschecks_time08:58:361
3949996953,10cyclictest31466-21ssh09:44:451
3949996953,10cyclictest31466-21ssh09:44:451
3949996950,13cyclictest21731-21latency_hist09:28:181
3949996943,19cyclictest29326-21kworker/u18:1+nfsiod12:03:011
3949996943,19cyclictest29326-21kworker/u18:1+nfsiod12:03:011
3949996852,10cyclictest18143-21switchtime11:18:511
3949996851,11cyclictest292-21dbus-daemon08:23:151
3949996751,10cyclictest6376-21ssh09:58:511
3949996751,10cyclictest6376-21ssh09:58:511
3949996749,11cyclictest1194-21latency_hist11:48:191
3949996749,11cyclictest1194-21latency_hist11:48:191
3949996748,13cyclictest11864-21cstates09:09:291
3949996647,13cyclictest16533-21systemctl09:18:131
3949996646,14cyclictest28862-21latency_hist12:38:161
191992660,7sleep30-21swapper/309:23:163
3949996547,12cyclictest3909-21cyclictest10:43:361
3949996547,12cyclictest10933-21diskmemload09:29:461
3949996545,14cyclictest28896-21sh09:40:181
3949996448,9cyclictest0-21swapper/108:43:521
3949996448,9cyclictest0-21swapper/108:43:521
3949996448,8cyclictest0-21swapper/108:08:141
3949996343,14cyclictest355-21runrttasks11:08:451
3949996343,14cyclictest25849-21sh11:33:311
3949996343,14cyclictest10269-21threads07:23:511
3949996340,17cyclictest27572-21fschecks_time10:38:371
3949996333,21cyclictest2839-21cron08:43:141
3949996248,9cyclictest31-21ksoftirqd/112:13:531
3949996245,9cyclictest0-21swapper/107:53:511
394999624,49cyclictest1569-21ssh11:48:351
3949996243,13cyclictest316-21rs:main1
3949996243,11cyclictest0-21swapper/112:03:491
394999623,42cyclictest22532-21latency_hist08:03:171
3949996228,24cyclictest11178-21apt-key09:08:161
3949996149,7cyclictest0-21swapper/109:23:161
3949996148,9cyclictest3909-21cyclictest10:15:041
3949996148,9cyclictest3909-21cyclictest10:15:041
3949996148,8cyclictest15574-21kworker/u17:1+xprtiod11:13:511
3949996148,8cyclictest15574-21kworker/u17:1+xprtiod11:13:511
3949996146,9cyclictest31-21ksoftirqd/110:08:191
3949996146,8cyclictest0-21swapper/110:23:321
394999614,35cyclictest355-21runrttasks08:33:141
3949996142,13cyclictest24225-21kworker/u19:2+events_unbound07:13:141
3949996141,15cyclictest32767-21ssh10:48:311
3949996141,15cyclictest32767-21ssh10:48:311
3949996047,9cyclictest20408-21kworker/u17:0-rpciod10:34:091
3949996045,8cyclictest0-21swapper/107:43:171
3949995943,8cyclictest0-21swapper/108:13:141
394999593,41cyclictest12458-21cron07:33:141
394999593,41cyclictest12458-21cron07:33:141
394999593,35cyclictest355-21runrttasks08:36:241
394999593,32cyclictest4716-21/usr/sbin/munin08:48:321
394999593,32cyclictest4716-21/usr/sbin/munin08:48:321
3949995929,20cyclictest27207-21modprobe08:18:131
3949995745,7cyclictest0-21swapper/107:43:421
3949995745,7cyclictest0-21swapper/107:43:421
3949995736,14cyclictest0-21swapper/107:48:381
394999554,35cyclictest355-21runrttasks07:22:441
398499513,33cyclictest10822-21chrt10:08:323
3984994911,22cyclictest895-21kworker/3:1+events07:43:303
3984994911,22cyclictest895-21kworker/3:1+events07:43:303
398499485,31cyclictest9487-21taskset10:05:033
398499474,27cyclictest895-21kworker/3:1+events09:23:423
398499473,32cyclictest11752-21taskset11:08:273
398499473,31cyclictest20789-21taskset07:58:143
398499473,30cyclictest20278-21chrt11:23:273
398499472,33cyclictest7134-21taskset11:58:263
398499472,33cyclictest7134-21taskset11:58:263
398499472,32cyclictest1576-21chrt11:48:373
398499472,31cyclictest25758-21taskset11:33:253
398499465,29cyclictest28385-21taskset10:38:563
398499464,30cyclictest5630-21taskset10:58:143
398499464,30cyclictest25880-21taskset10:34:513
398499464,30cyclictest23887-21taskset09:30:553
398499464,28cyclictest895-21kworker/3:1+events07:28:513
398499464,28cyclictest895-21kworker/3:1+events07:28:513
398499463,31cyclictest31742-21taskset08:29:163
398499463,31cyclictest18007-21taskset12:18:283
398499463,30cyclictest894-21chrt09:48:423
398499463,30cyclictest894-21chrt09:48:423
398499463,30cyclictest31471-21taskset10:45:083
398499463,30cyclictest16128-21taskset11:15:293
398499463,30cyclictest16128-21taskset11:15:293
398499463,30cyclictest11629-21taskset09:09:023
398499463,29cyclictest27515-21taskset08:18:273
398499463,29cyclictest27463-21taskset09:38:293
398499463,29cyclictest12593-21taskset12:08:233
398499463,28cyclictest31390-21taskset11:43:263
398499463,28cyclictest31390-21taskset11:43:263
398499463,28cyclictest16259-21taskset10:18:293
398499462,31cyclictest22195-21taskset10:28:553
398499454,29cyclictest23908-21taskset12:29:063
398499454,29cyclictest21145-21taskset12:23:433
398499454,29cyclictest21145-21taskset12:23:433
398499453,32cyclictest11046-21chrt11:06:573
398499453,32cyclictest11046-21chrt11:06:573
398499453,30cyclictest30298-21taskset09:43:353
398499453,30cyclictest30298-21taskset09:43:353
398499453,30cyclictest25413-21taskset09:33:463
398499453,30cyclictest17342-21chrt11:18:293
398499453,30cyclictest13347-21taskset07:33:483
398499453,29cyclictest6807-21taskset07:13:473
398499453,29cyclictest6377-21taskset09:58:513
398499453,29cyclictest6377-21taskset09:58:513
398499453,29cyclictest26148-21taskset08:13:373
398499453,29cyclictest25616-21taskset11:33:163
398499453,28cyclictest8819-21taskset11:03:173
398499453,27cyclictest31229-21taskset11:43:183
398499452,30cyclictest6690-21taskset08:53:433
398499452,30cyclictest28204-21taskset12:35:513
398499452,30cyclictest21187-21chrt07:58:343
398499452,30cyclictest14474-21taskset10:13:513
398499452,30cyclictest14474-21taskset10:13:513
398499452,30cyclictest14175-21taskset09:13:283
398499443,31cyclictest4880-21taskset11:53:443
398499443,30cyclictest9516-21chrt07:23:273
398499443,29cyclictest3128-21taskset08:43:303
398499443,29cyclictest3128-21taskset08:43:303
398499443,29cyclictest30148-21taskset11:40:503
398499443,28cyclictest7975-21chrt08:58:243
398499443,28cyclictest4752-21taskset07:09:383
398499443,28cyclictest19010-21taskset10:23:323
398499443,28cyclictest15181-21taskset12:13:253
398499443,27cyclictest7916-21taskset07:18:293
398499442,30cyclictest3626-21taskset09:55:263
398499442,29cyclictest1714-21chrt08:38:363
398499433,28cyclictest29118-21taskset08:23:253
398499433,27cyclictest24241-21taskset08:08:233
398499423,29cyclictest19208-21taskset07:53:163
398499423,26cyclictest4832-21taskset08:48:323
398499423,26cyclictest4832-21taskset08:48:323
398499422,29cyclictest15459-21taskset07:40:203
398499422,27cyclictest9817-21taskset12:03:243
398499422,27cyclictest22636-21taskset08:03:243
398499413,28cyclictest1149-21taskset08:37:313
3984993925,7cyclictest0-21swapper/309:03:563
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional