You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 13:10
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Thu Dec 04, 2025 00:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32039213414,82sleep032069-21ntpq19:04:120
576991303,115cyclictest7284-21munin-node23:53:460
576991182,25cyclictest26017-21/usr/sbin/munin20:23:500
576991142,62cyclictest5335-21/usr/sbin/munin21:59:000
576991142,27cyclictest27514-21irqstats00:29:060
5769911328,55cyclictest0-21swapper/019:13:520
576991112,26cyclictest26406-21rm22:34:290
576991112,25cyclictest6294-21seq22:56:170
576991102,24cyclictest25861-21ssh00:25:460
576991102,24cyclictest25861-21ssh00:25:460
576991092,25cyclictest30399-21rm21:46:280
576991092,22cyclictest26042-21/usr/sbin/munin21:39:010
5769910737,63cyclictest0-21swapper/022:59:000
5769910737,63cyclictest0-21swapper/022:59:000
576991072,25cyclictest10069-21sh21:11:360
576991072,25cyclictest10069-21sh21:11:360
5769910711,57cyclictest0-21swapper/020:49:000
576991063,7cyclictest0-21swapper/021:53:480
576991062,28cyclictest19013-21taskset20:03:190
5769910611,68cyclictest22278-21seq22:28:450
5769910611,68cyclictest22278-21seq22:28:450
5829910553,12cyclictest565-21cyclictest19:19:131
576991052,6cyclictest0-21swapper/022:38:460
576991052,6cyclictest0-21swapper/022:38:460
576991052,69cyclictest0-21swapper/023:39:130
576991052,24cyclictest19428-21rm21:28:250
586991043,78cyclictest3041-21apt-get19:13:502
576991042,23cyclictest5984-21rm23:51:190
576991032,22cyclictest9550-21ssh22:06:060
576991032,22cyclictest9550-21ssh22:06:060
5769910311,57cyclictest30448-21kworker/u17:4+nfsiod00:04:230
5769910311,57cyclictest30448-21kworker/u17:4+nfsiod00:04:230
5769910250,46cyclictest4277-21latency_hist22:53:380
5769910250,46cyclictest4277-21latency_hist22:53:380
576991022,82cyclictest14020-21apt-get21:18:490
576991022,24cyclictest20415-21taskset20:05:410
576991022,22cyclictest24003-21ssh23:26:520
5769910142,52cyclictest565-21cyclictest21:51:320
5769910127,51cyclictest0-21swapper/000:16:260
5769910127,51cyclictest0-21swapper/000:16:260
576991012,24cyclictest12842-21taskset23:08:220
5769910065,27cyclictest25904-21latency_hist20:23:390
576991002,25cyclictest14954-21rm22:15:060
576991002,24cyclictest21916-21sh22:28:250
57699994,83cyclictest2410-21latency_hist19:13:390
57699992,68cyclictest12517-21ssh00:03:050
57699992,68cyclictest12517-21ssh00:03:050
57699992,26cyclictest27212-21sh23:32:410
57699992,24cyclictest27623-21/usr/sbin/munin20:28:520
57699992,24cyclictest27623-21/usr/sbin/munin20:28:520
576999920,73cyclictest0-21swapper/022:08:550
57699983,22cyclictest22298-21runrttasks20:11:120
57699982,64cyclictest1244-21ssh23:43:450
57699982,64cyclictest1244-21ssh23:43:450
57699982,22cyclictest7433-21taskset19:25:070
57699972,69cyclictest31525-21ssh00:36:150
57699972,23cyclictest22951-21ssh00:20:280
57699972,23cyclictest22951-21ssh00:20:280
576999711,57cyclictest0-21swapper/022:23:320
576999711,57cyclictest0-21swapper/022:23:320
318082979,62sleep131845-21perl19:04:051
582999644,18cyclictest19601-21latency_hist21:28:381
57699963,84cyclictest0-21swapper/023:08:470
57699963,84cyclictest0-21swapper/023:08:470
57699963,12cyclictest7840-21diskmemload22:46:190
582999571,15cyclictest565-21cyclictest22:09:151
582999561,14cyclictest181rcu_preempt19:13:591
57699952,22cyclictest29538-21cat20:34:000
57699942,7cyclictest0-21swapper/021:33:490
57699942,56cyclictest7840-21diskmemload21:29:150
57699933,6cyclictest0-21swapper/020:13:500
576999322,42cyclictest0-21swapper/000:09:270
57699932,22cyclictest1490-21latency_hist20:48:380
57699924,81cyclictest0-21swapper/023:18:480
576999215,67cyclictest0-21swapper/023:38:380
576999215,67cyclictest0-21swapper/023:38:380
57699914,80cyclictest0-21swapper/021:13:500
576999128,56cyclictest0-21swapper/019:23:210
57699912,21cyclictest15601-21taskset19:51:110
57699912,21cyclictest15601-21taskset19:51:110
586999070,14cyclictest314-21in:imuxsock23:06:262
576999027,56cyclictest0-21swapper/021:03:580
57699902,26cyclictest9619-21apt-get19:33:480
57699883,58cyclictest0-21swapper/019:32:020
576998832,47cyclictest0-21swapper/019:43:460
576998832,47cyclictest0-21swapper/019:43:460
57699882,26cyclictest17173-21sed23:14:130
57699882,26cyclictest17173-21sed23:14:130
58699873,45cyclictest39-21ksoftirqd/222:24:012
57699873,70cyclictest0-21swapper/020:38:480
57699872,73cyclictest16069-21apt-get19:53:480
582998671,8cyclictest31-21ksoftirqd/123:18:391
582998655,9cyclictest181rcu_preempt21:53:501
58699856,43cyclictest39-21ksoftirqd/221:38:362
586998558,17cyclictest21758-21fschecks_time00:19:022
586998558,17cyclictest21758-21fschecks_time00:19:022
58699852,52cyclictest39-21ksoftirqd/200:09:542
58699852,39cyclictest39-21ksoftirqd/222:43:562
586998436,17cyclictest181rcu_preempt22:57:402
58699842,39cyclictest39-21ksoftirqd/219:59:512
582998460,16cyclictest29776-21latency_hist00:33:401
57699843,74cyclictest5487-21sendmail_mailqu20:59:090
57699843,74cyclictest5487-21sendmail_mailqu20:59:090
57699843,71cyclictest3239-21apt-get20:53:530
57699842,24cyclictest11925-21grep19:39:090
57699842,24cyclictest11925-21grep19:39:090
586998363,14cyclictest6525-21sh23:52:562
58699832,53cyclictest39-21ksoftirqd/220:54:072
58699832,51cyclictest39-21ksoftirqd/221:44:092
58699831,53cyclictest39-21ksoftirqd/220:24:102
582998365,13cyclictest32015-21fschecks_count21:49:251
582998349,9cyclictest181rcu_preempt21:34:101
586998237,16cyclictest181rcu_preempt23:31:592
586998162,14cyclictest17604-21processes21:24:132
58699812,37cyclictest39-21ksoftirqd/220:14:042
582998161,13cyclictest292-21dbus-daemon20:53:371
582998158,14cyclictest9428-21kworker/u17:2-rpciod23:35:441
582998158,14cyclictest9428-21kworker/u17:2-rpciod23:35:441
582998155,9cyclictest181rcu_preempt22:54:101
58699804,51cyclictest39-21ksoftirqd/200:29:122
586998040,16cyclictest181rcu_preempt19:39:092
586998040,16cyclictest181rcu_preempt19:39:092
58699803,42cyclictest39-21ksoftirqd/200:35:452
58699802,51cyclictest39-21ksoftirqd/221:53:522
58299806,30cyclictest9827-40ogrotate23:58:371
582998049,21cyclictest30789-21idleruntime-cro22:43:361
582998049,21cyclictest30789-21idleruntime-cro22:43:361
586997961,12cyclictest643-21latency_hist19:08:412
586997955,18cyclictest2222-21apt-key19:13:382
58699795,49cyclictest39-21ksoftirqd/221:53:142
58699792,47cyclictest39-21ksoftirqd/220:49:012
58699792,41cyclictest39-21ksoftirqd/223:54:002
582997956,16cyclictest13078-21ssh00:03:551
582997956,16cyclictest13078-21ssh00:03:551
582997937,12cyclictest181rcu_preempt22:22:001
582997937,12cyclictest181rcu_preempt22:22:001
58699782,50cyclictest39-21ksoftirqd/222:09:132
582997855,16cyclictest31256-21apt-key21:48:381
582997855,16cyclictest15855-21sh23:13:361
582997855,16cyclictest15855-21sh23:13:361
586997746,22cyclictest23059-21unixbench_singl23:24:122
58699774,37cyclictest39-21ksoftirqd/222:22:002
58699774,37cyclictest39-21ksoftirqd/222:22:002
586997736,23cyclictest39-21ksoftirqd/220:44:082
586997713,32cyclictest39-21ksoftirqd/221:08:552
586997713,32cyclictest39-21ksoftirqd/221:08:552
582997759,13cyclictest355-21runrttasks23:59:001
582997759,13cyclictest355-21runrttasks23:59:001
58299774,29cyclictest13689-21awk21:18:351
586997654,13cyclictest0-21swapper/221:19:142
586997652,9cyclictest181rcu_preempt21:31:372
586997637,13cyclictest181rcu_preempt20:04:152
586997636,16cyclictest181rcu_preempt19:50:402
586997636,16cyclictest181rcu_preempt19:50:402
586997629,12cyclictest0-21swapper/223:20:162
58699762,49cyclictest39-21ksoftirqd/219:34:082
582997657,13cyclictest355-21runrttasks22:17:091
582997656,12cyclictest0-21swapper/120:43:461
58299764,15cyclictest0-21swapper/121:28:491
586997556,13cyclictest7840-21diskmemload21:18:382
58699754,41cyclictest39-21ksoftirqd/222:38:362
586997539,7cyclictest181rcu_preempt22:29:042
586997539,7cyclictest181rcu_preempt22:29:042
58699752,42cyclictest39-21ksoftirqd/200:24:082
58699752,42cyclictest39-21ksoftirqd/200:24:082
586997521,36cyclictest39-21ksoftirqd/200:02:372
586997521,36cyclictest39-21ksoftirqd/200:02:372
586997515,50cyclictest381ktimers/200:13:592
586997515,50cyclictest381ktimers/200:13:592
582997556,12cyclictest355-21runrttasks23:21:431
582997554,14cyclictest0-21swapper/120:38:351
582997541,8cyclictest181rcu_preempt23:48:511
58299753,44cyclictest14740-21cut19:48:581
58299753,44cyclictest14740-21cut19:48:581
52632750,13sleep2381ktimers/220:59:022
52632750,13sleep2381ktimers/220:59:022
586997438,12cyclictest181rcu_preempt23:16:202
586997438,12cyclictest181rcu_preempt23:16:202
582997458,9cyclictest5915-21kworker/u17:2-rpciod21:19:221
58699738,27cyclictest39-21ksoftirqd/222:59:022
58699738,27cyclictest39-21ksoftirqd/222:59:022
586997354,13cyclictest31139-21tune2fs23:39:012
58699732,32cyclictest39-21ksoftirqd/222:48:532
58699732,32cyclictest39-21ksoftirqd/222:48:532
582997344,21cyclictest20772-21apt-key20:08:391
586997250,16cyclictest0-21swapper/221:43:382
58699723,29cyclictest39-21ksoftirqd/220:19:132
586997231,7cyclictest181rcu_preempt23:43:462
586997231,7cyclictest181rcu_preempt23:43:462
58699722,39cyclictest39-21ksoftirqd/200:08:352
58699722,39cyclictest39-21ksoftirqd/200:08:352
582997249,16cyclictest22585-21sh23:24:041
582997247,17cyclictest0-21swapper/122:53:351
582997247,17cyclictest0-21swapper/122:53:351
582997240,10cyclictest181rcu_preempt22:43:571
582997233,11cyclictest181rcu_preempt20:29:111
582997233,11cyclictest181rcu_preempt20:29:111
59099713,52cyclictest30231-21taskset23:37:593
59099713,52cyclictest30231-21taskset23:37:593
59099712,52cyclictest28274-21chrt00:29:313
586997154,10cyclictest0-21swapper/219:24:102
586997153,12cyclictest0-21swapper/219:19:132
586997143,12cyclictest181rcu_preempt23:09:472
586997143,12cyclictest181rcu_preempt23:09:472
586997142,11cyclictest181rcu_preempt22:38:522
586997142,11cyclictest181rcu_preempt22:38:522
58699713,25cyclictest39-21ksoftirqd/222:17:422
582997154,12cyclictest565-21cyclictest22:03:561
582997154,12cyclictest565-21cyclictest22:03:561
582997152,12cyclictest26155-21ssh21:39:041
582997147,17cyclictest2340-21unixbench_singl23:44:121
582997147,17cyclictest2340-21unixbench_singl23:44:121
582997142,11cyclictest0-21swapper/120:14:091
58299712,45cyclictest14201-21munin-run19:48:351
58299712,45cyclictest14201-21munin-run19:48:351
58699704,36cyclictest39-21ksoftirqd/222:08:392
58699704,36cyclictest39-21ksoftirqd/222:08:392
586997034,12cyclictest181rcu_preempt21:07:222
58699702,39cyclictest39-21ksoftirqd/223:34:022
58699702,39cyclictest39-21ksoftirqd/223:34:022
586997011,32cyclictest39-21ksoftirqd/221:58:542
582997040,12cyclictest0-21swapper/122:24:271
582997040,11cyclictest0-21swapper/120:19:111
58299703,45cyclictest21510-21fschecks_time20:10:031
582997033,7cyclictest0-21swapper/123:15:511
582997033,14cyclictest26120-21ssh23:29:311
58699692,36cyclictest39-21ksoftirqd/220:29:112
58699692,36cyclictest39-21ksoftirqd/220:29:112
586996911,24cyclictest30548-21cron20:38:352
58299693,27cyclictest1080-21ssh23:43:381
582996930,9cyclictest6080-21ssh21:59:211
582996924,9cyclictest181rcu_preempt00:24:091
582996924,9cyclictest181rcu_preempt00:24:091
58299692,29cyclictest24198-21taskset22:31:331
58299692,29cyclictest24198-21taskset22:31:331
582996919,36cyclictest7926-21latency_hist21:08:381
582996914,41cyclictest18320-21grep19:59:091
59099683,51cyclictest25822-21chrt00:25:383
59099683,51cyclictest25822-21chrt00:25:383
59099683,51cyclictest2348-21chrt21:53:583
59099683,50cyclictest31705-21taskset21:48:523
59099683,50cyclictest13268-21chrt22:13:353
59099682,52cyclictest23067-21taskset21:34:003
58699685,26cyclictest39-21ksoftirqd/219:28:422
586996839,8cyclictest181rcu_preempt20:13:112
586996838,11cyclictest181rcu_preempt20:38:532
58699683,32cyclictest39-21ksoftirqd/219:43:502
58699683,32cyclictest39-21ksoftirqd/219:43:502
582996852,10cyclictest8028-21ntp_kernel_pll_22:59:061
582996852,10cyclictest8028-21ntp_kernel_pll_22:59:061
582996851,12cyclictest565-21cyclictest00:14:221
582996851,12cyclictest565-21cyclictest00:14:221
582996850,11cyclictest762-21apt-get19:08:481
582996848,13cyclictest309-21systemd-logind19:33:371
582996838,7cyclictest181rcu_preempt21:03:371
582996838,7cyclictest181rcu_preempt21:03:371
58299683,47cyclictest11186-21latency_hist19:38:401
59099673,51cyclictest29878-21taskset00:33:493
582996748,11cyclictest13106-21taskset23:08:381
59099663,50cyclictest21162-21taskset20:08:503
59099663,49cyclictest9843-21taskset22:06:593
59099663,49cyclictest9843-21taskset22:06:593
59099663,49cyclictest5563-21taskset23:50:033
59099663,49cyclictest2161-21chrt20:49:063
59099663,48cyclictest3063-21taskset19:14:023
59099662,51cyclictest18265-21chrt23:17:073
59099662,51cyclictest18265-21chrt23:17:073
59099662,50cyclictest554-21taskset22:46:353
59099662,49cyclictest8320-21taskset23:55:103
59099662,49cyclictest4898-21chrt19:18:563
59099662,49cyclictest17640-21chrt22:19:233
59099662,49cyclictest17640-21chrt22:19:233
59099662,49cyclictest14620-21chrt22:14:163
58699662,33cyclictest39-21ksoftirqd/219:58:382
58299667,30cyclictest31-21ksoftirqd/100:19:041
58299667,30cyclictest31-21ksoftirqd/100:19:041
582996635,11cyclictest181rcu_preempt00:34:211
582996632,12cyclictest181rcu_preempt20:54:071
582996627,29cyclictest7945-21cat19:28:391
59099654,47cyclictest28794-21chrt20:31:413
59099654,47cyclictest28794-21chrt20:31:413
59099653,50cyclictest2225-21chrt23:44:113
59099653,50cyclictest2225-21chrt23:44:113
59099653,49cyclictest10153-21chrt19:34:073
59099653,49cyclictest0-21swapper/320:13:503
59099652,50cyclictest7373-21chrt19:24:193
59099652,50cyclictest25938-21chrt23:29:113
59099652,49cyclictest7942-21taskset22:59:043
59099652,49cyclictest7942-21taskset22:59:043
59099652,49cyclictest3058-21chrt22:50:203
59099652,49cyclictest3058-21chrt22:50:203
59099652,49cyclictest19133-21chrt21:27:313
59099652,49cyclictest1055-21chrt20:44:453
59099652,48cyclictest5409-21taskset20:59:073
59099652,48cyclictest5409-21taskset20:59:073
59099652,47cyclictest21658-21chrt21:31:253
582996549,9cyclictest0-21swapper/120:39:091
582996546,13cyclictest26587-21processes20:24:061
582996535,8cyclictest181rcu_preempt00:09:041
582996526,9cyclictest181rcu_preempt21:09:011
582996526,9cyclictest181rcu_preempt21:09:011
59099643,48cyclictest20366-21chrt23:19:393
59099643,48cyclictest0-21swapper/322:38:463
59099643,48cyclictest0-21swapper/322:38:463
59099643,47cyclictest11638-21taskset21:14:063
59099642,49cyclictest24071-21chrt22:31:113
59099642,49cyclictest24071-21chrt22:31:113
59099642,48cyclictest8353-21chrt19:29:013
59099642,48cyclictest25589-21chrt20:21:453
59099642,47cyclictest7600-21chrt21:05:513
59099642,46cyclictest17096-21chrt19:54:443
582996432,12cyclictest181rcu_preempt19:53:531
58299642,20cyclictest11498-21tune2fs19:38:581
58299642,20cyclictest11498-21tune2fs19:38:581
59099634,46cyclictest15767-21chrt19:53:193
59099634,46cyclictest15767-21chrt19:53:193
59099633,47cyclictest8176-21chrt21:08:513
59099633,47cyclictest8176-21chrt21:08:513
59099633,47cyclictest6033-21chrt22:55:313
59099633,47cyclictest20504-21chrt00:16:423
59099633,47cyclictest20504-21chrt00:16:423
59099633,46cyclictest31972-21taskset20:40:523
59099633,46cyclictest14639-21chrt21:19:083
59099632,48cyclictest13337-21chrt23:08:533
59099632,48cyclictest13337-21chrt23:08:533
59099632,48cyclictest11763-21chrt23:05:083
59099632,47cyclictest26416-21chrt22:34:313
59099632,47cyclictest25597-21chrt21:38:423
59099632,47cyclictest24172-21chrt00:23:403
59099632,47cyclictest24172-21chrt00:23:403
59099632,47cyclictest23343-21chrt23:24:543
59099632,47cyclictest15876-21chrt00:08:523
59099632,47cyclictest12863-21chrt19:43:513
59099632,47cyclictest12863-21chrt19:43:513
59099632,47cyclictest10203-21chrt23:58:593
59099632,47cyclictest10203-21chrt23:58:593
59099623,46cyclictest29690-21chrt20:34:043
59099622,48cyclictest13441-21taskset00:04:143
59099622,48cyclictest13441-21taskset00:04:143
59099622,47cyclictest3248-21taskset20:53:523
59099622,47cyclictest30472-21chrt21:46:413
59099622,47cyclictest12477-21chrt19:42:173
59099622,47cyclictest12477-21chrt19:42:173
59099622,46cyclictest21283-21chrt22:26:333
58299623,45cyclictest376-21sshd22:37:141
59099613,45cyclictest20318-21chrt20:04:243
59099612,48cyclictest32246-21chrt23:40:143
59099612,47cyclictest18789-21chrt20:00:233
59099612,46cyclictest6147-21taskset21:59:333
59099612,45cyclictest26012-21chrt20:23:473
49326012,27sleep30-21swapper/319:08:403
59099593,44cyclictest1860-21chrt19:10:373
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional