You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-02-23 - 17:15
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7 (updated Fri Feb 23, 2024 00:44:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6506212118,57sleep36538-21apt19:09:203
66809910948,11cyclictest0-21swapper/119:19:211
669799913,80cyclictest6018-21/usr/sbin/munin23:44:523
6689998931,20cyclictest939-21apt-get22:39:322
6689998931,20cyclictest939-21apt-get22:39:322
669799878,59cyclictest67-21kswapd000:19:443
669799878,59cyclictest67-21kswapd000:19:443
668099873,75cyclictest6248-21/usr/sbin/munin23:44:521
58142879,45sleep15867-21mailstats19:05:101
6689998665,14cyclictest21437-21idleruntime-cro22:19:182
23992850,4sleep32401-21grep20:34:223
23992850,4sleep32401-21grep20:34:223
6689998126,21cyclictest17973-21ssh23:09:222
6670998157,14cyclictest12117-21munin-plugin-st21:04:190
6697997738,12cyclictest141rcu_preempt20:24:223
669799773,56cyclictest41-21ksoftirqd/322:39:343
669799773,56cyclictest41-21ksoftirqd/322:39:343
6670997744,12cyclictest141rcu_preempt22:57:350
6680997653,14cyclictest25361-21latency_hist00:19:211
6680997645,7cyclictest141rcu_preempt21:09:541
6689997523,23cyclictest1646-21df00:34:472
6680997559,10cyclictest2577-21sendmail23:39:191
6697997446,7cyclictest141rcu_preempt22:54:483
6680997324,9cyclictest141rcu_preempt21:09:191
6697997238,18cyclictest41-21ksoftirqd/322:37:303
6697997238,18cyclictest41-21ksoftirqd/322:37:303
6697997227,22cyclictest0-21swapper/320:35:153
6689997223,29cyclictest13733-21diskmemload23:40:552
6680997251,8cyclictest141rcu_preempt00:39:211
6680997249,14cyclictest5624-21idleruntime-cro20:44:201
6680997248,17cyclictest23204-21runrttasks19:59:191
6680997213,41cyclictest25765-21apt-get00:19:441
6680997213,41cyclictest25765-21apt-get00:19:441
6697997131,12cyclictest141rcu_preempt21:14:503
6697997119,27cyclictest41-21ksoftirqd/321:30:463
6689997123,24cyclictest15700-21/usr/sbin/munin23:04:532
6689997120,26cyclictest399-21runrttasks22:25:452
6680997130,8cyclictest141rcu_preempt22:55:011
6670997145,20cyclictest6986-21fschecks_count22:49:490
6670997145,20cyclictest6986-21fschecks_count22:49:490
6670997127,14cyclictest0-21swapper/000:10:130
6689997021,19cyclictest26391-21ssh00:20:102
6689997021,19cyclictest26391-21ssh00:20:102
6689997021,18cyclictest0-21swapper/221:55:022
6689997020,21cyclictest17290-21latency_hist00:04:212
6680997040,9cyclictest141rcu_preempt23:33:021
6680997025,25cyclictest25-21ksoftirqd/123:40:051
6670997045,18cyclictest10249-21sendmail19:19:200
6697996928,13cyclictest41-21ksoftirqd/320:59:433
6680996941,17cyclictest4730-21timerandwakeup22:45:021
667099693,41cyclictest2711-21cpuspeed20:34:490
63862696,46sleep00-21swapper/019:07:460
669799683,37cyclictest8186-21modprobe19:14:193
6697996831,7cyclictest141rcu_preempt23:38:493
6689996824,17cyclictest0-21swapper/221:51:142
6689996823,17cyclictest26047-21taskset23:23:422
6689996823,17cyclictest26047-21taskset23:23:422
6689996822,18cyclictest399-21runrttasks20:56:342
6689996820,22cyclictest23495-21open_files00:15:002
6689996816,22cyclictest30228-21ssh22:34:302
6689996816,22cyclictest30228-21ssh22:34:302
6689996815,20cyclictest10266-21mailstats22:54:582
6670996848,13cyclictest25399-21apt-key00:19:210
6697996730,7cyclictest141rcu_preempt00:30:113
6697996712,26cyclictest0-21swapper/322:29:473
6697996712,26cyclictest0-21swapper/322:29:473
6689996717,21cyclictest26744-21apt-get20:09:372
6689996717,21cyclictest26744-21apt-get20:09:372
6689996716,19cyclictest18087-21users21:15:062
6680996726,14cyclictest0-21swapper/120:05:001
6680996719,25cyclictest25-21ksoftirqd/123:58:021
6680996719,25cyclictest25-21ksoftirqd/123:58:021
6689996638,20cyclictest10539-21apt-get19:19:372
6689996618,19cyclictest30918-21ssh00:29:192
6689996618,19cyclictest30918-21ssh00:29:192
6670996650,10cyclictest29715-21idleruntime-cro20:19:190
6670996650,10cyclictest28524-21unixbench_singl22:30:030
6670996650,10cyclictest28524-21unixbench_singl22:30:030
6670996627,25cyclictest25779-21ssh00:19:410
6670996627,25cyclictest25779-21ssh00:19:410
669799659,12cyclictest0-21swapper/323:52:593
669799659,12cyclictest0-21swapper/323:52:593
6697996549,11cyclictest276-21systemd-journal20:15:003
6697996521,16cyclictest0-21swapper/321:23:203
6689996526,28cyclictest30789-21rm21:38:392
6689996518,21cyclictest32441-21ssh23:34:432
6689996513,26cyclictest5953-21sh22:48:072
6680996551,10cyclictest179-21pr/ttyS023:01:481
6680996551,10cyclictest179-21pr/ttyS023:01:481
6680996550,9cyclictest22807-21unixbench_singl22:20:021
6680996545,12cyclictest0-21swapper/121:24:211
6680996543,13cyclictest9348-21fschecks_time20:54:501
6680996536,7cyclictest141rcu_preempt21:54:201
6680996530,12cyclictest347-21in:imklog20:59:471
6680996527,10cyclictest0-21swapper/121:14:501
6680996521,14cyclictest25-21ksoftirqd/119:10:031
6680996521,10cyclictest0-21swapper/121:25:311
667099657,32cyclictest12-21ksoftirqd/023:51:050
667099657,32cyclictest12-21ksoftirqd/023:51:050
6670996546,12cyclictest16620-21kworker/u8:021:34:220
6670996542,13cyclictest0-21swapper/000:25:070
6670996542,13cyclictest0-21swapper/000:25:070
6670996540,17cyclictest24856-21munin-plugin-st20:04:210
6670996538,15cyclictest19905-21munin-plugin-st00:09:190
667099652,12cyclictest21461-21sendmail-msp22:19:190
6697996420,24cyclictest41-21ksoftirqd/300:39:223
6697996418,16cyclictest41-21ksoftirqd/321:37:513
6697996410,30cyclictest141rcu_preempt23:20:033
6697996410,30cyclictest141rcu_preempt23:20:033
6689996438,17cyclictest13828-21latency_hist21:09:222
6689996425,14cyclictest0-21swapper/220:29:432
6689996425,14cyclictest0-21swapper/220:29:432
6689996424,15cyclictest0-21swapper/221:39:592
6680996433,9cyclictest141rcu_preempt22:14:351
668099642,20cyclictest9299-21ssh22:54:221
668099642,20cyclictest9299-21ssh22:54:221
6680996418,19cyclictest25-21ksoftirqd/100:28:251
6680996418,19cyclictest25-21ksoftirqd/100:28:251
6680996411,24cyclictest141rcu_preempt20:44:491
667099647,24cyclictest0-21swapper/022:39:410
667099647,24cyclictest0-21swapper/022:39:410
6670996446,12cyclictest131ktimers/021:39:490
6670996445,13cyclictest5208-21netstat21:49:530
6670996434,20cyclictest16738-21latency_hist21:14:210
6670996420,18cyclictest52-21kcompactd021:21:340
6697996339,13cyclictest0-21swapper/319:24:193
6697996337,17cyclictest11420-21unixbench_multi22:00:013
6697996337,14cyclictest0-21swapper/300:24:483
6697996337,14cyclictest0-21swapper/300:24:483
6697996320,24cyclictest41-21ksoftirqd/300:02:003
6689996336,19cyclictest13329-21grep22:04:442
6689996316,18cyclictest19870-21sh00:09:172
6680996337,17cyclictest347-21in:imklog22:29:481
6680996337,17cyclictest347-21in:imklog22:29:481
6680996330,12cyclictest0-21swapper/123:09:451
6680996318,11cyclictest141rcu_preempt21:35:041
667099637,26cyclictest12-21ksoftirqd/020:59:210
6697996244,12cyclictest5706-21latency_hist23:44:223
6697996243,12cyclictest31783-21sshd20:24:473
6697996242,11cyclictest8857-21latency_hist20:54:193
6697996221,24cyclictest41-21ksoftirqd/323:06:073
6689996239,16cyclictest5617-21idleruntime-cro20:44:202
6689996239,16cyclictest1190-21latency_hist00:34:202
6689996237,10cyclictest20911-21apt-get23:14:422
6689996236,11cyclictest0-21swapper/223:54:212
6689996236,11cyclictest0-21swapper/223:54:212
6689996230,24cyclictest28109-21/usr/sbin/munin22:29:592
6689996230,24cyclictest28109-21/usr/sbin/munin22:29:592
6689996217,17cyclictest13733-21diskmemload21:32:512
6689996215,17cyclictest13584-21ssh23:57:162
6689996215,17cyclictest13584-21ssh23:57:162
6689996214,18cyclictest20532-21ssh21:19:572
668099625,28cyclictest0-21swapper/122:09:431
6680996228,19cyclictest141rcu_preempt19:49:551
6680996228,19cyclictest141rcu_preempt19:49:551
667099629,7cyclictest0-21swapper/023:36:370
667099629,10cyclictest12-21ksoftirqd/023:05:040
6670996239,9cyclictest13247-21ssh22:04:390
6697996139,15cyclictest6656-21cyclictest21:05:023
6697996110,12cyclictest29813-21tune2fs23:29:483
668999618,45cyclictest2741-21cstates20:34:502
6689996141,13cyclictest16728-21sendmail19:39:192
6689996138,12cyclictest0-21swapper/219:34:222
6689996138,11cyclictest0-21swapper/219:44:222
6689996138,11cyclictest0-21swapper/219:44:222
6689996136,19cyclictest32767-21sshd20:27:082
6689996115,22cyclictest9385-21ntp_kernel_pll_19:14:532
6689996112,24cyclictest0-21swapper/220:19:392
668099615,23cyclictest13733-21diskmemload22:39:381
668099615,23cyclictest13733-21diskmemload22:39:381
6680996140,11cyclictest0-21swapper/100:05:111
6680996136,16cyclictest4398-21latency_hist21:49:211
6680996136,14cyclictest0-21swapper/119:45:031
6680996135,7cyclictest141rcu_preempt00:10:041
6680996126,19cyclictest179-21pr/ttyS022:29:191
6680996121,12cyclictest0-21swapper/120:15:011
6680996114,24cyclictest25-21ksoftirqd/123:59:521
667099619,31cyclictest1-21systemd19:26:450
6670996137,15cyclictest23588-21latency_hist23:19:210
667099613,30cyclictest12-21ksoftirqd/021:59:500
6670996124,12cyclictest0-21swapper/022:45:010
6697996041,12cyclictest1-21systemd22:25:073
6697996038,12cyclictest0-21swapper/321:24:533
6697996032,15cyclictest0-21swapper/320:09:473
6697996032,15cyclictest0-21swapper/320:09:473
6697996017,14cyclictest41-21ksoftirqd/320:04:413
6689996045,8cyclictest7094-21unixbench_multi23:45:112
6689996040,9cyclictest25125-21apt-get20:04:372
6680996037,8cyclictest141rcu_preempt19:30:021
6680996020,10cyclictest0-21swapper/121:33:141
667099608,8cyclictest0-21swapper/022:19:470
667099608,16cyclictest11772-21/usr/sbin/munin23:54:510
667099608,16cyclictest11772-21/usr/sbin/munin23:54:510
667099605,20cyclictest23915-21ssh23:19:440
667099605,20cyclictest23915-21ssh23:19:440
6670996047,8cyclictest7119-21rm23:45:120
6670996022,7cyclictest0-21swapper/019:55:010
6689995938,12cyclictest0-21swapper/220:54:192
6689995936,16cyclictest14946-21/usr/sbin/munin21:10:012
6689995935,14cyclictest276-21systemd-journal19:44:492
6689995933,19cyclictest23977-21awk19:59:542
6689995928,22cyclictest21817-21date19:54:232
6689995928,22cyclictest21817-21date19:54:232
6689995924,25cyclictest12990-21sed19:25:012
6689995915,17cyclictest31002-21ssh23:31:112
6689995913,19cyclictest6035-21df20:44:452
6680995918,18cyclictest141rcu_preempt20:29:191
6670995941,12cyclictest31789-21uniq20:24:470
6670995938,12cyclictest16332-21cut22:09:480
6670995937,12cyclictest5160-21ssh23:43:310
667099592,27cyclictest12-21ksoftirqd/022:27:380
6670995911,23cyclictest8880-21sh20:54:200
6697995840,12cyclictest10244-21latency_hist19:19:223
6697995835,12cyclictest0-21swapper/319:24:583
6697995832,14cyclictest41-21ksoftirqd/321:44:203
6697995824,12cyclictest41-21ksoftirqd/319:29:533
6689995837,11cyclictest20258-21apt-get00:09:452
6689995832,19cyclictest0-21swapper/222:19:382
6689995832,12cyclictest0-21swapper/221:24:472
6689995831,16cyclictest10961-21fschecks_count20:59:472
6689995829,21cyclictest26935-21tune2fs23:24:472
6689995827,24cyclictest17088-21unixbench_multi22:10:012
6680995843,8cyclictest0-21swapper/122:04:191
6680995824,12cyclictest347-21in:imklog20:19:561
6680995818,13cyclictest141rcu_preempt19:29:191
6670995841,10cyclictest131ktimers/019:44:490
6670995835,14cyclictest0-21swapper/021:45:060
667099582,36cyclictest12-21ksoftirqd/000:02:160
667099582,29cyclictest12-21ksoftirqd/021:35:550
6670995813,21cyclictest12-21ksoftirqd/021:16:350
6670995811,20cyclictest12-21ksoftirqd/000:31:470
6670995810,11cyclictest12-21ksoftirqd/019:30:020
58692588,28sleep25915-21switchtime19:05:112
6697995739,8cyclictest0-21swapper/319:44:193
6697995739,8cyclictest0-21swapper/319:44:193
6697995734,12cyclictest0-21swapper/320:49:193
6697995732,13cyclictest0-21swapper/322:19:213
669799573,15cyclictest20774-21ssh00:10:013
6697995722,10cyclictest41-21ksoftirqd/321:45:283
6697995715,19cyclictest141rcu_preempt19:54:213
6697995715,19cyclictest141rcu_preempt19:54:213
6689995738,11cyclictest0-21swapper/222:59:492
6689995738,11cyclictest0-21swapper/222:59:492
6689995737,10cyclictest0-21swapper/222:04:212
6689995737,10cyclictest0-21swapper/219:10:052
6689995731,16cyclictest1814-21/usr/sbin/munin21:45:032
6689995727,23cyclictest29721-21readlink20:19:192
6689995722,27cyclictest7321-21/usr/sbin/munin22:49:552
6689995722,27cyclictest7321-21/usr/sbin/munin22:49:552
668099578,21cyclictest141rcu_preempt23:24:471
6680995718,14cyclictest25-21ksoftirqd/119:34:391
6670995740,6cyclictest141rcu_preempt20:45:040
6670995736,15cyclictest29757-21cut23:29:470
669799566,11cyclictest0-21swapper/323:13:093
6697995642,8cyclictest18463-21tail00:05:113
6697995636,9cyclictest0-21swapper/322:45:013
6697995624,14cyclictest41-21ksoftirqd/319:34:443
6697995621,15cyclictest41-21ksoftirqd/300:14:473
6689995635,10cyclictest0-21swapper/219:54:482
6680995634,15cyclictest0-21swapper/122:39:191
6680995634,15cyclictest0-21swapper/122:39:191
6680995624,8cyclictest25-21ksoftirqd/123:15:021
6680995623,13cyclictest25-21ksoftirqd/121:42:031
6680995619,13cyclictest141rcu_preempt20:54:211
6680995617,15cyclictest25-21ksoftirqd/100:34:191
6680995612,18cyclictest141rcu_preempt21:55:011
6670995638,11cyclictest20546-21ssh23:14:190
667099563,14cyclictest12-21ksoftirqd/019:24:210
669799554,8cyclictest0-21swapper/320:39:463
6697995539,10cyclictest23256-21systemd-run19:59:203
6697995537,11cyclictest276-21systemd-journal20:55:513
6697995534,12cyclictest0-21swapper/319:49:213
6697995532,13cyclictest0-21swapper/322:20:013
669799553,11cyclictest0-21swapper/322:52:393
669799553,11cyclictest0-21swapper/322:52:393
6697995524,7cyclictest141rcu_preempt23:14:503
6697995513,18cyclictest141rcu_preempt22:05:073
6680995532,12cyclictest0-21swapper/123:09:211
6680995526,10cyclictest116-21pr/ttymxc023:20:141
6680995526,10cyclictest116-21pr/ttymxc023:20:141
6680995510,33cyclictest26374-21kworker/u8:120:34:421
667099558,21cyclictest12-21ksoftirqd/020:39:590
6670995531,14cyclictest0-21swapper/021:29:200
6670995530,13cyclictest141rcu_preempt21:05:380
6680995431,13cyclictest0-21swapper/120:00:021
667099549,11cyclictest7822-21awk19:10:060
667099548,22cyclictest20235-21/usr/sbin/munin19:50:050
667099548,22cyclictest20235-21/usr/sbin/munin19:50:050
667099545,10cyclictest12-21ksoftirqd/020:29:470
667099545,10cyclictest12-21ksoftirqd/020:29:470
6670995425,16cyclictest0-21swapper/000:36:060
6670995410,23cyclictest12-21ksoftirqd/023:27:050
669799533,13cyclictest141rcu_preempt23:54:513
669799533,13cyclictest141rcu_preempt23:54:513
6697995319,11cyclictest141rcu_preempt21:11:283
6697995317,8cyclictest141rcu_preempt21:59:193
6680995324,11cyclictest25-21ksoftirqd/119:24:201
6680995319,9cyclictest141rcu_preempt23:49:401
6680995319,9cyclictest141rcu_preempt23:49:401
667099535,14cyclictest12-21ksoftirqd/021:55:000
6670995323,10cyclictest7228-21kworker/u8:122:34:480
6670995323,10cyclictest7228-21kworker/u8:122:34:480
6670995311,11cyclictest12-21ksoftirqd/020:19:500
6697995228,8cyclictest141rcu_preempt23:28:463
6697995226,6cyclictest141rcu_preempt21:50:373
6680995219,8cyclictest141rcu_preempt22:04:251
6680995215,20cyclictest0-21swapper/120:29:561
6680995215,20cyclictest0-21swapper/120:29:561
6670995212,11cyclictest12-21ksoftirqd/022:59:480
6670995212,11cyclictest12-21ksoftirqd/022:59:480
6697995129,15cyclictest0-21swapper/323:00:023
6697995129,15cyclictest0-21swapper/323:00:023
6697995119,7cyclictest41-21ksoftirqd/319:59:443
6680995117,9cyclictest141rcu_preempt19:44:231
6680995117,9cyclictest141rcu_preempt19:44:231
6670995131,10cyclictest0-21swapper/020:14:220
6670995131,10cyclictest0-21swapper/020:14:220
668099502,9cyclictest141rcu_preempt20:09:581
668099502,9cyclictest141rcu_preempt20:09:581
6670995022,15cyclictest141rcu_preempt19:35:360
667099501,20cyclictest17818-21timerandwakeup19:40:010
667099501,20cyclictest17818-21timerandwakeup19:40:010
6670994931,11cyclictest0-21swapper/020:09:210
6697994812,14cyclictest41-21ksoftirqd/322:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional