You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-06-17 - 21:29
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Mon Jun 17, 2024 12:44:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5831991278,96cyclictest187962sleep010:19:410
364421120,4chrt3643-21ssh10:53:421
5831991006,76cyclictest0-21swapper/011:44:290
5831999818,50cyclictest23515-21latency_hist10:29:110
583199964,82cyclictest19514-21apt-get11:24:180
583199942,82cyclictest29667-21cut08:19:350
583199934,61cyclictest0-21swapper/012:37:570
583199932,82cyclictest0-21swapper/009:19:230
583199932,63cyclictest14412-21rm10:10:020
583199902,79cyclictest27195-21cstates09:34:270
54592905,14sleep20-21swapper/207:06:192
54592905,14sleep20-21swapper/207:06:192
5840998970,13cyclictest9060-21munin-plugin-st11:04:091
583199892,57cyclictest6139-21head08:44:520
585499884,51cyclictest30092-21apt-get11:44:223
5831998827,54cyclictest24969-21ssh09:29:480
583199873,73cyclictest22202-21apt-get11:29:220
583199872,77cyclictest17861-21find12:19:450
583199872,76cyclictest92892sleep011:04:120
5854998561,18cyclictest26679-21fschecks_time10:34:343
5840998561,16cyclictest16669-21munin-plugin-st11:19:091
583199852,52cyclictest28116-21mailstats11:39:450
585499842,21cyclictest22469-21fschecks_time11:29:323
583199844,68cyclictest5838-21sort07:09:130
583199844,68cyclictest5838-21sort07:09:130
583199843,74cyclictest27096-21idleruntime-cro11:39:100
583199843,73cyclictest15666-21/usr/sbin/munin09:14:120
583199843,51cyclictest31276-21taskset09:40:440
583199842,72cyclictest21598-21apt-get09:24:220
5831998422,54cyclictest0-21swapper/007:39:320
583199833,72cyclictest714-21fschecks_time11:49:330
583199833,70cyclictest15967-21apt-get09:14:260
583199832,74cyclictest20210-21/usr/sbin/munin12:24:420
583199832,73cyclictest1272-21sh10:49:100
583199832,55cyclictest5762-21rm10:56:510
583199832,55cyclictest5762-21rm10:56:510
583199823,71cyclictest27830-21apt-get08:14:420
583199823,67cyclictest6804-21apt-get08:49:240
583199823,67cyclictest6804-21apt-get08:49:240
5831998230,45cyclictest13034-21diskmemload12:14:220
583199822,57cyclictest13201-21latency_hist09:09:120
583199822,54cyclictest13034-21diskmemload10:04:030
585499813,59cyclictest25085-21apt-get12:34:373
585499813,19cyclictest4184-21unixbench_multi11:55:143
5840998155,17cyclictest8459-21latency_hist12:04:121
583199813,71cyclictest28698-21latency_hist10:39:120
583199812,70cyclictest12965-21apt-get07:29:220
5854998054,15cyclictest27650-21apt-config08:14:133
585499804,12cyclictest9063-21modprobe11:04:093
585499804,12cyclictest0-21swapper/310:44:363
5854998039,11cyclictest120-21pr/ttymxc012:18:513
5846998053,9cyclictest33-21ksoftirqd/207:49:122
5846998053,9cyclictest33-21ksoftirqd/207:49:122
5840998059,15cyclictest3291-21sendmail08:39:101
583199803,69cyclictest17293-21sort11:19:320
583199803,67cyclictest23816-21apt-get10:29:220
583199802,71cyclictest7394-21munin-run07:14:100
583199802,71cyclictest11779-21kernelversion12:09:380
583199802,69cyclictest12474-21/usr/sbin/munin11:09:350
583199802,52cyclictest4373-21ssh11:55:360
583199802,51cyclictest27561-21date08:14:120
5840997955,15cyclictest29877-21ssh11:44:121
583199793,67cyclictest362-21in:imuxsock12:04:120
583199793,66cyclictest6017-21/usr/sbin/munin09:54:410
583199793,63cyclictest0-21swapper/008:34:240
133162790,4sleep25819-21cyclictest07:29:392
585499783,14cyclictest15717-21/usr/sbin/munin10:14:443
583199783,67cyclictest5793-21/usr/sbin/munin11:59:320
583199783,67cyclictest22370-21/usr/sbin/munin12:29:290
583199782,68cyclictest416-21snmpd11:04:320
583199782,67cyclictest19327-21sed07:49:130
583199782,67cyclictest19327-21sed07:49:130
5831997822,48cyclictest0-21swapper/010:49:460
585499776,11cyclictest0-21swapper/312:24:463
5854997725,43cyclictest0-21swapper/311:54:123
5854997722,46cyclictest0-21swapper/311:44:133
585499772,20cyclictest9018-21grep07:14:523
583199773,67cyclictest8280-21fschecks_count07:14:310
583199773,67cyclictest16293-21/usr/sbin/munin10:14:440
583199773,65cyclictest22393-21modprobe07:59:100
583199773,65cyclictest22393-21modprobe07:59:100
583199772,67cyclictest11334-21/usr/sbin/munin07:24:340
583199772,66cyclictest9939-21grep07:19:320
583199772,66cyclictest555-21/usr/sbin/munin09:44:350
583199772,66cyclictest15284-21switchtime07:34:540
5831997723,45cyclictest0-21swapper/008:29:570
5854997622,46cyclictest0-21swapper/307:44:513
5854997622,46cyclictest0-21swapper/307:44:513
5840997654,13cyclictest18394-21latency_hist10:19:121
583199763,65cyclictest31273-21cut08:24:320
583199763,65cyclictest24946-21wc08:04:500
583199762,67cyclictest2927-21runrttasks09:49:320
585499755,16cyclictest25360-21grep09:29:543
5854997529,40cyclictest0-21swapper/310:14:093
5854997527,35cyclictest0-21swapper/308:54:443
5854997512,10cyclictest0-21swapper/307:19:483
5854997511,57cyclictest0-21swapper/310:24:353
583199753,66cyclictest0-21swapper/011:14:320
583199752,66cyclictest28884-21apt-get10:39:250
5831997523,44cyclictest0-21swapper/010:09:100
5831997523,44cyclictest0-21swapper/010:09:100
585499744,57cyclictest5014-21kworker/u8:210:19:323
585499744,15cyclictest4492-21ssh10:54:373
585499744,15cyclictest4492-21ssh10:54:373
585499743,19cyclictest8808-21fschecks_time12:04:303
585499743,16cyclictest25960-21ssh10:33:523
585499743,15cyclictest16975-21/usr/sbin/munin12:19:333
585499743,13cyclictest25162-21timerwakeupswit08:04:533
585499742,20cyclictest24085-21chrt08:04:113
5854997419,48cyclictest0-21swapper/311:34:323
5840997456,12cyclictest31222-21munin-plugin-st10:44:091
5840997423,28cyclictest141rcu_preempt09:04:281
583199743,61cyclictest22675-21/usr/sbin/munin07:59:460
54682746,29sleep10-21swapper/107:06:261
54682746,29sleep10-21swapper/107:06:261
585499733,13cyclictest30118-21cut09:39:333
585499732,17cyclictest14882-21fschecks_count11:14:323
5854997320,12cyclictest22770-21sh09:25:053
5840997347,17cyclictest14349-21munin-run11:14:091
5840997335,27cyclictest6138-21grep08:44:521
583199732,63cyclictest19479-21sendmail-mta07:49:270
585499728,18cyclictest29690-21fschecks_count08:19:363
585499724,14cyclictest4917-21cron08:44:093
585499723,20cyclictest19720-21timerandwakeup09:19:493
585499723,19cyclictest18579-21latency_hist09:19:113
585499723,18cyclictest2510-21tail10:49:473
585499723,16cyclictest11725-21idleruntime-cro11:09:093
585499723,11cyclictest17604-21latency_hist07:44:123
5854997222,43cyclictest24292-21t-helper12:33:333
584099727,31cyclictest18706-21apt-get10:19:411
5840997249,14cyclictest28163-21unixbench_singl09:34:521
5840997239,12cyclictest141rcu_preempt07:34:471
584099723,21cyclictest25-21ksoftirqd/108:14:301
5840997222,13cyclictest141rcu_preempt08:04:551
5840997221,17cyclictest25-21ksoftirqd/108:52:531
5840997221,17cyclictest25-21ksoftirqd/108:52:531
583199723,62cyclictest10038-21/usr/sbin/munin08:59:560
583199722,64cyclictest37492sleep008:39:370
583199722,62cyclictest9396-21unixbench_singl08:55:010
585499714,15cyclictest6074-21fschecks_count11:59:293
585499713,19cyclictest513-21fschecks_time09:44:343
585499713,17cyclictest276-21systemd-journal09:04:093
585499713,17cyclictest276-21systemd-journal07:34:103
585499712,17cyclictest19674-21fschecks_time07:49:353
5854997120,7cyclictest0-21swapper/310:04:453
5854997120,7cyclictest0-21swapper/310:04:453
585499708,12cyclictest0-21swapper/308:15:083
585499705,12cyclictest0-21swapper/311:19:323
585499703,19cyclictest12389-21grep12:09:523
585499703,17cyclictest22577-21latency_hist07:59:123
585499703,17cyclictest22577-21latency_hist07:59:123
5854997024,40cyclictest20775-21kworker/u8:008:54:103
5854997024,40cyclictest20775-21kworker/u8:008:54:103
585499702,17cyclictest8317-21fschecks_count09:59:313
5854997018,44cyclictest0-21swapper/311:11:353
5846997051,13cyclictest363-21in:imklog12:14:252
5846997047,12cyclictest0-21swapper/207:14:102
5840997033,9cyclictest141rcu_preempt12:04:141
5840997021,26cyclictest141rcu_preempt09:31:461
585499697,9cyclictest0-21swapper/310:40:243
585499693,21cyclictest6369-21unixbench_singl09:54:453
585499693,19cyclictest5245-21latency_hist09:54:133
585499693,18cyclictest338-21sort08:29:313
585499693,16cyclictest5967-21timerandwakeup08:44:503
585499693,16cyclictest15549-21sed07:34:573
585499692,17cyclictest400-21runrttasks11:24:333
585499692,17cyclictest27938-21threads09:34:503
585499692,16cyclictest3348-21latency_hist08:39:123
5854996910,9cyclictest0-21swapper/309:04:333
5846996943,9cyclictest33-21ksoftirqd/211:44:102
5846996917,11cyclictest141rcu_preempt07:39:322
5840996933,6cyclictest0-21swapper/112:32:591
5840996915,14cyclictest0-21swapper/111:54:341
585499687,11cyclictest0-21swapper/307:09:143
585499687,11cyclictest0-21swapper/307:09:143
585499684,13cyclictest7431-21latency_hist07:14:113
585499683,16cyclictest13551-21fschecks_count09:09:323
585499682,14cyclictest31274-21fschecks_count08:24:323
5854996812,10cyclictest0-21swapper/307:24:543
5840996851,11cyclictest1306-21sshd09:44:521
5840996829,10cyclictest141rcu_preempt11:44:571
5840996812,25cyclictest921-21sendmail-mta09:14:331
5846996741,16cyclictest19042-21fschecks_time10:19:522
5840996735,11cyclictest141rcu_preempt09:19:461
5840996722,23cyclictest5014-21kworker/u8:210:27:141
5846996645,13cyclictest8246-21modprobe12:04:092
5846996642,14cyclictest0-21swapper/208:24:102
5846996634,15cyclictest33-21ksoftirqd/207:49:292
5840996644,15cyclictest5819-21cyclictest09:24:331
584099663,48cyclictest25085-21apt-get12:34:231
5840996623,23cyclictest141rcu_preempt07:09:581
5840996617,18cyclictest25-21ksoftirqd/110:34:421
5846996542,9cyclictest33-21ksoftirqd/210:54:332
5846996542,9cyclictest33-21ksoftirqd/210:54:332
5846996541,14cyclictest27332-21idleruntime-cro08:14:102
5840996524,12cyclictest141rcu_preempt11:32:501
5840996510,28cyclictest25-21ksoftirqd/111:19:341
5846996441,10cyclictest33-21ksoftirqd/211:39:122
5846996438,15cyclictest0-21swapper/207:34:572
5846996429,13cyclictest363-21in:imklog08:54:112
5846996429,13cyclictest363-21in:imklog08:54:112
5846996417,29cyclictest33-21ksoftirqd/208:44:502
5840996440,17cyclictest9461-21idleruntime-cro07:19:101
584099643,33cyclictest27075-21latency_hist11:39:101
584099643,32cyclictest25-21ksoftirqd/109:14:101
5840996420,10cyclictest0-21swapper/107:54:521
5840996420,10cyclictest0-21swapper/107:54:521
5840996414,19cyclictest25-21ksoftirqd/109:50:041
5846996340,13cyclictest26055-21latency_hist10:34:092
5846996323,9cyclictest0-21swapper/208:44:092
584099639,33cyclictest25-21ksoftirqd/109:41:521
5840996343,7cyclictest141rcu_preempt10:04:371
5840996343,7cyclictest141rcu_preempt10:04:371
5840996327,23cyclictest141rcu_preempt12:14:271
5840996311,21cyclictest25-21ksoftirqd/110:55:481
5840996311,21cyclictest25-21ksoftirqd/110:55:481
584699627,32cyclictest0-21swapper/210:39:242
5846996242,14cyclictest33-21ksoftirqd/211:56:442
5846996241,12cyclictest0-21swapper/209:54:102
5846996232,13cyclictest0-21swapper/212:34:442
5846996229,17cyclictest141rcu_preempt09:49:122
5846996229,16cyclictest33-21ksoftirqd/208:29:532
5846996216,13cyclictest141rcu_preempt09:39:092
5846996211,13cyclictest141rcu_preempt11:19:312
5840996223,9cyclictest141rcu_preempt08:24:131
5840996218,25cyclictest141rcu_preempt10:10:501
5840996215,20cyclictest25-21ksoftirqd/107:49:291
5840996215,14cyclictest25-21ksoftirqd/109:56:441
5840996214,19cyclictest25-21ksoftirqd/108:59:201
5846996142,11cyclictest15943-21/usr/sbin/munin09:14:532
5846996141,12cyclictest5819-21cyclictest07:54:532
5846996141,12cyclictest5819-21cyclictest07:54:532
5846996110,27cyclictest33-21ksoftirqd/209:09:122
5840996140,7cyclictest141rcu_preempt07:31:081
5840996138,12cyclictest0-21swapper/110:49:101
5840996138,12cyclictest0-21swapper/107:19:521
5840996126,16cyclictest141rcu_preempt08:30:231
5840996118,18cyclictest25-21ksoftirqd/108:03:501
5846996042,11cyclictest363-21in:imklog10:24:312
5846996039,15cyclictest33-21ksoftirqd/208:59:122
5846996014,14cyclictest141rcu_preempt09:24:122
5846996010,17cyclictest141rcu_preempt11:24:462
5840996024,17cyclictest141rcu_preempt12:21:471
5840996010,27cyclictest25-21ksoftirqd/111:27:181
5846995943,9cyclictest120-21pr/ttymxc011:04:182
5846995941,13cyclictest27058-21kworker/u8:212:29:302
5846995941,11cyclictest15701-21systemd09:14:122
5846995939,13cyclictest25876-21latency_hist08:09:122
584099594,25cyclictest25-21ksoftirqd/112:24:351
5840995937,14cyclictest241ktimers/112:10:071
5840995936,12cyclictest0-21swapper/110:29:491
5840995916,25cyclictest141rcu_preempt11:50:551
5840995914,21cyclictest25-21ksoftirqd/107:24:371
9962580,7chrt0-21swapper/210:48:052
584699589,28cyclictest30092-21apt-get11:44:402
5846995843,8cyclictest0-21swapper/211:54:092
5846995842,9cyclictest0-21swapper/210:39:102
5846995840,10cyclictest0-21swapper/210:04:532
5846995840,10cyclictest0-21swapper/210:04:532
5846995839,11cyclictest18148-21sendmail10:19:102
5846995837,15cyclictest5819-21cyclictest09:44:102
5846995833,15cyclictest355-21systemd-logind11:04:102
5840995818,21cyclictest141rcu_preempt07:42:031
5846995737,13cyclictest337-21dbus-daemon09:04:102
5846995736,12cyclictest363-21in:imklog10:02:502
5846995735,13cyclictest0-21swapper/208:29:132
5846995723,10cyclictest141rcu_preempt11:34:122
5846995720,9cyclictest0-21swapper/209:59:122
5840995724,17cyclictest141rcu_preempt09:59:181
5840995724,13cyclictest141rcu_preempt08:24:421
5840995723,11cyclictest141rcu_preempt08:10:371
5840995722,13cyclictest141rcu_preempt08:40:441
584699567,9cyclictest0-21swapper/212:23:252
5840995613,27cyclictest141rcu_preempt11:06:281
5846995540,9cyclictest120-21pr/ttymxc011:09:332
5846995522,18cyclictest0-21swapper/212:09:402
5840995531,11cyclictest141rcu_preempt07:48:441
5840995531,11cyclictest141rcu_preempt07:48:441
584699549,20cyclictest0-21swapper/207:19:122
5846995429,13cyclictest0-21swapper/209:29:542
5846995416,10cyclictest141rcu_preempt08:19:122
5846995412,11cyclictest141rcu_preempt09:24:332
584099548,21cyclictest25-21ksoftirqd/108:55:021
584699536,16cyclictest141rcu_preempt11:14:372
5846995338,9cyclictest9965-21sort07:19:332
5846995321,24cyclictest0-21swapper/207:24:342
5846995231,11cyclictest0-21swapper/212:24:502
5846995227,16cyclictest363-21in:imklog10:14:112
5846995215,22cyclictest33-21ksoftirqd/208:04:092
5846995213,17cyclictest141rcu_preempt10:54:092
5846995128,12cyclictest0-21swapper/208:34:542
584699503,23cyclictest0-21swapper/212:04:562
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional