You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-25 - 06:16
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Thu Apr 25, 2024 00:44:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3026821000,6sleep30-21swapper/323:33:053
316899922,60cyclictest28508-21ssh22:35:282
316899922,60cyclictest28508-21ssh22:35:282
316899912,61cyclictest399-21runrttasks21:17:502
3168998846,35cyclictest25175-21idleruntime-cro22:30:122
235528710,34sleep00-21swapper/019:00:580
316899862,55cyclictest399-21runrttasks19:44:442
316899852,56cyclictest13367-21munin-run00:00:132
316899852,54cyclictest4826-21df21:55:312
316899842,52cyclictest29624-21awk20:30:492
3168998315,60cyclictest399-21runrttasks22:46:042
316899822,55cyclictest28676-21ssh21:40:422
316899819,64cyclictest445-21kworker/u8:321:10:592
316899819,64cyclictest445-21kworker/u8:321:10:592
316899813,67cyclictest7307-21diskmemload21:25:252
316899813,54cyclictest28405-21apt-get23:30:232
316899812,51cyclictest11070-21apt-get23:01:122
215528116,29sleep30-21swapper/319:00:493
3178998063,12cyclictest67-21kswapd023:00:383
316899802,70cyclictest21494-21ssh22:23:032
316899802,66cyclictest0-21swapper/200:20:262
316899802,52cyclictest7307-21diskmemload21:48:242
3168998011,46cyclictest7786-21apt-get23:50:482
3155998056,17cyclictest19522-21latency_hist22:20:150
3155998044,23cyclictest12-21ksoftirqd/023:17:160
3178997836,19cyclictest276-21systemd-journal23:18:213
316399782,51cyclictest1636-21apt-get22:45:261
3155997859,8cyclictest12-21ksoftirqd/022:41:090
3155997859,8cyclictest12-21ksoftirqd/022:41:090
316899762,46cyclictest0-21swapper/221:54:492
3163997657,12cyclictest10307-40logrotate23:55:141
3163997624,24cyclictest25-21ksoftirqd/123:15:301
3155997650,14cyclictest12-21ksoftirqd/020:45:380
3155997650,14cyclictest12-21ksoftirqd/020:45:380
316899752,43cyclictest31720-21sshd00:33:302
3163997550,18cyclictest19554-21latency_hist23:15:151
3163997532,16cyclictest141rcu_preempt22:00:301
3155997545,15cyclictest12-21ksoftirqd/022:02:180
316899742,64cyclictest6008-21idleruntime-cro19:15:122
316899742,64cyclictest18001-21/usr/sbin/munin19:55:522
316899742,64cyclictest0-21swapper/223:10:272
316899742,44cyclictest0-21swapper/223:39:232
3163997447,20cyclictest7448-21latency_hist22:00:151
3163997428,25cyclictest25-21ksoftirqd/122:12:141
3155997445,15cyclictest12-21ksoftirqd/019:15:250
3178997353,11cyclictest401ktimers/322:46:053
316899733,61cyclictest2857-21fschecks_count20:50:342
316899732,64cyclictest7833-21runrttasks22:00:332
316899732,45cyclictest7950-21apt-get22:55:242
316899731,48cyclictest20770-21ssh00:12:412
316899731,48cyclictest20770-21ssh00:12:412
3163997341,14cyclictest141rcu_preempt00:15:461
3163997336,18cyclictest141rcu_preempt21:37:251
3163997332,13cyclictest25-21ksoftirqd/121:52:161
3155997328,26cyclictest12-21ksoftirqd/000:26:330
316899722,61cyclictest14565-21awk19:45:442
316899722,44cyclictest0-21swapper/222:07:062
3155997231,13cyclictest0-21swapper/000:24:230
3155997229,23cyclictest141rcu_preempt23:34:350
3178997148,8cyclictest141rcu_preempt23:10:393
3178997139,8cyclictest141rcu_preempt21:46:313
316899713,60cyclictest3697-21kernelversion19:05:372
213027110,36sleep22153-21netstat19:00:482
3178997047,14cyclictest32298-21sendmail00:35:123
316899702,63cyclictest6546-21seq19:15:312
316899702,62cyclictest7307-21diskmemload23:06:552
316899702,61cyclictest26006-21memory23:25:382
316899702,61cyclictest18355-21ssh21:23:022
316899702,60cyclictest21948-21/usr/sbin/munin00:15:332
316899702,60cyclictest18407-21ssh22:17:392
316899702,59cyclictest30465-21fschecks_count20:35:322
3163997034,16cyclictest141rcu_preempt23:35:151
3155997038,21cyclictest12-21ksoftirqd/021:55:450
3155997026,27cyclictest12-21ksoftirqd/020:00:460
3178996942,12cyclictest141rcu_preempt19:15:313
316899693,58cyclictest139022sleep222:10:342
316899693,55cyclictest0-21swapper/220:55:252
316899693,55cyclictest0-21swapper/220:55:252
316899692,59cyclictest26086-21/usr/sbin/munin22:30:412
316899692,58cyclictest4968-21fschecks_time23:45:352
316899692,55cyclictest19136-21fschecks_count20:00:362
316899692,44cyclictest0-21swapper/222:53:182
3163996936,11cyclictest276-21systemd-journal00:12:121
3163996936,11cyclictest276-21systemd-journal00:12:121
3178996853,9cyclictest7856-21fschecks_count22:00:343
3178996845,7cyclictest141rcu_preempt00:05:253
3168996846,14cyclictest0-21swapper/222:45:132
3168996846,14cyclictest0-21swapper/222:45:132
316899682,59cyclictest7615-21ssh21:05:242
316899682,58cyclictest20287-21sh23:15:412
316899682,55cyclictest24713-21tail20:15:502
3163996832,9cyclictest141rcu_preempt20:30:381
3163996829,16cyclictest276-21systemd-journal19:15:311
3163996827,8cyclictest141rcu_preempt23:45:241
3155996843,16cyclictest10826-21latency_hist23:00:140
3155996834,26cyclictest12-21ksoftirqd/020:40:440
3155996822,20cyclictest141rcu_preempt22:35:170
3155996822,20cyclictest141rcu_preempt22:35:170
3178996751,10cyclictest32442-21unixbench_multi22:40:483
3178996751,10cyclictest32442-21unixbench_multi22:40:483
3178996744,8cyclictest141rcu_preempt21:35:313
3178996744,13cyclictest8266-21fschecks_time22:55:353
3178996743,8cyclictest141rcu_preempt23:35:333
3178996737,14cyclictest141rcu_preempt23:40:453
3178996737,14cyclictest141rcu_preempt23:40:453
3178996723,28cyclictest41-21ksoftirqd/322:54:253
316899673,55cyclictest13954-21cut00:00:342
316899672,56cyclictest17131-21cut00:05:482
316899672,55cyclictest27179-21df20:25:322
316899672,42cyclictest0-21swapper/219:26:492
316899672,40cyclictest26560-21apt-get00:25:262
3163996743,8cyclictest141rcu_preempt00:28:421
3163996729,12cyclictest141rcu_preempt22:44:131
3163996729,12cyclictest141rcu_preempt22:44:131
3163996727,17cyclictest25-21ksoftirqd/119:50:151
3155996741,18cyclictest4317-21sh21:55:120
3178996632,20cyclictest116-21pr/ttymxc021:56:193
3168996639,19cyclictest23530-21runrttasks20:15:132
3168996639,19cyclictest23530-21runrttasks20:15:132
316899662,56cyclictest22874-21tune2fs23:20:322
316899662,55cyclictest21471-21awk20:05:522
316899662,54cyclictest2087-21/usr/sbin/munin23:40:352
316899662,54cyclictest2087-21/usr/sbin/munin23:40:352
3163996647,13cyclictest11179-21kworker/u8:123:00:511
3155996645,15cyclictest10806-21modprobe23:55:180
3155996630,26cyclictest0-21swapper/021:07:490
3155996625,18cyclictest0-21swapper/023:00:410
3155996618,23cyclictest141rcu_preempt21:35:570
3178996547,12cyclictest19098-21munin-plugin-st21:25:133
3178996536,19cyclictest347-21in:imklog23:55:373
3178996532,18cyclictest276-21systemd-journal22:27:493
3178996512,28cyclictest0-21swapper/320:35:143
316899653,55cyclictest22370-21cpu21:30:312
316899652,55cyclictest25980-21open_inodes20:20:452
316899652,55cyclictest25433-21ssh21:35:342
316899652,53cyclictest32311-21/usr/sbin/munin20:40:402
316899652,53cyclictest17190-21sed19:55:152
3163996541,9cyclictest25-21ksoftirqd/119:40:501
3163996538,10cyclictest141rcu_preempt23:27:041
3163996533,14cyclictest141rcu_preempt20:10:541
3163996533,14cyclictest141rcu_preempt20:10:541
3155996548,9cyclictest24904-21kworker/u8:321:35:130
3155996543,16cyclictest12-21ksoftirqd/023:40:290
3155996543,16cyclictest12-21ksoftirqd/023:40:290
3155996539,16cyclictest10553-21sendmail19:35:130
3155996539,16cyclictest10553-21sendmail19:35:130
3155996538,11cyclictest141rcu_preempt21:18:330
3155996522,22cyclictest141rcu_preempt23:48:260
17742658,30sleep11818-21/usr/sbin/munin19:00:341
3178996449,8cyclictest0-21swapper/300:26:223
3178996426,14cyclictest141rcu_preempt20:00:143
3178996411,25cyclictest41-21ksoftirqd/323:20:393
316899642,55cyclictest11217-21cat19:35:332
316899642,53cyclictest8294-21idleruntime-cro19:25:122
3163996431,14cyclictest12397-21ssh21:12:191
3163996431,14cyclictest12397-21ssh21:12:191
3163996426,13cyclictest141rcu_preempt22:55:291
3163996426,11cyclictest347-21in:imklog22:37:581
3163996426,11cyclictest347-21in:imklog22:37:581
3163996422,24cyclictest25-21ksoftirqd/121:30:021
3155996443,15cyclictest10366-40gzip23:55:140
3155996425,11cyclictest141rcu_preempt20:20:510
3155996416,29cyclictest12-21ksoftirqd/021:13:040
3155996416,29cyclictest12-21ksoftirqd/021:13:040
3178996348,9cyclictest8340-21fschecks_time23:51:383
316899632,54cyclictest1732-21mailstats20:45:452
316899632,54cyclictest1732-21mailstats20:45:452
3163996346,11cyclictest2423-21munin-plugin-st20:50:131
3163996346,11cyclictest2423-21munin-plugin-st20:50:131
3163996329,13cyclictest141rcu_preempt23:35:501
3163996314,27cyclictest25-21ksoftirqd/123:56:271
315599636,31cyclictest141rcu_preempt22:30:250
3155996347,8cyclictest0-21swapper/022:55:120
3155996347,8cyclictest0-21swapper/019:30:130
3155996335,7cyclictest141rcu_preempt23:11:480
315599633,38cyclictest2183-21apt-get22:45:260
3155996332,24cyclictest0-21swapper/000:01:510
3155996331,14cyclictest13875-21ssh22:10:340
3155996330,12cyclictest141rcu_preempt00:10:480
3155996330,12cyclictest141rcu_preempt00:10:480
3155996313,13cyclictest141rcu_preempt19:10:150
3178996242,10cyclictest0-21swapper/320:25:523
3178996241,7cyclictest141rcu_preempt00:25:163
3178996234,11cyclictest141rcu_preempt22:30:353
3178996225,19cyclictest141rcu_preempt19:10:163
3178996223,22cyclictest41-21ksoftirqd/300:15:263
316899622,51cyclictest7443-21latency_hist21:05:152
3163996235,12cyclictest25-21ksoftirqd/120:50:291
3163996233,15cyclictest141rcu_preempt20:20:351
3163996231,12cyclictest141rcu_preempt23:41:121
3163996231,12cyclictest141rcu_preempt23:41:121
3155996247,9cyclictest32271-21cron00:35:110
3155996244,12cyclictest20433-21unixbench_multi21:25:510
3155996223,7cyclictest141rcu_preempt21:23:150
3155996215,8cyclictest141rcu_preempt23:40:150
3178996145,11cyclictest13383-21kworker/u8:221:15:123
3178996145,11cyclictest13383-21kworker/u8:221:15:123
3178996127,12cyclictest141rcu_preempt23:45:293
3178996125,22cyclictest141rcu_preempt21:17:293
3163996146,9cyclictest32109-21unixbench_multi21:45:481
3163996144,11cyclictest30896-21unixbench_singl00:30:531
3163996130,12cyclictest25-21ksoftirqd/122:10:141
3163996126,18cyclictest141rcu_preempt21:05:371
3163996126,15cyclictest141rcu_preempt22:17:551
3163996119,17cyclictest25-21ksoftirqd/122:21:241
3155996124,16cyclictest12-21ksoftirqd/019:45:380
317899607,22cyclictest0-21swapper/320:55:413
317899607,22cyclictest0-21swapper/320:55:413
3178996044,10cyclictest347-21in:imklog19:39:513
3178996035,16cyclictest0-21swapper/322:15:123
3178996028,9cyclictest141rcu_preempt21:40:483
317899602,31cyclictest399-21runrttasks00:01:443
316899602,49cyclictest10675-21latency_hist19:35:152
316899602,49cyclictest10675-21latency_hist19:35:152
3163996038,11cyclictest0-21swapper/100:05:521
3163996037,14cyclictest24513-1kworker/1:1H19:40:141
3163996023,8cyclictest116-21pr/ttymxc000:24:341
3155996044,11cyclictest13789-21runrttasks19:45:130
3155996033,14cyclictest12-21ksoftirqd/000:15:330
3155996023,16cyclictest347-21in:imklog20:08:130
3155996013,20cyclictest12-21ksoftirqd/023:22:070
3155996010,23cyclictest141rcu_preempt22:22:070
8322592,8sleep3909-21latency_hist20:45:153
44072590,5sleep34399-21ssh21:55:143
317899596,19cyclictest41-21ksoftirqd/320:35:323
3178995943,10cyclictest8686-21grep21:05:533
3178995942,9cyclictest24959-21systemd21:35:153
3178995937,15cyclictest276-21systemd-journal23:28:423
3178995929,13cyclictest276-21systemd-journal23:07:063
3178995927,10cyclictest141rcu_preempt19:25:243
3163995934,8cyclictest141rcu_preempt19:35:161
3163995934,7cyclictest141rcu_preempt20:20:141
3163995931,8cyclictest141rcu_preempt21:43:531
3163995928,8cyclictest116-21pr/ttymxc023:08:411
3155995940,13cyclictest7236-21kworker/u8:000:05:500
3178995837,8cyclictest141rcu_preempt20:50:343
3178995835,13cyclictest0-21swapper/320:20:503
317899583,33cyclictest41-21ksoftirqd/319:40:503
3178995831,12cyclictest347-21in:imklog19:45:493
3178995828,7cyclictest141rcu_preempt20:15:403
3178995827,17cyclictest141rcu_preempt21:30:143
3178995825,13cyclictest141rcu_preempt21:00:373
315599589,22cyclictest141rcu_preempt21:00:150
315599589,22cyclictest141rcu_preempt21:00:150
3155995842,10cyclictest29150-21unixbench_multi21:40:520
3155995839,13cyclictest23117-21unixbench_singl20:10:540
3155995839,13cyclictest23117-21unixbench_singl20:10:540
3155995839,11cyclictest0-21swapper/019:15:120
317899577,20cyclictest19496-21fschecks_time00:10:373
317899577,20cyclictest19496-21fschecks_time00:10:373
3178995736,8cyclictest141rcu_preempt22:40:063
3178995736,8cyclictest141rcu_preempt22:40:063
3178995733,8cyclictest141rcu_preempt20:50:163
3178995727,11cyclictest141rcu_preempt22:17:523
3163995739,10cyclictest23435-21timerandwakeup22:25:481
3163995736,11cyclictest0-21swapper/120:05:131
3163995729,7cyclictest141rcu_preempt19:25:191
3155995735,14cyclictest0-21swapper/020:30:230
3155995733,7cyclictest141rcu_preempt22:09:350
3155995724,14cyclictest141rcu_preempt19:22:450
3178995635,7cyclictest141rcu_preempt22:06:003
3178995631,13cyclictest0-21swapper/320:00:373
3163995634,12cyclictest0-21swapper/100:00:491
3163995629,7cyclictest25-21ksoftirqd/119:50:301
3163995628,8cyclictest141rcu_preempt22:32:041
3163995628,13cyclictest141rcu_preempt22:51:441
3163995626,7cyclictest141rcu_preempt19:25:121
316399562,12cyclictest25184-21modprobe23:25:121
3163995617,22cyclictest141rcu_preempt21:30:371
3163995617,19cyclictest141rcu_preempt20:45:121
315599568,24cyclictest141rcu_preempt19:55:520
3155995634,12cyclictest0-21swapper/019:55:120
3155995631,13cyclictest12-21ksoftirqd/022:25:460
3155995630,15cyclictest12-21ksoftirqd/023:05:460
3155995618,17cyclictest141rcu_preempt20:25:320
3178995538,10cyclictest0-21swapper/319:55:133
3178995530,8cyclictest0-21swapper/322:22:093
3178995530,10cyclictest141rcu_preempt19:10:213
3163995530,8cyclictest25-21ksoftirqd/119:07:491
3163995529,6cyclictest141rcu_preempt20:55:471
3163995529,6cyclictest141rcu_preempt20:55:471
3163995524,10cyclictest52-21kcompactd021:17:131
3155995534,14cyclictest339-21systemd-logind20:40:150
3155995529,14cyclictest141rcu_preempt19:35:280
3155995529,12cyclictest12-21ksoftirqd/020:50:340
3178995428,15cyclictest0-21swapper/320:10:513
3178995428,15cyclictest0-21swapper/320:10:513
3163995438,10cyclictest276-21systemd-journal20:35:281
3163995423,8cyclictest141rcu_preempt20:05:291
315599546,25cyclictest141rcu_preempt20:15:400
3155995414,14cyclictest141rcu_preempt21:05:120
317899537,29cyclictest0-21swapper/320:05:243
3163995333,11cyclictest276-21systemd-journal20:25:181
3155995324,10cyclictest12-21ksoftirqd/021:46:140
3178995230,8cyclictest0-21swapper/319:25:153
3178995212,22cyclictest41-21ksoftirqd/319:35:153
3178995212,22cyclictest41-21ksoftirqd/319:35:153
3163995232,11cyclictest179-21pr/ttyS021:20:391
3155995224,16cyclictest141rcu_preempt23:29:520
3163995133,12cyclictest17511-21fschecks_count19:55:381
3163995114,8cyclictest141rcu_preempt21:00:521
3163994914,10cyclictest141rcu_preempt19:35:131
3178994826,9cyclictest141rcu_preempt20:45:373
3163994816,17cyclictest141rcu_preempt19:10:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional