You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-25 - 20:03
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Thu Apr 25, 2024 12:44:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3126721308,92sleep231327-21grep07:00:532
3126721308,92sleep231327-21grep07:00:532
3213899952,53cyclictest19473-21apt-get09:25:301
32138999315,53cyclictest28494-21apt-get09:40:581
32138999315,53cyclictest28494-21apt-get09:40:581
32138999166,16cyclictest22148-21idleruntime-cro11:20:101
32138999114,60cyclictest0-21swapper/109:55:521
32155998337,27cyclictest141rcu_preempt08:40:313
32155998337,27cyclictest141rcu_preempt08:40:313
32138998259,14cyclictest25290-21latency_hist09:35:131
32155997837,20cyclictest19512-21kworker/u8:409:34:113
32155997830,9cyclictest141rcu_preempt11:55:133
32146997853,16cyclictest19920-21fschecks_time12:10:352
99052760,6sleep20-21swapper/209:10:132
32146997659,11cyclictest22744-21sendmail08:15:102
3213899762,12cyclictest7637-21cut11:50:131
3213899754,60cyclictest16644-21ssh10:15:221
3213899754,10cyclictest16265-21apt-get09:20:201
3142927524,24sleep10-21swapper/107:00:561
3142927524,24sleep10-21swapper/107:00:561
32155997445,7cyclictest141rcu_preempt08:50:123
3215599742,33cyclictest141rcu_preempt09:10:513
32146997455,13cyclictest26494-21unixbench_multi11:25:492
32146997455,13cyclictest26494-21unixbench_multi11:25:492
32146997437,10cyclictest141rcu_preempt07:25:312
32123997440,10cyclictest141rcu_preempt08:10:140
3213899734,58cyclictest1903-21apt-get11:40:221
3213899733,60cyclictest16406-21ssh11:10:241
3213899732,9cyclictest2702-21timerwakeupswit08:50:451
32138997325,39cyclictest10817-21ssh11:55:261
32138997325,39cyclictest10817-21ssh11:55:261
32155997247,17cyclictest1589-21munin-plugin-st11:40:103
32155997239,11cyclictest347-21in:imklog10:33:233
32155997234,8cyclictest141rcu_preempt10:55:523
3213899724,60cyclictest0-21swapper/109:11:091
3213899723,59cyclictest25746-21fschecks_count11:25:321
3213899723,59cyclictest25746-21fschecks_count11:25:321
3213899722,15cyclictest31317-21apt-get12:30:201
32155997148,13cyclictest10053-21wc09:10:143
32155997129,15cyclictest141rcu_preempt09:40:123
32146997112,22cyclictest33-21ksoftirqd/210:10:142
3213899714,57cyclictest0-21swapper/111:05:221
32138997144,15cyclictest0-21swapper/108:55:541
32138997138,18cyclictest25-21ksoftirqd/110:30:251
3213899713,60cyclictest26482-21grep09:35:491
3213899713,13cyclictest18124-21apt-get08:00:191
3213899712,13cyclictest23493-21/usr/sbin/munin12:15:511
32138997110,35cyclictest13315-21apt-get12:00:371
32155997045,15cyclictest25975-21idleruntime-cro08:25:103
32155997034,8cyclictest141rcu_preempt11:45:313
3213899704,57cyclictest6120-21grep07:20:501
3213899704,57cyclictest6120-21grep07:20:501
32138997044,18cyclictest13433-21latency_hist10:10:131
3213899703,13cyclictest7939-21apt-get10:00:271
3213899702,13cyclictest13285-21apt-get09:15:211
3213899702,13cyclictest12143-21/usr/sbin/munin07:40:391
32123997046,8cyclictest141rcu_preempt11:00:340
32123997046,8cyclictest141rcu_preempt11:00:340
32123997033,17cyclictest52-21kcompactd009:59:030
32123997030,12cyclictest141rcu_preempt09:11:310
32155996930,13cyclictest0-21swapper/308:55:353
32146996945,16cyclictest16232-21sendmail07:55:102
3213899697,53cyclictest20645-21head10:20:461
3213899693,58cyclictest23491-21tail11:20:481
3213899693,57cyclictest9333-21grep07:30:481
3213899693,57cyclictest9333-21grep07:30:481
3213899693,57cyclictest16540-21latency_hist10:15:141
3213899693,54cyclictest13514-21fschecks_time07:45:311
3213899693,54cyclictest13514-21fschecks_time07:45:311
3213899693,15cyclictest13109-21cat11:05:121
3213899693,15cyclictest13109-21cat11:05:121
3213899693,12cyclictest6428-21sed09:01:381
3213899692,14cyclictest1905-21apt-get07:10:241
3213899692,14cyclictest1905-21apt-get07:10:241
32123996947,15cyclictest22147-21munin-plugin-st11:20:100
32123996929,30cyclictest0-21swapper/010:45:200
3119426913,28sleep00-21swapper/007:00:510
3119426913,28sleep00-21swapper/007:00:510
32155996847,13cyclictest0-21swapper/310:40:103
32155996847,13cyclictest0-21swapper/310:40:103
32155996824,14cyclictest141rcu_preempt09:26:453
3213899684,13cyclictest31416-21apt-get11:35:231
3213899683,57cyclictest4449-21grep07:15:481
3213899683,56cyclictest29702-21fschecks_time08:35:291
3213899683,12cyclictest5774-21/usr/sbin/munin10:50:431
32138996831,28cyclictest0-21swapper/112:05:271
3213899682,9cyclictest23328-21/usr/sbin/munin08:15:341
3213899682,9cyclictest23328-21/usr/sbin/munin08:15:341
3213899682,9cyclictest10714-21/usr/sbin/munin07:35:451
3213899682,7cyclictest0-21swapper/109:45:231
3213899682,11cyclictest19954-21fschecks_count08:05:331
3213899682,10cyclictest8069-21taskset11:50:311
3213899682,10cyclictest1843-21/usr/sbin/munin10:45:201
32123996837,8cyclictest141rcu_preempt07:20:130
32123996826,13cyclictest347-21in:imklog07:44:030
32123996825,13cyclictest141rcu_preempt09:35:510
3215599675,34cyclictest5395-21seq09:00:283
32155996744,14cyclictest31248-21latency_hist12:30:173
32155996744,14cyclictest31248-21latency_hist12:30:173
32155996736,7cyclictest141rcu_preempt12:10:133
32155996734,18cyclictest141rcu_preempt08:30:493
3214699678,24cyclictest141rcu_preempt10:50:442
32146996733,9cyclictest141rcu_preempt10:37:412
32146996733,9cyclictest141rcu_preempt10:37:412
32146996723,27cyclictest33-21ksoftirqd/207:55:402
32146996713,38cyclictest33-21ksoftirqd/208:27:422
3213899674,55cyclictest0-21swapper/110:35:231
3213899674,55cyclictest0-21swapper/110:35:231
3213899673,56cyclictest15063-21sort07:50:291
3213899673,54cyclictest7928-21fschecks_time10:55:351
3213899673,11cyclictest28988-21/usr/sbin/munin11:30:391
3213899672,14cyclictest28401-21apt-get12:25:211
3213899672,14cyclictest28401-21apt-get12:25:211
3213899672,13cyclictest7531-21/usr/sbin/munin07:25:451
3213899672,12cyclictest28099-21idleruntime-cro12:25:101
3213899672,11cyclictest2283-21fschecks_time09:50:331
32123996738,7cyclictest141rcu_preempt11:10:250
32146996637,15cyclictest141rcu_preempt08:00:422
32146996627,21cyclictest141rcu_preempt11:00:552
32146996627,21cyclictest141rcu_preempt11:00:552
32146996623,28cyclictest33-21ksoftirqd/210:34:152
3213899664,54cyclictest22824-21grep08:15:121
3213899663,56cyclictest25536-21grep08:20:481
3213899663,55cyclictest20712-21sed12:10:491
3213899663,54cyclictest444-21sort08:45:341
3213899663,54cyclictest32016-21grep08:40:451
3213899663,54cyclictest32016-21grep08:40:451
3213899663,54cyclictest29380-21cat08:35:141
3213899662,14cyclictest32590-21/usr/sbin/munin07:05:321
3213899662,13cyclictest31896-21/usr/sbin/munin10:40:361
3213899662,13cyclictest23403-21timerandwakeup10:25:461
3213899662,12cyclictest10084-21latency_hist09:10:141
3213899662,10cyclictest17512-21/usr/sbin/munin07:56:021
32123996650,10cyclictest0-21swapper/010:15:110
32123996627,12cyclictest141rcu_preempt07:55:130
32146996511,38cyclictest0-21swapper/211:11:402
3213899653,9cyclictest26487-21fschecks_time08:25:321
32123996540,11cyclictest141rcu_preempt10:35:470
32123996540,11cyclictest141rcu_preempt10:35:470
32155996440,14cyclictest22194-21kworker/u8:211:20:123
32146996431,15cyclictest141rcu_preempt11:47:352
32146996431,13cyclictest33-21ksoftirqd/210:46:582
32146996428,11cyclictest0-21swapper/209:45:272
32146996425,23cyclictest141rcu_preempt12:01:272
32146996424,17cyclictest33-21ksoftirqd/211:32:272
32123996437,7cyclictest141rcu_preempt10:15:460
3212399643,39cyclictest4878-21apt-get09:00:240
32123996431,9cyclictest141rcu_preempt08:50:210
32123996425,29cyclictest0-21swapper/009:50:360
32123996425,11cyclictest141rcu_preempt08:45:460
3215599635,33cyclictest31026-21kworker/3:111:59:273
3215599635,33cyclictest31026-21kworker/3:111:59:273
32155996331,7cyclictest141rcu_preempt09:25:143
32155996328,12cyclictest141rcu_preempt08:05:543
32155996312,13cyclictest141rcu_preempt12:20:453
32146996344,13cyclictest3051-21unixbench_singl09:50:492
32146996340,12cyclictest0-21swapper/209:35:492
32146996326,17cyclictest33-21ksoftirqd/209:28:202
32146996317,24cyclictest0-21swapper/212:16:282
3212399639,32cyclictest0-21swapper/009:41:330
3212399639,32cyclictest0-21swapper/009:41:330
114932635,7sleep30-21swapper/310:05:443
32155996235,7cyclictest141rcu_preempt07:35:133
32155996235,7cyclictest141rcu_preempt07:35:133
32155996233,9cyclictest141rcu_preempt10:30:133
32146996242,14cyclictest3236-21cron08:55:102
32146996234,13cyclictest141rcu_preempt12:05:342
32146996225,15cyclictest141rcu_preempt09:25:142
32146996224,21cyclictest141rcu_preempt12:29:052
32146996224,21cyclictest141rcu_preempt12:29:052
32146996224,18cyclictest141rcu_preempt11:59:542
32146996224,18cyclictest141rcu_preempt11:59:542
32146996211,24cyclictest33-21ksoftirqd/207:40:132
32123996243,13cyclictest28267-21latency_hist12:25:120
32123996241,12cyclictest10532-21cron10:05:100
32155996148,9cyclictest52-21kcompactd011:10:483
32155996145,9cyclictest9011-21grep10:00:513
32155996126,8cyclictest141rcu_preempt12:20:133
32155996122,13cyclictest141rcu_preempt08:55:133
3214699613,35cyclictest0-21swapper/211:53:212
3214699613,28cyclictest0-21swapper/210:25:272
32146996125,10cyclictest141rcu_preempt11:35:552
32146996120,23cyclictest33-21ksoftirqd/208:15:442
32146996120,23cyclictest33-21ksoftirqd/208:15:442
32123996142,13cyclictest4546-21rm11:45:110
32123996142,12cyclictest32341-21unixbench_multi12:30:470
32123996140,8cyclictest141rcu_preempt09:08:110
32123996134,12cyclictest52-21kcompactd007:05:370
32155996040,12cyclictest22729-21idleruntime-cro08:15:103
32155996028,8cyclictest141rcu_preempt10:55:123
32146996041,11cyclictest0-21swapper/207:10:102
32146996040,7cyclictest141rcu_preempt08:25:122
32146996033,7cyclictest141rcu_preempt09:11:292
32146996028,8cyclictest141rcu_preempt12:31:472
32146996027,8cyclictest141rcu_preempt10:12:242
32146996011,26cyclictest141rcu_preempt08:40:452
32146996011,26cyclictest141rcu_preempt08:40:452
32123996035,19cyclictest30929-21ssh11:34:500
32123996034,13cyclictest10294-21kworker/u8:209:23:200
32123996025,8cyclictest141rcu_preempt12:00:130
32123996025,8cyclictest141rcu_preempt12:00:130
32123996022,13cyclictest141rcu_preempt08:30:140
32155995944,8cyclictest0-21swapper/310:40:473
32146995939,13cyclictest5869-21switchtime07:20:482
32146995939,13cyclictest5869-21switchtime07:20:482
32146995933,18cyclictest0-21swapper/208:30:492
3214699593,10cyclictest141rcu_preempt11:10:102
32146995918,12cyclictest141rcu_preempt08:05:342
32123995942,11cyclictest339-21systemd-logind08:40:110
32123995940,12cyclictest12-21ksoftirqd/011:38:120
32123995921,10cyclictest0-21swapper/011:00:130
32155995842,10cyclictest13954-21runrttasks10:10:303
32155995836,13cyclictest0-21swapper/308:30:143
32155995832,7cyclictest141rcu_preempt07:15:333
32155995831,16cyclictest347-21in:imklog11:00:283
32155995831,16cyclictest347-21in:imklog11:00:283
32155995830,9cyclictest141rcu_preempt11:40:503
32155995830,8cyclictest141rcu_preempt11:32:003
32155995826,9cyclictest141rcu_preempt07:20:273
32155995826,9cyclictest141rcu_preempt07:20:273
3214699588,25cyclictest0-21swapper/207:10:442
3214699588,25cyclictest0-21swapper/207:10:442
32146995841,9cyclictest0-21swapper/207:30:482
32146995841,9cyclictest0-21swapper/207:30:482
32146995833,11cyclictest0-21swapper/211:15:342
32146995821,15cyclictest33-21ksoftirqd/209:41:242
32146995821,15cyclictest33-21ksoftirqd/209:41:242
32146995821,15cyclictest141rcu_preempt09:15:192
3212399586,12cyclictest0-21swapper/007:10:240
3212399586,12cyclictest0-21swapper/007:10:240
32123995844,7cyclictest0-21swapper/009:20:110
32123995837,10cyclictest0-21swapper/012:30:130
32123995837,10cyclictest0-21swapper/012:30:130
32123995832,13cyclictest141rcu_preempt08:45:130
32123995832,13cyclictest141rcu_preempt08:45:130
32123995831,8cyclictest141rcu_preempt08:55:320
32123995829,9cyclictest141rcu_preempt11:45:370
32155995734,11cyclictest0-21swapper/312:01:193
32155995730,7cyclictest141rcu_preempt07:50:133
32155995730,7cyclictest141rcu_preempt07:50:133
32155995721,8cyclictest141rcu_preempt09:15:313
32155995717,10cyclictest141rcu_preempt12:34:293
32155995714,18cyclictest141rcu_preempt11:20:313
32146995724,12cyclictest141rcu_preempt09:57:432
3212399577,42cyclictest10840-21apt-get10:05:210
32123995742,8cyclictest6742-21diskmemload12:01:330
32123995735,6cyclictest141rcu_preempt11:28:130
32123995735,6cyclictest141rcu_preempt11:28:130
32123995729,8cyclictest141rcu_preempt07:35:140
32123995729,8cyclictest141rcu_preempt07:35:140
32123995728,8cyclictest141rcu_preempt09:25:240
32123995727,8cyclictest141rcu_preempt07:30:130
3212399572,25cyclictest28122-21ls10:35:100
32123995717,7cyclictest141rcu_preempt10:25:440
32123995717,12cyclictest141rcu_preempt10:20:310
32123995713,22cyclictest12-21ksoftirqd/011:05:360
32155995639,8cyclictest0-21swapper/308:00:463
32155995636,12cyclictest401ktimers/308:15:233
32155995636,12cyclictest401ktimers/308:15:233
32155995633,13cyclictest0-21swapper/311:25:273
32155995633,13cyclictest0-21swapper/311:25:273
32155995627,7cyclictest141rcu_preempt09:50:273
32146995641,8cyclictest179-21pr/ttyS011:43:352
32146995640,9cyclictest30831-21cron08:40:102
32146995639,10cyclictest10089-21idleruntime-cro11:00:102
32146995634,15cyclictest0-21swapper/207:45:112
3214699562,31cyclictest276-21systemd-journal09:01:002
32146995619,17cyclictest116-21pr/ttymxc008:55:192
32146995616,18cyclictest33-21ksoftirqd/210:23:132
32146995615,16cyclictest0-21swapper/212:20:232
32123995640,10cyclictest11370-21cron07:40:100
32155995523,14cyclictest141rcu_preempt07:15:133
32155995523,14cyclictest141rcu_preempt07:15:133
3214699556,21cyclictest0-21swapper/210:41:462
3214699553,35cyclictest0-21swapper/210:00:552
32146995510,8cyclictest141rcu_preempt11:20:222
3212399554,41cyclictest32166-21grep10:40:440
32123995537,12cyclictest16995-21fschecks_count12:05:340
32123995537,10cyclictest22915-21fschecks_time12:15:360
32123995522,12cyclictest141rcu_preempt11:54:370
32155995440,9cyclictest13939-21ntp_kernel_err11:05:403
3215599543,14cyclictest6837-21cpuspeed07:25:253
32155995412,18cyclictest41-21ksoftirqd/309:45:453
32146995435,7cyclictest141rcu_preempt09:30:142
32146995432,14cyclictest347-21in:imklog08:45:502
3212399543,43cyclictest23247-21fschecks_time08:15:320
3212399543,43cyclictest23247-21fschecks_time08:15:320
32123995424,8cyclictest141rcu_preempt10:55:120
3099625410,21sleep30-21swapper/307:00:423
3099625410,21sleep30-21swapper/307:00:423
32155995338,7cyclictest0-21swapper/309:41:363
32155995338,7cyclictest0-21swapper/309:41:363
32155995335,11cyclictest0-21swapper/312:15:093
32155995326,7cyclictest141rcu_preempt07:05:433
32155995319,8cyclictest141rcu_preempt09:55:493
3214699535,23cyclictest0-21swapper/210:16:352
32123995334,9cyclictest0-21swapper/012:15:120
3215599528,20cyclictest0-21swapper/310:45:463
32155995215,8cyclictest12072-21latency07:40:363
32155995215,16cyclictest0-21swapper/310:24:013
3214699521,7cyclictest141rcu_preempt07:15:332
32146995214,12cyclictest0-21swapper/207:46:092
32146995214,12cyclictest0-21swapper/207:46:092
3212399523,8cyclictest0-21swapper/008:00:190
3212399522,43cyclictest22520-21ssh09:30:290
32123995215,17cyclictest17870-21grep08:00:110
32123995214,8cyclictest141rcu_preempt08:30:290
3215599513,8cyclictest16637-21apt-get10:15:253
32155995136,8cyclictest0-21swapper/308:35:443
32155995133,11cyclictest52-21kcompactd007:53:443
32155995111,13cyclictest141rcu_preempt07:40:133
32123995134,9cyclictest0-21swapper/011:20:320
32123995123,7cyclictest141rcu_preempt09:45:350
32155995022,8cyclictest141rcu_preempt07:55:583
32123995032,11cyclictest0-21swapper/007:50:110
32123995032,11cyclictest0-21swapper/007:50:110
32123995023,14cyclictest0-21swapper/008:10:350
32123995021,8cyclictest141rcu_preempt08:20:280
3212399493,18cyclictest399-21runrttasks07:23:000
3212399493,18cyclictest399-21runrttasks07:23:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional