You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-26 - 21:36
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7.osadl.org (updated Fri Apr 26, 2024 12:44:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
765327690,14sleep17648-21fschecks_time07:00:341
765327690,14sleep17648-21fschecks_time07:00:341
88539910950,25cyclictest13059-21rm10:55:160
113521030,4chrt1134-21awk09:35:402
113521030,4chrt1134-21awk09:35:402
88789910147,9cyclictest18328-21dpkg09:10:062
8853999937,21cyclictest23864-21modprobe11:15:060
8853999737,24cyclictest4028-21sed12:30:090
8853999636,26cyclictest2838-21kernelversion08:25:290
8853999535,27cyclictest16626-21grep11:00:430
8853999436,20cyclictest32216-21mailstats10:30:420
8853999336,22cyclictest18688-21cut09:10:100
8853999336,21cyclictest0-21swapper/007:26:190
8853999333,27cyclictest17217-21expr10:05:210
8853999250,30cyclictest0-21swapper/012:35:080
8853999238,13cyclictest11964-21modprobe11:50:060
8853999233,31cyclictest3346-21diskmemload11:35:140
8853999233,23cyclictest7186-21sed11:40:430
8853999131,19cyclictest0-21swapper/011:55:050
8853999041,21cyclictest25628-21rm10:20:150
8853999037,25cyclictest408-21ssh09:35:220
8853999037,25cyclictest408-21ssh09:35:220
8853998955,8cyclictest4531-21ssh10:40:210
8853998939,30cyclictest399-21runrttasks08:04:410
8853998934,29cyclictest24937-21ssh12:10:330
8853998934,17cyclictest24386-21apt-get09:20:200
86822896,15sleep20-21swapper/207:04:472
86822896,15sleep20-21swapper/207:04:472
8853998840,12cyclictest5326-21modprobe09:45:060
8853998838,17cyclictest25229-21/usr/sbin/munin11:15:410
8853998837,20cyclictest28747-21seq11:22:260
8853998837,16cyclictest31352-21ssh11:26:310
8853998835,22cyclictest399-21runrttasks07:33:320
8878998711,29cyclictest29391-21/usr/sbin/munin08:05:162
886999873,63cyclictest4586-21/usr/sbin/munin12:31:001
8853998757,21cyclictest14077-21apt-get09:00:520
8853998757,21cyclictest14077-21apt-get09:00:520
8853998749,30cyclictest27297-21idleruntime-cro08:00:060
8853998749,28cyclictest688-21latency_hist12:25:090
8853998742,24cyclictest0-21swapper/010:52:240
8853998738,38cyclictest21244-21cat11:10:090
8853998737,19cyclictest32135-21ssh09:34:090
8853998737,19cyclictest17569-21snmpd07:40:060
8853998737,11cyclictest15308-21apt-get11:55:200
8853998734,20cyclictest15003-21if_eth010:01:280
8853998734,17cyclictest1062-21sort08:15:430
8853998733,23cyclictest13258-21ntp_kernel_pll_07:15:340
8853998725,17cyclictest0-21swapper/009:28:220
8853998639,19cyclictest15919-21sed07:25:090
8853998634,22cyclictest19292-21ssh09:11:210
8853998634,18cyclictest0-21swapper/007:50:470
8853998631,29cyclictest15436-21diskmemload09:18:410
8853998623,33cyclictest30415-21ssh10:27:540
8853998541,15cyclictest467-21ssh11:30:200
8853998541,15cyclictest467-21ssh11:30:200
8853998537,22cyclictest7470-21cstates08:40:220
8853998534,19cyclictest7223-21latency_hist08:40:090
8853998529,35cyclictest0-21swapper/012:00:180
758828510,38sleep07651-21fschecks_time07:00:340
758828510,38sleep07651-21fschecks_time07:00:340
8886998442,13cyclictest276-21systemd-journal09:00:073
8853998438,18cyclictest13577-21latency_hist09:00:100
8853998437,19cyclictest32403-21munin-plugin-st08:15:070
8853998436,18cyclictest21401-21cpu12:05:230
8853998436,18cyclictest21401-21cpu12:05:230
8853998435,19cyclictest23233-21latency10:15:310
8853998435,17cyclictest30809-21cat08:10:080
8853998428,20cyclictest12452-21apt07:15:070
8853998428,20cyclictest12452-21apt07:15:070
8853998335,21cyclictest399-21runrttasks09:55:240
8853998331,19cyclictest0-21swapper/009:48:450
8853998331,19cyclictest0-21swapper/009:48:450
8878998245,13cyclictest33-21ksoftirqd/209:55:292
886999825,40cyclictest14811-21ssh10:01:221
8853998237,14cyclictest9421-21ssh09:50:390
8853998236,16cyclictest22741-21sshd07:45:220
8853998232,22cyclictest11943-21modprobe08:55:060
8869998133,12cyclictest141rcu_preempt11:16:591
8853998130,18cyclictest0-21swapper/008:45:440
8853998129,20cyclictest0-21swapper/010:10:110
887899808,30cyclictest28618-21apt-get10:25:212
8853998037,16cyclictest29191-21ssh12:17:330
8878997961,12cyclictest12458-21munin-plugin-st07:15:072
8878997961,12cyclictest12458-21munin-plugin-st07:15:072
8878997952,10cyclictest33-21ksoftirqd/208:15:292
8878997937,15cyclictest347-21in:imklog08:50:232
8853997932,38cyclictest10552-21latency_hist07:10:090
8853997928,18cyclictest4773-21sensors08:30:380
8853997928,18cyclictest4773-21sensors08:30:380
8853997852,17cyclictest1639-21ssh10:35:150
8853997831,17cyclictest8788-21ssh10:46:490
8878997746,10cyclictest33-21ksoftirqd/211:50:422
8878997745,10cyclictest141rcu_preempt09:40:412
887899772,57cyclictest14326-21kworker/u8:210:00:392
8869997754,12cyclictest12873-21latency_hist10:55:081
29442770,2sleep1231rcuc/111:34:521
29442770,2sleep1231rcuc/111:34:521
8878997657,12cyclictest8315-21grep08:40:412
8878997642,12cyclictest33-21ksoftirqd/211:20:422
8878997640,13cyclictest33-21ksoftirqd/209:15:252
888699752,51cyclictest18910-21taskset09:10:243
8878997538,13cyclictest33-21ksoftirqd/211:40:472
8853997533,26cyclictest22179-21taskset07:42:110
8886997451,14cyclictest1337-21sendmail10:35:063
8878997453,14cyclictest5425-21runrttasks08:35:072
8878997453,14cyclictest5425-21runrttasks08:35:072
8869997458,10cyclictest28424-21latency_hist10:25:101
8878997350,9cyclictest33-21ksoftirqd/207:35:392
8878997326,15cyclictest347-21in:imklog10:12:302
8869997352,12cyclictest3833-21idleruntime-cro08:30:061
8869997348,7cyclictest30535-21ssh11:25:361
8886997212,28cyclictest12132-21latency_hist11:50:083
8886997149,13cyclictest6959-21munin-plugin-st12:35:063
8886997116,29cyclictest18795-21cstates11:05:243
887899717,45cyclictest4252-21apt-get12:30:562
8878997137,12cyclictest33-21ksoftirqd/212:26:282
8878997134,11cyclictest179-21pr/ttyS009:05:082
8878997134,11cyclictest179-21pr/ttyS009:05:082
8869997155,10cyclictest30136-21latency_hist12:20:061
8869997152,12cyclictest10958-21idleruntime-cro09:55:061
8869997130,13cyclictest1-21systemd07:20:071
8869997129,25cyclictest141rcu_preempt09:18:251
8853997126,18cyclictest2024-21runrttasks08:22:500
8886997013,27cyclictest17870-21ssh11:59:433
8878997051,12cyclictest276-21systemd-journal11:45:302
8878997033,10cyclictest33-21ksoftirqd/210:20:432
8878997030,14cyclictest33-21ksoftirqd/211:17:392
8869997027,23cyclictest0-21swapper/108:05:181
8886996952,10cyclictest14089-21idleruntime-cro07:20:063
8878996952,10cyclictest0-21swapper/207:30:082
8878996948,14cyclictest668-21fschecks_count11:30:272
8878996948,14cyclictest668-21fschecks_count11:30:272
887899693,32cyclictest0-21swapper/210:05:152
8869996945,8cyclictest141rcu_preempt08:35:281
8869996944,10cyclictest141rcu_preempt11:50:081
8869996941,7cyclictest141rcu_preempt08:10:401
8886996842,17cyclictest347-21in:imklog12:21:203
8878996843,15cyclictest0-21swapper/209:25:072
8878996838,7cyclictest33-21ksoftirqd/207:40:412
8869996846,13cyclictest18574-21sed09:10:091
8869996844,14cyclictest0-21swapper/109:45:281
8869996844,14cyclictest0-21swapper/109:45:281
8869996842,12cyclictest141rcu_preempt10:20:071
886999683,49cyclictest276-21systemd-journal09:00:301
886999683,49cyclictest276-21systemd-journal09:00:301
8886996743,14cyclictest28263-21latency_hist10:25:093
8886996715,21cyclictest26864-21runrttasks09:25:073
8886996710,24cyclictest17465-21if_eth010:05:283
8878996744,16cyclictest0-21swapper/211:25:332
8869996750,10cyclictest14021-21latency_hist10:00:091
8869996746,8cyclictest141rcu_preempt08:00:071
8869996736,8cyclictest141rcu_preempt12:11:591
8869996727,24cyclictest141rcu_preempt11:54:511
8869996719,14cyclictest0-21swapper/108:45:231
8869996717,20cyclictest25-21ksoftirqd/107:25:441
8886996615,19cyclictest200892sleep312:02:303
8886996610,23cyclictest27344-21latency_hist08:00:083
8878996646,11cyclictest19115-21mailstats12:00:402
8878996633,10cyclictest33-21ksoftirqd/211:35:432
8869996637,7cyclictest141rcu_preempt08:50:411
886999663,34cyclictest241ktimers/110:05:331
8869996628,22cyclictest141rcu_preempt11:06:121
8869996621,10cyclictest0-21swapper/110:25:521
8869996619,19cyclictest25-21ksoftirqd/107:10:481
8869996619,19cyclictest25-21ksoftirqd/107:10:481
8869996615,25cyclictest0-21swapper/110:10:311
8869996614,29cyclictest0-21swapper/108:19:231
8869996545,8cyclictest141rcu_preempt10:48:231
8869996544,8cyclictest141rcu_preempt11:10:351
8869996530,16cyclictest141rcu_preempt10:35:301
8869996530,14cyclictest141rcu_preempt09:40:311
8886996448,9cyclictest399-21runrttasks12:10:303
8886996441,13cyclictest18071-21kworker/u8:412:18:153
8886996412,20cyclictest23364-21sh11:13:323
8878996443,8cyclictest33-21ksoftirqd/212:06:102
8878996443,8cyclictest33-21ksoftirqd/212:06:102
8878996443,13cyclictest0-21swapper/209:00:072
8878996437,16cyclictest0-21swapper/208:50:072
8878996429,13cyclictest33-21ksoftirqd/209:46:322
8878996429,13cyclictest33-21ksoftirqd/209:46:322
8878996429,11cyclictest33-21ksoftirqd/210:34:282
8878996410,20cyclictest141rcu_preempt07:25:092
8869996430,15cyclictest141rcu_preempt11:20:251
8869996428,16cyclictest0-21swapper/109:28:541
8869996426,23cyclictest141rcu_preempt12:00:441
8869996426,17cyclictest141rcu_preempt09:35:291
8869996426,17cyclictest141rcu_preempt09:35:291
8869996425,11cyclictest141rcu_preempt10:30:131
8869996416,28cyclictest0-21swapper/109:31:241
8869996415,18cyclictest25-21ksoftirqd/110:59:371
8869996414,30cyclictest0-21swapper/107:05:411
888699635,11cyclictest141rcu_preempt07:50:303
8886996341,10cyclictest0-21swapper/307:10:483
8886996341,10cyclictest0-21swapper/307:10:483
8886996310,35cyclictest28789-21chrt08:03:473
887899635,12cyclictest27185-21apt-get09:25:232
8878996343,12cyclictest0-21swapper/212:15:062
8878996342,12cyclictest33-21ksoftirqd/211:00:422
8878996331,17cyclictest141rcu_preempt10:49:192
8878996328,13cyclictest33-21ksoftirqd/208:35:402
8878996315,21cyclictest0-21swapper/210:57:422
8869996328,9cyclictest141rcu_preempt11:38:111
8869996328,18cyclictest141rcu_preempt09:20:431
8869996319,25cyclictest0-21swapper/112:26:561
8869996313,18cyclictest25-21ksoftirqd/108:55:211
888699624,28cyclictest15457-21modprobe09:05:073
888699624,28cyclictest15457-21modprobe09:05:073
8886996242,12cyclictest15436-21diskmemload10:15:383
8886996214,8cyclictest141rcu_preempt11:55:063
8886996213,20cyclictest26193-21taskset11:18:313
8878996226,15cyclictest141rcu_preempt11:08:452
8869996238,7cyclictest141rcu_preempt07:44:271
8869996233,7cyclictest141rcu_preempt11:40:301
8869996225,6cyclictest0-21swapper/110:45:091
8869996218,21cyclictest25-21ksoftirqd/112:20:261
755326213,23sleep30-21swapper/307:00:333
755326213,23sleep30-21swapper/307:00:333
8878996143,12cyclictest276-21systemd-journal11:58:152
8878996142,7cyclictest33-21ksoftirqd/212:20:302
8878996138,9cyclictest33-21ksoftirqd/207:45:292
8878996127,11cyclictest33-21ksoftirqd/207:59:232
8869996127,9cyclictest141rcu_preempt09:15:061
8869996127,14cyclictest141rcu_preempt08:40:221
8869996123,9cyclictest0-21swapper/111:55:431
8869996117,17cyclictest25-21ksoftirqd/111:05:081
888699609,18cyclictest15436-21diskmemload10:58:503
888699607,19cyclictest10367-21runrttasks10:50:243
8886996042,10cyclictest0-21swapper/310:40:303
8886996012,12cyclictest21608-21ntp_states07:40:373
8878996041,12cyclictest0-21swapper/210:55:062
8878996040,8cyclictest33-21ksoftirqd/207:05:282
8878996026,13cyclictest276-21systemd-journal12:15:392
8878996022,21cyclictest179-21pr/ttyS009:15:082
8878996019,9cyclictest0-21swapper/211:10:402
8869996011,29cyclictest0-21swapper/108:31:221
8869996011,29cyclictest0-21swapper/108:31:221
888699598,25cyclictest141rcu_preempt08:50:233
888699598,19cyclictest3510-21cstates11:35:233
888699595,20cyclictest30035-21ntpq08:06:353
8886995928,21cyclictest15436-21diskmemload09:25:233
8886995911,18cyclictest31148-21cstates08:10:233
887899598,22cyclictest28741-21kworker/2:009:30:272
8878995939,11cyclictest347-21in:imklog07:55:082
8878995926,17cyclictest32395-21latency_hist08:15:072
8878995923,11cyclictest0-21swapper/209:50:252
8869995939,13cyclictest24063-21idleruntime-cro12:10:061
8869995939,13cyclictest24063-21idleruntime-cro12:10:061
8886995842,11cyclictest0-21swapper/310:50:093
888699583,14cyclictest14291-21/usr/sbin/munin10:01:223
8886995812,15cyclictest0-21swapper/307:25:083
887899583,27cyclictest3427-21unixbench_singl08:25:402
8878995823,14cyclictest141rcu_preempt10:19:312
8878995812,26cyclictest141rcu_preempt07:30:502
886999582,25cyclictest141rcu_preempt07:50:421
8869995817,18cyclictest25-21ksoftirqd/107:35:421
8886995737,10cyclictest0-21swapper/307:35:423
8886995714,16cyclictest8092-21sh11:42:383
8886995712,16cyclictest15436-21diskmemload11:27:473
8886995710,18cyclictest22210-21cat12:05:433
8886995710,18cyclictest22210-21cat12:05:433
8878995728,8cyclictest33-21ksoftirqd/208:00:292
8878995715,13cyclictest141rcu_preempt10:35:162
8869995729,7cyclictest141rcu_preempt08:00:391
8869995726,8cyclictest0-21swapper/107:30:431
8869995718,18cyclictest25-21ksoftirqd/107:45:271
888699567,17cyclictest8004-21run-parts08:40:383
888699566,19cyclictest8967-21taskset08:45:173
8886995638,12cyclictest18207-21latency07:30:363
888699562,22cyclictest9061-21ntpq09:50:323
8869995633,7cyclictest141rcu_preempt07:20:161
8886995538,11cyclictest2242-21systemd-run08:25:083
8886995535,11cyclictest5937-21fschecks_count08:35:293
8886995513,17cyclictest2352-21sh11:33:093
8886995513,17cyclictest2352-21sh11:33:093
888699551,20cyclictest22333-21ssh10:14:463
8886995437,11cyclictest347-21in:imklog07:45:233
8886995431,13cyclictest325-21apt-get09:35:193
8886995431,13cyclictest325-21apt-get09:35:193
8886995410,13cyclictest0-21swapper/309:15:283
8878995410,20cyclictest0-21swapper/210:41:352
888699539,16cyclictest15436-21diskmemload09:30:523
888699537,8cyclictest141rcu_preempt10:35:273
8886995339,8cyclictest0-21swapper/307:25:193
8886995332,12cyclictest3198-21apt-get09:40:203
8886995312,16cyclictest15436-21diskmemload11:21:323
8878995333,7cyclictest33-21ksoftirqd/207:15:302
888699525,19cyclictest15436-21diskmemload09:08:453
8886995236,9cyclictest0-21swapper/309:50:063
8886995236,9cyclictest0-21swapper/309:50:063
8886995236,8cyclictest0-21swapper/308:15:413
8886995235,9cyclictest0-21swapper/307:10:073
8886995233,12cyclictest4011-21apt-key12:30:103
8886995230,13cyclictest0-21swapper/310:26:413
8869995221,16cyclictest0-21swapper/108:25:081
8886995128,12cyclictest0-21swapper/311:05:093
8886995032,10cyclictest0-21swapper/309:55:183
888699483,13cyclictest141rcu_preempt08:30:063
8878994830,10cyclictest2212-21cron08:25:072
8886994717,20cyclictest4405-21/usr/sbin/munin08:30:273
8886994717,20cyclictest4405-21/usr/sbin/munin08:30:273
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional