You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-05-01 - 23:45
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Wed May 01, 2024 12:44:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
20303991283,88cyclictest6724-21latency_hist09:24:490
20303991173,97cyclictest27845-21apt-get11:50:000
2372621100,4chrt23725-21ssh10:49:311
20303991012,64cyclictest13353-21ssh09:35:150
2030399993,83cyclictest1247-21apt-get09:15:040
2030399993,83cyclictest1247-21apt-get09:15:040
2030399984,85cyclictest214072chrt10:45:050
20317999742,13cyclictest3377-21munin-plugin-st11:09:462
20303999734,56cyclictest0-21swapper/007:04:510
2030399943,64cyclictest0-21swapper/011:14:460
2030399943,59cyclictest28036-21ssh09:05:100
2030399943,57cyclictest26923-21ssh09:59:470
20327999340,27cyclictest0-21swapper/311:35:163
20317999333,15cyclictest15472-21cpuspeed10:35:012
2030399933,75cyclictest21427-21apt-get09:49:590
2015429315,43sleep120169-21modprobe07:04:471
2030399922,61cyclictest5042-21taskset10:16:110
2030399913,71cyclictest0-21swapper/008:35:530
2030399912,61cyclictest27374-21latency_hist07:24:500
2030399912,59cyclictest10671-21tr09:30:260
2030399902,76cyclictest7527-21ssh12:10:020
2030399892,58cyclictest7154-21irqstats11:15:110
2030399892,58cyclictest7154-21irqstats11:15:110
2030399883,76cyclictest17973-21apt10:39:460
2030399883,55cyclictest0-21swapper/007:40:240
2030399883,54cyclictest31304-21mailstats07:35:210
2030399882,78cyclictest28993-21/usr/sbin/munin07:30:210
2030399882,78cyclictest28993-21/usr/sbin/munin07:30:210
2030399873,74cyclictest16109-21/usr/sbin/munin09:40:110
2030399872,74cyclictest4481-21munin-node12:04:550
2030399872,59cyclictest823-21ssh11:58:500
2030399863,74cyclictest18888-21grep09:45:070
2030399863,74cyclictest18888-21grep09:45:070
2030399854,67cyclictest24404-21apt-get08:55:060
2030399853,53cyclictest629-21rm11:04:520
2030399852,56cyclictest27372-21diskmemload09:27:270
2030399852,56cyclictest27372-21diskmemload09:27:270
2030399852,53cyclictest9465-21latency_hist08:09:490
2030399852,53cyclictest3025-21/usr/sbin/munin07:50:120
2030399852,53cyclictest1727-21grep12:00:060
2030399843,73cyclictest399-21runrttasks11:32:570
2030399843,72cyclictest567-21latency_hist11:04:500
20303998411,65cyclictest19584-21latency_hist12:29:480
20303998411,65cyclictest19584-21latency_hist12:29:480
2030399834,72cyclictest19025-21awk10:40:180
2030399833,54cyclictest0-21swapper/010:50:140
2030399832,74cyclictest25527-21latency_hist07:19:490
2030399832,74cyclictest12652-21latency_hist11:24:500
2030399832,51cyclictest27372-21diskmemload10:27:190
2030399827,49cyclictest0-21swapper/011:35:290
20303998258,14cyclictest13224-21modprobe12:19:460
2030399823,71cyclictest13820-21ssh10:31:130
2030399822,60cyclictest28941-21sed07:29:490
46002810,7sleep30-21swapper/311:10:233
20311998157,14cyclictest19071-21fschecks_count11:35:061
2030399813,70cyclictest0-21swapper/011:40:150
2030399812,72cyclictest16725-21sed12:24:490
2030399812,68cyclictest6515-21cut08:00:100
2030399812,62cyclictest0-21swapper/011:47:200
2030399812,50cyclictest29271-21ssh10:58:140
2030399812,47cyclictest3361-21kworker/u8:011:25:200
2030399803,70cyclictest6803-21ls10:20:030
2030399802,71cyclictest0-21swapper/007:44:570
2030399802,71cyclictest0-21swapper/007:44:570
2030399802,70cyclictest14437-21fschecks_count12:21:160
2030399802,68cyclictest1025-21ssh10:10:090
2030399802,52cyclictest521-21sed10:09:490
2030399802,52cyclictest521-21sed10:09:490
2030399793,69cyclictest27372-21diskmemload10:01:190
2030399793,67cyclictest11192-21sshd08:15:010
2030399793,65cyclictest10064-21ntpq08:10:180
2030399792,68cyclictest20388-21/usr/sbin/munin12:30:150
20303997921,49cyclictest0-21swapper/008:25:080
20317997857,12cyclictest2504-21unixbench_multi12:00:222
20317997812,39cyclictest3888-21apt-get10:15:042
2030399783,67cyclictest22491-21latency_hist08:49:460
2030399783,66cyclictest12802-21apt-get08:20:020
2030399783,53cyclictest31031-21/usr/sbin/munin09:10:100
20311997758,12cyclictest31722-21runrttasks09:10:231
20311997734,13cyclictest0-21swapper/112:34:481
2030399772,50cyclictest0-21swapper/007:55:200
2030399772,50cyclictest0-21swapper/007:55:200
20317997553,14cyclictest22525-21sh12:34:462
20317997550,17cyclictest22498-21munin-plugin-st08:49:462
2031199759,56cyclictest13770-21apt-get12:20:001
2031199759,56cyclictest13770-21apt-get12:20:001
2031199757,39cyclictest18132-21diskstats08:36:121
2030399753,64cyclictest26933-21unixbench_multi09:00:270
2030399753,64cyclictest26933-21unixbench_multi09:00:270
2030399753,64cyclictest24250-21cat08:54:490
20317997339,19cyclictest141rcu_preempt11:20:092
20311997353,12cyclictest21842-21munin-plugin-st07:09:471
20311997353,12cyclictest21842-21munin-plugin-st07:09:471
20311997311,10cyclictest141rcu_preempt11:25:081
2030399733,61cyclictest22750-21fschecks_count07:10:110
2030399733,61cyclictest16996-21unixbench_singl08:30:210
2030399732,64cyclictest0-21swapper/008:40:010
158712730,5sleep20-21swapper/209:40:042
2031799722,18cyclictest13881-21ssh12:20:122
2031799722,18cyclictest13881-21ssh12:20:122
20311997255,11cyclictest7666-21idleruntime-cro08:04:461
2030399722,60cyclictest20544-21/usr/sbin/munin07:05:020
2030399722,60cyclictest20544-21/usr/sbin/munin07:05:020
2032799717,45cyclictest9558-21kworker/u8:310:15:433
2031199715,58cyclictest27522-21kworker/u8:410:15:111
2020627111,35sleep320224-21unin-run07:04:493
20317997052,11cyclictest29847-21munin-plugin-st10:59:462
20317997050,13cyclictest27150-21munin-plugin-st07:24:472
20327996927,11cyclictest0-21swapper/310:33:213
2031799696,9cyclictest0-21swapper/211:25:132
20317996951,12cyclictest14144-21cron08:24:472
20311996952,11cyclictest26795-21cron10:54:461
20327996814,35cyclictest41-21ksoftirqd/312:30:013
20317996850,11cyclictest16819-21sshd12:25:002
20317996850,11cyclictest16819-21sshd12:25:002
20311996842,17cyclictest26884-21sh11:47:551
20311996824,15cyclictest0-21swapper/110:14:481
20327996739,19cyclictest6049-21munin-plugin-st07:59:473
20327996739,19cyclictest6049-21munin-plugin-st07:59:473
20317996749,11cyclictest17391-21idleruntime-cro08:34:472
20311996710,49cyclictest27823-21cpuspeed10:00:591
20317996646,11cyclictest0-21swapper/207:09:472
20317996646,11cyclictest0-21swapper/207:09:472
2031799664,30cyclictest0-21swapper/211:34:572
20327996544,15cyclictest390-21cron10:09:463
20327996544,15cyclictest390-21cron10:09:463
20327996544,11cyclictest0-21swapper/311:44:493
20327996542,16cyclictest23906-21sendmail07:14:473
20327996541,12cyclictest0-21swapper/311:19:463
20327996541,12cyclictest0-21swapper/311:19:463
20327996526,22cyclictest6468-21kworker/u8:007:34:503
20327996526,22cyclictest6468-21kworker/u8:007:34:503
20327996521,13cyclictest141rcu_preempt11:25:243
20327996513,7cyclictest0-21swapper/308:14:583
20317996548,10cyclictest0-21swapper/211:40:092
2032799647,11cyclictest2400-21head07:45:233
2032799647,11cyclictest2400-21head07:45:233
20327996449,9cyclictest19592-21unixbench_multi09:45:213
20327996449,9cyclictest19592-21unixbench_multi09:45:213
20327996442,10cyclictest0-21swapper/307:40:243
20327996436,18cyclictest3533-21latency_hist10:14:493
2032799642,36cyclictest27668-21sort07:25:083
20317996438,8cyclictest33-21ksoftirqd/207:39:492
20327996342,13cyclictest0-21swapper/310:00:593
2032799634,14cyclictest27372-21diskmemload12:05:013
2032799632,35cyclictest19475-21cron12:29:453
2032799632,35cyclictest19475-21cron12:29:453
20327996318,9cyclictest141rcu_preempt08:50:113
20317996345,12cyclictest3569-21apt-key10:14:492
20317996341,15cyclictest30377-21sendmail07:34:472
20317996341,15cyclictest30377-21sendmail07:34:472
20317996339,15cyclictest31252-21idleruntime11:55:072
20317996337,8cyclictest33-21ksoftirqd/211:15:092
20317996337,8cyclictest33-21ksoftirqd/211:15:092
2031799633,38cyclictest17902-21apt-get08:34:592
2031199638,25cyclictest27372-21diskmemload11:44:471
20311996346,11cyclictest4260-21kworker/u8:208:59:461
20311996337,13cyclictest0-21swapper/109:44:491
20311996335,15cyclictest0-21swapper/111:14:461
20327996232,8cyclictest0-21swapper/311:31:533
20327996225,7cyclictest141rcu_preempt11:20:083
20317996244,11cyclictest10987-21unixbench_singl09:30:292
20317996241,14cyclictest15765-21munin-plugin-st08:29:462
20317996240,15cyclictest179-21pr/ttyS007:14:002
20317996239,16cyclictest276-21systemd-journal09:51:202
20317996239,13cyclictest10906-21idleruntime-cro08:14:472
2031799623,23cyclictest0-21swapper/212:05:132
20311996241,13cyclictest24116-21taskset07:14:501
20311996240,13cyclictest9506-21latency_hist11:19:481
20311996240,13cyclictest9506-21latency_hist11:19:481
20327996145,8cyclictest0-21swapper/311:50:073
2032799613,15cyclictest13065-21fschecks_time08:20:123
20327996127,19cyclictest116-21pr/ttymxc011:49:493
20317996122,12cyclictest33-21ksoftirqd/209:20:102
20317996118,29cyclictest141rcu_preempt12:18:362
20311996135,16cyclictest28026-21fschecks_time09:05:101
188322618,30sleep218820-21apt-get07:00:042
20327996034,14cyclictest0-21swapper/309:40:043
20327996026,19cyclictest52-21kcompactd011:00:123
20327996020,7cyclictest0-21swapper/309:10:233
2031799605,26cyclictest141rcu_preempt10:07:212
2031799605,26cyclictest141rcu_preempt10:07:212
20317996040,12cyclictest0-21swapper/207:50:232
20317996018,26cyclictest0-21swapper/212:10:432
20311996034,17cyclictest18493-21idleruntime-cro11:34:461
20311996021,17cyclictest347-21in:imklog12:29:341
20311996021,17cyclictest347-21in:imklog12:29:341
20327995940,12cyclictest347-21in:imklog08:04:553
20327995935,13cyclictest0-21swapper/308:29:483
20327995926,15cyclictest21454-21ssh10:45:083
20327995925,11cyclictest41-21ksoftirqd/311:05:083
2031799598,26cyclictest0-21swapper/208:55:172
2031799594,27cyclictest30633-21unin-run11:54:472
20317995939,12cyclictest0-21swapper/207:49:472
20317995939,12cyclictest0-21swapper/207:49:472
20317995934,14cyclictest0-21swapper/209:04:492
20317995934,14cyclictest0-21swapper/209:04:492
20317995933,17cyclictest347-21in:imklog09:12:482
2031799593,15cyclictest13870-21head09:35:242
2031799592,33cyclictest13491-21sed10:30:262
20317995919,25cyclictest141rcu_preempt09:27:502
20317995919,25cyclictest141rcu_preempt09:27:502
20317995910,28cyclictest33-21ksoftirqd/210:20:192
20311995941,11cyclictest179-21pr/ttyS011:08:241
20311995937,16cyclictest0-21swapper/109:54:461
20311995931,17cyclictest116-21pr/ttymxc008:05:091
20311995911,19cyclictest10659-21rm12:15:031
283932580,8sleep20-21swapper/210:01:152
2032799584,33cyclictest41-21ksoftirqd/309:25:333
2032799584,33cyclictest41-21ksoftirqd/309:25:333
20327995842,9cyclictest0-21swapper/312:10:273
20327995839,13cyclictest347-21in:imklog12:00:213
20327995836,11cyclictest0-21swapper/312:21:313
20327995836,11cyclictest0-21swapper/312:21:313
20327995836,11cyclictest0-21swapper/312:15:243
20327995835,13cyclictest0-21swapper/310:40:213
20317995842,10cyclictest0-21swapper/209:55:062
20317995838,12cyclictest347-21in:imklog08:50:102
20317995830,20cyclictest0-21swapper/209:15:102
20317995830,20cyclictest0-21swapper/209:15:102
20317995813,24cyclictest141rcu_preempt10:26:322
20317995812,11cyclictest28316-21sed07:25:222
20311995835,14cyclictest27372-21diskmemload09:26:521
20311995835,14cyclictest27372-21diskmemload09:26:521
20311995835,14cyclictest16304-21timerandwakeup10:35:231
20311995835,14cyclictest0-21swapper/109:39:471
2032799573,29cyclictest41-21ksoftirqd/310:54:473
20327995720,8cyclictest0-21swapper/309:30:123
20317995717,7cyclictest141rcu_preempt10:50:112
20311995740,11cyclictest0-21swapper/111:04:471
20311995736,11cyclictest0-21swapper/112:10:271
20311995735,12cyclictest0-21swapper/107:44:471
20311995735,12cyclictest0-21swapper/107:15:271
20311995734,12cyclictest0-21swapper/110:30:051
20311995723,11cyclictest141rcu_preempt11:55:011
2032799568,23cyclictest17357-21cron08:34:463
20327995633,12cyclictest0-21swapper/308:14:473
20327995631,13cyclictest0-21swapper/307:05:083
20327995631,13cyclictest0-21swapper/307:05:083
20327995630,17cyclictest0-21swapper/309:04:473
20327995630,17cyclictest0-21swapper/309:04:473
20327995624,11cyclictest141rcu_preempt08:55:143
20327995619,8cyclictest141rcu_preempt09:35:133
20327995618,9cyclictest0-21swapper/309:50:443
20327995616,8cyclictest0-21swapper/309:55:203
2031799566,12cyclictest0-21swapper/209:09:492
20317995634,12cyclictest0-21swapper/211:10:202
2031799563,34cyclictest30451-21grep11:00:112
20317995631,15cyclictest0-21swapper/210:40:202
20311995634,12cyclictest0-21swapper/111:50:111
20311995633,13cyclictest0-21swapper/110:40:061
2032799559,27cyclictest0-21swapper/309:07:113
2032799559,15cyclictest41-21ksoftirqd/310:55:073
20327995543,7cyclictest0-21swapper/308:45:063
20327995537,10cyclictest179-21pr/ttyS009:19:213
20327995537,10cyclictest179-21pr/ttyS009:19:213
20327995534,8cyclictest141rcu_preempt09:20:053
2032799553,32cyclictest0-21swapper/310:23:383
20327995529,7cyclictest0-21swapper/307:18:353
20327995529,15cyclictest0-21swapper/310:29:463
2031799559,28cyclictest141rcu_preempt11:47:392
2031799556,25cyclictest0-21swapper/209:45:012
2031799556,25cyclictest0-21swapper/209:45:012
2031799552,15cyclictest21464-21fschecks_count10:45:082
20311995539,11cyclictest0-21swapper/112:00:231
20311995537,12cyclictest32010-21munin-plugin-st07:39:481
20311995536,12cyclictest0-21swapper/109:34:461
20327995441,9cyclictest52-21kcompactd007:20:053
20327995439,7cyclictest0-21swapper/310:35:243
20327995437,8cyclictest0-21swapper/307:39:483
20327995435,10cyclictest7663-21munin-plugin-st08:04:463
20317995437,9cyclictest0-21swapper/211:30:262
20317995436,12cyclictest347-21in:imklog08:05:082
2031799543,29cyclictest0-21swapper/208:00:112
20317995431,11cyclictest0-21swapper/208:40:232
20317995429,14cyclictest0-21swapper/208:15:222
20311995433,11cyclictest0-21swapper/107:45:221
20311995433,11cyclictest0-21swapper/107:45:221
20311995431,12cyclictest0-21swapper/107:50:021
20311995431,11cyclictest0-21swapper/107:30:191
20311995431,11cyclictest0-21swapper/107:30:191
20317995333,11cyclictest0-21swapper/207:55:242
20317995333,11cyclictest0-21swapper/207:55:242
20317995330,15cyclictest25521-21modprobe07:19:472
2031199533,23cyclictest141rcu_preempt09:00:231
2031199533,23cyclictest141rcu_preempt09:00:231
20311995332,11cyclictest0-21swapper/107:29:491
20311995236,10cyclictest399-21runrttasks10:20:041
20311995227,16cyclictest3611-21sh09:19:101
20311995227,16cyclictest3611-21sh09:19:101
2031199521,20cyclictest16180-21diskstats08:30:051
2032799518,9cyclictest18681-21chrt08:36:343
20327995128,14cyclictest4392-21modprobe07:54:463
20327995124,14cyclictest0-21swapper/308:40:133
20317995120,12cyclictest0-21swapper/207:40:232
20311995131,9cyclictest0-21swapper/108:14:491
20311995131,11cyclictest0-21swapper/108:45:261
2031199512,19cyclictest13149-21irqstats08:20:151
20327995011,13cyclictest141rcu_preempt11:55:193
20311995032,12cyclictest0-21swapper/110:09:461
20311995032,12cyclictest0-21swapper/110:09:461
20311995023,14cyclictest0-21swapper/107:24:491
2031199502,18cyclictest24946-21ntp_states09:55:151
20311995020,20cyclictest0-21swapper/110:56:071
20311994932,10cyclictest4892-21fschecks_count07:55:111
20311994932,10cyclictest4892-21fschecks_count07:55:111
2031199491,21cyclictest4373-21diskstats09:20:091
20311994828,11cyclictest276-21systemd-journal10:27:001
2031199481,17cyclictest23285-21ntp_states08:50:141
20311994729,9cyclictest0-21swapper/108:40:201
20311994728,10cyclictest0-21swapper/108:25:081
20311994720,16cyclictest0-21swapper/109:45:211
20311994720,16cyclictest0-21swapper/109:45:211
2031199471,19cyclictest5937-21ssh12:06:221
20311994627,11cyclictest0-21swapper/108:15:131
20311994528,8cyclictest0-21swapper/111:20:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional