You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-09-13 - 00:24
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7.osadl.org (updated Thu Sep 12, 2024 12:44:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1458221150,5sleep114321-21latency_hist08:33:201
1597621059,67sleep30-21swapper/306:59:203
1597621059,67sleep30-21swapper/306:59:203
169129910026,15cyclictest7407-21fschecks_time11:23:400
16330210012,44sleep116431-21head06:59:301
16330210012,44sleep116431-21head06:59:301
16912999919,17cyclictest22003-21/usr/sbin/munin11:48:320
16912999919,17cyclictest22003-21/usr/sbin/munin11:48:320
16912999732,11cyclictest0-21swapper/009:43:170
16912999720,14cyclictest0-21swapper/009:53:530
1691299963,37cyclictest13964-21wc11:33:510
1691299944,16cyclictest16344-21latency_hist12:33:200
1691299943,11cyclictest24807-21apt-key09:03:200
16912999427,11cyclictest0-21swapper/009:28:190
16912999324,12cyclictest0-21swapper/011:43:170
1691299925,36cyclictest0-21swapper/010:43:390
1691299925,36cyclictest0-21swapper/010:43:390
16912999233,51cyclictest0-21swapper/009:18:330
16912999228,9cyclictest26113-21unixbench_singl11:53:530
16912999220,16cyclictest15248-21ntpq09:43:470
16912999220,16cyclictest15248-21ntpq09:43:470
16912999124,8cyclictest0-21swapper/012:28:190
1691299912,32cyclictest22623-21fschecks_count10:53:400
1691299908,26cyclictest12639-21kworker/u8:309:18:190
1691299899,74cyclictest25517-21rm10:58:340
1691299894,30cyclictest7193-21apt12:18:170
1691299894,29cyclictest23351-21apt-get09:58:310
16912998916,14cyclictest22943-21cron08:58:170
16931998870,12cyclictest0-21swapper/310:58:463
1691299885,11cyclictest26398-21apt-get10:03:470
1691299873,19cyclictest180942sleep009:48:470
16912998731,9cyclictest12639-21kworker/u8:309:04:060
16924998545,34cyclictest24945-21apt-get10:58:192
1691299854,29cyclictest13857-21apt-get10:38:290
1691299853,22cyclictest4740-21latency_hist08:03:200
16912998525,10cyclictest0-21swapper/007:33:200
16931998458,15cyclictest14725-21latency_hist08:33:213
1693199843,48cyclictest0-21swapper/309:03:343
1691299846,10cyclictest0-21swapper/012:03:310
1691299844,33cyclictest0-21swapper/009:38:180
1691299844,33cyclictest0-21swapper/009:38:180
16912998420,11cyclictest0-21swapper/012:08:380
16912998215,16cyclictest19958-1kworker/0:2H12:03:200
16920998136,38cyclictest31216-21fschecks_count12:03:361
1691299813,12cyclictest181irq_work/010:48:360
16931998055,17cyclictest12653-21idleruntime-cro08:28:173
16912998028,8cyclictest0-21swapper/008:37:490
16924997943,30cyclictest0-21swapper/212:13:172
1691299795,31cyclictest0-21swapper/011:08:190
1691299795,31cyclictest0-21swapper/011:08:190
16920997750,18cyclictest16304-21latency_hist12:33:201
1692099772,16cyclictest7493-21uname11:23:421
16931997540,13cyclictest141rcu_preempt07:50:533
1693199752,23cyclictest41-21ksoftirqd/310:03:353
1692099753,15cyclictest8154-21runrttasks10:28:301
1692099753,14cyclictest10782-21find11:28:501
16920997529,37cyclictest20515-21modprobe07:13:181
16920997528,39cyclictest0-21swapper/109:23:181
16912997519,11cyclictest0-21swapper/007:23:120
198752740,5sleep10-21swapper/111:43:491
16931997451,13cyclictest24564-21diskmemload11:23:343
16931997432,22cyclictest141rcu_preempt11:54:303
1692499742,42cyclictest233952ssh09:58:372
16920997428,37cyclictest0-21swapper/111:33:521
16920997427,37cyclictest0-21swapper/110:13:441
154382749,40sleep014920-21apt-get06:58:400
154382749,40sleep014920-21apt-get06:58:400
1693199732,48cyclictest41-21ksoftirqd/312:32:523
16920997310,13cyclictest26016-21sed10:03:201
16924997251,13cyclictest10103-21unixbench_multi09:33:522
16924997251,13cyclictest10103-21unixbench_multi09:33:522
16924997249,16cyclictest20512-21munin-plugin-st07:13:182
1692099725,58cyclictest22024-21apt-get11:48:311
1692099725,58cyclictest22024-21apt-get11:48:311
16920997253,13cyclictest20509-21unixbench_multi10:48:491
16920997251,13cyclictest28914-21munin-plugin-st10:08:181
1692099723,8cyclictest25036-21apt-get11:53:321
1692099723,15cyclictest28182-21grep07:33:491
1692099723,15cyclictest23339-21chrt07:18:521
1692099722,16cyclictest17156-21munin-plugin-st09:48:171
1692099722,16cyclictest17156-21munin-plugin-st09:48:171
16920997215,50cyclictest13222-21diskmemload12:28:191
16931997129,9cyclictest0-21swapper/310:38:173
16931997126,21cyclictest141rcu_preempt08:58:383
16931997117,22cyclictest41-21ksoftirqd/310:53:093
1692499713,39cyclictest141rcu_preempt09:56:552
1692099713,8cyclictest11301-21apt-get08:23:291
1692099713,60cyclictest25574-21latency_hist07:28:211
1692099713,19cyclictest17387-21grep08:38:521
1692099713,16cyclictest20915-21grep08:49:091
1692099713,16cyclictest20915-21grep08:49:091
1692099713,16cyclictest17754-21/usr/sbin/munin09:48:381
1692099713,15cyclictest4965-21modprobe10:23:171
1692099713,13cyclictest16677-21/usr/sbin/munin10:43:431
1692099713,13cyclictest16677-21/usr/sbin/munin10:43:431
1692099713,12cyclictest30711-21fschecks_count07:43:381
1692099712,15cyclictest21759-21unixbench_singl07:13:541
16931997049,11cyclictest0-21swapper/310:43:173
16931997032,16cyclictest8766-21kworker/u8:109:33:173
16931997031,13cyclictest141rcu_preempt10:53:443
16920997049,14cyclictest400-21runrttasks11:08:401
1692099703,16cyclictest4530-21modprobe08:03:171
1692099703,12cyclictest8009-21latency_hist10:28:201
1692099703,12cyclictest18191-21wc08:43:331
1692099703,11cyclictest29186-21/usr/sbin/munin10:08:421
16920997025,37cyclictest29039-21sh11:03:511
16920997025,37cyclictest29039-21sh11:03:511
1692099702,17cyclictest14801-21timerwakeupswit10:38:511
16920997012,50cyclictest25100-21sh10:58:201
16931996950,10cyclictest0-21swapper/309:58:213
16931996933,14cyclictest141rcu_preempt09:19:243
16931996932,13cyclictest141rcu_preempt09:48:403
16931996925,27cyclictest0-21swapper/307:28:213
16931996912,30cyclictest41-21ksoftirqd/307:19:073
16931996911,37cyclictest22803-21apt-get09:58:323
1692499694,56cyclictest8150-21apt-get10:28:312
1692499693,58cyclictest19663-21cat08:48:202
1692499693,54cyclictest16695-21apt-get10:43:302
1692499693,54cyclictest16695-21apt-get10:43:302
1692099698,53cyclictest0-21swapper/107:03:281
1692099693,57cyclictest8337-21uniq08:13:391
1692099693,57cyclictest7851-21latency_hist08:13:201
1692099693,15cyclictest27866-21cstates09:08:341
1692099693,14cyclictest28008-21/usr/sbin/munin11:58:391
1692099693,14cyclictest25863-21fschecks_time07:28:401
1692099693,12cyclictest5715-21sed08:03:511
16920996926,36cyclictest0-21swapper/111:13:521
1692099692,15cyclictest4413-21taskset11:18:381
1692099692,10cyclictest23460-21chrt08:58:381
1693199688,42cyclictest41-21ksoftirqd/311:45:093
1693199686,38cyclictest41-21ksoftirqd/309:26:163
1693199685,33cyclictest41-21ksoftirqd/310:21:013
16931996836,17cyclictest141rcu_preempt11:48:483
16931996836,17cyclictest141rcu_preempt11:48:483
16924996849,13cyclictest13344-21fschecks_time11:33:392
1692099684,7cyclictest0-21swapper/111:38:301
1692099683,8cyclictest0-21swapper/112:13:301
1692099683,16cyclictest20589-21tune2fs09:53:401
16920996823,37cyclictest0-21swapper/112:18:391
16920996823,37cyclictest0-21swapper/112:18:391
1692099682,17cyclictest4563-21grep09:23:531
1692099682,14cyclictest24945-21apt-get10:58:321
1692099682,13cyclictest1779-21taskset12:08:381
16912996833,11cyclictest12-21ksoftirqd/008:08:390
16931996744,9cyclictest141rcu_preempt11:41:263
16931996731,12cyclictest141rcu_preempt09:39:063
16931996716,24cyclictest41-21ksoftirqd/307:18:203
1692499673,54cyclictest17824-21latency_hist08:43:202
1692099673,56cyclictest31005-21/usr/sbin/munin09:13:421
1692099673,15cyclictest26238-21grep09:04:101
1692099673,14cyclictest1467-21cat07:53:201
1692099672,16cyclictest21571-21/usr/sbin/munin08:53:491
1693199663,42cyclictest41-21ksoftirqd/311:09:513
16931996620,25cyclictest363-21in:imklog09:46:243
16931996620,25cyclictest363-21in:imklog09:46:243
16931996620,25cyclictest26954-21kworker/u8:108:45:063
1692499668,50cyclictest416-21snmpd07:03:512
1692499662,56cyclictest0-21swapper/208:29:282
1692499662,55cyclictest25189-21taskset09:03:442
1692099663,55cyclictest16194-21modprobe08:38:181
1692099663,55cyclictest14383-21ssh09:43:181
1692099663,18cyclictest6609-21fschecks_time09:28:391
1692099662,15cyclictest9968-21grep09:33:511
1692099662,15cyclictest9968-21grep09:33:511
1692099662,15cyclictest11279-21fschecks_count10:33:381
16912996642,15cyclictest26033-21systemd-journal10:13:400
16912996620,24cyclictest12-21ksoftirqd/010:33:520
16931996538,15cyclictest0-21swapper/312:13:173
16931996534,18cyclictest141rcu_preempt11:58:303
16931996514,44cyclictest401ktimers/311:16:513
16931996511,32cyclictest41-21ksoftirqd/307:55:163
16931996511,32cyclictest41-21ksoftirqd/307:55:163
1692499653,54cyclictest4126-21head07:58:512
16924996531,26cyclictest0-21swapper/207:33:172
1692499652,56cyclictest1584-21sed09:18:502
1692499652,53cyclictest10477-21apt-get12:23:292
16924996525,16cyclictest24564-21diskmemload09:48:412
1692099653,12cyclictest29839-21grep07:38:511
1692099653,12cyclictest29839-21grep07:38:511
1692099652,9cyclictest9689-21apt-get08:18:291
1692099652,9cyclictest9689-21apt-get08:18:291
16912996533,8cyclictest141rcu_preempt07:48:140
1693199649,32cyclictest41-21ksoftirqd/308:34:283
16931996436,16cyclictest141rcu_preempt12:28:113
16931996434,7cyclictest141rcu_preempt07:43:353
16931996428,7cyclictest141rcu_preempt10:26:533
16931996424,13cyclictest141rcu_preempt07:38:483
16931996424,13cyclictest141rcu_preempt07:38:483
16931996422,10cyclictest0-21swapper/307:33:373
1692499648,49cyclictest19939-21apt-get08:48:342
1692499648,49cyclictest19939-21apt-get08:48:342
16924996449,9cyclictest22441-21timerwakeupswit08:53:512
1692499643,55cyclictest29004-21ps11:03:492
1692499643,55cyclictest29004-21ps11:03:492
16924996427,27cyclictest0-21swapper/210:28:202
16924996426,30cyclictest0-21swapper/211:23:402
1692499642,53cyclictest29591-21/usr/sbin/munin10:08:402
1692499642,53cyclictest26033-21systemd-journal08:03:292
1692099643,11cyclictest1748-21fschecks_count07:53:371
1692099643,11cyclictest1748-21fschecks_count07:53:371
16912996448,10cyclictest26033-21systemd-journal12:18:410
16912996448,10cyclictest26033-21systemd-journal12:18:410
1543526411,31sleep214920-21apt-get06:58:382
1543526411,31sleep214920-21apt-get06:58:382
1693199637,22cyclictest41-21ksoftirqd/312:22:553
1693199637,22cyclictest41-21ksoftirqd/312:22:553
16931996342,14cyclictest26033-21systemd-journal07:11:273
16931996329,12cyclictest141rcu_preempt08:50:023
16931996329,12cyclictest141rcu_preempt08:50:023
16931996325,20cyclictest141rcu_preempt08:05:573
16931996321,10cyclictest363-21in:imklog07:28:413
16931996312,8cyclictest41-21ksoftirqd/310:09:573
16931996311,23cyclictest41-21ksoftirqd/308:53:343
16924996343,14cyclictest12842-21latency_hist11:33:192
1692499633,53cyclictest23464-21fschecks_time08:58:382
1692499633,52cyclictest32054-21apt-get10:13:302
1692499633,50cyclictest3516-21/usr/sbin/munin10:18:522
1692499633,49cyclictest21016-21fschecks_time07:13:392
16924996333,9cyclictest30682-21munin-plugin-st12:03:172
1692499632,53cyclictest27743-21rm09:08:292
1692499632,43cyclictest15380-21ntp_kernel_pll_08:34:032
1692499632,39cyclictest31439-21if_err_eth011:08:382
16912996328,25cyclictest0-21swapper/010:28:290
16931996242,8cyclictest141rcu_preempt10:14:423
16931996236,18cyclictest7782-21kworker/u8:108:13:183
16931996232,9cyclictest141rcu_preempt10:31:353
16931996211,22cyclictest41-21ksoftirqd/311:06:413
16931996211,22cyclictest41-21ksoftirqd/311:06:413
16931996211,19cyclictest41-21ksoftirqd/309:13:403
16931996210,15cyclictest41-21ksoftirqd/307:58:413
1692499624,52cyclictest28042-21latency_hist11:03:182
1692499624,45cyclictest32366-21apt-get07:48:502
1692499623,49cyclictest4148-21ssh11:18:212
16924996228,26cyclictest0-21swapper/212:28:532
1692499622,53cyclictest13860-21ssh10:38:282
1692499622,50cyclictest6642-21/usr/sbin/munin08:08:392
16912996243,12cyclictest26033-21systemd-journal07:50:300
316852610,12sleep0161rcuc/010:13:140
196172610,6chrt0-21swapper/007:08:440
16931996140,15cyclictest401ktimers/312:05:113
16931996136,7cyclictest141rcu_preempt11:31:193
16931996128,11cyclictest141rcu_preempt08:15:143
16931996110,24cyclictest41-21ksoftirqd/312:14:003
1692499613,51cyclictest22769-21/usr/sbin/munin11:48:492
1692499613,51cyclictest22769-21/usr/sbin/munin11:48:492
1692499613,51cyclictest17239-21ls09:48:202
1692499613,51cyclictest17239-21ls09:48:202
1692499613,50cyclictest26398-21apt-get10:03:442
1692499612,48cyclictest12396-21/usr/sbin/munin09:38:432
1693199607,13cyclictest41-21ksoftirqd/309:12:383
16931996031,8cyclictest141rcu_preempt09:33:433
16931996031,8cyclictest141rcu_preempt09:33:433
16931996029,8cyclictest141rcu_preempt11:18:533
16931996028,16cyclictest141rcu_preempt08:20:513
16931996028,16cyclictest141rcu_preempt08:20:513
16931996023,10cyclictest141rcu_preempt07:03:343
16931996012,21cyclictest41-21ksoftirqd/310:48:023
16931996012,21cyclictest41-21ksoftirqd/310:48:023
16924996040,11cyclictest6764-21munin-plugin-st11:23:172
16924996035,15cyclictest16641-21ntp_offset11:38:452
1692499603,50cyclictest0-21swapper/207:43:312
1692499603,49cyclictest10288-21sh12:23:192
1692499603,49cyclictest10288-21sh12:23:192
1692499603,47cyclictest3126-21latency_hist07:58:212
1692499603,47cyclictest3126-21latency_hist07:58:212
1692499603,47cyclictest11836-21/usr/sbin/munin08:23:462
1692499603,17cyclictest19827-21fschecks_time10:48:372
1692499602,50cyclictest24282-21fschecks_time07:23:412
1692499602,50cyclictest0-21swapper/207:38:302
1692499602,50cyclictest0-21swapper/207:38:302
16931995930,8cyclictest141rcu_preempt11:34:173
16931995910,17cyclictest41-21ksoftirqd/308:42:573
1692499594,47cyclictest31280-21fschecks_time12:03:362
1692499593,49cyclictest27485-21fschecks_count07:33:362
1692499593,47cyclictest25576-21/usr/sbin/munin11:53:452
1692499593,47cyclictest19041-21apt-get11:43:292
1692499593,46cyclictest4623-21cstates12:13:332
1692499592,50cyclictest7141-21find09:28:512
1692499592,48cyclictest30877-21/usr/sbin/munin09:13:402
16912995946,8cyclictest12-21ksoftirqd/010:23:520
1692499583,45cyclictest9671-21/usr/sbin/munin08:18:282
1692499583,45cyclictest9671-21/usr/sbin/munin08:18:282
1692499582,48cyclictest8297-21cut08:13:392
1692499582,48cyclictest11008-21seq10:33:282
1692499582,48cyclictest0-21swapper/209:23:302
1691299584,21cyclictest400-21runrttasks11:33:200
16912995836,7cyclictest141rcu_preempt11:11:460
1692499573,47cyclictest23044-21taskset07:18:492
16912995714,8cyclictest141rcu_preempt08:23:200
16912995714,8cyclictest141rcu_preempt08:23:200
16912995637,12cyclictest26033-21systemd-journal09:08:350
16912995540,10cyclictest26033-21systemd-journal08:33:210
16912995540,10cyclictest0-21swapper/011:48:190
16912995525,23cyclictest21279-21cron08:53:180
16912995525,23cyclictest21279-21cron08:53:180
16912995439,9cyclictest0-21swapper/008:48:200
16912995436,11cyclictest0-21swapper/011:23:190
1691299537,10cyclictest141rcu_preempt08:23:390
16912995339,8cyclictest8069-21ssh09:31:070
16912995229,9cyclictest141rcu_preempt08:38:500
16912995229,12cyclictest0-21swapper/011:13:380
16912995223,23cyclictest12-21ksoftirqd/007:33:310
16912995219,11cyclictest141rcu_preempt10:18:420
1691299517,10cyclictest141rcu_preempt07:38:510
1691299517,10cyclictest141rcu_preempt07:38:510
16912995136,9cyclictest120-21pr/ttymxc008:08:190
16912995112,13cyclictest26033-21systemd-journal07:58:200
16912995112,13cyclictest26033-21systemd-journal07:58:200
1691299504,21cyclictest141rcu_preempt07:28:200
16912995031,12cyclictest26033-21systemd-journal08:13:500
16912995028,11cyclictest0-21swapper/007:18:070
16912994834,8cyclictest183-21pr/ttyS007:03:510
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional