You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-21 - 05:05
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Sun Apr 21, 2024 00:44:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
282769910230,63cyclictest0-21swapper/019:05:330
26782210128,41sleep20-21swapper/219:00:392
28283999634,13cyclictest0-21swapper/122:05:491
2827699969,9cyclictest141rcu_preempt23:31:060
2827699963,11cyclictest15566-21sort22:20:510
2827699957,7cyclictest0-21swapper/000:10:430
2827699953,13cyclictest3007-21true22:00:290
2827699934,10cyclictest3175-21apt-get19:25:420
28283999231,16cyclictest4816-21unixbench_multi21:07:021
28276999129,53cyclictest0-21swapper/019:22:110
28276999129,53cyclictest0-21swapper/019:22:110
2827699893,12cyclictest12122-21modprobe22:15:280
2827699893,10cyclictest12883-21grep23:11:020
2827699878,7cyclictest0-21swapper/021:38:300
2827699874,75cyclictest2900-21diskmemload21:35:310
2827699874,13cyclictest17762-21latency_hist23:20:300
2827699863,76cyclictest5651-21latency_hist23:55:300
2827699863,76cyclictest11229-21ssh21:18:370
2827699863,17cyclictest28853-21cron20:45:280
2827699863,13cyclictest2900-21diskmemload22:10:320
2827699863,11cyclictest3421-21/usr/sbin/munin21:05:370
2827699862,12cyclictest27192-21ntpq23:35:530
28276998534,44cyclictest17820-21ssh00:16:010
2827699853,14cyclictest20520-21modprobe23:25:280
2827699853,10cyclictest20764-21modprobe20:20:280
2827699852,12cyclictest1382-21/usr/sbin/munin22:51:050
2827699843,14cyclictest3506-21apt-get22:00:390
2827699843,12cyclictest0-21swapper/000:22:010
2827699843,12cyclictest0-21swapper/000:22:010
2827699842,19cyclictest6150-21acpi23:00:390
2827699842,19cyclictest6150-21acpi23:00:390
2827699842,14cyclictest399-21runrttasks23:30:310
2827699842,10cyclictest24830-21switchtime22:36:030
2827699833,9cyclictest0-21swapper/000:00:020
2827699833,9cyclictest0-21swapper/000:00:020
2827699833,17cyclictest24049-21/usr/sbin/munin21:40:560
2827699833,17cyclictest24049-21/usr/sbin/munin21:40:560
2827699833,15cyclictest22891-21grep20:25:500
2827699833,14cyclictest5844-21latency_hist23:00:310
28276998324,50cyclictest0-21swapper/019:36:050
2827699823,73cyclictest2609-21cron23:50:280
2827699823,18cyclictest12852-21latency_hist19:55:320
2827699823,17cyclictest6361-21latency_hist19:35:320
2827699823,17cyclictest21369-21fschecks_count22:30:480
2827699823,17cyclictest21369-21fschecks_count22:30:480
2827699823,11cyclictest30922-21ssh22:46:060
2827699823,11cyclictest0-21swapper/020:21:060
2827699823,11cyclictest0-21swapper/020:21:060
2827699822,8cyclictest11588-21latency_hist23:10:300
28276998223,51cyclictest0-21swapper/000:30:300
2827699822,18cyclictest32613-21tune2fs20:55:490
2827699822,15cyclictest8031-21/usr/sbin/munin19:41:130
2827699822,15cyclictest8031-21/usr/sbin/munin19:41:130
2827699822,12cyclictest11924-21kernelversion00:05:540
2827699822,12cyclictest11924-21kernelversion00:05:540
2827699822,10cyclictest31879-21munin-run19:15:290
28283998165,10cyclictest29571-21munin-plugin-st21:50:281
28283998165,10cyclictest29571-21munin-plugin-st21:50:281
2827699818,15cyclictest29532-21/usr/sbin/munin23:40:530
2827699813,70cyclictest0-21swapper/020:10:310
2827699813,14cyclictest19309-21sed22:26:080
2827699813,14cyclictest15304-21/usr/sbin/munin21:25:540
2827699813,13cyclictest14778-21grep20:00:500
2827699813,13cyclictest14778-21grep20:00:500
2827699813,13cyclictest11316-21latency_hist00:05:300
2827699813,11cyclictest25724-21grep00:30:500
2827699813,11cyclictest1744-21grep21:00:510
28276998127,47cyclictest27206-21ssh21:45:590
28276998127,47cyclictest27206-21ssh21:45:590
28276998119,54cyclictest0-21swapper/021:21:540
28296998055,16cyclictest6672-21fschecks_time22:05:503
2827699804,17cyclictest13079-21cut19:55:520
2827699804,15cyclictest32479-21cron21:55:280
2827699803,13cyclictest12680-21fschecks_count22:15:530
28291997954,16cyclictest22216-21unixbench_multi21:36:082
28283997964,8cyclictest9678-21ssh22:11:071
2827699794,12cyclictest30478-21cut20:50:290
2827699793,13cyclictest10605-21grep19:46:060
28276997924,48cyclictest0-21swapper/019:16:080
2827699792,13cyclictest25864-21/usr/sbin/munin20:36:010
2827699792,12cyclictest6668-21tune2fs21:10:550
2827699783,13cyclictest32085-21cron20:55:290
2827699782,15cyclictest29028-21taskset19:06:000
2827699782,12cyclictest26931-21ssh22:40:410
2827699782,12cyclictest26931-21ssh22:40:410
28296997741,28cyclictest0-21swapper/300:30:283
2827699778,11cyclictest0-21swapper/020:11:070
2827699773,12cyclictest25585-21modprobe20:35:290
2827699773,12cyclictest25585-21modprobe20:35:290
2827699772,13cyclictest11244-21/usr/sbin/munin19:50:540
2770127516,28sleep10-21swapper/119:01:151
28291997430,27cyclictest141rcu_preempt20:21:032
28291997430,27cyclictest141rcu_preempt20:21:032
28296997234,31cyclictest20831-21idleruntime-cro21:35:283
2829699722,61cyclictest8953-21fschecks_count00:00:523
28296997220,44cyclictest0-21swapper/319:06:043
28283997257,9cyclictest276-21systemd-journal22:01:311
2829699713,13cyclictest21782-21timerwakeupswit23:26:043
2829699703,58cyclictest14746-21fschecks_time20:00:503
2829699703,58cyclictest14746-21fschecks_time20:00:503
28296997033,29cyclictest0-21swapper/322:51:023
2829699703,16cyclictest30513-21apt-get19:10:423
2829699702,12cyclictest29122-21apt-get20:45:403
28291997049,12cyclictest24252-21fschecks_count22:35:492
28283997052,10cyclictest19120-21idleruntime-cro20:15:291
2829699694,60cyclictest0-21swapper/322:11:353
2829699693,55cyclictest24256-21/usr/sbin/munin20:30:453
2829699693,55cyclictest24256-21/usr/sbin/munin20:30:453
2829699693,14cyclictest14579-21apt-key23:15:303
2829699692,13cyclictest17880-21/usr/sbin/munin00:16:013
28291996950,12cyclictest22371-21munin-plugin-st00:25:282
28291996950,12cyclictest22371-21munin-plugin-st00:25:282
28291996947,15cyclictest32070-21kworker/u8:220:55:292
28283996951,11cyclictest2897-21latency_hist22:55:281
28283996951,10cyclictest2955-21systemd-run21:05:301
28283996920,33cyclictest20515-21kworker/u8:300:25:301
28283996920,33cyclictest20515-21kworker/u8:300:25:301
2829699683,14cyclictest13520-21find19:56:033
2829699682,9cyclictest2900-21diskmemload23:50:413
2829699682,13cyclictest3206-21/usr/sbin/munin22:56:033
2829699682,13cyclictest31023-21taskset20:50:543
2829699682,13cyclictest19275-21grep22:26:073
28296996820,13cyclictest141rcu_preempt19:20:323
28291996847,13cyclictest11595-21munin-plugin-st23:10:282
28296996744,13cyclictest179-21pr/ttyS023:05:513
2829699673,56cyclictest27248-21grep21:46:023
2829699673,56cyclictest27248-21grep21:46:023
2829699673,56cyclictest25100-21unixbench_singl22:36:063
2829699673,56cyclictest24023-21latency_hist20:30:303
2829699673,56cyclictest0-21swapper/300:20:403
2829699673,56cyclictest0-21swapper/300:20:403
2829699673,55cyclictest6372-21ssh21:10:433
2829699673,55cyclictest26867-21fschecks_count23:35:463
2829699673,54cyclictest15585-21sh22:20:523
2829699673,53cyclictest1746-21fschecks_count21:00:513
2829699673,12cyclictest21686-21/usr/sbin/munin22:30:553
2829699673,12cyclictest21686-21/usr/sbin/munin22:30:553
2829699673,12cyclictest11952-21taskset19:51:013
2829699673,12cyclictest11952-21taskset19:51:013
28296996726,10cyclictest179-21pr/ttyS020:20:283
2829699672,13cyclictest4854-21head21:07:023
2829699672,13cyclictest3579-21ssh22:00:523
28296996713,8cyclictest141rcu_preempt22:46:053
28291996745,12cyclictest0-21swapper/219:35:292
28283996752,9cyclictest12927-21unixbench_singl23:11:021
28283996750,10cyclictest12081-21apt-key21:20:311
28283996734,14cyclictest141rcu_preempt00:02:401
28283996726,10cyclictest141rcu_preempt21:30:511
2829699668,51cyclictest14045-21date00:10:303
2829699668,51cyclictest14045-21date00:10:303
2829699665,51cyclictest15327-21apt-get21:25:443
28296996648,12cyclictest20734-21latency_hist23:25:313
2829699663,54cyclictest26809-21sed20:36:033
2829699663,50cyclictest6412-21/usr/sbin/munin19:35:413
2829699663,15cyclictest3375-21fschecks_count19:25:513
2829699662,8cyclictest9310-21ssh21:15:443
2829699662,54cyclictest24602-21/usr/sbin/munin21:41:003
2829699662,54cyclictest24602-21/usr/sbin/munin21:41:003
2829699662,15cyclictest13317-21/usr/sbin/munin21:21:563
2829699662,13cyclictest21266-21tune2fs20:20:503
2829699662,13cyclictest21266-21tune2fs20:20:503
2829699662,12cyclictest6156-21chrt23:00:393
2829699662,12cyclictest6156-21chrt23:00:393
2829699662,12cyclictest27324-21/usr/sbin/munin22:40:513
2829699662,12cyclictest27324-21/usr/sbin/munin22:40:513
28291996650,8cyclictest0-21swapper/200:10:282
28291996650,8cyclictest0-21swapper/200:10:282
2829699653,55cyclictest21163-21/usr/sbin/munin21:36:023
2829699653,54cyclictest24339-21ntp_kernel_pll_23:31:003
2829699653,53cyclictest5025-21fschecks_time19:30:503
2829699653,16cyclictest28013-21ntpq20:40:563
2829699653,11cyclictest641-21/usr/sbin/munin23:45:583
2829699653,10cyclictest29867-21apt-get21:50:413
2829699652,8cyclictest12457-21apt-get22:15:423
2829699652,54cyclictest29532-21/usr/sbin/munin23:40:373
2829699652,53cyclictest16991-21latency_hist00:15:313
2829699652,12cyclictest920-21/usr/sbin/munin21:55:583
2829699652,10cyclictest2964-21latency_hist19:25:313
2829699652,10cyclictest2964-21latency_hist19:25:313
2829699643,54cyclictest15617-21fschecks_time23:16:383
2829699642,7cyclictest0-21swapper/300:30:433
2829699642,55cyclictest8245-21fschecks_count19:40:573
2829699642,55cyclictest8245-21fschecks_count19:40:573
2829699642,54cyclictest18017-21sort20:10:523
2829699642,53cyclictest5863-21/usr/sbin/munin23:55:473
2829699642,53cyclictest5863-21/usr/sbin/munin23:55:473
2829699642,13cyclictest16631-21/usr/sbin/munin20:05:563
28291996439,16cyclictest3447-21apt-key22:00:322
2829699633,13cyclictest11061-21latency_hist19:50:303
2829699632,10cyclictest32625-21tune2fs20:55:493
28291996342,15cyclictest18546-21fschecks_time21:30:522
28291996339,18cyclictest0-21swapper/222:10:492
2829199632,43cyclictest12569-21ssh21:21:062
2829699622,52cyclictest28325-21wc19:05:333
28291996241,17cyclictest0-21swapper/222:35:302
28291996241,17cyclictest0-21swapper/222:35:302
28291996227,8cyclictest3294-21wc21:05:322
2828399626,18cyclictest15047-21apt-get23:16:281
28283996246,10cyclictest1390-21unixbench_multi21:56:071
28291996144,12cyclictest29522-21idleruntime-cro22:45:282
28291996144,12cyclictest29522-21idleruntime-cro22:45:282
28291996138,15cyclictest0-21swapper/223:35:472
2828399618,28cyclictest347-21in:imklog23:50:571
28283996041,11cyclictest116-21pr/ttymxc019:51:021
28283996041,11cyclictest116-21pr/ttymxc019:51:021
28283996037,16cyclictest1054-21latency_hist19:20:331
28291995942,9cyclictest0-21swapper/200:11:032
28291995939,10cyclictest0-21swapper/219:56:062
28291995937,14cyclictest18553-21fschecks_time22:25:522
28291995931,18cyclictest0-21swapper/222:20:302
2828399594,32cyclictest6639-21ssh21:10:551
28283995942,9cyclictest0-21swapper/120:55:291
28283995942,8cyclictest0-21swapper/123:50:281
28283995938,13cyclictest0-21swapper/120:41:031
2829199588,10cyclictest141rcu_preempt23:50:312
28291995843,9cyclictest18732-21uptime20:11:082
28291995842,10cyclictest0-21swapper/200:05:282
28291995836,12cyclictest0-21swapper/221:25:392
28291995834,14cyclictest0-21swapper/222:45:502
28291995823,25cyclictest12053-21apt-config21:20:312
28283995843,10cyclictest28267-21cyclictest00:06:251
28283995843,10cyclictest28267-21cyclictest00:06:251
28283995837,14cyclictest0-21swapper/123:35:281
28283995835,15cyclictest26938-21ssh22:40:421
28283995835,15cyclictest26938-21ssh22:40:421
28283995832,19cyclictest347-21in:imklog22:31:091
28283995832,19cyclictest347-21in:imklog22:31:091
28283995821,13cyclictest141rcu_preempt22:25:591
28291995736,13cyclictest17484-21cron20:10:282
2829199573,30cyclictest0-21swapper/219:10:312
28283995742,9cyclictest3873-21ssh22:56:001
28283995741,9cyclictest0-21swapper/121:30:291
28283995741,10cyclictest276-21systemd-journal21:40:541
28283995741,10cyclictest276-21systemd-journal21:40:541
28283995740,9cyclictest0-21swapper/100:15:281
28283995740,11cyclictest179-21pr/ttyS023:26:381
28283995739,11cyclictest0-21swapper/119:06:041
2828399572,41cyclictest1554-21apt-get19:20:581
2828399572,41cyclictest1554-21apt-get19:20:581
28283995713,15cyclictest2739-21kworker/1:019:45:311
28283995713,15cyclictest2739-21kworker/1:019:45:311
2829199569,24cyclictest141rcu_preempt19:30:282
2829199565,15cyclictest141rcu_preempt23:40:482
28291995640,10cyclictest3960-21unixbench_multi23:51:052
28291995639,10cyclictest0-21swapper/222:25:292
28291995636,12cyclictest276-21systemd-journal21:06:512
2829199563,31cyclictest0-21swapper/219:15:562
2829199563,19cyclictest23278-21ntp_states00:26:002
28283995640,8cyclictest0-21swapper/123:56:071
28283995640,8cyclictest0-21swapper/123:56:071
28283995640,8cyclictest0-21swapper/119:26:061
28283995637,10cyclictest0-21swapper/120:56:031
28283995633,12cyclictest0-21swapper/123:41:021
28283995632,14cyclictest276-21systemd-journal21:35:551
28283995628,11cyclictest276-21systemd-journal20:20:501
28283995628,11cyclictest276-21systemd-journal20:20:501
28291995538,11cyclictest15512-21runrttasks23:16:362
28291995537,12cyclictest25822-21sh00:30:532
28291995530,15cyclictest0-21swapper/221:50:492
28291995529,17cyclictest347-21in:imklog19:50:432
28291995529,17cyclictest347-21in:imklog19:50:432
28291995527,20cyclictest0-21swapper/223:05:322
28291995527,20cyclictest0-21swapper/223:05:322
28283995538,11cyclictest116-21pr/ttymxc022:24:421
28283995529,10cyclictest116-21pr/ttymxc020:25:451
28283995524,11cyclictest0-21swapper/119:15:311
28283995510,13cyclictest25-21ksoftirqd/100:30:301
2829199545,20cyclictest141rcu_preempt23:11:022
28291995438,10cyclictest8390-21sh00:00:282
28291995438,10cyclictest8390-21sh00:00:282
28291995438,10cyclictest1325-21unixbench_singl22:51:042
28291995434,12cyclictest0-21swapper/220:50:292
28291995431,15cyclictest19672-21latency_hist00:20:312
28291995422,13cyclictest141rcu_preempt23:24:532
28283995434,12cyclictest0-21swapper/122:45:511
28283995431,8cyclictest141rcu_preempt19:55:541
28283995428,7cyclictest141rcu_preempt23:05:561
28291995331,16cyclictest0-21swapper/223:35:292
28291995313,17cyclictest141rcu_preempt22:55:502
28283995337,9cyclictest0-21swapper/119:36:061
2828399533,12cyclictest26780-21grep20:36:031
28291995233,9cyclictest0-21swapper/222:00:322
28291995228,9cyclictest8696-21rm22:09:392
28291995226,13cyclictest0-21swapper/223:30:312
28283995236,10cyclictest0-21swapper/122:15:541
28283995235,8cyclictest0-21swapper/100:16:021
28283995235,11cyclictest116-21pr/ttymxc022:35:511
28283995230,12cyclictest347-21in:imklog20:07:071
28283995221,12cyclictest141rcu_preempt23:35:491
28291995126,17cyclictest7826-21ssh21:12:132
28291995126,13cyclictest0-21swapper/220:25:502
2828399514,19cyclictest14587-21cpuspeed20:00:451
2828399514,19cyclictest14587-21cpuspeed20:00:451
2828399513,42cyclictest12447-21kworker/u8:421:21:391
28283995134,10cyclictest276-21systemd-journal21:50:501
28283995132,12cyclictest276-21systemd-journal20:15:521
28283995132,12cyclictest26470-21unixbench_singl00:31:041
28283995129,11cyclictest0-21swapper/123:20:551
2829199506,30cyclictest244-21jbd2/mmcblk2p2-19:21:092
2829199506,30cyclictest244-21jbd2/mmcblk2p2-19:21:092
97712490,6chrt0-21swapper/119:45:471
2829199494,16cyclictest141rcu_preempt19:10:512
28291994932,11cyclictest25159-21unixbench_multi20:31:062
28291994932,11cyclictest25159-21unixbench_multi20:31:062
28291994928,12cyclictest0-21swapper/219:50:282
2829199492,22cyclictest26951-21/usr/sbin/munin21:45:572
2829199492,22cyclictest26951-21/usr/sbin/munin21:45:572
2829199491,19cyclictest141rcu_preempt21:00:312
2828399499,11cyclictest179-21pr/ttyS023:00:501
2828399499,11cyclictest179-21pr/ttyS023:00:501
28283994935,7cyclictest0-21swapper/120:50:311
2828399493,22cyclictest25782-21latency_hist20:35:311
2828399493,22cyclictest25782-21latency_hist20:35:311
28291994833,10cyclictest7315-21unixbench_multi19:36:052
28291994832,9cyclictest0-21swapper/220:40:502
2829199483,17cyclictest15871-21modprobe20:05:292
2829199483,17cyclictest15871-21modprobe20:05:292
2829199482,7cyclictest141rcu_preempt21:41:102
2829199482,7cyclictest141rcu_preempt21:41:102
28291994826,10cyclictest0-21swapper/219:41:032
28291994826,10cyclictest0-21swapper/219:41:032
28291994825,12cyclictest0-21swapper/220:15:542
28291994527,8cyclictest0-21swapper/220:35:522
28283994529,10cyclictest179-21pr/ttyS019:30:511
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional