You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-27 - 07:07
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Sat Jul 27, 2024 00:44:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
69252990,4chrt12352-21diskmemload22:42:553
69252990,4chrt12352-21diskmemload22:42:553
512899973,64cyclictest18764-21/usr/sbin/munin23:56:482
145302930,4chrt14529-21ssh22:02:070
512899923,61cyclictest23354-21apt-get23:11:432
5139999025,55cyclictest7241-21apt-get00:31:443
248522900,9sleep0161rcuc/021:26:330
5128998850,13cyclictest141rcu_preempt20:01:332
5128998850,13cyclictest141rcu_preempt20:01:332
5139998733,38cyclictest12352-21diskmemload21:07:013
5116998770,11cyclictest26980-21munin-plugin-st20:11:300
57462850,8sleep30-21swapper/323:35:023
5128998359,16cyclictest4257-21munin-plugin-st20:41:302
37862819,31sleep20-21swapper/219:01:562
512899803,53cyclictest20445-21ssh21:18:052
512899803,53cyclictest20445-21ssh21:18:052
512899792,50cyclictest30307-21runrttasks21:35:222
512899792,50cyclictest30307-21runrttasks21:35:222
5139997840,30cyclictest41-21ksoftirqd/319:16:533
5139997840,30cyclictest41-21ksoftirqd/319:16:533
512899785,45cyclictest0-21swapper/222:21:062
359727811,41sleep33598-21munin-node19:01:403
5139997737,8cyclictest141rcu_preempt23:17:363
494027711,37sleep14974-21munin-run19:06:301
5139997648,19cyclictest7522-21latency_hist20:51:323
5128997657,12cyclictest7716-21unixbench_singl23:37:052
5128997657,12cyclictest7716-21unixbench_singl23:37:052
512899763,66cyclictest23180-21grep21:22:252
512899752,48cyclictest10975-21ssh22:50:472
5139997438,16cyclictest141rcu_preempt21:13:353
512899743,42cyclictest21502-21users19:52:112
511699742,42cyclictest4536-21timerwakeupswit23:32:030
5116997410,39cyclictest12-21ksoftirqd/000:17:060
284822740,7sleep10-21swapper/121:31:551
284822740,7sleep10-21swapper/121:31:551
5128997355,11cyclictest20755-21/usr/sbin/munin23:06:452
512899733,59cyclictest29197-21fschecks_time22:26:512
512899733,53cyclictest416-21snmpd22:37:152
512899732,62cyclictest17694-21timerwakeupswit22:07:032
512899732,62cyclictest11511-21rm22:51:352
5123997351,13cyclictest14769-21cstates22:56:471
5123997350,13cyclictest9433-21fschecks_time19:16:521
5123997350,13cyclictest9433-21fschecks_time19:16:521
5139997254,10cyclictest5106-21cyclictest20:36:313
5139997224,24cyclictest141rcu_preempt19:12:093
512899723,62cyclictest31246-21ssh23:22:362
5128997229,34cyclictest0-21swapper/221:41:312
5139997121,28cyclictest141rcu_preempt21:02:083
5139997121,28cyclictest141rcu_preempt21:02:083
512899719,56cyclictest0-21swapper/200:07:072
512899713,59cyclictest31081-21uname00:16:532
512899712,60cyclictest6113-21/usr/sbin/munin22:41:582
512899712,60cyclictest6113-21/usr/sbin/munin22:41:582
512899712,60cyclictest16658-21latency_hist22:06:332
512899711,44cyclictest8967-21ssh00:34:252
5139997035,10cyclictest141rcu_preempt22:11:523
5139997030,12cyclictest141rcu_preempt19:52:063
512899704,56cyclictest11264-21fschecks_time21:01:532
512899704,56cyclictest11264-21fschecks_time21:01:532
512899703,59cyclictest9851-21df_inode23:41:482
512899703,59cyclictest5510-21ssh00:27:522
512899703,58cyclictest1596-21unixbench_multi23:27:092
512899703,42cyclictest0-21swapper/222:13:182
5123997053,11cyclictest26247-21unixbench_singl21:27:301
5123997048,14cyclictest4356-21idleruntime-cro21:46:301
5123997045,17cyclictest8357-21taskset00:32:361
5123997032,9cyclictest141rcu_preempt22:36:541
5116997033,15cyclictest141rcu_preempt20:26:490
5139996952,10cyclictest29890-21unixbench_multi22:27:053
5139996948,8cyclictest141rcu_preempt23:02:373
5139996928,19cyclictest141rcu_preempt19:06:553
5139996910,26cyclictest41-21ksoftirqd/323:51:583
5128996952,10cyclictest183-21pr/ttyS022:23:542
512899694,57cyclictest19279-21latency19:46:542
5128996928,34cyclictest12352-21diskmemload21:51:462
512899692,61cyclictest0-21swapper/223:03:152
512899692,59cyclictest5915-21ssh21:47:302
5116996936,24cyclictest17317-21cut23:01:320
5116996925,23cyclictest141rcu_preempt22:24:580
5116996910,20cyclictest12-21ksoftirqd/023:02:540
5139996830,14cyclictest141rcu_preempt20:21:523
5128996849,13cyclictest362-21in:imuxsock21:16:302
512899684,58cyclictest2153-21kworker/u8:423:51:302
512899683,57cyclictest10116-21cat20:57:052
512899683,36cyclictest925-21ssh22:33:072
5128996831,27cyclictest0-21swapper/200:21:502
5128996831,27cyclictest0-21swapper/200:21:502
512899681,45cyclictest24008-21ssh00:05:222
5116996841,13cyclictest141rcu_preempt23:15:120
513999676,14cyclictest12632-21unixbench_singl22:52:093
5139996741,16cyclictest27399-21munin-plugin-st00:11:293
5139996733,12cyclictest141rcu_preempt22:10:083
5139996733,12cyclictest141rcu_preempt00:26:323
5139996733,12cyclictest141rcu_preempt00:26:323
5139996729,19cyclictest363-21in:imklog23:08:323
512899675,55cyclictest0-21swapper/223:17:222
5128996749,11cyclictest25404-21cut21:27:052
512899673,56cyclictest1611-21/usr/sbin/munin21:41:452
512899673,56cyclictest12538-21systemd-journal23:56:292
512899673,51cyclictest27239-21apt-get20:11:422
5139996614,10cyclictest0-21swapper/319:26:423
512899668,50cyclictest0-21swapper/222:01:342
512899664,54cyclictest0-21swapper/223:31:392
512899663,54cyclictest10112-21tail19:17:072
512899663,54cyclictest10112-21tail19:17:072
512899662,55cyclictest400-21runrttasks23:01:182
5123996634,18cyclictest141rcu_preempt22:06:551
5123996627,19cyclictest141rcu_preempt00:24:461
5123996627,19cyclictest141rcu_preempt00:24:461
5116996625,36cyclictest131ktimers/021:12:480
5139996538,16cyclictest639-21cpuspeed23:26:473
5139996535,9cyclictest141rcu_preempt21:41:323
5139996526,15cyclictest141rcu_preempt22:22:093
5139996516,25cyclictest141rcu_preempt19:32:033
512899653,54cyclictest27704-21apt-get00:11:412
5123996548,11cyclictest1-21systemd19:11:321
5116996543,15cyclictest14283-21idleruntime-cro22:56:300
5116996538,7cyclictest141rcu_preempt21:54:520
5116996533,16cyclictest141rcu_preempt21:07:270
513999647,17cyclictest141rcu_preempt20:46:313
5139996438,7cyclictest0-21swapper/322:20:143
5139996436,7cyclictest141rcu_preempt22:02:033
513999643,30cyclictest9847-21ssh23:41:483
5139996428,21cyclictest52-21kcompactd019:22:073
5139996414,30cyclictest141rcu_preempt20:51:593
5139996412,17cyclictest12249-21kworker/3:021:25:263
512899649,48cyclictest0-21swapper/219:11:372
5123996431,17cyclictest141rcu_preempt22:55:231
511699643,19cyclictest6954-21fschecks_time23:36:500
511699643,19cyclictest6954-21fschecks_time23:36:500
5116996430,19cyclictest141rcu_preempt19:46:450
5116996428,9cyclictest141rcu_preempt23:57:310
5139996343,12cyclictest9351-21idleruntime-cro23:41:293
5139996343,12cyclictest9351-21idleruntime-cro23:41:293
513999633,41cyclictest25310-21kworker/3:222:39:543
5139996328,18cyclictest141rcu_preempt19:47:043
5139996328,15cyclictest141rcu_preempt23:12:113
5139996320,23cyclictest141rcu_preempt23:50:173
5139996318,18cyclictest141rcu_preempt00:29:073
512899637,48cyclictest31992-21cut20:26:322
512899634,54cyclictest0-21swapper/220:01:412
512899634,54cyclictest0-21swapper/220:01:412
512899634,51cyclictest2675-21systemd-run20:36:322
512899634,50cyclictest12712-21/usr/sbin/munin19:26:522
512899633,52cyclictest6629-21latency_hist19:11:342
512899632,52cyclictest8155-21latency20:51:562
512899632,51cyclictest16162-21ntpq19:36:582
512899632,51cyclictest16162-21ntpq19:36:582
5123996336,15cyclictest141rcu_preempt19:58:241
5123996336,15cyclictest141rcu_preempt19:58:241
5123996331,16cyclictest141rcu_preempt00:18:281
5116996346,10cyclictest29910-21unixbench_multi22:27:060
5116996344,13cyclictest8242-21idleruntime-cro22:46:300
5116996344,13cyclictest8242-21idleruntime-cro22:46:300
5116996335,14cyclictest141rcu_preempt22:31:410
5116996334,15cyclictest141rcu_preempt00:35:200
5116996333,18cyclictest141rcu_preempt21:27:580
5116996332,7cyclictest141rcu_preempt00:25:160
5116996332,7cyclictest141rcu_preempt00:25:160
5116996325,21cyclictest141rcu_preempt21:51:080
5116996321,7cyclictest141rcu_preempt21:56:580
5116996312,25cyclictest12-21ksoftirqd/019:21:510
513999624,42cyclictest25622-21apt-get20:06:473
5139996236,8cyclictest141rcu_preempt20:56:533
5139996230,12cyclictest141rcu_preempt21:52:063
5139996221,20cyclictest141rcu_preempt21:48:493
5139996214,25cyclictest0-21swapper/300:06:163
5139996213,9cyclictest141rcu_preempt00:16:323
512899625,50cyclictest12902-21apt-get21:06:462
512899624,50cyclictest400-21runrttasks20:30:132
512899623,50cyclictest11739-21unixbench_multi19:22:052
512899622,50cyclictest28881-21apt-get20:16:412
512899622,50cyclictest28881-21apt-get20:16:412
5123996246,8cyclictest0-21swapper/119:36:311
5123996237,13cyclictest141rcu_preempt21:13:001
5123996231,11cyclictest141rcu_preempt21:57:581
5123996225,9cyclictest141rcu_preempt22:30:501
5123996211,13cyclictest241ktimers/123:31:331
5116996238,13cyclictest120-21pr/ttymxc020:16:330
5116996234,7cyclictest141rcu_preempt23:50:220
5116996234,7cyclictest141rcu_preempt21:42:000
5116996234,15cyclictest141rcu_preempt23:46:260
5116996233,8cyclictest141rcu_preempt22:40:500
5116996233,14cyclictest141rcu_preempt21:16:540
5116996233,14cyclictest141rcu_preempt21:16:540
513999614,11cyclictest19971-21tail21:17:033
513999614,11cyclictest19971-21tail21:17:033
5139996132,19cyclictest19859-21unixbench_singl23:57:073
5139996131,17cyclictest141rcu_preempt20:26:483
5139996114,19cyclictest141rcu_preempt21:46:133
512899613,49cyclictest6939-21sed20:47:042
512899613,48cyclictest4707-21fschecks_count20:41:512
5123996145,10cyclictest0-21swapper/100:27:021
5123996143,11cyclictest28612-21kworker/u8:320:16:301
5123996138,15cyclictest231rcuc/121:10:101
5123996133,7cyclictest141rcu_preempt23:48:251
5123996132,14cyclictest52-21kcompactd000:04:121
5123996132,13cyclictest141rcu_preempt21:54:091
5123996131,16cyclictest141rcu_preempt23:16:191
5123996130,11cyclictest141rcu_preempt22:16:131
5123996128,17cyclictest10131-21ssh22:48:271
5123996127,16cyclictest141rcu_preempt20:31:301
5116996146,10cyclictest131ktimers/023:26:540
5116996142,7cyclictest141rcu_preempt19:37:030
5116996142,7cyclictest141rcu_preempt19:37:030
5116996138,14cyclictest6860-21latency_hist19:11:330
5116996132,14cyclictest141rcu_preempt20:41:490
5116996130,12cyclictest141rcu_preempt00:31:100
5116996129,16cyclictest141rcu_preempt00:15:540
5116996128,16cyclictest141rcu_preempt21:05:160
5116996128,16cyclictest141rcu_preempt21:05:160
5116996121,28cyclictest141rcu_preempt19:18:090
5116996121,28cyclictest141rcu_preempt19:18:090
513999609,19cyclictest41-21ksoftirqd/319:56:573
513999609,19cyclictest41-21ksoftirqd/319:56:573
513999608,36cyclictest0-21swapper/320:12:073
513999608,26cyclictest24991-21kworker/3:220:21:303
513999608,26cyclictest24991-21kworker/3:220:21:303
5139996034,7cyclictest141rcu_preempt22:49:143
5139996033,9cyclictest141rcu_preempt20:06:323
5139996033,9cyclictest141rcu_preempt20:06:323
5139996032,11cyclictest141rcu_preempt19:46:333
512899604,46cyclictest400-21runrttasks20:08:362
5123996036,12cyclictest141rcu_preempt23:23:121
5123996033,13cyclictest141rcu_preempt22:24:121
5123996029,15cyclictest52-21kcompactd000:11:161
5123996010,20cyclictest231rcuc/121:02:041
5123996010,20cyclictest231rcuc/121:02:041
5116996043,11cyclictest131ktimers/019:51:540
5116996039,15cyclictest30215-21modprobe20:21:300
5116996039,15cyclictest30215-21modprobe20:21:300
5116996034,7cyclictest141rcu_preempt22:47:210
5116996033,8cyclictest141rcu_preempt21:36:300
5116996033,8cyclictest141rcu_preempt21:36:300
5116996033,8cyclictest141rcu_preempt20:31:500
5116996033,15cyclictest141rcu_preempt21:39:500
5139995927,15cyclictest41-21ksoftirqd/321:59:133
5139995924,10cyclictest141rcu_preempt22:56:453
512899593,46cyclictest17568-21fschecks_time19:41:522
512899592,48cyclictest14659-21/usr/sbin/munin19:32:032
5123995943,8cyclictest0-21swapper/123:06:451
5123995939,7cyclictest141rcu_preempt23:57:211
5123995938,11cyclictest0-21swapper/122:32:091
5123995935,6cyclictest141rcu_preempt23:34:441
5123995932,7cyclictest141rcu_preempt21:50:041
5123995929,8cyclictest141rcu_preempt19:38:051
5123995929,8cyclictest141rcu_preempt19:38:051
5123995917,22cyclictest241ktimers/122:01:571
5123995916,9cyclictest0-21swapper/119:12:021
5123995914,19cyclictest25-21ksoftirqd/123:20:541
5123995912,24cyclictest241ktimers/121:40:071
5116995930,16cyclictest141rcu_preempt23:56:140
5116995927,18cyclictest31638-21kworker/u8:123:18:070
5116995926,27cyclictest0-21swapper/000:02:020
5116995925,7cyclictest141rcu_preempt20:54:540
5116995924,21cyclictest141rcu_preempt20:36:490
5139995820,13cyclictest141rcu_preempt21:26:403
5139995814,11cyclictest141rcu_preempt00:16:533
5123995842,11cyclictest5106-21cyclictest23:51:331
5123995841,11cyclictest0-21swapper/120:06:301
5123995841,11cyclictest0-21swapper/120:06:301
511699589,16cyclictest120-21pr/ttymxc023:22:430
511699588,19cyclictest12-21ksoftirqd/020:22:020
5116995838,7cyclictest141rcu_preempt22:11:360
5116995837,12cyclictest0-21swapper/019:06:340
511699583,26cyclictest416-21snmpd20:01:520
511699583,26cyclictest416-21snmpd20:01:520
5116995831,7cyclictest141rcu_preempt23:07:140
5116995831,14cyclictest141rcu_preempt00:10:370
5116995829,17cyclictest141rcu_preempt19:31:100
5116995826,19cyclictest141rcu_preempt22:07:100
5139995721,13cyclictest0-21swapper/323:24:023
5139995717,12cyclictest141rcu_preempt19:36:543
5139995717,12cyclictest141rcu_preempt19:36:543
5139995715,20cyclictest141rcu_preempt22:32:073
5123995729,13cyclictest52-21kcompactd020:51:561
5123995727,16cyclictest141rcu_preempt22:20:371
5123995715,21cyclictest241ktimers/123:01:491
5123995714,20cyclictest241ktimers/123:38:001
5123995714,20cyclictest241ktimers/123:38:001
5123995712,21cyclictest25-21ksoftirqd/122:44:581
5123995712,21cyclictest25-21ksoftirqd/122:44:581
5116995742,9cyclictest6651-21ntp_offset20:46:570
5116995732,7cyclictest141rcu_preempt22:18:250
5116995718,27cyclictest141rcu_preempt19:36:160
5123995636,7cyclictest141rcu_preempt19:21:571
5123995631,7cyclictest141rcu_preempt23:42:121
5123995631,7cyclictest141rcu_preempt21:21:421
5123995622,8cyclictest141rcu_preempt20:41:431
5123995622,15cyclictest141rcu_preempt00:12:481
5123995622,13cyclictest141rcu_preempt21:18:161
5123995622,13cyclictest141rcu_preempt21:18:161
5116995631,12cyclictest141rcu_preempt20:59:450
5139995529,8cyclictest18560-21kworker/u8:021:34:093
5139995529,8cyclictest18560-21kworker/u8:021:34:093
5123995523,15cyclictest141rcu_preempt20:46:571
5123995523,10cyclictest19361-21kworker/u8:119:56:311
5123995522,25cyclictest231rcuc/120:32:021
511699553,20cyclictest12-21ksoftirqd/019:58:110
511699553,20cyclictest12-21ksoftirqd/019:58:110
5116995528,7cyclictest141rcu_preempt19:11:590
5139995436,12cyclictest363-21in:imklog20:37:063
5123995431,7cyclictest141rcu_preempt20:36:561
5123995424,15cyclictest52-21kcompactd019:47:031
5123995337,9cyclictest241ktimers/120:56:451
5123995327,7cyclictest141rcu_preempt20:17:321
5123995327,7cyclictest141rcu_preempt20:17:321
5123995327,11cyclictest141rcu_preempt19:29:411
5123995225,8cyclictest0-21swapper/120:06:341
5123995119,15cyclictest141rcu_preempt19:41:411
5116995133,6cyclictest141rcu_preempt19:44:370
5123995019,9cyclictest0-21swapper/120:21:531
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional