You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-06-15 - 13:03
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7.osadl.org (updated Sat Jun 15, 2024 00:44:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4246213411,91sleep10-21swapper/119:04:301
4246213411,91sleep10-21swapper/119:04:301
5731991092,80cyclictest28835-21apt-get00:24:373
5731991092,80cyclictest28835-21apt-get00:24:373
5720991025,38cyclictest9602-21/usr/sbin/munin19:19:580
5720991025,38cyclictest9602-21/usr/sbin/munin19:19:580
57209910228,15cyclictest28816-21acpi00:24:270
57209910228,15cyclictest28816-21acpi00:24:270
572099987,30cyclictest0-21swapper/000:30:110
5720999826,17cyclictest18612-21proc_pri22:14:530
5720999826,17cyclictest18612-21proc_pri22:14:530
5720999825,14cyclictest29948-21latency23:29:450
5720999710,33cyclictest0-21swapper/021:59:200
5720999629,13cyclictest0-21swapper/022:24:530
5720999629,13cyclictest0-21swapper/022:24:530
5720999626,14cyclictest10979-21diskmemload00:34:550
572099954,20cyclictest760-21switchtime21:44:560
572099953,36cyclictest32200-21latency_hist21:44:220
5720999520,10cyclictest0-21swapper/022:54:310
572099933,19cyclictest23594-21fschecks_count21:29:420
5720999233,11cyclictest11644-21kworker/u8:122:59:500
5720999219,13cyclictest0-21swapper/023:24:190
572099914,22cyclictest17372-21sort00:04:210
5720999125,15cyclictest0-21swapper/020:54:480
5720999125,15cyclictest0-21swapper/020:54:480
5720999120,13cyclictest0-21swapper/019:39:200
5720999120,13cyclictest0-21swapper/019:39:200
572099904,29cyclictest0-21swapper/023:24:300
5720999030,8cyclictest0-21swapper/021:09:570
5720999010,31cyclictest0-21swapper/022:23:570
572099893,34cyclictest14425-21ssh23:59:200
572099887,27cyclictest0-21swapper/023:39:570
572099883,12cyclictest2650-21awk23:39:200
572099874,15cyclictest337-21dbus-daemon19:59:580
572099862,16cyclictest9346-21ssh23:49:470
5720998620,18cyclictest0-21swapper/000:24:200
572099853,32cyclictest0-21swapper/021:50:390
445128511,40sleep34528-21fschecks_time19:04:573
445128511,40sleep34528-21fschecks_time19:04:573
230002840,6sleep30-21swapper/323:18:263
230002840,6sleep30-21swapper/323:18:263
5727998255,16cyclictest20855-21apt-key22:19:231
5727998255,16cyclictest20855-21apt-key22:19:231
572099823,13cyclictest337-21dbus-daemon21:09:210
5720998225,10cyclictest0-21swapper/020:24:350
261352820,7sleep10-21swapper/120:14:431
5731998152,15cyclictest0-21swapper/322:49:423
5727998132,34cyclictest28920-21sh00:24:451
5727998132,34cyclictest28920-21sh00:24:451
53892796,56sleep00-21swapper/019:06:450
53892796,56sleep00-21swapper/019:06:450
5720997856,15cyclictest7276-21latency_hist19:14:240
5729997638,31cyclictest0-21swapper/200:24:212
5729997425,43cyclictest362-21in:imuxsock19:44:192
5729997425,43cyclictest362-21in:imuxsock19:44:192
5720997448,15cyclictest8982-21runrttasks21:00:010
573199723,41cyclictest31857-21modprobe00:29:333
572999723,60cyclictest11984-21ls22:04:222
5727997244,17cyclictest13231-21munin-plugin-st19:34:191
5727997244,17cyclictest13231-21munin-plugin-st19:34:191
5727997222,26cyclictest0-21swapper/120:09:341
5727997151,13cyclictest16482-21idleruntime-cro19:44:191
5727997151,13cyclictest16482-21idleruntime-cro19:44:191
5727997145,13cyclictest0-21swapper/120:30:011
5720997152,12cyclictest0-21swapper/022:04:220
5729997054,9cyclictest25546-21idleruntime-cro20:14:192
572999703,55cyclictest7520-21munin-node19:14:272
572999703,55cyclictest7520-21munin-node19:14:272
572999702,59cyclictest17856-21ssh23:09:362
5727997049,13cyclictest0-21swapper/121:19:221
5720997049,14cyclictest17631-21kworker/u8:222:40:020
572999695,57cyclictest0-21swapper/200:04:362
5729996933,29cyclictest0-21swapper/222:44:562
572799698,40cyclictest276-21systemd-journal00:29:331
5731996852,9cyclictest7710-21modprobe20:59:203
5731996852,9cyclictest7710-21modprobe20:59:203
572999683,55cyclictest14414-21cpuspeed21:14:372
572999683,53cyclictest9561-21grep21:04:222
572999683,52cyclictest578-21fschecks_count22:39:452
572999683,52cyclictest23096-21grep00:14:222
572999682,59cyclictest3961-21unixbench_multi23:39:592
572999682,59cyclictest12692-21runrttasks22:05:042
572999682,57cyclictest32409-21apt-get21:44:322
5729996822,37cyclictest0-21swapper/223:39:232
572999673,55cyclictest1222-21cron20:39:192
572999673,54cyclictest4742-21apt-get20:49:352
572999672,57cyclictest6328-21tune2fs22:49:422
572999672,56cyclictest26607-21ssh23:24:302
572999672,56cyclictest13232-21munin-plugin-st19:34:192
572999672,56cyclictest13232-21munin-plugin-st19:34:192
572999672,14cyclictest23139-21wc20:04:562
5720996745,15cyclictest276-21systemd-journal19:24:470
5720996745,15cyclictest276-21systemd-journal19:24:470
5720996744,12cyclictest0-21swapper/021:34:430
5720996741,15cyclictest0-21swapper/021:19:200
5720996735,25cyclictest0-21swapper/000:14:200
5731996648,11cyclictest0-21swapper/323:59:203
572999664,54cyclictest9820-21grep19:19:422
572999664,54cyclictest9820-21grep19:19:422
572999663,55cyclictest31577-21latency_hist00:29:212
572999663,55cyclictest31577-21latency_hist00:29:212
572999663,50cyclictest27419-21latency_hist20:19:232
572999662,55cyclictest21245-21ssh22:19:542
572999662,55cyclictest17377-21ssh21:19:312
572999662,54cyclictest5672-21grep21:54:192
572999662,16cyclictest29200-21latency_hist23:29:232
572999662,13cyclictest12580-21awk23:54:542
572999662,13cyclictest12422-21/usr/sbin/munin21:10:002
5729996619,39cyclictest0-21swapper/220:54:562
5729996619,39cyclictest0-21swapper/220:54:562
573199652,54cyclictest20555-21apt-get22:19:333
572999653,55cyclictest26366-21/usr/sbin/munin21:34:592
572999653,55cyclictest14412-21grep19:35:012
572999653,55cyclictest14412-21grep19:35:012
572999653,55cyclictest0-21swapper/220:39:312
572999653,54cyclictest11728-21systemd22:59:212
572999653,53cyclictest24436-21grep21:29:572
572999653,53cyclictest15096-21ssh22:09:352
572999653,38cyclictest0-21swapper/221:55:002
572999652,57cyclictest5711-21cyclictest19:09:242
572999652,57cyclictest5711-21cyclictest19:09:242
572999652,55cyclictest10585-21grep21:05:012
572999652,54cyclictest22055-21cut20:04:202
572999652,12cyclictest29415-21apt-get21:39:342
572999652,11cyclictest9852-21ssh23:49:592
572999652,11cyclictest12248-21fschecks_count22:59:432
5727996548,11cyclictest1-21systemd22:34:201
5731996450,7cyclictest0-21swapper/321:29:423
5731996445,13cyclictest23098-21unin-run21:29:213
572999644,53cyclictest0-21swapper/219:09:322
572999644,53cyclictest0-21swapper/219:09:322
572999644,52cyclictest6984-21/usr/sbin/munin23:44:572
5729996445,13cyclictest20982-21latency_hist22:19:232
5729996445,13cyclictest20982-21latency_hist22:19:232
572999643,53cyclictest26939-21sort22:29:332
572999643,53cyclictest15029-21cstates23:04:362
572999643,52cyclictest24108-21/usr/sbin/munin22:24:362
572999643,52cyclictest24108-21/usr/sbin/munin22:24:362
572999643,51cyclictest11469-21fschecks_time19:24:522
572999643,51cyclictest11469-21fschecks_time19:24:522
572999643,14cyclictest27673-21tune2fs20:19:402
572999643,14cyclictest27673-21tune2fs20:19:402
5729996430,26cyclictest0-21swapper/223:19:582
572999642,9cyclictest29811-21ssh22:34:322
572999642,9cyclictest18023-21ps19:49:562
572999642,55cyclictest20612-21grep21:24:412
572999642,53cyclictest1976-21cat00:34:222
572999642,12cyclictest29282-21tune2fs20:24:402
5731996347,10cyclictest7521-21latency_hist19:14:243
5731996346,10cyclictest29082-21systemctl21:39:203
5731996341,12cyclictest0-21swapper/300:05:003
5731996330,23cyclictest20479-21apt-get00:09:373
572999633,52cyclictest3390-21sort20:44:432
572999633,52cyclictest20711-21/usr/sbin/munin23:14:452
572999633,52cyclictest20711-21/usr/sbin/munin23:14:452
572999632,6cyclictest0-21swapper/200:14:312
572999632,54cyclictest14906-21wc23:59:362
572999632,52cyclictest19384-21/usr/sbin/munin19:54:432
5727996322,19cyclictest141rcu_preempt21:35:391
5727996322,13cyclictest21114-21apt-get22:19:331
5731996247,8cyclictest0-21swapper/321:39:583
5731996241,12cyclictest11909-21latency_hist22:04:223
572999622,14cyclictest2332-21fschecks_time00:34:402
572999622,11cyclictest31640-21sed20:30:012
5727996245,11cyclictest32166-21latency_hist23:34:211
5727996242,13cyclictest420-21lldpd20:04:531
5727996240,12cyclictest0-21swapper/121:09:201
5727996226,11cyclictest141rcu_preempt23:16:421
5727996226,11cyclictest141rcu_preempt23:16:421
5727996213,30cyclictest0-21swapper/121:13:421
5720996238,19cyclictest12-21ksoftirqd/023:09:410
5720996226,12cyclictest141rcu_preempt20:34:410
572999612,10cyclictest337-21dbus-daemon19:48:002
572799619,30cyclictest141rcu_preempt23:54:191
572799618,25cyclictest12199-21apt-get22:04:381
5727996141,13cyclictest30443-21idleruntime-cro20:29:191
5727996138,14cyclictest0-21swapper/122:44:571
5727996125,18cyclictest141rcu_preempt23:54:371
5727996117,36cyclictest241ktimers/121:58:451
5720996141,12cyclictest0-21swapper/019:19:200
5720996141,12cyclictest0-21swapper/019:19:200
5720996140,10cyclictest0-21swapper/023:09:220
5731996044,9cyclictest0-21swapper/321:19:203
5731996033,12cyclictest141rcu_preempt23:29:423
573199603,22cyclictest14904-21latency_hist19:39:223
573199603,22cyclictest14904-21latency_hist19:39:223
573199602,18cyclictest28462-21apt-config00:24:213
5727996045,9cyclictest24284-21unixbench_singl00:14:561
5727996039,15cyclictest28309-21idleruntime-cro00:24:181
5727996034,14cyclictest0-21swapper/122:54:221
5727996033,16cyclictest0-21swapper/122:34:411
5727996027,13cyclictest141rcu_preempt21:40:241
5727996024,11cyclictest141rcu_preempt00:34:471
5727996010,24cyclictest25-21ksoftirqd/122:40:141
5720996041,11cyclictest0-21swapper/023:14:590
5720996041,11cyclictest0-21swapper/023:14:590
5731995945,8cyclictest24244-21unixbench_multi00:14:553
5731995943,9cyclictest22573-21fschecks_time20:04:423
5731995942,10cyclictest13047-21unixbench_multi23:00:003
5731995940,11cyclictest1-21systemd22:44:203
573199593,32cyclictest24186-21apt-get20:09:333
5731995922,10cyclictest0-21swapper/323:59:413
5727995944,10cyclictest231rcuc/123:20:041
5727995943,10cyclictest29051-21kworker/u8:121:49:201
5727995942,8cyclictest25424-21kworker/u8:020:39:221
5727995942,10cyclictest14849-21sendmail19:39:201
5727995942,10cyclictest14849-21sendmail19:39:201
5727995929,7cyclictest141rcu_preempt20:44:471
5727995925,7cyclictest141rcu_preempt21:34:031
5727995923,16cyclictest141rcu_preempt00:14:021
5720995945,7cyclictest0-21swapper/022:04:410
5720995941,12cyclictest8650-21kworker/u8:000:19:190
5720995941,12cyclictest6549-21processes19:10:000
5720995938,14cyclictest30028-21ssh22:34:410
572099593,46cyclictest6996-21ssh22:49:570
5731995842,10cyclictest12615-21systemd-run19:29:213
5731995842,10cyclictest12615-21systemd-run19:29:213
5731995838,9cyclictest0-21swapper/319:15:123
5731995838,9cyclictest0-21swapper/319:15:123
5731995835,13cyclictest3213-21fschecks_count21:49:413
5727995840,10cyclictest0-21swapper/122:10:021
5727995810,23cyclictest241ktimers/123:24:351
572099587,28cyclictest0-21swapper/020:09:200
572099585,20cyclictest3408-21fschecks_count22:44:400
5720995843,8cyclictest13212-21kworker/u8:119:54:200
5720995842,10cyclictest276-21systemd-journal20:49:300
5720995841,11cyclictest16509-21unin-run19:44:200
5720995841,11cyclictest16509-21unin-run19:44:200
572099583,44cyclictest18393-21grep21:19:580
5731995721,27cyclictest1496-21/usr/sbin/munin20:39:363
5731995721,25cyclictest27418-21cat20:19:233
5731995719,18cyclictest10152-21ssh23:50:293
5731995717,19cyclictest4160-21taskset00:39:163
5727995731,7cyclictest141rcu_preempt23:45:061
5727995731,13cyclictest0-21swapper/121:59:451
5727995726,14cyclictest141rcu_preempt22:59:301
573199569,16cyclictest10979-21diskmemload22:54:303
5731995629,16cyclictest0-21swapper/322:29:193
5731995629,16cyclictest0-21swapper/322:29:193
5731995628,8cyclictest10979-21diskmemload21:19:403
5727995640,9cyclictest0-21swapper/123:14:211
5727995632,14cyclictest120-21pr/ttymxc023:59:411
5727995628,15cyclictest141rcu_preempt22:26:271
5727995628,15cyclictest141rcu_preempt22:26:271
5727995628,15cyclictest141rcu_preempt21:49:391
5727995618,19cyclictest25-21ksoftirqd/120:59:081
5727995618,19cyclictest25-21ksoftirqd/120:59:081
5720995642,8cyclictest29051-21kworker/u8:122:30:180
5720995638,12cyclictest183-21pr/ttyS000:08:490
5720995632,13cyclictest0-21swapper/021:24:420
5731995540,9cyclictest6549-21runrttasks23:44:513
5731995538,11cyclictest10991-21idleruntime-cro21:09:203
5731995537,12cyclictest18900-21unixbench_singl23:10:023
5731995532,10cyclictest0-21swapper/322:05:173
5731995529,16cyclictest17726-21latency_hist23:09:233
5731995528,17cyclictest20678-21acpi19:59:323
5731995527,18cyclictest27349-21/usr/sbin/munin22:29:433
5731995523,22cyclictest5788-21latency_hist23:44:223
5727995532,12cyclictest0-21swapper/123:39:201
5727995524,7cyclictest141rcu_preempt22:56:041
5727995522,15cyclictest141rcu_preempt23:42:031
5720995540,10cyclictest0-21swapper/020:10:070
5720995525,12cyclictest12-21ksoftirqd/020:29:470
5720995518,11cyclictest141rcu_preempt20:44:430
5731995439,9cyclictest400-21runrttasks21:54:443
5731995432,13cyclictest0-21swapper/320:19:573
5731995432,13cyclictest0-21swapper/320:19:573
5731995431,15cyclictest0-21swapper/322:44:403
5731995423,19cyclictest29144-21/usr/sbin/munin20:24:353
5727995432,7cyclictest141rcu_preempt23:08:471
5727995429,7cyclictest141rcu_preempt20:02:351
5720995437,10cyclictest0-21swapper/019:54:400
5731995335,9cyclictest17946-21apt-get22:14:333
5731995335,9cyclictest17946-21apt-get22:14:333
573199533,31cyclictest0-21swapper/319:20:003
573199533,31cyclictest0-21swapper/319:20:003
5731995324,18cyclictest11871-21/usr/sbin/munin21:09:493
5731995324,17cyclictest15798-21/usr/sbin/munin19:40:063
5731995324,17cyclictest15798-21/usr/sbin/munin19:40:063
5731995323,19cyclictest32761-21/usr/sbin/munin20:34:453
5727995330,11cyclictest0-21swapper/119:14:241
5727995330,11cyclictest0-21swapper/119:14:241
5727995328,7cyclictest141rcu_preempt21:28:271
5727995328,17cyclictest0-21swapper/120:49:451
5720995331,13cyclictest15523-21kernelversion22:09:500
5720995323,21cyclictest0-21swapper/020:19:220
5731995237,8cyclictest0-21swapper/322:34:413
5731995231,11cyclictest0-21swapper/321:44:583
5731995229,12cyclictest0-21swapper/319:09:523
5731995223,19cyclictest3461-21/usr/sbin/munin20:44:443
573199522,24cyclictest9455-21latency_hist21:04:213
573199522,21cyclictest4931-21fschecks_count20:49:413
5727995228,7cyclictest141rcu_preempt19:15:241
5727995228,7cyclictest141rcu_preempt19:15:241
5727995226,8cyclictest141rcu_preempt00:05:261
5727995225,7cyclictest141rcu_preempt19:25:001
5727995225,7cyclictest141rcu_preempt19:25:001
5727995210,19cyclictest141rcu_preempt19:54:401
5720995236,9cyclictest276-21systemd-journal23:44:530
5731995124,17cyclictest32539-21/usr/sbin/munin23:34:353
5731995123,15cyclictest0-21swapper/319:49:223
572799518,20cyclictest2052-21ntp_states20:39:521
5727995128,13cyclictest0-21swapper/119:24:201
5727995128,13cyclictest0-21swapper/119:24:201
5727995124,15cyclictest141rcu_preempt21:22:361
5727995122,21cyclictest241ktimers/120:24:201
5727995122,21cyclictest241ktimers/120:24:201
5727995122,15cyclictest141rcu_preempt19:48:151
5727995116,17cyclictest0-21swapper/119:49:581
5731995030,13cyclictest10979-21diskmemload23:24:393
5731995024,17cyclictest15768-21/usr/sbin/munin22:09:573
5731995020,19cyclictest32053-21latency_hist20:34:223
5731995018,18cyclictest0-21swapper/323:19:493
572099503,40cyclictest416-21snmpd19:46:100
572099503,36cyclictest2861-21latency_hist20:44:220
5731994929,12cyclictest17434-21apt-get19:49:263
5731994924,17cyclictest19545-21/usr/sbin/munin19:54:493
5731994922,17cyclictest13245-21basename19:34:203
5731994922,17cyclictest13245-21basename19:34:203
5727994922,15cyclictest141rcu_preempt20:59:531
5720994935,9cyclictest120-21pr/ttymxc020:19:410
5720994935,9cyclictest120-21pr/ttymxc020:19:410
5720994727,12cyclictest0-21swapper/019:33:060
5720994727,12cyclictest0-21swapper/019:33:060
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional