You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-18 - 00:17
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Thu Jul 17, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
427218213,137sleep0506-21perl07:01:040
534213210,80sleep2575-21if_eth007:01:062
17199912148,14cyclictest0-21swapper/008:11:030
17199912148,14cyclictest0-21swapper/008:11:030
17199912030,31cyclictest0-21swapper/012:11:100
17199912030,31cyclictest0-21swapper/012:11:100
1719991128,53cyclictest3970-21apt-get08:51:350
1719999958,34cyclictest1196-21latency11:41:030
1719999958,34cyclictest1196-21latency11:41:030
23362970,7sleep20-21swapper/207:06:022
1729999162,20cyclictest23996-21sh09:31:062
1725999172,12cyclictest13569-21ssh09:13:341
1725999028,42cyclictest356-21runrttasks09:38:341
172599893,73cyclictest0-21swapper/110:45:491
172599893,73cyclictest0-21swapper/109:15:471
172599893,73cyclictest0-21swapper/109:15:471
172599879,63cyclictest20611-21apt-get09:26:071
1725998748,15cyclictest18835-21ssh10:20:531
1725998746,36cyclictest3050-21kworker/u18:2+events_unbound07:55:341
1719998772,8cyclictest0-21swapper/010:35:510
171999873,60cyclictest375-21snmpd12:05:510
1729998667,12cyclictest28237-21ssh12:26:122
1729998662,16cyclictest1708-21cyclictest11:05:562
1729998662,16cyclictest1708-21cyclictest11:05:562
1725998642,36cyclictest0-21swapper/112:11:071
1725998642,36cyclictest0-21swapper/112:11:071
1719998663,15cyclictest202-21systemd-journal09:59:050
1719998663,15cyclictest202-21systemd-journal09:59:050
1729998568,13cyclictest27564-21kworker/u18:3+events_unbound10:45:342
1729998556,19cyclictest5319-21idleruntime-cro07:15:352
1719998558,19cyclictest29575-21ntp_offset09:41:050
1729998448,13cyclictest181rcu_preempt12:00:362
1729998442,10cyclictest0-21swapper/207:26:112
1725998454,22cyclictest22356-21timerandwakeup12:16:131
1725998437,39cyclictest529-21cut12:35:361
172599842,77cyclictest0-21swapper/111:36:211
1719998459,17cyclictest21022-21latency_hist11:20:390
1719998422,12cyclictest0-21swapper/011:50:510
1729998359,17cyclictest18333-21latency_hist12:10:372
1729998356,21cyclictest26780-21fschecks_time08:20:562
1729998356,21cyclictest26780-21fschecks_time08:20:562
1725998361,16cyclictest31448-21sh09:45:341
172599834,73cyclictest0-21swapper/111:50:461
1719998346,26cyclictest0-21swapper/012:30:560
1729998265,11cyclictest15302-21kworker/u17:0-xprtiod10:35:342
1729998261,15cyclictest17607-21threads07:51:122
1729998259,16cyclictest1708-21cyclictest07:41:102
172599823,74cyclictest0-21swapper/107:10:501
172599823,68cyclictest0-21swapper/108:10:511
172599823,68cyclictest0-21swapper/108:10:511
1725998233,39cyclictest0-21swapper/107:26:111
1719998260,13cyclictest0-21swapper/010:21:140
1729998159,15cyclictest378-21rs:main2
1725998165,10cyclictest1708-21cyclictest10:57:061
172599812,69cyclictest0-21swapper/108:30:491
13612816,60sleep10-21swapper/107:02:521
1725998043,33cyclictest1708-21cyclictest11:05:511
1725998043,33cyclictest1708-21cyclictest11:05:511
172599803,67cyclictest5364-21modprobe08:55:341
1725998031,39cyclictest3699-21if_eth010:50:591
1719998060,14cyclictest8619-21diskmemload10:43:220
171999803,54cyclictest9359-21cstates11:00:490
1729997957,15cyclictest9566-21sh09:06:092
1729997957,15cyclictest9566-21sh09:06:092
1725997951,18cyclictest18641-21latency_hist10:20:371
1725997934,38cyclictest11554-21sh09:10:371
1725997934,38cyclictest11554-21sh09:10:371
172599792,66cyclictest0-21swapper/107:30:501
1729997859,13cyclictest26293-21idleruntime-cro08:20:342
1729997859,13cyclictest26293-21idleruntime-cro08:20:342
1729997823,30cyclictest39-21ksoftirqd/210:46:062
172599784,70cyclictest0-21swapper/107:35:511
172599784,69cyclictest0-21swapper/112:00:521
1725997832,37cyclictest0-21swapper/111:00:541
1719997859,12cyclictest8619-21diskmemload09:35:140
1719997858,14cyclictest5942-21latency_hist10:55:380
1719997858,14cyclictest23133-21sh11:23:570
1719997857,15cyclictest16262-21fschecks_count10:15:560
1719997855,9cyclictest181rcu_preempt11:09:460
1719997855,9cyclictest181rcu_preempt11:09:460
1719997846,26cyclictest0-21swapper/009:26:230
1719997833,34cyclictest0-21swapper/009:20:380
1719997833,34cyclictest0-21swapper/009:20:380
172599774,69cyclictest0-21swapper/109:30:481
172599773,65cyclictest0-21swapper/108:25:501
172599773,64cyclictest0-21swapper/107:20:481
172599773,63cyclictest0-21swapper/108:40:481
172599773,61cyclictest21650-21ssh11:21:051
1719997755,16cyclictest508-21latency_hist08:40:380
1719997748,9cyclictest181rcu_preempt12:00:520
171999772,52cyclictest31976-21ssh09:45:570
171999772,52cyclictest31976-21ssh09:45:570
40927616,39sleep30-21swapper/307:01:033
172999767,51cyclictest30338-21ssh11:36:122
172599763,68cyclictest0-21swapper/108:00:491
1725997631,38cyclictest0-21swapper/110:36:371
1719997657,12cyclictest26185-21/usr/sbin/munin09:35:590
1719997656,11cyclictest0-21swapper/008:45:500
1729997560,11cyclictest1708-21cyclictest12:31:122
172599755,65cyclictest0-21swapper/110:10:491
172599755,65cyclictest0-21swapper/110:10:491
172599754,50cyclictest17604-21apt-get09:20:521
172599754,50cyclictest17604-21apt-get09:20:521
1725997538,28cyclictest966-21fschecks_time11:40:591
1725997538,28cyclictest966-21fschecks_time11:40:591
172599753,62cyclictest0-21swapper/107:45:521
172599753,62cyclictest0-21swapper/107:45:521
172599753,62cyclictest0-21swapper/107:05:491
1725997535,32cyclictest0-21swapper/110:01:111
1725997531,35cyclictest0-21swapper/110:41:101
172599752,64cyclictest0-21swapper/111:25:511
172599752,64cyclictest0-21swapper/111:25:511
172599752,63cyclictest0-21swapper/110:05:501
172599752,51cyclictest356-21runrttasks12:25:421
1719997552,17cyclictest24900-21ssh11:26:120
1719997552,17cyclictest24900-21ssh11:26:120
1719997551,15cyclictest12045-21ssh09:10:580
171999754,43cyclictest10199-21latency_hist10:05:360
171999752,37cyclictest17915-21fschecks_time09:21:060
171999752,37cyclictest17915-21fschecks_time09:21:060
1729997449,19cyclictest17146-21apt-key09:20:382
1729997449,19cyclictest17146-21apt-key09:20:382
1729997411,23cyclictest381ktimers/212:23:502
1725997446,20cyclictest356-21runrttasks07:41:031
172599744,60cyclictest0-21swapper/109:00:531
172599743,66cyclictest0-21swapper/110:25:491
172599743,66cyclictest0-21swapper/110:25:491
172599743,66cyclictest0-21swapper/107:15:481
172599742,68cyclictest0-21swapper/108:20:471
172599742,68cyclictest0-21swapper/108:20:471
1719997442,26cyclictest6858-21ntp_kernel_pll_10:56:040
1719997435,11cyclictest0-21swapper/010:33:540
296062730,7sleep30-21swapper/311:35:333
1729997350,16cyclictest15719-21netstat11:11:042
1729997341,9cyclictest181rcu_preempt10:57:342
1729997333,18cyclictest4863-21ssh11:46:322
172999732,25cyclictest39-21ksoftirqd/211:42:012
172999732,25cyclictest39-21ksoftirqd/211:42:012
172599734,64cyclictest0-21swapper/109:45:481
172599734,64cyclictest0-21swapper/109:45:481
172599733,65cyclictest0-21swapper/112:05:481
172599733,60cyclictest26982-21apt-get11:30:471
1725997333,32cyclictest0-21swapper/107:56:131
172599732,66cyclictest0-21swapper/109:55:521
172599732,66cyclictest0-21swapper/109:55:521
1719997324,13cyclictest181rcu_preempt11:31:130
1729997252,7cyclictest181rcu_preempt09:55:292
1729997252,14cyclictest29020-21ssh09:40:512
1729997251,12cyclictest14284-21kworker/u17:2-rpciod11:18:302
172599723,64cyclictest0-21swapper/112:20:511
172599723,64cyclictest0-21swapper/108:05:511
172599723,61cyclictest2330-21/usr/sbin/munin08:45:581
172599723,61cyclictest2330-21/usr/sbin/munin08:45:581
1719997251,12cyclictest0-21swapper/007:35:570
1729997154,11cyclictest9266-21awk11:55:342
1729997151,7cyclictest181rcu_preempt12:16:482
1729997151,7cyclictest181rcu_preempt12:13:172
1729997151,7cyclictest181rcu_preempt12:13:172
1729997149,16cyclictest28562-21timerandwakeup10:36:532
172599714,57cyclictest24890-21memory10:31:011
172599712,65cyclictest0-21swapper/109:50:511
172599712,64cyclictest9598-21ssh11:55:481
172599712,64cyclictest0-21swapper/111:10:491
1719997152,13cyclictest365-21systemd-logind07:30:350
1719997142,22cyclictest0-21swapper/010:15:350
1719997142,22cyclictest0-21swapper/010:15:350
171999713,36cyclictest356-21runrttasks12:20:500
171999713,33cyclictest21467-21/usr/sbin/munin12:16:110
1719997116,12cyclictest181rcu_preempt09:50:470
1729997044,16cyclictest22043-21timerandwakeup11:21:132
1729997013,30cyclictest39-21ksoftirqd/208:39:072
1729997013,23cyclictest381ktimers/208:26:032
172599702,64cyclictest0-21swapper/111:45:441
1719997042,9cyclictest181rcu_preempt07:50:490
1719997036,28cyclictest21696-21ssh10:25:530
1719997036,28cyclictest21696-21ssh10:25:530
171999703,35cyclictest3633-21apt-get11:45:430
1729996944,13cyclictest181rcu_preempt10:26:002
1729996944,13cyclictest181rcu_preempt10:26:002
1729996941,11cyclictest181rcu_preempt12:05:272
172999693,46cyclictest8135-21grep07:21:102
172999693,42cyclictest4826-21switchtime08:52:092
1729996931,25cyclictest1761-21modprobe09:50:342
1729996931,25cyclictest1761-21modprobe09:50:342
1729996931,13cyclictest181rcu_preempt10:17:242
172999692,48cyclictest6376-21timerandwakeup07:16:112
172999692,47cyclictest12043-21latency_hist11:05:372
1729996923,14cyclictest181rcu_preempt11:30:372
1729996923,14cyclictest181rcu_preempt11:30:372
1729996913,22cyclictest381ktimers/210:55:352
172599694,60cyclictest0-21swapper/108:55:481
172599694,60cyclictest0-21swapper/108:55:481
172599694,58cyclictest20825-21ssh11:20:351
171999694,31cyclictest13619-21/usr/sbin/munin07:41:000
1719996941,14cyclictest181rcu_preempt10:10:250
171999693,34cyclictest16702-21latency_hist07:50:360
171999693,34cyclictest16702-21latency_hist07:50:360
1719996933,27cyclictest3228-21cron07:10:340
171999692,37cyclictest4608-21kworker/0:2+mm_percpu_wq10:46:110
1729996845,15cyclictest364-21ntpd07:30:562
1729996835,9cyclictest181rcu_preempt10:02:302
172999683,49cyclictest1180-21grep08:41:082
1729996831,12cyclictest181rcu_preempt10:08:412
1729996811,23cyclictest381ktimers/209:05:362
172599684,42cyclictest25163-21tune2fs08:15:541
172599684,42cyclictest25163-21tune2fs08:15:541
172599681,29cyclictest31580-21fschecks_count08:35:571
171999684,30cyclictest30427-21cpu11:36:240
171999683,44cyclictest5850-21grep08:55:550
171999683,44cyclictest5850-21grep08:55:550
1719996828,11cyclictest181rcu_preempt12:28:220
1719996822,30cyclictest2387-21fschecks_count08:46:040
1719996822,30cyclictest2387-21fschecks_count08:46:040
172999674,32cyclictest16149-21unixbench_multi07:46:142
172999674,32cyclictest16149-21unixbench_multi07:46:142
1729996742,7cyclictest181rcu_preempt09:57:192
1729996742,7cyclictest181rcu_preempt09:57:192
1729996737,15cyclictest181rcu_preempt10:23:022
172999673,55cyclictest17759-21expr09:20:592
172999673,55cyclictest17759-21expr09:20:592
1729996735,10cyclictest181rcu_preempt09:25:592
172999672,44cyclictest13471-21ssh10:10:562
172999672,44cyclictest13471-21ssh10:10:562
1719996737,21cyclictest8754-21cat09:05:370
1719996730,13cyclictest181rcu_preempt07:20:550
172999663,39cyclictest24730-21latency_hist08:15:372
172999663,39cyclictest24730-21latency_hist08:15:372
1729996629,19cyclictest0-21swapper/211:30:572
172999662,31cyclictest39-21ksoftirqd/209:40:032
1719996650,11cyclictest5807-21fschecks_time07:15:570
1719996646,14cyclictest16961-21ssh11:13:070
171999663,33cyclictest26291-21latency_hist08:20:360
171999663,33cyclictest26291-21latency_hist08:20:360
173499657,29cyclictest0-21swapper/310:36:343
172999653,43cyclictest3022-21timerwakeupswit08:46:242
172999653,43cyclictest3022-21timerwakeupswit08:46:242
172999653,43cyclictest20054-21latency_hist08:00:382
1719996547,13cyclictest170-21jbd2/mmcblk2p2-807:56:110
171999654,55cyclictest161ktimers/008:20:470
171999654,55cyclictest161ktimers/008:20:470
1719996543,17cyclictest0-21swapper/008:06:120
1719996543,15cyclictest10879-21rm09:08:530
1719996543,15cyclictest10879-21rm09:08:530
171999653,48cyclictest29585-21grep08:30:360
171999653,30cyclictest11758-21modprobe07:35:340
171999652,43cyclictest9557-21/usr/sbin/munin11:56:010
1719996523,33cyclictest20469-21kernelversion08:01:030
1729996438,7cyclictest181rcu_preempt08:04:312
172999642,26cyclictest39-21ksoftirqd/208:55:512
172999642,26cyclictest39-21ksoftirqd/208:55:512
1719996344,12cyclictest378-21rs:main0
171999634,27cyclictest356-21runrttasks08:33:590
172999624,32cyclictest13372-21latency_hist07:40:382
173499616,33cyclictest0-21swapper/311:36:003
172999617,25cyclictest39-21ksoftirqd/208:31:012
172999612,42cyclictest23263-21latency_hist08:10:382
173499602,42cyclictest22194-21taskset10:26:073
173499602,42cyclictest22194-21taskset10:26:073
173499573,48cyclictest0-21swapper/309:21:013
173499573,48cyclictest0-21swapper/309:21:013
173499563,41cyclictest1647-21taskset11:41:133
173499563,41cyclictest1647-21taskset11:41:133
173499562,38cyclictest10273-21chrt11:01:093
173499553,32cyclictest0-21swapper/309:26:073
173499552,38cyclictest17364-21chrt09:20:383
173499552,38cyclictest17364-21chrt09:20:383
173499552,35cyclictest15751-21taskset12:05:573
173499544,37cyclictest27288-21chrt09:36:153
173499543,33cyclictest5954-21chrt10:55:343
173499542,38cyclictest2941-21chrt09:51:113
173499542,38cyclictest15331-21chrt11:10:533
173499533,39cyclictest16394-21taskset10:15:583
173499533,35cyclictest14020-21taskset09:14:593
173499533,34cyclictest25484-21chrt12:21:123
173499532,40cyclictest8782-21taskset11:54:183
173499532,38cyclictest31844-21taskset10:43:013
173499532,38cyclictest11289-21chrt11:57:553
173499524,32cyclictest6144-21chrt07:16:073
173499523,36cyclictest23427-21chrt09:30:523
173499523,36cyclictest21686-21taskset12:15:583
173499523,36cyclictest11197-21taskset07:31:103
173499523,34cyclictest7694-21chrt07:21:013
173499523,34cyclictest7041-21chrt09:00:363
173499523,34cyclictest7041-21chrt09:00:363
173499523,33cyclictest20170-21chrt10:22:063
173499523,31cyclictest2005-21chrt09:50:383
173499523,31cyclictest2005-21chrt09:50:383
173499513,38cyclictest9782-21chrt09:06:113
173499513,38cyclictest9782-21chrt09:06:113
173499513,38cyclictest27663-21chrt12:25:593
173499513,36cyclictest5930-21taskset09:56:133
173499513,36cyclictest5930-21taskset09:56:133
173499513,36cyclictest3656-21taskset11:45:473
173499513,34cyclictest4896-21taskset07:11:133
173499513,32cyclictest714-21chrt12:35:393
173499512,36cyclictest526-21chrt10:45:493
173499512,35cyclictest22048-21chrt08:06:013
173499512,34cyclictest4950-21taskset08:52:103
173499512,34cyclictest31846-21taskset08:36:023
173499504,36cyclictest13163-21taskset12:01:073
173499503,35cyclictest7164-21chrt10:56:103
173499503,31cyclictest31329-21taskset08:35:373
173499502,31cyclictest15902-21chrt10:15:383
173499502,31cyclictest15902-21chrt10:15:383
173499494,36cyclictest25772-21chrt11:28:043
173499494,36cyclictest25772-21chrt11:28:043
173499494,32cyclictest29728-21taskset09:41:103
173499494,31cyclictest22369-21chrt11:21:433
173499494,28cyclictest28080-21chrt08:25:373
173499494,28cyclictest28080-21chrt08:25:373
173499493,35cyclictest20052-21chrt12:12:333
173499493,35cyclictest20052-21chrt12:12:333
173499492,36cyclictest20502-21taskset08:01:033
173499492,35cyclictest20781-21chrt11:20:343
173499492,32cyclictest28154-21taskset08:25:453
173499483,36cyclictest9243-21taskset10:02:463
173499483,33cyclictest10275-21chrt07:30:373
173499483,31cyclictest24991-21chrt08:15:493
173499483,31cyclictest24991-21chrt08:15:493
173499472,32cyclictest23986-21taskset08:11:113
173499472,32cyclictest23986-21taskset08:11:113
1734994633,8cyclictest451rcuc/309:03:023
173499462,31cyclictest2725-21taskset08:46:193
173499462,31cyclictest2725-21taskset08:46:193
173499462,31cyclictest14328-21chrt07:41:103
173499462,31cyclictest12388-21chrt10:09:063
173499453,30cyclictest2897-21chrt07:06:153
173499452,28cyclictest14984-21chrt11:10:373
173499452,28cyclictest14984-21chrt11:10:373
173499443,28cyclictest26768-21taskset10:35:153
173499442,28cyclictest18272-21taskset07:55:363
1734994320,12cyclictest0-21swapper/307:40:383
173499423,28cyclictest19838-21chrt08:00:343
173499423,26cyclictest16595-21chrt07:50:343
173499423,26cyclictest16595-21chrt07:50:343
1734994129,8cyclictest451rcuc/308:40:473
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional