You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 03:31
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Mon Nov 24, 2025 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
232639913331,65cyclictest24945-21unin-run23:44:150
232639912637,56cyclictest25042-21timerwakeupswit20:49:520
23263991133,34cyclictest866-21apt-get22:09:290
232639911332,72cyclictest14146-21ssh22:29:590
232639911332,72cyclictest14146-21ssh22:29:590
232739911065,38cyclictest20176-21timerandwakeup23:34:502
23263991093,26cyclictest9387-21kworker/u17:0+ext4-rsv-conversion19:09:190
232639910839,59cyclictest0-21swapper/019:59:520
2731421070,8sleep10-21swapper/123:47:461
232639910717,21cyclictest6027-21sh21:22:090
232639910660,40cyclictest28467-21fschecks_time23:49:380
232639910631,69cyclictest0-21swapper/021:54:490
232639910631,69cyclictest0-21swapper/021:54:490
23263991062,37cyclictest22580-21apt-get20:44:280
232639910535,65cyclictest0-21swapper/000:14:140
232739910466,31cyclictest0-21swapper/219:14:532
232639910447,52cyclictest23250-21cyclictest23:09:530
232639910447,52cyclictest23250-21cyclictest23:09:530
232639910441,57cyclictest0-21swapper/020:14:540
232639910438,59cyclictest0-21swapper/023:19:430
232739910361,35cyclictest7422-21latency_hist00:09:192
23273991033,36cyclictest12963-21rm21:34:292
23273991033,36cyclictest12963-21rm21:34:292
23263991032,33cyclictest22435-21cat20:44:170
232639910219,23cyclictest10501-21/usr/sbin/munin22:24:390
23263991013,24cyclictest14576-21ssh00:20:090
23271991003,64cyclictest22236-21apt-get00:34:331
23271991003,64cyclictest22236-21apt-get00:34:331
232639910043,50cyclictest0-21swapper/023:04:450
2326399995,31cyclictest14043-21/usr/sbin/munin21:34:540
2326399995,31cyclictest14043-21/usr/sbin/munin21:34:540
2326399993,24cyclictest355-21runrttasks23:17:420
2326399993,20cyclictest19856-21/usr/sbin/munin21:44:520
2327399983,34cyclictest30962-21fschecks_count21:09:372
2326399983,85cyclictest28412-21grep21:59:490
2326399983,36cyclictest1303-21modprobe23:04:140
2326399973,37cyclictest31786-21vmstat22:04:520
2326399972,33cyclictest22059-21cat00:34:180
23263999719,25cyclictest31237-21sed19:29:520
2327399964,28cyclictest13434-21seq23:24:312
23263999676,15cyclictest16064-21latency_hist20:24:190
2326399964,31cyclictest17643-21latency_hist20:29:190
23263999640,51cyclictest0-21swapper/019:40:000
2326399963,23cyclictest355-21runrttasks00:06:260
2326399963,23cyclictest24170-21ssh22:47:030
2326399963,22cyclictest7810-21sendmail_mailst21:24:500
2326399963,22cyclictest7810-21sendmail_mailst21:24:500
2326399963,20cyclictest26118-21grep23:44:500
2326399962,9cyclictest181rcu_preempt22:34:280
2326399962,9cyclictest181rcu_preempt22:34:280
2326399962,22cyclictest31409-21fschecks_time23:54:380
23263999621,70cyclictest0-21swapper/020:29:380
23263999621,70cyclictest0-21swapper/020:29:380
2326399958,29cyclictest27237-21expr19:19:340
2326399954,21cyclictest19537-21fschecks_count23:34:360
23263999534,55cyclictest12377-21munin-plugin-st20:14:170
2326399953,32cyclictest10001-21basename21:29:250
2326399953,30cyclictest28462-21fschecks_time22:54:380
2326399953,30cyclictest28462-21fschecks_time22:54:380
2326399953,28cyclictest4976-21latency19:49:410
2326399953,28cyclictest4976-21latency19:49:410
2327399942,35cyclictest1795-21kworker/u18:0+ext4-rsv-conversion00:34:492
2327399942,35cyclictest1795-21kworker/u18:0+ext4-rsv-conversion00:34:492
2327199949,78cyclictest882-21kworker/u18:0+flush-179:019:44:301
2326399945,29cyclictest23694-21runrttasks19:09:400
23263999440,47cyclictest23250-21cyclictest21:50:480
23263999440,47cyclictest0-21swapper/022:19:340
2326399943,37cyclictest6182-21rm22:16:580
2326399943,37cyclictest6182-21rm22:16:580
2326399943,34cyclictest355-21runrttasks23:28:530
2326399943,34cyclictest16345-21/usr/sbin/munin23:29:370
2326399943,32cyclictest28822-21latency_hist21:04:170
2326399943,29cyclictest3854-21systemd-run21:19:160
2326399943,27cyclictest30521-21latency_hist21:09:180
23263999420,7cyclictest0-21swapper/000:14:390
23263999420,7cyclictest0-21swapper/000:14:390
2185829410,58sleep121910-21expr19:04:361
2327399932,34cyclictest14334-21sed20:19:182
2327199933,53cyclictest14517-21apt-get20:19:401
2326399934,30cyclictest22738-21fschecks_time00:35:130
2326399934,30cyclictest22738-21fschecks_time00:35:130
2326399933,30cyclictest371-21unixbench_multi19:34:510
2326399933,30cyclictest16474-21ssh00:24:370
2326399933,30cyclictest16474-21ssh00:24:370
2326399933,29cyclictest9410-21/usr/sbin/munin20:04:350
23263999317,22cyclictest22114-21cat22:44:170
2327399922,38cyclictest28642-21cat21:59:522
23263999242,45cyclictest0-21swapper/019:14:240
2326399923,31cyclictest26009-21fschecks_count20:54:360
2326399923,29cyclictest25791-21netstat22:49:430
2326399923,28cyclictest7151-21/usr/sbin/munin19:54:530
2326399923,28cyclictest7151-21/usr/sbin/munin19:54:530
2326399923,25cyclictest30923-21/usr/sbin/munin21:09:370
23263999224,61cyclictest0-21swapper/023:59:380
23273999142,41cyclictest0-21swapper/200:19:502
23273999136,48cyclictest0-21swapper/219:44:512
23273999132,51cyclictest0-21swapper/221:39:302
23273999132,51cyclictest0-21swapper/221:39:302
2326399913,34cyclictest29080-21uname19:24:420
2326399913,32cyclictest20798-21latency_hist20:39:170
2326399913,32cyclictest20798-21latency_hist20:39:170
2326399913,25cyclictest16030-21taskset21:39:320
2326399913,25cyclictest16030-21taskset21:39:320
23278999010,68cyclictest13538-21chrt22:29:403
23278999010,68cyclictest13538-21chrt22:29:403
23273999071,9cyclictest0-21swapper/219:29:182
23273999067,13cyclictest0-21swapper/219:09:392
2327399904,34cyclictest10454-21/usr/sbin/munin23:19:482
23273999033,49cyclictest0-21swapper/222:44:402
23273999027,52cyclictest0-21swapper/200:14:182
2327199902,56cyclictest31-21ksoftirqd/100:12:241
2326399902,29cyclictest3828-21timerwakeupswit19:45:040
23273998967,15cyclictest368-21snmpd21:56:342
23273998967,15cyclictest368-21snmpd21:56:342
2327399894,36cyclictest12924-21idleruntime-cro22:29:152
2327399894,34cyclictest16901-21runrttasks00:24:482
2327399894,34cyclictest16901-21runrttasks00:24:482
2327399893,39cyclictest11017-21/usr/sbin/munin20:09:502
2327399893,36cyclictest7531-21/usr/sbin/munin22:19:372
2327399893,35cyclictest27833-21latency_hist23:49:172
2327399893,33cyclictest355-21runrttasks21:24:552
2327399893,33cyclictest355-21runrttasks21:24:552
2327399892,33cyclictest6938-21timerandwakeup19:54:512
2327399892,33cyclictest6938-21timerandwakeup19:54:512
2327199893,48cyclictest31-21ksoftirqd/123:21:521
2327399884,35cyclictest1203-21cpu21:14:322
2327399884,31cyclictest8747-21grep19:59:522
23273998842,39cyclictest0-21swapper/222:14:142
2327399883,32cyclictest355-21runrttasks20:45:372
2327399883,32cyclictest2698-21sed19:44:172
2327399883,32cyclictest16228-21rm22:34:182
2327399883,32cyclictest16228-21rm22:34:182
23273998830,51cyclictest0-21swapper/221:44:412
23273998821,35cyclictest8535-21grep23:14:512
23271998859,21cyclictest7776-21fschecks_time23:14:361
23273998740,41cyclictest0-21swapper/223:09:182
23273998733,47cyclictest6982-21sh22:19:152
23273998733,47cyclictest6982-21sh22:19:152
2327399873,33cyclictest21282-21if_err_eth020:39:392
2327399873,32cyclictest355-21runrttasks21:08:382
2327399872,28cyclictest20189-21grep00:29:532
2327399863,35cyclictest26495-21switchtime20:54:492
2327399863,34cyclictest27976-21wc19:19:522
2327399863,33cyclictest16384-21/usr/sbin/munin22:34:542
2327399863,33cyclictest16384-21/usr/sbin/munin22:34:542
23273998627,8cyclictest0-21swapper/220:04:492
23273998627,7cyclictest0-21swapper/223:59:312
23273998626,9cyclictest0-21swapper/223:39:262
2327199862,48cyclictest31-21ksoftirqd/123:41:041
23273998534,44cyclictest0-21swapper/222:04:522
2327399853,34cyclictest31834-21latency_hist19:34:192
2327399853,34cyclictest22812-21unixbench_multi21:49:522
2327399853,31cyclictest17714-21/usr/sbin/munin20:29:492
2327399853,31cyclictest17714-21/usr/sbin/munin20:29:492
23271998567,10cyclictest0-21swapper/121:39:161
23271998567,10cyclictest0-21swapper/121:39:161
23271998555,22cyclictest30990-21fschecks_time21:09:381
2327199852,32cyclictest31-21ksoftirqd/122:18:251
2327199852,32cyclictest31-21ksoftirqd/122:18:251
2327399844,24cyclictest19566-21/usr/sbin/munin22:39:362
23273998436,42cyclictest0-21swapper/222:54:382
23273998436,42cyclictest0-21swapper/222:54:382
2327399843,34cyclictest7281-21ssh23:14:172
2327399843,34cyclictest7281-21ssh23:14:172
2327399843,33cyclictest6763-21latency_hist21:24:182
2327399843,32cyclictest4622-21grep19:49:322
2327399843,32cyclictest4622-21grep19:49:322
2327399843,30cyclictest28136-21/usr/sbin/munin23:49:492
23273998429,47cyclictest0-21swapper/222:59:502
2327399842,35cyclictest831-21latency_hist19:39:192
23271998460,17cyclictest27413-21idleruntime-cro21:59:151
23271998460,17cyclictest27413-21idleruntime-cro21:59:151
23271998442,8cyclictest181rcu_preempt23:54:321
2327399834,35cyclictest28954-21taskset21:04:182
2327399834,32cyclictest16111-21/usr/sbin/munin20:24:482
2327399833,34cyclictest26312-21head22:49:522
2327399833,33cyclictest31374-21fschecks_time23:54:372
23273998326,47cyclictest0-21swapper/220:24:182
23271998361,13cyclictest292-21dbus-daemon00:19:161
23271998361,13cyclictest292-21dbus-daemon00:19:161
23271998349,10cyclictest181rcu_preempt00:08:351
2316228311,45sleep223177-21munin-run19:09:172
2327399823,43cyclictest25031-21munin-node21:29:252
2327399822,32cyclictest25616-21latency_hist20:54:172
2327399813,33cyclictest10741-21fschecks_time00:14:412
2327399813,33cyclictest10741-21fschecks_time00:14:412
2327399813,25cyclictest19522-21fschecks_count20:34:382
2327399813,25cyclictest19522-21fschecks_count20:34:382
23271998164,12cyclictest23250-21cyclictest22:09:151
2327199812,37cyclictest31-21ksoftirqd/119:14:121
2327199805,33cyclictest31-21ksoftirqd/121:27:121
2327199805,33cyclictest31-21ksoftirqd/121:27:121
2327199802,48cyclictest31-21ksoftirqd/121:30:221
2327199802,37cyclictest31-21ksoftirqd/122:50:561
2327399793,35cyclictest17153-21ssh23:29:512
23271997958,14cyclictest23250-21cyclictest22:28:441
2327199793,41cyclictest31-21ksoftirqd/119:17:411
23271997913,38cyclictest31-21ksoftirqd/120:09:281
2327199782,39cyclictest31-21ksoftirqd/119:33:041
2327199782,33cyclictest31-21ksoftirqd/120:43:541
23278997722,36cyclictest0-21swapper/322:10:493
2327199775,30cyclictest31-21ksoftirqd/121:04:431
23271997752,15cyclictest7310-21kworker/u17:2-xprtiod21:49:171
23271997732,12cyclictest181rcu_preempt20:45:351
23271997723,34cyclictest31-21ksoftirqd/120:59:181
23271997637,16cyclictest181rcu_preempt22:34:011
23271997637,16cyclictest181rcu_preempt22:34:011
2327199762,46cyclictest31-21ksoftirqd/122:45:091
2327199762,37cyclictest31-21ksoftirqd/119:36:521
23271997558,13cyclictest23250-21cyclictest23:33:001
23271997553,15cyclictest23250-21cyclictest19:25:141
2327199753,43cyclictest31-21ksoftirqd/123:09:291
2327199753,43cyclictest31-21ksoftirqd/123:09:291
2327199752,41cyclictest31-21ksoftirqd/119:52:541
2327199752,41cyclictest31-21ksoftirqd/119:52:541
23271997454,14cyclictest30844-21sh23:54:171
23271997450,16cyclictest7719-21interrupts22:19:401
23271997443,22cyclictest2579-21apt-key19:44:171
23271997437,12cyclictest181rcu_preempt19:55:301
23271997437,12cyclictest181rcu_preempt19:55:301
23271997350,13cyclictest0-21swapper/100:24:541
23271997350,13cyclictest0-21swapper/100:24:541
2327199733,30cyclictest31-21ksoftirqd/119:20:241
2327199732,24cyclictest31-21ksoftirqd/122:10:111
23271997247,19cyclictest12648-21kworker/u19:2+events_unbound22:44:141
2327199724,29cyclictest31-21ksoftirqd/123:26:181
23271997241,22cyclictest10965-21latency_hist20:09:181
2327199722,38cyclictest31-21ksoftirqd/122:35:081
2327199722,38cyclictest31-21ksoftirqd/122:35:081
2327199722,27cyclictest31-21ksoftirqd/100:31:551
23271997148,15cyclictest13513-21cstates00:19:321
2327199712,26cyclictest31-21ksoftirqd/120:04:191
23271997040,8cyclictest181rcu_preempt23:05:151
23271997036,12cyclictest181rcu_preempt21:01:531
2327199703,27cyclictest31-21ksoftirqd/121:51:161
2327199702,46cyclictest31-21ksoftirqd/120:26:531
2327199702,25cyclictest31-21ksoftirqd/123:36:251
2327199702,25cyclictest31-21ksoftirqd/120:36:541
2327199702,25cyclictest31-21ksoftirqd/120:36:541
23271996935,12cyclictest181rcu_preempt21:20:441
2327199693,46cyclictest1687-21ssh23:59:311
2327199693,38cyclictest31407-21fschecks_time22:59:361
2327199692,44cyclictest31-21ksoftirqd/120:54:111
2327199692,34cyclictest17343-21ssh21:40:501
2327199692,34cyclictest17343-21ssh21:40:501
23271996914,41cyclictest27737-21sort21:59:271
2327199682,32cyclictest31-21ksoftirqd/122:54:551
2327199682,32cyclictest31-21ksoftirqd/122:54:551
2327199674,41cyclictest355-21runrttasks20:33:161
2327199674,41cyclictest355-21runrttasks20:33:161
23271996741,16cyclictest0-21swapper/121:19:171
23271996613,28cyclictest31-21ksoftirqd/120:16:331
23278996531,22cyclictest26303-21chrt19:14:523
2327899636,49cyclictest0-21swapper/319:44:403
2327899624,41cyclictest24390-21taskset23:42:463
2327899623,43cyclictest24707-21chrt00:39:143
2327899623,43cyclictest24707-21chrt00:39:143
2327899614,41cyclictest16659-21taskset22:34:383
2327899614,41cyclictest16659-21taskset22:34:383
23278996014,25cyclictest0-21swapper/320:19:293
2327899594,41cyclictest10646-21chrt23:19:363
2327899593,41cyclictest29019-21chrt22:00:223
2327899584,35cyclictest30554-21chrt19:29:383
2327899574,40cyclictest31238-21taskset23:54:333
2327899573,41cyclictest4602-21taskset23:09:313
2327899573,41cyclictest4602-21taskset23:09:313
2327899563,40cyclictest5937-21chrt22:16:153
2327899563,40cyclictest5937-21chrt22:16:153
2327899563,38cyclictest15601-21chrt21:39:153
2327899563,38cyclictest15601-21chrt21:39:153
2327899554,38cyclictest19977-21taskset22:39:453
2327899553,40cyclictest21931-21chrt21:49:323
2327899553,39cyclictest28035-21taskset23:49:183
2327899553,38cyclictest19511-21chrt20:34:383
2327899553,38cyclictest19511-21chrt20:34:383
2327899553,33cyclictest16325-21taskset21:39:403
2327899553,33cyclictest16325-21taskset21:39:403
2327899552,35cyclictest895-21kworker/3:1+events20:49:293
2327899545,35cyclictest30941-21chrt22:04:363
2327899543,37cyclictest28291-21chrt20:59:533
2327899543,37cyclictest15349-21chrt00:22:283
2327899542,36cyclictest2195-21chrt21:14:523
2327899534,36cyclictest7751-21chrt19:59:183
2327899534,36cyclictest7751-21chrt19:59:183
2327899533,35cyclictest26217-21chrt21:55:533
2327899533,35cyclictest26217-21chrt21:55:533
2327899533,35cyclictest10981-21taskset20:09:193
2327899533,31cyclictest10688-21chrt22:24:423
2327899523,39cyclictest28462-21taskset23:49:383
2327899523,36cyclictest22028-21chrt23:39:173
2327899523,35cyclictest4269-21taskset23:09:153
2327899514,35cyclictest29033-21taskset21:04:273
2327899514,34cyclictest30731-21chrt21:09:293
2327899513,36cyclictest18909-21chrt00:29:173
2327899513,36cyclictest18909-21chrt00:29:173
2327899513,34cyclictest8136-21taskset23:14:463
2327899513,34cyclictest21228-21taskset00:32:543
2327899513,33cyclictest2693-21chrt00:00:003
2327899513,33cyclictest26739-21chrt20:54:513
2327899504,33cyclictest14018-21chrt20:19:153
2327899503,35cyclictest10322-21chrt00:14:183
2327899503,34cyclictest1664-21chrt19:39:553
2327899502,36cyclictest27236-21taskset19:19:343
2327899502,34cyclictest8016-21chrt21:24:533
2327899502,34cyclictest8016-21chrt21:24:533
2317925010,23sleep30-21swapper/319:09:183
2327899493,35cyclictest18862-21taskset23:33:543
2327899493,32cyclictest12686-21chrt21:34:173
2327899492,32cyclictest26423-21chrt22:50:063
2327899483,33cyclictest21480-21chrt20:39:443
23278994817,20cyclictest22579-21chrt20:44:263
2327899474,32cyclictest5955-21chrt21:21:573
2327899473,33cyclictest5501-21chrt00:04:543
2327899473,31cyclictest30139-21taskset22:57:243
2327899473,31cyclictest30139-21taskset22:57:243
2327899473,30cyclictest900-21taskset19:39:183
2327899473,29cyclictest12018-21chrt00:16:163
2327899473,29cyclictest12018-21chrt00:16:163
2327899472,35cyclictest28875-21chrt19:24:363
2327899464,33cyclictest16091-21chrt23:29:163
2327899464,32cyclictest21421-21taskset21:49:043
2327899463,33cyclictest9559-21taskset22:23:083
2327899463,31cyclictest4856-21taskset19:49:383
2327899463,31cyclictest4856-21taskset19:49:383
2327899462,33cyclictest23956-21chrt19:09:463
2327899453,32cyclictest8618-21chrt19:59:513
2327899453,31cyclictest25045-21chrt22:49:173
2327899443,29cyclictest18111-21taskset20:29:403
2327899443,29cyclictest18111-21taskset20:29:403
2327899443,29cyclictest16575-21taskset20:24:423
2327899412,30cyclictest31321-21chrt22:59:343
2327899403,26cyclictest12215-21taskset20:12:223
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional