You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-21 - 22:08
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Thu Aug 21, 2025 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23270211710,71sleep123283-21/usr/sbin/munin06:58:471
23270211710,71sleep123283-21/usr/sbin/munin06:58:471
24310991102,83cyclictest19608-21kworker/u19:0+flush-179:009:38:370
23421210911,64sleep223482-21mailstats06:58:562
23421210911,64sleep223482-21mailstats06:58:562
1466521080,7sleep00-21swapper/011:29:210
24314991052,33cyclictest141342ssh09:28:471
243149910441,12cyclictest0-21swapper/110:38:121
24310991042,66cyclictest25109-21apt-get11:48:470
243149910345,50cyclictest0-21swapper/111:49:451
24314991033,38cyclictest356-21runrttasks11:43:271
243149910327,14cyclictest0-21swapper/110:38:471
243149910327,14cyclictest0-21swapper/110:38:471
24310991032,70cyclictest375-21snmpd09:43:400
24310991032,70cyclictest375-21snmpd09:43:400
243149910036,10cyclictest0-21swapper/111:14:011
24314991003,41cyclictest8578-21latency_hist07:53:161
2431499984,39cyclictest19826-21find11:38:501
2431499984,39cyclictest19826-21find11:38:501
2431499973,38cyclictest31261-21fschecks_time11:58:321
2431499973,34cyclictest4252-21ssh12:08:161
24314999733,10cyclictest0-21swapper/108:28:401
24314999733,10cyclictest0-21swapper/108:28:401
24314999732,12cyclictest0-21swapper/109:33:521
2431499964,39cyclictest13914-21fschecks_count08:08:361
24314999633,10cyclictest0-21swapper/112:08:451
2431499953,37cyclictest11579-21grep11:23:491
24314999533,13cyclictest0-21swapper/108:23:141
24314999533,13cyclictest0-21swapper/108:23:141
24314999532,11cyclictest0-21swapper/110:13:491
2431499944,33cyclictest29763-21acpi07:18:251
24314999441,14cyclictest13375-21ssh12:23:351
2431499943,38cyclictest2868-21acpi10:08:231
2431499943,36cyclictest29353-21cut08:54:321
2431499943,36cyclictest29353-21cut08:54:321
24314999431,13cyclictest0-21swapper/109:44:211
24314999431,13cyclictest0-21swapper/109:44:211
2431499942,40cyclictest6900-21ls07:48:161
2431499942,36cyclictest26618-21cstates07:08:321
2431499934,35cyclictest26078-21munin-node09:58:241
2431499933,39cyclictest31768-21grep07:23:401
2431499933,35cyclictest20160-21latency_hist08:28:151
2431499933,35cyclictest20160-21latency_hist08:28:151
2431499933,32cyclictest9006-21kernelversion07:53:411
2431499932,27cyclictest11376-21awk09:23:421
2431499932,27cyclictest11376-21awk09:23:421
24320999227,25cyclictest25099-21munin-run08:43:142
24320999224,26cyclictest0-21swapper/211:38:242
24320999224,26cyclictest0-21swapper/211:38:242
2431499924,42cyclictest26277-21mailstats08:44:461
2431499924,37cyclictest32026-21latency_hist09:03:181
2431499924,37cyclictest13605-21/usr/sbin/munin10:28:381
24314999237,12cyclictest8227-21fschecks_count11:18:381
24314999237,12cyclictest8227-21fschecks_count11:18:381
2431499923,37cyclictest3192-21mailstats11:08:471
2431499923,34cyclictest2754-21latency_hist10:08:151
24314999229,12cyclictest0-21swapper/110:49:001
2431499922,36cyclictest2467-21latency_hist09:08:171
24320999129,29cyclictest18578-21seq08:23:292
24320999129,29cyclictest18578-21seq08:23:292
24320999129,23cyclictest5170-21seq11:13:162
2431499914,36cyclictest8144-21switchtime12:13:491
2431499913,37cyclictest356-21runrttasks11:03:071
2431499913,35cyclictest11599-21latency_hist08:03:171
2431499912,36cyclictest10484-21fschecks_count12:18:361
24314999120,30cyclictest8100-21apt-key09:18:171
24320999015,29cyclictest0-21swapper/207:38:162
2431499904,36cyclictest25818-21modprobe07:08:121
2432099893,67cyclictest12099-21apt-get08:03:282
24320998927,25cyclictest11173-21seq11:23:412
2431499894,34cyclictest16957-21timerandwakeup12:28:501
2431499894,31cyclictest28015-21timerwakeupswit08:48:511
24314998935,11cyclictest301ktimers/110:23:141
2431499893,37cyclictest12378-21fschecks_count08:04:321
2431099893,68cyclictest8266-21ssh09:18:300
24320998823,28cyclictest0-21swapper/209:30:292
2431499884,35cyclictest2155-21latency_hist11:08:151
2431499884,30cyclictest27232-21sendmail09:53:151
2431499883,37cyclictest29857-21latency_hist09:58:141
2431499883,35cyclictest29165-21grep11:54:011
2431499883,34cyclictest25299-21latency_hist08:43:151
2431499883,34cyclictest21500-21sh09:43:031
24320998728,23cyclictest12978-21cat12:23:152
24320998728,21cyclictest8650-21seq07:53:312
24320998727,26cyclictest4666-21runrttasks09:12:062
24320998725,22cyclictest356-21runrttasks12:28:162
24320998723,20cyclictest0-21swapper/208:33:312
2431499874,35cyclictest15179-21ssh11:30:571
2431499874,31cyclictest18845-21latency_hist11:38:141
2431499873,33cyclictest15714-21kernelversion08:13:381
24310998764,13cyclictest24298-21cyclictest07:18:360
24320998661,18cyclictest12800-21kworker/u17:3+events_unbound10:38:122
24320998623,30cyclictest11348-21/usr/sbin/munin09:23:422
24320998623,30cyclictest11348-21/usr/sbin/munin09:23:422
2431499864,34cyclictest8210-21/usr/sbin/munin09:19:361
2431499864,34cyclictest21631-21/usr/sbin/munin10:43:461
2431499863,37cyclictest720-21fschecks_count07:28:361
24320998527,23cyclictest28624-21seq09:53:562
24320998525,29cyclictest27258-21seq11:51:582
24310998539,36cyclictest0-21swapper/009:23:360
24310998539,36cyclictest0-21swapper/009:23:360
24320998441,36cyclictest0-21swapper/207:18:372
2431499843,34cyclictest356-21runrttasks10:23:201
24320998266,9cyclictest1146-21ssh09:04:232
24320998265,11cyclictest18736-21systemd-run12:33:132
24310998259,15cyclictest27966-21systemd-journal10:40:090
24310998259,15cyclictest27966-21systemd-journal10:40:090
24310998258,17cyclictest6382-21threads10:13:490
24310998213,43cyclictest26824-21grep10:53:300
24320998160,14cyclictest4366-21sh11:11:302
24320998130,19cyclictest1-21systemd07:53:142
2431499814,30cyclictest356-21runrttasks07:15:471
24310998157,15cyclictest32020-21diskmemload11:04:300
24310998135,9cyclictest24048-21apt-get10:48:280
24320998026,21cyclictest24839-21runrttasks09:48:282
24314998051,20cyclictest2404-21fschecks_time07:33:371
2431499803,42cyclictest28295-21ssh10:55:121
24310998059,14cyclictest27966-21systemd-journal09:17:060
24310998050,20cyclictest10979-21cpu10:23:310
2431099803,27cyclictest12099-21apt-get08:03:400
2431099802,60cyclictest29043-21apt-get08:54:020
2431099802,60cyclictest29043-21apt-get08:54:020
2415028028,28sleep30-21swapper/307:03:133
2415028028,28sleep30-21swapper/307:03:133
24320997963,10cyclictest27885-21munin-run07:13:132
24320997933,35cyclictest0-21swapper/210:53:282
24320997929,42cyclictest28458-21ssh11:53:412
24320997925,24cyclictest32745-21chrt12:00:512
24320997925,24cyclictest24842-21cut11:48:152
24320997920,25cyclictest0-21swapper/207:24:442
24310997915,25cyclictest16507-21kernelversion12:28:400
24310997912,25cyclictest31814-21timerandwakeup11:58:450
2432899783,46cyclictest0-21swapper/309:18:423
24320997860,12cyclictest331-21dbus-daemon10:18:132
24320997832,37cyclictest0-21swapper/209:13:242
24320997825,25cyclictest2236-21taskset10:07:022
24320997825,19cyclictest375-21snmpd07:13:452
24320997818,27cyclictest0-21swapper/210:38:242
24320997818,27cyclictest0-21swapper/210:38:242
2431099787,53cyclictest24170-21latency_hist07:03:170
2431099787,53cyclictest24170-21latency_hist07:03:170
24310997856,16cyclictest18947-21latency_hist10:38:180
24310997849,19cyclictest28109-21latency_hist07:13:160
24310997823,44cyclictest29459-21apt-get10:58:300
24320997758,12cyclictest4700-21kworker/u17:3-xprtiod11:19:092
24320997758,12cyclictest4700-21kworker/u17:3-xprtiod11:19:092
24320997729,25cyclictest356-21runrttasks08:44:472
24320997721,26cyclictest27705-21taskset08:48:452
24320997720,24cyclictest0-21swapper/207:28:262
24310997754,17cyclictest15169-21latency_hist08:13:150
24310997723,30cyclictest0-21swapper/008:33:490
24310997710,26cyclictest31634-21cut07:23:370
24320997655,16cyclictest17710-21kworker/u18:0+events_unbound08:58:122
24320997655,16cyclictest17710-21kworker/u18:0+events_unbound08:58:122
24320997651,16cyclictest27966-21systemd-journal11:33:062
24320997623,22cyclictest24700-21ssh10:48:542
24320997617,24cyclictest0-21swapper/209:18:232
24320997614,25cyclictest0-21swapper/209:44:152
24320997614,25cyclictest0-21swapper/209:44:152
24310997660,9cyclictest0-21swapper/007:33:400
24310997650,18cyclictest25181-21switchtime07:03:470
2431099763,37cyclictest16469-21fschecks_time09:33:340
24320997551,16cyclictest24298-21cyclictest10:28:502
24320997546,21cyclictest30867-21fschecks_count08:58:352
24320997513,30cyclictest0-21swapper/210:18:292
24320997513,26cyclictest0-21swapper/209:39:102
24310997560,9cyclictest8581-21ssh12:14:190
24310997550,17cyclictest4214-21sed12:08:160
24320997454,13cyclictest356-21runrttasks07:03:292
24320997419,23cyclictest0-21swapper/212:08:472
24320997417,24cyclictest0-21swapper/208:18:372
24320997417,24cyclictest0-21swapper/208:18:372
24320997415,26cyclictest0-21swapper/211:33:262
24320997412,27cyclictest0-21swapper/209:58:282
24314997446,18cyclictest5285-21latency_hist07:43:161
24310997450,17cyclictest14356-21ssh10:28:500
24310997435,29cyclictest0-21swapper/007:58:160
24320997323,19cyclictest29571-21cstates10:58:312
24320997319,31cyclictest0-21swapper/209:38:072
24310997356,11cyclictest1337-21kworker/u17:0+xprtiod11:18:150
24310997353,14cyclictest5246-21latency_hist07:43:150
24310997351,16cyclictest11626-21apt-key08:03:140
24310997343,25cyclictest24298-21cyclictest09:08:060
24310997334,30cyclictest13515-21taskset09:28:260
24310997331,23cyclictest0-21swapper/009:53:260
24320997255,11cyclictest27966-21systemd-journal12:18:122
24320997224,26cyclictest0-21swapper/212:03:232
24314997256,8cyclictest0-21swapper/109:13:121
24310997256,11cyclictest24298-21cyclictest10:08:400
24310997254,12cyclictest24298-21cyclictest12:11:330
24310997248,18cyclictest25058-21latency_hist08:43:150
2431099723,32cyclictest28426-21fschecks_time07:13:360
2431099723,31cyclictest8515-21fschecks_time10:18:350
24310997228,33cyclictest1975-21latency_hist07:33:150
24310997213,28cyclictest11238-21wc11:23:450
24320997147,17cyclictest11705-21ssh10:23:492
2431099713,29cyclictest27656-21cstates09:53:340
24310997118,18cyclictest31061-21grep09:58:500
24310997112,19cyclictest18779-21cron11:38:120
24320997051,11cyclictest24348-21kworker/u18:0+events_unbound10:13:142
24320997050,12cyclictest0-21swapper/210:48:142
24310997049,11cyclictest0-21swapper/009:08:390
24310997048,15cyclictest24298-21cyclictest08:44:490
24310997026,34cyclictest19352-21fschecks_count11:38:330
24310997026,34cyclictest19352-21fschecks_count11:38:330
2432899698,55cyclictest0-21swapper/311:49:203
24320996929,30cyclictest4029-21tune2fs07:38:372
24320996919,29cyclictest0-21swapper/207:59:512
24310996948,14cyclictest18302-21idleruntime-cro08:23:120
24310996948,14cyclictest18302-21idleruntime-cro08:23:120
24310996948,11cyclictest0-21swapper/012:18:520
24310996936,24cyclictest0-21swapper/012:23:230
24320996827,31cyclictest20903-21sensors_temp08:28:462
24320996827,31cyclictest20903-21sensors_temp08:28:462
24310996850,9cyclictest0-21swapper/008:58:520
2431099683,22cyclictest8365-21if_eth011:18:400
2431099683,22cyclictest8365-21if_eth011:18:400
24310996830,25cyclictest22039-21gpgv08:33:170
24310996817,30cyclictest22452-21cat11:43:400
2432899679,34cyclictest0-21swapper/308:33:493
24320996729,25cyclictest5954-21/usr/sbin/munin07:43:452
24320996715,22cyclictest0-21swapper/211:03:262
2431099675,27cyclictest21862-21latency_hist08:33:150
24310996749,12cyclictest21933-21ssh10:43:370
24310996749,11cyclictest0-21swapper/008:48:360
24310996747,14cyclictest28028-21/usr/sbin/munin11:53:370
24310996710,25cyclictest5005-21grep11:13:140
24320996631,26cyclictest13694-21/usr/sbin/munin08:08:462
24310996636,22cyclictest16776-21latency_hist08:18:150
2431099663,31cyclictest6792-21latency_hist07:48:150
24310996624,31cyclictest331-21dbus-daemon07:53:140
24310996611,10cyclictest0-21swapper/010:08:150
2432099654,39cyclictest356-21runrttasks08:13:262
2431499652,43cyclictest22366-21fschecks_count08:34:321
2431099652,34cyclictest19958-21cron08:28:120
2431099652,34cyclictest19958-21cron08:28:120
2432899589,27cyclictest0-21swapper/308:44:073
2432899572,38cyclictest29079-21chrt07:13:493
2432899563,38cyclictest1279-21chrt12:03:153
24328995611,33cyclictest0-21swapper/309:43:353
24328995611,33cyclictest0-21swapper/309:43:353
2432899554,39cyclictest5863-21taskset10:13:353
2432899553,38cyclictest30912-21chrt08:58:353
2432899544,39cyclictest4664-21taskset09:12:053
2432899544,35cyclictest0-21swapper/309:38:333
2432899544,33cyclictest17071-21chrt09:33:493
2432899542,42cyclictest6976-21chrt12:13:123
2432899533,36cyclictest13506-21chrt12:23:453
2432899532,37cyclictest27540-21chrt10:53:593
2432899532,36cyclictest635-21taskset09:03:583
2432899532,36cyclictest23778-21chrt08:38:303
2432899532,36cyclictest11965-21chrt11:24:463
2432899532,35cyclictest24936-21chrt11:48:163
2432899529,33cyclictest29096-21chrt08:54:023
2432899529,33cyclictest29096-21chrt08:54:023
2432899524,38cyclictest13515-21taskset09:28:263
2432899523,39cyclictest7156-21taskset11:16:393
2432899523,37cyclictest18776-21chrt12:33:143
2432899523,37cyclictest15976-21chrt10:33:123
2432899523,36cyclictest3885-21chrt07:38:323
2432899523,36cyclictest31532-21chrt11:02:133
2432899523,34cyclictest7318-21taskset07:48:363
2432899522,37cyclictest14812-21chrt11:29:483
2432899514,36cyclictest0-21swapper/308:03:403
2432899513,36cyclictest16316-21chrt11:33:293
2432899513,34cyclictest6061-21chrt09:13:473
2432899513,33cyclictest10469-21chrt11:23:163
2432899513,33cyclictest10469-21chrt11:23:163
2432899512,39cyclictest23622-21taskset10:47:513
2432899504,35cyclictest9091-21chrt07:53:433
2432899503,37cyclictest5445-21chrt07:43:303
2432899503,37cyclictest10538-21taskset07:58:373
2432899503,36cyclictest17682-21chrt08:18:503
2432899503,36cyclictest17682-21chrt08:18:503
2432899503,34cyclictest2935-21chrt10:08:293
2432899503,33cyclictest9448-21chrt10:19:193
2432899503,33cyclictest15140-21chrt08:13:153
2432899503,32cyclictest21905-21taskset08:33:163
2432899503,32cyclictest21905-21taskset08:33:163
2432899503,25cyclictest30085-21chrt11:56:353
2432899502,35cyclictest1068-21taskset07:28:443
2432899493,37cyclictest1059-21chrt11:04:443
2432899493,35cyclictest25324-21chrt09:48:403
2432899492,37cyclictest4588-21chrt11:12:143
2432899492,37cyclictest12449-21chrt09:25:423
2432899492,37cyclictest12449-21chrt09:25:423
2432899492,34cyclictest11268-21chrt12:18:513
2432899484,30cyclictest1597-21chrt10:04:403
2432899483,31cyclictest16673-21chrt10:33:373
2432899482,33cyclictest26852-21chrt07:08:393
2432899482,31cyclictest30381-21chrt09:58:373
2432899474,34cyclictest19303-21taskset08:23:483
2432899474,34cyclictest19303-21taskset08:23:483
2432899473,33cyclictest32168-21taskset07:23:503
2432899473,33cyclictest27550-21chrt09:53:303
2432899473,31cyclictest19471-21chrt11:38:353
2432899473,31cyclictest19471-21chrt11:38:353
2432899472,34cyclictest27196-21chrt08:48:313
2432899472,33cyclictest11711-21chrt10:23:483
2432899462,35cyclictest2707-21chrt12:04:213
2432899462,34cyclictest24961-21chrt07:03:413
2432899462,32cyclictest24070-21chrt10:48:283
2432899462,31cyclictest21183-21chrt10:42:443
2432899462,31cyclictest21183-21chrt10:42:443
2432899452,32cyclictest30754-21chrt07:18:523
2432899423,30cyclictest7492-21taskset12:13:323
2432899402,25cyclictest2996-21chrt07:33:503
2432899373,23cyclictest16135-21chrt08:13:483
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional