You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 15:48
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Wed Nov 26, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
102409912762,58cyclictest20670-21apt-key07:39:060
1027599125107,12cyclictest10217-21cyclictest11:09:242
10090212414,79sleep110125-21wc07:09:091
10090212414,79sleep110125-21wc07:09:091
102409912145,48cyclictest13249-21cstates08:54:240
10240991183,97cyclictest0-21swapper/009:14:190
10240991156,99cyclictest8366-21apt-get11:49:200
102409911388,17cyclictest314-21in:imuxsock12:30:310
10240991054,74cyclictest1614-21apt-get12:34:210
10240991023,92cyclictest0-21swapper/010:49:210
10240991023,92cyclictest0-21swapper/010:49:210
933521019,61sleep28622-21/usr/sbin/munin07:04:502
933521019,61sleep28622-21/usr/sbin/munin07:04:502
10251991014,34cyclictest25580-21ssh10:24:181
102409910137,58cyclictest0-21swapper/011:04:070
102409910137,58cyclictest0-21swapper/011:04:070
10275999977,16cyclictest10217-21cyclictest11:14:292
10251999939,14cyclictest22582-21latency_hist10:19:081
1024099992,87cyclictest0-21swapper/008:19:200
10240999920,53cyclictest355-21runrttasks10:01:140
10251999852,39cyclictest23122-21fschecks_count10:19:281
10251999852,39cyclictest23122-21fschecks_count10:19:281
1024099983,86cyclictest24135-21/usr/sbin/munin11:19:400
1025199973,33cyclictest27303-21kernelversion11:24:321
1025199973,33cyclictest27303-21kernelversion11:24:321
10240999740,52cyclictest0-21swapper/012:14:330
10240999739,51cyclictest0-21swapper/010:55:360
1024099973,85cyclictest9353-21ssh09:54:170
1024099973,82cyclictest3473-21apt-get08:24:180
1024099973,71cyclictest18646-21rm11:09:210
10240999724,46cyclictest0-21swapper/009:33:400
10240999642,50cyclictest10217-21cyclictest11:35:080
1024099962,83cyclictest0-21swapper/011:24:210
1024099962,83cyclictest0-21swapper/011:24:210
183902950,6sleep30-21swapper/307:29:403
183902950,6sleep30-21swapper/307:29:403
1025199953,44cyclictest3337-21latency_hist08:24:081
10251999533,10cyclictest0-21swapper/111:29:391
10251999533,10cyclictest0-21swapper/111:29:391
10251999529,13cyclictest0-21swapper/110:49:301
10251999529,13cyclictest0-21swapper/110:49:301
1024099953,83cyclictest0-21swapper/008:39:210
1024099953,79cyclictest28045-21latency_hist10:29:080
10275999471,16cyclictest0-21swapper/209:04:052
10275999420,42cyclictest4803-21sh11:42:482
1025199943,44cyclictest12961-21latency07:14:341
1025199943,40cyclictest32177-21grep10:34:421
1025199943,40cyclictest32177-21grep10:34:421
1024099943,64cyclictest30110-21/usr/sbin/munin11:29:320
1024099943,64cyclictest30110-21/usr/sbin/munin11:29:320
1025199934,34cyclictest26898-21switchtime09:24:441
10251999335,9cyclictest0-21swapper/112:10:131
1025199933,41cyclictest17351-21find10:09:091
1025199933,37cyclictest17553-21grep12:04:391
1025199933,37cyclictest17553-21grep12:04:391
1027599923,66cyclictest16054-21apt-get07:24:322
1025199925,32cyclictest17466-21/usr/sbin/munin10:10:311
1025199924,33cyclictest18078-21apt-get11:09:181
10251999238,8cyclictest0-21swapper/109:59:211
1024099923,79cyclictest22920-21apt-get10:19:200
1024099923,79cyclictest22920-21apt-get10:19:200
1024099923,76cyclictest18271-21grep11:09:070
1024099923,76cyclictest18271-21grep11:09:070
10240999233,52cyclictest0-21swapper/011:14:300
10275999152,32cyclictest17883-21latency_hist09:09:082
10275999152,32cyclictest17883-21latency_hist09:09:082
1025199915,33cyclictest29151-21xargs12:24:401
1025199914,35cyclictest18075-21latency_hist11:09:081
1025199914,35cyclictest18075-21latency_hist11:09:081
10251999135,9cyclictest0-21swapper/111:34:411
1025199913,35cyclictest31411-21grep12:29:301
1024099914,50cyclictest0-21swapper/010:45:100
1024099914,50cyclictest0-21swapper/010:45:100
10240999136,49cyclictest0-21swapper/007:09:370
10240999136,49cyclictest0-21swapper/007:09:370
10240999130,53cyclictest0-21swapper/009:44:240
10240999130,51cyclictest0-21swapper/012:26:390
1024099912,77cyclictest0-21swapper/009:19:200
10240999121,58cyclictest0-21swapper/010:14:080
10240999120,62cyclictest14629-21ssh10:04:090
1027599904,75cyclictest0-21swapper/210:19:202
1027599904,75cyclictest0-21swapper/210:19:202
1027599902,75cyclictest0-21swapper/212:04:212
1027599902,75cyclictest0-21swapper/212:04:212
1025199904,36cyclictest17299-21grep09:04:411
1025199904,36cyclictest17299-21grep09:04:411
1025199904,35cyclictest20615-21latency_hist07:39:091
10251999043,40cyclictest314-21in:imuxsock07:55:081
1025199904,33cyclictest6295-21ssh11:44:391
1025199903,37cyclictest25439-21/usr/sbin/munin12:19:401
1025199903,32cyclictest25846-21latency_hist09:24:071
10251999032,17cyclictest316-21rs:main1
10251999029,13cyclictest0-21swapper/107:48:321
10251999029,13cyclictest0-21swapper/107:48:321
1025199902,37cyclictest8949-21fschecks_time11:50:171
1024099903,57cyclictest17754-21diskmemload11:54:400
10240999033,48cyclictest0-21swapper/010:34:330
10240999033,48cyclictest0-21swapper/010:34:330
1024099902,75cyclictest0-21swapper/008:34:200
1024099902,61cyclictest8174-21ssh09:51:240
1024099902,60cyclictest16336-21taskset12:03:310
1025199895,32cyclictest12880-21/usr/sbin/munin10:59:291
1025199895,32cyclictest12880-21/usr/sbin/munin10:59:291
1025199893,36cyclictest355-21runrttasks08:40:041
1025199893,36cyclictest25058-21grep11:19:431
1025199893,36cyclictest13658-21ntp_kernel_pll_08:54:351
1025199892,36cyclictest32556-21latency_hist08:14:101
10251998914,32cyclictest4363-21latency_hist12:39:071
1025199890,39cyclictest18391-21switchtime07:29:401
1025199890,39cyclictest18391-21switchtime07:29:401
1024099895,78cyclictest0-21swapper/007:24:210
1024099893,75cyclictest19682-21ssh12:09:100
1024099893,65cyclictest31550-21ssh09:34:270
1024099893,60cyclictest32667-21apt-get08:14:190
10240998934,48cyclictest0-21swapper/012:19:430
947228812,52sleep09556-21unixbench_multi07:04:520
947228812,52sleep09556-21unixbench_multi07:04:520
1027599883,73cyclictest0-21swapper/209:24:222
1027599883,72cyclictest0-21swapper/207:44:192
1027599883,72cyclictest0-21swapper/207:44:192
1025199884,34cyclictest30994-21sh09:34:041
1025199883,39cyclictest32350-21grep09:34:431
1025199883,38cyclictest355-21runrttasks07:43:551
1025199883,37cyclictest5076-21wc10:44:411
1025199883,37cyclictest5076-21wc10:44:411
1025199883,36cyclictest3881-21seq11:39:511
10240998868,14cyclictest28575-21latency_hist09:29:080
1024099883,75cyclictest0-21swapper/009:39:190
10240998831,51cyclictest0-21swapper/012:04:420
10240998831,51cyclictest0-21swapper/012:04:420
10240998830,51cyclictest0-21swapper/008:44:430
10240998830,49cyclictest0-21swapper/010:29:360
10275998745,37cyclictest10217-21cyclictest11:23:222
1027599872,72cyclictest0-21swapper/208:14:382
1025199874,36cyclictest27774-21fschecks_count07:59:281
1025199874,35cyclictest14488-21fschecks_time07:19:301
1025199874,35cyclictest14488-21fschecks_time07:19:301
1025199874,33cyclictest16173-21cpuspeed07:24:571
1025199873,35cyclictest14258-21fschecks_time11:59:271
1025199872,41cyclictest10634-21switchtime08:44:401
1025199872,37cyclictest2378-21grep09:39:431
10251998717,24cyclictest21778-21sshd11:14:311
10240998768,11cyclictest17754-21diskmemload10:14:310
1024099873,75cyclictest21124-21/usr/sbin/munin07:40:530
10240998736,45cyclictest0-21swapper/010:39:390
1029199863,63cyclictest8424-21chrt11:49:263
1025199864,28cyclictest23293-21grep09:19:081
1025199863,37cyclictest368-21snmpd08:24:371
1025199863,35cyclictest10813-21grep07:09:481
1025199863,35cyclictest10813-21grep07:09:481
1025199863,34cyclictest18098-21ssh09:09:241
1024099863,72cyclictest24344-21/usr/sbin/munin07:49:270
1024099863,72cyclictest24344-21/usr/sbin/munin07:49:270
1024099863,69cyclictest0-21swapper/008:29:170
1024099862,74cyclictest12575-21apt-get07:14:200
1024099862,65cyclictest32381-21sed08:14:080
10275998546,34cyclictest10217-21cyclictest09:29:402
1027599853,74cyclictest0-21swapper/208:24:172
1027599852,62cyclictest1652-21sh12:34:222
1025199855,34cyclictest471-21fschecks_time08:14:451
1025199854,35cyclictest30230-21runrttasks10:32:111
1025199854,35cyclictest11669-21latency11:54:321
1025199853,36cyclictest29513-21idleruntime08:04:301
1025199853,34cyclictest6415-21grep09:49:071
1025199852,36cyclictest22608-21ssh12:14:221
1025199852,36cyclictest15026-21fschecks_time08:59:261
1024099853,74cyclictest0-21swapper/009:04:170
1024099853,74cyclictest0-21swapper/009:04:170
1027599843,67cyclictest0-21swapper/209:39:192
1027599842,67cyclictest0-21swapper/209:59:192
1025199844,31cyclictest6582-21latency_hist08:34:081
1025199843,36cyclictest9331-21/usr/sbin/munin09:54:331
1025199843,36cyclictest10932-21ntp_kernel_err10:55:301
1025199843,34cyclictest7523-21cat09:49:421
1024099843,76cyclictest0-21swapper/009:00:460
1024099843,76cyclictest0-21swapper/008:49:470
1024099843,76cyclictest0-21swapper/008:49:470
1024099843,68cyclictest0-21swapper/007:59:170
1024099842,69cyclictest17670-21apt-get07:29:170
1024099842,69cyclictest17670-21apt-get07:29:170
1024099842,66cyclictest14217-21apt-get07:19:190
1024099842,66cyclictest14217-21apt-get07:19:190
10275998359,18cyclictest6944-21fschecks_time09:49:292
10275998355,20cyclictest28590-21fschecks_time12:24:272
10275998336,39cyclictest0-21swapper/207:59:342
10275998335,41cyclictest0-21swapper/210:44:312
10275998335,41cyclictest0-21swapper/210:44:312
10275998334,42cyclictest0-21swapper/210:39:082
10275998334,42cyclictest0-21swapper/210:39:082
1027599832,70cyclictest17833-21kworker/u18:0+flush-179:010:54:192
1025199835,32cyclictest24344-21/usr/sbin/munin07:49:461
1025199835,32cyclictest24344-21/usr/sbin/munin07:49:461
1025199834,35cyclictest355-21runrttasks08:49:211
1025199834,35cyclictest355-21runrttasks08:49:211
1025199833,36cyclictest1485-21/usr/sbin/munin10:39:411
1024099832,74cyclictest3551-21switchtime11:39:410
1024099832,72cyclictest18511-21taskset09:09:330
10275998238,35cyclictest18075-21latency_hist11:09:082
10275998238,35cyclictest18075-21latency_hist11:09:082
1027599823,74cyclictest0-21swapper/209:19:202
1027599822,71cyclictest7086-21ssh10:49:212
1027599822,71cyclictest7086-21ssh10:49:212
1024099822,69cyclictest7995-21apt-config11:49:080
10275998161,9cyclictest0-21swapper/208:59:082
10275998130,37cyclictest0-21swapper/210:44:082
1027599812,68cyclictest0-21swapper/207:34:182
10240998128,45cyclictest0-21swapper/008:04:420
10240998127,46cyclictest0-21swapper/007:44:400
10240998127,46cyclictest0-21swapper/007:44:400
10275998063,8cyclictest0-21swapper/208:34:272
10275998058,14cyclictest0-21swapper/207:14:412
10275998055,17cyclictest1-21systemd12:21:312
10275998055,14cyclictest0-21swapper/209:14:092
1027599802,60cyclictest8416-21ssh11:49:252
1024099802,65cyclictest0-21swapper/007:54:180
10275997958,10cyclictest0-21swapper/210:59:422
10275997958,10cyclictest0-21swapper/210:59:422
1027599793,71cyclictest0-21swapper/210:29:192
1027599793,70cyclictest0-21swapper/211:54:182
10275997936,35cyclictest0-21swapper/208:49:332
10275997936,35cyclictest0-21swapper/208:49:332
1027599783,70cyclictest0-21swapper/208:29:172
1027599783,58cyclictest16516-21cron12:04:042
1027599782,31cyclictest10531-21apt-get07:09:332
1027599782,31cyclictest10531-21apt-get07:09:332
10275997812,60cyclictest26088-21netstat10:24:332
10275997758,9cyclictest0-21swapper/208:14:072
1027599772,67cyclictest0-21swapper/210:14:202
10275997725,40cyclictest0-21swapper/212:33:222
1027599772,31cyclictest18035-21fschecks_time10:10:192
10275997656,12cyclictest0-21swapper/207:19:292
10275997656,12cyclictest0-21swapper/207:19:292
1027599764,67cyclictest0-21swapper/207:29:172
1027599764,67cyclictest0-21swapper/207:29:172
10275997635,34cyclictest0-21swapper/212:09:202
1027599762,67cyclictest32387-21ssh11:34:212
10275997558,10cyclictest0-21swapper/208:09:082
10275997553,15cyclictest0-21swapper/210:09:082
1027599753,60cyclictest9926-21apt-get08:44:172
10275997533,36cyclictest0-21swapper/212:14:412
10275997518,52cyclictest21382-21cat07:40:202
1027599743,7cyclictest0-21swapper/209:54:202
1027599743,7cyclictest0-21swapper/208:19:202
1027599743,66cyclictest0-21swapper/211:29:172
1027599743,66cyclictest0-21swapper/211:29:172
1027599743,63cyclictest0-21swapper/207:54:182
1027599743,61cyclictest25717-21basename07:54:052
1027599743,61cyclictest25717-21basename07:54:052
10275997435,31cyclictest0-21swapper/209:34:222
10275997426,42cyclictest21187-21sort09:14:302
1027599742,34cyclictest4272-21ssh09:44:272
1027599733,58cyclictest8322-21apt-get08:39:192
1027599732,62cyclictest17754-21diskmemload11:24:212
1027599732,62cyclictest17754-21diskmemload11:24:212
1027599732,33cyclictest6079-21ssh11:44:332
1000727315,39sleep30-21swapper/307:09:073
1000727315,39sleep30-21swapper/307:09:073
1029199683,48cyclictest29744-21chrt08:04:353
1029199662,51cyclictest23009-21taskset09:18:363
1029199653,50cyclictest24745-21taskset11:19:363
1029199653,49cyclictest31175-21taskset12:29:253
10291996511,40cyclictest16103-21chrt07:24:463
1029199643,46cyclictest2895-21taskset09:41:053
1029199642,48cyclictest19617-21taskset07:34:303
1029199632,47cyclictest6043-21chrt11:44:323
1029199632,47cyclictest17352-21chrt11:06:393
1029199632,47cyclictest17352-21chrt11:06:393
10291996315,31cyclictest0-21swapper/312:34:233
107202620,8sleep30-21swapper/308:44:423
1029199623,46cyclictest26192-21taskset10:24:373
1029199623,45cyclictest7683-21chrt10:49:373
1029199623,45cyclictest7683-21chrt10:49:373
1029199622,48cyclictest13934-21taskset11:59:113
1029199622,47cyclictest31166-21chrt08:09:293
1029199613,44cyclictest5176-21chrt09:44:463
1029199613,44cyclictest30164-21taskset11:29:333
1029199613,44cyclictest30164-21taskset11:29:333
1029199612,44cyclictest27336-21taskset09:25:293
1029199609,31cyclictest0-21swapper/311:09:193
1029199604,34cyclictest0-21swapper/310:54:193
1029199603,46cyclictest20816-21taskset10:14:363
1029199603,45cyclictest4300-21chrt10:44:183
1029199603,45cyclictest4300-21chrt10:44:183
1029199603,44cyclictest18265-21chrt09:09:283
1029199602,47cyclictest1398-21taskset11:35:573
1029199602,47cyclictest11553-21taskset09:58:563
1029199602,46cyclictest32702-21chrt09:36:033
1029199602,46cyclictest29935-21chrt10:31:183
1029199602,46cyclictest29839-21chrt09:29:593
1029199602,46cyclictest22016-21taskset12:13:293
1029199602,46cyclictest16377-21taskset10:07:113
1029199602,46cyclictest15121-21chrt11:03:273
1029199602,46cyclictest15121-21chrt11:03:273
1029199602,45cyclictest14704-21chrt07:19:353
1029199602,45cyclictest14704-21chrt07:19:353
1029199602,44cyclictest7957-21taskset08:38:253
1029199602,44cyclictest25839-21chrt12:19:293
1029199602,44cyclictest22317-21chrt07:42:033
1029199602,44cyclictest19057-21taskset10:11:243
1029199602,43cyclictest22945-21chrt12:14:303
1029199593,44cyclictest29421-21taskset12:24:483
1029199593,44cyclictest28746-21chrt11:27:103
1029199593,44cyclictest28746-21chrt11:27:103
1029199593,43cyclictest3034-21chrt08:22:013
1029199592,45cyclictest16792-21taskset12:04:093
1029199592,45cyclictest16792-21taskset12:04:093
1029199592,44cyclictest11095-21taskset07:09:553
1029199592,44cyclictest11095-21taskset07:09:553
1029199592,43cyclictest14153-21taskset08:54:543
1029199583,43cyclictest13736-21chrt07:16:523
1029199583,42cyclictest1518-21taskset10:39:203
1029199583,39cyclictest32672-21chrt08:14:193
1029199582,45cyclictest6655-21taskset09:49:203
1029199582,45cyclictest13342-21chrt10:00:393
1029199582,44cyclictest17463-21chrt09:05:393
1029199582,44cyclictest17463-21chrt09:05:393
1029199582,44cyclictest12689-21chrt08:51:463
1029199582,44cyclictest12689-21chrt08:51:463
1029199582,44cyclictest11153-21chrt11:54:133
1029199582,43cyclictest28645-21chrt08:00:183
1029199582,43cyclictest23527-21chrt11:18:103
1029199582,42cyclictest31563-21chrt10:34:303
1029199582,42cyclictest31563-21chrt10:34:303
1029199574,39cyclictest9347-21chrt08:40:123
1029199573,42cyclictest25035-21chrt10:23:283
1029199573,42cyclictest25035-21chrt10:23:283
1029199573,42cyclictest23754-21chrt07:44:513
1029199573,42cyclictest23754-21chrt07:44:513
1029199573,41cyclictest24100-21taskset09:19:373
1029199573,40cyclictest26133-21chrt07:54:273
1029199572,43cyclictest4574-21chrt08:26:103
1029199572,43cyclictest2755-21taskset11:39:173
1029199572,42cyclictest5053-21chrt08:29:103
1029199572,42cyclictest15795-21chrt09:00:003
1029199562,43cyclictest25365-21chrt07:49:533
1029199562,43cyclictest25365-21chrt07:49:533
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional