You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-14 - 02:36
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Thu Nov 13, 2025 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
74799910942,58cyclictest17965-21latency_hist09:15:000
74799910942,58cyclictest17965-21latency_hist09:15:000
2987521070,3sleep00-21swapper/008:15:150
7401210511,66sleep17415-21latency_hist07:10:001
7329210412,51sleep27365-21munin-run07:09:572
74929910365,32cyclictest12570-21grep10:00:332
74929910262,34cyclictest3793-21/usr/sbin/munin10:40:342
7479991023,95cyclictest0-21swapper/009:20:130
7479991023,62cyclictest8210-21seq10:46:510
7479991022,59cyclictest9875-21seq08:53:020
74799910045,47cyclictest12469-21sh10:55:150
747999994,81cyclictest0-21swapper/009:45:130
7479999943,47cyclictest21343-21ssh11:10:160
7479999931,62cyclictest15350irq/292-2188000.ethernet10:20:110
747999984,87cyclictest0-21swapper/010:15:130
7479999839,37cyclictest316-21rs:main0
7479999837,54cyclictest0-21swapper/009:04:580
747999982,66cyclictest2113-21runrttasks11:32:300
747999982,66cyclictest2113-21runrttasks11:32:300
7492999781,11cyclictest7470-21cyclictest10:30:572
7479999741,49cyclictest0-21swapper/009:40:340
7479999736,54cyclictest0-21swapper/012:10:350
7479999637,51cyclictest0-21swapper/009:10:010
7479999637,51cyclictest0-21swapper/009:10:010
7479999634,53cyclictest0-21swapper/009:16:570
287672950,7sleep10-21swapper/109:32:581
749299944,77cyclictest3042-21ssh09:45:132
747999945,81cyclictest24381-21ssh09:25:370
747999945,81cyclictest24381-21ssh09:25:370
7479999438,50cyclictest0-21swapper/009:35:150
747999942,82cyclictest5903-21apt-get09:50:080
747999933,81cyclictest0-21swapper/008:40:150
747999933,81cyclictest0-21swapper/008:40:150
747999933,80cyclictest0-21swapper/007:45:130
7479999334,51cyclictest0-21swapper/007:30:320
747999932,84cyclictest6144-21find12:35:080
747999932,84cyclictest6144-21find12:35:080
749299923,81cyclictest0-21swapper/207:45:102
747999923,81cyclictest18325-21apt-get11:05:100
747999923,74cyclictest18037-21apt-get07:40:090
747999923,74cyclictest18037-21apt-get07:40:090
7479999235,51cyclictest0-21swapper/007:51:080
7479999235,51cyclictest0-21swapper/007:51:080
7479999230,54cyclictest18214-21latency_hist11:05:000
7479999230,54cyclictest18214-21latency_hist11:05:000
7492999148,36cyclictest0-21swapper/211:40:252
7492999141,41cyclictest0-21swapper/211:35:342
747999913,82cyclictest14906-21diskmemload12:25:390
7479999136,48cyclictest10619-21ssh11:46:380
7479999136,48cyclictest10619-21ssh11:46:380
747999913,59cyclictest18855-21ssh12:00:370
7479999133,53cyclictest0-21swapper/010:50:260
7479999133,53cyclictest0-21swapper/010:50:260
747999912,80cyclictest7999-21tail08:45:360
749299903,74cyclictest0-21swapper/209:05:142
749299903,74cyclictest0-21swapper/209:05:142
747999903,75cyclictest0-21swapper/007:35:110
7487998943,36cyclictest3459-21needreboot12:30:241
7479998967,15cyclictest2725-21kworker/u18:1-rpciod10:10:000
7479998941,42cyclictest26056-21cron08:04:570
747999893,76cyclictest0-21swapper/008:55:120
747999893,76cyclictest0-21swapper/008:05:120
747999893,58cyclictest18183-21time-dir10:08:320
7479998933,49cyclictest0-21swapper/012:20:160
747999892,80cyclictest0-21swapper/008:35:110
747999892,80cyclictest0-21swapper/008:35:110
7492998846,35cyclictest0-21swapper/212:00:112
7492998841,40cyclictest0-21swapper/212:35:332
7492998841,40cyclictest0-21swapper/212:35:332
749299883,71cyclictest0-21swapper/208:40:152
749299883,71cyclictest0-21swapper/208:40:152
7479998861,16cyclictest14711-21latency_hist11:54:590
7479998843,39cyclictest6827-21ssh11:40:270
747999883,78cyclictest27039-21find11:20:080
7479998830,53cyclictest0-21swapper/007:29:570
747999882,80cyclictest24164-21ssh11:15:110
747999882,78cyclictest0-21swapper/011:55:110
747999882,58cyclictest29347-21seq10:27:050
747999882,57cyclictest29398-21seq08:13:370
7492998770,10cyclictest0-21swapper/207:20:192
7492998770,10cyclictest0-21swapper/207:20:192
748799873,49cyclictest24223-21apt-get09:25:131
748799873,49cyclictest24223-21apt-get09:25:131
747999873,54cyclictest13288-21seq10:02:200
7479998729,52cyclictest9736-21unixbench_multi09:55:340
7492998660,15cyclictest0-21swapper/212:35:002
7492998647,33cyclictest29022-21sh11:23:072
749299863,72cyclictest0-21swapper/210:35:132
749299863,72cyclictest0-21swapper/210:35:132
7487998658,22cyclictest18798-21timerandwakeup07:40:331
7487998658,22cyclictest18798-21timerandwakeup07:40:331
7487998613,48cyclictest0-21swapper/109:20:201
747999864,70cyclictest7483-21wc07:10:010
747999863,77cyclictest0-21swapper/011:25:150
747999863,75cyclictest355-21runrttasks11:35:130
747999863,74cyclictest0-21swapper/012:15:120
7492998563,12cyclictest0-21swapper/212:10:312
7492998550,28cyclictest292-21dbus-daemon10:07:292
749299853,76cyclictest0-21swapper/209:20:132
747999854,69cyclictest30606-21apt-get10:30:100
747999853,77cyclictest0-21swapper/007:20:100
747999853,77cyclictest0-21swapper/007:20:100
747999853,75cyclictest31486-21cpuspeed08:20:130
747999853,75cyclictest28951-21rm09:33:310
747999853,74cyclictest0-21swapper/010:40:090
7479998532,43cyclictest0-21swapper/007:55:140
749299844,73cyclictest0-21swapper/208:05:122
749299843,74cyclictest0-21swapper/210:20:132
749299843,71cyclictest21733-21ssh10:15:132
749299842,72cyclictest0-21swapper/207:10:122
749299842,72cyclictest0-21swapper/207:10:122
7492998368,8cyclictest0-21swapper/211:45:312
7492998368,8cyclictest0-21swapper/211:45:312
7492998342,34cyclictest0-21swapper/211:15:292
749299833,71cyclictest0-21swapper/208:55:102
749299832,68cyclictest0-21swapper/208:45:122
7487998368,10cyclictest7470-21cyclictest09:35:321
7487998315,30cyclictest0-21swapper/109:10:421
7487998315,30cyclictest0-21swapper/109:10:421
7492998267,8cyclictest0-21swapper/209:50:132
7492998265,10cyclictest0-21swapper/211:25:292
7492998240,35cyclictest0-21swapper/208:50:092
749299823,73cyclictest0-21swapper/209:10:252
749299823,73cyclictest0-21swapper/209:10:252
7487998248,8cyclictest181rcu_preempt07:45:311
7487998215,29cyclictest0-21swapper/111:10:301
747999823,68cyclictest577-21apt-get08:25:100
747999822,74cyclictest1483-21netstat10:35:250
747999822,74cyclictest1483-21netstat10:35:250
749299814,69cyclictest0-21swapper/211:30:092
749299814,69cyclictest0-21swapper/211:30:092
7492998140,34cyclictest0-21swapper/208:35:212
7492998140,34cyclictest0-21swapper/208:35:212
7492998139,34cyclictest0-21swapper/211:14:572
7492998138,36cyclictest0-21swapper/209:35:162
749299813,72cyclictest0-21swapper/207:30:082
749299813,61cyclictest14862-21ssh11:55:112
7487998122,25cyclictest7427-21seq09:51:581
7479998128,47cyclictest0-21swapper/007:15:070
7479998128,45cyclictest0-21swapper/012:09:590
7492998064,9cyclictest0-21swapper/212:05:352
7492998057,13cyclictest0-21swapper/207:35:202
749299804,71cyclictest0-21swapper/209:00:122
7492998037,36cyclictest0-21swapper/211:05:362
7492998037,36cyclictest0-21swapper/211:01:572
7492998037,36cyclictest0-21swapper/211:01:572
7492998026,47cyclictest13627-21ls07:25:392
7487998021,28cyclictest24672-21grep10:20:141
747999803,64cyclictest2191-21apt-get08:30:100
70302805,14sleep30-21swapper/307:06:033
7492997961,8cyclictest0-21swapper/210:10:342
7492997957,10cyclictest0-21swapper/208:30:002
7492997957,10cyclictest0-21swapper/208:20:182
7487997951,20cyclictest9133-21kernelversion09:55:231
747999792,66cyclictest7920-21cut07:10:220
747999792,66cyclictest7920-21cut07:10:220
7492997858,10cyclictest0-21swapper/207:55:002
7492997856,12cyclictest0-21swapper/211:50:192
7492997856,10cyclictest0-21swapper/208:00:002
7492997854,18cyclictest11852-21sh10:54:102
7492997854,18cyclictest11852-21sh10:54:102
7492997837,34cyclictest0-21swapper/210:55:302
749299783,69cyclictest0-21swapper/209:40:082
7492997762,8cyclictest0-21swapper/210:45:182
7492997754,12cyclictest0-21swapper/209:30:012
7487997756,15cyclictest30251-21sh11:25:231
748799773,45cyclictest6428-21diskstats12:35:301
748799773,45cyclictest6428-21diskstats12:35:301
7487997732,29cyclictest24122-21sort11:15:071
7487997714,28cyclictest0-21swapper/110:50:311
7487997714,28cyclictest0-21swapper/110:50:311
7487997714,27cyclictest6465-21ssh10:44:591
749699764,56cyclictest30125-21taskset08:15:223
7492997658,9cyclictest0-21swapper/208:19:592
7492997656,10cyclictest0-21swapper/209:34:592
749299763,64cyclictest0-21swapper/208:10:192
749299762,66cyclictest0-21swapper/207:40:122
749299762,66cyclictest0-21swapper/207:40:122
7487997653,16cyclictest309-21systemd-logind09:09:581
7487997653,16cyclictest309-21systemd-logind09:09:581
7487997614,30cyclictest8597-21cat08:49:591
749699752,41cyclictest21534-21taskset11:10:203
7492997559,9cyclictest0-21swapper/209:25:392
749299753,67cyclictest0-21swapper/210:25:232
749299752,62cyclictest24723-21apt-get08:00:092
7487997550,18cyclictest7962-21idleruntime07:10:231
7487997550,18cyclictest7962-21idleruntime07:10:231
748799753,36cyclictest18506-21apt-get09:15:131
7487997531,33cyclictest0-21swapper/108:00:091
7492997459,8cyclictest0-21swapper/208:34:592
7492997458,9cyclictest18960-21kworker/u17:3-flush-179:012:15:072
7492997458,8cyclictest0-21swapper/207:54:562
749299744,64cyclictest0-21swapper/207:15:122
749299743,66cyclictest0-21swapper/209:55:102
7487997417,21cyclictest355-21runrttasks12:18:591
7487997416,16cyclictest4100-21uptime09:45:361
749699734,55cyclictest12992-21taskset10:01:263
749699733,57cyclictest27553-21chrt12:16:063
749299732,63cyclictest0-21swapper/209:15:312
7487997352,15cyclictest18629-21sh11:05:221
7487997341,24cyclictest27736-21apt-get10:25:231
7487997321,21cyclictest16027-21seq10:06:331
7487997318,23cyclictest355-21runrttasks08:36:251
7487997318,23cyclictest355-21runrttasks08:36:251
749699724,35cyclictest18533-21chrt09:15:153
749699723,36cyclictest15244-21chrt11:55:233
749699722,38cyclictest802-21chrt12:25:553
749299723,63cyclictest0-21swapper/212:25:412
7492997211,52cyclictest29359-21awk12:20:112
7487997254,13cyclictest14906-21diskmemload10:10:301
7487997248,15cyclictest17395-21cron11:59:561
7487997247,20cyclictest7470-21cyclictest11:50:191
7487997244,21cyclictest13076-21sh10:55:311
7487997237,24cyclictest1089-21timerwakeupswit09:40:331
7487997213,24cyclictest14906-21diskmemload11:21:261
7487997152,14cyclictest7470-21cyclictest11:00:151
7487997152,14cyclictest7470-21cyclictest11:00:151
7487997112,26cyclictest0-21swapper/107:19:421
749699704,34cyclictest20570-21taskset10:12:083
749699703,35cyclictest4834-21chrt12:32:483
749699702,34cyclictest31349-21chrt12:23:193
7487997042,19cyclictest6761-21sh11:40:251
7487997036,24cyclictest12930-21gpgv07:25:001
7487997036,24cyclictest12930-21gpgv07:25:001
7487997034,27cyclictest3302-21apt-get11:35:081
749699693,39cyclictest520-21chrt11:30:143
749699693,39cyclictest520-21chrt11:30:143
749699693,34cyclictest4421-21chrt10:40:293
749699693,34cyclictest15808-21taskset07:30:553
749699693,34cyclictest14522-21taskset11:54:543
748799699,25cyclictest0-21swapper/110:01:591
748799698,27cyclictest0-21swapper/107:25:281
7487996942,18cyclictest354-21apt-get11:30:121
7487996942,18cyclictest354-21apt-get11:30:121
7487996939,21cyclictest32176-21threads08:20:311
7487996932,26cyclictest23491-21apt-get12:10:131
7487996917,22cyclictest355-21runrttasks12:06:261
7487996911,24cyclictest14906-21diskmemload10:39:161
7487996911,24cyclictest14906-21diskmemload10:39:161
749699683,33cyclictest28586-21taskset09:32:073
749699683,29cyclictest6902-21chrt08:44:593
749699683,29cyclictest6902-21chrt08:44:593
7487996848,14cyclictest7470-21cyclictest08:54:591
7487996847,15cyclictest31139-21idleruntime-cro08:19:571
7487996839,20cyclictest29245-21apt-get12:20:081
7487996838,19cyclictest11942-21apt-get09:00:111
749699674,28cyclictest21679-21chrt07:50:523
749699674,28cyclictest21679-21chrt07:50:523
748799678,26cyclictest0-21swapper/107:55:131
7487996751,11cyclictest355-21runrttasks08:40:321
7487996751,11cyclictest355-21runrttasks08:40:321
7487996745,12cyclictest0-21swapper/110:30:311
7487996742,19cyclictest22441-21mailstats10:15:341
749699663,29cyclictest12265-21chrt09:00:223
749699663,24cyclictest18134-21chrt12:00:213
7487996651,8cyclictest0-21swapper/112:25:581
7487996648,13cyclictest7470-21cyclictest08:59:591
7487996643,15cyclictest1063-21ntp_kernel_err08:25:261
7487996630,25cyclictest6670-21apt-get10:45:091
749699659,46cyclictest0-21swapper/311:35:133
749699653,30cyclictest1144-21chrt10:35:183
749699653,30cyclictest1144-21chrt10:35:183
749699653,28cyclictest6965-21chrt12:35:443
749699653,28cyclictest6965-21chrt12:35:443
749699653,28cyclictest13981-21chrt07:25:493
749699653,25cyclictest22274-21chrt10:15:283
749699653,24cyclictest9826-21chrt11:45:303
749699653,24cyclictest9826-21chrt11:45:303
749699653,23cyclictest21138-21chrt07:49:593
749699653,23cyclictest12945-21taskset07:25:003
749699653,23cyclictest12945-21taskset07:25:003
749699652,32cyclictest6911-21taskset10:45:203
749699652,25cyclictest8856-21chrt08:50:173
749699652,25cyclictest15717-21chrt09:10:303
749699652,25cyclictest15717-21chrt09:10:303
7487996544,16cyclictest28970-21threads08:11:041
186202650,9sleep3451rcuc/310:09:533
749699644,8cyclictest0-21swapper/311:15:113
749699644,26cyclictest32302-21chrt08:20:323
749699643,50cyclictest29365-21chrt10:27:083
749699643,29cyclictest30621-21chrt10:30:123
749699643,28cyclictest24243-21taskset09:25:133
749699643,28cyclictest24243-21taskset09:25:133
749699643,25cyclictest27721-21chrt11:20:303
749699643,24cyclictest3526-21chrt09:45:253
749699643,22cyclictest5226-21taskset08:39:563
749699643,22cyclictest5226-21taskset08:39:563
749699643,22cyclictest23746-21chrt07:55:303
749699642,27cyclictest11103-21chrt07:19:583
749699642,24cyclictest26630-21chrt10:23:153
749699642,24cyclictest11697-21chrt08:59:573
749699642,23cyclictest8752-21chrt09:55:133
748799649,25cyclictest0-21swapper/111:46:341
748799649,25cyclictest0-21swapper/111:46:341
7487996441,13cyclictest0-21swapper/107:30:171
7487996440,12cyclictest16403-21apt-get07:35:111
749699633,28cyclictest29898-21chrt11:25:083
749699633,27cyclictest1352-21chrt09:40:513
749699633,24cyclictest25170-21chrt08:00:243
749699632,27cyclictest30164-21chrt09:35:243
749699632,27cyclictest29549-21taskset08:14:573
749699632,27cyclictest2635-21chrt08:30:253
749699632,23cyclictest8588-21chrt11:44:123
749699632,23cyclictest16863-21chrt11:01:493
749699632,23cyclictest16863-21chrt11:01:493
7487996327,26cyclictest20980-21apt-get07:50:101
7487996327,26cyclictest20980-21apt-get07:50:101
7487996327,25cyclictest2444-21cut08:30:211
749699623,26cyclictest7677-21chrt07:10:133
749699623,26cyclictest7677-21chrt07:10:133
749699623,24cyclictest16546-21chrt07:35:173
749699623,23cyclictest22157-21chrt09:20:343
749699622,26cyclictest20771-21chrt12:05:163
749699622,23cyclictest10183-21chrt10:50:263
749699622,23cyclictest10183-21chrt10:50:263
748799628,20cyclictest19425-21ssh12:02:161
7487996227,27cyclictest26621-21/usr/sbin/munin08:05:201
749699615,47cyclictest0-21swapper/308:25:193
749699613,22cyclictest8249-21chrt09:54:323
749699612,25cyclictest14700-21taskset10:59:013
749699607,44cyclictest0-21swapper/312:10:193
749699603,26cyclictest18254-21chrt11:05:013
749699603,26cyclictest14535-21chrt09:05:363
749699603,26cyclictest14535-21chrt09:05:363
749699602,22cyclictest18041-21chrt07:40:113
749699602,22cyclictest18041-21chrt07:40:113
749699602,21cyclictest8400-21chrt08:49:453
749699585,8cyclictest0-21swapper/308:05:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional