You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 15:50
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Mon Nov 24, 2025 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
15047212111,79sleep215072-21munin-run07:09:142
152219910885,15cyclictest12744-21latency_hist08:39:172
15221999776,12cyclictest0-21swapper/211:34:342
15213999678,8cyclictest0-21swapper/107:59:391
15221999574,11cyclictest0-21swapper/207:19:382
15221999573,12cyclictest0-21swapper/207:59:362
15221999573,12cyclictest0-21swapper/207:14:382
15213999577,8cyclictest0-21swapper/110:39:361
15213999577,8cyclictest0-21swapper/110:39:361
15213999571,13cyclictest0-21swapper/109:04:501
15221999472,12cyclictest0-21swapper/211:09:372
15221999472,12cyclictest0-21swapper/211:09:372
15203999361,23cyclictest23537-21latency_hist12:09:170
141862938,59sleep014210-21awk07:04:470
15221999272,10cyclictest0-21swapper/210:24:152
15221999270,16cyclictest15855-21switchtime11:54:502
142772928,58sleep114276-21ntp_states07:04:521
1520399903,69cyclictest1009-21apt-get10:29:300
1520399902,53cyclictest23755-21ssh12:09:340
1520399902,53cyclictest23755-21ssh12:09:340
15221998974,7cyclictest0-21swapper/210:49:452
15221998973,7cyclictest0-21swapper/208:09:382
15221998973,7cyclictest0-21swapper/208:09:382
15221998950,10cyclictest181rcu_preempt08:59:292
15221998950,10cyclictest181rcu_preempt08:59:292
15203998950,8cyclictest181rcu_preempt11:04:160
15203998938,17cyclictest207-21systemd-journal12:32:050
1520399886,44cyclictest15-21ksoftirqd/010:24:470
15203998810,32cyclictest15-21ksoftirqd/011:59:470
15221998753,8cyclictest181rcu_preempt09:04:472
15203998712,41cyclictest15-21ksoftirqd/010:22:490
15203998711,50cyclictest15-21ksoftirqd/007:24:360
15221998670,10cyclictest16966-21munin-node07:29:482
1520399862,42cyclictest22731-21awk09:09:310
15221998550,7cyclictest181rcu_preempt07:24:492
15221998548,9cyclictest181rcu_preempt09:39:312
15203998557,19cyclictest16807-21latency_hist07:14:160
15203998557,19cyclictest16807-21latency_hist07:14:160
1520399852,36cyclictest26666-21ssh10:17:170
15221998460,7cyclictest181rcu_preempt11:48:202
15221998449,7cyclictest181rcu_preempt09:12:292
15221998442,7cyclictest181rcu_preempt10:28:202
15213998459,16cyclictest17244-21systemd-run10:59:141
15213998459,16cyclictest17244-21systemd-run10:59:141
1520399847,40cyclictest15-21ksoftirqd/007:53:130
15203998457,7cyclictest181rcu_preempt11:34:550
15203998455,7cyclictest181rcu_preempt08:12:350
15203998455,7cyclictest181rcu_preempt08:12:350
15221998368,9cyclictest17778-21cpu08:54:292
15221998345,10cyclictest181rcu_preempt12:08:412
15203998345,12cyclictest355-21runrttasks08:41:240
1520399833,47cyclictest6807-21fschecks_count10:39:350
1520399833,47cyclictest6807-21fschecks_count10:39:350
15221998246,8cyclictest181rcu_preempt12:22:062
15203998255,11cyclictest181rcu_preempt11:44:490
15203998250,13cyclictest181rcu_preempt08:29:160
15203998243,7cyclictest181rcu_preempt12:18:400
15203998220,12cyclictest0-21swapper/008:01:340
15221998166,9cyclictest15191-21cyclictest12:31:082
15221998162,9cyclictest0-21swapper/210:35:382
15221998155,7cyclictest181rcu_preempt09:50:552
15221998155,7cyclictest181rcu_preempt09:50:552
15221998151,8cyclictest181rcu_preempt11:14:542
15221998151,8cyclictest181rcu_preempt11:14:542
15221998144,7cyclictest181rcu_preempt11:27:312
15213998164,7cyclictest0-21swapper/107:29:391
15203998153,13cyclictest17917-21ssh10:00:450
15203998149,9cyclictest181rcu_preempt10:44:480
15203998142,8cyclictest181rcu_preempt09:54:480
1520399813,25cyclictest161ktimers/009:46:190
15221998053,8cyclictest181rcu_preempt09:24:252
15221998052,8cyclictest181rcu_preempt07:44:312
15221998042,7cyclictest181rcu_preempt08:54:142
15221998041,7cyclictest181rcu_preempt12:00:502
15213998066,8cyclictest0-21swapper/109:59:311
15213998065,9cyclictest3462-21/usr/sbin/munin10:34:141
15203998049,8cyclictest181rcu_preempt08:49:360
15203998046,8cyclictest181rcu_preempt09:17:260
15203998039,11cyclictest181rcu_preempt12:23:350
1520399803,60cyclictest3955-21apt-get10:34:270
15203998032,12cyclictest181rcu_preempt10:08:260
15203998032,12cyclictest181rcu_preempt10:08:260
15203998029,13cyclictest0-21swapper/007:59:170
15203998020,15cyclictest0-21swapper/011:40:220
15221997954,8cyclictest181rcu_preempt09:32:592
15221997951,9cyclictest181rcu_preempt10:46:292
15221997945,7cyclictest181rcu_preempt08:42:242
15221997943,10cyclictest0-21swapper/212:09:352
15221997943,10cyclictest0-21swapper/212:09:352
15203997944,8cyclictest29728-21ssh09:22:420
1520399793,54cyclictest368-21snmpd11:14:450
1520399793,54cyclictest368-21snmpd11:14:450
15203997934,11cyclictest181rcu_preempt11:20:210
15203997922,25cyclictest0-21swapper/007:40:050
15221997863,10cyclictest15191-21cyclictest08:14:472
15221997853,7cyclictest181rcu_preempt10:41:382
15221997853,7cyclictest181rcu_preempt10:41:382
15221997852,8cyclictest181rcu_preempt12:25:162
15221997852,8cyclictest181rcu_preempt12:25:162
15221997850,19cyclictest14229-21kernelversion09:54:392
15221997849,9cyclictest181rcu_preempt11:00:222
15221997837,9cyclictest181rcu_preempt11:52:482
15221997837,15cyclictest18489-21ssh10:02:382
15213997854,14cyclictest0-21swapper/110:24:511
15203997854,18cyclictest11703-21ssh09:49:410
15203997854,18cyclictest11703-21ssh09:49:410
15221997753,7cyclictest181rcu_preempt09:19:372
15221997751,7cyclictest181rcu_preempt12:36:132
15221997750,9cyclictest181rcu_preempt12:15:422
15213997760,9cyclictest0-21swapper/111:29:401
15213997758,9cyclictest0-21swapper/109:34:421
15213997752,15cyclictest368-21snmpd11:24:161
1520399774,37cyclictest15-21ksoftirqd/007:24:170
15203997736,20cyclictest2906-21sh09:33:490
15203997719,22cyclictest0-21swapper/011:13:330
15203997719,22cyclictest0-21swapper/011:13:330
15203997712,39cyclictest15-21ksoftirqd/008:29:410
15203997712,22cyclictest161ktimers/008:04:380
15203997712,22cyclictest161ktimers/008:04:380
15221997647,10cyclictest181rcu_preempt11:32:352
15221997639,6cyclictest181rcu_preempt09:18:552
15221997638,10cyclictest26531-21ssh10:16:422
15221997635,8cyclictest181rcu_preempt10:05:572
15221997635,8cyclictest181rcu_preempt10:05:572
1522199763,38cyclictest22158-21apt-get10:09:502
15213997655,11cyclictest0-21swapper/112:16:121
15203997653,8cyclictest181rcu_preempt07:44:320
15203997652,9cyclictest181rcu_preempt09:24:420
15203997643,7cyclictest181rcu_preempt11:25:200
15203997638,16cyclictest181rcu_preempt08:47:570
15203997631,15cyclictest181rcu_preempt11:55:230
15203997631,11cyclictest0-21swapper/010:58:450
15203997631,11cyclictest0-21swapper/010:58:450
1520399762,22cyclictest5667-21latency_hist09:39:160
15203997610,27cyclictest12476-21fschecks_count11:49:350
15221997532,9cyclictest181rcu_preempt07:39:212
15213997559,10cyclictest5868-21mailstats12:34:481
15213997557,13cyclictest22346-21diskmemload10:10:191
15213997554,16cyclictest4061-21kworker/u18:0+events_unbound10:39:131
15213997554,11cyclictest0-21swapper/112:19:481
15203997547,13cyclictest32649-21ssh12:24:450
15203997547,13cyclictest32649-21ssh12:24:450
15203997537,10cyclictest181rcu_preempt09:04:340
1520399753,51cyclictest22158-21apt-get10:09:490
15203997520,14cyclictest0-21swapper/010:49:510
15203997510,39cyclictest15-21ksoftirqd/008:54:290
15221997450,7cyclictest181rcu_preempt10:55:012
15221997450,7cyclictest181rcu_preempt10:55:012
15221997447,7cyclictest181rcu_preempt09:44:182
15221997439,7cyclictest181rcu_preempt08:04:512
15221997439,7cyclictest181rcu_preempt08:04:512
15221997426,24cyclictest39-21ksoftirqd/209:35:062
15213997456,9cyclictest0-21swapper/109:54:391
15203997446,7cyclictest181rcu_preempt11:29:500
15213997356,8cyclictest0-21swapper/109:24:411
15213997356,8cyclictest0-21swapper/109:16:301
15213997355,10cyclictest0-21swapper/109:44:171
15213997354,9cyclictest0-21swapper/110:24:151
15213997354,13cyclictest12401-21sh09:50:151
15213997354,13cyclictest12401-21sh09:50:151
15213997354,12cyclictest15191-21cyclictest11:59:171
15213997353,10cyclictest0-21swapper/111:04:491
1520399734,41cyclictest15-21ksoftirqd/007:14:200
15203997334,7cyclictest181rcu_preempt12:36:570
15203997327,12cyclictest0-21swapper/007:29:420
1520399732,39cyclictest15-21ksoftirqd/011:05:180
15203997321,9cyclictest0-21swapper/009:39:250
15221997247,17cyclictest15900-21ntp_kernel_pll_07:09:422
15221997247,17cyclictest15900-21ntp_kernel_pll_07:09:422
15221997241,7cyclictest181rcu_preempt11:19:232
15213997253,9cyclictest0-21swapper/108:40:181
1521399723,30cyclictest31-21ksoftirqd/109:30:091
15203997234,12cyclictest181rcu_preempt08:59:290
15203997234,12cyclictest181rcu_preempt08:59:290
15221997150,10cyclictest0-21swapper/207:54:482
15221997147,7cyclictest181rcu_preempt11:04:352
15221997146,9cyclictest181rcu_preempt08:24:452
15221997133,10cyclictest181rcu_preempt08:33:492
1522199712,28cyclictest39-21ksoftirqd/207:36:102
15213997155,9cyclictest0-21swapper/110:14:531
15213997154,7cyclictest0-21swapper/111:09:321
15213997154,7cyclictest0-21swapper/111:09:321
15213997153,8cyclictest0-21swapper/112:24:341
15213997153,8cyclictest0-21swapper/112:24:341
15213997153,8cyclictest0-21swapper/109:46:221
15213997153,8cyclictest0-21swapper/107:44:381
15213997140,22cyclictest12629-21if_eth011:49:371
15221997050,9cyclictest181rcu_preempt10:33:062
15221997050,8cyclictest181rcu_preempt11:39:222
15221997046,7cyclictest181rcu_preempt07:49:382
15213997055,8cyclictest0-21swapper/108:54:401
15213997055,8cyclictest0-21swapper/108:14:131
15213997055,8cyclictest0-21swapper/108:14:131
15213997054,10cyclictest4569-21fschecks_count11:34:351
15213997053,10cyclictest0-21swapper/112:14:161
15213997053,10cyclictest0-21swapper/112:14:161
15213997051,13cyclictest20477-21latency_hist12:04:151
15213997050,14cyclictest0-21swapper/108:34:401
1520399706,34cyclictest15-21ksoftirqd/008:14:310
15203997032,7cyclictest181rcu_preempt08:34:340
15221996950,14cyclictest12949-21kworker/u17:0+events_unbound08:49:132
1521399696,25cyclictest31-21ksoftirqd/111:24:511
15213996950,9cyclictest0-21swapper/109:24:161
15213996932,8cyclictest181rcu_preempt11:46:111
1521399693,25cyclictest20148-21latency_hist11:04:161
15203996926,13cyclictest0-21swapper/008:19:410
15203996925,8cyclictest0-21swapper/007:39:160
15213996853,8cyclictest0-21swapper/107:39:141
15213996852,9cyclictest0-21swapper/111:44:121
15221996739,9cyclictest181rcu_preempt08:22:252
1521399677,34cyclictest31-21ksoftirqd/110:51:301
15213996751,8cyclictest0-21swapper/107:40:011
15213996750,8cyclictest0-21swapper/108:24:171
15213996750,12cyclictest20335-21kworker/u19:0+events_unbound07:54:131
1521399674,26cyclictest31-21ksoftirqd/108:14:511
1521399672,29cyclictest31-21ksoftirqd/108:05:181
1521399672,29cyclictest31-21ksoftirqd/108:05:181
1521399672,25cyclictest31-21ksoftirqd/110:04:441
1521399672,25cyclictest31-21ksoftirqd/110:04:441
1522899665,45cyclictest9603-21taskset11:44:313
15213996652,9cyclictest58-21kcompactd011:15:331
15213996652,9cyclictest58-21kcompactd011:15:331
1521399665,28cyclictest31-21ksoftirqd/108:34:141
15213996649,8cyclictest0-21swapper/107:24:161
15213996631,7cyclictest181rcu_preempt08:59:431
15213996631,7cyclictest181rcu_preempt08:59:431
1521399662,25cyclictest31-21ksoftirqd/107:14:251
1521399662,23cyclictest31-21ksoftirqd/112:29:321
1521399661,26cyclictest31-21ksoftirqd/107:29:151
15213996536,10cyclictest24851-21ssh09:13:561
15213996533,9cyclictest181rcu_preempt10:45:511
1521399653,25cyclictest31-21ksoftirqd/108:49:441
15213996530,13cyclictest207-21systemd-journal12:07:351
1521399652,29cyclictest31-21ksoftirqd/107:09:451
1521399652,29cyclictest31-21ksoftirqd/107:09:451
1521399652,26cyclictest31-21ksoftirqd/107:56:211
15228996432,21cyclictest29813-21chrt09:23:003
1521399642,27cyclictest31-21ksoftirqd/108:24:341
1522899637,40cyclictest0-21swapper/307:39:383
1522899633,44cyclictest31336-21chrt09:24:513
15228996330,20cyclictest12042-21chrt08:35:003
15228996231,20cyclictest15796-21chrt08:48:293
15228996228,22cyclictest30154-21chrt07:52:273
1521399624,29cyclictest31-21ksoftirqd/108:44:341
15228996132,19cyclictest25505-21chrt12:12:023
15228996132,19cyclictest25505-21chrt12:12:023
15228996131,19cyclictest12716-21chrt09:51:263
15228996131,19cyclictest12716-21chrt09:51:263
15228996129,21cyclictest2924-21chrt10:32:313
15228996128,21cyclictest2023-21chrt09:30:473
1522899606,43cyclictest0-21swapper/310:09:303
1522899604,40cyclictest14312-21taskset10:53:513
1522899603,41cyclictest0-21swapper/310:20:333
15228996030,20cyclictest25039-21chrt07:37:043
1522899602,44cyclictest31120-21taskset07:54:463
1522899593,38cyclictest2902-21chrt08:09:163
1522899593,38cyclictest2902-21chrt08:09:163
15228995929,20cyclictest22063-21chrt09:05:543
15228995929,19cyclictest14033-21chrt08:41:323
15228995929,18cyclictest28505-21chrt07:46:523
1522899592,44cyclictest16785-21taskset07:14:163
1522899592,44cyclictest16785-21taskset07:14:163
1522899585,36cyclictest6613-21chrt10:39:283
1522899585,36cyclictest6613-21chrt10:39:283
15228995828,19cyclictest18501-21chrt07:16:023
15228995813,31cyclictest23102-21taskset11:09:303
15228995813,31cyclictest23102-21taskset11:09:303
1522899575,37cyclictest31573-21chrt11:24:383
15228995728,18cyclictest21908-21chrt07:28:053
1522899563,41cyclictest6876-21chrt09:39:503
1522899563,40cyclictest32061-21taskset12:24:263
1522899563,40cyclictest32061-21taskset12:24:263
15228995627,19cyclictest9680-21chrt09:45:093
15228995627,19cyclictest23944-21chrt09:10:323
1522899562,42cyclictest31999-21taskset10:26:233
1522899562,37cyclictest17712-21chrt10:59:343
15228995621,24cyclictest21447-21taskset11:05:013
15228995617,27cyclictest0-21swapper/311:14:313
15228995617,27cyclictest0-21swapper/311:14:313
1522899555,35cyclictest11759-21chrt10:49:133
1522899554,34cyclictest27093-21taskset12:14:423
1522899553,34cyclictest15264-21taskset11:54:363
1522899552,41cyclictest4252-21taskset10:34:553
1522899552,37cyclictest16929-21taskset08:49:463
1522899544,36cyclictest25264-21chrt10:14:363
1522899542,38cyclictest5826-21taskset11:36:373
1522899542,35cyclictest3426-21chrt09:34:313
1522899534,37cyclictest5204-21chrt08:14:473
1522899533,31cyclictest6328-21chrt12:35:163
1522899532,39cyclictest1878-21taskset11:29:363
1522899532,36cyclictest26504-21chrt09:15:473
1522899532,33cyclictest22399-21taskset07:29:313
1522899524,36cyclictest19596-21chrt08:59:343
1522899524,36cyclictest19596-21chrt08:59:343
1522899523,35cyclictest29872-21chrt12:19:403
1522899523,35cyclictest12211-21chrt11:49:173
1522899522,38cyclictest13784-21taskset09:54:173
1522899513,35cyclictest19856-21chrt10:04:473
1522899513,35cyclictest19856-21chrt10:04:473
1522899513,32cyclictest28611-21chrt11:19:313
1522899503,37cyclictest3890-21chrt08:09:493
1522899503,37cyclictest3890-21chrt08:09:493
1522899503,34cyclictest18235-21taskset08:54:403
1522899503,31cyclictest8488-21chrt08:24:473
1522899502,36cyclictest16981-21chrt09:59:403
15228995019,20cyclictest16172-21taskset10:55:533
15228995019,20cyclictest16172-21taskset10:55:533
1522899494,34cyclictest6977-21chrt08:19:493
1522899494,34cyclictest22072-21chrt12:05:503
15228994933,8cyclictest4204-21chrt12:32:293
1522899493,34cyclictest19340-21chrt07:19:353
1522899493,33cyclictest9938-21chrt08:29:413
1522899484,32cyclictest32493-21chrt07:59:363
1522899484,32cyclictest15283-21chrt07:09:183
1522899483,34cyclictest8028-21chrt11:39:573
1522899482,34cyclictest19525-21chrt12:01:413
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional