You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-16 - 00:27
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Sat Nov 15, 2025 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1871621176,15sleep10-21swapper/107:09:311
188949910254,38cyclictest12295-21kworker/u17:1+events_unbound10:10:041
189069910056,12cyclictest181rcu_preempt12:39:502
189069910056,12cyclictest181rcu_preempt12:39:502
1890699997,77cyclictest30059-21apt-get12:10:042
18894999731,41cyclictest0-21swapper/111:28:581
18894999544,44cyclictest0-21swapper/112:10:031
18894999536,53cyclictest27421-21latency_hist07:34:511
1889499952,27cyclictest17132-21sh11:46:091
18888999470,16cyclictest18879-21cyclictest11:59:520
18888999470,16cyclictest18879-21cyclictest11:59:520
18894999380,7cyclictest0-21swapper/110:09:261
18894999329,37cyclictest0-21swapper/107:22:011
18894999329,37cyclictest0-21swapper/107:22:011
18894999270,16cyclictest18879-21cyclictest08:15:231
18888999267,19cyclictest1813-21interrupts11:20:130
18888999266,18cyclictest1221-21latency_hist11:19:510
18906999131,23cyclictest0-21swapper/211:15:072
18906999039,17cyclictest181rcu_preempt07:40:272
18906999038,17cyclictest181rcu_preempt07:10:262
18894999058,24cyclictest2126-21sh09:25:141
18906998946,8cyclictest181rcu_preempt10:49:062
18906998946,33cyclictest18250-21grep10:50:292
1889499893,72cyclictest0-21swapper/111:35:031
1889499893,72cyclictest0-21swapper/111:35:031
1889499893,72cyclictest0-21swapper/110:00:031
1889499893,72cyclictest0-21swapper/110:00:031
18894998934,36cyclictest0-21swapper/109:40:291
18906998865,15cyclictest26900-21acpi10:09:592
1890699885,72cyclictest4574-21ssh11:25:072
18906998839,13cyclictest181rcu_preempt10:40:072
1889499884,71cyclictest0-21swapper/108:55:061
1889499884,71cyclictest0-21swapper/108:30:061
18888998856,24cyclictest24773-21kernelversion07:25:150
18888998841,12cyclictest181rcu_preempt10:40:200
1888899883,45cyclictest979-21fschecks_time12:15:130
1888899883,45cyclictest979-21fschecks_time12:15:130
18906998764,16cyclictest14156-21kworker/u17:008:34:512
18906998729,19cyclictest181rcu_preempt09:34:392
18906998729,19cyclictest181rcu_preempt09:34:392
18894998741,39cyclictest0-21swapper/112:00:231
1889499872,69cyclictest26804-21apt-get12:04:591
18888998762,17cyclictest21818-21mailstats07:15:240
18888998762,17cyclictest21818-21mailstats07:15:240
183162878,55sleep218407-21sed07:05:372
18906998658,19cyclictest23560-21threads07:20:262
18906998658,19cyclictest23560-21threads07:20:262
1889499863,71cyclictest0-21swapper/109:15:051
1889499863,71cyclictest0-21swapper/109:15:051
1889499862,72cyclictest0-21swapper/108:10:021
18888998666,8cyclictest181rcu_preempt08:54:490
18888998663,15cyclictest25602-21ssh10:07:290
18888998660,18cyclictest25828-21diskmemload09:40:530
18906998554,11cyclictest181rcu_preempt07:35:242
18906998546,9cyclictest181rcu_preempt07:55:212
18906998546,13cyclictest181rcu_preempt10:56:032
1889499854,66cyclictest0-21swapper/107:25:031
18894998537,40cyclictest0-21swapper/108:04:001
18894998537,40cyclictest0-21swapper/108:04:001
18894998535,40cyclictest0-21swapper/111:34:501
1888899853,72cyclictest4547-21apt-get11:25:050
18906998437,16cyclictest181rcu_preempt12:23:132
18906998433,13cyclictest181rcu_preempt12:32:282
18906998433,13cyclictest181rcu_preempt11:02:152
18906998433,12cyclictest181rcu_preempt10:15:182
1889499844,69cyclictest0-21swapper/110:25:071
1889499844,67cyclictest0-21swapper/108:25:051
1889499843,66cyclictest0-21swapper/107:10:021
18888998442,7cyclictest181rcu_preempt11:00:250
1888899842,54cyclictest29498-21apt-get12:10:040
18906998339,12cyclictest181rcu_preempt08:29:502
18906998332,18cyclictest181rcu_preempt09:49:522
18906998332,18cyclictest181rcu_preempt09:49:522
18906998321,26cyclictest0-21swapper/212:29:452
18894998363,14cyclictest243-21rpcbind09:04:521
18894998360,16cyclictest30452-21sh11:13:561
18894998360,16cyclictest30452-21sh11:13:561
18894998332,42cyclictest0-21swapper/109:34:521
18894998332,42cyclictest0-21swapper/109:34:521
18906998251,9cyclictest29932-21ssh11:12:202
18906998251,9cyclictest29932-21ssh11:12:202
18906998249,11cyclictest181rcu_preempt11:49:212
18906998243,8cyclictest181rcu_preempt11:05:312
18906998231,18cyclictest181rcu_preempt12:16:322
18906998231,18cyclictest181rcu_preempt12:16:322
1889499824,72cyclictest0-21swapper/110:35:021
1889499824,72cyclictest0-21swapper/110:35:021
18894998234,40cyclictest0-21swapper/111:40:131
18888998263,13cyclictest6590-21latency_hist12:24:500
18888998259,18cyclictest32367-21apt-key10:19:510
18888998259,15cyclictest13236-21ssh09:45:180
18888998259,15cyclictest13236-21ssh09:45:180
18888998259,14cyclictest5522-21ssh10:29:440
18888998256,18cyclictest8610-21latency_hist10:34:520
18888998239,17cyclictest29998-21ssh09:16:040
18888998239,17cyclictest29998-21ssh09:16:040
18906998149,9cyclictest181rcu_preempt11:23:522
18906998135,11cyclictest181rcu_preempt10:00:222
18906998135,11cyclictest181rcu_preempt10:00:222
18906998134,12cyclictest181rcu_preempt11:54:252
18906998133,12cyclictest181rcu_preempt12:04:422
18906998131,13cyclictest181rcu_preempt12:04:592
1890699811,37cyclictest39-21ksoftirqd/208:14:222
18894998160,16cyclictest8083-21kworker/u17:1+events_unbound11:24:481
18894998158,16cyclictest25828-21diskmemload09:45:261
18894998158,16cyclictest25828-21diskmemload09:45:261
1889499813,69cyclictest0-21swapper/112:35:041
1889499813,69cyclictest0-21swapper/112:35:041
1889499813,66cyclictest0-21swapper/111:55:021
1889499813,66cyclictest0-21swapper/111:55:021
1889499812,7cyclictest0-21swapper/110:45:031
18888998160,14cyclictest292-21dbus-daemon11:49:490
18906998045,10cyclictest181rcu_preempt08:56:582
18906998042,11cyclictest18879-21cyclictest10:20:232
18906998042,11cyclictest18879-21cyclictest10:20:232
18906998032,12cyclictest181rcu_preempt09:53:352
18906998030,18cyclictest181rcu_preempt07:15:192
18906998030,18cyclictest181rcu_preempt07:15:192
18906998030,13cyclictest181rcu_preempt11:37:482
18906998030,13cyclictest181rcu_preempt11:37:482
18906998029,13cyclictest181rcu_preempt09:35:032
1889499804,71cyclictest0-21swapper/110:40:061
1889499803,65cyclictest0-21swapper/107:55:061
18894998033,41cyclictest0-21swapper/111:09:511
1889499802,66cyclictest341-21apt-get10:20:031
1889499802,66cyclictest341-21apt-get10:20:031
18894998021,33cyclictest355-21runrttasks11:16:161
18888998050,11cyclictest181rcu_preempt08:39:500
18888998048,12cyclictest181rcu_preempt09:50:240
18888998047,8cyclictest181rcu_preempt11:50:090
18888998045,8cyclictest181rcu_preempt08:05:110
18888998042,9cyclictest181rcu_preempt11:34:520
18888998028,31cyclictest1533-21systemctl09:24:490
18888998028,10cyclictest0-21swapper/012:39:420
18888998028,10cyclictest0-21swapper/012:39:420
18906997941,8cyclictest181rcu_preempt11:57:402
18906997941,8cyclictest181rcu_preempt11:57:402
18906997936,13cyclictest181rcu_preempt08:45:292
1889499794,70cyclictest0-21swapper/109:35:021
1889499794,32cyclictest9562-21idleruntime-cro12:29:481
18894997934,38cyclictest0-21swapper/110:19:521
18888997955,15cyclictest7358-21ssh09:35:060
18888997952,9cyclictest181rcu_preempt09:27:210
18888997944,16cyclictest181rcu_preempt11:35:560
18888997944,16cyclictest181rcu_preempt11:35:560
18906997860,10cyclictest18879-21cyclictest11:29:522
18906997841,13cyclictest181rcu_preempt09:55:012
18906997836,12cyclictest181rcu_preempt09:10:402
1889499784,67cyclictest0-21swapper/108:50:051
1889499784,63cyclictest0-21swapper/107:35:051
1889499783,69cyclictest0-21swapper/110:55:061
1889499783,67cyclictest0-21swapper/110:33:471
1889499783,61cyclictest26557-21irqstats09:10:151
1889499783,61cyclictest1719-21sort09:24:521
1889499783,32cyclictest583-21awk07:50:331
1889499782,6cyclictest0-21swapper/108:20:041
1889499782,60cyclictest18213-21apt-get09:55:021
18888997856,13cyclictest21221-21ssh10:56:200
18888997852,17cyclictest18879-21cyclictest12:01:360
1791627813,37sleep30-21swapper/307:05:263
18906997742,24cyclictest13226-21grep11:40:122
18906997738,10cyclictest181rcu_preempt07:45:052
18906997732,18cyclictest181rcu_preempt08:35:362
1889499773,64cyclictest14775-21apt-get08:35:051
18888997757,14cyclictest18879-21cyclictest12:31:400
18888997739,17cyclictest0-21swapper/008:14:220
18906997644,11cyclictest181rcu_preempt08:09:522
18906997641,8cyclictest181rcu_preempt09:15:192
18906997641,8cyclictest181rcu_preempt09:15:192
18894997656,14cyclictest309-21systemd-logind07:19:501
18894997656,14cyclictest309-21systemd-logind07:19:501
1889499763,33cyclictest6181-21rm12:24:021
18894997631,38cyclictest17557-21if_err_eth010:50:141
1889499762,32cyclictest12177-21rm12:33:401
1889499762,25cyclictest23972-21ssh11:01:191
18888997662,10cyclictest17487-21kworker/u18:2-rpciod09:56:250
18888997656,13cyclictest18879-21cyclictest10:54:450
18888997648,14cyclictest181rcu_preempt11:13:350
18888997648,14cyclictest181rcu_preempt11:13:350
18888997641,25cyclictest0-21swapper/007:56:520
18888997641,11cyclictest181rcu_preempt08:34:520
1888899762,60cyclictest32221-21apt-get10:20:030
1888899762,60cyclictest32221-21apt-get10:20:030
18906997544,8cyclictest181rcu_preempt08:40:202
18906997537,11cyclictest181rcu_preempt09:22:332
18906997530,9cyclictest181rcu_preempt08:50:152
18894997550,18cyclictest25832-21idleruntime-cro09:09:481
1889499753,65cyclictest0-21swapper/112:15:031
1889499753,65cyclictest0-21swapper/112:15:031
1889499753,64cyclictest0-21swapper/111:50:581
1889499753,61cyclictest0-21swapper/108:05:011
1889499752,32cyclictest31839-21/usr/sbin/munin07:45:281
1889499752,30cyclictest30359-21taskset07:41:571
18888997546,8cyclictest181rcu_preempt09:10:240
18888997543,13cyclictest181rcu_preempt07:20:080
18888997543,13cyclictest181rcu_preempt07:20:080
18888997540,11cyclictest181rcu_preempt12:05:060
18906997440,10cyclictest181rcu_preempt09:27:202
18906997439,10cyclictest181rcu_preempt10:05:062
18906997438,9cyclictest181rcu_preempt09:40:042
18906997430,15cyclictest181rcu_preempt07:30:192
1889499742,64cyclictest0-21swapper/109:50:031
1889499742,26cyclictest18375-21cut08:45:161
18888997440,9cyclictest181rcu_preempt07:40:190
18888997434,16cyclictest11269-21sendmail-msp10:39:490
18888997434,16cyclictest11269-21sendmail-msp10:39:490
18888997424,36cyclictest0-21swapper/008:46:110
18888997415,34cyclictest28376-21find07:35:250
18906997346,8cyclictest181rcu_preempt10:32:372
18906997337,10cyclictest181rcu_preempt09:04:502
18906997331,8cyclictest181rcu_preempt07:50:072
18888997351,15cyclictest11259-21latency_hist08:24:520
18888997341,10cyclictest181rcu_preempt07:50:070
18888997335,8cyclictest181rcu_preempt08:40:130
18888997334,11cyclictest181rcu_preempt09:05:100
18906997240,13cyclictest181rcu_preempt08:15:122
18906997229,11cyclictest181rcu_preempt10:25:332
18906997221,18cyclictest0-21swapper/207:25:182
18894997231,34cyclictest0-21swapper/108:40:191
18888997250,9cyclictest181rcu_preempt08:26:160
18888997250,8cyclictest181rcu_preempt10:10:370
18888997250,16cyclictest15328-21ssh10:45:260
18888997240,11cyclictest181rcu_preempt11:07:500
18888997231,16cyclictest181rcu_preempt07:30:440
18888997216,20cyclictest15-21ksoftirqd/010:00:320
18888997216,20cyclictest15-21ksoftirqd/010:00:320
1890699712,37cyclictest39-21ksoftirqd/208:00:062
1890699712,37cyclictest39-21ksoftirqd/208:00:062
18888997148,16cyclictest0-21swapper/007:09:530
18888997136,9cyclictest181rcu_preempt07:10:410
18888997133,11cyclictest0-21swapper/008:15:180
18888997123,37cyclictest7721-21timerandwakeup12:25:250
1890699703,48cyclictest8968-21fschecks_count10:35:122
1890699703,48cyclictest8968-21fschecks_count10:35:122
18906997030,8cyclictest181rcu_preempt09:09:512
18906997021,13cyclictest181rcu_preempt08:20:182
18888997039,9cyclictest181rcu_preempt11:40:130
18888996836,8cyclictest181rcu_preempt09:30:070
18888996836,8cyclictest181rcu_preempt09:30:070
18888996740,8cyclictest181rcu_preempt08:55:180
18888996722,10cyclictest181rcu_preempt09:00:270
1891299669,42cyclictest30165-21chrt12:10:183
1888899653,46cyclictest4606-21kworker/u18:208:04:480
1888899653,46cyclictest4606-21kworker/u18:208:04:480
18888996433,8cyclictest181rcu_preempt07:44:550
1891299578,38cyclictest417-21chrt10:20:133
1891299578,38cyclictest417-21chrt10:20:133
1891299563,40cyclictest9089-21chrt09:38:143
1891299544,38cyclictest9248-21taskset10:35:173
1891299544,38cyclictest9248-21taskset10:35:173
1891299543,37cyclictest12782-21chrt08:29:523
1891299533,35cyclictest4388-21chrt09:29:513
1891299532,36cyclictest28706-21taskset09:14:523
1891299524,35cyclictest19791-21taskset11:50:583
1891299523,37cyclictest5619-21taskset12:22:303
1891299523,37cyclictest24752-21chrt11:03:413
1891299523,36cyclictest22617-21chrt08:59:493
1891299523,36cyclictest19370-21taskset08:49:483
1891299522,35cyclictest9758-21chrt12:29:513
1891299513,35cyclictest20752-21taskset10:55:263
1891299513,29cyclictest1561-21chrt07:54:533
1891299512,37cyclictest32694-21taskset10:19:533
1891299503,34cyclictest23884-21taskset10:05:013
1891299503,33cyclictest22516-21chrt07:19:493
1891299503,33cyclictest22516-21chrt07:19:493
1891299503,33cyclictest15798-21taskset08:35:553
1891299502,36cyclictest24659-21chrt07:25:123
1891299502,35cyclictest11678-21chrt09:42:363
1891299502,34cyclictest4109-21chrt10:25:283
1891299502,32cyclictest6730-21chrt08:10:123
18912995013,21cyclictest0-21swapper/311:25:053
1891299494,31cyclictest20194-21chrt08:50:213
1891299493,35cyclictest7457-21chrt11:30:103
1891299493,34cyclictest12832-21chrt10:40:563
1891299493,33cyclictest27453-21chrt12:05:203
1891299493,33cyclictest20788-21taskset09:59:503
1891299493,29cyclictest26909-21taskset10:10:003
1891299492,35cyclictest10924-21chrt12:30:283
1891299492,34cyclictest30170-21taskset09:16:343
1891299492,34cyclictest30170-21taskset09:16:343
1891299483,37cyclictest3875-21chrt11:24:313
1891299483,32cyclictest6075-21chrt09:31:463
1891299483,32cyclictest6075-21chrt09:31:463
1891299483,32cyclictest372-21chrt09:21:123
1891299482,34cyclictest13838-21chrt09:45:363
1891299482,34cyclictest13838-21chrt09:45:363
1891299482,32cyclictest18134-21chrt09:54:523
1891299482,30cyclictest21066-21taskset10:00:033
1891299482,30cyclictest21066-21taskset10:00:033
1891299473,36cyclictest27862-21chrt07:35:123
1891299473,34cyclictest11200-21chrt11:35:293
1891299473,34cyclictest11200-21chrt11:35:293
1891299473,33cyclictest1138-21taskset11:19:493
1891299473,30cyclictest15309-21taskset10:45:253
1891299472,33cyclictest16688-21chrt08:40:133
1891299463,34cyclictest9550-21chrt08:19:513
1891299463,34cyclictest14169-21chrt12:35:473
1891299463,34cyclictest14169-21chrt12:35:473
1891299463,32cyclictest15841-21taskset11:45:063
1891299463,31cyclictest24296-21taskset07:24:523
1891299463,31cyclictest24296-21taskset07:24:523
1891299463,30cyclictest26846-21chrt07:30:263
1891299463,29cyclictest25153-21chrt09:05:243
1891299452,31cyclictest17829-21chrt10:50:243
1891299433,31cyclictest29733-21taskset07:40:193
1891299433,27cyclictest24276-21chrt09:04:513
1891299432,29cyclictest25371-21taskset12:01:023
1891299424,27cyclictest21399-21taskset11:55:033
1891299424,27cyclictest21399-21taskset11:55:033
1891299423,30cyclictest5023-21taskset08:05:093
1891299423,26cyclictest9728-21chrt08:20:043
1891299413,30cyclictest29527-21taskset11:11:063
1891299413,30cyclictest29527-21taskset11:11:063
1891299413,29cyclictest4006-21taskset08:00:233
1891299413,29cyclictest4006-21taskset08:00:233
1891299413,25cyclictest31555-21chrt07:45:253
1891299403,27cyclictest843-21taskset12:15:093
1891299403,27cyclictest843-21taskset12:15:093
1891299403,27cyclictest14039-21chrt11:40:283
1891299402,28cyclictest19237-21taskset07:10:093
1891299393,26cyclictest13076-21taskset08:30:083
1891299393,25cyclictest25478-21taskset11:05:023
1891299383,25cyclictest1770-21taskset07:55:093
1891299373,25cyclictest6887-21chrt10:30:273
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional