You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-03 - 04:44
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Sat Jan 03, 2026 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2424822860,28sleep30-21swapper/319:05:433
2424822860,28sleep30-21swapper/319:05:433
23956212512,77sleep223295-21/usr/sbin/munin19:05:392
23956212512,77sleep223295-21/usr/sbin/munin19:05:392
24793991114,89cyclictest15880-21apt-get22:25:090
24793991114,89cyclictest15880-21apt-get22:25:090
248099910731,23cyclictest0-21swapper/219:10:282
24801991064,39cyclictest352-21runrttasks22:21:591
248019910056,37cyclictest300-21rs:main1
119612990,6sleep00-21swapper/023:09:520
119612990,6sleep00-21swapper/023:09:520
24809999823,47cyclictest0-21swapper/220:40:102
24809999823,47cyclictest0-21swapper/220:40:102
24793999783,9cyclictest23873-21apt-get20:45:100
24793999670,18cyclictest18720-21latency_hist20:29:580
2480999955,61cyclictest9613-21ssh22:15:572
24793999447,20cyclictest181rcu_preempt21:49:220
24801999338,47cyclictest0-21swapper/122:45:291
24801999239,47cyclictest0-21swapper/121:59:561
2480199923,33cyclictest26307-21idleruntime-cro19:14:541
2480199902,31cyclictest5744-21cat22:59:571
24793999054,17cyclictest181rcu_preempt00:05:140
24793999054,17cyclictest181rcu_preempt00:05:140
24793999016,50cyclictest0-21swapper/022:15:570
2480199899,60cyclictest31944-21apt-get22:00:191
24801998940,42cyclictest0-21swapper/122:30:231
24793998952,29cyclictest31200-21ssh22:48:390
2480999883,71cyclictest16174-21/usr/sbin/munin22:25:272
2480999883,71cyclictest16174-21/usr/sbin/munin22:25:272
24801998839,42cyclictest0-21swapper/121:15:321
2480199883,30cyclictest6289-21fschecks_time23:00:181
2480199883,28cyclictest18882-21ssh21:39:591
24793998862,19cyclictest29888-21sh23:36:560
24793998860,20cyclictest31934-21diskmemload21:50:200
24793998847,32cyclictest22308-21latency_hist22:34:580
24809998770,11cyclictest22707-21ssh23:25:302
24809998770,11cyclictest22707-21ssh23:25:302
2480999872,52cyclictest21917-21apt-get22:35:082
2480199875,32cyclictest8481-21sh23:55:061
2480199873,35cyclictest18679-21ntpq00:10:281
2480199873,30cyclictest352-21runrttasks21:54:261
2480199873,28cyclictest27998-21grep20:55:311
2480199873,28cyclictest27998-21grep20:55:311
2480199872,34cyclictest24704-21rm23:29:531
2480199872,34cyclictest24704-21rm23:29:531
2480199872,32cyclictest12686-21cpuspeed21:30:111
2480199872,31cyclictest5056-21latency_hist23:49:571
2480199872,31cyclictest352-21runrttasks19:44:571
2480199872,30cyclictest16536-21grep20:20:321
24793998773,10cyclictest9549-21kworker/u18:2-rpciod22:23:030
24793998761,19cyclictest3743-21unixbench_multi22:05:320
24793998738,10cyclictest181rcu_preempt20:24:580
24809998631,33cyclictest0-21swapper/200:30:202
2480199869,63cyclictest9260-21apt-get22:15:101
2480199864,36cyclictest6157-21fschecks_time22:10:141
2480199864,34cyclictest352-21runrttasks00:34:571
2480199864,33cyclictest24423-21cpu00:20:111
24801998638,43cyclictest0-21swapper/121:10:301
2480199863,34cyclictest24178-21uniq20:45:461
2480199863,33cyclictest13231-21unixbench_multi20:10:321
2480199863,31cyclictest18696-21acpi23:20:061
2480199863,29cyclictest18738-21latency_hist21:39:571
24801998629,49cyclictest0-21swapper/123:50:261
2480199862,30cyclictest18889-21latency_hist22:29:571
2480199862,30cyclictest18889-21latency_hist22:29:571
24809998563,16cyclictest31934-21diskmemload23:55:162
24809998521,24cyclictest0-21swapper/200:04:452
2480199855,38cyclictest7410-21latency_hist19:54:581
2480199855,38cyclictest7410-21latency_hist19:54:581
2480199854,34cyclictest25908-21latency20:50:211
2480199853,35cyclictest29315-21mailstats19:20:301
2480199853,35cyclictest29315-21mailstats19:20:301
2480199853,30cyclictest9369-21/usr/sbin/munin21:25:271
2480199853,30cyclictest9265-21cstates23:05:121
2480199853,30cyclictest9265-21cstates23:05:121
2480199853,30cyclictest14133-21sort20:15:191
2480199852,31cyclictest21670-21latency00:15:211
24793998555,12cyclictest181rcu_preempt21:10:180
24793998541,12cyclictest181rcu_preempt22:36:230
24809998434,23cyclictest0-21swapper/222:50:302
24809998434,23cyclictest0-21swapper/222:50:302
2480199843,36cyclictest12452-21mailstats00:00:291
2480199843,34cyclictest28231-21grep00:25:251
2480199843,34cyclictest28231-21grep00:25:251
2480199843,33cyclictest30235-21/usr/sbin/munin19:25:101
2480199843,33cyclictest30235-21/usr/sbin/munin19:25:101
2480199843,33cyclictest22454-21/usr/sbin/munin22:35:061
2480199843,32cyclictest12179-21grep20:09:571
2480199843,31cyclictest9463-21kernelversion20:00:211
2480199843,30cyclictest16589-21sed23:15:331
2480199843,29cyclictest2481-21unixbench_multi00:35:311
2480199843,28cyclictest8919-21latency_hist19:59:571
2480199843,28cyclictest8919-21latency_hist19:59:571
2480199843,28cyclictest25036-21/usr/sbin/munin23:30:191
2480199843,28cyclictest22831-21ps21:45:271
2480199842,33cyclictest19259-21netstat20:30:241
2480199842,32cyclictest14823-21/usr/sbin/munin00:05:261
2480199842,32cyclictest14823-21/usr/sbin/munin00:05:261
24793998467,11cyclictest32486-21ssh22:00:550
24809998362,15cyclictest3707-21processes21:15:302
2480999832,48cyclictest23373-21apt-get20:45:082
2480199833,32cyclictest352-21runrttasks20:25:501
2480199833,31cyclictest26467-21mailstats22:40:291
2480199833,31cyclictest2357-21sendmail19:39:541
2480199833,30cyclictest22230-21fschecks_count20:40:151
2480199833,30cyclictest22230-21fschecks_count20:40:151
2480199833,30cyclictest12280-21/usr/sbin/munin23:10:271
2480199833,27cyclictest30571-21/usr/sbin/munin21:05:141
24793998354,9cyclictest181rcu_preempt22:57:420
24793998350,16cyclictest181rcu_preempt22:54:540
24793998350,16cyclictest181rcu_preempt22:54:540
24793998339,12cyclictest181rcu_preempt22:10:220
2480999824,68cyclictest31944-21apt-get22:00:122
24809998223,35cyclictest0-21swapper/219:17:042
2480199823,33cyclictest5148-21sh22:08:571
2480199823,30cyclictest20376-21/usr/sbin/munin20:35:281
2480199823,28cyclictest26803-21cpu19:15:041
24801998227,47cyclictest0-21swapper/123:44:581
2480199822,30cyclictest6926-21cut21:20:271
24793998243,12cyclictest0-21swapper/023:10:110
2479399823,36cyclictest15-21ksoftirqd/023:02:530
2480199814,64cyclictest28972-21apt-get21:00:081
2480199814,29cyclictest32012-21/usr/sbin/munin22:50:181
2480199814,29cyclictest32012-21/usr/sbin/munin22:50:181
2480199813,30cyclictest20019-21grep21:40:301
2480199812,32cyclictest843-21latency_hist19:34:571
2480199812,32cyclictest843-21latency_hist19:34:571
24793998145,16cyclictest181rcu_preempt20:15:190
24793998129,9cyclictest181rcu_preempt23:21:350
24809998056,18cyclictest1-21systemd19:49:472
24809998056,18cyclictest1-21systemd19:49:472
2480999802,36cyclictest2383-21wc00:35:302
24793998060,13cyclictest23443-21kworker/u18:2+rpciod23:27:200
24793998060,13cyclictest23443-21kworker/u18:2+rpciod23:27:200
24793998055,8cyclictest181rcu_preempt21:39:570
24793998055,8cyclictest181rcu_preempt21:39:570
24793998054,20cyclictest4460-21fschecks_count19:45:190
24793998054,20cyclictest4460-21fschecks_count19:45:190
24793998054,19cyclictest0-21swapper/020:41:470
24793998054,19cyclictest0-21swapper/020:41:470
24793998044,20cyclictest181rcu_preempt21:15:370
24793998041,29cyclictest0-21swapper/022:42:030
24809997963,9cyclictest7697-21ssh22:12:222
24809997933,23cyclictest181rcu_preempt19:39:572
2480199794,28cyclictest4237-21/usr/sbin/munin19:45:211
2480199794,28cyclictest4237-21/usr/sbin/munin19:45:211
24793997958,16cyclictest28125-21kworker/u18:2+events_unbound20:59:540
24793997958,16cyclictest28125-21kworker/u18:2+events_unbound20:59:540
24793997956,8cyclictest181rcu_preempt20:35:190
24793997954,12cyclictest181rcu_preempt21:25:320
24793997948,12cyclictest181rcu_preempt00:11:560
24793997927,24cyclictest181rcu_preempt21:55:490
24809997827,24cyclictest0-21swapper/221:22:432
24793997831,16cyclictest181rcu_preempt23:43:500
24793997817,39cyclictest2381-21cat00:35:300
24809997759,12cyclictest31934-21diskmemload22:55:172
24809997759,11cyclictest0-21swapper/223:24:562
24809997751,21cyclictest24783-21cyclictest21:39:572
24809997751,21cyclictest24783-21cyclictest21:39:572
2480199773,29cyclictest24825-21latency_hist19:09:591
2480199773,29cyclictest24825-21latency_hist19:09:591
24793997758,13cyclictest31002-21fschecks_count00:30:210
24793997755,15cyclictest29402-21threads19:20:320
24793997755,15cyclictest29402-21threads19:20:320
24793997742,15cyclictest181rcu_preempt19:28:240
24793997742,15cyclictest181rcu_preempt19:28:240
24793997742,13cyclictest181rcu_preempt19:50:170
24793997742,13cyclictest181rcu_preempt19:50:170
2479399773,43cyclictest5289-21/usr/sbin/munin23:50:110
2479399773,41cyclictest8724-21grep23:55:150
24809997654,15cyclictest5200-21latency_hist23:49:582
2479399764,37cyclictest12709-21unixbench_multi00:00:310
24793997636,9cyclictest181rcu_preempt19:19:040
24809997560,11cyclictest26517-21kworker/u17:1-writeback21:11:322
24809997553,16cyclictest31379-21latency_hist21:59:582
24809997550,19cyclictest25281-21timerandwakeup00:20:312
24793997553,15cyclictest803-21sendmail-mta20:04:080
24793997550,16cyclictest2932-21idleruntime19:40:180
24793997537,9cyclictest181rcu_preempt19:10:140
24809997454,14cyclictest16021-21sh23:15:222
2480999745,31cyclictest14303-21kernelversion20:15:232
24793997457,12cyclictest26627-21kworker/u17:2+xprtiod00:29:310
24793997457,12cyclictest26627-21kworker/u17:2+xprtiod00:29:310
24793997451,11cyclictest181rcu_preempt23:48:130
24793997445,22cyclictest15664-21latency_hist21:34:580
24793997439,14cyclictest181rcu_preempt19:55:240
24793997439,14cyclictest181rcu_preempt19:55:240
24793997345,11cyclictest181rcu_preempt20:05:230
24793997334,18cyclictest181rcu_preempt21:24:560
24793997333,10cyclictest181rcu_preempt19:30:300
24793997333,10cyclictest181rcu_preempt19:30:300
24793997317,34cyclictest15-21ksoftirqd/000:23:320
24809997255,11cyclictest5879-21kworker/u17:1-events_unbound22:34:562
24809997253,14cyclictest26971-21latency_hist20:54:572
24809997249,17cyclictest1-21systemd21:09:212
2480999723,48cyclictest0-21swapper/223:05:472
2480999723,48cyclictest0-21swapper/223:05:472
2480999723,28cyclictest0-21swapper/200:29:332
2480999723,28cyclictest0-21swapper/200:29:332
24809997218,41cyclictest3488-21rm22:05:302
24793997252,11cyclictest181rcu_preempt20:10:130
24793997239,20cyclictest181rcu_preempt23:30:180
24793997238,8cyclictest181rcu_preempt20:34:570
24793997234,17cyclictest181rcu_preempt20:50:120
2479399723,33cyclictest32037-21latency_hist21:09:570
24793997231,12cyclictest181rcu_preempt21:41:290
24809997148,17cyclictest1-21systemd19:53:462
24809997148,17cyclictest1-21systemd19:53:462
2480999713,34cyclictest6243-21grep23:00:172
2480999713,25cyclictest19408-21wc20:30:292
2480999711,39cyclictest25228-21ssh23:30:122
24793997133,17cyclictest181rcu_preempt23:19:550
24809997050,13cyclictest24783-21cyclictest23:42:082
24809997045,17cyclictest1-21systemd19:55:462
24809997045,17cyclictest1-21systemd19:55:462
2480999704,37cyclictest5342-21cpu23:50:112
2480999704,26cyclictest28445-21apt-get21:00:092
2480999703,38cyclictest0-21swapper/219:33:112
2480999703,38cyclictest0-21swapper/219:33:112
2480999703,32cyclictest31582-21modprobe19:29:542
2480999703,32cyclictest31582-21modprobe19:29:542
24809997019,22cyclictest0-21swapper/222:23:122
24809997017,22cyclictest9554-21needreboot20:00:222
24809997015,24cyclictest0-21swapper/222:41:352
24793997056,9cyclictest0-21swapper/000:19:130
24793997043,11cyclictest181rcu_preempt21:01:540
24809996850,12cyclictest14505-21runrttasks23:13:172
24809996847,15cyclictest21956-21ssh00:15:292
2480999684,32cyclictest17703-21munin-run00:09:552
2480999684,32cyclictest17703-21munin-run00:09:552
24809996842,20cyclictest19071-21cstates21:40:102
2480999682,52cyclictest17914-21switchtime20:25:302
2360226813,25sleep00-21swapper/019:05:240
2360226813,25sleep00-21swapper/019:05:240
24809996742,18cyclictest0-21swapper/220:20:192
24809996742,14cyclictest0-21swapper/220:10:192
2480999673,32cyclictest27755-21mailstats20:55:282
2480999673,32cyclictest27755-21mailstats20:55:282
24793996739,11cyclictest181rcu_preempt19:35:300
2480999664,40cyclictest28353-21latency_hist21:54:572
2480999663,43cyclictest352-21runrttasks22:45:542
2480999663,28cyclictest19017-21ls00:10:322
24809996624,32cyclictest10215-21switchtime21:25:302
2480999662,28cyclictest0-21swapper/221:30:282
2481499655,39cyclictest0-21swapper/323:59:553
24814996511,45cyclictest9619-21chrt22:15:583
24814996416,36cyclictest0-21swapper/322:25:113
24814996416,36cyclictest0-21swapper/322:25:113
2480999644,29cyclictest30059-21latency_hist19:24:572
2480999644,29cyclictest30059-21latency_hist19:24:572
2480999643,40cyclictest2747-21cstates19:40:122
2480999643,36cyclictest31266-21latency_hist23:39:572
24809996420,33cyclictest20622-21fschecks_time20:35:182
24809996339,11cyclictest0-21swapper/220:05:172
24809996237,18cyclictest0-21swapper/221:45:212
2481499604,26cyclictest0-21swapper/320:45:083
24814996027,22cyclictest25390-21taskset23:30:173
2481499583,40cyclictest22136-21chrt00:15:313
18382588,11sleep30-21swapper/322:53:273
18382588,11sleep30-21swapper/322:53:273
2481499563,39cyclictest19321-21taskset21:40:173
2481499544,37cyclictest3194-21chrt23:45:333
2481499543,38cyclictest28836-21taskset00:25:563
2481499543,38cyclictest28836-21taskset00:25:563
2481499543,38cyclictest26602-21taskset21:50:543
2481499543,35cyclictest482-21chrt21:10:273
2481499536,22cyclictest0-21swapper/322:35:083
2481499534,37cyclictest20184-21chrt00:13:123
2481499534,35cyclictest14658-21chrt21:32:593
2481499533,37cyclictest32517-21chrt23:40:303
2481499533,37cyclictest28074-21taskset22:43:463
2481499532,38cyclictest30743-21chrt21:05:163
2481499527,26cyclictest0-21swapper/322:00:183
2481499523,35cyclictest23010-21taskset20:40:313
2481499523,35cyclictest23010-21taskset20:40:313
2481499523,33cyclictest29286-21chrt23:35:323
2481499523,33cyclictest13679-21chrt22:20:453
2481499513,36cyclictest21567-21chrt23:24:543
2481499513,35cyclictest9417-21chrt20:00:193
2481499513,35cyclictest21771-21chrt22:34:343
2481499513,33cyclictest18738-21chrt20:29:583
2481499512,37cyclictest25425-21taskset19:10:233
2481499512,35cyclictest23598-21chrt23:27:053
2481499512,35cyclictest23598-21chrt23:27:053
2481499512,33cyclictest20738-21taskset20:35:213
2481499504,33cyclictest12399-21chrt00:00:283
2481499503,37cyclictest7514-21taskset23:01:373
2481499503,37cyclictest28886-21taskset19:20:193
2481499503,37cyclictest28886-21taskset19:20:193
2481499503,37cyclictest19533-21chrt20:30:313
2481499503,37cyclictest18427-21chrt23:19:563
2481499503,35cyclictest28371-21taskset19:19:543
2481499503,35cyclictest25179-21chrt00:20:293
2481499503,34cyclictest5343-21chrt23:50:113
2481499503,32cyclictest32202-21chrt19:30:193
2481499503,32cyclictest32202-21chrt19:30:193
2481499503,30cyclictest9201-21chrt21:24:573
2481499502,34cyclictest24042-21chrt21:47:303
2481499493,35cyclictest2355-21taskset19:39:543
2481499493,33cyclictest16034-21chrt20:20:233
2481499493,33cyclictest14226-21chrt20:15:213
2481499492,35cyclictest3920-21chrt22:05:493
2481499483,35cyclictest31161-21chrt22:48:323
2481499483,34cyclictest11447-21chrt20:05:293
2481499483,33cyclictest28796-21taskset20:59:583
2481499483,33cyclictest28796-21taskset20:59:583
2481499483,33cyclictest11289-21chrt23:08:093
2481499483,33cyclictest11289-21chrt23:08:093
2481499483,32cyclictest4032-21chrt19:44:563
2481499483,32cyclictest3147-21chrt21:15:133
2481499483,32cyclictest16105-21chrt21:35:183
2481499483,32cyclictest16105-21chrt21:35:183
2481499483,31cyclictest5175-21chrt19:45:333
2481499483,31cyclictest5175-21chrt19:45:333
2481499483,31cyclictest12600-21chrt20:10:193
2481499482,26cyclictest31576-21chrt21:59:583
2481499473,34cyclictest14575-21chrt23:13:283
2481499473,31cyclictest8855-21chrt22:14:573
2481499473,31cyclictest2878-21chrt00:36:223
2481499472,33cyclictest9698-21chrt21:25:183
2481499462,33cyclictest16545-21chrt00:07:003
2481499462,33cyclictest16545-21chrt00:07:003
2481499462,29cyclictest31388-21chrt00:30:303
2481499453,29cyclictest3232-21chrt22:55:213
2481499443,32cyclictest8815-21chrt19:59:543
2481499443,32cyclictest8815-21chrt19:59:543
2481499408,22cyclictest0-21swapper/321:01:073
2481499403,27cyclictest6312-21chrt19:50:233
2481499403,27cyclictest6312-21chrt19:50:233
2481499403,23cyclictest31598-21chrt19:29:543
2481499403,23cyclictest31598-21chrt19:29:543
2481499373,25cyclictest26660-21taskset20:52:493
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional