You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-08 - 20:14
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Mon Dec 08, 2025 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17595991462,35cyclictest21804-21ssh11:04:200
17595991462,35cyclictest21804-21ssh11:04:200
17445212910,84sleep017509-21sed07:08:210
17445212910,84sleep017509-21sed07:08:210
175959911937,76cyclictest0-21swapper/012:33:400
175959911829,79cyclictest7807-21ssh10:38:320
17601991123,73cyclictest12897-21apt-get10:48:321
17595991103,28cyclictest7949-21/usr/sbin/munin09:38:480
175959910950,53cyclictest24716-21diskmemload09:36:000
17595991082,43cyclictest16063-21apt-get09:53:310
175959910749,51cyclictest24344-21sh10:08:370
175959910749,51cyclictest24344-21sh10:08:370
17595991072,30cyclictest32442-21awk09:23:180
17595991072,30cyclictest32442-21awk09:23:180
17595991054,30cyclictest14479-21unixbench_multi11:48:550
17595991053,25cyclictest355-21runrttasks11:34:110
17595991052,94cyclictest25949-21kworker/u19:2+ext4-rsv-conversion10:48:470
17595991043,33cyclictest32704-21needreboot11:23:460
17595991043,33cyclictest32704-21needreboot11:23:460
17595991043,29cyclictest7199-21idleruntime08:13:410
17595991038,81cyclictest355-21runrttasks07:08:520
175959910340,55cyclictest0-21swapper/010:18:410
17595991028,83cyclictest10212-21apt-get11:43:410
175959910249,45cyclictest18640-21latency_hist10:58:210
175959910246,49cyclictest355-21runrttasks09:48:410
17595991023,31cyclictest26463-21latency_hist11:13:210
17595991023,21cyclictest11745-21/usr/sbin/munin08:28:440
17595991023,21cyclictest11745-21/usr/sbin/munin08:28:440
176149910150,8cyclictest181rcu_preempt11:59:352
175959910135,57cyclictest0-21swapper/010:13:470
17595991013,32cyclictest355-21runrttasks08:08:510
17595991003,30cyclictest26757-21/usr/sbin/munin11:13:340
17595991002,23cyclictest19436-21ps10:58:540
1759599994,31cyclictest25335-21grep12:08:540
17595999941,51cyclictest0-21swapper/009:58:410
17595999939,52cyclictest0-21swapper/009:03:400
1759599993,35cyclictest27451-21latency_hist09:13:200
1759599993,35cyclictest27451-21latency_hist09:13:200
1759599993,26cyclictest207-21systemd-journal10:48:180
1759599992,31cyclictest27076-21/usr/sbin/munin12:13:560
1759599992,31cyclictest27076-21/usr/sbin/munin12:13:560
1759599992,31cyclictest22279-21find12:03:540
1759599992,28cyclictest28587-21timerandwakeup09:13:570
1759599992,22cyclictest28067-21apt-get07:38:290
1759599982,34cyclictest18492-21/usr/sbin/munin08:48:570
1759599982,29cyclictest10681-21cpu09:43:350
1759599982,27cyclictest355-21runrttasks12:23:320
1759599982,24cyclictest32746-21seq09:23:320
1759599982,19cyclictest3691-21apt-get08:03:310
17595999752,40cyclictest0-21swapper/008:23:480
1759599973,24cyclictest5492-21kernelversion10:33:450
17595999723,66cyclictest19413-21diskstats11:59:350
1759599972,32cyclictest3327-21uname09:28:440
1759599972,30cyclictest21594-21/usr/sbin/munin07:18:460
1759599972,30cyclictest21594-21/usr/sbin/munin07:18:460
1759599972,28cyclictest415-21/usr/sbin/munin07:53:380
1759599964,30cyclictest21983-21fschecks_time08:58:420
1759599963,30cyclictest14777-21sshd08:38:170
1759599963,28cyclictest9928-21latency_hist08:23:200
1759599963,28cyclictest15036-21/usr/sbin/munin08:38:530
1759599963,18cyclictest32364-21fschecks_time10:23:420
1759599962,29cyclictest5606-21cron12:33:170
1759599962,29cyclictest5606-21cron12:33:170
1759599962,27cyclictest32513-21runrttasks07:52:040
1760199952,69cyclictest21553-21apt-get11:03:351
1760199952,69cyclictest21553-21apt-get11:03:351
1759599954,31cyclictest20620-21grep07:13:540
1759599954,28cyclictest26416-21/usr/sbin/munin07:33:460
1759599953,33cyclictest24809-21/usr/sbin/munin07:28:470
1759599953,29cyclictest2647-21fschecks_time10:28:420
171812957,15sleep20-21swapper/207:06:032
171812957,15sleep20-21swapper/207:06:032
1759599944,30cyclictest17133-21unixbench_multi11:53:580
1759599944,30cyclictest17133-21unixbench_multi11:53:580
17595999435,52cyclictest0-21swapper/010:03:570
1759599943,31cyclictest30910-21cat12:18:570
1759599943,29cyclictest30643-21unixbench_singl07:43:550
1759599943,29cyclictest16879-21/usr/sbin/munin08:43:310
1759599942,23cyclictest24684-21latency_hist07:28:200
1759599942,23cyclictest24684-21latency_hist07:28:200
17595999332,52cyclictest0-21swapper/011:43:210
1759599932,28cyclictest21050-21unixbench_singl08:53:550
17595999317,24cyclictest4951-21sh11:33:200
1759599923,27cyclictest2720-21mailstats07:58:520
1759599923,27cyclictest2720-21mailstats07:58:520
17595999211,24cyclictest30238-21/usr/sbin/munin11:18:530
17614999146,35cyclictest3605-21wc12:28:452
17614999146,35cyclictest3605-21wc12:28:452
17614999068,16cyclictest8848-21unixbench_multi11:38:562
17614999066,18cyclictest28348-21proc_pri09:13:542
1638629011,54sleep116390-21/usr/sbin/munin07:03:501
1638629011,54sleep116390-21/usr/sbin/munin07:03:501
17601998645,11cyclictest181rcu_preempt08:28:211
17601998564,15cyclictest21188-21apt-key11:03:211
17601998542,31cyclictest17584-21cyclictest07:08:391
1761499845,34cyclictest30331-21ssh09:18:302
1761499845,34cyclictest30331-21ssh09:18:302
17614998359,15cyclictest18682-21sh11:58:212
17614998359,15cyclictest18682-21sh11:58:212
17614998352,13cyclictest181rcu_preempt09:42:282
17614998338,34cyclictest32651-21kworker/u19:3+nfsiod09:08:452
17614998338,34cyclictest32651-21kworker/u19:3+nfsiod09:08:452
17601998363,8cyclictest181rcu_preempt11:38:541
1761499824,55cyclictest10212-21apt-get11:43:322
1762799819,50cyclictest0-21swapper/311:43:483
17614998152,10cyclictest181rcu_preempt08:18:202
17614998052,20cyclictest31219-21kworker/u19:0+nfsiod12:05:112
17601998055,11cyclictest181rcu_preempt08:50:361
17601998041,10cyclictest181rcu_preempt08:18:341
296842790,9sleep20-21swapper/210:18:412
17614997955,17cyclictest0-21swapper/210:49:002
17614997940,17cyclictest181rcu_preempt10:46:382
17601997956,17cyclictest4120-21sh12:28:551
17601997956,17cyclictest4120-21sh12:28:551
17601997953,18cyclictest26612-21fschecks_count07:33:401
17601997953,11cyclictest181rcu_preempt11:33:081
17601997936,13cyclictest0-21swapper/112:26:201
17614997862,10cyclictest13187-21systemd-run08:33:182
17614997862,10cyclictest13187-21systemd-run08:33:182
17614997854,16cyclictest355-21runrttasks11:28:402
17614997763,10cyclictest17584-21cyclictest09:29:022
17614997747,12cyclictest181rcu_preempt07:48:372
17601997758,14cyclictest17584-21cyclictest11:23:581
17601997758,14cyclictest17584-21cyclictest11:23:581
17601997751,10cyclictest27735-21ssh09:13:361
17601997739,11cyclictest0-21swapper/109:34:181
1762799763,44cyclictest0-21swapper/311:03:373
1762799763,44cyclictest0-21swapper/311:03:373
17614997643,9cyclictest181rcu_preempt07:23:222
17614997643,9cyclictest181rcu_preempt07:23:222
17614997620,18cyclictest39-21ksoftirqd/208:08:422
1760199763,50cyclictest13348-21/usr/sbin/munin09:48:401
17614997561,10cyclictest58-21kcompactd011:28:162
17614997561,10cyclictest58-21kcompactd011:28:162
17614997561,10cyclictest58-21kcompactd010:57:122
17614997556,13cyclictest17584-21cyclictest10:13:212
17614997556,13cyclictest17584-21cyclictest10:13:212
17614997554,14cyclictest17584-21cyclictest11:23:172
17614997546,13cyclictest181rcu_preempt10:33:412
17614997538,7cyclictest181rcu_preempt07:09:312
17614997530,16cyclictest181rcu_preempt10:02:392
17601997546,10cyclictest181rcu_preempt09:28:211
17601997527,12cyclictest181rcu_preempt07:43:331
1760199752,52cyclictest176-21jbd2/mmcblk2p2-811:59:121
1762799743,30cyclictest29059-21chrt10:18:183
1762799742,32cyclictest31669-21taskset12:20:543
17614997454,11cyclictest32362-21sh10:23:422
17614997452,15cyclictest0-21swapper/208:52:322
17614997444,9cyclictest181rcu_preempt08:48:202
17601997458,10cyclictest3428-21idleruntime-cro08:03:181
17601997458,10cyclictest3428-21idleruntime-cro08:03:181
17601997437,9cyclictest181rcu_preempt10:28:351
1760199741,30cyclictest15365-21ssh11:51:561
17614997348,14cyclictest181rcu_preempt07:58:102
1761499734,59cyclictest29969-21ssh12:18:362
17614997343,9cyclictest181rcu_preempt09:47:562
17614997337,17cyclictest181rcu_preempt08:22:062
17601997353,14cyclictest24838-21cpu07:28:341
17601997353,12cyclictest32366-21fschecks_time10:23:421
17601997351,16cyclictest355-21runrttasks10:38:321
17601997341,15cyclictest31058-21ssh11:20:371
17601997341,11cyclictest181rcu_preempt09:03:471
17601997320,8cyclictest181rcu_preempt11:53:321
17601997320,8cyclictest181rcu_preempt11:53:321
100752730,4sleep1291rcuc/109:42:301
1762799723,31cyclictest2079-21taskset07:58:333
1762799723,31cyclictest2079-21taskset07:58:333
17614997255,11cyclictest309-21systemd-logind08:03:182
17614997255,11cyclictest309-21systemd-logind08:03:182
17614997255,11cyclictest17584-21cyclictest11:52:302
17614997248,8cyclictest181rcu_preempt10:16:192
17614997246,8cyclictest181rcu_preempt10:38:372
17614997237,10cyclictest181rcu_preempt08:23:492
17614997210,39cyclictest39-21ksoftirqd/210:28:412
17601997255,11cyclictest21461-21latency_hist08:58:181
17601997250,15cyclictest22522-21unixbench_singl10:03:571
1760199722,45cyclictest31-21ksoftirqd/112:12:431
1762799715,40cyclictest0-21swapper/310:48:343
17614997150,15cyclictest207-21systemd-journal08:38:442
17614997143,10cyclictest181rcu_preempt08:53:362
17614997136,10cyclictest0-21swapper/209:48:502
17601997156,9cyclictest25718-21sh09:08:511
17601997156,9cyclictest25718-21sh09:08:511
17601997142,21cyclictest314-21in:imuxsock08:08:181
1760199713,35cyclictest19646-21/usr/sbin/munin09:58:561
17601997127,11cyclictest29054-21munin-plugin-st10:18:181
17601997117,30cyclictest32137-21timerwakeupswit07:48:551
1761499705,34cyclictest5908-21/usr/sbin/munin12:33:462
17614997048,8cyclictest181rcu_preempt09:23:362
17614997031,8cyclictest181rcu_preempt11:03:292
17614997031,8cyclictest181rcu_preempt11:03:292
17601997047,15cyclictest30701-21kernelversion09:18:441
17601997047,15cyclictest30701-21kernelversion09:18:441
17601997041,10cyclictest181rcu_preempt09:25:101
17601997038,13cyclictest181rcu_preempt12:06:351
17614996956,9cyclictest58-21kcompactd010:58:232
17614996947,9cyclictest181rcu_preempt12:27:392
17614996939,22cyclictest25679-21grep07:28:542
17614996930,17cyclictest181rcu_preempt11:13:542
17614996929,23cyclictest58-21kcompactd012:13:192
17614996929,11cyclictest181rcu_preempt09:53:562
17601996953,11cyclictest12471-21sh09:46:261
17601996946,17cyclictest27084-21fschecks_time11:13:401
17601996944,8cyclictest181rcu_preempt11:08:531
1760199692,23cyclictest17045-21tail09:53:561
1761499688,29cyclictest39-21ksoftirqd/207:48:222
17614996851,8cyclictest181rcu_preempt07:21:122
17614996851,8cyclictest181rcu_preempt07:21:122
17614996850,12cyclictest5711-21ntp_kernel_pll_11:33:472
17614996847,8cyclictest181rcu_preempt09:08:202
17614996845,7cyclictest181rcu_preempt08:33:552
1761499684,44cyclictest25142-21unixbench_multi11:08:552
17614996831,10cyclictest181rcu_preempt07:38:392
17614996828,8cyclictest181rcu_preempt12:18:212
17614996828,8cyclictest181rcu_preempt12:18:212
17601996854,10cyclictest17584-21cyclictest08:11:461
17601996852,11cyclictest17584-21cyclictest12:37:231
17601996847,14cyclictest13591-21fschecks_count08:33:391
17601996846,16cyclictest1-21systemd10:21:231
17601996840,9cyclictest6596-21ssh10:35:231
17601996840,13cyclictest181rcu_preempt07:21:291
17601996840,13cyclictest181rcu_preempt07:21:291
17601996838,13cyclictest30328-21ssh12:18:451
17601996819,14cyclictest0-21swapper/108:58:371
1762799674,22cyclictest30118-21chrt11:18:513
1762799674,21cyclictest12011-21chrt08:28:423
1762799674,21cyclictest12011-21chrt08:28:423
1762799672,24cyclictest18896-21taskset11:58:253
17614996752,10cyclictest26416-21/usr/sbin/munin07:33:472
17614996728,13cyclictest181rcu_preempt09:33:372
17614996725,23cyclictest39-21ksoftirqd/209:03:212
17601996752,10cyclictest58-21kcompactd008:18:171
17601996750,9cyclictest0-21swapper/110:43:551
17601996741,11cyclictest181rcu_preempt10:53:411
17601996736,8cyclictest181rcu_preempt11:33:331
1762799663,23cyclictest27895-21chrt07:38:203
1762799663,23cyclictest16129-21chrt10:53:353
1762799663,23cyclictest1610-21chrt11:26:413
1762799663,23cyclictest1610-21chrt11:26:413
1762799662,29cyclictest13962-21chrt08:33:493
1762799662,23cyclictest1200-21taskset10:25:313
17614996634,13cyclictest181rcu_preempt10:07:222
17601996642,8cyclictest181rcu_preempt08:43:461
17601996640,12cyclictest181rcu_preempt12:13:361
17601996640,12cyclictest181rcu_preempt12:13:361
1762799658,14cyclictest0-21swapper/309:03:553
1762799653,22cyclictest1240-21chrt12:23:543
1762799653,21cyclictest28451-21chrt11:16:073
1762799652,23cyclictest25615-21chrt11:10:283
17627996510,26cyclictest0-21swapper/307:08:523
1761499652,41cyclictest21475-21latency_hist07:18:212
1762799643,24cyclictest15775-21chrt09:53:183
1762799643,22cyclictest29730-21chrt10:18:423
1762799643,22cyclictest16876-21chrt08:43:283
1762799642,26cyclictest23044-21taskset10:05:343
1762799642,22cyclictest12803-21chrt09:47:303
1762799642,21cyclictest15214-21chrt11:51:283
1761499643,36cyclictest355-21runrttasks08:04:002
1760199648,45cyclictest10765-21apt-get11:43:351
1760199643,38cyclictest24561-21latency_hist07:28:201
1760199643,38cyclictest24561-21latency_hist07:28:201
1762799633,23cyclictest21551-21chrt12:03:273
1762799633,22cyclictest9060-21chrt11:39:213
1762799633,22cyclictest471-21chrt09:23:393
1762799633,22cyclictest26110-21chrt09:08:563
1762799633,22cyclictest26110-21chrt09:08:563
1762799633,22cyclictest24874-21chrt12:08:463
1762799633,21cyclictest4696-21chrt09:31:133
1762799632,24cyclictest7582-21chrt11:38:193
1762799632,24cyclictest27117-21chrt12:13:323
1762799632,24cyclictest27117-21chrt12:13:323
1762799632,23cyclictest721-21chrt07:53:433
1762799632,23cyclictest19424-21taskset09:58:533
1762799632,22cyclictest25296-21chrt10:09:063
1762799632,22cyclictest25296-21chrt10:09:063
1762799632,21cyclictest7114-21chrt10:37:143
1762799632,20cyclictest12860-21chrt10:48:163
17601996347,9cyclictest0-21swapper/110:08:521
17601996347,9cyclictest0-21swapper/110:08:521
17601996346,11cyclictest21326-21sh07:18:181
17601996341,16cyclictest17521-21kworker/u17:0+events_unbound07:58:181
17601996337,20cyclictest17521-21kworker/u17:0-events_unbound07:38:341
1762799628,12cyclictest0-21swapper/312:33:473
1762799623,20cyclictest18288-21chrt09:57:483
1762799622,22cyclictest4964-21chrt08:07:403
1762799622,22cyclictest24659-21chrt07:28:203
1762799622,22cyclictest24659-21chrt07:28:203
1762799622,21cyclictest20655-21chrt08:53:493
17601996238,10cyclictest181rcu_preempt08:28:531
17601996238,10cyclictest181rcu_preempt08:28:531
1762799613,23cyclictest5321-21chrt12:32:243
1762799613,23cyclictest5321-21chrt12:32:243
1762799613,22cyclictest7822-21chrt10:38:353
1762799613,22cyclictest29063-21chrt07:39:013
1762799613,21cyclictest6582-21chrt08:12:473
1762799613,21cyclictest22879-21chrt07:22:333
1762799613,21cyclictest22879-21chrt07:22:333
1762799613,20cyclictest7149-21chrt08:13:403
1762799613,19cyclictest26051-21chrt07:32:013
1762799612,25cyclictest6777-21taskset09:35:143
1762799612,24cyclictest4529-21chrt11:32:203
1762799612,23cyclictest31463-21chrt09:19:463
1762799612,23cyclictest31463-21chrt09:19:463
1762799612,23cyclictest20022-21taskset10:59:303
1762799612,22cyclictest18518-21chrt08:48:353
1762799612,21cyclictest9700-21chrt09:41:183
1762799612,21cyclictest32449-21chrt07:51:163
1762799612,21cyclictest29203-21taskset09:15:493
1762799612,21cyclictest22037-21chrt08:58:433
1762799612,20cyclictest3605-21chrt10:29:323
1762799612,20cyclictest21024-21chrt07:14:483
17601996140,14cyclictest16666-21apt-key08:43:211
1762799609,14cyclictest0-21swapper/307:08:243
1762799609,14cyclictest0-21swapper/307:08:243
1762799602,25cyclictest17394-21taskset11:54:363
1762799602,25cyclictest17394-21taskset11:54:363
1762799602,22cyclictest8522-21chrt08:18:263
1762799602,21cyclictest16254-21chrt08:41:383
1762799592,21cyclictest30794-21chrt07:45:373
17627995910,14cyclictest0-21swapper/308:23:373
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional