You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-07 - 02:13
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack0slot2.osadl.org (updated Mon Jul 07, 2025 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
282467899390,37cyclictest3424529-21latency_hist08:22:477
2824655993735,2cyclictest3228223-21kworker/u16:4+events_unbound06:53:094
2824655993734,2cyclictest3408261-21kworker/u16:3+flush-8:008:23:034
2824655993734,2cyclictest3240111-21kworker/u16:0+flush-8:006:58:024
2824678993429,3cyclictest3332022-21idleruntime-cro07:37:477
2824655993331,2cyclictest2899206-21kworker/u16:0+events_unbound05:03:234
282467899320,30cyclictest3013332-21latency_hist05:02:477
282467899320,30cyclictest2951683-21latency_hist04:32:477
2824655993229,2cyclictest3228223-21kworker/u16:4+events_unbound07:23:464
2824655993129,2cyclictest3393366-21kworker/u16:0+flush-8:008:48:174
2824655993129,2cyclictest3197451-21kworker/u16:3+events_unbound07:08:104
2824655993129,2cyclictest3074051-21kworker/u16:4+events_unbound05:32:464
2824655993129,2cyclictest3074051-21kworker/u16:4+events_unbound05:32:464
2824655993129,2cyclictest2920520-21kworker/u16:1+flush-8:004:23:124
2824655993028,2cyclictest3240111-21kworker/u16:0+events_unbound07:13:164
2824655993027,2cyclictest3413210-21kworker/u16:5+events_unbound08:58:174
282467899290,27cyclictest3260011-21latency_hist07:02:477
2824655992927,2cyclictest2814604-21kworker/u16:4+flush-8:003:49:584
2824655992926,2cyclictest2899206-21kworker/u16:0+events_unbound04:58:044
2824670992826,2cyclictest2899206-21kworker/u16:0+flush-8:005:18:056
2824655992826,2cyclictest3074051-21kworker/u16:4+flush-8:005:52:104
2824655992826,2cyclictest2878701-21kworker/u16:5+flush-8:004:38:164
2824655992825,2cyclictest3096357-21kworker/u16:1+flush-8:007:27:494
282467899270,25cyclictest3198366-21latency_hist06:32:477
2824655992725,2cyclictest3240111-21kworker/u16:0+events_unbound07:38:104
2824655992725,2cyclictest3197451-21kworker/u16:3+events_unbound06:48:234
2824655992725,2cyclictest2958201-21kworker/u16:3+events_unbound05:35:144
2824655992725,2cyclictest2958201-21kworker/u16:3+events_unbound05:35:144
2824655992725,2cyclictest2899206-21kworker/u16:0+events_unbound04:30:584
2824655992725,2cyclictest2878701-21kworker/u16:5+flush-8:004:43:124
2824655992725,2cyclictest2814604-21kworker/u16:4+events_unbound03:54:574
2824678992624,1cyclictest3465826-21awk08:42:467
2824655992624,2cyclictest3455161-21kworker/u16:7+flush-8:008:33:574
2824655992624,2cyclictest3395853-21kworker/u16:2+events_unbound08:18:174
2824655992624,2cyclictest3364079-21kworker/u16:4+flush-8:007:57:464
2824655992624,2cyclictest3364079-21kworker/u16:4+flush-8:007:57:464
2824655992624,2cyclictest3096357-21kworker/u16:1+flush-8:006:07:114
2824655992624,2cyclictest3023806-21kworker/u16:2+events_unbound05:25:144
2824678992522,2cyclictest3311421-21awk07:27:477
2824670992523,2cyclictest3074052-21kworker/u16:5+flush-8:005:48:186
2824655992523,2cyclictest3300119-21kworker/u16:3+flush-8:007:58:584
2824655992422,2cyclictest3002728-21kworker/u16:1+events_unbound04:53:584
2824655992422,2cyclictest2794536-21kworker/u16:3+events_unbound03:46:114
2824655992421,2cyclictest3393366-21kworker/u16:0+events_unbound08:54:564
2824678992322,0cyclictest0-21swapper/704:43:487
2824678992320,2cyclictest3188126-21awk06:27:477
2824655992321,2cyclictest3074052-21kworker/u16:5+events_unbound06:08:574
2824655992320,2cyclictest3364079-21kworker/u16:4+flush-8:008:02:574
282467899220,20cyclictest2992757-21latency_hist04:52:477
2824670992220,2cyclictest3127191-21kworker/u16:2+flush-8:006:12:476
2824655992220,2cyclictest3393366-21kworker/u16:0+flush-8:008:38:024
2824655992220,2cyclictest3228223-21kworker/u16:4+flush-8:007:33:164
2824655992220,2cyclictest3095312-21kworker/u16:0+events_unbound06:42:394
2824655992220,2cyclictest3095312-21kworker/u16:0+events_unbound06:01:104
2824655992220,2cyclictest3023806-21kworker/u16:2+flush-8:005:08:594
2824655992219,2cyclictest3074051-21kworker/u16:4+events_unbound05:56:344
2824678992118,2cyclictest2890000-21munin-run04:02:477
2824678992118,2cyclictest2890000-21munin-run04:02:477
2824655992119,2cyclictest3228223-21kworker/u16:4+flush-8:006:43:474
2824655992119,2cyclictest3095312-21kworker/u16:0+flush-8:006:24:104
2824655992119,2cyclictest3074051-21kworker/u16:4+events_unbound05:40:584
2824655992119,2cyclictest2899206-21kworker/u16:0+flush-8:004:14:024
2824678992019,0cyclictest0-21swapper/705:42:497
2824655992018,2cyclictest2899206-21kworker/u16:0+events_unbound04:51:024
2824655992017,2cyclictest3413210-21kworker/u16:5+flush-8:008:17:344
2824655992017,2cyclictest3364079-21kworker/u16:4+flush-8:008:07:504
2824678991918,0cyclictest0-21swapper/704:38:027
2824655991917,2cyclictest3413210-21kworker/u16:5+events_unbound08:31:254
2824655991917,2cyclictest3197451-21kworker/u16:3+flush-8:007:06:344
2824655991917,2cyclictest3095312-21kworker/u16:0+events_unbound06:18:084
2824655991917,2cyclictest3002728-21kworker/u16:1+events_unbound05:21:574
2824655991917,2cyclictest2878701-21kworker/u16:5+events_unbound04:20:024
2824655991917,2cyclictest2793015-21kworker/u16:2+events_unbound04:00:474
2824655991917,2cyclictest2793015-21kworker/u16:2+events_unbound04:00:474
2824655991917,2cyclictest2060216-21kworker/u16:0+events_unbound03:42:114
2824655991915,3cyclictest2899206-21kworker/u16:0+flush-8:004:33:094
2824655991815,2cyclictest3095312-21kworker/u16:0+events_unbound06:17:154
2824678991717,0cyclictest0-21swapper/705:23:097
282467899170,16cyclictest1313-21snmpd05:29:427
282467899170,16cyclictest1313-21snmpd05:29:427
2824662991715,2cyclictest2793015-21kworker/u16:2+flush-8:004:03:075
2824655991715,2cyclictest3353303-21kworker/u16:2+flush-8:007:52:414
2824655991715,2cyclictest3300119-21kworker/u16:3+flush-8:007:44:484
2824655991715,2cyclictest3002728-21kworker/u16:1+events_unbound05:15:064
2824655991715,2cyclictest2899206-21kworker/u16:0+flush-8:004:08:094
2824678991614,1cyclictest0-21swapper/706:13:487
2824678991614,1cyclictest0-21swapper/705:08:087
282467899160,14cyclictest1313-21snmpd06:46:167
282467899160,14cyclictest1313-21snmpd04:23:377
2824670991614,2cyclictest2920520-21kworker/u16:1+flush-8:004:23:186
2824670991612,3cyclictest3027940-21cat05:08:086
282466299160,2cyclictest3073766-21latency05:28:095
282466299160,2cyclictest3073766-21latency05:28:095
2824655991614,2cyclictest3413210-21kworker/u16:5+events_unbound08:44:344
2824655991614,2cyclictest3096357-21kworker/u16:1+events_unbound06:36:344
282467899152,12cyclictest27904-21gmain04:53:137
2824678991512,2cyclictest3298869-21kthreadcore07:18:107
2824678991511,3cyclictest3230697-21sed06:48:037
2824670991513,2cyclictest2899206-21kworker/u16:0+flush-8:005:13:036
282466299150,1cyclictest0-21swapper/506:03:015
2824655991513,2cyclictest2878701-21kworker/u16:5+events_unbound04:04:104
2824655991512,2cyclictest3127191-21kworker/u16:2+flush-8:006:32:474
2824678991411,2cyclictest3507016-21/usr/sbin/munin09:02:477
282467899141,12cyclictest27430-21gmain04:03:097
282467899141,12cyclictest27204-21gmain05:18:097
282467899140,13cyclictest3218957-21turbostat06:42:477
2824662991413,1cyclictest0-21swapper/506:13:155
282466299140,1cyclictest0-21swapper/507:18:215
282467899137,5cyclictest27642-21gmain07:09:097
282467899136,7cyclictest27924-21gmain05:58:177
282467899136,6cyclictest27763-21gvfs-afc-volume05:13:047
282467899135,6cyclictest27394-21gmain04:08:057
2824678991312,0cyclictest0-21swapper/703:51:117
2824678991311,1cyclictest2961939-21turbostat04:37:477
282467899130,2cyclictest3454511-21ntpq08:33:107
282467899130,12cyclictest3290833-21latency_hist07:17:477
282467899130,11cyclictest3363949-21cat07:53:017
282467899130,11cyclictest3363949-21cat07:53:017
282467899130,11cyclictest3022739-21ntpq05:03:177
2824670991312,1cyclictest0-21swapper/608:33:146
2824670991310,2cyclictest3300119-21kworker/u16:3+flush-8:007:53:176
2824670991310,2cyclictest3300119-21kworker/u16:3+flush-8:007:53:176
2824662991312,1cyclictest0-21swapper/506:23:115
2824662991312,1cyclictest0-21swapper/504:44:585
2824655991311,2cyclictest3096357-21kworker/u16:1+flush-8:007:19:434
282467899129,2cyclictest3241968-21cut06:53:067
2824678991211,0cyclictest0-21swapper/704:20:497
282467899120,3cyclictest3430519-21cut08:23:097
282467899120,2cyclictest3333254-21diskstats07:38:027
282467899120,11cyclictest3482577-21cut08:48:117
282467899120,11cyclictest3475174-21ntpq08:43:157
282467899120,11cyclictest3321661-21latency_hist07:32:477
282467899120,11cyclictest3269303-21ntpq07:03:127
282467899120,10cyclictest3374646-21cat07:58:017
2824670991211,1cyclictest0-21swapper/606:47:566
2824670991211,1cyclictest0-21swapper/606:24:266
2824670991211,1cyclictest0-21swapper/603:56:536
2824670991210,2cyclictest3240111-21kworker/u16:0+flush-8:007:22:476
2824662991211,1cyclictest0-21swapper/508:34:205
2824662991211,1cyclictest0-21swapper/507:28:175
2824662991211,1cyclictest0-21swapper/506:35:475
2824662991211,1cyclictest0-21swapper/505:22:485
2824662991211,1cyclictest0-21swapper/505:12:525
282466299120,1cyclictest0-21swapper/508:10:225
282466299120,1cyclictest0-21swapper/508:07:425
282466299120,1cyclictest0-21swapper/507:46:155
282466299120,1cyclictest0-21swapper/506:19:335
282466299120,1cyclictest0-21swapper/505:11:555
282466299120,1cyclictest0-21swapper/504:18:185
282466299120,1cyclictest0-21swapper/504:09:205
282466299120,12cyclictest0-21swapper/509:01:465
282466299120,12cyclictest0-21swapper/506:12:185
282466299120,12cyclictest0-21swapper/503:40:485
2824655991211,1cyclictest0-21swapper/405:43:284
282467899115,5cyclictest27763-21gvfs-afc-volume08:03:087
282467899110,1cyclictest0-21swapper/706:33:157
282467899110,10cyclictest3356148-21kthreadcore07:48:067
282467899110,10cyclictest3349093-21kthreadcore07:43:117
282467899110,10cyclictest27204-21gmain08:53:017
282467899110,10cyclictest1313-21snmpd08:32:187
282467899110,10cyclictest1313-21snmpd08:11:157
282467899110,10cyclictest1313-21snmpd06:10:287
282467899110,10cyclictest1313-21snmpd05:40:307
282467899110,10cyclictest1313-21snmpd05:36:517
282467899110,10cyclictest1313-21snmpd05:36:517
282467899110,10cyclictest1313-21snmpd03:56:577
282467899110,10cyclictest1313-21snmpd03:45:277
282467099118,2cyclictest1059-21lldpd04:46:036
2824670991110,1cyclictest0-21swapper/608:18:166
2824670991110,1cyclictest0-21swapper/608:15:356
2824670991110,1cyclictest0-21swapper/607:38:136
2824670991110,1cyclictest0-21swapper/607:23:526
2824670991110,1cyclictest0-21swapper/606:44:186
2824670991110,1cyclictest0-21swapper/606:32:296
2824670991110,1cyclictest0-21swapper/606:05:316
2824670991110,1cyclictest0-21swapper/605:57:196
2824670991110,1cyclictest0-21swapper/605:38:346
2824670991110,1cyclictest0-21swapper/605:28:476
2824670991110,1cyclictest0-21swapper/605:28:476
2824670991110,1cyclictest0-21swapper/604:56:006
2824670991110,1cyclictest0-21swapper/603:40:106
282467099110,1cyclictest0-21swapper/608:53:256
282467099110,1cyclictest0-21swapper/607:37:166
282467099110,1cyclictest0-21swapper/604:40:336
282467099110,1cyclictest0-21swapper/604:17:426
282467099110,11cyclictest0-21swapper/608:24:246
282467099110,11cyclictest0-21swapper/607:48:566
282467099110,11cyclictest0-21swapper/607:43:356
282467099110,11cyclictest0-21swapper/607:08:356
282467099110,11cyclictest0-21swapper/607:07:386
282467099110,11cyclictest0-21swapper/605:07:126
282467099110,11cyclictest0-21swapper/604:50:016
282467099110,11cyclictest0-21swapper/603:59:246
282467099110,11cyclictest0-21swapper/603:59:246
282466299117,3cyclictest3117736-21cat05:53:045
282466299117,3cyclictest2984010-21cat04:48:035
282466299116,4cyclictest3015906-21kthreadcore05:03:095
2824662991110,1cyclictest0-21swapper/508:43:105
2824662991110,1cyclictest0-21swapper/508:22:505
2824662991110,1cyclictest0-21swapper/507:27:205
2824662991110,1cyclictest0-21swapper/507:04:485
2824662991110,1cyclictest0-21swapper/504:57:535
2824662991110,1cyclictest0-21swapper/504:56:095
2824662991110,1cyclictest0-21swapper/504:41:305
282466299110,1cyclictest0-21swapper/508:31:395
282466299110,1cyclictest0-21swapper/508:20:095
282466299110,11cyclictest0-21swapper/508:56:245
282466299110,11cyclictest0-21swapper/508:38:455
282466299110,11cyclictest0-21swapper/508:00:365
282466299110,11cyclictest0-21swapper/505:41:335
282467899107,2cyclictest27177-21gmain08:13:017
282467899100,9cyclictest1313-21snmpd06:19:377
282467899100,9cyclictest1313-21snmpd05:47:547
282467899100,2cyclictest3137487-21/usr/sbin/munin06:03:157
282467899100,1cyclictest0-21swapper/705:53:017
282467899100,1cyclictest0-21swapper/704:13:167
282467099109,1cyclictest0-21swapper/607:59:396
282467099109,1cyclictest0-21swapper/607:02:166
282467099109,1cyclictest0-21swapper/606:54:146
282467099109,1cyclictest0-21swapper/605:35:536
282467099109,1cyclictest0-21swapper/605:35:536
282467099109,1cyclictest0-21swapper/604:05:146
282467099108,2cyclictest2878701-21kworker/u16:5+flush-8:004:37:476
282466299109,1cyclictest0-21swapper/506:01:255
282466299109,1cyclictest0-21swapper/504:17:415
282466299108,1cyclictest0-21swapper/506:38:085
282466299108,1cyclictest0-21swapper/505:33:105
282466299108,1cyclictest0-21swapper/505:33:105
28246789990,8cyclictest27174-21gnome-shell03:40:567
28246709998,1cyclictest0-21swapper/608:58:466
28246709998,1cyclictest0-21swapper/607:14:536
28246709998,1cyclictest0-21swapper/606:21:466
28246709990,8cyclictest0-21swapper/606:13:046
28246709990,1cyclictest0-21swapper/606:33:096
28246629998,1cyclictest0-21swapper/507:08:255
28246709987,1cyclictest0-21swapper/608:39:236
28246709987,1cyclictest0-21swapper/606:01:546
28246709986,2cyclictest2920520-21kworker/u16:1+events_unbound04:28:106
28246709985,3cyclictest2814604-21kworker/u16:4+flush-8:003:48:036
28246709984,3cyclictest3217874-21missed_timers06:38:116
28246709984,3cyclictest3096980-21tune2fs05:43:076
28246709983,4cyclictest3012371-21ntpq04:58:136
28246709983,4cyclictest2930174-21ntpq04:18:146
28246709983,4cyclictest2850863-21cut03:43:056
28246629987,1cyclictest0-21swapper/507:49:535
28246629987,1cyclictest0-21swapper/505:48:495
28246629986,1cyclictest0-21swapper/503:53:075
28246629985,2cyclictest1-21systemd07:17:475
28246629984,3cyclictest2954660-21cut04:33:065
28246629983,4cyclictest3096366-21/usr/sbin/munin05:43:075
28246629983,4cyclictest3053786-21sed05:18:165
28246629983,4cyclictest2944492-21cut04:28:135
28246629983,4cyclictest2882329-21grep03:58:065
28246629983,4cyclictest2882329-21grep03:58:065
28246629983,4cyclictest2860594-21awk03:48:035
28246709976,1cyclictest0-21swapper/608:11:106
28246709973,3cyclictest3055207-21grep05:23:036
28246709972,4cyclictest2902955-21cut04:08:056
28246629976,1cyclictest0-21swapper/508:15:445
28246629976,1cyclictest0-21swapper/507:34:355
28246629975,1cyclictest0-21swapper/506:27:595
28246629974,2cyclictest2850026-21entropy03:43:025
28246629973,3cyclictest3220216-21diskstats06:43:025
28246629972,4cyclictest2933787-21kthreadcore04:23:055
28246709964,1cyclictest0-21swapper/608:07:466
28246709960,1cyclictest0-21swapper/608:48:086
28246629965,1cyclictest0-21swapper/506:53:085
28246629965,1cyclictest0-21swapper/506:49:305
28246629960,2cyclictest0-21swapper/507:39:515
28246789950,4cyclictest2835256-21sendmail03:37:487
28246789950,4cyclictest2824086-21cron03:32:497
28246709950,4cyclictest3466541-21taskset08:42:576
28246709950,4cyclictest3444849-21chrt08:28:396
28246709950,4cyclictest3312134-21taskset07:27:586
28246629950,4cyclictest3476828-21chrt08:47:535
28246629950,4cyclictest3363553-21chrt07:52:485
28246629950,4cyclictest3363553-21chrt07:52:485
28246629950,4cyclictest3259165-21chrt06:58:165
28246559950,4cyclictest2835276-21903:37:484
28246629940,3cyclictest2835270-21sd-rmrf03:37:485
28246709930,2cyclictest0-21swapper/603:37:486
28246629921,1cyclictest0-21swapper/503:32:495
28246709910,1cyclictest0-21swapper/603:32:496
28246559910,1cyclictest0-21swapper/403:32:494
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional