You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 12:31
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack0slot2.osadl.org (updated Wed Nov 26, 2025 00:43:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32845022990,1sleep73284500-21ls08:33:177
265465825141,7sleep40-21swapper/403:23:144
265456425040,7sleep60-21swapper/603:21:566
2655028994644,2cyclictest2780347-21kworker/u16:3+events_unbound04:33:436
2655033994539,5cyclictest3122945-21latency_hist07:13:167
265454324535,7sleep50-21swapper/503:21:395
2655028994240,2cyclictest2644436-21kworker/u16:1+flush-8:003:43:476
2655033994140,0cyclictest0-21swapper/706:41:507
2655020994137,3cyclictest2749582-21kworker/u16:2+events_unbound04:38:424
2655033994039,0cyclictest0-21swapper/704:49:527
2655028994038,2cyclictest2644436-21kworker/u16:1+flush-8:003:33:446
26544882394,31sleep70-21swapper/703:20:537
2655033993837,0cyclictest0-21swapper/707:52:517
2655028993836,2cyclictest3203300-21kworker/u16:1+flush-8:007:58:546
2655028993836,2cyclictest1924053-21kworker/u16:3+events_unbound03:55:116
2655028993735,2cyclictest3032549-21kworker/u16:4+events_unbound06:48:456
2655020993735,2cyclictest2810253-21kworker/u16:4+flush-8:005:18:404
2655028993634,2cyclictest2780347-21kworker/u16:3+events_unbound04:53:416
2655033993530,4cyclictest3092691-21latency_hist06:58:177
2655028993533,2cyclictest3082073-21kworker/u16:0+events_unbound07:35:576
2655028993533,2cyclictest2931262-21kworker/u16:3+events_unbound05:48:206
2655028993432,2cyclictest3173536-21kworker/u16:3+flush-8:007:48:266
2655028993432,2cyclictest3032549-21kworker/u16:4+events_unbound07:09:286
2655028993432,2cyclictest2890476-21kworker/u16:1+flush-8:005:22:516
2655028993432,2cyclictest2749582-21kworker/u16:2+events_unbound04:40:496
2655028993432,2cyclictest2664933-21kworker/u16:4+events_unbound03:41:506
2655028993432,2cyclictest2664933-21kworker/u16:4+events_unbound03:41:506
2655033993332,0cyclictest0-21swapper/705:59:527
2655028993331,2cyclictest3173536-21kworker/u16:3+flush-8:007:53:396
2655028993331,2cyclictest3173536-21kworker/u16:3+events_unbound08:27:036
2655028993331,2cyclictest3032549-21kworker/u16:4+flush-8:007:39:286
2655028993331,2cyclictest2810253-21kworker/u16:4+events_unbound06:05:286
2655025993330,2cyclictest2890476-21kworker/u16:1+flush-8:005:23:175
2655033993231,0cyclictest0-21swapper/707:25:557
2655033993229,2cyclictest2950884-21unixbench_multi05:43:507
2655028993230,2cyclictest3062437-21kworker/u16:2+events_unbound06:47:516
2655020993230,2cyclictest2865481-21kworker/u16:2+flush-8:005:08:424
2655033993126,4cyclictest2941236-21latency_hist05:43:177
2655028993129,2cyclictest3244334-21kworker/u16:4+events_unbound08:28:436
2655028993129,2cyclictest3122767-21kworker/u16:2+events_unbound07:19:286
2655028993129,2cyclictest2931262-21kworker/u16:3+flush-8:006:18:146
2655028993129,2cyclictest2865481-21kworker/u16:2+events_unbound05:40:396
2655028993129,2cyclictest2644436-21kworker/u16:1+flush-8:004:00:516
2655020993129,2cyclictest2931262-21kworker/u16:3+flush-8:005:58:464
2655020993129,2cyclictest2890476-21kworker/u16:1+flush-8:005:23:474
265503399300,28cyclictest2910966-21latency_hist05:28:177
2655028993028,2cyclictest3244334-21kworker/u16:4+events_unbound08:21:196
2655028993028,2cyclictest3122767-21kworker/u16:2+flush-8:008:44:496
2655028993028,2cyclictest3122767-21kworker/u16:2+flush-8:007:15:036
2655028993028,2cyclictest3082073-21kworker/u16:0+events_unbound07:01:516
2655028993028,2cyclictest3052081-21kworker/u16:1+flush-8:006:40:356
2655028993028,2cyclictest2931262-21kworker/u16:3+flush-8:006:32:236
2655028993028,2cyclictest2912166-21kworker/u16:0+events_unbound06:37:276
2655028993028,2cyclictest2810253-21kworker/u16:4+flush-8:005:15:486
2655028993028,2cyclictest2749582-21kworker/u16:2+events_unbound04:15:306
2655020993028,2cyclictest3082073-21kworker/u16:0+flush-8:007:28:444
2655020993028,2cyclictest3052081-21kworker/u16:1+flush-8:006:43:174
2655020993028,2cyclictest2931262-21kworker/u16:3+flush-8:006:23:464
2655033992928,0cyclictest0-21swapper/708:35:517
2655033992928,0cyclictest0-21swapper/708:03:197
2655033992924,4cyclictest3173324-21latency_hist07:38:177
2655033992923,5cyclictest589235-21systemd-journal07:53:497
2655028992927,2cyclictest2810253-21kworker/u16:4+flush-8:006:01:556
2655028992927,2cyclictest2709027-21kworker/u16:0+events_unbound04:19:506
2655020992927,2cyclictest2912166-21kworker/u16:0+events_unbound05:53:314
2655020992927,2cyclictest2912166-21kworker/u16:0+events_unbound05:53:314
2655033992827,0cyclictest0-21swapper/704:26:277
2655033992827,0cyclictest0-21swapper/704:20:227
2655028992826,2cyclictest3304208-21kworker/u16:4+events_unbound08:40:116
2655028992826,2cyclictest3122767-21kworker/u16:2+events_unbound08:17:176
2655028992826,2cyclictest2810253-21kworker/u16:4+events_unbound05:55:466
2655028992826,2cyclictest2810253-21kworker/u16:4+events_unbound05:55:466
2655028992826,2cyclictest2809941-21kworker/u16:0+flush-8:005:11:506
2655020992826,2cyclictest3082073-21kworker/u16:0+flush-8:007:33:434
2655020992826,2cyclictest3016353-21kworker/u16:1+events_unbound06:28:304
2655020992826,2cyclictest2709027-21kworker/u16:0+flush-8:004:08:364
2655020992826,2cyclictest2644436-21kworker/u16:1+flush-8:004:28:334
2655033992721,5cyclictest2664968-21latency_hist03:28:177
2655028992725,2cyclictest3122767-21kworker/u16:2+flush-8:007:24:476
2655028992725,2cyclictest2865481-21kworker/u16:2+events_unbound05:26:276
2655028992725,2cyclictest2810253-21kworker/u16:4+events_unbound06:09:546
2655028992724,2cyclictest2809941-21kworker/u16:0+flush-8:004:58:276
2655025992723,3cyclictest3032549-21kworker/u16:4+flush-8:007:38:465
2655020992725,2cyclictest2810253-21kworker/u16:4+events_unbound05:38:304
2655033992625,0cyclictest0-21swapper/708:26:137
2655033992625,0cyclictest0-21swapper/705:00:187
265503399260,24cyclictest589235-21systemd-journal08:46:097
2655028992624,2cyclictest3173536-21kworker/u16:3+events_unbound08:07:396
2655028992624,2cyclictest3122767-21kworker/u16:2+events_unbound08:49:346
2655028992624,2cyclictest3032549-21kworker/u16:4+flush-8:006:57:266
2655028992624,2cyclictest3032549-21kworker/u16:4+events_unbound07:05:266
2655028992624,2cyclictest2912166-21kworker/u16:0+events_unbound06:23:156
2655028992624,2cyclictest2780347-21kworker/u16:3+events_unbound04:49:446
2655028992624,2cyclictest2749582-21kworker/u16:2+flush-8:004:11:156
2655028992624,2cyclictest2709027-21kworker/u16:0+flush-8:004:05:516
2655028992624,2cyclictest2644436-21kworker/u16:1+events_unbound03:32:196
2655020992624,2cyclictest3173536-21kworker/u16:3+flush-8:008:33:314
2655033992524,0cyclictest0-21swapper/707:38:327
2655033992523,1cyclictest0-21swapper/703:44:277
2655033992521,2cyclictest3242962-21ntpq08:08:427
2655033992521,2cyclictest3242962-21ntpq08:08:427
2655028992523,2cyclictest2865481-21kworker/u16:2+events_unbound05:06:486
2655028992523,2cyclictest2780347-21kworker/u16:3+flush-8:004:32:596
2655025992522,2cyclictest3173536-21kworker/u16:3+flush-8:008:23:465
2655020992523,2cyclictest3274084-21kworker/u16:1+flush-8:008:38:424
2655020992523,2cyclictest2664933-21kworker/u16:4+flush-8:003:28:234
2655020992521,3cyclictest3122767-21kworker/u16:2+flush-8:008:43:364
2655033992423,0cyclictest0-21swapper/707:17:317
2655033992423,0cyclictest0-21swapper/704:47:287
2655028992422,2cyclictest3203300-21kworker/u16:1+events_unbound08:12:386
2655028992422,2cyclictest3203300-21kworker/u16:1+events_unbound08:12:386
2655028992422,2cyclictest3173536-21kworker/u16:3+flush-8:008:35:266
2655028992422,2cyclictest2990791-21kworker/u16:2+flush-8:006:28:166
2655028992422,2cyclictest2749582-21kworker/u16:2+events_unbound04:45:076
2655028992422,2cyclictest2709027-21kworker/u16:0+events_unbound03:48:566
2655028992422,2cyclictest2644436-21kworker/u16:1+flush-8:003:25:276
2655025992421,2cyclictest2890476-21kworker/u16:1+flush-8:005:28:325
2655020992422,2cyclictest3102597-21kworker/u16:1+events_unbound07:08:324
2655020992422,2cyclictest2739412-21kworker/u16:4+events_unbound04:03:334
2655020992422,2cyclictest2644436-21kworker/u16:1+flush-8:004:23:124
2655020992420,3cyclictest2809941-21kworker/u16:0+flush-8:005:13:364
2655033992318,3cyclictest2688228-21/usr/sbin/munin03:33:437
2655028992321,2cyclictest2912166-21kworker/u16:0+flush-8:005:28:526
2655025992321,2cyclictest2780347-21kworker/u16:3+flush-8:004:33:445
2655020992321,2cyclictest3102597-21kworker/u16:1+flush-8:007:03:284
2655020992321,2cyclictest2912166-21kworker/u16:0+events_unbound06:33:314
2655020992321,2cyclictest2644436-21kworker/u16:1+flush-8:004:13:424
2655033992221,0cyclictest0-21swapper/707:18:487
2655033992221,0cyclictest0-21swapper/704:56:357
2655028992220,2cyclictest2931262-21kworker/u16:3+flush-8:005:47:306
2655028992220,2cyclictest2810253-21kworker/u16:4+events_unbound05:35:366
2655033992120,0cyclictest0-21swapper/704:06:517
2655033992115,5cyclictest3112886-21latency_hist07:08:177
2655028992119,2cyclictest2644436-21kworker/u16:1+events_unbound04:26:356
2655028992118,2cyclictest3082073-21kworker/u16:0+flush-8:007:29:166
2655020992119,2cyclictest3122767-21kworker/u16:2+flush-8:008:48:474
2655020992119,2cyclictest3032549-21kworker/u16:4+flush-8:007:38:444
2655033992018,1cyclictest2991926-21sed06:08:177
2655025992018,2cyclictest2912166-21kworker/u16:0+flush-8:005:43:335
2655020992018,2cyclictest2810253-21kworker/u16:4+flush-8:006:13:334
2655020992018,2cyclictest2810253-21kworker/u16:4+flush-8:005:36:184
2655020992018,2cyclictest2644436-21kworker/u16:1+flush-8:003:58:414
2655033991918,0cyclictest0-21swapper/708:51:187
2655020991917,2cyclictest3173536-21kworker/u16:3+flush-8:008:23:434
2655020991917,2cyclictest3032549-21kworker/u16:4+flush-8:007:23:334
2655020991917,2cyclictest2749582-21kworker/u16:2+flush-8:004:33:334
2655020991917,2cyclictest2709027-21kworker/u16:0+events_unbound03:49:174
2655020991917,2cyclictest1924053-21kworker/u16:3+flush-8:003:53:404
2655033991817,0cyclictest0-21swapper/708:18:417
2655025991813,4cyclictest2932317-21cat05:38:315
2655020991816,2cyclictest3173536-21kworker/u16:3+events_unbound08:28:354
2655020991816,2cyclictest2912166-21kworker/u16:0+events_unbound05:28:444
2655020991816,2cyclictest2810253-21kworker/u16:4+events_unbound06:08:324
2655033991716,1cyclictest0-21swapper/705:58:127
2655033991716,1cyclictest0-21swapper/705:58:127
2655033991716,0cyclictest0-21swapper/706:48:327
2655033991716,0cyclictest0-21swapper/705:32:507
2655033991711,5cyclictest589235-21systemd-journal07:47:497
2655028991715,2cyclictest3173536-21kworker/u16:3+flush-8:007:45:266
265502599170,16cyclictest2714677-21cut03:48:385
2655020991715,2cyclictest3122767-21kworker/u16:2+events_unbound08:03:444
2655020991715,2cyclictest3082073-21kworker/u16:0+events_unbound06:52:554
2655020991715,2cyclictest2809941-21kworker/u16:0+flush-8:005:03:394
2655020991714,2cyclictest3062437-21kworker/u16:2+flush-8:006:43:444
2655033991615,0cyclictest0-21swapper/705:17:287
2655033991615,0cyclictest0-21swapper/705:03:277
2655033991615,0cyclictest0-21swapper/704:12:527
2655033991614,1cyclictest0-21swapper/706:33:167
2655033991613,2cyclictest4175659-21snmpd05:49:167
2655033991610,5cyclictest589235-21systemd-journal04:38:177
2655020991614,2cyclictest3203300-21kworker/u16:1+flush-8:007:58:294
2655020991614,2cyclictest3203300-21kworker/u16:1+events_unbound07:53:444
2655020991614,2cyclictest2912166-21kworker/u16:0+flush-8:006:21:234
2655020991614,2cyclictest2780347-21kworker/u16:3+flush-8:004:55:174
265503399159,5cyclictest3223745-21latency_hist08:03:167
265503399158,6cyclictest0-21swapper/708:40:507
2655033991514,0cyclictest0-21swapper/707:01:327
2655033991514,0cyclictest0-21swapper/706:33:477
265503399150,13cyclictest976-21in:imjournal04:02:497
2655025991513,2cyclictest3032549-21kworker/u16:4+flush-8:007:13:335
2655020991513,2cyclictest3082073-21kworker/u16:0+flush-8:007:20:514
2655020991513,2cyclictest2809941-21kworker/u16:0+events_unbound04:48:324
2655020991513,2cyclictest2644436-21kworker/u16:1+events_unbound03:23:514
2655033991413,0cyclictest0-21swapper/703:53:377
2655025991410,3cyclictest3122767-21kworker/u16:2+flush-8:008:44:205
2655020991412,2cyclictest3203300-21kworker/u16:1+flush-8:008:09:544
2655020991412,2cyclictest3203300-21kworker/u16:1+flush-8:008:09:544
2655020991412,2cyclictest3122767-21kworker/u16:2+events_unbound08:19:184
2655020991412,2cyclictest3122767-21kworker/u16:2+events_unbound08:16:304
2655020991412,2cyclictest2990791-21kworker/u16:2+events_unbound06:05:184
2655020991412,2cyclictest2810253-21kworker/u16:4+events_unbound05:52:404
2655020991412,2cyclictest2664933-21kworker/u16:4+events_unbound03:38:324
2655020991412,2cyclictest2664933-21kworker/u16:4+events_unbound03:38:324
2655020991412,2cyclictest2644436-21kworker/u16:1+flush-8:004:23:454
2655020991412,2cyclictest2644436-21kworker/u16:1+events_unbound03:33:434
265503399138,4cyclictest3163217-21latency_hist07:33:177
265503399137,5cyclictest3022027-21latency_hist06:23:177
265502599139,3cyclictest3122767-21kworker/u16:2+flush-8:008:48:485
2655025991311,2cyclictest2931262-21kworker/u16:3+events_unbound05:48:365
2655025991310,2cyclictest2810253-21kworker/u16:4+flush-8:005:23:335
265502599130,1cyclictest0-21swapper/507:33:175
265502599130,13cyclictest0-21swapper/508:14:265
265502599130,12cyclictest3003512-21ps06:13:345
265502599130,0cyclictest0-21swapper/506:03:355
2655020991311,2cyclictest3122767-21kworker/u16:2+events_unbound07:44:514
2655020991311,2cyclictest2931262-21kworker/u16:3+events_unbound05:47:374
2655020991311,2cyclictest2810253-21kworker/u16:4+events_unbound05:03:114
2655020991311,2cyclictest2749582-21kworker/u16:2+flush-8:004:43:224
2655020991311,2cyclictest1924053-21kworker/u16:3+flush-8:003:48:054
265503399129,2cyclictest4175659-21snmpd06:28:017
265503399129,2cyclictest4175659-21snmpd03:28:287
265503399129,2cyclictest0-21swapper/703:48:417
265503399128,3cyclictest27924-21gmain03:38:397
265503399128,3cyclictest27924-21gmain03:38:397
265503399127,4cyclictest589235-21systemd-journal05:10:327
265503399120,11cyclictest0-21swapper/704:28:447
265502599128,3cyclictest3103966-21cat07:03:305
265502599128,3cyclictest2644436-21kworker/u16:1+flush-8:003:43:365
2655025991211,1cyclictest0-21swapper/504:29:005
265502599120,1cyclictest0-21swapper/505:15:215
265502599120,12cyclictest0-21swapper/506:20:275
265502599120,12cyclictest0-21swapper/505:11:055
2655020991210,2cyclictest3082073-21kworker/u16:0+events_unbound06:54:574
2655020991210,2cyclictest3032549-21kworker/u16:4+events_unbound07:13:254
265502099120,12cyclictest0-21swapper/406:58:214
265503399118,2cyclictest4175659-21snmpd04:41:387
265503399118,2cyclictest0-21swapper/705:18:327
265503399115,5cyclictest27174-21gnome-shell06:44:167
265503399111,9cyclictest3001892-21munin-run06:13:177
265503399110,4cyclictest2929369-21cut05:33:417
265503399110,10cyclictest3011024-21ntpq06:13:437
265502599119,2cyclictest2809941-21kworker/u16:0+flush-8:004:53:345
265502599117,3cyclictest2972755-21cat05:58:335
265502599116,4cyclictest3043443-21cat06:33:335
2655025991110,1cyclictest0-21swapper/508:22:385
2655025991110,1cyclictest0-21swapper/508:02:185
2655025991110,1cyclictest0-21swapper/507:28:245
2655025991110,1cyclictest0-21swapper/507:23:315
2655025991110,1cyclictest0-21swapper/507:18:375
2655025991110,1cyclictest0-21swapper/506:50:345
2655025991110,1cyclictest0-21swapper/505:06:505
2655025991110,1cyclictest0-21swapper/503:30:405
265502599110,1cyclictest0-21swapper/508:30:025
265502599110,11cyclictest0-21swapper/508:38:145
265502599110,11cyclictest0-21swapper/507:08:515
265502599110,11cyclictest0-21swapper/506:44:445
265502599110,11cyclictest0-21swapper/506:42:135
265502599110,11cyclictest0-21swapper/505:00:035
265502599110,11cyclictest0-21swapper/504:18:555
265502599110,11cyclictest0-21swapper/504:01:345
265502599110,0cyclictest0-21swapper/507:53:285
2655020991110,1cyclictest0-21swapper/407:48:574
265503399104,5cyclictest589235-21systemd-journal04:13:177
265503399100,9cyclictest4175659-21snmpd08:13:437
265502599109,1cyclictest0-21swapper/508:41:335
265502599109,1cyclictest0-21swapper/506:58:555
265502599109,1cyclictest0-21swapper/506:33:045
265502599109,1cyclictest0-21swapper/503:42:585
265502599109,1cyclictest0-21swapper/503:42:585
26550259998,1cyclictest0-21swapper/504:48:145
26550259998,1cyclictest0-21swapper/504:39:535
26550259998,1cyclictest0-21swapper/504:26:295
26550259998,1cyclictest0-21swapper/503:28:095
26550259987,1cyclictest0-21swapper/504:12:085
26550259987,1cyclictest0-21swapper/503:53:325
26550259987,1cyclictest0-21swapper/503:37:175
26550259984,3cyclictest3087992-21kthreadcore06:53:385
26550259983,4cyclictest3028302-21cut06:23:405
26550259983,4cyclictest2996711-21grep06:08:365
26550259983,4cyclictest2963832-21grep05:53:355
26550259983,4cyclictest2963832-21grep05:53:355
26550259983,4cyclictest2922332-21irqcore05:33:365
26550259980,1cyclictest0-21swapper/504:48:365
26550259974,2cyclictest3224847-21fschecks_time08:03:305
26550259973,3cyclictest3184230-21expr07:43:295
26550259973,3cyclictest2740248-21expr04:03:305
26550259972,4cyclictest3201185-21kthreadcore07:48:385
26550259972,4cyclictest2760273-21cat04:13:305
26550259960,1cyclictest0-21swapper/508:10:375
26550259960,1cyclictest0-21swapper/508:10:375
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional