You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-02-23 - 05:31
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot2.osadl.org (updated Fri Feb 23, 2024 00:43:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
396059225225,8sleep50-21swapper/519:05:125
39597992464,35sleep70-21swapper/719:02:497
396038023323,7sleep40-21swapper/419:03:104
3960977992722,4cyclictest634-21systemd-journal20:32:084
396066822312,8sleep60-21swapper/619:06:176
3960982992220,2cyclictest41788-21kworker/u16:3+events_unbound22:17:046
3960982992119,2cyclictest4012043-21kworker/u16:2+events_unbound19:42:416
3960982992117,3cyclictest4109623-21latency_hist20:22:086
396098699200,19cyclictest0-21swapper/722:17:157
396098699200,19cyclictest0-21swapper/722:17:087
3960982991917,2cyclictest4080306-21kworker/u16:1+flush-8:020:32:396
3960982991917,2cyclictest322255-21kworker/u16:0+events_unbound23:52:366
3960982991917,2cyclictest322255-21kworker/u16:0+events_unbound23:52:366
3960982991917,2cyclictest32158-21kworker/u16:1+events_unbound23:42:326
396098099193,5cyclictest0-21swapper/521:39:185
396098699180,16cyclictest0-21swapper/719:47:417
3960982991817,1cyclictest41788-21kworker/u16:3+events_unbound23:04:186
3960982991816,2cyclictest4186456-21kworker/u16:3+events_unbound21:03:596
3960982991816,2cyclictest41788-21kworker/u16:3+events_unbound22:52:386
3960982991816,2cyclictest4012043-21kworker/u16:2+flush-8:020:12:436
3960982991816,2cyclictest4012043-21kworker/u16:2+events_unbound19:32:436
3960980991813,4cyclictest4022687-21latency_hist19:37:095
3960977991815,2cyclictest1155-21snmpd19:51:514
396098699173,5cyclictest0-21swapper/720:24:527
396098699170,1cyclictest0-21swapper/720:32:177
3960982991715,2cyclictest4080306-21kworker/u16:1+events_unbound20:04:116
3960982991715,2cyclictest138067-21kworker/u16:0+flush-8:022:47:036
3960982991713,3cyclictest331901-21kworker/u16:4+events_unbound00:12:386
396098099170,1cyclictest0-21swapper/520:57:175
396098699160,6cyclictest0-21swapper/722:02:257
396098699160,14cyclictest0-21swapper/721:37:437
3960982991614,2cyclictest4080306-21kworker/u16:1+events_unbound20:46:536
3960982991614,2cyclictest32158-21kworker/u16:1+flush-8:021:57:296
3960982991614,2cyclictest253394-21kworker/u16:2+events_unbound00:19:566
3960982991612,3cyclictest32158-21kworker/u16:1+events_unbound22:27:266
3960977991614,1cyclictest31742-21taskset21:21:454
396097799160,1cyclictest0-21swapper/423:34:164
396097799160,1cyclictest0-21swapper/423:14:164
3960986991512,2cyclictest0-21swapper/722:07:397
396098699150,6cyclictest0-21swapper/723:27:217
396098699150,6cyclictest0-21swapper/723:03:147
396098699150,6cyclictest0-21swapper/721:09:177
396098699150,6cyclictest0-21swapper/700:27:407
396098699150,1cyclictest0-21swapper/722:01:177
396098699150,1cyclictest0-21swapper/720:18:177
3960982991513,2cyclictest41788-21kworker/u16:3+events_unbound22:35:526
3960982991513,2cyclictest3970833-21kworker/u16:4+events_unbound19:13:596
3960982991513,2cyclictest2403-21kworker/u16:4+events_unbound21:43:536
3960982991513,2cyclictest138067-21kworker/u16:0+events_unbound23:23:236
396098299150,1cyclictest0-21swapper/622:18:176
396098299150,1cyclictest0-21swapper/619:08:176
396098299150,1cyclictest0-21swapper/600:02:406
396098099158,2cyclictest1155-21snmpd22:11:015
396098099158,2cyclictest1155-21snmpd19:07:465
396098099150,1cyclictest0-21swapper/523:48:165
396098099150,1cyclictest0-21swapper/520:01:175
396097799152,5cyclictest0-21swapper/421:24:334
3960977991514,1cyclictest964-21runrttasks19:26:424
3960977991514,1cyclictest124027-21tr22:07:354
3960977991513,2cyclictest3187130-21gnome-shell00:05:164
3960977991513,1cyclictest92659-21kthreadcore21:52:334
3960977991513,1cyclictest323582-21kthreadcore23:52:364
3960977991513,1cyclictest323582-21kthreadcore23:52:364
396097799150,1cyclictest0-21swapper/423:41:164
396097799150,1cyclictest0-21swapper/423:03:164
396097799150,1cyclictest0-21swapper/423:01:164
396097799150,1cyclictest0-21swapper/422:13:174
396097799150,1cyclictest0-21swapper/420:45:174
396097799150,1cyclictest0-21swapper/420:11:174
396097799150,1cyclictest0-21swapper/420:06:174
396097799150,1cyclictest0-21swapper/419:56:174
396097799150,1cyclictest0-21swapper/419:42:174
396097799150,1cyclictest0-21swapper/400:29:164
396097799150,1cyclictest0-21swapper/400:01:164
396097799150,1cyclictest0-21swapper/400:01:164
396098699143,9cyclictest0-21swapper/723:39:007
396098699140,6cyclictest0-21swapper/723:55:077
396098699140,6cyclictest0-21swapper/723:55:077
396098699140,6cyclictest0-21swapper/723:43:427
396098699140,6cyclictest0-21swapper/722:24:557
396098699140,5cyclictest0-21swapper/721:12:147
396098699140,1cyclictest0-21swapper/719:18:177
396098699140,1cyclictest0-21swapper/719:18:177
396098699140,13cyclictest0-21swapper/723:34:097
396098699140,13cyclictest0-21swapper/720:45:407
3960982991412,2cyclictest4080306-21kworker/u16:1+flush-8:020:37:346
3960982991412,2cyclictest3868713-21kworker/u16:0+events_unbound19:52:396
3960982991412,2cyclictest277882-21kworker/u16:3+events_unbound23:37:376
3960982991412,2cyclictest105231-21kworker/u16:0+events_unbound22:02:156
3960982991412,1cyclictest70359-21cat21:42:086
396098299140,6cyclictest0-21swapper/620:47:306
396098299140,1cyclictest0-21swapper/600:30:446
396098099149,2cyclictest175808-21swap22:32:465
3960980991412,2cyclictest0-21swapper/519:48:335
3960980991411,1cyclictest0-21swapper/519:23:205
396098099140,1cyclictest0-21swapper/523:39:585
396098099140,1cyclictest0-21swapper/521:54:125
396098099140,1cyclictest0-21swapper/520:06:395
396098099140,1cyclictest0-21swapper/519:53:175
396098099140,1cyclictest0-21swapper/500:25:035
3960977991413,1cyclictest994-21lldpd00:32:364
3960977991413,1cyclictest4177358-21sed20:57:084
3960977991413,1cyclictest196096-21sed22:47:074
3960977991413,1cyclictest184641-21kthreadcore22:37:364
3960977991413,1cyclictest1606-21sendmail00:12:304
3960977991413,1cyclictest1155-21snmpd21:39:494
3960977991413,1cyclictest1155-21snmpd19:09:354
3960977991413,1cyclictest110069-21/usr/sbin/munin22:02:334
3960977991413,1cyclictest0-21swapper/421:12:314
3960977991412,1cyclictest46002-21kthreadcore21:27:354
3960977991412,1cyclictest46002-21kthreadcore21:27:354
3960977991411,3cyclictest3188205-21gmain20:22:274
396098699135,7cyclictest0-21swapper/722:57:597
3960986991311,1cyclictest0-21swapper/719:12:287
396098699130,6cyclictest0-21swapper/723:25:487
396098699130,6cyclictest0-21swapper/723:18:367
396098699130,6cyclictest0-21swapper/719:46:247
396098699130,1cyclictest0-21swapper/720:28:177
396098699130,12cyclictest0-21swapper/700:14:407
396098699130,11cyclictest0-21swapper/700:09:197
3960982991311,2cyclictest4138954-21kworker/u16:0+flush-8:020:52:416
3960982991311,2cyclictest4138954-21kworker/u16:0+events_unbound21:15:526
3960982991311,2cyclictest3960491-21kworker/u16:1+events_unbound19:27:306
3960982991311,2cyclictest277882-21kworker/u16:3+events_unbound23:31:226
3960982991311,2cyclictest277882-21kworker/u16:3+events_unbound00:00:176
3960982991311,2cyclictest277882-21kworker/u16:3+events_unbound00:00:176
3960982991311,2cyclictest253394-21kworker/u16:2+flush-8:023:47:386
396098299130,13cyclictest0-21swapper/619:17:296
396098299130,13cyclictest0-21swapper/619:17:296
3960980991312,1cyclictest0-21swapper/500:32:475
3960980991311,2cyclictest0-21swapper/521:19:215
3960980991310,1cyclictest0-21swapper/520:26:525
3960980991310,1cyclictest0-21swapper/520:20:245
396098099130,5cyclictest0-21swapper/521:57:135
396098099130,1cyclictest0-21swapper/520:47:395
396098099130,13cyclictest0-21swapper/522:22:535
396098099130,11cyclictest0-21swapper/520:33:295
396097799137,5cyclictest0-21swapper/423:12:074
3960977991312,1cyclictest905-21in:imjournal22:27:334
3960977991312,1cyclictest87274-21cut21:47:364
3960977991312,1cyclictest4024379-21kthreadcore19:37:344
3960977991312,1cyclictest1155-21snmpd23:49:074
3960977991311,2cyclictest3187130-21gnome-shell19:13:174
3960977991311,1cyclictest3045-21ssh21:07:104
396098699129,2cyclictest0-21swapper/721:27:427
396098699129,2cyclictest0-21swapper/721:27:427
396098699123,8cyclictest0-21swapper/722:43:357
396098699122,9cyclictest0-21swapper/719:54:217
3960986991211,0cyclictest0-21swapper/721:34:377
3960986991211,0cyclictest0-21swapper/721:34:377
396098699120,1cyclictest0-21swapper/719:23:177
396098699120,11cyclictest0-21swapper/720:57:417
396098699120,11cyclictest0-21swapper/700:05:137
3960982991211,1cyclictest0-21swapper/622:10:176
3960982991210,2cyclictest41788-21kworker/u16:3+events_unbound23:07:496
3960982991210,2cyclictest4138954-21kworker/u16:0+events_unbound21:19:336
3960982991210,2cyclictest4138954-21kworker/u16:0+events_unbound21:07:286
3960982991210,2cyclictest4080306-21kworker/u16:1+flush-8:020:07:296
3960982991210,2cyclictest277882-21kworker/u16:3+events_unbound00:11:576
3960982991210,2cyclictest1216842-21kworker/u16:3+events_unbound19:50:186
3960982991210,2cyclictest1216842-21kworker/u16:3+events_unbound19:22:376
396098299120,1cyclictest0-21swapper/622:25:436
396098299120,1cyclictest0-21swapper/621:53:246
396098299120,1cyclictest0-21swapper/620:28:076
396098299120,12cyclictest0-21swapper/620:59:116
396098099129,2cyclictest1155-21snmpd20:13:515
3960980991211,1cyclictest0-21swapper/523:54:095
3960980991211,1cyclictest0-21swapper/523:54:095
3960980991211,1cyclictest0-21swapper/523:04:115
3960980991211,1cyclictest0-21swapper/522:38:015
3960980991211,1cyclictest0-21swapper/521:50:155
3960980991211,1cyclictest0-21swapper/520:29:325
396097799129,2cyclictest4137742-21processes20:32:444
3960977991211,1cyclictest41788-21kworker/u16:3-events_unbound22:17:324
3960977991211,1cyclictest3986960-21kthreadcore19:17:314
3960977991211,1cyclictest3986960-21kthreadcore19:17:314
3960977991211,1cyclictest3960970-21cyclictest20:17:344
3960977991211,1cyclictest273431-21cpuspeed23:27:264
3960977991211,1cyclictest0-21swapper/423:17:354
3960977991211,1cyclictest0-21swapper/422:35:074
3960977991210,2cyclictest3187130-21gnome-shell20:51:174
3960977991210,1cyclictest377995-21ntpq00:17:454
3960977991210,1cyclictest2397-21PK-Backend21:04:004
3960977991210,1cyclictest0-21swapper/400:07:414
396097799120,5cyclictest72761-21kthreadcore21:42:354
396097799120,4cyclictest4003974-21cut19:27:274
396097799120,1cyclictest0-21swapper/420:59:164
396097799120,1cyclictest0-21swapper/419:57:194
396097799120,1cyclictest0-21swapper/400:25:164
396098699118,2cyclictest0-21swapper/720:14:127
396098699113,7cyclictest0-21swapper/700:37:087
396098699112,8cyclictest0-21swapper/722:50:177
3960986991110,0cyclictest0-21swapper/700:24:247
396098699110,10cyclictest0-21swapper/719:37:307
396098699110,10cyclictest0-21swapper/719:35:117
396098299119,2cyclictest41788-21kworker/u16:3+events_unbound22:48:546
396098299119,2cyclictest253394-21kworker/u16:2+events_unbound23:19:116
396098299119,2cyclictest2403-21kworker/u16:4+events_unbound21:49:336
396098299119,2cyclictest2403-21kworker/u16:4+events_unbound21:31:246
396098299119,2cyclictest2403-21kworker/u16:4+events_unbound21:31:246
396098299117,3cyclictest177226-21tr22:37:266
3960982991110,1cyclictest0-21swapper/623:01:406
396098299110,11cyclictest0-21swapper/621:32:556
396098299110,11cyclictest0-21swapper/621:32:556
396098099118,2cyclictest1155-21snmpd21:12:205
396098099118,2cyclictest1155-21snmpd20:45:555
396098099116,4cyclictest369677-21/usr/sbin/munin00:17:315
3960980991110,1cyclictest0-21swapper/523:46:265
3960980991110,1cyclictest0-21swapper/523:25:475
3960980991110,1cyclictest0-21swapper/522:28:055
3960980991110,1cyclictest0-21swapper/521:34:485
3960980991110,1cyclictest0-21swapper/521:34:485
3960980991110,1cyclictest0-21swapper/521:08:475
3960980991110,1cyclictest0-21swapper/519:22:045
3960980991110,1cyclictest0-21swapper/519:22:045
3960980991110,1cyclictest0-21swapper/519:13:245
396098099110,5cyclictest0-21swapper/520:08:395
396098099110,1cyclictest0-21swapper/522:59:085
396098099110,1cyclictest0-21swapper/522:45:265
396098099110,11cyclictest0-21swapper/523:16:485
396098099110,11cyclictest0-21swapper/521:06:075
396097799119,1cyclictest0-21swapper/422:47:344
396097799118,1cyclictest201rcu_preempt20:12:414
396097799111,5cyclictest65-21ksoftirqd/422:52:424
3960977991110,1cyclictest108202-21grep21:57:464
3960977991110,1cyclictest0-21swapper/423:42:324
3960977991110,1cyclictest0-21swapper/423:22:304
3960977991110,1cyclictest0-21swapper/421:32:384
3960977991110,1cyclictest0-21swapper/421:32:384
3960977991110,1cyclictest0-21swapper/420:37:364
3960977991110,1cyclictest0-21swapper/419:34:284
396097799110,11cyclictest0-21swapper/422:23:464
396098699109,0cyclictest0-21swapper/720:01:557
396098699109,0cyclictest0-21swapper/719:12:027
396098699100,9cyclictest0-21swapper/722:38:597
396098699100,9cyclictest0-21swapper/721:49:097
396098699100,8cyclictest0-21swapper/721:44:157
396098699100,4cyclictest0-21swapper/721:22:577
396098299108,2cyclictest4012043-21kworker/u16:2+flush-8:019:37:336
396098299108,2cyclictest378619-21kworker/u16:0+flush-8:000:33:256
396098299108,2cyclictest32158-21kworker/u16:1+events_unbound23:15:036
396098299108,2cyclictest277882-21kworker/u16:3+events_unbound23:35:486
396098299107,2cyclictest0-21swapper/600:22:276
396098299102,8cyclictest0-21swapper/621:22:366
396098099109,1cyclictest0-21swapper/523:32:155
396098099109,1cyclictest0-21swapper/500:08:215
396098099107,2cyclictest118273-21pool-org.gnome.22:03:595
396098099107,2cyclictest1155-21snmpd21:27:155
396098099107,2cyclictest1155-21snmpd21:27:155
396098099107,2cyclictest1155-21snmpd20:55:505
396098099102,8cyclictest0-21swapper/521:46:575
39609869998,0cyclictest0-21swapper/721:56:407
39609869992,4cyclictest0-21swapper/721:04:037
39609869990,8cyclictest0-21swapper/723:58:467
39609869990,8cyclictest0-21swapper/723:58:467
39609869990,8cyclictest0-21swapper/722:56:297
39609869990,8cyclictest0-21swapper/720:41:317
39609829997,2cyclictest4089314-21kworker/u16:5+events_unbound20:26:596
39609829997,2cyclictest4012043-21kworker/u16:2+events_unbound19:58:486
39609809998,1cyclictest0-21swapper/523:11:245
39609809998,1cyclictest0-21swapper/522:12:425
39609809998,1cyclictest0-21swapper/521:25:495
39609809998,1cyclictest0-21swapper/520:37:255
39609809998,1cyclictest0-21swapper/500:16:045
39609869987,0cyclictest0-21swapper/720:11:517
39609869985,2cyclictest0-21swapper/723:50:587
39609869981,4cyclictest0-21swapper/720:04:037
39609869980,6cyclictest2480362sleep723:12:377
39609869980,6cyclictest0-21swapper/720:52:367
39609809987,1cyclictest0-21swapper/522:49:135
39609809987,1cyclictest0-21swapper/519:43:305
39609809986,1cyclictest341036-21tr00:02:295
39609809986,1cyclictest0-21swapper/523:57:325
39609809986,1cyclictest0-21swapper/523:57:325
39609809986,1cyclictest0-21swapper/522:52:335
39609809983,4cyclictest274482-21sed23:27:315
39609809983,4cyclictest257834-21cut23:17:355
39609809980,8cyclictest0-21swapper/522:21:525
39609869976,0cyclictest0-21swapper/720:47:367
39609869976,0cyclictest0-21swapper/719:31:507
39609869974,2cyclictest0-21swapper/700:19:257
39609869970,5cyclictest166696-21taskset22:29:107
39609869970,2cyclictest0-21swapper/721:17:447
39609809972,4cyclictest389746-21cat00:27:375
39609809970,6cyclictest4023198-21cat19:37:285
39609809970,6cyclictest4003917-21grep19:27:275
39609869960,5cyclictest0-21swapper/722:32:287
39609869960,4cyclictest234751-21chrt23:07:157
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional