You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-28 - 03:16
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack0slot2.osadl.org (updated Sun Apr 28, 2024 00:43:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
148963425520,30sleep70-21swapper/719:06:127
2679189530,30rtkit-daemon0-21swapper/619:04:436
1490107993429,4cyclictest1906312-21latency_hist22:48:335
148963223423,8sleep50-21swapper/519:06:105
148950623323,7sleep40-21swapper/419:04:254
1490111993028,2cyclictest1435241-21kworker/u16:1+events_unbound19:13:386
1490107992826,2cyclictest1435241-21kworker/u16:1+flush-8:019:45:165
1490111992523,2cyclictest1857405-21kworker/u16:1+flush-8:022:58:586
1490111992521,3cyclictest1838057-21latency_hist22:13:346
1490107992521,3cyclictest2101727-21latency_hist00:28:335
1490111992321,2cyclictest1664272-21kworker/u16:2+events_unbound21:08:506
1490103992318,4cyclictest1499842-21latency_hist19:13:344
1490111992220,2cyclictest1941513-21kworker/u16:0+events_unbound23:24:066
1490103992220,2cyclictest1573778-21kworker/u16:2+flush-8:020:03:474
1490111992119,2cyclictest1523173-21kworker/u16:0+flush-8:019:24:026
1490111992018,2cyclictest1612984-21kworker/u16:3+flush-8:020:18:526
1490111992017,3cyclictest1740349-21kworker/u16:0+flush-8:021:23:596
1490111992017,3cyclictest1740349-21kworker/u16:0+flush-8:021:23:596
1490111991917,2cyclictest1906000-21kworker/u16:2+events_unbound23:13:566
1490111991917,2cyclictest1685936-21kworker/u16:1+flush-8:021:03:476
1490111991917,2cyclictest1664272-21kworker/u16:2+flush-8:021:18:476
1490111991917,2cyclictest1523173-21kworker/u16:0+flush-8:019:33:446
1490107991915,3cyclictest1603522-21latency_hist20:13:335
149010799190,18cyclictest0-21swapper/520:43:555
1490111991816,2cyclictest1857405-21kworker/u16:1+flush-8:022:39:106
1490111991816,2cyclictest1840121-21kworker/u16:3+flush-8:022:18:336
1490111991816,2cyclictest1577828-21kworker/u16:4+events_unbound20:34:016
1490111991816,2cyclictest1573778-21kworker/u16:2+flush-8:020:03:496
1490111991816,2cyclictest1532982-21kworker/u16:3+flush-8:019:53:476
1490111991815,2cyclictest1941513-21kworker/u16:0+flush-8:023:19:016
149010799180,17cyclictest0-21swapper/521:28:585
149010799180,17cyclictest0-21swapper/521:28:585
149011399174,5cyclictest0-21swapper/720:34:077
1490113991716,0cyclictest0-21swapper/720:19:127
1490111991715,2cyclictest1906815-21kworker/u16:3+flush-8:023:03:486
1490111991715,2cyclictest1906815-21kworker/u16:3+flush-8:022:53:466
1490111991715,2cyclictest1592692-21kworker/u16:1+flush-8:020:33:336
1490111991714,2cyclictest1435241-21kworker/u16:1+events_unbound19:18:356
149011199170,16cyclictest1690024-21kthreadcore20:54:076
149011399162,6cyclictest0-21swapper/721:49:337
1490111991614,2cyclictest2040346-21kworker/u16:0+events_unbound00:03:586
1490111991614,2cyclictest2040346-21kworker/u16:0+events_unbound00:03:586
1490111991614,2cyclictest1770130-21kworker/u16:1+flush-8:021:43:526
1490111991614,2cyclictest1770130-21kworker/u16:1+flush-8:021:38:456
1490111991614,2cyclictest1685936-21kworker/u16:1+flush-8:021:00:316
1490111991613,2cyclictest1685937-21kworker/u16:4+flush-8:021:33:546
1490107991614,2cyclictest1532982-21kworker/u16:3+flush-8:019:53:425
149010399163,5cyclictest0-21swapper/423:53:034
149011399156,8cyclictest0-21swapper/722:59:417
149011399156,8cyclictest0-21swapper/719:45:117
149011399156,8cyclictest0-21swapper/700:08:167
149011399156,8cyclictest0-21swapper/700:08:167
149011399150,6cyclictest0-21swapper/719:34:507
149011399150,13cyclictest0-21swapper/722:33:467
149011199157,8cyclictest0-21swapper/620:00:306
1490111991513,2cyclictest2102102-21kworker/u16:4+flush-8:000:34:046
1490111991513,2cyclictest2003116-21kworker/u16:1+flush-8:023:58:336
1490111991513,2cyclictest2003116-21kworker/u16:1+flush-8:023:58:336
1490111991513,2cyclictest2003116-21kworker/u16:1+flush-8:000:23:286
1490111991513,2cyclictest2003116-21kworker/u16:1+flush-8:000:13:476
1490111991513,2cyclictest1740349-21kworker/u16:0+events_unbound22:23:576
1490111991513,2cyclictest1740349-21kworker/u16:0+events_unbound22:05:216
1490111991513,2cyclictest1612984-21kworker/u16:3+flush-8:020:24:026
1490107991513,2cyclictest1523173-21kworker/u16:0+flush-8:019:23:485
149010799150,14cyclictest1847798-21munin-run22:18:335
1490103991513,2cyclictest2003116-21kworker/u16:1+flush-8:000:24:064
1490103991511,3cyclictest1643280-21kworker/u16:3+flush-8:020:49:044
1490103991510,4cyclictest1553285-21latency_hist19:43:344
149011399145,8cyclictest0-21swapper/719:56:277
149011399141,6cyclictest0-21swapper/721:32:577
149011399141,6cyclictest0-21swapper/721:32:577
149011399141,5cyclictest0-21swapper/721:57:397
149011399141,5cyclictest0-21swapper/721:44:037
149011399141,5cyclictest0-21swapper/720:17:367
149011399140,6cyclictest0-21swapper/723:54:007
149011399140,6cyclictest0-21swapper/723:54:007
149011399140,6cyclictest0-21swapper/722:57:267
149011399140,6cyclictest0-21swapper/722:22:167
149011399140,2cyclictest0-21swapper/721:14:117
149011399140,1cyclictest0-21swapper/723:03:597
1490111991412,2cyclictest2949790-21kworker/u16:2+events_unbound19:32:456
1490111991412,2cyclictest2102102-21kworker/u16:4+events_unbound00:28:516
1490111991412,2cyclictest1857405-21kworker/u16:1+events_unbound22:48:466
1490111991412,2cyclictest1818398-21kworker/u16:4+events_unbound22:32:296
1490111991412,2cyclictest1770130-21kworker/u16:1+flush-8:021:59:026
1490111991412,2cyclictest1740349-21kworker/u16:0+flush-8:021:54:236
1490111991412,2cyclictest1685937-21kworker/u16:4+flush-8:021:28:586
1490111991412,2cyclictest1685937-21kworker/u16:4+flush-8:021:28:586
1490111991412,2cyclictest1633009-21kworker/u16:0+events_unbound20:46:316
1490111991412,2cyclictest1577828-21kworker/u16:4+flush-8:020:38:356
1490111991412,2cyclictest1490305-21kworker/u16:4+events_unbound19:11:316
1490111991411,2cyclictest1984297-21kworker/u16:3+flush-8:023:28:336
149010799140,2cyclictest1514684-21iostat19:18:495
149010799140,13cyclictest1691453-21taskset20:55:285
1490103991412,2cyclictest1974399-21kworker/u16:2+flush-8:023:43:574
1490103991411,2cyclictest1135-21snmpd22:12:314
149011399135,7cyclictest0-21swapper/720:55:107
149011399134,8cyclictest0-21swapper/720:12:277
149011399134,8cyclictest0-21swapper/700:16:077
149011399131,5cyclictest0-21swapper/721:07:147
1490113991310,2cyclictest0-21swapper/723:39:487
149011399130,6cyclictest0-21swapper/723:52:457
149011399130,6cyclictest0-21swapper/723:46:127
149011399130,6cyclictest0-21swapper/722:25:527
149011399130,6cyclictest0-21swapper/722:15:167
149011399130,6cyclictest0-21swapper/721:35:217
149011399130,6cyclictest0-21swapper/720:44:347
149011399130,1cyclictest0-21swapper/720:28:567
149011199135,8cyclictest0-21swapper/619:50:086
1490111991311,2cyclictest1840121-21kworker/u16:3+flush-8:022:20:246
1490111991311,2cyclictest1818398-21kworker/u16:4+flush-8:022:44:326
1490111991311,2cyclictest1685936-21kworker/u16:1+events_unbound21:13:476
1490111991310,2cyclictest27516-21fwupd19:45:146
1490111991310,2cyclictest2012715-21kworker/u16:4+events_unbound23:39:076
149011199130,12cyclictest0-21swapper/622:33:466
1490107991311,2cyclictest2040346-21kworker/u16:0+flush-8:000:34:035
1490107991311,2cyclictest0-21swapper/500:06:335
1490107991311,2cyclictest0-21swapper/500:06:335
1490107991310,2cyclictest907-21gmain23:23:335
149010799130,1cyclictest0-21swapper/522:23:455
149010799130,13cyclictest0-21swapper/523:40:325
149010799130,13cyclictest0-21swapper/521:48:555
149010399138,4cyclictest1847791-21latency_hist22:18:334
149010399136,7cyclictest0-21swapper/421:38:064
1490103991310,2cyclictest1821215-21kthreadcore22:03:514
1490103991310,2cyclictest1135-21snmpd21:53:184
149010399130,5cyclictest0-21swapper/421:43:124
149010399130,0cyclictest0-21swapper/422:33:584
149011399129,2cyclictest0-21swapper/720:06:517
149011399129,2cyclictest0-21swapper/719:28:477
149011399120,6cyclictest0-21swapper/719:16:497
149011399120,2cyclictest0-21swapper/723:58:577
149011399120,2cyclictest0-21swapper/723:58:577
149011399120,2cyclictest0-21swapper/720:23:537
149011199124,8cyclictest0-21swapper/619:40:146
1490111991210,2cyclictest2040346-21kworker/u16:0+events_unbound00:11:326
1490111991210,2cyclictest2012715-21kworker/u16:4+flush-8:023:43:436
1490111991210,2cyclictest1941513-21kworker/u16:0+flush-8:023:33:336
1490111991210,2cyclictest1592692-21kworker/u16:1+flush-8:020:12:496
149010799129,3cyclictest1523173-21kworker/u16:0+flush-8:019:59:345
149010799129,2cyclictest2003319-21timerandwakeup23:34:005
149010799129,2cyclictest1135-21snmpd23:53:425
149010799129,2cyclictest1135-21snmpd23:53:425
149010799128,3cyclictest0-21swapper/521:58:565
1490107991211,1cyclictest1543922-21kworker/u16:4+flush-8:019:34:055
1490107991211,1cyclictest0-21swapper/522:28:585
149010799120,12cyclictest0-21swapper/521:18:295
149010799120,12cyclictest0-21swapper/521:04:085
149010799120,12cyclictest0-21swapper/519:14:445
149010799120,11cyclictest1-21systemd19:43:345
149010799120,0cyclictest0-21swapper/519:52:435
149010399129,2cyclictest1135-21snmpd23:00:594
149010399127,4cyclictest1534207-21cat19:28:514
149010399126,6cyclictest0-21swapper/420:17:214
149010399125,7cyclictest0-21swapper/423:58:184
149010399125,7cyclictest0-21swapper/423:58:184
149010399125,7cyclictest0-21swapper/422:31:314
1490103991210,2cyclictest1577828-21kworker/u16:4+flush-8:020:34:034
149010399120,1cyclictest0-21swapper/423:36:054
149010399120,1cyclictest0-21swapper/422:51:184
149010399120,1cyclictest0-21swapper/421:04:254
149010399120,12cyclictest0-21swapper/422:43:254
149010399120,12cyclictest0-21swapper/420:45:304
149011399118,2cyclictest0-21swapper/723:15:057
149011399118,2cyclictest0-21swapper/723:09:507
149011399118,2cyclictest0-21swapper/721:24:207
149011399118,2cyclictest0-21swapper/721:24:207
149011399110,4cyclictest0-21swapper/723:23:567
149011199119,2cyclictest2003116-21kworker/u16:1+events_unbound00:01:346
149011199119,2cyclictest2003116-21kworker/u16:1+events_unbound00:01:346
149011199119,2cyclictest1643280-21kworker/u16:3+flush-8:020:49:056
149011199119,2cyclictest1573778-21kworker/u16:2+events_unbound20:15:156
1490111991110,1cyclictest0-21swapper/600:24:066
149011199110,11cyclictest0-21swapper/623:09:326
149010799119,2cyclictest27491-21gmain22:05:575
149010799119,2cyclictest27430-21gmain21:00:395
149010799119,2cyclictest27326-21gmain23:27:125
149010799119,1cyclictest0-21swapper/522:58:515
149010799118,2cyclictest1135-21snmpd22:53:445
149010799118,2cyclictest1135-21snmpd22:20:525
149010799118,2cyclictest1135-21snmpd21:40:305
149010799118,2cyclictest1135-21snmpd21:11:445
149010799117,3cyclictest1612471-21processes20:14:005
149010799113,8cyclictest0-21swapper/521:20:075
149010799111,9cyclictest0-21swapper/523:30:485
1490107991110,1cyclictest0-21swapper/523:05:425
1490107991110,1cyclictest0-21swapper/521:45:175
1490107991110,1cyclictest0-21swapper/520:49:195
1490107991110,1cyclictest0-21swapper/519:13:095
1490107991110,1cyclictest0-21swapper/500:18:415
1490107991110,1cyclictest0-21swapper/500:09:425
149010799110,1cyclictest0-21swapper/522:42:035
149010799110,11cyclictest0-21swapper/522:53:055
149010799110,11cyclictest0-21swapper/522:37:195
149010799110,11cyclictest0-21swapper/522:10:125
149010799110,11cyclictest0-21swapper/521:56:105
149010799110,11cyclictest0-21swapper/520:37:025
149010799110,11cyclictest0-21swapper/500:00:435
149010799110,11cyclictest0-21swapper/500:00:435
149010399118,2cyclictest909-21gdbus19:45:174
149010399118,2cyclictest1515112-21kthreadcore19:18:514
149010399118,2cyclictest1135-21snmpd22:01:514
149010399118,2cyclictest1135-21snmpd19:49:204
149010399118,2cyclictest1135-21snmpd00:29:314
149010399114,7cyclictest0-21swapper/420:29:314
149010399113,7cyclictest0-21swapper/423:18:384
1490103991110,1cyclictest0-21swapper/423:03:554
1490103991110,1cyclictest0-21swapper/422:27:114
1490103991110,1cyclictest0-21swapper/421:24:554
1490103991110,1cyclictest0-21swapper/421:24:554
1490103991110,1cyclictest0-21swapper/420:09:054
1490103991110,1cyclictest0-21swapper/419:58:404
1490103991110,1cyclictest0-21swapper/419:16:354
1490103991110,1cyclictest0-21swapper/400:20:044
149010399110,6cyclictest0-21swapper/421:46:364
149010399110,1cyclictest0-21swapper/421:16:434
149010399110,11cyclictest0-21swapper/422:56:024
149010399110,11cyclictest0-21swapper/421:54:234
149010399110,11cyclictest0-21swapper/421:20:494
149010399110,11cyclictest0-21swapper/421:12:374
149010399110,11cyclictest0-21swapper/420:57:484
149010399110,11cyclictest0-21swapper/419:55:034
149010399110,11cyclictest0-21swapper/400:11:054
149010399110,11cyclictest0-21swapper/400:02:054
149010399110,11cyclictest0-21swapper/400:02:054
149011399107,2cyclictest0-21swapper/722:13:317
149011399102,7cyclictest0-21swapper/700:25:377
149011399102,2cyclictest0-21swapper/720:38:467
149011399101,8cyclictest0-21swapper/722:01:457
149011399101,2cyclictest16850irq/16-ehci_hcd:usb320:58:527
149011399101,2cyclictest0-21swapper/700:08:517
149011399100,5cyclictest0-21swapper/720:49:347
149011399100,5cyclictest0-21swapper/720:01:547
149011399100,4cyclictest1779406-21taskset21:42:427
149011399100,4cyclictest0-21swapper/700:34:057
149011199109,1cyclictest0-21swapper/621:51:396
149011199108,2cyclictest1974399-21kworker/u16:2+events_unbound23:49:176
149010799109,1cyclictest0-21swapper/523:50:375
149010799109,1cyclictest0-21swapper/521:26:415
149010799109,1cyclictest0-21swapper/521:26:415
149010799109,1cyclictest0-21swapper/519:33:205
149010799109,1cyclictest0-21swapper/500:29:155
149010799109,1cyclictest0-21swapper/500:17:075
149010799107,2cyclictest1760419-21/usr/sbin/munin21:33:495
149010799107,2cyclictest1659771-21kthreadcore20:38:555
149010799104,5cyclictest27233-21gdbus20:18:375
149010399109,1cyclictest0-21swapper/420:43:134
149010399109,1cyclictest0-21swapper/419:26:404
149010399108,2cyclictest1577828-21kworker/u16:4+flush-8:020:23:534
149010399107,2cyclictest1532982-21kworker/u16:3+flush-8:019:33:454
149010399107,2cyclictest1135-21snmpd23:30:094
149010399107,2cyclictest1135-21snmpd21:28:474
149010399107,2cyclictest1135-21snmpd21:28:474
149010399106,3cyclictest2004471-21tr23:38:444
14901139996,2cyclictest0-21swapper/700:30:527
14901139993,1cyclictest0-21swapper/722:03:507
14901139990,8cyclictest0-21swapper/721:19:027
14901139990,8cyclictest0-21swapper/719:38:597
14901139990,8cyclictest0-21swapper/719:21:527
14901139990,8cyclictest0-21swapper/700:19:107
14901139990,2cyclictest0-21swapper/723:34:007
14901139990,2cyclictest0-21swapper/722:48:597
14901139990,2cyclictest0-21swapper/722:43:487
14901079998,1cyclictest0-21swapper/523:08:515
14901079998,1cyclictest0-21swapper/520:03:365
14901079997,2cyclictest27326-21gmain23:46:165
14901079997,1cyclictest0-21swapper/523:13:495
14901079997,1cyclictest0-21swapper/520:28:495
14901079997,1cyclictest0-21swapper/520:23:545
14901039998,1cyclictest0-21swapper/422:20:244
14901039998,1cyclictest0-21swapper/400:03:404
14901039998,1cyclictest0-21swapper/400:03:404
14901039996,2cyclictest1135-21snmpd23:15:414
14901039996,2cyclictest1135-21snmpd23:09:114
14901039996,2cyclictest1135-21snmpd20:59:344
14901039996,2cyclictest1135-21snmpd00:36:284
14901039995,3cyclictest1897778-21cat22:43:504
14901139985,2cyclictest0-21swapper/719:08:507
14901139981,2cyclictest0-21swapper/719:23:537
14901139980,4cyclictest0-21swapper/721:09:507
14901139980,2cyclictest0-21swapper/723:21:187
14901139980,2cyclictest0-21swapper/722:39:017
14901039986,1cyclictest0-21swapper/423:24:044
14901039986,1cyclictest0-21swapper/420:18:364
14901039983,4cyclictest2078270-21cut00:13:524
14901139970,6cyclictest0-21swapper/723:28:567
14901139970,1cyclictest0-21swapper/722:28:537
14901139970,1cyclictest0-21swapper/719:53:347
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional