You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-20 - 14:35
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack0slot2.osadl.org (updated Tue May 20, 2025 12:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1329756994036,3cyclictest1443889-21kworker/u16:4+flush-8:018:45:105
1329759993936,2cyclictest1662439-21kworker/u16:5+flush-8:018:42:506
18650632360,1chrt0-21swapper/519:56:165
132976099330,31cyclictest1508695-21latency_hist17:00:587
1329752993230,2cyclictest1632570-21kworker/u16:2+flush-8:018:45:134
132976099310,29cyclictest643-21systemd-journal20:25:577
132976099310,29cyclictest1622250-21latency_hist17:55:587
132976099290,27cyclictest1549964-21latency_hist17:20:577
1329752992826,2cyclictest1304850-21kworker/u16:2+events_unbound16:07:124
1329752992724,2cyclictest1052578-21kworker/u16:0+flush-8:015:46:134
1329752992624,2cyclictest1859698-21kworker/u16:4+events_unbound20:02:204
1329752992624,2cyclictest1676219-21kworker/u16:3+events_unbound18:32:164
1329752992624,2cyclictest1262571-21kworker/u16:3+events_unbound15:38:014
1329752992523,2cyclictest1921615-21kworker/u16:0+events_unbound20:55:244
1329752992523,2cyclictest1676219-21kworker/u16:3+events_unbound18:47:044
1329752992523,2cyclictest1570283-21kworker/u16:1+events_unbound17:42:134
1329752992523,2cyclictest1466444-21kworker/u16:7+events_unbound16:50:334
1329752992523,2cyclictest1443889-21kworker/u16:4+events_unbound17:08:054
1329752992523,2cyclictest1262571-21kworker/u16:3+events_unbound16:17:244
1329752992522,2cyclictest1507664-21kworker/u16:0+events_unbound18:17:254
1329752992422,2cyclictest1262571-21kworker/u16:3+events_unbound17:27:044
1329752992422,2cyclictest1262571-21kworker/u16:3+events_unbound16:42:424
1329752992421,2cyclictest1262571-21kworker/u16:3+events_unbound16:01:014
1329752992321,2cyclictest1976076-21kworker/u16:3+events_unbound21:05:204
1329752992321,2cyclictest1942232-21kworker/u16:2+events_unbound20:32:204
1329752992321,2cyclictest1632570-21kworker/u16:2+events_unbound18:29:524
1329752992321,2cyclictest1632570-21kworker/u16:2+events_unbound18:03:054
1329752992321,2cyclictest1443889-21kworker/u16:4+events_unbound18:57:214
1329752992321,2cyclictest1329925-21kworker/u16:5+events_unbound15:44:284
1329752992321,2cyclictest1304850-21kworker/u16:2+events_unbound16:26:134
1329752992321,2cyclictest1304850-21kworker/u16:2+events_unbound16:15:174
1329752992321,2cyclictest1262571-21kworker/u16:3+flush-8:017:33:374
1329752992321,2cyclictest1262571-21kworker/u16:3+events_unbound16:22:524
132976099220,21cyclictest27326-21gnome-software18:45:237
1329752992220,2cyclictest1921615-21kworker/u16:0+events_unbound20:44:354
1329752992220,2cyclictest1921615-21kworker/u16:0+events_unbound20:44:354
1329752992220,2cyclictest1871686-21kworker/u16:2+events_unbound20:14:394
1329752992220,2cyclictest1859698-21kworker/u16:4+events_unbound20:50:234
1329752992220,2cyclictest1859698-21kworker/u16:4+events_unbound20:19:204
1329752992220,2cyclictest1798578-21kworker/u16:2+events_unbound19:29:244
1329752992220,2cyclictest1756669-21kworker/u16:1+events_unbound19:08:454
1329752992220,2cyclictest1756669-21kworker/u16:1+events_unbound19:08:454
1329752992220,2cyclictest1726825-21kworker/u16:0+events_unbound19:51:044
1329752992220,2cyclictest1443889-21kworker/u16:4+flush-8:019:22:474
1329752992220,2cyclictest1443889-21kworker/u16:4+flush-8:018:54:484
1329752992220,2cyclictest1329925-21kworker/u16:5+events_unbound16:38:404
1329752992220,2cyclictest1329925-21kworker/u16:5+events_unbound16:32:384
1329752992220,2cyclictest1304850-21kworker/u16:2+events_unbound15:53:104
1329760992120,0cyclictest0-21swapper/717:41:297
132976099210,20cyclictest1488118-21latency_hist16:50:587
132976099210,20cyclictest1384841-21latency_hist16:00:587
132976099210,20cyclictest1384841-21latency_hist16:00:587
1329752992119,2cyclictest1942232-21kworker/u16:2+events_unbound20:37:404
1329752992119,2cyclictest1756669-21kworker/u16:1+flush-8:020:28:404
1329752992118,2cyclictest1507664-21kworker/u16:0+events_unbound17:56:074
1329759992017,2cyclictest1507664-21kworker/u16:0+events_unbound17:41:146
1329752992018,2cyclictest1859698-21kworker/u16:4+events_unbound20:07:394
1329752992018,2cyclictest1839592-21kworker/u16:5+events_unbound19:58:084
1329752992018,2cyclictest1839592-21kworker/u16:5+events_unbound19:43:364
1329752992018,2cyclictest1756669-21kworker/u16:1+events_unbound19:46:444
1329752992018,2cyclictest1676219-21kworker/u16:3+events_unbound19:19:364
1329752992018,2cyclictest1632570-21kworker/u16:2+events_unbound18:12:524
1329752992018,2cyclictest1580056-21kworker/u16:2+flush-8:017:46:084
1329752992018,2cyclictest1570283-21kworker/u16:1+flush-8:018:07:164
1329752992018,2cyclictest1570283-21kworker/u16:1+flush-8:017:40:164
1329752992018,2cyclictest1466444-21kworker/u16:7+events_unbound17:24:124
1329752992018,2cyclictest1466444-21kworker/u16:7+events_unbound17:02:404
1329752992018,2cyclictest1443889-21kworker/u16:4+events_unbound18:38:044
1329752992018,2cyclictest1443889-21kworker/u16:4+events_unbound16:53:024
1329752992018,2cyclictest1262571-21kworker/u16:3+events_unbound17:12:564
1329760991917,1cyclictest1829631-21awk19:40:567
132976099190,17cyclictest1539651-21latency_hist17:15:587
1329752991917,2cyclictest1726825-21kworker/u16:0+events_unbound19:13:444
1329752991917,2cyclictest1726825-21kworker/u16:0+events_unbound19:04:594
1329752991917,2cyclictest1443889-21kworker/u16:4+flush-8:019:31:364
1329752991917,2cyclictest1443889-21kworker/u16:4+flush-8:018:21:364
1329759991816,2cyclictest1859698-21kworker/u16:4+flush-8:020:51:176
1329752991816,2cyclictest1942232-21kworker/u16:2+events_unbound20:58:554
1329752991816,2cyclictest1329925-21kworker/u16:5+events_unbound15:57:204
1329752991816,2cyclictest1329925-21kworker/u16:5+events_unbound15:57:204
132976099170,15cyclictest1313-21snmpd19:09:257
132976099170,15cyclictest1313-21snmpd19:09:257
1329759991714,2cyclictest1756669-21kworker/u16:1+flush-8:019:51:126
1329752991715,2cyclictest1798578-21kworker/u16:2+flush-8:019:40:274
1329752991715,2cyclictest1570283-21kworker/u16:1+events_unbound17:54:244
132976099168,7cyclictest27204-21gmain17:36:247
1329760991612,3cyclictest1685294-21idleruntime-cro18:30:577
1329760991612,2cyclictest1426093-21munin-run16:20:587
132976099160,1cyclictest0-21swapper/719:41:237
132976099160,15cyclictest1674944-21latency_hist18:25:577
1329752991614,2cyclictest1839592-21kworker/u16:5+events_unbound20:25:554
1329752991614,2cyclictest1329925-21kworker/u16:5+events_unbound16:59:074
1329752991614,2cyclictest1262571-21kworker/u16:3+events_unbound17:17:064
1329760991513,1cyclictest1974150-21basename20:50:577
1329760991513,1cyclictest1757656-21awk19:05:577
132976099150,14cyclictest1912244-21latency_hist20:20:587
132976099150,13cyclictest1313-21snmpd17:07:157
132976099150,13cyclictest0-21swapper/720:11:237
1329759991513,2cyclictest1329925-21kworker/u16:5+flush-8:015:56:286
1329759991513,2cyclictest1329925-21kworker/u16:5+flush-8:015:56:286
1329760991412,1cyclictest1870972-21awk20:00:577
1329760991410,3cyclictest1891596-21awk20:10:577
132976099140,13cyclictest1778177-21latency_hist19:15:577
132976099140,12cyclictest948-21NetworkManager17:01:237
1329759991412,2cyclictest1976076-21kworker/u16:3+events_unbound20:56:126
132975999140,1cyclictest0-21swapper/617:01:306
132975699140,14cyclictest0-21swapper/520:16:205
132976099136,6cyclictest27763-21gvfs-afc-volume18:05:587
1329760991310,2cyclictest1560276-21awk17:25:577
1329759991310,2cyclictest1756669-21kworker/u16:1+events_unbound20:16:116
1329756991312,1cyclictest0-21swapper/520:57:105
1329756991312,1cyclictest0-21swapper/520:51:295
1329756991312,1cyclictest0-21swapper/517:39:575
1329756991310,2cyclictest1774291-21kthreadcore19:11:185
132975699130,13cyclictest0-21swapper/518:11:575
132976099129,2cyclictest2005117-21awk21:05:577
132976099126,5cyclictest27763-21gvfs-afc-volume20:31:157
132976099121,10cyclictest0-21swapper/717:31:227
132976099120,11cyclictest1611941-21latency_hist17:50:587
132976099120,11cyclictest1313-21snmpd19:52:327
132976099120,11cyclictest1313-21snmpd18:36:307
132976099120,11cyclictest1313-21snmpd18:18:177
132976099120,11cyclictest1313-21snmpd15:44:027
132976099120,10cyclictest1489434-21tune2fs16:51:147
132976099120,10cyclictest0-21swapper/719:21:267
1329759991210,2cyclictest1338639-21kworker/u16:6+flush-8:015:51:136
132975999120,12cyclictest0-21swapper/618:08:396
132975999120,12cyclictest0-21swapper/615:43:096
132975999120,11cyclictest0-21swapper/620:31:006
132975699128,3cyclictest1717861-21cat18:46:145
132975699128,3cyclictest1603151-21cat17:46:145
1329756991211,1cyclictest0-21swapper/521:02:515
1329756991211,1cyclictest0-21swapper/518:21:155
1329756991211,1cyclictest0-21swapper/516:15:275
132975699120,1cyclictest0-21swapper/519:22:355
132975699120,1cyclictest0-21swapper/519:17:135
132975699120,1cyclictest0-21swapper/516:37:035
132975699120,1cyclictest0-21swapper/516:26:015
132976099117,3cyclictest27904-21gmain15:37:007
132976099115,5cyclictest27763-21gvfs-afc-volume15:51:157
132976099115,5cyclictest27642-21gmain17:59:247
132976099110,9cyclictest0-21swapper/718:56:137
132976099110,3cyclictest1981988-21grep20:51:227
132976099110,3cyclictest1072-21runrttasks16:31:137
132976099110,10cyclictest1313-21snmpd20:36:487
132976099110,10cyclictest1313-21snmpd19:33:207
132976099110,10cyclictest1313-21snmpd19:19:467
132976099110,10cyclictest1313-21snmpd16:42:517
132976099110,10cyclictest1313-21snmpd16:13:357
132976099110,10cyclictest1313-21snmpd16:06:177
132975999118,2cyclictest1798578-21kworker/u16:2+flush-8:019:37:036
132975999112,8cyclictest0-21swapper/620:01:126
1329759991110,1cyclictest0-21swapper/621:02:136
1329759991110,1cyclictest0-21swapper/619:27:386
1329759991110,1cyclictest0-21swapper/619:22:166
1329759991110,1cyclictest0-21swapper/617:59:026
1329759991110,1cyclictest0-21swapper/617:27:126
1329759991110,1cyclictest0-21swapper/616:44:006
132975999110,1cyclictest0-21swapper/616:16:446
132975999110,11cyclictest0-21swapper/617:36:496
132975999110,11cyclictest0-21swapper/616:04:266
132975699119,1cyclictest1358172-21kthreadcore15:46:175
132975699119,1cyclictest0-21swapper/520:01:105
1329756991110,1cyclictest0-21swapper/520:39:315
1329756991110,1cyclictest0-21swapper/519:52:325
1329756991110,1cyclictest0-21swapper/519:28:445
1329756991110,1cyclictest0-21swapper/517:23:055
1329756991110,1cyclictest0-21swapper/517:01:205
1329756991110,1cyclictest0-21swapper/516:42:065
1329756991110,1cyclictest0-21swapper/516:09:185
1329756991110,1cyclictest0-21swapper/516:01:255
132975699110,11cyclictest0-21swapper/520:14:465
132975699110,11cyclictest0-21swapper/520:08:375
132975699110,11cyclictest0-21swapper/519:47:105
132975699110,11cyclictest0-21swapper/515:58:455
132975699110,11cyclictest0-21swapper/515:58:455
132975699110,11cyclictest0-21swapper/515:54:205
132976099101,8cyclictest27430-21gmain18:01:247
1329760991010,0cyclictest0-21swapper/720:42:037
1329760991010,0cyclictest0-21swapper/720:42:037
1329760991010,0cyclictest0-21swapper/719:49:347
1329760991010,0cyclictest0-21swapper/717:29:257
1329760991010,0cyclictest0-21swapper/716:27:187
132976099100,9cyclictest1385583-21/usr/sbin/munin16:01:287
132976099100,9cyclictest1313-21snmpd20:56:197
132976099100,9cyclictest1313-21snmpd20:04:297
132976099100,9cyclictest1313-21snmpd18:51:007
132976099100,9cyclictest1313-21snmpd18:47:397
132976099100,9cyclictest1313-21snmpd15:48:057
132976099100,9cyclictest0-21swapper/720:26:237
132976099100,3cyclictest1313-21snmpd16:24:517
132976099100,2cyclictest1457795-21/usr/sbin/munin16:36:267
132976099100,1cyclictest1663575-21perf18:15:577
132976099100,1cyclictest0-21swapper/719:26:147
132975999109,1cyclictest0-21swapper/620:46:566
132975999109,1cyclictest0-21swapper/618:49:486
132975999109,1cyclictest0-21swapper/618:34:126
132975999109,1cyclictest0-21swapper/617:12:226
132975999109,1cyclictest0-21swapper/616:49:506
132975999106,3cyclictest1924098-21cat20:26:136
132975699109,1cyclictest1704065-21tr18:36:215
132975699109,1cyclictest0-21swapper/519:36:275
132975699109,1cyclictest0-21swapper/519:08:055
132975699109,1cyclictest0-21swapper/519:08:055
132975699109,1cyclictest0-21swapper/516:31:135
132975699108,1cyclictest0-21swapper/520:26:185
132975699108,1cyclictest0-21swapper/517:16:165
13297609990,1cyclictest0-21swapper/718:31:197
13297599998,1cyclictest0-21swapper/617:20:446
13297599998,1cyclictest0-21swapper/615:46:466
13297599994,4cyclictest1944754-21awk20:36:156
13297599990,8cyclictest0-21swapper/618:13:176
13297599990,1cyclictest0-21swapper/618:36:196
13297569998,1cyclictest1613429-21smartctl17:51:135
13297569998,1cyclictest1428208-21kthreadcore16:21:185
13297569998,1cyclictest0-21swapper/519:34:535
13297569998,1cyclictest0-21swapper/518:34:205
13297569998,1cyclictest0-21swapper/518:27:245
13297569998,1cyclictest0-21swapper/517:31:075
13297569998,1cyclictest0-21swapper/516:47:085
13297569997,1cyclictest1338497-21missed_timers15:36:215
13297569997,1cyclictest0-21swapper/519:41:185
13297569997,1cyclictest0-21swapper/516:16:115
13297569997,1cyclictest0-21swapper/515:41:015
13297569995,3cyclictest1561239-21grep17:26:105
13297569995,3cyclictest1489659-21cat16:51:155
13297599987,1cyclictest0-21swapper/620:15:056
13297599987,1cyclictest0-21swapper/618:52:576
13297599987,1cyclictest0-21swapper/616:38:106
13297599986,2cyclictest1262571-21kworker/u16:3+flush-8:015:37:586
13297599986,1cyclictest0-21swapper/616:51:176
13297599985,2cyclictest1571323-21/usr/sbin/munin17:31:296
13297599984,3cyclictest1670063-21cut18:21:126
13297599983,4cyclictest1761018-21grep19:06:156
13297599983,4cyclictest1761018-21grep19:06:156
13297599983,4cyclictest1440307-21cut16:26:176
13297599980,7cyclictest0-21swapper/619:56:146
13297599980,1cyclictest0-21swapper/620:06:156
13297569987,1cyclictest0-21swapper/518:05:295
13297569986,1cyclictest0-21swapper/518:51:115
13297569986,1cyclictest0-21swapper/517:11:155
13297569984,3cyclictest1746325-21cat18:56:215
13297569984,3cyclictest1592803-21cat17:41:155
13297569984,3cyclictest1521358-21kthreadcore17:06:165
13297569984,3cyclictest1499950-21cat16:56:165
13297569983,4cyclictest1955003-21grep20:41:125
13297569983,4cyclictest1955003-21grep20:41:125
13297569983,4cyclictest1917333-21cut20:21:185
13297569983,4cyclictest1750609-21cut19:01:175
13297599976,1cyclictest0-21swapper/616:32:206
13297599975,1cyclictest0-21swapper/618:56:136
13297599975,1cyclictest0-21swapper/616:21:156
13297599975,1cyclictest0-21swapper/616:11:136
13297599974,2cyclictest1955538-21kthreadcore20:41:146
13297599974,2cyclictest1955538-21kthreadcore20:41:146
13297599974,2cyclictest1834664-21kthreadcore19:41:176
13297599974,2cyclictest1634539-21kthreadcore18:01:166
13297599974,2cyclictest0-21swapper/619:11:186
13297599973,3cyclictest1779260-21expr19:16:126
13297599973,3cyclictest1677610-21cat18:26:176
13297599973,3cyclictest1621897-21unixbench_singl17:51:276
13297599973,3cyclictest1551788-21cut17:21:166
13297599972,4cyclictest1918363-21kthreadcore20:21:196
13297599972,4cyclictest1811253-21kthreadcore19:31:166
13297599972,4cyclictest1754300-21kthreadcore19:01:196
13297599972,4cyclictest1604633-21grep17:46:176
13297599972,4cyclictest1523288-21kthreadcore17:06:176
13297599972,4cyclictest1503851-21cut16:56:206
13297599972,4cyclictest1397056-21cut16:06:156
13297599970,3cyclictest1841150-21cat19:46:116
13297569976,1cyclictest0-21swapper/518:20:285
13297569972,4cyclictest1933826-21expr20:31:115
13297569972,4cyclictest1644327-21fschecks_time18:06:155
13297569972,4cyclictest1623450-21expr17:56:145
13297609950,4cyclictest1329728-21exim415:35:597
13297599953,1cyclictest16643042chrt18:16:106
13297569950,4cyclictest1964606-21sort20:46:105
13297569950,2cyclictest991rcuop/715:35:595
13297529940,3cyclictest1329758-21chrt15:35:594
13297599920,1cyclictest0-21swapper/615:35:596
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional