You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-28 - 14:12
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: reduced 
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot8.osadl.org (updated Thu Sep 28, 2023 00:44:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1221219994742,4cyclictest1355445-21kworker/u8:0+events_unbound21:51:091
1221238994640,4cyclictest1437753-21apt-get22:28:273
1221213994641,4cyclictest1338759-21kworker/u8:3+flush-8:021:20:310
1221213994338,4cyclictest1234232-21kworker/u8:4+flush-8:019:30:340
1221219994238,3cyclictest1487438-21kworker/u8:0+events_unbound23:51:111
1221238994037,2cyclictest1492911-21kworker/u8:2+events_unbound23:30:333
1221213994034,4cyclictest1350240-21apt-get21:10:200
1221238993936,2cyclictest1487438-21kworker/u8:0+flush-8:000:15:273
1221238993834,3cyclictest1297676-21kworker/u8:2+flush-8:020:55:013
1221227993834,3cyclictest1487438-21kworker/u8:0+flush-8:023:35:002
1221219993834,3cyclictest1338759-21kworker/u8:3+flush-8:021:45:251
1221238993633,2cyclictest1492911-21kworker/u8:2+flush-8:000:10:013
1221238993632,3cyclictest1415564-21kworker/u8:1+flush-8:022:15:303
1221213993632,3cyclictest1273299-21kworker/u8:0+flush-8:020:20:300
1221213993632,2cyclictest1536644-40updatedb.mlocat00:00:000
1221213993531,3cyclictest1492911-21kworker/u8:2+flush-8:023:20:210
1221238993429,4cyclictest1509349-21kworker/u8:3+flush-8:000:10:283
1221219993429,3cyclictest1349627-21python321:05:281
1221213993430,3cyclictest1215651-21kworker/u8:1+flush-8:019:10:250
1221219993330,2cyclictest1338759-21kworker/u8:3+events_unbound21:35:351
1221219993329,3cyclictest1338759-21kworker/u8:3+flush-8:021:10:231
1221219993328,4cyclictest1239878-21apt-get19:30:031
1221213993330,2cyclictest1459767-21kworker/u8:2+flush-8:022:50:230
1221213993329,3cyclictest1273299-21kworker/u8:0+flush-8:020:20:010
1221238993228,2cyclictest1382745-21kworker/3:1+pm21:51:093
1221238993226,3cyclictest1487438-21kworker/u8:0+flush-8:000:39:593
1221219993228,3cyclictest1415764-21kworker/u8:3+flush-8:023:25:011
1221219993127,3cyclictest1509349-21kworker/u8:3+events_unbound00:20:311
1221219993127,3cyclictest1355445-21kworker/u8:0+flush-8:022:05:211
1221213993128,2cyclictest1284260-21kworker/u8:1+flush-8:020:10:260
1221213993127,3cyclictest1487438-21kworker/u8:0+flush-8:000:05:310
1221238993026,3cyclictest1245637-21kworker/u8:1+events_unbound19:55:243
1221213993027,2cyclictest1399280-21kworker/u8:2+events_unbound22:25:300
1221213993027,2cyclictest1393791-21kworker/u8:4+events_unbound22:10:350
1221213993027,2cyclictest1355445-21kworker/u8:0+flush-8:021:45:000
1221213993026,3cyclictest1415564-21kworker/u8:1+flush-8:022:20:220
1221213993024,4cyclictest1438061-21apt-get22:30:210
1221238992926,2cyclictest1492911-21kworker/u8:2+flush-8:000:05:003
1221238992926,2cyclictest1338759-21kworker/u8:3+events_unbound21:15:303
1221219992926,2cyclictest1509349-21kworker/u8:3+events_unbound00:10:301
1221219992925,3cyclictest1262655-21kworker/u8:3+flush-8:019:50:221
1221213992924,3cyclictest1297676-21kworker/u8:2+flush-8:020:55:260
1221213992923,4cyclictest4170283-21systemd-journal23:00:000
1221238992825,2cyclictest1327846-21apt-get20:50:003
1221238992824,3cyclictest1234232-21kworker/u8:4+flush-8:019:35:253
1221227992822,4cyclictest1295388-21apt-get20:20:182
1221213992824,3cyclictest1297676-21kworker/u8:2+events_unbound20:30:300
1221238992724,2cyclictest1297676-21kworker/u8:2+flush-8:020:45:003
1221238992722,3cyclictest1372053-21apt-get21:30:273
1221238992722,2cyclictest1509349-21kworker/u8:3+events_unbound23:40:263
1221227992720,4cyclictest1377212-21latency_hist21:35:002
1221219992723,3cyclictest1448785-21kworker/u8:1+flush-8:022:50:011
1221213992724,2cyclictest1448785-21kworker/u8:1+events_unbound23:25:300
1221213992724,2cyclictest1355445-21kworker/u8:0+flush-8:021:50:310
1221213992724,2cyclictest1251460-21kworker/u8:0+flush-8:019:40:310
1221238992623,2cyclictest1338759-21kworker/u8:3+flush-8:021:45:263
1221238992623,2cyclictest1288968-21kworker/u8:3+flush-8:020:35:003
1221238992622,3cyclictest1525741-21kworker/u8:1+events_unbound23:55:013
1221238992620,4cyclictest1459719-21latency_hist22:50:003
1221219992622,3cyclictest1251460-21kworker/u8:0+flush-8:019:45:281
1221213992622,3cyclictest1335360-21kworker/u8:1+events_unbound21:40:000
1221238992522,2cyclictest1492911-21kworker/u8:2+events_unbound00:20:213
1221238992522,2cyclictest1278398-21kworker/u8:4+flush-8:020:40:013
1221238992522,2cyclictest1215651-21kworker/u8:1+flush-8:019:10:293
1221238992521,3cyclictest1448785-21kworker/u8:1+events_unbound23:05:013
1221227992522,2cyclictest1239912-21kworker/u8:0+flush-8:019:30:272
1221227992521,2cyclictest0-21swapper/200:00:002
1221227992518,4cyclictest1355419-21latency_hist21:15:002
1221219992521,3cyclictest1415764-21kworker/u8:3+flush-8:023:15:321
1221219992521,3cyclictest1338759-21kworker/u8:3+flush-8:021:20:211
1221213992522,2cyclictest1262655-21kworker/u8:3+events_unbound19:50:200
1221213992521,3cyclictest1278398-21kworker/u8:4+flush-8:020:50:310
1221213992521,3cyclictest1226771-21kworker/u8:2+flush-8:019:25:160
1221238992421,2cyclictest1443277-21kworker/u8:4+events_unbound22:35:333
1221238992421,2cyclictest1226771-21kworker/u8:2+events_unbound19:25:203
1221227992419,4cyclictest1509349-21kworker/u8:3+flush-8:000:25:012
1221227992419,3cyclictest1514465-21python323:35:312
1221227992419,3cyclictest1327846-21apt-get20:50:032
1221219992420,3cyclictest1487438-21kworker/u8:0+events_unbound00:35:321
1221213992421,2cyclictest1448785-21kworker/u8:1+flush-8:023:04:570
1221213992421,2cyclictest1278398-21kworker/u8:4+events_unbound20:05:250
1221238992319,3cyclictest1278398-21kworker/u8:4+events_unbound20:55:293
1221238992319,3cyclictest1278398-21kworker/u8:4+events_unbound20:15:223
1221238992319,3cyclictest1257187-21kworker/u8:2+events_unbound20:05:303
1221238992316,5cyclictest4170283-21systemd-journal00:34:593
1221227992318,3cyclictest4170283-21systemd-journal23:50:002
1221219992319,3cyclictest1492911-21kworker/u8:2+events_unbound23:35:241
1221219992319,3cyclictest1273299-21kworker/u8:0+flush-8:020:15:301
1221213992320,2cyclictest1399280-21kworker/u8:2+flush-8:022:00:270
1221238992219,2cyclictest1361149-21kworker/u8:2+flush-8:021:25:283
1221238992219,2cyclictest1273299-21kworker/u8:0+events_unbound20:00:313
1221238992218,2cyclictest1338759-21kworker/u8:3+flush-8:021:20:263
1221219992220,1cyclictest0-21swapper/100:00:001
1221219992219,1cyclictest1536920-21apt-get00:00:241
1221219992218,3cyclictest1399280-21kworker/u8:2+flush-8:022:00:261
1221219992218,3cyclictest1361149-21kworker/u8:2+events_unbound21:30:411
1221219992218,3cyclictest1333584-21munin-plugin-st20:55:001
1221219992218,3cyclictest1273299-21kworker/u8:0+flush-8:020:20:281
122121999220,21cyclictest0-21swapper/122:20:241
1221213992219,2cyclictest1487485-21systemd-run23:15:000
1221213992219,2cyclictest1245637-21kworker/u8:1+flush-8:019:45:350
1221213992218,3cyclictest1297676-21kworker/u8:2+flush-8:020:25:260
1221213992216,4cyclictest1399313-21apt-get21:55:190
1221238992119,1cyclictest1344511-21idleruntime-cro21:05:003
1221238992115,4cyclictest1525715-21latency_hist23:50:003
1221238992115,4cyclictest1300858-21apt-get20:25:223
1221238992115,4cyclictest1228995-21apt-get19:15:163
1221238992115,3cyclictest1448785-21kworker/u8:1+events_unbound23:25:293
1221227992114,4cyclictest1256960-21latency_hist19:45:012
1221219992118,2cyclictest1487438-21kworker/u8:0+flush-8:000:05:341
1221219992117,3cyclictest1338759-21kworker/u8:3+flush-8:021:00:311
1221219992115,4cyclictest1421613-21apt-get22:15:201
122121999210,20cyclictest0-21swapper/122:40:191
122121999210,20cyclictest0-21swapper/119:15:021
1221213992118,2cyclictest1509349-21kworker/u8:3+flush-8:023:45:230
1221213992118,2cyclictest1427180-21kworker/u8:0+events_unbound22:35:240
1221213992118,2cyclictest1338759-21kworker/u8:3+events_unbound21:25:230
1221213992118,2cyclictest1288968-21kworker/u8:3+flush-8:020:45:260
1221213992117,3cyclictest1487438-21kworker/u8:0+flush-8:000:15:220
1221213992117,3cyclictest1415764-21kworker/u8:3+flush-8:022:15:210
1221213992117,2cyclictest0-21swapper/020:05:000
1221238992016,3cyclictest1415764-21kworker/u8:3+events_unbound23:17:263
1221238992016,3cyclictest1393791-21kworker/u8:4+events_unbound21:58:283
1221238992015,3cyclictest1536644-40updatedb.mlocat00:00:013
1221219992017,2cyclictest1448785-21kworker/u8:1+events_unbound23:30:271
1221219992017,2cyclictest1297676-21kworker/u8:2+events_unbound20:25:251
1221219992017,2cyclictest1215651-21kworker/u8:1+events_unbound19:10:261
1221219992016,3cyclictest1415764-21kworker/u8:3+events_unbound22:30:251
1221219992014,4cyclictest1311521-21anacron20:34:081
122121999200,0cyclictest0-21swapper/122:25:291
1221213992018,1cyclictest4170283-21systemd-journal00:29:590
1221213992017,2cyclictest1297676-21kworker/u8:2+events_unbound20:35:330
1221213992015,3cyclictest1564033-21wc00:25:000
1221238991916,2cyclictest1410059-21cron22:05:003
1221238991915,3cyclictest1487438-21kworker/u8:0+events_unbound00:25:203
1221238991915,3cyclictest1245637-21kworker/u8:1+flush-8:019:40:213
1221227991914,3cyclictest1415764-21kworker/u8:3+flush-8:023:10:012
1221219991917,1cyclictest0-21swapper/120:11:341
1221219991916,2cyclictest1388139-21awk21:45:001
1221219991916,2cyclictest1317604-21kworker/u8:0+events_unbound20:45:211
1221219991915,3cyclictest1509349-21kworker/u8:3+events_unbound23:45:231
1221219991915,3cyclictest1288968-21kworker/u8:3+events_unbound20:40:241
1221238991814,3cyclictest1448785-21kworker/u8:1+events_unbound22:52:273
1221238991813,4cyclictest1393791-21kworker/u8:4+events_unbound22:10:313
1221238991813,3cyclictest1234232-21kworker/u8:4+events_unbound19:30:343
1221238991812,4cyclictest4170283-21systemd-journal21:40:263
1221227991815,2cyclictest1312216-21irqcore20:35:252
1221227991814,3cyclictest1415009-21needreboot22:05:262
1221227991813,3cyclictest0-21swapper/220:30:212
1221227991812,4cyclictest1547619-21latency_hist00:10:022
1221219991817,1cyclictest0-21swapper/120:35:011
1221219991816,1cyclictest0-21swapper/119:35:311
1221219991814,3cyclictest1459767-21kworker/u8:2+flush-8:022:55:211
122121999180,1cyclictest0-21swapper/120:56:091
122121999180,18cyclictest0-21swapper/123:44:551
122121999180,18cyclictest0-21swapper/121:25:351
122121999180,18cyclictest0-21swapper/100:15:091
122121999180,17cyclictest0-21swapper/123:12:181
122121999180,17cyclictest0-21swapper/119:44:091
122121999180,0cyclictest0-21swapper/123:00:361
1221213991816,1cyclictest1199861-21kworker/0:1+kacpid19:15:170
1221213991816,1cyclictest0-21swapper/020:00:000
1221213991814,3cyclictest1509349-21kworker/u8:3+flush-8:023:35:270
1221213991814,3cyclictest1278398-21kworker/u8:4+flush-8:021:00:210
1221213991812,4cyclictest4170283-21systemd-journal19:23:270
1221238991713,3cyclictest1415764-21kworker/u8:3+flush-8:023:15:013
1221238991713,3cyclictest1338759-21kworker/u8:3+events_unbound21:38:283
1221227991713,2cyclictest0-21swapper/222:55:002
1221219991716,1cyclictest0-21swapper/121:17:031
1221219991716,1cyclictest0-21swapper/119:26:241
1221219991716,1cyclictest0-21swapper/100:32:201
1221219991715,1cyclictest0-21swapper/122:35:161
1221219991715,1cyclictest0-21swapper/120:01:211
1221219991713,3cyclictest1564410-21kworker/u8:1+events_unbound00:25:331
122121999170,1cyclictest0-21swapper/122:11:401
122121999170,1cyclictest0-21swapper/119:56:301
122121999170,17cyclictest0-21swapper/122:50:121
122121999170,17cyclictest0-21swapper/121:57:041
122121999170,16cyclictest0-21swapper/119:20:211
122121999170,0cyclictest0-21swapper/123:28:181
122121999170,0cyclictest0-21swapper/123:07:151
1221213991715,1cyclictest1388706-21gltestperf21:45:250
1221213991712,4cyclictest1234232-21kworker/u8:4+flush-8:019:35:300
1221238991613,2cyclictest1338759-21kworker/u8:3+flush-8:021:05:263
1221238991612,3cyclictest1399280-21kworker/u8:2+events_unbound22:23:283
1221227991613,2cyclictest1478-21polkitd19:55:002
1221227991613,2cyclictest1318093-21irqrtprio20:40:262
1221227991610,4cyclictest1415893-21diskstats22:10:242
1221219991612,3cyclictest1278398-21kworker/u8:4+events_unbound20:05:311
1221213991615,0cyclictest1465419-21kworker/0:0+kacpid23:08:260
1221213991614,1cyclictest1481988-21kworker/0:1+kacpid23:30:240
1221213991613,2cyclictest1297676-21kworker/u8:2+events_unbound20:40:310
1221213991612,3cyclictest1448785-21kworker/u8:1+flush-8:022:40:200
1221238991512,2cyclictest1251460-21kworker/u8:0+flush-8:019:45:283
1221238991511,3cyclictest1338759-21kworker/u8:3+events_unbound21:10:343
1221238991511,3cyclictest1273299-21kworker/u8:0+events_unbound20:15:003
122123899150,13cyclictest1300600-21turbostat20:25:003
122122799159,4cyclictest4170283-21systemd-journal19:49:562
122122799159,4cyclictest1471054-21diskstats23:00:232
1221227991512,2cyclictest0-21swapper/220:50:012
1221227991511,3cyclictest1415764-21kworker/u8:3+flush-8:023:15:302
1221213991514,1cyclictest1481988-21kworker/0:1+kacpid00:35:340
1221213991513,1cyclictest0-21swapper/023:45:000
1221238991411,2cyclictest1183698-21kworker/u8:3+events_unbound19:22:303
1221238991410,3cyclictest1448785-21kworker/u8:1+flush-8:023:00:363
1221238991410,3cyclictest1415764-21kworker/u8:3+events_unbound22:41:283
1221238991410,3cyclictest1393791-21kworker/u8:4+events_unbound22:05:033
122122799147,5cyclictest1239517-21python319:25:232
1221227991411,2cyclictest1449504-21irqcore22:40:242
1221227991411,2cyclictest1384260-21kthreadcore21:40:242
1221227991411,2cyclictest0-21swapper/221:51:092
1221227991411,1cyclictest0-21swapper/221:00:002
1221213991410,3cyclictest1338759-21kworker/u8:3+events_unbound21:30:280
1221238991310,2cyclictest1492911-21kworker/u8:2+events_unbound23:35:323
1221238991310,2cyclictest1459767-21kworker/u8:2+events_unbound22:55:193
122122799139,3cyclictest1404594-21sendmail22:00:002
122122799139,3cyclictest1273328-21sendmail19:59:592
122122799139,2cyclictest1487438-21kworker/u8:0+flush-8:000:35:272
122122799137,3cyclictest4170283-21systemd-journal19:15:012
1221227991311,1cyclictest0-21swapper/223:40:252
1221227991310,2cyclictest1356097-21irqcore21:15:252
1221227991310,2cyclictest1301276-21irqcore20:25:272
1221227991310,2cyclictest0-21swapper/222:15:262
122122799130,1cyclictest0-21swapper/200:30:212
122121399139,3cyclictest1355445-21kworker/u8:0+events_unbound22:08:280
122121399138,3cyclictest1349984-21latency_hist21:10:000
1221213991310,2cyclictest1526404-21irqcore23:50:240
122123899127,3cyclictest1220882-21latency_hist19:10:023
122122799129,2cyclictest1553315-21apt-get00:15:192
122122799129,2cyclictest0-21swapper/220:00:232
122122799128,2cyclictest0-21swapper/223:00:002
122122799126,4cyclictest1553053-21latency_hist00:15:012
122122799120,1cyclictest0-21swapper/222:02:022
122121399129,2cyclictest1481988-21kworker/0:1+kacpi_notify23:15:400
122121399126,2cyclictest1574937-21turbostat00:35:000
122121399121,6cyclictest1553055-21turbostat00:15:000
1221213991210,1cyclictest1432249-21kworker/0:1+kacpid22:45:340
122123899117,3cyclictest1492911-21kworker/u8:2+events_unbound23:21:353
122123899117,3cyclictest1262655-21kworker/u8:3+events_unbound19:50:293
122122799119,1cyclictest0-21swapper/223:51:372
122122799119,1cyclictest0-21swapper/223:25:302
122122799118,1cyclictest0-21swapper/221:50:012
122122799118,1cyclictest0-21swapper/220:15:002
122122799117,3cyclictest1564410-21kworker/u8:1+flush-8:000:25:232
122122799116,3cyclictest1537058-21sort00:00:252
122122799115,4cyclictest1361278-21diskstats21:20:222
122121399117,2cyclictest0-21swapper/021:20:010
122123899106,2cyclictest1443308-21munin-plugin-st22:34:593
122122799108,1cyclictest0-21swapper/223:20:212
122122799107,2cyclictest4053-21cron19:25:012
122122799106,3cyclictest1415764-21kworker/u8:3+flush-8:022:29:592
122122799106,2cyclictest1443993-21irqcore22:35:292
122122799104,4cyclictest1432256-21sshd22:22:262
122122799101,8cyclictest381rcuc/221:25:262
122122799101,6cyclictest1284255-21true20:10:012
122121399107,2cyclictest0-21swapper/019:10:010
12212279997,1cyclictest1234493-21kworker/2:1+events_freezable_power_23:12:322
12212279997,1cyclictest0-21swapper/222:45:322
12212279997,1cyclictest0-21swapper/221:40:012
12212279997,1cyclictest0-21swapper/219:10:012
12212279996,2cyclictest0-21swapper/220:15:282
12212279993,4cyclictest1444-21dbus-daemon19:20:012
12212279993,4cyclictest1438061-21apt-get22:30:212
12212139996,2cyclictest1487438-21kworker/u8:0+events_unbound00:00:300
12212279981,5cyclictest1246258-21apt-get19:35:212
12212279981,4cyclictest1339045-21latency_hist21:00:012
12212279980,5cyclictest1344730-21/usr/sbin/munin21:05:182
41339870,4rtkit-daemon29787-21packagekitd19:05:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional