You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-08 - 00:58
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 792 highest latencies:
System rackdslot2.osadl.org (updated Thu Dec 07, 2023 12:46:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1235911530,36ptp4l401ktimersoftd/307:06:453
28489998512,14cyclictest0-21swapper/207:25:052
28490998414,15cyclictest5845-21awk08:40:013
28487998130,10cyclictest9-21ksoftirqd/011:50:380
28487998031,11cyclictest9-21ksoftirqd/010:40:220
28487997770,4cyclictest9-21ksoftirqd/009:40:180
28487997669,4cyclictest9-21ksoftirqd/008:40:210
2848999759,19cyclictest12050-21sed11:10:242
57698740,11rtkit-daemon0-21swapper/007:09:360
2848999749,19cyclictest27654-21grep09:25:272
28487997467,5cyclictest9-21ksoftirqd/011:20:190
28487997436,11cyclictest9-21ksoftirqd/010:25:300
281892741,20sleep20-21swapper/207:09:522
2848999738,19cyclictest4426-21sed09:45:012
28487997366,4cyclictest9-21ksoftirqd/010:10:190
28490997216,14cyclictest11564-21sed12:15:393
2848999729,13cyclictest9903-21ls08:45:402
28487997265,4cyclictest9-21ksoftirqd/010:35:210
2848999718,19cyclictest4666-21date08:35:322
2848999717,14cyclictest20190-21grep08:00:242
28487997163,5cyclictest9-21ksoftirqd/007:30:190
28487997063,4cyclictest9-21ksoftirqd/010:50:200
28487997033,5cyclictest9-21ksoftirqd/011:10:390
28487996961,5cyclictest9-21ksoftirqd/009:45:200
2848999688,16cyclictest13508-21sshd08:55:242
28487996861,4cyclictest9-21ksoftirqd/012:10:220
28487996833,5cyclictest9-21ksoftirqd/011:40:380
28487996831,9cyclictest9-21ksoftirqd/009:05:310
2848999678,13cyclictest14694-21fschecks_time11:15:262
28487996760,4cyclictest9-21ksoftirqd/012:15:190
28487996760,4cyclictest9-21ksoftirqd/007:10:190
28487996734,5cyclictest9-21ksoftirqd/007:40:220
28487996734,4cyclictest9-21ksoftirqd/010:15:340
28487996733,5cyclictest9-21ksoftirqd/010:20:360
28487996733,5cyclictest9-21ksoftirqd/008:30:390
28487996733,5cyclictest9-21ksoftirqd/007:50:250
28487996732,5cyclictest9-21ksoftirqd/011:05:270
28487996732,4cyclictest9-21ksoftirqd/008:15:230
279402674,11sleep10-21swapper/107:06:361
28490996611,14cyclictest7886-21kernelversion11:00:303
28489996613,49cyclictest8163-21latency_hist08:44:592
28487996659,4cyclictest9-21ksoftirqd/007:20:180
28487996639,4cyclictest9-21ksoftirqd/010:30:390
28487996632,5cyclictest9-21ksoftirqd/009:50:200
28487996632,4cyclictest9-21ksoftirqd/010:05:230
28487996631,6cyclictest9-21ksoftirqd/009:15:240
28490996512,12cyclictest0-21swapper/309:05:203
28487996533,4cyclictest9-21ksoftirqd/008:05:240
28487996532,4cyclictest9-21ksoftirqd/012:00:360
28487996532,4cyclictest9-21ksoftirqd/009:00:320
28487996532,4cyclictest9-21ksoftirqd/008:25:280
28487996529,7cyclictest9-21ksoftirqd/009:10:330
2848999648,14cyclictest24044-21latency_hist08:10:002
2848999647,9cyclictest15406-21ls10:05:392
28487996431,4cyclictest9-21ksoftirqd/011:35:340
28487996431,4cyclictest9-21ksoftirqd/011:15:210
28487996431,4cyclictest9-21ksoftirqd/010:55:380
28487996431,4cyclictest9-21ksoftirqd/010:55:370
28487996431,4cyclictest9-21ksoftirqd/008:50:360
28487996422,6cyclictest9-21ksoftirqd/007:50:000
2848999638,10cyclictest26477-21iostat11:40:272
2848999637,9cyclictest6128-21ls10:55:422
2848999637,9cyclictest6128-21ls10:55:412
2848999637,9cyclictest17469-21meminfo11:20:272
28487996330,4cyclictest9-21ksoftirqd/009:25:300
28487996328,5cyclictest9-21ksoftirqd/007:55:360
2848999625,14cyclictest15639-21cat09:00:202
28487996231,8cyclictest9-21ksoftirqd/012:30:320
28487996228,4cyclictest9-21ksoftirqd/011:45:270
28487996228,4cyclictest9-21ksoftirqd/011:25:370
28487996228,4cyclictest9-21ksoftirqd/008:45:390
2848999619,13cyclictest23837-21which09:15:352
28487996127,4cyclictest9-21ksoftirqd/011:00:290
28487996126,4cyclictest9-21ksoftirqd/009:55:420
28487996125,4cyclictest9-21ksoftirqd/012:10:000
2849099598,13cyclictest0-21swapper/307:20:283
2848999598,14cyclictest452-21latency_hist07:20:002
28487995951,3cyclictest9-21ksoftirqd/012:40:000
28487995948,3cyclictest9-21ksoftirqd/012:20:250
28487995947,4cyclictest9-21ksoftirqd/011:55:260
28487995946,4cyclictest9-21ksoftirqd/011:30:340
28487995924,5cyclictest9-21ksoftirqd/012:25:340
28487995924,5cyclictest9-21ksoftirqd/012:25:330
28487995924,4cyclictest9-21ksoftirqd/008:35:390
220182590,2sleep10-21swapper/111:30:281
158892590,8sleep12848899cyclictest09:00:221
2848999586,9cyclictest2751-21df_inode10:50:242
28487995852,3cyclictest9-21ksoftirqd/008:55:240
28487995852,3cyclictest9-21ksoftirqd/008:05:000
28487995817,8cyclictest9-21ksoftirqd/010:00:220
28490995711,14cyclictest7425-21perl09:50:243
28487995751,3cyclictest9-21ksoftirqd/008:25:010
2848799571,3cyclictest121rcu_preempt07:25:040
28488995649,4cyclictest25-21ksoftirqd/109:20:001
28488995649,4cyclictest25-21ksoftirqd/108:25:231
28487995625,7cyclictest9-21ksoftirqd/009:35:260
28487995621,5cyclictest9-21ksoftirqd/009:30:230
28490995511,14cyclictest29939-21sed07:10:353
28490995511,14cyclictest18105-21irqrtprio07:55:273
2848999558,13cyclictest11904-21df_abs10:00:232
28487995549,3cyclictest9-21ksoftirqd/007:19:590
28487995448,3cyclictest9-21ksoftirqd/009:20:340
28487995447,3cyclictest9-21ksoftirqd/007:35:310
2848999536,10cyclictest4929-21df_inode09:45:232
28488995346,4cyclictest25-21ksoftirqd/112:20:001
28487995346,4cyclictest1669-21sshd10:45:470
28487995345,4cyclictest9-21ksoftirqd/008:15:000
123591530,1ptp4l391rcuc/312:39:013
2848999526,8cyclictest29218-21apt-get09:30:032
2848999516,14cyclictest0-21swapper/211:35:222
2848999516,14cyclictest0-21swapper/211:25:392
28488995144,4cyclictest25-21ksoftirqd/110:15:221
28488995144,4cyclictest25-21ksoftirqd/107:45:001
28490995043,4cyclictest13691-21cron11:15:003
2848999506,13cyclictest28327-21cstates11:45:202
2848999505,10cyclictest8757-21iostat_ios07:35:272
28488995044,4cyclictest25-21ksoftirqd/108:20:001
28488995044,4cyclictest25-21ksoftirqd/107:19:591
28488995044,3cyclictest25-21ksoftirqd/112:00:001
28488995043,4cyclictest25-21ksoftirqd/110:00:221
28488995042,6cyclictest25-21ksoftirqd/107:30:001
2848999495,18cyclictest21173-21ps09:10:342
28488994942,4cyclictest25-21ksoftirqd/111:40:011
28488994942,4cyclictest25-21ksoftirqd/108:00:221
28488994842,4cyclictest25-21ksoftirqd/107:10:271
28488994841,4cyclictest25-21ksoftirqd/112:40:001
28488994841,4cyclictest25-21ksoftirqd/110:55:271
28488994841,4cyclictest25-21ksoftirqd/110:55:261
28488994841,4cyclictest25-21ksoftirqd/110:45:221
28488994841,4cyclictest25-21ksoftirqd/109:05:201
28488994841,4cyclictest25-21ksoftirqd/108:49:591
57698470,10rtkit-daemon0-21swapper/107:51:411
28489994712,19cyclictest1602-21sshd09:39:372
28488994742,3cyclictest25-21ksoftirqd/111:25:011
28488994740,4cyclictest25-21ksoftirqd/107:25:011
28490994612,18cyclictest1139-21snmpd12:08:363
28490994612,14cyclictest10728-21sed11:05:363
28488994640,4cyclictest25-21ksoftirqd/111:25:221
28488994639,4cyclictest25-21ksoftirqd/112:35:001
28488994639,4cyclictest25-21ksoftirqd/111:45:271
28488994638,4cyclictest25-21ksoftirqd/109:55:241
28488994615,4cyclictest25-21ksoftirqd/112:25:391
28488994615,4cyclictest25-21ksoftirqd/112:25:391
28490994511,18cyclictest1139-21snmpd10:38:213
28488994539,4cyclictest25-21ksoftirqd/108:10:001
28488994538,4cyclictest25-21ksoftirqd/111:44:591
28488994537,5cyclictest25-21ksoftirqd/107:40:001
28488994522,5cyclictest25-21ksoftirqd/108:10:231
28488994519,4cyclictest25-21ksoftirqd/108:50:401
28490994411,10cyclictest32047-21apt-get09:35:193
28490994411,10cyclictest29660-21/usr/sbin/munin09:30:303
28488994439,3cyclictest25-21ksoftirqd/110:35:231
28488994438,4cyclictest25-21ksoftirqd/109:55:011
123591440,1ptp4l401ktimersoftd/308:08:343
2849099437,11cyclictest0-21swapper/312:10:313
28490994311,9cyclictest6145-21ls09:45:393
28490994311,19cyclictest1139-21snmpd08:53:023
2848999439,10cyclictest31480-21switchtime10:40:352
2848999437,13cyclictest9152-21/usr/sbin/munin09:55:242
28489994337,4cyclictest33-21ksoftirqd/212:11:382
2848899438,4cyclictest25-21ksoftirqd/111:05:251
28488994337,4cyclictest25-21ksoftirqd/110:30:301
28488994337,4cyclictest25-21ksoftirqd/110:10:011
28488994337,4cyclictest25-21ksoftirqd/109:35:251
28488994337,3cyclictest25-21ksoftirqd/108:35:011
2848899431,3cyclictest121rcu_preempt10:14:591
123591430,1ptp4l401ktimersoftd/308:59:373
28490994235,4cyclictest41-21ksoftirqd/311:52:373
28490994235,4cyclictest41-21ksoftirqd/310:48:233
28490994212,15cyclictest4127-21ls12:00:373
28490994211,16cyclictest15608-21ls11:15:383
28490994210,19cyclictest6705-21idleruntime08:40:273
2848999428,9cyclictest17567-21ls10:10:372
2848999428,18cyclictest22111-21missed_timers11:30:312
2848999427,21cyclictest23817-21irqrtprio10:25:302
28489994236,4cyclictest33-21ksoftirqd/212:00:202
28489994235,4cyclictest33-21ksoftirqd/212:30:312
28488994235,4cyclictest25-21ksoftirqd/112:00:301
28488994235,4cyclictest25-21ksoftirqd/109:25:321
28488994235,4cyclictest25-21ksoftirqd/109:20:201
28488994235,4cyclictest25-21ksoftirqd/107:59:451
28488994234,4cyclictest25-21ksoftirqd/107:30:221
28490994134,4cyclictest41-21ksoftirqd/308:25:003
28490994111,15cyclictest24697-21df_inode08:10:203
28490994111,15cyclictest23491-21apt-get11:35:193
28490994110,18cyclictest12327-21apt-get12:20:203
2848999418,15cyclictest3383-21wc08:35:002
2848999417,13cyclictest18852-21df_abs10:15:222
28489994135,4cyclictest33-21ksoftirqd/208:11:372
28489994134,4cyclictest33-21ksoftirqd/207:41:192
28488994135,4cyclictest25-21ksoftirqd/110:25:311
28488994134,4cyclictest25-21ksoftirqd/111:52:401
28488994134,4cyclictest25-21ksoftirqd/108:40:001
28490994034,4cyclictest41-21ksoftirqd/311:31:393
28490994033,4cyclictest41-21ksoftirqd/308:28:053
28490994011,15cyclictest28117-21ls09:25:333
28490994011,15cyclictest26364-21meminfo10:30:313
28490994010,13cyclictest20277-21cstates09:10:223
2848999408,8cyclictest24893-21apt-get09:20:182
2848999407,10cyclictest7878-21irqstats11:00:292
2848999407,10cyclictest30449-21apt-get11:50:192
28489994034,3cyclictest33-21ksoftirqd/212:23:462
28489994034,3cyclictest33-21ksoftirqd/210:32:182
28488994035,3cyclictest25-21ksoftirqd/109:00:001
28488994035,3cyclictest25-21ksoftirqd/108:40:391
28488994034,4cyclictest25-21ksoftirqd/111:20:001
28488994034,3cyclictest25-21ksoftirqd/110:50:211
28488994034,3cyclictest25-21ksoftirqd/107:49:181
1236914028,5phc2sys41-21ksoftirqd/312:30:203
28490993933,4cyclictest41-21ksoftirqd/308:49:373
28490993933,3cyclictest41-21ksoftirqd/308:32:593
28490993932,4cyclictest41-21ksoftirqd/310:59:353
28490993932,4cyclictest41-21ksoftirqd/310:59:353
28490993910,15cyclictest13876-21apt-get10:05:203
2848999397,9cyclictest6755-21ls12:05:372
2848999397,9cyclictest20659-21ssh12:35:372
28489993932,4cyclictest33-21ksoftirqd/210:48:422
28489993931,5cyclictest33-21ksoftirqd/208:15:072
28488993933,4cyclictest25-21ksoftirqd/109:10:391
28488993932,4cyclictest25-21ksoftirqd/110:23:361
28488993932,4cyclictest25-21ksoftirqd/109:50:001
2849099384,11cyclictest0-21swapper/308:00:313
28490993832,4cyclictest41-21ksoftirqd/310:11:493
28490993832,3cyclictest41-21ksoftirqd/311:43:283
28490993831,4cyclictest41-21ksoftirqd/307:51:153
28490993831,4cyclictest41-21ksoftirqd/307:16:103
2848999387,18cyclictest21745-21meminfo10:20:302
2848999386,9cyclictest28885-21df07:10:212
28489993831,4cyclictest33-21ksoftirqd/208:24:012
28489993831,4cyclictest33-21ksoftirqd/207:21:472
28489993822,11cyclictest31283-21apt-get08:25:212
28488993832,4cyclictest25-21ksoftirqd/112:10:371
28488993832,4cyclictest25-21ksoftirqd/111:10:291
28488993831,4cyclictest25-21ksoftirqd/112:25:001
28488993831,4cyclictest25-21ksoftirqd/112:10:001
28488993831,4cyclictest25-21ksoftirqd/110:40:021
28488993831,4cyclictest25-21ksoftirqd/109:30:231
28490993731,3cyclictest41-21ksoftirqd/310:50:413
28490993730,4cyclictest41-21ksoftirqd/312:26:113
28490993730,4cyclictest41-21ksoftirqd/312:26:103
28490993730,4cyclictest41-21ksoftirqd/310:21:333
2848999378,9cyclictest10763-21apt-get08:50:202
2848999378,15cyclictest15936-21mailstats12:25:362
2848999378,15cyclictest15936-21mailstats12:25:362
28489993732,3cyclictest33-21ksoftirqd/207:50:352
28489993731,3cyclictest33-21ksoftirqd/211:56:232
28489993730,4cyclictest33-21ksoftirqd/211:06:532
28489993730,4cyclictest33-21ksoftirqd/207:33:202
28489993722,7cyclictest17726-21apt-get09:05:192
28489993721,9cyclictest6914-21apt-get09:50:192
2848899373,3cyclictest121rcu_preempt08:20:211
2849099365,9cyclictest0-21swapper/307:45:213
28490993630,3cyclictest41-21ksoftirqd/309:20:563
28490993617,16cyclictest208-21systemd-journal09:17:003
2848999366,18cyclictest208-21systemd-journal10:38:212
28489993630,4cyclictest33-21ksoftirqd/207:59:352
28488993630,3cyclictest25-21ksoftirqd/109:43:071
2848899361,3cyclictest121rcu_preempt11:02:131
28490993529,4cyclictest41-21ksoftirqd/309:03:493
28490993527,5cyclictest41-21ksoftirqd/310:01:503
28489993528,4cyclictest33-21ksoftirqd/212:16:412
28490993428,4cyclictest41-21ksoftirqd/311:21:393
28490993428,4cyclictest41-21ksoftirqd/310:17:073
28490993428,3cyclictest41-21ksoftirqd/308:18:303
28490993420,9cyclictest6138-21df07:30:223
28490993418,11cyclictest10182-21grep09:55:353
28490993417,13cyclictest9657-21grep07:35:383
28490993327,3cyclictest41-21ksoftirqd/311:27:193
28490993326,4cyclictest41-21ksoftirqd/310:42:163
28490993319,10cyclictest24277-21wc10:25:363
28490993318,11cyclictest28813-21sed11:45:263
28490993218,11cyclictest5020-21head07:25:383
28490993218,10cyclictest2906-21grep09:40:263
28490993218,10cyclictest1353-21needreboot11:55:353
28490993118,10cyclictest11012-21iwlist07:40:253
2848999317,9cyclictest12744-21apt-get07:45:222
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional