You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-03-17 - 10:31
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rackdslot2.osadl.org (updated Mon Mar 17, 2025 00:45:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
189702228168,49sleep10-21swapper/119:07:061
137391213169,33phc2sys0-21swapper/319:10:003
191642205172,22sleep20-21swapper/219:09:352
191152204170,23sleep00-21swapper/019:08:570
2696721340,2sleep20-21swapper/200:35:172
1954499730,25cyclictest25544-21chrt21:45:471
1954499720,32cyclictest22474-21chrt00:20:271
1954499690,34cyclictest4325-21/usr/sbin/munin20:55:301
1954499680,34cyclictest31786-21/usr/sbin/munin20:45:221
1954499671,33cyclictest26374-21dpkg00:35:001
1954499670,24cyclictest20823-1kworker/1:2H23:35:001
1954499661,32cyclictest2494-21gnome-shell00:17:581
1954499661,32cyclictest2494-21gnome-shell00:17:581
1954499660,33cyclictest9280-21apt22:30:141
1954499651,27cyclictest25108-21/usr/sbin/munin00:30:231
1954499641,26cyclictest26629-21/usr/sbin/munin21:50:121
1954499640,32cyclictest23895-21sshd23:04:181
1954499640,25cyclictest3559-21chrt19:44:371
1954499630,33cyclictest17017-21taskset21:22:151
1954499630,27cyclictest18167-21/usr/sbin/munin22:50:171
1954499611,26cyclictest29417-21/usr/sbin/munin19:30:181
1954499610,26cyclictest6990-21chrt23:43:411
1954499610,26cyclictest20799-21chrt20:23:471
1954499610,25cyclictest26339-21chrt19:23:351
1954499601,33cyclictest2494-21gnome-shell00:29:581
1954499601,25cyclictest27587-21sed20:35:341
1954499600,31cyclictest24405-21taskset23:07:511
1954499600,29cyclictest31367-21taskset23:20:531
1954499600,27cyclictest3206-21if_err_enp4s023:35:181
1954499591,25cyclictest23959-21/usr/sbin/munin21:40:211
1954499590,24cyclictest29210-21chrt23:18:471
1954499590,24cyclictest16754-21taskset22:49:531
1954499580,25cyclictest28477-21chrt21:55:061
1954499580,24cyclictest578-21dpkg20:50:151
1954499570,32cyclictest7511-21taskset21:04:461
1954499570,26cyclictest12937-21dpkg00:00:141
1954499570,23cyclictest18368-21chrt20:19:061
1954499562,23cyclictest20294-21latency_hist21:35:001
1954499561,26cyclictest10644-21/usr/sbin/munin20:00:261
1954499560,25cyclictest9956-21chrt21:08:011
1954499560,24cyclictest8674-21chrt22:25:341
1954499560,24cyclictest20691-21ps22:55:231
1954499560,22cyclictest135352sleep122:37:151
19543995627,9cyclictest9-21ksoftirqd/020:55:300
137291560,1ptp4l401ktimersoftd/300:26:593
1954499551,27cyclictest30216-21seq20:42:521
1954499550,26cyclictest31947-21/usr/sbin/munin19:35:171
1954499550,26cyclictest10028-21taskset23:51:401
1954499550,23cyclictest38422chrt19:45:211
1954499550,23cyclictest231302chrt20:28:211
1954499541,24cyclictest8076-21taskset22:20:531
1954499540,24cyclictest13439-21chrt20:06:421
1954499540,20cyclictest0-21swapper/123:46:251
1954499530,29cyclictest30869-21taskset22:00:111
1954499530,24cyclictest21606-21chrt19:13:561
1954499530,19cyclictest18211-21taskset00:14:301
19543995320,9cyclictest9-21ksoftirqd/019:40:130
137291530,1ptp4l401ktimersoftd/319:50:233
137291530,1ptp4l391rcuc/321:05:323
1954499520,25cyclictest8241-21/usr/sbin/munin19:55:261
1954499520,24cyclictest25326-21chrt20:31:141
1954499520,24cyclictest154252sleep100:06:571
1954499520,19cyclictest32231-21chrt23:28:581
1954499520,19cyclictest32231-21chrt23:28:571
19543995122,4cyclictest9-21ksoftirqd/023:30:110
1954499502,23cyclictest12568-21chrt23:58:491
1954499500,24cyclictest17630-21taskset21:27:091
19544994926,20cyclictest12487-21sh21:15:011
1954499490,25cyclictest15914-21chrt22:42:081
1954499490,23cyclictest57312sleep122:16:281
1954499480,22cyclictest26635-21chrt23:11:111
1954499480,22cyclictest1285-21runrttasks22:06:441
1954399483,6cyclictest29162-21ps21:55:200
19546994743,3cyclictest23830-21sshd19:17:053
130972470,1sleep20-21swapper/200:00:152
1954499460,24cyclictest6561-21chrt19:54:481
1954499460,24cyclictest23741-21chrt19:16:001
1954499460,23cyclictest20531-21/usr/sbin/munin21:35:201
1954399467,6cyclictest9-21ksoftirqd/019:35:260
19543994629,9cyclictest9-21ksoftirqd/000:35:000
19543994542,2cyclictest111rcuc/021:28:080
19543994541,2cyclictest111rcuc/020:10:160
19543994528,9cyclictest9-21ksoftirqd/000:20:170
1954399452,6cyclictest9-21ksoftirqd/020:49:590
1954399446,5cyclictest9-21ksoftirqd/022:10:000
1954399443,6cyclictest9-21ksoftirqd/022:30:160
19543994428,5cyclictest9-21ksoftirqd/019:25:150
19543994428,5cyclictest9-21ksoftirqd/019:25:150
19543994427,9cyclictest9-21ksoftirqd/023:20:240
137291440,0ptp4l401ktimersoftd/322:41:113
19546994339,2cyclictest25889-21sh21:50:013
1954399434,5cyclictest9-21ksoftirqd/021:15:260
19543994339,3cyclictest111rcuc/023:40:100
19543994339,2cyclictest111rcuc/019:24:230
19543994329,6cyclictest9-21ksoftirqd/022:50:200
1954499421,29cyclictest12708-21taskset21:15:031
1954399423,7cyclictest9-21ksoftirqd/021:40:210
19543994229,4cyclictest9-21ksoftirqd/020:40:130
19543994211,5cyclictest9-21ksoftirqd/020:50:210
1954399420,7cyclictest101ktimersoftd/019:10:130
1954399420,5cyclictest218-21systemd-journal00:06:310
1954499411,21cyclictest0-21swapper/122:13:351
1954499410,26cyclictest26705-21dpkg19:25:141
1954499410,26cyclictest26705-21dpkg19:25:131
1954399413,5cyclictest9-21ksoftirqd/023:50:010
19543994125,9cyclictest9-21ksoftirqd/023:50:010
137291410,1ptp4l401ktimersoftd/323:15:173
19546994035,3cyclictest41-21ksoftirqd/320:50:503
1954399404,8cyclictest9-21ksoftirqd/021:35:280
1954399404,5cyclictest9-21ksoftirqd/000:15:280
1954399404,5cyclictest9-21ksoftirqd/000:15:280
1954399402,5cyclictest9-21ksoftirqd/021:51:030
1954399401,36cyclictest23965-21latency_hist23:05:010
137291400,1ptp4l401ktimersoftd/300:07:483
19546993935,2cyclictest41-21ksoftirqd/300:00:013
1954499391,24cyclictest15943-21taskset20:13:201
1954399394,4cyclictest9-21ksoftirqd/000:33:000
1954399393,6cyclictest9-21ksoftirqd/022:55:130
19543993936,2cyclictest111rcuc/020:28:380
19543993928,5cyclictest9-21ksoftirqd/020:00:170
1954399392,6cyclictest9-21ksoftirqd/021:05:140
1954399392,5cyclictest9-21ksoftirqd/022:10:120
1954399390,7cyclictest5825-21sshd22:17:330
19546993826,9cyclictest41-21ksoftirqd/319:40:283
1954399385,7cyclictest9-21ksoftirqd/021:10:310
19543993835,2cyclictest111rcuc/019:31:100
19543993834,2cyclictest111rcuc/022:36:480
1954399382,6cyclictest9-21ksoftirqd/019:55:150
19543993826,4cyclictest9-21ksoftirqd/020:20:160
1954399382,5cyclictest9-21ksoftirqd/019:15:230
1954399381,5cyclictest9-21ksoftirqd/022:40:290
19543993722,6cyclictest9-21ksoftirqd/019:50:130
19543993712,8cyclictest9-21ksoftirqd/021:45:000
1954399370,7cyclictest101ktimersoftd/023:10:200
19546993631,3cyclictest41-21ksoftirqd/323:25:013
19546993631,3cyclictest41-21ksoftirqd/322:35:013
19546993631,3cyclictest41-21ksoftirqd/321:00:003
19546993631,3cyclictest41-21ksoftirqd/319:15:003
19546993630,4cyclictest41-21ksoftirqd/322:37:133
19546993630,4cyclictest41-21ksoftirqd/319:28:413
19546993630,4cyclictest41-21ksoftirqd/319:28:413
1954599365,4cyclictest121rcu_preempt22:00:002
19545993631,3cyclictest2786-21turbostat.cron20:55:012
19545993629,4cyclictest33-21ksoftirqd/200:26:222
19543993633,2cyclictest111rcuc/023:36:090
19543993633,2cyclictest111rcuc/020:36:290
19543993632,3cyclictest9-21ksoftirqd/022:20:280
19543993631,3cyclictest9-21ksoftirqd/022:00:200
19543993631,3cyclictest9-21ksoftirqd/020:20:010
19543993629,4cyclictest9-21ksoftirqd/000:03:050
19543993625,5cyclictest9-21ksoftirqd/021:00:350
19543993623,6cyclictest9-21ksoftirqd/000:10:320
137291360,1ptp4l391rcuc/322:00:193
42098350,7rtkit-daemon0-21swapper/320:30:073
42098350,1rtkit-daemon0-21swapper/323:13:073
19546993529,4cyclictest41-21ksoftirqd/320:19:483
19545993529,4cyclictest33-21ksoftirqd/220:24:012
1954399356,5cyclictest9-21ksoftirqd/021:20:310
19543993532,2cyclictest111rcuc/020:06:210
19543993530,3cyclictest9-21ksoftirqd/020:30:160
19543993529,4cyclictest9-21ksoftirqd/023:15:030
19543993522,5cyclictest9-21ksoftirqd/022:27:180
671623423,3sleep233-21ksoftirqd/222:20:202
19546993431,2cyclictest41-21ksoftirqd/320:05:153
19546993430,3cyclictest41-21ksoftirqd/319:23:193
19546993429,3cyclictest41-21ksoftirqd/322:55:013
19546993429,3cyclictest41-21ksoftirqd/320:50:013
19546993429,3cyclictest41-21ksoftirqd/320:14:593
19546993429,3cyclictest41-21ksoftirqd/319:55:203
19546993429,3cyclictest41-21ksoftirqd/300:36:403
19546993428,4cyclictest41-21ksoftirqd/323:46:443
19546993428,4cyclictest41-21ksoftirqd/321:56:133
19546993428,4cyclictest41-21ksoftirqd/320:42:073
19546993427,4cyclictest41-21ksoftirqd/323:50:073
1954699340,2cyclictest121rcu_preempt23:42:013
19545993429,3cyclictest33-21ksoftirqd/223:56:292
19545993428,4cyclictest33-21ksoftirqd/223:27:012
19545993428,4cyclictest33-21ksoftirqd/223:27:012
19545993428,4cyclictest33-21ksoftirqd/221:34:012
19545993428,4cyclictest33-21ksoftirqd/220:08:292
19543993431,2cyclictest111rcuc/022:45:370
19543993430,2cyclictest101ktimersoftd/021:30:180
19543993428,4cyclictest9-21ksoftirqd/000:29:050
19543993427,4cyclictest9-21ksoftirqd/019:48:280
19543993422,5cyclictest9-21ksoftirqd/023:55:170
19546993328,3cyclictest41-21ksoftirqd/322:50:143
19546993328,3cyclictest41-21ksoftirqd/321:26:033
19546993328,3cyclictest41-21ksoftirqd/300:20:153
19546993327,4cyclictest41-21ksoftirqd/323:25:513
19546993327,4cyclictest41-21ksoftirqd/323:25:503
19546993327,4cyclictest41-21ksoftirqd/323:03:553
19546993327,4cyclictest41-21ksoftirqd/321:43:203
19546993327,4cyclictest41-21ksoftirqd/321:20:133
19546993327,4cyclictest41-21ksoftirqd/321:15:563
19546993327,4cyclictest41-21ksoftirqd/320:26:393
19546993327,4cyclictest41-21ksoftirqd/319:35:563
19546993327,4cyclictest41-21ksoftirqd/300:11:033
19545993328,3cyclictest33-21ksoftirqd/220:03:022
19545993328,3cyclictest33-21ksoftirqd/219:40:412
19545993328,3cyclictest33-21ksoftirqd/219:17:002
19545993327,4cyclictest33-21ksoftirqd/223:38:292
19545993327,4cyclictest33-21ksoftirqd/222:06:342
19545993327,4cyclictest33-21ksoftirqd/220:45:382
19545993327,4cyclictest33-21ksoftirqd/220:13:022
19545993327,4cyclictest33-21ksoftirqd/219:13:242
19545993327,4cyclictest33-21ksoftirqd/200:10:542
19543993323,3cyclictest9-21ksoftirqd/023:26:060
19543993323,3cyclictest9-21ksoftirqd/023:26:060
19546993227,3cyclictest41-21ksoftirqd/300:30:133
19546993227,3cyclictest41-21ksoftirqd/300:15:193
19546993227,3cyclictest41-21ksoftirqd/300:15:193
19546993226,4cyclictest41-21ksoftirqd/322:20:153
19546993226,4cyclictest41-21ksoftirqd/322:11:283
19546993226,4cyclictest41-21ksoftirqd/320:01:073
19546993225,5cyclictest41-21ksoftirqd/322:07:073
19545993227,3cyclictest33-21ksoftirqd/223:49:162
19545993226,4cyclictest33-21ksoftirqd/222:48:462
19545993226,4cyclictest33-21ksoftirqd/222:25:032
19545993226,4cyclictest33-21ksoftirqd/222:18:462
19545993226,4cyclictest33-21ksoftirqd/221:53:502
19545993226,4cyclictest33-21ksoftirqd/221:24:512
19545993226,4cyclictest33-21ksoftirqd/221:16:432
19545993226,4cyclictest33-21ksoftirqd/221:06:162
19545993226,4cyclictest33-21ksoftirqd/221:01:082
19545993226,4cyclictest33-21ksoftirqd/220:28:022
19545993226,4cyclictest33-21ksoftirqd/200:20:492
19545993225,4cyclictest33-21ksoftirqd/220:17:432
137291320,0ptp4l401ktimersoftd/322:26:223
19546993126,3cyclictest41-21ksoftirqd/319:30:093
19546993125,4cyclictest41-21ksoftirqd/323:30:033
19546993125,4cyclictest41-21ksoftirqd/323:08:253
19546993125,4cyclictest41-21ksoftirqd/322:48:433
19546993125,4cyclictest41-21ksoftirqd/321:38:473
19546993125,4cyclictest41-21ksoftirqd/321:32:173
19546993125,4cyclictest41-21ksoftirqd/321:04:223
19545993126,3cyclictest33-21ksoftirqd/223:40:462
19545993125,4cyclictest33-21ksoftirqd/223:52:322
19545993125,4cyclictest33-21ksoftirqd/223:17:542
19545993125,4cyclictest33-21ksoftirqd/221:48:452
19545993125,4cyclictest33-21ksoftirqd/221:28:042
19545993125,4cyclictest33-21ksoftirqd/220:39:492
19545993125,4cyclictest33-21ksoftirqd/219:40:012
19545993122,7cyclictest33-21ksoftirqd/220:42:202
19543993128,2cyclictest111rcuc/023:06:520
137291310,0ptp4l401ktimersoftd/323:39:523
137291310,0ptp4l401ktimersoftd/321:10:523
19546993025,3cyclictest41-21ksoftirqd/320:40:003
19546993024,3cyclictest41-21ksoftirqd/320:25:003
19545993025,3cyclictest33-21ksoftirqd/223:22:282
19545993025,3cyclictest33-21ksoftirqd/223:01:432
19545993025,3cyclictest33-21ksoftirqd/222:58:582
19545993025,3cyclictest33-21ksoftirqd/219:56:012
19545993025,3cyclictest33-21ksoftirqd/200:32:062
19545993024,4cyclictest33-21ksoftirqd/221:36:542
19545993023,5cyclictest33-21ksoftirqd/222:40:112
137291300,0ptp4l401ktimersoftd/321:52:253
19546992923,4cyclictest41-21ksoftirqd/300:04:183
19545992924,3cyclictest33-21ksoftirqd/223:10:072
19545992924,3cyclictest33-21ksoftirqd/221:40:292
19545992924,3cyclictest33-21ksoftirqd/220:57:192
19545992924,3cyclictest33-21ksoftirqd/219:55:002
19545992924,3cyclictest33-21ksoftirqd/200:16:372
19545992924,3cyclictest33-21ksoftirqd/200:16:362
19545992924,3cyclictest33-21ksoftirqd/200:08:182
19545992923,4cyclictest33-21ksoftirqd/223:05:192
19545992923,4cyclictest33-21ksoftirqd/221:11:212
19545992823,3cyclictest33-21ksoftirqd/222:12:062
19545992823,3cyclictest33-21ksoftirqd/222:00:042
19546992722,3cyclictest41-21ksoftirqd/322:15:313
19546992721,4cyclictest41-21ksoftirqd/319:47:353
19545992722,3cyclictest33-21ksoftirqd/222:34:342
19545992722,3cyclictest33-21ksoftirqd/220:31:412
19545992721,4cyclictest33-21ksoftirqd/219:30:382
19545992621,3cyclictest33-21ksoftirqd/222:53:332
19545992621,3cyclictest33-21ksoftirqd/219:48:152
19545992520,3cyclictest33-21ksoftirqd/223:32:012
19545992420,3cyclictest33-21ksoftirqd/219:25:232
19545992420,3cyclictest33-21ksoftirqd/219:25:232
19545992318,3cyclictest33-21ksoftirqd/222:35:492
19545992115,4cyclictest33-21ksoftirqd/219:20:352
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional