You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-19 - 06:43
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rackdslot3.osadl.org (updated Sat Jul 19, 2025 00:45:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2748222270,6sleep233-21ksoftirqd/219:05:252
137191212178,23phc2sys0-21swapper/319:09:573
281482205172,22sleep00-21swapper/019:09:150
278912204172,21sleep10-21swapper/119:05:531
487021580,5sleep32866899cyclictest19:25:253
65002620,1sleep00-21swapper/022:50:260
2866899616,16cyclictest24319-21awk20:05:273
2866899557,16cyclictest28537-21cat21:25:003
2866899554,16cyclictest0-21swapper/321:40:213
28668995511,9cyclictest7574-21df_abs22:55:173
2866899537,16cyclictest28924-21latency_hist23:40:003
230262510,2sleep20-21swapper/221:10:262
190642510,2sleep10-21swapper/122:10:231
106522510,3sleep210656-21irqrtprio23:00:192
2866899507,16cyclictest1273-21latency_hist23:49:593
2866899507,15cyclictest18516-21mailstats23:15:253
2866899505,14cyclictest13562-21runrttasks23:05:233
2866899498,14cyclictest19211-21/usr/sbin/munin21:05:203
2866899497,8cyclictest5876-21load22:50:193
59298480,10rtkit-daemon0-21swapper/122:40:231
2866899478,7cyclictest13952-21fschecks_count00:15:163
2866899477,7cyclictest5579-21users23:55:283
2866899477,7cyclictest20870-21apt-get00:30:123
2866899477,7cyclictest20870-21apt-get00:30:123
2866899477,15cyclictest32078-21cat19:15:213
2866899472,10cyclictest29030-21df_abs21:25:143
10552470,2sleep30-21swapper/322:40:223
2866899467,8cyclictest23959-21apt-get21:15:113
2866899458,15cyclictest9449-21/usr/sbin/munin20:45:183
2866899457,7cyclictest20577-21apt-get22:15:123
2866899457,7cyclictest15270-21perl23:10:133
2866899457,7cyclictest10847-21meminfo23:00:203
28665994539,4cyclictest25-21ksoftirqd/122:19:261
2866899447,16cyclictest11884-21apt-config20:50:013
2866899445,7cyclictest6086-21apt-get21:45:143
2866899444,8cyclictest2075-21ps23:50:213
2866899436,7cyclictest2489-21apt-get22:45:133
2866899436,15cyclictest15907-21/usr/sbin/munin00:20:223
136991430,0ptp4l401ktimersoftd/323:40:443
136991430,0ptp4l401ktimersoftd/320:01:523
136991430,0ptp4l401ktimersoftd/319:35:313
2866899427,14cyclictest23631-21ps22:20:203
2866899425,8cyclictest7465-21ps00:00:233
2866899425,7cyclictest25415-21ls23:30:213
2866899425,7cyclictest25415-21ls23:30:213
28666994223,17cyclictest33-21ksoftirqd/223:25:592
136991420,0ptp4l401ktimersoftd/300:37:003
2866899417,13cyclictest26344-21ls22:25:173
136991410,0ptp4l401ktimersoftd/319:54:233
2866899405,15cyclictest21941-21/usr/sbin/munin23:25:143
28665994034,3cyclictest10372-21irqrtprio20:45:191
136991400,0ptp4l401ktimersoftd/322:30:213
136991400,0ptp4l401ktimersoftd/300:10:343
136991400,0ptp4l401ktimersoftd/300:09:533
59298390,8rtkit-daemon0-21swapper/000:07:510
2866899398,11cyclictest13853-21iostat_ios22:00:203
2866899397,14cyclictest30064-21/usr/sbin/munin22:35:203
59298380,4rtkit-daemon1039-21apt-get21:35:141
28665993833,3cyclictest25-21ksoftirqd/119:20:001
28665993832,3cyclictest25-21ksoftirqd/119:35:001
2866899378,11cyclictest19973-21diskstats23:20:163
28666993731,4cyclictest33-21ksoftirqd/221:35:142
28665993729,5cyclictest25-21ksoftirqd/121:44:151
2866699366,29cyclictest21796-21kworker/2:219:19:552
28666993630,4cyclictest33-21ksoftirqd/222:32:472
28666993630,4cyclictest33-21ksoftirqd/222:08:152
28666993630,4cyclictest33-21ksoftirqd/200:11:482
2866699362,25cyclictest8279-21kworker/2:121:22:272
28665993632,2cyclictest17306-21grep23:15:141
28665993631,3cyclictest25-21ksoftirqd/119:30:011
28665993630,4cyclictest25-21ksoftirqd/120:15:001
28665993630,3cyclictest25-21ksoftirqd/119:15:001
28664993631,3cyclictest9-21ksoftirqd/021:40:000
28664993631,3cyclictest9-21ksoftirqd/021:00:000
59298350,2rtkit-daemon0-21swapper/122:55:201
28668993530,3cyclictest41-21ksoftirqd/320:44:553
28668993530,3cyclictest41-21ksoftirqd/320:25:503
28666993529,4cyclictest33-21ksoftirqd/222:54:452
28666993529,4cyclictest33-21ksoftirqd/219:25:482
28666993529,4cyclictest33-21ksoftirqd/219:24:512
28666993529,4cyclictest33-21ksoftirqd/200:23:102
28666993528,4cyclictest33-21ksoftirqd/220:21:162
28665993530,3cyclictest25-21ksoftirqd/100:30:001
28665993529,4cyclictest25-21ksoftirqd/121:22:371
28665993528,5cyclictest25-21ksoftirqd/119:38:101
28664993530,3cyclictest9-21ksoftirqd/021:50:000
28664993530,3cyclictest9-21ksoftirqd/020:00:000
2866899348,16cyclictest2499-21cut20:30:143
2866899346,10cyclictest26755-21ls20:10:253
28668993428,4cyclictest41-21ksoftirqd/320:37:223
2866699345,27cyclictest8279-21kworker/2:120:31:432
28666993429,3cyclictest33-21ksoftirqd/223:10:092
28666993429,3cyclictest33-21ksoftirqd/221:06:002
28666993428,4cyclictest33-21ksoftirqd/221:56:202
28666993428,4cyclictest33-21ksoftirqd/220:11:502
28666993427,5cyclictest33-21ksoftirqd/219:41:282
28666993427,4cyclictest33-21ksoftirqd/222:50:012
28666993427,4cyclictest33-21ksoftirqd/220:01:182
28666993427,4cyclictest33-21ksoftirqd/219:57:212
2866699341,20cyclictest8279-21kworker/2:120:25:072
28665993430,2cyclictest25-21ksoftirqd/123:25:001
28665993429,3cyclictest25-21ksoftirqd/123:50:211
28665993429,3cyclictest25-21ksoftirqd/122:50:131
28665993429,3cyclictest25-21ksoftirqd/122:39:591
28665993429,3cyclictest25-21ksoftirqd/121:55:001
28665993428,4cyclictest25-21ksoftirqd/123:04:391
28665993428,4cyclictest25-21ksoftirqd/121:47:441
28665993428,4cyclictest25-21ksoftirqd/121:18:241
28665993428,4cyclictest25-21ksoftirqd/100:36:251
28665993428,3cyclictest25-21ksoftirqd/119:55:131
28665993427,5cyclictest25-21ksoftirqd/119:40:071
2866899337,16cyclictest17783-21/usr/sbin/munin19:55:243
2866899336,11cyclictest7466-21ls19:30:263
2866899335,15cyclictest1285-21snmpd20:55:203
2866899333,17cyclictest0-21swapper/321:10:143
28668993327,4cyclictest41-21ksoftirqd/320:18:523
28668993327,4cyclictest41-21ksoftirqd/319:45:123
2866699333,18cyclictest2909-21kworker/2:021:47:552
2866699333,10cyclictest8279-21kworker/2:121:03:512
28666993328,3cyclictest33-21ksoftirqd/223:33:012
28666993328,3cyclictest33-21ksoftirqd/223:33:012
28666993328,3cyclictest33-21ksoftirqd/200:28:112
28666993327,4cyclictest33-21ksoftirqd/221:42:402
28666993327,4cyclictest33-21ksoftirqd/221:32:412
28666993326,4cyclictest33-21ksoftirqd/219:10:122
2866699332,29cyclictest8279-21kworker/2:120:38:392
28665993329,3cyclictest25-21ksoftirqd/123:45:001
28665993328,3cyclictest25-21ksoftirqd/122:45:201
28665993328,3cyclictest25-21ksoftirqd/100:15:011
28665993327,4cyclictest25-21ksoftirqd/123:56:571
28665993327,4cyclictest25-21ksoftirqd/123:07:201
28665993327,4cyclictest25-21ksoftirqd/120:51:321
28665993327,4cyclictest25-21ksoftirqd/120:36:551
28665993327,4cyclictest25-21ksoftirqd/119:51:181
28665993327,3cyclictest25-21ksoftirqd/121:05:001
28665993327,3cyclictest25-21ksoftirqd/119:48:281
28665993326,5cyclictest25-21ksoftirqd/123:49:311
28664993328,3cyclictest9-21ksoftirqd/021:55:110
28664993327,4cyclictest9-21ksoftirqd/023:18:020
28664993327,4cyclictest9-21ksoftirqd/023:07:190
28664993327,4cyclictest9-21ksoftirqd/019:28:140
28664993327,4cyclictest9-21ksoftirqd/019:18:030
28664993327,4cyclictest9-21ksoftirqd/000:33:120
28664993327,4cyclictest9-21ksoftirqd/000:33:120
28668993226,4cyclictest41-21ksoftirqd/320:22:523
28666993226,4cyclictest33-21ksoftirqd/223:36:252
28666993226,4cyclictest33-21ksoftirqd/223:05:362
28666993226,4cyclictest33-21ksoftirqd/220:47:402
28666993226,4cyclictest33-21ksoftirqd/219:35:362
2866699322,29cyclictest2909-21kworker/2:023:53:312
2866699321,19cyclictest5446-21kworker/2:019:32:312
2866599323,2cyclictest121rcu_preempt20:29:451
28665993227,3cyclictest25-21ksoftirqd/121:30:181
28665993227,3cyclictest25-21ksoftirqd/100:02:181
28665993227,3cyclictest12931-21cron22:00:011
28665993226,4cyclictest25-21ksoftirqd/121:27:511
28665993226,4cyclictest25-21ksoftirqd/120:57:381
28665993226,4cyclictest25-21ksoftirqd/100:07:041
28665993226,3cyclictest25-21ksoftirqd/120:30:141
28665993225,4cyclictest25-21ksoftirqd/100:30:471
28665993225,4cyclictest25-21ksoftirqd/100:30:471
28664993227,3cyclictest9-21ksoftirqd/023:35:390
28664993227,3cyclictest9-21ksoftirqd/023:32:340
28664993227,3cyclictest9-21ksoftirqd/023:32:340
28664993227,3cyclictest9-21ksoftirqd/022:44:040
28664993227,3cyclictest9-21ksoftirqd/022:35:500
28664993227,3cyclictest9-21ksoftirqd/021:14:220
28664993227,3cyclictest9-21ksoftirqd/021:07:200
28664993227,3cyclictest9-21ksoftirqd/020:32:050
28664993227,3cyclictest9-21ksoftirqd/019:36:050
28664993226,4cyclictest9-21ksoftirqd/022:46:450
28664993226,4cyclictest9-21ksoftirqd/022:30:440
28664993226,4cyclictest9-21ksoftirqd/022:03:560
28664993226,4cyclictest9-21ksoftirqd/000:19:390
2866899318,10cyclictest18112-21gpgv22:10:013
2866899316,12cyclictest18935-21iostat00:25:203
28666993126,3cyclictest33-21ksoftirqd/223:15:132
28666993126,3cyclictest33-21ksoftirqd/220:09:312
28666993126,3cyclictest33-21ksoftirqd/200:39:262
28666993125,4cyclictest33-21ksoftirqd/223:22:092
28666993125,4cyclictest33-21ksoftirqd/222:17:222
28666993125,4cyclictest33-21ksoftirqd/220:55:462
28666993125,4cyclictest33-21ksoftirqd/220:16:312
28665993126,3cyclictest25-21ksoftirqd/123:25:191
28665993125,4cyclictest25-21ksoftirqd/120:01:131
28665993125,4cyclictest25-21ksoftirqd/100:17:251
28665993125,3cyclictest25-21ksoftirqd/120:10:001
28665993125,3cyclictest25-21ksoftirqd/119:22:201
28664993126,3cyclictest9-21ksoftirqd/023:40:410
28664993126,3cyclictest9-21ksoftirqd/022:58:470
28664993126,3cyclictest9-21ksoftirqd/021:54:430
28664993126,3cyclictest9-21ksoftirqd/021:22:590
28664993126,3cyclictest9-21ksoftirqd/020:00:360
28664993126,3cyclictest9-21ksoftirqd/000:37:330
28664993125,4cyclictest9-21ksoftirqd/023:25:440
28664993125,4cyclictest9-21ksoftirqd/023:03:170
28664993125,4cyclictest9-21ksoftirqd/019:48:040
28664993125,3cyclictest9-21ksoftirqd/022:07:410
2866899304,15cyclictest0-21swapper/319:10:393
28668993024,4cyclictest41-21ksoftirqd/321:52:323
2866699307,22cyclictest8279-21kworker/2:120:41:132
28666993026,3cyclictest33-21ksoftirqd/223:56:012
28666993024,4cyclictest33-21ksoftirqd/221:50:582
28666993024,4cyclictest33-21ksoftirqd/221:16:442
28665993026,3cyclictest25-21ksoftirqd/122:30:221
28665993025,3cyclictest25-21ksoftirqd/122:22:111
28665993024,4cyclictest25-21ksoftirqd/123:30:431
28665993024,4cyclictest25-21ksoftirqd/123:30:431
28665993024,4cyclictest25-21ksoftirqd/122:03:251
28665993024,3cyclictest25-21ksoftirqd/121:10:011
28664993025,3cyclictest9-21ksoftirqd/022:24:530
28664993025,3cyclictest9-21ksoftirqd/020:41:570
28664993025,3cyclictest9-21ksoftirqd/020:27:160
28664993025,3cyclictest9-21ksoftirqd/000:25:120
28664993024,4cyclictest9-21ksoftirqd/023:51:040
28664993024,4cyclictest9-21ksoftirqd/021:17:430
28664993024,4cyclictest9-21ksoftirqd/020:15:320
28664993024,4cyclictest9-21ksoftirqd/019:12:140
28668992922,4cyclictest41-21ksoftirqd/321:00:043
2866699295,2cyclictest121rcu_preempt00:06:072
28666992925,3cyclictest33-21ksoftirqd/219:48:512
28666992924,3cyclictest33-21ksoftirqd/223:44:162
28666992924,3cyclictest33-21ksoftirqd/222:01:022
28665992923,4cyclictest25-21ksoftirqd/123:13:221
28665992923,4cyclictest25-21ksoftirqd/120:41:531
28665992923,3cyclictest25-21ksoftirqd/121:15:001
28664992924,3cyclictest9-21ksoftirqd/023:14:210
28664992924,3cyclictest9-21ksoftirqd/020:06:260
28664992924,3cyclictest9-21ksoftirqd/000:12:180
28664992923,4cyclictest9-21ksoftirqd/023:20:350
28664992923,4cyclictest9-21ksoftirqd/022:27:030
28664992923,4cyclictest9-21ksoftirqd/019:41:270
28664992923,3cyclictest9-21ksoftirqd/020:36:230
28668992823,3cyclictest41-21ksoftirqd/319:25:003
28666992824,3cyclictest33-21ksoftirqd/220:52:512
28666992823,3cyclictest33-21ksoftirqd/222:57:222
28666992823,3cyclictest33-21ksoftirqd/222:10:032
28665992823,3cyclictest25-21ksoftirqd/100:25:001
28665992822,3cyclictest25-21ksoftirqd/123:38:231
28665992822,3cyclictest25-21ksoftirqd/122:25:021
28664992823,3cyclictest9-21ksoftirqd/023:55:060
28664992823,3cyclictest9-21ksoftirqd/021:27:090
28664992823,3cyclictest9-21ksoftirqd/020:50:590
28664992822,4cyclictest9-21ksoftirqd/021:41:580
28664992822,4cyclictest9-21ksoftirqd/019:54:540
28664992822,4cyclictest9-21ksoftirqd/019:34:280
59298270,8rtkit-daemon0-21swapper/122:07:211
2866899275,7cyclictest2165-21ls21:35:243
2866699274,3cyclictest10829-21kworker/2:200:16:062
28666992722,3cyclictest33-21ksoftirqd/222:26:392
28666992721,4cyclictest33-21ksoftirqd/221:26:432
28665992722,3cyclictest25-21ksoftirqd/120:24:561
28664992722,3cyclictest9-21ksoftirqd/022:10:420
28664992722,3cyclictest9-21ksoftirqd/020:10:440
28664992722,3cyclictest9-21ksoftirqd/019:20:300
28664992721,4cyclictest9-21ksoftirqd/023:45:100
28668992621,3cyclictest41-21ksoftirqd/321:55:333
28668992621,3cyclictest41-21ksoftirqd/319:41:473
2866699266,18cyclictest2909-21kworker/2:023:45:112
28666992621,3cyclictest33-21ksoftirqd/222:41:172
28666992620,4cyclictest33-21ksoftirqd/200:01:212
28665992620,4cyclictest25-21ksoftirqd/120:19:181
2866899259,13cyclictest16925-21sendmail_mailst22:05:243
28668992519,4cyclictest41-21ksoftirqd/321:34:233
28666992519,4cyclictest33-21ksoftirqd/200:32:462
28666992519,4cyclictest33-21ksoftirqd/200:32:462
28664992520,3cyclictest9-21ksoftirqd/020:50:000
28664992519,3cyclictest9-21ksoftirqd/000:04:200
28664992518,5cyclictest9-21ksoftirqd/021:04:550
28664992419,3cyclictest9-21ksoftirqd/022:18:040
28664992419,3cyclictest9-21ksoftirqd/020:22:220
28666992317,4cyclictest33-21ksoftirqd/222:21:342
28664992318,3cyclictest9-21ksoftirqd/021:30:280
2866699221,19cyclictest0-21swapper/219:54:062
28666992116,3cyclictest33-21ksoftirqd/222:37:162
28664992117,3cyclictest9-21ksoftirqd/000:20:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional