You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 00:18
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 792 highest latencies:
System rackdslot2.osadl.org (updated Wed Nov 26, 2025 12:45:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
109622218171,37sleep20-21swapper/207:05:592
109632207173,23sleep30-21swapper/307:06:003
112572206172,23sleep00-21swapper/007:09:430
110162205171,23sleep10-21swapper/107:06:411
1157799670,30cyclictest30332-21irqstats08:55:213
1157799631,24cyclictest29507-21irqstats12:15:203
11576996310,12cyclictest20433-21sed07:25:252
1157799620,23cyclictest32147-21ptp4l07:50:233
11576996210,11cyclictest2179-21awk07:55:252
201352600,4sleep020137-21sendmail_mailst07:25:230
11576995911,11cyclictest15681-21sed07:15:292
11576995910,11cyclictest1236-21unixbench_multi09:00:252
1157699588,18cyclictest6604-21cat09:13:382
1157699586,17cyclictest29408-21irqstats07:45:202
1157699586,16cyclictest31987-21/usr/sbin/munin10:05:192
1157699579,17cyclictest12278-21timedrift10:30:272
1157699579,15cyclictest32276-21awk07:50:252
1157699578,18cyclictest25063-21awk12:05:252
1157699578,17cyclictest15553-21sensors11:45:242
1157699577,19cyclictest0-21swapper/208:35:042
1157699577,19cyclictest0-21swapper/208:35:042
1157799561,24cyclictest24023-21taskset07:35:133
1157699569,10cyclictest24069-21awk10:55:242
1157699567,17cyclictest22743-21mailstats07:30:252
1157699557,18cyclictest29481-21taskset11:09:092
1157499558,9cyclictest8137-21cut08:10:120
1157699547,11cyclictest26237-21cpuspeed_turbos07:40:132
1157799530,31cyclictest10912-21grep09:20:263
1157699527,10cyclictest54712sleep212:34:542
1157699527,10cyclictest54712sleep212:34:542
11576995210,18cyclictest14433-21ps10:35:232
11576995210,18cyclictest14433-21ps10:35:232
1157699518,10cyclictest3284-21taskset10:13:472
1157699508,17cyclictest26742-21chrt08:49:252
1157699508,16cyclictest18537-21taskset11:53:592
1157699506,11cyclictest102022sleep210:26:052
11576995010,17cyclictest9149-21netstat08:10:212
1157499505,10cyclictest14356-21timerandwakeup08:20:280
1157699499,18cyclictest14796-21chrt08:23:222
1157699498,16cyclictest19603-21chrt08:34:162
1157699496,11cyclictest78672chrt10:21:312
11576994911,17cyclictest32127-21netstat12:20:232
1157699488,11cyclictest19087-21cat11:55:142
1157699487,11cyclictest23661-21sendmail_mailst08:40:272
1157699476,15cyclictest29793-21/usr/sbin/munin11:10:122
1157699476,11cyclictest22148-21cat09:45:192
1157699468,18cyclictest11222-21chrt11:37:172
1157699468,17cyclictest31327-21taskset08:57:352
1157699468,17cyclictest25909-21chrt09:53:352
1157699467,11cyclictest18531-21chrt09:37:592
136591450,35ptp4l0-21swapper/307:10:013
136591440,0ptp4l401ktimersoftd/309:56:023
1157699449,12cyclictest26404-21grep11:00:232
1157699447,11cyclictest27796-21cat09:55:262
1157799430,28cyclictest11758-21awk08:15:243
1157699437,19cyclictest13628-21chrt09:26:312
51698420,9rtkit-daemon0-21swapper/109:05:011
51698420,9rtkit-daemon0-21swapper/109:05:001
136591420,0ptp4l401ktimersoftd/311:16:543
136591420,0ptp4l401ktimersoftd/311:00:093
136591420,0ptp4l401ktimersoftd/310:38:423
136591420,0ptp4l401ktimersoftd/310:38:423
1157699424,16cyclictest11593-21/usr/sbin/munin07:10:142
1157699419,15cyclictest7063-21gzip12:35:232
1157699417,17cyclictest1780-21taskset11:18:402
1157799400,20cyclictest0-21swapper/310:35:003
1157699396,8cyclictest29972-21systemctl12:15:252
1157699392,14cyclictest0-21swapper/210:16:552
1157699377,18cyclictest0-21swapper/208:19:572
11577993613,20cyclictest27617-21tr11:05:123
1157799361,18cyclictest0-21swapper/308:15:013
1157799360,34cyclictest18544-21latency_hist07:25:013
1157699362,18cyclictest0-21swapper/210:50:122
11577993514,18cyclictest30964-21/usr/sbin/munin12:20:133
11577993512,20cyclictest18923-21sed09:40:003
1157799350,1cyclictest0-21swapper/308:00:213
1157699357,7cyclictest28451-21python08:50:252
1157699357,10cyclictest7809-21irqrtprio11:30:172
1157699356,19cyclictest6389-21runrttasks11:26:022
11576993510,11cyclictest15828-21grep09:30:322
11574993524,6cyclictest9-21ksoftirqd/011:55:140
1157499351,2cyclictest121rcu_preempt12:03:560
1157799349,22cyclictest16158-21sh07:20:013
1157799343,26cyclictest16514-21apt-get11:50:143
1157799342,25cyclictest6031-21apt-get10:20:133
11577993412,19cyclictest16705-21grep10:40:233
11577993411,20cyclictest8667-21expr10:25:143
11577993410,21cyclictest19817-21wc08:35:013
1157799340,31cyclictest9973-21/usr/sbin/munin11:35:183
1157699347,16cyclictest8122-21ls09:15:222
1157699347,16cyclictest21793-21cat12:00:162
1157699343,19cyclictest0-21swapper/209:07:572
1157699343,19cyclictest0-21swapper/209:07:572
11575993429,3cyclictest25-21ksoftirqd/109:50:201
11575993428,4cyclictest25-21ksoftirqd/107:50:071
136591330,0ptp4l401ktimersoftd/309:06:113
136591330,0ptp4l401ktimersoftd/309:06:113
1157799339,21cyclictest7749-21cron08:10:013
1157799339,21cyclictest7749-21cron08:10:013
11577993326,4cyclictest41-21ksoftirqd/310:05:073
11577993313,18cyclictest5769-21latency_hist10:20:013
11577993313,18cyclictest5718-21/usr/sbin/munin12:35:263
1157799331,29cyclictest26800-21latency_hist08:50:003
11577993310,20cyclictest27042-21irqstats07:40:203
1157799330,30cyclictest32533-21grep09:00:163
1157699337,16cyclictest26843-21cat12:10:182
1157699333,17cyclictest0-21swapper/210:45:212
11575993327,4cyclictest25-21ksoftirqd/112:06:541
11575993327,3cyclictest25-21ksoftirqd/112:20:191
11574993327,4cyclictest9-21ksoftirqd/009:04:310
1157799329,20cyclictest28718-21awk08:50:273
1157799329,19cyclictest22927-21unixbench_multi12:00:273
1157799328,21cyclictest17250-21sshd08:29:223
1157799327,22cyclictest8072-21needreboot11:30:203
11577993222,7cyclictest41-21ksoftirqd/310:54:593
1157799321,28cyclictest6103-21grep09:10:283
11577993210,19cyclictest12752-21/usr/sbin/munin08:20:253
1157799320,20cyclictest16432-21cat09:35:003
1157699327,22cyclictest172132sleep208:28:592
1157699324,17cyclictest0-21swapper/208:00:282
11576993226,4cyclictest33-21ksoftirqd/209:20:412
1157699322,18cyclictest0-21swapper/211:20:202
11575993227,3cyclictest25-21ksoftirqd/110:55:141
11575993227,3cyclictest25-21ksoftirqd/109:40:201
11575993227,3cyclictest25-21ksoftirqd/109:00:191
11575993227,3cyclictest25-21ksoftirqd/107:30:591
11575993227,3cyclictest25-21ksoftirqd/107:20:011
11575993226,4cyclictest25-21ksoftirqd/111:16:301
11575993226,4cyclictest25-21ksoftirqd/111:10:531
11575993226,4cyclictest25-21ksoftirqd/111:05:551
11575993226,4cyclictest25-21ksoftirqd/110:10:191
11575993226,4cyclictest25-21ksoftirqd/108:22:591
11575993226,3cyclictest25-21ksoftirqd/110:20:201
11575993225,5cyclictest25-21ksoftirqd/109:57:201
11574993226,4cyclictest9-21ksoftirqd/011:39:520
11574993226,4cyclictest9-21ksoftirqd/010:19:530
11574993226,4cyclictest9-21ksoftirqd/010:08:590
11574993226,4cyclictest9-21ksoftirqd/009:49:520
11574993226,4cyclictest9-21ksoftirqd/008:17:570
11574993226,4cyclictest9-21ksoftirqd/008:09:230
11574993226,4cyclictest9-21ksoftirqd/008:09:220
11574993226,4cyclictest9-21ksoftirqd/007:55:470
1157799313,24cyclictest923-21apt-get12:25:123
1157799313,22cyclictest14145-21apt-get11:45:123
11577993125,4cyclictest41-21ksoftirqd/307:48:243
1157799312,24cyclictest0-21swapper/312:30:123
1157799312,24cyclictest0-21swapper/312:30:113
1157799311,27cyclictest22026-1kworker/3:2H12:10:023
11577993112,17cyclictest21368-21apt07:30:113
1157799310,28cyclictest30263-21grep11:10:153
1157799310,21cyclictest4599-21egrep11:25:143
11576993125,4cyclictest33-21ksoftirqd/211:42:432
11575993127,3cyclictest25-21ksoftirqd/111:04:591
11575993127,3cyclictest25-21ksoftirqd/109:20:001
11575993126,3cyclictest25-21ksoftirqd/110:50:191
11575993126,3cyclictest25-21ksoftirqd/110:08:581
11575993126,3cyclictest25-21ksoftirqd/108:58:281
11575993126,3cyclictest25-21ksoftirqd/108:50:001
11575993125,4cyclictest25-21ksoftirqd/112:31:491
11575993125,4cyclictest25-21ksoftirqd/112:31:481
11575993125,4cyclictest25-21ksoftirqd/111:22:121
11575993125,4cyclictest25-21ksoftirqd/108:06:321
11575993125,4cyclictest25-21ksoftirqd/108:06:311
11575993125,3cyclictest25-21ksoftirqd/107:30:001
11574993126,3cyclictest9-21ksoftirqd/007:49:140
11574993125,4cyclictest9-21ksoftirqd/012:34:000
11574993125,4cyclictest9-21ksoftirqd/012:34:000
11574993125,4cyclictest9-21ksoftirqd/011:34:510
11574993125,4cyclictest9-21ksoftirqd/009:29:180
11577993024,4cyclictest41-21ksoftirqd/311:56:083
11577993024,4cyclictest41-21ksoftirqd/308:43:353
1157799302,24cyclictest786-21apt-get07:55:123
1157799302,24cyclictest19919-21apt-get08:35:123
1157799302,24cyclictest19919-21apt-get08:35:123
1157799302,23cyclictest2176-21apt-get11:20:123
1157799301,26cyclictest23214-21unixbench_multi09:45:303
1157799301,25cyclictest26217-21apt-get12:10:133
1157799300,28cyclictest0-21swapper/311:45:003
1157699303,17cyclictest0-21swapper/207:20:222
11576993024,4cyclictest33-21ksoftirqd/208:05:402
11576993024,4cyclictest33-21ksoftirqd/208:05:392
11575993026,3cyclictest25-21ksoftirqd/110:30:001
11575993026,3cyclictest25-21ksoftirqd/108:50:121
11575993026,2cyclictest25-21ksoftirqd/110:40:201
11575993025,3cyclictest25-21ksoftirqd/111:30:171
11575993025,3cyclictest25-21ksoftirqd/111:26:411
11575993025,3cyclictest25-21ksoftirqd/108:25:191
11575993025,3cyclictest25-21ksoftirqd/108:10:191
11575993024,4cyclictest25-21ksoftirqd/108:38:571
11575993024,4cyclictest25-21ksoftirqd/108:38:571
11575993024,4cyclictest25-21ksoftirqd/107:47:021
11575993024,4cyclictest25-21ksoftirqd/107:22:191
1157499305,11cyclictest2259-21gnome-shell11:40:250
11574993026,3cyclictest9-21ksoftirqd/008:48:480
11574993024,4cyclictest9-21ksoftirqd/012:07:280
11574993024,4cyclictest9-21ksoftirqd/011:52:240
11574993024,4cyclictest9-21ksoftirqd/009:33:310
11574993024,4cyclictest9-21ksoftirqd/009:20:350
11574993024,4cyclictest9-21ksoftirqd/008:58:250
1157799297,20cyclictest13914-21latency_hist09:29:593
1157799296,20cyclictest1403-21grep10:10:143
1157799295,21cyclictest18009-21sort10:45:143
11577992923,4cyclictest41-21ksoftirqd/310:02:193
1157799292,22cyclictest7028-21apt-get09:15:133
1157799291,25cyclictest23916-21apt-get09:50:143
1157799291,24cyclictest226-21systemd-journal09:45:003
1157799290,26cyclictest22672-21/usr/sbin/munin10:55:203
1157699296,7cyclictest2494-21grep12:25:252
1157699295,10cyclictest0-21swapper/210:44:352
1157699293,16cyclictest0-21swapper/207:37:382
11576992924,3cyclictest33-21ksoftirqd/209:40:562
11575992925,3cyclictest25-21ksoftirqd/110:45:191
11575992925,3cyclictest25-21ksoftirqd/110:15:191
11575992924,3cyclictest25-21ksoftirqd/111:52:321
11575992924,3cyclictest25-21ksoftirqd/107:40:141
11575992923,4cyclictest25-21ksoftirqd/109:34:531
11575992923,4cyclictest25-21ksoftirqd/107:11:561
11575992923,3cyclictest25-21ksoftirqd/109:30:001
11574992923,4cyclictest9-21ksoftirqd/010:44:090
11574992923,4cyclictest9-21ksoftirqd/008:01:170
11574992922,5cyclictest9-21ksoftirqd/012:22:220
1157799280,24cyclictest19026-21cat07:25:113
11576992823,3cyclictest33-21ksoftirqd/210:04:452
11575992824,2cyclictest25-21ksoftirqd/111:45:131
11575992823,3cyclictest25-21ksoftirqd/111:56:481
11575992823,3cyclictest25-21ksoftirqd/111:40:201
11575992823,3cyclictest25-21ksoftirqd/108:45:001
11575992823,3cyclictest25-21ksoftirqd/108:30:201
11575992823,3cyclictest25-21ksoftirqd/108:19:101
11575992822,4cyclictest25-21ksoftirqd/110:36:511
11575992822,4cyclictest25-21ksoftirqd/110:36:511
11575992822,4cyclictest25-21ksoftirqd/110:34:381
11575992822,4cyclictest25-21ksoftirqd/109:38:101
11575992822,3cyclictest25-21ksoftirqd/112:15:201
1157499286,9cyclictest23718-21/usr/sbin/munin07:35:260
11574992823,3cyclictest9-21ksoftirqd/011:50:000
11574992823,3cyclictest9-21ksoftirqd/008:39:480
11574992823,3cyclictest9-21ksoftirqd/008:39:480
11574992822,4cyclictest9-21ksoftirqd/011:16:450
11574992822,4cyclictest9-21ksoftirqd/011:06:340
11574992822,4cyclictest9-21ksoftirqd/010:48:130
11574992822,4cyclictest9-21ksoftirqd/009:59:110
11575992724,2cyclictest25-21ksoftirqd/112:00:181
11575992723,3cyclictest25-21ksoftirqd/107:35:131
11575992722,3cyclictest25-21ksoftirqd/110:01:291
11575992722,3cyclictest25-21ksoftirqd/109:10:201
11575992722,3cyclictest25-21ksoftirqd/108:00:481
11574992722,3cyclictest9-21ksoftirqd/011:12:560
11574992722,3cyclictest9-21ksoftirqd/010:11:590
11574992722,3cyclictest9-21ksoftirqd/009:51:080
11574992722,3cyclictest9-21ksoftirqd/008:51:160
11574992722,3cyclictest9-21ksoftirqd/007:41:240
11574992721,4cyclictest9-21ksoftirqd/011:01:290
11574992721,4cyclictest9-21ksoftirqd/009:40:460
11575992622,3cyclictest25-21ksoftirqd/112:25:181
11575992621,3cyclictest25-21ksoftirqd/111:37:511
11574992620,4cyclictest9-21ksoftirqd/012:16:030
11574992620,4cyclictest9-21ksoftirqd/010:24:300
11574992620,4cyclictest9-21ksoftirqd/007:21:080
11575992520,3cyclictest25-21ksoftirqd/109:45:081
11575992520,3cyclictest25-21ksoftirqd/109:20:111
11574992520,3cyclictest9-21ksoftirqd/010:36:430
11574992520,3cyclictest9-21ksoftirqd/010:36:420
11574992520,3cyclictest9-21ksoftirqd/010:26:400
11574992520,3cyclictest9-21ksoftirqd/009:39:220
11574992520,3cyclictest9-21ksoftirqd/008:30:000
11575992419,3cyclictest25-21ksoftirqd/112:10:371
11575992419,3cyclictest25-21ksoftirqd/107:55:271
1157499243,9cyclictest1307-21runrttasks08:32:230
11574992419,3cyclictest9-21ksoftirqd/010:55:510
11574992419,3cyclictest9-21ksoftirqd/010:31:520
11574992419,3cyclictest9-21ksoftirqd/007:12:030
11574992418,4cyclictest9-21ksoftirqd/012:13:340
11575992319,3cyclictest25-21ksoftirqd/112:35:181
11574992318,3cyclictest9-21ksoftirqd/011:26:020
11574992317,4cyclictest9-21ksoftirqd/012:25:580
11574992317,4cyclictest9-21ksoftirqd/009:05:080
11574992317,4cyclictest9-21ksoftirqd/009:05:070
11574992217,3cyclictest9-21ksoftirqd/007:17:560
11574992216,4cyclictest9-21ksoftirqd/012:38:050
11574992216,4cyclictest9175-21sh09:20:010
11574992116,3cyclictest9-21ksoftirqd/011:22:590
11574992115,4cyclictest9-21ksoftirqd/009:14:230
11574992115,4cyclictest9-21ksoftirqd/007:34:090
11574992015,3cyclictest9-21ksoftirqd/010:53:370
11574992014,4cyclictest9-21ksoftirqd/007:52:340
11574992010,7cyclictest29259-21awk10:00:170
1157499191,5cyclictest9-21ksoftirqd/008:40:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional