You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-29 - 03:46
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Sat Nov 29, 2025 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31259991039,81cyclictest24379-21dpkg23:29:241
31259991032,37cyclictest355-21runrttasks22:49:091
815921020,9sleep20-21swapper/219:34:282
815921020,9sleep20-21swapper/219:34:282
29794210114,41sleep10-21swapper/119:04:161
31259999819,32cyclictest1021-21load21:54:211
31259999819,32cyclictest1021-21load21:54:211
31259999734,8cyclictest0-21swapper/120:09:321
31259999734,8cyclictest0-21swapper/120:09:321
31259999634,14cyclictest0-21swapper/100:24:091
31259999634,14cyclictest0-21swapper/100:24:091
31259999537,10cyclictest0-21swapper/122:44:161
3125999944,35cyclictest27180-21/usr/sbin/munin23:34:251
3125999944,35cyclictest27180-21/usr/sbin/munin23:34:251
3125999943,34cyclictest30314-21/usr/sbin/munin20:44:081
31259999431,8cyclictest0-21swapper/122:59:331
31259999430,13cyclictest0-21swapper/119:14:321
3125999942,35cyclictest27805-21grep20:34:341
3125999933,35cyclictest12205-21/usr/sbin/munin00:04:321
3125999933,33cyclictest25695-21rm21:40:481
3125999933,33cyclictest25695-21rm21:40:481
3125999933,32cyclictest7419-21unixbench_multi23:54:321
31259999329,9cyclictest0-21swapper/122:39:321
31259999329,9cyclictest0-21swapper/122:39:321
31259999325,17cyclictest0-21swapper/121:38:581
3125999932,35cyclictest9288-21grep19:39:151
3125999924,41cyclictest30112-21/usr/sbin/munin23:39:291
31259999239,45cyclictest0-21swapper/119:59:121
3125999923,42cyclictest24031-21latency20:24:211
3125999923,36cyclictest21596-21fschecks_count22:29:171
3125999923,34cyclictest13259-21unixbench_multi19:49:381
3125999923,33cyclictest7658-21/usr/sbin/munin22:04:461
3125999923,33cyclictest15526-21wc21:24:281
31259999230,13cyclictest0-21swapper/122:13:581
31259999229,15cyclictest0-21swapper/122:19:101
31259999229,15cyclictest0-21swapper/122:19:101
3125999922,33cyclictest12177-21fschecks_count21:19:171
31259999215,33cyclictest29524-21cut00:34:201
31259999146,37cyclictest18242-21cpuspeed21:29:351
3125999913,36cyclictest18344-21cstates23:19:141
3125999913,36cyclictest18344-21cstates23:19:141
3125999913,35cyclictest25509-21grep20:29:201
3125999913,35cyclictest21677-21mailstats00:19:311
3125999913,32cyclictest5694-21latency_hist19:28:581
3125999913,30cyclictest18302-21grep00:14:181
3125999912,39cyclictest13584-21grep22:14:371
3125999912,39cyclictest13584-21grep22:14:371
3125999912,36cyclictest891-21latency_hist20:53:581
3125999912,34cyclictest4564-21fschecks_count21:04:171
3125999912,34cyclictest318-21/usr/sbin/munin19:13:561
31257999166,16cyclictest5811-21latency_hist19:28:590
3125999903,32cyclictest6494-21latency_hist22:03:591
3125999903,32cyclictest6494-21latency_hist22:03:591
3125999903,32cyclictest15739-21seq00:09:251
3125999903,32cyclictest12127-21latency_hist19:48:581
3125999903,31cyclictest17163-21/usr/sbin/munin20:04:281
3125999902,38cyclictest6930-21timerandwakeup21:09:311
3125999894,37cyclictest8937-21/usr/sbin/munin21:14:161
3125999894,37cyclictest8937-21/usr/sbin/munin21:14:161
3125999894,33cyclictest355-21runrttasks23:09:521
3125999894,33cyclictest355-21runrttasks23:09:521
3125999893,40cyclictest24543-21cut22:34:181
3125999893,38cyclictest21579-21latency23:24:221
3125999893,38cyclictest21579-21latency23:24:221
3125999893,34cyclictest4112-21ssh22:54:251
3125999893,34cyclictest4112-21ssh22:54:251
3125999893,33cyclictest8940-21latency_hist19:38:581
3125999893,33cyclictest8940-21latency_hist19:38:581
3125999893,32cyclictest411-21latency_hist21:53:591
3125999893,31cyclictest3007-21sort20:59:181
3125999893,31cyclictest16224-21timerwakeupswit23:14:311
3125999893,30cyclictest22214-21uniq20:19:191
31259998925,11cyclictest0-21swapper/123:53:591
31259998925,11cyclictest0-21swapper/123:53:591
3125999892,37cyclictest1365-21fschecks_time20:54:181
3125999892,37cyclictest1365-21fschecks_time20:54:181
3125999892,33cyclictest14573-21mailstats19:54:281
3125999892,32cyclictest27543-21cat21:44:211
3125999892,32cyclictest10334-21sed23:04:311
31264998839,13cyclictest181rcu_preempt23:38:572
31264998839,13cyclictest181rcu_preempt23:38:572
3126499883,45cyclictest29860-21apt-config20:43:572
3125999884,31cyclictest28467-21/usr/sbin/munin20:39:171
3125999883,32cyclictest9762-21latency23:59:211
3125999882,31cyclictest18393-21/usr/sbin/munin22:24:321
3125999882,31cyclictest18393-21/usr/sbin/munin22:24:321
31264998764,15cyclictest207-21systemd-journal22:42:122
31264998764,15cyclictest207-21systemd-journal22:42:122
3126499874,62cyclictest24276-21apt-get23:29:262
3125999873,33cyclictest28202-21ssh00:31:541
3125999872,36cyclictest3385-21switchtime19:19:311
3125999872,33cyclictest7212-21modprobe19:33:551
3125999872,33cyclictest7212-21modprobe19:33:551
31257998761,20cyclictest7405-21sh22:04:430
3125999863,35cyclictest21282-21unixbench_multi20:14:331
3125999863,32cyclictest355-21runrttasks23:46:431
31264998565,14cyclictest6327-21latency_hist22:58:582
31264998565,14cyclictest6327-21latency_hist22:58:582
31264998561,8cyclictest181rcu_preempt22:25:532
31264998561,8cyclictest181rcu_preempt22:25:532
3125799858,31cyclictest27094-21apt-get21:44:070
31257998460,15cyclictest27102-21diskmemload23:33:580
31257998361,15cyclictest5708-21diskmemload23:44:190
31257998316,19cyclictest29232-21awk00:34:120
31264998264,12cyclictest31243-21cyclictest21:44:332
31264998234,17cyclictest181rcu_preempt23:14:052
31257998259,14cyclictest864-21ssh21:54:190
31257998259,14cyclictest864-21ssh21:54:190
3125799824,39cyclictest12532-21fschecks_count19:49:190
3109928210,46sleep0691-21cron19:08:560
307732829,36sleep20-21swapper/219:04:462
31264998158,15cyclictest207-21systemd-journal23:05:122
31264998128,11cyclictest0-21swapper/223:58:262
3125799818,29cyclictest21370-21ssh22:29:070
3125799813,41cyclictest355-21runrttasks21:18:420
3125799813,41cyclictest355-21runrttasks21:18:420
31257998126,18cyclictest15645-21if_eth023:14:180
3125799811,57cyclictest27630-21/usr/sbin/munin22:39:190
3125799811,57cyclictest27630-21/usr/sbin/munin22:39:190
31264998042,12cyclictest0-21swapper/221:57:112
31264998042,12cyclictest0-21swapper/221:57:112
31264998041,12cyclictest24228-21kworker/u18:1-nfsiod22:49:222
31257998056,13cyclictest5708-21diskmemload22:59:320
31264997937,11cyclictest181rcu_preempt21:14:132
31264997937,11cyclictest181rcu_preempt21:14:132
31257997921,36cyclictest0-21swapper/020:04:230
31264997855,17cyclictest15388-21fschecks_count00:09:182
3126499783,61cyclictest18138-21ssh21:29:242
31264997821,32cyclictest231-21systemd-udevd00:21:002
31257997859,13cyclictest5708-21diskmemload23:04:310
31257997849,22cyclictest0-21swapper/022:00:470
31257997849,22cyclictest0-21swapper/022:00:470
31257997833,37cyclictest0-21swapper/022:54:080
31257997833,37cyclictest0-21swapper/022:54:080
31257997812,26cyclictest0-21swapper/020:39:100
31264997751,17cyclictest28484-21kworker/u17:3-writeback21:53:592
31264997740,13cyclictest0-21swapper/222:21:522
31264997740,13cyclictest0-21swapper/222:21:522
31264997731,17cyclictest181rcu_preempt19:29:062
31264997731,17cyclictest181rcu_preempt19:29:062
3125799779,26cyclictest15263-21irqstats21:24:200
31264997656,13cyclictest2376-21latency_hist19:18:592
31264997655,8cyclictest16397-21kworker/u17:0+xprtiod00:34:092
3126499765,50cyclictest3990-21ssh21:59:532
3126499765,50cyclictest3990-21ssh21:59:532
31257997652,16cyclictest207-21systemd-journal00:32:460
3125799764,58cyclictest19094-21ssh23:19:320
3125799764,58cyclictest19094-21ssh23:19:320
31257997632,20cyclictest0-21swapper/021:20:590
31257997628,30cyclictest2695-21ssh22:52:200
31257997625,32cyclictest0-21swapper/000:04:300
31257997620,16cyclictest16181-21/usr/sbin/munin22:19:320
31257997620,16cyclictest16181-21/usr/sbin/munin22:19:320
31257997615,27cyclictest19002-21switchtime21:29:540
31264997560,11cyclictest31243-21cyclictest21:38:582
31264997551,15cyclictest27152-21latency_hist22:38:592
31264997551,14cyclictest24206-21sh23:28:592
31264997539,11cyclictest181rcu_preempt22:18:212
31264997539,11cyclictest181rcu_preempt22:18:212
31264997537,13cyclictest181rcu_preempt00:04:232
3126499752,37cyclictest39-21ksoftirqd/219:09:122
31264997523,14cyclictest0-21swapper/221:03:592
31257997557,11cyclictest9046-21systemd-run23:58:560
31257997557,11cyclictest14929-21ssh23:13:540
31257997557,11cyclictest14929-21ssh23:13:540
31257997547,18cyclictest12199-21latency_hist19:48:580
3125799754,39cyclictest9923-21ntpq23:59:250
31257997526,8cyclictest161ktimers/019:41:230
31257997518,24cyclictest355-21runrttasks23:26:270
31257997518,24cyclictest355-21runrttasks23:26:270
31257997516,21cyclictest26967-21true22:38:560
31257997510,23cyclictest0-21swapper/021:49:080
31264997459,9cyclictest16397-21kworker/u17:0-nfsiod23:28:562
31264997454,15cyclictest31243-21cyclictest20:28:572
31264997453,15cyclictest25046-21sh21:40:122
31264997453,15cyclictest25046-21sh21:40:122
31264997434,19cyclictest9562-21fschecks_count23:59:172
31264997430,14cyclictest0-21swapper/220:04:112
3125799747,22cyclictest15-21ksoftirqd/020:09:270
3125799747,22cyclictest15-21ksoftirqd/020:09:270
31257997454,13cyclictest691-21cron22:13:560
3125799745,36cyclictest31934-21/usr/sbin/munin20:49:230
31257997445,20cyclictest21929-21latency_hist20:18:580
31257997440,23cyclictest4584-21grep21:04:170
31257997422,12cyclictest15356-21ssh00:09:180
31264997348,7cyclictest181rcu_preempt20:54:292
31264997348,7cyclictest181rcu_preempt20:54:292
31264997327,9cyclictest0-21swapper/220:34:312
31257997348,19cyclictest339-21sh22:48:550
31257997322,11cyclictest1759-21grep20:54:290
31257997322,11cyclictest1759-21grep20:54:290
31257997312,35cyclictest18393-21/usr/sbin/munin22:24:340
31257997312,35cyclictest18393-21/usr/sbin/munin22:24:340
31264997255,11cyclictest355-21runrttasks23:48:562
31264997255,10cyclictest0-21swapper/222:09:332
31264997244,7cyclictest181rcu_preempt20:49:322
31264997235,12cyclictest0-21swapper/221:23:592
3126499722,35cyclictest39-21ksoftirqd/221:09:092
31257997252,14cyclictest13407-21threads22:14:350
31257997252,14cyclictest13407-21threads22:14:350
31257997251,12cyclictest22281-21ssh21:35:100
31257997213,40cyclictest0-21swapper/020:29:090
31257997213,23cyclictest24171-21apt-get21:39:120
31257997213,23cyclictest24171-21apt-get21:39:120
31257997210,25cyclictest770-21ntp_kernel_pll_19:14:040
3055927215,39sleep30-21swapper/319:04:433
31264997155,11cyclictest31243-21cyclictest23:51:472
31264997155,11cyclictest31243-21cyclictest23:51:472
31264997152,13cyclictest25820-21proc_pri20:29:302
31264997152,12cyclictest292-21dbus-daemon00:33:562
31264997144,17cyclictest28464-21kworker/u17:2+ext4-rsv-conversion20:44:102
31264997126,16cyclictest181rcu_preempt22:59:492
3125799713,38cyclictest355-21runrttasks23:39:070
3125799713,38cyclictest355-21runrttasks00:22:330
31257997116,14cyclictest14618-21taskset19:54:290
31264997053,10cyclictest4008-21cron19:23:562
31264997048,8cyclictest181rcu_preempt19:43:582
31264997046,15cyclictest17644-21apt-config21:28:582
31264997032,10cyclictest0-21swapper/220:14:192
31264997031,13cyclictest0-21swapper/223:43:412
3126499702,43cyclictest39-21ksoftirqd/223:19:142
3126499702,43cyclictest39-21ksoftirqd/223:19:142
3126499702,34cyclictest39-21ksoftirqd/221:04:102
3125799704,25cyclictest3326-21grep20:59:280
3125799703,31cyclictest24598-21head20:24:330
31257997021,12cyclictest28108-21cat23:34:320
31257997021,12cyclictest28108-21cat23:34:320
31264996952,12cyclictest10739-21cpu19:44:132
31264996948,16cyclictest354-21kworker/u18:0+events_unbound19:53:552
31264996938,10cyclictest181rcu_preempt22:05:382
3125799698,20cyclictest29829-21apt-get20:44:080
3125799693,52cyclictest355-21runrttasks23:51:390
3125799693,52cyclictest355-21runrttasks23:51:390
3125799693,46cyclictest2647-21/usr/sbin/munin19:19:170
31257996916,17cyclictest8146-21tr19:34:280
31257996916,17cyclictest8146-21tr19:34:280
31257996916,13cyclictest6293-21/usr/sbin/munin21:09:170
31257996914,18cyclictest16969-21sed20:03:580
31264996856,9cyclictest31243-21cyclictest20:19:392
3126499683,51cyclictest18705-21/usr/sbin/munin00:14:282
31264996834,8cyclictest0-21swapper/219:24:182
3126499683,36cyclictest14195-21fschecks_time19:54:182
31264996831,8cyclictest181rcu_preempt19:58:592
31257996848,13cyclictest23761-21cstates00:24:140
31257996848,13cyclictest23761-21cstates00:24:140
31257996817,12cyclictest32158-21taskset19:09:320
31264996738,20cyclictest30603-21cut22:44:202
31264996732,13cyclictest0-21swapper/200:24:322
31264996732,13cyclictest0-21swapper/200:24:322
3125799675,31cyclictest18433-21/usr/sbin/munin00:14:200
31257996747,12cyclictest28451-21latency_hist20:38:590
3125799674,42cyclictest5859-21/usr/sbin/munin19:29:170
3125799674,42cyclictest5859-21/usr/sbin/munin19:29:170
31257996710,24cyclictest22588-21/usr/sbin/munin20:19:300
60062660,7sleep30-21swapper/322:03:013
60062660,7sleep30-21swapper/322:03:013
31264996630,8cyclictest0-21swapper/220:09:282
31264996630,8cyclictest0-21swapper/220:09:282
31264996627,16cyclictest181rcu_preempt23:09:582
31264996627,16cyclictest181rcu_preempt23:09:582
31264996423,14cyclictest0-21swapper/222:29:012
216052640,5sleep30-21swapper/323:24:223
216052640,5sleep30-21swapper/323:24:223
144582600,7sleep30-21swapper/300:07:543
3127199593,41cyclictest29439-21taskset21:48:103
3127199584,36cyclictest18403-21chrt23:19:153
3127199584,36cyclictest18403-21chrt23:19:153
3127199572,32cyclictest0-21swapper/323:58:573
3127199564,43cyclictest756-21taskset22:49:133
3127199564,37cyclictest16803-21taskset22:20:123
3127199564,37cyclictest16803-21taskset22:20:123
3127199563,38cyclictest3415-21taskset19:19:313
3127199562,33cyclictest0-21swapper/321:29:203
3127199554,36cyclictest21806-21chrt00:19:333
3127199543,38cyclictest3538-21chrt21:59:003
3127199543,38cyclictest3538-21chrt21:59:003
3127199543,38cyclictest10006-21chrt21:14:333
3127199543,38cyclictest10006-21chrt21:14:333
3127199543,33cyclictest26745-21chrt20:33:593
3127199533,41cyclictest29979-21chrt00:34:323
3127199532,41cyclictest19862-21taskset22:25:303
3127199532,41cyclictest19862-21taskset22:25:303
3127199523,37cyclictest30103-21chrt20:43:593
3127199523,35cyclictest32090-21taskset19:09:313
3127199522,35cyclictest30834-21chrt21:49:313
3127199514,33cyclictest14537-21chrt22:17:123
3127199514,33cyclictest14537-21chrt22:17:123
3127199513,34cyclictest24725-21chrt00:24:353
3127199513,34cyclictest24725-21chrt00:24:353
3127199512,35cyclictest2437-21taskset23:46:473
3127199504,37cyclictest22559-21chrt21:35:573
3127199503,36cyclictest12373-21chrt23:09:103
3127199503,36cyclictest12373-21chrt23:09:103
3127199503,34cyclictest9945-21chrt23:59:273
3127199503,34cyclictest30244-21chrt23:39:133
3127199503,34cyclictest29028-21chrt23:36:423
3127199503,34cyclictest29028-21chrt23:36:423
3127199503,33cyclictest15952-21chrt19:59:203
3127199503,33cyclictest15276-21chrt19:58:553
3127199502,39cyclictest17215-21chrt23:17:003
3127199502,35cyclictest7218-21chrt22:59:303
3127199502,35cyclictest5570-21taskset23:52:263
3127199502,35cyclictest5570-21taskset23:52:263
3127199502,35cyclictest12596-21chrt19:49:203
3127199502,32cyclictest9863-21chrt19:39:283
3127199494,33cyclictest12174-21chrt23:08:583
3127199493,37cyclictest1978-21chrt20:54:323
3127199493,37cyclictest1978-21chrt20:54:323
3127199493,35cyclictest28847-21chrt00:33:553
3127199493,33cyclictest3006-21chrt20:59:183
3127199493,33cyclictest24354-21chrt23:29:203
3127199493,31cyclictest20584-21chrt20:14:183
3127199493,31cyclictest18353-21chrt00:14:183
3127199493,31cyclictest14405-21chrt21:23:333
3127199492,36cyclictest30541-21taskset22:44:193
3127199492,33cyclictest2476-21chrt19:18:583
3127199482,36cyclictest667-21chrt20:51:533
3127199482,32cyclictest3866-21taskset22:54:193
3127199482,32cyclictest3866-21taskset22:54:193
3127199474,33cyclictest21372-21taskset22:29:073
3127199473,31cyclictest29023-21chrt22:41:313
3127199473,31cyclictest29023-21chrt22:41:313
3127199473,31cyclictest23146-21chrt20:21:113
3127199473,31cyclictest19292-21taskset20:09:243
3127199473,31cyclictest19292-21taskset20:09:243
3127199472,34cyclictest9861-21chrt22:09:263
3127199472,31cyclictest4987-21chrt19:24:293
3127199463,31cyclictest25693-21taskset21:40:483
3127199463,31cyclictest25693-21taskset21:40:483
3127199453,29cyclictest15167-21chrt21:24:183
3127199452,30cyclictest6023-21taskset21:09:063
3127199444,31cyclictest30869-21chrt20:44:523
3127199442,29cyclictest8085-21chrt22:05:333
3127199432,29cyclictest24736-21chrt22:34:223
3127199432,26cyclictest4981-21chrt21:04:283
3127199432,26cyclictest26975-21chrt20:34:153
3127199423,27cyclictest17905-21taskset00:13:583
3127199413,28cyclictest6747-21chrt19:29:313
3127199413,28cyclictest6747-21chrt19:29:313
3127199402,29cyclictest18511-21chrt20:08:563
3127199402,24cyclictest9075-21taskset19:38:593
3127199393,26cyclictest7930-21chrt19:34:213
3127199392,24cyclictest10701-21taskset19:44:013
3127199382,27cyclictest23648-21chrt20:24:113
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional