You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-29 - 23:38
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Sat Nov 29, 2025 12:44:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
20374214111,84sleep020418-21switchtime07:04:390
21173991213,91cyclictest23894-21apt-get09:59:082
211619911543,35cyclictest0-21swapper/010:54:090
211619910777,18cyclictest161ktimers/009:59:240
211619910437,28cyclictest6061-21/usr/sbin/munin10:24:230
211619910437,28cyclictest6061-21/usr/sbin/munin10:24:230
211619910341,26cyclictest1-21systemd09:13:550
211619910340,28cyclictest20365-21vmstat10:49:350
211619910340,28cyclictest20365-21vmstat10:49:350
211619910340,25cyclictest13765-21apt-get10:39:060
211619910334,44cyclictest24993-21/usr/sbin/munin07:19:320
211619910334,44cyclictest24993-21/usr/sbin/munin07:19:320
211619910329,29cyclictest22905-21latency_hist07:13:570
211619910238,31cyclictest28889-21ps12:04:290
211619910237,29cyclictest31644-21/usr/sbin/munin12:09:230
211619910237,29cyclictest31644-21/usr/sbin/munin12:09:230
1380821020,5sleep013807-21needreboot12:34:220
211619910139,27cyclictest5867-21ssh09:26:340
211619910138,28cyclictest5416-21/usr/sbin/munin12:19:340
211619910138,28cyclictest5416-21/usr/sbin/munin12:19:340
211619910137,22cyclictest27257-21grep10:04:310
211619910137,22cyclictest27257-21grep10:04:310
211619910132,28cyclictest8906-21/usr/sbin/munin08:09:220
211619910132,28cyclictest8906-21/usr/sbin/munin08:09:220
211619910128,30cyclictest11487-21latency_hist11:33:570
211619910128,30cyclictest11487-21latency_hist11:33:570
211619910041,28cyclictest8304-21apt-get10:29:090
211619910029,37cyclictest24340-21head07:14:330
21161999937,23cyclictest4753-21/usr/sbin/munin07:54:280
21161999933,27cyclictest31724-21ssh11:10:480
2117399977,41cyclictest7719-21ssh09:30:062
21161999737,24cyclictest12746-21ssh12:33:180
21161999737,24cyclictest12746-21ssh12:33:180
21161999737,23cyclictest8932-21ssh09:32:020
21161999734,29cyclictest30742-21rm10:10:180
21161999734,29cyclictest30742-21rm10:10:180
21161999732,27cyclictest207-21systemd-journal11:55:130
21161999732,26cyclictest12891-21ssh11:35:110
21161999725,51cyclictest0-21swapper/008:14:160
21161999636,23cyclictest26140-21ssh11:00:320
21161999636,23cyclictest26140-21ssh11:00:320
21161999635,23cyclictest11053-21taskset10:34:060
21161999635,23cyclictest11053-21taskset10:34:060
21161999631,29cyclictest0-21swapper/008:39:140
2117399953,33cyclictest26544-21latency_hist09:03:562
2116599959,66cyclictest7101-21apt-get09:29:121
21161999540,25cyclictest3686-21wc10:20:120
21161999536,23cyclictest9682-21rm12:27:580
21161999533,29cyclictest21442-21ssh09:54:270
21161999531,26cyclictest16660-21ssh11:43:250
21161999531,26cyclictest16660-21ssh11:43:250
21161999529,43cyclictest772-21cat10:14:430
21161999522,38cyclictest0-21swapper/009:18:570
21161999522,38cyclictest0-21swapper/009:18:570
21161999517,43cyclictest0-21swapper/009:44:110
21173999435,8cyclictest0-21swapper/212:24:352
2117399943,38cyclictest15125-21latency_hist09:43:552
21173999428,15cyclictest0-21swapper/212:29:212
21173999428,15cyclictest0-21swapper/212:29:212
2117399942,39cyclictest24021-21wc08:54:262
2117399942,39cyclictest24021-21wc08:54:262
21161999462,20cyclictest28224-21apt-get07:29:090
21161999438,22cyclictest2381-21taskset12:14:310
21161999435,47cyclictest30102-21latency_hist11:08:570
21161999435,47cyclictest30102-21latency_hist11:08:570
21161999434,25cyclictest16498-21ssh10:43:580
21161999434,18cyclictest3859-21ntp_states11:19:260
21161999433,31cyclictest8071-21taskset11:27:390
21161999433,25cyclictest14046-21ssh09:40:450
21161999432,24cyclictest11449-21ssh09:36:180
21161999432,24cyclictest11449-21ssh09:36:180
2117399933,40cyclictest3887-21latency_hist07:53:562
2117399933,39cyclictest22593-21ntp_offset10:54:232
2117399933,35cyclictest4994-21grep07:54:302
21173999332,11cyclictest0-21swapper/207:33:562
21173999325,9cyclictest0-21swapper/210:41:552
2117399932,37cyclictest25372-21netstat10:59:232
2117399932,37cyclictest25372-21netstat10:59:232
21173999317,12cyclictest17650jbd2/mmcblk2p2-809:49:222
21173999317,12cyclictest17650jbd2/mmcblk2p2-809:49:222
21161999334,21cyclictest19162-21ssh09:49:410
21161999334,21cyclictest19162-21ssh09:49:410
21161999331,27cyclictest17088-21ssh11:43:590
21161999319,36cyclictest0-21swapper/008:24:130
2117399923,39cyclictest25631-21sendmail_mailqu07:19:292
2117399923,39cyclictest25631-21sendmail_mailqu07:19:292
2117399923,34cyclictest1651-21cstates09:19:112
2117399923,30cyclictest28015-21latency_hist07:28:552
2117399923,30cyclictest0-21swapper/211:53:572
21173999226,11cyclictest0-21swapper/212:23:552
21173999226,11cyclictest0-21swapper/212:23:552
21161999233,23cyclictest21810-21taskset08:49:000
21161999233,23cyclictest21810-21taskset08:49:000
21173999141,44cyclictest0-21swapper/210:09:412
21173999141,44cyclictest0-21swapper/210:09:412
21173999141,41cyclictest0-21swapper/211:33:572
21173999141,41cyclictest0-21swapper/211:33:572
2117399913,37cyclictest19251-21/usr/sbin/munin10:49:332
2117399913,37cyclictest19251-21/usr/sbin/munin10:49:332
2117399913,37cyclictest15567-21fschecks_count08:29:152
2117399913,35cyclictest11155-21mailstats08:14:272
2117399913,34cyclictest27650-21runrttasks09:04:332
2117399913,34cyclictest13691-21latency_hist08:23:572
2117399913,32cyclictest22644-21/usr/sbin/munin11:54:302
21173999132,52cyclictest0-21swapper/207:11:442
21173999127,13cyclictest0-21swapper/209:13:572
21161999136,26cyclictest27494-21ssh12:02:430
21161999136,21cyclictest2134-21/usr/sbin/munin09:19:350
21161999135,21cyclictest7848-21taskset08:04:250
21161999135,21cyclictest7848-21taskset08:04:250
21173999035,10cyclictest0-21swapper/209:18:532
21173999035,10cyclictest0-21swapper/209:18:532
2117399903,39cyclictest22754-21grep08:49:312
2117399903,39cyclictest22754-21grep08:49:312
2117399903,30cyclictest19132-21latency_hist10:48:562
2117399902,37cyclictest14491-21sh11:39:132
2117399902,37cyclictest14491-21sh11:39:132
2117399902,29cyclictest11864-21switchtime10:34:302
2117399902,29cyclictest11864-21switchtime10:34:302
2116599903,53cyclictest31-21ksoftirqd/110:34:101
2116599903,53cyclictest31-21ksoftirqd/110:34:101
21161999064,16cyclictest2463-21apt-get07:49:110
21161999032,27cyclictest12851-21sensors_temp08:19:270
21161999032,22cyclictest20558-21cat08:44:180
21161999032,22cyclictest20558-21cat08:44:180
21161999030,44cyclictest0-21swapper/011:49:140
21161999030,22cyclictest0-21swapper/007:39:290
2117399896,73cyclictest2213-21apt-get10:19:362
2117399895,33cyclictest355-21runrttasks07:14:042
2117399894,33cyclictest9080-21switchtime10:29:292
21173998941,41cyclictest0-21swapper/208:09:252
21173998941,41cyclictest0-21swapper/208:09:252
2117399893,40cyclictest13600-21fschecks_time12:34:172
2117399893,39cyclictest27547-21rm11:04:092
2117399893,39cyclictest27547-21rm11:04:092
2117399893,38cyclictest8075-21latency_hist10:28:552
2117399893,38cyclictest8075-21latency_hist10:28:552
2117399893,37cyclictest8822-21latency_hist08:08:562
2117399893,37cyclictest8822-21latency_hist08:08:562
2117399893,36cyclictest1757-21fschecks_count12:14:172
2117399893,35cyclictest32700-21latency_hist11:13:552
2117399893,35cyclictest14037-21grep08:24:182
2117399893,34cyclictest29110-21modprobe10:08:532
2117399893,34cyclictest29110-21modprobe10:08:532
2117399893,33cyclictest6002-21/usr/sbin/munin11:24:132
2117399893,33cyclictest355-21runrttasks08:34:332
2117399893,32cyclictest31070-21/usr/sbin/munin12:09:132
2117399893,32cyclictest31070-21/usr/sbin/munin12:09:132
2117399893,31cyclictest11890-21fschecks_time11:34:172
2117399892,34cyclictest30864-21timerandwakeup07:34:332
2117399892,34cyclictest20492-21fschecks_time08:44:172
2117399892,34cyclictest20492-21fschecks_time08:44:172
2117399892,31cyclictest1209-21kernelversion07:44:192
21161998935,20cyclictest24790-21munin-run08:58:530
21161998935,20cyclictest24790-21munin-run08:58:530
21161998934,18cyclictest26768-21df07:24:130
21161998927,33cyclictest28025-21kworker/u17:209:08:530
21161998926,25cyclictest0-21swapper/011:14:140
2117399883,35cyclictest15574-21fschecks_count09:44:162
2117399883,34cyclictest17867-21grep11:44:302
2117399883,33cyclictest12343-21ssh09:38:562
2117399883,33cyclictest12343-21ssh09:38:562
2117399883,32cyclictest355-21runrttasks08:00:092
2117399883,31cyclictest32471-21cut10:14:232
2117399883,31cyclictest18600-21/usr/sbin/munin08:39:272
2117399882,31cyclictest32054-21kernelversion07:39:202
21173998734,46cyclictest0-21swapper/212:03:572
2117399873,36cyclictest4440-21runrttasks11:19:522
2117399873,36cyclictest4135-21/usr/sbin/munin09:24:202
2117399873,36cyclictest28202-21/usr/sbin/munin12:04:252
2117399873,33cyclictest20802-21/usr/sbin/munin09:54:192
21161998727,27cyclictest1567-21sed07:44:280
21173998634,45cyclictest0-21swapper/211:14:172
21161998666,11cyclictest30073-21apt-get07:34:110
21161998635,22cyclictest18076-21taskset08:35:490
21161998633,19cyclictest5682-21/usr/sbin/munin07:59:180
21161998630,24cyclictest0-21swapper/008:29:230
21165998568,9cyclictest0-21swapper/111:29:131
21165998568,9cyclictest0-21swapper/111:29:131
21165998454,11cyclictest31412-21snmpd07:14:171
21161998347,27cyclictest0-21swapper/008:59:080
2117399813,33cyclictest21021-21latency_hist07:08:592
2116599814,36cyclictest31-21ksoftirqd/111:40:101
2116599814,36cyclictest31-21ksoftirqd/111:40:101
21165998050,21cyclictest23772-21latency_hist09:58:581
21165998034,13cyclictest181rcu_preempt12:34:171
2118099784,63cyclictest23916-21chrt09:59:093
21165997863,11cyclictest58-21kcompactd007:58:571
21165997859,13cyclictest309-21systemd-logind08:48:541
21165997859,13cyclictest309-21systemd-logind08:48:541
21165997847,11cyclictest181rcu_preempt11:56:561
21165997839,9cyclictest181rcu_preempt08:19:211
21165997763,8cyclictest0-21swapper/109:59:241
21165997743,12cyclictest181rcu_preempt11:17:401
21165997655,15cyclictest21992-21/usr/sbin/munin10:54:161
21165997625,28cyclictest0-21swapper/112:04:281
21165997624,10cyclictest181rcu_preempt09:04:181
21165997556,14cyclictest3862-21sh11:19:261
21165997551,9cyclictest181rcu_preempt07:39:051
21165997535,8cyclictest0-21swapper/112:09:121
21165997535,8cyclictest0-21swapper/112:09:121
21165997460,10cyclictest21151-21cyclictest09:52:161
21165997460,10cyclictest21151-21cyclictest09:52:161
21165997450,9cyclictest181rcu_preempt08:09:301
21165997450,9cyclictest181rcu_preempt08:09:301
21165997450,18cyclictest21583-21sh10:53:341
21165997450,18cyclictest21583-21sh10:53:341
21165997449,11cyclictest181rcu_preempt08:38:461
21165997429,12cyclictest181rcu_preempt08:28:141
21165997353,14cyclictest2423-21apt-key10:18:561
21165997350,17cyclictest309-21systemd-logind11:08:541
21165997350,17cyclictest309-21systemd-logind11:08:541
21165997328,13cyclictest181rcu_preempt10:29:531
21165997258,9cyclictest58-21kcompactd008:04:391
21165997258,9cyclictest58-21kcompactd008:04:391
21165997252,15cyclictest29562-21latency_hist07:33:561
21165997237,11cyclictest181rcu_preempt12:29:091
21165997237,11cyclictest181rcu_preempt12:29:091
21165997230,8cyclictest181rcu_preempt07:53:571
21165997151,15cyclictest21151-21cyclictest09:39:151
21165997145,8cyclictest181rcu_preempt08:14:281
21165997145,11cyclictest181rcu_preempt12:15:571
21165997141,8cyclictest181rcu_preempt12:22:171
21165997141,8cyclictest181rcu_preempt12:22:171
21165997124,13cyclictest181rcu_preempt07:24:311
2116599704,33cyclictest27706-21unixbench_singl10:04:391
2116599704,33cyclictest27706-21unixbench_singl10:04:391
21165997034,13cyclictest181rcu_preempt08:03:561
21165997026,35cyclictest3154-21rm10:19:571
2116599702,36cyclictest31-21ksoftirqd/111:49:191
2118099694,39cyclictest0-21swapper/310:19:373
21180996935,22cyclictest31064-21chrt12:09:043
21180996935,22cyclictest31064-21chrt12:09:043
21180996932,24cyclictest11324-21chrt09:35:513
21180996932,24cyclictest11324-21chrt09:35:513
21180996932,23cyclictest25079-21chrt08:59:083
21180996930,27cyclictest19218-21taskset10:48:593
21180996930,27cyclictest19218-21taskset10:48:593
21165996954,8cyclictest0-21swapper/109:34:231
21165996954,8cyclictest0-21swapper/109:34:231
21165996953,11cyclictest21151-21cyclictest08:59:301
21165996946,14cyclictest22680-21idleruntime-cro07:13:531
21165996945,17cyclictest58-21kcompactd009:28:261
21165996941,9cyclictest181rcu_preempt11:38:551
21165996937,10cyclictest181rcu_preempt08:43:561
21165996933,12cyclictest181rcu_preempt11:13:461
21165996911,33cyclictest29257-21grep09:09:301
21180996834,23cyclictest25580-21chrt11:59:113
21180996833,24cyclictest10643-21chrt10:33:393
21180996833,23cyclictest13784-21chrt10:39:103
21180996831,25cyclictest7255-21taskset11:25:023
21180996828,26cyclictest513-21taskset07:43:113
21180996827,26cyclictest21422-21chrt08:47:093
21180996827,26cyclictest21422-21chrt08:47:093
21165996849,13cyclictest355-21runrttasks09:19:091
21165996846,16cyclictest25350-21needreboot10:59:221
21165996846,16cyclictest25350-21needreboot10:59:221
21165996843,8cyclictest181rcu_preempt09:14:111
21165996843,8cyclictest181rcu_preempt09:14:111
2116599683,43cyclictest30299-21fschecks_time07:34:201
21165996829,23cyclictest355-21runrttasks09:44:431
21180996734,21cyclictest4858-21chrt11:21:133
21180996733,22cyclictest2612-21chrt07:49:143
21180996731,24cyclictest21541-21taskset11:51:293
21180996731,22cyclictest23669-21taskset07:14:193
21180996729,24cyclictest12896-21taskset12:33:443
21180996729,24cyclictest12896-21taskset12:33:443
21165996745,16cyclictest7235-21sh11:24:581
21165996744,8cyclictest181rcu_preempt08:29:171
21165996743,14cyclictest0-21swapper/108:53:531
21165996743,14cyclictest0-21swapper/108:53:531
21165996741,17cyclictest0-21swapper/111:44:241
21165996737,15cyclictest181rcu_preempt10:24:521
21165996737,15cyclictest181rcu_preempt10:24:521
21165996736,14cyclictest181rcu_preempt12:28:161
21165996733,13cyclictest181rcu_preempt11:59:431
21165996721,34cyclictest1368-21grep07:44:221
21165996716,32cyclictest30112-21proc_pri10:09:371
21165996716,32cyclictest30112-21proc_pri10:09:371
2013926713,37sleep30-21swapper/307:04:293
21180996633,22cyclictest8073-21chrt12:24:483
21180996633,21cyclictest29449-21chrt09:09:333
21180996633,20cyclictest2591-21chrt09:19:423
21180996633,20cyclictest12065-21chrt10:34:323
21180996633,20cyclictest12065-21chrt10:34:323
21180996632,21cyclictest17975-21chrt08:34:383
21180996631,23cyclictest14812-21chrt11:39:213
21180996631,23cyclictest14812-21chrt11:39:213
21180996631,22cyclictest4307-21chrt12:18:573
21180996631,22cyclictest4307-21chrt12:18:573
21180996631,22cyclictest31500-21chrt10:12:433
21180996631,22cyclictest31500-21chrt10:12:433
21180996631,22cyclictest26176-21chrt07:21:483
21180996631,22cyclictest26176-21chrt07:21:483
21180996631,21cyclictest10810-21chrt08:14:173
21180996630,25cyclictest2267-21chrt11:16:513
21180996630,23cyclictest17955-21taskset10:45:343
21180996629,24cyclictest31896-21taskset11:11:203
21180996629,24cyclictest22625-21taskset07:13:263
21180996628,23cyclictest18930-21taskset11:47:033
21165996650,11cyclictest21151-21cyclictest08:54:281
21165996650,11cyclictest21151-21cyclictest08:54:281
21165996636,10cyclictest181rcu_preempt10:48:561
21165996610,30cyclictest14061-21fschecks_time10:39:181
21180996532,21cyclictest27049-21chrt10:04:243
21180996532,21cyclictest27049-21chrt10:04:243
21180996532,20cyclictest3545-21chrt12:17:153
21180996531,20cyclictest22193-21chrt10:54:143
21180996531,20cyclictest19259-21chrt08:39:283
21180996529,24cyclictest20801-21taskset09:54:063
21180996529,24cyclictest12516-21taskset11:34:313
21180996527,25cyclictest27852-21taskset07:27:443
21165996532,17cyclictest25713-21switchtime07:19:301
21165996532,17cyclictest25713-21switchtime07:19:301
21180996433,20cyclictest19105-21chrt09:49:333
21180996433,20cyclictest19105-21chrt09:49:333
21180996431,21cyclictest6625-21chrt09:28:543
21180996431,21cyclictest28879-21chrt11:05:203
21180996431,21cyclictest28879-21chrt11:05:203
21180996431,21cyclictest13783-21chrt08:24:093
21180996430,22cyclictest27835-21chrt09:06:333
21180996430,21cyclictest32746-21chrt10:14:303
21180996430,21cyclictest25919-21chrt10:59:333
21180996430,21cyclictest25919-21chrt10:59:333
21180996430,21cyclictest11133-21chrt11:33:293
21180996430,21cyclictest11133-21chrt11:33:293
21180996429,24cyclictest7220-21chrt09:29:313
21180996429,24cyclictest14152-21taskset09:41:063
21180996429,23cyclictest13265-21taskset12:34:053
21180996416,31cyclictest0-21swapper/307:34:103
210512649,35sleep121060-21sh07:08:561
21180996333,18cyclictest28233-21chrt07:29:093
21180996331,20cyclictest23447-21chrt08:54:063
21180996331,20cyclictest23447-21chrt08:54:063
21180996331,19cyclictest31850-21chrt09:14:283
21180996331,19cyclictest31850-21chrt09:14:283
21180996330,21cyclictest6167-21chrt10:24:273
21180996330,21cyclictest6167-21chrt10:24:273
21180996330,20cyclictest8511-21chrt08:07:123
21180996330,20cyclictest8511-21chrt08:07:123
21180996329,22cyclictest24350-21chrt11:56:193
21180996325,26cyclictest6698-21taskset07:59:393
21180996231,20cyclictest9928-21chrt08:09:433
21180996231,20cyclictest9928-21chrt08:09:433
21180996230,20cyclictest30131-21chrt12:06:573
21180996230,19cyclictest4324-21chrt07:54:163
21180996229,20cyclictest16637-21chrt09:45:193
21180996229,20cyclictest13408-21chrt08:22:473
21180996229,19cyclictest21925-21taskset08:49:123
21180996229,19cyclictest21925-21taskset08:49:123
21180996129,19cyclictest15635-21chrt08:29:153
21180996028,19cyclictest861-21chrt07:44:083
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional