You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-20 - 21:36
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Mon Oct 20, 2025 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
147899125101,18cyclictest15280-21rm09:20:212
1470991252,83cyclictest14988-21ssh12:10:091
14649912542,25cyclictest0-21swapper/010:50:260
14649912542,25cyclictest0-21swapper/010:50:260
147899122102,13cyclictest23644-21cpu09:35:202
281821070,7sleep10-21swapper/110:52:471
281821070,7sleep10-21swapper/110:52:471
14789910245,48cyclictest25122-21cstates10:35:082
14649910083,8cyclictest0-21swapper/012:10:210
1470999858,33cyclictest21394-21timerwakeupswit09:30:231
147099983,47cyclictest32074-21apt-get09:50:041
147099983,47cyclictest32074-21apt-get09:50:041
4442977,21sleep30-21swapper/307:05:333
9292956,16sleep20-21swapper/207:06:172
1470999574,15cyclictest24081-21latency_hist12:24:521
147099953,60cyclictest15198-21ssh09:20:111
147099943,40cyclictest372-21snmpd11:00:271
147099932,81cyclictest8118-21apt-get10:05:161
147899923,27cyclictest23507-21cut11:30:112
147899923,27cyclictest23507-21cut11:30:112
147899922,42cyclictest15021-21ssh12:10:142
1470999274,12cyclictest25275-21systemctl08:19:501
147899913,36cyclictest11935-21/usr/sbin/munin11:10:112
1478999042,41cyclictest7374-21idleruntime-cro09:04:492
1478999042,41cyclictest7374-21idleruntime-cro09:04:492
1478999038,45cyclictest0-21swapper/212:09:522
1478999031,9cyclictest0-21swapper/209:45:292
1478999017,29cyclictest4662-21timerandwakeup10:55:262
1470999047,36cyclictest0-21swapper/108:30:281
147899893,67cyclictest8399-21apt-get07:30:042
1478998934,45cyclictest0-21swapper/207:44:542
147899893,35cyclictest2944-21cut08:50:122
147899892,33cyclictest15800-21latency_hist07:49:542
1464998969,14cyclictest14002-21apt-key07:44:540
147899883,34cyclictest11780-21sort11:09:522
1470998870,12cyclictest23733-21cstates09:35:201
1470998841,41cyclictest8831-21fschecks_time12:00:151
1470998841,41cyclictest8831-21fschecks_time12:00:151
147899873,36cyclictest7042-21/usr/sbin/munin07:25:232
147899873,35cyclictest31709-21rm10:45:272
147899873,30cyclictest20392-21latency_hist09:29:522
1478998731,48cyclictest0-21swapper/212:25:262
1478998731,48cyclictest0-21swapper/212:25:262
147099873,34cyclictest3781-21/usr/sbin/munin10:55:121
147899863,34cyclictest28871-21runrttasks08:30:092
147899863,33cyclictest352-21runrttasks08:41:172
147899863,32cyclictest24131-21latency_hist12:24:522
147899863,31cyclictest4412-21processes07:15:242
147899863,29cyclictest26420-21ssh11:35:132
1478998627,8cyclictest0-21swapper/212:30:242
147899862,65cyclictest8156-21ssh10:05:082
1478998625,11cyclictest0-21swapper/211:50:192
1478998621,12cyclictest0-21swapper/207:50:112
1470998645,33cyclictest0-21swapper/111:10:231
147099863,34cyclictest25530-21/usr/sbin/munin08:20:221
147099863,33cyclictest15013-21sort09:19:541
148399853,50cyclictest15232-21chrt09:20:163
147899855,29cyclictest20992-21ssh12:19:532
147899854,33cyclictest21632-21unixbench_singl08:05:272
147899854,33cyclictest21632-21unixbench_singl08:05:272
1478998541,35cyclictest0-21swapper/211:20:052
1478998535,42cyclictest0-21swapper/209:50:142
1478998535,42cyclictest0-21swapper/209:50:142
1478998533,44cyclictest0-21swapper/209:34:522
147899853,33cyclictest14994-21apt-key09:19:542
147899853,31cyclictest30812-21fschecks_time12:35:122
147899853,31cyclictest10731-21fschecks_count07:35:132
147899853,27cyclictest8333-21ssh11:02:182
147899853,27cyclictest31161-21cat08:35:252
1478998528,8cyclictest0-21swapper/212:03:592
1478998528,8cyclictest0-21swapper/212:03:592
1478998526,7cyclictest0-21swapper/208:24:532
147899852,37cyclictest16351-21systemctl10:19:502
147899852,36cyclictest1802-21cpu07:10:222
147899852,35cyclictest28590-21latency_hist08:29:522
147899852,31cyclictest20952-21ntpq11:25:192
147099854,37cyclictest15106-21sshd10:16:191
147099853,37cyclictest12220-21/usr/sbin/munin12:05:191
147099852,37cyclictest22497-21fschecks_count08:10:131
147099852,37cyclictest22497-21fschecks_count08:10:131
1464998563,15cyclictest2387-21/usr/sbin/munin09:55:160
147899843,33cyclictest32694-21timerandwakeup11:45:262
147899843,33cyclictest32694-21timerandwakeup11:45:262
147899843,31cyclictest8215-21modprobe11:59:492
147899843,25cyclictest5436-21/usr/sbin/munin07:20:252
1478998426,8cyclictest0-21swapper/211:15:242
1478998426,10cyclictest0-21swapper/210:44:152
147899842,32cyclictest24298-21ssh10:33:402
147899842,32cyclictest24298-21ssh10:33:402
147899842,32cyclictest19075-21acpi08:00:022
147899842,32cyclictest19075-21acpi08:00:022
147899842,28cyclictest2052-21tail10:50:292
147899842,28cyclictest2052-21tail10:50:292
147099843,37cyclictest7563-21/usr/sbin/munin07:25:191
147099843,35cyclictest21426-21/usr/sbin/munin11:25:281
147899833,29cyclictest11448-21uname10:10:172
147899833,26cyclictest352-21runrttasks09:09:442
1478998331,45cyclictest0-21swapper/208:49:532
1478998320,11cyclictest0-21swapper/209:40:202
1470998363,13cyclictest8983-21diskmemload10:10:141
1470998355,20cyclictest1377-21apt-key07:09:551
147099833,39cyclictest22765-21/usr/sbin/munin10:30:181
147099833,39cyclictest22765-21/usr/sbin/munin10:30:181
1470998326,50cyclictest0-21swapper/108:05:111
1470998326,50cyclictest0-21swapper/108:05:111
147099832,37cyclictest27064-21ls12:29:491
147099832,37cyclictest27064-21ls12:29:491
147899823,32cyclictest9559-21fschecks_time09:10:142
147899823,32cyclictest7906-21latency_hist10:04:532
147899823,26cyclictest4999-21find08:55:232
147899823,26cyclictest19016-21latency_hist07:59:542
147899823,26cyclictest19016-21latency_hist07:59:542
1478998231,8cyclictest0-21swapper/208:15:262
1478998220,15cyclictest207-21systemd-journal10:24:492
1470998256,16cyclictest1455-21cyclictest07:59:541
1470998256,16cyclictest1455-21cyclictest07:59:541
147099823,37cyclictest5084-21idleruntime-cro11:54:491
147099823,37cyclictest31980-21/usr/sbin/munin08:39:581
147099823,34cyclictest8983-21cpuspeed07:30:181
147099823,32cyclictest28932-21taskset11:40:041
147099823,27cyclictest1156-21apt-get08:45:021
1470998223,50cyclictest0-21swapper/107:14:541
1464998257,19cyclictest14561-21apt-key12:09:520
147899813,27cyclictest19449-21/usr/sbin/munin10:25:112
147899813,24cyclictest2816-21sh09:55:142
147099813,31cyclictest32011-21fschecks_count11:45:121
147099813,31cyclictest32011-21fschecks_count11:45:121
147099813,30cyclictest31919-21latency_hist09:49:521
147099813,28cyclictest352-21runrttasks07:37:551
147099813,21cyclictest7688-21cat10:04:511
147099812,31cyclictest20641-21rm10:25:441
1464998165,10cyclictest28206-21ssh12:30:260
1464998144,9cyclictest0-21swapper/009:30:210
147899803,25cyclictest29232-21fschecks_time11:40:122
147099804,35cyclictest17911-21/usr/sbin/munin09:25:111
147099803,8cyclictest0-21swapper/110:38:311
147099803,35cyclictest3438-21sed09:55:251
147099803,34cyclictest6419-21/usr/sbin/munin11:55:281
147099803,34cyclictest23468-21ssh11:30:101
147099803,34cyclictest23468-21ssh11:30:101
147099803,33cyclictest9273-21/usr/sbin/munin09:10:251
147099802,37cyclictest31722-21sh08:39:501
1464998066,10cyclictest1455-21cyclictest11:10:220
147899793,25cyclictest23113-21wc08:10:412
147899793,25cyclictest23113-21wc08:10:412
1470997941,31cyclictest0-21swapper/110:40:011
147099793,36cyclictest9754-21/usr/sbin/munin11:05:151
147099793,32cyclictest5623-21/usr/sbin/munin07:20:121
147099793,31cyclictest6183-21fschecks_count09:00:111
147099793,31cyclictest6183-21fschecks_count09:00:111
147099793,31cyclictest17691-21/usr/sbin/munin11:20:181
147099793,27cyclictest30634-21/usr/sbin/munin10:45:111
147099792,36cyclictest30486-21ssh12:35:021
1464997958,15cyclictest9410-21timerandwakeup12:00:280
1464997958,15cyclictest9410-21timerandwakeup12:00:280
1464997942,30cyclictest0-21swapper/012:35:110
119727912,43sleep01227-21munin-run07:09:500
147099783,32cyclictest12847-21/usr/sbin/munin07:40:311
147099783,31cyclictest352-21runrttasks07:11:301
147099782,37cyclictest16719-21/usr/sbin/munin07:50:251
147099782,35cyclictest26197-21ssh11:35:071
147099782,26cyclictest7628-21apt-get09:05:061
1470997822,46cyclictest0-21swapper/108:25:121
147099773,36cyclictest18380-21idleruntime12:15:131
147099773,30cyclictest19289-21fschecks_count08:00:131
147099773,30cyclictest19289-21fschecks_count08:00:131
147099773,13cyclictest0-21swapper/111:15:171
147099772,34cyclictest352-21runrttasks07:45:031
147099772,24cyclictest27922-21/usr/sbin/munin12:30:191
147099772,23cyclictest5269-21grep08:55:271
1464997761,10cyclictest1455-21cyclictest10:44:300
1464997760,11cyclictest352-21runrttasks11:04:520
1464997756,15cyclictest5711-21fschecks_time07:20:140
147099763,35cyclictest16824-21ssh10:20:101
147099763,31cyclictest26439-21/usr/sbin/munin09:40:171
1464997660,10cyclictest23480-21forks11:30:110
1464997660,10cyclictest23480-21forks11:30:110
147099753,30cyclictest3130-21uname08:50:161
1464997553,16cyclictest20191-21latency_hist11:24:520
1464997462,8cyclictest13462-21kworker/u17:1+rpciod10:55:290
1464997456,12cyclictest17493-21ssh09:24:320
1464997456,12cyclictest14831-21unixbench_singl10:15:280
1464997454,14cyclictest1-21systemd08:44:150
1464997454,13cyclictest24785-21ssh10:34:530
1464997454,13cyclictest24785-21ssh10:34:530
1464997355,12cyclictest300-21in:imuxsock08:19:510
1464997353,14cyclictest17387-21sendmail11:19:490
1464997343,23cyclictest0-21swapper/012:22:160
1464997339,27cyclictest0-21swapper/009:35:200
146499733,41cyclictest5222-21latency_hist11:54:520
1464997256,10cyclictest29799-21kworker/u18:2-rpciod09:49:310
1464997254,12cyclictest25404-21ssh10:35:150
1464997250,15cyclictest8983-21diskmemload09:25:150
1464997235,28cyclictest0-21swapper/010:05:220
146499714,33cyclictest14740-21latency_hist09:19:520
1464997135,27cyclictest27732-21ps08:25:220
146499713,35cyclictest24271-21apt-get12:25:000
146499713,35cyclictest24271-21apt-get12:25:000
1464997056,10cyclictest1455-21cyclictest09:40:270
146499705,34cyclictest20050-21sshd10:25:190
1464997053,10cyclictest2258-21ssh09:54:530
1464997053,10cyclictest2258-21ssh09:54:530
1464997050,12cyclictest475-21kworker/u19:0+flush-179:011:26:000
1464997048,15cyclictest17181-21munin-plugin-st07:54:520
146499704,38cyclictest352-21runrttasks10:23:330
1464996951,10cyclictest0-21swapper/011:55:170
1464996948,14cyclictest11854-21latency_hist09:14:520
1464996934,28cyclictest0-21swapper/011:47:570
1464996934,28cyclictest0-21swapper/011:47:570
1464996931,27cyclictest23634-21modprobe08:14:490
1464996931,27cyclictest23634-21modprobe08:14:490
146499685,43cyclictest352-21runrttasks08:56:300
1464996851,11cyclictest29311-21ssh11:40:150
1464996851,11cyclictest11627-21ssh11:09:490
146499684,54cyclictest7813-21cat10:04:530
146499684,53cyclictest352-21runrttasks10:13:450
146499683,32cyclictest15080-21timerwakeupswit07:45:400
1464996750,10cyclictest18208-21cstates12:15:070
146499673,35cyclictest6226-21tune2fs09:00:120
146499673,35cyclictest6226-21tune2fs09:00:120
146499673,34cyclictest26198-21seq11:35:070
146499672,52cyclictest4772-1kworker/0:1H+mmc_complete08:39:520
146499665,51cyclictest1219-21apt-get07:10:060
1464996652,9cyclictest10140-21sshd07:33:390
146499664,52cyclictest30634-21/usr/sbin/munin10:45:180
1464996645,15cyclictest4747-21unixbench_singl07:15:280
1464996640,18cyclictest8515-21apt-key07:29:530
263792650,7sleep30-21swapper/312:28:053
263792650,7sleep30-21swapper/312:28:053
146499656,29cyclictest22213-21runrttasks08:09:530
146499656,29cyclictest22213-21runrttasks08:09:530
1464996542,9cyclictest181rcu_preempt08:00:300
1464996542,9cyclictest181rcu_preempt08:00:300
1464996535,13cyclictest181rcu_preempt07:59:500
1464996535,13cyclictest181rcu_preempt07:59:500
1464996532,22cyclictest11237-21cat07:35:260
146499643,50cyclictest352-21runrttasks08:47:130
1464996422,27cyclictest181rcu_preempt09:05:250
1464996347,11cyclictest22095-21kworker/u19:2-flush-179:008:34:260
1464996241,15cyclictest3029-21fschecks_time08:50:130
1464996230,23cyclictest26128-21grep08:20:230
148399535,33cyclictest20127-21chrt10:25:223
148399532,35cyclictest7749-21taskset10:04:523
148399524,36cyclictest13750-21taskset12:08:113
148399523,38cyclictest8000-21taskset11:59:243
148399523,36cyclictest4192-21chrt09:58:163
148399523,36cyclictest32581-21chrt08:40:203
148399516,34cyclictest12407-21chrt07:40:163
148399516,12cyclictest0-21swapper/309:35:163
148399513,36cyclictest813-21taskset11:46:363
148399513,36cyclictest813-21taskset11:46:363
148399513,35cyclictest16376-21chrt10:19:513
148399513,34cyclictest18169-21chrt09:25:143
148399512,38cyclictest27073-21taskset10:38:273
148399504,34cyclictest4571-21chrt07:15:273
148399503,36cyclictest10441-21chrt10:09:063
148399503,35cyclictest28039-21chrt11:37:573
148399503,35cyclictest20885-21chrt11:25:173
148399503,34cyclictest7188-21taskset11:00:183
148399503,34cyclictest6796-21chrt07:24:503
148399503,34cyclictest4720-21chrt11:53:533
148399503,34cyclictest4204-21chrt10:55:163
148399502,35cyclictest32277-21chrt09:50:143
148399502,35cyclictest32277-21chrt09:50:143
148399502,35cyclictest21223-21chrt09:30:213
148399502,35cyclictest10234-21chrt07:34:503
148399502,30cyclictest905-21taskset10:49:533
148399493,34cyclictest9612-21chrt12:00:303
148399493,34cyclictest9612-21chrt12:00:303
148399493,34cyclictest29998-21chrt09:45:243
148399493,34cyclictest17408-21chrt07:54:543
148399493,32cyclictest10423-21taskset09:10:363
148399492,35cyclictest22384-21chrt08:10:083
148399492,35cyclictest22384-21chrt08:10:083
148399492,35cyclictest22266-21taskset12:20:273
148399484,32cyclictest11583-21chrt10:10:193
148399484,31cyclictest6189-21chrt09:00:113
148399484,31cyclictest6189-21chrt09:00:113
148399483,34cyclictest28729-21taskset12:31:043
148399483,34cyclictest19258-21taskset08:00:123
148399483,34cyclictest19258-21taskset08:00:123
148399483,32cyclictest24172-21chrt08:15:143
148399483,31cyclictest15027-21chrt11:15:133
148399482,32cyclictest325-21chrt12:38:383
148399474,31cyclictest25367-21chrt11:33:273
148399474,31cyclictest25367-21chrt11:33:273
148399473,32cyclictest3138-21taskset07:14:503
148399473,31cyclictest7398-21chrt07:25:143
148399473,31cyclictest30671-21chrt11:42:353
1483994633,8cyclictest0-21swapper/312:10:233
148399463,31cyclictest19182-21taskset10:24:503
148399463,30cyclictest12604-21taskset11:10:233
148399462,32cyclictest18667-21chrt11:20:273
148399454,27cyclictest22837-21chrt10:30:213
148399454,27cyclictest22837-21chrt10:30:213
148399453,32cyclictest1244-21chrt10:50:133
148399453,32cyclictest1244-21chrt10:50:133
148399453,30cyclictest14967-21chrt07:45:393
148399453,29cyclictest2837-21chrt08:50:073
148399452,29cyclictest22100-21taskset08:09:523
148399452,29cyclictest22100-21taskset08:09:523
148399443,29cyclictest12700-21taskset09:15:223
148399443,27cyclictest1472-21taskset08:45:143
148399442,29cyclictest7636-21taskset09:05:053
148399433,31cyclictest17786-21taskset07:55:143
148399433,31cyclictest17786-21taskset07:55:143
148399433,28cyclictest29757-21taskset08:30:323
148399432,29cyclictest19189-21taskset12:15:333
148399423,27cyclictest11645-21chrt07:37:213
148399423,26cyclictest4526-21taskset08:55:093
148399422,27cyclictest9845-21taskset11:05:163
148399412,28cyclictest30863-21taskset08:35:183
148399404,27cyclictest29847-21chrt10:43:173
148399403,27cyclictest28915-21chrt09:44:313
148399403,26cyclictest26078-21chrt08:20:203
148399402,27cyclictest27487-21taskset08:25:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional