You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-24 - 07:09
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Wed Apr 24, 2024 00:44:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
440211914,78sleep1491-21wc19:02:011
440211914,78sleep1491-21wc19:02:011
10262895,14sleep30-21swapper/319:04:073
10262895,14sleep30-21swapper/319:04:073
323182827,55sleep00-21swapper/019:01:270
323182827,55sleep00-21swapper/019:01:270
58182810,2sleep00-21swapper/019:20:290
1279998156,12cyclictest141rcu_preempt19:10:421
1270998161,12cyclictest11764-21unixbench_singl21:10:540
128799803,18cyclictest19722-21/usr/sbin/munin21:25:562
1287997962,10cyclictest29644-21apt22:35:162
1279997941,30cyclictest0-21swapper/119:50:551
1279997938,33cyclictest1657-21fschecks_count19:05:411
1279997937,36cyclictest22424-21munin-run21:30:161
1270997920,23cyclictest0-21swapper/000:18:160
128799784,8cyclictest0-21swapper/221:40:282
128799783,16cyclictest27263-21cut00:20:182
128799782,17cyclictest7653-21rm22:50:552
128799782,17cyclictest22917-21timerwakeupswit00:10:532
127999783,34cyclictest27553-21apt-get00:20:291
127999783,34cyclictest27553-21apt-get00:20:291
128799774,13cyclictest15634-21apt-get23:05:332
128799774,13cyclictest15634-21apt-get23:05:332
128799773,66cyclictest16712-21unixbench_singl19:50:552
128799773,17cyclictest491-21/usr/sbin/munin22:40:262
128799773,17cyclictest3999-21latency_hist23:40:182
128799773,14cyclictest8716-21grep21:05:522
1279997739,30cyclictest0-21swapper/100:15:161
1270997727,24cyclictest141rcu_preempt22:31:450
1270997610,24cyclictest2596-21apt-get21:50:300
128799754,61cyclictest1714-21grep19:05:412
128799754,16cyclictest13576-21latency_hist21:15:192
128799754,15cyclictest15155-21grep19:46:462
128799753,21cyclictest5952-21fschecks_count19:20:332
128799753,18cyclictest5348-21grep20:55:572
128799753,17cyclictest27143-21cpu20:25:302
128799753,17cyclictest27143-21cpu20:25:302
128799753,17cyclictest12950-21fschecks_time23:00:412
128799753,13cyclictest14186-21timerandwakeup23:55:522
128799753,11cyclictest1289-21latency_hist19:05:202
128799753,11cyclictest1289-21latency_hist19:05:202
1287997523,45cyclictest0-21swapper/221:05:192
128799752,18cyclictest3227-21ls21:51:262
128799752,17cyclictest1551-21latency20:45:432
128799752,15cyclictest24684-21ssh23:20:282
128799752,15cyclictest24684-21ssh23:20:282
1279997540,28cyclictest0-21swapper/119:55:541
127999752,44cyclictest4768-21ssh22:46:141
1270997550,16cyclictest2845-21latency_hist00:35:190
1270997528,9cyclictest141rcu_preempt20:10:200
128799745,17cyclictest21225-21grep22:20:402
128799744,63cyclictest1054-21latency_hist23:35:192
128799744,63cyclictest1054-21latency_hist23:35:192
128799744,17cyclictest2610-21latency_hist00:35:182
128799743,9cyclictest0-21swapper/219:25:322
128799743,13cyclictest24580-21latency_hist23:20:202
1270997454,12cyclictest19248-21unixbench_singl22:16:350
1270997432,11cyclictest141rcu_preempt00:23:140
1270997432,11cyclictest141rcu_preempt00:23:140
1270997428,35cyclictest0-21swapper/022:40:530
1270997420,23cyclictest0-21swapper/020:13:440
1295997350,14cyclictest16718-21unixbench_singl23:05:553
1295997350,14cyclictest16718-21unixbench_singl23:05:553
1287997354,12cyclictest1-21systemd19:10:282
128799733,9cyclictest30062-21modprobe20:35:162
128799733,14cyclictest28695-21tail23:25:532
128799733,14cyclictest20045-21tail00:05:502
128799733,14cyclictest18296-21sed19:55:542
128799733,11cyclictest3554-21ssh22:45:292
128799732,19cyclictest19588-21cat23:10:522
128799732,13cyclictest8664-21seq22:00:322
128799732,10cyclictest16478-21fschecks_count00:00:402
1287997310,56cyclictest1-21systemd23:55:162
1287997310,56cyclictest1-21systemd23:55:162
1270997340,13cyclictest141rcu_preempt20:15:360
128799724,60cyclictest5600-21latency_hist19:20:192
128799723,18cyclictest14964-21fschecks_time22:10:382
128799723,16cyclictest7456-21/usr/sbin/munin23:45:342
128799723,16cyclictest14427-21runrttasks22:10:182
128799723,13cyclictest9614-21ssh22:55:302
128799723,13cyclictest9614-21ssh22:55:302
128799723,11cyclictest24188-21fschecks_count22:25:392
128799723,10cyclictest25803-21ssh21:35:322
128799723,10cyclictest0-21swapper/219:35:332
128799722,15cyclictest32070-21cstates20:40:342
128799722,14cyclictest17034-21tune2fs21:20:372
1287997220,43cyclictest0-21swapper/220:35:392
1270997234,33cyclictest2306-21apt-key21:50:190
1270997234,33cyclictest2306-21apt-key21:50:190
1270997214,10cyclictest0-21swapper/022:10:510
128799716,11cyclictest19001-21apt-get20:00:312
128799714,59cyclictest25756-21grep20:20:392
128799714,20cyclictest12351-21cpuspeed19:40:332
128799714,20cyclictest12351-21cpuspeed19:40:332
128799714,18cyclictest28738-21grep00:21:552
128799714,18cyclictest28738-21grep00:21:552
128799713,61cyclictest327-21tail21:45:532
128799713,61cyclictest327-21tail21:45:532
128799713,59cyclictest21580-21head20:05:532
128799713,58cyclictest4513-21grep23:40:382
128799712,13cyclictest23187-21unixbench_singl20:10:542
1279997137,27cyclictest0-21swapper/123:00:161
1279997137,27cyclictest0-21swapper/123:00:161
1270997132,9cyclictest141rcu_preempt23:49:390
1295997047,16cyclictest7603-21fschecks_time23:45:403
128799704,14cyclictest368-21cat22:40:192
128799704,14cyclictest14720-21ls21:15:572
128799703,57cyclictest4127-21run-parts20:55:162
128799703,15cyclictest25235-21idleruntime-cro20:20:172
128799703,15cyclictest18236-21/usr/sbin/munin22:16:052
128799702,11cyclictest6549-21sed21:55:522
127999705,35cyclictest4260-21/usr/sbin/munin23:40:261
127999702,42cyclictest26856-21sh23:23:551
127999702,42cyclictest26856-21sh23:23:551
1270997035,12cyclictest141rcu_preempt20:35:400
1295996918,21cyclictest141rcu_preempt22:16:203
128799694,57cyclictest9992-21grep19:30:562
128799694,57cyclictest9992-21grep19:30:562
128799694,13cyclictest31038-21find00:26:022
1279996934,27cyclictest0-21swapper/123:05:291
1279996934,27cyclictest0-21swapper/123:05:291
1279996931,31cyclictest9245-21kworker/u8:523:12:161
1279996922,12cyclictest141rcu_preempt22:51:091
1295996852,10cyclictest22690-21unixbench_multi23:15:513
128799683,18cyclictest23402-21ps21:30:482
1279996844,15cyclictest11345-21unixbench_multi23:50:561
1279996844,15cyclictest11345-21unixbench_multi23:50:561
1270996844,15cyclictest28460-21idleruntime-cro20:30:170
1270996844,15cyclictest28460-21idleruntime-cro20:30:170
1270996831,11cyclictest141rcu_preempt23:30:420
1270996831,11cyclictest141rcu_preempt23:30:420
1270996823,24cyclictest141rcu_preempt23:05:550
1270996823,24cyclictest141rcu_preempt23:05:550
1295996750,11cyclictest27161-21latency_hist00:20:183
1295996746,12cyclictest30299-21fschecks_time22:35:403
127999673,58cyclictest0-21swapper/121:35:321
1279996729,30cyclictest0-21swapper/123:20:191
1270996745,13cyclictest22163-21ssh21:29:370
1270996733,18cyclictest141rcu_preempt22:00:390
1270996728,9cyclictest141rcu_preempt19:10:270
129599664,29cyclictest5293-21ssh21:55:163
1279996645,8cyclictest141rcu_preempt22:15:171
127999663,56cyclictest4126-21idleruntime-cro20:55:161
127999662,54cyclictest24358-21/usr/sbin/munin20:15:421
1270996643,16cyclictest2503-21idleruntime-cro20:50:170
1270996638,8cyclictest141rcu_preempt19:35:570
1270996637,9cyclictest141rcu_preempt19:10:190
1270996632,13cyclictest141rcu_preempt19:29:450
1270996630,8cyclictest141rcu_preempt20:40:390
1270996626,17cyclictest141rcu_preempt23:11:520
127099662,21cyclictest14504-21ssh23:03:020
1295996546,12cyclictest4580-21unixbench_multi22:45:523
1295996540,19cyclictest17104-21munin-plugin-st19:55:173
1279996547,12cyclictest25122-21kernelversion00:15:441
1279996546,11cyclictest179-21pr/ttyS019:25:451
1270996544,15cyclictest20355-21munin-plugin-st20:05:160
1270996529,13cyclictest141rcu_preempt00:04:580
1270996526,9cyclictest141rcu_preempt23:20:550
1270996526,9cyclictest141rcu_preempt23:20:550
1270996526,12cyclictest141rcu_preempt21:30:520
1295996449,9cyclictest22822-21cpuspeed21:30:333
1279996429,27cyclictest0-21swapper/100:00:451
127999642,54cyclictest4177-21cut23:40:191
127999642,52cyclictest22576-21/usr/sbin/munin20:10:421
127999642,51cyclictest29009-21/usr/sbin/munin20:30:381
1279996423,34cyclictest23620-21cron22:25:171
127099647,23cyclictest161rcuc/021:05:200
1270996443,8cyclictest141rcu_preempt23:55:080
1270996443,8cyclictest141rcu_preempt23:55:080
1270996438,8cyclictest141rcu_preempt22:59:450
1270996438,8cyclictest141rcu_preempt22:59:450
1270996431,12cyclictest141rcu_preempt21:55:310
1270996411,17cyclictest12-21ksoftirqd/022:45:500
1270996410,28cyclictest12-21ksoftirqd/020:55:470
129599637,33cyclictest13041-21/usr/sbin/munin23:55:183
129599637,33cyclictest13041-21/usr/sbin/munin23:55:183
1295996332,11cyclictest347-21in:imklog23:32:293
1295996332,11cyclictest347-21in:imklog23:32:293
127999634,49cyclictest16740-21ssh21:20:271
127999632,52cyclictest12615-21/usr/sbin/munin19:40:411
127999632,52cyclictest12615-21/usr/sbin/munin19:40:411
127999632,51cyclictest998-21rm22:40:411
127999632,51cyclictest19622-21/usr/sbin/munin20:00:491
127999632,50cyclictest32515-21/usr/sbin/munin20:40:471
127999632,40cyclictest11242-21ssh21:10:441
1270996348,9cyclictest9245-21kworker/u8:523:30:160
1270996346,10cyclictest0-21swapper/021:40:380
1270996336,7cyclictest141rcu_preempt20:23:360
1270996331,14cyclictest141rcu_preempt22:08:000
1270996327,22cyclictest141rcu_preempt20:55:180
1270996318,24cyclictest141rcu_preempt21:16:250
27022622,11sleep02709-21ssh23:36:500
1295996247,9cyclictest19323-21proc_pri23:10:493
127999624,49cyclictest399-21runrttasks21:50:311
1279996241,13cyclictest25427-21modprobe21:35:161
127999623,52cyclictest28776-21apt-get21:40:281
127999623,47cyclictest8718-21unixbench_multi21:05:521
127999622,52cyclictest30120-21sh00:25:221
127999622,52cyclictest15462-21modprobe19:50:161
127999622,51cyclictest7382-21diskmemload00:30:301
127999622,51cyclictest31346-21/usr/sbin/munin20:35:551
127999622,50cyclictest4858-21/usr/sbin/munin20:55:461
127999622,50cyclictest2430-21latency_hist21:50:201
127999622,50cyclictest2430-21latency_hist21:50:201
127099626,35cyclictest32265-21ssh00:29:390
1270996245,9cyclictest161rcuc/022:20:500
1270996238,14cyclictest17281-21runrttasks19:55:200
1270996235,7cyclictest141rcu_preempt21:39:070
1270996230,13cyclictest141rcu_preempt22:39:120
1295996142,11cyclictest22496-21fschecks_time20:10:403
127999613,49cyclictest1326-21fschecks_count20:45:391
1279996125,28cyclictest0-21swapper/122:15:521
127999612,51cyclictest29798-21grep22:35:191
127999612,49cyclictest9972-21head19:30:561
127999612,49cyclictest9972-21head19:30:561
127999612,49cyclictest13159-21/usr/sbin/munin23:00:451
127999612,49cyclictest10656-21apt-get19:35:331
1279996120,34cyclictest399-21runrttasks21:56:421
127099619,39cyclictest12-21ksoftirqd/020:30:480
1270996134,14cyclictest141rcu_preempt22:26:510
1295996040,13cyclictest0-21swapper/321:50:213
1295996024,11cyclictest347-21in:imklog22:29:523
127999605,45cyclictest0-21swapper/123:25:391
127999602,51cyclictest11874-21fschecks_count22:05:371
127999602,50cyclictest6794-21timerandwakeup21:00:531
127999602,50cyclictest25754-21/usr/sbin/munin20:20:391
127999602,50cyclictest19248-21/usr/sbin/munin00:05:341
127099607,36cyclictest12-21ksoftirqd/000:10:320
1270996039,8cyclictest141rcu_preempt22:52:220
1270996026,12cyclictest141rcu_preempt23:55:310
1270996012,21cyclictest131ktimers/000:05:360
1270996012,21cyclictest12-21ksoftirqd/021:00:400
1295995945,8cyclictest0-21swapper/321:00:203
1295995938,10cyclictest0-21swapper/322:05:193
1295995923,11cyclictest141rcu_preempt19:46:463
127999593,48cyclictest13986-21proc_pri23:55:491
127999593,47cyclictest27297-21fschecks_count20:25:351
127999593,47cyclictest27297-21fschecks_count20:25:351
127999592,49cyclictest6017-21cut19:20:341
127999592,49cyclictest29989-21rm22:35:291
127999592,49cyclictest14056-21cut21:15:421
127999592,48cyclictest20974-21/usr/sbin/munin20:05:411
127999592,47cyclictest5642-21latency_hist19:20:191
127999592,35cyclictest25221-21ssh22:26:221
1270995942,10cyclictest276-21systemd-journal19:45:160
1270995942,10cyclictest276-21systemd-journal19:45:160
1270995929,8cyclictest141rcu_preempt21:20:410
1270995922,10cyclictest52-21kcompactd023:18:150
1270995919,19cyclictest0-21swapper/019:55:400
67952582,5sleep00-21swapper/023:44:510
1295995831,13cyclictest0-21swapper/321:05:373
1295995829,18cyclictest20822-21grep21:25:583
127999582,48cyclictest8588-21ssh22:00:231
127999582,48cyclictest7593-21tune2fs23:45:391
1279995824,25cyclictest0-21swapper/123:30:241
1279995824,25cyclictest0-21swapper/123:30:241
1270995820,31cyclictest14119-21apt-get19:45:320
1295995736,15cyclictest1254-21cyclictest00:15:183
1295995733,12cyclictest0-21swapper/319:15:183
1270995732,7cyclictest141rcu_preempt19:30:570
1270995732,7cyclictest141rcu_preempt19:30:570
129599569,28cyclictest41-21ksoftirqd/321:17:073
129599569,14cyclictest41-21ksoftirqd/323:00:453
1270995619,16cyclictest0-21swapper/019:19:380
1295995540,8cyclictest0-21swapper/322:05:523
1295995536,7cyclictest141rcu_preempt23:28:223
129599553,23cyclictest7382-21diskmemload21:40:173
1295995521,14cyclictest141rcu_preempt19:05:433
1295995437,9cyclictest0-21swapper/321:00:543
1295995434,10cyclictest0-21swapper/322:10:363
129599543,18cyclictest25431-21wc20:20:203
1295995429,13cyclictest0-21swapper/300:30:513
1295995427,16cyclictest0-21swapper/321:15:193
1295995421,7cyclictest141rcu_preempt20:50:443
1295995416,17cyclictest179-21pr/ttyS022:51:193
129599538,23cyclictest141rcu_preempt20:40:453
129599537,22cyclictest141rcu_preempt23:40:263
1295995337,10cyclictest12027-21systemd-run19:40:173
1295995337,10cyclictest0-21swapper/320:20:463
1295995331,12cyclictest0-21swapper/322:55:513
1295995331,12cyclictest0-21swapper/322:55:513
1295995325,16cyclictest141rcu_preempt20:00:273
1295995324,7cyclictest141rcu_preempt20:30:463
1295995322,14cyclictest141rcu_preempt22:31:073
129599525,25cyclictest141rcu_preempt19:30:403
129599525,25cyclictest141rcu_preempt19:30:403
1295995237,10cyclictest391rcuc/320:36:423
1295995233,7cyclictest141rcu_preempt21:47:233
1295995233,7cyclictest141rcu_preempt00:20:343
1295995233,7cyclictest141rcu_preempt00:20:343
1295995231,11cyclictest0-21swapper/300:10:163
1295995225,14cyclictest141rcu_preempt22:23:073
1295995224,7cyclictest141rcu_preempt20:45:233
1295995223,7cyclictest141rcu_preempt23:35:363
129599522,14cyclictest41-21ksoftirqd/319:58:213
1295995213,15cyclictest41-21ksoftirqd/323:55:313
1295995212,20cyclictest41-21ksoftirqd/319:29:083
1295995211,20cyclictest41-21ksoftirqd/322:43:213
1295995134,7cyclictest0-21swapper/320:25:363
1295995134,7cyclictest0-21swapper/320:25:363
129599512,19cyclictest303432diskstats00:25:353
129599503,21cyclictest5821-21cpu19:20:293
1295995021,16cyclictest141rcu_preempt21:55:473
1295995014,17cyclictest399-21runrttasks19:40:273
1295995014,17cyclictest399-21runrttasks19:40:273
1295995012,15cyclictest0-21swapper/321:40:233
1295994927,12cyclictest0-21swapper/321:20:413
1295994926,14cyclictest0-21swapper/323:20:373
1295994926,14cyclictest0-21swapper/323:20:373
1295994913,14cyclictest0-21swapper/300:02:363
129599484,15cyclictest0-21swapper/320:05:333
129599447,18cyclictest41-21ksoftirqd/319:17:283
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional