You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-27 - 12:14
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Sat Apr 27, 2024 00:44:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30430213524,52sleep125-21ksoftirqd/119:05:051
30578999754,36cyclictest8774-21munin-run23:45:043
3056699943,55cyclictest3277-21sh23:35:490
45302900,6sleep10-21swapper/123:36:521
30578998975,8cyclictest0-21swapper/323:05:553
3057799894,63cyclictest399-21runrttasks22:55:412
30578998844,37cyclictest17736-21unixbench_multi20:00:393
30577998829,18cyclictest0-21swapper/223:15:332
30578998631,46cyclictest0-21swapper/319:10:433
24542864,17sleep01986-21/usr/sbin/munin19:15:310
3057799858,45cyclictest0-21swapper/221:10:502
3057799858,45cyclictest0-21swapper/221:10:502
3057799845,47cyclictest0-21swapper/223:05:382
30566998430,20cyclictest141rcu_preempt23:19:280
30578998357,18cyclictest0-21swapper/322:50:353
30571998338,28cyclictest141rcu_preempt22:25:241
30566998347,8cyclictest141rcu_preempt20:05:070
3057899828,23cyclictest21804-21grep22:20:433
3057899823,24cyclictest3771-21wc23:36:223
3057899823,23cyclictest28005-21unixbench_singl22:30:443
30578998230,44cyclictest0-21swapper/319:50:433
30578998228,41cyclictest0-21swapper/323:34:103
30578998228,41cyclictest0-21swapper/323:34:103
30578998226,48cyclictest0-21swapper/321:30:153
30578998215,12cyclictest0-21swapper/323:30:063
30578998215,12cyclictest0-21swapper/323:30:063
3016528212,47sleep330175-21awk19:01:503
3057899813,25cyclictest26303-21cut23:20:273
3057899813,23cyclictest19069-21fschecks_count21:25:253
3057899813,23cyclictest15461-21fschecks_time19:55:243
30578998123,7cyclictest0-21swapper/300:35:043
30578998010,8cyclictest182052/usr/sbin/munin00:00:183
30578998010,8cyclictest182052/usr/sbin/munin00:00:183
30566998040,10cyclictest141rcu_preempt19:40:070
3057899794,19cyclictest28459-21latency_hist20:35:083
30578997934,12cyclictest0-21swapper/323:45:263
30578997934,12cyclictest0-21swapper/323:45:263
3057899793,33cyclictest14957-21latency_hist23:55:063
3057899793,22cyclictest7793-21unixbench_singl19:30:393
3057899793,19cyclictest6412-21grep21:55:433
3057899793,19cyclictest25216-21find21:35:153
30578997927,10cyclictest0-21swapper/323:10:243
3057899792,21cyclictest4675-21timerandwakeup22:45:423
3057899786,17cyclictest374-21sed20:45:423
3057899786,17cyclictest374-21sed20:45:423
3057899785,24cyclictest28969-21fschecks_time20:36:233
3057899784,21cyclictest248972unixbench_multi22:25:423
3057899783,20cyclictest4520-21sed19:20:413
3057899783,20cyclictest10966-21unixbench_multi21:11:403
3057899783,20cyclictest10966-21unixbench_multi21:11:403
3057899783,19cyclictest31234-21grep20:40:433
30578997830,39cyclictest0-21swapper/300:10:063
3057899782,20cyclictest399-21runrttasks22:06:573
3057899782,20cyclictest399-21runrttasks22:06:573
3057899774,29cyclictest18001-21sh21:23:453
3057899773,20cyclictest26142-21head20:25:413
30578997714,12cyclictest0-21swapper/321:01:243
30578997712,14cyclictest0-21swapper/320:10:053
30578997711,11cyclictest0-21swapper/322:35:263
30578997711,11cyclictest0-21swapper/322:35:263
30578997711,10cyclictest0-21swapper/322:56:383
3057799778,48cyclictest0-21swapper/221:31:222
3057799776,45cyclictest7073-21apt-get22:50:182
30577997726,17cyclictest21680-21ssh23:12:342
30577997716,31cyclictest2206-21/usr/sbin/munin21:50:362
30571997743,21cyclictest141rcu_preempt22:37:001
30571997743,21cyclictest141rcu_preempt22:37:001
30571997721,27cyclictest0-21swapper/123:40:191
3057899763,20cyclictest14753-21sshd22:10:213
3057899763,19cyclictest25123-21grep00:10:403
3057899763,19cyclictest1893-21timerwakeupswit20:50:413
30578997629,39cyclictest0-21swapper/319:46:133
30578997624,9cyclictest0-21swapper/319:40:233
30578997624,9cyclictest0-21swapper/319:40:233
3057899762,21cyclictest31213-21apt-config21:45:073
30577997661,9cyclictest6725-21apt-key22:50:072
3057899754,18cyclictest276-21systemd-journal21:20:043
3057899753,21cyclictest909-21fschecks_count22:40:253
3057899753,21cyclictest5598-21sed19:25:283
30578997514,12cyclictest0-21swapper/321:00:073
30571997531,29cyclictest141rcu_preempt22:14:331
30571997524,18cyclictest141rcu_preempt21:20:041
3057899743,29cyclictest681-21cron00:25:043
3057899743,21cyclictest11171-21modprobe22:05:043
3057899743,21cyclictest11171-21modprobe22:05:043
3057899743,19cyclictest22356-21latency20:15:293
30578997421,8cyclictest0-21swapper/322:15:263
30577997415,24cyclictest12380-21/usr/sbin/munin23:50:252
30566997452,14cyclictest14901-40ogrotate23:55:040
3057899733,20cyclictest21195-21timerwakeupswit20:11:323
3057899733,20cyclictest13654-21sshd23:00:213
3057899733,19cyclictest32111-21latency_hist19:10:073
30577997355,11cyclictest28150-21munin-plugin-st20:35:052
30577997323,18cyclictest31912-21/usr/sbin/munin20:45:202
30577997323,18cyclictest31912-21/usr/sbin/munin20:45:202
30577997319,21cyclictest14140-21/usr/sbin/munin19:50:392
30571997325,24cyclictest141rcu_preempt00:20:311
3057899724,19cyclictest28011-21ls00:15:453
30577997222,17cyclictest0-21swapper/221:07:372
30571997239,8cyclictest141rcu_preempt23:14:181
30566997253,11cyclictest17921-21fschecks_count22:15:250
3057899718,14cyclictest0-21swapper/321:46:223
3057899718,14cyclictest0-21swapper/321:46:223
30571997147,16cyclictest30554-21cyclictest21:05:071
30571997132,11cyclictest141rcu_preempt20:11:071
30571997125,21cyclictest0-21swapper/121:08:081
30571997123,18cyclictest141rcu_preempt21:38:381
30566997139,8cyclictest141rcu_preempt22:11:260
30566997131,24cyclictest141rcu_preempt21:42:260
3057899704,19cyclictest649-21idleruntime00:25:263
3057899704,19cyclictest649-21idleruntime00:25:263
3057899703,21cyclictest6239-21/usr/sbin/munin21:05:423
3057899702,20cyclictest25024-21latency_hist20:25:073
30577997052,12cyclictest0-21swapper/200:15:032
30577997026,19cyclictest5921-21diskmemload23:56:522
30571997043,9cyclictest141rcu_preempt23:20:411
30571997040,15cyclictest141rcu_preempt20:15:251
30571997038,10cyclictest141rcu_preempt19:35:341
30571997017,26cyclictest25-21ksoftirqd/122:00:211
30571997017,26cyclictest25-21ksoftirqd/122:00:211
30566997036,10cyclictest6155-21/usr/sbin/munin23:40:200
30566997034,8cyclictest0-21swapper/022:50:080
30566997022,13cyclictest141rcu_preempt20:10:060
30577996951,12cyclictest18510-21munin-plugin-st21:25:042
30577996933,7cyclictest141rcu_preempt19:35:222
30577996925,16cyclictest1599-21taskset19:13:462
30577996920,20cyclictest399-21runrttasks22:15:022
30577996917,23cyclictest6888-21sed23:40:392
30571996943,17cyclictest19903-21unixbench_singl21:25:411
30571996938,8cyclictest141rcu_preempt22:30:241
30571996934,9cyclictest141rcu_preempt20:55:061
30571996934,12cyclictest141rcu_preempt20:40:421
30571996924,24cyclictest0-21swapper/123:51:271
30571996912,23cyclictest141rcu_preempt20:36:111
30566996928,16cyclictest141rcu_preempt00:23:020
30566996910,7cyclictest0-21swapper/021:46:090
30566996910,7cyclictest0-21swapper/021:46:090
30571996836,8cyclictest141rcu_preempt19:08:531
30571996836,14cyclictest141rcu_preempt20:55:281
30571996832,8cyclictest0-21swapper/121:20:231
30571996830,13cyclictest141rcu_preempt19:35:081
30571996829,16cyclictest141rcu_preempt19:50:071
30571996821,22cyclictest0-21swapper/122:44:291
30571996818,25cyclictest0-21swapper/121:54:031
30566996845,14cyclictest18127-21unixbench_multi23:06:280
30566996826,15cyclictest141rcu_preempt00:30:210
30566996819,27cyclictest0-21swapper/022:27:560
2974626824,19sleep00-21swapper/019:01:420
30578996738,13cyclictest41-21ksoftirqd/323:15:433
30577996749,9cyclictest0-21swapper/221:46:112
30577996749,9cyclictest0-21swapper/221:46:112
30577996724,16cyclictest5921-21diskmemload22:22:292
30577996713,23cyclictest20658-21chrt00:04:362
30577996713,23cyclictest20658-21chrt00:04:362
30571996732,16cyclictest2899-21kworker/u8:400:34:441
30571996727,22cyclictest141rcu_preempt22:56:401
30571996726,22cyclictest0-21swapper/120:30:071
30571996724,12cyclictest141rcu_preempt00:05:271
30566996740,8cyclictest141rcu_preempt00:26:360
30566996740,8cyclictest141rcu_preempt00:26:360
30566996733,11cyclictest141rcu_preempt21:31:370
30566996730,16cyclictest141rcu_preempt23:47:500
30566996730,16cyclictest141rcu_preempt23:47:500
30566996721,17cyclictest52-21kcompactd020:55:250
3057899663,16cyclictest2960-21grep19:15:403
30577996626,15cyclictest31544-21taskset20:44:052
30571996639,10cyclictest141rcu_preempt21:31:401
30571996635,15cyclictest141rcu_preempt20:30:311
30571996629,9cyclictest141rcu_preempt00:04:281
30571996629,9cyclictest141rcu_preempt00:04:281
30571996626,27cyclictest141rcu_preempt21:12:331
30571996626,27cyclictest141rcu_preempt21:12:331
3057199662,39cyclictest22190-21kworker/u8:321:45:341
3057199662,39cyclictest22190-21kworker/u8:321:45:341
30571996620,32cyclictest0-21swapper/123:55:461
30571996612,22cyclictest0-21swapper/119:10:271
3056699665,44cyclictest14909-21kworker/u8:021:10:500
3056699665,44cyclictest14909-21kworker/u8:021:10:500
30566996643,16cyclictest26322-21cut23:20:270
30566996631,20cyclictest141rcu_preempt23:10:080
30566996627,22cyclictest141rcu_preempt22:05:530
30566996627,22cyclictest141rcu_preempt22:05:530
30577996548,9cyclictest0-21swapper/223:25:042
30571996542,11cyclictest0-21swapper/100:20:071
30571996537,14cyclictest141rcu_preempt19:55:371
30571996534,12cyclictest141rcu_preempt22:55:081
30571996529,14cyclictest141rcu_preempt00:25:251
30571996529,14cyclictest141rcu_preempt00:25:251
30571996527,8cyclictest141rcu_preempt22:22:031
30571996527,18cyclictest141rcu_preempt21:43:391
30571996520,23cyclictest141rcu_preempt22:48:151
30571996514,25cyclictest141rcu_preempt23:30:191
30571996514,25cyclictest141rcu_preempt23:30:191
30571996512,22cyclictest25-21ksoftirqd/123:04:061
30566996543,11cyclictest0-21swapper/019:10:170
30566996541,7cyclictest141rcu_preempt19:25:080
30566996536,7cyclictest141rcu_preempt00:08:320
30566996536,12cyclictest141rcu_preempt21:23:190
3056699653,40cyclictest0-21swapper/021:19:390
30566996532,13cyclictest141rcu_preempt21:52:010
30566996527,17cyclictest141rcu_preempt23:02:180
30566996516,9cyclictest141rcu_preempt21:05:260
3057899644,21cyclictest399-21runrttasks23:59:033
3057799644,50cyclictest399-21runrttasks21:00:302
30577996442,12cyclictest0-21swapper/219:30:402
30577996439,13cyclictest0-21swapper/219:25:392
30577996415,19cyclictest11343-21taskset19:43:572
30577996415,19cyclictest11343-21taskset19:43:572
30577996414,18cyclictest5921-21diskmemload00:16:492
30577996411,21cyclictest1224-21df_inode20:50:242
30571996444,13cyclictest18699-21unixbench_multi22:15:401
30571996424,12cyclictest141rcu_preempt20:20:271
30571996424,10cyclictest141rcu_preempt00:12:291
30571996421,13cyclictest0-21swapper/123:06:161
30571996419,23cyclictest0-21swapper/120:05:031
30571996418,21cyclictest0-21swapper/119:40:181
30571996418,21cyclictest0-21swapper/119:40:181
30566996434,10cyclictest141rcu_preempt22:43:120
30566996425,30cyclictest0-21swapper/021:36:210
30577996347,11cyclictest30554-21cyclictest20:00:242
30577996313,22cyclictest2709-21apt-get20:55:182
30571996334,11cyclictest141rcu_preempt20:45:301
30571996334,11cyclictest141rcu_preempt20:45:301
30571996322,24cyclictest141rcu_preempt19:15:281
30571996315,21cyclictest141rcu_preempt21:55:081
30571996314,13cyclictest141rcu_preempt23:29:591
30571996314,13cyclictest141rcu_preempt23:29:591
30566996337,12cyclictest141rcu_preempt20:19:440
30566996334,8cyclictest141rcu_preempt22:56:400
3056699633,22cyclictest131ktimers/023:55:450
3056699633,12cyclictest12-21ksoftirqd/022:32:460
30566996317,24cyclictest0-21swapper/023:31:130
30566996317,24cyclictest0-21swapper/023:31:130
2911326310,22sleep20-21swapper/219:00:262
30577996228,21cyclictest23852-21apt-get22:25:172
30577996215,21cyclictest25244-21sh21:35:172
30577996213,21cyclictest2873-21munin-run00:30:042
30577996213,21cyclictest2873-21munin-run00:30:042
30577996213,19cyclictest18022-21ssh22:15:262
30571996234,11cyclictest141rcu_preempt23:15:461
30571996231,13cyclictest141rcu_preempt20:05:191
30571996218,26cyclictest25-21ksoftirqd/122:08:191
30571996218,26cyclictest25-21ksoftirqd/122:08:191
30571996211,21cyclictest25-21ksoftirqd/119:50:221
30566996231,12cyclictest141rcu_preempt20:11:010
30566996220,16cyclictest12-21ksoftirqd/019:40:350
30566996220,16cyclictest12-21ksoftirqd/019:40:350
30566996212,11cyclictest12-21ksoftirqd/000:15:300
3057899614,22cyclictest399-21runrttasks19:39:283
30577996141,13cyclictest0-21swapper/221:45:042
30577996117,16cyclictest7591-21diskmemload21:58:422
30577996113,18cyclictest30221-21ssh23:25:422
30577996113,18cyclictest30221-21ssh23:25:422
30571996125,9cyclictest141rcu_preempt23:49:491
30571996125,9cyclictest141rcu_preempt23:49:491
30566996145,10cyclictest26811-21latency_hist00:15:060
30566996139,14cyclictest0-21swapper/020:45:270
30566996139,14cyclictest0-21swapper/020:45:270
30566996134,7cyclictest141rcu_preempt19:30:210
30566996124,16cyclictest141rcu_preempt20:25:260
30566996121,32cyclictest23778-21fschecks_count20:20:260
30577996044,10cyclictest23295-21cron20:20:052
30577996040,13cyclictest1-21systemd19:25:062
30577996039,14cyclictest21677-21munin-plugin-st20:15:042
30577996030,22cyclictest9665-21awk23:45:332
30577996030,22cyclictest9665-21awk23:45:332
30571996034,12cyclictest141rcu_preempt19:25:401
30566996042,11cyclictest30517-21fschecks_time20:40:260
3056699603,31cyclictest4567-21apt-get21:00:340
30566996028,13cyclictest0-21swapper/021:25:270
30566996012,25cyclictest12-21ksoftirqd/000:00:390
30566996012,25cyclictest12-21ksoftirqd/000:00:390
30566996012,15cyclictest0-21swapper/021:55:200
39472590,7sleep20-21swapper/223:36:282
30577995913,17cyclictest29000-21perl20:36:232
30577995912,20cyclictest26559-21munin-run20:30:062
30571995920,15cyclictest0-21swapper/119:25:071
30566995939,12cyclictest28555-21latency_hist20:35:080
30566995934,7cyclictest141rcu_preempt19:45:560
30566995931,8cyclictest141rcu_preempt22:51:300
30566995926,17cyclictest11968-21kworker/u8:319:55:250
30566995915,15cyclictest12-21ksoftirqd/019:05:320
30577995842,11cyclictest0-21swapper/222:00:472
30577995842,11cyclictest0-21swapper/222:00:472
30577995830,20cyclictest0-21swapper/222:40:162
30577995811,19cyclictest22941-21ssh00:07:402
30566995841,10cyclictest31348-21ssh23:28:400
30566995841,10cyclictest31348-21ssh23:28:400
30566995835,15cyclictest131ktimers/020:40:070
30566995831,7cyclictest141rcu_preempt22:38:200
30566995831,7cyclictest141rcu_preempt22:38:200
30566995828,8cyclictest141rcu_preempt22:01:210
30566995828,8cyclictest141rcu_preempt22:01:210
30578995726,25cyclictest41-21ksoftirqd/321:50:263
30577995741,8cyclictest0-21swapper/222:05:452
30577995741,8cyclictest0-21swapper/222:05:452
30577995734,12cyclictest0-21swapper/223:35:062
30577995734,12cyclictest0-21swapper/223:35:062
30577995730,18cyclictest32203-21cut19:10:072
30577995720,13cyclictest0-21swapper/221:28:422
30577995711,18cyclictest17569-21snmpd19:47:082
3057799569,19cyclictest2455-21memory19:15:312
30577995638,12cyclictest15434-21munin-plugin-st21:20:042
30577995626,22cyclictest13483-21ssh23:00:142
30577995621,21cyclictest26932-21apt-get22:30:172
30577995610,20cyclictest30339-21perl22:35:262
30577995610,20cyclictest30339-21perl22:35:262
30566995623,16cyclictest141rcu_preempt22:21:220
30566995530,7cyclictest141rcu_preempt19:51:150
30577995426,20cyclictest24192-21awk20:20:382
30566995418,12cyclictest0-21swapper/020:55:070
30566995410,19cyclictest141rcu_preempt19:20:400
3057799538,16cyclictest3150-21/usr/sbin/munin00:30:262
30577995324,21cyclictest16595-21taskset20:00:062
30577995316,12cyclictest0-21swapper/200:22:592
30577995225,18cyclictest18705-21fschecks_time20:05:252
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional