You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-10-04 - 05:31
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7 (updated Wed Oct 04, 2023 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1100921150,4sleep2321ktimers/220:06:482
1100921150,4sleep2321ktimers/220:06:482
23328999962,23cyclictest2137-21munin-plugin-st22:06:463
2332899958,47cyclictest0-21swapper/322:52:013
271042930,4chrt27103-21sed23:37:292
271042930,4chrt27103-21sed23:37:292
2332899933,57cyclictest18301-21diskstats22:32:213
2251429212,43sleep00-21swapper/019:02:340
23328998448,15cyclictest41-21ksoftirqd/323:22:153
23328998141,33cyclictest41-21ksoftirqd/323:39:073
23328998141,33cyclictest41-21ksoftirqd/323:39:073
23328998135,22cyclictest41-21ksoftirqd/322:58:453
23304998134,12cyclictest141rcu_preempt23:07:130
23304998126,31cyclictest361-21in:imklog23:55:010
2210028130,23sleep10-21swapper/119:02:191
23328998041,30cyclictest41-21ksoftirqd/319:42:173
23328997810,27cyclictest0-21swapper/322:47:053
23328997810,27cyclictest0-21swapper/322:47:053
23320997841,13cyclictest141rcu_preempt23:10:212
23304997831,26cyclictest183-21pr/ttyS000:29:040
23304997831,26cyclictest183-21pr/ttyS000:29:040
23328997756,12cyclictest41-21ksoftirqd/319:17:243
23328997746,8cyclictest41-21ksoftirqd/322:27:203
2330499775,37cyclictest31649-21latency_hist22:01:480
23328997643,27cyclictest41-21ksoftirqd/300:12:323
23320997632,20cyclictest183-21pr/ttyS022:38:312
23312997623,21cyclictest17408-21http20:26:461
23312997622,22cyclictest9954-21taskset22:17:591
23328997552,14cyclictest5672-21diskmemload21:17:283
23328997529,21cyclictest0-21swapper/323:57:223
23328997529,21cyclictest0-21swapper/323:57:223
23320997528,22cyclictest361-21in:imklog21:21:112
23320997439,13cyclictest141rcu_preempt22:28:182
23320997433,18cyclictest52-21kcompactd020:22:242
23312997456,11cyclictest7439-21cron23:06:451
23304997429,19cyclictest361-21in:imklog22:37:480
23328997336,14cyclictest41-21ksoftirqd/319:16:493
23320997333,21cyclictest25835-21kworker/u8:300:30:292
23320997333,21cyclictest25835-21kworker/u8:300:30:292
23304997332,25cyclictest141rcu_preempt21:42:440
23304997332,25cyclictest141rcu_preempt21:42:440
23320997238,11cyclictest361-21in:imklog22:22:532
23320997238,11cyclictest361-21in:imklog22:22:532
23320997237,20cyclictest0-21swapper/222:32:122
23320997234,12cyclictest141rcu_preempt19:12:562
23312997257,9cyclictest14148-21sendmail20:16:461
23312997257,9cyclictest14148-21sendmail20:16:461
23312997255,11cyclictest8305-21idleruntime-cro22:16:451
23312997254,11cyclictest0-21swapper/119:56:461
23312997254,11cyclictest0-21swapper/119:56:461
23312997253,12cyclictest10898-21basename20:06:461
23312997253,12cyclictest10898-21basename20:06:461
23312997215,21cyclictest21379-21taskset21:43:571
23312997215,21cyclictest21379-21taskset21:43:571
23304997242,10cyclictest141rcu_preempt23:15:460
23304997242,10cyclictest141rcu_preempt23:15:460
23320997138,8cyclictest141rcu_preempt23:32:542
23312997127,15cyclictest18931-21timerwakeupswit00:17:471
23312997124,23cyclictest0-21swapper/100:24:191
23304997133,16cyclictest141rcu_preempt21:33:550
23304997132,12cyclictest141rcu_preempt20:47:270
23304997132,12cyclictest141rcu_preempt20:47:270
221292719,37sleep222168-21irqstats19:02:202
23328997049,15cyclictest0-21swapper/323:56:473
23328997048,15cyclictest8828-21unixbench_multi23:07:303
23328997039,14cyclictest141rcu_preempt23:20:503
23328997036,10cyclictest141rcu_preempt21:57:213
23328997031,13cyclictest2639-21ssh21:12:313
23328997028,34cyclictest41-21ksoftirqd/320:22:273
23328997025,7cyclictest141rcu_preempt19:52:293
23328997025,7cyclictest141rcu_preempt19:52:293
23328997025,12cyclictest0-21swapper/321:37:163
23320997030,10cyclictest52-21kcompactd019:57:212
23320997025,24cyclictest361-21in:imklog22:17:492
23312997055,9cyclictest19248-21munin-plugin-st20:31:451
23312997053,10cyclictest24156-21unixbench_multi23:32:251
23304997051,13cyclictest1436-21latency_hist22:56:480
23304997030,16cyclictest141rcu_preempt20:32:460
23304997027,16cyclictest141rcu_preempt22:34:100
241142690,7chrt0-21swapper/120:46:451
2332899695,32cyclictest0-21swapper/320:27:043
23328996948,8cyclictest41-21ksoftirqd/300:23:173
23328996944,9cyclictest41-21ksoftirqd/321:31:063
23328996943,19cyclictest41-21ksoftirqd/322:42:393
23328996937,13cyclictest41-21ksoftirqd/320:37:273
23320996938,16cyclictest141rcu_preempt19:25:122
23320996931,11cyclictest141rcu_preempt22:05:542
23320996928,20cyclictest141rcu_preempt19:11:102
23312996923,14cyclictest11720-21apt-get22:22:061
23312996923,14cyclictest11720-21apt-get22:22:061
23312996917,18cyclictest23769-21/usr/sbin/munin19:07:161
23304996933,22cyclictest141rcu_preempt22:16:270
2330499693,10cyclictest131ktimers/020:42:280
23304996930,16cyclictest361-21in:imklog22:31:260
23304996930,12cyclictest276-21systemd-journal22:26:460
23304996930,12cyclictest276-21systemd-journal22:26:460
23304996927,28cyclictest52-21kcompactd000:15:190
23328996842,18cyclictest41-21ksoftirqd/300:35:053
23328996842,18cyclictest41-21ksoftirqd/300:35:053
23320996843,7cyclictest141rcu_preempt22:09:262
23320996843,7cyclictest141rcu_preempt22:09:262
23320996839,15cyclictest141rcu_preempt21:39:582
23320996836,7cyclictest141rcu_preempt23:21:312
23320996833,16cyclictest141rcu_preempt23:14:122
23320996833,16cyclictest141rcu_preempt23:14:122
23320996827,20cyclictest361-21in:imklog00:02:482
23304996841,8cyclictest141rcu_preempt21:17:080
23304996840,8cyclictest141rcu_preempt23:31:450
23304996839,15cyclictest141rcu_preempt23:24:330
23304996829,26cyclictest141rcu_preempt00:17:510
23304996817,22cyclictest0-21swapper/021:24:310
23304996814,30cyclictest131ktimers/020:27:350
23328996747,12cyclictest0-21swapper/319:36:463
23328996747,12cyclictest0-21swapper/319:36:463
23328996740,14cyclictest401ktimers/322:12:033
2332899673,38cyclictest5776-21latency_hist19:51:463
23328996733,13cyclictest141rcu_preempt23:43:193
23320996740,8cyclictest141rcu_preempt21:31:412
23320996739,8cyclictest141rcu_preempt00:32:222
23320996739,8cyclictest141rcu_preempt00:32:222
23320996739,7cyclictest141rcu_preempt21:10:092
23320996732,9cyclictest141rcu_preempt20:47:122
23320996732,9cyclictest141rcu_preempt20:47:122
23320996730,13cyclictest361-21in:imklog00:12:262
23312996723,16cyclictest0-21swapper/122:06:211
23312996714,20cyclictest14771-21/usr/sbin/munin20:17:141
23304996742,8cyclictest141rcu_preempt00:10:170
23304996732,28cyclictest0-21swapper/022:46:470
23328996647,11cyclictest0-21swapper/323:32:083
23328996630,14cyclictest41-21ksoftirqd/322:40:063
23328996629,15cyclictest41-21ksoftirqd/323:12:123
23328996629,15cyclictest41-21ksoftirqd/323:12:123
23328996619,22cyclictest141rcu_preempt19:22:183
23320996636,8cyclictest141rcu_preempt19:41:362
23320996630,10cyclictest141rcu_preempt22:54:312
23320996626,15cyclictest141rcu_preempt21:03:272
23320996622,18cyclictest141rcu_preempt20:11:572
23320996622,18cyclictest141rcu_preempt20:11:572
23312996613,23cyclictest18873-21rm21:40:201
23312996611,23cyclictest20053-21/usr/sbin/munin23:27:031
23304996645,8cyclictest141rcu_preempt22:07:360
23304996645,8cyclictest141rcu_preempt22:07:360
23304996640,7cyclictest141rcu_preempt19:52:360
23304996640,7cyclictest141rcu_preempt19:52:360
23304996633,12cyclictest141rcu_preempt23:50:430
23304996633,12cyclictest141rcu_preempt23:50:430
23304996629,13cyclictest141rcu_preempt22:19:160
23304996626,22cyclictest361-21in:imklog21:27:470
23304996616,22cyclictest0-21swapper/023:06:160
23328996540,15cyclictest27351-21cron20:56:453
23328996539,13cyclictest0-21swapper/320:42:293
23328996534,25cyclictest41-21ksoftirqd/323:47:013
23328996534,25cyclictest41-21ksoftirqd/323:47:013
23328996534,17cyclictest397-21runrttasks20:17:113
23328996534,13cyclictest41-21ksoftirqd/321:32:113
23320996546,12cyclictest22450-21unin-run21:46:472
23320996546,12cyclictest22450-21unin-run21:46:472
23320996534,8cyclictest141rcu_preempt22:47:452
23320996534,8cyclictest141rcu_preempt22:47:452
23320996534,10cyclictest141rcu_preempt22:01:332
23320996533,8cyclictest141rcu_preempt21:34:302
23320996531,15cyclictest141rcu_preempt22:45:222
23320996529,17cyclictest141rcu_preempt23:52:172
23320996527,16cyclictest141rcu_preempt20:34:162
23320996526,17cyclictest141rcu_preempt19:21:462
23320996516,19cyclictest33-21ksoftirqd/200:24:102
2331299655,23cyclictest7618-21/usr/sbin/munin21:22:101
23312996547,11cyclictest19894-21kernelversion20:32:211
23312996529,21cyclictest0-21swapper/121:12:311
23304996543,14cyclictest2649-21unixbench_singl21:12:310
23304996537,8cyclictest141rcu_preempt22:47:510
23304996537,8cyclictest141rcu_preempt22:47:510
23304996532,7cyclictest141rcu_preempt19:36:490
23304996532,7cyclictest141rcu_preempt19:36:490
23304996531,12cyclictest141rcu_preempt00:35:590
23304996531,12cyclictest141rcu_preempt00:35:590
23304996523,21cyclictest361-21in:imklog21:56:390
23328996445,12cyclictest28470-21idleruntime-cro21:56:463
23328996430,9cyclictest0-21swapper/322:18:483
23328996412,30cyclictest141rcu_preempt22:24:453
23328996412,30cyclictest141rcu_preempt22:24:453
23320996433,12cyclictest141rcu_preempt00:19:142
23320996431,12cyclictest183-21pr/ttyS023:02:282
23320996429,12cyclictest141rcu_preempt23:58:082
23320996429,12cyclictest141rcu_preempt23:58:082
23320996426,18cyclictest141rcu_preempt21:47:162
23320996423,23cyclictest141rcu_preempt20:52:202
23320996422,19cyclictest52-21kcompactd023:42:182
23320996421,22cyclictest361-21in:imklog23:24:232
23320996421,20cyclictest361-21in:imklog22:15:092
23320996416,20cyclictest361-21in:imklog21:25:192
23312996416,20cyclictest27643-21sh21:54:331
23304996434,7cyclictest141rcu_preempt20:37:270
23304996432,8cyclictest141rcu_preempt23:17:140
23304996431,15cyclictest141rcu_preempt20:25:580
23304996420,20cyclictest0-21swapper/023:45:480
23328996338,12cyclictest141rcu_preempt21:09:453
23328996337,9cyclictest141rcu_preempt21:24:513
23328996334,24cyclictest41-21ksoftirqd/320:01:483
23328996333,17cyclictest41-21ksoftirqd/323:02:193
23328996322,20cyclictest141rcu_preempt20:57:103
23320996336,7cyclictest141rcu_preempt23:27:032
23320996331,7cyclictest141rcu_preempt19:54:542
23320996331,7cyclictest141rcu_preempt19:54:542
2331299638,18cyclictest0-21swapper/121:50:291
23312996346,10cyclictest5968-21latency_hist23:56:491
23312996339,9cyclictest141rcu_preempt23:22:131
23312996312,18cyclictest32712-21/usr/sbin/munin19:32:261
23312996312,18cyclictest32712-21/usr/sbin/munin19:32:261
23304996348,9cyclictest28524-21munin-plugin-st19:21:460
23304996339,7cyclictest141rcu_preempt23:33:200
23304996333,10cyclictest141rcu_preempt19:44:180
23304996331,20cyclictest141rcu_preempt00:22:110
23304996327,22cyclictest141rcu_preempt22:03:500
23304996319,16cyclictest0-21swapper/019:30:190
23304996313,22cyclictest361-21in:imklog23:59:550
23304996313,22cyclictest361-21in:imklog23:59:550
2332899622,21cyclictest25567-21latency_hist21:51:483
23320996247,8cyclictest0-21swapper/219:27:292
23320996236,6cyclictest141rcu_preempt19:47:282
23320996235,8cyclictest141rcu_preempt23:50:572
23320996235,8cyclictest141rcu_preempt23:50:572
23320996229,9cyclictest141rcu_preempt20:41:472
2331299624,49cyclictest8273-21kworker/u8:622:37:001
23312996244,10cyclictest2555-21latency_hist19:41:451
23312996242,11cyclictest0-21swapper/123:57:301
23312996242,11cyclictest0-21swapper/123:57:301
23304996237,19cyclictest26970-21unixbench_singl20:52:270
23304996228,19cyclictest141rcu_preempt21:47:350
23304996228,13cyclictest361-21in:imklog21:08:080
2332899614,37cyclictest23775-21sh00:27:083
2332899614,37cyclictest23775-21sh00:27:083
23328996130,10cyclictest141rcu_preempt00:05:123
23320996136,6cyclictest141rcu_preempt22:58:152
23320996134,7cyclictest141rcu_preempt20:42:062
23320996129,16cyclictest141rcu_preempt00:09:292
2331299614,32cyclictest716-21switchtime23:47:251
2331299614,32cyclictest716-21switchtime23:47:251
23312996143,10cyclictest17430-21cron00:16:451
23312996122,16cyclictest27392-21fschecks_time19:17:111
23312996120,14cyclictest0-21swapper/121:21:201
23304996131,16cyclictest141rcu_preempt20:10:180
23304996128,8cyclictest141rcu_preempt20:00:190
23304996124,18cyclictest141rcu_preempt19:37:290
23304996119,18cyclictest361-21in:imklog19:47:280
23304996112,19cyclictest131ktimers/020:57:280
2332899606,41cyclictest14845-21kworker/u8:200:17:183
23328996043,9cyclictest0-21swapper/320:16:463
23328996043,9cyclictest0-21swapper/320:16:463
23328996039,13cyclictest22806-21systemd23:31:473
23328996020,8cyclictest141rcu_preempt22:07:233
23328996020,8cyclictest141rcu_preempt22:07:233
23320996040,14cyclictest321ktimers/219:32:152
23320996040,14cyclictest321ktimers/219:32:152
23320996033,7cyclictest141rcu_preempt21:12:292
23320996031,8cyclictest141rcu_preempt20:57:522
23320996029,16cyclictest141rcu_preempt19:45:092
23320996027,7cyclictest141rcu_preempt21:52:092
23320996014,17cyclictest0-21swapper/220:30:572
2331299609,30cyclictest397-21runrttasks23:09:481
2331299609,19cyclictest28920-21cat19:22:111
23312996043,11cyclictest0-21swapper/119:51:481
23304996038,8cyclictest141rcu_preempt19:07:210
23304996033,14cyclictest141rcu_preempt23:38:190
23304996033,14cyclictest141rcu_preempt23:38:190
23304996024,9cyclictest141rcu_preempt00:05:020
23304996011,24cyclictest12-21ksoftirqd/021:37:190
23304996010,12cyclictest141rcu_preempt19:22:190
2310526012,25sleep30-21swapper/319:06:203
2332899592,27cyclictest9562-21taskset20:02:083
2332899592,27cyclictest9562-21taskset20:02:083
23328995921,15cyclictest141rcu_preempt20:32:303
23320995926,17cyclictest141rcu_preempt20:07:192
2331299599,17cyclictest2282-21ntp_states22:57:231
23312995945,8cyclictest31511-21kworker/u8:322:01:451
23312995916,14cyclictest13345-21ssh00:08:131
23312995914,17cyclictest28108-21ssh00:34:231
23312995914,17cyclictest28108-21ssh00:34:231
23312995910,20cyclictest9640-21ntp_states00:02:211
23312995910,19cyclictest31504-21sed21:07:211
23304995930,15cyclictest141rcu_preempt22:58:060
23304995914,23cyclictest131ktimers/019:16:470
23328995822,13cyclictest141rcu_preempt00:08:333
2331299587,19cyclictest8910-21users19:57:301
23312995825,22cyclictest10863-21apt-get23:12:061
23312995825,22cyclictest10863-21apt-get23:12:061
23312995810,19cyclictest13824-21ssh21:32:111
23304995830,7cyclictest141rcu_preempt20:12:240
23304995830,7cyclictest141rcu_preempt20:12:240
23304995828,8cyclictest141rcu_preempt21:04:430
23328995734,17cyclictest41-21ksoftirqd/319:27:103
2331299579,18cyclictest11805-21mailstats20:07:281
23312995732,17cyclictest18703-21/usr/sbin/munin22:32:321
23312995718,15cyclictest0-21swapper/121:31:221
23312995710,19cyclictest2442-21sh22:06:591
23312995710,19cyclictest2442-21sh22:06:591
23304995730,12cyclictest141rcu_preempt20:18:450
23328995624,7cyclictest41-21ksoftirqd/321:42:113
23328995624,7cyclictest41-21ksoftirqd/321:42:113
23320995631,7cyclictest141rcu_preempt20:21:122
23312995614,17cyclictest16417-21ssh23:21:131
23312995612,18cyclictest26307-21diskmemload22:44:371
23312995611,20cyclictest31560-21diskmemload23:45:381
23312995611,18cyclictest28485-21taskset23:40:351
23312995611,18cyclictest28485-21taskset23:40:351
2332899554,18cyclictest397-21runrttasks20:07:183
23312995527,20cyclictest31040-21apt-get22:52:251
23312995514,13cyclictest27731-21ssh22:47:051
23312995514,13cyclictest27731-21ssh22:47:051
23304995519,19cyclictest12-21ksoftirqd/020:02:170
23304995519,19cyclictest12-21ksoftirqd/020:02:170
23328995429,18cyclictest41-21ksoftirqd/321:02:263
23328995410,19cyclictest361-21in:imklog20:47:203
23328995410,19cyclictest361-21in:imklog20:47:203
2331299546,21cyclictest28631-21users20:57:291
23312995412,25cyclictest16994-21ssh22:30:341
23312995327,15cyclictest31908-21wc19:31:481
2331299528,17cyclictest29614-21irqstats21:02:151
23312995228,15cyclictest0-21swapper/120:37:111
23312995210,19cyclictest26540-21ntp_states20:52:211
2332899516,25cyclictest141rcu_preempt19:37:443
23328995128,7cyclictest41-21ksoftirqd/319:11:493
2331299515,18cyclictest25800-21ssh00:30:491
2331299515,18cyclictest25800-21ssh00:30:491
23312995129,11cyclictest0-21swapper/120:49:351
23312995129,11cyclictest0-21swapper/120:49:351
2331299506,16cyclictest397-21runrttasks19:13:071
23312995034,9cyclictest183-21pr/ttyS019:42:251
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional