You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-28 - 17:49
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Sun Apr 28, 2024 12:44:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
87219910846,13cyclictest0-21swapper/109:09:581
241482980,4chrt24147-21ssh11:11:141
241482980,4chrt24147-21ssh11:11:141
8721999068,14cyclictest0-21swapper/112:05:231
8721999068,14cyclictest0-21swapper/112:05:231
8711998670,10cyclictest22877-21sshd10:15:290
8711998125,27cyclictest12872-21sh10:52:550
753128116,31sleep20-21swapper/207:00:402
8711997964,9cyclictest10465-21latency_hist07:10:020
8743997833,25cyclictest0-21swapper/309:01:223
8743997833,25cyclictest0-21swapper/309:01:223
8721997848,8cyclictest25-21ksoftirqd/111:55:351
8711997826,28cyclictest26158-21sh11:15:290
8721997745,9cyclictest25-21ksoftirqd/108:23:291
8721997745,9cyclictest25-21ksoftirqd/108:23:291
8711997751,7cyclictest141rcu_preempt09:56:250
8711997739,9cyclictest141rcu_preempt09:49:010
8721997447,19cyclictest10582-21munin-plugin-st09:54:581
8721997443,9cyclictest25-21ksoftirqd/110:30:261
8721997443,9cyclictest25-21ksoftirqd/110:30:261
8721997422,10cyclictest141rcu_preempt10:45:211
8743997328,29cyclictest141rcu_preempt10:03:513
8711997357,10cyclictest346-21in:imuxsock11:24:580
8711997354,13cyclictest5798-21latency_hist08:40:000
290052730,6sleep20-21swapper/209:30:222
8743997253,12cyclictest10672-21idleruntime-cro10:49:583
8743997251,13cyclictest8969-21munin-plugin-st08:49:583
8743997244,9cyclictest141rcu_preempt09:37:473
8743997237,12cyclictest141rcu_preempt10:43:333
8743997218,27cyclictest17245-1kworker/3:0H12:06:003
8743997218,27cyclictest17245-1kworker/3:0H12:06:003
8735997219,24cyclictest22766-21apt-get10:15:182
8721997257,9cyclictest21040-21grep11:05:491
8743997128,8cyclictest141rcu_preempt12:30:143
8735997144,17cyclictest12326-21munin-run07:14:592
8735997140,8cyclictest141rcu_preempt08:55:332
8735997134,8cyclictest141rcu_preempt12:30:212
8721997144,14cyclictest0-21swapper/109:10:381
8721997138,8cyclictest25-21ksoftirqd/107:45:191
8721997133,11cyclictest25-21ksoftirqd/108:00:311
871199712,12cyclictest8981-21grep09:50:340
8743997041,7cyclictest141rcu_preempt10:10:153
8743997041,7cyclictest141rcu_preempt10:10:153
8743997016,24cyclictest0-21swapper/310:52:203
8743997012,32cyclictest41-21ksoftirqd/308:20:143
8743997012,32cyclictest41-21ksoftirqd/308:20:143
8735997037,13cyclictest141rcu_preempt11:53:402
8721997036,8cyclictest141rcu_preempt12:26:271
8721997035,10cyclictest25-21ksoftirqd/110:37:421
8711997044,8cyclictest141rcu_preempt09:31:250
83662705,43sleep10-21swapper/107:02:181
76162709,38sleep07619-21/usr/sbin/munin07:00:420
8743996928,9cyclictest141rcu_preempt08:05:223
8735996927,8cyclictest141rcu_preempt12:05:372
8735996927,8cyclictest141rcu_preempt12:05:372
8735996927,19cyclictest141rcu_preempt09:10:382
8735996917,26cyclictest52-21kcompactd011:15:472
8721996947,8cyclictest25-21ksoftirqd/111:45:011
8721996947,8cyclictest25-21ksoftirqd/111:45:011
8721996947,12cyclictest25-21ksoftirqd/108:40:201
8711996941,7cyclictest141rcu_preempt11:10:070
8711996925,11cyclictest0-21swapper/011:45:000
8711996925,11cyclictest0-21swapper/011:45:000
8743996847,13cyclictest27239-21unixbench_multi10:21:453
8743996837,8cyclictest0-21swapper/307:35:373
8735996842,7cyclictest141rcu_preempt10:57:142
8735996824,19cyclictest33-21ksoftirqd/211:35:302
8735996813,28cyclictest33-21ksoftirqd/210:24:592
8721996827,16cyclictest141rcu_preempt12:13:501
8721996827,16cyclictest141rcu_preempt12:13:501
8721996820,24cyclictest141rcu_preempt08:50:011
8711996854,9cyclictest16189-21kworker/u8:308:30:310
8711996843,7cyclictest141rcu_preempt12:12:020
8711996843,7cyclictest141rcu_preempt12:12:020
8711996810,35cyclictest12-21ksoftirqd/012:20:010
8743996730,16cyclictest19450-21ssh11:04:273
8743996724,23cyclictest141rcu_preempt08:10:233
8743996710,26cyclictest41-21ksoftirqd/309:50:003
8735996752,9cyclictest6947-21runrttasks08:40:352
8735996735,11cyclictest141rcu_preempt09:21:352
8735996726,16cyclictest141rcu_preempt09:19:422
8735996719,23cyclictest33-21ksoftirqd/211:45:222
8721996744,15cyclictest0-21swapper/110:55:021
8721996743,9cyclictest25-21ksoftirqd/107:55:011
8721996732,13cyclictest25-21ksoftirqd/108:15:241
8721996731,8cyclictest25-21ksoftirqd/111:35:381
8711996746,8cyclictest141rcu_preempt09:41:230
8743996644,15cyclictest2393-21/usr/sbin/munin09:40:193
8743996643,14cyclictest14071-21kworker/u8:209:09:583
8743996638,8cyclictest141rcu_preempt12:00:263
8743996638,8cyclictest141rcu_preempt12:00:263
8743996638,7cyclictest141rcu_preempt08:44:323
8743996636,14cyclictest141rcu_preempt10:16:303
8743996633,10cyclictest141rcu_preempt07:45:333
8743996631,9cyclictest141rcu_preempt11:54:413
8743996631,8cyclictest141rcu_preempt11:18:223
8743996630,14cyclictest141rcu_preempt11:30:083
8743996628,13cyclictest141rcu_preempt08:15:153
8743996611,25cyclictest345-21kworker/3:111:14:483
8743996611,25cyclictest345-21kworker/3:111:14:483
8735996644,15cyclictest10592-21idleruntime-cro08:54:592
8735996633,7cyclictest141rcu_preempt11:00:362
8735996623,15cyclictest141rcu_preempt08:15:242
8735996613,27cyclictest141rcu_preempt08:10:322
8721996645,8cyclictest25-21ksoftirqd/111:29:441
8721996645,10cyclictest0-21swapper/108:59:581
8721996641,18cyclictest0-21swapper/108:54:591
8721996635,16cyclictest25-21ksoftirqd/109:30:171
872199663,31cyclictest26846-21runrttasks10:21:361
8721996632,13cyclictest24688-21ssh10:18:091
8743996515,20cyclictest24294-21chrt07:54:593
8735996545,10cyclictest321ktimers/208:45:182
8735996538,7cyclictest141rcu_preempt10:41:412
8735996534,9cyclictest141rcu_preempt12:20:122
8735996526,17cyclictest141rcu_preempt08:30:012
8735996513,25cyclictest7746-21ssh09:50:012
8721996544,14cyclictest5734-21idleruntime-cro08:39:581
8711996547,10cyclictest25411-21munin-plugin-st12:09:580
8711996547,10cyclictest25411-21munin-plugin-st12:09:580
8711996546,12cyclictest0-21swapper/008:14:590
8711996543,12cyclictest0-21swapper/009:05:140
874399648,22cyclictest41-21ksoftirqd/308:58:033
8743996439,7cyclictest141rcu_preempt10:08:143
8743996436,12cyclictest141rcu_preempt07:10:293
8743996426,8cyclictest0-21swapper/311:09:443
8743996424,9cyclictest141rcu_preempt11:38:543
8743996413,28cyclictest41-21ksoftirqd/308:53:323
8743996411,27cyclictest345-21kworker/3:110:32:323
8743996411,27cyclictest345-21kworker/3:110:32:323
8735996438,11cyclictest141rcu_preempt08:05:352
8735996429,9cyclictest141rcu_preempt07:24:582
8735996424,17cyclictest141rcu_preempt10:50:222
8721996442,7cyclictest25-21ksoftirqd/111:30:091
8721996435,8cyclictest25-21ksoftirqd/109:40:161
8721996426,15cyclictest25-21ksoftirqd/109:25:331
8721996426,15cyclictest25-21ksoftirqd/109:25:331
8711996442,14cyclictest15963-21ssh10:57:430
8711996437,7cyclictest141rcu_preempt12:02:420
8711996437,7cyclictest141rcu_preempt12:02:420
743126415,24sleep30-21swapper/307:00:373
8743996348,9cyclictest29230-21latency12:15:223
8743996344,13cyclictest19774-21sh09:14:593
8743996340,7cyclictest141rcu_preempt12:10:523
8743996340,7cyclictest141rcu_preempt12:10:523
8743996334,12cyclictest141rcu_preempt09:50:203
8743996332,7cyclictest141rcu_preempt09:55:203
8743996313,15cyclictest41-21ksoftirqd/308:00:083
8735996346,10cyclictest321ktimers/211:25:172
8735996332,7cyclictest141rcu_preempt07:55:222
8735996324,13cyclictest141rcu_preempt07:45:232
8735996322,10cyclictest141rcu_preempt07:50:222
8735996320,17cyclictest33-21ksoftirqd/207:40:282
8735996318,17cyclictest33-21ksoftirqd/210:30:312
8735996318,17cyclictest33-21ksoftirqd/210:30:312
8735996314,24cyclictest33-21ksoftirqd/211:55:172
8721996338,9cyclictest25-21ksoftirqd/107:10:271
8721996337,10cyclictest25-21ksoftirqd/107:35:231
8721996320,24cyclictest141rcu_preempt11:46:501
8721996312,22cyclictest141rcu_preempt08:10:341
8711996331,15cyclictest141rcu_preempt10:35:040
8711996331,13cyclictest141rcu_preempt11:48:220
874399628,22cyclictest0-21swapper/307:15:153
8743996237,7cyclictest141rcu_preempt10:25:363
8743996220,13cyclictest141rcu_preempt11:55:123
8743996219,16cyclictest41-21ksoftirqd/309:20:393
873599627,23cyclictest33-21ksoftirqd/207:05:462
8735996229,17cyclictest141rcu_preempt11:11:332
8735996229,17cyclictest141rcu_preempt11:11:332
8735996222,21cyclictest141rcu_preempt10:40:002
8735996221,17cyclictest33-21ksoftirqd/207:35:232
8735996219,20cyclictest33-21ksoftirqd/211:20:192
8721996241,8cyclictest25-21ksoftirqd/109:20:391
8721996239,8cyclictest25-21ksoftirqd/107:23:311
8721996239,13cyclictest0-21swapper/108:31:121
8721996235,7cyclictest141rcu_preempt10:12:271
8721996235,7cyclictest141rcu_preempt10:12:271
871199629,13cyclictest12-21ksoftirqd/007:25:330
871199629,13cyclictest12-21ksoftirqd/007:25:330
8711996244,11cyclictest10558-21kworker/u8:110:31:000
8711996244,11cyclictest10558-21kworker/u8:110:31:000
8711996244,11cyclictest0-21swapper/010:14:590
8711996244,11cyclictest0-21swapper/010:14:590
8711996242,12cyclictest2266-21true08:29:590
871199624,10cyclictest12-21ksoftirqd/011:35:380
8711996216,31cyclictest12-21ksoftirqd/011:55:170
8711996211,24cyclictest12-21ksoftirqd/009:20:400
8711996211,13cyclictest13880-21apt-get11:50:070
8743996134,7cyclictest141rcu_preempt08:30:573
8743996130,14cyclictest141rcu_preempt07:05:193
8735996143,11cyclictest0-21swapper/212:14:582
8735996143,11cyclictest0-21swapper/212:14:582
8735996138,14cyclictest14079-21kworker/u8:409:04:582
8735996138,14cyclictest14079-21kworker/u8:409:04:582
8735996136,7cyclictest141rcu_preempt09:25:472
8735996136,7cyclictest141rcu_preempt09:25:472
8735996123,20cyclictest4026-21kworker/u8:111:33:542
8735996117,8cyclictest141rcu_preempt10:00:522
8735996115,16cyclictest26001-21kworker/2:012:17:442
8735996111,29cyclictest141rcu_preempt09:05:212
8721996141,13cyclictest8141-21sendmail12:34:581
8721996128,13cyclictest141rcu_preempt11:05:001
8721996124,13cyclictest141rcu_preempt09:39:401
8721996118,25cyclictest141rcu_preempt10:07:421
8721996117,8cyclictest141rcu_preempt08:25:161
8711996146,9cyclictest27562-21latency_hist08:04:590
8711996140,12cyclictest0-21swapper/010:00:180
8711996136,17cyclictest0-21swapper/010:25:250
871199613,38cyclictest12-21ksoftirqd/008:40:240
8743996037,7cyclictest141rcu_preempt07:23:313
8743996023,23cyclictest141rcu_preempt11:24:333
8743996021,18cyclictest0-21swapper/309:32:283
8743996019,11cyclictest0-21swapper/308:27:253
8743996018,17cyclictest345-21kworker/3:111:30:013
8743996013,20cyclictest0-21swapper/310:57:173
8735996043,9cyclictest321ktimers/210:05:162
8735996037,14cyclictest22581-21latency_hist12:05:012
8735996037,14cyclictest22581-21latency_hist12:05:012
8735996034,7cyclictest141rcu_preempt09:55:152
8735996032,8cyclictest141rcu_preempt08:30:572
8735996029,9cyclictest141rcu_preempt10:30:022
8735996013,27cyclictest141rcu_preempt10:49:122
8735996012,12cyclictest141rcu_preempt11:45:002
8735996012,12cyclictest141rcu_preempt11:45:002
8721996041,10cyclictest0-21swapper/107:19:591
8721996038,12cyclictest0-21swapper/109:00:111
8721996038,12cyclictest0-21swapper/109:00:111
872199603,33cyclictest13983-21/usr/sbin/munin10:00:161
8721996030,7cyclictest141rcu_preempt11:15:391
8721996030,16cyclictest25-21ksoftirqd/108:05:261
8721996029,12cyclictest141rcu_preempt12:23:021
8721996027,7cyclictest0-21swapper/110:27:501
872199602,24cyclictest141rcu_preempt09:45:191
8711996029,11cyclictest141rcu_preempt12:29:590
8711996019,33cyclictest0-21swapper/010:41:400
8743995932,7cyclictest141rcu_preempt07:30:013
8743995932,7cyclictest141rcu_preempt07:30:013
8743995932,15cyclictest141rcu_preempt11:46:213
8743995931,8cyclictest141rcu_preempt10:35:263
8735995929,15cyclictest141rcu_preempt07:20:012
8735995920,18cyclictest141rcu_preempt11:05:362
8735995915,20cyclictest33-21ksoftirqd/209:50:122
8735995914,24cyclictest33-21ksoftirqd/210:10:212
8735995914,24cyclictest33-21ksoftirqd/210:10:212
8735995914,19cyclictest141rcu_preempt07:27:122
8735995914,19cyclictest141rcu_preempt07:27:122
8735995912,30cyclictest321ktimers/208:00:192
872199597,30cyclictest0-21swapper/110:56:091
8721995932,15cyclictest0-21swapper/112:00:201
8721995932,15cyclictest0-21swapper/112:00:201
8721995928,8cyclictest25-21ksoftirqd/111:20:331
8721995928,14cyclictest25-21ksoftirqd/110:41:411
8711995938,7cyclictest141rcu_preempt09:25:450
8711995938,7cyclictest141rcu_preempt09:25:450
8711995938,11cyclictest0-21swapper/010:05:280
8711995936,12cyclictest0-21swapper/008:24:590
8711995936,12cyclictest0-21swapper/008:24:590
8711995935,9cyclictest141rcu_preempt11:13:530
8711995935,9cyclictest141rcu_preempt11:13:530
8711995935,15cyclictest3378-21unixbench_singl11:30:410
8711995912,22cyclictest7572-21seq12:33:310
8743995824,16cyclictest0-21swapper/309:28:263
8743995824,16cyclictest0-21swapper/309:28:263
8743995817,12cyclictest0-21swapper/311:40:153
8743995817,12cyclictest0-21swapper/311:40:153
8743995812,17cyclictest41-21ksoftirqd/307:55:383
8735995841,9cyclictest321ktimers/207:30:372
8721995830,14cyclictest141rcu_preempt09:15:231
871199588,43cyclictest399-21runrttasks10:20:460
871199584,21cyclictest16971-21/usr/sbin/munin11:00:190
8711995841,8cyclictest0-21swapper/007:40:370
8711995837,7cyclictest141rcu_preempt09:38:580
8711995835,9cyclictest141rcu_preempt12:24:230
8711995830,8cyclictest141rcu_preempt10:47:350
871199582,18cyclictest9405-21cut08:50:210
8711995810,29cyclictest1-21systemd07:10:100
8743995735,13cyclictest17828-21basename07:34:583
8743995734,7cyclictest141rcu_preempt08:35:333
8743995733,7cyclictest141rcu_preempt09:18:043
8743995728,8cyclictest141rcu_preempt12:21:403
8743995724,12cyclictest116-21pr/ttymxc012:25:443
873599573,10cyclictest2191-21apt-get09:40:122
8735995718,17cyclictest33-21ksoftirqd/208:35:332
8721995736,12cyclictest399-21runrttasks07:05:371
871199574,26cyclictest12-21ksoftirqd/009:19:070
8711995723,24cyclictest0-21swapper/009:00:010
8735995614,18cyclictest33-21ksoftirqd/209:35:302
8721995639,9cyclictest0-21swapper/107:34:581
8721995631,10cyclictest141rcu_preempt11:50:301
8721995625,13cyclictest141rcu_preempt07:55:241
871199564,20cyclictest18331-21vmstat09:10:400
8711995640,10cyclictest25946-21idleruntime-cro07:59:590
8711995638,9cyclictest0-21swapper/007:54:590
8711995633,9cyclictest141rcu_preempt07:20:280
871199563,35cyclictest12-21ksoftirqd/008:15:200
871199563,33cyclictest12-21ksoftirqd/007:20:020
871199563,20cyclictest12-21ksoftirqd/007:35:250
873599557,18cyclictest33-21ksoftirqd/212:25:032
8721995530,7cyclictest25-21ksoftirqd/112:15:301
8711995532,12cyclictest0-21swapper/007:45:180
8711995522,13cyclictest141rcu_preempt08:06:140
8711995521,13cyclictest141rcu_preempt11:25:040
8711995512,12cyclictest12-21ksoftirqd/007:30:280
8743995413,19cyclictest41-21ksoftirqd/307:40:263
8721995424,17cyclictest25-21ksoftirqd/107:25:241
8721995424,17cyclictest25-21ksoftirqd/107:25:241
871199544,17cyclictest14082-21sh09:04:580
871199544,17cyclictest14082-21sh09:04:580
8711995421,13cyclictest141rcu_preempt08:45:460
8721995325,16cyclictest25-21ksoftirqd/110:00:011
873599523,20cyclictest33-21ksoftirqd/208:22:532
873599523,20cyclictest33-21ksoftirqd/208:22:532
8721995224,7cyclictest141rcu_preempt07:40:341
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional