You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-08-16 - 07:09

x86 Intel Atom D2700 @2130 MHz, Linux 5.15.49-rt47 (Profile)

Latency plot of system in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot6.osadl.org (updated Tue Aug 16, 2022 00:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2911390,0irq_work/2311rcuc/219:59:342
2911250,0irq_work/2311rcuc/222:43:512
2911180,0irq_work/2311rcuc/223:18:332
2911140,0irq_work/2311rcuc/223:14:282
2911130,0irq_work/2311rcuc/222:58:372
2911130,0irq_work/2311rcuc/221:52:432
2911130,0irq_work/2311rcuc/219:23:502
2911100,0irq_work/2311rcuc/220:33:212
2911100,0irq_work/2311rcuc/219:33:412
2911090,0irq_work/2311rcuc/223:08:492
2911090,0irq_work/2311rcuc/222:58:202
2911090,0irq_work/2311rcuc/222:33:362
2911090,0irq_work/2311rcuc/221:53:522
2911080,0irq_work/2311rcuc/223:03:582
2911070,0irq_work/2311rcuc/223:23:462
2911070,0irq_work/2311rcuc/221:31:552
2911070,0irq_work/2311rcuc/221:31:552
2911070,0irq_work/2311rcuc/221:18:472
2911060,0irq_work/2311rcuc/223:58:002
2911060,0irq_work/2311rcuc/222:27:572
2911060,0irq_work/2311rcuc/222:03:442
2911050,0irq_work/2311rcuc/219:43:482
2911040,0irq_work/2311rcuc/219:18:412
2911040,0irq_work/2311rcuc/219:18:412
2911020,0irq_work/2311rcuc/223:28:412
2911020,0irq_work/2311rcuc/219:08:582
2911000,0irq_work/2311rcuc/223:33:532
291990,0irq_work/2311rcuc/200:31:522
291980,0irq_work/2311rcuc/220:23:202
291970,0irq_work/2311rcuc/222:40:182
291970,0irq_work/2311rcuc/222:40:182
291970,0irq_work/2311rcuc/219:58:212
291960,0irq_work/2311rcuc/222:13:562
291960,0irq_work/2311rcuc/220:53:412
291960,0irq_work/2311rcuc/219:32:412
291950,0irq_work/2311rcuc/221:03:472
291940,0irq_work/2311rcuc/220:48:212
291940,0irq_work/2311rcuc/200:17:462
291930,0irq_work/2311rcuc/221:33:412
291930,0irq_work/2311rcuc/221:33:412
103350930,0irq/28-eth0-tx-7650irq/26-ahci[00021:23:330
291920,0irq_work/2311rcuc/222:48:542
291920,0irq_work/2311rcuc/221:44:102
291920,0irq_work/2311rcuc/221:13:422
291920,0irq_work/2311rcuc/221:08:222
291910,0irq_work/2311rcuc/221:58:382
221910,0irq_work/1241rcuc/119:28:321
291900,0irq_work/2311rcuc/221:27:492
291900,0irq_work/2311rcuc/200:09:452
291900,0irq_work/2311rcuc/200:05:032
181890,0irq_work/0161rcuc/021:58:210
291880,0irq_work/2311rcuc/222:19:532
291880,0irq_work/2311rcuc/222:09:022
291870,0irq_work/2311rcuc/220:08:162
291870,0irq_work/2311rcuc/200:27:462
291860,0irq_work/2311rcuc/223:40:082
291860,0irq_work/2311rcuc/220:28:152
291860,0irq_work/2311rcuc/220:28:152
291850,0irq_work/2311rcuc/223:58:422
291840,0irq_work/2311rcuc/223:53:122
291840,0irq_work/2311rcuc/219:41:562
2172999842,0cyclictest210099rtkit-daemon20:08:583
291830,0irq_work/2311rcuc/223:43:512
291830,0irq_work/2311rcuc/223:43:512
291830,0irq_work/2311rcuc/220:08:402
291830,0irq_work/2311rcuc/219:52:242
2172999831,0cyclictest210099rtkit-daemon19:44:373
291820,0irq_work/2311rcuc/221:38:552
291820,0irq_work/2311rcuc/221:02:362
291810,0irq_work/2311rcuc/222:31:452
291810,0irq_work/2311rcuc/220:17:432
291800,0irq_work/2311rcuc/219:13:352
291780,0irq_work/2311rcuc/220:48:382
291780,0irq_work/2311rcuc/220:39:202
291780,0irq_work/2311rcuc/220:33:192
291780,0irq_work/2311rcuc/220:33:192
291760,0irq_work/2311rcuc/200:33:402
221760,0irq_work/1241rcuc/122:38:491
221760,0irq_work/1241rcuc/122:38:491
291720,0irq_work/2311rcuc/200:18:432
181700,0irq_work/0161rcuc/019:28:530
103350700,0irq/28-eth0-tx-7650irq/26-ahci[00021:40:330
2172999686,0cyclictest3799migration/321:23:463
2172999676,0cyclictest3799migration/319:13:383
2172999666,0cyclictest3799migration/321:30:553
2172999666,0cyclictest3799migration/321:30:553
181660,0irq_work/0161rcuc/021:03:380
2172999655,0cyclictest3799migration/319:53:503
2172999654,0cyclictest3799migration/322:17:143
181630,0irq_work/0161rcuc/022:21:040
2172999628,0cyclictest3799migration/321:22:133
2172999626,0cyclictest3799migration/323:48:543
361610,0irq_work/3381rcuc/323:15:193
2172999615,0cyclictest3799migration/320:16:023
103350600,0irq/28-eth0-tx-7650irq/26-ahci[00000:06:330
181590,0irq_work/0161rcuc/023:03:480
181590,0irq_work/0161rcuc/020:53:410
181590,0irq_work/0161rcuc/019:38:490
2172999582,0cyclictest28712-21grep23:18:363
2172999573,0cyclictest28330-21munin-node21:58:523
2172999573,0cyclictest26772-21cpuspeed_turbos20:38:383
181570,0irq_work/0161rcuc/023:58:480
181570,0irq_work/0161rcuc/022:26:240
181570,0irq_work/0161rcuc/021:50:400
141570,0rcu_preempt25-21ksoftirqd/121:38:211
141570,0rcu_preempt25-21ksoftirqd/121:38:211
141570,0rcu_preempt23748-21timerandwakeup20:28:533
141570,0rcu_preempt23748-21timerandwakeup20:28:533
221560,0irq_work/1241rcuc/119:23:471
221560,0irq_work/1241rcuc/100:13:151
2172999565,0cyclictest3799migration/321:03:173
2172999563,0cyclictest21206-21munin-node21:43:333
181560,0irq_work/0161rcuc/023:28:210
181560,0irq_work/0161rcuc/020:18:110
141550,0rcu_preempt39-21ksoftirqd/323:28:563
221540,0irq_work/1241rcuc/123:08:561
221540,0irq_work/1241rcuc/121:18:451
2172999542,0cyclictest26408-21munin-run20:38:203
21729995414,0cyclictest13827-21grep23:58:553
181540,0irq_work/0161rcuc/000:24:290
221530,0irq_work/1241rcuc/123:48:491
221530,0irq_work/1241rcuc/121:43:471
221530,0irq_work/1241rcuc/120:28:511
221530,0irq_work/1241rcuc/120:28:511
2172999533,0cyclictest5598-21unixbench_singl23:38:583
2172999533,0cyclictest31929-21switchtime20:48:483
2172999532,0cyclictest5189-21gltestperf21:03:383
2172999532,0cyclictest3799migration/321:41:583
2172999532,0cyclictest1920-21tr19:38:353
181530,0irq_work/0161rcuc/023:56:510
181530,0irq_work/0161rcuc/023:48:260
181530,0irq_work/0161rcuc/023:18:220
141530,0rcu_preempt25-21ksoftirqd/122:43:441
141530,0rcu_preempt25-21ksoftirqd/120:58:501
361520,0irq_work/3381rcuc/323:08:443
2172999523,0cyclictest6312-21cpuspeed_turbos23:43:353
2172999523,0cyclictest6312-21cpuspeed_turbos23:43:353
21729995225,0cyclictest8947-21grep22:28:543
2172999522,0cyclictest0-21swapper/322:08:213
21729995213,0cyclictest21144-21fschecks_time00:18:383
2172999521,0cyclictest210099rtkit-daemon22:40:253
2172999521,0cyclictest210099rtkit-daemon22:40:253
2172999521,0cyclictest210099rtkit-daemon20:01:443
2172999520,0cyclictest3799migration/321:13:233
181520,0irq_work/0161rcuc/023:11:050
181520,0irq_work/0161rcuc/023:00:580
361510,0irq_work/3381rcuc/323:33:473
361510,0irq_work/3381rcuc/319:28:443
221510,0irq_work/1241rcuc/120:23:151
2172999513,0cyclictest16734-21cut00:08:393
2172999513,0cyclictest11735-21grep23:53:543
2172999512,0cyclictest831-17auditd19:18:343
2172999512,0cyclictest831-17auditd19:18:343
2172999512,0cyclictest3799migration/300:05:053
2172999512,0cyclictest21676-21timerandwakeup20:23:513
2172999512,0cyclictest21676-21timerandwakeup20:23:513
2172999512,0cyclictest13575-21unixbench_multi20:03:563
2172999511,0cyclictest210099rtkit-daemon21:51:023
181510,0irq_work/0161rcuc/019:16:390
141510,0rcu_preempt39-21ksoftirqd/300:13:393
141510,0rcu_preempt25-21ksoftirqd/121:08:531
141510,0rcu_preempt21282-21unixbench_multi22:58:541
141510,0rcu_preempt0-21swapper/319:07:393
361500,0irq_work/3381rcuc/319:08:363
221500,0irq_work/1241rcuc/122:28:201
2172999503,0cyclictest3799migration/320:47:393
2172999503,0cyclictest26621-21timerwakeupswit21:53:583
2172999503,0cyclictest0-21swapper/323:27:283
2172999502,0cyclictest3799migration/300:30:513
2172999502,0cyclictest31701-21tr22:08:403
2172999502,0cyclictest20590-21kernelversion22:58:413
2171999502,0cyclictest210099rtkit-daemon23:38:510
2171999501,0cyclictest210099rtkit-daemon00:10:010
181500,0irq_work/0161rcuc/021:29:580
181500,0irq_work/0161rcuc/021:29:580
141500,0rcu_preempt8344-21missed_timers22:28:430
141500,0rcu_preempt39-21ksoftirqd/322:43:533
141500,0rcu_preempt25-21ksoftirqd/121:15:131
141500,0rcu_preempt25-21ksoftirqd/120:23:431
141500,0rcu_preempt25-21ksoftirqd/120:23:431
141500,0rcu_preempt16324-21runrttasks00:08:101
7650490,0irq/26-ahci[00021509-21ls19:08:210
241490,0rcuc/1141rcu_preempt19:16:221
2172999494,0cyclictest1234-21runrttasks22:23:503
21729994922,0cyclictest23858-21sendmail_mailst00:23:503
2172999492,0cyclictest847-21gltestperf20:53:373
2172999492,0cyclictest27050-21cpuspeed_turbos00:33:363
2172999492,0cyclictest22320-21fschecks_count23:03:473
2172999492,0cyclictest17579-21sed21:33:433
2172999492,0cyclictest17579-21sed21:33:433
2172999492,0cyclictest16068-21fschecks_count22:48:373
21729994912,0cyclictest29293-21unixbench_multi19:23:533
2171999491,0cyclictest210099rtkit-daemon00:31:440
181490,0irq_work/0161rcuc/022:56:150
181490,0irq_work/0161rcuc/022:44:480
181490,0irq_work/0161rcuc/020:00:320
181490,0irq_work/0161rcuc/019:50:210
161490,0rcuc/0141rcu_preempt00:21:520
141490,0rcu_preempt25-21ksoftirqd/122:48:511
141490,0rcu_preempt0-21swapper/020:22:020
2172999483,0cyclictest6127-21cpuspeed_turbos19:48:343
2172999482,0cyclictest6837-21tr21:08:363
2172999482,0cyclictest604-21proc_pri19:33:473
2172999482,0cyclictest4753-21grep22:18:533
141480,0rcu_preempt25-21ksoftirqd/121:58:381
141480,0rcu_preempt13-21ksoftirqd/022:13:490
141480,0rcu_preempt13-21ksoftirqd/019:08:460
221470,0irq_work/1241rcuc/120:48:221
221470,0irq_work/1241rcuc/100:28:511
2172999472,0cyclictest17991-21tr22:53:343
21729994711,0cyclictest9998-21fschecks_time22:33:373
2172699472,0cyclictest210099rtkit-daemon00:33:461
181470,0irq_work/0161rcuc/022:48:470
181470,0irq_work/0161rcuc/019:56:290
141470,0rcu_preempt25-21ksoftirqd/123:43:541
141470,0rcu_preempt25-21ksoftirqd/123:43:541
141470,0rcu_preempt25-21ksoftirqd/123:18:501
141470,0rcu_preempt25-21ksoftirqd/122:03:411
141470,0rcu_preempt25-21ksoftirqd/121:29:381
141470,0rcu_preempt25-21ksoftirqd/121:29:381
141470,0rcu_preempt25-21ksoftirqd/120:38:451
141470,0rcu_preempt25-21ksoftirqd/120:33:211
141470,0rcu_preempt25-21ksoftirqd/120:08:461
141470,0rcu_preempt25-21ksoftirqd/119:43:531
141470,0rcu_preempt25-21ksoftirqd/119:33:221
141470,0rcu_preempt18398-21kworker/1:000:13:371
311460,0rcuc/220143-21grep19:03:372
221460,0irq_work/1241rcuc/123:28:581
2172999462,0cyclictest18862-21gltestperf20:18:373
2172699461,0cyclictest210099rtkit-daemon23:42:221
141460,0rcu_preempt28201-21timerwakeupswit00:33:540
141460,0rcu_preempt25-21ksoftirqd/122:53:511
141460,0rcu_preempt25-21ksoftirqd/122:28:481
141460,0rcu_preempt25-21ksoftirqd/122:13:521
141460,0rcu_preempt25-21ksoftirqd/121:23:441
141460,0rcu_preempt25-21ksoftirqd/119:58:401
141460,0rcu_preempt25-21ksoftirqd/119:53:361
141460,0rcu_preempt25-21ksoftirqd/119:18:511
141460,0rcu_preempt25-21ksoftirqd/119:18:511
241450,0rcuc/120575-21nfs_client19:03:431
221450,0irq_work/1241rcuc/123:28:211
2171999451,0cyclictest210099rtkit-daemon23:34:550
181450,0irq_work/0161rcuc/019:45:500
181450,0irq_work/0161rcuc/019:33:470
141450,0rcu_preempt25-21ksoftirqd/123:53:501
141450,0rcu_preempt25-21ksoftirqd/123:03:521
141450,0rcu_preempt25-21ksoftirqd/120:53:221
141450,0rcu_preempt25-21ksoftirqd/119:08:501
2172699447,0cyclictest2399migration/100:27:541
2172699446,0cyclictest2399migration/100:18:381
2171999445,0cyclictest1799migration/021:33:590
2171999445,0cyclictest1799migration/021:33:590
181440,0irq_work/0161rcuc/021:58:370
181440,0irq_work/0161rcuc/020:31:040
181440,0irq_work/0161rcuc/020:31:040
181440,0irq_work/0161rcuc/019:27:380
141440,0rcu_preempt25-21ksoftirqd/123:13:591
141440,0rcu_preempt25-21ksoftirqd/122:38:211
2172699435,0cyclictest2399migration/120:16:011
2172699430,0cyclictest2399migration/121:39:011
181430,0irq_work/0161rcuc/023:31:550
181430,0irq_work/0161rcuc/022:12:140
181430,0irq_work/0161rcuc/021:10:090
181430,0irq_work/0161rcuc/020:08:490
141430,0rcu_preempt13495-21timerwakeupswit20:03:551
221420,0irq_work/1241rcuc/123:58:501
141420,0rcu_preempt2556-21latency19:38:431
141420,0rcu_preempt25-21ksoftirqd/123:33:461
141420,0rcu_preempt25-21ksoftirqd/119:48:361
181410,0irq_work/0161rcuc/020:48:210
141410,0rcu_preempt25-21ksoftirqd/121:53:461
141410,0rcu_preempt25-21ksoftirqd/121:03:531
141410,0rcu_preempt25-21ksoftirqd/120:43:531
2171999405,0cyclictest1799migration/022:38:350
2171999405,0cyclictest1799migration/022:38:350
181400,0irq_work/0161rcuc/020:03:470
141400,0rcu_preempt25-21ksoftirqd/122:11:541
141400,0rcu_preempt25-21ksoftirqd/121:48:591
141400,0rcu_preempt0-21swapper/122:18:411
21719993927,0cyclictest29724-21munin-node22:03:410
2171999392,0cyclictest1799migration/023:13:430
181390,0irq_work/0161rcuc/023:43:590
181390,0irq_work/0161rcuc/023:43:590
181390,0irq_work/0161rcuc/022:37:070
141390,0rcu_preempt26057-21taskset20:33:580
2171999382,0cyclictest21821-21unixbench_multi20:23:530
2171999382,0cyclictest21821-21unixbench_multi20:23:530
181380,0irq_work/0161rcuc/020:58:530
181380,0irq_work/0161rcuc/020:50:530
181380,0irq_work/0141rcu_preempt21:48:200
141380,0rcu_preempt26627-21taskset20:38:280
181370,0irq_work/0161rcuc/021:17:590
141370,0rcu_preempt13-21ksoftirqd/000:13:580
161340,0rcuc/0141rcu_preempt21:21:540
141340,0rcu_preempt13-21ksoftirqd/019:22:390
141340,0rcu_preempt13-21ksoftirqd/019:22:390
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional