You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-12-09 - 16:43

x86 Intel Atom D2700 @2130 MHz, Linux 5.15.49-rt47 (Profile)

Latency plot of system in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot6.osadl.org (updated Fri Dec 09, 2022 12:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2911380,0irq_work/2311rcuc/209:09:522
2911370,0irq_work/2311rcuc/209:59:582
2911370,0irq_work/2311rcuc/209:59:582
2911350,0irq_work/2311rcuc/208:14:242
2911310,0irq_work/2311rcuc/207:29:252
2911260,0irq_work/2311rcuc/212:09:232
2911250,0irq_work/2311rcuc/207:24:512
2911240,0irq_work/2311rcuc/211:09:542
2911240,0irq_work/2311rcuc/211:09:542
2911230,0irq_work/2311rcuc/207:49:562
2911230,0irq_work/2311rcuc/207:49:562
2911220,0irq_work/2311rcuc/208:24:482
2911200,0irq_work/2311rcuc/209:54:512
2911180,0irq_work/2311rcuc/211:42:072
2911160,0irq_work/2311rcuc/209:24:412
2911150,0irq_work/2311rcuc/208:54:562
2911150,0irq_work/2311rcuc/208:54:562
2911140,0irq_work/2311rcuc/209:19:382
2911130,0irq_work/2311rcuc/212:29:482
2911120,0irq_work/2311rcuc/210:14:432
2911110,0irq_work/2311rcuc/212:26:362
2911110,0irq_work/2311rcuc/210:19:562
2911100,0irq_work/2311rcuc/210:09:582
2911100,0irq_work/2311rcuc/209:34:422
2911100,0irq_work/2311rcuc/208:06:232
2911090,0irq_work/2311rcuc/211:44:562
2911090,0irq_work/2311rcuc/210:49:252
2911090,0irq_work/2311rcuc/208:50:222
2911080,0irq_work/2311rcuc/209:30:232
2911080,0irq_work/2311rcuc/208:19:512
2911060,0irq_work/2311rcuc/207:23:102
2911040,0irq_work/2311rcuc/212:02:502
2911040,0irq_work/2311rcuc/209:44:482
2911040,0irq_work/2311rcuc/207:04:542
2911030,0irq_work/2311rcuc/212:10:052
2911030,0irq_work/2311rcuc/210:36:562
2911030,0irq_work/2311rcuc/209:14:442
2911020,0irq_work/2311rcuc/208:14:562
2911020,0irq_work/2311rcuc/207:17:012
2911010,0irq_work/2311rcuc/210:44:352
2911010,0irq_work/2311rcuc/209:49:512
291990,0irq_work/2311rcuc/212:19:562
291990,0irq_work/2311rcuc/207:44:242
291990,0irq_work/2311rcuc/207:34:462
291980,0irq_work/2311rcuc/211:23:212
291980,0irq_work/2311rcuc/210:28:562
291980,0irq_work/2311rcuc/207:14:242
291970,0irq_work/2311rcuc/211:38:192
291970,0irq_work/2311rcuc/207:59:412
291960,0irq_work/2311rcuc/208:39:492
291930,0irq_work/2311rcuc/212:14:502
291930,0irq_work/2311rcuc/212:14:502
291920,0irq_work/2311rcuc/211:49:512
291920,0irq_work/2311rcuc/210:59:562
291910,0irq_work/2311rcuc/211:30:562
291910,0irq_work/2311rcuc/210:04:432
291910,0irq_work/2311rcuc/210:04:432
291910,0irq_work/2311rcuc/209:39:552
291900,0irq_work/2311rcuc/211:54:592
291900,0irq_work/2311rcuc/211:24:512
291900,0irq_work/2311rcuc/208:30:212
291890,0irq_work/2311rcuc/211:15:212
291890,0irq_work/2311rcuc/211:04:582
291890,0irq_work/2311rcuc/209:04:402
291880,0irq_work/2311rcuc/208:35:002
291880,0irq_work/2311rcuc/207:54:442
291870,0irq_work/2311rcuc/208:59:562
291840,0irq_work/2311rcuc/210:43:442
291830,0irq_work/2311rcuc/210:54:532
361790,0irq_work/3381rcuc/312:24:503
291790,0irq_work/2311rcuc/210:32:382
291770,0irq_work/2311rcuc/208:44:492
291770,0irq_work/2311rcuc/207:45:562
361710,0irq_work/3381rcuc/308:34:493
221670,0irq_work/1241rcuc/111:54:241
221650,0irq_work/1241rcuc/109:14:511
221640,0irq_work/1241rcuc/111:29:521
221640,0irq_work/1241rcuc/108:39:191
221600,0irq_work/1241rcuc/109:54:231
361590,0irq_work/3381rcuc/308:29:453
1799590,0migration/025762-21cpu06:59:360
361580,0irq_work/3381rcuc/311:39:563
221580,0irq_work/1241rcuc/108:29:521
27797995511,0cyclictest830-17auditd08:55:230
27797995511,0cyclictest830-17auditd08:55:230
2779799551,0cyclictest210099rtkit-daemon11:59:520
221550,0irq_work/1241rcuc/110:44:241
2779799545,0cyclictest1799migration/009:24:480
221540,0irq_work/1241rcuc/107:39:231
141540,0rcu_preempt31927-21ps12:24:501
2779799532,0cyclictest851-17sedispatch10:39:240
2779799532,0cyclictest14910-21munin-node09:09:420
27797995319,0cyclictest648-21runrttasks09:52:330
181530,0irq_work/0161rcuc/011:44:480
141530,0rcu_preempt25-21ksoftirqd/109:59:231
2779799521,0cyclictest4625-21wget11:19:420
141520,0rcu_preempt26990-21ls06:59:503
141520,0rcu_preempt25-21ksoftirqd/107:44:531
2780199511,0cyclictest210099rtkit-daemon12:19:423
2780199511,0cyclictest210099rtkit-daemon11:29:103
2780199511,0cyclictest210099rtkit-daemon08:49:363
2780199511,0cyclictest210099rtkit-daemon08:06:293
2779799513,0cyclictest1799migration/012:18:340
2779799513,0cyclictest1799migration/012:18:340
141510,0rcu_preempt25-21ksoftirqd/111:19:391
2780199502,0cyclictest210099rtkit-daemon11:51:293
2780199501,0cyclictest210099rtkit-daemon09:12:563
2780199501,0cyclictest210099rtkit-daemon08:15:493
27797995024,0cyclictest20319-21munin-node08:04:420
2779799501,0cyclictest1799migration/007:24:420
241500,0rcuc/119373-21fschecks_time09:19:381
141500,0rcu_preempt22499-21tr08:09:380
2780199491,0cyclictest210099rtkit-daemon09:45:333
2779999496,0cyclictest2399migration/107:09:491
2779799493,0cyclictest1799migration/010:14:510
2779799492,0cyclictest1799migration/012:13:510
2779799492,0cyclictest1799migration/009:59:280
2779799492,0cyclictest1799migration/009:59:280
2779799491,0cyclictest21873-21munin-node10:44:440
141490,0rcu_preempt8750-21timerwakeupswit10:09:591
141490,0rcu_preempt39-21ksoftirqd/309:59:233
141490,0rcu_preempt13-21ksoftirqd/007:09:390
141490,0rcu_preempt0-21swapper/112:04:491
361480,0irq_work/3381rcuc/309:39:453
2780199482,0cyclictest210099rtkit-daemon09:05:003
2780199482,0cyclictest210099rtkit-daemon07:46:533
2780199481,0cyclictest210099rtkit-daemon09:32:123
2780199481,0cyclictest210099rtkit-daemon08:01:243
2779799484,0cyclictest1799migration/007:38:330
2779799483,0cyclictest1799migration/009:56:070
2779799483,0cyclictest1799migration/009:44:420
27797994822,0cyclictest20415-21tail09:19:550
27797994821,0cyclictest8435-21forks11:29:400
2779799482,0cyclictest0-21swapper/008:59:440
27797994818,0cyclictest6170-21cpu11:24:350
2779799481,0cyclictest210099rtkit-daemon10:29:300
221480,0irq_work/1241rcuc/112:19:511
221480,0irq_work/1241rcuc/109:29:251
181480,0irq_work/0161rcuc/007:29:370
141480,0rcu_preempt25-21ksoftirqd/108:04:571
141480,0rcu_preempt25-21ksoftirqd/107:14:521
141480,0rcu_preempt13-21ksoftirqd/011:09:490
141480,0rcu_preempt13-21ksoftirqd/011:09:490
361470,0irq_work/3381rcuc/310:14:183
2780199471,0cyclictest210099rtkit-daemon07:27:123
2779999471,0cyclictest210099rtkit-daemon10:22:111
2779799473,0cyclictest1799migration/011:55:050
27797994723,0cyclictest9257-21crond10:14:230
2779799472,0cyclictest1799migration/011:51:450
27797994717,0cyclictest19153-21netstat07:59:470
2779799471,0cyclictest13235-21taskset11:39:500
221470,0irq_work/1241rcuc/107:24:511
181470,0irq_work/0161rcuc/009:39:430
141470,0rcu_preempt13-21ksoftirqd/008:39:560
141470,0rcu_preempt13-21ksoftirqd/007:44:430
361460,0irq_work/3381rcuc/310:59:543
311460,0rcuc/226246-21cut06:59:392
2780199461,0cyclictest210099rtkit-daemon08:47:273
2780199461,0cyclictest210099rtkit-daemon07:50:273
2780199461,0cyclictest210099rtkit-daemon07:50:273
2779999466,0cyclictest2399migration/110:29:211
2779999465,0cyclictest2399migration/111:10:111
2779999465,0cyclictest2399migration/111:10:111
2779799463,0cyclictest210099rtkit-daemon07:21:460
2779799463,0cyclictest1799migration/009:29:410
2779799463,0cyclictest1799migration/007:16:000
27797994626,0cyclictest30871-21diskstats08:29:410
27797994622,0cyclictest14677-21wc07:49:400
27797994622,0cyclictest14677-21wc07:49:400
2779799462,0cyclictest25312-21needreboot08:14:490
2779799462,0cyclictest0-21swapper/012:24:540
2779799462,0cyclictest0-21swapper/010:49:490
27797994614,0cyclictest26436-21wc09:34:550
27797994614,0cyclictest1327-21latency12:29:480
27797994613,0cyclictest28557-21tail10:59:410
2779799461,0cyclictest22664-21munin-node12:04:450
2779799461,0cyclictest0-21swapper/010:39:500
241460,0rcuc/127325-21runrttasks07:01:171
181460,0irq_work/0161rcuc/009:18:590
141460,0rcu_preempt3907-21ls11:19:243
141460,0rcu_preempt25-21ksoftirqd/108:54:391
141460,0rcu_preempt25-21ksoftirqd/108:54:391
361450,0irq_work/3381rcuc/307:04:573
2780199451,0cyclictest210099rtkit-daemon09:27:013
27797994519,0cyclictest1234-21runrttasks12:22:490
27797994519,0cyclictest1234-21runrttasks08:37:270
2779799450,0cyclictest1799migration/010:21:500
221450,0irq_work/1241rcuc/110:59:501
221450,0irq_work/1241rcuc/107:59:501
141450,0rcu_preempt25-21ksoftirqd/111:49:241
141450,0rcu_preempt25-21ksoftirqd/109:04:551
141450,0rcu_preempt17579-21unixbench_multi07:54:573
2780199442,0cyclictest210099rtkit-daemon10:56:393
27797994421,0cyclictest26986-21gltestperf08:19:410
27797994420,0cyclictest10513-21fschecks_count11:34:400
2779799442,0cyclictest3394-21unixbench_multi11:14:590
2779799442,0cyclictest1234-21runrttasks11:09:020
27797994416,0cyclictest5454-21netstat08:44:520
27797994415,0cyclictest17273-21sensors_temp07:54:520
27797994413,0cyclictest26986-21proc_pri10:54:570
241440,0rcuc/127789-21cyclictest07:04:241
141440,0rcu_preempt25-21ksoftirqd/107:54:401
141440,0rcu_preempt25-21ksoftirqd/107:44:241
103250440,0irq/27-eth0-rx-860-21find11:09:543
103250440,0irq/27-eth0-rx-860-21find11:09:543
361430,0irq_work/3381rcuc/311:44:463
361430,0irq_work/3381rcuc/311:04:453
2780199431,0cyclictest210099rtkit-daemon12:18:493
2780199431,0cyclictest210099rtkit-daemon12:18:493
2779999436,0cyclictest2399migration/111:54:391
27797994325,0cyclictest0-21swapper/008:24:460
2779799432,0cyclictest12834-21systemd-cgroups09:04:250
27797994315,0cyclictest832-13audispd07:09:240
221430,0irq_work/1241rcuc/110:14:531
141430,0rcu_preempt25-21ksoftirqd/109:09:541
141430,0rcu_preempt21774-21sed10:44:243
361420,0irq_work/3381rcuc/312:06:403
2780199425,0cyclictest3799migration/309:34:423
2780199424,0cyclictest3799migration/308:54:403
2780199424,0cyclictest3799migration/308:54:403
2779999425,0cyclictest2399migration/111:25:061
2779999422,0cyclictest210099rtkit-daemon10:04:531
2779999422,0cyclictest210099rtkit-daemon10:04:531
2779799422,0cyclictest1799migration/008:49:330
27797994215,0cyclictest1234-21runrttasks10:04:440
27797994215,0cyclictest1234-21runrttasks10:04:440
27797994215,0cyclictest10529-21sh07:39:460
141420,0rcu_preempt25-21ksoftirqd/109:24:481
141420,0rcu_preempt25-21ksoftirqd/108:59:391
141420,0rcu_preempt25-21ksoftirqd/107:29:361
141420,0rcu_preempt2005-21fschecks_count07:19:463
141420,0rcu_preempt0-21swapper/010:24:410
381410,0rcuc/3141rcu_preempt11:20:143
361410,0irq_work/3381rcuc/311:34:513
2780199415,0cyclictest3799migration/310:30:273
2780199413,0cyclictest3799migration/307:39:283
2780199413,0cyclictest18671-21munin-node10:34:503
221410,0irq_work/1241rcuc/109:34:511
141410,0rcu_preempt25-21ksoftirqd/108:09:471
141410,0rcu_preempt0-21swapper/110:55:031
381400,0rcuc/3141rcu_preempt09:14:383
361400,0irq_work/3381rcuc/311:34:243
27801994026,0cyclictest24073-21grep10:49:423
27801994025,0cyclictest8451-21grep07:34:423
2780199402,0cyclictest3799migration/310:24:433
141400,0rcu_preempt3450-21grep11:14:591
141400,0rcu_preempt28086-21timerwakeupswit12:14:561
141400,0rcu_preempt28086-21timerwakeupswit12:14:561
141400,0rcu_preempt25-21ksoftirqd/112:10:041
141400,0rcu_preempt25-21ksoftirqd/109:39:441
141400,0rcu_preempt0-21swapper/111:39:561
2779999391,0cyclictest210099rtkit-daemon10:45:021
141390,0rcu_preempt25-21ksoftirqd/111:05:011
141390,0rcu_preempt2399migration/110:00:181
141390,0rcu_preempt2399migration/110:00:181
141390,0rcu_preempt2399migration/108:41:381
141390,0rcu_preempt0-21swapper/107:49:521
141390,0rcu_preempt0-21swapper/107:49:521
361380,0irq_work/3381rcuc/310:15:563
2779999383,0cyclictest2399migration/111:35:201
141380,0rcu_preempt39-21ksoftirqd/308:13:043
141380,0rcu_preempt25-21ksoftirqd/108:49:381
141380,0rcu_preempt25-21ksoftirqd/108:19:521
141380,0rcu_preempt24655-21latency10:49:501
381370,0rcuc/3614-21tr12:29:383
361370,0irq_work/3381rcuc/310:19:523
361370,0irq_work/3381rcuc/309:04:053
27801993724,0cyclictest897-21grep07:14:583
27801993723,0cyclictest3989-21kernelversion09:59:453
27801993723,0cyclictest3989-21kernelversion09:59:453
2780199372,0cyclictest20742-21grep11:59:363
2779999376,0cyclictest2399migration/108:45:091
27799993722,0cyclictest20736-21grep11:59:361
221370,0irq_work/1241rcuc/107:20:141
141370,0rcu_preempt39-21ksoftirqd/307:29:523
141370,0rcu_preempt25-21ksoftirqd/108:14:561
381360,0rcuc/30-21swapper/308:24:363
361360,0irq_work/3381rcuc/310:44:543
27799993620,0cyclictest19164-21grep10:34:581
141360,0rcu_preempt39-21ksoftirqd/310:04:483
141360,0rcu_preempt39-21ksoftirqd/310:04:483
141360,0rcu_preempt25545-21missed_timers12:09:553
141360,0rcu_preempt1751-21switchtime12:29:561
241350,0rcuc/129081-21grep08:24:401
141350,0rcu_preempt30077-21latency09:44:451
361340,0irq_work/3381rcuc/309:49:543
381330,0rcuc/330694-21cat07:09:483
27799993318,0cyclictest16426-21latency10:29:471
361320,0irq_work/3381rcuc/311:54:583
2780199324,0cyclictest0-21swapper/309:19:553
27801993218,0cyclictest3216-21cat08:39:473
2780199320,0cyclictest3799migration/308:19:413
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional