You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-12-05 - 03:33

x86 Intel Atom D2700 @2130 MHz, Linux 5.15.49-rt47 (Profile)

Latency plot of system in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot6.osadl.org (updated Mon Dec 05, 2022 00:43:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2211360,0irq_work/1241rcuc/122:14:321
2211360,0irq_work/1241rcuc/122:14:321
1811210,0irq_work/0161rcuc/023:34:320
1811150,0irq_work/0161rcuc/020:39:280
2911120,0irq_work/2311rcuc/219:39:022
2211120,0irq_work/1241rcuc/119:29:331
1811110,0irq_work/0161rcuc/000:19:490
2211100,0irq_work/1241rcuc/119:44:321
1811090,0irq_work/0161rcuc/020:19:580
2211080,0irq_work/1241rcuc/121:09:271
2211080,0irq_work/1241rcuc/121:09:271
2211070,0irq_work/1241rcuc/121:39:581
2211050,0irq_work/1241rcuc/120:20:001
2911040,0irq_work/2311rcuc/219:19:502
2211040,0irq_work/1241rcuc/123:35:001
2211040,0irq_work/1241rcuc/121:54:591
1811030,0irq_work/0161rcuc/021:39:470
2211020,0irq_work/1241rcuc/123:04:591
2911010,0irq_work/2311rcuc/223:34:472
3611000,0irq_work/3381rcuc/323:44:323
2211000,0irq_work/1241rcuc/120:54:321
2211000,0irq_work/1241rcuc/120:45:041
1811000,0irq_work/0161rcuc/020:00:320
1811000,0irq_work/0161rcuc/020:00:320
181990,0irq_work/0161rcuc/000:30:310
181990,0irq_work/0161rcuc/000:30:310
221980,0irq_work/1241rcuc/122:00:021
221980,0irq_work/1241rcuc/121:04:321
221980,0irq_work/1241rcuc/121:04:321
221980,0irq_work/1241rcuc/120:29:521
221960,0irq_work/1241rcuc/119:09:591
181960,0irq_work/0161rcuc/022:59:330
291940,0irq_work/2311rcuc/220:09:332
221940,0irq_work/1241rcuc/121:29:321
221930,0irq_work/1241rcuc/122:34:321
221930,0irq_work/1241rcuc/100:34:271
221930,0irq_work/1241rcuc/100:34:271
221920,0irq_work/1241rcuc/123:24:321
221920,0irq_work/1241rcuc/123:24:321
221910,0irq_work/1241rcuc/121:30:311
181910,0irq_work/0161rcuc/020:24:570
221900,0irq_work/1241rcuc/119:49:321
221900,0irq_work/1241rcuc/100:15:011
181900,0irq_work/0161rcuc/021:49:280
181900,0irq_work/0161rcuc/021:15:000
181900,0irq_work/0161rcuc/000:00:040
221890,0irq_work/1241rcuc/120:44:281
221890,0irq_work/1241rcuc/120:09:271
181890,0irq_work/0161rcuc/023:24:270
181890,0irq_work/0161rcuc/023:24:270
291880,0irq_work/2311rcuc/220:00:012
291880,0irq_work/2311rcuc/220:00:012
181880,0irq_work/0161rcuc/023:19:270
291870,0irq_work/2311rcuc/221:44:582
181870,0irq_work/0161rcuc/020:30:020
103350870,0irq/28-eth0-tx-7650irq/26-ahci[00021:50:040
221860,0irq_work/1241rcuc/123:55:001
221860,0irq_work/1241rcuc/122:20:001
221860,0irq_work/1241rcuc/122:14:591
221860,0irq_work/1241rcuc/122:14:591
221860,0irq_work/1241rcuc/120:35:011
221850,0irq_work/1241rcuc/119:34:331
181850,0irq_work/0161rcuc/022:20:040
291840,0irq_work/2311rcuc/200:10:062
221840,0irq_work/1241rcuc/120:14:271
221840,0irq_work/1241rcuc/119:35:021
221840,0irq_work/1241rcuc/100:05:071
181840,0irq_work/0161rcuc/021:00:040
181840,0irq_work/0161rcuc/021:00:040
221830,0irq_work/1241rcuc/122:39:491
181830,0irq_work/0161rcuc/019:24:280
361820,0irq_work/3381rcuc/321:44:473
181820,0irq_work/0161rcuc/020:09:590
181820,0irq_work/0161rcuc/019:44:570
291810,0irq_work/2311rcuc/222:00:012
181810,0irq_work/0161rcuc/023:49:320
181810,0irq_work/0161rcuc/023:44:320
181810,0irq_work/0161rcuc/020:44:550
181810,0irq_work/0161rcuc/019:50:000
181810,0irq_work/0161rcuc/000:14:500
291800,0irq_work/2311rcuc/222:10:052
291800,0irq_work/2311rcuc/222:10:052
291800,0irq_work/2311rcuc/221:50:042
291800,0irq_work/2311rcuc/200:04:332
221800,0irq_work/1241rcuc/122:55:331
221800,0irq_work/1241rcuc/121:44:571
221800,0irq_work/1241rcuc/120:15:021
181800,0irq_work/0161rcuc/022:29:270
291790,0irq_work/2311rcuc/220:04:482
221790,0irq_work/1241rcuc/123:30:021
221790,0irq_work/1241rcuc/122:50:051
221790,0irq_work/1241rcuc/121:19:561
221790,0irq_work/1241rcuc/119:19:581
221790,0irq_work/1241rcuc/100:09:491
181790,0irq_work/0161rcuc/023:09:480
181790,0irq_work/0161rcuc/019:55:320
181790,0irq_work/0161rcuc/019:55:320
181790,0irq_work/0161rcuc/019:30:030
291780,0irq_work/2311rcuc/223:45:022
221780,0irq_work/1241rcuc/119:05:011
181780,0irq_work/0161rcuc/022:09:270
181780,0irq_work/0161rcuc/021:25:000
361770,0irq_work/3381rcuc/322:50:093
181770,0irq_work/0161rcuc/021:30:030
181770,0irq_work/0161rcuc/021:10:000
361760,0irq_work/3381rcuc/323:04:333
291760,0irq_work/2311rcuc/223:24:322
291760,0irq_work/2311rcuc/223:24:322
291760,0irq_work/2311rcuc/221:24:452
221760,0irq_work/1241rcuc/123:49:331
221760,0irq_work/1241rcuc/123:39:571
103350760,0irq/28-eth0-tx-7650irq/26-ahci[00023:56:580
291750,0irq_work/2311rcuc/200:29:552
291750,0irq_work/2311rcuc/200:29:552
221750,0irq_work/1241rcuc/100:24:331
181750,0irq_work/0161rcuc/020:09:320
291740,0irq_work/2311rcuc/220:24:462
221740,0irq_work/1241rcuc/123:19:321
221740,0irq_work/1241rcuc/122:44:491
221740,0irq_work/1241rcuc/122:25:021
181740,0irq_work/0161rcuc/022:09:330
181740,0irq_work/0161rcuc/022:09:330
181740,0irq_work/0161rcuc/020:39:580
291720,0irq_work/2311rcuc/221:29:482
221720,0irq_work/1241rcuc/123:29:271
221720,0irq_work/1241rcuc/123:29:271
221720,0irq_work/1241rcuc/123:00:051
221720,0irq_work/1241rcuc/120:29:331
181720,0irq_work/0161rcuc/023:00:050
181720,0irq_work/0161rcuc/022:44:570
181720,0irq_work/0161rcuc/022:34:570
181720,0irq_work/0161rcuc/020:50:030
291710,0irq_work/2311rcuc/219:50:032
291710,0irq_work/2311rcuc/219:14:332
291710,0irq_work/2311rcuc/218:59:342
291710,0irq_work/2311rcuc/218:59:342
221710,0irq_work/1241rcuc/123:59:511
221710,0irq_work/1241rcuc/122:39:271
181710,0irq_work/0161rcuc/023:04:480
181710,0irq_work/0161rcuc/021:59:510
181710,0irq_work/0161rcuc/021:34:590
291700,0irq_work/2311rcuc/223:05:012
221700,0irq_work/1241rcuc/123:44:331
221700,0irq_work/1241rcuc/119:49:481
221700,0irq_work/1241rcuc/100:19:571
181700,0irq_work/0161rcuc/021:54:480
291690,0irq_work/2311rcuc/221:34:512
291690,0irq_work/2311rcuc/220:34:462
291690,0irq_work/2311rcuc/200:25:002
221690,0irq_work/1241rcuc/121:09:491
221690,0irq_work/1241rcuc/119:15:021
181690,0irq_work/0161rcuc/019:39:280
181690,0irq_work/0161rcuc/019:19:280
181690,0irq_work/0161rcuc/000:09:330
361680,0irq_work/3381rcuc/323:54:573
291680,0irq_work/2311rcuc/200:04:572
221680,0irq_work/1241rcuc/121:54:331
221680,0irq_work/1241rcuc/121:34:491
181680,0irq_work/0161rcuc/020:59:320
181670,0irq_work/0161rcuc/022:44:320
361660,0irq_work/3381rcuc/322:35:063
291650,0irq_work/2311rcuc/221:13:452
181650,0irq_work/0161rcuc/023:49:580
181650,0irq_work/0161rcuc/023:24:570
181650,0irq_work/0161rcuc/023:24:570
181650,0irq_work/0161rcuc/021:19:590
291640,0irq_work/2311rcuc/222:24:592
221640,0irq_work/1241rcuc/121:14:471
181640,0irq_work/0161rcuc/023:34:590
181640,0irq_work/0161rcuc/000:24:590
361630,0irq_work/3381rcuc/321:44:283
291630,0irq_work/2311rcuc/223:44:262
221630,0irq_work/1241rcuc/120:54:481
221630,0irq_work/1241rcuc/120:04:331
221630,0irq_work/1241rcuc/120:04:331
181630,0irq_work/0161rcuc/021:04:330
181630,0irq_work/0161rcuc/021:04:330
181630,0irq_work/0161rcuc/019:09:330
291620,0irq_work/2311rcuc/222:34:332
291620,0irq_work/2311rcuc/222:20:052
291620,0irq_work/2311rcuc/221:44:252
291620,0irq_work/2311rcuc/219:30:012
291620,0irq_work/2311rcuc/219:24:462
181620,0irq_work/0161rcuc/022:50:030
181620,0irq_work/0161rcuc/019:25:000
361610,0irq_work/3381rcuc/320:44:333
361610,0irq_work/3381rcuc/320:38:073
291610,0irq_work/2311rcuc/200:19:592
221610,0irq_work/1241rcuc/119:59:331
221610,0irq_work/1241rcuc/119:59:331
181610,0irq_work/0161rcuc/022:19:320
181610,0irq_work/0161rcuc/022:19:320
181610,0irq_work/0161rcuc/019:39:500
291600,0irq_work/2311rcuc/220:19:462
221600,0irq_work/1241rcuc/123:09:331
221600,0irq_work/1241rcuc/122:04:481
361590,0irq_work/3381rcuc/319:54:333
291590,0irq_work/2311rcuc/222:55:062
181590,0irq_work/0161rcuc/020:14:580
181590,0irq_work/0161rcuc/019:14:330
361580,0irq_work/3381rcuc/322:25:043
291570,0irq_work/2311rcuc/221:14:452
291570,0irq_work/2311rcuc/220:50:082
361560,0irq_work/3381rcuc/319:39:043
291560,0irq_work/2311rcuc/221:00:322
291560,0irq_work/2311rcuc/221:00:322
291560,0irq_work/2311rcuc/220:59:322
291560,0irq_work/2311rcuc/220:49:292
291550,0irq_work/2311rcuc/222:42:032
291550,0irq_work/2311rcuc/222:39:332
291550,0irq_work/2311rcuc/200:14:572
181550,0irq_work/0161rcuc/000:04:510
361540,0irq_work/3381rcuc/323:24:593
361540,0irq_work/3381rcuc/323:24:593
291540,0irq_work/2311rcuc/222:59:572
361530,0irq_work/3381rcuc/321:23:303
291530,0irq_work/2311rcuc/223:59:332
291530,0irq_work/2311rcuc/222:49:572
361520,0irq_work/3381rcuc/320:24:523
361520,0irq_work/3381rcuc/319:44:473
361520,0irq_work/3381rcuc/300:14:283
291520,0irq_work/2311rcuc/223:25:012
291520,0irq_work/2311rcuc/223:25:012
291520,0irq_work/2311rcuc/222:14:532
291520,0irq_work/2311rcuc/222:14:532
291520,0irq_work/2311rcuc/220:30:032
181520,0irq_work/0161rcuc/022:34:320
141520,0rcu_preempt0-21swapper/319:09:333
103350520,0irq/28-eth0-tx-1234-21runrttasks19:00:040
103350520,0irq/28-eth0-tx-1234-21runrttasks19:00:040
361510,0irq_work/3381rcuc/321:51:043
361510,0irq_work/3381rcuc/300:04:333
291510,0irq_work/2311rcuc/223:49:562
2735699511,0cyclictest210099rtkit-daemon20:55:123
141510,0rcu_preempt0-21swapper/319:01:473
141510,0rcu_preempt0-21swapper/319:01:473
291500,0irq_work/2311rcuc/223:14:522
141500,0rcu_preempt0-21swapper/119:02:561
141500,0rcu_preempt0-21swapper/119:02:561
361490,0irq_work/3381rcuc/319:56:153
361490,0irq_work/3381rcuc/319:56:153
291490,0irq_work/2311rcuc/221:22:542
2735699491,0cyclictest210099rtkit-daemon21:09:373
361480,0irq_work/3381rcuc/300:19:493
2735699481,0cyclictest210099rtkit-daemon21:05:043
2735699481,0cyclictest210099rtkit-daemon21:05:043
2735699481,0cyclictest210099rtkit-daemon20:52:373
2735599488,0cyclictest3099migration/219:55:202
2735599488,0cyclictest3099migration/219:55:202
2735599481,0cyclictest210099rtkit-daemon21:04:342
2735599481,0cyclictest210099rtkit-daemon21:04:342
141480,0rcu_preempt39-21ksoftirqd/319:30:043
291470,0irq_work/2311rcuc/220:14:332
291470,0irq_work/2311rcuc/219:04:492
2735699471,0cyclictest210099rtkit-daemon20:30:093
141470,0rcu_preempt39-21ksoftirqd/323:09:573
141470,0rcu_preempt39-21ksoftirqd/320:05:033
361460,0irq_work/3381rcuc/319:14:053
291460,0irq_work/2311rcuc/219:39:532
2735699461,0cyclictest210099rtkit-daemon20:47:053
2735699461,0cyclictest210099rtkit-daemon20:22:343
291450,0irq_work/2311rcuc/221:54:572
2735699451,0cyclictest210099rtkit-daemon21:00:343
2735699451,0cyclictest210099rtkit-daemon21:00:343
2735599457,0cyclictest3099migration/223:30:302
141450,0rcu_preempt0-21swapper/222:05:042
291440,0irq_work/2311rcuc/223:13:522
2735599445,0cyclictest3099migration/222:44:362
27355994429,0cyclictest32-21ksoftirqd/219:14:502
141440,0rcu_preempt39-21ksoftirqd/300:14:503
141440,0rcu_preempt24133-21latency_hist00:09:333
361430,0irq_work/3381rcuc/320:11:453
2735699434,0cyclictest210099rtkit-daemon23:29:363
2735699431,0cyclictest210099rtkit-daemon22:29:353
2735699431,0cyclictest210099rtkit-daemon21:14:473
141430,0rcu_preempt23486-21ls21:29:553
103250430,0irq/27-eth0-rx-1211-21snmpd22:45:333
361420,0irq_work/3381rcuc/322:20:413
2735699423,0cyclictest210099rtkit-daemon21:36:583
27356994228,0cyclictest729-21grep21:54:453
141420,0rcu_preempt39-21ksoftirqd/322:39:503
381410,0rcuc/30-21swapper/319:41:263
2735699411,0cyclictest210099rtkit-daemon23:05:283
141410,0rcu_preempt39-21ksoftirqd/321:24:483
141410,0rcu_preempt0-21swapper/323:53:073
361400,0irq_work/3381rcuc/323:15:043
27356994022,0cyclictest26933-21grep22:55:063
2735699401,0cyclictest210099rtkit-daemon23:34:393
2735699401,0cyclictest210099rtkit-daemon22:12:373
2735699401,0cyclictest210099rtkit-daemon22:12:373
2735699401,0cyclictest210099rtkit-daemon22:00:253
141400,0rcu_preempt32003-21grep19:14:513
361390,0irq_work/3381rcuc/300:25:013
291390,0irq_work/2311rcuc/219:44:442
2735699393,0cyclictest3799migration/319:24:553
2735699392,0cyclictest210099rtkit-daemon22:15:003
2735699392,0cyclictest210099rtkit-daemon22:15:003
2735699392,0cyclictest210099rtkit-daemon19:19:353
2735699391,0cyclictest210099rtkit-daemon23:21:013
2735699391,0cyclictest210099rtkit-daemon23:21:013
141390,0rcu_preempt39-21ksoftirqd/320:14:553
291380,0irq_work/2311rcuc/220:41:292
2735699372,0cyclictest210099rtkit-daemon23:45:333
361360,0irq_work/3381rcuc/300:30:013
361360,0irq_work/3381rcuc/300:30:013
2735699353,0cyclictest3799migration/322:04:583
381340,0rcuc/318504-21kernelversion19:59:553
381340,0rcuc/318504-21kernelversion19:59:553
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional