You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-09-30 - 18:47
/usr/bin/Xorg /usr/bin/Xorg

x86 Intel Atom D2700 @2130 MHz, Linux 5.15.49-rt47 (Profile)

Latency plot of system in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot6.osadl.org (updated Fri Sep 30, 2022 12:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3611260,0irq_work/3381rcuc/310:16:493
1811010,0irq_work/0161rcuc/009:02:440
1811000,0irq_work/0161rcuc/012:02:440
181950,0irq_work/0161rcuc/009:51:400
181940,0irq_work/0161rcuc/009:36:450
181940,0irq_work/0161rcuc/008:37:130
181890,0irq_work/0161rcuc/011:07:120
181880,0irq_work/0161rcuc/009:27:110
181880,0irq_work/0161rcuc/009:27:110
181860,0irq_work/0161rcuc/011:57:100
181850,0irq_work/0161rcuc/008:27:110
181840,0irq_work/0161rcuc/012:32:440
181820,0irq_work/0161rcuc/011:12:100
181810,0irq_work/0161rcuc/010:22:120
103350800,0irq/28-eth0-tx-7650irq/26-ahci[00008:34:310
181790,0irq_work/0161rcuc/007:51:460
361770,0irq_work/3381rcuc/308:42:173
181770,0irq_work/0161rcuc/010:07:130
181770,0irq_work/0161rcuc/007:32:440
181760,0irq_work/0161rcuc/007:56:460
181760,0irq_work/0161rcuc/007:31:450
181750,0irq_work/0161rcuc/007:46:460
633899742,0cyclictest210099rtkit-daemon12:12:021
361740,0irq_work/3381rcuc/307:23:313
181740,0irq_work/0161rcuc/012:12:010
181740,0irq_work/0161rcuc/011:56:450
181740,0irq_work/0161rcuc/008:16:460
181740,0irq_work/0161rcuc/007:17:150
181730,0irq_work/0161rcuc/011:06:450
181730,0irq_work/0161rcuc/008:02:440
181720,0irq_work/0161rcuc/009:01:400
181700,0irq_work/0161rcuc/011:41:450
633899690,0cyclictest2399migration/111:51:211
633899690,0cyclictest2399migration/111:51:211
361690,0irq_work/3381rcuc/308:07:063
181690,0irq_work/0161rcuc/008:52:150
633899682,0cyclictest210099rtkit-daemon10:16:451
633899675,0cyclictest2399migration/109:41:281
633899664,0cyclictest2399migration/110:26:011
6338996522,0cyclictest6027-21latency12:17:051
291650,0irq_work/2311rcuc/211:37:082
291650,0irq_work/2311rcuc/209:46:462
181650,0irq_work/0161rcuc/011:47:130
181650,0irq_work/0161rcuc/011:47:130
181650,0irq_work/0161rcuc/009:17:120
181640,0irq_work/0161rcuc/010:47:140
181640,0irq_work/0161rcuc/010:32:080
181640,0irq_work/0161rcuc/010:32:080
181640,0irq_work/0161rcuc/010:17:100
181640,0irq_work/0161rcuc/010:16:450
633899635,0cyclictest2399migration/110:57:211
181630,0irq_work/0161rcuc/011:17:110
181630,0irq_work/0161rcuc/009:42:000
633899622,0cyclictest2399migration/108:34:151
291620,0irq_work/2311rcuc/211:01:462
181620,0irq_work/0161rcuc/011:26:460
633899612,0cyclictest2399migration/111:18:271
221610,0irq_work/1241rcuc/110:32:311
221610,0irq_work/1241rcuc/110:32:311
181610,0irq_work/0161rcuc/010:57:100
633899604,0cyclictest2399migration/111:22:131
633899602,0cyclictest2399migration/109:45:201
291600,0irq_work/2311rcuc/210:07:142
181600,0irq_work/0161rcuc/010:56:450
633899596,0cyclictest2399migration/110:41:351
633899596,0cyclictest2399migration/110:41:351
361590,0irq_work/3381rcuc/309:47:163
181590,0irq_work/0161rcuc/009:52:130
181590,0irq_work/0161rcuc/009:22:000
181590,0irq_work/0161rcuc/009:07:010
181590,0irq_work/0161rcuc/007:57:120
181590,0irq_work/0161rcuc/007:11:400
633899583,0cyclictest2399migration/109:56:531
6338995823,0cyclictest28827-21crond08:01:461
361580,0irq_work/3381rcuc/312:32:133
181580,0irq_work/0161rcuc/012:07:020
181580,0irq_work/0161rcuc/009:12:000
633899572,0cyclictest6866-21fschecks_count11:01:591
633899572,0cyclictest2399migration/108:26:491
181570,0irq_work/0161rcuc/008:21:460
181570,0irq_work/0161rcuc/008:21:460
181570,0irq_work/0161rcuc/008:11:400
6338995615,0cyclictest2025-21cat08:12:141
633899552,0cyclictest2399migration/108:22:531
633899552,0cyclictest2399migration/108:22:531
6338995516,0cyclictest15909-21http_loadtime08:47:031
6338995513,0cyclictest18380-21open_files08:52:121
633899550,0cyclictest2399migration/109:13:161
361550,0irq_work/3381rcuc/309:16:453
181550,0irq_work/0161rcuc/009:37:110
633899540,0cyclictest2399migration/112:33:351
633899540,0cyclictest2399migration/107:23:401
633899540,0cyclictest2399migration/107:15:291
633899540,0cyclictest2399migration/107:15:291
3799540,0migration/36151-21latency_hist07:06:453
291540,0irq_work/2311rcuc/211:31:562
633899533,0cyclictest2399migration/110:43:021
6338995315,0cyclictest23359-21sed07:47:011
6338995221,0cyclictest20178-21grep07:37:231
6338995218,0cyclictest19618-21cpuspeed_turbos08:56:581
6338995218,0cyclictest17323-21iostat07:32:051
6338995217,0cyclictest12435-21timerandwakeup08:37:171
181520,0irq_work/0161rcuc/010:01:450
6338995124,0cyclictest29384-21cpuspeed_turbos11:56:571
6338995116,0cyclictest29309-21fschecks_count08:01:591
6338995114,0cyclictest25108-21expr07:51:571
6338995113,0cyclictest8255-21fschecks_time09:47:001
633899510,0cyclictest2399migration/112:07:591
361510,0irq_work/3381rcuc/310:40:313
361510,0irq_work/3381rcuc/310:40:313
361510,0irq_work/3381rcuc/309:41:313
361510,0irq_work/3381rcuc/307:51:403
6338995019,0cyclictest1234-21runrttasks10:03:141
6338995017,0cyclictest21193-21tr11:36:581
6338995015,0cyclictest9840-21sed11:07:161
6338995013,0cyclictest1339-21Xorg09:36:441
633899501,0cyclictest210099rtkit-daemon07:44:301
633899500,0cyclictest2399migration/109:09:301
141500,0rcu_preempt32-21ksoftirqd/210:57:052
141500,0rcu_preempt0-21swapper/207:04:282
633899492,0cyclictest2399migration/112:29:191
6338994919,0cyclictest32323-21timerandwakeup08:07:161
6338994919,0cyclictest27555-21fschecks_time11:52:001
6338994918,0cyclictest12313-21crond07:21:451
6338994917,0cyclictest16283-21grep10:06:591
6338994914,0cyclictest31006-21switchtime09:22:141
6338994913,0cyclictest698-21cut10:47:021
361490,0irq_work/3381rcuc/307:17:173
291490,0irq_work/2311rcuc/209:12:062
161490,0rcuc/05338-21munin-node07:02:130
6338994820,0cyclictest13810-21gltestperf08:42:041
6338994815,0cyclictest8228-21needreboot12:22:101
6338994814,0cyclictest22965-21unixbench_multi09:02:171
6338994814,0cyclictest15604-21proc_pri07:27:131
6338994814,0cyclictest10118-21wc09:52:021
6338994811,0cyclictest20673-21fschecks_time10:17:021
291480,0irq_work/2311rcuc/212:07:092
241480,0rcuc/14512-21munin-node07:02:001
181480,0irq_work/0161rcuc/012:27:000
181480,0irq_work/0161rcuc/010:42:010
181480,0irq_work/0161rcuc/008:51:450
181480,0irq_work/0161rcuc/007:22:150
633999477,0cyclictest3099migration/212:14:302
633999477,0cyclictest3099migration/209:39:062
6338994716,0cyclictest28713-21ntpq09:17:111
6338994715,0cyclictest0-21swapper/111:27:121
6338994714,0cyclictest12442-21seq11:15:251
6338994713,0cyclictest25627-21cat10:27:151
361470,0irq_work/3381rcuc/312:17:313
291470,0irq_work/2311rcuc/210:17:202
291470,0irq_work/2311rcuc/209:04:392
181470,0irq_work/0161rcuc/008:12:110
6338994614,0cyclictest3946-21open_inodes08:17:111
361460,0irq_work/3381rcuc/310:52:093
181460,0irq_work/0161rcuc/012:22:120
181460,0irq_work/0161rcuc/012:21:450
181460,0irq_work/0161rcuc/011:41:590
181460,0irq_work/0161rcuc/011:41:590
181460,0irq_work/0161rcuc/007:37:140
634099457,0cyclictest3799migration/307:55:113
634099454,0cyclictest3799migration/310:45:583
633999458,0cyclictest3099migration/211:46:412
633999458,0cyclictest3099migration/211:46:412
633999458,0cyclictest3099migration/210:32:402
633999458,0cyclictest3099migration/210:32:402
633999458,0cyclictest3099migration/208:48:032
633999454,0cyclictest3099migration/209:00:482
633899453,0cyclictest2399migration/107:08:191
6338994516,0cyclictest2651-21expr10:51:591
361450,0irq_work/3381rcuc/312:01:453
141450,0rcu_preempt25-21ksoftirqd/111:43:311
141450,0rcu_preempt25-21ksoftirqd/111:43:311
634099447,0cyclictest3799migration/312:04:333
6340994433,0cyclictest9662-21proc_pri11:07:133
633999447,0cyclictest3099migration/211:18:332
633999447,0cyclictest3099migration/211:16:452
633999447,0cyclictest3099migration/207:24:512
633999446,0cyclictest3099migration/209:33:012
6338994419,0cyclictest31422-21cpu12:01:571
6338994418,0cyclictest19768-21missed_timers11:32:061
361440,0irq_work/3381rcuc/310:11:593
291440,0irq_work/2311rcuc/212:27:052
181440,0irq_work/0161rcuc/008:42:020
141440,0rcu_preempt3583-21sed10:52:162
141440,0rcu_preempt32-21ksoftirqd/212:21:582
634099435,0cyclictest3799migration/311:52:463
634099435,0cyclictest3799migration/311:32:203
634099435,0cyclictest3799migration/309:30:133
634099435,0cyclictest3799migration/309:30:133
634099434,0cyclictest3799migration/311:12:173
634099434,0cyclictest3799migration/308:24:473
634099434,0cyclictest3799migration/308:24:473
634099433,0cyclictest3799migration/308:39:523
634099432,0cyclictest3799migration/310:49:083
634099432,0cyclictest20085-21grep07:37:233
633999437,0cyclictest3099migration/209:52:062
633999437,0cyclictest3099migration/208:36:162
633999436,0cyclictest3099migration/207:48:252
633999434,0cyclictest3099migration/209:26:262
361430,0irq_work/3381rcuc/312:21:513
141430,0rcu_preempt32-21ksoftirqd/208:52:092
141430,0rcu_preempt10981-21gltestperf07:17:052
634099427,0cyclictest3799migration/311:48:393
634099427,0cyclictest3799migration/311:48:393
634099425,0cyclictest3799migration/310:27:543
634099425,0cyclictest3799migration/307:12:433
634099425,0cyclictest3799migration/307:12:433
634099424,0cyclictest3799migration/307:32:443
634099423,0cyclictest3799migration/312:15:163
6340994226,0cyclictest830-17auditd07:46:453
633999426,0cyclictest3099migration/208:15:102
633999425,0cyclictest3099migration/210:42:252
633999425,0cyclictest3099migration/209:57:042
633999425,0cyclictest3099migration/209:07:092
633999424,0cyclictest3099migration/208:19:112
633999423,0cyclictest3099migration/211:29:012
633999423,0cyclictest3099migration/210:14:362
6338994215,0cyclictest31969-21cpuspeed_turbos09:26:571
6338994215,0cyclictest31969-21cpuspeed_turbos09:26:571
291420,0irq_work/2311rcuc/211:22:132
291420,0irq_work/2311rcuc/210:47:182
181420,0irq_work/0161rcuc/007:15:310
181420,0irq_work/0161rcuc/007:15:310
141420,0rcu_preempt39-21ksoftirqd/309:46:453
141420,0rcu_preempt0-21swapper/209:17:142
634099416,0cyclictest3799migration/309:57:083
634099416,0cyclictest3799migration/309:34:213
634099414,0cyclictest3799migration/309:02:093
634099413,0cyclictest3799migration/310:57:593
633999415,0cyclictest3099migration/207:58:272
633999414,0cyclictest3099migration/209:27:592
633999414,0cyclictest3099migration/209:27:592
633999413,0cyclictest3099migration/207:37:542
6339994127,0cyclictest25595-21fschecks_time11:47:012
6339994127,0cyclictest25595-21fschecks_time11:47:012
633799414,0cyclictest1799migration/010:37:210
633799414,0cyclictest1799migration/010:37:210
361410,0irq_work/3381rcuc/309:52:013
361410,0irq_work/3381rcuc/308:12:123
361410,0irq_work/3381rcuc/307:27:093
141410,0rcu_preempt39-21ksoftirqd/308:02:143
141410,0rcu_preempt25584-21ls09:11:463
6340994025,0cyclictest6880-21grep07:07:023
633999403,0cyclictest3099migration/212:20:062
6339994028,0cyclictest25425-21gltestperf07:52:012
361400,0irq_work/3381rcuc/311:36:453
291400,0irq_work/2311rcuc/207:32:122
141400,0rcu_preempt39-21ksoftirqd/312:26:463
141400,0rcu_preempt32-21ksoftirqd/209:42:172
141400,0rcu_preempt15548-21kernelversion11:22:053
634099394,0cyclictest3799migration/312:07:033
634099394,0cyclictest3799migration/309:24:133
634099390,0cyclictest3799migration/309:01:453
633999395,0cyclictest3099migration/212:32:132
633999395,0cyclictest3099migration/207:12:142
633999395,0cyclictest3099migration/207:12:142
633999393,0cyclictest3099migration/208:46:232
6339993925,0cyclictest31895-21grep12:02:022
6337993924,0cyclictest24806-21cut10:27:010
633799392,0cyclictest210099rtkit-daemon11:29:560
361390,0irq_work/3381rcuc/310:02:163
291390,0irq_work/2311rcuc/210:39:202
291390,0irq_work/2311rcuc/210:39:202
291390,0irq_work/2311rcuc/210:24:542
291390,0irq_work/2311rcuc/207:10:322
141390,0rcu_preempt39-21ksoftirqd/311:42:063
141390,0rcu_preempt39-21ksoftirqd/311:42:063
141390,0rcu_preempt32-21ksoftirqd/211:52:002
634099381,0cyclictest3799migration/310:22:433
633799381,0cyclictest210099rtkit-daemon11:33:100
291380,0irq_work/2311rcuc/210:26:452
291380,0irq_work/2311rcuc/207:27:592
634099375,0cyclictest3799migration/311:02:163
634099374,0cyclictest3799migration/308:53:273
634099372,0cyclictest3799migration/308:36:303
633999372,0cyclictest3099migration/208:39:262
291370,0irq_work/2311rcuc/210:02:062
141370,0rcu_preempt39-21ksoftirqd/311:27:143
141370,0rcu_preempt17301-21switchtime10:07:163
7650360,0irq/26-ahci[00015920-21crond10:06:450
634099364,0cyclictest3799migration/309:21:013
633999362,0cyclictest3099migration/208:22:002
633999362,0cyclictest3099migration/208:22:002
634099353,0cyclictest3799migration/311:19:023
634099353,0cyclictest3799migration/310:32:013
634099353,0cyclictest3799migration/310:32:013
291350,0irq_work/2141rcu_preempt11:57:042
141350,0rcu_preempt9171-21gltestperf11:07:032
634099344,0cyclictest3799migration/307:57:333
634099342,0cyclictest3799migration/308:20:193
633999342,0cyclictest3099migration/208:04:112
361340,0irq_work/3141rcu_preempt08:27:153
311340,0rcuc/27897-21missed_timers08:27:072
291340,0irq_work/2311rcuc/207:42:002
6340993219,0cyclictest15527-21cat08:46:593
141310,0rcu_preempt32-21ksoftirqd/208:07:182
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional