You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-26 - 15:31
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot6.osadl.org (updated Tue Sep 26, 2023 12:43:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2211420,0irq_work/1241rcuc/109:29:041
3611000,0irq_work/3381rcuc/308:05:373
361800,0irq_work/3381rcuc/309:49:293
361800,0irq_work/3381rcuc/309:49:293
181790,0irq_work/0161rcuc/008:03:590
1454599771,0cyclictest218599rtkit-daemon08:43:010
291760,0irq_work/2311rcuc/209:14:052
181750,0irq_work/0161rcuc/012:19:120
291720,0irq_work/2311rcuc/212:03:592
291720,0irq_work/2311rcuc/212:03:592
291670,0irq_work/2311rcuc/210:42:372
1455899673,0cyclictest218599rtkit-daemon08:04:041
361650,0irq_work/3381rcuc/309:04:033
181650,0irq_work/0161rcuc/010:58:550
181650,0irq_work/0161rcuc/010:58:550
141640,0rcu_preempt0-21swapper/207:03:562
221620,0irq_work/1241rcuc/112:28:361
361610,0irq_work/3381rcuc/310:28:373
181610,0irq_work/0161rcuc/010:13:370
1455899602,0cyclictest218599rtkit-daemon07:54:021
291590,0irq_work/2311rcuc/210:04:092
141590,0rcu_preempt13-21ksoftirqd/007:04:080
1454599586,0cyclictest218599rtkit-daemon09:23:320
361570,0irq_work/3381rcuc/309:54:003
361570,0irq_work/3381rcuc/309:54:003
181570,0irq_work/0161rcuc/010:46:030
14566995739,0cyclictest23967-21unixbench_multi08:39:162
361560,0irq_work/3381rcuc/311:39:083
181560,0irq_work/0161rcuc/007:14:110
1455899562,0cyclictest218599rtkit-daemon10:53:571
141560,0rcu_preempt13-21ksoftirqd/007:48:370
361550,0irq_work/3381rcuc/312:29:033
361550,0irq_work/3381rcuc/309:29:053
181550,0irq_work/0161rcuc/010:27:510
361540,0irq_work/3381rcuc/311:04:143
361540,0irq_work/3381rcuc/308:48:573
291540,0irq_work/2311rcuc/209:08:362
361530,0irq_work/3381rcuc/309:19:093
181530,0irq_work/0161rcuc/008:19:050
181530,0irq_work/0161rcuc/007:34:000
181530,0irq_work/0161rcuc/007:34:000
181530,0irq_work/0161rcuc/007:28:360
1455899531,0cyclictest218599rtkit-daemon07:39:001
141530,0rcu_preempt13-21ksoftirqd/008:11:300
361520,0irq_work/3381rcuc/310:00:523
361520,0irq_work/3381rcuc/308:23:413
221520,0irq_work/1241rcuc/110:08:361
291510,0irq_work/2311rcuc/210:52:132
291510,0irq_work/2311rcuc/210:09:172
181510,0irq_work/0161rcuc/009:24:380
1455899512,0cyclictest218599rtkit-daemon11:59:101
181500,0irq_work/0161rcuc/012:35:460
181500,0irq_work/0161rcuc/011:17:430
181500,0irq_work/0161rcuc/007:48:430
1455899507,0cyclictest2399migration/108:28:371
1455899506,0cyclictest2399migration/110:48:531
291490,0irq_work/2311rcuc/211:35:332
241490,0rcuc/112831-21munin-node07:04:001
181490,0irq_work/0161rcuc/010:57:460
1455899492,0cyclictest218599rtkit-daemon08:23:591
361480,0irq_work/3381rcuc/312:09:093
361480,0irq_work/3381rcuc/310:54:083
361480,0irq_work/3381rcuc/310:15:583
361480,0irq_work/3381rcuc/307:37:353
361480,0irq_work/3381rcuc/307:37:353
361480,0irq_work/3381rcuc/307:17:063
291480,0irq_work/2311rcuc/207:54:192
181480,0irq_work/0161rcuc/012:25:400
181480,0irq_work/0161rcuc/011:38:550
181480,0irq_work/0161rcuc/011:23:510
361470,0irq_work/3381rcuc/311:54:393
361470,0irq_work/3381rcuc/308:37:103
361470,0irq_work/3381rcuc/307:54:243
291470,0irq_work/2311rcuc/208:07:022
221470,0irq_work/1241rcuc/111:53:361
221470,0irq_work/1241rcuc/108:09:151
181470,0irq_work/0161rcuc/012:04:460
181470,0irq_work/0161rcuc/012:04:460
181470,0irq_work/0161rcuc/008:24:080
181470,0irq_work/0161rcuc/007:23:270
1455899478,0cyclictest2399migration/108:57:041
141470,0rcu_preempt39-21ksoftirqd/312:04:053
141470,0rcu_preempt39-21ksoftirqd/312:04:053
141470,0rcu_preempt21909-21perl09:49:020
141470,0rcu_preempt21909-21perl09:49:020
141470,0rcu_preempt20991-21if_eth008:34:042
181460,0irq_work/0161rcuc/010:41:480
181460,0irq_work/0161rcuc/009:58:560
181460,0irq_work/0161rcuc/008:58:450
181460,0irq_work/0161rcuc/008:16:540
181460,0irq_work/0161rcuc/007:56:080
181460,0irq_work/0161rcuc/007:11:110
14573994628,0cyclictest10795-21tr09:24:023
1455899467,0cyclictest2399migration/107:09:201
1455899466,0cyclictest2399migration/109:13:091
141460,0rcu_preempt32-21ksoftirqd/211:29:072
141460,0rcu_preempt0-21swapper/009:56:300
141460,0rcu_preempt0-21swapper/009:56:300
103550460,0irq/27-eth0-rx-12779-21cat07:04:003
361450,0irq_work/3381rcuc/309:18:333
361450,0irq_work/3381rcuc/309:03:033
291450,0irq_work/2311rcuc/210:14:142
291450,0irq_work/2311rcuc/209:40:062
221450,0irq_work/1241rcuc/112:33:061
181450,0irq_work/0161rcuc/011:22:590
181450,0irq_work/0161rcuc/011:09:020
181450,0irq_work/0161rcuc/008:55:190
181450,0irq_work/0161rcuc/008:34:180
1456699459,0cyclictest3099migration/211:08:282
1455899456,0cyclictest2399migration/110:28:561
1455899453,0cyclictest218599rtkit-daemon11:17:511
1455899451,0cyclictest218599rtkit-daemon10:24:051
141450,0rcu_preempt32-21ksoftirqd/207:24:172
141450,0rcu_preempt19414-21sed08:29:182
361440,0irq_work/3381rcuc/308:55:543
361440,0irq_work/3381rcuc/308:12:133
291440,0irq_work/2311rcuc/208:48:572
291440,0irq_work/2311rcuc/208:02:012
14573994431,0cyclictest30528-21grep10:08:573
1455899446,0cyclictest2399migration/111:10:551
1455899443,0cyclictest218599rtkit-daemon12:07:171
1455899443,0cyclictest218599rtkit-daemon12:07:171
1455899443,0cyclictest218599rtkit-daemon11:55:361
1455899441,0cyclictest218599rtkit-daemon11:22:421
1455899441,0cyclictest218599rtkit-daemon10:14:151
1455899441,0cyclictest218599rtkit-daemon09:23:551
381430,0rcuc/319748-21gltestperf09:44:023
361430,0irq_work/3381rcuc/308:03:043
361430,0irq_work/3381rcuc/307:44:123
291430,0irq_work/2311rcuc/212:12:512
291430,0irq_work/2311rcuc/210:02:272
181430,0irq_work/0161rcuc/012:28:370
181430,0irq_work/0161rcuc/012:14:150
1455899436,0cyclictest2399migration/108:52:171
1455899433,0cyclictest218599rtkit-daemon12:36:051
1455899432,0cyclictest5600-21missed_timers07:59:091
1455899431,0cyclictest218599rtkit-daemon10:42:101
141430,0rcu_preempt39-21ksoftirqd/311:48:553
141430,0rcu_preempt3679-21awk11:33:543
141430,0rcu_preempt32-21ksoftirqd/210:54:032
141430,0rcu_preempt13430-21wc10:43:383
141430,0rcu_preempt1255-21runrttasks11:35:151
381420,0rcuc/329273-21nscd11:19:013
361420,0irq_work/3381rcuc/310:03:383
311420,0rcuc/230957-21wget07:44:032
221420,0irq_work/1241rcuc/107:44:011
181420,0irq_work/0161rcuc/009:44:070
1455899426,0cyclictest2399migration/111:23:561
1455899426,0cyclictest2399migration/109:45:031
1455899425,0cyclictest2399migration/111:07:291
1455899425,0cyclictest2399migration/107:28:051
1455899423,0cyclictest218599rtkit-daemon07:36:261
1455899423,0cyclictest218599rtkit-daemon07:36:261
14558994223,0cyclictest9322-21grep10:33:551
14558994222,0cyclictest23288-21crond09:53:361
14558994222,0cyclictest23288-21crond09:53:361
1455899422,0cyclictest218599rtkit-daemon09:17:091
141420,0rcu_preempt39-21ksoftirqd/311:59:103
141420,0rcu_preempt25-21ksoftirqd/107:19:081
141420,0rcu_preempt12032-21awk08:14:043
381410,0rcuc/3141rcu_preempt08:44:093
381410,0rcuc/3141rcu_preempt08:44:093
361410,0irq_work/3381rcuc/312:14:093
311410,0rcuc/2141rcu_preempt11:03:002
311410,0rcuc/2141rcu_preempt11:03:002
291410,0irq_work/2311rcuc/210:32:322
241410,0rcuc/132144-21grep08:59:041
181410,0irq_work/0161rcuc/010:04:040
1457399414,0cyclictest3799migration/311:31:203
14566994129,0cyclictest31694-21grep12:34:162
1455899416,0cyclictest2399migration/112:10:391
14558994128,0cyclictest15540-21ls09:34:061
14558994124,0cyclictest9947-21crond09:23:371
1455899411,0cyclictest218599rtkit-daemon12:18:201
1455899411,0cyclictest218599rtkit-daemon09:54:521
1455899411,0cyclictest218599rtkit-daemon09:54:521
1455899411,0cyclictest218599rtkit-daemon07:29:561
1454599416,0cyclictest1799migration/007:39:070
141410,0rcu_preempt39-21ksoftirqd/310:29:013
141410,0rcu_preempt39-21ksoftirqd/308:32:373
141410,0rcu_preempt39-21ksoftirqd/307:19:033
141410,0rcu_preempt25-21ksoftirqd/108:44:321
141410,0rcu_preempt25-21ksoftirqd/108:44:321
141410,0rcu_preempt25-21ksoftirqd/107:13:571
291400,0irq_work/2311rcuc/211:19:432
291400,0irq_work/2311rcuc/210:34:252
1457399402,0cyclictest3799migration/312:24:553
14566994021,0cyclictest24020-21grep09:53:592
14566994021,0cyclictest24020-21grep09:53:592
1455899405,0cyclictest2399migration/112:19:161
1455899403,0cyclictest218599rtkit-daemon10:20:201
1455899401,0cyclictest218599rtkit-daemon11:29:301
1455899401,0cyclictest218599rtkit-daemon08:39:161
14545994022,0cyclictest8417-21grep10:29:200
141400,0rcu_preempt39-21ksoftirqd/312:33:553
141400,0rcu_preempt39-21ksoftirqd/307:39:063
141400,0rcu_preempt32-21ksoftirqd/212:29:072
141400,0rcu_preempt25-21ksoftirqd/108:36:181
361390,0irq_work/3381rcuc/310:39:383
361390,0irq_work/3381rcuc/310:19:453
291390,0irq_work/2311rcuc/208:58:572
181390,0irq_work/0161rcuc/011:29:070
161390,0rcuc/024934-21cpuspeed_turbos08:44:000
161390,0rcuc/024934-21cpuspeed_turbos08:44:000
14566993925,0cyclictest5114-21grep09:09:192
14558993924,0cyclictest14614-21netstat08:19:101
1455899391,0cyclictest218599rtkit-daemon11:43:561
1455899391,0cyclictest218599rtkit-daemon10:59:201
1455899391,0cyclictest218599rtkit-daemon10:59:201
14545993923,0cyclictest377-17jbd2/sda2-810:38:370
141390,0rcu_preempt39-21ksoftirqd/310:34:143
141390,0rcu_preempt39-21ksoftirqd/308:19:023
141390,0rcu_preempt32-21ksoftirqd/210:18:582
141390,0rcu_preempt32-21ksoftirqd/207:14:102
141390,0rcu_preempt22545-21hostname11:03:570
141390,0rcu_preempt17831-21missed_timers09:39:070
141390,0rcu_preempt15979-21switchtime07:09:122
141390,0rcu_preempt0-21swapper/311:18:373
103550390,0irq/27-eth0-rx-1272-21lldpd09:12:263
361380,0irq_work/3381rcuc/309:43:173
291380,0irq_work/2311rcuc/212:13:572
291380,0irq_work/2311rcuc/209:45:052
291380,0irq_work/2311rcuc/209:27:582
291380,0irq_work/2311rcuc/209:22:182
291380,0irq_work/2311rcuc/208:25:012
241380,0rcuc/10-21swapper/110:44:151
181380,0irq_work/0161rcuc/010:51:340
181380,0irq_work/0161rcuc/009:08:370
1456699386,0cyclictest3099migration/210:27:162
14566993827,0cyclictest7293-21grep11:39:182
1456699381,0cyclictest1255-21runrttasks12:24:402
1455899381,0cyclictest218599rtkit-daemon07:49:381
141380,0rcu_preempt25-21ksoftirqd/109:06:181
141380,0rcu_preempt13973-21fschecks_count08:19:002
141380,0rcu_preempt12875-21fschecks_count09:29:002
361370,0irq_work/3381rcuc/307:09:103
291370,0irq_work/2311rcuc/211:57:482
291370,0irq_work/2311rcuc/211:27:422
291370,0irq_work/2311rcuc/207:51:592
181370,0irq_work/0161rcuc/012:08:590
181370,0irq_work/0161rcuc/010:19:020
14573993730,0cyclictest381rcuc/310:49:013
1457399373,0cyclictest3799migration/311:12:523
14573993724,0cyclictest21603-21grep07:23:583
14566993723,0cyclictest14116-21grep10:44:022
14558993721,0cyclictest31765-21grep10:09:181
141370,0rcu_preempt39-21ksoftirqd/308:39:083
291360,0irq_work/2311rcuc/209:37:212
181360,0irq_work/0161rcuc/011:59:030
181360,0irq_work/0161rcuc/011:56:170
1457399364,0cyclictest3799migration/311:44:253
14566993624,0cyclictest22332-21grep09:49:092
14566993624,0cyclictest22332-21grep09:49:092
1456699362,0cyclictest0-21swapper/207:39:162
14558993618,0cyclictest12913-21grep08:14:201
14545993622,0cyclictest14898-21grep09:33:580
14545993619,0cyclictest5295-21grep07:59:030
141360,0rcu_preempt32-21ksoftirqd/208:53:372
361350,0irq_work/3381rcuc/311:24:053
181350,0irq_work/0161rcuc/011:34:070
181350,0irq_work/0161rcuc/009:13:430
161350,0rcuc/019693-21taskset08:30:020
14573993523,0cyclictest15022-21df09:33:593
1456699353,0cyclictest3099migration/207:32:292
14566993523,0cyclictest9536-21fschecks_count08:09:032
14566993521,0cyclictest26638-21crond08:48:362
14566993521,0cyclictest26638-21crond08:48:362
14558993519,0cyclictest27318-21grep09:59:181
1454599354,0cyclictest1799migration/010:16:480
14545993522,0cyclictest4886-21cat09:09:160
14545993521,0cyclictest12663-21grep09:28:570
14545993519,0cyclictest10534-21awk11:48:590
141350,0rcu_preempt9031-21munin-node11:44:110
141350,0rcu_preempt39-21ksoftirqd/310:59:083
141350,0rcu_preempt39-21ksoftirqd/310:59:083
381340,0rcuc/324028-21tune2fs12:18:583
381340,0rcuc/30-21swapper/307:28:463
291340,0irq_work/2311rcuc/211:17:402
291340,0irq_work/2311rcuc/208:14:222
14545993420,0cyclictest28127-21sendmail_mailst08:49:120
14545993416,0cyclictest23990-21sshd07:29:040
141340,0rcu_preempt39-21ksoftirqd/307:49:073
141340,0rcu_preempt25-21ksoftirqd/111:39:071
1456699335,0cyclictest3099migration/212:18:582
141330,0rcu_preempt32-21ksoftirqd/211:09:502
1456699323,0cyclictest3099migration/212:03:362
14566993222,0cyclictest19795-21sed07:19:012
14566993217,0cyclictest9275-21expr11:44:152
14566993119,0cyclictest10558-21fschecks_count11:48:592
14558993119,0cyclictest16908-21seq09:38:551
141310,0rcu_preempt0-21swapper/207:34:062
141310,0rcu_preempt0-21swapper/207:34:062
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional