You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-03 - 02:38
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot6.osadl.org (updated Sun Dec 03, 2023 00:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2911130,0irq_work/2311rcuc/222:26:442
2911110,0irq_work/2311rcuc/223:46:522
2911110,0irq_work/2311rcuc/223:46:522
1811080,0irq_work/0161rcuc/021:41:250
1811060,0irq_work/0161rcuc/019:16:450
1811030,0irq_work/0161rcuc/021:16:240
2911020,0irq_work/2311rcuc/221:02:102
291970,0irq_work/2311rcuc/220:03:062
181930,0irq_work/0161rcuc/023:27:030
361900,0irq_work/3381rcuc/321:23:453
361900,0irq_work/3381rcuc/321:23:453
181890,0irq_work/0161rcuc/000:12:510
181880,0irq_work/0161rcuc/019:06:450
291870,0irq_work/2311rcuc/200:22:062
291860,0irq_work/2311rcuc/219:26:462
181860,0irq_work/0161rcuc/021:51:250
291850,0irq_work/2311rcuc/221:21:502
291850,0irq_work/2311rcuc/221:21:502
181830,0irq_work/0161rcuc/000:07:080
221820,0irq_work/1241rcuc/121:36:261
361810,0irq_work/3381rcuc/323:56:243
291790,0irq_work/2311rcuc/223:11:262
103650780,0irq/28-eth0-tx-7650irq/26-ahci[00020:58:270
181750,0irq_work/0161rcuc/022:12:510
361740,0irq_work/3381rcuc/322:52:023
291740,0irq_work/2311rcuc/223:26:462
291740,0irq_work/2311rcuc/223:04:092
361730,0irq_work/3381rcuc/321:36:203
291730,0irq_work/2311rcuc/220:27:062
291730,0irq_work/2311rcuc/200:16:412
181730,0irq_work/0161rcuc/023:36:590
181730,0irq_work/0161rcuc/022:21:200
181730,0irq_work/0161rcuc/020:11:250
361720,0irq_work/3381rcuc/321:07:073
291720,0irq_work/2311rcuc/219:51:592
291720,0irq_work/2311rcuc/219:21:522
181710,0irq_work/0161rcuc/020:32:050
291700,0irq_work/2311rcuc/222:54:262
291700,0irq_work/2311rcuc/220:56:252
291700,0irq_work/2311rcuc/219:16:482
181700,0irq_work/0161rcuc/020:42:000
181700,0irq_work/0141rcu_preempt22:31:520
181700,0irq_work/0141rcu_preempt22:31:520
291690,0irq_work/2311rcuc/223:17:062
221690,0irq_work/1241rcuc/119:06:061
291680,0irq_work/2311rcuc/220:57:252
361670,0irq_work/3381rcuc/320:14:383
361670,0irq_work/3381rcuc/320:14:383
291670,0irq_work/2311rcuc/221:56:572
291670,0irq_work/2311rcuc/220:26:252
181670,0irq_work/0161rcuc/019:31:250
291660,0irq_work/2311rcuc/222:07:042
181660,0irq_work/0161rcuc/019:36:250
6117996556,0cyclictest29812-21cpu23:51:500
181650,0irq_work/0161rcuc/020:16:250
181650,0irq_work/0161rcuc/020:16:250
6117996442,0cyclictest16057-21crond23:21:250
221640,0irq_work/1241rcuc/120:16:261
221640,0irq_work/1241rcuc/120:16:261
181640,0irq_work/0161rcuc/020:51:250
611799633,0cyclictest218599rtkit-daemon23:08:440
291630,0irq_work/2311rcuc/221:32:012
6122996258,0cyclictest0-21swapper/123:09:591
291620,0irq_work/2311rcuc/222:32:052
291620,0irq_work/2311rcuc/222:32:052
291620,0irq_work/2311rcuc/219:31:492
291620,0irq_work/2311rcuc/200:32:062
181620,0irq_work/0161rcuc/019:51:250
181620,0irq_work/0161rcuc/000:26:250
6129996136,0cyclictest6461-21tr19:06:463
611799612,0cyclictest218599rtkit-daemon22:02:030
291610,0irq_work/2311rcuc/220:01:252
611799603,0cyclictest218599rtkit-daemon23:42:100
611799603,0cyclictest218599rtkit-daemon23:42:100
361600,0irq_work/3381rcuc/300:01:413
291600,0irq_work/2311rcuc/221:37:122
291600,0irq_work/2311rcuc/219:15:512
291600,0irq_work/2311rcuc/219:15:512
291600,0irq_work/2311rcuc/200:06:562
181600,0irq_work/0161rcuc/021:56:250
181600,0irq_work/0161rcuc/021:06:270
181600,0irq_work/0161rcuc/019:16:200
181600,0irq_work/0161rcuc/019:16:200
361590,0irq_work/3381rcuc/320:32:033
291590,0irq_work/2311rcuc/223:41:252
291590,0irq_work/2311rcuc/223:41:252
291590,0irq_work/2311rcuc/220:06:592
181590,0irq_work/0161rcuc/021:02:000
612999582,0cyclictest218599rtkit-daemon20:58:233
361580,0irq_work/3381rcuc/320:06:443
291580,0irq_work/2311rcuc/220:36:542
221580,0irq_work/1241rcuc/120:47:091
221580,0irq_work/1241rcuc/100:16:511
181580,0irq_work/0161rcuc/022:46:250
181580,0irq_work/0161rcuc/000:06:250
361570,0irq_work/3381rcuc/321:36:483
361570,0irq_work/3381rcuc/320:46:523
291570,0irq_work/2311rcuc/222:16:582
291570,0irq_work/2311rcuc/221:06:462
221570,0irq_work/1241rcuc/119:26:461
6122995646,0cyclictest23826-21needreboot20:57:001
612299560,0cyclictest2399migration/121:29:311
612299560,0cyclictest2399migration/121:29:311
291560,0irq_work/2311rcuc/222:42:512
221560,0irq_work/1241rcuc/123:23:361
181560,0irq_work/0161rcuc/022:11:250
181560,0irq_work/0161rcuc/021:56:560
181560,0irq_work/0161rcuc/021:27:020
181560,0irq_work/0161rcuc/021:27:020
181560,0irq_work/0161rcuc/020:27:040
161560,0rcuc/04881-21munin-node19:01:500
291550,0irq_work/2311rcuc/219:38:282
611799542,0cyclictest218599rtkit-daemon22:57:020
361540,0irq_work/3381rcuc/323:10:363
291540,0irq_work/2311rcuc/223:26:252
291540,0irq_work/2311rcuc/222:51:022
291540,0irq_work/2311rcuc/222:12:162
291540,0irq_work/2311rcuc/222:06:252
221540,0irq_work/1241rcuc/121:07:011
181540,0irq_work/0161rcuc/021:41:540
291530,0irq_work/2311rcuc/222:36:492
291530,0irq_work/2311rcuc/222:36:492
291530,0irq_work/2311rcuc/221:16:562
221530,0irq_work/1241rcuc/121:32:011
221530,0irq_work/1241rcuc/120:06:521
181530,0irq_work/0161rcuc/022:26:250
611799522,0cyclictest218599rtkit-daemon23:33:100
361520,0irq_work/3381rcuc/319:47:103
291520,0irq_work/2311rcuc/223:06:422
291520,0irq_work/2311rcuc/222:56:422
291520,0irq_work/2311rcuc/221:26:592
291520,0irq_work/2311rcuc/221:26:592
221520,0irq_work/1241rcuc/122:36:561
221520,0irq_work/1241rcuc/122:36:561
221520,0irq_work/1241rcuc/121:21:071
361510,0irq_work/3381rcuc/320:41:583
291510,0irq_work/2311rcuc/221:52:512
291510,0irq_work/2311rcuc/200:00:202
221510,0irq_work/1241rcuc/122:09:311
181510,0irq_work/0161rcuc/022:56:250
181510,0irq_work/0161rcuc/020:36:510
361500,0irq_work/3381rcuc/319:37:063
361500,0irq_work/3381rcuc/319:27:103
291500,0irq_work/2311rcuc/220:36:252
291500,0irq_work/2311rcuc/220:17:032
291500,0irq_work/2311rcuc/220:17:032
221500,0irq_work/1241rcuc/120:02:241
221500,0irq_work/1241rcuc/119:53:541
181500,0irq_work/0161rcuc/020:02:010
181500,0irq_work/0161rcuc/019:52:070
611799492,0cyclictest218599rtkit-daemon00:36:050
611799491,0cyclictest218599rtkit-daemon23:15:570
361490,0irq_work/3381rcuc/319:19:523
291490,0irq_work/2311rcuc/223:38:492
291490,0irq_work/2311rcuc/221:41:422
291490,0irq_work/2311rcuc/220:41:272
181490,0irq_work/0161rcuc/022:46:550
181490,0irq_work/0161rcuc/022:41:240
181490,0irq_work/0161rcuc/022:41:240
381480,0rcuc/351221processes19:01:543
291480,0irq_work/2311rcuc/220:13:502
291480,0irq_work/2311rcuc/220:13:502
241480,0rcuc/115033-21cat23:17:001
221480,0irq_work/1241rcuc/122:25:541
141480,0rcu_preempt25-21ksoftirqd/123:56:251
612999471,0cyclictest218599rtkit-daemon20:54:243
611799471,0cyclictest218599rtkit-daemon22:28:000
611799471,0cyclictest218599rtkit-daemon00:30:210
361470,0irq_work/3381rcuc/322:37:083
361470,0irq_work/3381rcuc/322:37:083
361470,0irq_work/3381rcuc/320:28:303
361470,0irq_work/3381rcuc/319:33:063
361470,0irq_work/3381rcuc/319:22:013
181470,0irq_work/0161rcuc/021:26:250
181470,0irq_work/0161rcuc/021:26:250
612699468,0cyclictest3099migration/223:52:582
361460,0irq_work/3381rcuc/323:24:283
311460,0rcuc/24891-21sort19:01:502
291460,0irq_work/2311rcuc/219:06:522
141460,0rcu_preempt32-21ksoftirqd/221:47:012
612699458,0cyclictest3099migration/221:15:012
611799451,0cyclictest218599rtkit-daemon23:04:540
381450,0rcuc/330747-21aten_r1power_vo20:01:443
361450,0irq_work/3381rcuc/323:46:553
361450,0irq_work/3381rcuc/323:46:553
291450,0irq_work/2311rcuc/220:49:312
221450,0irq_work/1241rcuc/121:21:561
221450,0irq_work/1241rcuc/121:21:561
181450,0irq_work/0161rcuc/020:52:010
141450,0rcu_preempt39-21ksoftirqd/322:57:033
612999441,0cyclictest218599rtkit-daemon21:13:183
612699444,0cyclictest3099migration/200:14:472
361440,0irq_work/3381rcuc/300:21:253
291440,0irq_work/2311rcuc/200:27:122
221440,0irq_work/1241rcuc/122:32:021
221440,0irq_work/1241rcuc/122:32:021
221440,0irq_work/1241rcuc/121:56:481
221440,0irq_work/1241rcuc/121:02:011
221440,0irq_work/1241rcuc/100:23:471
181440,0irq_work/0161rcuc/021:31:580
141440,0rcu_preempt13-21ksoftirqd/019:56:260
611799432,0cyclictest218599rtkit-daemon19:39:530
611799431,0cyclictest218599rtkit-daemon19:22:090
381430,0rcuc/30-21swapper/322:46:393
291430,0irq_work/2311rcuc/222:26:242
291430,0irq_work/2311rcuc/219:49:382
221430,0irq_work/1241rcuc/122:17:201
221430,0irq_work/1241rcuc/100:26:531
221430,0irq_work/1141rcu_preempt23:41:241
221430,0irq_work/1141rcu_preempt19:22:041
181430,0irq_work/0161rcuc/000:16:460
141430,0rcu_preempt39-21ksoftirqd/322:16:263
612299422,0cyclictest218599rtkit-daemon23:03:401
361420,0irq_work/3381rcuc/322:21:243
361420,0irq_work/3381rcuc/322:01:393
361420,0irq_work/3381rcuc/319:42:053
291420,0irq_work/2311rcuc/200:03:252
221420,0irq_work/1241rcuc/119:16:501
221420,0irq_work/1141rcu_preempt22:12:031
221420,0irq_work/1141rcu_preempt20:31:511
181420,0irq_work/0161rcuc/021:17:080
141420,0rcu_preempt39-21ksoftirqd/322:31:453
141420,0rcu_preempt39-21ksoftirqd/322:31:453
141420,0rcu_preempt11879-21sed21:41:583
612299414,0cyclictest2399migration/122:01:481
612299414,0cyclictest1439-21taskset00:02:001
611799412,0cyclictest9213-21crond20:26:250
221410,0irq_work/1241rcuc/123:14:461
221410,0irq_work/1141rcu_preempt23:31:471
221410,0irq_work/1141rcu_preempt20:26:251
221410,0irq_work/1141rcu_preempt20:01:261
221410,0irq_work/1141rcu_preempt19:11:251
221410,0irq_work/1141rcu_preempt00:01:241
6126994022,0cyclictest22721-21crond23:36:242
612699402,0cyclictest1211-21snmpd19:42:062
612299403,0cyclictest1255-21runrttasks21:42:051
6122994021,0cyclictest3189-21cut20:11:491
6122994021,0cyclictest3189-21cut20:11:491
612299402,0cyclictest218599rtkit-daemon00:14:081
6117994022,0cyclictest5417-21cut20:16:510
6117994022,0cyclictest5417-21cut20:16:510
381400,0rcuc/30-21swapper/323:32:093
361400,0irq_work/3381rcuc/323:36:523
221400,0irq_work/1241rcuc/120:51:461
221400,0irq_work/1141rcu_preempt23:47:111
221400,0irq_work/1141rcu_preempt23:47:111
221400,0irq_work/1141rcu_preempt23:42:111
221400,0irq_work/1141rcu_preempt23:42:111
221400,0irq_work/1141rcu_preempt22:56:421
221400,0irq_work/1141rcu_preempt22:46:241
221400,0irq_work/1141rcu_preempt21:47:081
221400,0irq_work/1141rcu_preempt19:36:531
221400,0irq_work/1141rcu_preempt19:16:251
221400,0irq_work/1141rcu_preempt19:16:251
181400,0irq_work/0161rcuc/023:57:250
181400,0irq_work/0161rcuc/019:42:010
141400,0rcu_preempt39-21ksoftirqd/321:31:253
141400,0rcu_preempt39-21ksoftirqd/321:31:253
141400,0rcu_preempt0-21swapper/323:16:403
103650400,0irq/28-eth0-tx-1274-21lldpd23:24:330
6122993926,0cyclictest10113-21grep20:26:521
361390,0irq_work/3381rcuc/323:29:453
221390,0irq_work/1241rcuc/123:27:061
221390,0irq_work/1241rcuc/122:51:531
221390,0irq_work/1141rcu_preempt21:56:261
221390,0irq_work/1141rcu_preempt20:36:461
221390,0irq_work/1141rcu_preempt19:36:251
221390,0irq_work/1141rcu_preempt00:31:541
141390,0rcu_preempt39-21ksoftirqd/321:16:463
612999384,0cyclictest3799migration/300:09:133
612999383,0cyclictest3799migration/300:27:293
361380,0irq_work/3381rcuc/322:24:313
361380,0irq_work/3381rcuc/321:46:453
361380,0irq_work/3381rcuc/300:21:523
221380,0irq_work/1141rcu_preempt22:26:421
221380,0irq_work/1141rcu_preempt20:41:481
221380,0irq_work/1141rcu_preempt19:46:531
221380,0irq_work/1141rcu_preempt19:42:021
221380,0irq_work/1141rcu_preempt00:11:251
141380,0rcu_preempt39-21ksoftirqd/319:57:013
612999375,0cyclictest3799migration/323:42:093
612999375,0cyclictest3799migration/323:42:093
612999374,0cyclictest3799migration/322:00:013
612999373,0cyclictest3799migration/322:26:293
611799371,0cyclictest218599rtkit-daemon23:50:410
611799371,0cyclictest218599rtkit-daemon23:50:410
361370,0irq_work/3381rcuc/323:58:003
361370,0irq_work/3381rcuc/323:13:513
221370,0irq_work/1141rcu_preempt21:12:011
141370,0rcu_preempt0-21swapper/321:01:473
141370,0rcu_preempt0-21swapper/320:37:563
103550370,0irq/27-eth0-rx-7238-21munin-node20:21:493
612999364,0cyclictest3799migration/322:06:473
612999364,0cyclictest3799migration/319:51:533
221360,0irq_work/1141rcu_preempt22:46:561
612999354,0cyclictest3799migration/321:51:513
612999352,0cyclictest3799migration/300:14:043
612999345,0cyclictest3799migration/300:31:483
612999344,0cyclictest3799migration/322:43:053
612999342,0cyclictest3799migration/323:02:533
361340,0irq_work/3381rcuc/319:11:453
361340,0irq_work/3381rcuc/319:11:453
612999333,0cyclictest3799migration/320:16:493
612999333,0cyclictest3799migration/320:16:493
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional