You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-08 - 04:47
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot6.osadl.org (updated Fri Dec 08, 2023 00:43:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291850,0irq_work/2311rcuc/223:19:052
361790,0irq_work/3381rcuc/300:26:163
291750,0irq_work/2311rcuc/220:01:032
181730,0irq_work/0161rcuc/023:36:480
181730,0irq_work/0161rcuc/022:36:480
291720,0irq_work/2311rcuc/221:06:472
181700,0irq_work/0161rcuc/019:46:150
21367996849,0cyclictest8022sleep320:44:353
181670,0irq_work/0161rcuc/000:21:580
181660,0irq_work/0161rcuc/019:21:150
21361996534,0cyclictest30943-21taskset20:38:302
21361996534,0cyclictest30943-21taskset20:38:302
21367996411,0cyclictest1255-21runrttasks20:38:303
21367996411,0cyclictest1255-21runrttasks20:38:303
141640,0rcu_preempt1295-21Xorg19:46:321
21351996357,0cyclictest0-21swapper/021:36:370
291620,0irq_work/2311rcuc/220:03:102
181620,0irq_work/0161rcuc/020:42:420
291610,0irq_work/2311rcuc/221:02:032
181590,0irq_work/0161rcuc/000:16:560
141590,0rcu_preempt32-21ksoftirqd/221:56:162
141590,0rcu_preempt0-21swapper/119:03:021
221580,0irq_work/1241rcuc/122:51:511
221580,0irq_work/1241rcuc/122:51:511
221580,0irq_work/1241rcuc/120:31:151
221580,0irq_work/1241rcuc/120:06:501
181580,0irq_work/0161rcuc/023:41:510
361570,0irq_work/3381rcuc/322:26:533
2136799578,0cyclictest20044-21crond20:16:163
21367995713,0cyclictest0-21swapper/319:56:453
181570,0irq_work/0161rcuc/020:36:500
181570,0irq_work/0161rcuc/020:36:500
181570,0irq_work/0161rcuc/019:36:510
181560,0irq_work/0161rcuc/021:21:500
181550,0irq_work/0161rcuc/020:01:160
361540,0irq_work/3381rcuc/321:46:433
361540,0irq_work/3381rcuc/321:46:433
21367995419,0cyclictest0-21swapper/323:41:403
181540,0irq_work/0161rcuc/020:01:520
361530,0irq_work/3381rcuc/320:31:493
361530,0irq_work/3381rcuc/320:31:493
361530,0irq_work/3141rcu_preempt22:36:423
21361995338,0cyclictest1552-21timerwakeupswit00:22:052
21361995320,0cyclictest3099migration/223:27:522
181530,0irq_work/0161rcuc/019:51:560
141530,0rcu_preempt39-21ksoftirqd/321:11:413
141530,0rcu_preempt1312-21ps19:31:503
361520,0irq_work/3381rcuc/323:51:403
361520,0irq_work/3381rcuc/300:03:393
361520,0irq_work/3381rcuc/300:03:393
21367995232,0cyclictest9785-21grep19:51:423
21361995227,0cyclictest28038-21ntp_kernel_pll_20:31:492
21361995227,0cyclictest28038-21ntp_kernel_pll_20:31:492
21361995215,0cyclictest3099migration/221:11:422
21356995235,0cyclictest6911-21proc_pri23:21:511
361510,0irq_work/3381rcuc/323:06:313
361510,0irq_work/3381rcuc/322:32:413
361510,0irq_work/3381rcuc/321:51:503
361510,0irq_work/3381rcuc/321:18:123
361510,0irq_work/3381rcuc/319:20:273
361510,0irq_work/3381rcuc/300:19:353
291510,0irq_work/2311rcuc/219:29:322
291510,0irq_work/2311rcuc/219:29:322
21361995136,0cyclictest22014-21runrttasks22:43:412
21361995135,0cyclictest1255-21runrttasks00:15:232
181510,0irq_work/0161rcuc/022:06:540
181510,0irq_work/0161rcuc/019:31:150
181510,0irq_work/0161rcuc/019:31:150
181510,0irq_work/0161rcuc/000:32:000
141510,0rcu_preempt25-21ksoftirqd/122:01:331
141510,0rcu_preempt25-21ksoftirqd/122:01:151
141510,0rcu_preempt13-21ksoftirqd/020:56:450
381500,0rcuc/3141rcu_preempt19:26:353
381500,0rcuc/3141rcu_preempt19:26:353
361500,0irq_work/3381rcuc/322:01:203
291500,0irq_work/2311rcuc/223:22:522
2136799502,0cyclictest218599rtkit-daemon00:01:103
21361995016,0cyclictest3099migration/222:02:052
141500,0rcu_preempt25-21ksoftirqd/120:36:341
141500,0rcu_preempt25-21ksoftirqd/120:36:341
141500,0rcu_preempt13-21ksoftirqd/023:08:090
141500,0rcu_preempt0-21swapper/219:03:082
361490,0irq_work/3381rcuc/322:12:273
361490,0irq_work/3381rcuc/320:56:303
361490,0irq_work/3381rcuc/319:39:283
361490,0irq_work/3381rcuc/319:07:513
2136799492,0cyclictest30967-21sshd23:04:163
21361994918,0cyclictest3099migration/221:51:342
21361994916,0cyclictest3099migration/223:06:502
2135699493,0cyclictest2712-21cpuspeed_turbos00:26:391
181490,0irq_work/0161rcuc/023:31:490
161490,0rcuc/020450-21munin-node19:01:500
361480,0irq_work/3381rcuc/319:50:173
291480,0irq_work/2311rcuc/219:11:362
21367994810,0cyclictest5781-21munin-node00:31:593
21351994822,0cyclictest0-21swapper/020:11:530
181480,0irq_work/0161rcuc/021:31:520
141480,0rcu_preempt32-21ksoftirqd/223:31:552
141480,0rcu_preempt32-21ksoftirqd/220:16:562
141480,0rcu_preempt13-21ksoftirqd/021:46:400
141480,0rcu_preempt13-21ksoftirqd/021:46:400
361470,0irq_work/3381rcuc/321:31:173
361470,0irq_work/3381rcuc/319:12:223
21367994740,0cyclictest0-21swapper/322:16:293
21367994722,0cyclictest0-21swapper/321:37:323
2135699474,0cyclictest218599rtkit-daemon19:20:251
2135699471,0cyclictest218599rtkit-daemon19:14:451
2135199476,0cyclictest1799migration/022:35:120
21351994725,0cyclictest26081-21cpuspeed_turbos21:41:350
21351994725,0cyclictest26081-21cpuspeed_turbos21:41:350
181470,0irq_work/0161rcuc/022:26:520
181470,0irq_work/0161rcuc/020:26:530
181470,0irq_work/0161rcuc/000:06:570
141470,0rcu_preempt39-21ksoftirqd/323:26:413
141470,0rcu_preempt13-21ksoftirqd/019:21:430
141470,0rcu_preempt13-21ksoftirqd/019:21:430
141470,0rcu_preempt13-21ksoftirqd/019:11:540
361460,0irq_work/3381rcuc/323:34:493
361460,0irq_work/3381rcuc/322:06:523
361460,0irq_work/3381rcuc/320:01:543
221460,0irq_work/1241rcuc/123:01:461
21361994613,0cyclictest3099migration/221:31:432
2135699461,0cyclictest218599rtkit-daemon19:39:081
2135199467,0cyclictest1799migration/022:11:270
21351994639,0cyclictest9066-21ntp_kernel_pll_23:26:480
103550460,0irq/27-eth0-rx-28350-21sed22:56:563
103550460,0irq/27-eth0-rx-28350-21sed22:56:563
361450,0irq_work/3381rcuc/320:51:553
291450,0irq_work/2311rcuc/221:47:342
291450,0irq_work/2311rcuc/221:47:342
2135199458,0cyclictest1799migration/020:46:170
2135199457,0cyclictest2648-21cpuspeed_turbos00:26:390
161450,0rcuc/030850-21fschecks_time21:51:400
141450,0rcu_preempt39-21ksoftirqd/300:16:153
141450,0rcu_preempt32-21ksoftirqd/221:36:522
141450,0rcu_preempt25-21ksoftirqd/121:26:161
141450,0rcu_preempt25-21ksoftirqd/120:01:561
141450,0rcu_preempt13-21ksoftirqd/022:56:420
141450,0rcu_preempt13-21ksoftirqd/022:56:420
141450,0rcu_preempt13-21ksoftirqd/022:21:400
141450,0rcu_preempt0-21swapper/319:05:563
361440,0irq_work/3381rcuc/322:25:083
361440,0irq_work/3381rcuc/321:27:303
361440,0irq_work/3381rcuc/321:03:233
361440,0irq_work/3381rcuc/320:46:533
361440,0irq_work/3381rcuc/320:23:533
361440,0irq_work/3381rcuc/319:44:503
21367994411,0cyclictest28910-21munin-node19:21:413
21367994411,0cyclictest28910-21munin-node19:21:413
21361994412,0cyclictest3099migration/220:08:432
21361994412,0cyclictest3099migration/219:11:112
21361994411,0cyclictest3099migration/200:26:172
2135699441,0cyclictest218599rtkit-daemon19:30:581
2135699441,0cyclictest218599rtkit-daemon19:30:581
2135199444,0cyclictest1799migration/022:02:550
21351994413,0cyclictest19957-21kernelversion23:51:480
141440,0rcu_preempt8730-21smartctl21:01:421
141440,0rcu_preempt25-21ksoftirqd/122:06:481
141440,0rcu_preempt25-21ksoftirqd/100:31:541
141440,0rcu_preempt25-21ksoftirqd/100:02:041
141440,0rcu_preempt25-21ksoftirqd/100:02:041
141440,0rcu_preempt13-21ksoftirqd/019:36:160
361430,0irq_work/3381rcuc/322:42:513
361430,0irq_work/3381rcuc/320:18:453
2136799437,0cyclictest3799migration/300:21:493
21367994326,0cyclictest26118-21grep21:41:353
21367994326,0cyclictest26118-21grep21:41:353
21367994326,0cyclictest17223-21grep21:21:363
2136199439,0cyclictest3099migration/220:44:482
21361994332,0cyclictest11228-21latency_hist22:21:162
21361994319,0cyclictest3099migration/200:05:432
21361994319,0cyclictest3099migration/200:05:432
2136199431,0cyclictest218599rtkit-daemon00:09:392
2135699431,0cyclictest218599rtkit-daemon19:10:551
2135199436,0cyclictest1799migration/021:16:340
2135199434,0cyclictest1799migration/022:41:550
2135199433,0cyclictest1799migration/000:12:320
2135199431,0cyclictest218599rtkit-daemon19:09:110
181430,0irq_work/0161rcuc/022:16:490
141430,0rcu_preempt39-21ksoftirqd/323:46:423
141430,0rcu_preempt32-21ksoftirqd/223:01:372
141430,0rcu_preempt32-21ksoftirqd/219:21:492
141430,0rcu_preempt32-21ksoftirqd/219:21:492
141430,0rcu_preempt25-21ksoftirqd/121:36:401
141430,0rcu_preempt25-21ksoftirqd/120:21:431
381420,0rcuc/3141rcu_preempt21:56:483
361420,0irq_work/3381rcuc/323:22:413
361420,0irq_work/3381rcuc/323:19:073
361420,0irq_work/3381rcuc/320:26:423
221420,0irq_work/1241rcuc/123:31:501
2136799427,0cyclictest3799migration/320:06:503
21361994225,0cyclictest3512-21head19:36:482
21361994219,0cyclictest20052-21munin-run20:16:162
21361994214,0cyclictest3099migration/220:53:452
2135699423,0cyclictest218599rtkit-daemon21:10:001
2135699422,0cyclictest218599rtkit-daemon19:32:211
2135699421,0cyclictest218599rtkit-daemon23:47:151
2135199427,0cyclictest29741-21fschecks_time23:01:380
181420,0irq_work/0161rcuc/021:11:520
141420,0rcu_preempt39-21ksoftirqd/322:46:553
141420,0rcu_preempt32-21ksoftirqd/223:01:152
141420,0rcu_preempt32-21ksoftirqd/219:46:462
141420,0rcu_preempt25-21ksoftirqd/121:31:561
141420,0rcu_preempt25-21ksoftirqd/119:43:311
141420,0rcu_preempt13-21ksoftirqd/023:21:410
361410,0irq_work/3381rcuc/322:51:473
361410,0irq_work/3381rcuc/322:51:473
361410,0irq_work/3381rcuc/321:06:353
2136199415,0cyclictest0-21swapper/222:11:552
21361994121,0cyclictest25822-21munin-node21:41:562
21361994121,0cyclictest25822-21munin-node21:41:562
21361994111,0cyclictest3099migration/219:55:152
2135699412,0cyclictest218599rtkit-daemon20:55:451
2135699411,0cyclictest218599rtkit-daemon19:25:561
2135699411,0cyclictest218599rtkit-daemon19:25:561
2135199412,0cyclictest23288-21irqstats22:46:460
141410,0rcu_preempt7324-21entropy19:46:370
141410,0rcu_preempt13-21ksoftirqd/023:57:010
141410,0rcu_preempt13-21ksoftirqd/021:11:160
141410,0rcu_preempt13-21ksoftirqd/021:01:540
141410,0rcu_preempt13-21ksoftirqd/000:06:110
141410,0rcu_preempt13-21ksoftirqd/000:06:110
291400,0irq_work/2311rcuc/222:37:452
291400,0irq_work/2311rcuc/222:27:432
291400,0irq_work/2311rcuc/220:28:522
221400,0irq_work/1241rcuc/122:36:371
2136199403,0cyclictest3099migration/220:57:122
21361994023,0cyclictest17896-21tr23:46:502
21361994019,0cyclictest2104-21runrttasks19:34:572
21361994018,0cyclictest814-21rs:main2
21361994016,0cyclictest3099migration/222:56:432
21361994014,0cyclictest2215-21latency20:46:432
21361994011,0cyclictest0-21swapper/223:51:442
2135199403,0cyclictest1799migration/020:21:130
181400,0irq_work/0161rcuc/023:21:150
181400,0irq_work/0161rcuc/020:06:530
141400,0rcu_preempt25-21ksoftirqd/123:06:501
141400,0rcu_preempt25-21ksoftirqd/100:21:151
2136199395,0cyclictest3099migration/222:31:342
21361993922,0cyclictest6104-21runrttasks22:06:562
21361993921,0cyclictest3099migration/223:57:072
2135699391,0cyclictest218599rtkit-daemon22:28:151
141390,0rcu_preempt6763-21missed_timers20:56:451
141390,0rcu_preempt39-21ksoftirqd/323:36:543
141390,0rcu_preempt25-21ksoftirqd/123:41:561
141390,0rcu_preempt25-21ksoftirqd/120:46:331
141390,0rcu_preempt17080-21switchtime22:31:501
361380,0irq_work/3141rcu_preempt00:06:383
291380,0irq_work/2311rcuc/223:42:562
291380,0irq_work/2311rcuc/222:47:462
221380,0irq_work/1241rcuc/121:53:041
21361993822,0cyclictest19152-21munin-node21:26:432
21361993821,0cyclictest25260-21fschecks_time22:51:412
21361993821,0cyclictest25260-21fschecks_time22:51:412
21361993820,0cyclictest1211-21snmpd21:22:152
21361993819,0cyclictest3099migration/221:16:392
21361993819,0cyclictest27374-21ls19:16:572
21361993811,0cyclictest3099migration/200:17:142
2135699386,0cyclictest2399migration/119:54:571
2135699384,0cyclictest2399migration/123:36:391
2135699384,0cyclictest2399migration/122:16:371
2135699384,0cyclictest19340-21munin-node23:51:401
21356993824,0cyclictest19589-21grep20:11:561
2135699382,0cyclictest2399migration/122:21:341
2135699381,0cyclictest8363-21taskset23:26:361
141380,0rcu_preempt28071-21aten_r1power_en00:11:331
141380,0rcu_preempt27587-21grep22:56:421
141380,0rcu_preempt27587-21grep22:56:421
141380,0rcu_preempt25738-21netstat22:51:490
141380,0rcu_preempt25738-21netstat22:51:490
141380,0rcu_preempt25-21ksoftirqd/121:16:401
141380,0rcu_preempt1255-21runrttasks20:52:480
2136199373,0cyclictest218599rtkit-daemon20:22:092
21361993721,0cyclictest11428-21aten_r1power_fr22:21:352
2135699375,0cyclictest2399migration/121:29:101
2135699374,0cyclictest2399migration/121:11:491
2135699371,0cyclictest26644-21iostat21:41:431
2135699371,0cyclictest26644-21iostat21:41:431
181370,0irq_work/0161rcuc/023:46:550
141370,0rcu_preempt32-21ksoftirqd/200:31:482
141370,0rcu_preempt25-21ksoftirqd/122:46:441
21361993620,0cyclictest2823-21unixbench-2d23:11:552
21361993620,0cyclictest12677-21cpuspeed_turbos23:36:352
2135699365,0cyclictest2399migration/120:41:371
2135699364,0cyclictest2399migration/120:34:241
2135699364,0cyclictest2399migration/120:34:241
2135699364,0cyclictest2399migration/100:07:061
2135699363,0cyclictest2399migration/123:18:201
21356993621,0cyclictest7521-21grep22:11:411
2135699362,0cyclictest218599rtkit-daemon22:43:191
21351993619,0cyclictest2211-21gdm-simple-gree20:31:460
21351993619,0cyclictest2211-21gdm-simple-gree20:31:460
161360,0rcuc/01061-21cat21:56:470
141360,0rcu_preempt25-21ksoftirqd/123:11:451
141360,0rcu_preempt25-21ksoftirqd/119:56:431
141360,0rcu_preempt13-21ksoftirqd/020:21:160
141360,0rcu_preempt0-21swapper/323:16:153
2135699354,0cyclictest2399migration/123:57:101
2135199355,0cyclictest1799migration/023:15:420
141350,0rcu_preempt0-21swapper/021:29:490
141340,0rcu_preempt941-21latency00:21:501
141340,0rcu_preempt21080-21mii-tool20:16:431
21356993319,0cyclictest28362-21grep21:46:361
21356993319,0cyclictest28362-21grep21:46:361
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional