You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-23 - 09:35
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot6.osadl.org (updated Sat Sep 23, 2023 00:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291990,0irq_work/2311rcuc/200:32:472
291990,0irq_work/2311rcuc/200:32:472
2340999820,0cyclictest218599rtkit-daemon20:17:080
2341199752,0cyclictest218599rtkit-daemon20:33:442
291720,0irq_work/2311rcuc/221:39:192
181690,0irq_work/0161rcuc/021:39:170
291670,0irq_work/2311rcuc/222:40:542
291660,0irq_work/2311rcuc/219:04:452
2341099661,0cyclictest218599rtkit-daemon21:47:431
361650,0irq_work/3381rcuc/300:13:433
221650,0irq_work/1241rcuc/121:32:111
181650,0irq_work/0161rcuc/000:04:190
2340999641,0cyclictest218599rtkit-daemon19:39:270
361630,0irq_work/3381rcuc/323:39:093
2340999631,0cyclictest218599rtkit-daemon20:39:130
361620,0irq_work/3381rcuc/319:55:123
2340999621,0cyclictest218599rtkit-daemon21:46:290
2340999621,0cyclictest218599rtkit-daemon20:02:310
221610,0irq_work/1241rcuc/100:29:051
221610,0irq_work/1241rcuc/100:29:051
221610,0irq_work/1241rcuc/100:04:201
361600,0irq_work/3381rcuc/323:44:013
291600,0irq_work/2311rcuc/220:24:042
2340999601,0cyclictest218599rtkit-daemon23:38:270
2340999601,0cyclictest218599rtkit-daemon22:21:110
181600,0irq_work/0161rcuc/022:08:430
181600,0irq_work/0161rcuc/022:08:430
181600,0irq_work/0161rcuc/020:54:230
181600,0irq_work/0161rcuc/000:29:000
181600,0irq_work/0161rcuc/000:29:000
2340999591,0cyclictest218599rtkit-daemon21:33:420
2340999591,0cyclictest218599rtkit-daemon21:01:510
2340999591,0cyclictest218599rtkit-daemon21:01:510
291580,0irq_work/2311rcuc/219:59:022
2340999561,0cyclictest218599rtkit-daemon19:34:020
2340999561,0cyclictest218599rtkit-daemon00:14:360
181560,0irq_work/0161rcuc/022:34:210
361550,0irq_work/3381rcuc/319:59:453
361550,0irq_work/3381rcuc/319:15:413
291550,0irq_work/2311rcuc/220:14:272
2341099551,0cyclictest218599rtkit-daemon00:13:561
361540,0irq_work/3381rcuc/323:03:433
361540,0irq_work/3381rcuc/300:14:103
291540,0irq_work/2311rcuc/223:14:042
221540,0irq_work/1241rcuc/120:39:051
361530,0irq_work/3381rcuc/323:28:113
291530,0irq_work/2311rcuc/223:29:132
291530,0irq_work/2311rcuc/222:25:222
291530,0irq_work/2311rcuc/221:45:442
291530,0irq_work/2311rcuc/200:04:072
2340999534,0cyclictest218599rtkit-daemon19:30:450
2340999533,0cyclictest218599rtkit-daemon21:55:230
2340999531,0cyclictest218599rtkit-daemon23:58:050
2340999531,0cyclictest218599rtkit-daemon20:04:060
221530,0irq_work/1241rcuc/121:54:011
221520,0irq_work/1241rcuc/123:42:221
221520,0irq_work/1241rcuc/123:14:121
221520,0irq_work/1241rcuc/120:28:441
221520,0irq_work/1241rcuc/119:24:051
361510,0irq_work/3381rcuc/320:20:473
2340999512,0cyclictest218599rtkit-daemon22:40:120
2340999511,0cyclictest218599rtkit-daemon20:12:510
2340999511,0cyclictest218599rtkit-daemon19:09:270
221510,0irq_work/1241rcuc/121:01:031
221510,0irq_work/1241rcuc/121:01:031
221510,0irq_work/1241rcuc/119:20:141
181510,0irq_work/0161rcuc/019:56:000
181510,0irq_work/0161rcuc/019:25:160
241500,0rcuc/122454-21munin-node19:04:161
2341199502,0cyclictest218599rtkit-daemon21:00:432
2341199502,0cyclictest218599rtkit-daemon21:00:432
2340999502,0cyclictest218599rtkit-daemon00:26:250
221500,0irq_work/1241rcuc/120:29:151
361490,0irq_work/3381rcuc/300:35:023
23411994910,0cyclictest6444-21crond22:08:432
23411994910,0cyclictest6444-21crond22:08:432
2340999491,0cyclictest218599rtkit-daemon23:12:110
2340999491,0cyclictest218599rtkit-daemon20:29:350
2340999491,0cyclictest218599rtkit-daemon20:27:200
2340999491,0cyclictest218599rtkit-daemon00:22:140
221490,0irq_work/1241rcuc/121:04:131
181490,0irq_work/0161rcuc/023:02:440
181490,0irq_work/0161rcuc/019:14:170
361480,0irq_work/3381rcuc/323:59:273
361480,0irq_work/3381rcuc/300:25:443
291480,0irq_work/2311rcuc/219:54:542
2341099486,0cyclictest2399migration/122:09:461
2341099486,0cyclictest2399migration/122:09:461
2340999481,0cyclictest218599rtkit-daemon20:46:200
221480,0irq_work/1241rcuc/122:25:001
181480,0irq_work/0161rcuc/021:09:310
141480,0rcu_preempt39-21ksoftirqd/320:39:243
141480,0rcu_preempt32-21ksoftirqd/200:13:592
141480,0rcu_preempt25-21ksoftirqd/121:53:431
141480,0rcu_preempt0-21swapper/023:47:570
361470,0irq_work/3381rcuc/300:33:333
361470,0irq_work/3381rcuc/300:33:333
291470,0irq_work/2311rcuc/222:34:572
2341099479,0cyclictest2399migration/120:55:471
2341099478,0cyclictest2399migration/121:18:561
2340999471,0cyclictest218599rtkit-daemon19:20:520
221470,0irq_work/1241rcuc/121:14:231
161470,0rcuc/021647-21grep19:04:070
141470,0rcu_preempt25-21ksoftirqd/122:52:251
361460,0irq_work/3381rcuc/322:10:573
361460,0irq_work/3381rcuc/322:10:573
291460,0irq_work/2311rcuc/223:59:112
291460,0irq_work/2311rcuc/223:00:292
2341199461,0cyclictest218599rtkit-daemon22:56:062
2341099468,0cyclictest2399migration/100:36:111
2341099466,0cyclictest2399migration/119:50:061
2341099466,0cyclictest2399migration/119:50:061
2340999461,0cyclictest218599rtkit-daemon22:12:490
2340999461,0cyclictest218599rtkit-daemon22:12:490
221460,0irq_work/1241rcuc/123:59:571
221460,0irq_work/1241rcuc/122:36:561
221460,0irq_work/1241rcuc/121:38:351
221460,0irq_work/1241rcuc/120:14:291
181460,0irq_work/0161rcuc/023:29:210
141460,0rcu_preempt39-21ksoftirqd/322:43:443
381450,0rcuc/322802-21unixbench_singl19:04:203
361450,0irq_work/3381rcuc/321:54:213
291450,0irq_work/2311rcuc/223:50:012
291450,0irq_work/2311rcuc/220:43:362
291450,0irq_work/2311rcuc/219:24:142
2341099456,0cyclictest2399migration/119:48:371
2341099456,0cyclictest2399migration/119:31:501
2340999451,0cyclictest218599rtkit-daemon21:08:050
221450,0irq_work/1241rcuc/123:04:071
221450,0irq_work/1241rcuc/121:09:071
141450,0rcu_preempt25-21ksoftirqd/122:55:391
361440,0irq_work/3381rcuc/320:59:163
361440,0irq_work/3381rcuc/320:59:163
361440,0irq_work/3381rcuc/320:34:153
2341099445,0cyclictest2399migration/122:30:541
2341099443,0cyclictest2399migration/122:22:041
2340999441,0cyclictest218599rtkit-daemon20:36:080
2340999441,0cyclictest218599rtkit-daemon19:43:490
221440,0irq_work/1241rcuc/119:34:221
141440,0rcu_preempt18062-21timerandwakeup23:44:212
3799430,0migration/323369-21latency_hist19:08:443
361430,0irq_work/3381rcuc/322:20:193
361430,0irq_work/3381rcuc/321:44:243
291430,0irq_work/2311rcuc/221:55:172
291430,0irq_work/2311rcuc/221:37:152
291430,0irq_work/2311rcuc/221:11:062
2341299433,0cyclictest3799migration/320:16:293
2341099436,0cyclictest2399migration/100:24:381
2341099434,0cyclictest2399migration/123:46:511
2341099434,0cyclictest2399migration/120:43:551
2341099433,0cyclictest2399migration/119:57:101
2340999431,0cyclictest218599rtkit-daemon21:15:040
2340999431,0cyclictest218599rtkit-daemon20:23:160
221430,0irq_work/1241rcuc/123:19:031
221430,0irq_work/1241rcuc/123:19:031
221430,0irq_work/1241rcuc/123:10:341
221430,0irq_work/1241rcuc/119:09:271
141430,0rcu_preempt39-21ksoftirqd/321:39:073
141430,0rcu_preempt25-21ksoftirqd/122:14:071
141430,0rcu_preempt20052-21grep23:49:190
361420,0irq_work/3381rcuc/322:44:253
361420,0irq_work/3381rcuc/322:04:023
361420,0irq_work/3381rcuc/322:04:023
2341299424,0cyclictest3799migration/300:04:353
23412994226,0cyclictest7138-21grep20:54:233
2341099423,0cyclictest2399migration/123:26:051
23409994211,0cyclictest8635-21crond00:38:430
2340999421,0cyclictest218599rtkit-daemon22:45:190
221420,0irq_work/1241rcuc/119:39:251
181420,0irq_work/0161rcuc/022:00:560
181420,0irq_work/0161rcuc/021:19:540
141420,0rcu_preempt39-21ksoftirqd/320:44:023
141420,0rcu_preempt32-21ksoftirqd/219:34:062
141420,0rcu_preempt25-21ksoftirqd/100:09:201
141420,0rcu_preempt13-21ksoftirqd/023:18:430
361410,0irq_work/3381rcuc/322:30:123
361410,0irq_work/3381rcuc/320:11:253
291410,0irq_work/2311rcuc/223:58:382
291410,0irq_work/2311rcuc/223:24:062
291410,0irq_work/2311rcuc/220:52:302
241410,0rcuc/128963-21munin-node20:34:101
2341299412,0cyclictest3799migration/323:58:343
2341299412,0cyclictest3799migration/319:36:283
2341199413,0cyclictest3099migration/221:14:062
2341099416,0cyclictest2399migration/123:01:251
2341099415,0cyclictest2399migration/120:00:061
2341099414,0cyclictest2399migration/121:24:431
2341099413,0cyclictest2399migration/122:07:101
2341099413,0cyclictest2399migration/122:07:101
2341099413,0cyclictest2399migration/121:40:511
221410,0irq_work/1241rcuc/123:54:051
181410,0irq_work/0161rcuc/021:24:190
141410,0rcu_preempt28299-21unixbench_singl22:54:243
291400,0irq_work/2311rcuc/222:50:272
291400,0irq_work/2311rcuc/221:06:062
291400,0irq_work/2311rcuc/220:04:202
291400,0irq_work/2311rcuc/200:20:082
23412994025,0cyclictest10340-21grep21:04:023
2341299402,0cyclictest3799migration/323:29:433
2341299401,0cyclictest3799migration/322:34:023
2341299401,0cyclictest3799migration/321:49:133
2341199401,0cyclictest16715-21munin-node21:19:092
23409994027,0cyclictest15073-21grep23:39:100
221400,0irq_work/1241rcuc/100:19:031
141400,0rcu_preempt13689-21timerwakeupswit23:34:221
141400,0rcu_preempt0-21swapper/122:39:121
381390,0rcuc/30-21swapper/319:38:513
361390,0irq_work/3381rcuc/323:19:083
361390,0irq_work/3381rcuc/323:19:083
361390,0irq_work/3381rcuc/322:49:043
311390,0rcuc/26253-21sed20:54:062
291390,0irq_work/2311rcuc/221:25:112
2341299393,0cyclictest3799migration/323:07:453
23412993924,0cyclictest1419-21grep00:19:273
2341299392,0cyclictest3799migration/323:34:523
2341299391,0cyclictest3799migration/319:25:533
23409993925,0cyclictest10215-21switchtime22:14:190
2340999391,0cyclictest218599rtkit-daemon22:57:370
181390,0irq_work/0161rcuc/023:04:050
141390,0rcu_preempt7596-21latency22:09:132
141390,0rcu_preempt7596-21latency22:09:132
141390,0rcu_preempt39-21ksoftirqd/321:59:213
141390,0rcu_preempt39-21ksoftirqd/321:19:093
141390,0rcu_preempt32-21ksoftirqd/221:49:252
141390,0rcu_preempt22264-21cut22:44:022
141390,0rcu_preempt13-21ksoftirqd/022:49:150
141390,0rcu_preempt13-21ksoftirqd/021:34:270
141390,0rcu_preempt11409-21sed23:29:221
141390,0rcu_preempt0-21swapper/220:09:262
103650390,0irq/28-eth0-tx-9552-21sshd23:26:120
103550390,0irq/27-eth0-rx-1474-21tune2fs23:09:073
361380,0irq_work/3381rcuc/320:49:123
291380,0irq_work/2311rcuc/223:19:042
291380,0irq_work/2311rcuc/223:19:042
291380,0irq_work/2311rcuc/223:09:342
291380,0irq_work/2311rcuc/219:39:522
2341299384,0cyclictest3799migration/319:47:483
23411993826,0cyclictest9200-21grep22:14:012
2341199381,0cyclictest3099migration/219:14:182
141380,0rcu_preempt39-21ksoftirqd/319:49:033
141380,0rcu_preempt39-21ksoftirqd/319:49:033
141380,0rcu_preempt24313-21latency19:09:102
141380,0rcu_preempt0-21swapper/121:59:391
291370,0irq_work/2311rcuc/219:44:532
291370,0irq_work/2311rcuc/219:21:122
291370,0irq_work/2311rcuc/200:25:152
291370,0irq_work/2141rcu_preempt20:19:242
2341299371,0cyclictest3799migration/322:25:453
2341199374,0cyclictest3099migration/200:09:022
2340999370,0cyclictest218599rtkit-daemon19:53:090
2340999370,0cyclictest218599rtkit-daemon19:53:090
141370,0rcu_preempt19432-21cut23:49:073
141370,0rcu_preempt0-21swapper/322:15:483
141370,0rcu_preempt0-21swapper/223:04:222
361360,0irq_work/3381rcuc/319:21:053
291360,0irq_work/2311rcuc/223:40:382
2341099366,0cyclictest19520-21grep23:49:081
2341099362,0cyclictest2399migration/119:16:491
23410993618,0cyclictest18412-21fschecks_time20:09:091
23409993630,0cyclictest23341-21munin-node23:59:200
2340999361,0cyclictest1799migration/023:21:170
2340999361,0cyclictest1799migration/023:21:170
221360,0irq_work/1241rcuc/120:48:441
181360,0irq_work/0161rcuc/022:29:190
141360,0rcu_preempt25-21ksoftirqd/122:48:441
141360,0rcu_preempt25-21ksoftirqd/120:19:201
361350,0irq_work/3381rcuc/321:09:423
361350,0irq_work/3381rcuc/320:04:343
311350,0rcuc/211535-21forks22:19:072
291350,0irq_work/2311rcuc/222:29:142
291350,0irq_work/2311rcuc/221:30:122
291350,0irq_work/2311rcuc/219:49:542
291350,0irq_work/2311rcuc/219:49:542
2341299353,0cyclictest3799migration/323:17:473
23412993523,0cyclictest27147-21grep20:29:073
2341299352,0cyclictest3799migration/321:14:553
141350,0rcu_preempt867-21missed_timers19:29:192
141350,0rcu_preempt39-21ksoftirqd/319:33:443
141350,0rcu_preempt13-21ksoftirqd/021:49:040
291340,0irq_work/2311rcuc/223:38:432
2341299341,0cyclictest26015-21ls20:24:263
2341199344,0cyclictest3099migration/222:01:362
23409993419,0cyclictest3353-21seq20:49:020
2340999340,0cyclictest218599rtkit-daemon22:24:370
23412993326,0cyclictest24974-21unixbench_singl21:34:243
23411993322,0cyclictest7298-21grep00:34:072
2341099332,0cyclictest2399migration/120:04:021
23409993320,0cyclictest29391-21unixbench_singl00:09:210
361320,0irq_work/3381rcuc/321:28:183
2341299321,0cyclictest3799migration/321:29:043
291300,0irq_work/2311rcuc/220:44:052
23411992814,0cyclictest29398-21cut20:34:082
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional