You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-01 - 04:10
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack1slot6.osadl.org (updated Thu Nov 30, 2023 12:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
181980,0irq_work/0161rcuc/011:42:100
291940,0irq_work/2311rcuc/211:25:022
291940,0irq_work/2311rcuc/211:25:022
14588999383,0cyclictest25728-21ls09:07:081
361810,0irq_work/3381rcuc/312:32:023
361760,0irq_work/3381rcuc/308:55:413
291760,0irq_work/2311rcuc/210:15:462
291740,0irq_work/2311rcuc/208:12:132
181710,0irq_work/0161rcuc/011:47:060
361670,0irq_work/3381rcuc/312:28:283
361670,0irq_work/3381rcuc/310:12:023
361670,0irq_work/3381rcuc/309:31:513
181660,0irq_work/0161rcuc/010:02:060
221630,0irq_work/1241rcuc/110:17:091
181630,0irq_work/0161rcuc/009:37:460
361620,0irq_work/3381rcuc/311:02:003
361620,0irq_work/3381rcuc/309:21:573
14588996257,0cyclictest0-21swapper/107:33:471
1458499623,0cyclictest1799migration/008:12:090
181610,0irq_work/0161rcuc/012:11:550
181610,0irq_work/0161rcuc/011:12:100
1458899601,0cyclictest0-21swapper/109:54:081
291590,0irq_work/2311rcuc/210:17:022
291590,0irq_work/2311rcuc/209:02:092
14601995920,0cyclictest29008-21fschecks_time11:51:583
181580,0irq_work/0161rcuc/011:37:040
181580,0irq_work/0161rcuc/008:57:050
1460199581,0cyclictest3799migration/311:50:463
1458499583,0cyclictest218599rtkit-daemon08:51:530
221570,0irq_work/1241rcuc/111:22:081
221570,0irq_work/1241rcuc/111:22:081
1460199560,0cyclictest3799migration/309:16:083
1460199560,0cyclictest3799migration/309:16:083
1460199560,0cyclictest3799migration/308:34:453
291550,0irq_work/2311rcuc/212:26:512
221550,0irq_work/1241rcuc/111:52:091
221550,0irq_work/1241rcuc/109:46:301
1460199550,0cyclictest3799migration/312:21:023
1460199550,0cyclictest3799migration/310:37:173
1460199550,0cyclictest3799migration/308:28:163
14588995543,0cyclictest8312-21seq08:27:481
1458899552,0cyclictest218599rtkit-daemon10:07:021
1458499552,0cyclictest0-21swapper/012:32:070
361540,0irq_work/3381rcuc/309:38:213
291540,0irq_work/2311rcuc/212:01:252
221540,0irq_work/1241rcuc/111:41:581
181540,0irq_work/0161rcuc/012:08:170
181540,0irq_work/0161rcuc/011:17:520
1460199543,0cyclictest0-21swapper/308:41:343
1460199540,0cyclictest3799migration/311:16:233
1458499542,0cyclictest0-21swapper/012:21:570
1458499541,0cyclictest218599rtkit-daemon10:31:550
141540,0rcu_preempt13-21ksoftirqd/007:40:040
1458499532,0cyclictest0-21swapper/008:37:100
291520,0irq_work/2311rcuc/207:27:462
181520,0irq_work/0161rcuc/010:42:120
14601995223,0cyclictest30979-21grep08:02:133
14601995223,0cyclictest30979-21grep08:02:133
14601995217,0cyclictest24198-21cut07:41:553
14594995210,0cyclictest1255-21runrttasks11:30:202
1458899523,0cyclictest218599rtkit-daemon11:15:091
1458499524,0cyclictest1799migration/007:27:590
1458499522,0cyclictest0-21swapper/009:51:500
141520,0rcu_preempt12335-21latency_hist07:01:303
361510,0irq_work/3381rcuc/310:17:093
291510,0irq_work/2311rcuc/211:20:402
291510,0irq_work/2311rcuc/210:46:292
291510,0irq_work/2311rcuc/208:55:522
291510,0irq_work/2311rcuc/207:20:052
14601995119,0cyclictest3001-21runrttasks12:09:063
14601995118,0cyclictest28197-21vmstat07:52:163
14601995117,0cyclictest2080-21cat10:47:093
14601995113,0cyclictest2211-21gdm-simple-gree10:32:133
1458499512,0cyclictest8076-21crond09:46:300
291500,0irq_work/2311rcuc/208:16:552
241500,0rcuc/113247-21kernelversion07:01:561
181500,0irq_work/0161rcuc/008:35:290
181500,0irq_work/0161rcuc/007:23:420
14601995022,0cyclictest25149-21grep10:26:583
14601995017,0cyclictest3809-21fschecks_count10:51:513
14601995016,0cyclictest4548-21open_files12:12:063
1458899501,0cyclictest218599rtkit-daemon10:21:591
361490,0irq_work/3381rcuc/309:02:183
291490,0irq_work/2311rcuc/211:04:252
14601994921,0cyclictest14641-21seq07:06:593
14601994921,0cyclictest13129-21grep11:16:523
14601994917,0cyclictest32053-21fschecks_count08:06:543
14601994917,0cyclictest32053-21fschecks_count08:06:543
14601994916,0cyclictest17570-21seq10:11:123
14594994936,0cyclictest27498-21missed_timers07:52:032
291480,0irq_work/2311rcuc/211:56:242
291480,0irq_work/2311rcuc/210:00:352
291480,0irq_work/2311rcuc/208:56:522
291480,0irq_work/2311rcuc/207:33:502
221480,0irq_work/1241rcuc/112:17:131
181480,0irq_work/0161rcuc/010:27:090
1460199485,0cyclictest3799migration/311:34:033
1460199485,0cyclictest3799migration/311:34:033
14601994823,0cyclictest32728-21grep10:42:183
14601994819,0cyclictest1255-21runrttasks11:26:043
14601994819,0cyclictest1255-21runrttasks11:26:043
14601994817,0cyclictest20951-21missed_timers08:57:003
14601994815,0cyclictest15438-21cpu07:11:493
1460199480,0cyclictest3799migration/309:41:593
291470,0irq_work/2311rcuc/212:08:562
291470,0irq_work/2311rcuc/208:43:002
291470,0irq_work/2311rcuc/208:34:382
291470,0irq_work/2311rcuc/207:41:532
221470,0irq_work/1241rcuc/109:22:001
14601994718,0cyclictest21757-21munin-node11:36:553
14601994718,0cyclictest18279-21missed_timers11:27:013
14601994717,0cyclictest9128-21http_loadtime11:06:573
14601994714,0cyclictest25086-21sed09:06:553
1458899476,0cyclictest2399migration/111:57:141
14584994741,0cyclictest0-21swapper/007:08:110
141470,0rcu_preempt32-21ksoftirqd/210:26:532
361460,0irq_work/3381rcuc/309:52:073
311460,0rcuc/212844-21fschecks_count07:01:522
291460,0irq_work/2311rcuc/210:32:282
291460,0irq_work/2311rcuc/210:22:382
291460,0irq_work/2311rcuc/209:24:542
221460,0irq_work/1241rcuc/109:41:251
221460,0irq_work/1241rcuc/109:36:301
181460,0irq_work/0161rcuc/010:55:370
181460,0irq_work/0161rcuc/010:07:090
181460,0irq_work/0161rcuc/007:59:130
14601994619,0cyclictest1802-21expr08:11:533
14601994617,0cyclictest5086-21switchtime08:17:103
14601994616,0cyclictest23375-21ntpdc10:22:033
14601994613,0cyclictest32502-21if_eth009:26:593
1460199460,0cyclictest3799migration/310:06:303
1458899467,0cyclictest2399migration/107:36:491
1458899466,0cyclictest2399migration/112:15:371
1458899465,0cyclictest2399migration/107:51:581
1458899462,0cyclictest218599rtkit-daemon10:51:241
14584994624,0cyclictest0-21swapper/011:01:450
141460,0rcu_preempt0-21swapper/010:58:310
361450,0irq_work/3381rcuc/309:16:303
361450,0irq_work/3141rcu_preempt07:56:593
291450,0irq_work/2311rcuc/207:24:082
181450,0irq_work/0161rcuc/008:28:400
181450,0irq_work/0161rcuc/007:11:500
14601994519,0cyclictest8683-21crond09:51:303
14601994519,0cyclictest19683-21awk07:22:073
14601994516,0cyclictest12142-21nfs_client08:37:063
14601994514,0cyclictest15519-21basename08:46:473
14601994513,0cyclictest9227-21sed12:22:123
14588994533,0cyclictest1113-21nscd11:06:501
14588994528,0cyclictest28275-21grep10:32:121
1458899452,0cyclictest24044-21taskset09:04:501
14584994529,0cyclictest776-21grep08:07:130
14584994529,0cyclictest776-21grep08:07:130
141450,0rcu_preempt25-21ksoftirqd/112:22:051
291440,0irq_work/2311rcuc/211:41:192
291440,0irq_work/2311rcuc/209:11:512
291440,0irq_work/2311rcuc/209:11:512
291440,0irq_work/2311rcuc/208:49:162
291440,0irq_work/2311rcuc/208:09:342
291440,0irq_work/2311rcuc/208:09:342
291440,0irq_work/2311rcuc/207:49:302
181440,0irq_work/0161rcuc/011:22:150
181440,0irq_work/0161rcuc/011:22:150
181440,0irq_work/0161rcuc/010:17:240
181440,0irq_work/0161rcuc/010:13:130
181440,0irq_work/0161rcuc/009:36:210
161440,0rcuc/012438-21munin-node07:01:490
14601994417,0cyclictest1255-21runrttasks12:02:503
14601994416,0cyclictest22180-21ls07:32:013
14601994416,0cyclictest12641-21timerwakeupswit09:57:093
1459499447,0cyclictest3099migration/211:50:382
1459499446,0cyclictest3099migration/211:34:372
1459499446,0cyclictest3099migration/211:34:372
1459499446,0cyclictest3099migration/207:58:052
1458899441,0cyclictest218599rtkit-daemon12:27:241
141440,0rcu_preempt32-21ksoftirqd/209:37:132
291430,0irq_work/2311rcuc/211:12:262
291430,0irq_work/2311rcuc/209:10:142
291430,0irq_work/2311rcuc/208:37:012
291430,0irq_work/2311rcuc/207:09:232
181430,0irq_work/0161rcuc/008:46:400
1460199433,0cyclictest3799migration/308:26:193
14601994316,0cyclictest31520-21cat11:57:013
14601994315,0cyclictest20629-21seq07:31:153
1459499436,0cyclictest3099migration/208:31:222
14594994336,0cyclictest0-21swapper/209:31:452
14588994327,0cyclictest25143-21grep10:26:581
1458899431,0cyclictest218599rtkit-daemon10:55:341
1458899431,0cyclictest218599rtkit-daemon09:49:181
1458899431,0cyclictest218599rtkit-daemon08:50:161
141430,0rcu_preempt32-21ksoftirqd/208:02:052
141430,0rcu_preempt32-21ksoftirqd/208:02:052
141430,0rcu_preempt25-21ksoftirqd/108:17:031
291420,0irq_work/2311rcuc/211:11:132
291420,0irq_work/2311rcuc/210:01:492
181420,0irq_work/0161rcuc/010:40:270
181420,0irq_work/0161rcuc/008:20:550
14601994217,0cyclictest1255-21runrttasks10:58:543
14601994214,0cyclictest1255-21runrttasks11:45:293
1459499426,0cyclictest3099migration/212:32:382
1459499426,0cyclictest3099migration/210:52:042
1459499426,0cyclictest3099migration/209:48:362
1459499425,0cyclictest3099migration/212:11:382
1459499425,0cyclictest3099migration/209:51:562
1459499425,0cyclictest3099migration/207:13:322
1459499423,0cyclictest3099migration/209:43:052
1458899422,0cyclictest2399migration/110:37:311
1458899421,0cyclictest218599rtkit-daemon11:05:091
181410,0irq_work/0161rcuc/011:06:530
1460199410,0cyclictest3799migration/307:51:283
1459499415,0cyclictest3099migration/212:17:462
1458899418,0cyclictest5677-21runrttasks10:58:011
1458899415,0cyclictest2399migration/112:06:521
1458899415,0cyclictest2399migration/107:45:541
14588994130,0cyclictest241rcuc/110:41:591
1458899411,0cyclictest218599rtkit-daemon12:03:391
1458899411,0cyclictest218599rtkit-daemon07:51:281
141410,0rcu_preempt13-21ksoftirqd/011:56:300
221400,0irq_work/1241rcuc/109:21:291
181400,0irq_work/0161rcuc/012:01:450
181400,0irq_work/0161rcuc/008:26:070
1459499406,0cyclictest3099migration/210:06:532
1458899407,0cyclictest23241-21timerwakeupswit11:37:101
14588994031,0cyclictest25-21ksoftirqd/111:27:331
14588994023,0cyclictest12778-21cut12:31:561
1458899401,0cyclictest218599rtkit-daemon11:49:281
141400,0rcu_preempt13-21ksoftirqd/009:22:520
141400,0rcu_preempt1295-21Xorg07:42:010
103650400,0irq/28-eth0-tx-20885-21munin-node07:31:430
241390,0rcuc/1141rcu_preempt09:12:061
241390,0rcuc/1141rcu_preempt09:12:061
221390,0irq_work/1241rcuc/111:16:551
181390,0irq_work/0161rcuc/010:51:300
181390,0irq_work/0161rcuc/007:47:130
14601993916,0cyclictest2211-21gdm-simple-gree07:37:163
1458899397,0cyclictest2399migration/108:06:501
1458899397,0cyclictest2399migration/108:06:501
1458899396,0cyclictest2399migration/107:14:461
1458899395,0cyclictest2399migration/108:45:011
1458899395,0cyclictest15293-21runrttasks10:05:301
1458899393,0cyclictest2399migration/108:57:071
1458899391,0cyclictest218599rtkit-daemon08:37:321
14584993924,0cyclictest23728-21unixbench_multi09:02:120
14584993923,0cyclictest25990-21grep09:07:120
141390,0rcu_preempt32-21ksoftirqd/210:41:302
141390,0rcu_preempt13-21ksoftirqd/012:27:070
291380,0irq_work/2311rcuc/212:25:062
291380,0irq_work/2311rcuc/209:28:102
14601993814,0cyclictest2211-21gdm-simple-gree07:17:153
1458899385,0cyclictest2399migration/108:25:381
1458899385,0cyclictest2399migration/107:57:461
1458899384,0cyclictest32537-21iostat_ios09:27:001
1458899383,0cyclictest2399migration/107:21:181
141380,0rcu_preempt25-21ksoftirqd/108:12:101
141380,0rcu_preempt2055-21sed10:47:082
291370,0irq_work/2311rcuc/211:46:202
291370,0irq_work/2311rcuc/208:26:302
291370,0irq_work/2311rcuc/207:39:282
14584993721,0cyclictest12687-21grep09:57:100
141370,0rcu_preempt8409-21taskset09:47:570
141370,0rcu_preempt25-21ksoftirqd/110:13:171
221360,0irq_work/1241rcuc/111:32:071
221360,0irq_work/1241rcuc/111:32:071
181360,0irq_work/0161rcuc/009:11:450
181360,0irq_work/0161rcuc/009:11:450
181360,0irq_work/0141rcu_preempt08:42:070
1459499363,0cyclictest3099migration/212:04:222
14588993624,0cyclictest141rcu_preempt08:03:321
14588993624,0cyclictest141rcu_preempt08:03:321
14584993619,0cyclictest2211-21gdm-simple-gree11:56:460
141360,0rcu_preempt13-21ksoftirqd/011:32:130
141360,0rcu_preempt13-21ksoftirqd/011:32:130
291350,0irq_work/2141rcu_preempt09:21:292
221350,0irq_work/1241rcuc/109:57:031
181350,0irq_work/0161rcuc/010:23:480
181350,0irq_work/0141rcu_preempt09:31:300
161350,0rcuc/07289-21vmstat12:17:140
181340,0irq_work/0141rcu_preempt07:52:040
14594993417,0cyclictest2211-21gdm-simple-gree10:59:242
14588993420,0cyclictest18416-21grep08:51:541
14584993421,0cyclictest30340-21load08:02:010
14584993421,0cyclictest30340-21load08:02:010
14584993420,0cyclictest2211-21gdm-simple-gree11:30:010
241330,0rcuc/114583-21sed07:06:301
14588993319,0cyclictest18678-21grep07:21:501
1458499325,0cyclictest1799migration/007:18:320
141320,0rcu_preempt0-21swapper/009:17:450
14588992917,0cyclictest20652-21crond07:31:301
1458899290,0cyclictest2399migration/108:31:501
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional