You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-11-29 - 15:46
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 792 highest latencies:
System rack1slot1.osadl.org (updated Wed Nov 29, 2023 12:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2665222530,4sleep30-21swapper/307:05:293
270392199162,24sleep00-21swapper/007:09:490
249472198158,26sleep20-21swapper/207:05:142
270352197159,26sleep10-21swapper/107:09:461
315142170,1sleep32715999cyclictest11:35:293
27136991513,1cyclictest884-21cron12:20:000
27136991511,2cyclictest259-21systemd-journal07:14:590
143822150,0sleep10-21swapper/112:35:301
27136991412,2cyclictest19939-21diskmemload11:00:280
2713699139,3cyclictest14901-21egrep07:50:200
2713699133,9cyclictest29588-21ssh10:08:130
27136991311,2cyclictest0-21swapper/011:39:160
27136991311,1cyclictest0-21swapper/011:27:370
206142130,0sleep00-21swapper/012:02:570
27159991211,1cyclictest35-21ksoftirqd/310:35:003
27159991210,1cyclictest35-21ksoftirqd/307:30:013
2715199120,11cyclictest30038-21ssh10:09:212
2715199120,0cyclictest0-21swapper/209:35:432
2713699122,9cyclictest29416-21ssh10:53:150
27136991211,1cyclictest0-21swapper/012:13:240
27136991211,1cyclictest0-21swapper/010:24:470
27136991211,1cyclictest0-21swapper/008:15:190
27136991210,2cyclictest4043-21ssh12:22:560
2713699119,2cyclictest0-21swapper/011:50:290
2713699119,2cyclictest0-21swapper/008:35:240
2713699119,1cyclictest19939-21diskmemload09:22:080
2713699119,1cyclictest0-21swapper/009:45:560
2713699118,2cyclictest4006-21processes07:25:310
2713699117,3cyclictest19939-21diskmemload10:02:530
2713699117,3cyclictest19939-21diskmemload09:10:500
2713699116,4cyclictest7872-21ssh12:27:370
2713699116,4cyclictest17662-21ssh11:19:390
2713699116,4cyclictest15410-21users11:55:280
2713699112,8cyclictest19939-21diskmemload10:11:490
2713699112,8cyclictest0-21swapper/007:35:260
2713699112,1cyclictest19939-21diskmemload09:41:190
2713699112,1cyclictest13430-21ssh10:33:010
27136991110,1cyclictest32681-21ssh10:56:290
27136991110,1cyclictest19939-21diskmemload12:05:400
27136991110,1cyclictest19939-21diskmemload11:24:130
27136991110,1cyclictest11675-21ssh12:32:140
27136991110,1cyclictest10599-21ssh11:10:280
27136991110,1cyclictest0-21swapper/011:46:410
27136991110,1cyclictest0-21swapper/011:08:070
27136991110,1cyclictest0-21swapper/010:40:200
27136991110,1cyclictest0-21swapper/008:45:210
277232100,1sleep20-21swapper/212:10:582
2715199100,7cyclictest19939-21diskmemload11:50:322
2713699109,1cyclictest19939-21diskmemload10:25:390
2713699109,1cyclictest19939-21diskmemload10:16:590
2713699109,1cyclictest0-21swapper/011:43:480
2713699109,1cyclictest0-21swapper/010:45:540
2713699109,1cyclictest0-21swapper/010:35:290
2713699109,1cyclictest0-21swapper/009:56:300
2713699108,1cyclictest19939-21diskmemload12:38:090
2713699108,1cyclictest19939-21diskmemload11:31:200
2713699108,1cyclictest19939-21diskmemload09:39:340
2713699107,1cyclictest9-21ksoftirqd/008:59:020
2713699107,1cyclictest9-21ksoftirqd/008:59:020
2713699106,3cyclictest0-21swapper/009:53:040
2713699106,3cyclictest0-21swapper/009:30:230
2713699103,6cyclictest24971-21ssh09:15:290
2713699102,1cyclictest770-21snmpd07:47:350
8296290,0sleep20-21swapper/210:25:042
271599998,1cyclictest35-21ksoftirqd/310:40:003
271599998,1cyclictest35-21ksoftirqd/309:40:003
271599992,5cyclictest0-21swapper/311:19:483
271519998,1cyclictest11094-21sh09:42:482
271519992,1cyclictest0-21swapper/211:00:222
271519991,5cyclictest13191-21tail12:35:232
271519990,1cyclictest0-21swapper/207:50:192
271439990,5cyclictest0-21swapper/110:50:191
271439990,4cyclictest0-21swapper/109:42:031
271369998,1cyclictest32765-21cat08:25:340
271369995,3cyclictest19939-21diskmemload09:27:400
271369994,2cyclictest259-21systemd-journal07:24:590
271369993,5cyclictest5059-21/usr/sbin/munin07:30:270
271599987,1cyclictest35-21ksoftirqd/311:15:003
271599987,1cyclictest35-21ksoftirqd/311:05:013
271599987,1cyclictest35-21ksoftirqd/308:30:003
271599986,1cyclictest770-21snmpd12:26:413
271599986,1cyclictest770-21snmpd10:55:353
271599986,1cyclictest770-21snmpd08:36:193
271599986,1cyclictest770-21snmpd08:01:543
271599986,1cyclictest770-21snmpd07:59:573
271599986,1cyclictest770-21snmpd07:50:503
271599986,1cyclictest1422-21ntpd08:55:303
271599986,1cyclictest1422-21ntpd08:55:293
271599985,3cyclictest19939-21diskmemload09:14:323
271599980,7cyclictest0-21swapper/311:20:243
271599980,4cyclictest19939-21diskmemload10:15:143
271599980,3cyclictest27665-21netstat10:50:253
271519988,0cyclictest0-21swapper/212:08:452
271519988,0cyclictest0-21swapper/210:11:412
271519986,1cyclictest770-21snmpd09:20:372
271519986,1cyclictest61250irq/122-eno112:19:592
271519986,1cyclictest61250irq/122-eno111:55:212
271519986,1cyclictest61250irq/122-eno110:33:182
271519986,1cyclictest61250irq/122-eno110:25:012
271519986,1cyclictest61250irq/122-eno109:55:292
271519986,1cyclictest61250irq/122-eno109:50:292
271519986,1cyclictest28-21ksoftirqd/208:14:592
271519986,1cyclictest0-21swapper/208:35:192
271519983,2cyclictest0-21swapper/207:55:272
271439980,5cyclictest0-21swapper/110:44:201
271439980,4cyclictest0-21swapper/109:48:181
271369986,1cyclictest770-21snmpd07:57:050
271369983,4cyclictest12710-21awk08:55:010
271369982,4cyclictest0-21swapper/008:40:210
271369980,1cyclictest0-21swapper/007:15:190
271599977,0cyclictest35-21ksoftirqd/310:15:003
271599976,1cyclictest35-21ksoftirqd/312:35:243
271599976,1cyclictest35-21ksoftirqd/309:55:003
271599976,1cyclictest21185-21cat09:55:233
271599975,1cyclictest770-21snmpd11:41:493
271599975,1cyclictest770-21snmpd11:27:183
271599975,1cyclictest24459-21runrttasks12:07:433
271599973,3cyclictest19939-21diskmemload09:25:173
271599972,4cyclictest13622-21apt-get11:55:163
271599972,3cyclictest0-21swapper/311:31:423
271599972,2cyclictest19939-21diskmemload12:10:063
271599972,2cyclictest19939-21diskmemload10:05:023
271599970,3cyclictest0-21swapper/310:40:023
271599970,2cyclictest0-21swapper/312:15:213
271599970,2cyclictest0-21swapper/311:47:103
271519976,1cyclictest61250irq/122-eno112:20:142
271519976,1cyclictest61250irq/122-eno112:00:182
271519976,1cyclictest61250irq/122-eno111:45:162
271519976,1cyclictest61250irq/122-eno111:16:172
271519976,1cyclictest61250irq/122-eno109:25:152
271519976,1cyclictest61250irq/122-eno109:17:252
271519976,1cyclictest61250irq/122-eno108:40:302
271519976,0cyclictest61250irq/122-eno111:30:202
271519975,1cyclictest770-21snmpd12:29:422
271519975,1cyclictest770-21snmpd10:18:042
271519975,1cyclictest770-21snmpd09:30:482
271519975,1cyclictest770-21snmpd08:20:242
271519975,1cyclictest770-21snmpd07:12:432
271519975,1cyclictest7163-21cron07:35:002
271519975,1cyclictest61250irq/122-eno110:55:252
271519973,2cyclictest259-21systemd-journal09:14:592
271519970,4cyclictest0-21swapper/211:40:122
271519970,4cyclictest0-21swapper/211:05:282
271519970,4cyclictest0-21swapper/210:01:362
271519970,3cyclictest24995-21runrttasks10:47:092
271519970,1cyclictest259-21systemd-journal08:59:012
271519970,1cyclictest259-21systemd-journal08:59:012
271519970,1cyclictest0-21swapper/208:50:272
271439973,1cyclictest0-21swapper/112:20:311
271439971,2cyclictest0-21swapper/107:10:411
271439971,1cyclictest0-21swapper/111:00:321
271439970,3cyclictest0-21swapper/111:13:521
271439970,2cyclictest0-21swapper/112:16:161
271439970,1cyclictest0-21swapper/111:40:201
271369975,1cyclictest0-21swapper/007:40:190
271369974,2cyclictest29848-21ls08:20:260
271369974,2cyclictest25387-21tr08:10:300
271369974,2cyclictest22463-21perl08:05:200
271369974,2cyclictest20437-21grep08:00:250
271369974,2cyclictest18499-21sort09:05:210
271369974,2cyclictest1807-21perl08:30:230
271599966,0cyclictest35-21ksoftirqd/310:00:003
271599965,1cyclictest770-21snmpd09:06:313
271599965,1cyclictest7699-21ssh10:24:073
271599965,1cyclictest6026-21ssh11:05:203
271599965,1cyclictest35-21ksoftirqd/309:45:163
271599965,1cyclictest2488-21ssh09:30:243
271599965,1cyclictest1422-21ntpd10:45:293
271599965,1cyclictest1422-21ntpd08:45:293
271599965,1cyclictest12031-21ssh11:51:503
271599965,1cyclictest1201-21ssh12:20:103
271599964,1cyclictest770-21snmpd10:25:223
271599964,1cyclictest770-21snmpd07:39:083
271599964,1cyclictest35-21ksoftirqd/309:00:143
271599964,1cyclictest1422-21ntpd07:40:303
271599963,1cyclictest259-21systemd-journal09:23:153
271599962,1cyclictest19939-21diskmemload09:41:083
271599961,2cyclictest10862-21unixbench-2d12:30:303
271599960,4cyclictest0-21swapper/309:15:493
271599960,2cyclictest0-21swapper/308:05:173
271599960,1cyclictest0-21swapper/312:01:173
271519966,0cyclictest61250irq/122-eno111:35:042
271519966,0cyclictest61250irq/122-eno111:25:202
271519966,0cyclictest61250irq/122-eno110:41:252
271519966,0cyclictest61250irq/122-eno108:45:202
271519965,1cyclictest61250irq/122-eno111:20:372
271519965,1cyclictest61250irq/122-eno107:15:192
271519965,1cyclictest16613-21ntpq09:00:262
271519964,1cyclictest61250irq/122-eno112:30:242
271519964,1cyclictest61250irq/122-eno110:50:212
271519964,1cyclictest61250irq/122-eno108:30:242
271519964,1cyclictest61250irq/122-eno108:00:222
271519964,1cyclictest61250irq/122-eno107:25:252
271519964,1cyclictest10079-21apt11:10:242
271519963,1cyclictest28-21ksoftirqd/208:10:002
271519962,3cyclictest26486-21apt-get08:15:192
271519960,6cyclictest0-21swapper/209:45:032
271519960,4cyclictest259-21systemd-journal09:10:002
271519960,4cyclictest0-21swapper/210:35:042
271519960,1cyclictest0-21swapper/208:25:152
271439965,1cyclictest0-21swapper/109:55:241
271439964,2cyclictest0-21swapper/112:02:541
271439964,1cyclictest0-21swapper/111:21:061
271439961,5cyclictest0-21swapper/110:13:101
271439961,2cyclictest0-21swapper/112:30:041
271439961,2cyclictest0-21swapper/108:50:301
271439960,6cyclictest0-21swapper/108:10:551
271439960,6cyclictest0-21swapper/107:35:171
271439960,5cyclictest0-21swapper/111:25:261
271439960,5cyclictest0-21swapper/110:55:311
271439960,5cyclictest0-21swapper/110:23:571
271439960,5cyclictest0-21swapper/107:32:061
271439960,4cyclictest0-21swapper/111:16:321
271439960,4cyclictest0-21swapper/109:35:011
271439960,4cyclictest0-21swapper/107:45:051
271439960,3cyclictest0-21swapper/109:14:541
271439960,2cyclictest0-21swapper/112:27:001
271439960,2cyclictest0-21swapper/112:07:411
271439960,2cyclictest0-21swapper/111:35:331
271439960,2cyclictest0-21swapper/109:30:231
271439960,2cyclictest0-21swapper/109:21:521
271439960,2cyclictest0-21swapper/109:16:551
271439960,1cyclictest0-21swapper/107:15:011
271439960,0cyclictest0-21swapper/111:07:441
271369963,2cyclictest15721-21grep09:00:210
271599954,1cyclictest770-21snmpd08:14:513
271599954,0cyclictest1153-21apt-get08:30:203
271599953,1cyclictest2277-21taskset07:22:473
271599952,1cyclictest259-21systemd-journal08:25:003
271599952,1cyclictest13107-21sed07:45:233
271599950,3cyclictest0-21swapper/307:30:313
271599950,2cyclictest0-21swapper/307:15:183
271599950,1cyclictest0-21swapper/308:15:183
271519955,0cyclictest61250irq/122-eno107:40:222
271519954,1cyclictest61250irq/122-eno107:35:192
271519950,4cyclictest303-21/usr/sbin/munin07:20:232
271519950,1cyclictest0-21swapper/207:45:222
271439955,0cyclictest0-21swapper/111:56:421
271439954,1cyclictest0-21swapper/110:00:121
271439953,1cyclictest7844-21chrt08:43:141
271439953,1cyclictest0-21swapper/110:35:201
271439953,1cyclictest0-21swapper/108:20:231
271439951,2cyclictest0-21swapper/111:50:231
271439951,1cyclictest0-21swapper/111:30:261
271439951,1cyclictest0-21swapper/109:25:181
271439951,1cyclictest0-21swapper/107:20:261
271439950,5cyclictest0-21swapper/109:08:161
271439950,5cyclictest0-21swapper/108:55:051
271439950,5cyclictest0-21swapper/108:55:041
271439950,5cyclictest0-21swapper/108:35:061
271439950,4cyclictest0-21swapper/111:50:011
271439950,4cyclictest0-21swapper/110:48:141
271439950,3cyclictest0-21swapper/112:10:211
271439950,3cyclictest0-21swapper/108:45:201
271439950,2cyclictest0-21swapper/110:30:201
271439950,2cyclictest0-21swapper/109:52:001
271439950,1cyclictest0-21swapper/107:50:241
271439950,1cyclictest0-21swapper/107:25:221
271599943,1cyclictest11097-21df08:50:253
271599942,1cyclictest27604-21cut07:10:203
271599940,0cyclictest0-21swapper/308:40:043
271439944,0cyclictest0-21swapper/110:27:251
271439944,0cyclictest0-21swapper/108:32:191
271439944,0cyclictest0-21swapper/108:05:381
271439942,1cyclictest0-21swapper/107:40:221
271439940,3cyclictest0-21swapper/109:00:181
271439940,3cyclictest0-21swapper/108:28:431
271439940,3cyclictest0-21swapper/108:01:561
271439940,3cyclictest0-21swapper/107:55:271
271439940,1cyclictest0-21swapper/110:05:081
271439940,0cyclictest0-21swapper/110:15:031
271439931,1cyclictest0-21swapper/108:15:231
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional