You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-08-16 - 07:00

x86 Intel Core i9-9900K @3600 MHz, Linux 5.15.49-rt47 (Profile)

Latency plot of system in rack #0, slot #8
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Up99 -i200 -h400 -q done
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8.osadl.org (updated Tue Aug 16, 2022 00:45:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
296602999480,40cyclictest1987702-21chrome00:23:2413
296602199484,19cyclictest3379639-21ThreadPoolForeg23:43:289
296602199480,40cyclictest1987702-21chrome20:28:259
296603499470,41cyclictest1987702-21chrome23:08:524
296603299470,40cyclictest1987702-21chrome00:03:412
296603499460,40cyclictest1987702-21chrome23:14:484
296603799454,24cyclictest3143408-21ThreadPoolForeg21:43:256
2966032994420,2cyclictest7356-21gnome-shell19:36:322
2966030994412,25cyclictest1987702-21chrome23:36:3014
2966034994324,19cyclictest0-21swapper/1222:47:254
2966037994210,26cyclictest0-21swapper/1400:01:036
296603399420,2cyclictest483-21chrome21:12:523
296602999420,41cyclictest3272113-21pkla-check-auth22:48:5513
2966037994141,0cyclictest0-21swapper/1421:12:516
2966034994110,25cyclictest0-21swapper/1223:46:264
2966034994110,25cyclictest0-21swapper/1222:57:424
2966033994110,27cyclictest0-21swapper/1123:29:033
2966032994110,25cyclictest0-21swapper/1022:40:282
2966032994110,25cyclictest0-21swapper/1020:17:452
296603199412,13cyclictest0-21swapper/921:43:2415
2966031994110,26cyclictest0-21swapper/921:39:5315
2966029994111,25cyclictest0-21swapper/720:40:5813
2966024994110,25cyclictest0-21swapper/520:30:2011
2966024994110,25cyclictest0-21swapper/519:35:5911
296602499410,1cyclictest1987702-21chrome23:10:2611
2966023994114,26cyclictest431irq_work/419:24:4510
2966021994112,27cyclictest29753591kworker/u32:019:57:589
296602099419,25cyclictest1987702-21chrome23:51:468
2966038994014,26cyclictest1201irq_work/1521:26:017
2966038994012,27cyclictest1201irq_work/1519:36:357
296603799409,25cyclictest0-21swapper/1400:11:096
296603799409,25cyclictest0-21swapper/1400:11:086
2966037994010,25cyclictest0-21swapper/1422:38:456
2966037994010,25cyclictest0-21swapper/1420:11:586
2966037994010,25cyclictest0-21swapper/1419:28:596
2966036994014,26cyclictest1061irq_work/1323:33:475
296603499409,25cyclictest0-21swapper/1223:29:484
296603499409,25cyclictest0-21swapper/1222:44:454
296603499409,25cyclictest0-21swapper/1200:12:434
296603499409,25cyclictest0-21swapper/1200:12:434
296603499409,25cyclictest0-21swapper/1200:00:404
296603499408,26cyclictest0-21swapper/1200:33:254
2966033994014,26cyclictest921irq_work/1120:28:343
296603299409,25cyclictest0-21swapper/1023:03:042
296603299405,0cyclictest0-21swapper/1019:41:402
296603199409,26cyclictest0-21swapper/920:41:3515
2966029994014,26cyclictest641irq_work/721:05:2613
2966025994014,26cyclictest571irq_work/622:20:1712
2966025994014,26cyclictest2975359-21kworker/u32:020:16:2212
2966025994013,26cyclictest571irq_work/623:30:3212
2966025994013,26cyclictest571irq_work/622:51:4412
2966025994013,26cyclictest571irq_work/621:54:2712
296602499409,25cyclictest0-21swapper/520:21:4811
296602499408,26cyclictest9867-21TaskCon~ller11
2966024994014,26cyclictest501irq_work/522:42:3511
2966024994014,26cyclictest501irq_work/521:21:2011
2966024994014,26cyclictest501irq_work/521:21:1911
296602399409,26cyclictest0-21swapper/419:38:0510
296602399409,26cyclictest0-21swapper/400:09:4810
2966023994013,26cyclictest431irq_work/422:35:2910
2966021994014,25cyclictest361irq_work/323:29:259
296602099409,25cyclictest0-21swapper/219:49:358
296603899399,25cyclictest0-21swapper/1523:22:467
2966038993913,25cyclictest1201irq_work/1521:09:317
2966038993912,26cyclictest29753591kworker/u32:020:40:457
296603799398,25cyclictest0-21swapper/1400:35:036
296603799390,14cyclictest0-21swapper/1420:49:396
296603799390,14cyclictest0-21swapper/1419:33:536
2966036993914,25cyclictest1061irq_work/1322:50:145
2966036993913,26cyclictest1061irq_work/1322:19:355
2966036993912,26cyclictest1061irq_work/1323:20:335
296603499399,25cyclictest0-21swapper/1223:59:224
296603499398,26cyclictest0-21swapper/1220:39:444
296603499398,25cyclictest0-21swapper/1223:44:324
2966034993914,25cyclictest991irq_work/1219:38:004
2966034993912,26cyclictest991irq_work/1219:40:034
296603399398,25cyclictest0-21swapper/1122:40:503
296603399398,25cyclictest0-21swapper/1120:13:323
2966033993913,26cyclictest921irq_work/1123:31:193
296603399390,14cyclictest0-21swapper/1122:55:223
296603399390,14cyclictest0-21swapper/1119:13:563
296603299399,25cyclictest0-21swapper/1022:34:072
296603299399,24cyclictest0-21swapper/1022:51:132
296603199390,15cyclictest0-21swapper/900:08:3415
296603099391,37cyclictest3445311-21meminfo00:15:2214
296603099391,37cyclictest3445311-21meminfo00:15:2214
296602999399,25cyclictest0-21swapper/722:37:0813
296602999398,26cyclictest0-21swapper/700:03:2313
2966029993913,26cyclictest641irq_work/723:23:0013
2966029993913,26cyclictest641irq_work/721:03:0813
2966029993912,26cyclictest641irq_work/720:53:5013
296602599399,25cyclictest0-21swapper/622:36:1812
296602599398,26cyclictest0-21swapper/622:15:5712
2966025993913,26cyclictest571irq_work/621:28:4112
296602499399,25cyclictest0-21swapper/523:42:5111
2966024993914,25cyclictest501irq_work/519:41:0811
2966024993914,25cyclictest501irq_work/519:34:3611
2966024993913,26cyclictest501irq_work/521:09:4211
2966024993913,25cyclictest501irq_work/523:00:5911
2966023993912,26cyclictest431irq_work/400:32:0610
296602399390,14cyclictest0-21swapper/423:19:1910
296602199398,25cyclictest0-21swapper/321:08:459
296602199398,25cyclictest0-21swapper/320:14:539
296602199398,25cyclictest0-21swapper/319:49:569
2966021993914,25cyclictest361irq_work/322:06:059
2966021993913,26cyclictest361irq_work/321:40:049
2966021993913,26cyclictest361irq_work/319:30:129
2966021993912,27cyclictest361irq_work/323:38:509
296602099398,25cyclictest0-21swapper/219:56:498
2966020993914,25cyclictest291irq_work/221:10:308
2966018993912,25cyclictest221irq_work/100:18:021
2966018993912,25cyclictest221irq_work/100:18:021
296603899387,26cyclictest0-21swapper/1520:05:347
2966038993812,26cyclictest1201irq_work/1521:52:277
2966038993812,25cyclictest1201irq_work/1520:18:407
2966038993811,26cyclictest1201irq_work/1521:10:257
296603799387,26cyclictest0-21swapper/1421:03:096
296603799386,12cyclictest3261588-21ThreadPoolForeg22:43:266
296603699387,6cyclictest124450irq/131-enp1s0-23:43:265
2966036993812,25cyclictest1061irq_work/1322:45:195
2966036993811,26cyclictest1061irq_work/1322:58:405
296603499387,26cyclictest0-21swapper/1220:33:484
296603499387,25cyclictest0-21swapper/1222:12:294
296603499387,25cyclictest0-21swapper/1220:18:334
296603499387,25cyclictest0-21swapper/1220:10:144
2966034993813,25cyclictest9950irq_work/1219:27:144
2966034993813,25cyclictest991irq_work/1219:20:524
2966034993813,25cyclictest991irq_work/1200:19:104
2966034993813,25cyclictest991irq_work/1200:19:094
2966034993812,26cyclictest991irq_work/1222:00:324
2966033993813,25cyclictest921irq_work/1123:10:483
2966033993812,25cyclictest921irq_work/1119:25:013
2966033993812,24cyclictest921irq_work/1120:07:443
296603299387,25cyclictest0-21swapper/1021:12:092
296603299386,26cyclictest0-21swapper/1020:06:482
296603199387,31cyclictest0-21swapper/921:48:3015
296603199380,35cyclictest0-21swapper/919:46:2615
296603199380,14cyclictest0-21swapper/920:34:5815
2966030993814,24cyclictest711irq_work/823:17:1814
2966030993813,25cyclictest711irq_work/823:04:1014
296603099380,14cyclictest0-21swapper/819:53:1514
296602999388,25cyclictest0-21swapper/722:16:0913
296602999388,25cyclictest0-21swapper/721:10:3713
2966029993813,25cyclictest641irq_work/720:03:3313
2966029993812,26cyclictest641irq_work/720:21:5513
296602599387,26cyclictest0-21swapper/622:43:3612
2966025993823,9cyclictest3817392-21chrome23:43:2612
2966025993813,25cyclictest571irq_work/621:02:5112
2966025993811,26cyclictest571irq_work/621:25:0012
2966025993811,26cyclictest571irq_work/621:25:0012
296602499388,25cyclictest0-21swapper/522:53:4811
296602499388,25cyclictest0-21swapper/519:59:0311
2966024993813,25cyclictest501irq_work/520:50:3711
2966024993811,26cyclictest501irq_work/521:45:2911
296602499380,13cyclictest0-21swapper/523:34:2311
296602399387,25cyclictest0-21swapper/423:37:3810
2966023993812,26cyclictest431irq_work/423:05:3110
2966023993812,26cyclictest431irq_work/400:10:5710
2966023993812,26cyclictest431irq_work/400:10:5610
2966021993813,25cyclictest361irq_work/323:54:169
2966021993813,25cyclictest361irq_work/322:55:289
2966021993813,25cyclictest361irq_work/321:58:339
2966021993813,25cyclictest361irq_work/321:22:409
2966021993813,25cyclictest361irq_work/321:22:409
2966021993811,26cyclictest361irq_work/319:15:129
2966020993813,25cyclictest291irq_work/221:47:218
2966020993813,25cyclictest291irq_work/220:54:058
2966020993811,26cyclictest291irq_work/220:17:318
2966018993814,24cyclictest221irq_work/123:57:491
2966018993814,24cyclictest221irq_work/119:26:001
2966018993813,24cyclictest221irq_work/121:39:251
2966017993813,25cyclictest181irq_work/023:36:330
2966017993813,25cyclictest181irq_work/019:40:080
2966017993813,24cyclictest181irq_work/022:50:550
296601799380,14cyclictest0-21swapper/023:22:390
296603899377,25cyclictest0-21swapper/1520:22:077
296603899377,25cyclictest0-21swapper/1500:32:367
2966038993712,25cyclictest1201irq_work/1522:39:547
2966038993711,26cyclictest1201irq_work/1519:30:337
2966037993712,25cyclictest1131irq_work/1422:16:376
2966037993710,27cyclictest1131irq_work/1420:44:466
2966036993712,25cyclictest1061irq_work/1322:03:595
2966036993711,25cyclictest1061irq_work/1320:45:165
2966036993711,25cyclictest1061irq_work/1320:36:185
296603699370,12cyclictest0-21swapper/1319:33:315
296603699370,11cyclictest0-21swapper/1319:24:305
296603499376,25cyclictest0-21swapper/1221:19:434
296603499376,25cyclictest0-21swapper/1221:19:434
296603499376,25cyclictest0-21swapper/1220:09:534
296603499376,25cyclictest0-21swapper/1219:17:214
2966034993712,25cyclictest991irq_work/1223:15:394
2966034993712,25cyclictest991irq_work/1200:28:574
296603399377,30cyclictest921irq_work/1120:15:323
296603399376,25cyclictest0-21swapper/1122:50:423
296603399376,25cyclictest0-21swapper/1119:44:373
296603399374,13cyclictest1006847-21ThreadPoolForeg21:43:253
2966033993712,25cyclictest921irq_work/1100:27:143
2966033993712,25cyclictest921irq_work/1100:19:233
2966033993712,25cyclictest921irq_work/1100:19:233
296603299376,25cyclictest0-21swapper/1000:25:092
2966032993712,25cyclictest851irq_work/1021:31:192
2966032993711,26cyclictest851irq_work/1019:57:582
2966030993713,24cyclictest711irq_work/821:29:5414
2966030993713,24cyclictest711irq_work/820:54:5614
2966029993711,26cyclictest641irq_work/720:57:3613
2966029993710,26cyclictest641irq_work/700:35:1813
296602599377,25cyclictest0-21swapper/623:13:0312
2966025993712,25cyclictest571irq_work/621:13:5212
296602499377,25cyclictest0-21swapper/521:55:3111
2966024993712,25cyclictest501irq_work/523:59:1711
2966024993711,26cyclictest501irq_work/523:19:4911
2966024993711,26cyclictest501irq_work/522:38:3311
2966024993710,26cyclictest501irq_work/522:30:4311
296602399377,25cyclictest0-21swapper/400:27:4910
296602399376,26cyclictest0-21swapper/422:34:4110
296602399376,25cyclictest0-21swapper/422:02:4710
296602399376,25cyclictest0-21swapper/420:53:4910
2966023993710,26cyclictest431irq_work/419:46:2610
296602199376,25cyclictest0-21swapper/323:20:219
2966021993711,26cyclictest361irq_work/300:31:049
2966018993712,24cyclictest221irq_work/119:48:161
2966018993712,24cyclictest221irq_work/119:44:021
2966017993713,24cyclictest181irq_work/000:38:500
2966017993712,25cyclictest1850irq_work/000:04:300
2966017993712,25cyclictest181irq_work/019:52:280
2966017993712,25cyclictest181irq_work/000:15:560
2966017993712,25cyclictest181irq_work/000:15:560
2966017993712,24cyclictest181irq_work/020:52:340
296603899365,26cyclictest0-21swapper/1500:36:387
296603899365,26cyclictest0-21swapper/1500:06:357
296603799365,25cyclictest0-21swapper/1423:08:586
2966037993610,26cyclictest1131irq_work/1423:56:466
2966037993610,26cyclictest1131irq_work/1420:21:486
296603699364,27cyclictest905490-21DOM22:54:445
2966036993610,25cyclictest0-21swapper/1300:16:035
2966036993610,25cyclictest0-21swapper/1300:16:035
296603499369,17cyclictest1006843-21ThreadPoolForeg21:43:254
296603499366,25cyclictest0-21swapper/1219:32:454
296603499365,26cyclictest0-21swapper/1221:24:094
296603499365,26cyclictest0-21swapper/1221:24:084
2966034993610,26cyclictest991irq_work/1220:26:254
296603399364,32cyclictest0-21swapper/1121:56:473
2966033993612,24cyclictest921irq_work/1119:50:213
2966032993610,26cyclictest851irq_work/1022:07:092
296603199365,31cyclictest0-21swapper/921:32:5715
2966031993611,24cyclictest0-21swapper/900:00:5815
2966030993613,23cyclictest711irq_work/820:25:2914
2966030993612,24cyclictest711irq_work/823:45:1514
2966030993612,1cyclictest711irq_work/821:06:0114
2966030993610,24cyclictest0-21swapper/820:02:5314
296602999366,25cyclictest0-21swapper/722:53:4313
296602999366,25cyclictest0-21swapper/722:05:1413
296602999366,25cyclictest0-21swapper/720:19:0713
2966029993611,25cyclictest641irq_work/720:10:4913
2966029993610,26cyclictest641irq_work/723:02:4213
296602599366,25cyclictest0-21swapper/622:04:1012
2966025993611,25cyclictest571irq_work/620:24:4312
2966025993611,25cyclictest571irq_work/600:35:2512
296602499369,26cyclictest501irq_work/521:44:2111
296602499366,25cyclictest0-21swapper/500:20:2611
2966024993610,26cyclictest501irq_work/520:59:1611
2966023993612,24cyclictest431irq_work/419:32:3010
2966023993611,1cyclictest431irq_work/420:18:0010
2966023993610,26cyclictest0-21swapper/400:21:1010
296602199369,26cyclictest361irq_work/300:00:279
296602199365,25cyclictest0-21swapper/322:20:509
2966021993611,25cyclictest361irq_work/322:25:209
2966020993611,25cyclictest291irq_work/222:00:478
2966020993611,25cyclictest291irq_work/221:50:238
2966020993611,25cyclictest291irq_work/221:41:508
296602099360,11cyclictest0-21swapper/221:29:318
2966018993612,23cyclictest221irq_work/121:15:011
2966018993612,23cyclictest221irq_work/121:15:011
2966017993612,24cyclictest181irq_work/021:17:320
2966017993612,24cyclictest181irq_work/021:17:320
2966017993612,1cyclictest181irq_work/020:31:430
2966017993611,24cyclictest181irq_work/021:28:030
2966017993610,25cyclictest0-21swapper/022:55:440
296601799360,13cyclictest0-21swapper/021:31:300
296603899355,25cyclictest0-21swapper/1520:59:337
296603899355,25cyclictest0-21swapper/1519:49:577
2966038993510,25cyclictest0-21swapper/1523:59:477
2966038993510,1cyclictest0-21swapper/1522:51:537
296603799359,26cyclictest1131irq_work/1400:21:346
296603799354,26cyclictest0-21swapper/1419:21:186
296603699358,26cyclictest1061irq_work/1319:58:345
296603699358,26cyclictest0-21swapper/1321:20:295
296603699358,26cyclictest0-21swapper/1321:20:295
2966036993510,25cyclictest0-21swapper/1322:34:255
2966036993510,25cyclictest0-21swapper/1321:45:035
2966036993510,25cyclictest0-21swapper/1320:51:485
2966036993510,1cyclictest1061irq_work/1300:02:445
2966034993510,25cyclictest991irq_work/1223:52:304
296603399354,25cyclictest0-21swapper/1120:33:023
296603399353,25cyclictest1987702-21chrome23:54:013
2966033993510,25cyclictest0-21swapper/1122:14:073
2966032993510,25cyclictest851irq_work/1020:44:152
2966032993510,25cyclictest3459990-21kworker/u32:300:31:582
2966032993510,25cyclictest0-21swapper/1019:45:512
2966032993510,24cyclictest0-21swapper/1000:05:412
296603199355,25cyclictest0-21swapper/922:14:1615
296603199351,27cyclictest26250irq/127-xhci_hc00:17:2015
296603199351,27cyclictest26250irq/127-xhci_hc00:17:2015
2966031993511,24cyclictest781irq_work/923:18:2215
2966031993510,24cyclictest0-21swapper/919:40:1415
2966030993511,24cyclictest711irq_work/823:26:5114
2966030993511,24cyclictest0-21swapper/823:50:2514
2966030993510,24cyclictest0-21swapper/821:17:4314
2966030993510,24cyclictest0-21swapper/821:17:4214
2966030993510,24cyclictest0-21swapper/819:43:0514
296602999358,26cyclictest641irq_work/719:26:0413
296602999355,25cyclictest0-21swapper/721:52:2513
296602999355,25cyclictest0-21swapper/721:36:4313
296602999355,25cyclictest0-21swapper/720:31:2313
296602999354,26cyclictest0-21swapper/721:48:0313
296602999354,26cyclictest0-21swapper/719:52:0213
2966029993511,1cyclictest641irq_work/720:25:3313
2966029993510,25cyclictest641irq_work/723:30:5413
2966029993510,25cyclictest641irq_work/719:15:3213
296602599355,25cyclictest0-21swapper/600:29:1212
296602599354,26cyclictest0-21swapper/622:49:5812
2966024993510,25cyclictest0-21swapper/500:10:2511
2966024993510,25cyclictest0-21swapper/500:10:2411
2966024993510,25cyclictest0-21swapper/500:02:2811
296602399359,26cyclictest0-21swapper/423:45:4610
296602399354,26cyclictest0-21swapper/420:36:0010
2966023993510,25cyclictest431irq_work/422:09:5510
2966023993510,25cyclictest431irq_work/421:40:4510
296602199354,26cyclictest0-21swapper/319:50:079
296602099354,25cyclictest0-21swapper/221:16:548
296602099354,25cyclictest0-21swapper/221:16:548
2966020993510,25cyclictest291irq_work/221:55:358
2966018993511,24cyclictest221irq_work/100:37:101
2966018993510,25cyclictest221irq_work/122:17:231
2966018993510,24cyclictest221irq_work/120:53:261
2966017993511,24cyclictest181irq_work/019:36:190
296603899349,25cyclictest1201irq_work/1522:04:557
296603899348,26cyclictest1201irq_work/1523:16:497
296603899348,26cyclictest1201irq_work/1522:22:307
296603899345,27cyclictest3202710-21ThreadPoolForeg22:14:597
296603899344,25cyclictest0-21swapper/1523:28:557
296603899344,25cyclictest0-21swapper/1520:50:477
296603899344,25cyclictest0-21swapper/1500:11:227
296603899344,25cyclictest0-21swapper/1500:11:217
296603799349,25cyclictest1131irq_work/1422:02:596
296603799348,26cyclictest1131irq_work/1420:37:296
296603799343,26cyclictest0-21swapper/1421:25:196
2966037993410,24cyclictest0-21swapper/1400:26:236
296603699349,25cyclictest0-21swapper/1319:41:205
296603699347,26cyclictest1061irq_work/1323:25:045
296603699343,25cyclictest0-21swapper/1322:35:555
296603499349,25cyclictest991irq_work/1219:56:214
296603499349,25cyclictest0-21swapper/1220:46:084
296603499343,25cyclictest0-21swapper/1221:45:184
296603399349,25cyclictest921irq_work/1121:38:113
296603399343,25cyclictest0-21swapper/1119:45:463
2966033993410,24cyclictest921irq_work/1120:57:123
2966033993410,0cyclictest0-21swapper/1121:23:013
2966033993410,0cyclictest0-21swapper/1121:23:013
296603299349,25cyclictest851irq_work/1023:55:452
296603299349,25cyclictest0-21swapper/1023:43:512
296603299349,24cyclictest0-21swapper/1019:54:512
296603299341,8cyclictest1006844-21ThreadPoolForeg23:45:512
2966032993410,24cyclictest0-21swapper/1022:01:412
296603199349,24cyclictest0-21swapper/922:37:0715
296603199349,24cyclictest0-21swapper/921:06:2015
2966031993411,23cyclictest0-21swapper/919:53:4915
296603199340,34cyclictest0-21swapper/923:45:5215
296603199340,1cyclictest0-21swapper/920:39:4215
296603099349,24cyclictest0-21swapper/820:19:3814
2966030993410,24cyclictest711irq_work/822:53:0014
2966030993410,24cyclictest711irq_work/821:00:3214
2966030993410,24cyclictest0-21swapper/819:47:3714
2966030993410,24cyclictest0-21swapper/819:37:1514
296602999348,26cyclictest641irq_work/721:25:1413
296602999348,26cyclictest641irq_work/721:21:4213
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional