You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-26 - 20:01
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack3slot0.osadl.org (updated Tue Sep 26, 2023 12:43:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1178524826,8sleep50-21swapper/507:08:015
1156624826,8sleep70-21swapper/707:04:597
941424230,8sleep00-21swapper/007:04:280
1184324128,9sleep10-21swapper/107:08:461
1172624129,9sleep40-21swapper/407:07:124
941124028,8sleep60-21swapper/607:04:266
927324028,8sleep30-21swapper/307:04:233
942523928,8sleep20-21swapper/207:04:382
298482380,0sleep60-21swapper/607:39:406
12009991411,2cyclictest8842-21expr07:59:415
12009991411,2cyclictest8842-21expr07:59:415
12007991412,1cyclictest0-21swapper/312:29:523
1200699143,8cyclictest33-21ksoftirqd/210:26:372
1200599140,5cyclictest6393-21proc_pri08:49:481
1200599140,5cyclictest27207-21expr08:29:501
12006991311,1cyclictest33-21ksoftirqd/210:49:222
1200599130,5cyclictest9368-21which07:59:451
1200599130,5cyclictest9368-21which07:59:451
1200599130,5cyclictest19752-21pmu-power10:09:491
249372120,0sleep10-21swapper/111:14:511
1200999125,1cyclictest0-21swapper/509:34:225
1200599120,5cyclictest0-21swapper/110:39:451
1200599120,5cyclictest0-21swapper/107:49:421
1200999114,1cyclictest21620-21idleruntime-cro09:19:215
1200999113,2cyclictest0-21swapper/508:21:395
1200999110,1cyclictest0-21swapper/510:48:195
1200699119,1cyclictest33-21ksoftirqd/207:49:472
12006991110,1cyclictest33-21ksoftirqd/208:04:512
12006991110,1cyclictest33-21ksoftirqd/208:04:512
1200599110,7cyclictest0-21swapper/111:04:291
1200599110,5cyclictest0-21swapper/110:46:501
1200599110,4cyclictest0-21swapper/112:05:181
1200499119,1cyclictest16767-21munin-run08:14:210
1201899100,1cyclictest15891-21awk10:59:467
1200999100,1cyclictest0-21swapper/511:42:305
1200899109,1cyclictest49-21ksoftirqd/411:04:214
1200699109,1cyclictest33-21ksoftirqd/212:09:412
1200699108,1cyclictest33-21ksoftirqd/210:14:212
1200699108,1cyclictest33-21ksoftirqd/209:14:212
1200699100,10cyclictest0-21swapper/209:14:482
1200599100,9cyclictest777-21snmpd09:41:151
1200599100,9cyclictest25362-21cut10:19:501
1200599100,8cyclictest7243-21df_abs09:49:421
1200599100,8cyclictest31372-21sed07:39:501
1200599100,5cyclictest0-21swapper/112:17:291
1200599100,5cyclictest0-21swapper/111:58:141
1200599100,5cyclictest0-21swapper/109:27:211
1200599100,5cyclictest0-21swapper/109:15:281
1200599100,5cyclictest0-21swapper/109:13:171
1200599100,4cyclictest0-21swapper/107:29:211
1200599100,3cyclictest0-21swapper/110:59:391
1200599100,10cyclictest0-21swapper/111:24:381
120149999,0cyclictest0-21swapper/608:09:226
120149999,0cyclictest0-21swapper/608:09:226
120149990,0cyclictest0-21swapper/607:34:506
120099998,1cyclictest0-21swapper/510:38:105
120099990,9cyclictest0-21swapper/508:34:475
120099990,9cyclictest0-21swapper/508:34:475
120079990,9cyclictest0-21swapper/310:04:273
120069998,1cyclictest33-21ksoftirqd/211:44:212
120069998,1cyclictest33-21ksoftirqd/210:49:422
120069998,1cyclictest33-21ksoftirqd/207:59:472
120069998,1cyclictest33-21ksoftirqd/207:59:472
120069997,1cyclictest33-21ksoftirqd/211:34:212
120069997,1cyclictest33-21ksoftirqd/208:54:422
120069997,1cyclictest33-21ksoftirqd/207:29:512
120069990,9cyclictest0-21swapper/208:39:482
120069990,0cyclictest0-21swapper/208:24:492
120059994,4cyclictest0-21swapper/109:04:431
120059993,5cyclictest777-21snmpd12:24:161
120059993,5cyclictest777-21snmpd09:29:321
120059993,5cyclictest777-21snmpd08:48:431
120059993,5cyclictest777-21snmpd08:26:511
120059993,5cyclictest12640-21unixbench_singl08:59:571
120059992,6cyclictest26559-21httpd10:34:441
120059992,6cyclictest22119-21tr09:19:381
120059992,6cyclictest19996-21httpd11:19:521
120059992,6cyclictest19996-21httpd11:19:521
120059990,9cyclictest0-21swapper/112:35:211
120059990,9cyclictest0-21swapper/112:35:211
120059990,9cyclictest0-21swapper/112:00:121
120059990,9cyclictest0-21swapper/112:00:121
120059990,9cyclictest0-21swapper/111:49:481
120059990,9cyclictest0-21swapper/111:40:071
120059990,9cyclictest0-21swapper/111:10:041
120059990,9cyclictest0-21swapper/110:30:281
120059990,9cyclictest0-21swapper/110:24:391
120059990,9cyclictest0-21swapper/110:02:361
120059990,9cyclictest0-21swapper/109:35:101
120059990,9cyclictest0-21swapper/108:17:201
120059990,9cyclictest0-21swapper/108:10:301
120059990,9cyclictest0-21swapper/107:13:211
120059990,5cyclictest0-21swapper/112:12:261
120059990,5cyclictest0-21swapper/111:44:491
120059990,5cyclictest0-21swapper/110:56:191
120059990,5cyclictest0-21swapper/110:15:451
120059990,5cyclictest0-21swapper/108:55:221
120059990,5cyclictest0-21swapper/108:37:541
120059990,5cyclictest0-21swapper/108:37:541
120059990,5cyclictest0-21swapper/108:05:561
120059990,5cyclictest0-21swapper/108:05:561
120059990,5cyclictest0-21swapper/107:19:411
120059990,5cyclictest0-21swapper/107:14:411
120059990,4cyclictest0-21swapper/112:32:071
120059990,4cyclictest0-21swapper/111:30:311
120059990,4cyclictest0-21swapper/110:51:431
120059990,4cyclictest0-21swapper/109:44:481
120059990,4cyclictest0-21swapper/108:22:111
120059990,4cyclictest0-21swapper/107:33:451
120099988,0cyclictest0-21swapper/511:04:425
120099983,5cyclictest0-21swapper/509:24:265
120099983,1cyclictest0-21swapper/511:20:025
120099983,1cyclictest0-21swapper/511:20:025
120099980,1cyclictest0-21swapper/511:33:445
120099980,0cyclictest0-21swapper/511:35:025
120089986,1cyclictest49-21ksoftirqd/410:49:214
120089986,1cyclictest49-21ksoftirqd/409:54:214
120089986,1cyclictest49-21ksoftirqd/407:59:214
120089986,1cyclictest49-21ksoftirqd/407:59:214
120089983,2cyclictest0-21swapper/411:26:204
120089983,1cyclictest0-21swapper/410:15:394
120079986,1cyclictest41-21ksoftirqd/309:39:123
120079983,2cyclictest16774-21perf08:14:213
120079982,3cyclictest0-21swapper/311:14:213
120069988,0cyclictest33-21ksoftirqd/207:54:442
120069988,0cyclictest33-21ksoftirqd/207:54:442
120069988,0cyclictest0-21swapper/208:09:512
120069987,1cyclictest33-21ksoftirqd/211:14:402
120069987,1cyclictest33-21ksoftirqd/209:19:462
120069987,1cyclictest33-21ksoftirqd/208:49:392
120069986,2cyclictest33-21ksoftirqd/211:04:212
120069986,1cyclictest33-21ksoftirqd/211:04:432
120069986,1cyclictest33-21ksoftirqd/210:34:212
120069986,1cyclictest33-21ksoftirqd/208:29:432
120069983,2cyclictest0-21swapper/212:26:252
120069983,2cyclictest0-21swapper/211:53:182
120069983,0cyclictest0-21swapper/209:31:142
120059983,4cyclictest31920-21df_abs08:39:401
120059982,5cyclictest777-21snmpd12:24:371
120059982,5cyclictest777-21snmpd07:54:411
120059982,5cyclictest777-21snmpd07:54:411
120059980,8cyclictest0-21swapper/110:04:381
120059980,8cyclictest0-21swapper/109:54:501
120059980,7cyclictest784-21runrttasks07:34:441
120059980,7cyclictest2257-21tr11:34:371
120059980,4cyclictest0-21swapper/107:45:031
120049988,0cyclictest0-21swapper/012:09:530
120049982,3cyclictest0-21swapper/007:24:220
120049981,4cyclictest0-21swapper/008:44:210
23590270,0sleep60-21swapper/608:24:416
120189977,0cyclictest0-21swapper/708:09:497
120189975,1cyclictest73-21ksoftirqd/711:28:027
120189975,1cyclictest73-21ksoftirqd/709:43:297
120189975,1cyclictest73-21ksoftirqd/707:56:257
120189975,1cyclictest73-21ksoftirqd/707:56:257
120189975,1cyclictest5026-21iostat_ios12:34:437
120189975,1cyclictest5026-21iostat_ios12:34:437
120189973,2cyclictest0-21swapper/710:55:197
120189973,2cyclictest0-21swapper/710:41:107
120189972,2cyclictest22686-21munin-run11:14:217
120189972,2cyclictest0-21swapper/710:09:337
120149977,0cyclictest0-21swapper/607:29:576
120149975,1cyclictest65-21ksoftirqd/610:34:206
120149975,1cyclictest65-21ksoftirqd/608:02:206
120149975,1cyclictest65-21ksoftirqd/608:02:206
120149974,2cyclictest917-21irqstats07:44:466
120149974,2cyclictest30827-21cstates09:34:416
120149973,1cyclictest0-21swapper/612:39:166
120149973,1cyclictest0-21swapper/612:39:166
120149972,1cyclictest0-21swapper/612:12:466
120149970,4cyclictest0-21swapper/611:45:486
120099975,1cyclictest57-21ksoftirqd/507:49:515
120099975,1cyclictest28292-21latency_hist08:34:225
120099974,2cyclictest14813-21nfsd411:54:465
120099972,2cyclictest0-21swapper/511:29:205
120099972,1cyclictest0-21swapper/509:12:175
120099970,4cyclictest0-21swapper/511:48:035
120089975,1cyclictest49-21ksoftirqd/409:25:564
120089975,1cyclictest49-21ksoftirqd/407:29:224
120089972,2cyclictest0-21swapper/409:21:234
120089972,2cyclictest0-21swapper/409:11:194
120089972,2cyclictest0-21swapper/408:54:524
120089972,2cyclictest0-21swapper/408:34:514
120089972,2cyclictest0-21swapper/408:34:514
120089972,1cyclictest0-21swapper/408:59:584
120089970,7cyclictest0-21swapper/411:54:404
120089970,1cyclictest0-21swapper/409:56:124
120079975,1cyclictest41-21ksoftirqd/308:51:173
120079975,1cyclictest41-21ksoftirqd/307:24:223
120079974,2cyclictest23499-21ntp_states09:19:463
120079972,4cyclictest4352-21df09:44:413
120079972,2cyclictest0-21swapper/311:49:093
120069977,0cyclictest0-21swapper/208:44:222
120069976,1cyclictest33-21ksoftirqd/212:29:442
120069976,1cyclictest33-21ksoftirqd/211:44:432
120069976,1cyclictest33-21ksoftirqd/210:34:462
120069976,1cyclictest33-21ksoftirqd/208:34:482
120069976,1cyclictest33-21ksoftirqd/208:34:482
120069976,1cyclictest33-21ksoftirqd/207:44:222
120069976,1cyclictest33-21ksoftirqd/207:34:422
120069975,1cyclictest33-21ksoftirqd/212:19:382
120069975,1cyclictest33-21ksoftirqd/212:14:432
120069975,1cyclictest33-21ksoftirqd/212:04:482
120069975,1cyclictest33-21ksoftirqd/211:24:422
120069975,1cyclictest33-21ksoftirqd/211:19:442
120069975,1cyclictest33-21ksoftirqd/211:19:442
120069975,1cyclictest33-21ksoftirqd/210:54:382
120069975,1cyclictest33-21ksoftirqd/210:14:382
120069975,1cyclictest33-21ksoftirqd/209:59:222
120069975,1cyclictest33-21ksoftirqd/209:49:482
120069975,1cyclictest33-21ksoftirqd/209:44:512
120069975,1cyclictest33-21ksoftirqd/209:39:432
120069975,1cyclictest33-21ksoftirqd/209:24:492
120069975,1cyclictest33-21ksoftirqd/208:59:572
120069975,1cyclictest33-21ksoftirqd/208:19:452
120069975,1cyclictest33-21ksoftirqd/208:14:442
120069975,1cyclictest33-21ksoftirqd/207:46:272
120069975,1cyclictest33-21ksoftirqd/207:19:462
120069975,1cyclictest33-21ksoftirqd/207:14:212
120069975,1cyclictest311rcuc/212:39:212
120069975,1cyclictest311rcuc/212:39:212
120069975,1cyclictest253241nfsd410:19:502
120069973,4cyclictest33-21ksoftirqd/210:04:472
120069970,4cyclictest0-21swapper/211:54:442
120049977,0cyclictest0-21swapper/008:51:530
120049975,1cyclictest9-21ksoftirqd/011:19:210
120049975,1cyclictest9-21ksoftirqd/011:13:000
120049975,1cyclictest9-21ksoftirqd/009:15:510
120049975,1cyclictest9-21ksoftirqd/008:44:420
120049972,5cyclictest0-21swapper/010:04:270
120049972,2cyclictest0-21swapper/007:28:230
120049971,3cyclictest10143-21munin-run08:59:210
120049970,7cyclictest0-21swapper/009:39:580
120049970,7cyclictest0-21swapper/008:06:000
120049970,7cyclictest0-21swapper/008:06:000
120049970,1cyclictest0-21swapper/009:59:210
120189966,0cyclictest0-21swapper/712:24:447
120189965,1cyclictest73-21ksoftirqd/709:03:347
120189964,1cyclictest907-21awk07:44:457
120189964,1cyclictest8453-21sensors_fan09:49:517
120189964,1cyclictest73-21ksoftirqd/711:24:217
120189964,1cyclictest73-21ksoftirqd/711:24:217
120189964,1cyclictest73-21ksoftirqd/709:10:057
120189964,1cyclictest73-21ksoftirqd/707:19:227
120189964,1cyclictest29243-21cat07:39:227
120189964,1cyclictest2389-21df08:44:437
120189964,1cyclictest18514-21/usr/sbin/munin07:19:417
120189964,1cyclictest1213-21users08:39:507
120189963,2cyclictest777-21snmpd12:06:577
120189963,2cyclictest4637-21sed07:49:517
120189963,0cyclictest0-21swapper/711:42:547
120189963,0cyclictest0-21swapper/709:35:507
120189961,2cyclictest0-21swapper/711:34:117
120189961,2cyclictest0-21swapper/708:54:437
120189960,5cyclictest9747-21sed10:49:457
120189960,3cyclictest0-21swapper/709:07:497
120189960,1cyclictest0-21swapper/708:51:587
120189960,1cyclictest0-21swapper/707:59:457
120189960,1cyclictest0-21swapper/707:59:457
120149964,1cyclictest777-21snmpd12:32:046
120149964,1cyclictest65-21ksoftirqd/612:00:216
120149964,1cyclictest65-21ksoftirqd/612:00:216
120149964,1cyclictest65-21ksoftirqd/608:47:136
120149964,1cyclictest4263-21nfsd410:39:456
120149964,1cyclictest17403-21cpuspeed_turbos11:04:406
120149964,1cyclictest15718-21chrt09:07:326
120149964,1cyclictest1256-21sed11:29:576
120149963,3cyclictest0-21swapper/610:08:006
120149963,3cyclictest0-21swapper/608:39:226
120149963,3cyclictest0-21swapper/608:39:226
120149963,2cyclictest777-21snmpd11:13:506
120149963,2cyclictest777-21snmpd10:54:016
120149963,2cyclictest777-21snmpd10:44:346
120149963,2cyclictest777-21snmpd08:43:406
120149963,2cyclictest12136-21processes08:59:546
120149962,3cyclictest26663-21timerandwakeup09:24:526
120149960,6cyclictest0-21swapper/612:14:416
120149960,6cyclictest0-21swapper/610:38:106
120149960,5cyclictest222532sleep608:20:566
120149960,3cyclictest0-21swapper/607:50:206
120149960,3cyclictest0-21swapper/607:27:506
120149960,0cyclictest0-21swapper/607:15:556
120099964,1cyclictest57-21ksoftirqd/510:56:135
120099964,1cyclictest57-21ksoftirqd/510:09:215
120099964,1cyclictest57-21ksoftirqd/507:39:225
120099964,1cyclictest57-21ksoftirqd/507:09:245
120099964,1cyclictest17432-21latency11:59:485
120099964,1cyclictest17432-21latency11:59:485
120099964,1cyclictest16120-21interrupts07:14:455
120099963,2cyclictest777-21snmpd12:29:345
120099963,2cyclictest777-21snmpd11:53:155
120099963,2cyclictest777-21snmpd11:18:055
120099963,2cyclictest777-21snmpd10:53:285
120099963,2cyclictest777-21snmpd10:28:135
120099963,2cyclictest777-21snmpd09:55:095
120099963,2cyclictest777-21snmpd09:05:165
120099963,2cyclictest777-21snmpd09:00:405
120099963,2cyclictest777-21snmpd08:15:575
120099963,2cyclictest777-21snmpd07:57:445
120099963,2cyclictest777-21snmpd07:57:445
120099963,2cyclictest777-21snmpd07:40:385
120099963,2cyclictest777-21snmpd07:25:505
120099963,2cyclictest32394-21cat08:39:435
120099963,2cyclictest22677-21latency_hist11:14:215
120099963,2cyclictest15347-21cat10:59:435
120099960,5cyclictest23806-21/usr/sbin/munin07:29:485
120099960,3cyclictest0-21swapper/508:05:155
120099960,3cyclictest0-21swapper/508:05:155
120099960,2cyclictest7671-21/usr/sbin/munin08:54:465
120099960,1cyclictest0-21swapper/508:49:225
120089965,1cyclictest49-21ksoftirqd/411:19:594
120089965,1cyclictest49-21ksoftirqd/411:19:594
120089965,1cyclictest49-21ksoftirqd/410:34:214
120089964,1cyclictest777-21snmpd10:25:014
120089964,1cyclictest777-21snmpd10:04:244
120089964,1cyclictest49-21ksoftirqd/407:53:084
120089964,1cyclictest25220-21runrttasks11:15:064
120089963,2cyclictest777-21snmpd12:28:494
120089963,2cyclictest777-21snmpd11:53:034
120089963,2cyclictest777-21snmpd10:20:334
120089963,2cyclictest777-21snmpd09:37:024
120089963,2cyclictest777-21snmpd09:16:534
120089963,2cyclictest777-21snmpd08:45:284
120089963,2cyclictest19705-21sendmail_mailtr07:19:504
120089963,2cyclictest15172-21if_em208:09:454
120089963,1cyclictest0-21swapper/412:10:124
120089961,2cyclictest0-21swapper/410:11:214
120089961,2cyclictest0-21swapper/407:29:504
120089960,6cyclictest0-21swapper/407:11:434
120089960,5cyclictest0-21swapper/411:34:224
120089960,3cyclictest0-21swapper/408:28:484
120089960,3cyclictest0-21swapper/408:04:304
120089960,3cyclictest0-21swapper/408:04:304
120089960,3cyclictest0-21swapper/407:19:214
120079964,1cyclictest777-21snmpd08:36:303
120079964,1cyclictest777-21snmpd08:36:303
120079964,1cyclictest4105-21cut10:39:443
120079964,1cyclictest41-21ksoftirqd/312:29:213
120079964,1cyclictest41-21ksoftirqd/311:38:123
120079964,1cyclictest41-21ksoftirqd/309:04:213
120079964,1cyclictest26842-21timerandwakeup12:14:483
120079964,1cyclictest24755-21/usr/sbin/munin07:29:503
120079964,1cyclictest23845-21irqstats08:24:433
120079964,1cyclictest2368-21sshd10:35:513
120079964,1cyclictest1393-21awk07:44:483
120079964,1cyclictest12195-21latency_hist09:59:213
120079963,2cyclictest9437-21cat07:59:453
120079963,2cyclictest9437-21cat07:59:453
120079963,2cyclictest8542-21cat08:54:433
120079963,2cyclictest777-21snmpd12:03:003
120079963,2cyclictest777-21snmpd12:03:003
120079963,2cyclictest777-21snmpd11:33:473
120079963,2cyclictest777-21snmpd11:24:563
120079963,2cyclictest777-21snmpd11:23:023
120079963,2cyclictest777-21snmpd11:23:023
120079963,2cyclictest777-21snmpd11:00:533
120079963,2cyclictest777-21snmpd10:52:253
120079963,2cyclictest777-21snmpd10:24:463
120079963,2cyclictest777-21snmpd10:19:513
120079963,2cyclictest777-21snmpd10:13:003
120079963,2cyclictest777-21snmpd09:16:323
120079963,2cyclictest777-21snmpd08:39:493
120079963,2cyclictest777-21snmpd07:42:053
120079963,2cyclictest777-21snmpd07:16:133
120079963,2cyclictest30595-21cat10:29:453
120079963,2cyclictest30348-21idleruntime-cro12:24:213
120079963,2cyclictest23995-21cat11:14:433
120079961,5cyclictest401ktimersoftd/309:53:123
120079961,2cyclictest0-21swapper/308:08:033
120079961,2cyclictest0-21swapper/308:08:033
120079960,3cyclictest0-21swapper/308:16:543
120079960,1cyclictest0-21swapper/309:39:423
120069965,1cyclictest33-21ksoftirqd/211:34:372
120069965,1cyclictest33-21ksoftirqd/211:09:372
120069965,1cyclictest33-21ksoftirqd/209:04:282
120069964,1cyclictest33-21ksoftirqd/211:59:422
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional