You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-09-17 - 00:07
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 792 highest latencies:
System rack1slot1.osadl.org (updated Mon Sep 16, 2024 12:45:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
219252213142,25sleep20-21swapper/207:08:362
217122198161,24sleep00-21swapper/007:05:470
217902196158,26sleep10-21swapper/107:06:501
199242196159,24sleep30-21swapper/307:05:063
15772230,0sleep10-21swapper/107:30:201
299942140,0sleep00-21swapper/009:26:050
22130991210,1cyclictest28-21ksoftirqd/211:35:012
73502110,0sleep00-21swapper/009:38:200
2213699119,1cyclictest35-21ksoftirqd/311:20:003
22136991111,0cyclictest35-21ksoftirqd/310:54:593
2213699110,5cyclictest15427-21apt-get09:10:113
2213099119,1cyclictest28-21ksoftirqd/208:45:012
22130991110,1cyclictest28-21ksoftirqd/210:50:002
2213099110,1cyclictest0-21swapper/210:04:182
67572100,0sleep10-21swapper/110:50:421
2213699108,1cyclictest35-21ksoftirqd/307:59:593
2213699100,4cyclictest28490-21apt-get10:40:113
2211499109,1cyclictest9-21ksoftirqd/008:40:000
2211499106,2cyclictest10479-1kworker/u9:112:16:320
2211499104,5cyclictest7635-21perf12:10:000
2211499103,4cyclictest21870-21cat11:10:000
2211499102,4cyclictest6570-21perf11:30:000
221369998,1cyclictest35-21ksoftirqd/312:00:003
221369997,1cyclictest35-21ksoftirqd/311:10:143
221369990,5cyclictest26313-21apt-get10:00:153
221309998,1cyclictest28-21ksoftirqd/212:05:012
221309997,1cyclictest28-21ksoftirqd/207:20:002
221309995,2cyclictest28-21ksoftirqd/212:10:002
221309994,2cyclictest254-21systemd-journal08:35:002
221309990,8cyclictest1114-21cat08:40:002
221149998,1cyclictest9-21ksoftirqd/007:20:010
221149997,1cyclictest11746-21/usr/sbin/munin10:20:140
221149992,6cyclictest18665-1kworker/u9:010:30:240
221149990,2cyclictest26162-21apt-get08:25:020
221369988,0cyclictest35-21ksoftirqd/311:05:013
221369987,1cyclictest35-21ksoftirqd/309:45:003
221369987,1cyclictest35-21ksoftirqd/309:40:003
221369986,1cyclictest793-21snmpd11:46:063
221369986,1cyclictest793-21snmpd10:05:043
221369986,1cyclictest793-21snmpd09:22:493
221369986,1cyclictest793-21snmpd09:06:123
221369986,1cyclictest793-21snmpd08:27:243
221369986,1cyclictest793-21snmpd08:08:533
221369986,1cyclictest793-21snmpd07:52:453
221369986,1cyclictest35-21ksoftirqd/311:07:493
221369985,1cyclictest35-21ksoftirqd/308:15:003
221369980,5cyclictest21654-21apt-get08:15:133
221309988,0cyclictest28-21ksoftirqd/212:15:142
221309986,1cyclictest793-21snmpd11:10:292
221309986,1cyclictest793-21snmpd07:26:392
221309986,1cyclictest60950irq/123-eno111:53:582
221309986,1cyclictest60950irq/123-eno111:05:182
221309986,1cyclictest32247-21/usr/sbin/munin09:30:222
221309986,1cyclictest32247-21/usr/sbin/munin09:30:212
221309985,2cyclictest31312-21/usr/sbin/munin11:20:132
221309984,3cyclictest15151-21diskmemload09:10:122
221309983,1cyclictest0-21swapper/210:55:132
221309980,2cyclictest0-21swapper/207:10:112
221149987,1cyclictest16303-21ssh09:10:160
221149986,1cyclictest793-21snmpd08:48:240
221149981,6cyclictest5242-21ssh10:12:140
221149981,1cyclictest16984-21python10:25:200
221149981,1cyclictest16984-21python10:25:200
221149980,7cyclictest8157-21ssh10:53:360
221149980,7cyclictest16682-21ssh11:03:140
221149980,7cyclictest0-21swapper/010:57:370
221149980,1cyclictest16743-21ssh09:49:250
221149980,1cyclictest0-21swapper/012:35:010
221369976,1cyclictest793-21snmpd12:16:333
221369976,1cyclictest793-21snmpd09:17:263
221369976,1cyclictest793-21snmpd08:34:543
221369976,1cyclictest35-21ksoftirqd/312:25:013
221369976,1cyclictest35-21ksoftirqd/311:00:003
221369976,1cyclictest35-21ksoftirqd/310:35:143
221369976,1cyclictest35-21ksoftirqd/310:35:013
221369976,1cyclictest35-21ksoftirqd/308:45:003
221369975,1cyclictest793-21snmpd12:36:123
221369975,1cyclictest793-21snmpd11:37:533
221369975,1cyclictest793-21snmpd10:45:243
221369975,1cyclictest793-21snmpd10:14:113
221369975,1cyclictest793-21snmpd07:13:023
221369975,1cyclictest3771-21ssh11:25:173
221369975,1cyclictest35-21ksoftirqd/312:30:203
221369975,1cyclictest35-21ksoftirqd/312:15:003
221369975,1cyclictest31173-21ssh09:28:333
221369975,1cyclictest23004-21ssh12:26:393
221369975,1cyclictest17906-21ssh11:42:223
221369975,1cyclictest10620-21ssh10:19:313
221369974,3cyclictest567-21nscd12:00:133
221369974,1cyclictest31142-21sed07:25:153
221369971,3cyclictest0-21swapper/308:55:133
221369970,6cyclictest26154-21dump-pmu-power08:25:003
221369970,4cyclictest0-21swapper/309:47:343
221309976,1cyclictest9202-21cat08:55:172
221309976,1cyclictest60950irq/123-eno111:47:512
221309976,1cyclictest60950irq/123-eno111:41:342
221309976,1cyclictest60950irq/123-eno111:15:132
221309976,1cyclictest60950irq/123-eno110:38:322
221309976,1cyclictest28-21ksoftirqd/212:25:002
221309976,1cyclictest28-21ksoftirqd/210:05:142
221309976,1cyclictest28-21ksoftirqd/209:54:592
221309976,1cyclictest28-21ksoftirqd/209:15:132
221309976,1cyclictest28-21ksoftirqd/207:35:012
221309975,1cyclictest793-21snmpd08:50:172
221309975,1cyclictest793-21snmpd08:10:192
221309975,1cyclictest60950irq/123-eno112:35:172
221309975,1cyclictest60950irq/123-eno109:45:122
221309975,1cyclictest60950irq/123-eno109:35:192
221309975,1cyclictest60950irq/123-eno108:25:012
221309975,1cyclictest28-21ksoftirqd/212:25:002
221309975,1cyclictest28-21ksoftirqd/210:55:002
221309975,1cyclictest28-21ksoftirqd/207:49:592
221309975,1cyclictest0-21swapper/207:55:202
221309974,2cyclictest28-21ksoftirqd/212:10:532
221309973,2cyclictest254-21systemd-journal09:40:002
221309972,1cyclictest60950irq/123-eno111:25:222
221309971,0cyclictest10685-21apt-get09:00:102
221309970,6cyclictest11526-21/usr/sbin/munin11:35:142
221309970,6cyclictest0-21swapper/210:17:352
221239970,3cyclictest0-21swapper/111:20:111
221239970,1cyclictest0-21swapper/112:13:271
221239970,0cyclictest0-21swapper/107:47:171
221149976,1cyclictest2410-21ntp_states08:40:170
221149976,1cyclictest15425-1kworker/u9:009:50:200
221149975,1cyclictest8938-21ssh10:15:540
221149975,1cyclictest30964-21aten_r1power_vo11:20:120
221149975,1cyclictest30261-21ssh11:56:500
221149975,1cyclictest23124-1kworker/u9:110:37:400
221149975,1cyclictest13951-21users07:55:210
221149975,1cyclictest1098-21cut09:30:260
221149975,1cyclictest1098-21cut09:30:260
221149973,0cyclictest0-21swapper/012:20:050
221149970,6cyclictest8752-1kworker/u9:211:30:550
221149970,6cyclictest3093-21ssh12:04:160
221149970,6cyclictest27386-21ssh09:24:430
221149970,6cyclictest27353-21memory10:00:210
221149970,6cyclictest2519-21ssh10:46:010
221149970,6cyclictest23018-21tune2fs11:10:160
221149970,6cyclictest21142-21df_inode12:25:140
221149970,6cyclictest20148-21perl11:45:150
221149970,6cyclictest0-21swapper/009:40:160
221149970,6cyclictest0-21swapper/008:55:210
221149970,6cyclictest0-21swapper/008:15:170
221149970,5cyclictest0-21swapper/011:41:060
221149970,1cyclictest30423-21cstates07:25:120
221149970,1cyclictest29145-21ssh11:17:070
221149970,1cyclictest26532-21sed12:30:240
221149970,1cyclictest24175-21ssh09:56:160
221149970,1cyclictest23528-21ssh11:50:010
221149970,1cyclictest21332-21ssh09:15:460
18611270,0sleep10-21swapper/110:28:031
18611270,0sleep10-21swapper/110:28:031
13474270,0sleep00-21swapper/011:36:510
221369966,0cyclictest35-21ksoftirqd/310:20:133
221369965,1cyclictest8623-21ssh11:30:213
221369965,1cyclictest35-21ksoftirqd/308:55:003
221369965,1cyclictest35-21ksoftirqd/308:40:003
221369965,1cyclictest35-21ksoftirqd/308:05:013
221369965,1cyclictest35-21ksoftirqd/307:50:003
221369965,1cyclictest25013-21ntpq07:15:013
221369965,1cyclictest20458-21ssh09:53:053
221369964,1cyclictest793-21snmpd09:02:263
221369964,1cyclictest793-21snmpd07:21:493
221369964,1cyclictest5850-21cat07:40:153
221369964,1cyclictest23553-21ssh11:50:053
221369962,2cyclictest1243-21apt-get08:40:103
221369961,2cyclictest742-21runrttasks12:05:173
221369960,4cyclictest317-21diskstats09:30:203
221369960,4cyclictest317-21diskstats09:30:203
221369960,2cyclictest0-21swapper/311:20:133
221369960,1cyclictest0-21swapper/309:56:273
221309966,0cyclictest60950irq/123-eno112:30:272
221309966,0cyclictest60950irq/123-eno111:55:012
221309966,0cyclictest60950irq/123-eno110:40:242
221309966,0cyclictest60950irq/123-eno110:31:372
221309966,0cyclictest60950irq/123-eno110:10:052
221309966,0cyclictest60950irq/123-eno109:25:052
221309966,0cyclictest60950irq/123-eno109:20:072
221309966,0cyclictest60950irq/123-eno108:15:132
221309966,0cyclictest60950irq/123-eno107:43:112
221309965,1cyclictest24747-21ssh09:57:292
221309965,1cyclictest0-21swapper/207:50:072
221309965,0cyclictest60950irq/123-eno110:25:162
221309965,0cyclictest60950irq/123-eno110:25:162
221309964,1cyclictest60950irq/123-eno110:20:172
221309964,1cyclictest60950irq/123-eno109:05:112
221309964,1cyclictest60950irq/123-eno108:05:142
221309964,1cyclictest60950irq/123-eno107:20:162
221309964,1cyclictest28-21ksoftirqd/208:45:132
221309964,1cyclictest28-21ksoftirqd/208:25:172
221309963,2cyclictest13798-21tr11:00:132
221309960,6cyclictest0-21swapper/207:35:152
221239961,2cyclictest0-21swapper/110:49:571
221239960,6cyclictest0-21swapper/112:20:121
221239960,6cyclictest0-21swapper/112:08:241
221239960,6cyclictest0-21swapper/110:44:461
221239960,6cyclictest0-21swapper/110:30:051
221239960,6cyclictest0-21swapper/109:55:291
221239960,6cyclictest0-21swapper/108:50:471
221239960,6cyclictest0-21swapper/108:45:101
221239960,6cyclictest0-21swapper/107:50:571
221239960,5cyclictest0-21swapper/110:10:121
221239960,5cyclictest0-21swapper/108:25:231
221239960,2cyclictest0-21swapper/108:00:131
221239960,1cyclictest0-21swapper/112:29:521
221239960,1cyclictest0-21swapper/110:20:161
221239960,1cyclictest0-21swapper/108:15:171
221239960,0cyclictest0-21swapper/111:37:321
221239960,0cyclictest0-21swapper/110:55:161
221239960,0cyclictest0-21swapper/107:55:191
221149965,1cyclictest793-21snmpd07:39:310
221149964,1cyclictest793-21snmpd08:34:200
221149964,1cyclictest793-21snmpd07:14:090
221149964,1cyclictest24028-21/usr/sbin/munin08:20:200
221149962,3cyclictest101rcuc/008:04:590
221149961,5cyclictest7907-21ssh12:10:110
221149961,2cyclictest0-21swapper/009:05:140
221149960,6cyclictest0-21swapper/008:51:490
221149960,6cyclictest0-21swapper/007:20:250
221149960,1cyclictest22105-21cyclictest10:05:140
221149960,0cyclictest0-21swapper/010:40:110
221149960,0cyclictest0-21swapper/009:00:520
221149960,0cyclictest0-21swapper/007:50:560
221149960,0cyclictest0-21swapper/007:46:410
221149960,0cyclictest0-21swapper/007:30:220
221369951,1cyclictest15514-21ssh10:25:093
221369951,1cyclictest15514-21ssh10:25:093
221369950,1cyclictest2383-21apt-get07:35:103
221309950,4cyclictest14595-21apt-get08:00:122
221239955,0cyclictest0-21swapper/111:14:441
221239955,0cyclictest0-21swapper/109:17:351
221239953,1cyclictest27940-21chrt09:25:111
221239953,1cyclictest0-21swapper/108:55:151
221239951,2cyclictest0-21swapper/109:23:321
221239951,2cyclictest0-21swapper/107:10:101
221239950,5cyclictest0-21swapper/112:37:591
221239950,5cyclictest0-21swapper/111:48:001
221239950,5cyclictest0-21swapper/111:33:411
221239950,5cyclictest0-21swapper/109:54:051
221239950,5cyclictest0-21swapper/108:37:291
221239950,5cyclictest0-21swapper/108:33:241
221239950,5cyclictest0-21swapper/108:20:141
221239950,4cyclictest0-21swapper/111:41:531
221239950,3cyclictest0-21swapper/111:05:441
221239950,3cyclictest0-21swapper/110:20:011
221239950,2cyclictest0-21swapper/110:05:181
221239950,2cyclictest0-21swapper/109:30:171
221239950,2cyclictest0-21swapper/109:30:171
221239950,1cyclictest0-21swapper/112:15:111
221239950,1cyclictest0-21swapper/109:45:131
221239950,0cyclictest0-21swapper/112:00:201
221239950,0cyclictest0-21swapper/111:56:061
221239950,0cyclictest0-21swapper/110:01:091
221239950,0cyclictest0-21swapper/109:02:011
221149951,1cyclictest254-21systemd-journal07:40:000
221149950,4cyclictest0-21swapper/008:10:100
221149950,0cyclictest0-21swapper/008:05:240
31164240,0sleep10-21swapper/107:25:171
221369941,2cyclictest32449-21apt-get07:30:103
221239944,0cyclictest0-21swapper/108:40:051
221239943,1cyclictest0-21swapper/111:15:211
221239941,3cyclictest0-21swapper/107:37:141
221239940,4cyclictest0-21swapper/107:15:001
221239940,3cyclictest0-21swapper/111:25:121
221239940,2cyclictest0-21swapper/112:30:211
221239940,2cyclictest0-21swapper/109:40:381
221239940,2cyclictest0-21swapper/109:39:361
221239940,2cyclictest0-21swapper/109:10:111
221239940,1cyclictest0-21swapper/111:00:031
221239940,0cyclictest0-21swapper/110:37:361
221239940,0cyclictest0-21swapper/108:11:521
221239940,0cyclictest0-21swapper/108:05:201
221239940,0cyclictest0-21swapper/107:20:201
221239932,1cyclictest23554-21chrt11:50:051
221239930,1cyclictest0-21swapper/109:05:201
221239930,1cyclictest0-21swapper/107:40:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional