You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-09 - 06:07
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot6.osadl.org (updated Fri May 09, 2025 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31445521120,95pltrace0-21swapper/119:05:251
31445521120,105pltrace0-21swapper/619:05:146
31445521080,104pltrace0-21swapper/219:10:002
4208521040,0pltrace0-21swapper/123:05:041
31445521040,40pltrace0-21swapper/719:05:137
3144552980,89pltrace0-21swapper/519:05:065
315552910,0pltrace0-21swapper/022:05:030
3144552910,82pltrace0-21swapper/319:05:123
116725944,11sleep00-21swapper/019:06:180
112425944,11sleep40-21swapper/419:05:494
2198852350,5pltrace0-21swapper/700:35:147
1667452340,4pltrace0-21swapper/719:35:197
190752280,4pltrace0-21swapper/719:10:167
1773252270,0pltrace0-21swapper/220:35:032
1773252270,0pltrace0-21swapper/220:35:032
1773252220,5pltrace0-21swapper/720:36:137
1773252220,5pltrace0-21swapper/720:36:137
315552210,8pltrace0-21swapper/722:06:477
2822252210,0pltrace0-21swapper/722:52:307
1495052200,0pltrace0-21swapper/720:33:077
1429652200,6pltrace1132-21snmpd22:25:287
938152190,0pltrace0-21swapper/720:20:047
553552190,8pltrace0-21swapper/619:18:296
1708252190,12pltrace17644-21diskstats22:30:131
1708252190,11pltrace17249-21runrttasks22:30:117
2051752180,12pltrace21788-21cat20:40:153
2051752180,12pltrace21788-21cat20:40:153
2051752180,10pltrace20937-21expr20:40:112
2051752180,10pltrace20937-21expr20:40:112
1633991814,3cyclictest0-21swapper/623:09:026
806752170,12pltrace8972-21gltestperf00:10:151
806752170,11pltrace0-21swapper/700:10:147
659552170,4pltrace0-21swapper/720:17:437
3207452170,10pltrace0-21swapper/523:56:185
3207452170,10pltrace0-21swapper/523:56:185
2822252170,12pltrace0-21swapper/222:54:102
2716352170,0pltrace0-21swapper/721:50:207
2159352170,10pltrace0-21swapper/521:40:145
1708252170,12pltrace18336-21memory22:30:172
163899171,8cyclictest0-21swapper/721:48:517
163399171,15cyclictest0-21swapper/623:45:406
1621991711,5cyclictest0-21swapper/322:03:043
1621991711,5cyclictest0-21swapper/319:11:403
1616991711,5cyclictest0-21swapper/220:49:192
1616991711,5cyclictest0-21swapper/200:02:432
160999178,5cyclictest26794-21expr23:45:140
160999173,13cyclictest0-21swapper/019:29:280
1609991711,3cyclictest95152pltrace20:06:200
1256652170,0pltrace0-21swapper/723:20:417
806752160,11pltrace0-21swapper/200:10:172
3273652160,13pltrace0-21swapper/722:01:397
3100752160,3pltrace0-21swapper/722:55:547
2822252160,11pltrace0-21swapper/322:54:273
1708252160,11pltrace0-21swapper/522:30:195
1638991611,3cyclictest0-21swapper/722:38:407
163399161,8cyclictest0-21swapper/622:46:316
163399160,2cyclictest0-21swapper/619:55:166
163399160,15cyclictest0-21swapper/621:20:386
163099161,14cyclictest0-21swapper/520:35:165
163099161,14cyclictest0-21swapper/520:35:165
161699161,4cyclictest0-21swapper/221:54:372
161699160,4cyclictest0-21swapper/200:39:302
699652150,7pltrace1129-21modem-manager23:15:017
420852150,1pltrace0-21swapper/323:10:003
2927852150,3pltrace0-21swapper/023:51:560
2927852150,3pltrace0-21swapper/023:51:560
2649052150,5pltrace0-21swapper/723:45:307
2198852150,8pltrace24110-21taskset00:36:081
2198852150,1pltrace0-21swapper/300:40:013
2051752150,10pltrace21852-21missed_timers20:40:167
2051752150,10pltrace21852-21missed_timers20:40:167
1773252150,3pltrace19626-21cat20:35:193
1773252150,3pltrace19626-21cat20:35:193
163899150,8cyclictest0-21swapper/721:31:327
163899150,7cyclictest0-21swapper/722:48:087
1633991512,2cyclictest0-21swapper/621:05:186
163399150,14cyclictest0-21swapper/623:57:536
163399150,14cyclictest0-21swapper/623:57:536
163399150,14cyclictest0-21swapper/623:36:096
163399150,14cyclictest0-21swapper/620:41:256
163399150,14cyclictest0-21swapper/620:41:256
163099150,2cyclictest0-21swapper/523:35:175
162599150,8cyclictest0-21swapper/400:00:314
162599150,7cyclictest0-21swapper/420:15:534
162599150,4cyclictest0-21swapper/422:11:314
162599150,4cyclictest0-21swapper/419:24:434
1621991512,2cyclictest1023-21NetworkManager21:50:013
1621991512,2cyclictest0-21swapper/320:10:273
1621991511,3cyclictest0-21swapper/323:02:503
162199150,14cyclictest0-21swapper/320:06:273
162199150,14cyclictest0-21swapper/319:59:083
161699159,5cyclictest0-21swapper/223:36:122
161699151,7cyclictest0-21swapper/222:06:432
161699150,7cyclictest0-21swapper/220:21:512
161699150,14cyclictest0-21swapper/200:09:462
160999157,4cyclictest15-21ksoftirqd/023:25:180
160999157,2cyclictest32103-21basename23:55:120
160999157,2cyclictest32103-21basename23:55:120
160999156,8cyclictest0-21swapper/021:27:320
160999156,4cyclictest15-21ksoftirqd/022:55:140
160999152,12cyclictest0-21swapper/019:15:170
160999150,8cyclictest0-21swapper/000:30:270
160999150,7cyclictest0-21swapper/021:10:030
160999150,7cyclictest0-21swapper/020:37:550
160999150,7cyclictest0-21swapper/020:37:540
160999150,7cyclictest0-21swapper/020:27:580
160999150,4cyclictest0-21swapper/022:00:000
160999150,14cyclictest0-21swapper/019:10:130
1495052150,3pltrace17181-21chrt20:32:413
1429652150,3pltrace15397-21interrupts22:25:165
1429652150,1pltrace0-21swapper/322:25:153
1389152150,2pltrace15501-21sensors19:30:217
1151252150,0pltrace0-21swapper/722:24:387
806752140,9pltrace0-21swapper/500:10:175
699652140,8pltrace8132-21kernelversion23:10:163
594252140,0pltrace0-21swapper/722:10:377
3273652140,10pltrace1090-21dbus-daemon22:05:005
2264952140,6pltrace0-21swapper/522:42:025
2264952140,5pltrace0-21swapper/122:43:141
2159352140,8pltrace22709-21uname21:40:167
2159352140,5pltrace0-21swapper/121:40:331
2051752140,3pltrace21947-21ntp_kernel_pll_20:40:161
2051752140,3pltrace21947-21ntp_kernel_pll_20:40:161
1638991411,1cyclictest0-21swapper/723:57:487
1638991411,1cyclictest0-21swapper/723:57:487
163899140,6cyclictest0-21swapper/700:05:057
163899140,2cyclictest0-21swapper/719:55:197
163399149,3cyclictest0-21swapper/622:54:176
163399141,12cyclictest0-21swapper/622:35:406
163399140,8cyclictest0-21swapper/621:00:416
163399140,8cyclictest0-21swapper/620:37:006
163399140,8cyclictest0-21swapper/620:37:006
163399140,7cyclictest0-21swapper/619:31:566
163399140,13cyclictest0-21swapper/619:40:136
1630991411,2cyclictest1069-17sedispatch20:20:015
163099140,1cyclictest0-21swapper/520:30:155
162599147,6cyclictest0-21swapper/400:20:134
162599140,7cyclictest0-21swapper/421:14:514
162599140,13cyclictest0-21swapper/423:08:274
1621991411,2cyclictest0-21swapper/323:49:433
162199140,8cyclictest0-21swapper/322:41:353
162199140,7cyclictest0-21swapper/300:19:233
162199140,13cyclictest0-21swapper/321:24:363
161699149,3cyclictest0-21swapper/222:22:052
161699140,1cyclictest0-21swapper/219:55:132
160999147,2cyclictest22789-21latency21:40:160
160999146,4cyclictest15-21ksoftirqd/019:30:180
160999146,4cyclictest15-21ksoftirqd/000:15:250
160999145,3cyclictest23879-21ls23:40:140
160999144,4cyclictest15-21ksoftirqd/023:10:160
160999144,4cyclictest15-21ksoftirqd/022:37:080
160999144,4cyclictest15-21ksoftirqd/021:30:200
160999144,4cyclictest15-21ksoftirqd/019:45:180
1609991412,1cyclictest0-21swapper/020:58:110
1609991411,2cyclictest1384-21pltrace-run23:00:040
1609991411,2cyclictest0-21swapper/021:07:070
160999141,12cyclictest0-21swapper/000:21:400
1429652140,1pltrace16637-21taskset22:29:171
1389152140,9pltrace14510-21fschecks_count19:30:143
1389152140,10pltrace0-21swapper/119:30:051
1256652140,3pltrace13808-21cat23:20:203
1045652140,9pltrace78-21ksoftirqd/721:20:207
978052130,3pltrace10743-21sed23:15:133
978052130,1pltrace0-21swapper/523:15:205
806752130,7pltrace0-21swapper/300:10:173
69812130,1sleep4521rcuc/400:05:214
528352130,5pltrace0-21swapper/500:09:565
3207452130,5pltrace0-21swapper/123:57:471
3207452130,5pltrace0-21swapper/123:57:471
2887152130,3pltrace30819-21grep20:55:233
2887152130,1pltrace29145-21cpuspeed_turbos20:55:141
2543352130,3pltrace26347-21gltestperf22:45:163
2264952130,4pltrace0-21swapper/722:41:437
2198852130,5pltrace0-21swapper/500:36:145
2159352130,5pltrace0-21swapper/221:40:062
2159352130,4pltrace0-21swapper/321:40:153
2051752130,1pltrace0-21swapper/520:40:135
2051752130,1pltrace0-21swapper/520:40:135
1986452130,9pltrace20222-21tr22:35:145
1880652130,4pltrace0-21swapper/521:36:575
163899130,12cyclictest0-21swapper/723:33:377
163899130,12cyclictest0-21swapper/721:59:277
163899130,12cyclictest0-21swapper/719:21:027
163399138,2cyclictest0-21swapper/619:24:116
163399137,5cyclictest0-21swapper/619:13:256
163399130,13cyclictest0-21swapper/600:30:016
163399130,12cyclictest0-21swapper/621:52:366
163399130,12cyclictest0-21swapper/619:27:286
163099139,3cyclictest1028-17auditd22:00:015
1630991312,1cyclictest0-21swapper/520:25:045
1630991310,2cyclictest0-21swapper/521:30:395
1630991310,2cyclictest0-21swapper/521:22:515
1630991310,2cyclictest0-21swapper/519:28:435
163099130,12cyclictest0-21swapper/519:37:025
1625991312,1cyclictest0-21swapper/422:44:184
162599130,13cyclictest0-21swapper/422:35:144
162599130,12cyclictest0-21swapper/423:59:194
162599130,12cyclictest0-21swapper/423:59:194
162599130,12cyclictest0-21swapper/421:32:364
162599130,12cyclictest0-21swapper/419:35:164
162599130,12cyclictest0-21swapper/419:10:164
162599130,1cyclictest0-21swapper/419:55:214
162199134,8cyclictest20694-21fschecks_time22:35:163
162199133,8cyclictest0-21swapper/321:03:473
162199130,7cyclictest0-21swapper/322:16:153
162199130,13cyclictest0-21swapper/323:30:243
162199130,12cyclictest0-21swapper/323:55:073
162199130,12cyclictest0-21swapper/323:55:073
162199130,12cyclictest0-21swapper/322:13:453
162199130,12cyclictest0-21swapper/320:45:293
162199130,1cyclictest0-21swapper/300:30:153
161699138,4cyclictest0-21swapper/219:39:012
161699138,3cyclictest0-21swapper/221:06:252
1616991312,1cyclictest0-21swapper/200:15:242
161699130,6cyclictest0-21swapper/223:08:452
161699130,12cyclictest0-21swapper/222:14:592
161699130,12cyclictest0-21swapper/220:31:002
161699130,12cyclictest0-21swapper/200:33:232
161399130,13cyclictest0-21swapper/123:45:231
161399130,13cyclictest0-21swapper/120:05:201
160999137,2cyclictest15-21ksoftirqd/020:50:110
160999136,4cyclictest15-21ksoftirqd/019:40:130
160999136,4cyclictest0-21swapper/023:15:210
160999136,2cyclictest15-21ksoftirqd/022:50:140
160999135,4cyclictest15-21ksoftirqd/021:15:130
160999135,4cyclictest15-21ksoftirqd/000:05:170
160999133,6cyclictest15-21ksoftirqd/019:55:140
1609991311,1cyclictest0-21swapper/023:08:220
1609991311,1cyclictest0-21swapper/019:35:090
1609991310,2cyclictest700-21expr21:00:200
1609991310,2cyclictest26867-21ntpq22:45:190
1609991310,2cyclictest2586-21cut00:00:140
1609991310,2cyclictest19388-21fschecks_count21:35:140
1609991310,2cyclictest17101-21cut22:30:100
1609991310,2cyclictest12942-21ntpq22:20:160
1609991310,2cyclictest1132-21snmpd20:17:060
1609991310,2cyclictest0-21swapper/000:35:200
160999130,13cyclictest0-21swapper/022:10:140
160999130,12cyclictest0-21swapper/023:37:280
1429652130,1pltrace0-21swapper/222:25:232
1389152130,5pltrace0-21swapper/519:30:125
1151252130,3pltrace11810-21grep22:20:123
1151252130,1pltrace0-21swapper/522:24:595
978052120,1pltrace0-21swapper/723:15:127
528352120,3pltrace7623-21sshd00:09:181
3059952120,4pltrace30638-21wget20:00:103
2887152120,3pltrace0-21swapper/520:55:245
2608852120,3pltrace0-21swapper/520:50:445
247452120,1pltrace3025-21cstates00:00:153
247452120,0pltrace0-21swapper/700:02:487
2264952120,5pltrace0-21swapper/222:40:502
1920652120,8pltrace1132-21snmpd00:33:365
1880652120,9pltrace1132-21snmpd21:36:141
1708252120,4pltrace0-21swapper/322:30:233
163899126,5cyclictest0-21swapper/723:04:217
163899126,5cyclictest0-21swapper/722:16:377
163899124,7cyclictest0-21swapper/719:15:137
163399129,2cyclictest7317-21fschecks_time20:15:146
163399129,1cyclictest0-21swapper/622:31:316
163399128,3cyclictest1132-21snmpd00:28:096
163399127,4cyclictest3854-21head00:00:206
163399126,5cyclictest0-21swapper/623:29:366
163399126,5cyclictest0-21swapper/620:45:316
163399126,4cyclictest0-21swapper/623:43:256
1633991211,1cyclictest0-21swapper/623:30:256
1633991210,1cyclictest0-21swapper/619:35:236
163099129,2cyclictest0-21swapper/522:57:115
1625991211,1cyclictest0-21swapper/400:36:214
162599120,11cyclictest0-21swapper/420:35:094
162599120,11cyclictest0-21swapper/420:35:094
162599120,11cyclictest0-21swapper/420:09:594
162599120,1cyclictest0-21swapper/421:17:134
162199127,2cyclictest0-21swapper/323:41:113
162199126,5cyclictest0-21swapper/321:33:303
162199126,5cyclictest0-21swapper/321:16:423
162199126,5cyclictest0-21swapper/320:17:143
162199123,8cyclictest0-21swapper/320:25:513
1621991211,1cyclictest0-21swapper/319:35:223
161699126,5cyclictest0-21swapper/223:20:452
161699126,5cyclictest0-21swapper/220:02:322
161699126,5cyclictest0-21swapper/219:24:012
1616991211,1cyclictest0-21swapper/223:15:102
1616991210,1cyclictest0-21swapper/223:45:172
1616991210,1cyclictest0-21swapper/221:10:012
1613991211,1cyclictest0-21swapper/122:10:201
161399120,11cyclictest0-21swapper/122:15:011
160999129,2cyclictest8300-21expr00:10:130
160999129,2cyclictest5492-21sed20:10:230
160999129,2cyclictest31356-21tune2fs20:00:140
160999129,2cyclictest26379-21cat19:50:160
160999129,2cyclictest25825-21ntpq21:45:180
160999129,2cyclictest25358-21awk20:45:240
160999129,2cyclictest22059-21processes20:40:180
160999129,2cyclictest22059-21processes20:40:180
160999129,2cyclictest12777-21/usr/sbin/munin23:20:170
160999129,2cyclictest1271-21gltestperf22:00:160
160999126,3cyclictest15-21ksoftirqd/022:15:130
1609991210,1cyclictest0-21swapper/021:51:270
1609991210,1cyclictest0-21swapper/019:20:530
138052120,4pltrace54-21ksoftirqd/423:00:284
1216552120,0pltrace0-21swapper/120:26:131
1151252120,1pltrace0-21swapper/122:25:011
1085252120,4pltrace12304-21ntp_states00:15:211
1085252120,4pltrace11979-21irqstats_TLB00:15:205
872652110,1pltrace0-21swapper/222:15:142
806752110,3pltrace0-21swapper/600:10:146
699652110,3pltrace7228-21cpuspeed_turbos23:10:132
2822252110,3pltrace0-21swapper/522:54:225
2608852110,4pltrace191rcu_preempt20:50:161
2608852110,1pltrace78-21ksoftirqd/720:54:087
1945852110,2pltrace0-21swapper/719:40:117
1880652110,2pltrace21212-21munin-run21:40:013
1813752110,1pltrace0-21swapper/523:31:425
163899119,1cyclictest0-21swapper/720:10:107
163899117,3cyclictest1132-21snmpd19:26:567
163899116,3cyclictest0-21swapper/720:46:417
163899116,3cyclictest0-21swapper/700:32:507
163899115,5cyclictest0-21swapper/700:25:087
1638991110,1cyclictest0-21swapper/723:37:247
163899110,9cyclictest0-21swapper/721:17:177
163899110,11cyclictest0-21swapper/723:29:577
163899110,11cyclictest0-21swapper/721:27:327
163899110,11cyclictest0-21swapper/720:26:027
163899110,11cyclictest0-21swapper/700:19:487
163899110,10cyclictest0-21swapper/721:05:127
163899110,1cyclictest0-21swapper/723:53:557
163899110,1cyclictest0-21swapper/723:53:557
163899110,1cyclictest0-21swapper/720:08:237
163399119,1cyclictest0-21swapper/622:25:016
163399118,2cyclictest2824-21latency_hist22:04:596
163399118,2cyclictest13536-21gltestperf23:20:186
163399118,2cyclictest12343-21fschecks_time22:20:126
163399118,2cyclictest0-21swapper/623:11:326
163399117,3cyclictest9547-21timerwakeupswit21:15:236
163399116,3cyclictest0-21swapper/621:11:176
163399116,3cyclictest0-21swapper/600:15:476
163399115,5cyclictest0-21swapper/621:41:206
163399115,3cyclictest0-21swapper/620:20:076
163399114,4cyclictest0-21swapper/620:55:176
1633991110,1cyclictest0-21swapper/619:45:146
163399110,9cyclictest0-21swapper/600:23:216
163399110,11cyclictest0-21swapper/600:37:286
163399110,1cyclictest0-21swapper/622:55:276
163399110,1cyclictest0-21swapper/621:57:006
163399110,1cyclictest0-21swapper/620:02:046
163099118,2cyclictest1132-21snmpd21:12:115
1630991110,1cyclictest0-21swapper/500:20:095
163099110,11cyclictest0-21swapper/520:46:455
163099110,10cyclictest0-21swapper/520:07:435
163099110,1cyclictest0-21swapper/523:26:235
163099110,1cyclictest0-21swapper/521:19:185
163099110,1cyclictest0-21swapper/519:11:235
162599118,2cyclictest32595-21gltestperf21:00:164
162599118,2cyclictest26111-21awk21:45:224
162599118,2cyclictest19246-21runrttasks22:31:434
162599118,2cyclictest10727-21sed22:15:244
162599113,4cyclictest0-21swapper/423:25:174
162599112,4cyclictest54-21ksoftirqd/423:40:134
1625991110,1cyclictest0-21swapper/423:30:074
1625991110,1cyclictest0-21swapper/421:28:334
1625991110,1cyclictest0-21swapper/420:23:564
162599110,11cyclictest0-21swapper/422:25:414
162599110,11cyclictest0-21swapper/421:54:274
162599110,11cyclictest0-21swapper/420:40:574
162599110,11cyclictest0-21swapper/420:40:564
162599110,1cyclictest0-21swapper/423:52:284
162599110,1cyclictest0-21swapper/423:52:284
162599110,1cyclictest0-21swapper/421:23:514
162599110,1cyclictest0-21swapper/420:27:544
162199116,3cyclictest0-21swapper/323:36:543
162199115,5cyclictest1028-13auditd21:10:013
1621991110,1cyclictest0-21swapper/319:45:243
162199110,9cyclictest0-21swapper/320:20:183
162199110,6cyclictest0-21swapper/321:55:523
162199110,11cyclictest0-21swapper/322:05:383
161699118,2cyclictest1132-21snmpd00:28:382
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional