You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-21 - 00:25
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot6.osadl.org (updated Sat Dec 20, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3510521190,56pltrace0-21swapper/707:05:037
3510521020,93pltrace0-21swapper/107:05:161
2097152990,1pltrace601rcuc/512:22:505
404729559,10sleep60-21swapper/607:05:036
351052880,80pltrace0-21swapper/207:07:092
627428044,10sleep30-21swapper/307:07:413
351052650,57pltrace0-21swapper/507:05:145
622325945,10sleep00-21swapper/007:06:580
613525844,10sleep40-21swapper/407:05:414
2950652340,5pltrace0-21swapper/707:50:187
2393152340,4pltrace0-21swapper/707:40:137
656952320,3pltrace0-21swapper/709:05:157
2500652320,4pltrace0-21swapper/708:40:167
1278952310,1pltrace12793-21pltrace-run07:17:495
1153352310,1pltrace0-21swapper/711:07:517
1557552290,3pltrace0-21swapper/707:24:177
1710852270,1pltrace17112-21pltrace-run11:17:505
421552200,0pltrace0-21swapper/711:55:167
665299190,3cyclictest0-21swapper/111:33:191
2331452190,13pltrace0-21swapper/209:35:112
1818352190,5pltrace0-21swapper/712:20:167
766952180,0pltrace0-21swapper/710:03:057
6673991812,5cyclictest0-21swapper/610:28:546
6669991812,5cyclictest0-21swapper/509:59:415
665299184,8cyclictest15885-21cat07:25:011
2654352180,11pltrace28682-21sensors12:35:223
6676991712,3cyclictest0-21swapper/709:42:327
667699171,8cyclictest0-21swapper/711:03:477
3275052170,1pltrace0-21swapper/610:50:526
2654352170,11pltrace0-21swapper/012:35:210
2375552170,0pltrace0-21swapper/712:30:177
766952160,13pltrace54-21ksoftirqd/410:07:354
667699161,8cyclictest0-21swapper/711:21:497
667699161,14cyclictest0-21swapper/708:20:187
667699160,9cyclictest0-21swapper/711:42:307
6673991610,5cyclictest0-21swapper/607:51:386
6673991610,3cyclictest0-21swapper/607:15:076
667399161,14cyclictest0-21swapper/609:06:586
667399161,14cyclictest0-21swapper/608:44:186
667399160,8cyclictest0-21swapper/609:54:486
667399160,4cyclictest0-21swapper/608:08:176
666999160,11cyclictest0-21swapper/507:42:215
6660991610,5cyclictest0-21swapper/312:24:313
6660991610,5cyclictest0-21swapper/311:00:373
665599161,8cyclictest0-21swapper/208:06:082
665599160,2cyclictest20793-21ntp_states10:25:192
665599160,15cyclictest0-21swapper/211:35:322
665599160,15cyclictest0-21swapper/211:00:172
665599160,15cyclictest0-21swapper/209:29:482
592452160,10pltrace54-21ksoftirqd/411:01:004
484952160,11pltrace54-21ksoftirqd/409:59:244
2654352160,8pltrace27864-21cut12:35:162
2439052160,3pltrace0-21swapper/710:33:117
2160252160,0pltrace0-21swapper/710:28:107
2114652160,1pltrace21150-21pltrace-run07:32:496
936552150,3pltrace12019-21chrt09:10:576
6676991510,3cyclictest0-21swapper/710:45:497
667699150,7cyclictest0-21swapper/708:17:187
667699150,4cyclictest0-21swapper/710:11:327
667699150,14cyclictest0-21swapper/710:57:197
667699150,14cyclictest0-21swapper/708:50:007
6673991512,2cyclictest19824-21iostat_ios12:20:196
667399150,8cyclictest0-21swapper/611:09:466
667399150,3cyclictest0-21swapper/612:32:286
666999157,7cyclictest1478-21lldpd11:27:495
6669991512,2cyclictest0-21swapper/509:30:105
6669991510,4cyclictest0-21swapper/512:07:135
6669991510,4cyclictest0-21swapper/510:36:145
666999151,8cyclictest0-21swapper/511:55:005
666999150,14cyclictest0-21swapper/512:31:215
6660991510,3cyclictest0-21swapper/311:55:063
666099150,7cyclictest0-21swapper/309:07:223
666099150,7cyclictest0-21swapper/307:13:223
665599159,5cyclictest0-21swapper/210:22:062
665599159,5cyclictest0-21swapper/210:22:062
665599151,7cyclictest0-21swapper/212:25:312
665599150,7cyclictest0-21swapper/211:58:182
665599150,7cyclictest0-21swapper/209:17:552
665599150,4cyclictest0-21swapper/209:21:482
665599150,3cyclictest0-21swapper/210:42:502
665599150,14cyclictest0-21swapper/209:54:562
665599150,14cyclictest0-21swapper/208:17:482
665599150,14cyclictest0-21swapper/207:41:472
665299155,3cyclictest17250-21timerandwakeup09:20:221
664799150,14cyclictest0-21swapper/012:20:230
592452150,10pltrace0-21swapper/011:01:080
3058152150,9pltrace0-21swapper/608:48:026
2888852150,9pltrace1474-21lldpd09:46:480
1539852150,11pltrace0-21swapper/012:12:540
667699148,2cyclictest78-21ksoftirqd/712:00:137
667699140,14cyclictest0-21swapper/711:45:197
667699140,10cyclictest32218-21/usr/sbin/munin09:50:217
667699140,1cyclictest0-21swapper/710:40:147
667399148,5cyclictest0-21swapper/611:34:536
667399148,3cyclictest0-21swapper/612:36:016
667399148,3cyclictest0-21swapper/609:41:266
6673991412,1cyclictest0-21swapper/611:45:116
6673991411,2cyclictest1055-17auditd09:20:006
667399141,1cyclictest0-21swapper/608:15:206
667399140,3cyclictest0-21swapper/611:02:496
667399140,1cyclictest0-21swapper/610:05:176
666999149,4cyclictest0-21swapper/510:50:005
666999145,8cyclictest1146-21snmpd08:12:295
666999140,13cyclictest0-21swapper/509:20:285
666999140,11cyclictest0-21swapper/510:17:275
6660991410,2cyclictest0-21swapper/307:18:473
666099141,12cyclictest0-21swapper/308:53:253
666099140,7cyclictest0-21swapper/309:00:403
666099140,3cyclictest0-21swapper/311:23:323
666099140,12cyclictest0-21swapper/311:40:593
666099140,1cyclictest0-21swapper/311:25:203
665599148,5cyclictest0-21swapper/211:24:142
6655991411,2cyclictest0-21swapper/207:55:142
6655991410,3cyclictest0-21swapper/210:03:292
665599140,13cyclictest0-21swapper/207:30:182
665299144,8cyclictest3789-21cpu10:55:141
665299142,5cyclictest21590-21expr09:30:131
665299141,12cyclictest0-21swapper/109:50:001
665299140,2cyclictest0-21swapper/111:54:091
3275052140,6pltrace0-21swapper/010:48:210
2672052140,10pltrace0-21swapper/007:47:000
1153352140,4pltrace0-21swapper/711:10:037
667699139,2cyclictest0-21swapper/712:06:067
667699138,3cyclictest761rcuc/710:38:117
667699138,2cyclictest78-21ksoftirqd/711:25:147
667699137,2cyclictest78-21ksoftirqd/708:35:157
667699137,2cyclictest21632-21awk09:30:147
667699132,7cyclictest78-21ksoftirqd/712:15:227
6676991310,2cyclictest3235-21timerandwakeup08:55:237
6676991310,2cyclictest28414-21cat12:35:197
6676991310,2cyclictest16589-21basename10:20:137
6676991310,2cyclictest16589-21basename10:20:137
6676991310,2cyclictest15521-21aten_r4power_cu09:20:127
6676991310,2cyclictest12206-21taskset09:13:167
6676991310,2cyclictest1178-21runrttasks11:15:267
667699131,6cyclictest78-21ksoftirqd/710:15:027
667699131,1cyclictest771ktimers/709:15:337
667699130,12cyclictest0-21swapper/711:35:197
667399138,4cyclictest11190-21cpuspeed_turbos10:10:116
6673991312,1cyclictest0-21swapper/608:27:496
667399130,2cyclictest0-21swapper/610:46:196
667399130,12cyclictest0-21swapper/612:08:196
667399130,12cyclictest0-21swapper/610:55:226
666999136,6cyclictest1146-21snmpd11:55:485
666999135,3cyclictest1178-21runrttasks11:12:215
666999134,2cyclictest1178-21runrttasks12:02:555
666999134,2cyclictest1146-21snmpd08:08:205
666999133,3cyclictest16177-21tr12:15:135
666999132,3cyclictest27598-21gltestperf09:40:175
6669991310,2cyclictest0-21swapper/509:17:015
666999130,12cyclictest0-21swapper/509:52:415
666999130,12cyclictest0-21swapper/507:49:465
666499138,4cyclictest1055-17auditd07:50:014
6664991310,2cyclictest0-21swapper/412:09:244
666499130,12cyclictest0-21swapper/412:00:464
666499130,1cyclictest0-21swapper/407:26:104
666099137,5cyclictest0-21swapper/311:45:453
6660991310,2cyclictest1146-21snmpd08:55:283
6655991310,2cyclictest1146-21snmpd07:15:402
665599130,7cyclictest0-21swapper/208:35:172
665599130,13cyclictest0-21swapper/208:25:202
665599130,1cyclictest0-21swapper/211:27:102
665299139,3cyclictest12266-21timerandwakeup07:15:221
665299135,7cyclictest26661-21chrt12:34:271
665299134,4cyclictest32299-21pltrace-run07:52:491
665299134,4cyclictest16034-21pltrace-run10:17:491
665299134,3cyclictest30243-21latency_hist10:45:011
665299133,3cyclictest27770-21cstates07:45:141
6652991312,1cyclictest0-21swapper/109:43:021
665299130,7cyclictest0-21swapper/111:17:531
665299130,6cyclictest0-21swapper/107:11:311
665299130,13cyclictest0-21swapper/111:20:211
665299130,12cyclictest0-21swapper/108:46:161
664799130,6cyclictest0-21swapper/008:39:150
664799130,6cyclictest0-21swapper/007:41:460
3229452130,5pltrace0-21swapper/307:56:573
2331452130,7pltrace54-21ksoftirqd/409:35:064
2331452130,5pltrace0-21swapper/509:35:115
2331452130,0pltrace0-21swapper/309:35:113
2222352130,5pltrace0-21swapper/308:37:203
1881852130,4pltrace19895-21df10:25:135
1836052130,5pltrace0-21swapper/207:29:162
1836052130,2pltrace18457-21runrttasks07:29:083
1603052130,3pltrace17794-21sed10:20:186
1603052130,3pltrace17794-21sed10:20:186
667699129,2cyclictest8562-21cat07:10:217
667699129,2cyclictest4664-21taskset09:55:267
667699129,2cyclictest4543-21cpuspeed_turbos09:00:117
667699129,2cyclictest27486-21cut07:45:137
667699129,2cyclictest24209-21cstates09:35:117
667699129,2cyclictest22737-21latency12:25:177
667699129,2cyclictest2060-21cat10:50:177
667699129,2cyclictest16598-21expr07:25:167
667699129,2cyclictest1146-21snmpd08:12:447
667699129,2cyclictest10022-21sed07:15:017
667699128,1cyclictest191rcu_preempt09:25:137
667699127,2cyclictest78-21ksoftirqd/708:33:427
667699124,4cyclictest78-21ksoftirqd/712:13:307
667699122,7cyclictest78-21ksoftirqd/708:05:147
667699122,7cyclictest78-21ksoftirqd/707:30:367
667699121,8cyclictest78-21ksoftirqd/709:46:517
667399126,5cyclictest0-21swapper/611:36:386
667399126,3cyclictest0-21swapper/611:14:416
6673991211,1cyclictest0-21swapper/610:18:156
6673991211,1cyclictest0-21swapper/608:33:106
6673991210,1cyclictest0-21swapper/608:04:216
667399120,12cyclictest0-21swapper/609:22:526
667399120,12cyclictest0-21swapper/607:59:106
667399120,11cyclictest0-21swapper/608:21:216
667399120,1cyclictest0-21swapper/611:18:196
666999125,6cyclictest1146-21snmpd08:23:215
666999124,2cyclictest30396-21cstates07:50:135
666999124,2cyclictest22976-21missed_timers07:35:185
666999124,2cyclictest19150-21fschecks_time09:25:155
666999123,4cyclictest6890-21proc_pri10:00:195
666999123,2cyclictest19183-21unixbench_singl08:25:235
666999123,2cyclictest102752chrt10:05:295
666999123,2cyclictest0-21swapper/511:45:175
666999122,4cyclictest14048-21fschecks_time07:20:155
6669991211,1cyclictest0-21swapper/508:50:115
6669991210,1cyclictest0-21swapper/512:35:145
666999121,4cyclictest23787-21grep11:30:145
666999120,5cyclictest836-21/usr/sbin/munin10:50:215
666999120,5cyclictest619-21cut07:55:135
666999120,5cyclictest4293-21chrt09:00:075
666999120,5cyclictest28649-21gltestperf10:40:155
666999120,5cyclictest19258-21swap11:20:215
666999120,5cyclictest17475-21tr07:25:215
666999120,5cyclictest1621-21/usr/sbin/munin08:55:135
666999120,5cyclictest1178-21runrttasks11:40:045
666999120,4cyclictest9709-21expr11:05:135
666999120,4cyclictest7148-21aten_r4power_en09:05:125
666999120,4cyclictest4117-21gltestperf08:00:155
666999120,4cyclictest3837-21cpuspeed_turbos10:55:145
666999120,4cyclictest29370-21basename08:45:165
666999120,4cyclictest23712-21gltestperf08:35:165
666999120,4cyclictest23345-21sed12:25:225
666999120,4cyclictest20687-21fschecks_time08:30:155
666999120,4cyclictest19264-21cstates07:30:145
666999120,4cyclictest12327-21fschecks_time08:15:145
666999120,4cyclictest10270-21cstates09:10:135
666999120,11cyclictest0-21swapper/507:12:435
666499129,2cyclictest0-21swapper/409:48:264
666499126,5cyclictest1055-13auditd10:40:004
6664991211,1cyclictest0-21swapper/409:21:174
6664991210,1cyclictest0-21swapper/411:24:304
6664991210,1cyclictest0-21swapper/409:25:104
6664991210,1cyclictest0-21swapper/408:14:174
666499120,11cyclictest0-21swapper/410:44:594
666499120,11cyclictest0-21swapper/408:28:504
666499120,1cyclictest0-21swapper/412:10:584
666099129,2cyclictest1-21systemd09:20:003
666099126,5cyclictest0-21swapper/310:31:273
6660991211,1cyclictest0-21swapper/312:05:093
6660991211,1cyclictest0-21swapper/310:45:203
665599126,5cyclictest0-21swapper/211:11:072
665599120,11cyclictest0-21swapper/211:06:192
665599120,11cyclictest0-21swapper/210:45:192
665299129,2cyclictest18284-21/usr/sbin/munin09:25:211
665299129,2cyclictest1146-21snmpd08:17:451
665299127,4cyclictest20235-21fschecks_time10:25:151
665299127,4cyclictest10197-21head11:05:141
665299125,6cyclictest5571-21chrt08:03:551
665299125,6cyclictest26025-21iostat_ios10:35:171
665299125,6cyclictest18713-21taskset10:21:351
665299125,6cyclictest18713-21taskset10:21:351
665299125,6cyclictest18089-21tail08:25:151
665299125,6cyclictest1146-21snmpd10:05:051
665299125,6cyclictest1146-21snmpd09:38:261
665299125,5cyclictest4556-21grep09:00:111
665299125,5cyclictest10643-21cpuspeed_turbos12:05:101
665299125,2cyclictest27662-21cut12:35:161
665299124,6cyclictest13297-21tune2fs09:15:131
665299124,5cyclictest23747-21/usr/sbin/munin10:30:211
665299123,3cyclictest26937-21netstat08:40:161
6652991211,1cyclictest0-21swapper/108:55:171
6652991210,1cyclictest0-21swapper/107:25:151
665299120,11cyclictest0-21swapper/110:51:121
665299120,1cyclictest0-21swapper/111:40:541
665299120,1cyclictest0-21swapper/108:53:011
665299120,1cyclictest0-21swapper/108:09:571
6647991210,1cyclictest0-21swapper/009:04:570
548852120,0pltrace0-21swapper/308:06:293
484952120,5pltrace0-21swapper/009:59:280
421552120,0pltrace0-21swapper/711:53:197
3229452120,7pltrace78-21ksoftirqd/707:55:197
3058152120,5pltrace0-21swapper/308:48:043
2717752120,1pltrace27181-21tail10:37:503
2439052120,5pltrace0-21swapper/210:35:582
1774052120,5pltrace0-21swapper/309:23:093
1664952120,3pltrace17203-21aten_r4power_cu08:25:133
1432252120,5pltrace0-21swapper/311:13:183
1107952120,5pltrace0-21swapper/308:16:003
1045752120,5pltrace0-21swapper/210:11:502
701752110,0pltrace0-21swapper/312:01:043
667699118,2cyclictest9770-21grep10:05:207
667699118,2cyclictest3402-21cpuspeed_turbos08:00:137
667699118,2cyclictest32476-21needreboot08:50:197
667699118,2cyclictest23839-21fschecks_count11:30:147
667699116,2cyclictest78-21ksoftirqd/707:35:157
667399118,2cyclictest14555-21cat07:20:186
667399116,3cyclictest0-21swapper/609:59:086
667399115,5cyclictest1055-17auditd08:15:016
6673991110,1cyclictest0-21swapper/607:37:486
667399110,11cyclictest0-21swapper/612:02:016
667399110,11cyclictest0-21swapper/611:27:036
667399110,11cyclictest0-21swapper/609:04:356
667399110,11cyclictest0-21swapper/607:42:106
667399110,11cyclictest0-21swapper/607:10:556
667399110,1cyclictest0-21swapper/608:38:316
666999110,5cyclictest1178-21runrttasks08:40:015
666999110,4cyclictest7045-21fschecks_count11:00:155
666999110,4cyclictest29580-21cpu09:45:125
666999110,4cyclictest26578-21fschecks_count11:35:155
666999110,4cyclictest16603-21aten_r4power_en10:20:135
666999110,4cyclictest16603-21aten_r4power_en10:20:135
666999110,4cyclictest13260-21cpu12:10:115
666999110,4cyclictest1178-21runrttasks10:30:125
666999110,4cyclictest11038-21/usr/sbin/munin10:10:115
666499118,2cyclictest0-21swapper/410:26:544
666499117,3cyclictest1146-21snmpd12:29:014
666499117,3cyclictest1146-21snmpd10:21:034
666499117,3cyclictest1146-21snmpd10:21:034
666499117,3cyclictest1146-21snmpd07:58:054
666499112,4cyclictest4730-21if_err_eth010:55:174
6664991110,1cyclictest0-21swapper/409:06:044
6664991110,1cyclictest0-21swapper/408:20:054
6664991110,1cyclictest0-21swapper/407:50:474
666499110,11cyclictest0-21swapper/411:50:344
666499110,11cyclictest0-21swapper/411:19:194
666499110,11cyclictest0-21swapper/411:06:414
666499110,11cyclictest0-21swapper/409:31:004
666499110,11cyclictest0-21swapper/409:01:224
666499110,1cyclictest0-21swapper/408:09:434
666499110,1cyclictest0-21swapper/407:43:304
666499110,1cyclictest0-21swapper/407:39:074
666499110,1cyclictest0-21swapper/407:14:214
666099118,2cyclictest26413-21grep07:40:223
666099118,2cyclictest18284-21/usr/sbin/munin09:25:133
666099118,2cyclictest14098-21cut10:15:133
666099117,3cyclictest27365-21needreboot11:35:183
666099117,3cyclictest1146-21snmpd09:14:163
6660991110,1cyclictest0-21swapper/312:30:543
6660991110,1cyclictest0-21swapper/308:30:093
666099111,8cyclictest0-21swapper/310:00:473
666099110,11cyclictest0-21swapper/310:22:113
666099110,11cyclictest0-21swapper/310:22:113
666099110,11cyclictest0-21swapper/309:47:323
666099110,1cyclictest0-21swapper/312:16:293
666099110,1cyclictest0-21swapper/312:14:333
666099110,1cyclictest0-21swapper/311:30:313
666099110,1cyclictest0-21swapper/311:19:203
666099110,1cyclictest0-21swapper/310:41:473
666099110,1cyclictest0-21swapper/307:38:013
665599117,3cyclictest0-21swapper/209:31:042
665599115,5cyclictest0-21swapper/208:31:032
665599115,2cyclictest0-21swapper/210:05:142
6655991110,1cyclictest0-21swapper/211:40:362
6655991110,1cyclictest0-21swapper/209:43:422
665599110,11cyclictest0-21swapper/212:13:082
665599110,11cyclictest0-21swapper/211:31:322
665599110,11cyclictest0-21swapper/209:04:512
665599110,11cyclictest0-21swapper/208:52:032
665599110,11cyclictest0-21swapper/208:43:192
665599110,10cyclictest0-21swapper/212:18:582
665599110,1cyclictest0-21swapper/212:20:552
665299117,3cyclictest5789-21ls10:00:131
665299116,4cyclictest950-21latency09:50:201
665299115,5cyclictest9184-21grep12:00:201
665299115,5cyclictest8628-21/usr/sbin/munin09:05:201
665299115,5cyclictest6820-21expr11:00:141
665299115,5cyclictest6094-21needreboot11:55:171
665299115,5cyclictest405-21cut07:55:121
665299115,5cyclictest3536-21tr09:55:141
665299115,5cyclictest26625-21fschecks_count11:35:151
665299115,5cyclictest25986-21processes07:40:201
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional