You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 07:23
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack5slot1.osadl.org (updated Mon Dec 01, 2025 00:44:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
269312910,0sleep0104-21kswapd000:20:090
683239840,0EcMasterDemoSyn684129tAtEmLog_019:05:205
929439820,0EcMasterDemoSyn930429tAtEmLog_019:10:255
3251339820,0EcMasterDemoSyn3252329tAtEmLog_023:39:105
2894839820,0EcMasterDemoSyn2895829tAtEmLog_019:50:595
2783139820,0EcMasterDemoSyn2784129tAtEmLog_021:27:195
180539820,0EcMasterDemoSyn181529tAtEmLog_023:18:545
1715539820,0EcMasterDemoSyn1716529tAtEmLog_023:29:025
127339820,0EcMasterDemoSyn128329tAtEmLog_020:01:075
1239839820,0EcMasterDemoSyn1240829tAtEmLog_019:15:305
1239839820,0EcMasterDemoSyn1240829tAtEmLog_019:15:295
599239810,0EcMasterDemoSyn600229tAtEmLog_020:11:175
3130539810,0EcMasterDemoSyn3131529tAtEmLog_019:56:045
2655239810,0EcMasterDemoSyn2656229tAtEmLog_023:13:485
1315539810,0EcMasterDemoSyn1316529tAtEmLog_020:26:285
124439810,0EcMasterDemoSyn125429tAtEmLog_000:24:495
7757992523,2cyclictest5603-21kworker/1:0+events22:33:141
306762220,1sleep0221rcuc/000:00:000
7769992116,3cyclictest3094-21turbostat23:20:004
775299190,15cyclictest805-21runrttasks00:16:070
59992190,0sleep00-21swapper/023:00:030
7752991815,2cyclictest5758-21turbostat00:09:540
775299180,14cyclictest8153-21sh22:17:580
775299180,10cyclictest26070-21sh23:13:230
775299180,10cyclictest20363-21sh00:39:110
775299180,10cyclictest11713-21sh21:59:200
7769991712,4cyclictest26939-21turbostat19:50:004
775799170,9cyclictest11121-21diskmemload21:16:221
775799170,9cyclictest11121-21diskmemload21:16:211
775299174,9cyclictest8954-21aten2_r5power_c22:40:120
775299170,9cyclictest805-21runrttasks00:29:490
775299170,9cyclictest738-21snmpd22:58:330
775299170,9cyclictest24561-21sh22:07:170
775299170,9cyclictest22957-21sh22:49:360
775299170,9cyclictest14651-21runrttasks23:26:490
775299170,9cyclictest13845-21sh21:18:410
775299170,9cyclictest13845-21sh21:18:410
775299170,9cyclictest12473-21turbostat.cron22:04:550
775299170,9cyclictest10896-21rm00:30:490
775299170,8cyclictest29066-21cat22:10:160
775299170,13cyclictest805-21runrttasks20:36:060
775299170,12cyclictest8134-21sh23:22:490
775299170,12cyclictest8134-21sh23:22:480
1788499170,10tDemoTimingTask18242-21chrt20:40:015
1516399170,7tDemoTimingTask19239-21chrt23:51:165
1154499170,10tDemoTimingTask11639-21chrt23:03:445
824199160,9tDemoTimingTask10372-21taskset22:19:555
776999161,11cyclictest16-1pr/legacy20:01:064
7761991612,3cyclictest29934-21aten2_r5power_p21:50:122
776199161,11cyclictest16-1pr/legacy21:07:012
776199160,9cyclictest16120-21/usr/sbin/munin20:35:252
775299160,9cyclictest805-21runrttasks19:33:390
775299160,8cyclictest16153-21sh00:12:310
775299160,13cyclictest805-21runrttasks19:49:550
775299160,13cyclictest25829-21sh22:50:300
3242199160,14tDemoTimingTask30035-21sleep21:09:565
1516399160,6tDemoTimingTask15868-21taskset23:49:575
816999150,9tDemoTimingTask8178-21taskset00:29:535
776999151,9cyclictest18518-21basename19:30:174
776999150,8cyclictest20346-21ssh23:09:574
776699157,6cyclictest31161-21grep21:05:173
776699153,8cyclictest25008-21processes21:25:203
776199159,3cyclictest2367439EcMasterDemoSyn22:28:112
776199154,8cyclictest8162-21grep20:15:242
776199151,10cyclictest8255-21ssh00:07:052
776199151,10cyclictest25224-21ssh00:18:572
776199150,9cyclictest30618-21sh23:15:402
776199150,9cyclictest29038-21taskset19:51:412
776199150,11cyclictest29843-21chrt22:54:162
776199150,10cyclictest17813-21/usr/sbin/munin19:30:122
776199150,10cyclictest10719-21ssh23:02:562
775799154,6cyclictest8586-21cat21:35:261
7757991510,4cyclictest18956-21turbostat00:15:001
775299156,7cyclictest18840-21ssh23:30:120
775299151,11cyclictest27395-21sh21:48:170
775299151,10cyclictest16875-21sh21:40:450
775299151,10cyclictest11625-21sh23:46:060
775299150,8cyclictest805-21runrttasks20:27:450
775299150,7cyclictest4687-21cat21:55:010
775299150,10cyclictest18911-21ssh23:50:580
1980039150,1EcMasterDemoSyn19807-21runintdemo5min21:22:153
776999145,8cyclictest8123-21/usr/sbin/munin21:35:194
776999144,6cyclictest19412-21tune2fs23:30:144
776999141,9cyclictest5368-21/usr/sbin/munin21:55:224
776999141,9cyclictest2722-21taskset22:14:534
776999141,9cyclictest19810-21ssh21:43:184
776999141,9cyclictest12541-21runrttasks23:46:524
776999141,9cyclictest10558-21latency_hist22:19:594
7769991411,2cyclictest15354-21idleruntime-cro21:20:004
7769991411,2cyclictest15354-21idleruntime-cro21:20:004
776999141,10cyclictest738-21snmpd00:20:534
776999141,10cyclictest1171-21/usr/sbin/munin21:10:154
776999141,10cyclictest1171-21/usr/sbin/munin21:10:144
7769991410,3cyclictest23044-21latency_hist21:25:014
7769991410,3cyclictest19852-21grep22:25:204
776999140,9cyclictest5136-21ssh22:37:134
776999140,9cyclictest17354-21ssh22:02:394
776999140,10cyclictest738-21snmpd21:31:194
776999140,10cyclictest2575-21ssh21:53:184
776999140,10cyclictest23719-21/usr/sbin/munin19:40:224
776999140,10cyclictest22783-21/usr/sbin/munin23:10:254
776999140,10cyclictest16-1pr/legacy00:09:354
776699146,6cyclictest30245-21grep23:15:253
776699143,8cyclictest32741-21sshd21:51:223
776699141,9cyclictest8825-21ssh00:07:373
776699141,9cyclictest805-21runrttasks21:03:393
776699141,9cyclictest10307-21ssh21:15:403
776699141,9cyclictest10307-21ssh21:15:393
7766991411,2cyclictest728-21gdbus21:55:023
7766991410,3cyclictest717-21in:imuxsock19:45:013
776699140,9cyclictest23821-21/usr/sbin/munin23:55:203
776699140,7cyclictest22243-21/usr/sbin/munin21:45:133
776699140,10cyclictest805-21runrttasks23:44:353
776699140,10cyclictest7926-21sh23:22:383
776699140,10cyclictest7926-21sh23:22:373
776699140,10cyclictest623-21chrt00:24:153
776699140,10cyclictest5656-21sh00:04:543
776699140,10cyclictest5607-21ssh22:15:443
776699140,10cyclictest4171-21runrttasks21:33:013
776699140,10cyclictest3777-21rm22:36:003
776699140,10cyclictest31422-21chrt19:57:093
776699140,10cyclictest2882-21ssh21:10:393
776699140,10cyclictest2882-21ssh21:10:393
776699140,10cyclictest27914-21/usr/sbin/munin19:50:183
776699140,10cyclictest2268-21ssh22:56:563
776699140,10cyclictest21716-21ssh23:53:323
776699140,10cyclictest20726-21head23:30:253
776699140,10cyclictest19382-21/usr/sbin/munin00:15:203
776699140,10cyclictest17811-21rm00:36:163
776699140,10cyclictest14699-21taskset00:11:133
776699140,10cyclictest12074-21sed23:25:183
776699140,10cyclictest11877-21sh21:38:023
776699140,10cyclictest10896-21sh22:20:063
776199146,6cyclictest778-21postgres21:35:282
776199144,5cyclictest14086-21cpuspeed_turbos20:30:152
776199143,8cyclictest353-21systemd-udevd20:59:412
776199141,9cyclictest623-21ssh21:10:062
776199141,9cyclictest623-21ssh21:10:052
776199141,9cyclictest24590-21latency_hist19:45:012
776199141,9cyclictest19923-21sh23:09:332
776199141,10cyclictest1894-21sh21:31:032
7761991410,3cyclictest1430-21aten2_r5power_v22:35:152
776199140,9cyclictest28394-21cpuspeed22:10:132
776199140,9cyclictest20583-21/usr/sbin/munin22:05:212
776199140,7cyclictest2585-21ssh00:01:592
776199140,10cyclictest9458-21fschecks_count20:20:172
776199140,10cyclictest8369-21df19:10:152
776199140,10cyclictest6104-21runrttasks20:12:132
776199140,10cyclictest5989-21ssh23:42:452
776199140,10cyclictest5946-21sh22:16:032
776199140,10cyclictest5926-21ssh23:20:512
776199140,10cyclictest5926-21ssh23:20:502
776199140,10cyclictest5368-21/usr/sbin/munin21:55:162
776199140,10cyclictest3544-21ssh22:58:082
776199140,10cyclictest31375-21runrttasks19:56:282
776199140,10cyclictest27974-21/usr/sbin/munin21:00:252
776199140,10cyclictest27673-21ssh22:30:232
776199140,10cyclictest27291-21rm00:20:132
776199140,10cyclictest26353-21ssh23:56:012
776199140,10cyclictest25552-21hddtemp_smartct19:45:142
776199140,10cyclictest25039-21uname21:25:202
776199140,10cyclictest23418-21sh23:10:592
776199140,10cyclictest2273-21diskstats20:05:142
776199140,10cyclictest22040-21sh21:45:022
776199140,10cyclictest21967-21chrt19:36:452
776199140,10cyclictest20937-21ssh23:30:362
776199140,10cyclictest19537-21ntp_kernel_err20:40:192
776199140,10cyclictest17563-21needreboot22:45:202
776199140,10cyclictest16335-21sh00:12:392
776199140,10cyclictest15865-21ssh22:01:182
776199140,10cyclictest15850-21/usr/sbin/munin21:20:142
776199140,10cyclictest15278-21/usr/sbin/munin00:35:162
776199140,10cyclictest14651-21runrttasks22:21:572
776199140,10cyclictest14038-21sh00:34:182
776199140,10cyclictest13899-21wc19:20:192
776199140,10cyclictest11872-21ssh22:41:202
776199140,10cyclictest11741-21ssh23:46:092
776199140,10cyclictest11363-21sort19:15:182
776199140,10cyclictest11363-21sort19:15:172
776199140,10cyclictest11134-21ssh21:16:232
776199140,10cyclictest11134-21ssh21:16:222
775799144,8cyclictest28023-21/usr/sbin/munin21:00:121
775799141,9cyclictest8312-21runintdemo5min22:18:051
775799141,9cyclictest24889-21/usr/sbin/munin19:45:141
775799141,9cyclictest22176-21sh23:53:581
775799141,9cyclictest16709-21sh23:06:351
775799141,9cyclictest15662-21ssh21:20:001
775799141,9cyclictest13618-21ssh23:25:551
775799141,10cyclictest6034-21chrt22:38:001
775799140,9cyclictest8911-21/usr/sbin/munin22:40:191
775799140,9cyclictest4093-21runrttasks00:25:281
775799140,10cyclictest9956-21awk00:30:221
775799140,10cyclictest805-21runrttasks19:55:511
775799140,10cyclictest805-21runrttasks19:33:091
775799140,10cyclictest7413-21ssh23:22:111
775799140,10cyclictest7413-21ssh23:22:101
775799140,10cyclictest6588-21diskstats23:00:151
775799140,10cyclictest5978-21latency_hist00:05:001
775799140,10cyclictest5368-21/usr/sbin/munin21:55:231
775799140,10cyclictest4979-21ssh21:33:451
775799140,10cyclictest30809-21rm23:15:501
775799140,10cyclictest30235-21ssh21:29:261
775799140,10cyclictest29112-21ssh00:20:421
775799140,10cyclictest27986-21ssh22:10:011
775799140,10cyclictest27245-21/usr/sbin/munin19:50:221
775799140,10cyclictest25259-21sh22:29:341
775799140,10cyclictest23684-21latency_hist22:50:011
775799140,10cyclictest23396-21sh23:32:441
775799140,10cyclictest22242-21/usr/sbin/munin21:45:131
775799140,10cyclictest21417-21irqstats22:05:181
775799140,10cyclictest20646-21proc_pri00:15:161
775799140,10cyclictest16991-21ssh22:02:191
775799140,10cyclictest15530-21grep20:31:351
775799140,10cyclictest14043-21ssh22:21:261
775799140,10cyclictest11061-21munin-run20:24:591
775299143,8cyclictest14237-21cut19:20:220
775299143,3cyclictest805-21runrttasks19:52:410
775299141,9cyclictest805-21runrttasks22:37:050
775299141,9cyclictest1492-21taskset20:03:310
775299141,7cyclictest26987-21hddtemp_smartct22:30:170
775299141,10cyclictest2323-21diskmemload21:31:250
775299140,7cyclictest16593-21sh21:20:140
775299140,7cyclictest1282-21sh00:00:440
775299140,12cyclictest28452-21sh21:27:520
775299140,12cyclictest22376-21wc20:45:260
775299140,12cyclictest19281-21cut22:25:160
775299140,11cyclictest146-1kworker/0:1H+kblockd22:20:140
775299140,10cyclictest9682-21idleruntime20:20:180
775299140,10cyclictest805-21runrttasks21:00:150
775299140,10cyclictest805-21runrttasks20:11:430
775299140,10cyclictest5090-21ssh23:41:550
775299140,10cyclictest5051-21sh21:12:380
775299140,10cyclictest5051-21sh21:12:370
775299140,10cyclictest31937-21sh23:16:520
775299140,10cyclictest26463-21sh23:35:080
775299140,10cyclictest25302-21latency_hist20:55:000
775299140,10cyclictest15817-21ssh23:05:490
775299140,10cyclictest15679-21chrt20:33:180
775299140,10cyclictest12642-21runrttasks19:18:160
775299140,10cyclictest12642-21runrttasks19:18:150
775299140,10cyclictest10569-21ssh21:36:570
2264499140,11tDemoTimingTask26193-21chrt23:55:535
110402140,7sleep411056-21ssh23:03:144
776999131,9cyclictest10041-21runrttasks20:20:214
776999131,8cyclictest22104-21switchtime22:05:244
776999131,7cyclictest2294950latency_hist20:50:014
776999130,9cyclictest805-21runrttasks23:41:084
776999130,9cyclictest805-21runrttasks23:25:044
776999130,9cyclictest805-21runrttasks21:09:294
776999130,9cyclictest7885-21latency_hist19:10:014
776999130,9cyclictest738-21snmpd20:38:514
776999130,9cyclictest738-21snmpd19:50:034
776999130,9cyclictest4322-21aten2_r5power_e20:10:164
776999130,9cyclictest3411-21ssh23:20:014
776999130,9cyclictest3411-21ssh23:20:004
776999130,9cyclictest32570-21sed00:00:184
776999130,9cyclictest31684-21ssh22:33:414
776999130,9cyclictest31171-21/usr/sbin/munin22:55:174
776999130,9cyclictest29712-21taskset21:01:344
776999130,9cyclictest29592-21/usr/sbin/munin19:55:124
776999130,9cyclictest26400-21ssh23:56:054
776999130,9cyclictest26392-21latency20:55:184
776999130,9cyclictest25068-21ssh22:50:224
776999130,9cyclictest23250-21/usr/sbin/munin20:50:154
776999130,9cyclictest22247-21sh21:45:134
776999130,9cyclictest1949-21gpgconf00:25:004
776999130,9cyclictest19252-21rm00:15:004
776999130,9cyclictest18736-21tr20:40:144
776999130,9cyclictest17215-21runrttasks19:26:084
776999130,9cyclictest16426-21/usr/sbin/munin23:50:204
776999130,9cyclictest13764-21/usr/sbin/munin20:30:144
776999130,9cyclictest13470-21diskstats19:20:164
776999130,9cyclictest12360-21unixbench_singl19:15:274
776999130,9cyclictest12360-21unixbench_singl19:15:264
776999130,9cyclictest10968-21/usr/sbin/munin22:20:134
776999130,9cyclictest10721-21sed22:40:274
776999130,7cyclictest8373-21df19:10:154
776999130,10cyclictest9340-21fschecks_time00:30:174
776999130,10cyclictest3452-21grep20:05:234
776999130,10cyclictest26486-21ssh23:35:094
776999130,10cyclictest25037-21sh21:25:204
776999130,10cyclictest20420-21cut19:35:134
776999130,10cyclictest18457-21ssh00:37:054
776999130,10cyclictest17919-21ssh00:14:094
776999130,10cyclictest17909-21diskmemload22:45:234
776699133,7cyclictest25270-21sort19:45:133
776699133,6cyclictest9117-21open_inodes19:10:213
776699131,9cyclictest2270-21df_inode20:05:143
776699131,9cyclictest16386-21sh22:45:063
776699130,9cyclictest9756-21sh22:40:183
776699130,9cyclictest8235-21ssh23:00:353
776699130,9cyclictest752-21ntpd20:55:503
776699130,9cyclictest738-21snmpd19:15:093
776699130,9cyclictest738-21snmpd19:15:083
776699130,9cyclictest6618-21/usr/sbin/munin20:15:123
776699130,9cyclictest4268-21/usr/sbin/munin20:10:203
776699130,9cyclictest31965-21/usr/sbin/munin20:00:133
776699130,9cyclictest28323-21sh22:30:383
776699130,9cyclictest28178-21/usr/sbin/munin22:10:173
776699130,9cyclictest27726-21ntp_kernel_pll_23:35:213
776699130,9cyclictest2479-21cpuspeed_turbos00:25:123
776699130,9cyclictest23916-21/usr/sbin/munin22:50:283
776699130,9cyclictest23250-21/usr/sbin/munin20:50:193
776699130,9cyclictest20933-21/usr/sbin/munin23:10:173
776699130,9cyclictest20583-21/usr/sbin/munin22:05:143
776699130,9cyclictest20166-21/usr/sbin/munin19:35:133
776699130,9cyclictest19681-21latency22:25:203
776699130,9cyclictest18494-21/usr/sbin/munin20:40:133
776699130,9cyclictest17813-21/usr/sbin/munin19:30:173
776699130,9cyclictest17287-21runrttasks19:27:083
776699130,9cyclictest16120-21/usr/sbin/munin20:35:163
776699130,9cyclictest15515-21ssh23:05:323
776699130,9cyclictest14946-21runrttasks19:22:163
776699130,9cyclictest14669-21/usr/sbin/munin22:00:253
776699130,9cyclictest13152-21sh23:47:263
776699130,9cyclictest10864-21taskset20:22:313
776699130,10cyclictest8734-21ssh00:30:123
776699130,10cyclictest22615-21chrt20:46:183
776699130,10cyclictest16691-21ssh21:40:343
776699130,10cyclictest15661-21chrt20:33:023
776199135,6cyclictest23303-21aten2_r5power_e20:50:142
776199135,6cyclictest11901-21sh23:25:162
776199130,9cyclictest805-21runrttasks20:48:052
776199130,9cyclictest7911-21latency_hist19:10:012
776199130,9cyclictest32166-21cpuspeed20:00:132
776199130,9cyclictest2238-21/usr/sbin/munin00:25:142
776199130,9cyclictest19233-21ssh23:51:162
776199130,9cyclictest15445-21runrttasks19:25:032
776199130,9cyclictest11365-21/usr/sbin/munin20:25:132
776199130,10cyclictest27322-21ssh23:35:182
776199130,10cyclictest17272-21runrttasks21:41:062
775799139,3cyclictest16766-21switchtime00:35:261
775799132,8cyclictest754-21lldpd00:05:511
775799131,9cyclictest1378-21sh23:40:011
775799130,9cyclictest805-21runrttasks23:45:221
775799130,9cyclictest805-21runrttasks20:09:571
775799130,9cyclictest8021-21/usr/sbin/munin19:10:211
775799130,9cyclictest7477-21cat20:15:191
775799130,9cyclictest741-21/usr/sbin/munin21:10:141
775799130,9cyclictest741-21/usr/sbin/munin21:10:131
775799130,9cyclictest4268-21/usr/sbin/munin20:10:191
775799130,9cyclictest31970-21apt-get20:00:121
775799130,9cyclictest31261-21aten2_r5power_p22:55:131
775799130,9cyclictest30629-21/usr/sbin/munin21:05:151
775799130,9cyclictest29621-21ssh21:50:011
775799130,9cyclictest26701-21aten2_r5power_v23:35:151
775799130,9cyclictest25602-21/usr/sbin/munin20:55:171
775799130,9cyclictest23621-21forks20:50:151
775799130,9cyclictest22539-21/usr/sbin/munin19:40:171
775799130,9cyclictest22461-21latency_hist19:40:021
775799130,9cyclictest21462-21fschecks_time20:45:171
775799130,9cyclictest16294-21ssh22:45:021
775799130,9cyclictest16120-21/usr/sbin/munin20:35:161
775799130,9cyclictest15453-21/usr/sbin/munin19:25:221
775799130,9cyclictest14881-21runrttasks19:21:231
775799130,9cyclictest14644-21/usr/sbin/munin21:40:251
775799130,9cyclictest11357-21/usr/sbin/munin20:25:251
775799130,9cyclictest10562-21apt-config19:15:011
775799130,9cyclictest10562-21apt-config19:15:001
775799130,11cyclictest20705-21latency_hist20:45:001
775799130,10cyclictest25741-21ssh23:55:271
775799130,10cyclictest20840-21sh23:10:081
775299130,9cyclictest8224-21unixbench_singl20:15:260
775299130,9cyclictest805-21runrttasks19:12:420
775299130,9cyclictest2383-21fschecks_time20:05:140
775299130,9cyclictest21442-21swap19:35:250
775299130,10cyclictest31504-21runrttasks21:05:190
775299130,10cyclictest30762-21/usr/sbin/munin19:55:180
775299130,10cyclictest25994-21fschecks_count20:55:170
775299130,10cyclictest23874-21sed19:40:240
775299130,10cyclictest19757-21python320:40:240
775299130,10cyclictest15704-21cpuspeed_turbos19:25:110
776999124,6cyclictest11787-21awk20:25:154
776999120,9cyclictest6979-21sh20:15:144
776699120,9cyclictest11452-21aten2_r5power_v20:25:143
775799121,7cyclictest792250cat19:10:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional