You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 03:21
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack5slot1.osadl.org (updated Sun Nov 30, 2025 00:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
88339900,0EcMasterDemoSyn89329tAtEmLog_021:06:585
2846139840,0EcMasterDemoSyn2847129tAtEmLog_021:52:375
667939820,0EcMasterDemoSyn668929tAtEmLog_019:05:165
519639820,0EcMasterDemoSyn520629tAtEmLog_021:12:025
3272939820,0EcMasterDemoSyn3273929tAtEmLog_022:43:195
2869839820,0EcMasterDemoSyn2870829tAtEmLog_000:39:575
2693239820,0EcMasterDemoSyn2694229tAtEmLog_019:45:505
2693239820,0EcMasterDemoSyn2694229tAtEmLog_019:45:505
1809839820,0EcMasterDemoSyn1810829tAtEmLog_023:18:485
1173539820,0EcMasterDemoSyn1174529tAtEmLog_020:21:205
1675239810,0EcMasterDemoSyn1676229tAtEmLog_020:31:285
112539810,0EcMasterDemoSyn113529tAtEmLog_021:32:205
427339800,0EcMasterDemoSyn428329tAtEmLog_020:06:075
2148839800,0EcMasterDemoSyn2149829tAtEmLog_020:41:375
3044339300,1EcMasterDemoSyn30450-21runintdemo5min22:17:571
220842200,1sleep2391rcuc/221:24:352
161232190,0sleep40-21swapper/422:54:304
1243439190,0EcMasterDemoSyn0-21swapper/221:17:062
839999181,9cyclictest11532-21runrttasks23:14:130
2519829180,10tAtEmLog_026700-21chrt23:25:015
2486699180,8tDemoTimingTask27902-21chrt00:10:455
841299170,10cyclictest738-21snmpd23:51:173
839999175,9cyclictest26723-21sh23:25:010
839999170,9cyclictest2193-21smartctl23:30:150
839999170,9cyclictest11828-21sh23:36:490
839999170,9cyclictest11828-21sh23:36:480
2486699171,7tDemoTimingTask25196-21chrt00:09:505
2457139170,0EcMasterDemoSyn0-21swapper/519:40:465
841699164,7cyclictest2746-21processes23:30:184
8412991612,3cyclictest724-21polkitd22:10:003
8407991613,2cyclictest1-21systemd00:00:012
8407991612,2cyclictest171rcu_preempt23:47:252
840399165,6cyclictest334-21systemd-journal20:25:001
840399163,9cyclictest2317-21latency_hist20:05:001
840399161,9cyclictest805-21runrttasks19:31:041
8403991614,2cyclictest11710-21kworker/1:2+events00:04:251
8403991613,2cyclictest0-21swapper/121:25:161
839999165,8cyclictest21423-21sh23:44:180
839999164,9cyclictest805-21runrttasks21:05:440
839999164,8cyclictest12481-21entropy00:00:160
839999163,9cyclictest28812-21awk00:40:000
839999160,9cyclictest22201-21sh22:11:260
839999160,13cyclictest805-21runrttasks20:27:330
839999160,13cyclictest22699-21runrttasks22:58:580
839999160,10cyclictest146-1kworker/0:1H+kblockd22:29:540
1418829160,9tAtEmLog_014260-21taskset23:39:105
1418829160,9tAtEmLog_014260-21taskset23:39:105
1410229160,9tAtEmLog_014134-21taskset20:26:275
841699158,5cyclictest13866-21tune2fs00:25:184
841699157,6cyclictest6117-21grep23:10:144
841699154,8cyclictest738-21snmpd21:41:324
841699154,6cyclictest26066-21cat21:50:264
841699153,8cyclictest781-21postgres20:50:154
8416991513,2cyclictest25751-21kworker/4:0+events19:40:464
8416991511,3cyclictest704-21dbus-daemon21:25:004
8416991511,3cyclictest6979-21sh22:24:594
841699150,9cyclictest26051-21sh22:38:184
841699150,8cyclictest11936-21sh22:27:594
841299158,6cyclictest15636-21grep22:30:193
841299158,5cyclictest28710-21seq00:39:573
8412991512,2cyclictest21776-21kworker/3:0+events20:36:333
841299151,10cyclictest6181-21ssh22:47:053
841299150,9cyclictest23530-21diskmemload22:35:463
840799159,4cyclictest16017-21needreboot20:30:212
840799158,5cyclictest8394-21cyclictest20:03:392
840799154,8cyclictest24042-21cut21:25:192
8407991511,3cyclictest704-21dbus-daemon22:20:002
840799150,9cyclictest11506-21diskmemload00:23:412
840799150,8cyclictest17102-21ssh22:08:082
840399154,8cyclictest754-21lldpd21:31:341
840399154,8cyclictest738-21snmpd23:58:301
840399151,10cyclictest8429-21runrttasks23:11:141
840399150,11cyclictest25165-21sed00:35:251
840399150,10cyclictest29310-21chrt21:53:271
840399150,10cyclictest17027-21chrt23:17:441
839999157,7cyclictest8394-21cyclictest00:17:230
839999156,7cyclictest29904-21sh21:30:120
839999151,7cyclictest14927-21wc22:30:130
839999150,9cyclictest5156-21sh23:09:510
839999150,7cyclictest21188-21ssh21:46:580
839999150,7cyclictest21188-21ssh21:46:570
839999150,6cyclictest23359-21cat00:35:000
839999150,6cyclictest19692-21fschecks_time00:05:170
839999150,11cyclictest805-21runrttasks22:01:380
664299150,9tDemoTimingTask6947-21taskset20:14:575
1763539150,2EcMasterDemoSyn651irq_work/500:29:495
113529151,12tAtEmLog_056372sleep521:35:235
979499140,7tDemoTimingTask9799-21taskset22:02:455
841699143,6cyclictest704-21dbus-daemon23:17:004
841699142,9cyclictest12820-21fschecks_time20:25:154
841699142,8cyclictest28150-21proc_pri23:25:234
841699141,12cyclictest7225-21sh21:36:384
841699140,9cyclictest939550cat20:20:004
841699140,9cyclictest25092-21taskset00:35:224
841699140,9cyclictest19799-21rm22:33:554
841699140,11cyclictest10634-21basename20:20:184
841699140,10cyclictest805-21runrttasks19:14:124
841699140,10cyclictest7782-21ssh23:56:404
841699140,10cyclictest738-21snmpd22:47:474
841699140,10cyclictest32676-21cpuspeed00:15:134
841699140,10cyclictest28840-21python319:50:214
841699140,10cyclictest27079-21runrttasks21:27:414
841699140,10cyclictest19649-21ssh23:20:034
841699140,10cyclictest1668-21sh21:10:134
841699140,10cyclictest15881-21cut19:25:134
841699140,10cyclictest15580-21sh23:40:084
841699140,10cyclictest12102-21ssh21:16:484
841699140,10cyclictest11870-21rm00:00:034
841699140,10cyclictest11516-21ssh23:36:314
841699140,10cyclictest11516-21ssh23:36:304
841299147,5cyclictest677-21head21:05:273
841299143,8cyclictest738-21snmpd20:46:013
841299142,9cyclictest738-21snmpd00:25:333
841299142,9cyclictest7203-21cut00:20:173
841299142,8cyclictest6456-21unixbench_multi20:10:273
841299141,9cyclictest3191-21ssh23:07:543
841299141,9cyclictest20258-21ssh21:22:443
841299141,9cyclictest1887-21cpuspeed21:10:153
841299141,10cyclictest1803-21ssh21:56:353
841299140,9cyclictest11189-21/usr/sbin/munin19:15:163
841299140,8cyclictest10528-21munin-run21:40:003
841299140,10cyclictest805-21runrttasks00:30:393
841299140,10cyclictest7605-21/usr/sbin/munin20:15:143
841299140,10cyclictest6643-21ssh23:55:333
841299140,10cyclictest5770-21/usr/sbin/munin23:10:193
841299140,10cyclictest3641-21sh22:21:433
841299140,10cyclictest31030-21sh23:27:423
841299140,10cyclictest29336-21ssh00:12:173
841299140,10cyclictest29171-21/usr/sbin/munin21:00:153
841299140,10cyclictest28315-21ssh23:02:213
841299140,10cyclictest26212-21/usr/sbin/munin22:15:243
841299140,10cyclictest2408-21ssh21:33:373
841299140,10cyclictest21844-21ssh23:20:393
841299140,10cyclictest21762-21python319:35:233
841299140,10cyclictest2148-21gltestperf23:30:143
841299140,10cyclictest20521-21ssh22:56:563
841299140,10cyclictest19350-21sh23:42:133
841299140,10cyclictest17969-21aten2_r5power_p21:45:153
841299140,10cyclictest17969-21aten2_r5power_p21:45:143
841299140,10cyclictest16917-21ssh21:44:323
841299140,10cyclictest15120-21ssh00:01:343
841299140,10cyclictest15005-21chrt23:15:443
841299140,10cyclictest14694-21ssh23:39:363
841299140,10cyclictest14694-21ssh23:39:353
841299140,10cyclictest12590-21cpuspeed_turbos20:25:143
841299140,10cyclictest10814-21sh21:15:353
841299140,10cyclictest1071-21ssh22:44:023
840799145,6cyclictest25811-21kworker/u24:0+nfsiod00:01:002
840799144,6cyclictest41-21ksoftirqd/223:00:262
840799144,6cyclictest30067-21cat21:00:202
840799142,8cyclictest41-21ksoftirqd/222:04:532
840799141,9cyclictest41-21ksoftirqd/223:15:432
840799141,9cyclictest41-21ksoftirqd/219:20:212
840799141,9cyclictest23814-21missed_timers19:40:192
840799141,9cyclictest21901-21ssh00:06:242
840799141,10cyclictest29365-21runrttasks19:51:262
840799141,10cyclictest28786-21runrttasks20:59:202
840799140,9cyclictest19363-21taskset20:39:272
840799140,10cyclictest9931-21python323:35:222
840799140,10cyclictest9931-21python323:35:212
840799140,10cyclictest8270-21ssh23:11:062
840799140,10cyclictest6348-21sh21:35:462
840799140,10cyclictest5503-21ssh23:32:342
840799140,10cyclictest4087-21sh22:22:072
840799140,10cyclictest3348-21ssh00:17:022
840799140,10cyclictest31525-21/usr/sbin/munin21:05:162
840799140,10cyclictest31012-21grep19:55:202
840799140,10cyclictest3081-21fschecks_time20:05:182
840799140,10cyclictest29237-21ssh00:12:102
840799140,10cyclictest26902-21ssh23:25:112
840799140,10cyclictest25864-21ssh00:35:542
840799140,10cyclictest2578-21/usr/sbin/munin22:45:172
840799140,10cyclictest24580-21/usr/sbin/munin21:50:142
840799140,10cyclictest22718-21ssh23:21:302
840799140,10cyclictest22465-21sh22:11:402
840799140,10cyclictest21550-21ssh22:57:522
840799140,10cyclictest17235-21ssh00:29:102
840799140,10cyclictest12910-21grep21:40:262
840799140,10cyclictest12109-21ssh22:50:472
840399146,6cyclictest4063-21cat20:05:261
840399146,6cyclictest17833-21hddtemp_smartct22:55:171
840399143,6cyclictest1469-21atemtest23:30:111
840399142,8cyclictest5435-21fschecks_time20:10:171
840399141,9cyclictest7394-21rm21:36:481
840399141,9cyclictest6496-21ssh22:00:201
840399141,9cyclictest3644-21ssh21:10:321
840399141,9cyclictest30764-21ssh22:41:211
840399141,9cyclictest22123-21sh21:47:511
840399141,9cyclictest22123-21sh21:47:501
840399141,9cyclictest10986-21runrttasks21:15:481
840399140,11cyclictest805-21runrttasks19:56:221
840399140,10cyclictest9232-21unixbench_singl22:25:261
840399140,10cyclictest8666-21/usr/sbin/munin19:10:141
840399140,10cyclictest8629-21threads20:15:251
840399140,10cyclictest805-21runrttasks22:45:591
840399140,10cyclictest805-21runrttasks20:57:031
840399140,10cyclictest762-21sshd21:55:321
840399140,10cyclictest5495-21ssh22:23:321
840399140,10cyclictest4839-21sh00:18:401
840399140,10cyclictest32180-21ssh23:50:541
840399140,10cyclictest31363-21/usr/sbin/munin23:05:251
840399140,10cyclictest31092-21runrttasks21:03:231
840399140,10cyclictest29962-21ssh23:26:411
840399140,10cyclictest27771-21sh23:01:481
840399140,10cyclictest27587-21/usr/sbin/munin19:50:261
840399140,10cyclictest27523-21sed00:10:261
840399140,10cyclictest25235-21/usr/sbin/munin19:45:211
840399140,10cyclictest25235-21/usr/sbin/munin19:45:211
840399140,10cyclictest23500-21runrttasks22:35:441
840399140,10cyclictest22603-21taskset00:34:051
840399140,10cyclictest21252-21ssh00:05:431
840399140,10cyclictest20507-21/usr/sbin/munin19:35:201
840399140,10cyclictest15657-21sh23:40:131
840399140,10cyclictest12591-21/usr/sbin/munin22:05:151
840399140,10cyclictest12356-21/usr/sbin/munin20:25:251
840399140,10cyclictest1164-21latency_hist21:10:001
840399140,10cyclictest10314-21ssh00:22:231
839999145,7cyclictest32402-21rm23:51:080
839999145,7cyclictest27666-21sh00:10:300
839999144,5cyclictest11600-21grep20:20:260
839999143,8cyclictest32691-21forks20:00:170
839999143,8cyclictest30306-21entropy19:55:150
839999143,8cyclictest11108-21proc_pri22:50:220
839999141,9cyclictest30543-21sh22:41:030
839999141,9cyclictest19661-21timerwakeupswit19:30:260
839999141,7cyclictest11590-21ssh00:23:460
839999141,10cyclictest3660-21ssh21:10:330
8399991410,3cyclictest32434-21sh22:20:000
839999140,9cyclictest805-21runrttasks20:09:520
839999140,7cyclictest6610-21ssh21:36:010
839999140,7cyclictest32272-21sed21:55:190
839999140,7cyclictest27982-21run-parts20:55:220
839999140,7cyclictest27811-21sed22:40:010
839999140,7cyclictest26975-21ssh21:50:590
839999140,7cyclictest25781-21sh23:46:420
839999140,7cyclictest22591-21ssh23:21:240
839999140,7cyclictest13194-21/usr/sbin/munin22:05:180
839999140,12cyclictest17074-21cat21:20:200
839999140,10cyclictest9863-21sendmail_mailqu19:10:200
839999140,10cyclictest9736-21rm23:58:310
839999140,10cyclictest805-21runrttasks00:25:280
839999140,10cyclictest609-21grep22:20:140
839999140,10cyclictest3341-21sh22:45:170
839999140,10cyclictest27788-21cpuspeed19:50:150
839999140,10cyclictest25152-21ssh21:25:450
839999140,10cyclictest21011-21fschecks_time19:35:170
839999140,10cyclictest19170-21taskset20:36:490
839999140,10cyclictest17637-21ssh23:18:220
839999140,10cyclictest17045-21sensors_temp19:25:210
839999140,10cyclictest16002-21sh21:43:320
839999140,10cyclictest15419-21chrt19:24:050
839999140,10cyclictest15008-21chrt20:30:120
839999140,10cyclictest11259-21sh21:16:020
2128399140,8tDemoTimingTask21291-21taskset23:44:105
1498099140,8tDemoTimingTask15073-21taskset22:53:305
841699135,6cyclictest466-21gltestperf20:00:184
841699134,5cyclictest30635-21cat19:55:184
841699131,9cyclictest19568-21/usr/sbin/munin00:05:154
841699130,9cyclictest906-21diskmemload21:55:374
841699130,9cyclictest7441-21runrttasks00:20:194
841699130,9cyclictest738-21snmpd19:21:064
841699130,9cyclictest4915-21/usr/sbin/munin20:10:264
841699130,9cyclictest4402-21runrttasks20:07:204
841699130,9cyclictest31289-21sh23:05:064
841699130,9cyclictest30536-21cut23:50:154
841699130,9cyclictest29800-21expr21:00:184
841699130,9cyclictest29683-21sh21:30:074
841699130,9cyclictest28662-21taskset20:57:394
841699130,9cyclictest24567-21/usr/sbin/munin23:00:154
841699130,9cyclictest21568-21sh00:32:344
841699130,9cyclictest21265-21kernelversion19:35:194
841699130,9cyclictest19497-21wget22:10:124
841699130,9cyclictest17690-21sh21:45:014
841699130,9cyclictest17690-21sh21:45:014
841699130,9cyclictest17124-21/usr/sbin/munin22:55:204
841699130,9cyclictest15232-21cpuspeed_turbos20:30:154
841699130,9cyclictest14035-21switchtime22:05:264
841699130,9cyclictest11020-21/usr/sbin/munin19:15:154
841699130,10cyclictest851-21taskset21:06:404
841699130,10cyclictest6436-21hddtemp_smartct22:00:194
841699130,10cyclictest30197-21ssh22:40:464
841699130,10cyclictest27918-21runrttasks00:10:464
841699130,10cyclictest27036-21chrt19:46:454
841699130,10cyclictest27036-21chrt19:46:444
841699130,10cyclictest26212-21/usr/sbin/munin22:15:164
841699130,10cyclictest23471-21sh23:45:194
841699130,10cyclictest20020-21runrttasks19:32:374
841299133,7cyclictest9953-21smartctl19:10:223
841299131,9cyclictest21293-21sh00:05:453
841299131,8cyclictest24636-21runrttasks23:45:373
841299130,9cyclictest805-21runrttasks19:55:143
841299130,9cyclictest7407-21/usr/sbin/munin22:25:253
841299130,9cyclictest26325-21taskset20:52:493
841299130,9cyclictest2560-21/usr/sbin/munin20:05:223
841299130,9cyclictest25235-21/usr/sbin/munin19:45:173
841299130,9cyclictest25235-21/usr/sbin/munin19:45:163
841299130,9cyclictest24903-21ssh21:25:303
841299130,9cyclictest22882-21/usr/sbin/munin19:40:143
841299130,9cyclictest19976-21chrt19:32:053
841299130,9cyclictest19739-21/usr/sbin/munin20:40:203
841299130,9cyclictest15798-21/usr/sbin/munin19:25:193
841299130,9cyclictest13446-21/usr/sbin/munin19:20:213
841299130,10cyclictest8626-21sed19:10:013
841299130,10cyclictest5604-21sh22:00:093
841299130,10cyclictest28160-21perl19:50:163
841299130,10cyclictest24371-21sh21:50:013
841299130,10cyclictest2420-21sh00:16:003
841299130,10cyclictest21479-21ssh22:10:403
841299130,10cyclictest10632-21idleruntime22:50:183
840799135,6cyclictest16172-21forks19:25:152
840799133,6cyclictest41-21ksoftirqd/221:55:172
840799132,7cyclictest41-21ksoftirqd/222:31:262
840799131,7cyclictest20081-21/usr/sbin/munin20:40:162
840799130,9cyclictest916-21sh21:32:092
840799130,9cyclictest8666-21/usr/sbin/munin19:10:242
840799130,9cyclictest7784-21cstates22:25:142
840799130,9cyclictest7264-21/usr/sbin/munin20:15:142
840799130,9cyclictest4915-21/usr/sbin/munin20:10:162
840799130,9cyclictest41-21ksoftirqd/223:40:232
840799130,9cyclictest41-21ksoftirqd/219:45:142
840799130,9cyclictest41-21ksoftirqd/219:45:142
840799130,9cyclictest31363-21/usr/sbin/munin23:05:152
840799130,9cyclictest27988-21ssh22:40:072
840799130,9cyclictest20493-21runrttasks19:35:012
840799130,9cyclictest20236-21sh00:30:452
840799130,9cyclictest19345-21grep21:45:262
840799130,9cyclictest19345-21grep21:45:252
840799130,9cyclictest18730-21fw_packets19:30:182
840799130,9cyclictest11020-21/usr/sbin/munin19:15:232
840799130,9cyclictest10531-21gltestperf20:20:162
840799130,8cyclictest41-21ksoftirqd/222:35:132
840799130,10cyclictest29903-21wget23:50:132
840799130,10cyclictest2099-21diskstats21:10:172
840799130,10cyclictest13394-21nfs4_client20:25:192
840399135,6cyclictest25099-21gltestperf20:50:161
840399131,9cyclictest21839-21sh23:20:391
840399131,9cyclictest16789-21runintdemo5min20:31:321
840399130,9cyclictest9847-21/usr/sbin/munin22:50:141
840399130,9cyclictest762-21sshd19:26:261
840399130,9cyclictest25203-21ssh23:46:101
840399130,9cyclictest20169-21sh22:10:161
840399130,9cyclictest19739-21/usr/sbin/munin20:40:221
840399130,9cyclictest17364-21/usr/sbin/munin20:35:141
840399130,9cyclictest16050-21/usr/sbin/munin21:20:181
840399130,9cyclictest10857-21ssh21:40:021
840399130,11cyclictest8855-21cpuspeed_turbos23:35:121
840399130,11cyclictest8855-21cpuspeed_turbos23:35:121
840399130,10cyclictest782-21postgres19:17:131
840399130,10cyclictest22089-21/usr/sbin/munin20:45:181
840399130,10cyclictest15979-21ssh22:30:221
840399130,10cyclictest15047-21unixbench_singl19:20:251
840399130,10cyclictest14060-21sh00:25:191
839999131,9cyclictest805-21runrttasks23:00:020
839999130,9cyclictest805-21runrttasks20:43:190
839999130,9cyclictest30766-21unixbench_singl21:00:260
839999130,9cyclictest23502-21gltestperf19:40:180
839999130,9cyclictest22489-21grep20:45:160
839999130,7cyclictest146-1kworker/0:1H+kblockd20:10:170
839999130,10cyclictest9070-21runrttasks20:16:420
839999130,10cyclictest25649-21grep19:45:170
839999130,10cyclictest25649-21grep19:45:170
839999130,10cyclictest24938-21perl20:50:150
695639130,0EcMasterDemoSyn0-21swapper/523:34:015
398799130,11tDemoTimingTask5298-21chrt23:09:585
30454991312,0tDemoTimingTask0-21swapper/522:23:005
1235199130,10tDemoTimingTask14111-21chrt19:20:175
1069239130,0EcMasterDemoSyn0-21swapper/523:59:235
841699120,9cyclictest805-21runrttasks20:35:594
841699120,9cyclictest23344-21smartctl20:45:224
841699120,9cyclictest20983-21/usr/sbin/munin20:40:234
841299125,6cyclictest10374-21fschecks_count20:20:173
841299120,9cyclictest32556-21cpuspeed_turbos20:00:163
841299120,9cyclictest15130-21cpu20:30:143
841299120,7cyclictest27147-21df20:55:153
840799120,9cyclictest22438-21df20:45:162
840799120,8cyclictest8485-21cat19:10:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional