You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 07:20
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack5slot1.osadl.org (updated Tue Nov 18, 2025 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2289391860,0EcMasterDemoSyn0-21swapper/423:27:354
16592391080,0EcMasterDemoSyn0-21swapper/420:55:274
22052950,1sleep30-21swapper/323:05:223
395039910,0EcMasterDemoSyn396029tAtEmLog_022:01:235
95632900,0sleep50-21swapper/522:05:195
51062830,1sleep45104-21apt-config19:25:004
2721439830,0EcMasterDemoSyn2722429tAtEmLog_023:22:305
984239820,0EcMasterDemoSyn985229tAtEmLog_022:26:445
803539820,0EcMasterDemoSyn804529tAtEmLog_022:47:015
450739820,0EcMasterDemoSyn451729tAtEmLog_019:24:105
2358139820,0EcMasterDemoSyn2359129tAtEmLog_000:28:265
1431739820,0EcMasterDemoSyn1432729tAtEmLog_019:44:275
577039810,0EcMasterDemoSyn578029tAtEmLog_021:41:065
2376939810,0EcMasterDemoSyn2377929tAtEmLog_020:04:445
2279939810,0EcMasterDemoSyn2280929tAtEmLog_021:30:575
1667439810,0EcMasterDemoSyn1668429tAtEmLog_019:49:305
2612839800,0EcMasterDemoSyn2613829tAtEmLog_020:09:485
2106639580,0EcMasterDemoSyn2107629tAtEmLog_021:56:175
29978992924,4cyclictest13571-21turbostat22:30:000
2997899190,15cyclictest28125-21sh23:00:300
106599192,9tDemoTimingTask3389-21taskset20:30:015
2997899185,9cyclictest805-21runrttasks20:27:450
2997899180,14cyclictest20237-21sh21:50:300
2997899180,10cyclictest7336-21sh22:04:230
2138939180,0EcMasterDemoSyn0-21swapper/519:59:395
1903239180,0EcMasterDemoSyn0-21swapper/119:54:351
2998399171,12cyclictest16-1pr/legacy20:24:591
2998399171,12cyclictest16-1pr/legacy20:24:591
2997899170,9cyclictest805-21runrttasks19:43:190
2997899170,9cyclictest10741-21rm21:23:580
2997899170,7cyclictest146-1kworker/0:1H+kblockd23:12:520
1748899170,9tDemoTimingTask21238-21taskset22:35:015
1482629170,7tAtEmLog_017924-21taskset21:28:355
1423339170,0EcMasterDemoSyn0-21swapper/520:50:235
2999199168,6cyclictest9622-21grep23:10:213
2999199165,6cyclictest334-21systemd-journal23:23:503
2999199161,11cyclictest16-1pr/legacy19:49:303
2999199160,9cyclictest11786-21idleruntime-cro00:20:013
2998799165,9cyclictest27334-21/usr/sbin/munin21:55:252
29987991612,3cyclictest4922-21sh00:15:002
2998399160,10cyclictest15165-21rm23:35:371
2997899163,9cyclictest14909-21sh22:09:230
2997899160,9cyclictest32340-21sh22:41:460
2997899160,10cyclictest384-21sh21:16:380
2981329160,7tAtEmLog_032162-21chrt19:13:595
2280929160,9tAtEmLog_027675-21chrt21:35:005
987439150,1EcMasterDemoSyn9880-21ps23:32:390
710829150,14tAtEmLog_04223-21turbostat21:24:555
2999699153,9cyclictest738-21snmpd23:12:344
2999699153,8cyclictest738-21snmpd21:05:024
2999699153,8cyclictest4815-21switchtime20:30:244
2999699152,9cyclictest6388-21ssh22:03:314
2999699150,9cyclictest23153-21runrttasks22:35:264
2999699150,9cyclictest18764-21sh21:29:214
2999699150,9cyclictest14106-21ssh22:08:374
2999699150,6cyclictest29284-21ssh23:45:214
2999199154,6cyclictest16312-21needreboot20:55:213
2999199153,9cyclictest704-21dbus-daemon23:50:233
2999199150,10cyclictest8674-21fschecks_count20:40:163
2998799157,7cyclictest1041-21apache223:11:232
2998799157,6cyclictest5438-21grep21:20:152
2998799155,8cyclictest927-21/usr/sbin/munin19:15:162
2998799155,5cyclictest12841-21uniq00:20:152
2998799154,8cyclictest281-21jbd2/md0-823:38:402
2998799154,8cyclictest21526-21load23:40:222
2998799151,10cyclictest25827-21memory00:05:182
2998799151,10cyclictest11228-21gltestperf20:45:172
2998399154,6cyclictest28281-21expr21:35:161
29983991511,3cyclictest4705-21cpuspeed_turbos22:45:181
2998399151,10cyclictest25439-21taskset23:20:531
2998399150,10cyclictest22391-21sh22:56:591
2997899156,7cyclictest12057-21awk21:45:200
2997899154,8cyclictest22676-21wc22:35:220
2997899153,10cyclictest2384-21mailstats23:05:240
2997899152,9cyclictest28881-21grep20:15:140
2997899152,7cyclictest18159-21ssh22:54:420
2997899151,10cyclictest24828-21sh21:10:410
2997899150,9cyclictest3663-21sh00:37:460
2997899150,8cyclictest805-21runrttasks21:07:110
2997899150,7cyclictest17767-21sh22:32:020
2997899150,6cyclictest3082-21ssh23:28:160
2997899150,10cyclictest29258-21chrt23:24:230
2999699144,8cyclictest29679-21/usr/sbin/munin21:15:134
2999699143,8cyclictest20449-21postgres23:18:054
2999699141,10cyclictest805-21runrttasks21:51:154
2999699141,10cyclictest30981-21cat20:20:004
29996991410,3cyclictest728-21gdbus00:15:004
2999699140,9cyclictest27056-21sh21:55:214
2999699140,10cyclictest805-21runrttasks21:31:164
2999699140,10cyclictest805-21runrttasks21:00:174
2999699140,10cyclictest805-21runrttasks00:37:334
2999699140,10cyclictest7938-21diskstats19:30:144
2999699140,10cyclictest7362-21rm21:21:014
2999699140,10cyclictest4255-21ssh22:45:064
2999699140,10cyclictest334-21systemd-journal22:50:074
2999699140,10cyclictest2843-21chrt20:25:354
2999699140,10cyclictest28203-21ssh21:14:034
2999699140,10cyclictest27778-21ssh21:35:064
2999699140,10cyclictest26495-21/usr/sbin/munin20:10:254
2999699140,10cyclictest26107-21runrttasks23:43:494
2999699140,10cyclictest25536-21/usr/sbin/munin00:30:264
2999699140,10cyclictest23530-21runrttasks22:15:044
2999699140,10cyclictest22903-21ssh00:27:424
2999699140,10cyclictest1892-21chrt19:16:054
2999699140,10cyclictest17212-21rm22:31:344
2999199143,8cyclictest6989-21switchtime23:30:243
2999199141,9cyclictest31767-21ssh22:41:163
2999199141,9cyclictest24809-21runrttasks23:42:403
2999199141,9cyclictest21574-21diskmemload00:26:133
29991991411,2cyclictest22754-21aten2_r5power_e21:10:163
2999199141,10cyclictest27423-21ssh22:39:093
2999199140,8cyclictest29479-21sh22:18:513
2999199140,12cyclictest32481-21/usr/sbin/munin22:20:193
2999199140,10cyclictest8431-21ssh21:43:263
2999199140,10cyclictest805-21runrttasks22:51:373
2999199140,10cyclictest805-21runrttasks19:40:493
2999199140,10cyclictest6168-21expr20:35:183
2999199140,10cyclictest5219-21/usr/sbin/munin19:25:143
2999199140,10cyclictest4768-21/usr/sbin/munin21:20:253
2999199140,10cyclictest32318-21sh21:37:373
2999199140,10cyclictest32157-21ssh23:04:163
2999199140,10cyclictest3162-21sh22:00:413
2999199140,10cyclictest30520-21ssh21:57:503
2999199140,10cyclictest30417-21ssh00:33:403
2999199140,10cyclictest30247-21/usr/sbin/munin19:10:203
2999199140,10cyclictest2630-21wget19:20:153
2999199140,10cyclictest23738-21runrttasks20:04:273
2999199140,10cyclictest21315-21ssh22:13:203
2999199140,10cyclictest21315-21ssh22:13:203
2999199140,10cyclictest20591-21tr21:30:143
2999199140,10cyclictest19916-21sh22:34:033
2999199140,10cyclictest18417-21find19:50:283
2999199140,10cyclictest18246-21perl00:00:163
2999199140,10cyclictest17347-21sh00:23:473
2999199140,10cyclictest15623-21/usr/sbin/munin23:15:233
2999199140,10cyclictest15426-21ssh21:26:243
2999199140,10cyclictest13622-21ssh23:56:533
2999199140,10cyclictest13622-21ssh23:56:523
2999199140,10cyclictest12370-21chrt20:46:083
2999199140,10cyclictest1211-21wget20:25:143
2999199140,10cyclictest10795-21ssh22:05:423
2999199140,10cyclictest10665-21sh21:45:073
2999199140,10cyclictest10479-21sh22:49:083
2999199140,10cyclictest10236-21sh22:27:043
2998799146,6cyclictest19452-21cut19:55:152
2998799143,8cyclictest19077-21sed21:00:252
2998799143,6cyclictest14428-21basename19:44:592
2998799142,8cyclictest17450-21cut19:50:162
2998799141,9cyclictest24095-21ssh21:10:252
2998799141,9cyclictest13810-21ssh21:46:282
2998799141,10cyclictest805-21runrttasks19:14:202
2998799141,10cyclictest6553-21ssh23:30:172
2998799141,10cyclictest13881-21missed_timers20:50:192
29987991410,3cyclictest28623-21gltestperf21:35:182
29987991410,3cyclictest1067-21latency_hist20:25:012
29987991410,3cyclictest1067-21latency_hist20:25:002
2998799140,9cyclictest9118-21ssh22:26:052
2998799140,9cyclictest25223-21diskmemload22:59:372
2998799140,11cyclictest827-21rm22:20:272
2998799140,10cyclictest9685-21ssh21:44:322
2998799140,10cyclictest9575-21sh22:48:182
2998799140,10cyclictest8578-21/usr/sbin/munin22:05:222
2998799140,10cyclictest805-21runrttasks21:50:552
2998799140,10cyclictest805-21runrttasks00:28:132
2998799140,10cyclictest7281-21rm22:04:192
2998799140,10cyclictest6589-21ssh23:52:052
2998799140,10cyclictest3530-21/usr/sbin/munin20:30:152
2998799140,10cyclictest32159-21/usr/sbin/munin00:35:182
2998799140,10cyclictest31565-21ssh21:15:292
2998799140,10cyclictest30521-21rm23:46:072
2998799140,10cyclictest29728-21ssh23:02:012
2998799140,10cyclictest27632-21sh23:22:532
2998799140,10cyclictest24194-21ssh22:36:142
2998799140,10cyclictest20265-21sh00:01:082
2998799140,10cyclictest17653-21rm22:31:572
2998799140,10cyclictest16760-21latency_hist19:50:002
2998799140,10cyclictest15082-21rm23:58:092
2998799140,10cyclictest15082-21rm23:58:092
2998799140,10cyclictest12494-21/usr/sbin/munin22:50:162
2998399147,6cyclictest4149-21grep20:30:151
2998399146,6cyclictest26732-21cut23:00:161
2998399146,6cyclictest12472-21/usr/sbin/munin21:25:171
2998399143,8cyclictest754-21lldpd19:25:101
2998399143,8cyclictest23883-21cut22:15:171
2998399143,6cyclictest20409-21sensors00:25:221
2998399142,9cyclictest752-21ntpd23:05:221
2998399141,9cyclictest8803-21ssh21:43:431
2998399141,9cyclictest805-21runrttasks23:16:281
2998399141,9cyclictest10454-21ssh23:33:081
2998399141,10cyclictest26941-21fschecks_count20:10:171
2998399141,10cyclictest12237-21ssh23:55:311
2998399141,10cyclictest12237-21ssh23:55:311
2998399140,9cyclictest2959-21/usr/sbin/munin23:50:241
2998399140,10cyclictest9403-21runrttasks19:32:011
2998399140,10cyclictest8373-21ssh00:16:261
2998399140,10cyclictest805-21runrttasks20:00:141
2998399140,10cyclictest805-21runrttasks00:20:031
2998399140,10cyclictest4446-21ssh22:01:471
2998399140,10cyclictest3568-21sh22:45:001
2998399140,10cyclictest32042-21ssh00:35:071
2998399140,10cyclictest31922-21fschecks_time00:10:181
2998399140,10cyclictest31628-21sh00:34:581
2998399140,10cyclictest26549-21runrttasks22:38:211
2998399140,10cyclictest24693-21ssh00:05:021
2998399140,10cyclictest2255-21ssh21:18:151
2998399140,10cyclictest20561-21/usr/sbin/munin23:40:251
2998399140,10cyclictest20267-21ssh21:30:081
2998399140,10cyclictest1938-21sh23:27:171
2998399140,10cyclictest1935-21taskset19:16:401
2998399140,10cyclictest19064-21hddtemp_smartct21:50:191
2998399140,10cyclictest18877-21awk21:00:221
2998399140,10cyclictest18183-21sh22:10:361
2998399140,10cyclictest18183-21sh22:10:361
2998399140,10cyclictest14890-21ssh23:14:481
2998399140,10cyclictest14009-21ssh22:30:081
2998399140,10cyclictest13786-21sh21:46:271
2998399140,10cyclictest10949-21sh22:05:511
2998399140,10cyclictest10732-21sh22:27:301
2997899145,7cyclictest25740-21ssh21:55:060
2997899145,7cyclictest14519-21sh23:57:400
2997899145,7cyclictest14519-21sh23:57:400
2997899143,8cyclictest13981-21latency23:35:170
2997899141,9cyclictest5569-21ssh00:15:040
2997899141,9cyclictest28957-21missed_timers23:45:180
2997899141,7cyclictest805-21runrttasks00:28:050
2997899140,7cyclictest28044-21aten2_r5power_p21:35:150
2997899140,7cyclictest24174-21/usr/sbin/munin22:15:190
2997899140,7cyclictest15932-21ssh22:10:030
2997899140,7cyclictest15932-21ssh22:10:030
2997899140,7cyclictest15206-21sh00:21:330
2997899140,7cyclictest12620-21ssh21:25:170
2997899140,12cyclictest9359-21awk20:40:200
2997899140,12cyclictest21532-21grep21:30:190
2997899140,12cyclictest20886-21sh00:01:440
2997899140,12cyclictest19581-21perl22:55:170
2997899140,12cyclictest16026-21diskstats23:15:160
2997899140,10cyclictest6838-21sh22:45:570
2997899140,10cyclictest5384-21rm23:50:580
2997899140,10cyclictest31310-21rm22:20:110
2997899140,10cyclictest27644-21sh00:30:440
2997899140,10cyclictest26093-21chrt20:09:270
2997899140,10cyclictest24047-21ssh23:41:570
2745999148,5tDemoTimingTask2745829tAtEmLog_019:08:335
2999699130,9cyclictest8223-21runintdemo5min20:40:134
2999699130,9cyclictest805-21runrttasks20:51:074
2999699130,9cyclictest7285-21if_err_enp5s022:25:184
2999699130,9cyclictest5452-21sh23:30:104
2999699130,9cyclictest31229-21/usr/sbin/munin20:20:144
2999699130,9cyclictest31229-21/usr/sbin/munin20:20:144
2999699130,9cyclictest30148-21latency_hist19:10:004
2999699130,9cyclictest2987-21sh21:40:034
2999699130,9cyclictest28772-21rm22:40:024
2999699130,9cyclictest2684-21ssh23:50:014
2999699130,9cyclictest24142-21/usr/sbin/munin20:05:214
2999699130,9cyclictest22692-21sshd00:03:244
2999699130,9cyclictest21445-21chrt19:59:544
2999699130,9cyclictest20266-21sendmail_mailqu22:55:224
2999699130,9cyclictest16497-21df22:10:144
2999699130,9cyclictest16497-21df22:10:144
2999699130,9cyclictest14131-21chrt19:42:034
2999699130,9cyclictest13464-21df_inode23:35:144
2999699130,9cyclictest12077-21mailstats21:45:204
2999699130,9cyclictest10617-21/usr/sbin/munin20:45:194
2999699130,9cyclictest10283-21/usr/sbin/munin23:55:194
2999699130,9cyclictest10283-21/usr/sbin/munin23:55:184
2999699130,10cyclictest905-21sh23:05:084
2999699130,10cyclictest6956-21cat20:35:244
2999699130,10cyclictest5485-21cat00:15:024
2999699130,10cyclictest3879-21switchtime19:20:264
2999699130,10cyclictest31129-21rm22:20:014
2999699130,10cyclictest29239-21ssh23:01:334
2999699130,10cyclictest27136-21sh00:05:524
2999699130,10cyclictest23655-21cut23:20:164
2999699130,10cyclictest23278-21cat20:00:254
2999699130,10cyclictest17233-21cpuspeed19:50:154
2999699130,10cyclictest15164-21cut19:45:184
2999699130,10cyclictest14164-21ssh00:20:294
2999699130,10cyclictest11981-21runrttasks19:39:264
2999199131,9cyclictest20590-21sendmail_mailqu19:55:233
2999199131,9cyclictest13846-21missed_timers20:50:193
2999199130,9cyclictest9985-21/usr/sbin/munin19:35:143
2999199130,9cyclictest30276-21runrttasks23:25:033
2999199130,9cyclictest28846-21/usr/sbin/munin20:15:143
2999199130,9cyclictest28343-21runrttasks20:13:043
2999199130,9cyclictest27259-21ssh00:06:003
2999199130,9cyclictest24142-21/usr/sbin/munin20:05:213
2999199130,9cyclictest2296-21sh00:36:163
2999199130,9cyclictest18584-21sed21:00:193
2999199130,9cyclictest18119-21sh21:50:043
2999199130,9cyclictest12930-21ssh23:35:043
2999199130,9cyclictest1210-21netstat19:15:183
2999199130,9cyclictest1111-21ssh00:10:503
2999199130,10cyclictest9016-21timerandwakeup19:30:243
2999199130,10cyclictest805-21runrttasks21:06:093
2999199130,10cyclictest585-21chrt20:22:333
2999199130,10cyclictest585-21chrt20:22:333
2999199130,10cyclictest4403-21memory20:30:173
2999199130,10cyclictest30162-21fschecks_time21:15:163
2999199130,10cyclictest27788-21sh23:45:063
2999199130,10cyclictest19364-21df22:55:163
2998799131,9cyclictest29798-21missed_timers20:15:192
2998799130,9cyclictest9564-21runrttasks20:40:232
2998799130,9cyclictest805-21runrttasks23:05:452
2998799130,9cyclictest7577-21/usr/sbin/munin19:30:212
2998799130,9cyclictest6406-21mailstats19:25:212
2998799130,9cyclictest30911-21ssh22:40:302
2998799130,9cyclictest30352-21ssh23:25:062
2998799130,9cyclictest27485-21ssh00:30:352
2998799130,9cyclictest26194-21idleruntime-cro20:10:002
2998799130,9cyclictest23622-21sshd20:02:542
2998799130,9cyclictest20045-21/usr/sbin/munin21:05:202
2998799130,9cyclictest17553-21diskmemload22:10:222
2998799130,9cyclictest17553-21diskmemload22:10:222
2998799130,9cyclictest15436-21rm23:15:022
2998799130,9cyclictest13739-21ps21:25:242
2998799130,8cyclictest22750-21ssh21:30:542
2998799130,10cyclictest805-21runrttasks22:15:102
2998799130,10cyclictest7563-21runrttasks20:35:452
2998799130,10cyclictest5963-21tr00:15:152
2998799130,10cyclictest2931-21diskstats19:20:172
2998799130,10cyclictest27307-21cat20:10:192
2998799130,10cyclictest15536-21taskset20:55:172
2998799130,10cyclictest1525-21cstates20:25:152
2998399138,2cyclictest1431739EcMasterDemoSyn19:44:261
2998399131,9cyclictest29195-21processes23:45:201
2998399130,9cyclictest805-21runrttasks20:26:361
2998399130,9cyclictest4592-21sh21:20:031
2998399130,9cyclictest32024-21chrt19:12:071
2998399130,9cyclictest28846-21/usr/sbin/munin20:15:261
2998399130,9cyclictest2669-21aten2_r5power_v19:20:161
2998399130,9cyclictest25899-21/usr/sbin/munin21:55:141
2998399130,9cyclictest25219-21sh21:11:021
2998399130,9cyclictest21934-21chrt21:07:571
2998399130,9cyclictest19417-21/usr/sbin/munin19:55:181
2998399130,9cyclictest10617-21/usr/sbin/munin20:45:141
2998399130,10cyclictest805-21runrttasks20:05:421
2998399130,10cyclictest6543-21cut20:35:201
2998399130,10cyclictest17472-21ssh00:00:031
2998399130,10cyclictest16340-21grep19:45:281
2998399130,10cyclictest1540-21ssh22:21:051
2998399130,10cyclictest14865-21runrttasks20:53:041
2998399130,10cyclictest12705-21sh22:50:181
2997899131,7cyclictest13329-21/usr/sbin/munin20:50:150
2997899130,9cyclictest9535-21runrttasks19:33:490
2997899130,9cyclictest5090-21gpgconf19:24:590
2997899130,9cyclictest16905-21unixbench_multi20:55:290
2997899130,9cyclictest12088-21sed20:45:260
2997899130,7cyclictest146-1kworker/0:1H+kblockd20:31:340
2997899130,7cyclictest146-1kworker/0:1H+kblockd20:21:220
2997899130,7cyclictest146-1kworker/0:1H+kblockd20:21:220
2997899130,7cyclictest146-1kworker/0:1H+kblockd20:03:130
2997899130,7cyclictest146-1kworker/0:1H+kblockd19:35:260
2997899130,11cyclictest31310-21ssh00:10:080
2997899130,11cyclictest19973-21cat21:05:000
2997899130,11cyclictest17561-21fschecks_time19:50:170
2997899130,10cyclictest805-21runrttasks19:45:500
2997899130,10cyclictest6331-21grep20:35:190
2997899130,10cyclictest4350-21runrttasks19:22:120
2997899130,10cyclictest3680-21uniq21:40:160
2997899130,10cyclictest328-21/usr/sbin/munin19:15:140
2997899130,10cyclictest30524-21cpuspeed_turbos19:10:150
2997899130,10cyclictest26409-21sh00:05:240
2997899130,10cyclictest21038-21grep19:55:280
1541099131,10tDemoTimingTask17483-21chrt22:54:045
823899120,10tDemoTimingTask9990-21chrt20:40:455
717799120,10tDemoTimingTask9523-21taskset19:33:395
585399120,10tDemoTimingTask7798-21chrt20:38:565
3188139120,2EcMasterDemoSyn31987-21chrt21:15:515
3024429120,2tAtEmLog_01756-21taskset00:35:415
2999699120,9cyclictest30247-21/usr/sbin/munin19:10:194
2998799120,9cyclictest10322-21cstates19:35:142
2998799120,7cyclictest30003-21latency_hist19:10:012
2998699120,10tDemoTimingTask3536-21chrt23:06:075
2998399120,9cyclictest8410-21runrttasks20:40:141
2998399120,9cyclictest15802-21fschecks_time20:55:191
2998399120,9cyclictest10368-21fschecks_count19:35:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional