You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-22 - 20:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Wed Jul 22, 2026 12:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2729321210,4sleep23056-21PK-Backend08:15:222
139099890,77rtkit-daemon0-21swapper/207:05:562
267327739,32sleep10-21swapper/107:09:011
251327539,13sleep30-21swapper/307:07:133
100326142,13sleep00-21swapper/007:05:050
503842500,3sleep00-21swapper/009:20:230
139099450,4rtkit-daemon3056-21PK-Backend10:55:112
139099450,3rtkit-daemon0-21swapper/207:35:122
212002440,4sleep20-21swapper/208:00:022
288332370,7sleep2341ksoftirqd/208:20:192
2907992516,7cyclictest883-21dbus-daemon09:19:551
139099250,7rtkit-daemon0-21swapper/209:59:432
2907992417,4cyclictest41801-21turbostat.cron09:00:001
810772230,4sleep33056-21PK-Backend10:50:003
291699231,6cyclictest33396-21apt08:35:123
2903992310,11cyclictest38883-21expr08:50:100
139099230,16rtkit-daemon3056-21PK-Backend11:41:203
1028782230,2sleep23056-21PK-Backend11:50:182
692322220,2sleep33056-21PK-Backend10:15:153
388842220,3sleep342-21ksoftirqd/308:50:113
2916992220,1cyclictest0-21swapper/309:14:243
2916992219,2cyclictest3056-21PK-Backend09:20:523
2916992219,2cyclictest3056-21PK-Backend08:55:383
2916992219,2cyclictest3056-21PK-Backend07:47:453
291699220,20cyclictest3056-21PK-Backend12:24:463
291699220,20cyclictest3056-21PK-Backend09:33:033
2907992220,1cyclictest0-21swapper/110:10:101
2907992220,1cyclictest0-21swapper/107:10:511
2907992216,4cyclictest82702-21apt-get10:55:001
2907992213,6cyclictest104921-21python311:55:221
2907992210,6cyclictest20754-21pluginstate08:00:001
2903992213,7cyclictest29424-21sshd08:22:240
139099220,15rtkit-daemon3056-21PK-Backend08:20:293
139099220,14rtkit-daemon3056-21PK-Backend09:04:073
139099220,14rtkit-daemon0-21swapper/209:16:132
2916992119,1cyclictest3056-21PK-Backend11:37:353
2916992119,1cyclictest3056-21PK-Backend11:18:203
2916992119,1cyclictest3056-21PK-Backend10:59:153
2916992119,1cyclictest3056-21PK-Backend10:39:213
2916992119,1cyclictest3056-21PK-Backend09:35:543
2916992119,1cyclictest3056-21PK-Backend08:32:123
2916992119,1cyclictest3056-21PK-Backend07:14:323
2916992119,1cyclictest0-21swapper/312:36:073
2916992119,1cyclictest0-21swapper/311:31:473
2916992119,1cyclictest0-21swapper/310:30:433
2916992119,1cyclictest0-21swapper/309:15:263
2916992119,1cyclictest0-21swapper/308:40:183
2916992119,1cyclictest0-21swapper/308:28:343
291699210,20cyclictest3056-21PK-Backend11:07:543
291699210,20cyclictest3056-21PK-Backend10:53:143
291699210,20cyclictest3056-21PK-Backend09:49:393
291699210,20cyclictest3056-21PK-Backend07:28:123
291699210,19cyclictest3056-21PK-Backend10:25:153
291699210,19cyclictest3056-21PK-Backend10:05:143
2912992119,1cyclictest0-21swapper/212:27:202
2912992115,4cyclictest1053622sleep211:58:132
291299210,20cyclictest0-21swapper/210:30:462
291299210,20cyclictest0-21swapper/207:51:282
291299210,20cyclictest0-21swapper/207:51:272
291299210,19cyclictest3056-21PK-Backend07:10:042
291299210,19cyclictest0-21swapper/211:16:082
291299210,19cyclictest0-21swapper/210:10:192
2907992119,1cyclictest3056-21PK-Backend08:04:321
2907992119,1cyclictest0-21swapper/111:16:081
2907992119,1cyclictest0-21swapper/110:39:171
2907992119,1cyclictest0-21swapper/110:28:361
2907992119,1cyclictest0-21swapper/110:09:091
2907992119,1cyclictest0-21swapper/109:23:341
2907992119,1cyclictest0-21swapper/108:30:141
2907992114,5cyclictest26-21ksoftirqd/109:55:201
2907992113,6cyclictest96126-21ntpq11:30:181
2907992113,5cyclictest1095-21runrttasks09:35:081
2907992110,7cyclictest79243-21basename10:45:001
2907992110,7cyclictest79243-21basename10:44:591
290799210,20cyclictest0-21swapper/112:30:501
290799210,20cyclictest0-21swapper/112:13:541
290799210,20cyclictest0-21swapper/111:41:231
290799210,20cyclictest0-21swapper/111:35:341
2903992118,2cyclictest3056-21PK-Backend08:01:130
2903992113,7cyclictest41403-21users08:55:240
2903992113,6cyclictest60323-21expr09:50:120
2903992113,6cyclictest60323-21expr09:50:110
2903992113,6cyclictest33135-21/usr/sbin/munin08:35:000
2903992113,6cyclictest116461-21apt12:30:100
2903992113,6cyclictest100330-21/usr/sbin/munin11:45:000
2903992112,7cyclictest436-20systemd-journal09:45:000
290399210,20cyclictest3056-21PK-Backend12:25:240
2916992018,1cyclictest3056-21PK-Backend10:23:253
2916992018,1cyclictest3056-21PK-Backend10:03:253
2916992018,1cyclictest3056-21PK-Backend09:08:093
2916992018,1cyclictest3056-21PK-Backend08:14:083
2916992018,1cyclictest3056-21PK-Backend07:15:553
2916992018,1cyclictest0-21swapper/308:04:073
2916992017,2cyclictest3056-21PK-Backend09:56:083
291699200,1cyclictest3056-21PK-Backend09:53:173
291699200,1cyclictest3056-21PK-Backend09:53:173
291699200,1cyclictest3056-21PK-Backend07:40:213
291699200,19cyclictest3056-21PK-Backend07:24:133
291699200,19cyclictest0-21swapper/308:48:223
2912992018,1cyclictest3056-21PK-Backend11:35:502
2912992018,1cyclictest0-21swapper/212:14:232
2912992018,1cyclictest0-21swapper/211:23:112
2912992018,1cyclictest0-21swapper/211:14:202
2912992018,1cyclictest0-21swapper/209:23:142
2912992018,1cyclictest0-21swapper/209:01:092
2912992018,1cyclictest0-21swapper/208:25:212
2912992018,1cyclictest0-21swapper/208:06:282
291299200,1cyclictest0-21swapper/209:34:352
291299200,19cyclictest0-21swapper/211:46:372
291299200,19cyclictest0-21swapper/210:07:282
291299200,19cyclictest0-21swapper/209:13:552
291299200,19cyclictest0-21swapper/209:08:262
291299200,19cyclictest0-21swapper/208:10:402
291299200,19cyclictest0-21swapper/207:20:382
291299200,18cyclictest0-21swapper/210:00:582
290799209,8cyclictest436-20systemd-journal11:25:011
290799207,6cyclictest17501-21/usr/sbin/munin07:50:201
290799207,6cyclictest17501-21/usr/sbin/munin07:50:201
2907992018,1cyclictest3056-21PK-Backend09:10:481
2907992018,1cyclictest3056-21PK-Backend08:46:211
2907992018,1cyclictest0-21swapper/110:56:001
2907992018,1cyclictest0-21swapper/110:45:331
2907992017,2cyclictest3056-21PK-Backend08:42:441
2907992017,2cyclictest3056-21PK-Backend08:25:471
2907992017,2cyclictest0-21swapper/111:49:341
2907992017,2cyclictest0-21swapper/109:44:141
2907992017,2cyclictest0-21swapper/108:51:491
2907992017,1cyclictest0-21swapper/109:30:161
2907992016,3cyclictest107535-21grep12:05:131
2907992016,3cyclictest0-21swapper/112:25:161
2907992012,6cyclictest94029-21df11:25:131
2907992012,6cyclictest51159-21apt09:25:141
2907992012,6cyclictest42842-21awk09:00:211
290799201,15cyclictest1095-21runrttasks07:15:131
290799200,1cyclictest0-21swapper/107:36:331
290799200,19cyclictest0-21swapper/110:20:181
290799200,19cyclictest0-21swapper/109:50:441
290799200,19cyclictest0-21swapper/109:50:441
290799200,19cyclictest0-21swapper/109:05:221
290799200,19cyclictest0-21swapper/107:24:221
2903992018,1cyclictest0-21swapper/010:15:450
2903992017,2cyclictest0-21swapper/007:24:370
2903992015,3cyclictest978-21snmpd12:14:230
2903992015,3cyclictest85709-21ntp_states11:00:200
2903992015,3cyclictest2156-21gnome-shell12:00:090
2903992013,5cyclictest89898-21cron11:15:000
2903992013,5cyclictest105733-21sed12:00:010
2903992012,6cyclictest54365-21/usr/sbin/munin09:35:010
2903992012,6cyclictest49182-21systemctl09:20:000
2903992012,6cyclictest119661-21turbostat.cron12:39:590
2903992012,6cyclictest111782-21proc_pri12:15:200
290399200,19cyclictest0-21swapper/010:54:460
290399200,19cyclictest0-21swapper/010:32:560
290399200,19cyclictest0-21swapper/010:22:210
290399200,19cyclictest0-21swapper/007:54:310
290399200,19cyclictest0-21swapper/007:54:310
139099200,14rtkit-daemon3056-21PK-Backend12:10:443
139099200,13rtkit-daemon3056-21PK-Backend11:50:493
2916991917,1cyclictest3056-21PK-Backend10:12:303
2916991917,1cyclictest3056-21PK-Backend08:18:213
2916991916,2cyclictest3056-21PK-Backend09:40:343
291699190,1cyclictest3056-21PK-Backend11:25:113
291699190,18cyclictest3056-21PK-Backend08:09:553
291699190,18cyclictest3056-21PK-Backend07:55:573
2912991917,1cyclictest0-21swapper/212:35:322
2912991917,1cyclictest0-21swapper/211:31:362
2912991917,1cyclictest0-21swapper/208:51:032
2912991917,1cyclictest0-21swapper/208:40:552
2912991917,1cyclictest0-21swapper/208:32:382
2912991917,1cyclictest0-21swapper/207:55:402
2912991917,1cyclictest0-21swapper/207:44:392
2912991912,5cyclictest94382-21ntpq11:25:182
2912991912,5cyclictest78385-21wc10:40:202
2912991912,5cyclictest78385-21wc10:40:202
291299190,1cyclictest0-21swapper/212:01:242
291299190,1cyclictest0-21swapper/210:39:392
291299190,1cyclictest0-21swapper/209:45:592
291299190,1cyclictest0-21swapper/209:35:372
291299190,1cyclictest0-21swapper/209:28:552
291299190,1cyclictest0-21swapper/208:39:542
291299190,18cyclictest0-21swapper/210:53:582
291299190,18cyclictest0-21swapper/209:43:082
2907991917,1cyclictest0-21swapper/108:12:561
2907991914,3cyclictest87372-21grep11:05:171
2907991912,5cyclictest102951-21cat11:50:191
2907991912,4cyclictest89918-21sh11:15:011
2907991911,5cyclictest85634-21needreboot11:00:181
290799190,1cyclictest0-21swapper/110:02:401
290799190,1cyclictest0-21swapper/108:23:571
2903991917,1cyclictest0-21swapper/010:57:570
2903991913,4cyclictest24225-21sshd08:07:250
2903991913,4cyclictest1095-21runrttasks10:45:110
2903991912,5cyclictest92954-21cat11:20:210
2903991912,5cyclictest3809-21ntpq07:10:170
2903991911,6cyclictest42656-21wc09:00:180
2903991911,6cyclictest1-21systemd11:10:000
2903991910,7cyclictest2636-21xdg-desktop-por10:29:550
290399190,1cyclictest0-21swapper/009:05:290
290399190,1cyclictest0-21swapper/007:33:150
290399190,18cyclictest0-21swapper/010:10:440
290399190,18cyclictest0-21swapper/007:25:250
113792190,3chrt3056-21PK-Backend07:30:193
378672180,4sleep03056-21PK-Backend08:45:230
2916991816,1cyclictest3056-21PK-Backend10:41:093
2916991816,1cyclictest3056-21PK-Backend10:41:093
2916991816,1cyclictest3056-21PK-Backend09:28:233
291299182,12cyclictest36863-21apt08:45:132
2912991816,1cyclictest0-21swapper/210:48:442
2912991816,1cyclictest0-21swapper/209:53:292
2912991816,1cyclictest0-21swapper/209:53:282
2912991812,4cyclictest10258-21sh07:30:012
290799182,12cyclictest105928-21cpuspeed_turbos12:00:131
2907991816,1cyclictest3056-21PK-Backend08:36:291
2907991816,1cyclictest0-21swapper/110:34:511
2907991816,1cyclictest0-21swapper/109:45:171
2907991816,1cyclictest0-21swapper/108:15:321
2907991816,1cyclictest0-21swapper/107:49:451
2907991813,3cyclictest10259-21idleruntime-cro07:30:001
290799181,15cyclictest114216-21munin-run12:24:591
2907991810,5cyclictest12009-21idleruntime-cro07:35:001
290399184,12cyclictest103819-21unin-run11:55:010
290399182,14cyclictest51821-21tail09:25:200
290399182,14cyclictest108978-21cron12:10:000
290399182,12cyclictest61897-21apt09:55:130
2903991816,1cyclictest0-21swapper/011:48:460
2903991815,2cyclictest0-21swapper/011:25:210
2903991814,3cyclictest59107-21timerwakeupswit09:45:210
2903991813,3cyclictest25195-21kernelversion08:10:170
2903991813,3cyclictest19798-21latency07:55:190
2903991812,5cyclictest64305-21cut10:00:190
2903991812,4cyclictest6883-21sed07:20:010
2903991811,5cyclictest90506-21cat11:15:130
2903991811,5cyclictest14975-21cat07:40:210
291699170,15cyclictest116860-21hddtemp_smartct12:30:163
2907991715,1cyclictest0-21swapper/110:16:401
2907991715,1cyclictest0-21swapper/107:44:441
2907991712,3cyclictest111184-21cstates12:15:141
2903991713,3cyclictest45532-21apt09:10:100
2903991712,4cyclictest13-21ksoftirqd/010:45:000
2903991712,4cyclictest13-21ksoftirqd/010:45:000
2903991711,5cyclictest36583-21cron08:45:000
2903991711,5cyclictest16680-21ls07:45:220
290399170,3cyclictest96870-21basename11:35:000
290399170,16cyclictest0-21swapper/012:21:040
50012160,4chrt0-21swapper/207:15:022
2916991614,1cyclictest3056-21PK-Backend11:24:223
2916991614,1cyclictest3056-21PK-Backend07:39:333
291699161,13cyclictest104033-21/usr/sbin/munin11:55:183
291299169,5cyclictest40452-21grep08:55:152
2912991614,1cyclictest0-21swapper/207:47:152
2912991612,3cyclictest70983-21tr10:20:112
2912991610,5cyclictest116877-21ls12:30:162
2907991614,1cyclictest0-21swapper/108:09:321
290399169,5cyclictest55337-21ntpq09:35:190
290399169,5cyclictest12349-21ls07:35:120
2903991610,5cyclictest34872-21/usr/sbin/munin08:40:010
290399160,14cyclictest98622-20unin-run11:39:590
291699150,13cyclictest114431-21/usr/sbin/munin12:25:233
291299158,5cyclictest69018-21tr10:15:132
291299157,6cyclictest85422-21if_err_enp2s011:00:172
2912991511,2cyclictest34-21ksoftirqd/212:15:242
2912991510,3cyclictest73464-21switchtime10:25:222
2903991513,1cyclictest0-21swapper/008:16:010
2903991512,2cyclictest30039-21tr08:25:130
2903991511,3cyclictest76895-21sensors_temp10:35:200
2903991511,3cyclictest13-21ksoftirqd/010:07:110
1112362150,4sleep33056-21PK-Backend12:15:133
2916991412,1cyclictest3056-21PK-Backend11:49:413
2916991412,1cyclictest3056-21PK-Backend11:12:193
2916991412,1cyclictest3056-21PK-Backend11:01:043
2916991411,2cyclictest3056-21PK-Backend07:54:213
2916991411,2cyclictest3056-21PK-Backend07:54:213
291299141,11cyclictest107709-21cpuspeed_turbos12:05:142
290799142,10cyclictest118330-21grep12:35:121
291699139,3cyclictest3056-21PK-Backend12:01:473
291299138,3cyclictest34-21ksoftirqd/207:32:562
291299136,6cyclictest34-21ksoftirqd/212:20:412
291299136,5cyclictest87040-21expr11:05:142
291699125,5cyclictest108996-21sh12:10:003
291299129,2cyclictest98835-21/usr/sbin/munin11:40:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional