You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-07 - 18:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Tue Jul 07, 2026 12:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
403439722840,8sleep30-21swapper/307:05:253
415166821380,4sleep10-21swapper/112:35:211
409973021270,4sleep20-21swapper/210:10:142
405890221250,2sleep33056-21PK-Backend08:15:103
405890221250,2sleep33056-21PK-Backend08:15:103
413674521030,4sleep33056-21PK-Backend11:55:133
403498827941,14sleep00-21swapper/007:09:160
403505627146,16sleep10-21swapper/107:10:011
403488926847,14sleep20-21swapper/207:08:012
41458342510,5sleep24035186-21cyclictest12:20:142
41298832460,2sleep20-21swapper/211:35:112
41408652400,2sleep00-21swapper/012:05:160
40997932340,2sleep10-21swapper/110:10:181
41287432300,3sleep10-21swapper/111:30:201
403519799296,8cyclictest4149390-21df_inode12:30:141
403519799296,7cyclictest4091452-21taskset09:47:421
4035197992922,6cyclictest0-21swapper/110:35:111
403519799286,9cyclictest4112090-21memory10:45:171
403519799286,12cyclictest0-21swapper/108:36:291
403519799284,15cyclictest0-21swapper/111:20:171
403519799278,11cyclictest4134223-21chrt11:46:001
403519799276,9cyclictest4095854-21cstates10:00:141
403519799275,13cyclictest4139847-21taskset12:02:181
40842232260,4sleep20-21swapper/209:25:232
4035197992613,11cyclictest0-21swapper/112:25:171
4035197992611,13cyclictest1-21systemd09:20:001
403520499251,7cyclictest3056-21PK-Backend09:30:133
4035197992519,4cyclictest4082593-21users09:20:231
4035197992516,7cyclictest4041052-21cron07:25:011
4035192992513,8cyclictest4037740-21sort07:15:030
403519799249,13cyclictest4093540-21wc09:55:011
403519799245,6cyclictest4093713-21cat09:55:111
403519799245,6cyclictest4093713-21cat09:55:111
4035197992417,5cyclictest4071434-21cat08:50:181
4035197992414,8cyclictest4114711-21latency_hist10:55:001
4035192992415,7cyclictest4107383-21users10:30:230
403519299240,3cyclictest4125890-21munin-run11:25:000
40562622230,2sleep10-21swapper/108:05:221
4035204992312,10cyclictest42-21ksoftirqd/312:10:163
403519799238,13cyclictest4176944-21snapd08:11:111
403519799238,13cyclictest4116945-21find11:00:101
4035197992319,3cyclictest0-21swapper/107:49:301
4035197992314,7cyclictest4140985-21chrt12:05:171
4035197992314,7cyclictest4132079-21ntpq11:40:191
4035197992313,8cyclictest4036847-21apt-get07:14:591
403519299234,17cyclictest4127631-21perf11:30:000
4035204992220,1cyclictest3056-21PK-Backend10:34:413
4035204992219,1cyclictest3056-21PK-Backend07:55:153
4035204992218,2cyclictest3056-21PK-Backend10:47:543
403520499220,21cyclictest3056-21PK-Backend11:29:373
403520499220,20cyclictest3056-21PK-Backend11:17:063
403520499220,20cyclictest3056-21PK-Backend09:53:383
403520499220,20cyclictest3056-21PK-Backend08:01:373
403520499220,20cyclictest3056-21PK-Backend07:28:183
4035201992220,1cyclictest0-21swapper/211:25:062
4035201992220,1cyclictest0-21swapper/209:40:432
4035201992220,1cyclictest0-21swapper/209:19:342
4035201992220,1cyclictest0-21swapper/208:59:192
4035201992220,1cyclictest0-21swapper/207:46:402
403520199220,21cyclictest0-21swapper/211:08:372
403519799224,6cyclictest4067401-21apt08:40:121
403519799222,8cyclictest0-21swapper/108:55:181
4035197992220,1cyclictest0-21swapper/111:51:021
4035197992220,1cyclictest0-21swapper/107:16:591
4035197992214,6cyclictest4087382-21tail09:35:181
4035197992214,6cyclictest4061113-21needreboot08:20:191
4035197992213,7cyclictest4131093-21cron11:40:001
4035197992213,7cyclictest4089959-21latency_hist09:45:011
4035197992213,7cyclictest4042199-21python307:25:201
4035197992212,8cyclictest4120150-21cron11:10:001
4035197992212,8cyclictest4098314-21find10:05:211
4035197992212,5cyclictest4062199-21apt08:25:121
4035197992211,9cyclictest4147024-21awk12:25:001
4035192992215,6cyclictest1-21systemd10:40:000
4035204992119,1cyclictest3056-21PK-Backend12:36:423
4035204992119,1cyclictest3056-21PK-Backend11:11:123
4035204992119,1cyclictest3056-21PK-Backend10:43:203
4035204992119,1cyclictest3056-21PK-Backend09:24:453
4035204992119,1cyclictest3056-21PK-Backend08:54:443
4035204992118,2cyclictest3056-21PK-Backend12:18:583
4035204992118,2cyclictest3056-21PK-Backend12:07:423
4035204992118,2cyclictest3056-21PK-Backend11:35:113
4035204992118,2cyclictest3056-21PK-Backend10:11:363
4035204992118,2cyclictest3056-21PK-Backend09:13:573
4035204992118,2cyclictest3056-21PK-Backend08:43:563
4035204992111,3cyclictest42-21ksoftirqd/309:05:113
403520499210,19cyclictest3056-21PK-Backend10:36:513
403520499210,19cyclictest3056-21PK-Backend09:28:373
403520499210,19cyclictest3056-21PK-Backend08:28:353
403520499210,19cyclictest3056-21PK-Backend07:35:023
4035201992119,1cyclictest0-21swapper/212:25:562
4035201992119,1cyclictest0-21swapper/212:12:042
4035201992119,1cyclictest0-21swapper/212:05:082
4035201992119,1cyclictest0-21swapper/210:31:262
4035201992119,1cyclictest0-21swapper/210:16:182
4035201992119,1cyclictest0-21swapper/210:04:572
4035201992119,1cyclictest0-21swapper/209:22:522
4035201992119,1cyclictest0-21swapper/209:11:172
4035201992119,1cyclictest0-21swapper/209:09:202
4035201992119,1cyclictest0-21swapper/208:26:202
4035201992119,1cyclictest0-21swapper/208:07:142
4035201992119,1cyclictest0-21swapper/207:55:592
4035201992119,1cyclictest0-21swapper/207:25:372
4035201992117,3cyclictest34-21ksoftirqd/208:55:012
403520199210,20cyclictest0-21swapper/211:10:122
403520199210,20cyclictest0-21swapper/210:48:362
403520199210,20cyclictest0-21swapper/209:46:442
403520199210,20cyclictest0-21swapper/209:04:532
403520199210,20cyclictest0-21swapper/208:40:342
403519799217,11cyclictest886-21NetworkManager11:25:121
403519799216,11cyclictest0-21swapper/111:59:291
4035197992119,1cyclictest0-21swapper/112:10:491
4035197992119,1cyclictest0-21swapper/110:43:501
4035197992119,1cyclictest0-21swapper/110:34:241
4035197992115,5cyclictest0-21swapper/111:15:011
4035197992114,5cyclictest4050728-21needreboot07:50:161
4035197992113,6cyclictest4144306-21kernelversion12:15:181
4035197992113,6cyclictest4116476-21apt-key11:00:001
4035197992113,6cyclictest4101774-21needreboot10:15:171
4035197992113,6cyclictest4085180-21cpuspeed_turbos09:30:121
4035197992113,6cyclictest4058899-21cpuspeed_turbos08:15:111
4035197992113,6cyclictest4058899-21cpuspeed_turbos08:15:101
4035197992113,6cyclictest4054380-21python308:00:201
4035197992113,6cyclictest4046480-21latency_hist07:40:001
4035197992113,6cyclictest4043195-21grep07:30:151
4035197992111,8cyclictest4063909-21/usr/sbin/munin08:30:121
4035197992110,9cyclictest4120799-21taskset11:10:131
4035197992110,9cyclictest4083257-21taskset09:25:121
403519799210,20cyclictest0-21swapper/107:43:221
4035192992119,1cyclictest0-21swapper/011:41:330
4035192992118,2cyclictest4091577-21chrt09:49:180
403519299210,2cyclictest4129891-21expr11:35:110
403519299210,20cyclictest0-21swapper/012:20:330
403519299210,20cyclictest0-21swapper/011:11:590
403519299210,20cyclictest0-21swapper/008:09:420
139099210,3rtkit-daemon1389-21rtkit-daemon12:30:453
4035204992018,1cyclictest3056-21PK-Backend11:33:363
4035204992018,1cyclictest3056-21PK-Backend10:02:373
4035204992018,1cyclictest3056-21PK-Backend09:41:013
4035204992018,1cyclictest3056-21PK-Backend08:22:403
4035204992018,1cyclictest3056-21PK-Backend07:18:323
4035204992017,2cyclictest3056-21PK-Backend08:32:143
4035204992015,3cyclictest978-21snmpd10:05:243
403520499200,2cyclictest141rcu_preempt09:50:013
403520499200,1cyclictest3056-21PK-Backend11:51:403
403520499200,1cyclictest3056-21PK-Backend10:15:423
403520499200,1cyclictest3056-21PK-Backend08:56:403
403520499200,19cyclictest3056-21PK-Backend12:26:283
403520499200,19cyclictest3056-21PK-Backend10:28:123
403520499200,19cyclictest3056-21PK-Backend07:41:093
403520499200,19cyclictest0-21swapper/309:35:473
4035201992018,1cyclictest0-21swapper/211:51:022
4035201992018,1cyclictest0-21swapper/210:42:282
4035201992018,1cyclictest0-21swapper/209:56:312
4035201992018,1cyclictest0-21swapper/209:56:312
4035201992018,1cyclictest0-21swapper/209:36:032
4035201992018,1cyclictest0-21swapper/208:01:132
4035201992017,2cyclictest0-21swapper/211:04:042
4035201992012,6cyclictest4060835-21df_inode08:20:152
403520199200,1cyclictest0-21swapper/212:38:342
403520199200,1cyclictest0-21swapper/210:09:492
403520199200,19cyclictest3056-21PK-Backend07:10:242
403520199200,19cyclictest0-21swapper/211:19:182
403520199200,19cyclictest0-21swapper/210:25:322
403520199200,19cyclictest0-21swapper/207:39:162
403520199200,19cyclictest0-21swapper/207:31:322
403520199200,19cyclictest0-21swapper/207:21:182
403519799208,10cyclictest4103370-21cat10:20:171
4035197992013,5cyclictest4074982-21runrttasks09:00:141
4035197992012,6cyclictest4104731-21cut10:25:121
4035197992012,6cyclictest4078714-21tail09:10:171
4035197992011,7cyclictest4052189-21/usr/sbin/munin07:55:151
403519799200,19cyclictest0-21swapper/109:09:001
4035192992018,1cyclictest0-21swapper/009:16:130
4035192992018,1cyclictest0-21swapper/008:42:200
4035192992013,6cyclictest1-21systemd12:00:000
4035192992012,6cyclictest4115555-21memory10:55:180
4035192992012,6cyclictest4109768-21awk10:40:110
4035192992012,6cyclictest4099920-21latency10:10:180
4035192992012,6cyclictest4061539-21awk08:20:240
4035192992011,7cyclictest918-21rs:main0
403519299200,1cyclictest0-21swapper/007:21:220
403519299200,19cyclictest0-21swapper/012:38:170
403519299200,19cyclictest0-21swapper/009:52:020
403519299200,19cyclictest0-21swapper/009:26:280
403519299200,19cyclictest0-21swapper/008:01:110
403520499199,7cyclictest42-21ksoftirqd/307:50:183
4035204991917,1cyclictest3056-21PK-Backend11:04:293
4035204991917,1cyclictest3056-21PK-Backend10:24:413
4035204991917,1cyclictest3056-21PK-Backend08:10:573
4035204991917,1cyclictest3056-21PK-Backend07:34:133
4035204991914,3cyclictest4036122-21ntp_states07:10:193
4035204991911,3cyclictest42-21ksoftirqd/310:00:003
4035204991911,3cyclictest42-21ksoftirqd/310:00:003
403520499190,1cyclictest3056-21PK-Backend11:49:173
403520499190,1cyclictest3056-21PK-Backend10:51:383
403520499190,1cyclictest3056-21PK-Backend09:16:273
403520499190,1cyclictest3056-21PK-Backend08:46:263
403520499190,1cyclictest3056-21PK-Backend08:38:553
403520499190,18cyclictest3056-21PK-Backend10:59:563
4035201991917,1cyclictest0-21swapper/212:19:002
4035201991917,1cyclictest0-21swapper/212:03:322
4035201991917,1cyclictest0-21swapper/210:39:302
4035201991914,4cyclictest34-21ksoftirqd/211:00:002
4035201991912,5cyclictest4038585-21ntpq07:15:202
403520199190,1cyclictest0-21swapper/211:40:272
403520199190,1cyclictest0-21swapper/209:31:232
403520199190,1cyclictest0-21swapper/208:38:372
403520199190,1cyclictest0-21swapper/207:54:372
403520199190,18cyclictest3056-21PK-Backend12:30:292
4035197991910,7cyclictest4069902-21awk08:45:201
4035192991917,1cyclictest0-21swapper/012:10:050
4035192991917,1cyclictest0-21swapper/011:06:050
4035192991917,1cyclictest0-21swapper/010:05:550
4035192991917,1cyclictest0-21swapper/009:55:350
4035192991917,1cyclictest0-21swapper/009:55:340
4035192991917,1cyclictest0-21swapper/009:35:330
4035192991917,1cyclictest0-21swapper/008:49:160
4035192991915,3cyclictest0-21swapper/007:30:000
4035192991914,4cyclictest13-21ksoftirqd/007:10:160
4035192991913,5cyclictest4064309-21diskstats08:30:160
4035192991912,5cyclictest4059440-21ntpq08:15:160
4035192991912,5cyclictest4059440-21ntpq08:15:160
403519299190,1cyclictest0-21swapper/011:17:540
403519299190,1cyclictest0-21swapper/011:01:310
403519299190,1cyclictest0-21swapper/010:52:250
403519299190,1cyclictest0-21swapper/009:42:430
403519299190,1cyclictest0-21swapper/009:03:090
403519299190,18cyclictest0-21swapper/012:28:030
403519299190,18cyclictest0-21swapper/012:00:590
403519299190,18cyclictest0-21swapper/010:47:520
403519299190,18cyclictest0-21swapper/009:30:190
403519299190,18cyclictest0-21swapper/009:11:470
4035204991816,1cyclictest3056-21PK-Backend11:05:173
4035201991816,1cyclictest0-21swapper/211:56:362
4035201991816,1cyclictest0-21swapper/209:54:212
403519299189,7cyclictest4057377-21fw_packets08:10:150
4035192991816,1cyclictest0-21swapper/012:19:450
4035192991816,1cyclictest0-21swapper/011:45:320
4035192991816,1cyclictest0-21swapper/008:59:300
4035192991816,1cyclictest0-21swapper/007:32:110
4035192991812,5cyclictest4081825-21expr09:20:130
4035192991812,4cyclictest1095-21runrttasks07:35:210
4035192991811,6cyclictest4103740-21sed10:20:210
4035192991811,5cyclictest4102564-21cron10:20:010
403519299180,1cyclictest0-21swapper/007:52:520
403520199176,9cyclictest0-21swapper/211:35:012
4035201991715,1cyclictest0-21swapper/210:51:472
4035201991715,1cyclictest0-21swapper/210:21:592
4035201991715,1cyclictest0-21swapper/208:46:212
4035201991715,1cyclictest0-21swapper/208:18:242
4035201991715,1cyclictest0-21swapper/208:18:232
403519299179,6cyclictest4052063-21expr07:55:120
403519299178,8cyclictest4105288-21grep10:25:170
4035192991715,1cyclictest0-21swapper/011:50:180
4035192991710,6cyclictest4067085-21seq08:39:370
4035192991710,5cyclictest4077970-21sed09:10:000
4035204991614,1cyclictest3056-21PK-Backend12:00:123
4035204991614,1cyclictest3056-21PK-Backend09:01:403
4035201991612,3cyclictest4046566-21/usr/sbin/munin07:40:142
4035192991614,1cyclictest0-21swapper/007:49:480
4035192991613,2cyclictest4128632-21ntpq11:30:190
4035192991611,4cyclictest4071338-21awk08:50:160
4035204991513,1cyclictest3056-21PK-Backend08:06:183
4035204991513,1cyclictest3056-21PK-Backend07:24:473
403520199158,5cyclictest4134009-21cat11:45:212
4035201991513,1cyclictest0-21swapper/208:32:292
4035201991510,4cyclictest34-21ksoftirqd/208:11:592
403519299158,5cyclictest4096069-21basename10:00:170
4035192991511,3cyclictest4149668-21needreboot12:30:180
403519299151,11cyclictest4046592-21apt07:40:110
4035204991412,1cyclictest3056-21PK-Backend11:40:453
4035204991410,3cyclictest3056-21PK-Backend07:48:203
4035204991410,2cyclictest42-21ksoftirqd/311:20:113
403520499140,12cyclictest3056-21PK-Backend12:20:193
4035201991410,3cyclictest1095-21runrttasks11:24:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional