You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-24 - 10:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun May 24, 2026 00:44:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
404303021590,3sleep13056-21PK-Backend19:55:181
406169021300,8sleep234-21ksoftirqd/220:50:002
404861921280,4sleep03056-21PK-Backend20:10:200
412060421170,2sleep10-21swapper/123:35:201
405553921010,4sleep10-21swapper/120:30:181
139099820,70rtkit-daemon0-21swapper/119:05:531
402565227344,20sleep20-21swapper/219:06:072
402529027145,16sleep00-21swapper/019:05:240
402448527146,16sleep30-21swapper/319:05:113
41169962440,3sleep13056-21PK-Backend23:25:181
40570922400,3sleep10-21swapper/120:35:161
4026101993733,3cyclictest34-21ksoftirqd/200:07:072
4026101993715,12cyclictest34-21ksoftirqd/200:30:012
41088412350,3sleep13056-21PK-Backend23:05:011
402610199327,5cyclictest34-21ksoftirqd/221:25:012
402610699296,2cyclictest141rcu_preempt22:10:003
402610199297,10cyclictest34-21ksoftirqd/223:50:002
402610199288,5cyclictest34-21ksoftirqd/221:30:002
402610699271,24cyclictest4037045-21sed19:40:013
139099270,4rtkit-daemon1389-21rtkit-daemon23:51:051
40575302260,5sleep24057535-21timerwakeupswit20:35:212
402610199269,5cyclictest34-21ksoftirqd/223:55:002
4026101992613,5cyclictest34-21ksoftirqd/221:05:012
402610199261,13cyclictest34-21ksoftirqd/221:20:012
40706782250,3chrt0-21swapper/121:15:141
4026106992519,4cyclictest4027760-21munin-plugin-st19:15:003
402609999250,8cyclictest4138666-21apt-key00:30:011
139099250,4rtkit-daemon1389-21rtkit-daemon21:27:010
402610199243,6cyclictest34-21ksoftirqd/221:40:122
402610199243,11cyclictest34-21ksoftirqd/221:05:182
4026101992420,3cyclictest331rcuc/221:50:012
4026101992417,5cyclictest34-21ksoftirqd/220:52:042
402610199240,6cyclictest4094698-21cut22:25:002
402610199240,6cyclictest34-21ksoftirqd/219:40:002
402610199240,10cyclictest34-21ksoftirqd/222:25:132
40904392230,3sleep30-21swapper/322:10:173
402610699232,2cyclictest141rcu_preempt22:05:003
4026106992313,8cyclictest4031120-21python319:20:213
4026106992311,10cyclictest4102373-21irqstats22:45:193
4026106992311,10cyclictest4102373-21irqstats22:45:193
402610699230,1cyclictest141rcu_preempt00:35:003
402610199234,9cyclictest34-21ksoftirqd/200:35:002
402610199234,5cyclictest34-21ksoftirqd/223:55:212
402610199233,10cyclictest34-21ksoftirqd/223:35:132
402610199232,11cyclictest34-21ksoftirqd/222:00:142
4026101992311,3cyclictest141rcu_preempt22:08:292
402610199230,8cyclictest34-21ksoftirqd/221:35:002
402609999230,21cyclictest3056-21PK-Backend19:35:121
4026095992321,1cyclictest0-21swapper/020:03:280
402609599231,20cyclictest4119478-21cron23:35:000
402610699227,6cyclictest42-21ksoftirqd/320:15:033
4026106992213,7cyclictest436-20systemd-journal21:40:003
402610699221,2cyclictest141rcu_preempt00:00:003
402610699220,1cyclictest0-21swapper/300:26:083
402610199229,2cyclictest141rcu_preempt23:40:022
402610199224,10cyclictest34-21ksoftirqd/222:35:002
402610199222,7cyclictest34-21ksoftirqd/220:55:172
4026101992211,3cyclictest141rcu_preempt20:29:282
402610199220,10cyclictest40338112cat19:30:132
4026099992219,2cyclictest0-21swapper/120:45:151
4026099992213,7cyclictest4069748-21users21:10:221
4026095992220,1cyclictest0-21swapper/023:26:470
4026095992220,1cyclictest0-21swapper/022:14:550
4026095992220,1cyclictest0-21swapper/019:45:440
4026095992220,1cyclictest0-21swapper/019:32:050
4026095992220,1cyclictest0-21swapper/019:21:510
402609599220,21cyclictest0-21swapper/000:37:250
402609599220,20cyclictest0-21swapper/019:55:030
4026106992118,2cyclictest0-21swapper/321:46:283
4026106992110,9cyclictest4084053-21unin-run21:55:003
402610699210,20cyclictest3056-21PK-Backend22:33:333
402610699210,20cyclictest3056-21PK-Backend22:29:143
402610699210,20cyclictest0-21swapper/320:31:043
402610199215,5cyclictest34-21ksoftirqd/220:45:002
402610199212,7cyclictest34-21ksoftirqd/222:10:122
402610199212,14cyclictest4049050-21apt-get20:15:032
4026101992119,1cyclictest0-21swapper/223:16:022
4026101992119,1cyclictest0-21swapper/223:03:182
4026101992119,1cyclictest0-21swapper/221:36:112
4026101992119,1cyclictest0-21swapper/220:01:072
4026101992119,1cyclictest0-21swapper/219:52:152
4026101992119,1cyclictest0-21swapper/219:52:152
402610199211,8cyclictest34-21ksoftirqd/221:10:122
4026101992117,3cyclictest331rcuc/222:40:302
402610199210,8cyclictest34-21ksoftirqd/200:20:002
402610199210,20cyclictest0-21swapper/222:46:482
402610199210,20cyclictest0-21swapper/222:46:482
402610199210,20cyclictest0-21swapper/219:49:522
4026099992114,6cyclictest4046734-21cat20:05:221
4026099992113,6cyclictest957-21gdbus21:32:061
4026099992113,3cyclictest26-21ksoftirqd/122:40:121
402609999210,20cyclictest3056-21PK-Backend21:53:221
402609999210,20cyclictest3056-21PK-Backend19:21:261
402609999210,20cyclictest0-21swapper/121:04:081
402609999210,19cyclictest3056-21PK-Backend19:41:081
4026095992119,1cyclictest0-21swapper/022:48:140
4026095992119,1cyclictest0-21swapper/022:48:140
4026095992119,1cyclictest0-21swapper/022:30:510
4026095992119,1cyclictest0-21swapper/021:31:020
4026095992119,1cyclictest0-21swapper/020:56:410
4026095992119,1cyclictest0-21swapper/000:22:310
4026095992119,1cyclictest0-21swapper/000:09:060
4026095992114,5cyclictest4115121-21load23:20:160
402609599210,20cyclictest0-21swapper/023:47:150
402609599210,20cyclictest0-21swapper/022:54:230
402609599210,20cyclictest0-21swapper/020:53:580
402609599210,20cyclictest0-21swapper/019:38:470
139099210,3rtkit-daemon1389-21rtkit-daemon21:46:071
4026106992018,1cyclictest3056-21PK-Backend21:22:013
4026106992018,1cyclictest0-21swapper/323:30:453
4026106992018,1cyclictest0-21swapper/323:14:503
4026106992018,1cyclictest0-21swapper/323:08:273
4026106992018,1cyclictest0-21swapper/322:18:253
4026106992018,1cyclictest0-21swapper/321:31:003
4026106992017,2cyclictest3056-21PK-Backend20:45:243
4026106992013,6cyclictest42-21ksoftirqd/319:15:113
4026106992012,6cyclictest4094297-21awk22:20:233
4026106992012,6cyclictest4039876-21ls19:45:223
4026106992011,7cyclictest42-21ksoftirqd/323:20:013
402610699201,17cyclictest1895009-21snapd19:43:433
4026106992011,5cyclictest42-21ksoftirqd/320:35:133
402610699200,1cyclictest0-21swapper/322:51:103
402610699200,1cyclictest0-21swapper/319:53:073
402610699200,1cyclictest0-21swapper/319:53:063
402610699200,19cyclictest3056-21PK-Backend23:54:033
402610699200,19cyclictest3056-21PK-Backend00:18:173
402610699200,19cyclictest0-21swapper/323:27:343
402610699200,19cyclictest0-21swapper/320:42:273
402610699200,19cyclictest0-21swapper/320:26:113
402610699200,19cyclictest0-21swapper/319:58:263
402610699200,19cyclictest0-21swapper/319:32:313
402610699200,19cyclictest0-21swapper/319:26:363
402610699200,19cyclictest0-21swapper/300:21:413
402610699200,19cyclictest0-21swapper/300:04:523
402610199209,4cyclictest34-21ksoftirqd/222:19:312
402610199204,9cyclictest34-21ksoftirqd/200:35:142
4026101992018,1cyclictest0-21swapper/223:21:362
4026101992018,1cyclictest0-21swapper/223:06:292
4026101992018,1cyclictest0-21swapper/219:55:262
4026101992018,1cyclictest0-21swapper/219:43:432
4026101992016,3cyclictest34-21ksoftirqd/220:21:212
4026101992016,3cyclictest34-21ksoftirqd/219:12:392
4026101992013,5cyclictest34-21ksoftirqd/223:15:002
4026101992011,6cyclictest34-21ksoftirqd/200:20:232
4026101992010,3cyclictest34-21ksoftirqd/223:32:282
402610199200,8cyclictest34-21ksoftirqd/219:15:022
402610199200,7cyclictest4054791-21apt20:30:102
402610199200,1cyclictest0-21swapper/222:59:192
4026099992018,1cyclictest3056-21PK-Backend00:00:031
4026099992018,1cyclictest0-21swapper/100:30:181
4026099992017,2cyclictest3056-21PK-Backend21:56:531
402609999201,6cyclictest4029997-21munin-run19:20:001
4026099992013,5cyclictest4074450-21load21:25:171
4026099992012,6cyclictest4127397-21ntpq23:55:181
402609999200,1cyclictest3056-21PK-Backend23:24:341
402609999200,1cyclictest3056-21PK-Backend21:40:041
402609999200,19cyclictest3056-21PK-Backend20:59:281
402609999200,19cyclictest3056-21PK-Backend20:14:531
402609999200,19cyclictest3056-21PK-Backend19:32:151
402609999200,19cyclictest3056-21PK-Backend00:37:071
402609999200,19cyclictest0-21swapper/123:49:491
402609999200,19cyclictest0-21swapper/122:59:331
402609999200,19cyclictest0-21swapper/122:22:351
402609999200,19cyclictest0-21swapper/121:35:101
402609999200,19cyclictest0-21swapper/121:08:131
4026095992018,1cyclictest0-21swapper/023:19:380
4026095992018,1cyclictest0-21swapper/021:59:480
4026095992018,1cyclictest0-21swapper/021:44:200
4026095992018,1cyclictest0-21swapper/021:35:550
4026095992018,1cyclictest0-21swapper/021:19:530
4026095992018,1cyclictest0-21swapper/019:44:080
4026095992018,1cyclictest0-21swapper/019:13:120
4026095992018,1cyclictest0-21swapper/000:14:060
4026095992017,2cyclictest3056-21PK-Backend20:47:080
4026095992016,2cyclictest13-21ksoftirqd/000:25:230
402609599200,19cyclictest3056-21PK-Backend20:40:390
402609599200,19cyclictest0-21swapper/023:44:390
402609599200,19cyclictest0-21swapper/023:11:400
402609599200,19cyclictest0-21swapper/022:23:340
402609599200,19cyclictest0-21swapper/021:54:070
402609599200,19cyclictest0-21swapper/020:23:360
402610699193,7cyclictest42-21ksoftirqd/323:00:203
4026106991917,1cyclictest0-21swapper/300:11:413
4026106991915,3cyclictest978-21snmpd00:08:133
4026106991915,3cyclictest4045578-21dump-pmu-power20:05:003
402610699191,2cyclictest141rcu_preempt23:20:003
4026106991912,5cyclictest4122526-21sed23:40:223
4026106991912,5cyclictest4100871-21ntpq22:40:173
4026106991911,6cyclictest42-21ksoftirqd/323:35:163
4026106991911,2cyclictest42-21ksoftirqd/321:05:193
402610699190,6cyclictest3056-21PK-Backend00:35:183
402610699190,5cyclictest0-21swapper/320:55:203
402610699190,2cyclictest3056-21PK-Backend20:50:163
402610699190,1cyclictest3056-21PK-Backend21:00:593
402610699190,18cyclictest0-21swapper/322:55:093
402610699190,18cyclictest0-21swapper/321:42:573
402610699190,18cyclictest0-21swapper/320:21:193
402610199192,7cyclictest34-21ksoftirqd/222:50:152
4026101991917,1cyclictest0-21swapper/223:25:352
4026101991917,1cyclictest0-21swapper/222:38:172
4026101991915,3cyclictest331rcuc/220:08:332
402610199191,13cyclictest4031122-21runrttasks19:20:202
402610199190,5cyclictest0-21swapper/221:55:092
402610199190,1cyclictest0-21swapper/219:25:452
402610199190,18cyclictest0-21swapper/220:10:062
402609999198,2cyclictest141rcu_preempt20:40:051
402609999197,5cyclictest141rcu_preempt20:50:461
402609999196,5cyclictest141rcu_preempt23:18:191
402609999192,13cyclictest4089754-21apt22:10:111
4026099991917,1cyclictest3056-21PK-Backend22:38:441
4026099991917,1cyclictest3056-21PK-Backend20:01:141
4026099991917,1cyclictest3056-21PK-Backend19:51:001
4026099991917,1cyclictest3056-21PK-Backend19:51:001
4026099991917,1cyclictest0-21swapper/123:11:161
4026099991917,1cyclictest0-21swapper/122:00:251
4026099991917,1cyclictest0-21swapper/119:14:581
402609999190,1cyclictest3056-21PK-Backend20:26:021
402609999190,1cyclictest3056-21PK-Backend19:29:171
402609999190,1cyclictest3056-21PK-Backend00:08:141
402609999190,1cyclictest0-21swapper/120:21:081
402609999190,18cyclictest3056-21PK-Backend23:44:011
402609999190,18cyclictest3056-21PK-Backend22:51:491
402609999190,18cyclictest3056-21PK-Backend19:45:401
4026095991917,1cyclictest3056-21PK-Backend23:00:450
4026095991917,1cyclictest3056-21PK-Backend19:15:230
4026095991917,1cyclictest0-21swapper/022:26:310
4026095991917,1cyclictest0-21swapper/020:32:480
4026095991917,1cyclictest0-21swapper/019:53:290
4026095991917,1cyclictest0-21swapper/019:53:280
4026095991911,7cyclictest4127340-21grep23:55:170
402609599190,3cyclictest4057987-21perf20:40:000
402609599190,1cyclictest0-21swapper/023:08:290
402609599190,1cyclictest0-21swapper/022:57:340
402609599190,1cyclictest0-21swapper/022:35:100
402609599190,1cyclictest0-21swapper/021:06:480
402610699187,5cyclictest42-21ksoftirqd/322:35:173
402610699181,15cyclictest4046716-21python320:05:203
402610699180,2cyclictest141rcu_preempt21:25:203
402610699180,1cyclictest3056-21PK-Backend21:12:193
402610199183,8cyclictest34-21ksoftirqd/200:03:592
402609999186,5cyclictest141rcu_preempt00:19:421
402609999181,8cyclictest141rcu_preempt00:20:131
4026099991816,1cyclictest3056-21PK-Backend00:14:021
4026099991816,1cyclictest0-21swapper/122:34:111
4026099991810,6cyclictest26-21ksoftirqd/121:20:171
402609999180,1cyclictest3056-21PK-Backend20:15:021
402609599189,7cyclictest2164-21gdbus20:07:020
4026095991812,5cyclictest4120139-21seq23:35:150
139099180,15rtkit-daemon0-21swapper/122:25:081
402610699179,6cyclictest42-21ksoftirqd/321:15:203
4026106991710,5cyclictest4123619-21ls23:45:173
402610699170,15cyclictest4084902-21missed_timers21:55:173
402610699170,15cyclictest0-21swapper/320:10:103
4026101991715,1cyclictest0-21swapper/221:48:562
402609999176,5cyclictest141rcu_preempt23:34:151
4026099991715,1cyclictest3056-21PK-Backend22:46:281
4026099991715,1cyclictest3056-21PK-Backend22:46:281
4026099991715,1cyclictest0-21swapper/122:16:541
4026099991714,2cyclictest3056-21PK-Backend23:09:401
4026099991713,2cyclictest26-21ksoftirqd/122:05:181
402609599179,6cyclictest4139842-21ntp_states00:30:190
4026095991715,1cyclictest3056-21PK-Backend21:49:130
4026095991715,1cyclictest0-21swapper/022:42:540
4026095991715,1cyclictest0-21swapper/022:08:260
4026095991715,1cyclictest0-21swapper/000:04:050
4026095991713,3cyclictest13-21ksoftirqd/020:30:010
4026095991711,4cyclictest4066866-21wc21:05:010
402609599171,12cyclictest4086291-21apt22:00:130
4026095991710,5cyclictest4073684-21wc21:25:020
402610199169,5cyclictest4132277-21df_inode00:10:152
4026095991614,1cyclictest0-21swapper/023:54:530
4026095991614,1cyclictest0-21swapper/019:26:580
4026095991610,4cyclictest4091668-21cpuspeed_turbos22:15:130
4026095991511,3cyclictest4051054-21latency_hist20:20:020
402609599149,3cyclictest4134102-21if_err_enp2s000:15:170
4026095991410,3cyclictest1095-21runrttasks21:14:340
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional