You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-27 - 06:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu Aug 27, 2026 00:44:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
327759521410,3sleep10-21swapper/123:55:131
326022321360,2sleep13056-21PK-Backend23:05:131
323426421200,3sleep30-21swapper/321:50:203
325457121040,3sleep20-21swapper/222:50:142
1390991030,58rtkit-daemon0-21swapper/119:08:161
317510027144,16sleep00-21swapper/019:05:240
317572926444,14sleep30-21swapper/319:09:453
317546026242,14sleep20-21swapper/219:06:132
32375292550,4sleep30-21swapper/322:00:143
32588272520,3sleep23056-21PK-Backend23:00:202
32863512510,3sleep00-21swapper/000:20:140
32619772490,3sleep30-21swapper/323:10:123
32341332380,2sleep10-21swapper/121:50:181
3176003993632,3cyclictest13-21ksoftirqd/020:09:280
32042502330,2sleep10-21swapper/120:25:131
31769962280,4sleep33056-21PK-Backend19:10:163
3176003992814,4cyclictest13-21ksoftirqd/023:05:020
32919652260,4sleep23056-21PK-Backend00:35:162
317601499269,4cyclictest34-21ksoftirqd/200:34:212
3176014992611,4cyclictest34-21ksoftirqd/220:48:492
3176014992611,4cyclictest34-21ksoftirqd/200:09:232
317600799260,24cyclictest3256717-21cut22:55:171
3176003992619,5cyclictest3223738-21ntp_states21:20:200
3176017992519,5cyclictest0-21swapper/320:34:503
317601499259,4cyclictest34-21ksoftirqd/221:22:052
3176014992512,3cyclictest34-21ksoftirqd/222:08:082
3176014992511,3cyclictest34-21ksoftirqd/219:19:152
3176014992510,4cyclictest34-21ksoftirqd/222:57:052
3176003992521,3cyclictest3183710-21nm-dispatcher19:29:240
139099250,4rtkit-daemon1389-21rtkit-daemon23:36:370
317601799242,20cyclictest0-21swapper/320:11:253
3176014992410,4cyclictest34-21ksoftirqd/220:06:222
3176017992321,1cyclictest0-21swapper/300:28:523
3176017992320,1cyclictest3056-21PK-Backend23:40:193
3176017992318,4cyclictest957-21gdbus19:49:273
3176017992310,12cyclictest3185511-21latency_hist19:34:593
3176014992319,3cyclictest34-21ksoftirqd/219:40:012
3176014992317,4cyclictest34-21ksoftirqd/220:30:002
3176014992311,4cyclictest34-21ksoftirqd/221:19:072
3176014992311,4cyclictest34-21ksoftirqd/220:50:132
3176014992310,4cyclictest34-21ksoftirqd/222:23:402
3176007992321,1cyclictest0-21swapper/120:40:521
3176007992318,3cyclictest3230296-21df_inode21:40:131
317600799230,21cyclictest3056-21PK-Backend19:43:331
317600399232,9cyclictest13-21ksoftirqd/019:47:320
3176003992321,1cyclictest0-21swapper/022:42:130
3176003992320,2cyclictest161rcuc/019:18:170
139099230,4rtkit-daemon1389-21rtkit-daemon23:11:550
3176017992220,1cyclictest3056-21PK-Backend22:45:173
3176017992219,2cyclictest3056-21PK-Backend21:47:513
3176017992217,4cyclictest3191716-21if_err_enp2s019:50:163
317601799220,21cyclictest3056-21PK-Backend00:01:553
317601499229,4cyclictest34-21ksoftirqd/220:23:312
317601499228,4cyclictest34-21ksoftirqd/222:26:132
3176014992217,4cyclictest34-21ksoftirqd/219:49:282
3176014992212,4cyclictest34-21ksoftirqd/220:33:442
317601499220,21cyclictest3287967-21basename00:25:112
3176007992220,1cyclictest3056-21PK-Backend20:32:001
3176007992220,1cyclictest0-21swapper/122:01:091
3176007992220,1cyclictest0-21swapper/119:26:021
317600399229,3cyclictest141rcu_preempt19:42:100
3176003992220,1cyclictest0-21swapper/023:49:110
3176003992220,1cyclictest0-21swapper/023:05:040
3176003992220,1cyclictest0-21swapper/021:35:140
3176003992220,1cyclictest0-21swapper/021:34:250
3176003992220,1cyclictest0-21swapper/021:29:320
3176003992220,1cyclictest0-21swapper/000:36:490
3176003992215,5cyclictest13-21ksoftirqd/022:25:000
3176003992214,6cyclictest883-21dbus-daemon20:25:000
3176003992213,7cyclictest3264874-21cron23:20:000
317600399220,6cyclictest13-21ksoftirqd/022:25:170
317600399220,3cyclictest3056-21PK-Backend21:45:010
139099220,4rtkit-daemon1389-21rtkit-daemon22:33:550
139099220,15rtkit-daemon0-21swapper/019:30:170
317601799214,5cyclictest3198212-21basename20:10:013
3176017992119,1cyclictest3056-21PK-Backend23:00:113
3176017992119,1cyclictest3056-21PK-Backend22:17:183
3176017992119,1cyclictest3056-21PK-Backend21:38:113
3176017992119,1cyclictest3056-21PK-Backend21:24:123
3176017992119,1cyclictest0-21swapper/321:10:403
3176017992119,1cyclictest0-21swapper/320:38:363
3176017992119,1cyclictest0-21swapper/319:44:353
3176017992119,1cyclictest0-21swapper/319:27:383
3176017992119,1cyclictest0-21swapper/300:19:523
3176017992115,4cyclictest3209280-21cpuspeed_turbos20:40:113
3176017992114,6cyclictest3270928-21memory23:35:173
317601799210,20cyclictest3056-21PK-Backend20:53:373
317601799210,20cyclictest0-21swapper/323:55:193
317601799210,19cyclictest3056-21PK-Backend22:25:353
317601499219,2cyclictest141rcu_preempt23:40:312
317601499218,9cyclictest2013-21gdbus23:49:352
317601499216,8cyclictest34-21ksoftirqd/222:40:112
3176014992116,3cyclictest436-20systemd-journal20:39:592
317601499210,20cyclictest0-21swapper/220:18:262
3176007992119,1cyclictest3056-21PK-Backend19:37:381
3176007992119,1cyclictest3056-21PK-Backend19:32:311
3176007992119,1cyclictest0-21swapper/123:53:091
3176007992119,1cyclictest0-21swapper/123:45:451
3176007992119,1cyclictest0-21swapper/122:29:141
3176007992119,1cyclictest0-21swapper/121:37:161
3176007992119,1cyclictest0-21swapper/119:50:361
3176007992119,1cyclictest0-21swapper/100:14:311
317600799210,20cyclictest3056-21PK-Backend22:52:531
317600799210,20cyclictest0-21swapper/123:37:201
317600799210,19cyclictest0-21swapper/123:25:111
317600399219,4cyclictest13-21ksoftirqd/023:31:550
317600399217,4cyclictest13-21ksoftirqd/023:27:010
317600399216,5cyclictest13-21ksoftirqd/023:55:090
317600399213,9cyclictest13-21ksoftirqd/020:55:170
317600399212,9cyclictest13-21ksoftirqd/022:05:010
317600399212,10cyclictest13-21ksoftirqd/022:46:070
3176003992119,1cyclictest3056-21PK-Backend20:40:460
3176003992119,1cyclictest0-21swapper/023:42:020
3176003992119,1cyclictest0-21swapper/021:48:260
3176003992119,1cyclictest0-21swapper/020:48:030
3176003992119,1cyclictest0-21swapper/020:14:500
3176003992118,2cyclictest161rcuc/019:10:540
317600399210,2cyclictest3190722-21apt-get19:50:030
317600399210,20cyclictest0-21swapper/021:06:280
139099210,18rtkit-daemon3281447-21strings00:05:210
139099210,14rtkit-daemon0-21swapper/120:07:161
3176017992018,1cyclictest3056-21PK-Backend19:16:303
3176017992018,1cyclictest0-21swapper/320:20:173
3176017992018,1cyclictest0-21swapper/300:10:073
3176017992013,5cyclictest42-21ksoftirqd/322:40:203
3176017992011,7cyclictest3242069-21munin-run22:15:003
317601799200,1cyclictest3056-21PK-Backend22:57:463
317601799200,1cyclictest0-21swapper/323:46:073
317601799200,19cyclictest3056-21PK-Backend20:49:513
317601799200,19cyclictest0-21swapper/322:54:353
317601799200,19cyclictest0-21swapper/321:55:203
317601799200,19cyclictest0-21swapper/320:04:083
317601499209,4cyclictest34-21ksoftirqd/222:37:592
317601499209,4cyclictest34-21ksoftirqd/200:18:002
317601499207,10cyclictest883-21dbus-daemon19:29:272
317601499204,8cyclictest34-21ksoftirqd/223:11:022
317601499202,4cyclictest34-21ksoftirqd/221:35:182
3176014992018,1cyclictest0-21swapper/223:17:452
3176014992018,1cyclictest0-21swapper/222:34:522
3176014992017,2cyclictest3056-21PK-Backend21:55:262
3176014992017,2cyclictest3056-21PK-Backend19:10:052
317601499201,3cyclictest0-21swapper/222:10:172
3176014992012,6cyclictest3283159-21fw_packets00:10:142
317601499201,15cyclictest3226465-21seq21:30:122
317601499200,19cyclictest3056-21PK-Backend22:18:242
317601499200,19cyclictest0-21swapper/223:38:192
317601499200,19cyclictest0-21swapper/220:00:552
317601499200,19cyclictest0-21swapper/219:40:192
317601499200,19cyclictest0-21swapper/200:04:222
3176007992018,1cyclictest3056-21PK-Backend20:19:351
3176007992018,1cyclictest0-21swapper/123:40:451
3176007992018,1cyclictest0-21swapper/122:35:361
3176007992018,1cyclictest0-21swapper/122:16:561
3176007992018,1cyclictest0-21swapper/122:05:421
3176007992018,1cyclictest0-21swapper/121:46:071
3176007992018,1cyclictest0-21swapper/120:52:061
3176007992018,1cyclictest0-21swapper/120:00:571
3176007992016,3cyclictest1095-21runrttasks23:01:311
3176007992012,6cyclictest3292601-21sshd00:36:121
3176007992012,6cyclictest3281673-21sed00:05:241
3176007992012,6cyclictest3248246-21ntpq22:30:181
317600799200,19cyclictest3056-21PK-Backend21:27:021
317600799200,19cyclictest3056-21PK-Backend20:59:241
317600799200,19cyclictest0-21swapper/122:21:301
317600799200,19cyclictest0-21swapper/121:56:361
317600799200,19cyclictest0-21swapper/120:23:551
317600799200,19cyclictest0-21swapper/119:20:211
317600799200,19cyclictest0-21swapper/100:20:201
317600399209,3cyclictest141rcu_preempt22:10:590
317600399207,4cyclictest13-21ksoftirqd/021:14:410
317600399207,3cyclictest141rcu_preempt20:17:310
317600399206,7cyclictest13-21ksoftirqd/000:12:040
317600399205,4cyclictest13-21ksoftirqd/000:15:210
317600399203,8cyclictest13-21ksoftirqd/000:25:160
317600399201,9cyclictest13-21ksoftirqd/019:55:140
3176003992018,1cyclictest0-21swapper/022:18:130
3176003992018,1cyclictest0-21swapper/019:37:320
3176003992018,1cyclictest0-21swapper/019:24:480
317600399200,9cyclictest3279133-21taskset00:00:120
317600399200,1cyclictest3056-21PK-Backend21:57:170
317600399200,19cyclictest0-21swapper/022:06:580
317600399200,19cyclictest0-21swapper/021:53:320
317600399200,11cyclictest13-21ksoftirqd/020:00:190
139099200,14rtkit-daemon3056-21PK-Backend21:09:281
32512042190,4chrt0-21swapper/122:40:101
3176017991918,1cyclictest0-21swapper/320:28:563
3176017991917,1cyclictest3056-21PK-Backend22:39:423
3176017991917,1cyclictest3056-21PK-Backend21:16:213
3176017991917,1cyclictest0-21swapper/323:29:303
3176017991917,1cyclictest0-21swapper/321:44:063
3176017991917,1cyclictest0-21swapper/321:30:063
3176017991917,1cyclictest0-21swapper/320:19:303
3176017991917,1cyclictest0-21swapper/319:37:533
3176017991915,3cyclictest978-21snmpd22:32:593
3176017991914,3cyclictest0-21swapper/323:05:113
3176017991912,6cyclictest883-21dbus-daemon19:58:033
317601799190,1cyclictest3056-21PK-Backend22:21:033
317601799190,1cyclictest3056-21PK-Backend22:07:253
317601499198,3cyclictest34-21ksoftirqd/222:48:272
317601499198,2cyclictest141rcu_preempt23:30:022
317601499198,2cyclictest141rcu_preempt00:23:032
317601499191,8cyclictest3277034-21turbostat23:55:012
3176014991917,1cyclictest0-21swapper/222:00:462
3176014991917,1cyclictest0-21swapper/221:07:202
3176014991911,6cyclictest3232181-21cat21:45:162
3176014991911,6cyclictest3216765-21ntpq21:00:192
3176014991911,5cyclictest28650irq/123-ahci[0021:25:212
317601499190,1cyclictest0-21swapper/223:25:092
317601499190,1cyclictest0-21swapper/221:44:432
317601499190,1cyclictest0-21swapper/220:14:402
317601499190,18cyclictest0-21swapper/221:53:502
317601499190,18cyclictest0-21swapper/219:57:232
317601499190,18cyclictest0-21swapper/219:30:402
3176007991912,5cyclictest3288525-21sed00:25:171
317600799190,1cyclictest3056-21PK-Backend22:12:231
317600799190,1cyclictest3056-21PK-Backend20:12:191
317600799190,1cyclictest3056-21PK-Backend19:46:291
317600799190,1cyclictest3056-21PK-Backend19:14:061
317600799190,1cyclictest0-21swapper/122:48:531
317600799190,1cyclictest0-21swapper/120:46:461
317600399196,7cyclictest13-21ksoftirqd/023:23:320
317600399196,4cyclictest13-21ksoftirqd/022:53:410
317600399192,9cyclictest13-21ksoftirqd/022:35:580
3176003991917,1cyclictest0-21swapper/021:16:010
3176003991917,1cyclictest0-21swapper/020:52:360
3176003991917,1cyclictest0-21swapper/020:36:270
3176003991917,1cyclictest0-21swapper/020:32:070
3176003991915,3cyclictest161rcuc/022:55:280
3176003991912,6cyclictest3202850-21sed20:20:230
3176003991911,7cyclictest3276297-21ntp_states23:50:190
139099190,13rtkit-daemon3056-21PK-Backend19:56:371
3176017991816,1cyclictest0-21swapper/321:29:193
3176017991816,1cyclictest0-21swapper/321:07:083
3176017991812,5cyclictest3269076-21interrupts23:30:183
3176017991812,5cyclictest3265271-21expr23:20:123
3176017991811,6cyclictest3277035-21/usr/sbin/munin23:55:003
317601799181,15cyclictest3263768-21df23:15:133
317601499189,4cyclictest34-21ksoftirqd/223:23:062
317601499189,4cyclictest34-21ksoftirqd/219:54:142
317601499188,2cyclictest141rcu_preempt23:09:172
317601499187,4cyclictest34-21ksoftirqd/220:40:022
3176014991814,3cyclictest34-21ksoftirqd/219:25:002
3176014991811,5cyclictest3277767-21awk23:55:162
3176007991816,1cyclictest3056-21PK-Backend21:34:191
3176007991816,1cyclictest3056-21PK-Backend21:02:551
3176007991815,2cyclictest3056-21PK-Backend23:22:481
3176007991811,5cyclictest3208122-21cat20:35:191
317600399186,3cyclictest13-21ksoftirqd/021:01:010
317601799171,15cyclictest3216899-21/usr/sbin/munin21:00:193
317601499178,4cyclictest34-21ksoftirqd/220:58:302
317601499175,6cyclictest141rcu_preempt21:12:402
3176007991715,1cyclictest3056-21PK-Backend23:10:371
3176007991715,1cyclictest3056-21PK-Backend21:14:041
3176007991715,1cyclictest0-21swapper/121:22:421
3176007991715,1cyclictest0-21swapper/121:16:141
3176007991710,6cyclictest3180430-21sed19:20:001
3176003991710,6cyclictest3290192-21uname00:30:140
317601799169,5cyclictest3293066-21date00:40:013
317601799169,4cyclictest32896872sleep300:30:093
3176017991614,1cyclictest3056-21PK-Backend19:24:073
317601799161,11cyclictest82332-21update-manager20:59:313
317600799169,6cyclictest3286082-21sed00:20:011
3176007991614,1cyclictest0-21swapper/100:30:061
317601799153,11cyclictest3281395-21missed_timers00:05:213
317600799157,6cyclictest3263529-21sort23:15:111
3176007991513,1cyclictest0-21swapper/100:04:451
317601799140,13cyclictest3286228-21/usr/sbin/munin00:20:223
317600799122,9cyclictest3056-21PK-Backend23:35:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional