You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-16 - 21:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Sat May 16, 2026 12:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
404351721150,3sleep10-21swapper/110:55:151
396495221050,2sleep00-21swapper/007:15:040
404649921010,3sleep10-21swapper/111:05:021
139099860,75rtkit-daemon0-21swapper/007:09:470
396228327839,14sleep20-21swapper/207:07:552
396217827138,27sleep10-21swapper/107:06:351
40369832660,3sleep20-21swapper/210:35:222
396232026442,15sleep30-21swapper/307:08:263
40329022530,4sleep10-21swapper/110:25:161
40278722510,2sleep10-21swapper/110:10:191
40744012470,4sleep04074390-21perf12:24:590
40744012470,4sleep04074390-21perf12:24:590
40184592430,2sleep10-21swapper/109:45:131
40112222360,3sleep00-21swapper/009:25:110
3962590992620,5cyclictest4026755-21basename10:10:002
396258599260,24cyclictest1389-21rtkit-daemon09:31:590
139099260,4rtkit-daemon1389-21rtkit-daemon08:37:431
139099250,4rtkit-daemon1389-21rtkit-daemon09:13:581
139099250,4rtkit-daemon1389-21rtkit-daemon07:50:081
139099250,3rtkit-daemon1389-21rtkit-daemon08:05:520
40699442240,4sleep03056-21PK-Backend12:10:160
3962590992413,9cyclictest3967233-21kernelversion07:20:172
396259099241,21cyclictest3997161-21/usr/sbin/munin08:45:152
40683782230,6sleep013-21ksoftirqd/012:05:190
39635072230,3chrt3056-21PK-Backend07:10:190
3962590992321,1cyclictest0-21swapper/211:13:312
3962590992321,1cyclictest0-21swapper/209:34:142
3962590992315,5cyclictest28650irq/123-ahci[0012:35:202
396259099231,20cyclictest0-21swapper/210:47:022
396258799230,21cyclictest4064414-21irqstats11:55:161
139099230,4rtkit-daemon1389-21rtkit-daemon09:07:141
139099230,4rtkit-daemon1389-21rtkit-daemon08:54:321
396259399220,20cyclictest3056-21PK-Backend09:23:533
396259399220,20cyclictest3056-21PK-Backend07:58:363
396259399220,20cyclictest1389-21rtkit-daemon11:41:153
3962590992220,1cyclictest0-21swapper/209:57:332
3962590992220,1cyclictest0-21swapper/209:53:292
3962590992220,1cyclictest0-21swapper/208:21:422
3962590992220,1cyclictest0-21swapper/207:31:392
3962590992214,6cyclictest28650irq/123-ahci[0010:55:222
3962590992213,6cyclictest28650irq/123-ahci[0011:25:202
396259099220,21cyclictest0-21swapper/211:08:102
396259099220,21cyclictest0-21swapper/210:10:052
3962587992220,1cyclictest0-21swapper/112:30:031
3962587992220,1cyclictest0-21swapper/111:52:121
3962587992220,1cyclictest0-21swapper/110:30:331
3962587992220,1cyclictest0-21swapper/109:40:371
3962587992219,2cyclictest3056-21PK-Backend12:16:581
3962587992219,2cyclictest3056-21PK-Backend08:49:131
396258799220,9cyclictest39672412tr07:20:171
396258799220,21cyclictest0-21swapper/107:42:481
396258799220,20cyclictest3056-21PK-Backend10:39:581
3962585992220,1cyclictest0-21swapper/009:53:090
3962585992214,6cyclictest4004576-21seq09:05:230
396258599221,16cyclictest4074653-21seq12:25:120
396258599220,20cyclictest3986496-21diskstats08:15:150
139099220,3rtkit-daemon1389-21rtkit-daemon09:19:221
139099220,3rtkit-daemon1389-21rtkit-daemon07:30:590
3962593992119,1cyclictest3056-21PK-Backend10:21:183
396259399210,20cyclictest0-21swapper/309:29:003
396259399210,20cyclictest0-21swapper/307:12:063
396259399210,19cyclictest3056-21PK-Backend08:56:153
3962590992119,1cyclictest0-21swapper/212:05:092
3962590992119,1cyclictest0-21swapper/210:33:162
3962590992119,1cyclictest0-21swapper/209:45:312
3962590992119,1cyclictest0-21swapper/209:39:212
3962590992119,1cyclictest0-21swapper/208:04:582
3962590992119,1cyclictest0-21swapper/207:53:102
3962590992119,1cyclictest0-21swapper/207:17:072
3962590992113,6cyclictest3994016-21awk08:35:182
3962590992112,7cyclictest3985255-21python308:10:212
396259099210,20cyclictest0-21swapper/210:40:402
396259099210,20cyclictest0-21swapper/210:04:432
396259099210,20cyclictest0-21swapper/208:44:122
396259099210,20cyclictest0-21swapper/207:57:022
3962587992119,1cyclictest0-21swapper/112:23:561
3962587992119,1cyclictest0-21swapper/112:23:561
3962587992119,1cyclictest0-21swapper/110:53:451
3962587992119,1cyclictest0-21swapper/109:50:041
3962587992119,1cyclictest0-21swapper/109:23:001
3962587992119,1cyclictest0-21swapper/108:55:091
3962587992119,1cyclictest0-21swapper/107:57:021
3962587992113,6cyclictest4055600-21expr11:30:131
3962587992113,6cyclictest4015754-21python309:35:211
396258799210,20cyclictest0-21swapper/108:07:301
396258799210,20cyclictest0-21swapper/107:32:491
396258599219,8cyclictest436-20systemd-journal10:45:010
3962585992113,6cyclictest4005756-21awk09:10:130
396258599210,19cyclictest4028468-21cron10:15:000
139099210,14rtkit-daemon3056-21PK-Backend12:07:561
139099210,14rtkit-daemon0-21swapper/112:00:241
3962593992018,1cyclictest0-21swapper/311:10:273
3962593992018,1cyclictest0-21swapper/310:50:183
3962593992018,1cyclictest0-21swapper/308:41:023
3962593992016,3cyclictest42-21ksoftirqd/308:10:013
396259399200,1cyclictest3056-21PK-Backend08:51:213
396259399200,1cyclictest0-21swapper/310:47:013
396259399200,19cyclictest0-21swapper/310:42:423
396259399200,19cyclictest0-21swapper/307:39:103
396259399200,18cyclictest3056-21PK-Backend09:09:203
396259399200,18cyclictest3056-21PK-Backend08:00:383
3962590992018,1cyclictest0-21swapper/211:01:482
3962590992018,1cyclictest0-21swapper/210:26:532
3962590992018,1cyclictest0-21swapper/210:21:342
3962590992018,1cyclictest0-21swapper/209:08:392
3962590992018,1cyclictest0-21swapper/209:02:442
3962590992012,6cyclictest4073783-21python312:20:222
3962590992012,6cyclictest4073783-21python312:20:212
3962590992012,6cyclictest3999234-21ntpq08:50:192
396259099200,1cyclictest0-21swapper/209:42:262
396259099200,1cyclictest0-21swapper/209:27:062
396259099200,1cyclictest0-21swapper/208:58:402
396259099200,19cyclictest3056-21PK-Backend08:31:552
396259099200,19cyclictest0-21swapper/212:18:412
396259099200,18cyclictest3991317-21cron08:30:002
3962587992018,1cyclictest0-21swapper/112:27:001
3962587992018,1cyclictest0-21swapper/111:14:531
3962587992018,1cyclictest0-21swapper/111:07:301
3962587992018,1cyclictest0-21swapper/110:15:311
3962587992018,1cyclictest0-21swapper/108:25:481
396258799200,1cyclictest0-21swapper/110:05:511
396258799200,19cyclictest3056-21PK-Backend09:29:091
396258799200,19cyclictest3056-21PK-Backend08:31:021
396258799200,19cyclictest0-21swapper/111:29:271
396258799200,19cyclictest0-21swapper/108:16:211
396258799200,19cyclictest0-21swapper/108:00:061
3962585992018,1cyclictest0-21swapper/012:18:150
3962585992018,1cyclictest0-21swapper/008:56:240
3962585992017,2cyclictest0-21swapper/008:20:150
3962585992011,7cyclictest4079587-21apt-get12:40:010
3962585992011,7cyclictest4065418-21idleruntime-cro12:00:010
396258599200,1cyclictest3056-21PK-Backend11:20:360
396258599200,1cyclictest3056-21PK-Backend11:20:360
396258599200,1cyclictest3056-21PK-Backend10:05:540
396258599200,1cyclictest0-21swapper/008:12:040
396258599200,19cyclictest0-21swapper/011:11:110
396258599200,19cyclictest0-21swapper/010:25:130
396258599200,19cyclictest0-21swapper/008:26:160
396258599200,19cyclictest0-21swapper/007:41:430
3962593991917,1cyclictest3056-21PK-Backend11:35:423
3962593991917,1cyclictest3056-21PK-Backend08:10:193
3962593991917,1cyclictest0-21swapper/312:37:543
3962593991917,1cyclictest0-21swapper/310:27:403
3962593991917,1cyclictest0-21swapper/308:45:413
3962593991916,2cyclictest0-21swapper/312:00:153
3962593991914,3cyclictest4036343-21hddtemp_smartct10:35:163
3962593991910,7cyclictest4068026-21df12:05:143
396259399190,1cyclictest3056-21PK-Backend12:22:453
396259399190,1cyclictest3056-21PK-Backend12:22:453
396259399190,1cyclictest3056-21PK-Backend11:16:493
396259399190,1cyclictest3056-21PK-Backend09:33:053
396259399190,1cyclictest0-21swapper/312:33:483
396259399190,1cyclictest0-21swapper/310:33:023
396259399190,18cyclictest0-21swapper/311:22:093
396259399190,18cyclictest0-21swapper/311:22:093
396259099193,14cyclictest4041618-21/usr/sbin/munin10:50:132
3962590991917,1cyclictest0-21swapper/212:28:412
3962590991917,1cyclictest0-21swapper/212:14:362
3962590991917,1cyclictest0-21swapper/212:00:022
3962590991917,1cyclictest0-21swapper/209:18:542
3962590991914,3cyclictest4058473-21latency_hist11:40:002
3962590991912,5cyclictest3972811-21smartctl07:35:232
396259099190,1cyclictest3056-21PK-Backend07:13:072
396259099190,18cyclictest0-21swapper/208:16:422
3962587991917,1cyclictest0-21swapper/111:45:011
3962587991917,1cyclictest0-21swapper/108:23:581
3962587991917,1cyclictest0-21swapper/107:48:171
3962587991917,1cyclictest0-21swapper/107:28:501
3962587991917,1cyclictest0-21swapper/107:28:501
3962587991917,1cyclictest0-21swapper/107:19:571
3962587991917,1cyclictest0-21swapper/107:13:141
3962587991912,5cyclictest4078778-21tr12:35:171
396258799190,1cyclictest3056-21PK-Backend08:43:321
396258799190,1cyclictest0-21swapper/109:02:051
396258599192,15cyclictest4041139-21latency_hist10:50:000
3962585991917,1cyclictest0-21swapper/010:50:140
3962585991917,1cyclictest0-21swapper/009:16:050
3962585991917,1cyclictest0-21swapper/008:31:310
3962585991917,1cyclictest0-21swapper/007:46:080
3962585991917,1cyclictest0-21swapper/007:27:570
3962585991917,1cyclictest0-21swapper/007:27:560
3962585991912,6cyclictest4061363-21sed11:45:200
3962585991912,5cyclictest3966975-21cat07:20:120
3962585991912,5cyclictest13-21ksoftirqd/008:35:220
3962585991912,5cyclictest1095-21runrttasks11:15:180
3962585991911,6cyclictest4029480-21ntpq10:15:180
396258599190,1cyclictest3056-21PK-Backend10:35:420
396258599190,1cyclictest0-21swapper/009:44:450
396258599190,1cyclictest0-21swapper/008:50:300
396258599190,1cyclictest0-21swapper/008:49:290
396258599190,18cyclictest0-21swapper/012:03:090
396258599190,18cyclictest0-21swapper/010:20:540
396258599190,18cyclictest0-21swapper/009:39:380
396258599190,18cyclictest0-21swapper/009:22:270
39769052180,4sleep33056-21PK-Backend07:49:553
3962593991816,1cyclictest3056-21PK-Backend09:19:483
3962593991816,1cyclictest3056-21PK-Backend08:26:543
3962593991816,1cyclictest0-21swapper/312:16:513
3962593991815,2cyclictest3056-21PK-Backend11:30:353
3962593991813,4cyclictest3975192-21dump-pmu-power07:45:003
3962593991813,3cyclictest4046606-21apt11:05:123
3962593991812,5cyclictest4021073-21ls09:50:213
3962593991812,4cyclictest1095-21runrttasks11:25:223
3962593991811,5cyclictest436-20systemd-journal11:02:543
396259399180,1cyclictest3056-21PK-Backend09:02:103
3962590991816,1cyclictest0-21swapper/209:12:462
3962590991816,1cyclictest0-21swapper/207:47:422
3962590991816,1cyclictest0-21swapper/207:44:042
3962590991815,2cyclictest0-21swapper/211:31:222
3962590991814,3cyclictest978-21snmpd12:33:292
3962590991814,3cyclictest1095-21runrttasks11:56:082
3962587991816,1cyclictest0-21swapper/108:11:341
3962587991814,2cyclictest4058467-21sh11:40:001
3962587991811,6cyclictest4014353-21cron09:35:001
396258799180,1cyclictest0-21swapper/110:21:541
396258599184,13cyclictest1095-21runrttasks10:00:570
396258599182,12cyclictest0-21swapper/009:00:120
3962585991816,1cyclictest0-21swapper/011:30:370
3962585991816,1cyclictest0-21swapper/011:02:460
3962585991816,1cyclictest0-21swapper/007:53:380
3962585991813,4cyclictest13-21ksoftirqd/010:30:180
3962585991813,3cyclictest4077006-21missed_timers12:30:180
3962585991811,6cyclictest3981067-21which08:00:160
3962585991810,6cyclictest4057767-21awk11:35:210
396258599180,1cyclictest0-21swapper/008:42:120
3962593991715,1cyclictest3056-21PK-Backend12:25:503
3962593991715,1cyclictest3056-21PK-Backend10:59:453
3962593991715,1cyclictest3056-21PK-Backend08:20:543
3962593991715,1cyclictest0-21swapper/310:09:373
3962593991710,5cyclictest3977393-21tr07:50:123
396259399170,15cyclictest4015359-21/usr/sbin/munin09:35:183
396259399170,15cyclictest3965746-21ntp_states07:15:203
396259099179,6cyclictest4052172-21df_inode11:20:142
396259099179,6cyclictest4052172-21df_inode11:20:142
3962590991713,3cyclictest3982539-21tr08:05:132
3962590991712,3cyclictest978-21snmpd11:45:592
3962587991715,1cyclictest3056-21PK-Backend12:10:501
3962587991715,1cyclictest3056-21PK-Backend07:38:361
3962587991715,1cyclictest0-21swapper/111:19:121
3962585991715,1cyclictest0-21swapper/007:55:410
3962585991712,3cyclictest1095-21runrttasks10:55:190
3962585991710,5cyclictest4060363-21ls11:45:000
3962585991710,5cyclictest4022813-21cat09:55:190
3962585991710,5cyclictest13-21ksoftirqd/009:45:230
396258599170,15cyclictest3972514-21sed07:35:190
3962593991614,1cyclictest3056-21PK-Backend10:14:573
3962593991612,2cyclictest4069398-21apt12:10:113
396259399160,3cyclictest3986124-21apt08:15:133
396259399160,14cyclictest4062100-21sed11:50:003
396259099168,6cyclictest4009879-21awk09:20:192
3962590991611,3cyclictest4059177-21if_enp2s011:40:162
3962590991611,3cyclictest3968643-21tr07:25:132
3962590991611,3cyclictest3968643-21tr07:25:122
396258799169,5cyclictest4022092-21cat09:55:121
3962587991612,3cyclictest4023640-21cpuspeed10:00:141
3962585991610,5cyclictest4063372-21seq11:51:230
396258599160,15cyclictest0-21swapper/011:27:330
3962593991513,1cyclictest3056-21PK-Backend07:27:133
3962593991513,1cyclictest3056-21PK-Backend07:27:123
3962593991511,3cyclictest4063911-21/usr/sbin/munin11:55:153
3962593991510,4cyclictest4024994-21sshd10:04:553
396259399150,2cyclictest39944142sleep308:35:313
3962590991513,1cyclictest0-21swapper/211:16:502
3962590991513,1cyclictest0-21swapper/210:19:312
396258799158,5cyclictest4039735-21cat10:45:121
3962587991513,1cyclictest0-21swapper/111:40:551
3962587991510,4cyclictest4039120-21sshd10:44:531
396259399149,4cyclictest3056-21PK-Backend09:43:393
396259399149,3cyclictest40287552sleep310:15:113
3962593991410,3cyclictest3056-21PK-Backend11:50:223
396259099147,5cyclictest4062182-21acpi11:50:122
396258799149,3cyclictest4052399-21missed_timers11:20:181
396258799149,3cyclictest4052399-21missed_timers11:20:171
3962585991410,3cyclictest4046606-21apt11:05:120
396259399139,3cyclictest3056-21PK-Backend09:57:583
396259399139,3cyclictest1095-21runrttasks08:31:573
3962593991310,2cyclictest3056-21PK-Backend09:46:163
396259399130,1cyclictest3056-21PK-Backend09:14:283
396259399130,11cyclictest42-21ksoftirqd/307:30:133
3962593991210,1cyclictest3056-21PK-Backend07:22:063
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional