You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-09 - 20:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Fri Jan 09, 2026 12:44:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
340690722810,15sleep126-21ksoftirqd/107:05:271
340690722810,15sleep126-21ksoftirqd/107:05:261
349590121150,3sleep30-21swapper/311:20:013
197399980,9rtkit-daemon1972-21rtkit-daemon07:46:411
340755499831,45cyclictest3488513-21ls11:00:003
340724127839,32sleep30-21swapper/307:08:593
340724127839,32sleep30-21swapper/307:08:583
340716927340,28sleep00-21swapper/007:08:040
340716927340,28sleep00-21swapper/007:08:040
340755499682,41cyclictest3504400-21munin-run11:45:003
340755499681,42cyclictest0-21swapper/308:55:013
340755499671,41cyclictest0-21swapper/309:50:283
340755499671,41cyclictest0-21swapper/309:50:273
340755499661,48cyclictest3520334-21perf12:30:003
340755499652,33cyclictest3509663-21runrttasks12:00:003
340755499631,40cyclictest3436367-21expr08:30:133
340755499631,39cyclictest3457178-21latency_hist09:30:033
340697526344,13sleep20-21swapper/207:05:322
340697526344,13sleep20-21swapper/207:05:322
340755499621,32cyclictest3464095-21ls09:50:023
340755499621,31cyclictest3480016-21latency_hist10:35:013
3407547996244,6cyclictest34-21ksoftirqd/211:56:442
340755499617,34cyclictest34485612expr09:05:143
197399610,9rtkit-daemon1972-21rtkit-daemon11:41:402
340755499601,31cyclictest3417306-21cpuspeed_turbos07:35:133
197399600,9rtkit-daemon1972-21rtkit-daemon11:07:113
197399600,9rtkit-daemon1972-21rtkit-daemon11:07:113
197399600,9rtkit-daemon1972-21rtkit-daemon10:50:462
340755499591,37cyclictest3429532-21ls08:10:143
340755499591,33cyclictest3421808-21taskset07:46:353
340755499591,29cyclictest3455320-21sort09:25:023
197399590,9rtkit-daemon1972-21rtkit-daemon09:18:253
340755499581,31cyclictest3446688-21cat09:00:123
340755499581,30cyclictest3468095-21cpuspeed_turbos10:00:113
340755499581,30cyclictest3422534-21expr07:50:123
340755499581,29cyclictest3476506-21cat10:25:023
340755499581,29cyclictest3457699-21basename09:30:193
340755499581,29cyclictest3457699-21basename09:30:193
340755499581,28cyclictest3432663-21latency_hist08:20:013
340755499581,27cyclictest0-21swapper/312:20:213
197399580,9rtkit-daemon1972-21rtkit-daemon12:09:253
197399580,9rtkit-daemon1972-21rtkit-daemon11:52:302
197399580,9rtkit-daemon1972-21rtkit-daemon11:52:292
197399580,9rtkit-daemon1972-21rtkit-daemon09:05:582
197399580,8rtkit-daemon1972-21rtkit-daemon10:21:052
340755499572,27cyclictest3483402-21latency_hist10:45:023
340755499571,34cyclictest3478083-21taskset10:29:463
197399570,9rtkit-daemon1972-21rtkit-daemon09:38:203
197399570,9rtkit-daemon1972-21rtkit-daemon07:27:332
340755499561,37cyclictest3499382-21latency_hist11:30:033
340755499561,33cyclictest3425442-21chrt07:58:513
340755499561,31cyclictest3513886-21cat12:10:213
340755499561,31cyclictest3511350-21cron12:05:003
340755499561,31cyclictest3416052-21needreboot07:30:223
340755499561,31cyclictest3416052-21needreboot07:30:223
340755499561,30cyclictest3523809-21taskset12:37:163
340755499561,30cyclictest3523809-21taskset12:37:153
340755499561,30cyclictest3446176-21taskset08:58:273
340755499561,30cyclictest3438078-21cpuspeed_turbos08:35:113
340755499561,28cyclictest3522411-21cut12:35:023
340755499561,28cyclictest3497507-21ls11:25:013
197399560,8rtkit-daemon1972-21rtkit-daemon10:00:002
34933372550,3sleep23493340-21proc_pri11:10:272
340755499552,22cyclictest0-21swapper/311:30:293
340755499552,22cyclictest0-21swapper/311:30:283
340755499551,50cyclictest3428930-21taskset08:09:133
340755499551,49cyclictest3412654-21grep07:20:223
340755499551,48cyclictest3483775-21cstates10:45:143
3407554995512,38cyclictest3419362-21cat07:40:183
340755499542,29cyclictest3474660-21perf10:20:013
340755499542,26cyclictest3516445-21taskset12:18:533
340755499541,27cyclictest3427170-21taskset08:03:503
197399540,8rtkit-daemon1972-21rtkit-daemon10:40:222
34642552530,20sleep126-21ksoftirqd/109:50:131
34642552530,20sleep126-21ksoftirqd/109:50:131
340755499531,29cyclictest3471005-21taskset10:07:473
340755499531,27cyclictest3441033-21taskset08:44:023
340755499531,27cyclictest3441033-21taskset08:44:023
340755499531,26cyclictest3481087-21sed10:35:273
197399530,8rtkit-daemon1972-21rtkit-daemon09:19:232
34922802520,6chrt3492274-21perf11:10:010
34922802520,6chrt3492274-21perf11:10:000
340755499521,30cyclictest3502706-21latency_hist11:40:013
340755499521,27cyclictest3414788-21runrttasks07:26:033
340755499521,27cyclictest0-21swapper/310:50:303
3407554995210,20cyclictest3434661-21cpuspeed_turbos08:25:143
340755499512,28cyclictest3410563-21diskstats07:15:163
340755499511,46cyclictest0-21swapper/308:20:193
340755499511,29cyclictest0-21swapper/309:11:263
340755499511,28cyclictest0-21swapper/309:57:593
340755499511,27cyclictest3489678-21sed11:00:273
340755499511,27cyclictest3489678-21sed11:00:273
197399510,6rtkit-daemon1972-21rtkit-daemon08:04:042
197399510,44rtkit-daemon34-21ksoftirqd/209:30:212
197399510,44rtkit-daemon34-21ksoftirqd/209:30:212
340755499502,24cyclictest3472213-21ntp_states10:10:253
3407547994924,20cyclictest3515836-21ntp_states12:15:252
340755499481,25cyclictest3492862-21cstates11:10:133
3407540994830,4cyclictest34958982latency_hist11:20:010
3407554994721,22cyclictest3507398-21timerwakeupswit11:50:283
3407554994721,22cyclictest3507398-21timerwakeupswit11:50:273
340755499471,41cyclictest3460831-21cpuspeed_turbos09:40:153
340755499471,30cyclictest0-21swapper/311:45:263
340755499471,28cyclictest0-21swapper/307:12:543
340755499471,26cyclictest0-21swapper/308:49:463
197399470,40rtkit-daemon2753-21gmain08:13:422
34706212460,5sleep2331rcuc/210:05:392
3407540994634,9cyclictest13-21ksoftirqd/011:56:420
197399460,39rtkit-daemon2753-21gmain08:31:052
34872512450,5sleep20-21swapper/210:55:142
34404002440,7sleep126-21ksoftirqd/108:40:241
34404002440,7sleep126-21ksoftirqd/108:40:241
34294672440,7sleep126-21ksoftirqd/108:10:121
340754399438,11cyclictest141rcu_preempt09:18:171
3407540994326,12cyclictest3451500-21apt-get09:15:000
197399430,36rtkit-daemon927-21gmain09:27:172
34940332420,5sleep1251rcuc/111:15:001
34203882420,15sleep013-21ksoftirqd/007:45:000
3407540994215,11cyclictest3423813-21cron07:55:000
197399420,36rtkit-daemon1039-21gmain08:57:292
3407540994114,11cyclictest3445528-21grep08:55:210
3407540994112,12cyclictest1-21systemd12:10:020
3407547994030,5cyclictest34-21ksoftirqd/208:45:122
3407540994021,14cyclictest3493290-21ntp_states11:10:260
197399400,33rtkit-daemon0-21swapper/211:20:532
35104812390,6chrt0-21swapper/112:00:211
3407547993924,11cyclictest3522202-21dpkg12:35:012
3407547993924,11cyclictest3480761-21missed_timers10:35:202
3407547993920,15cyclictest3516796-21/usr/sbin/munin12:20:272
3407547993918,17cyclictest3413413-21unin-run07:25:012
3407543993922,5cyclictest26-21ksoftirqd/109:25:461
3407540993920,15cyclictest3461482-21ntp_states09:40:260
3407540993912,8cyclictest13-21ksoftirqd/012:16:000
340754099391,20cyclictest438-20systemd-journal12:05:000
3407540993911,8cyclictest13-21ksoftirqd/011:31:100
3407540993911,8cyclictest13-21ksoftirqd/011:31:090
34860392380,7chrt0-21swapper/010:50:240
3407547993816,18cyclictest3448080-21unin-run09:05:012
340754799381,15cyclictest34947472irqstats11:15:192
3407543993820,14cyclictest3426187-21df_inode08:00:141
3407543993819,5cyclictest26-21ksoftirqd/109:31:221
3407543993819,5cyclictest26-21ksoftirqd/109:31:211
340754099386,14cyclictest924-21NetworkManager08:09:570
340754099383,14cyclictest3501200-21grep11:35:100
3407540993821,12cyclictest3506770-21entropy11:50:170
3407540993821,12cyclictest3506770-21entropy11:50:160
3407540993811,11cyclictest13-21ksoftirqd/007:58:210
197399380,29rtkit-daemon1074-21lldpd08:50:562
340754799374,16cyclictest34-21ksoftirqd/210:05:002
3407547993727,6cyclictest34-21ksoftirqd/210:30:022
3407547993721,12cyclictest1-21systemd07:15:002
3407547993720,13cyclictest34-21ksoftirqd/211:40:012
3407547993720,12cyclictest438-20systemd-journal07:45:542
3407547993717,15cyclictest34-21ksoftirqd/212:05:002
340754799371,32cyclictest3460137-21taskset09:35:552
340754099379,11cyclictest13-21ksoftirqd/010:46:190
340754099377,5cyclictest141rcu_preempt07:15:020
3407540993723,10cyclictest3415100-21cron07:30:000
3407540993722,11cyclictest3483300-21cron10:45:000
3407540993721,11cyclictest3465695-21apt-get09:55:090
197399370,30rtkit-daemon0-21swapper/212:12:272
197399370,30rtkit-daemon0-21swapper/210:33:232
197399370,30rtkit-daemon0-21swapper/207:58:302
35231792360,11sleep03407534-21cyclictest12:35:210
35231792360,11sleep03407534-21cyclictest12:35:210
34298332360,7sleep0161rcuc/008:10:190
34238062360,6sleep14497-21PK-Backend07:55:011
3407547993630,4cyclictest0-21swapper/208:41:132
3407547993630,4cyclictest0-21swapper/208:41:122
3407547993620,5cyclictest34-21ksoftirqd/208:40:032
3407547993620,12cyclictest3505193-21sed11:45:192
3407547993620,12cyclictest3454484-21strings09:20:212
3407547993618,13cyclictest3491494-21ntpq11:05:242
3407547993618,13cyclictest3491494-21ntpq11:05:232
3407543993628,4cyclictest4497-21PK-Backend10:05:041
3407543993618,13cyclictest3479112-21ntpq10:30:241
340754399361,14cyclictest3464075-21latency_hist09:50:011
340754099369,10cyclictest13-21ksoftirqd/009:25:050
340754099368,5cyclictest141rcu_preempt10:35:280
340754099367,5cyclictest141rcu_preempt09:15:280
3407540993621,11cyclictest3490256-21dpkg11:05:010
3407540993621,11cyclictest3490256-21dpkg11:05:000
3407540993618,14cyclictest3519556-21ntp_states12:25:250
340754099361,14cyclictest3518456-21sed12:25:010
340754099361,10cyclictest4497-21PK-Backend11:20:260
3407540993610,12cyclictest141rcu_preempt10:15:130
197399360,29rtkit-daemon0-21swapper/212:35:172
197399360,29rtkit-daemon0-21swapper/212:35:172
197399360,29rtkit-daemon0-21swapper/210:19:142
197399360,29rtkit-daemon0-21swapper/208:15:452
34109302350,8sleep14497-21PK-Backend07:15:231
3407547993524,6cyclictest34-21ksoftirqd/211:35:002
3407547993524,6cyclictest34-21ksoftirqd/211:34:592
3407547993521,10cyclictest3484233-21grep10:45:212
3407547993519,12cyclictest3511869-21expr12:05:122
3407547993519,12cyclictest3463963-21cron09:50:002
3407547993518,13cyclictest3497724-21awk11:25:102
3407547993517,13cyclictest3410550-21df07:15:152
340754399355,11cyclictest26-21ksoftirqd/107:23:331
3407543993521,10cyclictest35123732chrt12:05:231
3407543993519,5cyclictest26-21ksoftirqd/107:40:141
3407543993519,12cyclictest3487748-21awk10:55:221
340754099359,9cyclictest0-21swapper/011:40:270
340754099358,8cyclictest13-21ksoftirqd/009:07:060
340754099358,11cyclictest141rcu_preempt09:30:150
340754099358,11cyclictest141rcu_preempt09:30:150
3407540993525,6cyclictest13-21ksoftirqd/007:35:010
3407540993525,6cyclictest13-21ksoftirqd/007:35:010
340754099352,11cyclictest3433108-21df08:20:140
3407540993521,10cyclictest3426542-21ntp_kernel_pll_08:00:230
340754099351,6cyclictest4497-21PK-Backend09:50:130
340754099351,6cyclictest4497-21PK-Backend09:50:130
340754099351,30cyclictest3453814-21apt09:20:120
197399350,28rtkit-daemon0-21swapper/211:03:182
197399350,28rtkit-daemon0-21swapper/211:03:182
197399350,28rtkit-daemon0-21swapper/209:51:132
197399350,28rtkit-daemon0-21swapper/209:51:132
197399350,28rtkit-daemon0-21swapper/208:20:052
197399350,28rtkit-daemon0-21swapper/207:30:572
197399350,28rtkit-daemon0-21swapper/207:30:572
197399350,26rtkit-daemon1972-21rtkit-daemon12:29:432
3407547993419,10cyclictest34-21ksoftirqd/210:15:002
340754399349,7cyclictest141rcu_preempt11:00:181
340754399349,7cyclictest141rcu_preempt11:00:181
3407543993418,5cyclictest26-21ksoftirqd/112:16:501
3407543993418,12cyclictest923-21dbus-daemon11:50:001
340754399341,4cyclictest4497-21PK-Backend11:31:031
340754399341,4cyclictest4497-21PK-Backend11:31:031
340754399341,14cyclictest4497-21PK-Backend10:54:341
340754399341,13cyclictest3435568-21awk08:25:311
340754099346,11cyclictest13-21ksoftirqd/008:29:120
340754099346,11cyclictest13-21ksoftirqd/008:19:490
340754099343,11cyclictest13-21ksoftirqd/008:41:260
340754099343,11cyclictest13-21ksoftirqd/008:41:260
3407540993419,11cyclictest3513848-21cut12:10:210
3407540993418,11cyclictest3438944-21timerwakeupswit08:35:260
340754099341,6cyclictest141rcu_preempt07:40:010
3407540993415,8cyclictest141rcu_preempt09:37:550
340754099341,14cyclictest3479355-21sort10:30:290
340754099341,13cyclictest3462655-21cpuspeed_turbos09:45:120
340754099341,11cyclictest4497-21PK-Backend07:15:130
340754099341,11cyclictest3412515-21sort07:20:200
197399340,28rtkit-daemon0-21swapper/207:41:382
197399340,27rtkit-daemon0-21swapper/207:36:542
3407547993323,6cyclictest34-21ksoftirqd/208:25:252
3407547993323,5cyclictest34-21ksoftirqd/208:10:022
3407547993319,10cyclictest3450385-21expr09:10:142
3407547993318,11cyclictest3461715-21sed09:40:292
3407543993326,5cyclictest0-21swapper/108:20:001
3407543993320,9cyclictest26-21ksoftirqd/111:50:141
3407543993320,9cyclictest26-21ksoftirqd/111:50:141
3407543993318,11cyclictest3496593-21grep11:20:211
3407543993317,12cyclictest3441975-21memory08:45:201
3407543993317,12cyclictest3428413-21python308:05:241
340754399331,4cyclictest4497-21PK-Backend11:25:071
340754399331,4cyclictest4497-21PK-Backend11:07:361
340754399331,4cyclictest4497-21PK-Backend11:07:361
340754399331,4cyclictest4497-21PK-Backend08:36:521
340754399331,29cyclictest4497-21PK-Backend08:34:221
340754399331,29cyclictest4497-21PK-Backend07:12:261
340754399331,28cyclictest4497-21PK-Backend12:20:091
340754399331,10cyclictest4497-21PK-Backend07:55:151
3407540993325,5cyclictest0-21swapper/008:45:290
3407540993318,11cyclictest3521547-21grep12:30:290
3407540993318,11cyclictest3498459-21ntpq11:25:240
3407540993318,11cyclictest3421247-21wc07:45:200
3407540993317,12cyclictest3505524-21python311:45:270
3407540993316,12cyclictest438-20systemd-journal10:05:000
340754099331,29cyclictest4497-21PK-Backend10:14:370
3407540993310,7cyclictest2069-21gnome-shell10:23:580
3407547993218,10cyclictest3423461-21seq07:50:352
3407543993225,4cyclictest4497-21PK-Backend10:25:041
3407543993224,5cyclictest4497-21PK-Backend09:05:391
3407543993222,6cyclictest3524083-21sh12:39:591
3407543993222,6cyclictest3524083-21sh12:39:591
3407543993218,9cyclictest3480928-21cut10:35:261
3407543993218,10cyclictest3508759-21sed11:55:231
3407543993218,10cyclictest3447553-21sed09:00:271
3407543993216,12cyclictest923-21dbus-daemon10:49:561
340754399321,4cyclictest4497-21PK-Backend12:33:181
340754399321,4cyclictest4497-21PK-Backend12:12:391
3407543993214,13cyclictest438-20systemd-journal09:55:031
340754399321,28cyclictest4497-21PK-Backend09:13:281
340754399321,28cyclictest4497-21PK-Backend08:23:261
340754399321,10cyclictest4497-21PK-Backend10:24:481
3407540993223,6cyclictest161rcuc/008:52:100
3407540993218,10cyclictest3446689-21cat09:00:120
340754099321,6cyclictest141rcu_preempt10:05:390
340754099321,4cyclictest0-21swapper/008:34:140
340754099321,28cyclictest4497-21PK-Backend10:59:000
340754399313,11cyclictest4497-21PK-Backend07:30:131
340754399313,11cyclictest4497-21PK-Backend07:30:121
3407543993117,10cyclictest3460043-21sed09:35:291
340754399311,5cyclictest4497-21PK-Backend07:29:561
340754399311,4cyclictest4497-21PK-Backend08:56:341
340754399311,4cyclictest0-21swapper/107:39:571
3407543993114,12cyclictest438-20systemd-journal11:17:551
340754399311,27cyclictest4497-21PK-Backend10:01:001
340754099311,11cyclictest3476765-21cat10:25:130
3407543993024,4cyclictest0-21swapper/110:19:071
3407543993020,7cyclictest3454479-21needreboot09:20:211
340754399301,6cyclictest4497-21PK-Backend11:36:451
3407543993016,10cyclictest3519800-21grep12:25:301
340754399301,4cyclictest4497-21PK-Backend11:43:071
340754399301,14cyclictest0-21swapper/110:12:071
3407543992922,4cyclictest4497-21PK-Backend10:41:201
340754399291,5cyclictest4497-21PK-Backend08:50:241
340754399281,9cyclictest26-21ksoftirqd/109:40:131
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional