You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-02 - 17:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Fri Jan 02, 2026 12:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3466721212064,52sleep30-21swapper/307:07:373
3466630210188,7sleep20-21swapper/207:06:272
346658628654,10sleep10-21swapper/107:05:541
346660527855,6sleep00-21swapper/007:06:090
3467033992511,9cyclictest3542608-21taskset08:18:303
346703399250,20cyclictest3483548-21taskset07:20:183
3467033992415,6cyclictest3626672-21turbostat09:40:003
34794442231,6sleep03479470-21munin-run07:20:000
34794442231,6sleep03479470-21munin-run07:20:000
3467033992320,1cyclictest0-21swapper/310:35:003
3467033992219,1cyclictest0-21swapper/309:30:173
346703399217,7cyclictest3579353-21taskset08:53:293
3467033992118,2cyclictest0-21swapper/311:40:163
3467033992118,1cyclictest0-21swapper/312:25:153
3467033992115,3cyclictest3816128-21turbostat12:40:003
3467033992017,1cyclictest0-21swapper/310:25:133
3467033992016,2cyclictest0-21swapper/309:05:003
346703399195,12cyclictest0-21swapper/311:20:193
346703399191,8cyclictest3532244-21turbostat08:10:003
3467033991917,1cyclictest0-21swapper/307:45:153
3467033991916,1cyclictest0-21swapper/309:50:013
3467033991916,1cyclictest0-21swapper/308:35:153
3467033991916,1cyclictest0-21swapper/307:35:113
3467033991915,2cyclictest0-21swapper/311:05:013
3467033991915,2cyclictest0-21swapper/310:50:203
3467033991915,2cyclictest0-21swapper/307:50:163
3467033991915,2cyclictest0-21swapper/307:10:173
37257732181,13sleep13725980-21grep11:10:281
35712902181,5sleep13572998-21grep08:45:181
3467033991815,1cyclictest0-21swapper/310:40:163
3467033991815,1cyclictest0-21swapper/308:25:163
3467033991815,1cyclictest0-21swapper/307:29:593
3467033991814,2cyclictest0-21swapper/311:45:193
3467033991813,2cyclictest3768367-21turbostat11:54:593
346703399181,14cyclictest3490680-21taskset07:30:163
346703399181,14cyclictest0-21swapper/307:40:483
346703399181,11cyclictest3593087-21taskset09:05:153
35003282170,4sleep00-21swapper/007:39:240
346703399175,8cyclictest3710413-21taskset10:56:023
346703399172,10cyclictest3695284-21taskset10:45:113
346703399171,9cyclictest3722782-21taskset11:10:173
3467033991714,2cyclictest0-21swapper/309:20:223
3467033991714,2cyclictest0-21swapper/309:20:223
3467033991714,1cyclictest0-21swapper/312:05:183
3467033991714,1cyclictest0-21swapper/311:05:163
3467033991713,1cyclictest0-21swapper/310:20:133
3467033991712,2cyclictest0-21swapper/310:00:013
346703399171,13cyclictest0-21swapper/310:15:213
346703399171,13cyclictest0-21swapper/309:15:123
346703399171,12cyclictest0-21swapper/308:40:113
346703399171,11cyclictest3547991-21taskset08:24:553
346703399171,10cyclictest3650790-21taskset10:00:153
346703399171,10cyclictest3599215-21taskset09:10:163
37889342161,10sleep23788975-21ntpq12:10:212
37107292161,11sleep03710789-21sleep11:00:010
35116032161,10sleep23511724-21grep07:50:122
346703399167,6cyclictest3631704-21taskset09:42:553
346703399161,9cyclictest3805894-21taskset12:30:013
346703399161,9cyclictest3795950-21taskset12:20:153
346703399161,9cyclictest3774182-21taskset12:00:073
346703399161,9cyclictest3579721-21taskset08:55:013
3467033991614,1cyclictest0-21swapper/312:15:133
3467033991613,1cyclictest0-21swapper/312:10:283
346703399161,13cyclictest0-21swapper/311:30:233
346703399161,12cyclictest0-21swapper/311:55:203
346703399161,11cyclictest3726870-21chrt11:15:143
346703399161,11cyclictest0-21swapper/311:35:133
346703399160,13cyclictest0-21swapper/308:30:203
346703399160,13cyclictest0-21swapper/308:10:093
346703399160,13cyclictest0-21swapper/307:55:263
346703399160,10cyclictest3641696-21taskset09:50:213
346703399160,10cyclictest3526972-21taskset08:04:393
37151412151,9sleep23715265-21systemctl11:00:232
37005182151,10sleep23700598-21tr10:50:132
346703399155,6cyclictest3658260-21taskset10:08:283
346703399151,9cyclictest3742125-21taskset11:29:003
346703399151,9cyclictest3663573-21taskset10:14:293
3467033991513,1cyclictest0-21swapper/310:35:183
346703399150,12cyclictest0-21swapper/309:25:213
36897692141,9sleep03689791-21/usr/sbin/munin10:40:010
36862152141,4sleep23687913-21tr10:35:182
36432002141,9sleep23643332-21head09:55:142
35432952148,3sleep23543389-21fschecks_time08:20:132
346703399144,6cyclictest3479145-21taskset07:16:173
346703399144,6cyclictest3479145-21taskset07:16:173
346703399140,9cyclictest3569260-21taskset08:45:113
37760882131,2sleep13777837-21kthreadcore12:00:181
37415222137,2sleep1321rcuc/111:25:271
36681862131,9sleep03668257-21find10:15:280
36638772131,6sleep10-21swapper/110:15:121
36442652131,7sleep13645959-21kthreadcore09:55:171
35377722131,8sleep13537779-21cpu08:15:161
34881132131,8sleep13488759-21missed_timers07:25:181
37742102121,7sleep03774440-21cat12:00:130
37474662121,7sleep13747688-21cut11:35:011
36151752121,8sleep03615219-21/usr/sbin/munin09:25:210
35549782121,3sleep23556693-21cut08:30:162
35014552121,7sleep03503070-21cut07:40:160
34810922121,7sleep13482783-21kthreadcore07:20:171
37620842111,5sleep03762126-21ntpq11:45:220
37545012111,6sleep03756250-21kthreadcore11:40:170
36887532111,5sleep13688799-21ntpq10:35:211
36417502111,5sleep03641809-21cut09:50:230
36290532115,3sleep13630674-21latency09:40:171
36275252111,6sleep00-21swapper/009:40:160
35427282111,5sleep03542800-21munin-run08:20:000
35103382111,6sleep23510392-21ld07:45:232
37852612100,1sleep00-21swapper/012:10:150
37847052101,5sleep13784740-21basename12:10:001
37797172100,6sleep20-21swapper/212:05:122
37624342101,6sleep20-21swapper/211:45:282
37526782101,4sleep03752861-21cat11:40:010
37483092104,2sleep23749629-21cut11:35:182
37168402101,5sleep20-21swapper/211:05:162
37054702101,4sleep11-21systemd10:55:011
36952622101,4sleep23695281-21apt-get10:45:122
36901232101,4sleep03690336-21df_inode10:40:130
36791232100,1sleep20-21swapper/210:27:312
36521432100,1sleep10-21swapper/110:00:181
36415572101,4sleep13641646-21ntpq09:50:201
36321542101,4sleep03632410-21df09:45:130
36216842101,4sleep03621937-21df09:35:110
36104242101,4sleep23610609-21users09:20:262
36104242101,4sleep23610609-21users09:20:252
35960122100,6sleep13597577-21kthreadcore09:10:151
35953932101,4sleep03595412-21apt-get09:10:100
35732172101,4sleep03573266-21ntpq08:45:210
35482582101,4sleep23548276-21apt-get08:25:112
35430362101,4sleep13543294-21df_inode08:20:131
35377882101,4sleep23538037-21df_inode08:15:172
35322342101,5sleep23532333-21ls08:10:002
35102852101,4sleep13510336-21ntpq07:45:221
34920002101,5sleep10-21swapper/107:30:191
34783452101,4sleep13478425-21ntpq07:15:201
34783452101,4sleep13478425-21ntpq07:15:201
3790851290,2sleep20-21swapper/212:15:182
3790204291,3sleep13790223-21apt-get12:15:131
3778268290,1sleep20-21swapper/212:00:232
3773905291,4sleep01-21systemd12:00:010
3756805291,3sleep20-21swapper/211:40:192
3747727291,3sleep10-21swapper/111:35:131
3737187291,3sleep23737205-21apt-get11:25:122
3680144291,4sleep23680800-21kthreadcore10:30:172
3674388290,1sleep10-21swapper/110:25:111
3659204291,4sleep13659990-21grep10:10:161
3619901291,4sleep03620268-21sed09:30:180
3616473290,2sleep20-21swapper/209:30:122
3612601292,4sleep13613273-21kthreadcore09:25:181
3610007291,4sleep03610055-21ntpq09:20:200
3610007291,4sleep03610055-21ntpq09:20:200
3599723291,4sleep23599789-21sed09:10:242
3594107291,3sleep13594147-21ntpq09:05:181
3588581291,4sleep23588808-21grep09:00:182
3553493291,3sleep03553512-21apt-get08:30:100
3532493290,4sleep03532512-21apt-get08:10:100
3501050291,4sleep13501083-21cut07:40:131
3500398291,4sleep13500389-31find07:39:251
3499684291,4sleep23499721-21switchtime07:35:252
3486398291,4sleep03488112-21tr07:25:170
3469193291,4sleep03470956-21cut07:10:180
3809880281,3sleep23809929-21ntpq12:30:212
3806460281,3sleep13806917-21kthreadcore12:30:161
3788842280,4sleep10-21swapper/112:10:201
3773641280,1sleep20-21swapper/211:56:422
3768356281,3sleep10-21swapper/111:55:011
3763382280,1sleep00-21swapper/011:50:110
3747452280,1sleep20-21swapper/211:34:542
3712532281,3sleep03714285-21tr11:00:160
3711300281,4sleep13711468-21gltestperf11:00:141
3684534281,3sleep13684547-21latency_hist10:35:011
3678271281,3sleep03678319-21ntpq10:25:180
3667853281,2sleep20-21swapper/210:15:222
3658249280,1sleep00-21swapper/010:08:220
3652009281,4sleep03652142-21netstat10:00:170
3641647281,3sleep23641694-21ntpq09:50:212
3601192281,3sleep03601332-21tr09:15:150
3595018280,1sleep20-21swapper/209:08:362
3580282281,3sleep23580339-21kthreadcore08:55:172
3578787281,3sleep03578858-21find08:50:260
3568429281,3sleep23568614-21users08:40:262
3568294280,1sleep10-21swapper/108:40:251
3563704281,3sleep03563736-21munin-run08:40:000
3562673280,4sleep23562718-21ntpq08:35:172
3558724280,1sleep10-21swapper/108:35:081
3531545280,5sleep10-21swapper/108:05:261
3522048281,2sleep23522067-21apt-get08:00:102
3500765280,4sleep20-21swapper/207:40:062
3472272281,3sleep20-21swapper/207:15:012
3472272281,3sleep20-21swapper/207:15:002
3816117270,3sleep03816363-21date12:40:010
3805601270,1sleep00-21swapper/012:29:260
3796046271,3sleep03797554-21kthreadcore12:20:170
3795718271,3sleep10-21swapper/112:20:141
3784019270,1sleep00-21swapper/012:05:260
3767375270,2sleep20-21swapper/211:50:212
3753250270,3sleep13753426-21grep11:40:141
3742793271,3sleep03742988-21idleruntime11:30:150
3741856270,1sleep00-21swapper/011:25:340
3731338270,1sleep00-21swapper/011:15:520
3726716270,2sleep10-21swapper/111:15:131
3726401271,3sleep03726402-21sshd11:14:230
3716268270,2sleep10-21swapper/111:05:111
3710560270,1sleep20-21swapper/210:57:542
3694999270,1sleep10-21swapper/110:44:591
3670655271,2sleep03672380-21kthreadcore10:20:170
3669447270,4sleep10-21swapper/110:20:141
3669157270,2sleep20-21swapper/210:20:112
3653430270,2sleep20-21swapper/210:05:122
3649134271,2sleep20-21swapper/210:00:152
3647848270,1sleep00-21swapper/009:59:150
3631541270,1sleep20-21swapper/209:40:512
3616451270,2sleep10-21swapper/109:30:111
3586818271,3sleep13588580-21tr09:00:171
3574495271,3sleep23574530-21cpu08:50:122
3554062271,3sleep10-21swapper/108:30:151
3553168270,1sleep10-21swapper/108:29:151
3543560271,3sleep03543939-21tr08:20:160
3536509270,2sleep10-21swapper/108:10:181
3532772270,1sleep20-21swapper/208:10:122
3531276271,3sleep03527260-21/usr/sbin/munin08:05:210
3523862271,2sleep00-21swapper/008:00:160
3521710270,1sleep20-21swapper/207:59:182
3518654271,3sleep13520440-21grep07:55:161
3484937270,2sleep23484956-21apt-get07:25:112
3480016271,3sleep20-21swapper/207:20:142
3815865260,3sleep10-21swapper/112:36:491
3811119260,1sleep20-21swapper/212:35:102
3805428260,1sleep20-21swapper/212:27:162
3795435260,1sleep20-21swapper/212:20:072
3768900260,1sleep10-21swapper/111:55:131
3763033260,1sleep10-21swapper/111:48:551
3735761260,1sleep20-21swapper/211:20:192
3726738261,2sleep20-21swapper/211:15:142
3722044261,2sleep20-21swapper/211:10:172
3721156260,4sleep00-21swapper/011:09:140
3710718260,4sleep10-21swapper/110:59:541
3700105261,2sleep10-21swapper/110:48:231
3636833260,4sleep10-21swapper/109:46:351
3635880260,2sleep23632131-21/usr/sbin/munin09:45:202
3626531260,1sleep10-21swapper/109:38:411
3625747260,2sleep20-21swapper/209:35:222
3616190260,2sleep20-21swapper/209:30:002
3601333261,2sleep10-21swapper/109:15:161
3600952261,2sleep20-21swapper/209:15:132
3590143260,1sleep00-21swapper/009:05:010
3585518260,3sleep03586817-21grep09:00:160
3579131260,1sleep10-21swapper/108:50:391
3573000260,2sleep23573165-21strings08:45:192
3565238261,2sleep00-21swapper/008:40:160
3553049260,1sleep00-21swapper/008:27:420
3525615261,2sleep13525947-21strings08:00:161
3516791261,2sleep00-21swapper/007:55:110
3511581260,1sleep10-21swapper/107:50:111
3511265260,4sleep00-21swapper/007:49:180
3490750260,2sleep00-21swapper/007:30:180
3480294261,2sleep00-21swapper/007:20:160
3467503260,2sleep10-21swapper/107:10:141
3467202260,1sleep20-21swapper/207:10:082
3805909250,2sleep00-21swapper/012:30:110
3804761250,2sleep10-21swapper/112:25:231
3795060250,3sleep00-21swapper/012:18:480
3731975250,1sleep00-21swapper/011:20:130
3679871251,1sleep00-21swapper/010:30:150
3659991250,2sleep23661691-21kthreadcore10:10:172
3658642250,2sleep00-21swapper/010:10:110
3608915250,1sleep10-21swapper/109:20:171
3608915250,1sleep10-21swapper/109:20:171
3514674250,1sleep00-21swapper/007:50:170
3494234250,1sleep20-21swapper/207:30:242
3732232240,1sleep10-21swapper/111:20:131
3700786240,1sleep00-21swapper/010:50:150
3695052240,1sleep00-21swapper/010:45:020
3690050240,1sleep20-21swapper/210:40:112
3653731240,1sleep10-21swapper/110:05:151
3580085240,1sleep10-21swapper/108:55:161
3580017240,1sleep00-21swapper/008:55:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional