You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-18 - 21:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Thu Dec 18, 2025 12:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
34025612990,7sleep04497-21PK-Backend11:45:150
197399790,68rtkit-daemon0-21swapper/007:07:370
33058212760,4sleep30-21swapper/307:10:173
330492927350,15sleep30-21swapper/307:10:013
330491927354,14sleep10-21swapper/107:09:541
197399680,9rtkit-daemon1972-21rtkit-daemon11:58:420
197399670,9rtkit-daemon1972-21rtkit-daemon09:36:590
197399660,8rtkit-daemon1972-21rtkit-daemon10:18:110
330428226442,15sleep20-21swapper/207:05:212
330507199632,34cyclictest3386040-40lsb_release10:56:062
33621352620,17sleep013-21ksoftirqd/009:50:130
197399590,9rtkit-daemon1972-21rtkit-daemon10:02:250
197399590,9rtkit-daemon1972-21rtkit-daemon08:18:300
197399590,7rtkit-daemon1972-21rtkit-daemon12:32:270
330507199582,27cyclictest3345693-21munin-run09:05:012
197399580,9rtkit-daemon1972-21rtkit-daemon11:27:210
197399580,9rtkit-daemon1972-21rtkit-daemon09:59:150
197399580,9rtkit-daemon1972-21rtkit-daemon09:59:150
197399570,9rtkit-daemon1972-21rtkit-daemon12:11:350
197399570,9rtkit-daemon1972-21rtkit-daemon11:51:040
197399570,9rtkit-daemon1972-21rtkit-daemon10:28:370
197399570,9rtkit-daemon1972-21rtkit-daemon09:44:460
197399570,9rtkit-daemon1972-21rtkit-daemon07:56:160
197399570,8rtkit-daemon1972-21rtkit-daemon09:32:410
197399570,8rtkit-daemon1972-21rtkit-daemon08:08:320
197399570,8rtkit-daemon1972-21rtkit-daemon07:21:160
197399560,9rtkit-daemon1972-21rtkit-daemon12:06:400
197399560,9rtkit-daemon1972-21rtkit-daemon12:01:510
197399560,9rtkit-daemon1972-21rtkit-daemon09:47:140
197399560,9rtkit-daemon1972-21rtkit-daemon09:28:060
197399560,9rtkit-daemon1972-21rtkit-daemon09:21:170
197399560,9rtkit-daemon1972-21rtkit-daemon09:07:570
197399560,9rtkit-daemon1972-21rtkit-daemon08:30:510
197399560,9rtkit-daemon1972-21rtkit-daemon07:52:200
197399560,9rtkit-daemon1972-21rtkit-daemon07:52:200
197399560,8rtkit-daemon1972-21rtkit-daemon12:37:110
197399560,8rtkit-daemon1972-21rtkit-daemon08:42:340
197399560,8rtkit-daemon1972-21rtkit-daemon07:42:380
197399560,8rtkit-daemon1972-21rtkit-daemon07:33:500
3305071995520,29cyclictest438-20systemd-journal10:24:102
3305071995520,29cyclictest438-20systemd-journal10:24:102
3305070995521,7cyclictest3380896-21perf10:45:001
3305070995519,8cyclictest3319727-21ls07:50:141
3305070995519,8cyclictest3319727-21ls07:50:131
197399550,9rtkit-daemon1972-21rtkit-daemon10:15:381
197399550,8rtkit-daemon1972-21rtkit-daemon11:40:040
197399550,8rtkit-daemon1972-21rtkit-daemon11:39:230
197399550,8rtkit-daemon1972-21rtkit-daemon10:10:050
197399550,8rtkit-daemon1972-21rtkit-daemon09:16:410
197399550,8rtkit-daemon1972-21rtkit-daemon09:11:420
197399550,8rtkit-daemon1972-21rtkit-daemon08:50:040
197399550,8rtkit-daemon1972-21rtkit-daemon08:38:270
197399550,8rtkit-daemon1972-21rtkit-daemon08:21:350
197399550,8rtkit-daemon1972-21rtkit-daemon08:21:340
197399550,8rtkit-daemon1972-21rtkit-daemon07:29:290
330507199542,27cyclictest3414727-21diskstats12:20:182
3305071995418,32cyclictest1-21systemd10:15:012
3305070995416,12cyclictest3385243-21df10:55:141
197399540,8rtkit-daemon1972-21rtkit-daemon12:23:560
197399540,8rtkit-daemon1972-21rtkit-daemon08:27:060
197399540,8rtkit-daemon1972-21rtkit-daemon08:12:200
197399540,8rtkit-daemon1972-21rtkit-daemon07:12:340
3305071995320,28cyclictest3395668-21df_inode11:25:162
330507199522,27cyclictest3363286-21munin-run09:55:002
197399520,8rtkit-daemon1972-21rtkit-daemon12:24:491
197399520,8rtkit-daemon1972-21rtkit-daemon10:30:501
197399520,8rtkit-daemon1972-21rtkit-daemon08:58:331
3305076995126,20cyclictest3339384-21diskstats08:45:163
3305076995126,20cyclictest3339384-21diskstats08:45:153
330507199512,27cyclictest3309601-21diskstats07:20:162
330507199512,26cyclictest3387047-21hddtemp_smartct11:00:192
330507199511,27cyclictest3383083-21seq10:50:142
3305066995126,8cyclictest13-21ksoftirqd/010:20:220
3305066995126,8cyclictest13-21ksoftirqd/010:20:210
3305066995110,38cyclictest1972-21rtkit-daemon12:25:190
3305071995019,26cyclictest3377222-21date10:35:012
3305071995018,27cyclictest3335722-21grep08:35:152
3305071995017,28cyclictest3326950-21df08:10:152
330506699500,47cyclictest1972-21rtkit-daemon07:17:480
3305071994918,26cyclictest3381903-21ntpq10:45:252
3305071994918,26cyclictest3323039-21cut08:00:022
330507199491,29cyclictest1054-21snmpd12:25:142
33183192480,8sleep10-21swapper/107:45:201
330507199482,26cyclictest3403303-21users11:45:322
3305071994817,26cyclictest3326090-21seq08:07:152
3305071994815,28cyclictest3398485-21sh11:35:002
197399480,35rtkit-daemon0-21swapper/311:10:033
330507199473,39cyclictest3353658-21wc09:25:262
3305071994719,24cyclictest3319385-21cron07:54:552
3305071994719,24cyclictest3319385-21cron07:54:552
3305071994717,26cyclictest3317699-21systemd07:45:022
3305071994716,27cyclictest3340532-21sed08:50:022
3305071994716,27cyclictest3340532-21sed08:50:012
3305071994716,26cyclictest3378435-21grep10:35:292
3305071994715,28cyclictest923-21dbus-daemon09:35:012
3305071994715,27cyclictest3419512-21/usr/sbin/munin12:35:162
3305071994714,28cyclictest3379741-21df10:40:152
3305071994615,26cyclictest3410948-21cpuspeed12:10:142
3305071994613,29cyclictest3308930-21seq07:19:552
3305071994612,29cyclictest3333695-21sed08:30:022
33726972450,5sleep30-21swapper/310:20:193
33726972450,5sleep30-21swapper/310:20:193
330507199452,38cyclictest3392566-21tr11:15:242
330507199452,37cyclictest2746-21gmain10:00:152
330507199452,23cyclictest3418290-21irqstats12:30:192
3305071994518,23cyclictest3374049-21cpuspeed_turbos10:25:142
3305071994518,23cyclictest3312416-21latency_hist07:30:012
3305071994515,25cyclictest3389229-21cut11:05:282
330507199451,39cyclictest3312661-21/usr/sbin/munin07:30:212
3305070994524,4cyclictest4497-21PK-Backend07:30:331
33549322440,8sleep14497-21PK-Backend09:30:141
3305076994423,16cyclictest3390037-21latency_hist11:10:013
3305071994419,20cyclictest3399749-21grep11:35:292
3305071994417,23cyclictest3338304-21sort08:40:292
3305071994417,23cyclictest1137-21runrttasks12:03:012
330507199441,38cyclictest438-20systemd-journal08:54:592
3305071994412,28cyclictest3363749-21cat09:55:132
3305071994412,28cyclictest3363749-21cat09:55:132
3305071994411,29cyclictest3328584-21cat08:15:132
3305066994414,15cyclictest13-21ksoftirqd/010:36:590
197399440,8rtkit-daemon3395007-21cat11:24:553
330507199438,30cyclictest3356533-21seq09:35:142
3305071994317,22cyclictest3331711-21ls08:25:022
3305071994317,22cyclictest3331711-21ls08:25:012
3305071994310,29cyclictest959-21udisksd09:49:572
3305071994310,28cyclictest3404954-21cat11:50:302
3305071994310,28cyclictest3305999-21cat07:10:192
330507199428,29cyclictest3331800-21cpu08:25:152
3305071994219,19cyclictest1137-21runrttasks11:55:292
3305071994218,20cyclictest3414167-21latency_hist12:20:022
330507199421,37cyclictest3391571-21latency_hist11:15:012
330507199421,36cyclictest3401956-21latency_hist11:45:012
330507199421,36cyclictest3352580-21latency_hist09:25:022
3305071994213,24cyclictest438-20systemd-journal08:56:102
3305071994211,27cyclictest3347980-21cat09:10:152
330506699422,18cyclictest13-21ksoftirqd/010:50:350
330506699421,20cyclictest13-21ksoftirqd/007:49:090
330507199419,28cyclictest3370814-21expr10:15:132
330507199412,35cyclictest1137-21runrttasks09:40:162
3305071994114,23cyclictest3346474-21latency09:05:202
3305071994114,22cyclictest3314423-21cpu07:35:132
330506699412,15cyclictest0-21swapper/008:46:290
330506699412,15cyclictest0-21swapper/008:46:290
197399410,32rtkit-daemon3305060-21cyclictest12:18:420
197399410,32rtkit-daemon3305060-21cyclictest10:05:180
3305076994016,19cyclictest3375076-21sshd10:26:123
3305071994014,22cyclictest3349603-21cstates09:15:152
330507199401,35cyclictest3319193-21taskset07:47:332
330506699407,11cyclictest13-21ksoftirqd/011:00:000
330506699404,17cyclictest13-21ksoftirqd/011:05:370
330506699401,15cyclictest3343659-21seq08:56:410
197399400,31rtkit-daemon3305060-21cyclictest09:02:010
197399400,31rtkit-daemon3305060-21cyclictest08:04:410
33246422390,6chrt411rcuc/308:05:013
330507199397,6cyclictest33238462grep08:00:232
330507199393,31cyclictest3393922-21expr11:20:152
3305071993911,24cyclictest3368258-21seq10:07:372
3305066993911,17cyclictest161rcuc/010:41:590
330507699381,14cyclictest33443412cut09:00:133
3305071993813,21cyclictest3409363-21cstates12:05:142
330506699382,12cyclictest0-21swapper/011:15:060
33972712370,5sleep14497-21PK-Backend11:30:161
3305076993722,11cyclictest42-21ksoftirqd/311:39:493
3305076993721,12cyclictest1-21systemd09:35:023
3305076993719,14cyclictest3409060-21cat12:05:023
330507699371,31cyclictest438-20systemd-journal12:15:003
330507699371,14cyclictest33928062cat11:15:313
330506699377,12cyclictest13-21ksoftirqd/011:11:570
330506699376,15cyclictest13-21ksoftirqd/010:47:430
330506699371,19cyclictest3314391-21/usr/sbin/munin07:35:230
197399370,29rtkit-daemon0-21swapper/109:00:021
330507699362,6cyclictest3395883-21memory11:25:223
3305076993622,11cyclictest42-21ksoftirqd/310:35:013
3305076993622,10cyclictest3337952-21strings08:40:233
3305076993621,11cyclictest3387229-21memory11:00:223
3305076993620,12cyclictest3420247-21ntp_kernel_err12:35:213
3305076993620,11cyclictest1033-21gdbus10:41:073
3305076993619,12cyclictest438-20systemd-journal09:15:003
3305076993619,12cyclictest438-20systemd-journal08:39:553
330507099367,11cyclictest1137-21runrttasks09:29:431
330506699366,11cyclictest13-21ksoftirqd/011:33:460
330506699361,13cyclictest13-21ksoftirqd/011:00:530
197399360,25rtkit-daemon0-21swapper/310:49:523
3305076993526,6cyclictest3404624-21strings11:50:233
3305076993523,8cyclictest3308359-21python307:15:253
3305076993521,10cyclictest923-21dbus-daemon09:19:563
3305076993521,10cyclictest3369779-21cat10:10:303
3305076993519,12cyclictest42-21ksoftirqd/312:20:183
3305076993519,12cyclictest3398100-21cat11:30:303
3305076993519,11cyclictest3327180-21grep08:10:223
3305076993518,13cyclictest3318343-21which07:45:203
3305076993517,13cyclictest3400898-21which11:40:213
3305076993517,13cyclictest3362179-21df_inode09:50:143
330507699351,30cyclictest3366991-21basename10:05:093
3305076993510,20cyclictest0-21swapper/308:20:203
3305076993510,20cyclictest0-21swapper/308:20:203
3305070993519,12cyclictest3305658-21fw_packets07:10:131
3305070993514,6cyclictest141rcu_preempt10:40:011
3305070993511,6cyclictest141rcu_preempt11:45:001
330507099351,13cyclictest4497-21PK-Backend11:21:481
330507099351,13cyclictest4497-21PK-Backend11:18:511
330507099351,13cyclictest4497-21PK-Backend08:48:121
330507099351,13cyclictest4497-21PK-Backend08:48:121
330507099351,13cyclictest4497-21PK-Backend07:22:051
330507099351,11cyclictest4497-21PK-Backend08:40:311
330506699354,17cyclictest13-21ksoftirqd/010:30:270
3305076993423,7cyclictest3402621-21hddtemp_smartct11:45:193
3305076993422,8cyclictest3358686-21diskstats09:40:163
330507699342,28cyclictest3315966-21munin-run07:40:023
330507699342,27cyclictest3332367-21memory08:25:233
3305076993420,10cyclictest42-21ksoftirqd/307:50:133
3305076993420,10cyclictest42-21ksoftirqd/307:50:123
3305076993419,11cyclictest3363887-21expr09:55:153
3305076993419,11cyclictest3363887-21expr09:55:143
3305076993418,12cyclictest3406555-21sensors_temp11:55:263
3305076993416,15cyclictest3386043-40apt-check10:56:053
3305076993416,14cyclictest959-21udisksd12:09:573
3305076993416,14cyclictest3325217-21df08:05:133
3305076993415,15cyclictest3417570-21unin-run12:30:013
3305076993412,18cyclictest3352855-21apt09:25:123
330507699341,14cyclictest3310964-21grep07:25:133
3305070993427,4cyclictest0-21swapper/111:12:161
330507099341,9cyclictest4497-21PK-Backend10:24:141
330507099341,9cyclictest4497-21PK-Backend10:24:141
330507099341,9cyclictest4497-21PK-Backend10:09:531
330507099341,9cyclictest4497-21PK-Backend08:37:291
330507099341,9cyclictest4497-21PK-Backend08:24:331
330507099341,9cyclictest4497-21PK-Backend08:24:331
3305070993417,13cyclictest3346636-21awk09:05:221
3305070993417,12cyclictest3416440-21df12:25:141
330507099341,5cyclictest4497-21PK-Backend10:00:211
3305070993413,17cyclictest3365001-21idleruntime-cro09:59:591
3305070993413,17cyclictest3365001-21idleruntime-cro09:59:591
330507099341,30cyclictest4497-21PK-Backend08:52:111
330507099341,29cyclictest4497-21PK-Backend11:01:581
330507099341,29cyclictest4497-21PK-Backend09:45:591
330507099341,15cyclictest4497-21PK-Backend07:41:351
330507099341,13cyclictest4497-21PK-Backend09:19:351
330507099341,10cyclictest4497-21PK-Backend08:19:261
330506699345,14cyclictest13-21ksoftirqd/011:20:230
3305076993324,6cyclictest42-21ksoftirqd/312:30:243
3305076993322,7cyclictest3365672-21hddtemp_smartct10:00:173
3305076993322,7cyclictest3341490-21ntp_states08:50:253
3305076993320,9cyclictest42-21ksoftirqd/308:20:003
3305076993319,10cyclictest42-21ksoftirqd/309:35:303
3305076993318,11cyclictest3334625-21awk08:30:253
3305076993318,11cyclictest3322066-21grep07:55:233
3305076993318,11cyclictest3316583-21entropy07:40:173
3305076993316,13cyclictest3383252-21df10:50:163
3305076993316,13cyclictest3313431-21uname07:30:253
3305076993316,12cyclictest438-20systemd-journal12:19:553
330507699331,28cyclictest3343923-21sh09:00:003
330507699331,13cyclictest3345932-21apt09:05:123
330507699331,13cyclictest3309619-21/usr/sbin/munin07:20:173
330507699331,11cyclictest3351355-21expr09:20:143
3305070993326,4cyclictest4497-21PK-Backend09:35:381
3305070993325,5cyclictest4497-21PK-Backend12:17:561
3305070993323,6cyclictest3349131-21munin-run09:15:001
330507099331,9cyclictest4497-21PK-Backend12:36:301
330507099331,9cyclictest4497-21PK-Backend12:31:021
330507099331,9cyclictest4497-21PK-Backend12:00:401
3305070993319,9cyclictest3322402-21sed07:55:291
330507099331,8cyclictest4497-21PK-Backend08:33:401
3305070993317,12cyclictest3384246-21seq10:54:001
330507099331,29cyclictest4497-21PK-Backend10:14:261
330507099331,29cyclictest4497-21PK-Backend08:26:501
330507099331,28cyclictest4497-21PK-Backend09:50:211
330507099331,11cyclictest4497-21PK-Backend08:12:391
33957302320,7chrt4497-21PK-Backend11:25:201
330507699322,25cyclictest3360945-21python309:45:263
3305076993219,9cyclictest3379149-21sed10:40:023
3305070993221,7cyclictest3307460-21/usr/sbin/munin07:15:251
330507099321,9cyclictest4497-21PK-Backend11:52:111
330507099321,8cyclictest4497-21PK-Backend10:28:361
3305070993217,11cyclictest3399841-21awk11:35:301
3305070993217,10cyclictest3352143-21grep09:20:291
3305070993215,13cyclictest3323860-21ntpq08:00:221
330507099321,13cyclictest4497-21PK-Backend11:07:521
330507699311,10cyclictest3370562-21cat10:15:113
3305070993119,8cyclictest3315217-21ps07:35:261
3305070993118,9cyclictest3411819-21sed12:10:301
330507099311,27cyclictest4497-21PK-Backend07:29:191
330507099311,17cyclictest4497-21PK-Backend11:45:351
330507099311,10cyclictest4497-21PK-Backend12:07:001
3305070993023,4cyclictest4497-21PK-Backend08:08:061
3305070993013,13cyclictest3406370-21cut11:55:231
197399300,24rtkit-daemon4497-21PK-Backend09:41:321
330507099281,13cyclictest4497-21PK-Backend10:48:201
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional