You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-10 - 10:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Sat Jan 10, 2026 00:43:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
396145622090,2sleep23961455-21unin-run22:25:002
3753961213364,63sleep30-21swapper/319:05:263
396064821180,2sleep03960650-21find22:20:210
375422128452,10sleep10-21swapper/119:08:361
375404528365,10sleep20-21swapper/219:06:212
375399527455,13sleep00-21swapper/019:05:430
39717202320,3sleep23971724-21unixbench_singl22:30:262
3754445992825,1cyclictest0-21swapper/300:10:003
375444599249,10cyclictest3882200-21taskset21:06:013
3754445992421,1cyclictest0-21swapper/320:00:013
3754445992219,1cyclictest0-21swapper/323:00:003
3754445992218,2cyclictest0-21swapper/322:25:183
375444599221,17cyclictest4030193-21sleep23:30:003
375444599221,17cyclictest3876966-21taskset21:01:013
39341192211,5sleep23934166-21ntpq21:55:192
3754445992017,2cyclictest0-21swapper/320:05:003
3754445992017,1cyclictest0-21swapper/319:50:153
3754445992016,2cyclictest0-21swapper/320:25:003
375444599199,7cyclictest3956723-21taskset22:20:123
3754445991916,1cyclictest0-21swapper/322:45:013
3754445991916,1cyclictest0-21swapper/322:05:213
3754445991916,1cyclictest0-21swapper/319:35:213
3754445991915,2cyclictest0-21swapper/323:15:233
3754445991915,2cyclictest0-21swapper/300:40:003
375444599191,13cyclictest3798483-21sleep19:50:003
375444599182,10cyclictest3983457-21taskset22:45:133
3754445991815,1cyclictest0-21swapper/323:35:133
3754445991815,1cyclictest0-21swapper/322:00:103
3754445991815,1cyclictest0-21swapper/321:30:183
375444599181,12cyclictest4051463-21chrt23:50:103
375444599175,1cyclictest0-21swapper/321:50:133
375444599172,10cyclictest4072492-21taskset00:10:103
3754445991714,1cyclictest0-21swapper/323:20:113
375444599171,13cyclictest3913892-21turbostat21:40:003
375444599171,13cyclictest0-21swapper/320:25:113
375444599171,13cyclictest0-21swapper/300:00:113
375444599171,12cyclictest3851198-21turbostat20:40:003
375444599171,11cyclictest3867095-21chrt20:55:113
375444599171,10cyclictest3754617-21taskset19:10:033
375444599170,15cyclictest0-21swapper/320:05:123
375444599170,13cyclictest0-21swapper/321:15:183
37599892161,7sleep13761313-21kthreadcore19:15:021
375444599165,8cyclictest4056442-21turbostat23:59:553
375444599162,7cyclictest3956224-21taskset22:20:003
375444599161,9cyclictest3882774-21taskset21:10:063
375444599161,9cyclictest3856642-21taskset20:45:013
3754445991614,1cyclictest0-21swapper/322:50:173
3754445991613,1cyclictest0-21swapper/321:55:173
3754445991613,1cyclictest0-21swapper/320:19:563
3754445991612,2cyclictest0-21swapper/323:10:153
375444599161,12cyclictest3977365-21turbostat22:40:003
375444599161,12cyclictest0-21swapper/323:40:183
375444599161,12cyclictest0-21swapper/322:30:183
375444599161,11cyclictest3866829-21turbostat20:55:013
375444599160,14cyclictest0-21swapper/321:25:153
375444599160,13cyclictest0-21swapper/323:45:243
375444599160,13cyclictest0-21swapper/320:40:233
375444599160,13cyclictest0-21swapper/320:10:193
375444599160,13cyclictest0-21swapper/300:20:123
375444599160,10cyclictest4093271-21taskset00:26:333
375444599160,10cyclictest4082409-21taskset00:16:313
38918872150,10sleep13891976-21ntpq21:15:191
38147442151,4sleep13814952-21expr20:05:121
375444599151,9cyclictest4035237-21taskset23:33:023
375444599151,9cyclictest3897061-21taskset21:20:163
375444599151,9cyclictest3766877-21taskset19:19:563
3754445991512,1cyclictest0-21swapper/323:05:253
375444599151,10cyclictest3945963-21chrt22:10:103
375444599150,12cyclictest0-21swapper/321:40:103
375444599150,12cyclictest0-21swapper/319:30:003
375444599150,12cyclictest0-21swapper/300:30:123
40072482141,8sleep24008012-21sed23:05:172
38449342141,8sleep13840935-21/usr/sbin/munin20:30:221
3754445991411,2cyclictest0-21swapper/321:45:123
3754445991411,1cyclictest0-21swapper/323:00:143
3754445991411,1cyclictest0-21swapper/319:30:193
375444599140,12cyclictest0-21swapper/320:30:133
375444599140,12cyclictest0-21swapper/319:40:153
375444599140,12cyclictest0-21swapper/319:20:153
40924472131,3sleep24092535-21ntpq00:25:172
40299012130,1sleep10-21swapper/123:26:411
40291432131,8sleep04029191-21modprobe23:25:190
39837402131,2sleep10-21swapper/122:45:161
39423442131,7sleep13944118-21cut22:05:161
38972022131,8sleep03897248-21/usr/sbin/munin21:20:180
40947112121,7sleep04096382-21cut00:30:160
40237592121,6sleep04023900-21ntpq23:20:180
40041142122,5sleep04004132-21apt-get23:05:120
39738962121,7sleep03975657-21kthreadcore22:35:170
39196982121,5sleep13919822-21perl21:45:121
38885722121,7sleep23889390-21kthreadcore21:15:152
37937792121,7sleep23793912-21gltestperf19:45:142
37893502121,7sleep13790328-21kthreadcore19:40:151
37821472121,7sleep23777775-21/usr/sbin/munin19:30:262
37777972121,7sleep03777952-21cstates19:30:120
40776982110,1sleep20-21swapper/200:15:112
40519872111,6sleep24052445-21kthreadcore23:50:142
40320792111,6sleep14033843-21kthreadcore23:30:171
39619552111,5sleep23962168-21hddtemp_smartct22:25:132
39551062111,5sleep13955157-21ntpq22:15:181
39446702111,5sleep03944720-21ntpq22:05:190
39404042111,6sleep23940603-21latency_hist22:05:012
38985622110,1sleep10-21swapper/121:25:131
38427812111,6sleep23844581-21tr20:30:182
40978392101,4sleep14097892-21ntp_states00:30:211
40901132101,4sleep04091855-21grep00:25:150
40888202101,5sleep14088971-21expr00:25:121
40660302101,4sleep24066077-21ntpq00:00:202
40567102101,4sleep24056965-21df23:55:112
40541522101,4sleep04055270-21memory23:50:170
40463352101,4sleep14046458-21diskstats23:45:121
40187002101,4sleep14018757-21ntp_states23:15:231
40134092101,5sleep04013455-21ntpq23:10:200
39833332101,4sleep23983456-21diskstats22:45:132
39551592101,4sleep23955205-21ntpq22:15:192
39290092101,4sleep13929226-21users21:50:251
39248142101,4sleep03924937-21diskstats21:50:120
38882942104,3sleep00-21swapper/021:15:140
38778272101,4sleep23878036-21hddtemp_smartct21:05:132
38762792101,4sleep23876328-21ntpq21:00:202
38606262101,4sleep23860673-21ntpq20:45:212
38461902101,3sleep20-21swapper/220:35:112
38291552101,3sleep23829196-21ntpq20:15:192
37935602101,4sleep13793778-21df_inode19:45:121
37671902101,4sleep13767448-21df_inode19:20:111
4100029291,4sleep24101685-21tr00:35:152
4068504291,4sleep14070212-21tr00:05:161
4060649291,4sleep14060701-21ld23:55:181
4050171291,4sleep04050221-21ntpq23:45:190
4045911291,3sleep14045928-21munin-run23:45:001
4020992291,4sleep24022686-21cut23:20:162
4008889291,2sleep10-21swapper/123:07:011
3998838290,3sleep23998857-21apt-get23:00:122
3994110291,4sleep23994276-21cat22:55:152
3988688290,1sleep20-21swapper/222:50:142
3968607291,4sleep03970354-21grep22:30:160
3924032290,1sleep00-21swapper/021:45:310
3923289291,4sleep23923382-21ld21:45:182
3914730291,4sleep13915935-21cat21:40:151
3909449291,4sleep23909969-21cat21:35:162
3909181291,4sleep03909246-21grep21:35:130
3906365291,3sleep23907483-21memory21:30:182
3878110291,4sleep03879415-21cut21:05:160
3871007291,4sleep23871053-21ntpq20:55:192
3870865291,4sleep13871006-21ntpq20:55:181
3866128290,5sleep13866368-21unixbench_multi20:50:251
3861887291,4sleep03862145-21df_inode20:50:120
3860583291,3sleep13860624-21ntpq20:45:201
3850027291,4sleep13850167-21ntp_192.168.11420:35:181
3840934291,3sleep03840953-21apt-get20:30:120
3835674291,3sleep00-21swapper/020:25:110
3835359291,3sleep2800-21snmpd20:24:202
3831044291,4sleep13832494-21cat20:20:161
3814722291,3sleep03814741-21apt-get20:05:110
3805079291,4sleep03806759-21cut19:55:170
3800601291,4sleep13802391-21cut19:50:171
3799278291,4sleep03800600-21cut19:50:160
3792542291,4sleep23792752-21grep19:40:242
3767168291,2sleep00-21swapper/019:20:100
4088275281,3sleep14088473-21latency_hist00:25:001
4087118281,3sleep04087259-21ntpq00:20:180
4082437280,1sleep10-21swapper/100:16:541
4067269281,2sleep04067288-21apt-get00:05:110
4050275282,3sleep24046170-21/usr/sbin/munin23:45:212
4039540281,3sleep04039629-21ntpq23:35:170
4034392281,3sleep04034440-21ntpq23:30:200
4030712281,3sleep20-21swapper/223:30:142
4024916281,3sleep14024983-21latency_hist23:25:001
4018274280,1sleep20-21swapper/223:15:202
3997867281,3sleep13998126-21unixbench_multi22:55:261
3971464280,4sleep13971716-21grep22:30:261
3962247281,3sleep03963733-21cut22:25:150
3956456281,3sleep13956599-21/usr/sbin/munin22:20:121
3918198280,1sleep20-21swapper/221:40:202
3903694281,2sleep00-21swapper/021:30:120
3903432281,3sleep03903461-21perf21:30:000
3877703281,3sleep13877826-21diskstats21:05:121
3834367280,3sleep03834415-21ntpq20:20:190
3829648281,3sleep13829814-21vmstat20:15:261
3824676280,1sleep00-21swapper/020:11:570
3814953280,1sleep20-21swapper/220:05:132
3814186281,3sleep13814254-21systemd-run20:05:001
3809704281,3sleep23809801-21ps20:00:162
3797443281,3sleep03797492-21ntpq19:45:190
3788308281,2sleep00-21swapper/019:40:100
3786979281,3sleep13787017-21ntpq19:35:191
3786513280,1sleep00-21swapper/019:35:170
3777291281,4sleep1248-21systemd-journal19:27:121
4104163270,5sleep00-21swapper/000:38:420
4099310270,2sleep10-21swapper/100:35:101
4084397270,3sleep24086102-21grep00:20:172
4081994271,3sleep00-21swapper/000:15:250
4072470270,2sleep20-21swapper/200:10:112
4014254270,1sleep20-21swapper/223:13:142
4009660271,2sleep10-21swapper/123:10:131
4003222271,2sleep10-21swapper/123:00:271
3997595271,2sleep00-21swapper/022:55:220
3988135270,4sleep10-21swapper/122:50:011
3983168271,2sleep00-21swapper/022:45:110
3981632271,2sleep10-21swapper/122:40:191
3963734271,2sleep13965427-21ls22:25:161
3955842270,1sleep00-21swapper/022:16:090
3935378270,1sleep10-21swapper/122:00:071
3930133270,2sleep10-21swapper/121:55:101
3914160270,2sleep03914179-21apt-get21:40:100
3908545270,5sleep10-21swapper/121:33:491
3898207270,1sleep20-21swapper/221:24:532
3884050271,3sleep03885781-21kthreadcore21:10:160
3871756270,1sleep00-21swapper/020:56:370
3840373270,1sleep20-21swapper/220:26:132
3829938270,3sleep00-21swapper/020:16:570
3824264270,4sleep10-21swapper/120:10:251
3777982270,4sleep10-21swapper/119:30:131
3766583270,1sleep00-21swapper/019:16:140
3762766271,2sleep20-21swapper/219:15:152
3759484270,1sleep10-21swapper/119:13:091
4098434260,4sleep20-21swapper/200:30:432
4077164260,1sleep00-21swapper/000:11:360
4077102260,4sleep10-21swapper/100:10:501
4071851260,4sleep20-21swapper/200:05:292
4061975260,2sleep14061994-21apt-get00:00:121
4056273260,3sleep10-21swapper/123:53:021
4045690260,1sleep20-21swapper/223:42:122
4045683260,1sleep00-21swapper/023:42:060
4037052261,2sleep20-21swapper/223:35:162
4036210261,2sleep10-21swapper/123:35:141
4015227261,2sleep00-21swapper/023:15:170
3988416261,2sleep00-21swapper/022:50:120
3981537260,2sleep03981625-21ntpq22:40:180
3977660260,3sleep20-21swapper/222:40:132
3977195260,4sleep20-21swapper/222:37:592
3950822260,4sleep00-21swapper/022:13:540
3950615260,4sleep20-21swapper/222:11:142
3935952261,2sleep00-21swapper/022:00:150
3930428261,2sleep00-21swapper/021:55:130
3924648260,1sleep20-21swapper/221:50:102
3913523260,1sleep10-21swapper/121:35:351
3898473260,1sleep20-21swapper/221:25:062
3894098261,2sleep10-21swapper/121:20:151
3887611260,1sleep20-21swapper/221:13:142
3887479260,1sleep10-21swapper/121:11:331
3876657260,3sleep00-21swapper/021:00:260
3872307260,2sleep10-21swapper/121:00:111
3866703260,1sleep20-21swapper/220:53:392
3855406260,2sleep00-21swapper/020:40:180
3851463260,4sleep10-21swapper/120:40:071
3846470261,2sleep00-21swapper/020:35:130
3824691260,1sleep20-21swapper/220:12:062
3808862260,1sleep20-21swapper/219:59:242
3803353260,1sleep20-21swapper/219:51:102
3783598261,2sleep20-21swapper/219:35:152
3777269260,3sleep20-21swapper/219:26:572
3777196260,1sleep00-21swapper/019:26:000
3759301260,1sleep20-21swapper/219:10:502
4065844250,1sleep00-21swapper/000:00:190
4061435250,1sleep00-21swapper/023:57:050
4030153250,1sleep20-21swapper/223:29:552
3944994250,2sleep20-21swapper/222:05:252
3840496250,3sleep10-21swapper/120:27:481
3804195250,1sleep10-21swapper/119:55:141
3767449250,1sleep20-21swapper/219:20:122
3999119240,1sleep00-21swapper/023:00:140
3972666240,1sleep10-21swapper/122:35:141
3946237240,1sleep10-21swapper/122:10:141
3856950240,1sleep00-21swapper/020:45:150
3851755240,1sleep20-21swapper/220:40:132
3809409240,1sleep00-21swapper/020:00:140
3754651240,1sleep00-21swapper/019:10:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional