You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 09:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot4s.osadl.org (updated Mon Dec 01, 2025 00:44:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
396736723060,2sleep3481ktimers/321:11:253
395718222000,2sleep3586-21kworker/3:2-mm_percpu_wq21:01:163
390526021950,2sleep3481ktimers/320:11:173
404733921850,2sleep3586-21kworker/3:2+mm_percpu_wq22:35:003
39469512860,2sleep3586-21kworker/3:2-events20:51:183
40818242770,1chrt0-21swapper/123:06:281
3841470996554,8cyclictest3947395-21ntpq20:51:200
3841475996154,5cyclictest3933708-21/usr/sbin/munin20:41:261
40353072550,2sleep20-21swapper/222:21:132
3841470995448,5cyclictest4061561-21kworker/u8:0+events_unbound23:01:090
3841470995448,4cyclictest3841346-21kworker/u8:3+flush-8:019:16:250
3841479995346,5cyclictest4131957-21kworker/u8:2+flush-8:000:16:192
3841475994940,6cyclictest4157550-21latency_hist00:30:011
3841475994939,8cyclictest282-21systemd-journal19:15:051
3841470994938,8cyclictest4136915-21apt-get00:05:010
3841470994840,7cyclictest3841346-21kworker/u8:3+flush-8:019:41:200
3841470994839,7cyclictest282-21systemd-journal21:21:160
3841475994641,4cyclictest3841346-21kworker/u8:3+events_unbound20:01:251
3841470994537,5cyclictest4087095-21latency_hist23:15:010
3841475994440,3cyclictest3958426-21kworker/u8:2+flush-8:021:26:161
3841475994433,8cyclictest3877401-21ntpq19:41:221
3841470994438,4cyclictest3883156-21gpgv19:50:000
3841470994437,5cyclictest282-21systemd-journal23:16:150
3841470994433,8cyclictest4017428-21kworker/u8:0+flush-8:022:06:210
3841475994338,4cyclictest4047361-21kworker/u8:1+events_unbound23:41:191
3841479994237,4cyclictest3918737-21kworker/u8:3+flush-8:020:31:212
3841475994032,6cyclictest3958426-21kworker/u8:2+events_unbound22:21:181
3841475994031,7cyclictest4101090-21ntpq23:26:191
3841470994036,3cyclictest4017428-21kworker/u8:0+events_unbound22:26:270
3841470994035,3cyclictest4047361-21kworker/u8:1+events_unbound23:50:000
3841475993931,6cyclictest4117140-21/usr/sbin/munin23:46:181
3841475993828,7cyclictest282-21systemd-journal20:29:161
3841475993828,7cyclictest282-21systemd-journal20:29:161
3841470993827,8cyclictest282-21systemd-journal21:56:160
384147099380,35cyclictest4072306-21true23:00:000
384147099380,35cyclictest4072306-21true23:00:000
3841475993728,7cyclictest3863688-21apt-get19:31:061
3841470993732,4cyclictest3958426-21kworker/u8:2+events_unbound21:16:170
3841470993732,4cyclictest3831782-21kworker/u8:1+flush-8:020:11:260
3841470993728,7cyclictest282-21systemd-journal23:31:160
3841470993728,6cyclictest282-21systemd-journal23:25:010
3841475993631,4cyclictest4047361-21kworker/u8:1+events_unbound00:11:211
3841475993629,5cyclictest4126666-21kworker/u8:3+flush-8:000:01:161
3841470993630,4cyclictest3873229-21munin-run19:40:000
3841470993628,6cyclictest3942435-21ntpq20:46:200
38412552363,10sleep10-21swapper/119:09:361
3841479993529,4cyclictest3863119-21apt-get19:30:002
3841479993526,6cyclictest3982890-21cat21:30:002
3841475993528,5cyclictest4042743-21apt-get22:31:071
3841475993528,5cyclictest3858188-21latency_hist19:25:001
3841475993527,6cyclictest3893238-21latency_hist20:00:001
3841470993530,4cyclictest3831782-21kworker/u8:1+flush-8:019:11:130
3841470993530,3cyclictest3892447-21kworker/u8:0+events_unbound20:01:150
3841470993529,4cyclictest4002607-21kworker/u8:3+events_unbound21:51:190
3841470993529,4cyclictest3892447-21kworker/u8:0+events_unbound21:36:240
3841470993527,6cyclictest4147960-21/usr/sbin/munin00:21:160
3841479993422,9cyclictest4007809-21awk21:55:002
3841475993427,5cyclictest4157546-21apt-get00:30:031
3841470993428,5cyclictest3958426-21kworker/u8:2+flush-8:021:26:190
3841470993427,5cyclictest282-21systemd-journal20:06:150
3841470993425,7cyclictest4032824-21/usr/sbin/munin22:21:170
3841475993328,4cyclictest3813148-21kworker/u8:0+flush-8:019:26:091
3841475993326,6cyclictest4032217-21kworker/u8:3+flush-8:022:56:261
3841475993326,6cyclictest4032217-21kworker/u8:3+flush-8:022:56:261
3841475993325,5cyclictest4111910-21latency_hist23:40:011
3841475993324,7cyclictest3988128-21/usr/sbin/munin21:36:191
3841475993324,6cyclictest3943294-21latency_hist20:50:011
3841475993323,8cyclictest3883137-21apt-get19:50:021
3841470993329,3cyclictest3958426-21kworker/u8:2+events_unbound22:11:100
3841470993329,3cyclictest3841346-21kworker/u8:3+flush-8:019:56:220
3841470993328,4cyclictest4096576-21kworker/u8:2+events_unbound23:51:170
3841470993327,4cyclictest3958426-21kworker/u8:2+events_unbound22:36:220
3841470993323,7cyclictest282-21systemd-journal22:35:000
3841479993227,4cyclictest3958426-21kworker/u8:2+events_unbound21:16:192
3841475993227,4cyclictest4106628-21kworker/u8:0+flush-8:023:51:221
3841475993223,7cyclictest4018003-21/usr/sbin/munin22:06:191
3841470993227,3cyclictest3841346-21kworker/u8:3+events_unbound20:16:150
3841470993227,3cyclictest0-21swapper/000:05:000
3841470993226,4cyclictest4131883-21cron00:00:000
3841470993226,4cyclictest4111519-21unixbench_multi23:36:280
3841470993226,4cyclictest4101953-21cron23:30:000
3841470993225,5cyclictest4061561-21kworker/u8:0+events_unbound23:06:160
3841470993225,5cyclictest3958426-21kworker/u8:2+flush-8:021:06:210
3841470993225,5cyclictest3932462-21needreboot20:36:190
3841470993224,7cyclictest3892447-21kworker/u8:0+events_unbound20:31:300
3841479993126,4cyclictest4142905-21kworker/u8:0+events_unbound00:26:222
3841479993126,4cyclictest4061561-21kworker/u8:0+events_unbound22:56:142
3841479993126,4cyclictest4061561-21kworker/u8:0+events_unbound22:56:142
3841475993126,4cyclictest3943527-21kworker/u8:1+flush-8:020:51:121
3841475993126,3cyclictest4129440-21kthreadcore23:56:161
3841475993124,5cyclictest3957801-21unixbench_multi21:01:261
3841470993126,3cyclictest3913970-21aten2_r5power_p20:21:100
3841470993126,3cyclictest3883967-21fschecks_count19:51:120
384147099312,24cyclictest3831366-21kworker/0:1+events21:30:300
3841475993026,3cyclictest3958426-21kworker/u8:2+events_unbound21:20:341
3841475993025,4cyclictest4126666-21kworker/u8:3+flush-8:000:36:261
3841475993025,4cyclictest4002607-21kworker/u8:3+events_unbound22:01:101
3841475993025,3cyclictest4148703-21irqrtprio00:21:121
3841475993023,5cyclictest3831782-21kworker/u8:1+flush-8:020:11:121
3841475993022,6cyclictest282-21systemd-journal19:40:001
3841470993026,2cyclictest3998967-21irqrtprio21:46:140
3841470993026,2cyclictest3938403-21systemd-run20:45:010
3841470993024,5cyclictest3841346-21kworker/u8:3+events_unbound19:21:240
3841470993022,6cyclictest282-21systemd-journal21:45:000
384147099300,27cyclictest410-21mta-sts-daemon22:16:260
3841479992919,7cyclictest4062377-21awk22:50:002
3841475992924,4cyclictest4032217-21kworker/u8:3+flush-8:023:11:251
3841475992923,4cyclictest3846400-21systemd-run19:15:001
3841470992924,4cyclictest4142905-21kworker/u8:0+flush-8:000:26:150
3841470992924,4cyclictest4061561-21kworker/u8:0+flush-8:022:51:210
3841470992924,4cyclictest4061561-21kworker/u8:0+flush-8:022:51:210
3841470992924,3cyclictest3953783-21cpuspeed_turbos21:01:090
3841470992924,3cyclictest3919132-21fschecks_count20:26:120
3841470992924,3cyclictest3919132-21fschecks_count20:26:120
3841470992923,4cyclictest4138321-21cpuspeed_turbos00:11:050
3841470992923,4cyclictest3831782-21kworker/u8:1+events_unbound19:31:190
3841479992825,2cyclictest3904026-21fschecks_count20:11:132
3841479992824,2cyclictest3988503-21fschecks_time21:36:112
3841479992823,3cyclictest4122596-21fschecks_count23:51:102
3841479992817,9cyclictest4137766-21awk00:10:002
3841475992824,3cyclictest3892447-21kworker/u8:0+events_unbound21:00:011
3841475992823,3cyclictest3972648-21unixbench_singl21:16:271
3841475992823,3cyclictest3958426-21kworker/u8:2+events_unbound21:56:161
3841475992823,3cyclictest3924036-21cpuspeed_turbos20:31:101
3841475992823,3cyclictest3909622-21irqrtprio20:16:141
3841475992822,4cyclictest4096576-21kworker/u8:2+flush-8:023:31:191
3841475992822,4cyclictest4092379-21apt-get23:21:061
3841475992822,4cyclictest4032217-21kworker/u8:3+flush-8:022:46:211
3841475992822,4cyclictest4002893-21systemd-run21:50:001
3841475992822,4cyclictest3883369-21apt-key19:50:001
3841475992821,6cyclictest4032126-21unixbench_multi22:16:281
3841475992820,5cyclictest4047891-21apt-get22:36:081
3841475992818,8cyclictest4063242-21sort22:51:151
3841475992818,8cyclictest4063242-21sort22:51:141
384147099286,18cyclictest4017720-21kworker/0:0+events22:45:060
3841470992823,3cyclictest4158336-21cpuspeed_turbos00:31:120
3841470992823,3cyclictest4112567-21fschecks_time23:41:100
3841470992822,4cyclictest3953212-21sendmail21:00:000
3841470992822,4cyclictest3862237-21ntpq19:26:160
3841479992723,3cyclictest3903029-21unixbench_multi20:06:252
3841475992723,2cyclictest0-21swapper/120:40:011
3841475992721,5cyclictest3958725-21cpuspeed_turbos21:06:111
3841470992721,4cyclictest4017685-21cron22:05:000
3841479992622,2cyclictest3919157-21fschecks_time20:26:122
3841479992622,2cyclictest3919157-21fschecks_time20:26:122
3841479992615,9cyclictest3893253-21awk19:59:592
3841475992622,3cyclictest4057102-21unixbench_multi22:41:271
3841475992620,4cyclictest3997869-21apt-key21:45:011
3841475992619,5cyclictest3963487-21aten2_r5power_c21:11:071
38410652263,3sleep00-21swapper/019:07:100
40805702250,6sleep30-21swapper/323:06:183
3841479992522,2cyclictest4117515-21fschecks_time23:46:112
3841479992520,3cyclictest3878732-21cpuspeed_turbos19:46:082
3841479992518,4cyclictest282-21systemd-journal19:40:002
3841475992518,6cyclictest4042061-21unixbench_multi22:26:271
3841470992520,3cyclictest4047361-21kworker/u8:1+events_unbound22:41:090
3841470992519,4cyclictest4147255-21unixbench_multi00:16:230
3841479992420,2cyclictest4022268-21unixbench_multi22:06:272
3841479992419,3cyclictest4102380-21aten2_r5power_p23:31:082
3841479992416,6cyclictest4142676-21cat00:15:002
3841475992419,3cyclictest0-21swapper/123:16:141
384147999234,3cyclictest38647422tr19:31:152
3841479992320,2cyclictest4043008-21fschecks_count22:31:112
3841479992320,2cyclictest3953873-21fschecks_time21:01:102
3841479992320,2cyclictest3948958-21fschecks_time20:56:122
3841479992319,3cyclictest3912741-21kworker/u8:2+events_unbound20:16:232
3841479992314,6cyclictest3983193-21munin-node21:31:052
3841479992313,8cyclictest0-21swapper/222:30:002
3841475992319,2cyclictest0-21swapper/123:01:061
3841475992318,3cyclictest0-21swapper/120:10:001
384147099234,18cyclictest4017720-21kworker/0:0+events00:36:010
3841470992318,4cyclictest3958426-21kworker/u8:2+events_unbound21:11:280
3841479992217,3cyclictest4148336-21fschecks_time00:21:092
3841479992217,3cyclictest4127484-21cpuspeed_turbos23:56:092
3841479992217,3cyclictest3887918-21unixbench_multi19:51:282
3841479992214,6cyclictest4067331-21turbostat.cron22:55:002
3841479992214,6cyclictest4067331-21turbostat.cron22:55:002
3841475992217,3cyclictest0-21swapper/122:15:001
3841475992216,4cyclictest4143001-21apt-get00:16:021
41334792210,6sleep24134443-21sed00:01:162
3841479992118,2cyclictest3938823-21aten2_r5power_v20:46:092
3841479992116,3cyclictest3929050-21fschecks_count20:36:132
3841479992113,6cyclictest4092017-21cron23:20:012
3841479992111,2cyclictest0-21swapper/200:30:022
3841475992116,3cyclictest0-21swapper/121:31:091
3841475992115,4cyclictest3913968-21wget20:21:101
3841479992017,2cyclictest4013202-21aten2_r5power_p22:01:092
3841479992017,2cyclictest3948385-21latency_hist20:55:012
3841479992016,2cyclictest4107548-21fschecks_time23:36:122
3841479992016,2cyclictest4086712-21unixbench_multi23:11:272
3841479992016,2cyclictest4008409-21fschecks_count21:56:132
3841479992015,4cyclictest3853239-21sendmail19:20:002
3841479992011,7cyclictest4112603-21fschecks_time23:41:102
384147999201,10cyclictest38757062kthreadcore19:41:152
3841479992010,8cyclictest3977934-21awk21:25:002
3841475992014,4cyclictest4006043-21kthreadcore21:51:161
38376682201,14sleep23838475-21kthreadcore19:06:152
384147999199,7cyclictest4055137-21tr22:41:162
384147999192,12cyclictest3893822-21apt-get20:01:082
3841479991915,3cyclictest3958426-21kworker/u8:2+flush-8:021:05:362
3841479991915,3cyclictest0-21swapper/223:21:132
3841479991915,2cyclictest3845441-21needreboot19:11:192
384147999191,13cyclictest3933733-21apt-get20:41:082
3841479991813,4cyclictest3997844-21latency_hist21:45:012
384147999181,12cyclictest4163051-21apt-get00:36:102
384147999181,12cyclictest4022982-21apt-get22:11:072
384147999181,12cyclictest3913885-21apt-get20:21:092
3841475991812,4cyclictest831-21mta-sts-daemon00:06:021
38411302183,8sleep30-21swapper/319:08:013
384147999179,6cyclictest4100346-21tr23:26:172
384147999178,7cyclictest3998542-21uniq21:46:112
384147999173,8cyclictest4072621-21apt-get23:01:062
3841479991712,3cyclictest4048250-21fschecks_time22:36:132
3841479991712,3cyclictest3967111-21needreboot21:11:182
384147999167,7cyclictest3856371-21cut19:21:182
384147999162,7cyclictest4077557-21apt-get23:06:082
384147999162,13cyclictest0-21swapper/222:15:182
41534272133,7sleep30-21swapper/300:26:133
41026642132,7sleep30-21swapper/323:31:113
40978262131,7sleep30-21swapper/323:26:143
40843832133,6sleep30-21swapper/323:11:173
40316702133,6sleep30-21swapper/322:16:233
40128102132,6sleep30-21swapper/322:00:013
39732962132,7sleep30-21swapper/321:21:053
41636782122,6sleep30-21swapper/300:36:173
41619632122,6sleep30-21swapper/300:31:273
41384822120,5chrt0-21swapper/300:11:063
41360632122,6sleep30-21swapper/300:01:213
41315902122,6sleep30-21swapper/323:56:283
41110132122,6sleep30-21swapper/323:36:213
40923832122,6sleep30-21swapper/323:21:073
40226402122,6sleep30-21swapper/322:10:013
39982552122,6sleep30-21swapper/321:46:093
39933912122,6sleep30-21swapper/321:41:113
39878362122,6sleep30-21swapper/321:35:013
39787842122,6sleep30-21swapper/321:26:143
39689912122,6sleep30-21swapper/321:16:143
39629832122,6sleep30-21swapper/321:08:253
39523222122,6sleep30-21swapper/320:56:213
39374342122,6sleep30-21swapper/320:41:203
39135432122,6sleep30-21swapper/320:20:013
39091552122,6sleep30-21swapper/320:16:113
38940692122,6sleep30-21swapper/320:01:113
38926942122,6sleep30-21swapper/319:56:253
38841712123,6sleep30-21swapper/319:51:153
38782052122,5sleep30-21swapper/319:45:013
38778802122,6sleep30-21swapper/319:41:283
38532742122,6sleep30-21swapper/319:20:013
41525182112,6sleep30-21swapper/300:24:013
41466132112,5sleep30-21swapper/300:16:143
41375562112,6sleep30-21swapper/300:07:313
41217852110,6sleep30-21swapper/323:50:013
41209092112,5sleep30-21swapper/323:46:203
41125022112,6sleep30-21swapper/323:41:113
40876272112,5sleep30-21swapper/323:16:103
40723142111,5sleep30-21swapper/323:00:013
40675752112,5sleep30-21swapper/322:55:163
40675752112,5sleep30-21swapper/322:55:163
40630892112,5sleep30-21swapper/322:51:143
40630892112,5sleep30-21swapper/322:51:143
40590012112,5sleep30-21swapper/322:46:173
40531112112,5sleep30-21swapper/322:41:103
40378422112,5sleep30-21swapper/322:26:103
40366722112,5sleep30-21swapper/322:21:203
40235002112,5sleep30-21swapper/322:11:133
40086492112,5sleep30-21swapper/321:56:163
40040882112,5sleep30-21swapper/321:51:143
39877052112,6sleep30-21swapper/321:33:413
39391212112,5sleep30-21swapper/320:46:133
39287642112,5sleep30-21swapper/320:36:103
39250472111,6sleep30-21swapper/320:31:163
39193372112,6sleep30-21swapper/320:26:153
39193372112,6sleep30-21swapper/320:26:153
39027662112,5sleep30-21swapper/320:06:243
38688602112,5sleep30-21swapper/319:36:103
38636922112,6sleep30-21swapper/319:31:093
38528062112,6sleep30-21swapper/319:16:283
38417052112,5sleep30-21swapper/319:11:093
40473392101,5sleep30-21swapper/322:35:013
38584112102,5sleep30-21swapper/319:25:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional