You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 09:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot4s.osadl.org (updated Tue Dec 02, 2025 00:44:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
37639862760,4chrt0-21swapper/121:36:151
3614835996258,3cyclictest3676504-21kworker/u8:1+events_unbound21:11:081
3614833996248,12cyclictest3583283-21kworker/u8:0+flush-8:019:16:230
3614835995854,3cyclictest3768257-21kworker/u8:3+events_unbound21:51:241
3614842995744,4cyclictest2796699-21kworker/u8:4+flush-8:020:56:222
3614835995753,3cyclictest3583283-21kworker/u8:0+events_unbound19:16:241
3614835995450,3cyclictest3870068-21kworker/u8:3+events_unbound23:26:081
3614835995450,3cyclictest3870068-21kworker/u8:3+events_unbound23:26:081
3614835995450,3cyclictest3850467-21kworker/u8:2+events_unbound23:31:131
3614835995450,3cyclictest3850467-21kworker/u8:2+events_unbound23:31:121
3614835995450,3cyclictest3621879-21kworker/u8:2+events_unbound20:06:131
36573092530,4chrt0-21swapper/019:51:120
3614835995349,3cyclictest3755821-21kworker/u8:3+events_unbound21:31:071
3614833995343,8cyclictest282-21systemd-journal23:31:210
3614833995343,8cyclictest282-21systemd-journal23:31:210
3614842995243,7cyclictest3904976-40gzip00:00:012
3614835995248,3cyclictest2796699-21kworker/u8:4+events_unbound20:11:231
3614835995147,3cyclictest3870068-21kworker/u8:3+events_unbound23:46:251
3614842995045,3cyclictest3835042-21kworker/u8:2+events_unbound22:51:232
3614835995046,3cyclictest3621879-21kworker/u8:2+events_unbound20:01:221
3614833995042,5cyclictest3686145-21latency_hist20:20:010
3614835994946,2cyclictest3730631-21kworker/u8:2+events_unbound21:41:291
3614835994945,3cyclictest3889495-21kworker/u8:0+events_unbound00:06:071
3614835994945,3cyclictest3819897-21kworker/u8:0+events_unbound22:43:091
3614835994945,3cyclictest3761052-21kworker/u8:1+events_unbound22:21:221
3614835994939,8cyclictest3655700-21ntpq19:46:201
3614833994940,6cyclictest3870040-21latency_hist23:25:010
3614835994844,3cyclictest3761052-21kworker/u8:1+events_unbound22:16:271
3614835994844,3cyclictest3676504-21kworker/u8:1+events_unbound20:31:111
3614835994844,3cyclictest3676504-21kworker/u8:1+events_unbound20:31:111
3614835994844,3cyclictest2796699-21kworker/u8:4+events_unbound19:26:131
3614833994844,3cyclictest3676504-21kworker/u8:1+events_unbound20:21:090
3614833994843,3cyclictest3662028-21diskstats19:56:110
3614842994742,4cyclictest3870068-21kworker/u8:3+flush-8:023:51:242
3614842994742,3cyclictest3730631-21kworker/u8:2+events_unbound21:35:012
3614842994741,4cyclictest3850467-21kworker/u8:2+events_unbound23:41:212
3614835994743,3cyclictest2796699-21kworker/u8:4+events_unbound20:51:121
3614835994743,3cyclictest2796699-21kworker/u8:4+events_unbound20:41:261
3614835994737,7cyclictest3646634-21latency_hist19:40:011
3614835994642,3cyclictest2796699-21kworker/u8:4+events_unbound21:08:071
3614835994642,3cyclictest2796699-21kworker/u8:4+events_unbound19:30:041
3614833994635,8cyclictest3820483-21/usr/sbin/munin22:36:200
3614835994542,2cyclictest3768257-21kworker/u8:3+events_unbound22:31:081
3614835994541,3cyclictest3905191-21kworker/u8:4+events_unbound00:31:061
3614833994536,7cyclictest3943619-21ntpq00:36:200
3614835994441,2cyclictest3661465-21kworker/u8:0+events_unbound21:03:081
3614835994440,3cyclictest3819897-21kworker/u8:0+events_unbound23:11:171
3614835994440,3cyclictest3621879-21kworker/u8:2+events_unbound19:51:231
3614835994439,4cyclictest3844938-21kworker/u8:3+events_unbound22:59:231
3614835994439,4cyclictest3761052-21kworker/u8:1+events_unbound23:56:111
3614842994337,4cyclictest2796699-21kworker/u8:4+flush-8:020:51:192
3614835994338,4cyclictest3889495-21kworker/u8:0+events_unbound00:01:151
3614835994338,3cyclictest3761052-21kworker/u8:1+events_unbound23:41:441
3614833994337,4cyclictest3768257-21kworker/u8:3+flush-8:022:41:260
3614833994336,4cyclictest3790473-21latency_hist22:05:010
3614833994335,5cyclictest3768257-21kworker/u8:3+flush-8:021:51:210
3614833994334,7cyclictest3805929-21diskstats22:21:100
361448424328,9sleep10-21swapper/119:07:511
3614835994238,3cyclictest2796699-21kworker/u8:4+events_unbound20:36:271
3614833994237,4cyclictest2796699-21kworker/u8:4+events_unbound19:26:170
3614842994136,4cyclictest3730631-21kworker/u8:2+flush-8:021:46:212
3614842994136,4cyclictest3676504-21kworker/u8:1+flush-8:021:06:142
3614835994137,3cyclictest3905191-21kworker/u8:4+events_unbound00:28:141
3614835994137,3cyclictest3850467-21kworker/u8:2+events_unbound23:15:081
3614835994137,3cyclictest3768257-21kworker/u8:3+flush-8:022:26:121
39031472400,3sleep00-21swapper/023:56:190
3614842994034,4cyclictest3914062-21kworker/u8:1+events_unbound00:16:242
3614835994037,2cyclictest3661465-21kworker/u8:0+events_unbound20:46:441
3614835994036,3cyclictest3676504-21kworker/u8:1+events_unbound21:28:081
3614835994036,3cyclictest3583283-21kworker/u8:0+events_unbound19:23:061
3614835994035,4cyclictest2796699-21kworker/u8:4+events_unbound20:55:031
3614833994034,4cyclictest3621879-21kworker/u8:2+flush-8:020:01:250
3614833994030,7cyclictest3819897-21kworker/u8:0+flush-8:022:46:190
3614833994030,7cyclictest282-21systemd-journal20:55:010
3614842993935,3cyclictest3761052-21kworker/u8:1+flush-8:022:06:202
3614842993932,5cyclictest3819897-21kworker/u8:0+events_unbound22:46:232
3614835993936,2cyclictest3914062-21kworker/u8:1+events_unbound00:14:131
3614835993935,3cyclictest3870068-21kworker/u8:3+events_unbound00:21:241
3614835993935,3cyclictest3850467-21kworker/u8:2+events_unbound23:23:081
3614835993935,3cyclictest3768257-21kworker/u8:3+events_unbound22:03:081
3614835993935,3cyclictest3730631-21kworker/u8:2+events_unbound21:16:261
3614835993929,7cyclictest282-21systemd-journal22:11:071
3614833993932,5cyclictest282-21systemd-journal22:14:070
3614833993929,8cyclictest3711189-21apt-get20:46:060
3614835993834,3cyclictest3730631-21kworker/u8:2+events_unbound21:20:381
3614835993834,3cyclictest3621879-21kworker/u8:2+events_unbound19:40:071
3614835993834,3cyclictest2796699-21kworker/u8:4+events_unbound20:28:091
3614833993834,3cyclictest3676504-21kworker/u8:1+flush-8:021:01:170
3614833993833,3cyclictest3730631-21kworker/u8:2+flush-8:021:40:370
3614833993832,4cyclictest3768257-21kworker/u8:3+flush-8:021:56:120
3614833993831,5cyclictest3755821-21kworker/u8:3+events_unbound21:35:350
38247902370,3sleep2371rcuc/222:36:282
3614835993733,3cyclictest3850467-21kworker/u8:2+events_unbound23:38:121
3614835993733,3cyclictest3583283-21kworker/u8:0+events_unbound19:10:031
3614833993732,3cyclictest3870068-21kworker/u8:3+events_unbound00:11:170
3614833993732,3cyclictest2796699-21kworker/u8:4+flush-8:020:31:170
3614833993732,3cyclictest2796699-21kworker/u8:4+flush-8:020:31:170
3614833993731,4cyclictest3790720-21kworker/u8:2+events_unbound22:06:070
3614835993632,3cyclictest3870068-21kworker/u8:3+events_unbound23:54:391
3614835993632,3cyclictest3819897-21kworker/u8:0+events_unbound22:39:151
3614833993631,3cyclictest3755821-21kworker/u8:3+flush-8:021:31:250
3614833993630,4cyclictest3730631-21kworker/u8:2+flush-8:021:16:240
3614833993627,7cyclictest3636489-21apt-get19:30:010
3614833993625,8cyclictest3854432-21ntpq23:06:230
3614842993531,3cyclictest2796699-21kworker/u8:4+events_unbound19:15:022
3614842993530,4cyclictest3768257-21kworker/u8:3+flush-8:022:01:202
3614842993530,4cyclictest3761052-21kworker/u8:1+events_unbound22:00:012
3614842993523,9cyclictest3730609-21awk21:05:012
3614833993530,4cyclictest3676504-21kworker/u8:1+events_unbound21:11:150
3614833993530,4cyclictest2796699-21kworker/u8:4+flush-8:020:36:170
3614833993527,5cyclictest3889946-21cp23:45:010
3614842993429,4cyclictest3870068-21kworker/u8:3+flush-8:023:36:222
3614835993431,2cyclictest3676504-21kworker/u8:1+events_unbound20:21:261
3614835993430,3cyclictest3870068-21kworker/u8:3+events_unbound00:18:121
3614835993430,3cyclictest3784943-21kworker/u8:0+events_unbound21:57:131
3614833993429,4cyclictest3850467-21kworker/u8:2+flush-8:023:26:260
3614833993429,4cyclictest3850467-21kworker/u8:2+flush-8:023:26:260
3614833993429,4cyclictest3790720-21kworker/u8:2+events_unbound22:16:190
3614833993427,5cyclictest3761052-21kworker/u8:1+flush-8:022:26:190
3614833993426,6cyclictest3745793-21/usr/sbin/munin21:21:170
3614835993329,3cyclictest3761052-21kworker/u8:1+events_unbound22:48:101
3614835993329,3cyclictest3730631-21kworker/u8:2+events_unbound21:45:021
3614833993328,3cyclictest3676504-21kworker/u8:1+events_unbound21:30:010
3614842993228,3cyclictest2796699-21kworker/u8:4+flush-8:019:21:112
3614835993229,2cyclictest2796699-21kworker/u8:4+events_unbound20:18:501
3614833993227,3cyclictest3861015-21irqcore23:16:130
3614833993226,4cyclictest3905191-21kworker/u8:4+events_unbound00:01:250
3614842993126,4cyclictest3850467-21kworker/u8:2+flush-8:023:06:152
3614835993127,3cyclictest3905191-21kworker/u8:4+events_unbound00:37:131
3614835993127,3cyclictest3835042-21kworker/u8:2+events_unbound22:51:341
3614835993127,3cyclictest3761052-21kworker/u8:1+events_unbound23:08:081
3614835993127,3cyclictest3761052-21kworker/u8:1+events_unbound23:02:131
3614835993127,3cyclictest3761052-21kworker/u8:1+events_unbound22:09:211
3614835993127,3cyclictest3621879-21kworker/u8:2+events_unbound19:56:271
3614833993126,4cyclictest3621879-21kworker/u8:2+flush-8:020:06:150
3614833993126,3cyclictest3761052-21kworker/u8:1+events_unbound23:11:070
3614833993126,3cyclictest3676504-21kworker/u8:1+events_unbound20:26:180
3614833993126,3cyclictest3583283-21kworker/u8:0+events_unbound19:21:240
3614833993119,10cyclictest282-21systemd-journal21:10:000
3614833993026,3cyclictest3889495-21kworker/u8:0+flush-8:000:06:180
3614833993025,4cyclictest3905191-21kworker/u8:4+flush-8:000:26:430
3614833993025,3cyclictest3840349-21systemd-run22:55:000
3614833993025,3cyclictest3761052-21kworker/u8:1+flush-8:023:01:120
3614842992925,3cyclictest3905191-21kworker/u8:4+flush-8:000:36:242
3614842992923,4cyclictest3840657-21apt-get22:56:062
3614842992919,8cyclictest3755797-21cp21:30:002
3614833992925,3cyclictest3835042-21kworker/u8:2+flush-8:022:56:190
3614833992925,2cyclictest0-21swapper/020:45:010
3614833992923,5cyclictest3768257-21kworker/u8:3+flush-8:022:31:210
3614833992824,3cyclictest2796699-21kworker/u8:4+events_unbound20:56:210
3614833992824,2cyclictest3656508-21sh19:49:590
3614833992823,3cyclictest3905191-21kworker/u8:4+flush-8:000:31:220
3614842992723,2cyclictest3745933-21aten2_r5power_v21:21:082
3614833992722,3cyclictest0-21swapper/023:50:000
3614833992720,4cyclictest3884897-21latency_hist23:40:010
3614833992717,8cyclictest282-21systemd-journal00:25:010
3614833992620,4cyclictest3621879-21kworker/u8:2+flush-8:019:41:200
3614833992620,4cyclictest3621879-21kworker/u8:2+events_unbound19:36:220
3614842992518,5cyclictest3651241-21grep19:41:272
3614833992523,1cyclictest2796699-21kworker/u8:4+events_unbound20:11:150
3614842992421,2cyclictest3691463-21aten2_r5power_c20:26:092
3614842992414,8cyclictest3929675-21turbostat.cron00:25:012
361483399245,18cyclictest3676275-21kworker/0:1+events00:15:140
3614833992420,3cyclictest3583283-21kworker/u8:0+events_unbound19:11:250
3614842992319,2cyclictest3915435-21cpuspeed_turbos00:11:112
3614842992319,2cyclictest3810695-21aten2_r5power_e22:26:082
3614842992314,7cyclictest3745467-21awk21:20:002
3614842992219,2cyclictest3701529-21tr20:36:072
3614842992219,2cyclictest3686764-21fschecks_count20:21:112
3614842992218,2cyclictest3860797-21fschecks_time23:16:102
3614842992217,3cyclictest3870068-21kworker/u8:3+events_unbound23:25:022
3614842992217,3cyclictest3870068-21kworker/u8:3+events_unbound23:25:022
361484299220,5cyclictest37080931kthreadcore20:41:152
361483399221,20cyclictest3676275-21kworker/0:1+events23:50:060
361484299213,12cyclictest3890411-21apt-get23:46:052
361484299212,7cyclictest0-21swapper/221:36:102
3614842992118,2cyclictest3939211-21unixbench_singl00:31:242
3614842992117,3cyclictest3768257-21kworker/u8:3+events_unbound22:33:212
3614842992110,8cyclictest3638089-21tr19:31:152
361484299210,8cyclictest36324411irqcore19:26:142
3614833992118,1cyclictest0-21swapper/021:46:140
361484299202,4cyclictest0-21swapper/221:43:242
361484299202,13cyclictest3676587-21apt-get20:11:082
3614842992013,5cyclictest3700980-21munin-run20:35:002
3614842992013,5cyclictest3700980-21munin-run20:35:002
361484299200,11cyclictest36654571needreboot19:56:182
36144382203,11sleep00-21swapper/019:07:140
361484299194,4cyclictest36823451irqrtprio20:16:142
361484299192,5cyclictest0-21swapper/222:21:082
3614842991916,2cyclictest3661111-21unixbench_singl19:51:262
3614842991915,2cyclictest3711221-21aten2_r5power_c20:46:092
3614842991915,2cyclictest0-21swapper/200:01:252
36101292194,9sleep20-21swapper/219:06:082
361484299186,4cyclictest0-21swapper/222:16:092
361484299183,6cyclictest0-21swapper/221:51:082
361484299182,7cyclictest0-21swapper/200:26:212
361484299182,4cyclictest0-21swapper/219:11:062
3614842991814,2cyclictest3795897-21aten2_r5power_v22:11:072
3614842991813,4cyclictest3879608-21unixbench_singl23:31:272
3614842991813,4cyclictest3879608-21unixbench_singl23:31:272
361484299181,12cyclictest3735904-21apt-get21:11:072
3614842991810,6cyclictest3646615-21cron19:39:592
36143842183,8sleep30-21swapper/319:06:323
3614842991714,2cyclictest0-21swapper/200:06:242
3614842991713,3cyclictest3761052-21kworker/u8:1+events_unbound23:11:382
3614842991713,2cyclictest0-21swapper/222:45:002
3614842991712,4cyclictest3621879-21kworker/u8:2+events_unbound19:46:552
3614842991712,3cyclictest3670421-21grep20:01:192
361484299167,7cyclictest3672915-21tr20:06:182
361484299166,7cyclictest3845636-21apt-get23:01:082
361484299152,1cyclictest0-21swapper/223:23:362
39140022142,8sleep30-21swapper/300:06:183
36960192142,7sleep30-21swapper/320:30:003
36605892142,8sleep30-21swapper/319:51:203
38755802132,8sleep30-21swapper/323:31:123
38755802132,8sleep30-21swapper/323:31:123
38552052132,6sleep30-21swapper/323:10:013
38444362132,7sleep30-21swapper/322:56:203
38201992132,6sleep30-21swapper/322:35:003
38056832132,7sleep30-21swapper/322:21:093
37358782132,7sleep30-21swapper/321:11:053
37247472132,7sleep30-21swapper/320:56:213
39349092122,6sleep30-21swapper/300:31:033
39210232123,6sleep30-21swapper/300:16:143
39041932122,6sleep30-21swapper/323:56:283
38909122122,6sleep30-21swapper/323:46:123
38799492122,6sleep30-21swapper/323:35:013
38701862122,5sleep30-21swapper/323:25:023
38701862122,5sleep30-21swapper/323:25:013
38595582122,6sleep30-21swapper/323:11:253
38453282122,6sleep30-21swapper/323:00:013
38400002122,6sleep30-21swapper/322:51:273
38168652123,6sleep30-21swapper/322:31:163
38111302123,6sleep30-21swapper/322:26:143
38050252122,6sleep30-21swapper/322:16:273
37817832122,6sleep30-21swapper/321:56:173
37714852122,6sleep30-21swapper/321:46:133
37656342122,6sleep30-21swapper/321:39:073
37507682122,6sleep30-21swapper/321:26:113
37493862122,6sleep30-21swapper/321:21:173
37454562122,6sleep30-21swapper/321:20:013
37300202122,6sleep30-21swapper/321:01:273
37163772122,6sleep30-21swapper/320:51:113
37108432122,5sleep30-21swapper/320:45:013
36967392122,6sleep30-21swapper/320:31:133
36967392122,6sleep30-21swapper/320:31:133
36806522122,6sleep30-21swapper/320:11:253
36716872122,6sleep30-21swapper/320:06:133
36672232123,5sleep30-21swapper/320:01:153
36617832122,6sleep30-21swapper/319:56:093
36563522122,6sleep30-21swapper/319:48:073
36507272122,6sleep30-21swapper/319:41:213
36396682122,6sleep30-21swapper/319:31:173
36193942122,6sleep30-21swapper/319:11:273
39399332112,6sleep30-21swapper/300:36:093
39335962112,6sleep30-21swapper/300:26:183
39257472111,6sleep30-21swapper/300:21:113
39160952112,5sleep30-21swapper/300:11:173
39052662112,5sleep30-21swapper/300:01:053
38990792112,5sleep30-21swapper/323:51:183
38861092112,6sleep30-21swapper/323:41:153
38659392112,5sleep30-21swapper/323:21:123
38607322112,5sleep30-21swapper/323:16:113
38347542112,5sleep30-21swapper/322:46:253
38260482112,5sleep30-21swapper/322:41:113
37957442112,5sleep30-21swapper/322:10:533
37908492112,5sleep30-21swapper/322:06:083
37861072112,5sleep30-21swapper/322:01:093
37797982112,5sleep30-21swapper/321:51:233
37696232112,5sleep30-21swapper/321:41:203
37604972113,5sleep30-21swapper/321:34:293
37309482112,5sleep30-21swapper/321:06:083
37136542112,5sleep30-21swapper/320:46:183
37056622112,6sleep30-21swapper/320:36:473
36861892111,6sleep30-21swapper/320:20:013
36828682112,5sleep30-21swapper/320:16:163
36458712112,6sleep30-21swapper/319:36:233
36364432112,6sleep30-21swapper/319:29:373
36276722112,5sleep30-21swapper/319:21:143
36219652112,5sleep30-21swapper/319:16:083
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional