You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-05 - 03:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot4s.osadl.org (updated Fri Dec 05, 2025 00:44:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2998917997464,7cyclictest2961811-21kworker/u8:1+events_unbound19:56:181
32700672730,3chrt281rcuc/123:41:111
29968152690,4sleep00-21swapper/019:06:120
2998917996855,9cyclictest3145199-21kworker/u8:3+flush-8:021:51:151
32190032650,6chrt0-21swapper/222:46:262
2998917996556,7cyclictest2330870-21kworker/u8:2+flush-8:020:31:231
2998913996556,6cyclictest3318679-21latency_hist00:30:010
2998913996556,6cyclictest3318679-21latency_hist00:30:010
2998913996454,7cyclictest3051075-21/usr/sbin/munin20:01:240
2998917996154,5cyclictest3214829-21/usr/sbin/munin22:46:181
2998917995953,4cyclictest3224414-21kworker/u8:2+events_unbound23:26:251
2998917995951,5cyclictest3313768-21latency_hist00:25:011
2998913995948,9cyclictest3183742-21ntpq22:11:170
2998917995848,7cyclictest3140367-21/usr/sbin/munin21:31:221
2998917995846,9cyclictest282-21systemd-journal19:15:001
2998913995849,7cyclictest282-21systemd-journal20:15:010
2998917995751,4cyclictest2894841-21kworker/u8:3+flush-8:020:21:241
2998913995749,5cyclictest3163976-21ntpq21:51:150
2998917995650,4cyclictest3159495-21kworker/u8:0+events_unbound23:06:201
2998917995649,5cyclictest3060515-21kworker/u8:0+events_unbound20:51:221
2998917995648,5cyclictest3100500-21kworker/u8:1+events_unbound21:36:221
2998917995648,5cyclictest3100500-21kworker/u8:1+events_unbound21:36:221
2998913995647,6cyclictest3115350-21latency_hist21:05:010
2998917995549,4cyclictest3145199-21kworker/u8:3+flush-8:021:56:221
2998917995548,5cyclictest3060515-21kworker/u8:0+events_unbound20:56:201
2998917995446,6cyclictest2330870-21kworker/u8:2+flush-8:019:31:221
2998917995442,9cyclictest282-21systemd-journal22:06:191
30158982530,2sleep3471rcuc/319:25:013
2998917995346,5cyclictest3145199-21kworker/u8:3+flush-8:022:21:071
2998917995346,5cyclictest3060515-21kworker/u8:0+flush-8:021:06:211
2998917995344,6cyclictest3060515-21kworker/u8:0+flush-8:020:46:241
2998917995342,8cyclictest3233710-21ntpq23:01:181
2998917995342,8cyclictest282-21systemd-journal22:01:191
2998913995345,5cyclictest3085617-21latency_hist20:35:010
2998913995342,8cyclictest282-21systemd-journal21:21:190
2998917995247,4cyclictest2330870-21kworker/u8:2+flush-8:019:36:081
2998917995246,4cyclictest3145199-21kworker/u8:3+flush-8:023:11:261
2998917995246,4cyclictest3060515-21kworker/u8:0+flush-8:020:36:091
2998917995242,7cyclictest3154905-21latency_hist21:45:001
2998917995145,4cyclictest3145199-21kworker/u8:3+flush-8:022:31:121
2998917995137,10cyclictest282-21systemd-journal22:41:181
2998913995133,5cyclictest3044680-21ntpq19:51:160
32941102500,2sleep3586-21kworker/3:2-events00:05:013
32245722500,4chrt3224570-21latency_hist22:55:002
2998917995045,4cyclictest3159495-21kworker/u8:0+flush-8:022:51:221
2998917995044,4cyclictest2894841-21kworker/u8:3+events_unbound19:46:171
2998917995041,7cyclictest3284297-21/usr/sbin/munin23:56:181
32572202490,2sleep3471rcuc/323:26:163
2998917994944,4cyclictest3169837-21kworker/u8:1+flush-8:022:16:181
2998917994941,6cyclictest2894841-21kworker/u8:3+flush-8:019:26:161
2998917994939,7cyclictest3180038-21apt-get22:11:051
2998917994843,3cyclictest2961811-21kworker/u8:1+events_unbound19:15:021
2998917994842,4cyclictest3224414-21kworker/u8:2+flush-8:023:51:231
2998917994840,5cyclictest3100500-21kworker/u8:1+events_unbound21:11:211
2998917994838,8cyclictest3244707-21/usr/sbin/munin23:16:161
2998917994742,4cyclictest2330870-21kworker/u8:2+flush-8:020:01:221
2998917994741,4cyclictest3224414-21kworker/u8:2+flush-8:022:56:231
2998913994738,6cyclictest3298925-21latency_hist00:10:010
2998913994735,9cyclictest3066382-21diskstats20:16:080
299891399460,43cyclictest1-21systemd23:15:000
2998917994438,4cyclictest3120547-21kworker/u8:2+flush-8:021:16:231
2998917994437,5cyclictest3060515-21kworker/u8:0+flush-8:020:11:201
2998917994337,4cyclictest2330870-21kworker/u8:2+flush-8:020:26:191
2998917994336,5cyclictest3283611-21kworker/u8:0+events_unbound00:01:201
2998917994235,5cyclictest3283611-21kworker/u8:0+events_unbound00:31:191
2998917994136,3cyclictest3100500-21kworker/u8:1+flush-8:021:46:121
2998917994135,4cyclictest3060515-21kworker/u8:0+events_unbound21:26:091
2998917994132,7cyclictest3274406-21/usr/sbin/munin23:46:171
2998913994133,5cyclictest3020618-21latency_hist19:30:010
2998913994129,9cyclictest282-21systemd-journal20:41:170
2998920994032,6cyclictest282-21systemd-journal22:25:192
2998920993930,7cyclictest3144971-21turbostat.cron21:35:002
2998920993928,8cyclictest3030764-21cat19:40:002
2998917993933,4cyclictest3304204-21apt-get00:16:041
2998917993932,5cyclictest3120547-21kworker/u8:2+events_unbound21:21:251
2998913993931,6cyclictest282-21systemd-journal21:06:180
2998913993928,9cyclictest282-21systemd-journal20:55:000
2998913993927,9cyclictest3224396-21apt-get22:55:000
2998917993832,5cyclictest3060515-21kworker/u8:0+events_unbound20:16:181
2998913993828,7cyclictest3040597-21latency_hist19:50:000
2998913993824,11cyclictest282-21systemd-journal22:10:010
2998917993732,4cyclictest3249053-21kworker/u8:1+flush-8:023:36:131
2998917993730,5cyclictest2330870-21kworker/u8:2+events_unbound20:06:191
2998913993729,5cyclictest3060638-21latency_hist20:10:010
299891399371,34cyclictest2988683-21kworker/0:2+events19:31:160
2998917993632,3cyclictest2330870-21kworker/u8:2+flush-8:020:41:181
2998917993631,4cyclictest2330870-21kworker/u8:2+flush-8:019:41:201
2998917993629,5cyclictest3283611-21kworker/u8:0+flush-8:000:06:241
2998917993526,7cyclictest3114475-21ntpq21:01:211
2998917993526,6cyclictest3199730-21latency_hist22:30:011
2998913993425,7cyclictest3109487-21ntpq20:56:180
2998920993325,6cyclictest3051101-21apt-get20:01:052
2998917993329,3cyclictest2961811-21kworker/u8:1+flush-8:019:21:071
2998917993327,4cyclictest3288999-21kworker/u8:1+events_unbound00:11:221
2998917993327,4cyclictest3224414-21kworker/u8:2+flush-8:023:31:211
2998917993327,4cyclictest3169837-21kworker/u8:1+flush-8:022:36:191
2998913993326,5cyclictest282-21systemd-journal22:36:180
299891399332,30cyclictest3055727-21kworker/0:0+events22:29:480
2998920993221,8cyclictest3328574-21awk00:40:012
2998917993226,5cyclictest3224414-21kworker/u8:2+flush-8:023:21:161
299891399322,28cyclictest3234575-21kworker/0:1+events23:31:200
2998913993222,8cyclictest3139175-21ntpq21:26:170
299891399322,28cyclictest3055727-21kworker/0:0+events22:46:150
2998917993125,4cyclictest3283611-21kworker/u8:0+events_unbound00:26:171
2998917993125,4cyclictest3283611-21kworker/u8:0+events_unbound00:26:161
2998913993125,4cyclictest3100500-21kworker/u8:1+events_unbound21:16:070
2998913993123,6cyclictest282-21systemd-journal22:05:010
2998913993122,6cyclictest3293950-21latency_hist00:05:010
2998920993026,3cyclictest3159495-21kworker/u8:0+events_unbound22:15:012
2998920993025,4cyclictest3159495-21kworker/u8:0+flush-8:022:01:152
2998913993025,3cyclictest3249386-21systemd-run23:20:010
2998913993024,4cyclictest282-21systemd-journal22:45:000
2998913993022,6cyclictest282-21systemd-journal21:40:000
2998913993022,6cyclictest282-21systemd-journal21:40:000
2998920992924,4cyclictest3060515-21kworker/u8:0+flush-8:020:31:272
299891399296,21cyclictest3234575-21kworker/0:1+events23:26:200
2998913992925,3cyclictest3279056-21idleruntime-cro23:50:000
2998913992925,3cyclictest3060515-21kworker/u8:0+events_unbound20:46:250
2998913992910,17cyclictest3055727-21kworker/0:0+events21:41:120
299891399290,27cyclictest3055727-21kworker/0:0+events21:47:320
299891399290,27cyclictest2988683-21kworker/0:2+events19:21:200
2998920992820,6cyclictest282-21systemd-journal19:40:022
2998920992819,7cyclictest3319006-21apt-get00:31:052
299891399289,17cyclictest3055727-21kworker/0:0+events23:01:040
2998913992824,2cyclictest3050794-21systemd-run20:00:010
2998913992823,3cyclictest405-21dbus-daemon21:15:000
2998913992818,8cyclictest282-21systemd-journal22:20:000
2998913992817,9cyclictest282-21systemd-journal00:35:000
299891399281,26cyclictest3055727-21kworker/0:0+events22:21:160
299891399281,25cyclictest3055727-21kworker/0:0+events23:06:200
2998920992723,2cyclictest3091004-21aten2_r5power_p20:41:072
2998920992722,3cyclictest3185030-21aten2_r5power_p22:16:042
2998920992719,6cyclictest3045945-21cat19:55:002
2998920992717,8cyclictest3071090-21apt-get20:21:042
2998917992721,4cyclictest3283611-21kworker/u8:0+events_unbound00:36:121
2998917992720,5cyclictest2330870-21kworker/u8:2+events_unbound19:51:071
299891399279,16cyclictest2988683-21kworker/0:2+events19:11:200
299891399276,16cyclictest3234575-21kworker/0:1+events23:55:200
2998913992723,2cyclictest3313789-21systemd-run00:25:000
2998913992719,5cyclictest3328570-21latency_hist00:40:000
2998913992711,15cyclictest3055727-21kworker/0:0+events20:36:190
2998913992710,16cyclictest3234575-21kworker/0:1+events23:51:110
2998913992710,15cyclictest2988683-21kworker/0:2+events19:16:040
2998920992622,3cyclictest3234954-21aten2_r5power_p23:06:082
2998920992622,3cyclictest3200371-21fschecks_time22:31:122
2998920992616,8cyclictest3049824-21grep19:56:172
299892099260,10cyclictest33022262grep00:11:162
2998913992622,3cyclictest3074769-21needreboot20:21:170
2998920992522,2cyclictest3110992-21fschecks_count21:01:132
2998920992521,2cyclictest3279674-21fschecks_count23:51:082
2998920992521,2cyclictest3105099-21unixbench_singl20:51:292
2998920992516,6cyclictest3110370-21awk21:00:002
299891399259,15cyclictest3055727-21kworker/0:0+events22:56:200
2998913992521,3cyclictest3283611-21kworker/u8:0+flush-8:000:14:300
2998913992520,3cyclictest3166121-21kthreadcore21:56:140
2998920992421,2cyclictest3086255-21fschecks_time20:36:092
2998920992418,4cyclictest3129832-21unixbench_singl21:16:242
299891399248,15cyclictest2988683-21kworker/0:2+events19:40:240
299891399247,16cyclictest3234575-21kworker/0:1+events23:36:190
299891399247,16cyclictest3234575-21kworker/0:1+events23:21:200
2998913992419,4cyclictest3030094-21unixbench_multi19:36:240
2998920992319,2cyclictest0-21swapper/222:55:012
2998920992319,2cyclictest0-21swapper/220:15:012
2998920992318,3cyclictest3006121-21aten2_r5power_c19:16:042
29986522238,8sleep30-21swapper/319:08:493
299892099222,2cyclictest30169582cut19:26:112
2998920992218,2cyclictest0-21swapper/200:01:192
2998920992214,3cyclictest3193935-21ntpq22:21:192
2998913992218,2cyclictest0-21swapper/021:31:100
299892099212,14cyclictest3249691-21apt-get23:21:052
2998920992118,2cyclictest3015351-21unixbench_singl19:21:222
2998920992117,2cyclictest3095960-21aten2_r5power_p20:46:062
2998920992117,2cyclictest0-21swapper/221:41:142
2998920992115,5cyclictest3288999-21kworker/u8:1+flush-8:000:09:032
2998920992112,7cyclictest3177478-21cut22:06:152
299891399214,16cyclictest3234575-21kworker/0:1+events00:16:120
2998913992117,2cyclictest3078243-21runrttasks20:26:130
2998913992111,8cyclictest3203757-21ntpq22:31:200
29941822214,11sleep10-21swapper/119:05:421
299892099209,8cyclictest3145297-21apt-get21:36:032
299892099209,8cyclictest3145297-21apt-get21:36:032
2998920992016,2cyclictest0-21swapper/222:41:192
29941302204,10sleep20-21swapper/219:05:032
299892099192,12cyclictest3264518-21apt-get23:36:062
2998920991916,2cyclictest3115910-21cpuspeed_turbos21:06:092
2998920991914,4cyclictest3160542-21cut21:51:072
2998920991914,4cyclictest3159495-21kworker/u8:0+events_unbound21:56:102
2998920991914,3cyclictest3288569-21unixbench_multi23:56:262
2998920991912,5cyclictest3244408-21cat23:15:002
2998920991910,6cyclictest3003766-21sshd19:14:182
2998913991914,3cyclictest0-21swapper/023:41:080
33190092182,10sleep30-21swapper/300:31:063
299892099183,11cyclictest3204966-21apt-get22:36:052
299892099182,12cyclictest3254647-21apt-get23:26:052
2998920991814,2cyclictest3274781-21fschecks_time23:46:102
2998920991814,2cyclictest3269526-21aten2_r5power_c23:41:042
2998920991814,2cyclictest0-21swapper/223:16:232
2998920991813,3cyclictest3100500-21kworker/u8:1-ext4-rsv-conversion21:25:002
299892099179,6cyclictest3123005-21tr21:11:142
299892099179,6cyclictest3059314-21tr20:06:152
299892099178,7cyclictest3138735-21tr21:26:162
299892099172,11cyclictest3317800-21ntpq00:26:222
299892099172,11cyclictest3317800-21ntpq00:26:222
2998920991713,2cyclictest3068067-21kthreadcore20:16:132
2998920991713,2cyclictest0-21swapper/223:31:192
2998920991713,2cyclictest0-21swapper/219:45:072
2998920991711,4cyclictest0-21swapper/223:01:132
299892099171,11cyclictest3155253-21apt-get21:46:092
31840182166,6sleep30-21swapper/322:11:253
299892099168,6cyclictest3024708-21sshd19:31:162
299892099162,2cyclictest0-21swapper/220:26:052
299892099162,13cyclictest0-21swapper/200:16:032
2998920991613,2cyclictest0-21swapper/200:21:242
31552562152,8sleep30-21swapper/321:46:093
30710882152,8sleep30-21swapper/320:21:053
30510982152,8sleep30-21swapper/320:01:063
31950712142,8sleep30-21swapper/322:26:063
32996272132,7sleep30-21swapper/300:11:123
32954762132,7sleep30-21swapper/300:06:143
32831252132,7sleep30-21swapper/323:51:173
32691912132,6sleep30-21swapper/323:40:023
32194662132,6sleep30-21swapper/322:50:013
32005522132,7sleep30-21swapper/322:31:153
31502702132,7sleep30-21swapper/321:41:033
33146142122,6sleep30-21swapper/300:26:173
33146142122,6sleep30-21swapper/300:26:173
33044542122,6sleep30-21swapper/300:16:083
32850862121,7sleep30-21swapper/323:56:153
32748132122,6sleep30-21swapper/323:46:123
32639632122,6sleep30-21swapper/323:32:193
32535122122,6sleep30-21swapper/323:21:193
32444192122,6sleep30-21swapper/323:15:013
32403222122,6sleep30-21swapper/323:11:153
32310802122,6sleep30-21swapper/323:01:133
32293152122,6sleep30-21swapper/322:56:273
32105552122,6sleep30-21swapper/322:41:143
32054912122,6sleep30-21swapper/322:36:133
31701502122,6sleep30-21swapper/322:01:053
31449832122,6sleep30-21swapper/321:35:013
31449832122,6sleep30-21swapper/321:35:003
31390442122,6sleep30-21swapper/321:26:183
31255762122,6sleep30-21swapper/321:16:073
31206172122,6sleep30-21swapper/321:11:053
31104172122,6sleep30-21swapper/321:00:003
30996982123,6sleep30-21swapper/320:46:203
30909242122,6sleep30-21swapper/320:41:073
30856162122,6sleep30-21swapper/320:35:013
30760292122,6sleep30-21swapper/320:26:063
30661052122,6sleep30-21swapper/320:16:033
30649842122,6sleep30-21swapper/320:11:253
30606262121,6sleep30-21swapper/320:10:013
30501562122,6sleep30-21swapper/319:56:253
30398352122,6sleep30-21swapper/319:46:223
30309462122,6sleep30-21swapper/319:40:123
30246912122,6sleep30-21swapper/319:31:173
30036092122,6sleep30-21swapper/319:12:183
33279712112,5sleep30-21swapper/300:36:283
33090962113,5sleep30-21swapper/300:20:513
32644922112,5sleep30-21swapper/323:36:043
32348712112,5sleep30-21swapper/323:06:073
32148862110,7sleep30-21swapper/322:46:083
31939332112,5sleep30-21swapper/322:21:193
31851902112,5sleep30-21swapper/322:16:063
31750192112,6sleep30-21swapper/322:05:193
31654442112,5sleep30-21swapper/321:56:123
31640702112,5sleep30-21swapper/321:51:183
31407652112,5sleep30-21swapper/321:31:163
31304902112,5sleep30-21swapper/321:21:043
31155702112,6sleep30-21swapper/321:05:033
31007862112,5sleep30-21swapper/320:51:073
30830762112,5sleep30-21swapper/320:31:203
30409032112,5sleep30-21swapper/319:51:033
30258902112,5sleep30-21swapper/319:36:063
30159432112,6sleep30-21swapper/319:25:353
30043332112,5sleep30-21swapper/319:15:033
31062512100,6sleep30-21swapper/320:56:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional