You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-18 - 08:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot4s.osadl.org (updated Thu Dec 18, 2025 00:44:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
113221021620,4sleep3575-21kworker/3:2+events23:14:043
8975482690,2chrt575-21kworker/3:2+events19:19:003
889927995845,10cyclictest1178836-21/usr/sbin/munin00:00:000
889927995845,10cyclictest1178836-21/usr/sbin/munin00:00:000
889936995652,3cyclictest1000738-21kworker/u8:2+events_unbound23:09:152
889932995445,7cyclictest1168994-21latency_hist23:50:001
889936995248,3cyclictest1031634-21kworker/u8:4+events_unbound21:34:152
889936995045,3cyclictest1000738-21kworker/u8:2+events_unbound21:10:302
889932995040,7cyclictest971031-21latency_hist20:30:011
889932994943,5cyclictest1040717-21kworker/u8:0+flush-8:021:49:001
889932994943,4cyclictest1149065-21kworker/u8:0+flush-8:000:20:311
889927994936,10cyclictest278-21systemd-journal00:30:000
889936994833,8cyclictest1134518-21kworker/u8:2+events_unbound00:24:042
889932994843,3cyclictest1031634-21kworker/u8:4+events_unbound21:43:581
889932994741,4cyclictest1134518-21kworker/u8:2+events_unbound23:53:591
889932994736,9cyclictest1124189-21apt-get23:05:001
889932994640,5cyclictest911934-21kworker/u8:3+events_unbound19:40:001
889932994640,4cyclictest941725-21kworker/u8:0+events_unbound20:05:011
889932994640,4cyclictest1050564-21kworker/u8:1+flush-8:022:59:141
889932994639,5cyclictest880046-21kworker/u8:1+flush-8:019:34:151
889932994540,4cyclictest951376-21kworker/u8:3+events_unbound20:15:011
889932994540,4cyclictest1000738-21kworker/u8:2+flush-8:022:08:591
889932994538,5cyclictest981361-21kworker/u8:3+events_unbound20:59:121
889932994535,7cyclictest894778-21latency_hist19:15:011
889932994439,3cyclictest920964-21kworker/u8:2+events_unbound19:45:001
889932994439,3cyclictest920964-21kworker/u8:2+events_unbound19:45:001
889932994339,3cyclictest1000738-21kworker/u8:2+events_unbound21:04:001
889932994337,5cyclictest1000738-21kworker/u8:2+events_unbound22:53:591
889932994336,5cyclictest981361-21kworker/u8:3+events_unbound20:49:101
889936994237,3cyclictest1050564-21kworker/u8:1+events_unbound22:35:122
889932994237,4cyclictest1134518-21kworker/u8:2+flush-8:023:59:151
889932994237,4cyclictest1134518-21kworker/u8:2+flush-8:023:59:151
889932994237,4cyclictest1000738-21kworker/u8:2+events_unbound21:25:011
889932994136,4cyclictest920964-21kworker/u8:2+flush-8:019:48:591
889932994134,6cyclictest1134518-21kworker/u8:2+flush-8:023:34:011
889932994132,6cyclictest1030364-21apt-get21:30:001
889927994114,13cyclictest1055818-21apt-get21:58:550
889932994036,3cyclictest1184012-21kworker/u8:3+flush-8:000:13:591
889932994035,4cyclictest981361-21kworker/u8:3+events_unbound20:43:591
889932994034,4cyclictest1050564-21kworker/u8:1+events_unbound22:33:581
889932994033,5cyclictest1149065-21kworker/u8:0+events_unbound00:34:101
889927994018,13cyclictest1011217-21seq21:13:560
889927994013,10cyclictest990620-21cat20:49:140
889927994011,16cyclictest927730-21/usr/sbin/munin19:49:010
889932993934,4cyclictest1050564-21kworker/u8:1+flush-8:022:14:151
889932993931,6cyclictest1184012-21kworker/u8:3+flush-8:000:09:031
889927993913,12cyclictest912260-21apt-get19:33:570
889932993834,3cyclictest880046-21kworker/u8:1+flush-8:019:58:591
889932993834,3cyclictest1149065-21kworker/u8:0+events_unbound00:40:011
889932993834,3cyclictest1134518-21kworker/u8:2+flush-8:023:19:071
889932993832,4cyclictest1129322-21kworker/u8:3+events_unbound23:29:021
889932993832,4cyclictest1000738-21kworker/u8:2+events_unbound21:10:011
88992799389,14cyclictest1040139-21cat21:39:160
889927993816,11cyclictest1198151-21ntpq00:19:100
889927993813,12cyclictest1019978-21ntpq21:19:120
889927993811,16cyclictest576-21runrttasks23:18:570
889927993811,11cyclictest1212808-21cat00:34:070
889927993810,12cyclictest970534-21wc20:29:130
8853022384,9sleep10-21swapper/119:07:191
889936993733,3cyclictest911934-21kworker/u8:3+events_unbound19:39:232
889932993731,4cyclictest880046-21kworker/u8:1+flush-8:019:24:031
889932993730,5cyclictest1031634-21kworker/u8:4+events_unbound21:58:551
889932993729,6cyclictest920964-21kworker/u8:2+events_unbound19:53:581
88992799379,17cyclictest1061003-21/usr/sbin/munin22:04:010
88992799379,15cyclictest1218520-21munin-run00:40:000
88992799379,12cyclictest894611-21grep19:14:120
88992799378,17cyclictest1139992-21df23:24:010
889927993731,5cyclictest1031634-21kworker/u8:4+flush-8:021:53:590
889927993729,5cyclictest907082-21sshd19:28:540
889927993713,13cyclictest1066038-21/usr/sbin/munin22:09:010
889927993713,10cyclictest980519-21ntpq20:39:120
889927993711,15cyclictest1118588-21ntpq22:59:100
889936993632,3cyclictest1050564-21kworker/u8:1+events_unbound23:49:092
889932993630,4cyclictest1031634-21kworker/u8:4+flush-8:021:39:061
889932993629,5cyclictest1129322-21kworker/u8:3+flush-8:023:24:021
889932993629,5cyclictest1000738-21kworker/u8:2+flush-8:022:04:161
88992799368,19cyclictest975967-21apt-get20:35:000
889927993616,10cyclictest906023-21ntpq19:24:130
889927993613,9cyclictest950750-21ntpq20:09:120
889927993612,9cyclictest995041-21cat20:54:070
889927993612,15cyclictest942017-21find20:03:560
889927993612,12cyclictest1149658-21/usr/sbin/munin23:33:580
889927993612,12cyclictest1049791-21ntpq21:49:120
889927993611,14cyclictest952153-21/usr/sbin/munin20:13:570
889927993611,14cyclictest922389-21/usr/sbin/munin19:44:010
889927993611,14cyclictest922389-21/usr/sbin/munin19:44:000
889927993611,14cyclictest1005179-21ntp_states21:04:130
889927993611,12cyclictest1189878-21hddtemp_smartct00:14:010
889927993610,9cyclictest1098945-21ntpq22:39:090
889927993610,9cyclictest1089075-21/usr/sbin/munin22:29:090
889927993610,17cyclictest981932-21/usr/sbin/munin20:44:000
889927993610,15cyclictest1168396-21ntpq23:49:090
889936993531,3cyclictest880046-21kworker/u8:1+events_unbound20:30:122
889936993530,4cyclictest981361-21kworker/u8:3+events_unbound20:44:132
889932993529,5cyclictest1050564-21kworker/u8:1+events_unbound23:39:161
889932993529,4cyclictest1129852-21sshd23:13:541
889932993529,4cyclictest1031634-21kworker/u8:4+flush-8:021:54:091
889932993529,4cyclictest1016131-21apt-get21:18:561
88992799359,8cyclictest1128679-21ld23:09:100
88992799359,7cyclictest1120122-21sort23:04:000
88992799359,14cyclictest1080706-21/usr/sbin/munin22:23:590
88992799358,16cyclictest1144941-21/usr/sbin/munin23:28:580
88992799358,16cyclictest1090536-21df22:33:580
88992799358,14cyclictest960388-21cat20:19:080
889927993522,8cyclictest1031145-21apt-get21:33:550
889927993514,7cyclictest1006648-21/usr/sbin/munin21:09:020
889927993513,10cyclictest1203122-21ntpq00:24:090
889927993513,10cyclictest1044850-21ntpq21:44:100
889927993512,10cyclictest932598-21hddtemp_smartct19:54:010
889927993512,10cyclictest1070788-21df22:13:580
889927993511,13cyclictest901058-21ntp_states19:19:130
889927993511,12cyclictest1104124-21python322:44:150
889927993510,14cyclictest1159968-21diskstats23:43:580
889927993510,13cyclictest940921-21ntp_states19:59:110
889927993510,13cyclictest1179669-21aten2_r5power_e00:03:590
889932993429,4cyclictest1000738-21kworker/u8:2+events_unbound22:49:171
889932993429,4cyclictest1000738-21kworker/u8:2+events_unbound22:49:171
889932993428,5cyclictest1134518-21kworker/u8:2+flush-8:023:44:101
88992799349,16cyclictest1026225-21/usr/sbin/munin21:28:590
88992799348,16cyclictest1110411-21if_enp4s022:54:010
889927993422,10cyclictest1130101-21diskstats23:13:590
889927993412,10cyclictest965326-21memory20:24:080
889927993412,10cyclictest1158372-21ntp_states23:39:130
889927993412,10cyclictest1025430-21munin-run21:25:000
889927993411,11cyclictest1108808-21ntpq22:49:100
889927993411,11cyclictest1108808-21ntpq22:49:100
889927993410,8cyclictest1184737-21ls00:09:010
889927993410,12cyclictest1079352-21ntp_states22:19:120
889936993329,3cyclictest1000738-21kworker/u8:2+events_unbound21:59:122
889936993328,3cyclictest880046-21kworker/u8:1+events_unbound19:40:022
889936993328,3cyclictest880046-21kworker/u8:1+events_unbound19:40:022
889932993328,4cyclictest951376-21kworker/u8:3+events_unbound20:24:091
889932993328,4cyclictest1149065-21kworker/u8:0+events_unbound00:29:151
889932993328,4cyclictest1050564-21kworker/u8:1+flush-8:000:04:041
889932993326,5cyclictest970586-21kworker/u8:2+flush-8:020:39:131
889932993326,5cyclictest1010693-21kworker/u8:0+events_unbound21:14:121
88992799338,12cyclictest1173770-21unixbench_singl23:54:150
889927993311,11cyclictest996666-21diskstats20:58:590
889927993311,11cyclictest917465-21df_inode19:39:000
889936993228,3cyclictest1129322-21kworker/u8:3+events_unbound23:24:022
889936993228,3cyclictest1000738-21kworker/u8:2+events_unbound21:15:012
889936993227,4cyclictest1050564-21kworker/u8:1+events_unbound23:10:022
889932993228,3cyclictest1050564-21kworker/u8:1+flush-8:022:29:161
889932993226,4cyclictest941725-21kworker/u8:0+flush-8:020:09:171
889932993226,4cyclictest1000738-21kworker/u8:2+events_unbound22:44:151
8852112324,9sleep20-21swapper/219:06:082
889936993127,3cyclictest1000738-21kworker/u8:2+events_unbound21:04:182
889932993127,3cyclictest1031735-21irqcore21:34:021
889932993122,7cyclictest1095184-21/usr/sbin/munin22:39:091
889936993025,4cyclictest1149065-21kworker/u8:0+events_unbound00:39:172
889932993026,3cyclictest1149065-21kworker/u8:0+flush-8:000:19:071
889932993024,5cyclictest1070234-21kworker/u8:0+events_unbound22:18:591
889936992925,3cyclictest1149065-21kworker/u8:0+events_unbound00:19:172
889936992925,3cyclictest1010693-21kworker/u8:0+flush-8:021:24:192
889936992925,3cyclictest1000738-21kworker/u8:2+events_unbound22:05:102
889936992922,5cyclictest906444-21grep19:24:172
889936992920,7cyclictest916668-21cat19:35:002
889936992919,8cyclictest1030742-21cat21:30:012
889932992924,3cyclictest901851-21kworker/u8:2+events_unbound19:29:051
889932992923,4cyclictest951376-21kworker/u8:3+flush-8:020:19:151
889927992917,9cyclictest0-21swapper/019:10:010
889936992824,3cyclictest1000738-21kworker/u8:2+events_unbound21:09:342
889932992822,5cyclictest1050564-21kworker/u8:1+events_unbound22:24:121
889936992724,2cyclictest1204372-21aten2_r5power_e00:28:582
889936992723,2cyclictest1070868-21fschecks_time22:14:002
889936992720,5cyclictest1178824-21apt-get00:00:012
889936992720,5cyclictest1178824-21apt-get00:00:002
889936992719,6cyclictest278-21systemd-journal20:10:542
889932992723,3cyclictest976004-21kworker/u8:0+events_unbound20:54:091
889932992722,4cyclictest1050564-21kworker/u8:1+flush-8:023:05:071
889932992719,6cyclictest897218-21/usr/sbin/munin19:19:141
889936992622,3cyclictest1050564-21kworker/u8:1+events_unbound23:52:352
889936992621,4cyclictest961204-21idleruntime-cro20:20:002
889936992521,3cyclictest1134518-21kworker/u8:2+events_unbound23:17:322
889936992521,2cyclictest1051149-21fschecks_count21:53:582
889936992420,3cyclictest1149065-21kworker/u8:0+events_unbound00:10:022
889936992415,7cyclictest1124280-21cp23:05:022
88993699241,11cyclictest161rcu_preempt22:58:562
88993699240,14cyclictest1154920-21ls23:39:022
889932992420,3cyclictest970586-21kworker/u8:2+events_unbound20:33:591
88993699238,3cyclictest161rcu_preempt19:15:002
889936992319,3cyclictest981361-21kworker/u8:3+events_unbound20:54:152
889936992319,3cyclictest976004-21kworker/u8:0+events_unbound20:45:242
889936992319,3cyclictest1070234-21kworker/u8:0+events_unbound22:20:082
88993699231,13cyclictest1105057-21seq22:48:582
88993699231,13cyclictest1105057-21seq22:48:572
88993699230,7cyclictest977096-21grep20:39:022
88993699230,7cyclictest946449-21apt20:05:002
88993699230,7cyclictest1153136-21taskset23:34:072
88993699230,1cyclictest0-21swapper/219:54:042
88993699230,14cyclictest1160309-21irqcore23:44:022
88993699228,2cyclictest161rcu_preempt21:50:002
88993699227,2cyclictest161rcu_preempt21:40:002
88993699226,2cyclictest161rcu_preempt22:44:052
889936992218,3cyclictest951376-21kworker/u8:3+events_unbound20:29:562
889936992217,4cyclictest901604-21sendmail19:20:002
889936992217,3cyclictest907330-21cpuspeed_turbos19:28:582
88993699221,1cyclictest161rcu_preempt00:09:142
88993699220,8cyclictest931518-21grep19:49:162
88993699220,10cyclictest965982-21grep20:24:162
88993699215,3cyclictest161rcu_preempt20:10:002
88993699212,2cyclictest0-21swapper/222:54:592
889936992117,3cyclictest1149065-21kworker/u8:0+events_unbound00:30:092
889936992117,3cyclictest1070234-21kworker/u8:0+events_unbound22:31:072
889936992116,4cyclictest20643-21cron20:00:002
88993699211,2cyclictest1000856-21cat21:00:012
88993699210,6cyclictest1041758-21sed21:44:032
88993699210,12cyclictest1089096-21awk22:29:092
88993699205,3cyclictest161rcu_preempt22:04:062
88993699202,2cyclictest161rcu_preempt23:29:032
889936992017,2cyclictest0-21swapper/222:19:052
88993699190,2cyclictest161rcu_preempt00:04:042
8852072183,8sleep30-21swapper/319:06:053
11386422156,5sleep30-21swapper/323:19:143
11149002142,8sleep30-21swapper/322:58:563
10152772143,8sleep30-21swapper/321:14:163
9759522132,7sleep30-21swapper/320:35:003
9661422132,6sleep30-21swapper/320:25:013
9625092133,6sleep30-21swapper/320:24:053
9437132133,7sleep30-21swapper/320:04:073
9044652133,6sleep30-21swapper/319:24:073
11935032132,7sleep30-21swapper/300:14:153
11714502132,7sleep30-21swapper/323:54:053
11598262132,7sleep30-21swapper/323:43:583
11484062132,7sleep30-21swapper/323:29:093
10841032132,7sleep30-21swapper/322:24:103
10700182132,6sleep30-21swapper/322:10:013
10464632132,7sleep30-21swapper/321:49:033
9966592123,5sleep30-21swapper/320:59:003
9935192122,6sleep30-21swapper/320:54:073
9821392122,6sleep30-21swapper/320:44:033
9507462123,5sleep30-21swapper/320:09:123
9392002122,6sleep30-21swapper/319:59:053
9263502123,5sleep30-21swapper/319:44:173
9263502123,5sleep30-21swapper/319:44:173
9114572120,7sleep30-21swapper/319:30:003
12095962122,6sleep30-21swapper/300:34:013
12047922122,6sleep30-21swapper/300:29:043
11980092121,7sleep30-21swapper/300:19:093
11496302122,6sleep30-21swapper/323:33:593
11401292123,5sleep30-21swapper/323:24:043
11053142122,6sleep30-21swapper/322:49:013
11053142122,6sleep30-21swapper/322:49:013
10946742122,5sleep30-21swapper/322:35:013
10906672122,6sleep30-21swapper/322:34:013
10854312122,6sleep30-21swapper/322:28:593
10698292122,6sleep30-21swapper/322:09:173
10638992122,6sleep30-21swapper/322:04:083
10268612122,6sleep30-21swapper/321:29:053
10250492122,6sleep30-21swapper/321:24:153
10101322122,6sleep30-21swapper/321:09:163
10055142122,6sleep30-21swapper/321:04:173
9904432112,5sleep30-21swapper/320:49:133
9567592113,5sleep30-21swapper/320:18:273
9521452112,5sleep30-21swapper/320:13:583
9270512112,5sleep30-21swapper/319:45:573
9170122112,5sleep30-21swapper/319:36:393
9119772113,5sleep30-21swapper/319:30:413
8941952112,5sleep30-21swapper/319:14:083
12144962112,5sleep30-21swapper/300:39:033
11992692112,5sleep30-21swapper/300:22:553
11839792112,5sleep30-21swapper/300:05:003
11793052112,5sleep30-21swapper/300:00:033
11741932112,5sleep30-21swapper/323:55:543
11741932112,5sleep30-21swapper/323:55:543
11643432112,5sleep30-21swapper/323:45:303
11549762112,5sleep30-21swapper/323:39:043
11250632112,5sleep30-21swapper/323:08:593
11236502112,5sleep30-21swapper/323:04:143
11001352112,5sleep30-21swapper/322:43:563
10752122112,5sleep30-21swapper/322:15:233
10555442113,5sleep30-21swapper/321:55:413
10508872112,5sleep30-21swapper/321:53:563
10408792113,5sleep30-21swapper/321:42:133
10358052112,6sleep30-21swapper/321:35:233
10161352112,5sleep30-21swapper/321:18:583
9764462102,5sleep30-21swapper/320:35:213
9318912102,5sleep30-21swapper/319:50:033
11098092102,5sleep30-21swapper/322:51:583
10308272102,5sleep30-21swapper/321:30:093
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional