You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 04:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot4s.osadl.org (updated Tue Nov 25, 2025 00:44:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10774412710,4sleep01077434-21ntpq20:26:230
997285997064,4cyclictest1158320-21kworker/u8:2+events_unbound21:51:271
997285996761,4cyclictest1232389-21kworker/u8:2+flush-8:023:16:241
997285996760,5cyclictest1183057-21kworker/u8:1+events_unbound23:26:231
997285996757,7cyclictest1075544-21apt-get20:26:191
997285996352,7cyclictest1127981-21latency_hist21:20:011
997282996242,17cyclictest1138104-21latency_hist21:30:010
12725842620,3chrt586-21kworker/3:2-events23:46:113
12725842620,3chrt586-21kworker/3:2-events23:46:113
997285996156,3cyclictest1183057-21kworker/u8:1+flush-8:022:36:271
997285996155,5cyclictest945057-21kworker/u8:3+flush-8:019:21:221
997285996054,5cyclictest160499-21kworker/u8:0+flush-8:021:31:241
997285996054,4cyclictest1158320-21kworker/u8:2+events_unbound22:01:251
997285996052,6cyclictest1118315-21kworker/u8:3+flush-8:021:36:251
997285996050,7cyclictest1167987-21latency_hist22:00:011
997285995847,8cyclictest1177054-21ntpq22:06:221
997285995752,3cyclictest1108430-21kworker/u8:1+events_unbound21:06:231
997285995750,5cyclictest1272396-21kworker/u8:3+flush-8:023:51:241
997285995652,3cyclictest1118315-21kworker/u8:3+events_unbound22:51:271
997285995652,3cyclictest1118315-21kworker/u8:3+events_unbound22:51:271
997285995651,4cyclictest1212844-21kworker/u8:0+flush-8:023:11:241
997285995651,3cyclictest160499-21kworker/u8:0+events_unbound20:01:161
997285995650,4cyclictest1183057-21kworker/u8:1+flush-8:022:56:281
11779822560,3sleep00-21swapper/022:10:010
997285995551,3cyclictest997417-21kworker/u8:1+events_unbound19:36:271
997285995550,4cyclictest945057-21kworker/u8:3+events_unbound20:36:121
997285995549,5cyclictest945057-21kworker/u8:3+flush-8:020:16:251
997285995547,6cyclictest1316938-21kworker/u8:1+flush-8:000:31:261
997282995535,17cyclictest1122218-21ntpq21:11:250
997282995531,21cyclictest1113731-21diskstats21:06:130
10702622550,6sleep30-21swapper/320:21:203
997285995449,4cyclictest160499-21kworker/u8:0+flush-8:020:46:241
997285995449,3cyclictest1108430-21kworker/u8:1+events_unbound21:01:201
997285995447,5cyclictest160499-21kworker/u8:0+flush-8:019:51:231
997285995445,7cyclictest1158320-21kworker/u8:2+flush-8:022:11:231
997282995434,18cyclictest1032887-21ntpq19:41:240
997282995434,18cyclictest1032887-21ntpq19:41:240
11834032540,3sleep2371rcuc/222:16:132
997285995348,4cyclictest945057-21kworker/u8:3+events_unbound19:56:211
997285995347,5cyclictest160499-21kworker/u8:0+flush-8:020:56:251
997285995347,4cyclictest1043835-21kworker/u8:2+events_unbound20:06:211
997285995247,3cyclictest1118315-21kworker/u8:3+events_unbound22:41:231
997285995243,7cyclictest1183139-21/usr/sbin/munin22:16:221
997285995242,8cyclictest1092318-21ntpq20:41:241
997285995146,3cyclictest1212844-21kworker/u8:0+flush-8:023:56:301
997285995146,3cyclictest1212844-21kworker/u8:0+flush-8:023:56:301
997285995046,3cyclictest1272396-21kworker/u8:3+events_unbound00:21:251
997285995042,6cyclictest1183057-21kworker/u8:1+events_unbound23:31:241
997285995040,8cyclictest1023101-21ntpq19:31:261
997285994943,4cyclictest1272396-21kworker/u8:3+events_unbound00:26:231
997285994842,5cyclictest997417-21kworker/u8:1+flush-8:019:26:251
997285994842,4cyclictest1212844-21kworker/u8:0+flush-8:000:16:241
997285994841,5cyclictest1212844-21kworker/u8:0+flush-8:023:01:251
997285994841,5cyclictest1212844-21kworker/u8:0+flush-8:000:11:251
997285994743,3cyclictest1232389-21kworker/u8:2+events_unbound23:25:001
997285994742,3cyclictest160499-21kworker/u8:0+events_unbound21:25:011
997285994641,4cyclictest160499-21kworker/u8:0+flush-8:020:51:221
997285994640,5cyclictest1282519-21kworker/u8:2+flush-8:000:01:231
997285994639,5cyclictest1183057-21kworker/u8:1+flush-8:023:06:241
997285994639,5cyclictest1183057-21kworker/u8:1+flush-8:022:26:241
997285994540,3cyclictest1271297-21ntpq23:41:221
997285994540,3cyclictest1271297-21ntpq23:41:221
997285994539,5cyclictest1043835-21kworker/u8:2+flush-8:020:21:191
997285994539,4cyclictest160499-21kworker/u8:0+flush-8:019:16:141
997285994538,5cyclictest1212844-21kworker/u8:0+flush-8:023:46:241
997285994538,5cyclictest1212844-21kworker/u8:0+flush-8:023:46:241
997282994535,8cyclictest1252948-21diskstats23:26:140
997285994440,3cyclictest1118315-21kworker/u8:3+events_unbound22:35:001
997285994438,4cyclictest997417-21kworker/u8:1+flush-8:019:41:251
997285994438,4cyclictest997417-21kworker/u8:1+flush-8:019:41:251
997282994437,5cyclictest1157008-21ntpq21:46:240
997282994430,9cyclictest282-21systemd-journal00:01:190
997282994426,16cyclictest829-21mta-sts-daemon22:35:000
997285994338,4cyclictest160499-21kworker/u8:0+events_unbound21:41:171
997285994338,4cyclictest1118315-21kworker/u8:3+flush-8:021:26:281
997285994338,3cyclictest1232389-21kworker/u8:2+flush-8:023:36:201
997285994334,7cyclictest997132-21latency_hist19:10:011
997282994333,7cyclictest282-21systemd-journal20:24:590
997282994324,16cyclictest1163050-21latency_hist21:55:010
997287994235,5cyclictest1297217-21cat00:10:012
997285994235,5cyclictest160499-21kworker/u8:0+events_unbound21:46:141
997282994233,7cyclictest1102997-21apt-get20:55:010
997282994223,17cyclictest1312280-21munin-plugin-st00:25:010
997282994223,16cyclictest1058416-21cron20:10:000
997285994135,5cyclictest997417-21kworker/u8:1+flush-8:019:11:251
997282994122,17cyclictest1043223-21unixbench_multi19:51:310
997282994119,8cyclictest1282627-21apt-get23:56:100
997282994119,8cyclictest1282627-21apt-get23:56:090
997282994119,3cyclictest1128302-21latency_hist21:20:010
997285994036,3cyclictest1043835-21kworker/u8:2+events_unbound20:11:241
997285994034,4cyclictest1118315-21kworker/u8:3+flush-8:021:11:191
997282994022,16cyclictest1009004-21cron19:20:000
997282994022,16cyclictest0-21swapper/023:31:150
997282994019,3cyclictest1083232-21idleruntime-cro20:35:000
997282994018,4cyclictest0-21swapper/021:36:190
997282994017,7cyclictest1317201-21cron00:30:010
99728299401,17cyclictest587-21runrttasks00:06:110
997285993934,4cyclictest1043835-21kworker/u8:2+flush-8:020:31:281
997282993918,8cyclictest1067495-21ntpq20:16:270
997287993830,5cyclictest1277126-21apt-get23:50:002
997287993830,5cyclictest1277126-21apt-get23:49:592
997285993833,4cyclictest1282519-21kworker/u8:2+flush-8:000:36:181
997282993818,2cyclictest0-21swapper/022:15:000
997287993728,7cyclictest1282278-21cron23:55:002
997285993732,3cyclictest1118315-21kworker/u8:3+flush-8:022:21:171
997282993727,7cyclictest282-21systemd-journal22:30:000
997282993717,6cyclictest282-21systemd-journal23:50:000
997282993717,6cyclictest282-21systemd-journal23:50:000
99728299371,21cyclictest1-21systemd23:20:000
997285993631,3cyclictest160499-21kworker/u8:0+events_unbound19:46:271
997282993627,6cyclictest1237331-21latency_hist23:10:010
997282993618,2cyclictest829-21mta-sts-daemon22:36:120
13103862360,5sleep3451irq_work/300:21:223
10191222360,5sleep0141ktimers/019:30:010
997285993531,3cyclictest1118315-21kworker/u8:3+events_unbound22:46:121
997282993529,4cyclictest1093131-21cron20:45:000
997282993517,4cyclictest1270768-21kthreadcore23:41:200
997282993517,4cyclictest1270768-21kthreadcore23:41:200
997282993515,4cyclictest832-21mta-sts-daemon22:16:180
997282993428,4cyclictest160499-21kworker/u8:0+flush-8:020:50:240
997282993327,4cyclictest997526-21apt-get19:11:100
997282993324,6cyclictest1098043-21latency_hist20:50:010
997282993315,3cyclictest1108812-21fschecks_count21:01:120
997282993226,4cyclictest1048559-21sh20:00:010
99728299321,8cyclictest1024694-21if_err_enp4s019:36:180
13263492320,2sleep2371rcuc/200:36:262
11822152320,4sleep3481ktimers/322:11:283
11571112320,4sleep2381ktimers/221:46:272
997287993122,7cyclictest1317212-21turbostat.cron00:30:012
997287993121,8cyclictest1058425-21turbostat.cron20:10:002
997282993025,3cyclictest1138702-21fschecks_count21:31:140
997282993023,5cyclictest1307097-21sendmail00:20:010
997287992925,2cyclictest1043278-21unixbench_singl19:51:312
997287992922,5cyclictest282-21systemd-journal22:37:192
997287992921,6cyclictest1142702-21grep21:31:302
997282992924,3cyclictest1238122-21cpuspeed_turbos23:11:120
997282992924,3cyclictest0-21swapper/000:15:010
997282992921,5cyclictest1217551-21latency_hist22:50:010
997287992824,2cyclictest1147974-21sh21:40:002
997287992823,3cyclictest1048178-21unixbench_multi19:56:292
997282992824,2cyclictest1252378-21systemd-run23:25:000
997282992823,3cyclictest1088201-21systemd-run20:40:010
997282992823,3cyclictest1053504-21cron20:05:010
997282992822,5cyclictest1118315-21kworker/u8:3+events_unbound21:41:290
997282992822,4cyclictest1323029-21irqcore00:36:160
997287992722,3cyclictest1247842-21aten2_r5power_e23:21:102
997287992721,4cyclictest1238481-21sed23:11:162
997287992717,8cyclictest1014002-21cat19:25:002
997282992722,4cyclictest160499-21kworker/u8:0+flush-8:019:45:230
997282992722,3cyclictest1321259-21needreboot00:31:230
997287992621,3cyclictest1128652-21aten2_r5power_p21:21:122
997287992616,8cyclictest282-21systemd-journal20:35:002
997285992621,3cyclictest1292827-21cpuspeed_turbos00:06:141
997282992621,4cyclictest1212844-21kworker/u8:0+events_unbound23:51:190
997282992620,4cyclictest1266851-21unixbench_multi23:36:320
997282992620,4cyclictest1022959-21needreboot19:31:220
997287992521,2cyclictest1073873-21fschecks_count20:26:142
997287992521,2cyclictest1063756-21aten2_r5power_e20:16:142
997287992518,5cyclictest1175457-21kthreadcore22:06:172
99728799250,9cyclictest11882002cut22:21:102
997287992421,2cyclictest1103789-21cpuspeed_turbos20:56:132
997287992419,3cyclictest1322168-21cat00:35:012
997287992419,3cyclictest1118084-21idleruntime-cro21:10:002
997287992418,4cyclictest1227410-21idleruntime-cro23:00:002
997287992417,5cyclictest1287241-40logrotate23:59:592
997287992417,5cyclictest1287241-40logrotate00:00:002
997287992415,7cyclictest405-21dbus-daemon23:05:002
997282992420,3cyclictest1208058-21aten2_r5power_p22:41:110
997282992420,2cyclictest1223330-21if_err_enp4s0.322:56:170
997282992419,3cyclictest1188877-21irqrtprio22:21:160
997282992415,7cyclictest282-21systemd-journal21:21:180
997287992320,2cyclictest1302519-21aten2_r5power_c00:16:122
997287992320,2cyclictest1098492-21aten2_r5power_v20:51:122
997287992320,2cyclictest1001802-21unixbench_multi19:11:302
997287992319,2cyclictest1158510-21aten2_r5power_p21:51:092
997282992318,3cyclictest1229933-21kthreadcore23:01:190
997282992317,4cyclictest1172926-21cron22:04:590
997282992317,4cyclictest1167620-21unixbench_multi21:56:290
99728799223,13cyclictest1118423-21apt-get21:11:112
997287992219,2cyclictest1297773-21cpuspeed_turbos00:11:142
997287992216,4cyclictest18912-21cron22:15:002
997282992218,2cyclictest0-21swapper/022:51:110
997282992218,2cyclictest0-21swapper/022:51:110
997282992217,3cyclictest0-21swapper/019:21:300
9970362223,3sleep00-21swapper/019:09:030
997287992118,2cyclictest1014603-21fschecks_count19:26:142
997287992117,2cyclictest1133510-21aten2_r5power_c21:26:112
997287992113,5cyclictest1063363-21cat20:14:592
99728799210,3cyclictest12166162tr22:46:212
99728799202,13cyclictest1148328-21apt-get21:41:122
997287992017,2cyclictest1108787-21fschecks_count21:01:112
997287992016,3cyclictest1038666-21turbostat.cron19:50:002
997287992015,3cyclictest0-21swapper/221:20:012
997287992014,5cyclictest0-21swapper/219:41:292
997287992014,5cyclictest0-21swapper/219:41:282
997282992015,3cyclictest1063357-21cron20:15:000
9968252204,9sleep20-21swapper/219:06:322
130208822012,5sleep30-21swapper/300:14:213
99728799192,13cyclictest1193123-21apt-get22:26:112
99728799192,11cyclictest1163384-21apt-get21:56:092
997287991916,2cyclictest1097671-21unixbench_multi20:46:312
997287991913,4cyclictest1263296-21irqrtprio23:36:192
997287991911,6cyclictest282-21systemd-journal23:35:002
997287991911,5cyclictest1170993-21tr22:01:182
99728799182,12cyclictest1052692-21ntpq20:01:252
997287991814,2cyclictest1307738-21aten2_r5power_e00:21:142
997287991814,2cyclictest1198115-21aten2_r5power_e22:31:102
997287991814,2cyclictest0-21swapper/200:01:302
997287991812,4cyclictest0-21swapper/222:41:202
9925052183,9sleep30-21swapper/319:05:453
997287991713,2cyclictest0-21swapper/219:36:242
997287991712,3cyclictest0-21swapper/223:10:012
997287991712,3cyclictest0-21swapper/220:21:322
99728799167,7cyclictest1253148-21awk23:26:162
99728799167,7cyclictest1084299-21sed20:36:162
99728799162,13cyclictest0-21swapper/223:40:232
99728799162,13cyclictest0-21swapper/223:40:232
997287991612,2cyclictest0-21swapper/223:16:312
997287991611,3cyclictest1008133-21ntpq19:16:212
997287991512,2cyclictest0-21swapper/220:41:262
997287991510,3cyclictest1118315-21kworker/u8:3+flush-8:022:51:132
997287991510,3cyclictest1118315-21kworker/u8:3+flush-8:022:51:122
997287991510,3cyclictest0-21swapper/219:31:192
11483112152,8sleep30-21swapper/321:41:113
12897852142,8sleep30-21swapper/300:01:213
13255792130,3sleep3586-21kworker/3:2-mm_percpu_wq00:36:193
12634692132,7sleep30-21swapper/323:36:213
12494952130,6sleep30-21swapper/323:21:193
12383782132,7sleep30-21swapper/323:11:163
12274532132,6sleep30-21swapper/323:00:013
12216132132,7sleep30-21swapper/322:51:223
12216132132,7sleep30-21swapper/322:51:223
12026672132,6sleep30-21swapper/322:35:013
10898992132,6sleep30-21swapper/320:41:203
10392572132,7sleep30-21swapper/319:51:163
10180462132,7sleep30-21swapper/319:26:243
13024612122,6sleep30-21swapper/300:16:103
12930942122,6sleep30-21swapper/300:06:183
12864692122,6sleep30-21swapper/323:56:273
12864692122,6sleep30-21swapper/323:56:273
12523762122,5sleep30-21swapper/323:25:003
12468352123,5sleep30-21swapper/323:16:303
12348832122,6sleep30-21swapper/323:06:193
12079972122,6sleep30-21swapper/322:41:113
12035062122,6sleep30-21swapper/322:36:173
11884002122,6sleep30-21swapper/322:21:123
11874222122,6sleep30-21swapper/322:16:313
11680272122,5sleep30-21swapper/322:00:013
11678912122,6sleep30-21swapper/321:59:013
11557082122,6sleep30-21swapper/321:46:213
11430372122,5sleep30-21swapper/321:35:013
11242642123,5sleep30-21swapper/321:16:193
11181002122,5sleep30-21swapper/321:10:013
11102242123,5sleep30-21swapper/321:01:183
10983832122,6sleep30-21swapper/320:51:113
10880512122,6sleep30-21swapper/320:38:513
10826182122,6sleep30-21swapper/320:31:293
10596462122,6sleep30-21swapper/320:11:183
10578732122,6sleep30-21swapper/320:06:293
10483162122,6sleep30-21swapper/319:57:023
10378022122,6sleep30-21swapper/319:46:223
10192572122,6sleep30-21swapper/319:31:133
10090732122,6sleep30-21swapper/319:20:013
10049012123,5sleep30-21swapper/319:16:163
10012682122,6sleep30-21swapper/319:11:233
13180312112,6sleep30-21swapper/300:31:173
13125642112,6sleep30-21swapper/300:25:533
12776662112,5sleep30-21swapper/323:51:093
12678172112,6sleep30-21swapper/323:41:143
12678172112,6sleep30-21swapper/323:41:133
12575432112,6sleep30-21swapper/323:30:033
12273522112,6sleep30-21swapper/322:59:213
12133722112,5sleep30-21swapper/322:46:163
11928282112,5sleep30-21swapper/322:25:003
11732192112,6sleep30-21swapper/322:05:473
11584842112,5sleep30-21swapper/321:51:113
11384412112,5sleep30-21swapper/321:31:123
11337292112,5sleep30-21swapper/321:26:143
11314212112,6sleep30-21swapper/321:21:203
11172452112,5sleep30-21swapper/321:06:243
11040102112,6sleep30-21swapper/320:56:173
10935222111,6sleep30-21swapper/320:46:133
10738692112,5sleep30-21swapper/320:26:153
10636712112,5sleep30-21swapper/320:16:123
10489162112,5sleep30-21swapper/320:01:113
10295092112,5sleep30-21swapper/319:41:163
10295092112,5sleep30-21swapper/319:41:163
10286362112,6sleep30-21swapper/319:38:193
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional