You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 11:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot4s.osadl.org (updated Sun Nov 30, 2025 00:44:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2581521950,2sleep3586-21kworker/3:2-events22:01:183
41727012930,3sleep3451irq_work/321:11:303
956882730,2sleep2371rcuc/223:11:162
4046844995954,4cyclictest73112-21kworker/u8:1+flush-8:022:51:231
4046844995853,4cyclictest4143843-21kworker/u8:0+events_unbound21:41:251
4046844995753,3cyclictest4032036-21kworker/u8:0+flush-8:019:26:241
4046844995449,4cyclictest4032036-21kworker/u8:0+events_unbound20:15:001
4046844995447,5cyclictest28810-21kworker/u8:3+events_unbound22:06:291
45232530,6chrt371rcuc/221:41:102
4046844995347,4cyclictest4143843-21kworker/u8:0+events_unbound21:11:271
4046844995247,3cyclictest68075-21kworker/u8:0+events_unbound23:36:251
4046844995244,6cyclictest18665-21kworker/u8:1+flush-8:022:01:231
4046844995139,9cyclictest4097832-21ntpq19:56:211
4046843995142,6cyclictest53799-21latency_hist22:30:010
4046844995045,4cyclictest68075-21kworker/u8:0+events_unbound22:46:221
4046844995045,4cyclictest4032036-21kworker/u8:0+flush-8:020:21:251
4046844995044,5cyclictest3221998-21kworker/u8:1+flush-8:019:31:171
4046844995041,7cyclictest4069341-21kworker/u8:3+flush-8:020:46:211
4046844995041,7cyclictest4069341-21kworker/u8:3+flush-8:020:46:211
4046844995039,8cyclictest23326-21ntpq21:56:241
4046844994942,4cyclictest28810-21kworker/u8:3+events_unbound22:11:141
4046844994841,5cyclictest3221998-21kworker/u8:1+flush-8:019:41:221
4046843994842,4cyclictest28810-21kworker/u8:3+events_unbound23:16:200
4046843994741,4cyclictest4143534-21apt-get20:46:070
4046843994741,4cyclictest4143534-21apt-get20:46:070
4046844994640,4cyclictest93010-21kworker/u8:2+events_unbound23:51:101
4046844994640,4cyclictest93010-21kworker/u8:2+events_unbound23:51:101
4046844994640,4cyclictest18665-21kworker/u8:1+events_unbound22:16:271
4046844994538,5cyclictest73112-21kworker/u8:1+events_unbound23:06:221
4046843994535,8cyclictest18128-21ntpq21:51:230
4046844994338,3cyclictest4069341-21kworker/u8:3+events_unbound20:06:181
4046844994337,4cyclictest28810-21kworker/u8:3+flush-8:023:11:151
4046844994333,7cyclictest107362-21ntpq23:21:191
4046844994329,11cyclictest282-21systemd-journal00:35:001
404684399430,41cyclictest4163150-21true21:05:010
4046844994236,4cyclictest93010-21kworker/u8:2+events_unbound00:36:261
4046844994235,5cyclictest4032036-21kworker/u8:0+flush-8:020:16:221
4046844994234,6cyclictest4069341-21kworker/u8:3+flush-8:020:31:261
4046844994234,6cyclictest152969-21latency_hist00:10:011
4046844994234,6cyclictest152969-21latency_hist00:10:011
4046844994136,3cyclictest93010-21kworker/u8:2+events_unbound00:20:001
4046844994136,3cyclictest4177147-21kworker/u8:2+events_unbound21:26:121
4046844994135,4cyclictest28810-21kworker/u8:3+flush-8:023:16:261
4046844994133,6cyclictest143307-21kworker/u8:1+flush-8:000:11:201
4046844994133,6cyclictest143307-21kworker/u8:1+flush-8:000:11:191
4046843994135,4cyclictest4193167-21apt-get21:36:060
4046843994134,5cyclictest163238-21apt-key00:20:010
4046843994131,8cyclictest282-21systemd-journal19:30:000
4046844994033,5cyclictest143307-21kworker/u8:1+events_unbound00:01:241
4046844993933,4cyclictest58998-21kworker/u8:1+flush-8:022:41:151
4046844993931,6cyclictest282-21systemd-journal19:15:061
4046844993929,8cyclictest4092924-21ntpq19:51:211
4046844993929,8cyclictest13222-21ntpq21:46:241
4046844993831,5cyclictest4128548-21kworker/u8:1+flush-8:021:06:231
4046844993829,7cyclictest168384-21apt-get00:26:081
4046844993829,7cyclictest142909-21cat00:00:001
4046844993829,7cyclictest128320-21/usr/sbin/munin23:46:201
4046844993829,7cyclictest128320-21/usr/sbin/munin23:46:201
4046844993828,7cyclictest52912-21ntpq22:26:181
4046843993829,6cyclictest4173334-21/usr/sbin/munin21:16:230
4046844993732,3cyclictest87601-21ntpq23:01:221
4046844993627,7cyclictest4128317-21latency_hist20:30:011
4046843993626,8cyclictest4057745-21ntpq19:16:220
4046849993527,6cyclictest72798-21ntpq22:46:232
4046844993529,5cyclictest4158412-21kworker/u8:2+flush-8:021:01:281
4046844993529,4cyclictest4138672-21tr20:41:111
4046844993428,4cyclictest4143843-21kworker/u8:0+events_unbound21:36:231
4046844993427,5cyclictest8774-21kworker/u8:2+events_unbound22:31:171
4046844993427,5cyclictest68075-21kworker/u8:0+events_unbound22:56:241
4046849993328,4cyclictest58998-21kworker/u8:1+flush-8:022:41:122
4046849993322,8cyclictest4051836-21cp19:15:002
4046844993326,5cyclictest3221998-21kworker/u8:1+flush-8:019:21:121
4046844993325,3cyclictest4177147-21kworker/u8:2+flush-8:021:30:311
4046843993328,3cyclictest4098692-21systemd-run19:59:590
4046849993227,3cyclictest4069341-21kworker/u8:3+events_unbound20:51:112
4046849993227,3cyclictest4069341-21kworker/u8:3+events_unbound20:51:112
4046844993225,5cyclictest4133610-21apt-get20:36:071
4046844993224,6cyclictest93010-21kworker/u8:2+flush-8:023:31:201
4046843993225,5cyclictest3221998-21kworker/u8:1+flush-8:019:31:130
4046843993222,7cyclictest113194-21latency_hist23:30:010
4046843993221,9cyclictest282-21systemd-journal23:59:590
4046849993126,4cyclictest137897-21kworker/u8:3+events_unbound00:21:242
4046844993125,4cyclictest93010-21kworker/u8:2+flush-8:023:41:241
4046844993124,5cyclictest4178296-21apt-get21:21:081
4046844993122,7cyclictest4074037-21/usr/sbin/munin19:36:211
4046843993127,3cyclictest4069341-21kworker/u8:3+flush-8:020:26:150
4046843993122,7cyclictest282-21systemd-journal23:50:000
4046843993122,7cyclictest282-21systemd-journal23:50:000
4046844993025,4cyclictest4143843-21kworker/u8:0+flush-8:021:16:271
4046844993025,4cyclictest137897-21kworker/u8:3+events_unbound00:21:161
4046844993021,7cyclictest47989-21ntpq22:21:201
4046843993026,2cyclictest13999-21idleruntime-cro21:50:000
4046843993024,4cyclictest4032036-21kworker/u8:0+flush-8:019:11:270
4046844992925,3cyclictest2622304-21kworker/u8:3+flush-8:019:11:221
4046844992921,6cyclictest4152434-21ntpq20:51:231
4046844992921,6cyclictest4152434-21ntpq20:51:231
4046843992924,4cyclictest3221998-21kworker/u8:1+flush-8:019:41:310
4046843992921,5cyclictest29086-21latency_hist22:05:010
4046843992919,7cyclictest83517-21latency_hist23:00:010
4046843992918,8cyclictest162844-21latency_hist00:20:000
4046849992824,2cyclictest43910-21sendmail22:19:592
4046849992823,4cyclictest4069341-21kworker/u8:3+events_unbound20:56:262
4046849992816,10cyclictest4182953-21/usr/sbin/munin21:25:012
4046849992810,3cyclictest926952processes23:06:232
4046844992823,4cyclictest4069341-21kworker/u8:3+events_unbound20:56:301
4046844992822,4cyclictest3221998-21kworker/u8:1+flush-8:019:46:231
4046843992825,2cyclictest74547-21irqcore22:51:150
4046843992822,4cyclictest9060-21systemd-run21:45:010
4046843992818,8cyclictest93410-21/usr/sbin/munin23:10:000
4046849992724,2cyclictest113782-21fschecks_count23:31:092
4046849992723,3cyclictest8774-21kworker/u8:2+events_unbound22:36:232
4046849992723,2cyclictest168607-21cpuspeed_turbos00:26:112
4046849992722,4cyclictest28810-21kworker/u8:3+events_unbound23:26:282
4046849992721,4cyclictest4192456-21unixbench_multi21:31:262
4046844992721,4cyclictest4103262-21unixbench_multi20:01:301
4046843992723,3cyclictest4108553-21kworker/u8:2+events_unbound20:41:180
4046843992722,3cyclictest73317-21unixbench_singl22:46:290
4046843992720,5cyclictest133278-21apt-get23:51:070
4046843992720,5cyclictest133278-21apt-get23:51:070
740232260,7sleep3451irq_work/322:51:103
4046849992623,2cyclictest4143908-21fschecks_time20:46:132
4046849992623,2cyclictest4143908-21fschecks_time20:46:132
4046849992623,2cyclictest123467-21aten2_r5power_c23:41:112
4046849992621,3cyclictest122750-21unixbench_singl23:36:262
4046849992617,7cyclictest282-21systemd-journal23:20:002
4046844992622,3cyclictest93010-21kworker/u8:2+events_unbound23:26:151
4046844992622,2cyclictest63742-21sendmail22:40:001
4046844992618,6cyclictest4143843-21kworker/u8:0+events_unbound21:51:101
4046843992622,3cyclictest18665-21kworker/u8:1+flush-8:021:58:560
4046843992621,4cyclictest3221998-21kworker/u8:1+flush-8:019:51:120
4046843992620,4cyclictest4118087-21unixbench_singl20:16:270
4046843992620,4cyclictest4032036-21kworker/u8:0+flush-8:019:36:220
4046843992620,4cyclictest0-21swapper/021:00:000
4046849992522,2cyclictest4069371-21fschecks_count19:31:132
4046849992522,2cyclictest34405-21aten2_r5power_e22:11:082
4046849992521,2cyclictest158491-21cpuspeed_turbos00:16:102
4046849992521,2cyclictest128573-21cpuspeed_turbos23:46:112
4046849992521,2cyclictest128573-21cpuspeed_turbos23:46:112
4046843992521,3cyclictest4128871-21cpuspeed_turbos20:31:110
4046843992521,3cyclictest28810-21kworker/u8:3+events_unbound23:15:010
4046843992521,2cyclictest4163631-21cpuspeed21:06:110
4046843992521,2cyclictest38996-21systemd-run22:15:000
4046843992520,4cyclictest3221998-21kworker/u8:1+flush-8:019:46:130
4046843992520,2cyclictest0-21swapper/000:11:080
4046843992520,2cyclictest0-21swapper/000:11:080
4046843992519,4cyclictest30336-21kthreadcore22:06:190
4046843992518,5cyclictest282-21systemd-journal00:10:000
4046843992518,5cyclictest282-21systemd-journal00:10:000
4046843992517,7cyclictest142857-21kworker/0:1+events00:39:410
4046843992513,2cyclictest0-21swapper/022:41:120
404643522510,9sleep10-21swapper/119:07:201
404684999242,20cyclictest4083868-21awk19:45:002
4046849992421,2cyclictest4089359-21cpuspeed_turbos19:51:102
4046849992420,3cyclictest4098671-21idleruntime-cro20:00:002
4046849992420,2cyclictest173346-21aten2_r5power_e00:31:092
4046843992421,2cyclictest104369-21irqrtprio23:21:140
4046843992418,5cyclictest4108553-21kworker/u8:2+events_unbound20:36:200
4046843992418,5cyclictest4069341-21kworker/u8:3+flush-8:020:11:130
4046843992417,5cyclictest87740-21sshd23:01:250
4046843992417,5cyclictest28810-21kworker/u8:3+events_unbound22:21:200
4046843992415,6cyclictest172960-21latency_hist00:30:010
4046849992320,2cyclictest4123396-21cron20:25:002
4046849992320,2cyclictest14622-21fschecks_count21:51:132
4046849992320,2cyclictest133567-21fschecks_count23:51:112
4046849992320,2cyclictest133567-21fschecks_count23:51:112
4046849992314,7cyclictest3876-21awk21:40:002
4046849992313,8cyclictest4143255-21cat20:45:002
4046849992313,8cyclictest34022-21awk22:10:012
404684399235,17cyclictest4027167-21kworker/0:2+events22:31:260
4046843992319,3cyclictest4182568-21unixbench_multi21:21:280
4046843992318,3cyclictest0-21swapper/023:31:130
4046843992317,4cyclictest4062734-21ntpq19:21:200
4046843992317,4cyclictest39927-21irqcore22:16:170
404684399230,22cyclictest4027167-21kworker/0:2+events21:26:100
4046849992218,3cyclictest84184-21fschecks_time23:01:122
4046849992217,4cyclictest0-21swapper/220:30:002
4046849992217,3cyclictest4163132-21idleruntime-cro21:05:012
4046849992213,6cyclictest4108709-21sed20:10:012
404684399227,14cyclictest4027167-21kworker/0:2+events20:51:180
404684399227,14cyclictest4027167-21kworker/0:2+events20:51:170
404684399225,16cyclictest4027167-21kworker/0:2+events23:36:260
404684399224,17cyclictest4027167-21kworker/0:2+events20:06:220
4046843992218,2cyclictest0-21swapper/021:31:210
404684399220,21cyclictest4027167-21kworker/0:2+events20:01:100
4046849992113,6cyclictest4099421-21awk20:01:142
4046849992112,6cyclictest24313-21grep22:00:012
4046843992118,2cyclictest0-21swapper/020:21:110
4046843992116,4cyclictest143307-21kworker/u8:1+flush-8:000:01:290
4046843992116,3cyclictest127596-21timerwakeupswit23:41:300
4046843992116,3cyclictest0-21swapper/000:31:270
4046843992113,6cyclictest62942-21ntpq22:36:220
40465932214,9sleep20-21swapper/219:09:222
40425272213,12sleep00-21swapper/019:06:140
41325032200,2sleep2371rcuc/220:31:242
404684999208,8cyclictest73960-21apt-get22:51:082
404684999204,2cyclictest161rcu_preempt19:46:232
4046849992017,2cyclictest49544-21fschecks_time22:26:102
4046849992016,2cyclictest4059304-21fschecks_time19:21:112
4046849992016,2cyclictest0-21swapper/219:40:012
4046849992016,2cyclictest0-21swapper/200:00:012
4046849992012,6cyclictest83626-21cat23:00:012
4046849992012,6cyclictest4114198-21sort20:16:122
404684999192,12cyclictest4133610-21apt-get20:36:092
4046849991915,2cyclictest24678-21cpuspeed_turbos22:01:122
4046849991915,2cyclictest0-21swapper/222:21:282
404684999191,13cyclictest4173360-21apt-get21:16:092
4046849991910,7cyclictest4108911-21cut20:11:112
4046849991814,2cyclictest0-21swapper/200:36:232
4046849991813,4cyclictest13675-21unixbench_singl21:46:292
4046849991810,6cyclictest4169756-21cut21:11:172
404684999180,8cyclictest568511kthreadcore22:31:172
40420212183,8sleep30-21swapper/319:05:133
404684999172,11cyclictest148391-21apt-get00:06:092
404684999172,11cyclictest148391-21apt-get00:06:092
4046849991713,2cyclictest0-21swapper/219:15:032
4046849991713,2cyclictest0-21swapper/200:01:242
4046849991712,3cyclictest0-21swapper/221:26:162
404684999171,12cyclictest4163466-21apt-get21:06:102
4046849991710,5cyclictest157550-21tail00:11:282
4046849991710,5cyclictest157550-21tail00:11:282
41616802160,2sleep3471rcuc/321:01:183
404684999162,13cyclictest0-21swapper/223:23:212
404684399169,4cyclictest587-21runrttasks21:11:110
788692152,8sleep30-21swapper/322:56:093
41036112155,5sleep30-21swapper/320:05:013
4046849991511,2cyclictest0-21swapper/219:25:062
41733432142,8sleep30-21swapper/321:16:083
41421672142,8sleep30-21swapper/320:41:213
1819632142,8sleep30-21swapper/300:36:213
934292132,6sleep30-21swapper/323:10:013
901552132,6sleep30-21swapper/323:06:163
438722132,7sleep30-21swapper/322:20:013
41820632132,7sleep30-21swapper/321:21:223
41150822132,7sleep30-21swapper/320:16:173
41126482132,7sleep30-21swapper/320:11:233
40578192133,6sleep30-21swapper/319:16:253
28452132,7sleep30-21swapper/321:36:193
189222132,7sleep30-21swapper/321:55:013
1644092132,7sleep30-21swapper/300:21:173
1529832132,6sleep30-21swapper/300:10:013
1529832132,6sleep30-21swapper/300:10:003
1022812134,5sleep30-21swapper/323:16:193
838752122,6sleep30-21swapper/323:01:103
579832122,6sleep30-21swapper/322:31:253
442432122,6sleep30-21swapper/322:21:093
41636272122,6sleep30-21swapper/321:06:133
41535322122,6sleep30-21swapper/320:56:103
41475302122,6sleep30-21swapper/320:46:263
41475302122,6sleep30-21swapper/320:46:263
41335842122,6sleep30-21swapper/320:36:063
41283412122,6sleep30-21swapper/320:30:013
41246242122,6sleep30-21swapper/320:26:163
41046872122,6sleep30-21swapper/320:06:173
40934412122,6sleep30-21swapper/319:51:293
40846282122,6sleep30-21swapper/319:46:153
40835752122,6sleep30-21swapper/319:41:293
40786482121,7sleep30-21swapper/319:40:013
40687942122,6sleep30-21swapper/319:30:013
40628152122,6sleep30-21swapper/319:21:233
343202122,5sleep30-21swapper/322:11:063
297962122,6sleep30-21swapper/322:06:173
198482122,6sleep30-21swapper/321:56:133
1332502122,6sleep30-21swapper/323:51:073
1332502122,6sleep30-21swapper/323:51:073
1286902122,6sleep30-21swapper/323:46:133
1286902122,6sleep30-21swapper/323:46:133
1181642122,6sleep30-21swapper/323:35:013
1088752122,6sleep30-21swapper/323:26:143
1057832122,6sleep30-21swapper/323:21:173
93802112,5sleep30-21swapper/321:46:093
686562111,5sleep30-21swapper/322:45:013
643172112,5sleep30-21swapper/322:41:123
593052112,5sleep30-21swapper/322:36:093
491642112,6sleep30-21swapper/322:26:033
44152112,5sleep30-21swapper/321:41:083
41881882112,6sleep30-21swapper/321:30:523
41871342112,5sleep30-21swapper/321:26:243
41524312112,5sleep30-21swapper/320:51:243
41524312112,5sleep30-21swapper/320:51:243
41191662112,5sleep30-21swapper/320:21:133
40945252110,7sleep30-21swapper/319:56:143
40644862112,5sleep30-21swapper/319:26:123
40470342112,5sleep30-21swapper/319:11:073
1732052112,6sleep30-21swapper/300:30:243
1686372112,5sleep30-21swapper/300:26:143
1582312112,5sleep30-21swapper/300:16:073
1486352112,5sleep30-21swapper/300:06:123
1486352112,5sleep30-21swapper/300:06:123
1472712112,5sleep30-21swapper/300:01:263
1385812112,5sleep30-21swapper/323:56:123
1233252112,6sleep30-21swapper/323:40:023
1140012112,5sleep30-21swapper/323:31:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional