You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-17 - 05:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Mon Nov 17, 2025 00:39:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
758018995039,8cyclictest898444-21ssh23:35:011
758019994940,6cyclictest931391-21idleruntime-cro00:20:012
758018994739,5cyclictest815883-21sendmail21:40:011
758017994725,16cyclictest151ktimers/019:29:130
758017994725,16cyclictest151ktimers/019:29:130
758019994637,6cyclictest876894-21systemd-run23:05:012
758019994635,7cyclictest3046038-21cups-browsed23:53:592
758018994638,5cyclictest945549-21systemd-run00:40:021
758017994640,4cyclictest880419-21munin-plugin-st23:10:010
758019994535,6cyclictest869567-21systemd-run22:55:012
758019994534,7cyclictest421-21avahi-daemon23:44:102
758019994429,12cyclictest800730-21sh21:19:152
758019994429,12cyclictest800730-21sh21:19:152
758018994435,5cyclictest794004-21grep21:10:011
758019994234,5cyclictest894839-21munin-run23:30:002
758019994233,6cyclictest897207-21sh23:34:102
758019994233,6cyclictest266-21systemd-journal22:05:012
758019994233,5cyclictest815154-21proc_pri21:39:242
758018994233,6cyclictest883170-21ntpq23:14:161
758018994232,5cyclictest0-21swapper/100:19:171
758019994133,5cyclictest887695-21sendmail23:20:012
758019994133,5cyclictest854162-21needreboot22:34:102
758019994130,7cyclictest421-21avahi-daemon22:05:322
75801899415,29cyclictest836418-21ssh22:09:141
758018994132,6cyclictest833351-21unixbench_multi22:04:241
758018994131,6cyclictest780499-21cron20:25:011
758018994128,6cyclictest0-21swapper/122:24:141
758017994137,2cyclictest0-21swapper/023:00:010
758019994031,6cyclictest920419-21systemd-run00:05:012
758019994029,7cyclictest266-21systemd-journal22:12:532
758018994031,6cyclictest1-21systemd23:10:011
758018994026,7cyclictest0-21swapper/100:11:031
758017994034,4cyclictest942234-21systemd-run00:35:010
758017994030,5cyclictest151ktimers/022:39:090
758019993933,3cyclictest0-21swapper/223:00:012
758018993930,6cyclictest491-21gdbus22:49:191
758018993930,6cyclictest421-21avahi-daemon00:34:431
758017993919,14cyclictest151ktimers/019:49:160
758019993830,5cyclictest938146-21mailstats00:29:192
758019993828,5cyclictest0-21swapper/222:19:132
758019993816,3cyclictest171rcu_preempt21:49:092
75801999381,33cyclictest880442-21systemd-run23:10:012
758018993830,5cyclictest900685-21/usr/sbin/munin23:39:031
758018993830,5cyclictest873214-21plymouth23:00:011
758018993830,5cyclictest850834-21proc_pri22:29:161
758018993830,4cyclictest0-21swapper/123:04:161
758019993729,5cyclictest916842-21latency_hist00:00:032
758019993729,5cyclictest916842-21latency_hist00:00:032
758019993729,5cyclictest828919-21kernelversion21:59:092
758019993729,4cyclictest0-21swapper/200:24:192
758019993726,8cyclictest784322-21needreboot20:39:092
75801999371,18cyclictest36-21ksoftirqd/219:54:122
758018993731,4cyclictest0-21swapper/100:00:011
758018993731,4cyclictest0-21swapper/100:00:011
758018993731,2cyclictest0-21swapper/123:24:001
758018993729,5cyclictest766996-21munin-run19:40:011
758018993728,5cyclictest0-21swapper/100:09:121
758018993726,8cyclictest797427-21switchtime21:14:231
75801899371,29cyclictest822049-21ssh21:49:121
758017993731,4cyclictest944167-21sort00:39:050
758017993725,6cyclictest12450irq/32-eth023:29:200
75801799372,29cyclictest822123-21ssh21:49:160
758017993719,12cyclictest12450irq/32-eth022:14:200
758017993715,17cyclictest12450irq/32-eth023:14:140
758019993628,5cyclictest1-21systemd19:40:022
758019993628,4cyclictest0-21swapper/223:24:172
758019993620,5cyclictest171rcu_preempt21:09:112
758018993630,3cyclictest0-21swapper/122:54:101
758018993629,4cyclictest824975-21aten_r2power_fr21:53:591
758018993628,5cyclictest854670-21timerandwakeup22:34:241
758018993628,4cyclictest0-21swapper/123:54:091
758018993628,4cyclictest0-21swapper/122:14:181
758017993629,5cyclictest422-21dbus-daemon20:49:200
758017993628,5cyclictest3046038-21cups-browsed21:34:040
758017993628,5cyclictest3046038-21cups-browsed21:34:040
75801799362,32cyclictest0-21swapper/022:54:090
758017993614,19cyclictest807822-21ssh21:29:100
758017993614,19cyclictest807822-21ssh21:29:100
75801799360,31cyclictest931409-21grep00:20:010
758019993529,3cyclictest0-21swapper/222:40:012
758019993527,5cyclictest783519-21systemd-run20:35:012
758019993523,9cyclictest786504-21/usr/sbin/munin20:45:022
758019993523,8cyclictest944331-21kernelversion00:39:092
758018993530,2cyclictest0-21swapper/123:19:131
758018993528,4cyclictest828578-21aten_r2power_po21:59:031
758018993526,5cyclictest776033-21latency_hist20:10:021
758018993526,5cyclictest776033-21latency_hist20:10:021
758018993526,4cyclictest0-21swapper/123:44:101
758018993517,13cyclictest422-21dbus-daemon21:34:151
758018993517,13cyclictest422-21dbus-daemon21:34:151
758017993522,10cyclictest453-21systemd-logind19:10:010
75801999343,23cyclictest926795-21ssh00:14:122
758019993428,3cyclictest0-21swapper/200:35:012
758019993422,9cyclictest851634-21sh22:30:012
75801999342,25cyclictest797127-21ssh21:14:152
758019993420,5cyclictest865836-21kworker/u8:5+xprtiod22:50:022
758018993427,3cyclictest0-21swapper/121:24:161
758018993427,3cyclictest0-21swapper/121:24:161
758018993421,5cyclictest957-21mta-sts-daemon23:25:161
758017993429,2cyclictest0-21swapper/019:45:010
758017993427,5cyclictest421-21avahi-daemon19:24:160
758017993427,5cyclictest421-21avahi-daemon19:24:160
758017993427,4cyclictest825641-21ntp_kernel_pll_21:54:110
758017993421,10cyclictest912558-21grep23:54:220
758017993420,11cyclictest935067-21cron00:25:010
758017993420,11cyclictest772994-21systemctl20:00:010
758017993420,11cyclictest772994-21systemctl20:00:010
758019993325,5cyclictest861560-21ssh22:44:222
758019993325,5cyclictest861560-21ssh22:44:222
758019993324,6cyclictest421-21avahi-daemon23:50:012
758019993324,4cyclictest0-21swapper/223:39:092
758019993322,8cyclictest790979-21/usr/sbin/munin21:00:022
758018993327,3cyclictest0-21swapper/122:39:151
758018993326,3cyclictest0-21swapper/122:44:061
758018993326,3cyclictest0-21swapper/122:44:061
758018993325,5cyclictest801619-21basename21:20:011
758018993325,5cyclictest801619-21basename21:20:011
758018993325,5cyclictest453-21systemd-logind20:05:011
758018993325,4cyclictest789476-21cron20:55:011
758018993325,3cyclictest0-21swapper/122:19:181
75801799333,24cyclictest864976-21ssh22:49:120
758017993326,4cyclictest905344-21grep23:44:210
758017993326,4cyclictest781899-21unixbench_multi20:29:220
758017993326,4cyclictest781899-21unixbench_multi20:29:220
758017993324,6cyclictest3046038-21cups-browsed22:43:390
758017993324,6cyclictest3046038-21cups-browsed22:43:390
758017993321,9cyclictest794000-21diskmemload21:19:230
758017993321,9cyclictest794000-21diskmemload21:19:230
758017993320,10cyclictest875923-21ntp_kernel_pll_23:04:110
758017993319,9cyclictest266-21systemd-journal22:29:520
75801799331,27cyclictest926881-21ssh00:14:150
758019993225,4cyclictest779010-21awk20:20:012
758019993225,4cyclictest752833-21kworker/u8:8-nfsiod20:50:022
758019993224,5cyclictest923305-21grep00:09:182
758019993224,5cyclictest819451-21grep21:45:012
758019993224,5cyclictest788237-21/usr/sbin/munin20:54:102
75801999322,26cyclictest847103-21ssh22:24:092
758019993219,5cyclictest957-21mta-sts-daemon19:29:112
758019993219,5cyclictest957-21mta-sts-daemon19:29:112
758019993218,10cyclictest807799-21ntp_kernel_err21:29:102
758019993218,10cyclictest807799-21ntp_kernel_err21:29:102
758018993224,3cyclictest0-21swapper/121:29:141
758018993224,3cyclictest0-21swapper/121:29:141
758018993223,3cyclictest0-21swapper/100:04:141
758018993221,6cyclictest0-21swapper/119:19:131
758017993227,3cyclictest758016-21cyclictest00:09:140
758017993227,2cyclictest0-21swapper/020:55:020
758017993225,4cyclictest834745-21kworker/u8:16-rpciod22:22:530
758017993217,9cyclictest151ktimers/020:24:160
758017993217,11cyclictest151ktimers/023:24:090
758017993216,11cyclictest12450irq/32-eth023:39:110
758019993124,5cyclictest0-21swapper/219:45:012
758019993123,4cyclictest0-21swapper/221:34:122
758019993123,4cyclictest0-21swapper/221:34:122
758019993123,4cyclictest0-21swapper/221:24:132
758019993123,4cyclictest0-21swapper/221:24:132
75801999311,7cyclictest0-21swapper/219:34:122
75801999311,7cyclictest0-21swapper/219:34:122
75801799313,22cyclictest815110-21ssh21:39:230
758017993126,3cyclictest775902-21tail20:09:240
758017993126,3cyclictest775902-21tail20:09:240
758017993124,5cyclictest474-21in:imuxsock23:35:000
758017993124,3cyclictest0-21swapper/023:49:160
758017993114,13cyclictest919493-21ssh00:04:120
75801799311,24cyclictest3046013-21cupsd21:59:160
758019993022,5cyclictest759502-21munin-plugin-st19:15:012
758019993021,7cyclictest775765-21timerandwakeup20:09:242
758019993021,7cyclictest775765-21timerandwakeup20:09:242
758019993014,3cyclictest171rcu_preempt19:59:102
758019993014,3cyclictest171rcu_preempt19:59:102
758018993023,3cyclictest0-21swapper/100:24:151
758018993022,5cyclictest421-21avahi-daemon23:48:361
758018993022,5cyclictest0-21swapper/120:44:121
758017993024,4cyclictest887057-21switchtime23:19:230
758017993024,4cyclictest767004-21systemctl19:40:010
758017993022,4cyclictest0-21swapper/022:09:110
75801799302,22cyclictest843678-21ssh22:19:180
758017993020,6cyclictest0-21swapper/021:14:170
758019992922,4cyclictest0-21swapper/219:25:012
758019992922,4cyclictest0-21swapper/219:25:012
758019992921,5cyclictest3046038-21cups-browsed19:17:442
758019992919,3cyclictest171rcu_preempt21:53:592
758019992918,6cyclictest171rcu_preempt20:04:022
758019992912,4cyclictest171rcu_preempt20:24:112
75801999291,21cyclictest883187-21ssh23:14:172
758018992924,2cyclictest0-21swapper/120:20:011
758018992922,3cyclictest0-21swapper/121:42:451
758018992919,7cyclictest959-21mta-sts-daemon19:31:511
758018992919,7cyclictest959-21mta-sts-daemon19:31:511
758018992919,7cyclictest787894-21unixbench_multi20:49:211
758018992918,8cyclictest761909-21ntp_kernel_pll_19:24:081
758018992918,8cyclictest761909-21ntp_kernel_pll_19:24:081
758017992922,5cyclictest909314-21kworker/u8:1+xprtiod00:30:010
758017992922,5cyclictest0-21swapper/019:35:000
758017992922,5cyclictest0-21swapper/019:35:000
75801799292,23cyclictest832874-21ssh22:04:160
758019992822,3cyclictest0-21swapper/221:05:012
758019992816,3cyclictest171rcu_preempt19:49:102
75801999281,5cyclictest776933-21ntpq20:14:122
75801999281,5cyclictest776933-21ntpq20:14:122
758018992824,2cyclictest0-21swapper/121:00:011
75801899282,20cyclictest937974-21ssh00:29:141
758018992821,4cyclictest780987-21cpuspeed20:29:021
758018992821,4cyclictest780987-21cpuspeed20:29:021
758018992819,6cyclictest3046038-21cups-browsed20:39:141
758017992824,2cyclictest0-21swapper/019:19:020
758017992822,3cyclictest0-21swapper/022:35:020
758017992821,4cyclictest955-21mta-sts-daemon21:24:060
758017992821,4cyclictest955-21mta-sts-daemon21:24:060
758017992821,4cyclictest819612-21latency_hist21:45:020
758017992816,8cyclictest12450irq/32-eth023:59:140
758017992816,8cyclictest12450irq/32-eth023:59:140
758017992812,13cyclictest794042-21latency_hist21:10:020
758019992720,4cyclictest956-21mta-sts-daemon20:29:072
758019992720,4cyclictest956-21mta-sts-daemon20:29:072
758018992720,3cyclictest0-21swapper/119:59:161
758018992720,3cyclictest0-21swapper/119:59:161
758017992722,2cyclictest0-21swapper/020:45:020
758017992721,3cyclictest770475-21cpuspeed19:54:010
758017992720,4cyclictest792500-21sh21:05:010
758018992619,4cyclictest791701-21latency21:04:061
758018992519,3cyclictest0-21swapper/119:30:001
758018992519,3cyclictest0-21swapper/119:30:001
758018992517,5cyclictest955-21mta-sts-daemon19:15:021
75801799253,15cyclictest12450irq/32-eth020:04:110
758019992410,10cyclictest266-21systemd-journal19:10:022
758018992418,3cyclictest752833-21kworker/u8:8-xprtiod20:34:181
758018992418,3cyclictest0-21swapper/120:14:051
758018992418,3cyclictest0-21swapper/120:14:051
758017992417,5cyclictest790867-21unixbench_multi20:59:190
758017992413,8cyclictest0-21swapper/020:30:370
758018992318,2cyclictest0-21swapper/119:49:201
758017992314,6cyclictest957-21mta-sts-daemon19:10:490
75801799227,11cyclictest191rcuc/020:39:170
758017992216,3cyclictest151ktimers/020:14:130
758017992216,3cyclictest151ktimers/020:14:130
75801899219,8cyclictest261rcuc/119:44:191
758018992116,2cyclictest0-21swapper/119:54:101
758018992116,2cyclictest0-21swapper/119:10:021
758017992116,3cyclictest959-21mta-sts-daemon20:16:500
75802099182,7cyclictest0-21swapper/321:59:183
75802099173,8cyclictest0-21swapper/323:14:163
75802099172,8cyclictest0-21swapper/323:59:133
75802099172,8cyclictest0-21swapper/323:59:133
75802099172,8cyclictest0-21swapper/322:39:153
75802099171,8cyclictest0-21swapper/323:34:203
75802099162,8cyclictest0-21swapper/322:09:143
75802099161,8cyclictest0-21swapper/322:44:083
75802099161,8cyclictest0-21swapper/322:44:083
75802099152,8cyclictest0-21swapper/323:44:163
75802099152,8cyclictest0-21swapper/320:39:173
75802099152,8cyclictest0-21swapper/300:39:163
75802099152,7cyclictest0-21swapper/322:29:143
75802099152,7cyclictest0-21swapper/321:14:123
75802099152,6cyclictest0-21swapper/321:39:153
75802099152,6cyclictest0-21swapper/320:04:143
75802099152,6cyclictest0-21swapper/319:59:083
75802099152,6cyclictest0-21swapper/319:59:083
75802099151,8cyclictest0-21swapper/321:54:123
75802099151,7cyclictest0-21swapper/321:49:103
75802099151,7cyclictest0-21swapper/320:44:183
75802099151,7cyclictest0-21swapper/320:24:113
75802099151,7cyclictest0-21swapper/319:24:113
75802099151,7cyclictest0-21swapper/319:24:113
75802099151,7cyclictest0-21swapper/300:14:123
75802099151,6cyclictest0-21swapper/319:34:093
75802099151,6cyclictest0-21swapper/319:34:093
75802099143,7cyclictest0-21swapper/322:19:173
75802099142,7cyclictest0-21swapper/322:24:113
75802099142,7cyclictest0-21swapper/321:19:053
75802099142,7cyclictest0-21swapper/321:19:053
75802099142,7cyclictest0-21swapper/319:39:103
75802099142,7cyclictest0-21swapper/300:04:183
75802099142,6cyclictest0-21swapper/323:24:103
75802099142,5cyclictest0-21swapper/319:54:083
75802099142,5cyclictest0-21swapper/319:44:193
75802099142,5cyclictest0-21swapper/319:29:103
75802099142,5cyclictest0-21swapper/319:29:103
75802099141,9cyclictest0-21swapper/323:49:153
75802099141,8cyclictest0-21swapper/323:39:213
75802099141,8cyclictest0-21swapper/323:04:143
75802099141,8cyclictest0-21swapper/322:04:173
75802099141,7cyclictest0-21swapper/321:04:143
75802099141,6cyclictest0-21swapper/320:29:153
75802099141,6cyclictest0-21swapper/320:29:153
75802099141,6cyclictest0-21swapper/319:49:133
75802099135,6cyclictest0-21swapper/322:14:143
75802099133,7cyclictest0-21swapper/323:19:063
75802099133,7cyclictest0-21swapper/322:49:183
75802099133,6cyclictest0-21swapper/300:24:123
75802099133,5cyclictest0-21swapper/300:29:083
75802099132,7cyclictest0-21swapper/323:54:083
75802099132,6cyclictest0-21swapper/323:09:023
75802099132,6cyclictest0-21swapper/322:34:013
75802099132,6cyclictest0-21swapper/300:09:093
75802099132,5cyclictest0-21swapper/320:09:123
75802099132,5cyclictest0-21swapper/320:09:123
75802099131,8cyclictest0-21swapper/323:29:163
75802099131,8cyclictest0-21swapper/321:44:133
75802099131,7cyclictest0-21swapper/321:34:043
75802099131,7cyclictest0-21swapper/321:34:043
75802099131,6cyclictest0-21swapper/321:25:523
75802099131,6cyclictest0-21swapper/321:25:523
75802099131,6cyclictest0-21swapper/320:54:113
75802099131,6cyclictest0-21swapper/320:34:093
75802099131,6cyclictest0-21swapper/320:19:083
75802099131,6cyclictest0-21swapper/319:19:103
75802099122,6cyclictest0-21swapper/322:54:093
75802099122,6cyclictest0-21swapper/300:19:093
75802099122,4cyclictest0-21swapper/321:09:093
75802099121,6cyclictest0-21swapper/322:56:333
75802099121,6cyclictest0-21swapper/320:59:103
75802099121,5cyclictest0-21swapper/319:14:123
75802099101,6cyclictest0-21swapper/321:24:123
75802099101,6cyclictest0-21swapper/321:24:123
75802099101,6cyclictest0-21swapper/300:34:113
7580209991,5cyclictest0-21swapper/320:45:523
7580209981,4cyclictest0-21swapper/319:10:023
7580209981,3cyclictest0-21swapper/320:14:123
7580209981,3cyclictest0-21swapper/320:14:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional