You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 09:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Tue Nov 25, 2025 00:39:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21046995545,6cyclictest83933-21ssh21:45:021
21047995341,9cyclictest174093-21mailstats23:44:252
21046995042,3cyclictest0-21swapper/123:04:171
21046995041,5cyclictest42879-21/usr/sbin/munin20:24:121
21046995026,6cyclictest171rcu_preempt19:59:141
21047994738,5cyclictest186160-21kthreadd00:00:022
21046994738,6cyclictest46536-21systemd-run20:35:011
21045994738,7cyclictest955-21mta-sts-daemon22:29:040
21046994623,7cyclictest171rcu_preempt19:34:171
21047994538,3cyclictest0-21swapper/223:40:012
21047994515,5cyclictest171rcu_preempt22:44:212
21046994535,6cyclictest196777-21timerandwakeup00:14:261
21045994533,9cyclictest12450irq/32-eth019:19:120
21045994533,9cyclictest12450irq/32-eth019:19:120
21045994533,9cyclictest122021-21cut22:35:010
21045994526,16cyclictest12550irq/33-eth022:49:160
21047994338,2cyclictest0-21swapper/200:30:012
21047994335,5cyclictest24049-21sendmail19:20:002
21047994335,5cyclictest24049-21sendmail19:20:002
2104699436,31cyclictest166231-21ssh23:34:201
21046994333,7cyclictest30579-21fschecks_count19:44:081
21045994336,4cyclictest103054-21cut22:10:010
21045994330,10cyclictest71277-21missed_timers21:29:100
21047994231,8cyclictest125862-21sh22:40:012
21047994231,7cyclictest2319198-21cups-browsed22:14:262
21046994234,5cyclictest97809-21fschecks_count22:04:061
21046994234,5cyclictest60754-21head21:14:271
21046994234,5cyclictest118504-21latency_hist22:30:021
21047994133,5cyclictest140588-21ssh23:00:012
21047994133,5cyclictest133168-21diskmemload22:50:012
21047994133,4cyclictest0-21swapper/223:29:192
21047994132,6cyclictest421-21avahi-daemon00:39:042
21047994132,6cyclictest2319198-21cups-browsed22:29:082
21046994134,5cyclictest125421-21unixbench-2d22:39:251
21046994133,5cyclictest591-21sshd22:17:131
21046994133,5cyclictest45031-21grep20:30:011
21046994131,7cyclictest106292-21switchtime22:14:241
21045994134,5cyclictest30065-21systemd-run19:40:020
2104799404,29cyclictest60268-21ssh21:14:202
21047994032,5cyclictest200811-21sendmail-msp00:20:012
21047994032,5cyclictest136768-21munin-plugin-st22:55:012
21047994031,6cyclictest421-21avahi-daemon23:24:042
21047994030,7cyclictest957-21mta-sts-daemon23:49:192
21046994031,5cyclictest957-21mta-sts-daemon21:55:011
21046994030,5cyclictest957-21mta-sts-daemon00:29:171
21047993933,3cyclictest0-21swapper/223:19:182
21047993931,5cyclictest266-21systemd-journal21:30:012
21046993931,5cyclictest95340-21grep22:00:001
21046993931,5cyclictest95340-21grep22:00:001
21046993930,5cyclictest959-21mta-sts-daemon23:19:331
21046993929,7cyclictest114225-21grep22:24:251
21045993933,4cyclictest186132-21kworker/u8:11-xprtiod00:00:020
21045993926,10cyclictest91178-21unixbench_singl21:54:240
21047993831,4cyclictest0-21swapper/222:24:062
21047993830,5cyclictest49543-21systemd-run20:45:012
21047993830,5cyclictest31557-21systemd-run19:45:022
21047993830,5cyclictest167147-21latency_hist23:35:022
21047993828,6cyclictest1-21systemd19:55:022
21046993831,4cyclictest147749-21unixbench_multi23:09:241
21046993830,5cyclictest21576-21fschecks_time19:14:071
21046993830,5cyclictest186138-21systemd-run00:00:021
21046993829,7cyclictest57047-21sh21:10:011
21046993829,7cyclictest47346-21needreboot20:39:101
21046993829,6cyclictest1-21systemd21:30:021
21046993828,7cyclictest2319198-21cups-browsed23:27:121
21045993832,4cyclictest21044-21cyclictest00:28:200
21045993832,3cyclictest0-21swapper/023:49:150
21045993831,4cyclictest113811-21proc_pri22:24:190
21045993829,7cyclictest51556-21fschecks_count20:54:060
21045993829,4cyclictest0-21swapper/023:19:180
21047993730,4cyclictest0-21swapper/221:09:122
21047993728,5cyclictest143417-21ntp_kernel_pll_23:04:182
21046993729,5cyclictest422-21dbus-daemon00:05:011
21046993729,5cyclictest1-21systemd21:50:021
21045993731,4cyclictest106906-21systemd-run22:15:010
21045993730,4cyclictest42060-21systemd-run20:20:010
21045993729,5cyclictest211090-21latency_hist00:35:020
21045993729,5cyclictest200816-21systemd-run00:20:010
2104599372,28cyclictest181408-21ssh23:54:160
21047993628,5cyclictest110752-21sendmail22:20:012
2104799362,26cyclictest196476-21ssh00:14:212
21046993628,5cyclictest80042-21basename21:40:011
21046993628,5cyclictest170926-21ssh23:40:011
21046993628,4cyclictest40538-21kworker/u8:020:15:011
21046993628,4cyclictest40538-21kworker/u8:020:15:011
21046993627,5cyclictest957-21mta-sts-daemon00:34:321
21046993626,5cyclictest0-21swapper/123:44:211
21045993629,4cyclictest1-21systemd19:35:020
21045993628,5cyclictest2319198-21cups-browsed19:54:160
21047993528,3cyclictest0-21swapper/222:34:132
21047993527,5cyclictest204217-21munin-plugin-st00:25:012
21047993525,8cyclictest0-21swapper/223:09:152
21046993527,5cyclictest186163-21cups-browsed00:06:141
21046993527,5cyclictest140546-21grep23:00:011
2104699352,25cyclictest67693-21ssh21:24:141
2104599353,25cyclictest60249-21ssh21:14:190
21045993529,4cyclictest65050-21latency_hist21:20:030
21045993528,4cyclictest99201-21munin-plugin-st22:05:010
21045993528,4cyclictest80037-21munin-plugin-st21:40:010
21045993527,6cyclictest2319198-21cups-browsed22:40:010
21045993527,5cyclictest2319198-21cups-browsed23:12:530
21045993526,6cyclictest266-21systemd-journal20:24:120
21045993526,5cyclictest189366-21threads00:04:260
21045993522,10cyclictest157818-21/usr/sbin/munin23:24:070
2104799346,5cyclictest171rcu_preempt20:54:182
21047993427,3cyclictest0-21swapper/221:44:132
21047993426,5cyclictest74566-21sh21:34:042
2104799342,5cyclictest171rcu_preempt20:09:182
21046993427,4cyclictest178204-21unixbench_singl23:49:251
21046993426,5cyclictest159186-21head23:24:271
21045993428,4cyclictest129521-21munin-run22:45:010
21045993428,3cyclictest118405-21kworker/u8:0-xprtiod23:30:010
21045993427,4cyclictest36591-21fschecks_time20:04:080
21045993426,5cyclictest486-21gmain22:54:260
21045993426,5cyclictest474-21in:imuxsock00:05:130
21045993424,7cyclictest110204-21threads22:19:250
21045993418,12cyclictest12450irq/32-eth021:59:160
21045993418,12cyclictest12450irq/32-eth021:59:160
21045993410,4cyclictest0-21swapper/020:59:180
21047993324,4cyclictest0-21swapper/221:54:142
21047993324,4cyclictest0-21swapper/221:24:202
2104799332,26cyclictest94395-21ssh21:59:142
2104799332,26cyclictest94395-21ssh21:59:142
21046993323,6cyclictest181249-21needreboot23:54:111
21046993322,8cyclictest135735-21needreboot22:54:101
2104599334,10cyclictest174784-21ssh23:45:020
21045993324,6cyclictest955-21mta-sts-daemon23:04:320
21045993324,5cyclictest2319198-21cups-browsed21:23:160
21045993319,10cyclictest151ktimers/019:24:130
21047993225,5cyclictest0-21swapper/221:20:012
21047993225,3cyclictest0-21swapper/200:04:222
21047993220,9cyclictest41889-21unixbench-2d20:19:272
21047993218,9cyclictest35457-21awk19:59:142
21046993225,5cyclictest121483-21timerandwakeup22:34:231
21046993223,6cyclictest64829-21munin-run21:20:011
21046993223,5cyclictest213872-21unixbench_multi00:39:271
21046993222,7cyclictest51845-21needreboot20:54:101
21046993221,8cyclictest202862-21sort00:24:071
21045993226,4cyclictest46542-21systemd-run20:35:020
21045993226,3cyclictest200814-21kworker/u8:9-events_unbound00:24:120
21045993225,5cyclictest12450irq/32-eth023:34:160
21045993224,5cyclictest147188-21ssh23:09:130
21045993220,4cyclictest27010-21unixbench_singl19:29:250
2104799317,13cyclictest351ktimers/219:49:172
21047993124,5cyclictest0-21swapper/221:40:012
21047993123,5cyclictest956-21mta-sts-daemon20:49:202
21047993123,3cyclictest0-21swapper/221:50:022
21047993122,5cyclictest209644-21aten2.4-expect00:34:072
21047993121,5cyclictest0-21swapper/220:59:132
21047993120,5cyclictest0-21swapper/222:09:182
21046993125,3cyclictest0-21swapper/122:49:261
21045993121,6cyclictest74936-21if_wlan021:34:090
21045993116,12cyclictest0-21swapper/021:44:260
2104599311,4cyclictest171rcu_preempt21:49:170
21047993024,3cyclictest0-21swapper/219:35:022
21047993023,4cyclictest30022-21cron19:40:012
21047993023,4cyclictest0-21swapper/220:40:012
21047993023,3cyclictest0-21swapper/223:54:172
2104799301,4cyclictest171rcu_preempt22:02:002
21046993024,3cyclictest955-21mta-sts-daemon23:11:311
21046993024,3cyclictest0-21swapper/121:34:251
21046993024,2cyclictest0-21swapper/121:00:011
21046993022,5cyclictest33568-21fschecks_time19:54:081
2104699302,24cyclictest38423-21sshd20:09:121
21046993020,7cyclictest23057-21entropy19:19:061
21046993020,7cyclictest23057-21entropy19:19:061
2104599309,17cyclictest169931-21ssh23:39:150
21045993025,2cyclictest0-21swapper/023:00:010
21045993022,6cyclictest491-21gdbus20:09:080
21045993022,5cyclictest49549-21systemd-run20:45:020
21045993021,7cyclictest56355-21needreboot21:09:100
21045993021,7cyclictest266-21systemd-journal19:15:020
21045993017,5cyclictest0-21swapper/020:14:190
21045993017,5cyclictest0-21swapper/020:14:190
21046992922,4cyclictest957-21mta-sts-daemon22:40:351
21046992922,4cyclictest42016-21cron20:20:001
21046992922,4cyclictest0-21swapper/119:40:011
21045992924,3cyclictest196927-21unixbench_multi00:14:270
21047992819,7cyclictest190587-21sh00:06:022
21046992822,3cyclictest0-21swapper/100:19:191
2104599283,6cyclictest0-21swapper/019:49:170
21045992812,10cyclictest12450irq/32-eth019:44:130
21047992720,3cyclictest0-21swapper/223:14:162
21047992719,5cyclictest453-21systemd-logind20:30:022
21047992719,5cyclictest37080-21proc_pri20:04:212
21047992715,5cyclictest0-21swapper/219:24:132
21046992722,3cyclictest37534-21debian-sa120:05:011
2104699272,20cyclictest102067-21ssh22:09:121
21046992718,7cyclictest32209-21kernelversion19:49:081
21045992722,3cyclictest213866-21grep00:39:270
21045992721,4cyclictest2319198-21cups-browsed19:59:250
21047992621,2cyclictest0-21swapper/220:35:012
21047992619,5cyclictest0-21swapper/220:24:262
21047992619,4cyclictest25765-21kworker/u8:3-xprtiod20:15:002
21047992619,4cyclictest25765-21kworker/u8:3-xprtiod20:15:002
21047992618,5cyclictest22124-21systemctl19:14:212
21045992616,7cyclictest47378-21ntp_kernel_err20:39:110
2104899252,15cyclictest0-21swapper/322:09:173
21047992517,5cyclictest55528-21debian-sa121:05:012
21046992519,3cyclictest0-21swapper/119:10:021
21046992519,3cyclictest0-21swapper/119:10:021
21046992518,4cyclictest956-21mta-sts-daemon20:46:501
21046992516,6cyclictest959-21mta-sts-daemon20:40:091
2104599257,14cyclictest50393-21sed20:49:150
2104599254,3cyclictest0-21swapper/020:29:160
21045992510,12cyclictest151ktimers/021:04:120
21047992415,6cyclictest25959-21aten_r2power_vo19:29:042
21046992419,2cyclictest0-21swapper/119:30:021
21046992417,4cyclictest55387-21unixbench-2d21:04:231
2104699232,16cyclictest25121-21ps19:24:191
2104899223,11cyclictest0-21swapper/323:54:153
2104899223,11cyclictest0-21swapper/323:04:223
2104899223,11cyclictest0-21swapper/321:19:153
2104899222,11cyclictest0-21swapper/319:19:123
2104899222,11cyclictest0-21swapper/319:19:123
2104899222,11cyclictest0-21swapper/300:04:163
2104899214,9cyclictest0-21swapper/322:34:153
2104899213,8cyclictest0-21swapper/323:14:173
2104899213,12cyclictest0-21swapper/321:14:153
2104899213,12cyclictest0-21swapper/300:14:143
2104899213,11cyclictest0-21swapper/321:29:123
21047992116,2cyclictest0-21swapper/219:10:012
21047992116,2cyclictest0-21swapper/219:10:012
2104899204,9cyclictest0-21swapper/321:39:203
2104899204,8cyclictest0-21swapper/319:49:123
2104899203,10cyclictest0-21swapper/300:19:163
2104899202,11cyclictest0-21swapper/323:09:143
2104899202,10cyclictest0-21swapper/321:24:133
2104899193,9cyclictest0-21swapper/320:04:143
2104899193,9cyclictest0-21swapper/300:39:183
2104899192,9cyclictest0-21swapper/322:24:133
2104899192,10cyclictest0-21swapper/300:29:213
2104899192,10cyclictest0-21swapper/300:24:183
2104899191,11cyclictest0-21swapper/323:19:203
2104899184,10cyclictest0-21swapper/322:29:143
2104899183,9cyclictest0-21swapper/323:49:153
2104899183,9cyclictest0-21swapper/323:34:153
2104899183,9cyclictest0-21swapper/321:44:143
2104899183,7cyclictest0-21swapper/319:39:143
2104899183,10cyclictest0-21swapper/322:14:153
2104899183,10cyclictest0-21swapper/321:09:123
2104899182,9cyclictest0-21swapper/320:44:153
2104899182,8cyclictest0-21swapper/320:29:153
2104899182,11cyclictest0-21swapper/321:49:173
2104899182,10cyclictest0-21swapper/323:59:133
2104899182,10cyclictest0-21swapper/322:49:143
2104599180,15cyclictest21065-21latency_hist19:10:020
2104599180,15cyclictest21065-21latency_hist19:10:020
2104899174,7cyclictest0-21swapper/322:54:113
2104899173,9cyclictest0-21swapper/323:39:143
2104899173,9cyclictest0-21swapper/300:34:123
2104899173,8cyclictest0-21swapper/322:44:033
2104899172,8cyclictest0-21swapper/322:39:123
2104899172,10cyclictest0-21swapper/320:59:173
2104899171,9cyclictest0-21swapper/323:24:103
2104899164,8cyclictest0-21swapper/322:04:113
2104899163,9cyclictest0-21swapper/323:44:143
2104899163,9cyclictest0-21swapper/322:19:153
2104899163,9cyclictest0-21swapper/321:59:073
2104899163,9cyclictest0-21swapper/321:59:073
2104899163,7cyclictest0-21swapper/320:19:153
2104899163,7cyclictest0-21swapper/320:14:143
2104899163,7cyclictest0-21swapper/320:14:143
2104899163,7cyclictest0-21swapper/319:34:133
2104899162,9cyclictest0-21swapper/321:54:123
2104899162,8cyclictest0-21swapper/322:57:123
2104899162,8cyclictest0-21swapper/319:14:123
2104899162,8cyclictest0-21swapper/300:09:173
2104899162,10cyclictest0-21swapper/323:29:123
2104899153,9cyclictest0-21swapper/321:34:153
2104899152,8cyclictest0-21swapper/320:54:123
2104899152,8cyclictest0-21swapper/319:59:113
2104899152,8cyclictest0-21swapper/319:29:133
2104899152,7cyclictest0-21swapper/320:34:053
2104899151,8cyclictest0-21swapper/319:44:133
2104899151,8cyclictest0-21swapper/319:24:133
2104899143,7cyclictest0-21swapper/319:54:193
2104899142,7cyclictest0-21swapper/320:09:123
2104899141,9cyclictest0-21swapper/321:04:103
2104899141,8cyclictest0-21swapper/320:39:153
2104899132,8cyclictest0-21swapper/320:49:153
2104899131,7cyclictest0-21swapper/320:24:183
2104899102,5cyclictest0-21swapper/319:10:013
2104899102,5cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional