You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 14:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Thu Nov 27, 2025 12:39:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
808519995730,5cyclictest985292-21idleruntime-cro12:35:012
808519995628,6cyclictest916772-21telnet10:59:052
808519995226,5cyclictest816003-21sed07:34:292
80851999504,40cyclictest810002-21/usr/sbin/munin07:15:022
808518995020,24cyclictest0-21swapper/112:34:161
808517995044,4cyclictest808516-21cyclictest07:44:160
80851799492,28cyclictest422-21dbus-daemon09:59:190
808519994837,6cyclictest0-21swapper/208:51:102
808519994739,5cyclictest954890-21cron11:55:012
80851899473,34cyclictest972241-21ssh12:19:151
808518994718,26cyclictest0-21swapper/112:14:171
808518994718,26cyclictest0-21swapper/112:14:171
80851899463,7cyclictest826005-21ntpq08:09:211
80851899462,14cyclictest437-21mta-sts-daemon12:04:151
808518994535,6cyclictest266-21systemd-journal08:05:011
80851899451,19cyclictest884447-21ssh10:09:181
808518994424,17cyclictest848527-21grep09:15:011
808518994420,20cyclictest951155-21threads11:49:321
808517994433,8cyclictest937940-21ssh11:30:010
808519994335,5cyclictest958336-21latency_hist12:00:022
808518994335,5cyclictest878499-21systemd-run10:00:021
808518994323,17cyclictest891520-21unixbench_multi10:19:281
808518994316,2cyclictest0-21swapper/111:59:171
808518994316,2cyclictest0-21swapper/110:54:151
808517994335,5cyclictest872081-21systemd-run09:50:010
808519994234,5cyclictest973261-21basename12:20:012
808519994234,5cyclictest891837-21sendmail10:20:012
808519994233,7cyclictest904833-21sh10:40:012
808519994233,7cyclictest904833-21sh10:40:012
808519994233,6cyclictest844556-21munin-plugin-st09:10:012
808519994233,4cyclictest0-21swapper/211:24:262
808519994231,8cyclictest927695-21idleruntime-cro11:15:002
808518994235,4cyclictest829601-21latency_hist08:20:021
808518994230,9cyclictest868732-21latency_hist09:45:021
808518994217,22cyclictest830965-21unixbench_multi08:24:271
808518994217,22cyclictest830965-21unixbench_multi08:24:271
808518994215,3cyclictest0-21swapper/111:54:161
808518994214,3cyclictest0-21swapper/110:14:281
808518994214,3cyclictest0-21swapper/109:29:231
808518994133,5cyclictest422-21dbus-daemon11:45:021
808518994125,13cyclictest0-21swapper/112:09:041
808518994120,18cyclictest0-21swapper/110:44:241
808518994120,18cyclictest0-21swapper/110:44:241
808518994118,4cyclictest1-21systemd10:50:011
808518994118,4cyclictest1-21systemd10:50:011
808518994118,20cyclictest0-21swapper/108:30:001
808518994116,22cyclictest0-21swapper/109:39:241
808518994114,2cyclictest0-21swapper/107:14:221
808517994134,5cyclictest887017-21aten_r2power_vo10:14:080
808517994134,5cyclictest453-21systemd-logind10:30:020
808517994129,9cyclictest968975-21unixbench_multi12:14:270
808517994129,9cyclictest968975-21unixbench_multi12:14:270
808517994128,9cyclictest981275-21sh12:30:010
808517994125,13cyclictest838776-40mandb08:51:100
808519994033,5cyclictest0-21swapper/208:19:052
808519994031,6cyclictest266-21systemd-journal12:05:022
808518994031,5cyclictest422-21dbus-daemon07:25:011
808518994023,13cyclictest0-21swapper/110:34:141
808518994015,2cyclictest0-21swapper/108:39:091
808518994014,3cyclictest0-21swapper/111:19:191
808518994014,3cyclictest0-21swapper/110:30:011
808518994013,6cyclictest940067-21/usr/sbin/munin11:34:081
808517994031,6cyclictest868068-21switchtime09:44:250
80851999394,27cyclictest937205-21grep11:29:242
808519993928,8cyclictest947614-21proc_pri11:44:232
80851899393,21cyclictest937075-21diskmemload11:29:201
808518993931,5cyclictest917993-21grep11:00:011
808518993916,2cyclictest0-21swapper/108:51:061
80851899391,13cyclictest894064-21ntpq10:24:181
80851899391,13cyclictest837949-21ntpq08:49:191
808517993921,15cyclictest474-21in:imuxsock10:03:050
808519993831,4cyclictest885211-21latency_hist10:10:022
808519993829,6cyclictest0-21swapper/209:40:012
808518993828,6cyclictest980884-21unixbench-2d12:29:311
80851899382,13cyclictest976261-21ssh12:24:131
808518993816,19cyclictest0-21swapper/110:40:011
808518993816,19cyclictest0-21swapper/110:40:011
808518993815,2cyclictest0-21swapper/109:34:321
80851899381,18cyclictest854197-21ssh09:24:151
80851899381,18cyclictest854197-21ssh09:24:151
80851899381,16cyclictest988414-21ssh12:39:171
80851899381,15cyclictest871228-21ssh09:49:181
808517993832,3cyclictest151ktimers/011:19:150
808517993830,6cyclictest873894-21/usr/sbin/munin09:54:340
808517993830,5cyclictest474-21in:imuxsock09:35:010
808517993824,11cyclictest0-21swapper/010:49:100
808517993824,11cyclictest0-21swapper/010:49:100
808519993730,4cyclictest882664-21kworker/u8:9+rpciod10:45:012
808519993730,4cyclictest882664-21kworker/u8:9+rpciod10:45:012
808519993729,5cyclictest814575-21systemd-run07:30:012
808519993729,5cyclictest814575-21systemd-run07:30:012
808519993728,5cyclictest474-21in:imuxsock12:05:032
808519993726,7cyclictest4168385-21cups-browsed10:49:102
808519993726,7cyclictest4168385-21cups-browsed10:49:102
808518993726,7cyclictest850544-21fschecks_count09:19:091
808518993718,16cyclictest0-21swapper/109:54:061
808518993714,20cyclictest0-21swapper/108:00:011
808517993731,4cyclictest934582-21latency_hist11:25:030
808517993730,4cyclictest973251-21idleruntime-cro12:20:010
808517993730,4cyclictest927692-21grep11:15:000
808517993729,5cyclictest987799-21wc12:39:050
808519993628,5cyclictest979648-21aten_r2power_en12:29:052
80851899363,8cyclictest881052-21ntpq10:04:151
80851899363,10cyclictest437-21mta-sts-daemon11:01:021
80851899363,10cyclictest437-21mta-sts-daemon11:01:021
808518993615,3cyclictest0-21swapper/111:14:251
808518993613,3cyclictest0-21swapper/111:05:401
808518993613,3cyclictest0-21swapper/111:05:401
808518993613,20cyclictest0-21swapper/109:04:061
808517993628,5cyclictest4168385-21cups-browsed11:41:560
808517993621,12cyclictest4168358-21cupsd08:39:160
80851899353,11cyclictest955-21mta-sts-daemon11:38:211
80851899351,27cyclictest933674-21ssh11:24:181
808517993527,5cyclictest421-21avahi-daemon11:53:450
808517993526,6cyclictest4168385-21cups-browsed10:20:590
808517993524,8cyclictest976894-21sed12:24:280
808517993522,10cyclictest0-21swapper/012:04:180
808517993520,9cyclictest854207-21ssh09:24:160
808517993520,9cyclictest854207-21ssh09:24:160
808519993428,3cyclictest0-21swapper/211:04:222
808519993428,3cyclictest0-21swapper/211:04:222
808519993426,5cyclictest875607-21latency_hist09:55:032
808519993426,5cyclictest837589-21fschecks_count08:49:122
808519993426,5cyclictest0-21swapper/211:10:002
808519993426,5cyclictest0-21swapper/211:10:002
808519993425,6cyclictest4168385-21cups-browsed10:53:252
808518993418,13cyclictest955-21mta-sts-daemon08:11:011
808517993428,4cyclictest906631-21aten_r2power_cu10:44:040
808517993428,4cyclictest906631-21aten_r2power_cu10:44:040
808517993427,4cyclictest951296-21unixbench-2d11:49:330
808517993427,4cyclictest851834-21ssh09:20:010
808517993426,5cyclictest4168385-21cups-browsed12:33:410
808517993421,10cyclictest904349-21timerandwakeup10:39:270
808517993421,10cyclictest904349-21timerandwakeup10:39:270
808517993414,17cyclictest657-21snmpd08:19:170
808519993328,2cyclictest0-21swapper/207:20:012
808519993325,5cyclictest867562-21wc09:44:112
808519993325,5cyclictest825065-21cut08:05:012
808519993324,3cyclictest0-21swapper/212:23:452
808518993326,3cyclictest0-21swapper/109:10:021
808517993327,4cyclictest816052-21idleruntime-cro07:35:000
808517993327,4cyclictest810017-21systemd-run07:15:010
808517993325,5cyclictest819395-21aten_r2power_fr07:49:060
808517993325,4cyclictest919648-21aten_r2power_cu11:04:060
808517993325,4cyclictest919648-21aten_r2power_cu11:04:060
808517993324,5cyclictest841706-21latency_hist09:00:030
808517993322,8cyclictest846948-21aten_r2power_cu09:14:040
808517993311,16cyclictest965124-21sort12:09:250
808517993310,14cyclictest151ktimers/009:09:140
808519993224,5cyclictest955-21mta-sts-daemon11:19:142
808519993224,5cyclictest0-21swapper/210:00:012
808519993223,6cyclictest826569-21true08:10:012
808519993223,6cyclictest819063-21munin-run07:45:012
808519993221,8cyclictest900349-21needreboot10:34:122
808519993221,7cyclictest857131-21gltestperf09:29:102
808518993224,5cyclictest841539-21grep09:00:011
808517993225,5cyclictest917888-21head10:59:400
808517993225,4cyclictest474-21in:imuxsock09:30:010
808517993224,5cyclictest817580-21sendmail-msp07:40:010
808517993223,7cyclictest838546-21/usr/sbin/munin08:50:020
808517993221,7cyclictest941357-21munin-plugin-st11:35:010
808517993220,9cyclictest830953-21unixbench_multi08:24:270
808517993220,9cyclictest830953-21unixbench_multi08:24:270
808519993125,3cyclictest0-21swapper/210:22:052
808519993125,3cyclictest0-21swapper/209:20:012
808519993124,4cyclictest988895-21unixbench-2d12:39:262
808519993123,5cyclictest843073-21systemd-run09:05:012
808518993123,5cyclictest1-21systemd08:35:021
808517993124,3cyclictest0-21swapper/010:54:150
808517993123,5cyclictest921412-21kworker/u8:13+xprtiod11:40:010
808517993122,5cyclictest0-21swapper/010:09:160
808517993119,9cyclictest865159-21systemd09:40:020
808519993023,4cyclictest957-21mta-sts-daemon09:34:442
808519993023,4cyclictest817537-21cron07:40:012
808519993023,3cyclictest0-21swapper/209:49:242
808518993022,5cyclictest813599-21fschecks_count07:29:091
808518993022,5cyclictest813599-21fschecks_count07:29:091
808517993022,6cyclictest890911-21ntp_kernel_pll_10:19:170
808517993018,4cyclictest0-21swapper/008:35:010
808519992923,4cyclictest959-21mta-sts-daemon11:33:182
808519992922,4cyclictest835924-21aten_r2power_po08:44:042
808519992922,3cyclictest0-21swapper/211:40:022
808519992922,3cyclictest0-21swapper/209:11:542
808519992921,5cyclictest835535-21cron08:40:012
808519992921,4cyclictest0-21swapper/210:14:172
808519992920,7cyclictest880799-21kernelversion10:04:102
808518992923,3cyclictest802036-21kworker/u8:3-xprtiod07:43:051
808517992922,4cyclictest959-21mta-sts-daemon11:56:430
808517992922,4cyclictest12550irq/33-eth010:34:180
80851799291,23cyclictest923409-21ssh11:09:160
80851799291,23cyclictest923409-21ssh11:09:160
808519992822,3cyclictest0-21swapper/209:24:262
808519992822,3cyclictest0-21swapper/209:24:262
808519992821,4cyclictest834047-21munin-run08:35:012
808519992815,10cyclictest821446-21sshd07:54:182
808517992820,5cyclictest0-21swapper/007:20:010
808517992820,4cyclictest0-21swapper/008:30:010
808517992813,5cyclictest0-21swapper/007:29:170
808517992813,5cyclictest0-21swapper/007:29:170
80851999272,19cyclictest897113-21ssh10:29:172
808519992718,6cyclictest819615-21gltestperf07:49:112
808518992723,2cyclictest0-21swapper/107:20:011
808519992621,2cyclictest0-21swapper/211:50:022
808519992621,2cyclictest0-21swapper/207:25:012
808519992619,4cyclictest959-21mta-sts-daemon12:14:212
808519992619,4cyclictest959-21mta-sts-daemon12:14:212
808519992619,4cyclictest841542-21sendmail09:00:012
808519992618,5cyclictest823671-21latency_hist08:00:022
808519992617,6cyclictest4168385-21cups-browsed08:21:052
808519992617,6cyclictest4168385-21cups-browsed08:21:052
808517992621,3cyclictest151ktimers/007:54:190
808517992621,2cyclictest0-21swapper/008:44:080
808517992618,5cyclictest824986-21unixbench_multi08:04:240
808517992615,8cyclictest825948-21ntp_kernel_pll_08:09:190
808518992516,5cyclictest4168385-21cups-browsed08:44:161
808517992515,7cyclictest842477-21ntp_kernel_pll_09:04:140
808518992418,3cyclictest816071-21kworker/u8:4-nfsiod07:35:011
808518992418,3cyclictest0-21swapper/107:40:021
808518992416,4cyclictest0-21swapper/107:49:141
808517992419,2cyclictest0-21swapper/007:59:150
808517992415,7cyclictest422-21dbus-daemon08:15:020
808517992411,9cyclictest151ktimers/007:24:190
808519992317,3cyclictest826936-21sh08:14:042
808519992314,6cyclictest831567-21/usr/sbin/munin08:29:052
808518992314,6cyclictest821215-21uname07:54:131
808519992116,2cyclictest0-21swapper/207:10:012
80852099194,6cyclictest0-21swapper/307:49:213
80852099193,9cyclictest0-21swapper/310:54:143
80852099193,8cyclictest0-21swapper/311:34:243
80852099192,8cyclictest0-21swapper/312:04:073
808518991813,2cyclictest0-21swapper/107:10:021
80852099173,8cyclictest0-21swapper/312:34:183
80852099172,9cyclictest0-21swapper/311:29:243
80852099172,9cyclictest0-21swapper/311:24:193
80852099163,8cyclictest0-21swapper/308:51:173
80852099163,7cyclictest0-21swapper/310:34:143
80852099163,6cyclictest0-21swapper/312:29:173
80852099162,9cyclictest0-21swapper/312:24:033
80852099162,9cyclictest0-21swapper/311:19:193
80852099162,8cyclictest0-21swapper/309:54:053
80852099162,7cyclictest0-21swapper/310:39:173
80852099162,7cyclictest0-21swapper/310:39:173
80852099161,8cyclictest0-21swapper/311:59:263
80852099161,8cyclictest0-21swapper/309:49:163
80852099161,8cyclictest0-21swapper/309:40:013
80851799161,13cyclictest808577-21latency_hist07:10:020
80852099154,4cyclictest0-21swapper/307:29:153
80852099154,4cyclictest0-21swapper/307:29:153
80852099153,7cyclictest0-21swapper/309:14:173
80852099152,9cyclictest0-21swapper/312:17:043
80852099152,8cyclictest0-21swapper/312:39:133
80852099152,8cyclictest0-21swapper/312:09:163
80852099152,8cyclictest0-21swapper/310:14:233
80852099152,8cyclictest0-21swapper/309:10:013
80852099152,7cyclictest0-21swapper/310:09:163
80852099151,8cyclictest0-21swapper/312:14:163
80852099151,8cyclictest0-21swapper/312:14:163
80852099151,8cyclictest0-21swapper/310:49:203
80852099151,8cyclictest0-21swapper/310:49:203
80852099151,8cyclictest0-21swapper/309:44:073
80852099151,7cyclictest0-21swapper/310:59:153
80852099151,7cyclictest0-21swapper/309:24:083
80852099151,7cyclictest0-21swapper/309:24:083
80852099144,5cyclictest0-21swapper/311:14:133
80852099143,7cyclictest0-21swapper/311:49:193
80852099142,8cyclictest0-21swapper/310:04:143
80852099142,7cyclictest0-21swapper/310:24:143
80852099142,7cyclictest0-21swapper/309:59:153
80852099142,6cyclictest0-21swapper/307:24:173
80852099141,8cyclictest0-21swapper/311:44:073
80852099141,7cyclictest0-21swapper/311:09:073
80852099141,7cyclictest0-21swapper/311:09:073
80852099141,7cyclictest0-21swapper/308:59:143
80852099141,7cyclictest0-21swapper/308:39:143
80852099141,6cyclictest0-21swapper/308:24:183
80852099141,6cyclictest0-21swapper/308:24:183
80852099141,6cyclictest0-21swapper/307:39:133
80852099141,6cyclictest0-21swapper/307:14:163
80852099135,6cyclictest0-21swapper/311:39:173
80852099132,7cyclictest0-21swapper/311:04:263
80852099132,7cyclictest0-21swapper/311:04:263
80852099132,7cyclictest0-21swapper/309:34:203
80852099132,7cyclictest0-21swapper/309:29:183
80852099132,6cyclictest0-21swapper/310:19:153
80852099132,6cyclictest0-21swapper/307:54:183
80852099131,8cyclictest0-21swapper/311:54:153
80852099131,6cyclictest0-21swapper/308:49:263
80852099131,6cyclictest0-21swapper/308:29:133
80852099131,5cyclictest0-21swapper/307:44:153
80852099124,5cyclictest0-21swapper/310:44:193
80852099124,5cyclictest0-21swapper/310:44:193
80852099122,7cyclictest0-21swapper/309:19:063
80852099122,6cyclictest0-21swapper/308:20:013
80852099121,7cyclictest0-21swapper/308:34:193
80852099121,6cyclictest0-21swapper/308:44:133
80852099121,6cyclictest0-21swapper/308:09:353
80852099121,6cyclictest0-21swapper/307:59:213
80852099111,7cyclictest0-21swapper/310:29:113
80852099111,6cyclictest0-21swapper/308:04:173
80852099101,5cyclictest0-21swapper/309:04:103
8085209995,3cyclictest0-21swapper/307:34:033
8085209993,5cyclictest0-21swapper/307:10:013
8085209993,4cyclictest0-21swapper/307:15:033
8085209991,6cyclictest0-21swapper/308:14:043
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional