You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-01 - 16:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Tue Sep 01, 2026 12:42:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
481587995243,6cyclictest554366-21latency_hist10:10:021
481587995234,10cyclictest60869550kworker/u8:8+xprtiod11:40:011
481586994526,15cyclictest508655-21latency_hist08:40:020
481588994336,4cyclictest526806-21cut09:25:012
481588994234,5cyclictest627289-21systemd-run12:10:022
481588994234,5cyclictest627289-21systemd-run12:10:022
481588994230,10cyclictest536001-21sendmail09:40:012
481588994230,10cyclictest536001-21sendmail09:40:012
481586994233,6cyclictest530378-21/usr/sbin/munin09:30:540
481587994133,5cyclictest560389-21ssh10:20:021
481587994133,5cyclictest530386-21aten_r2power_cu09:30:541
481587994132,6cyclictest1-21systemd12:15:011
481587994132,6cyclictest1-21systemd12:15:011
481586994131,7cyclictest572479-21/usr/sbin/munin10:40:020
481587994032,5cyclictest572487-21systemctl10:40:011
481586994032,5cyclictest582548-21sshd10:56:040
481586994025,10cyclictest12650irq/32-eth012:16:050
481587993933,3cyclictest0-21swapper/108:40:021
481587993931,5cyclictest1-21systemd08:55:021
481587993930,5cyclictest549120-21cron10:01:001
481587993929,7cyclictest641078-21timerandwakeup12:31:181
481588993830,5cyclictest507126-21systemd-run08:35:022
481588993829,6cyclictest567-21gdbus09:01:102
48158899382,28cyclictest640699-21ssh12:31:072
481587993831,3cyclictest0-21swapper/108:06:031
481587993830,5cyclictest525596-21unixbench_singl09:21:211
481587993830,5cyclictest421-21dbus-daemon11:31:061
481586993829,6cyclictest563410-21unin-run10:25:020
481586993829,6cyclictest551227-21idleruntime-cro10:05:010
481587993730,5cyclictest637596-21sshd12:26:041
481587993729,5cyclictest536830-21gltestperf09:40:571
481587993729,5cyclictest536830-21gltestperf09:40:571
481587993729,5cyclictest523819-21latency_hist09:20:021
481587993727,6cyclictest420-21avahi-daemon12:00:071
481586993728,7cyclictest599874-21munin-plugin-st11:25:000
481586993726,9cyclictest0-21swapper/007:11:150
481588993628,5cyclictest551245-21systemd-run10:05:012
481588993628,5cyclictest548145-21cron10:00:002
481588993627,6cyclictest619977-21unixbench_singl11:56:202
481587993629,4cyclictest545114-21grep09:55:001
481587993629,4cyclictest545114-21grep09:55:001
481587993628,5cyclictest490456-21unixbench_multi07:36:211
481587993628,5cyclictest490456-21unixbench_multi07:36:211
481587993627,6cyclictest484632-21latency_hist07:20:021
481587993627,6cyclictest3847959-21cupsd11:23:011
481586993631,3cyclictest0-21swapper/009:05:010
481586993625,8cyclictest601617-21unixbench-2d11:26:170
481588993529,3cyclictest0-21swapper/211:10:012
481588993528,3cyclictest0-21swapper/207:40:002
481588993528,3cyclictest0-21swapper/207:40:002
481588993527,5cyclictest575571-21systemd10:45:022
481588993527,4cyclictest0-21swapper/211:11:052
481588993524,7cyclictest420-21avahi-daemon09:50:122
481588993524,7cyclictest420-21avahi-daemon09:50:122
481587993528,4cyclictest562128-21grep10:21:201
481587993528,4cyclictest517603-21diskmemload10:31:141
481587993528,3cyclictest0-21swapper/110:49:241
481587993527,5cyclictest529855-21ls09:30:011
481587993525,7cyclictest420-21avahi-daemon08:16:091
481586993529,3cyclictest0-21swapper/009:16:160
481586993521,9cyclictest517603-21diskmemload11:11:100
481586993516,15cyclictest12750irq/33-eth010:16:040
481588993428,3cyclictest0-21swapper/212:20:012
481588993427,4cyclictest537764-21unixbench_multi09:41:162
481588993427,4cyclictest537764-21unixbench_multi09:41:162
481588993426,4cyclictest0-21swapper/211:21:122
481587993425,6cyclictest439-21systemd-logind08:00:021
481587993425,6cyclictest1-21systemd12:20:011
481586993429,3cyclictest1003-21mta-sts-daemon12:40:020
481586993429,3cyclictest1003-21mta-sts-daemon12:40:020
481586993428,3cyclictest0-21swapper/011:35:010
481586993427,4cyclictest590740-21ssh11:10:010
481586993425,7cyclictest639539-21latency_hist12:30:020
48158699341,28cyclictest151ktimers/011:56:130
481588993327,4cyclictest481585-21cyclictest10:51:142
481588993325,5cyclictest421-21dbus-daemon11:35:012
481587993327,3cyclictest0-21swapper/112:21:151
481587993327,3cyclictest0-21swapper/108:35:011
481587993326,3cyclictest0-21swapper/112:06:001
481587993326,3cyclictest0-21swapper/112:06:001
481587993325,5cyclictest514584-21cron09:00:011
481587993324,6cyclictest585934-21ntpq11:01:101
481587993323,7cyclictest510118-21/usr/sbin/munin08:45:021
481586993328,3cyclictest636418-21munin-plugin-st12:25:010
481586993325,5cyclictest485-21in:imuxsock08:20:000
481586993321,9cyclictest587695-21cron11:05:010
481588993225,4cyclictest523759-21kworker/u8:8+xprtiod09:20:012
481588993225,3cyclictest0-21swapper/212:35:112
481588993225,3cyclictest0-21swapper/212:35:112
481588993224,5cyclictest554240-21cron10:10:012
481588993224,5cyclictest499562-21head08:06:162
481588993222,6cyclictest3847961-21cups-browsed10:20:302
481587993226,3cyclictest0-21swapper/110:15:011
481587993225,4cyclictest493596-21kworker/u8:1-writeback07:55:021
481587993224,5cyclictest591815-21missed_timers11:11:031
481587993224,5cyclictest533909-21rm09:36:011
481587993224,5cyclictest533909-21rm09:36:011
481587993224,5cyclictest517603-21diskmemload10:51:201
481586993226,4cyclictest524332-21aten2.4-expect09:20:570
481586993225,4cyclictest0-21swapper/010:46:120
481588993125,3cyclictest0-21swapper/211:40:012
481588993124,5cyclictest595575-21sed11:16:182
481588993124,5cyclictest595575-21sed11:16:182
481588993124,5cyclictest0-21swapper/207:50:022
481588993124,4cyclictest1003-21mta-sts-daemon10:57:462
481588993123,5cyclictest501022-21tail08:11:192
481588993122,6cyclictest634497-21needreboot12:21:012
481588993120,7cyclictest616094-21latency11:51:012
481587993124,4cyclictest596836-21sendmail11:20:011
481587993124,4cyclictest596836-21sendmail11:20:011
481587993122,7cyclictest516390-21aten_r2power_cu09:05:561
48158699318,3cyclictest517603-21diskmemload10:11:020
481586993124,5cyclictest265-21systemd-journal09:14:140
481586993124,5cyclictest265-21systemd-journal09:10:010
481586993124,4cyclictest596869-21kworker/u8:13-rpciod12:05:560
481586993124,4cyclictest596869-21kworker/u8:13-rpciod12:05:560
481586993123,6cyclictest0-21swapper/010:35:010
481586993122,6cyclictest1000-21mta-sts-daemon12:01:090
481586993120,8cyclictest579982-21uname10:51:140
48158699311,25cyclictest665-21snmpd08:06:040
481586993110,17cyclictest616086-21sed11:51:010
481588993025,2cyclictest0-21swapper/210:40:012
481588993024,3cyclictest0-21swapper/210:20:012
481588993023,5cyclictest610800-21unixbench_multi11:41:192
481588993023,4cyclictest502589-21kworker/u8:11-xprtiod09:26:182
481588993023,3cyclictest0-21swapper/212:01:112
481588993023,3cyclictest0-21swapper/207:30:012
481588993022,5cyclictest1000-21mta-sts-daemon10:14:572
481588993020,7cyclictest491400-21needreboot07:41:042
481587993024,3cyclictest0-21swapper/112:36:141
481587993024,3cyclictest0-21swapper/112:36:141
481587993024,3cyclictest0-21swapper/112:00:011
481587993024,3cyclictest0-21swapper/111:30:001
481587993022,5cyclictest518189-21telnet09:10:541
481587993020,6cyclictest0-21swapper/111:46:091
481586993023,4cyclictest615092-21ls11:50:010
481586993023,4cyclictest490703-21latency_hist07:40:020
481586993023,4cyclictest490703-21latency_hist07:40:020
481586993022,5cyclictest1-21systemd10:41:130
481586993020,4cyclictest630323-21true12:15:010
481586993020,4cyclictest630323-21true12:15:010
481588992923,3cyclictest0-21swapper/209:10:012
481588992922,4cyclictest636926-21acpi12:25:542
481588992922,4cyclictest1001-21mta-sts-daemon09:11:322
481588992922,4cyclictest1000-21mta-sts-daemon11:01:112
481588992922,4cyclictest1000-21mta-sts-daemon09:31:122
481588992922,4cyclictest0-21swapper/211:46:042
481588992919,8cyclictest577205-21timerwakeupswit10:46:162
481587992923,3cyclictest0-21swapper/111:00:011
48158799292,22cyclictest610217-21ssh11:41:061
481587992921,5cyclictest501001-21unixbench_multi08:11:191
481587992921,4cyclictest0-21swapper/111:06:081
481587992920,5cyclictest516130-21systemd-run09:05:011
481587992920,5cyclictest490955-21aten_r2power_po07:40:551
481586992924,2cyclictest0-21swapper/009:45:000
481586992924,2cyclictest0-21swapper/009:45:000
481586992923,4cyclictest487653-21latency_hist07:30:020
481586992923,3cyclictest595586-21head11:16:190
481586992923,3cyclictest595586-21head11:16:190
481586992918,8cyclictest607185-21ntpq11:36:060
48158699291,7cyclictest528102-21ssh09:26:100
48158699291,22cyclictest640774-21ssh12:31:110
481588992822,3cyclictest0-21swapper/208:20:012
481588992822,2cyclictest0-21swapper/211:30:022
481588992820,5cyclictest514596-21sendmail09:00:012
481588992820,5cyclictest508177-21sendmail_mailqu08:36:142
481588992820,5cyclictest3847961-21cups-browsed08:54:322
48158799285,19cyclictest539836-21ssh09:45:581
48158799285,19cyclictest539836-21ssh09:45:581
481587992822,3cyclictest0-21swapper/110:41:071
481587992822,3cyclictest0-21swapper/109:56:091
481587992821,3cyclictest0-21swapper/110:28:461
481587992819,6cyclictest485-21in:imuxsock07:25:011
481586992822,4cyclictest505499-21tail08:26:160
481586992822,4cyclictest488102-21fschecks_time07:30:590
481586992822,4cyclictest488102-21fschecks_time07:30:590
481586992821,4cyclictest265-21systemd-journal07:55:010
481586992820,6cyclictest421-21dbus-daemon08:33:150
481586992820,6cyclictest0-21swapper/009:50:590
481586992820,6cyclictest0-21swapper/009:50:590
481586992818,8cyclictest554357-21latency_hist10:10:020
481586992814,9cyclictest12650irq/32-eth010:26:130
481588992721,3cyclictest0-21swapper/210:31:122
481588992719,5cyclictest485109-21uniq07:20:592
481588992719,5cyclictest483062-21cron07:15:012
481588992718,6cyclictest421-21dbus-daemon07:10:012
481588992712,11cyclictest3847961-21cups-browsed08:41:082
481587992721,3cyclictest1002-21mta-sts-daemon07:31:161
481587992721,3cyclictest1002-21mta-sts-daemon07:31:161
481587992720,3cyclictest0-21swapper/111:51:181
481587992719,5cyclictest1-21systemd07:15:011
481587992718,6cyclictest420-21avahi-daemon08:01:181
481586992722,2cyclictest0-21swapper/010:00:010
481586992721,4cyclictest493155-21systemctl07:46:130
481586992720,4cyclictest529872-21kworker/u8:12-rpciod09:35:590
481586992720,4cyclictest529872-21kworker/u8:12-rpciod09:35:590
481588992620,4cyclictest1003-21mta-sts-daemon12:10:162
481588992620,4cyclictest1003-21mta-sts-daemon12:10:162
481588992620,3cyclictest0-21swapper/209:45:552
481588992620,3cyclictest0-21swapper/209:45:552
481588992618,5cyclictest503097-21cpuspeed08:20:562
481588992618,5cyclictest503097-21cpuspeed08:20:562
481588992618,5cyclictest3847961-21cups-browsed07:15:352
481588992611,7cyclictest421-21dbus-daemon07:35:012
481588992611,7cyclictest421-21dbus-daemon07:35:012
481587992619,3cyclictest0-21swapper/108:26:051
481586992621,3cyclictest1001-21mta-sts-daemon09:45:110
481586992621,3cyclictest1001-21mta-sts-daemon09:45:110
481586992618,5cyclictest481604-21latency_hist07:10:020
481586992614,10cyclictest497287-21kernelversion08:01:020
481588992519,3cyclictest0-21swapper/208:00:012
481588992518,5cyclictest0-21swapper/210:26:182
481588992517,5cyclictest495071-21idleruntime-cro07:55:012
481587992517,5cyclictest503164-21fschecks_time08:20:581
481587992517,5cyclictest503164-21fschecks_time08:20:581
481587992517,5cyclictest493482-21tail07:46:171
481587992514,8cyclictest511018-21ntp_kernel_pll_08:46:061
48158699259,13cyclictest151ktimers/008:41:160
481586992520,2cyclictest0-21swapper/011:41:080
481586992519,4cyclictest3847959-21cupsd08:22:550
481586992519,4cyclictest3847959-21cupsd08:22:550
481586992515,8cyclictest12650irq/32-eth008:00:010
48158699249,13cyclictest151ktimers/008:11:120
481586992417,5cyclictest439-21systemd-logind09:00:020
481588992316,3cyclictest0-21swapper/208:05:002
481588992315,5cyclictest490607-21kworker/u8:8+xprtiod08:46:162
481588992312,8cyclictest421-21dbus-daemon08:30:012
48158799234,13cyclictest428-21mta-sts-daemon07:26:071
481587992314,6cyclictest481591-21systemd-run07:10:011
481586992318,3cyclictest490607-21kworker/u8:8-events_unbound08:45:550
481586992317,3cyclictest151ktimers/008:51:110
481586992315,6cyclictest1001-21mta-sts-daemon07:15:580
48158699211,15cyclictest151ktimers/007:21:060
48158999192,8cyclictest0-21swapper/310:51:143
481586991914,3cyclictest151ktimers/007:41:070
48158999182,8cyclictest0-21swapper/312:01:053
48158999172,9cyclictest0-21swapper/312:26:043
48158999171,9cyclictest0-21swapper/310:01:093
48158999164,7cyclictest0-21swapper/309:21:103
48158999162,8cyclictest0-21swapper/311:45:553
48158999161,8cyclictest0-21swapper/310:21:093
48158999153,6cyclictest0-21swapper/310:41:073
48158999153,6cyclictest0-21swapper/310:31:113
48158999152,8cyclictest0-21swapper/311:31:063
48158999152,8cyclictest0-21swapper/311:01:133
48158999152,7cyclictest0-21swapper/310:36:083
48158999151,8cyclictest0-21swapper/312:20:013
48158999151,8cyclictest0-21swapper/312:11:083
48158999151,8cyclictest0-21swapper/312:11:083
48158999151,8cyclictest0-21swapper/312:06:123
48158999151,8cyclictest0-21swapper/312:06:123
48158999151,8cyclictest0-21swapper/311:16:053
48158999151,8cyclictest0-21swapper/311:16:053
48158999151,6cyclictest0-21swapper/308:01:063
48158999151,6cyclictest0-21swapper/307:31:073
48158999151,6cyclictest0-21swapper/307:31:073
48158999144,8cyclictest0-21swapper/308:16:093
48158999142,7cyclictest0-21swapper/311:51:053
48158999142,7cyclictest0-21swapper/309:37:323
48158999142,7cyclictest0-21swapper/309:37:323
48158999142,6cyclictest0-21swapper/310:26:053
48158999142,6cyclictest0-21swapper/309:56:053
48158999142,6cyclictest0-21swapper/308:11:103
48158999142,6cyclictest0-21swapper/307:36:083
48158999142,6cyclictest0-21swapper/307:36:083
48158999142,5cyclictest0-21swapper/308:51:083
48158999141,8cyclictest0-21swapper/312:31:093
48158999141,8cyclictest0-21swapper/312:21:073
48158999141,8cyclictest0-21swapper/311:11:063
48158999141,6cyclictest0-21swapper/307:46:103
48158999141,6cyclictest0-21swapper/307:16:043
48158999132,6cyclictest0-21swapper/311:40:553
48158999132,6cyclictest0-21swapper/307:45:013
48158999131,7cyclictest0-21swapper/312:36:083
48158999131,7cyclictest0-21swapper/312:36:083
48158999131,7cyclictest0-21swapper/311:26:063
48158999131,7cyclictest0-21swapper/311:20:553
48158999131,7cyclictest0-21swapper/309:31:123
48158999131,7cyclictest0-21swapper/309:16:153
48158999131,7cyclictest0-21swapper/308:30:013
48158999131,6cyclictest0-21swapper/308:46:053
48158999131,5cyclictest0-21swapper/309:06:053
48158999122,7cyclictest0-21swapper/310:16:043
48158999122,7cyclictest0-21swapper/309:50:553
48158999122,7cyclictest0-21swapper/309:50:553
48158999122,7cyclictest0-21swapper/308:06:033
48158999122,6cyclictest0-21swapper/309:40:263
48158999122,6cyclictest0-21swapper/309:40:263
48158999122,6cyclictest0-21swapper/309:11:033
48158999121,7cyclictest0-21swapper/311:35:593
48158999121,7cyclictest0-21swapper/310:56:113
48158999121,7cyclictest0-21swapper/310:10:583
48158999121,7cyclictest0-21swapper/310:10:013
48158999121,7cyclictest0-21swapper/309:01:073
48158999121,6cyclictest0-21swapper/309:45:563
48158999121,6cyclictest0-21swapper/309:45:563
48158999121,6cyclictest0-21swapper/309:26:083
48158999121,6cyclictest0-21swapper/308:20:583
48158999121,6cyclictest0-21swapper/308:20:583
48158999121,6cyclictest0-21swapper/307:56:033
48158999121,6cyclictest0-21swapper/307:26:053
48158999121,5cyclictest0-21swapper/311:55:043
48158999112,4cyclictest0-21swapper/308:40:573
48158999111,7cyclictest0-21swapper/307:15:013
48158999111,6cyclictest0-21swapper/308:36:003
48158999111,5cyclictest0-21swapper/311:05:363
48158999111,5cyclictest0-21swapper/308:30:563
48158999101,7cyclictest0-21swapper/310:45:543
48158999101,5cyclictest0-21swapper/307:25:023
48158999101,4cyclictest0-21swapper/308:55:123
4815899991,5cyclictest0-21swapper/307:50:553
4815899981,5cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional