You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-07 - 11:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Mon Sep 07, 2026 00:42:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2394596995541,9cyclictest1-21systemd22:55:022
2394596995044,4cyclictest2394593-21cyclictest00:01:122
2394596995041,4cyclictest0-21swapper/200:06:132
2394595994838,8cyclictest2534329-21munin-run00:00:011
2394595994835,9cyclictest2452062-21idleruntime-cro21:45:011
2394594994837,8cyclictest2503908-21cron23:10:010
2394594994537,6cyclictest0-21swapper/021:00:010
2394595994436,5cyclictest2531317-21systemd-run23:55:011
2394596994234,5cyclictest2430600-21systemd-run21:10:022
2394596994232,5cyclictest0-21swapper/200:11:112
2394594994228,11cyclictest2432175-21switchtime21:11:280
2394596994133,5cyclictest2436730-21awk21:20:012
2394596994133,5cyclictest2436730-21awk21:20:012
2394596994131,6cyclictest2396103-21systemd-run19:15:012
2394595994126,12cyclictest2541620-21needreboot00:11:061
2394596994033,3cyclictest0-21swapper/222:05:012
2394596994033,3cyclictest0-21swapper/222:05:012
2394596994032,5cyclictest2438524-21unixbench_multi21:21:192
2394595994032,5cyclictest2538267-21fschecks_time00:06:031
2394594994034,4cyclictest2468141-21fschecks_count22:11:040
2394595993931,5cyclictest2418600-21systemd-run20:30:011
2394595993931,5cyclictest2409565-21sh20:00:011
2394595993928,7cyclictest2523367-21ntpq23:41:111
2394594993932,4cyclictest2531900-21aten_r2power_en23:56:010
2394594993932,4cyclictest2494913-21latency_hist22:55:030
2394594993929,8cyclictest2408074-21sh19:55:010
2394596993830,4cyclictest2458833-21aten_r2power_vo21:55:582
2394596993828,6cyclictest2433635-21cron21:15:012
2394596993826,8cyclictest2509988-21sendmail23:20:012
2394595993831,4cyclictest2474639-21modprobe22:21:091
2394595993830,5cyclictest2508110-21needreboot23:16:101
2394595993829,6cyclictest421-21dbus-daemon21:30:011
2394595993829,6cyclictest1-21systemd21:40:011
2394594993831,4cyclictest2523595-21proc_pri23:41:210
2394594993830,6cyclictest1000-21mta-sts-daemon00:06:110
2394596993730,3cyclictest0-21swapper/219:20:012
2394596993729,5cyclictest2503926-21idleruntime-cro23:10:012
2394596993728,6cyclictest1-21systemd21:40:022
239459699372,27cyclictest2471628-21ssh22:16:102
2394595993730,4cyclictest2414070-21cron20:15:011
2394595993729,5cyclictest485-21in:imuxsock20:05:011
2394595993729,5cyclictest2500913-21latency_hist23:05:021
2394595993728,6cyclictest1-21systemd20:50:011
2394594993731,4cyclictest2417091-21munin-plugin-st20:25:010
2394594993731,3cyclictest0-21swapper/022:16:070
2394594993730,4cyclictest2455084-21ssh21:50:010
2394594993730,4cyclictest2426217-21latency_hist20:55:020
2394594993721,12cyclictest2517285-21ssh23:31:080
2394594993721,12cyclictest2517285-21ssh23:31:080
2394596993627,6cyclictest567-21gdbus20:06:142
2394596993627,6cyclictest2456247-21needreboot21:51:082
2394596993624,8cyclictest421-21dbus-daemon23:41:112
2394595993630,3cyclictest0-21swapper/100:30:011
2394595993628,5cyclictest2487415-21unixbench-2d22:41:211
2394594993632,2cyclictest0-21swapper/021:41:080
2394594993630,3cyclictest0-21swapper/021:26:140
2394594993629,4cyclictest2554448-21tail00:31:200
2394594993629,4cyclictest2491710-21cron22:50:010
2394594993628,6cyclictest2499554-21timerwakeupswit23:01:210
2394594993628,5cyclictest420-21avahi-daemon23:14:200
2394596993529,3cyclictest0-21swapper/222:40:012
2394596993527,5cyclictest2465095-21fschecks_count22:06:042
2394596993523,9cyclictest2450706-21timerwakeupswit21:41:212
2394595993528,4cyclictest2423088-21idleruntime-cro20:45:001
2394595993528,4cyclictest2423088-21idleruntime-cro20:45:001
2394595993526,6cyclictest0-21swapper/121:20:011
2394595993526,6cyclictest0-21swapper/121:20:011
2394594993527,6cyclictest12650irq/32-eth021:56:070
2394594993527,5cyclictest151ktimers/000:21:090
2394594993524,9cyclictest1002-21mta-sts-daemon00:40:020
2394594993523,9cyclictest2446114-21latency_hist21:35:020
2394596993426,5cyclictest695189-21cups-browsed23:26:282
2394596993425,6cyclictest2400602-21idleruntime-cro19:30:002
2394595993428,3cyclictest2534397-21kworker/u8:8-xprtiod00:05:011
2394595993428,3cyclictest2397626-21kworker/u8:12-xprtiod19:30:011
2394595993426,5cyclictest695189-21cups-browsed20:57:141
2394595993426,5cyclictest2534398-21cups-browsed00:40:011
2394595993426,5cyclictest2489997-21ntp_offset22:46:151
2394595993426,5cyclictest2470450-21latency_hist22:15:021
2394594993427,5cyclictest2430575-21diskmemload21:21:070
2394594993427,4cyclictest2429080-21sh21:05:010
2394594993427,4cyclictest2409595-21awk20:00:010
2394594993426,5cyclictest695189-21cups-browsed22:57:390
2394594993425,6cyclictest2513025-21cron23:25:010
2394596993327,3cyclictest2390603-21kworker/u8:15-xprtiod19:25:002
2394596993327,3cyclictest2390603-21kworker/u8:15-xprtiod19:25:002
2394596993327,3cyclictest0-21swapper/223:35:012
2394596993327,3cyclictest0-21swapper/223:35:012
2394596993326,4cyclictest2444483-21threads21:31:232
2394596993325,5cyclictest2430575-21diskmemload00:00:012
2394596993325,5cyclictest2415567-21cron20:20:012
2394596993322,8cyclictest2418582-21/usr/sbin/munin20:30:022
2394595993325,5cyclictest2430568-21cron21:10:011
2394595993323,7cyclictest2526288-21missed_timers23:46:051
2394594993327,4cyclictest2405739-21kernelversion19:46:020
2394594993326,4cyclictest2477468-21kernelversion22:26:030
2394594993326,4cyclictest2436739-21awk21:20:010
2394594993326,4cyclictest2436739-21awk21:20:010
2394594993322,7cyclictest0-21swapper/020:30:010
2394594993318,12cyclictest2447142-21ssh21:36:090
2394596993226,3cyclictest0-21swapper/200:40:002
2394596993225,4cyclictest2474386-21kernelversion22:21:032
2394595993226,3cyclictest2403619-21kworker/u8:10-xprtiod19:40:021
239459599321,26cyclictest2496130-21ssh22:56:141
239459599321,23cyclictest2438061-21ssh21:21:121
2394594993226,4cyclictest2400194-21mailstats19:26:170
2394594993225,5cyclictest421-21dbus-daemon22:23:210
2394594993224,5cyclictest2550121-21/usr/sbin/munin00:26:160
2394594993217,11cyclictest2520518-21ssh23:36:150
2394594993217,11cyclictest2520518-21ssh23:36:150
2394596993122,6cyclictest420-21avahi-daemon20:05:012
2394595993126,2cyclictest0-21swapper/120:40:011
2394595993125,3cyclictest0-21swapper/123:25:011
2394595993124,5cyclictest2547729-21needreboot00:21:061
2394595993124,4cyclictest1001-21mta-sts-daemon00:15:291
2394595993123,5cyclictest420-21avahi-daemon22:16:101
2394595993123,5cyclictest2455118-21systemd-run21:50:021
2394595993123,5cyclictest2415584-21systemctl20:20:011
2394595993122,6cyclictest793993-21munin-node23:11:461
239459499317,11cyclictest421-21dbus-daemon20:40:010
2394594993125,2cyclictest0-21swapper/019:11:110
239459499312,25cyclictest2483838-21ssh22:36:100
2394596993024,3cyclictest0-21swapper/223:15:012
2394596993023,3cyclictest0-21swapper/222:46:142
2394596993022,5cyclictest567-21gdbus20:46:162
2394596993022,5cyclictest2510548-21aten_r2power_cu23:20:582
2394596993022,5cyclictest1000-21mta-sts-daemon22:59:462
2394596993021,6cyclictest2487410-21unixbench-2d22:41:212
2394596993021,6cyclictest2455106-21idleruntime-cro21:50:012
2394596993019,8cyclictest2549627-21sh00:25:022
2394596993019,8cyclictest2480638-21missed_timers22:31:052
2394595993024,3cyclictest0-21swapper/119:20:011
2394595993023,4cyclictest1003-21mta-sts-daemon22:29:591
2394595993023,3cyclictest0-21swapper/121:51:151
2394595993023,3cyclictest0-21swapper/119:21:151
2394595993023,3cyclictest0-21swapper/119:21:151
2394595993022,5cyclictest0-21swapper/122:00:021
2394595993021,6cyclictest695189-21cups-browsed19:31:031
2394595993020,8cyclictest2480252-21cpu22:31:001
2394594993024,3cyclictest2487490-21unixbench_singl22:41:220
2394594993022,6cyclictest2422395-21needreboot20:41:050
2394594993022,6cyclictest2422395-21needreboot20:41:050
239459499302,26cyclictest151ktimers/021:06:260
2394594993018,9cyclictest420-21avahi-daemon19:21:080
2394594993018,9cyclictest420-21avahi-daemon19:21:080
2394594993017,10cyclictest0-21swapper/022:31:200
2394596992923,3cyclictest0-21swapper/219:35:002
2394596992922,4cyclictest2551436-21unixbench_singl00:26:212
2394596992919,7cyclictest1001-21mta-sts-daemon19:51:122
239459699291,15cyclictest2530046-21sed23:51:222
239459599294,11cyclictest2430575-21diskmemload00:32:061
2394595992923,3cyclictest0-21swapper/119:55:011
2394595992922,4cyclictest2430575-21diskmemload23:30:021
2394595992921,5cyclictest2405091-21grep19:45:011
2394595992920,6cyclictest2501197-21ssh23:05:211
2394594992921,6cyclictest151ktimers/021:51:170
2394594992921,6cyclictest1001-21mta-sts-daemon00:01:170
2394594992921,5cyclictest2544777-21sh00:16:100
2394594992920,6cyclictest420-21avahi-daemon23:18:250
2394594992919,7cyclictest2527183-21sh23:46:410
2394594992919,7cyclictest0-21swapper/020:15:590
239459699282,21cyclictest2526644-21ssh23:46:152
2394596992821,4cyclictest2555718-21sh00:35:022
2394596992821,4cyclictest1002-21mta-sts-daemon20:36:132
2394596992820,5cyclictest1002-21mta-sts-daemon19:46:082
2394596992818,6cyclictest695189-21cups-browsed21:28:382
2394595992821,4cyclictest2397627-21kworker/u8:13-xprtiod19:50:011
2394595992821,4cyclictest1002-21mta-sts-daemon23:33:051
2394595992821,4cyclictest1002-21mta-sts-daemon23:33:051
2394595992821,4cyclictest1002-21mta-sts-daemon21:31:531
2394595992821,4cyclictest1000-21mta-sts-daemon20:50:171
2394595992821,3cyclictest0-21swapper/123:35:281
2394595992821,3cyclictest0-21swapper/123:35:281
2394595992821,3cyclictest0-21swapper/121:11:531
2394595992820,5cyclictest2463082-21vmstat22:01:201
2394595992820,5cyclictest2463082-21vmstat22:01:201
239459599281,22cyclictest2493094-21ssh22:51:171
2394594992822,3cyclictest0-21swapper/019:35:010
2394594992821,5cyclictest0-21swapper/023:51:210
2394594992821,4cyclictest2462994-21unixbench_multi22:01:190
2394594992821,4cyclictest2462994-21unixbench_multi22:01:190
2394594992820,6cyclictest2423977-21ntp_kernel_pll_20:46:090
2394594992812,11cyclictest2465495-21ssh22:06:110
239459499281,13cyclictest2513546-21/usr/sbin/munin23:26:040
2394596992721,3cyclictest2431077-21kworker/u8:17-xprtiod22:25:582
2394596992721,3cyclictest1001-21mta-sts-daemon00:16:002
2394596992721,3cyclictest0-21swapper/223:36:092
2394596992721,3cyclictest0-21swapper/223:36:092
2394596992721,3cyclictest0-21swapper/222:11:162
2394596992719,5cyclictest2409604-21systemd-run20:00:012
2394595992720,4cyclictest1000-21mta-sts-daemon20:23:421
2394595992718,7cyclictest2419967-21tail20:31:191
2394595992716,8cyclictest2465450-21sed22:06:101
2394594992722,3cyclictest2397627-21kworker/u8:13-xprtiod20:06:190
2394594992710,13cyclictest151ktimers/019:41:100
2394596992619,4cyclictest2498342-21/usr/sbin/munin23:01:162
2394596992619,4cyclictest2397627-21kworker/u8:13-xprtiod20:51:192
239459599261,18cyclictest2483825-21ssh22:36:091
2394594992619,5cyclictest485-21in:imuxsock19:40:010
2394594992616,7cyclictest695186-21cupsd20:12:280
2394594992614,9cyclictest265-21systemd-journal00:15:010
2394596992519,4cyclictest2428925-21unixbench-2d21:01:202
2394596992519,3cyclictest0-21swapper/220:56:102
2394596992519,3cyclictest0-21swapper/219:45:012
2394596992517,6cyclictest2421920-21aten_r2power_fr20:40:582
2394596992517,6cyclictest2421920-21aten_r2power_fr20:40:582
2394595992517,5cyclictest2429080-21cron21:05:011
239459499259,13cyclictest0-21swapper/020:31:190
2394594992513,10cyclictest0-21swapper/020:01:160
239459699243,13cyclictest1000-21mta-sts-daemon20:11:102
2394596992417,4cyclictest2417088-21idleruntime-cro20:25:012
2394595992419,2cyclictest0-21swapper/119:11:001
2394594992415,7cyclictest2396977-21ntp_kernel_pll_19:16:090
239459699232,15cyclictest695189-21cups-browsed19:36:112
239459799224,10cyclictest0-21swapper/322:06:103
239459799222,12cyclictest0-21swapper/323:26:093
2394595992216,3cyclictest0-21swapper/120:06:091
239459799213,13cyclictest0-21swapper/321:11:133
239459799213,12cyclictest0-21swapper/321:56:093
239459799213,11cyclictest0-21swapper/322:56:083
239459799213,11cyclictest0-21swapper/322:26:123
239459799213,10cyclictest0-21swapper/321:41:163
239459799212,12cyclictest0-21swapper/322:51:143
239459799212,12cyclictest0-21swapper/300:21:163
239459799212,12cyclictest0-21swapper/300:16:133
239459799212,11cyclictest0-21swapper/300:26:103
239459799205,8cyclictest0-21swapper/323:41:123
239459799205,12cyclictest0-21swapper/322:21:093
239459799204,10cyclictest0-21swapper/323:56:133
239459799204,10cyclictest0-21swapper/321:31:113
239459799203,9cyclictest0-21swapper/322:16:073
239459799203,12cyclictest0-21swapper/323:21:093
239459799203,11cyclictest0-21swapper/323:06:093
239459799203,11cyclictest0-21swapper/321:51:133
239459799203,11cyclictest0-21swapper/321:26:153
239459799203,10cyclictest0-21swapper/322:46:143
239459799202,9cyclictest0-21swapper/323:46:093
239459799201,9cyclictest0-21swapper/319:16:093
2394596992014,3cyclictest0-21swapper/220:35:002
239459799195,8cyclictest0-21swapper/320:41:073
239459799195,8cyclictest0-21swapper/320:41:073
239459799194,7cyclictest0-21swapper/319:41:133
239459799193,9cyclictest0-21swapper/323:11:133
239459799193,9cyclictest0-21swapper/322:41:083
239459799193,9cyclictest0-21swapper/319:56:103
239459799193,10cyclictest0-21swapper/323:01:093
239459799192,9cyclictest0-21swapper/323:16:063
239459799192,9cyclictest0-21swapper/319:11:113
239459799192,12cyclictest0-21swapper/321:36:073
239459799192,11cyclictest0-21swapper/322:36:133
239459799192,10cyclictest0-21swapper/323:36:073
239459799192,10cyclictest0-21swapper/323:36:073
239459799192,10cyclictest0-21swapper/300:11:073
239459799191,11cyclictest0-21swapper/322:31:123
239459799191,11cyclictest0-21swapper/321:46:133
239459799191,11cyclictest0-21swapper/300:36:153
239459799191,11cyclictest0-21swapper/300:01:103
239459799191,10cyclictest0-21swapper/323:31:113
239459799191,10cyclictest0-21swapper/323:31:113
2394595991914,2cyclictest0-21swapper/119:10:011
2394594991914,3cyclictest2394532-21kworker/u8:0-xprtiod19:10:010
239459799184,8cyclictest0-21swapper/322:11:123
239459799183,9cyclictest0-21swapper/321:16:083
239459799183,9cyclictest0-21swapper/321:16:083
239459799183,8cyclictest0-21swapper/320:46:103
239459799183,10cyclictest0-21swapper/322:05:003
239459799183,10cyclictest0-21swapper/322:05:003
239459799182,9cyclictest0-21swapper/320:11:093
239459799182,10cyclictest0-21swapper/323:51:123
239459799174,6cyclictest0-21swapper/321:06:253
239459799172,9cyclictest0-21swapper/320:56:103
239459799172,8cyclictest0-21swapper/321:20:593
239459799172,8cyclictest0-21swapper/320:36:083
239459799172,8cyclictest0-21swapper/320:21:093
239459799172,8cyclictest0-21swapper/319:51:063
239459799172,8cyclictest0-21swapper/319:21:083
239459799172,8cyclictest0-21swapper/319:21:083
239459799172,8cyclictest0-21swapper/300:31:123
239459799172,10cyclictest0-21swapper/300:05:443
239459799171,9cyclictest0-21swapper/320:01:113
239459799171,9cyclictest0-21swapper/319:46:073
239459799171,8cyclictest0-21swapper/320:06:103
239459799162,9cyclictest0-21swapper/320:16:103
239459799162,7cyclictest0-21swapper/320:31:063
2394596991612,2cyclictest0-21swapper/219:10:012
239459799152,8cyclictest0-21swapper/320:26:063
239459799152,8cyclictest0-21swapper/319:31:123
239459799152,7cyclictest0-21swapper/319:36:073
239459799142,8cyclictest0-21swapper/320:51:083
239459799142,7cyclictest0-21swapper/319:26:083
239459799142,6cyclictest0-21swapper/321:01:073
239459799122,8cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional