You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-06 - 03:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Sun Sep 06, 2026 00:42:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
556278995041,6cyclictest619661-21cut21:55:011
556278994638,5cyclictest577295-21systemd-run20:20:011
556277994640,4cyclictest571249-21sendmail20:00:010
556279994434,6cyclictest698180-21cron00:05:002
556279994235,4cyclictest637828-21idleruntime-cro22:25:012
556279994233,5cyclictest1001-21mta-sts-daemon22:54:312
556278994234,5cyclictest704195-21idleruntime-cro00:15:011
556278994234,5cyclictest1-21systemd00:40:021
556277994236,4cyclictest1-21systemd23:45:020
556278994133,5cyclictest689085-21cron23:50:011
556278994133,5cyclictest610684-21idleruntime-cro21:40:011
556278994031,6cyclictest1-21systemd21:15:011
556278994030,7cyclictest634787-21cron22:20:001
55627999393,30cyclictest596557-21ssh21:16:072
556279993932,4cyclictest610692-21munin-plugin-st21:40:012
556279993932,4cyclictest601490-21grep21:25:012
556279993930,5cyclictest665-21snmpd22:56:162
556279993929,6cyclictest669772-21tail23:16:192
556279993929,6cyclictest563754-21cron19:35:012
556279993929,6cyclictest560794-21true19:25:012
556278993930,6cyclictest421-21dbus-daemon22:25:491
556278993930,5cyclictest421-21dbus-daemon23:42:511
556277993927,3cyclictest0-21swapper/022:36:220
556279993832,3cyclictest0-21swapper/221:46:212
556279993830,5cyclictest689095-21munin-run23:50:012
556279993826,8cyclictest705823-21timerandwakeup00:16:182
556278993832,3cyclictest0-21swapper/121:20:011
556278993831,4cyclictest707228-21sendmail00:20:011
556277993830,5cyclictest711681-21systemctl00:26:160
556279993731,3cyclictest631759-21kworker/u8:10-xprtiod22:40:012
556279993729,5cyclictest630508-21unixbench_multi22:11:222
556279993729,5cyclictest630508-21unixbench_multi22:11:222
556279993729,5cyclictest587780-21systemd-run20:55:012
556279993729,5cyclictest587780-21systemd-run20:55:012
55627899374,10cyclictest708485-21ntpq00:21:101
556278993729,5cyclictest612534-21grep21:41:321
556278993729,5cyclictest607622-21idleruntime-cro21:35:011
556278993712,5cyclictest28-21ksoftirqd/121:26:111
556279993628,5cyclictest566789-21systemd-run19:45:012
55627899363,10cyclictest588737-21ntpq20:56:121
55627899363,10cyclictest588737-21ntpq20:56:121
556278993628,5cyclictest678539-21mailstats23:31:151
556278993628,5cyclictest675282-21ntpq23:26:101
556278993628,5cyclictest560799-21systemd-run19:25:011
556278993611,4cyclictest28-21ksoftirqd/123:00:421
556278993611,4cyclictest28-21ksoftirqd/123:00:421
55627799366,10cyclictest602634-21sh21:26:060
556277993631,3cyclictest607624-21grep21:35:010
556277993629,4cyclictest578923-21latency_hist20:25:020
556277993621,3cyclictest171rcu_preempt19:30:030
556277993619,3cyclictest171rcu_preempt22:55:010
556277993610,10cyclictest14-21ksoftirqd/021:00:010
556277993610,10cyclictest14-21ksoftirqd/021:00:010
556279993528,4cyclictest717759-21mailstats00:36:202
556279993527,5cyclictest672753-21sed23:21:202
556279993526,6cyclictest621505-21unixbench_singl21:56:222
556279993525,8cyclictest699328-21needreboot00:06:092
556278993529,4cyclictest713370-21latency_hist00:30:021
556278993527,5cyclictest592268-21idleruntime-cro21:10:001
556278993527,5cyclictest592268-21idleruntime-cro21:10:001
556278993527,5cyclictest563754-21cron19:35:011
556278993525,6cyclictest590788-21systemd-run21:05:011
556277993528,4cyclictest593646-21sh21:11:130
556277993528,4cyclictest574145-21unixbench_multi20:06:260
556277993527,5cyclictest1-21systemd00:10:010
556277993510,21cyclictest12750irq/33-eth022:16:080
556279993426,5cyclictest680073-21grep23:35:012
556279993426,5cyclictest647707-21fschecks_time22:41:012
556279993426,5cyclictest612541-21ssh21:41:322
556279993426,5cyclictest603341-21unixbench_singl21:26:202
556279993426,5cyclictest561297-21fschecks_count19:26:002
556279993426,4cyclictest0-21swapper/221:31:162
556279993421,3cyclictest171rcu_preempt22:45:152
556279993418,7cyclictest665-21snmpd22:26:102
556279993415,10cyclictest428-21mta-sts-daemon00:21:122
556278993427,4cyclictest654621-21timerwakeupswit22:51:241
556278993425,6cyclictest559315-21latency_hist19:20:021
556278993422,8cyclictest3037047-21cups-browsed22:12:361
556278993422,8cyclictest3037047-21cups-browsed22:12:361
556278993412,5cyclictest28-21ksoftirqd/122:56:081
556278993410,5cyclictest28-21ksoftirqd/122:44:171
556277993429,3cyclictest0-21swapper/022:50:010
556277993425,7cyclictest702992-21unixbench_singl00:11:210
556277993420,4cyclictest556219-21kworker/u8:11-xprtiod19:21:070
556279993325,5cyclictest625711-21idleruntime-cro22:05:012
556279993324,6cyclictest3136330-21munin-node21:50:372
556279993324,5cyclictest1001-21mta-sts-daemon19:16:092
556279993314,3cyclictest171rcu_preempt20:21:102
55627899338,7cyclictest623887-21ntpq22:01:081
55627899337,8cyclictest564104-21aten_r2power_po19:35:581
55627899334,12cyclictest670060-21sh23:17:001
556278993327,3cyclictest0-21swapper/123:10:011
556278993327,3cyclictest0-21swapper/123:10:011
556278993326,4cyclictest628840-21latency_hist22:10:031
556278993325,5cyclictest638020-21latency_hist22:25:021
556278993325,5cyclictest574286-21systemd-run20:10:021
556278993323,8cyclictest592269-21diskmemload23:11:151
556278993323,8cyclictest592269-21diskmemload23:11:151
556278993321,8cyclictest3037047-21cups-browsed20:21:101
556278993318,9cyclictest584155-21ntpq20:41:081
55627799336,6cyclictest14-21ksoftirqd/000:25:010
556277993329,2cyclictest0-21swapper/023:20:010
556277993326,4cyclictest637941-21latency_hist22:25:020
556277993325,5cyclictest420-21avahi-daemon21:44:180
556277993325,5cyclictest3037047-21cups-browsed21:24:300
556279993225,4cyclictest1002-21mta-sts-daemon23:42:502
556279993224,5cyclictest0-21swapper/200:26:162
556279993223,7cyclictest683084-21munin-run23:40:012
556279993223,6cyclictest3037047-21cups-browsed22:17:542
556279993222,7cyclictest716255-21/usr/sbin/munin00:35:022
55627999322,20cyclictest690285-21sh23:51:072
556279993217,9cyclictest693312-21sh23:56:072
556279993217,9cyclictest693312-21sh23:56:072
556279993213,6cyclictest3037047-21cups-browsed23:25:252
55627899328,5cyclictest28-21ksoftirqd/121:21:011
55627899324,7cyclictest568285-21systemd-run19:50:011
556278993226,3cyclictest0-21swapper/121:46:011
556278993225,3cyclictest0-21swapper/123:56:051
556278993225,3cyclictest0-21swapper/123:56:051
556278993224,5cyclictest690053-21kernelversion23:51:031
556278993224,5cyclictest621363-21timerwakeupswit21:56:211
556278993224,4cyclictest0-21swapper/119:56:081
556278993223,6cyclictest420-21avahi-daemon20:31:151
556278993222,8cyclictest572770-21munin-plugin-st20:05:021
55627799328,7cyclictest14-21ksoftirqd/023:23:250
55627799326,8cyclictest14-21ksoftirqd/023:28:070
55627799326,7cyclictest14-21ksoftirqd/000:39:530
556277993226,3cyclictest541551-21kworker/u8:8-xprtiod19:40:010
556277993226,3cyclictest191rcuc/021:58:500
556277993225,4cyclictest643830-21sh22:35:010
556277993223,3cyclictest191rcuc/020:51:170
556277993223,3cyclictest191rcuc/020:51:170
556277993216,2cyclictest171rcu_preempt22:58:410
55627799321,27cyclictest648186-21ssh22:41:120
556279993122,6cyclictest3037047-21cups-browsed22:10:012
556279993115,3cyclictest171rcu_preempt21:15:012
556279993112,3cyclictest171rcu_preempt20:06:212
556278993122,6cyclictest1002-21mta-sts-daemon22:45:361
55627899312,23cyclictest672419-21ssh23:21:151
556278993121,6cyclictest420-21avahi-daemon00:33:371
556278993117,9cyclictest587223-21ntpq20:51:131
556278993117,9cyclictest587223-21ntpq20:51:131
556278993116,11cyclictest561676-21ntpq19:26:091
556278993110,3cyclictest28-21ksoftirqd/119:11:091
556277993122,3cyclictest171rcu_preempt23:54:400
556277993117,6cyclictest171rcu_preempt00:16:530
556279993023,4cyclictest1001-21mta-sts-daemon21:01:112
556279993023,3cyclictest0-21swapper/223:11:182
556279993023,3cyclictest0-21swapper/223:11:182
556279993022,5cyclictest659904-21kernelversion23:01:032
556279993022,5cyclictest659904-21kernelversion23:01:032
556279993013,3cyclictest171rcu_preempt23:09:392
556279993013,3cyclictest171rcu_preempt23:09:392
556279993013,3cyclictest171rcu_preempt22:31:142
55627899309,5cyclictest28-21ksoftirqd/123:36:111
55627899303,24cyclictest642074-21sh22:31:111
55627899302,24cyclictest583258-21munin-run20:40:011
55627799308,6cyclictest14-21ksoftirqd/000:31:130
55627799306,7cyclictest14-21ksoftirqd/023:01:120
55627799306,7cyclictest14-21ksoftirqd/023:01:120
556277993025,4cyclictest556276-21cyclictest23:35:410
556277993023,5cyclictest3037047-21cups-browsed23:31:180
55627799301,9cyclictest14-21ksoftirqd/021:16:060
556277993018,4cyclictest151ktimers/000:01:120
556279992921,5cyclictest568775-21fschecks_count19:51:022
556279992921,5cyclictest556783-21fschecks_count19:11:002
556279992919,3cyclictest171rcu_preempt20:41:112
556279992913,3cyclictest171rcu_preempt19:56:142
556279992910,3cyclictest0-21swapper/220:31:062
55627899298,5cyclictest28-21ksoftirqd/120:46:071
55627899297,5cyclictest28-21ksoftirqd/119:51:041
55627899296,5cyclictest592269-21diskmemload00:02:341
55627899295,14cyclictest0-21swapper/120:11:061
556278992922,3cyclictest0-21swapper/100:06:121
556277992923,4cyclictest608455-21ssh21:36:010
556277992923,4cyclictest191rcuc/020:41:120
556277992921,3cyclictest171rcu_preempt21:01:120
556277992917,3cyclictest171rcu_preempt19:51:050
556277992916,3cyclictest171rcu_preempt21:50:060
556277992915,11cyclictest191rcuc/019:40:470
556277992914,8cyclictest3037039-21cupsd20:37:400
556277992914,3cyclictest57-21kcompactd023:48:170
556277992912,4cyclictest0-21swapper/022:11:020
556277992912,4cyclictest0-21swapper/022:11:020
556277992912,2cyclictest171rcu_preempt21:06:080
556277992912,2cyclictest171rcu_preempt21:06:080
556277992910,8cyclictest191rcuc/022:01:110
556277992910,3cyclictest0-21swapper/019:31:070
55627999289,10cyclictest36-21ksoftirqd/219:36:102
55627999284,4cyclictest36-21ksoftirqd/220:56:122
55627999284,4cyclictest36-21ksoftirqd/220:56:122
556279992819,6cyclictest420-21avahi-daemon20:01:192
556279992818,3cyclictest591673-21ntpq21:06:102
556279992818,3cyclictest591673-21ntpq21:06:102
556279992815,3cyclictest171rcu_preempt20:16:082
556279992815,3cyclictest171rcu_preempt20:11:022
556279992813,3cyclictest171rcu_preempt00:14:322
556279992811,6cyclictest171rcu_preempt20:26:192
556279992810,15cyclictest341rcuc/220:40:022
55627899289,12cyclictest271ktimers/119:41:091
55627799286,9cyclictest151ktimers/020:15:570
55627799286,11cyclictest151ktimers/022:10:020
55627799286,10cyclictest151ktimers/021:45:590
556277992822,4cyclictest559289-21sendmail19:20:010
556277992822,3cyclictest673051-21kworker/u8:0-xprtiod23:56:200
556277992822,3cyclictest673051-21kworker/u8:0-xprtiod23:56:200
556277992821,4cyclictest663739-21unixbench_multi23:06:200
556277992821,4cyclictest663739-21unixbench_multi23:06:200
556277992820,3cyclictest171rcu_preempt23:10:350
556277992820,3cyclictest171rcu_preempt23:10:350
556277992819,5cyclictest191rcuc/022:27:520
556277992818,3cyclictest171rcu_preempt20:46:000
556277992815,2cyclictest171rcu_preempt20:01:070
556277992812,3cyclictest191rcuc/019:11:080
55627899279,4cyclictest28-21ksoftirqd/122:35:031
556278992720,3cyclictest0-21swapper/120:26:091
556277992719,3cyclictest171rcu_preempt20:31:080
556277992718,4cyclictest151ktimers/019:46:120
556277992717,7cyclictest1002-21mta-sts-daemon20:11:020
556277992712,4cyclictest12750irq/33-eth019:10:020
55627999269,4cyclictest171rcu_preempt20:46:102
55627999266,4cyclictest36-21ksoftirqd/219:46:142
55627799266,9cyclictest151ktimers/020:26:030
55627999246,4cyclictest341rcuc/219:10:012
556278992216,3cyclictest0-21swapper/119:10:011
55628099213,10cyclictest0-21swapper/322:26:093
55628099212,11cyclictest0-21swapper/322:16:083
55628099193,8cyclictest0-21swapper/321:21:093
55628099193,11cyclictest0-21swapper/322:01:063
55628099192,9cyclictest0-21swapper/321:56:103
55628099192,9cyclictest0-21swapper/321:36:103
55628099192,10cyclictest0-21swapper/323:46:083
55628099192,10cyclictest0-21swapper/323:36:093
55628099184,7cyclictest0-21swapper/321:16:073
55628099184,10cyclictest0-21swapper/300:36:213
55628099183,9cyclictest0-21swapper/322:21:033
55628099183,9cyclictest0-21swapper/300:16:103
55628099183,7cyclictest0-21swapper/321:46:083
55628099183,7cyclictest0-21swapper/319:15:013
55628099182,9cyclictest0-21swapper/322:56:083
55628099182,9cyclictest0-21swapper/321:26:083
55628099182,8cyclictest0-21swapper/322:11:133
55628099182,8cyclictest0-21swapper/322:11:133
55628099182,7cyclictest0-21swapper/300:30:513
55628099182,10cyclictest0-21swapper/323:26:103
55628099182,10cyclictest0-21swapper/300:11:153
55628099181,10cyclictest0-21swapper/323:56:073
55628099181,10cyclictest0-21swapper/323:56:073
55628099173,9cyclictest0-21swapper/323:51:133
55628099173,9cyclictest0-21swapper/321:11:113
55628099173,8cyclictest0-21swapper/322:51:193
55628099173,8cyclictest0-21swapper/322:31:103
55628099172,9cyclictest0-21swapper/322:46:083
55628099172,9cyclictest0-21swapper/321:31:153
55628099172,7cyclictest0-21swapper/320:20:013
55628099171,9cyclictest0-21swapper/319:41:093
55628099171,7cyclictest0-21swapper/320:51:083
55628099171,7cyclictest0-21swapper/320:51:083
55628099171,11cyclictest0-21swapper/323:31:093
55628099171,10cyclictest0-21swapper/323:05:573
55628099171,10cyclictest0-21swapper/323:05:573
55628099171,10cyclictest0-21swapper/322:06:123
55628099164,8cyclictest0-21swapper/323:00:583
55628099164,8cyclictest0-21swapper/323:00:583
55628099162,9cyclictest0-21swapper/300:21:063
55628099162,9cyclictest0-21swapper/300:01:053
55628099162,8cyclictest0-21swapper/323:11:013
55628099162,8cyclictest0-21swapper/323:11:013
55628099162,8cyclictest0-21swapper/322:35:593
55628099162,7cyclictest0-21swapper/322:45:033
55628099162,7cyclictest0-21swapper/320:31:093
55628099161,9cyclictest0-21swapper/321:51:103
55628099161,9cyclictest0-21swapper/321:41:143
55628099161,9cyclictest0-21swapper/300:06:183
55628099161,10cyclictest0-21swapper/323:21:073
55628099153,7cyclictest0-21swapper/319:31:093
55628099153,4cyclictest0-21swapper/319:21:073
55628099152,9cyclictest0-21swapper/323:40:553
55628099152,8cyclictest0-21swapper/321:01:123
55628099152,8cyclictest0-21swapper/320:26:083
55628099152,8cyclictest0-21swapper/320:21:103
55628099152,8cyclictest0-21swapper/319:55:013
55628099152,8cyclictest0-21swapper/300:26:023
55628099152,7cyclictest0-21swapper/323:15:573
55628099151,7cyclictest0-21swapper/319:16:093
55628099142,7cyclictest0-21swapper/319:26:093
55628099142,6cyclictest0-21swapper/321:06:103
55628099142,6cyclictest0-21swapper/321:06:103
55628099141,7cyclictest0-21swapper/320:41:083
55628099141,7cyclictest0-21swapper/319:36:083
55628099141,6cyclictest0-21swapper/320:46:073
55628099132,6cyclictest0-21swapper/320:36:063
55628099131,7cyclictest0-21swapper/320:00:563
55628099131,7cyclictest0-21swapper/319:46:173
55628099131,6cyclictest0-21swapper/320:06:063
55628099114,5cyclictest0-21swapper/320:11:073
55628099113,6cyclictest0-21swapper/320:56:053
55628099113,6cyclictest0-21swapper/320:56:053
55628099112,5cyclictest0-21swapper/319:55:583
5562809981,4cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional