You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-26 - 12:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Wed Aug 26, 2026 00:45:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
75077699118114,2cyclictest0-21swapper/023:30:020
750776999524,43cyclictest0-21swapper/021:50:180
750776999516,15cyclictest0-21swapper/023:35:020
750776999516,15cyclictest0-21swapper/023:35:020
750776999312,37cyclictest0-21swapper/022:03:480
750776999256,26cyclictest771373-21sh21:40:030
750776999117,41cyclictest813803-21wc23:10:010
750776999035,16cyclictest0-21swapper/021:43:150
750776998713,44cyclictest0-21swapper/000:05:010
750776998417,33cyclictest761921-21latency_hist21:20:020
750776997940,35cyclictest757424-21ssh21:10:240
750776997741,32cyclictest807560-21ssh22:56:400
750776997740,32cyclictest833305-21ssh23:50:420
750776997740,32cyclictest833305-21ssh23:50:420
750776997730,22cyclictest806806-21latency_hist22:55:020
750776997723,8cyclictest814877-21rm23:12:040
750776997723,8cyclictest814877-21rm23:12:040
750776997622,14cyclictest2129045-1kworker/0:2H00:00:000
750776997621,20cyclictest773876-21ssh21:45:210
750776997619,26cyclictest796951-21ssh22:34:340
750776997529,15cyclictest849063-21ssh00:24:270
750776997524,34cyclictest845744-21ssh00:17:120
750776997519,24cyclictest767709-21ssh21:32:280
750776997339,31cyclictest840132-21ssh00:05:060
750776997323,19cyclictest753340-21systemd-run20:00:010
750776997323,14cyclictest852173-21ssh00:30:280
750776997321,18cyclictest785942-21ssh22:10:490
750776997223,16cyclictest757039-21ls21:10:010
750776997130,7cyclictest828382-21ssh23:40:220
750776997130,7cyclictest828382-21ssh23:40:220
750776997120,15cyclictest763551-21sshd21:23:490
750776997022,18cyclictest855407-21ssh00:37:260
750776997022,18cyclictest844154-21ssh00:13:550
750776996932,32cyclictest831968-21ssh23:48:060
750776996824,13cyclictest826066-21ssh23:35:320
750776996814,17cyclictest811107-21ssh23:04:210
750776996814,17cyclictest811107-21ssh23:04:210
750776996721,12cyclictest0-21swapper/000:26:280
750787996628,13cyclictest37-21ksoftirqd/300:25:003
750776996624,13cyclictest783635-21ssh22:06:000
750776996624,13cyclictest783635-21ssh22:06:000
750776996622,13cyclictest794268-21ssh22:28:490
750776996618,19cyclictest0-21swapper/023:21:300
750776996618,19cyclictest0-21swapper/023:21:300
750776996525,36cyclictest792537-21latency_hist22:25:020
750776996521,40cyclictest0-21swapper/022:17:490
750776996423,37cyclictest752272-21munin-plugin-st19:39:590
750776996343,15cyclictest818440-21sshd23:19:490
750776996334,6cyclictest0-21swapper/020:50:020
750776996323,11cyclictest0-21swapper/021:27:200
750776996228,4cyclictest750969-21munin-run19:15:000
750776996220,4cyclictest752545-21munin-run19:45:010
750776996219,37cyclictest1-21systemd22:45:010
750776996118,12cyclictest752883-21cat19:50:020
750783995950,6cyclictest794814-21kworker/2:300:02:482
750776995951,4cyclictest756742-21munin-run21:05:010
750776995951,4cyclictest756742-21munin-run21:05:000
750776995943,13cyclictest751492-21sshd19:24:490
75078799581,53cyclictest809155-21ssh23:00:013
750776995844,10cyclictest754925-21munin-run20:30:000
750776995822,5cyclictest803080-21ssh22:47:140
750776995742,11cyclictest779240-21sshd21:56:490
750776995742,11cyclictest779240-21sshd21:56:490
750776995741,12cyclictest798993-21ssh22:38:530
750776995732,19cyclictest751227-21modprobe19:20:000
750783995646,6cyclictest514710-21kworker/2:219:35:002
750783995537,14cyclictest275-21dbus-daemon00:30:012
750776995542,9cyclictest754386-21munin-run20:20:020
750776995539,11cyclictest751724-21sshd19:25:450
750776995539,11cyclictest751724-21sshd19:25:450
750776995538,12cyclictest755705-21munin-run20:45:000
750776995536,15cyclictest364-21snmpd20:13:030
750783995446,5cyclictest755468-21kworker/2:100:38:572
750779995439,10cyclictest3467602-21systemd-journal00:29:201
750776995434,16cyclictest3468525-21sendmail-mta20:31:000
750787995345,6cyclictest37-21ksoftirqd/320:50:023
750783995345,5cyclictest794814-21kworker/2:323:05:542
750783995344,6cyclictest794814-21kworker/2:300:12:032
750783995335,14cyclictest275-21dbus-daemon23:59:592
750779995336,12cyclictest790139-21sh22:20:001
750776995341,9cyclictest754591-21sshd20:20:560
750776995331,17cyclictest753053-21munin-plugin-st19:55:000
75078799527,40cyclictest851926-21ssh00:30:033
750783995244,5cyclictest794814-21kworker/2:323:30:292
750783995244,5cyclictest794814-21kworker/2:323:30:292
750783995244,5cyclictest755468-21kworker/2:121:36:412
750783995231,15cyclictest275-21dbus-daemon00:20:012
75077999526,38cyclictest23-21ksoftirqd/100:00:021
75077999523,3cyclictest131rcu_preempt21:50:001
75077999522,17cyclictest750975-21true19:15:011
750776995240,9cyclictest756177-21sshd20:51:030
750776995218,9cyclictest753580-21latency_hist20:05:030
750787995135,12cyclictest3467602-21systemd-journal21:30:003
750783995143,5cyclictest794814-21kworker/2:323:19:342
750783995143,5cyclictest755468-21kworker/2:121:41:562
750783995141,7cyclictest755468-21kworker/2:122:10:022
750783995141,7cyclictest755468-21kworker/2:122:10:022
750783995114,9cyclictest30-21ksoftirqd/221:30:162
75077999516,8cyclictest23-21ksoftirqd/122:56:301
75077999512,6cyclictest0-21swapper/100:36:541
750776995132,14cyclictest750877-21sed19:10:020
75078399509,6cyclictest131rcu_preempt21:27:422
750783995042,5cyclictest794814-21kworker/2:323:22:142
750783995042,5cyclictest794814-21kworker/2:323:22:142
750783995042,5cyclictest794814-21kworker/2:300:34:582
750783995042,5cyclictest755468-21kworker/2:122:53:382
750783995041,6cyclictest794814-21kworker/2:300:21:522
750783995036,10cyclictest3467602-21systemd-journal20:55:012
75078399502,10cyclictest30-21ksoftirqd/223:01:162
75078399502,10cyclictest30-21ksoftirqd/223:01:152
75078399501,16cyclictest0-21swapper/200:08:182
750776995043,4cyclictest756471-21munin-run21:00:010
750776995035,11cyclictest364-21snmpd20:07:120
750776995035,11cyclictest364-21snmpd20:07:110
750776995029,17cyclictest753571-21cron20:05:000
750776995022,24cyclictest755631-21cat20:40:020
750783994942,4cyclictest755468-21kworker/2:122:57:512
750783994941,5cyclictest755468-21kworker/2:122:45:242
750783994941,5cyclictest755468-21kworker/2:122:01:262
750783994914,5cyclictest131rcu_preempt22:21:082
750783994911,6cyclictest131rcu_preempt22:17:092
75077999495,5cyclictest131rcu_preempt21:40:021
750779994936,10cyclictest1-21systemd22:55:011
750779994929,14cyclictest275-21dbus-daemon23:25:011
750779994929,14cyclictest275-21dbus-daemon23:25:011
750776994941,4cyclictest364-21snmpd19:31:500
750787994828,14cyclictest275-21dbus-daemon23:25:023
750787994828,14cyclictest275-21dbus-daemon23:25:023
75078799482,4cyclictest131rcu_preempt19:31:493
750787994813,32cyclictest0-21swapper/321:45:023
750783994840,5cyclictest768784-21kworker/2:021:48:442
750783994840,5cyclictest755468-21kworker/2:123:52:042
750783994840,5cyclictest755468-21kworker/2:123:52:042
750783994840,5cyclictest755468-21kworker/2:122:14:222
75078399482,3cyclictest131rcu_preempt21:00:002
75077999484,3cyclictest131rcu_preempt00:22:361
75077999481,7cyclictest817063-21ssh23:16:381
75078399476,6cyclictest131rcu_preempt21:05:012
75078399476,6cyclictest131rcu_preempt21:05:012
750783994739,5cyclictest755468-21kworker/2:122:35:342
750783994739,5cyclictest755468-21kworker/2:121:22:452
750783994739,5cyclictest754382-21kworker/2:321:10:442
750783994739,5cyclictest743104-21kworker/2:020:20:032
750783994733,10cyclictest275-21dbus-daemon20:00:012
750783994732,11cyclictest275-21dbus-daemon21:10:012
75078399472,4cyclictest131rcu_preempt22:41:072
750787994630,10cyclictest1-21systemd21:20:013
750787994629,12cyclictest37-21ksoftirqd/319:10:023
75078799461,5cyclictest131rcu_preempt00:00:003
75078799461,23cyclictest810215-21kworker/u8:223:20:013
750783994638,5cyclictest755468-21kworker/2:122:26:182
750783994638,5cyclictest755468-21kworker/2:121:58:122
750783994638,5cyclictest755468-21kworker/2:121:58:122
750783994638,5cyclictest755468-21kworker/2:121:53:352
750783994631,12cyclictest30-21ksoftirqd/220:30:002
75078399462,4cyclictest131rcu_preempt23:11:272
75078399462,4cyclictest131rcu_preempt23:11:272
750779994630,11cyclictest275-21dbus-daemon00:35:011
750779994621,15cyclictest23-21ksoftirqd/122:10:021
750779994621,15cyclictest23-21ksoftirqd/122:10:021
750779994611,6cyclictest131rcu_preempt23:13:261
750779994611,6cyclictest131rcu_preempt23:13:261
750787994534,8cyclictest37-21ksoftirqd/322:18:103
750787994530,10cyclictest309-21in:imuxsock00:20:013
750787994524,9cyclictest275-21dbus-daemon21:55:013
75078399455,8cyclictest30-21ksoftirqd/223:45:002
75078399455,8cyclictest30-21ksoftirqd/223:45:002
750783994537,5cyclictest743104-21kworker/2:020:13:172
75078399452,3cyclictest131rcu_preempt23:50:022
75077999453,5cyclictest131rcu_preempt20:20:021
75077999452,12cyclictest826399-21ssh23:36:141
75077999451,4cyclictest793529-21ssh22:27:071
750779994511,5cyclictest131rcu_preempt22:33:591
750787994430,10cyclictest3467602-21systemd-journal22:07:493
750787994430,10cyclictest3467602-21systemd-journal22:07:483
750787994416,5cyclictest131rcu_preempt23:49:373
750783994436,5cyclictest754382-21kworker/2:321:15:142
75078399441,3cyclictest131rcu_preempt23:40:022
750779994430,10cyclictest3467602-21systemd-journal19:34:591
750779994419,5cyclictest23-21ksoftirqd/120:35:011
750787994327,12cyclictest37-21ksoftirqd/322:27:143
75078799432,6cyclictest781639-21ssh22:01:493
750787994317,6cyclictest131rcu_preempt21:59:273
750787994317,6cyclictest131rcu_preempt21:59:273
75078799431,38cyclictest797697-21sshd22:35:493
750783994330,9cyclictest30-21ksoftirqd/222:32:512
750783994330,10cyclictest30-21ksoftirqd/220:45:002
750783994310,5cyclictest131rcu_preempt20:24:492
75077999436,6cyclictest131rcu_preempt23:25:431
75077999436,3cyclictest0-21swapper/123:48:021
75077999434,7cyclictest0-21swapper/121:11:191
75077999433,36cyclictest1-21systemd23:05:011
75077999433,36cyclictest1-21systemd23:05:011
750779994330,10cyclictest275-21dbus-daemon22:40:011
750779994328,12cyclictest23-21ksoftirqd/122:13:261
75077999432,6cyclictest801721-21sh22:44:331
750779994326,12cyclictest1-21systemd21:35:001
750779994311,6cyclictest131rcu_preempt00:00:241
750787994233,7cyclictest37-21ksoftirqd/322:12:553
750787994231,8cyclictest275-21dbus-daemon19:20:003
750787994231,7cyclictest825846-21sed23:35:023
750787994231,7cyclictest825846-21sed23:35:023
75078799422,5cyclictest131rcu_preempt23:10:003
750787994222,15cyclictest275-21dbus-daemon21:00:013
75078799421,4cyclictest131rcu_preempt00:15:003
75078399423,4cyclictest131rcu_preempt19:20:012
75078399421,7cyclictest755941-21sshd20:49:482
75077999423,4cyclictest131rcu_preempt00:07:271
75077999423,3cyclictest131rcu_preempt19:40:001
750779994232,6cyclictest274-21cron19:20:011
750779994228,9cyclictest3467602-21systemd-journal19:55:011
75077999421,6cyclictest764872-21ssh21:26:191
75077999421,11cyclictest829278-21ssh23:42:261
75077999421,11cyclictest829278-21ssh23:42:261
75077999421,11cyclictest823906-21rm23:31:001
75077999421,11cyclictest823906-21rm23:31:001
750787994132,5cyclictest309-21in:imuxsock21:35:013
750787994127,11cyclictest37-21ksoftirqd/321:45:373
750787994126,10cyclictest309-21in:imuxsock21:07:503
750783994132,5cyclictest514710-21kworker/2:219:40:012
750783994128,10cyclictest750887-21date19:10:022
75078399411,9cyclictest821217-21ssh23:25:152
75077999415,4cyclictest131rcu_preempt23:52:091
75077999415,4cyclictest131rcu_preempt23:52:091
75077999414,4cyclictest131rcu_preempt21:41:401
75077999413,3cyclictest131rcu_preempt21:50:381
75077999413,3cyclictest131rcu_preempt00:17:531
750779994131,6cyclictest753306-21sendmail19:59:591
75077999411,8cyclictest756995-21diskmemload23:06:541
75077999411,7cyclictest23-21ksoftirqd/122:03:561
75078799406,4cyclictest131rcu_preempt22:45:023
750787994026,11cyclictest37-21ksoftirqd/300:05:023
750787994026,10cyclictest37-21ksoftirqd/323:35:503
750787994025,11cyclictest37-21ksoftirqd/322:48:103
750787994024,13cyclictest1-21systemd23:05:013
750787994024,13cyclictest1-21systemd23:05:013
750787994020,9cyclictest37-21ksoftirqd/322:33:253
75078799401,5cyclictest131rcu_preempt23:15:003
75078799401,5cyclictest131rcu_preempt23:15:003
75078399402,4cyclictest131rcu_preempt20:40:012
75077999405,5cyclictest131rcu_preempt19:26:061
75077999405,5cyclictest131rcu_preempt19:26:061
75077999404,3cyclictest131rcu_preempt00:10:151
750779994029,8cyclictest757027-21ssh21:10:011
750779994025,11cyclictest275-21dbus-daemon20:26:481
75077999401,6cyclictest0-21swapper/121:22:221
750787993928,9cyclictest37-21ksoftirqd/300:05:483
750787993927,9cyclictest37-21ksoftirqd/321:10:173
750787993927,8cyclictest37-21ksoftirqd/320:20:023
75078799392,6cyclictest131rcu_preempt19:25:023
750783993923,12cyclictest275-21dbus-daemon19:45:012
75077999396,3cyclictest131rcu_preempt22:22:041
75077999394,3cyclictest131rcu_preempt22:47:001
750779993930,6cyclictest780619-21sendmail22:00:011
750779993930,6cyclictest780619-21sendmail22:00:011
750787993827,8cyclictest275-21dbus-daemon21:05:023
750787993827,8cyclictest275-21dbus-daemon21:05:013
750787993827,5cyclictest309-21in:imuxsock21:40:013
750787993826,9cyclictest3467602-21systemd-journal19:40:483
750787993824,10cyclictest275-21dbus-daemon20:45:013
75078799382,3cyclictest131rcu_preempt22:55:003
750787993822,11cyclictest309-21in:imuxsock20:25:013
75078799381,4cyclictest131rcu_preempt22:24:343
75078399385,4cyclictest131rcu_preempt20:10:022
75078399385,4cyclictest131rcu_preempt20:10:012
75078399382,5cyclictest131rcu_preempt19:30:012
75078399382,5cyclictest131rcu_preempt19:30:002
750783993824,10cyclictest309-21in:imuxsock19:50:012
75077999384,4cyclictest131rcu_preempt19:43:491
75077999382,4cyclictest131rcu_preempt21:18:391
750779993824,9cyclictest275-21dbus-daemon20:10:001
750779993824,9cyclictest275-21dbus-daemon20:10:001
750779993824,11cyclictest275-21dbus-daemon20:25:011
750779993822,11cyclictest1-21systemd20:50:011
750787993728,5cyclictest753894-21latency_hist20:10:013
750787993728,5cyclictest753894-21latency_hist20:10:013
750787993727,8cyclictest37-21ksoftirqd/323:28:143
750787993726,8cyclictest3467602-21systemd-journal19:25:033
750787993726,8cyclictest3467602-21systemd-journal19:25:023
750787993721,12cyclictest1-21systemd00:40:013
75078799371,12cyclictest834391-21rm23:53:223
75078799371,12cyclictest834391-21rm23:53:223
750783993728,7cyclictest30-21ksoftirqd/219:14:592
75078399371,5cyclictest131rcu_preempt20:33:502
75078399371,4cyclictest131rcu_preempt19:55:022
75077999375,5cyclictest131rcu_preempt20:41:431
750779993724,9cyclictest1-21systemd19:50:011
750787993625,8cyclictest37-21ksoftirqd/319:15:033
750787993625,8cyclictest3467602-21systemd-journal20:50:023
750787993625,7cyclictest37-21ksoftirqd/323:41:173
750787993625,7cyclictest37-21ksoftirqd/323:41:173
75078799362,3cyclictest131rcu_preempt20:15:003
750787993622,11cyclictest37-21ksoftirqd/320:05:013
750787993622,11cyclictest37-21ksoftirqd/320:05:003
750787993620,13cyclictest37-21ksoftirqd/300:31:543
750783993620,11cyclictest309-21in:imuxsock19:25:012
75078399361,3cyclictest131rcu_preempt20:05:012
75078399361,3cyclictest131rcu_preempt20:05:002
750779993624,9cyclictest23-21ksoftirqd/120:05:011
750779993624,9cyclictest23-21ksoftirqd/120:05:011
75077999362,3cyclictest131rcu_preempt19:21:481
750779993621,11cyclictest275-21dbus-daemon20:55:011
750779993612,19cyclictest756449-21cron21:00:001
75078799358,6cyclictest131rcu_preempt19:50:133
75078799355,21cyclictest0-21swapper/320:00:013
75078799351,5cyclictest131rcu_preempt21:23:563
75077999352,3cyclictest131rcu_preempt20:14:491
750779993520,11cyclictest755437-21systemctl20:40:011
75077999351,3cyclictest131rcu_preempt19:10:011
750787993418,13cyclictest37-21ksoftirqd/320:26:513
75078799341,6cyclictest150-21pr/ttyS219:40:003
750787993320,10cyclictest275-21dbus-daemon20:35:013
750779993218,10cyclictest756722-21sshd21:03:491
750779993218,10cyclictest756722-21sshd21:03:481
750787993123,6cyclictest752795-21modprobe19:50:003
750787993123,5cyclictest755428-21latency_hist20:40:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional