You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-11-17 - 14:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Tue Nov 17, 2026 00:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1937794998126,16cyclictest2008733-21ssh23:27:403
1937794998122,14cyclictest1942014-21date20:30:023
1937794997925,16cyclictest1994709-21ssh22:58:073
1937794997522,14cyclictest1971410-21ssh22:07:583
1937794997224,13cyclictest2022500-21ssh23:56:463
193779099723,6cyclictest131rcu_preempt19:20:012
1937790997220,13cyclictest1-21systemd20:50:012
1937794997124,14cyclictest2019986-21ssh23:51:303
1937790997126,10cyclictest3467602-21systemd-journal21:30:312
1937794997020,14cyclictest2006078-21ssh23:21:523
1937790997033,31cyclictest3467602-21systemd-journal00:20:002
1937790997022,12cyclictest3467602-21systemd-journal21:25:002
1937794996922,15cyclictest1937994-21unin-run19:15:023
1937794996917,15cyclictest1988987-21ssh22:45:293
1937790996922,8cyclictest309-21in:imuxsock22:00:012
1937794996853,12cyclictest1967384-21ssh21:59:323
1937790996727,35cyclictest1988545-21latency_hist22:45:022
1937794996637,24cyclictest150-21pr/ttyS221:24:263
1937790996631,29cyclictest309-21in:imuxsock20:05:002
1937790996630,32cyclictest30-21ksoftirqd/221:12:282
1937790996620,5cyclictest309-21in:imuxsock00:15:022
1937790996620,5cyclictest309-21in:imuxsock00:15:012
1937790996616,43cyclictest309-21in:imuxsock23:24:592
1937785996630,11cyclictest23-21ksoftirqd/100:05:011
193778099662,8cyclictest1969987-21kworker/0:222:32:090
1937794996531,11cyclictest364-21snmpd20:21:383
1937794996519,9cyclictest2024312-21ssh00:00:333
1937790996511,12cyclictest1-21systemd23:05:012
1937780996556,6cyclictest1990903-21latency_hist22:50:020
1937780996515,27cyclictest0-21swapper/021:45:020
193779099647,11cyclictest1977031-21modprobe22:20:002
1937790996430,29cyclictest30-21ksoftirqd/221:25:522
1937790996425,34cyclictest309-21in:imuxsock20:25:022
1937790996424,5cyclictest2017785-21sh23:46:472
1937790996424,5cyclictest2017785-21sh23:46:472
1937794996350,10cyclictest2002861-21ssh23:15:053
1937794996348,12cyclictest2013066-21ssh23:36:393
1937794996348,12cyclictest2013066-21ssh23:36:393
193779099636,21cyclictest1939041-21unin-run19:35:002
1937790996325,33cyclictest2038095-21latency_hist00:30:022
1937794996254,4cyclictest2001865-21sshd23:13:173
1937794996249,10cyclictest1957847-21ssh21:39:173
1937794996242,16cyclictest3467602-21systemd-journal21:29:493
1937790996224,8cyclictest30-21ksoftirqd/200:40:002
1937790996224,6cyclictest2019402-21sh23:50:052
193778099629,10cyclictest12-21ksoftirqd/000:15:510
1937780996228,3cyclictest20235871ssh23:59:240
1937794996148,10cyclictest1998903-21ssh23:06:463
1937794996148,10cyclictest1998903-21ssh23:06:453
1937794996147,11cyclictest1975488-21ssh22:16:253
1937794996141,16cyclictest3467602-21systemd-journal00:17:263
1937794996121,15cyclictest1974697-21munin-run22:15:023
193779099614,23cyclictest3467602-21systemd-journal23:00:002
1937790996120,7cyclictest30-21ksoftirqd/221:55:012
1937790996120,5cyclictest2004733-21sh23:19:162
193778099619,9cyclictest12-21ksoftirqd/023:03:340
1937780996121,16cyclictest12-21ksoftirqd/023:20:010
1937794996042,14cyclictest1959459-21ssh21:42:313
1937794996040,15cyclictest1996510-21ssh23:01:403
1937790996028,28cyclictest30-21ksoftirqd/200:08:512
1937790996026,8cyclictest30-21ksoftirqd/220:35:012
1937790996010,14cyclictest275-21dbus-daemon22:15:002
1937785996015,9cyclictest23-21ksoftirqd/121:12:301
1937794995944,12cyclictest2034015-21ssh00:21:163
1937794995944,11cyclictest3467602-21systemd-journal22:23:523
1937794995943,13cyclictest1991367-21ssh22:50:403
1937794995943,13cyclictest1987866-21ssh22:43:273
1937794995943,12cyclictest1961291-21ssh21:46:213
1937794995942,14cyclictest2039573-21ssh00:33:203
1937794995942,14cyclictest1946067-21ssh21:14:413
1937794995925,12cyclictest150-21pr/ttyS221:16:213
1937794995925,12cyclictest150-21pr/ttyS221:16:213
193779099594,17cyclictest1940846-21modprobe20:10:002
1937790995925,5cyclictest1939565-21sh19:45:002
1937785995917,17cyclictest23-21ksoftirqd/121:30:361
1937794995844,11cyclictest1953767-21ssh21:30:233
1937794995844,10cyclictest2036363-21sshd00:26:163
1937794995843,12cyclictest2015088-21ssh23:40:563
1937794995842,13cyclictest2026530-21ssh00:05:183
1937794995841,13cyclictest1979807-21ssh22:26:083
1937794995841,13cyclictest1968288-21ssh22:01:083
1937790995848,5cyclictest1929957-21kworker/2:121:40:002
1937790995822,9cyclictest30-21ksoftirqd/223:12:102
1937790995821,8cyclictest30-21ksoftirqd/222:04:532
193778099586,46cyclictest0-21swapper/022:25:000
193778099582,5cyclictest131rcu_preempt00:30:020
1937794995745,9cyclictest2028932-21ssh00:10:273
1937794995745,9cyclictest2028932-21ssh00:10:263
1937794995743,12cyclictest2017555-21ssh23:46:103
1937794995743,12cyclictest2017555-21ssh23:46:103
1937794995742,12cyclictest1985123-21ssh22:37:303
1937794995742,12cyclictest1981901-21ssh22:30:293
1937794995741,11cyclictest1942923-21munin-run20:50:013
1937794995740,13cyclictest2010387-21ssh23:31:023
1937790995741,9cyclictest2014422-21cron23:40:012
1937790995741,9cyclictest2014422-21cron23:40:002
1937785995747,7cyclictest23-21ksoftirqd/123:50:011
1937785995747,7cyclictest23-21ksoftirqd/123:50:001
1937780995730,12cyclictest2009659-21ssh23:29:470
1937794995643,10cyclictest2041636-21ssh00:37:473
1937794995642,12cyclictest1963166-21ssh21:50:063
1937794995632,19cyclictest1-21systemd20:10:013
193779099566,16cyclictest2023885-21cron00:00:002
1937790995627,6cyclictest1985502-21sh22:38:332
1937790995624,9cyclictest30-21ksoftirqd/222:31:052
1937790995624,5cyclictest2010014-21sh23:30:032
193778599566,11cyclictest23-21ksoftirqd/122:34:501
193778599562,11cyclictest23-21ksoftirqd/123:04:591
193778599561,13cyclictest23-21ksoftirqd/100:33:591
1937780995628,13cyclictest2011624-21ssh23:33:570
193778099562,39cyclictest2000214-21ssh23:09:490
193778099562,39cyclictest2000214-21ssh23:09:480
193778099561,8cyclictest284-21systemd-logind22:10:000
193778099561,8cyclictest1-21systemd00:25:010
1937794995543,9cyclictest1942356-21sshd20:35:183
1937794995542,10cyclictest1943935-21sshd21:08:183
1937794995540,12cyclictest1938746-21sshd19:27:183
1937794995535,16cyclictest364-21snmpd20:13:583
1937790995520,11cyclictest30-21ksoftirqd/222:28:592
193778599556,9cyclictest23-21ksoftirqd/100:30:001
1937785995517,10cyclictest23-21ksoftirqd/119:56:181
1937785995515,10cyclictest23-21ksoftirqd/122:10:071
1937780995530,9cyclictest1976994-21ssh22:19:480
1937780995524,15cyclictest2024539-21ssh00:01:020
193778099552,37cyclictest2017046-21ssh23:45:030
193778099552,37cyclictest2017046-21ssh23:45:030
1937794995439,12cyclictest1939779-21sshd19:46:183
1937794995437,13cyclictest1939532-21sshd19:42:183
193779099543,20cyclictest2016826-21ssh23:45:002
193779099542,42cyclictest309-21in:imuxsock20:00:012
1937790995423,4cyclictest1971741-21sh22:08:412
1937790995422,12cyclictest30-21ksoftirqd/223:07:222
1937790995422,12cyclictest30-21ksoftirqd/223:07:222
1937790995417,9cyclictest30-21ksoftirqd/220:45:012
1937790995417,10cyclictest30-21ksoftirqd/219:30:002
193778599547,7cyclictest23-21ksoftirqd/122:16:241
1937785995448,4cyclictest221rcuc/122:57:571
193778599543,14cyclictest23-21ksoftirqd/121:23:151
1937785995420,9cyclictest23-21ksoftirqd/121:17:321
1937785995420,9cyclictest23-21ksoftirqd/121:17:311
1937780995443,6cyclictest16950irq/224-800000020:20:000
1937780995426,12cyclictest1994160-21ssh22:56:450
1937780995424,14cyclictest2013788-21ssh23:38:320
1937780995424,14cyclictest2013788-21ssh23:38:310
193778099541,38cyclictest1962813-21ssh21:49:490
1937794995346,4cyclictest1943156-21sshd20:54:183
1937794995341,9cyclictest1941314-21sshd20:15:193
1937794995340,9cyclictest364-21snmpd20:04:273
1937794995334,14cyclictest364-21snmpd19:20:563
1937794995333,15cyclictest1942128-21cron20:35:003
1937790995314,12cyclictest30-21ksoftirqd/200:21:222
193778599535,9cyclictest23-21ksoftirqd/123:37:121
193778599535,9cyclictest23-21ksoftirqd/123:37:111
193778599534,6cyclictest23-21ksoftirqd/121:40:181
1937785995311,5cyclictest131rcu_preempt00:06:081
1937780995327,12cyclictest1967848-21ssh22:00:110
193778099531,36cyclictest1956640-21ssh21:36:380
1937780995313,36cyclictest12-21ksoftirqd/021:15:280
1937780995313,36cyclictest12-21ksoftirqd/021:15:270
1937790995243,6cyclictest1979199-21kworker/2:222:45:252
1937790995219,10cyclictest30-21ksoftirqd/221:44:522
193778599525,7cyclictest23-21ksoftirqd/123:15:001
193778599524,9cyclictest23-21ksoftirqd/121:45:401
193778599524,6cyclictest23-21ksoftirqd/121:27:291
1937785995213,8cyclictest0-21swapper/100:00:021
1937785995211,13cyclictest23-21ksoftirqd/122:10:001
193778099528,32cyclictest131rcu_preempt23:14:540
1937780995233,14cyclictest3467602-21systemd-journal21:55:010
1937780995226,11cyclictest2039466-21ssh00:33:070
1937780995225,12cyclictest1992711-21ssh22:53:530
1937780995224,13cyclictest2020107-21ssh23:51:450
1937780995211,12cyclictest12-21ksoftirqd/023:23:370
1937794995142,5cyclictest1939291-21munin-run19:40:013
1937794995142,5cyclictest1939291-21munin-run19:40:003
1937794995136,10cyclictest1940078-21systemd-run19:55:013
1937794995127,20cyclictest1937910-21latency_hist19:10:023
193779099513,18cyclictest3467602-21systemd-journal20:25:002
1937790995120,10cyclictest30-21ksoftirqd/221:45:152
193779099511,19cyclictest2040362-21sed00:35:022
193778599517,8cyclictest23-21ksoftirqd/100:20:011
193778599516,7cyclictest23-21ksoftirqd/100:14:571
193778599516,7cyclictest23-21ksoftirqd/100:14:561
1937785995138,8cyclictest221rcuc/122:45:001
1937785995136,12cyclictest1-21systemd23:35:001
1937785995118,11cyclictest23-21ksoftirqd/123:20:031
1937785995114,6cyclictest131rcu_preempt22:45:161
1937780995131,15cyclictest275-21dbus-daemon20:30:020
1937780995124,12cyclictest1952249-21ssh21:27:240
1937780995112,11cyclictest12-21ksoftirqd/022:29:190
1937780995111,36cyclictest12-21ksoftirqd/019:20:000
1937794995038,9cyclictest364-21snmpd20:56:463
1937794995037,10cyclictest364-21snmpd19:16:253
193778599509,11cyclictest23-21ksoftirqd/121:54:241
1937785995040,5cyclictest1937788-21grep19:10:011
193778599501,9cyclictest23-21ksoftirqd/122:25:001
1937785995013,10cyclictest23-21ksoftirqd/122:37:131
1937780995038,7cyclictest1942139-21munin-plugin-st20:35:000
1937780995026,9cyclictest1967152-21ssh21:59:040
1937780995020,12cyclictest2028672-21latency_hist00:10:020
1937780995019,15cyclictest2030802-21ssh00:14:530
1937780995019,15cyclictest2030802-21ssh00:14:530
1937780995017,28cyclictest310-21in:imklog23:44:020
1937780995013,9cyclictest12-21ksoftirqd/022:36:370
1937794994937,9cyclictest364-21snmpd21:03:213
1937794994932,13cyclictest364-21snmpd20:41:153
1937794994931,13cyclictest364-21snmpd19:55:393
1937790994941,6cyclictest2005086-21kworker/2:300:03:052
1937790994912,13cyclictest275-21dbus-daemon19:55:012
193778599497,7cyclictest23-21ksoftirqd/123:53:291
193778599497,7cyclictest23-21ksoftirqd/122:02:091
193778599497,10cyclictest23-21ksoftirqd/123:45:001
193778599496,7cyclictest23-21ksoftirqd/120:15:001
193778599496,11cyclictest23-21ksoftirqd/119:40:011
193778599496,11cyclictest23-21ksoftirqd/119:40:011
193778599495,8cyclictest23-21ksoftirqd/122:27:481
193778599494,7cyclictest23-21ksoftirqd/100:35:141
1937785994936,10cyclictest23-21ksoftirqd/122:52:371
193778099492,10cyclictest1988702-21latency_hist22:45:010
1937794994833,10cyclictest364-21snmpd19:33:263
1937790994838,7cyclictest1929957-21kworker/2:119:15:012
193778599487,11cyclictest23-21ksoftirqd/121:36:431
193778599486,7cyclictest23-21ksoftirqd/123:20:301
193778599484,6cyclictest23-21ksoftirqd/121:55:311
193778599484,10cyclictest23-21ksoftirqd/120:55:001
1937785994831,13cyclictest1-21systemd19:25:011
1937780994831,13cyclictest1944215-21ssh21:10:170
1937780994829,15cyclictest1942416-21tr20:40:000
1937780994823,21cyclictest12-21ksoftirqd/022:14:180
1937780994818,6cyclictest12-21ksoftirqd/021:33:450
1937790994740,5cyclictest1929957-21kworker/2:122:24:082
1937790994739,5cyclictest1929957-21kworker/2:121:05:022
1937790994722,7cyclictest30-21ksoftirqd/222:54:362
193778599474,13cyclictest23-21ksoftirqd/121:00:001
1937785994738,5cyclictest1938774-21munin-plugin-st19:30:001
1937785994736,8cyclictest1938254-21cut19:20:011
193778099475,36cyclictest0-21swapper/021:00:010
1937780994738,6cyclictest16950irq/224-800000019:40:010
1937780994738,6cyclictest16950irq/224-800000019:40:000
1937780994735,8cyclictest12-21ksoftirqd/021:25:000
1937780994729,14cyclictest2040871-21ssh00:35:440
1937790994623,6cyclictest30-21ksoftirqd/223:28:142
193778599466,6cyclictest23-21ksoftirqd/120:45:021
1937785994629,12cyclictest275-21dbus-daemon19:35:001
193778599461,9cyclictest23-21ksoftirqd/100:23:281
1937785994610,4cyclictest131rcu_preempt23:25:131
1937790994537,5cyclictest1929957-21kworker/2:121:17:042
1937790994537,5cyclictest1929957-21kworker/2:121:17:042
193778599456,8cyclictest23-21ksoftirqd/120:25:011
193778599455,7cyclictest23-21ksoftirqd/123:06:161
193778599455,7cyclictest23-21ksoftirqd/123:06:151
1937785994530,11cyclictest23-21ksoftirqd/121:05:011
1937780994534,9cyclictest12-21ksoftirqd/019:55:020
1937780994533,8cyclictest16950irq/224-800000021:10:020
1937790994416,8cyclictest1943467-21systemctl21:00:002
193778599445,8cyclictest23-21ksoftirqd/120:30:001
193778599443,8cyclictest23-21ksoftirqd/120:46:441
193778599441,12cyclictest23-21ksoftirqd/120:05:011
193778599435,11cyclictest23-21ksoftirqd/119:52:281
1937785994325,13cyclictest3467602-21systemd-journal19:50:001
193778599431,8cyclictest0-21swapper/121:08:411
193778599431,7cyclictest0-21swapper/120:09:381
193778099437,33cyclictest1699551-21munin-node19:30:550
1937780994334,6cyclictest12-21ksoftirqd/019:58:170
1937790994230,8cyclictest1939280-21sh19:40:002
1937790994230,8cyclictest1939280-21sh19:40:002
1937785994234,6cyclictest23-21ksoftirqd/120:40:021
1937785994233,6cyclictest23-21ksoftirqd/120:20:011
193778599422,7cyclictest23-21ksoftirqd/119:15:011
193778599421,7cyclictest0-21swapper/120:32:251
193778099422,36cyclictest1939785-21sshd19:47:190
1937790994125,12cyclictest1-21systemd20:20:012
1937790994124,13cyclictest1-21systemd19:50:002
1937790994124,11cyclictest1942400-21run-parts20:40:012
1937780994132,6cyclictest19350irq/34-mmc120:05:380
1937780994128,10cyclictest1940578-21sshd20:04:190
1937790993929,7cyclictest1944131-21sed21:10:032
193778599392,5cyclictest0-21swapper/119:40:141
1937780993927,9cyclictest12-21ksoftirqd/019:15:000
1937780993920,15cyclictest1938753-21sshd19:28:180
1937780993913,6cyclictest131rcu_preempt20:50:020
1937790993829,7cyclictest30-21ksoftirqd/220:15:012
1937780993728,6cyclictest19350irq/34-mmc119:23:370
1937780993727,6cyclictest1941626-21cron20:25:000
1937780993726,7cyclictest12-21ksoftirqd/019:45:010
1937780993726,6cyclictest12-21ksoftirqd/020:45:010
193778099371,32cyclictest0-21swapper/021:05:010
193779099362,5cyclictest131rcu_preempt19:25:032
1937780993629,5cyclictest12-21ksoftirqd/020:55:020
1937780993625,6cyclictest1937782-21latency_hist19:10:010
1937790993522,10cyclictest3467602-21systemd-journal20:53:022
1937780993427,4cyclictest16950irq/224-800000020:12:170
1937790992919,7cyclictest0-21swapper/219:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional