You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-22 - 21:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Wed Jul 22, 2026 12:45:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1896719998985,2cyclictest0-21swapper/011:20:020
1896719998824,13cyclictest1927578-21ssh10:01:450
1896719998783,2cyclictest0-21swapper/009:15:030
1896719998617,63cyclictest16850irq/134-800000011:48:440
1896719998417,62cyclictest16850irq/134-800000011:27:260
1896719998328,14cyclictest1981118-21ssh11:54:200
1896719998317,61cyclictest16850irq/134-800000012:07:590
1896719998220,12cyclictest1915782-21ssh09:36:400
1896719998220,12cyclictest1915782-21ssh09:36:400
1896719998218,59cyclictest16850irq/134-800000012:01:190
1896719998218,59cyclictest16850irq/134-800000012:01:190
189671999811,40cyclictest1963863-21ssh11:18:120
1896719998113,6cyclictest16850irq/134-800000009:28:590
1896719998028,46cyclictest3467602-21systemd-journal08:15:000
1896719997921,15cyclictest1950281-21ssh10:49:450
1896719997919,12cyclictest1998353-21ssh12:30:510
1896719997919,12cyclictest1998353-21ssh12:30:500
1896719997826,16cyclictest1991172-21ssh12:15:460
1896719997813,60cyclictest16850irq/134-800000010:55:230
189671999781,27cyclictest0-21swapper/009:12:260
1896719997771,4cyclictest0-21swapper/009:25:020
1896719997712,6cyclictest16850irq/134-800000007:43:340
1896719997712,6cyclictest16850irq/134-800000007:43:340
1896719997625,46cyclictest1933704-21cron10:14:590
1896719997617,12cyclictest1951508-21ssh10:52:020
1896719997528,15cyclictest1918210-21ssh09:41:450
1896719997526,15cyclictest1957950-21ssh11:05:260
1896719997526,15cyclictest1957950-21ssh11:05:260
1896719997425,14cyclictest1938060-21ssh10:23:540
1896719997414,50cyclictest16850irq/134-800000011:10:090
1896719997319,50cyclictest16850irq/134-800000011:38:440
1896719997317,46cyclictest16850irq/134-800000010:40:320
1896719997314,13cyclictest1913974-21ssh09:32:560
1896719997223,12cyclictest1941359-21ssh10:30:360
1896719997222,12cyclictest1921550-21ssh09:49:260
189671999721,66cyclictest1945612-21cron10:40:000
1896731997125,11cyclictest1969449-21pluginstate11:30:022
1896719997127,16cyclictest1994568-21ssh12:23:040
189671999711,54cyclictest1971385-21ssh11:33:580
189671999703,7cyclictest1900287-21cron08:20:010
189671999701,26cyclictest0-21swapper/011:43:250
1896731996930,35cyclictest3467602-21systemd-journal07:29:252
1896731996925,5cyclictest1924771-21sh09:55:532
1896731996925,4cyclictest1902666-21latency_hist09:05:012
1896719996927,11cyclictest1934579-21ssh10:16:330
1896719996923,9cyclictest1956078-21ssh11:01:420
1896719996920,12cyclictest1930025-21ssh10:07:000
1896719996920,12cyclictest1925757-21ssh09:58:050
1896719996919,13cyclictest2001539-21ssh12:37:460
1896719996916,48cyclictest16850irq/134-800000009:51:040
1896719996913,13cyclictest1996941-21ssh12:28:030
1896719996912,52cyclictest16850irq/134-800000010:29:220
1896724996811,42cyclictest1882931-21munin-node07:25:011
1896719996717,13cyclictest1988414-21ssh12:10:050
1896719996717,13cyclictest1988414-21ssh12:10:050
1896734996517,14cyclictest37-21ksoftirqd/311:10:013
1896734996517,14cyclictest37-21ksoftirqd/311:10:003
1896731996523,5cyclictest1931235-21sh10:09:462
1896719996521,9cyclictest1982513-21ssh11:57:030
1896734996425,11cyclictest37-21ksoftirqd/312:24:593
1896731996427,5cyclictest1975922-21sh11:43:272
1896731996424,6cyclictest1995445-21sh12:25:022
1896731996423,7cyclictest1917223-21modprobe09:40:012
1896731996423,7cyclictest1917223-21modprobe09:40:002
1896731996423,4cyclictest1910143-21latency_hist09:25:012
1896731996322,5cyclictest1961752-21sh11:13:522
1896731996220,9cyclictest1-21systemd10:15:002
189673199615,3cyclictest131rcu_preempt11:35:002
1896731996124,5cyclictest1923182-21sh09:52:442
1896731996025,7cyclictest1902355-21cron09:00:002
1896731996025,4cyclictest1950409-21latency_hist10:50:022
189671999605,21cyclictest1899248-21modprobe08:00:000
1896719996024,8cyclictest12-21ksoftirqd/008:32:400
1896719996023,5cyclictest1898235-21sendmail07:40:010
1896731995925,5cyclictest1902046-21latency_hist08:50:032
1896731995923,4cyclictest1944521-21sh10:37:212
1896731995921,8cyclictest30-21ksoftirqd/211:21:402
1896731995822,5cyclictest1902112-21latency_hist08:55:012
1896731995818,10cyclictest1983610-1kworker/u9:112:07:022
1896731995818,10cyclictest1-21systemd11:55:012
189671999583,21cyclictest1897179-21modprobe07:20:000
189671999583,21cyclictest1897179-21modprobe07:20:000
189673499574,9cyclictest37-21ksoftirqd/309:20:023
189673199574,3cyclictest131rcu_preempt10:42:332
1896731995725,5cyclictest1904605-21sh09:13:272
1896731995721,8cyclictest30-21ksoftirqd/212:39:592
1896731995721,7cyclictest30-21ksoftirqd/210:50:442
189673499562,12cyclictest37-21ksoftirqd/309:53:253
189673499561,9cyclictest1905351-21latency_hist09:15:023
189673499561,10cyclictest37-21ksoftirqd/310:25:013
1896731995623,4cyclictest1974304-21latency_hist11:40:012
1896731995622,7cyclictest30-21ksoftirqd/211:59:082
189671999561,28cyclictest0-21swapper/008:30:020
189671999561,28cyclictest0-21swapper/008:30:010
189671999561,12cyclictest1902450-21latency_hist09:00:010
189673499557,10cyclictest37-21ksoftirqd/311:40:013
1896734995546,5cyclictest37-21ksoftirqd/310:30:023
1896731995523,6cyclictest1955126-21munin-plugin-st11:00:002
1896731995523,5cyclictest1899421-21latency_hist08:00:022
1896731995520,10cyclictest30-21ksoftirqd/210:33:182
189673499549,26cyclictest37-21ksoftirqd/310:35:023
189673499542,11cyclictest37-21ksoftirqd/311:05:013
1896734995412,5cyclictest131rcu_preempt11:52:453
1896731995445,6cyclictest1901369-21kworker/2:109:42:232
1896724995420,10cyclictest23-21ksoftirqd/112:22:471
189673499537,10cyclictest37-21ksoftirqd/310:54:093
189673499532,14cyclictest2002563-21sendmail12:40:003
1896731995336,13cyclictest1-21systemd07:35:022
1896731995321,11cyclictest30-21ksoftirqd/209:33:152
1896731995318,8cyclictest1899855-21date08:10:022
1896724995319,19cyclictest23-21ksoftirqd/111:00:281
1896734995211,5cyclictest131rcu_preempt10:44:453
1896734995211,12cyclictest37-21ksoftirqd/307:50:013
189673199527,3cyclictest131rcu_preempt09:20:012
189671999521,47cyclictest1900565-21modprobe08:24:590
189673499512,11cyclictest1943519-21ssh10:35:083
189673499511,19cyclictest0-21swapper/309:38:103
189673499511,19cyclictest0-21swapper/309:38:093
1896731995142,6cyclictest1910180-21kworker/2:010:20:452
1896731995122,8cyclictest30-21ksoftirqd/209:48:062
1896724995139,8cyclictest23-21ksoftirqd/110:45:021
1896724995139,7cyclictest1952790-21cron10:55:011
189672499513,18cyclictest131rcu_preempt09:20:011
1896724995116,11cyclictest23-21ksoftirqd/110:39:591
189671999513,16cyclictest1901608-21sh08:45:000
189671999511,26cyclictest0-21swapper/008:53:030
189673499509,8cyclictest37-21ksoftirqd/311:48:353
189673499509,5cyclictest131rcu_preempt10:10:003
189673499507,5cyclictest131rcu_preempt10:10:093
1896734995034,13cyclictest275-21dbus-daemon09:55:493
1896731995042,5cyclictest1910180-21kworker/2:012:14:122
1896731995042,5cyclictest1910180-21kworker/2:012:14:112
1896731995042,5cyclictest1910180-21kworker/2:011:47:432
1896731995042,5cyclictest1910180-21kworker/2:010:17:152
1896731995041,5cyclictest1910180-21kworker/2:011:01:152
1896731995023,6cyclictest30-21ksoftirqd/212:01:362
1896731995023,6cyclictest30-21ksoftirqd/212:01:362
189673199501,3cyclictest131rcu_preempt11:15:032
1896724995013,5cyclictest131rcu_preempt09:25:331
1896724995012,10cyclictest131rcu_preempt09:40:031
1896724995012,10cyclictest131rcu_preempt09:40:021
1896719995023,6cyclictest1898085-21latency_hist07:35:020
1896734994936,7cyclictest0-21swapper/312:15:033
1896734994936,7cyclictest0-21swapper/312:15:023
1896731994941,5cyclictest1910180-21kworker/2:012:16:592
1896731994941,5cyclictest1910180-21kworker/2:009:29:222
1896731994941,5cyclictest1897992-21kworker/2:007:45:012
1896731994941,5cyclictest1897992-21kworker/2:007:45:012
1896731994931,14cyclictest1938938-21sshd10:25:242
189672499492,27cyclictest1974187-21kworker/u8:411:45:031
189673499488,6cyclictest131rcu_preempt12:33:163
189673499488,6cyclictest131rcu_preempt12:33:153
189673499488,5cyclictest131rcu_preempt11:21:083
189673499488,12cyclictest37-21ksoftirqd/309:43:273
189673499487,10cyclictest37-21ksoftirqd/312:09:483
189673499484,9cyclictest37-21ksoftirqd/311:29:463
189673499483,8cyclictest37-21ksoftirqd/309:46:343
189673499482,12cyclictest37-21ksoftirqd/310:49:593
189673499482,11cyclictest37-21ksoftirqd/312:16:433
1896734994818,6cyclictest131rcu_preempt09:20:463
1896734994812,5cyclictest131rcu_preempt09:31:253
1896731994840,5cyclictest1910180-21kworker/2:011:09:232
1896731994840,5cyclictest1910180-21kworker/2:011:09:222
1896731994839,6cyclictest1901369-21kworker/2:110:01:322
1896731994839,5cyclictest1895884-21kworker/2:408:35:032
189672499486,4cyclictest131rcu_preempt09:12:371
1896724994832,12cyclictest1-21systemd09:35:011
1896724994830,13cyclictest23-21ksoftirqd/111:20:011
189672499481,8cyclictest23-21ksoftirqd/111:00:011
189673499476,9cyclictest37-21ksoftirqd/310:16:253
189673499475,11cyclictest37-21ksoftirqd/311:59:003
189673499474,8cyclictest37-21ksoftirqd/309:26:053
189673499474,5cyclictest131rcu_preempt10:57:183
1896734994739,6cyclictest361rcuc/312:04:263
1896734994739,6cyclictest361rcuc/312:04:253
189673499473,7cyclictest37-21ksoftirqd/308:45:013
189673499473,10cyclictest37-21ksoftirqd/308:10:023
1896734994723,20cyclictest37-21ksoftirqd/311:20:013
189673499472,11cyclictest37-21ksoftirqd/310:01:183
1896734994719,5cyclictest131rcu_preempt11:40:533
1896734994716,5cyclictest131rcu_preempt11:14:213
1896734994710,6cyclictest131rcu_preempt11:34:313
1896731994738,6cyclictest1979009-21kworker/2:212:31:282
1896731994738,6cyclictest1979009-21kworker/2:212:31:282
189672499474,4cyclictest131rcu_preempt11:58:491
189672499473,8cyclictest23-21ksoftirqd/108:05:001
1896724994737,7cyclictest23-21ksoftirqd/108:45:011
1896724994733,11cyclictest120-21kswapd012:35:011
1896724994733,11cyclictest120-21kswapd012:35:001
1896724994731,12cyclictest275-21dbus-daemon11:53:241
1896724994731,11cyclictest2002557-21cron12:40:001
189672499472,5cyclictest131rcu_preempt08:10:021
1896724994714,8cyclictest150-21pr/ttyS210:31:291
1896724994711,5cyclictest131rcu_preempt11:35:001
1896724994710,6cyclictest131rcu_preempt11:22:121
189671999473,4cyclictest131rcu_preempt07:30:010
1896734994631,12cyclictest275-21dbus-daemon07:55:013
1896734994629,12cyclictest37-21ksoftirqd/309:00:013
1896734994628,12cyclictest3467602-21systemd-journal07:35:013
189673499462,14cyclictest37-21ksoftirqd/312:25:133
1896734994615,10cyclictest37-21ksoftirqd/307:40:003
1896734994612,5cyclictest131rcu_preempt07:45:023
1896734994612,5cyclictest131rcu_preempt07:45:013
189672499469,9cyclictest23-21ksoftirqd/110:00:021
1896724994629,12cyclictest3467602-21systemd-journal09:50:241
1896724994628,10cyclictest23-21ksoftirqd/109:25:011
1896724994619,8cyclictest131rcu_preempt12:04:261
1896724994619,8cyclictest131rcu_preempt12:04:251
1896724994615,22cyclictest131rcu_preempt12:07:461
189671999461,27cyclictest0-21swapper/008:48:180
189673499453,6cyclictest37-21ksoftirqd/307:10:013
1896734994527,13cyclictest275-21dbus-daemon08:05:013
189673499451,5cyclictest0-21swapper/308:49:293
189673199454,4cyclictest131rcu_preempt12:27:522
189673199453,4cyclictest131rcu_preempt08:15:012
1896731994530,11cyclictest1901345-21tr08:40:002
189673199452,3cyclictest131rcu_preempt08:25:012
1896731994516,9cyclictest30-21ksoftirqd/207:40:032
1896724994535,7cyclictest23-21ksoftirqd/112:10:131
1896724994535,7cyclictest23-21ksoftirqd/112:10:121
189671999452,39cyclictest1896907-21cron07:15:000
1896734994428,11cyclictest37-21ksoftirqd/308:30:023
1896734994428,11cyclictest37-21ksoftirqd/308:30:013
1896731994436,5cyclictest1901369-21kworker/2:109:08:482
189673199442,4cyclictest131rcu_preempt07:55:012
1896731994423,15cyclictest275-21dbus-daemon07:50:012
189672499445,4cyclictest131rcu_preempt09:49:431
1896724994419,16cyclictest23-21ksoftirqd/107:45:031
1896724994419,16cyclictest23-21ksoftirqd/107:45:021
1896724994417,17cyclictest23-21ksoftirqd/110:16:581
1896724994415,18cyclictest23-21ksoftirqd/108:20:021
189673499439,5cyclictest131rcu_preempt09:05:013
1896734994333,7cyclictest37-21ksoftirqd/307:25:013
1896734994332,7cyclictest309-21in:imuxsock08:35:033
1896734994327,12cyclictest37-21ksoftirqd/308:39:593
189672499434,4cyclictest131rcu_preempt08:50:011
189672499433,5cyclictest131rcu_preempt09:40:091
1896724994332,9cyclictest23-21ksoftirqd/111:09:031
1896724994332,9cyclictest23-21ksoftirqd/111:09:021
189672499431,9cyclictest0-21swapper/111:35:091
1896724994317,17cyclictest23-21ksoftirqd/108:10:021
1896724994311,5cyclictest131rcu_preempt12:20:011
1896719994329,9cyclictest309-21in:imuxsock07:25:000
189673499428,6cyclictest131rcu_preempt07:13:243
189673499428,5cyclictest131rcu_preempt08:15:013
189673499422,9cyclictest37-21ksoftirqd/309:10:023
189673499421,7cyclictest1-21systemd08:25:003
1896731994230,8cyclictest3467602-21systemd-journal07:20:002
1896731994230,8cyclictest3467602-21systemd-journal07:20:002
189672499428,5cyclictest131rcu_preempt10:12:251
1896724994227,10cyclictest3467602-21systemd-journal12:30:011
1896724994226,13cyclictest23-21ksoftirqd/108:55:001
189672499422,5cyclictest131rcu_preempt07:40:011
1896724994216,6cyclictest221rcuc/111:13:151
1896724994213,16cyclictest23-21ksoftirqd/110:47:571
189671999422,36cyclictest1903133-21sed09:10:030
189673499411,9cyclictest0-21swapper/307:17:353
189673499411,9cyclictest0-21swapper/307:17:343
189673199416,5cyclictest131rcu_preempt08:45:002
1896731994126,12cyclictest1899586-21date08:05:022
189673199412,5cyclictest131rcu_preempt08:20:002
189672499417,19cyclictest1900003-21kworker/u8:108:30:021
189672499417,19cyclictest1900003-21kworker/u8:108:30:021
1896724994134,5cyclictest221rcuc/110:21:331
1896724994130,8cyclictest1900545-21sshd08:21:251
1896724994126,11cyclictest3467602-21systemd-journal07:49:251
1896724994118,5cyclictest23-21ksoftirqd/111:26:431
189672499411,7cyclictest0-21swapper/110:00:531
1896724994115,17cyclictest23-21ksoftirqd/110:09:401
189672499411,4cyclictest131rcu_preempt11:44:301
189672499411,10cyclictest23-21ksoftirqd/110:28:071
189671999413,4cyclictest0-21swapper/008:10:020
189673499405,8cyclictest37-21ksoftirqd/308:55:003
189673199402,4cyclictest131rcu_preempt07:20:222
189671999403,9cyclictest12-21ksoftirqd/007:45:570
1896719994017,13cyclictest0-21swapper/007:10:010
189673499398,5cyclictest131rcu_preempt08:20:013
189673199392,7cyclictest0-21swapper/207:15:022
189671999392,31cyclictest1-21systemd09:05:010
1896734993831,5cyclictest37-21ksoftirqd/307:30:033
1896731993823,11cyclictest275-21dbus-daemon08:30:012
1896731993823,11cyclictest275-21dbus-daemon08:30:002
1896724993829,7cyclictest23-21ksoftirqd/109:05:011
189672499381,5cyclictest131rcu_preempt07:15:021
189673499371,7cyclictest150-21pr/ttyS207:57:383
1896731993725,9cyclictest1896728-21latency_hist07:10:012
189672499375,4cyclictest1899483-21latency_hist08:00:031
189672499371,4cyclictest131rcu_preempt08:33:261
189671999372,7cyclictest3467602-21systemd-journal07:50:250
189671999371,32cyclictest1901299-21sshd08:36:250
189671999361,31cyclictest3467602-21systemd-journal08:01:100
189672499359,6cyclictest131rcu_preempt07:27:261
189672499357,5cyclictest131rcu_preempt07:33:241
189672499354,5cyclictest131rcu_preempt08:57:251
189672499353,6cyclictest131rcu_preempt09:10:001
189672499351,5cyclictest0-21swapper/107:54:241
189672499341,7cyclictest0-21swapper/107:16:521
189672499341,7cyclictest0-21swapper/107:16:521
1896724993410,5cyclictest131rcu_preempt08:40:001
189672499281,4cyclictest131rcu_preempt07:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional