You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-14 - 14:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sun Jun 14, 2026 00:45:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1963566997624,16cyclictest1969651-21latency_hist21:05:011
1963566997416,18cyclictest2032924-21ssh23:31:421
1963566997218,21cyclictest2004703-21ssh22:28:411
1963560997164,2cyclictest0-21swapper/000:15:010
1963566997012,21cyclictest2024878-21ssh23:13:461
1963566997012,21cyclictest2024878-21ssh23:13:461
1963566997012,18cyclictest2006525-21ssh22:32:331
1963566996813,19cyclictest2010597-21sh22:41:291
1963566996717,15cyclictest1978275-21sh21:29:121
1963566996713,20cyclictest2043407-21ssh23:55:081
1963566996713,15cyclictest2019912-21ssh23:02:321
1963566996713,15cyclictest2019912-21ssh23:02:311
1963566996611,19cyclictest1985989-21ssh21:46:211
1963566996513,15cyclictest1981277-21ssh21:35:421
1963566996512,15cyclictest1973477-21ssh21:18:211
1963560996557,6cyclictest19350irq/34-mmc120:45:030
1963560996557,6cyclictest19350irq/34-mmc120:45:020
196357399647,5cyclictest131rcu_preempt21:40:013
196356699648,18cyclictest2041720-21ssh23:51:301
1963566996417,11cyclictest23-21ksoftirqd/122:04:381
1963566996410,20cyclictest2057417-21ssh00:26:481
1963566996311,19cyclictest2029166-21sh23:23:191
1963560996350,6cyclictest16950irq/224-800000023:44:590
1963560996326,33cyclictest12-21ksoftirqd/021:13:130
1963566996216,15cyclictest2027138-21sh23:18:501
1963566996213,18cyclictest2059983-21sh00:32:321
1963560996243,13cyclictest1-21systemd22:10:010
1963566996116,7cyclictest23-21ksoftirqd/100:21:251
1963566996115,10cyclictest23-21ksoftirqd/100:35:251
196356099611,41cyclictest131rcu_preempt00:25:520
1963573996012,6cyclictest131rcu_preempt22:04:363
1963569996051,6cyclictest1967667-21kworker/2:222:37:332
1963569996051,6cyclictest1967667-21kworker/2:222:37:322
1963569996024,18cyclictest30-21ksoftirqd/222:28:482
1963566996013,11cyclictest23-21ksoftirqd/100:00:211
1963566996012,14cyclictest1974890-21ssh21:21:181
1963566996011,14cyclictest275-21dbus-daemon20:45:021
1963566996011,14cyclictest275-21dbus-daemon20:45:011
1963560996020,32cyclictest12-21ksoftirqd/022:40:010
1963560996020,32cyclictest12-21ksoftirqd/022:40:000
196356699596,17cyclictest2009358-21rm22:39:091
196356699596,17cyclictest2009358-21rm22:39:081
1963566995917,7cyclictest23-21ksoftirqd/100:05:151
1963566995916,9cyclictest23-21ksoftirqd/122:14:521
1963566995914,13cyclictest2040352-21ssh23:48:321
1963566995913,13cyclictest1964547-21munin-run19:30:001
1963560995925,7cyclictest12-21ksoftirqd/023:22:120
1963569995849,5cyclictest2023178-21kworker/2:123:36:212
196356699589,19cyclictest1985117-21ssh21:44:411
1963566995834,16cyclictest275-21dbus-daemon20:30:001
1963566995810,17cyclictest1967555-21cat20:25:021
1963560995847,7cyclictest16850irq/134-800000000:10:020
1963566995715,11cyclictest0-21swapper/121:53:391
1963566995715,10cyclictest23-21ksoftirqd/100:18:211
1963566995714,9cyclictest23-21ksoftirqd/122:58:241
1963566995713,17cyclictest2000316-21sh22:18:441
1963566995713,17cyclictest2000316-21sh22:18:431
1963566995713,14cyclictest0-21swapper/123:27:331
1963566995713,10cyclictest23-21ksoftirqd/122:49:081
1963566995712,13cyclictest2050827-21ssh00:11:561
1963560995724,7cyclictest12-21ksoftirqd/023:47:330
1963560995718,36cyclictest12-21ksoftirqd/020:05:020
196356699568,16cyclictest1965830-21munin-run19:55:001
1963566995614,7cyclictest23-21ksoftirqd/122:22:231
196356099564,45cyclictest2000851-21munin-run22:20:010
196356099564,45cyclictest2000851-21munin-run22:20:000
196356099564,32cyclictest16950irq/224-800000022:00:020
1963560995637,14cyclictest1-21systemd21:05:000
1963573995516,6cyclictest131rcu_preempt19:40:013
1963569995547,5cyclictest2023178-21kworker/2:123:22:422
1963569995547,5cyclictest1967667-21kworker/2:220:45:022
1963569995547,5cyclictest1967667-21kworker/2:220:45:022
196356699558,15cyclictest1980860-21munin-run21:35:021
196356699558,14cyclictest2022910-21ssh23:09:311
1963560995528,11cyclictest2022307-21ssh23:07:580
1963560995523,29cyclictest12-21ksoftirqd/023:35:490
196357399545,9cyclictest37-21ksoftirqd/300:40:013
1963573995437,13cyclictest1-21systemd21:45:013
1963573995415,9cyclictest131rcu_preempt22:16:073
1963573995415,9cyclictest131rcu_preempt22:16:073
1963569995425,21cyclictest30-21ksoftirqd/221:27:052
1963569995423,22cyclictest30-21ksoftirqd/200:35:172
1963569995423,21cyclictest30-21ksoftirqd/222:00:062
1963566995412,14cyclictest1969731-21diskmemload23:39:551
1963560995440,7cyclictest0-21swapper/023:20:010
1963560995422,3cyclictest0-21swapper/022:30:180
1963573995334,14cyclictest2045335-21systemctl23:59:593
196356999535,5cyclictest131rcu_preempt19:45:002
1963569995345,5cyclictest2023178-21kworker/2:123:28:302
1963569995345,5cyclictest2023178-21kworker/2:100:11:302
1963569995345,5cyclictest2023178-21kworker/2:100:03:382
1963569995345,5cyclictest1967667-21kworker/2:221:39:342
1963569995345,4cyclictest1964305-21kworker/2:020:18:312
1963569995344,6cyclictest1967667-21kworker/2:221:20:262
1963569995336,12cyclictest1-21systemd00:00:002
1963566995332,10cyclictest309-21in:imuxsock21:10:011
1963566995332,10cyclictest309-21in:imuxsock21:10:001
1963566995330,18cyclictest1991187-21ssh21:58:091
1963560995337,12cyclictest1-21systemd20:15:020
1963560995335,4cyclictest16850irq/134-800000023:55:010
1963560995326,12cyclictest2015885-21ssh22:53:430
196356099532,36cyclictest1987418-21ssh21:49:510
196356099531,28cyclictest12-21ksoftirqd/000:00:210
1963573995218,6cyclictest131rcu_preempt21:49:443
1963573995213,6cyclictest131rcu_preempt00:11:433
1963569995244,5cyclictest2023178-21kworker/2:100:32:232
1963569995244,5cyclictest1967667-21kworker/2:221:40:112
1963569995244,5cyclictest1967667-21kworker/2:200:20:232
1963569995244,5cyclictest1967667-21kworker/2:200:16:302
1963569995243,6cyclictest1964305-21kworker/2:021:47:342
1963566995214,9cyclictest23-21ksoftirqd/122:51:561
1963566995214,6cyclictest2037887-21ssh23:42:571
1963560995230,16cyclictest275-21dbus-daemon22:45:010
1963560995224,3cyclictest131rcu_preempt22:28:450
1963560995223,14cyclictest2032478-21ssh23:30:310
1963560995222,26cyclictest1998677-21ssh22:15:020
196356099521,7cyclictest2045310-40mandb00:00:010
196357399517,6cyclictest131rcu_preempt20:29:593
1963573995115,5cyclictest131rcu_preempt22:53:183
1963569995143,5cyclictest2023178-21kworker/2:123:33:172
1963569995143,5cyclictest1987536-21kworker/2:323:05:532
1963569995143,5cyclictest1987536-21kworker/2:322:51:002
1963569995143,5cyclictest1987536-21kworker/2:322:19:342
1963569995143,5cyclictest1987536-21kworker/2:322:19:342
1963569995143,5cyclictest1967667-21kworker/2:221:53:022
1963569995143,5cyclictest1967667-21kworker/2:220:50:182
1963569995143,5cyclictest1967667-21kworker/2:220:40:022
1963569995143,5cyclictest1964305-21kworker/2:021:34:422
1963569995143,5cyclictest1964305-21kworker/2:021:14:042
1963569995142,6cyclictest2023178-21kworker/2:123:54:302
1963569995142,6cyclictest1987536-21kworker/2:322:34:202
1963569995142,5cyclictest2023178-21kworker/2:123:46:262
1963569995142,5cyclictest1987536-21kworker/2:322:59:142
1963569995131,15cyclictest275-21dbus-daemon20:05:002
1963573995040,7cyclictest361rcuc/300:20:013
1963573995033,12cyclictest3467602-21systemd-journal21:05:003
196357399501,8cyclictest2056525-21sh00:25:013
1963573995014,6cyclictest131rcu_preempt00:06:203
1963573995012,6cyclictest131rcu_preempt22:58:183
1963573995012,6cyclictest131rcu_preempt22:34:493
1963573995011,7cyclictest131rcu_preempt00:30:003
1963569995043,4cyclictest2023178-21kworker/2:100:29:282
1963569995042,5cyclictest2023178-21kworker/2:100:06:192
1963569995042,5cyclictest1987536-21kworker/2:323:03:162
1963569995042,5cyclictest1987536-21kworker/2:323:03:152
1963569995042,5cyclictest1987536-21kworker/2:322:45:042
1963569995042,5cyclictest1987536-21kworker/2:322:11:192
1963569995042,5cyclictest1967667-21kworker/2:223:10:152
1963569995042,5cyclictest1967667-21kworker/2:223:10:152
1963569995042,5cyclictest1967667-21kworker/2:222:05:592
1963569995041,6cyclictest1964305-21kworker/2:021:16:272
1963569995040,6cyclictest1987536-21kworker/2:321:56:202
1963566995036,10cyclictest1970681-21ssh21:11:541
196356099504,3cyclictest131rcu_preempt00:22:420
1963560995041,5cyclictest16850irq/134-800000021:10:020
1963560995041,5cyclictest16850irq/134-800000021:10:010
1963560995025,19cyclictest364-21snmpd21:21:290
196356099502,3cyclictest131rcu_preempt21:51:530
1963560995022,3cyclictest131rcu_preempt00:19:050
1963560995019,15cyclictest2019060-21ssh23:00:150
1963560995019,15cyclictest2019060-21ssh23:00:140
196357399499,5cyclictest131rcu_preempt22:43:203
196357399499,5cyclictest131rcu_preempt19:20:023
196357399499,16cyclictest131rcu_preempt23:38:013
196357399496,8cyclictest37-21ksoftirqd/320:00:003
196357399496,8cyclictest37-21ksoftirqd/320:00:003
1963573994931,12cyclictest1-21systemd23:50:003
1963573994916,23cyclictest131rcu_preempt21:30:023
1963573994914,5cyclictest131rcu_preempt23:44:323
1963573994912,6cyclictest131rcu_preempt23:53:493
1963573994912,6cyclictest131rcu_preempt21:51:523
1963569994941,5cyclictest2023178-21kworker/2:123:17:502
1963569994941,5cyclictest1942844-21kworker/2:219:26:062
1963569994940,6cyclictest1967667-21kworker/2:222:40:552
1963569994940,5cyclictest1987536-21kworker/2:322:24:242
196356699497,14cyclictest1966091-21munin-run20:00:011
196356699497,14cyclictest1966091-21munin-run20:00:001
1963566994936,10cyclictest1994944-21ssh22:06:181
1963566994926,16cyclictest1964784-21cron19:35:001
196356099496,29cyclictest0-21swapper/022:55:120
1963560994923,11cyclictest1984843-21ssh21:43:540
1963560994918,28cyclictest1980838-21latency_hist21:35:020
1963573994836,9cyclictest37-21ksoftirqd/319:50:013
1963573994810,6cyclictest131rcu_preempt23:05:493
1963569994841,5cyclictest1967667-21kworker/2:220:46:232
1963569994840,5cyclictest1967667-21kworker/2:223:40:042
1963566994832,11cyclictest1968956-21munin-run20:55:001
196356099482,3cyclictest131rcu_preempt21:29:530
1963573994736,6cyclictest361rcuc/321:25:003
1963573994716,18cyclictest131rcu_preempt21:56:353
1963573994713,6cyclictest131rcu_preempt00:32:033
1963573994712,7cyclictest131rcu_preempt23:03:583
1963573994712,7cyclictest131rcu_preempt23:03:583
1963573994712,6cyclictest131rcu_preempt21:20:013
1963573994710,8cyclictest131rcu_preempt23:20:383
1963569994739,5cyclictest1942844-21kworker/2:219:21:232
1963569994739,5cyclictest1942844-21kworker/2:219:21:222
1963560994740,4cyclictest16950irq/224-800000022:50:000
196357399469,6cyclictest131rcu_preempt23:13:453
196357399469,6cyclictest131rcu_preempt23:13:443
196357399469,6cyclictest131rcu_preempt22:39:173
196357399469,6cyclictest131rcu_preempt22:39:163
196357399461,5cyclictest0-21swapper/320:20:023
1963573994614,5cyclictest131rcu_preempt22:26:593
1963573994614,5cyclictest131rcu_preempt22:25:013
1963573994612,5cyclictest131rcu_preempt21:31:103
1963569994639,4cyclictest1967667-21kworker/2:220:57:502
1963569994638,5cyclictest1964305-21kworker/2:021:05:042
1963569994638,5cyclictest1964305-21kworker/2:021:05:042
1963569994638,5cyclictest1964305-21kworker/2:020:20:272
1963569994638,5cyclictest1964305-21kworker/2:020:11:312
1963569994638,5cyclictest1964305-21kworker/2:020:05:342
1963569994638,5cyclictest1964305-21kworker/2:019:52:202
1963569994638,5cyclictest1964305-21kworker/2:019:46:292
1963569994638,5cyclictest1964305-21kworker/2:019:39:592
1963569994638,5cyclictest1964305-21kworker/2:019:30:512
196356699468,23cyclictest1965045-21modprobe19:40:001
1963566994634,9cyclictest1968132-21sshd20:37:321
1963566994633,10cyclictest1963768-21munin-run19:15:011
1963566994624,17cyclictest1968678-21cron20:50:001
196356099463,14cyclictest1972134-21modprobe21:15:030
1963573994535,7cyclictest37-21ksoftirqd/322:05:303
1963573994512,6cyclictest131rcu_preempt22:11:463
1963573994512,5cyclictest131rcu_preempt23:29:413
1963573994511,5cyclictest131rcu_preempt23:19:003
1963569994538,4cyclictest1942844-21kworker/2:219:10:222
1963569994537,5cyclictest1964305-21kworker/2:019:57:212
1963569994537,5cyclictest1964305-21kworker/2:019:57:202
196356099454,38cyclictest0-21swapper/000:40:000
196356099452,3cyclictest131rcu_preempt23:27:330
1963560994512,25cyclictest12-21ksoftirqd/022:01:040
196357399446,5cyclictest131rcu_preempt20:15:003
196357399442,5cyclictest131rcu_preempt19:32:493
1963573994417,5cyclictest131rcu_preempt20:05:023
1963573994417,23cyclictest361rcuc/321:10:023
1963573994417,23cyclictest361rcuc/321:10:023
1963573994413,6cyclictest131rcu_preempt23:35:013
1963573994411,6cyclictest131rcu_preempt21:11:513
1963569994436,5cyclictest1967667-21kworker/2:220:33:372
1963569994435,6cyclictest1964305-21kworker/2:021:00:192
1963566994431,10cyclictest1963970-21sshd19:15:331
1963560994436,6cyclictest12-21ksoftirqd/022:22:290
1963560994428,12cyclictest3467602-21systemd-journal20:33:040
1963560994428,12cyclictest2023970-21ssh23:11:340
1963560994428,12cyclictest2023970-21ssh23:11:340
1963560994427,14cyclictest2060418-21ssh00:33:400
196356099442,36cyclictest3467602-21systemd-journal19:15:020
196356099441,38cyclictest1965019-21sshd19:36:330
1963573994334,7cyclictest37-21ksoftirqd/319:55:023
1963573994323,14cyclictest1-21systemd00:00:053
1963573994310,7cyclictest131rcu_preempt22:47:113
1963569994335,5cyclictest1942844-21kworker/2:219:15:032
1963566994331,9cyclictest1966333-21sshd20:02:331
1963566994330,9cyclictest48-21kcompactd019:23:061
1963566994330,9cyclictest48-21kcompactd019:23:061
1963566994329,10cyclictest48-21kcompactd019:49:441
1963566994329,10cyclictest1965342-21munin-run19:45:021
1963566994328,10cyclictest48-21kcompactd020:59:391
1963566994328,10cyclictest1966865-21munin-run20:15:001
1963560994336,4cyclictest16950irq/224-800000019:55:010
1963560994336,4cyclictest16950irq/224-800000019:25:020
1963560994336,4cyclictest16950irq/224-800000019:25:010
1963560994327,12cyclictest1967572-21sed20:25:020
1963560994326,14cyclictest1981746-21ssh21:36:450
1963573994232,7cyclictest37-21ksoftirqd/320:25:013
196357399421,6cyclictest0-21swapper/319:21:003
196357399421,6cyclictest0-21swapper/319:20:593
1963569994234,5cyclictest1964305-21kworker/2:020:25:182
196356699429,21cyclictest1967116-21modprobe20:20:001
1963560994225,13cyclictest1968123-21sshd20:35:330
1963573994122,14cyclictest1-21systemd20:55:013
1963573994121,13cyclictest37-21ksoftirqd/320:49:593
1963573994116,17cyclictest37-21ksoftirqd/320:56:333
1963566994127,10cyclictest1966614-21munin-run20:10:011
1963566994123,14cyclictest364-21snmpd20:32:251
1963560994131,7cyclictest1-21systemd20:20:000
1963560994127,9cyclictest309-21in:imuxsock19:19:590
1963560994113,18cyclictest0-21swapper/019:10:010
1963573994031,7cyclictest37-21ksoftirqd/320:06:153
196357399401,8cyclictest150-21pr/ttyS219:42:593
196357399401,6cyclictest150-21pr/ttyS220:40:493
196357399401,6cyclictest150-21pr/ttyS220:40:493
1963560994031,6cyclictest1968677-21cron20:49:590
196356099402,6cyclictest1965590-21munin-run19:50:010
1963560994025,11cyclictest1969175-21sshd20:58:320
196356099401,35cyclictest0-21swapper/019:43:380
1963573993931,6cyclictest37-21ksoftirqd/320:36:063
196357399391,8cyclictest0-21swapper/319:26:453
196357399391,7cyclictest150-21pr/ttyS219:13:063
196357399391,6cyclictest150-21pr/ttyS220:34:343
196356099394,6cyclictest3467602-21systemd-journal20:00:010
196356099394,6cyclictest3467602-21systemd-journal20:00:000
1963560993928,8cyclictest12-21ksoftirqd/019:35:020
196356099384,4cyclictest131rcu_preempt20:30:010
1963560993826,10cyclictest12-21ksoftirqd/019:30:010
196357399372,5cyclictest131rcu_preempt19:10:013
196356699373,5cyclictest131rcu_preempt19:10:031
1963560993726,9cyclictest12-21ksoftirqd/020:55:020
196356099351,5cyclictest1966628-21munin-run20:10:010
1963569993323,7cyclictest1963572-21sed19:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional