You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-12 - 23:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Thu Mar 12, 2026 12:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1920980998030,13cyclictest275-21dbus-daemon09:35:021
1920976997832,42cyclictest1922464-21cron07:40:000
1920980997435,12cyclictest3467602-21systemd-journal10:12:321
192098699682,61cyclictest1980622-21ssh11:10:013
1920986996822,40cyclictest3467602-21systemd-journal10:00:003
192098699672,12cyclictest3467602-21systemd-journal10:12:233
192098699671,20cyclictest1995398-21ssh11:42:463
192098699671,20cyclictest1995398-21ssh11:42:463
192098699671,11cyclictest1983485-21sshd11:16:043
1920986996611,51cyclictest1959201-21ssh10:21:133
192098699652,59cyclictest1921468-21munin-run07:20:023
192098699652,17cyclictest1998451-21sh11:49:493
192098699651,4cyclictest1942377-21ssh09:43:553
192098699651,14cyclictest1976687-21ssh11:00:113
1920986996422,38cyclictest1931211-21sh09:18:473
1920986996421,38cyclictest1961290-21sh10:25:483
192098699641,8cyclictest1928020-21ssh09:11:313
192098699641,5cyclictest1966469-21ssh10:37:353
192098699641,16cyclictest1957297-21ssh10:17:093
192098699641,11cyclictest1990438-21ssh11:31:263
192098699632,16cyclictest1972218-21ssh10:50:043
192098699631,21cyclictest1932212-21ssh09:20:443
192098699631,20cyclictest1987604-21ssh11:25:063
192098699631,20cyclictest1937686-21ssh09:33:183
192098699631,17cyclictest1971240-21ssh10:48:173
192098699631,10cyclictest2017145-21ssh12:31:233
192098699631,10cyclictest1951066-21ssh10:03:103
192098699631,10cyclictest1934717-21ssh09:26:263
192098699622,9cyclictest3467602-21systemd-journal11:55:443
192098699622,6cyclictest2007469-21munin-run12:10:013
192098699621,9cyclictest2009916-21ssh12:15:073
192098699621,9cyclictest2009916-21ssh12:15:073
192098699621,9cyclictest1992373-21ssh11:35:533
192098699621,8cyclictest1968379-21ssh10:41:463
1920976996233,9cyclictest2000751-21munin-run11:55:000
192097699621,40cyclictest1921688-21kworker/u8:107:30:030
192098699612,8cyclictest2019508-21ssh12:36:473
192098699612,7cyclictest1981017-21ssh11:10:273
192098699611,9cyclictest1938825-21ssh09:35:373
192098699611,14cyclictest1946833-21ssh09:53:533
192098699611,14cyclictest1946833-21ssh09:53:533
192098699611,10cyclictest2012595-21sshd12:21:113
192098699611,10cyclictest1986573-21ssh11:23:143
192098699602,6cyclictest1952529-21ssh10:06:173
192098699602,54cyclictest2014922-21sh12:26:293
192098699601,8cyclictest1964224-21ssh10:32:363
192098699601,8cyclictest1944505-21ssh09:48:343
192098699601,7cyclictest1998873-21ssh11:50:263
192098699601,5cyclictest2004859-21ssh12:04:073
1920983996049,7cyclictest1906355-21kworker/2:311:39:372
1920976996028,15cyclictest1929278-21ssh09:14:400
192098699592,17cyclictest1974847-21ssh10:56:053
1920983995924,7cyclictest1928028-21sh09:11:322
1920976995928,3cyclictest131rcu_preempt10:41:150
1920976995928,14cyclictest1992555-21ssh11:36:180
192098699582,4cyclictest2007922-21ssh12:10:373
192098699581,14cyclictest1927260-21date09:10:013
192098699572,5cyclictest1924292-21munin-run08:15:013
192098699572,16cyclictest1922470-21munin-run07:40:003
192098699571,4cyclictest1921145-21sshd07:13:203
1920983995750,4cyclictest1927145-21diskmemload11:30:042
1920983995749,5cyclictest1906355-21kworker/2:310:00:062
192097699574,26cyclictest0-21swapper/010:17:510
192097699573,4cyclictest131rcu_preempt12:30:360
1920976995730,11cyclictest2019444-21ssh12:36:380
1920976995729,13cyclictest1950704-21ssh10:02:190
1920976995725,28cyclictest12-21ksoftirqd/012:20:020
1920976995725,28cyclictest12-21ksoftirqd/012:20:020
1920976995724,29cyclictest12-21ksoftirqd/010:51:460
1920976995719,33cyclictest16850irq/134-800000011:19:540
192098699562,6cyclictest364-21snmpd07:27:433
1920976995645,7cyclictest16850irq/134-800000011:05:010
1920976995620,32cyclictest16850irq/134-800000011:14:240
1920976995618,33cyclictest16850irq/134-800000009:25:320
192098699552,6cyclictest1923008-21munin-run07:50:023
192098699551,5cyclictest1925544-21sshd08:35:383
1920976995521,28cyclictest0-21swapper/011:30:030
1920976995520,4cyclictest2005285-21latency_hist12:05:020
1920976995519,32cyclictest16850irq/134-800000011:32:580
1920976995519,32cyclictest16850irq/134-800000009:49:560
1920976995519,31cyclictest16850irq/134-800000009:32:120
1920976995518,6cyclictest16850irq/134-800000010:39:370
192098699542,6cyclictest1923742-21sshd08:00:203
192098699542,4cyclictest1926342-21sshd08:53:203
192098699542,46cyclictest1921053-21grep07:10:023
192098699542,16cyclictest1924028-21munin-run08:10:003
1920983995444,5cyclictest1906355-21kworker/2:307:30:002
1920976995435,16cyclictest12-21ksoftirqd/009:40:020
1920976995426,12cyclictest1963515-21ssh10:30:460
192097699542,44cyclictest0-21swapper/010:23:260
1920976995419,18cyclictest16850irq/134-800000011:48:310
1920976995419,17cyclictest16850irq/134-800000011:09:400
1920976995418,7cyclictest16850irq/134-800000010:11:180
1920976995417,33cyclictest16850irq/134-800000010:07:070
192098699532,5cyclictest1925324-21munin-run08:35:003
192098699532,5cyclictest1925324-21munin-run08:35:003
192098699531,5cyclictest1922210-21munin-run07:35:003
1920986995311,37cyclictest1925061-21cron08:30:003
1920986995311,37cyclictest1925061-21cron08:30:003
192098699531,12cyclictest1926591-21sshd08:58:213
1920983995345,5cyclictest1960783-21kworker/2:211:15:462
1920983995345,5cyclictest1960783-21kworker/2:211:14:592
1920983995345,5cyclictest1960783-21kworker/2:211:05:292
1920983995345,5cyclictest1906355-21kworker/2:312:32:012
1920983995345,5cyclictest1906355-21kworker/2:312:27:472
1920983995345,5cyclictest1906355-21kworker/2:311:55:012
1920983995345,5cyclictest1906355-21kworker/2:310:06:222
1920983995345,5cyclictest1906355-21kworker/2:308:25:002
1920983995345,5cyclictest1906355-21kworker/2:308:25:002
1920983995344,5cyclictest1927107-21kworker/2:009:36:482
1920983995344,5cyclictest1906355-21kworker/2:310:29:232
1920983995343,6cyclictest1906355-21kworker/2:309:51:362
1920983995343,6cyclictest1906355-21kworker/2:309:51:362
192098099531,29cyclictest1955205-21kworker/u8:110:15:031
1920976995319,6cyclictest16850irq/134-800000012:20:590
1920976995319,30cyclictest16850irq/134-800000010:47:350
1920976995319,17cyclictest16850irq/134-800000009:44:030
1920976995318,5cyclictest16850irq/134-800000009:59:500
1920976995318,18cyclictest16850irq/134-800000012:12:030
1920976995318,17cyclictest16850irq/134-800000010:56:370
1920976995317,6cyclictest16850irq/134-800000009:51:560
1920976995317,6cyclictest16850irq/134-800000009:51:560
1920976995317,32cyclictest16850irq/134-800000011:22:430
1920983995244,5cyclictest1906355-21kworker/2:312:22:082
1920983995244,5cyclictest1906355-21kworker/2:312:08:242
1920983995244,5cyclictest1906355-21kworker/2:311:47:282
1920983995244,5cyclictest1906355-21kworker/2:311:42:292
1920983995244,5cyclictest1906355-21kworker/2:311:42:292
1920983995244,5cyclictest1906355-21kworker/2:310:41:012
1920983995244,5cyclictest1906355-21kworker/2:310:18:512
1920983995243,5cyclictest1906355-21kworker/2:312:00:212
1920983995243,5cyclictest1906355-21kworker/2:311:59:472
1920980995233,15cyclictest2020865-21systemd-run12:40:011
1920976995242,6cyclictest1921428-21sendmail07:20:010
1920976995217,18cyclictest16850irq/134-800000010:25:420
1920976995217,17cyclictest16850irq/134-800000011:58:180
1920976995217,17cyclictest16850irq/134-800000011:42:360
1920976995217,17cyclictest16850irq/134-800000011:42:360
1920976995216,18cyclictest16850irq/134-800000009:20:290
1920976995216,17cyclictest16850irq/134-800000009:15:400
192098699511,5cyclictest1-21systemd08:00:003
192098699511,4cyclictest1926088-21sshd08:48:203
192098699511,4cyclictest1926088-21sshd08:48:203
1920983995143,5cyclictest1960783-21kworker/2:211:20:472
1920983995143,5cyclictest1960783-21kworker/2:211:05:022
1920983995143,5cyclictest1906355-21kworker/2:310:35:202
1920983995143,5cyclictest1906355-21kworker/2:310:32:432
1920983995143,5cyclictest1906355-21kworker/2:309:18:342
1920983995142,5cyclictest1906355-21kworker/2:312:39:062
1920983995142,5cyclictest1906355-21kworker/2:310:55:002
1920983995142,5cyclictest1906355-21kworker/2:309:22:192
1920983995141,6cyclictest1960783-21kworker/2:211:30:012
1920983995140,7cyclictest1906355-21kworker/2:312:17:412
1920983995140,7cyclictest1906355-21kworker/2:312:17:412
192098399513,4cyclictest131rcu_preempt12:14:472
1920983995042,5cyclictest1906355-21kworker/2:310:47:172
1920983995042,5cyclictest1906355-21kworker/2:309:46:032
1920983995041,5cyclictest1906355-21kworker/2:310:10:532
1920976995042,5cyclictest16950irq/224-800000008:15:020
192098699492,42cyclictest1924527-21cron08:20:003
192098699491,6cyclictest364-21snmpd07:42:463
192098699491,3cyclictest1923211-21modprobe07:50:213
192098699491,11cyclictest364-21snmpd07:20:073
1920983994941,5cyclictest1906355-21kworker/2:310:57:552
1920983994940,5cyclictest1906355-21kworker/2:310:23:492
1920983994940,5cyclictest1906355-21kworker/2:309:57:542
1920980994930,13cyclictest275-21dbus-daemon08:30:001
1920980994930,13cyclictest275-21dbus-daemon08:30:001
192097699492,33cyclictest151rcuc/009:05:020
192097699492,33cyclictest151rcuc/009:05:020
1920976994921,13cyclictest2006268-21ssh12:07:060
1920976994914,18cyclictest16850irq/134-800000012:27:130
192098699481,3cyclictest364-21snmpd08:42:573
192098699481,12cyclictest364-21snmpd09:03:303
192098699481,12cyclictest364-21snmpd09:03:303
1920983994840,5cyclictest1927107-21kworker/2:009:30:032
1920983994840,4cyclictest1906355-21kworker/2:307:10:022
1920980994836,8cyclictest309-21in:imuxsock09:00:001
1920976994830,13cyclictest3467602-21systemd-journal08:05:010
192098699472,6cyclictest1924807-21cron08:25:003
192098699472,6cyclictest1924807-21cron08:25:003
1920983994729,12cyclictest1-21systemd08:05:012
1920980994736,7cyclictest1924551-21sendmail08:20:001
1920980994731,12cyclictest1-21systemd07:34:591
1920976994735,9cyclictest1-21systemd08:20:010
1920983994638,5cyclictest1906355-21kworker/2:309:26:532
1920983994637,6cyclictest1906355-21kworker/2:307:53:152
1920983994630,11cyclictest275-21dbus-daemon08:30:002
1920983994630,11cyclictest275-21dbus-daemon08:29:592
1920983994619,6cyclictest30-21ksoftirqd/208:40:002
192098399461,3cyclictest131rcu_preempt08:50:102
1920980994630,12cyclictest1-21systemd10:10:011
1920976994631,11cyclictest1-21systemd07:15:000
192097699462,39cyclictest120-21kswapd009:10:010
1920983994535,6cyclictest1906355-21kworker/2:307:50:032
1920983994532,10cyclictest3467602-21systemd-journal08:14:212
192098099453,4cyclictest131rcu_preempt12:32:421
1920980994531,11cyclictest1-21systemd08:45:021
1920980994529,11cyclictest275-21dbus-daemon10:45:011
192097699452,39cyclictest1922097-21latency_hist07:30:010
1920976994517,25cyclictest16850irq/134-800000008:30:020
1920976994517,25cyclictest16850irq/134-800000008:30:010
1920983994436,5cyclictest1906355-21kworker/2:309:45:012
1920980994431,9cyclictest2015280-21sshd12:27:201
1920980994431,10cyclictest23-21ksoftirqd/110:45:301
1920980994430,11cyclictest23-21ksoftirqd/110:53:031
1920980994429,11cyclictest23-21ksoftirqd/111:25:001
1920983994322,12cyclictest1927158-21modprobe09:10:002
192098099434,3cyclictest131rcu_preempt09:50:001
1920980994327,13cyclictest3467602-21systemd-journal09:55:191
1920980994327,12cyclictest275-21dbus-daemon12:20:011
1920980994327,12cyclictest275-21dbus-daemon12:20:011
1920980994326,13cyclictest2003178-21wc12:00:021
1920980994324,12cyclictest309-21in:imuxsock09:45:011
1920976994328,11cyclictest3467602-21systemd-journal09:00:000
1920983994235,4cyclictest1906355-21kworker/2:308:42:232
1920983994234,5cyclictest1906355-21kworker/2:307:41:252
1920983994227,9cyclictest274-21cron09:00:002
1920983994220,15cyclictest30-21ksoftirqd/207:15:002
192098099423,3cyclictest131rcu_preempt08:05:011
1920980994228,11cyclictest23-21ksoftirqd/111:27:471
1920980994228,11cyclictest1984815-21sshd11:19:201
1920980994227,11cyclictest275-21dbus-daemon07:45:011
1920980994224,13cyclictest3467602-21systemd-journal11:05:001
1920983994126,11cyclictest309-21in:imuxsock08:17:002
192098099413,3cyclictest131rcu_preempt10:40:001
1920980994130,7cyclictest3467602-21systemd-journal11:40:001
1920980994128,10cyclictest23-21ksoftirqd/110:27:401
192098099412,6cyclictest0-21swapper/109:36:001
1920980994125,11cyclictest23-21ksoftirqd/109:22:201
192098099412,3cyclictest131rcu_preempt09:14:541
192097699411,10cyclictest1926363-21munin-run08:55:000
1920983994031,6cyclictest1906355-21kworker/2:307:21:532
1920983994013,24cyclictest1926906-21unin-run09:05:012
1920983994013,24cyclictest1926906-21unin-run09:05:002
1920980994027,10cyclictest23-21ksoftirqd/112:10:051
1920980994024,13cyclictest1991864-21latency_hist11:35:021
1920976994033,4cyclictest16950irq/224-800000008:32:200
1920976994033,4cyclictest16950irq/224-800000008:32:200
1920976994029,8cyclictest12-21ksoftirqd/008:10:000
1920983993931,5cyclictest1906355-21kworker/2:307:35:002
1920983993926,10cyclictest275-21dbus-daemon08:10:012
192098399391,3cyclictest131rcu_preempt08:35:022
192098399391,3cyclictest131rcu_preempt08:35:012
1920980993929,8cyclictest23-21ksoftirqd/109:27:131
1920980993928,8cyclictest23-21ksoftirqd/112:10:001
1920980993918,6cyclictest23-21ksoftirqd/110:31:121
192098099391,3cyclictest131rcu_preempt10:24:201
1920976993924,12cyclictest1926073-21sshd08:45:210
1920976993924,12cyclictest1926073-21sshd08:45:200
1920976993922,12cyclictest12-21ksoftirqd/008:40:010
1920983993830,5cyclictest1906355-21kworker/2:307:35:482
1920980993826,9cyclictest23-21ksoftirqd/112:23:011
1920980993826,10cyclictest23-21ksoftirqd/111:48:131
1920980993826,10cyclictest23-21ksoftirqd/110:03:421
192098099382,3cyclictest131rcu_preempt08:25:001
192098099382,3cyclictest131rcu_preempt08:24:591
1920980993823,12cyclictest23-21ksoftirqd/111:10:021
1920980993822,13cyclictest23-21ksoftirqd/108:40:011
1920980993822,11cyclictest364-21snmpd09:55:011
1920980993822,11cyclictest364-21snmpd09:55:011
1920976993832,4cyclictest16950irq/224-800000007:42:200
192097699382,6cyclictest1925859-21munin-run08:45:010
192097699382,6cyclictest1923525-21sendmail-msp08:00:000
192097699382,31cyclictest1924820-21munin-run08:25:010
192097699382,31cyclictest1924820-21munin-run08:25:000
1920976993822,12cyclictest1-21systemd07:50:010
192098399373,26cyclictest1926111-21grep08:50:012
192098399373,26cyclictest1926111-21grep08:50:012
1920983993727,7cyclictest1923503-21munin-plugin-st08:00:002
1920980993730,4cyclictest1976399-21sendmail11:00:011
1920980993730,4cyclictest1-21systemd08:35:001
1920980993730,4cyclictest1-21systemd08:35:001
1920980993728,7cyclictest23-21ksoftirqd/112:00:321
192098099372,7cyclictest0-21swapper/111:11:191
1920980993726,8cyclictest23-21ksoftirqd/107:30:011
1920980993725,9cyclictest23-21ksoftirqd/107:15:011
1920980993724,10cyclictest1-21systemd07:25:011
1920980993722,10cyclictest1930627-21ssh09:17:251
1920980993721,12cyclictest275-21dbus-daemon08:55:001
1920980993721,11cyclictest275-21dbus-daemon07:40:001
1920980993721,11cyclictest23-21ksoftirqd/107:20:001
192097699371,34cyclictest0-21swapper/007:25:010
192098099368,8cyclictest0-21swapper/107:10:021
192098099364,3cyclictest131rcu_preempt11:51:461
1920980993626,7cyclictest23-21ksoftirqd/107:55:011
192098099362,3cyclictest131rcu_preempt08:15:011
192098099361,3cyclictest131rcu_preempt11:45:001
192098099361,3cyclictest131rcu_preempt11:45:001
1920976993624,8cyclictest275-21dbus-daemon07:55:000
192098399356,15cyclictest1921417-21modprobe07:20:012
1920980993525,7cyclictest23-21ksoftirqd/107:50:021
1920980993523,9cyclictest23-21ksoftirqd/108:10:001
1920980993521,10cyclictest23-21ksoftirqd/109:10:011
1920980993422,8cyclictest1923498-21cron08:00:001
1920980993316,13cyclictest1-21systemd08:50:011
1920980993316,13cyclictest1-21systemd08:50:011
192098099302,24cyclictest48-21kcompactd009:00:501
192098099302,24cyclictest48-21kcompactd009:00:501
1920976993018,9cyclictest1921124-21date07:10:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional