You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-22 - 23:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Wed Apr 22, 2026 12:45:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
191760699871,4cyclictest150-21pr/ttyS207:20:020
191760699871,4cyclictest150-21pr/ttyS207:20:010
191760699845,15cyclictest131rcu_preempt10:05:030
1917617998220,24cyclictest1985319-21ssh11:32:142
1917617997926,21cyclictest1937833-21ssh09:42:122
1917617997725,14cyclictest1981920-21ssh11:24:362
1917617997721,19cyclictest1978090-21ssh11:15:232
1917617997721,19cyclictest1978090-21ssh11:15:232
1917617997623,22cyclictest2005695-21sh12:19:452
1917617997325,14cyclictest1954175-21latency_hist10:20:022
1917617997219,17cyclictest1935299-21ssh09:36:182
1917617997122,14cyclictest1987499-21ssh11:37:212
1917617997020,15cyclictest1996802-21ssh11:59:072
1917617997020,15cyclictest1996802-21ssh11:59:072
1917617997017,16cyclictest1972037-21rm11:01:202
1917617996920,16cyclictest1975045-21ssh11:08:342
1917617996917,15cyclictest1943177-21ssh09:54:562
1917617996828,14cyclictest1988992-21ssh11:40:362
1917617996823,12cyclictest0-21swapper/212:01:102
1917617996815,16cyclictest1924415-21ssh09:11:032
1917617996717,15cyclictest1999981-21ssh12:06:012
1917617996716,16cyclictest2007372-21ssh12:23:312
1917617996624,7cyclictest1934494-21ssh09:34:442
1917617996620,14cyclictest1923970-21sed09:10:022
1917617996515,15cyclictest1944593-21ssh09:58:012
1917617996513,20cyclictest1945833-21rm10:00:322
1917617996426,33cyclictest2003675-21cron12:15:002
1917617996417,16cyclictest1958803-21ssh10:30:352
1917617996417,14cyclictest1939962-21ssh09:47:152
1917617996415,13cyclictest1930231-21ssh09:24:522
1917617996414,15cyclictest1976632-21ssh11:12:092
191760699645,52cyclictest0-21swapper/010:50:000
1917617996324,9cyclictest0-21swapper/210:37:512
1917617996320,12cyclictest1928151-21munin-run09:20:012
1917617996317,12cyclictest0-21swapper/209:26:572
1917617996241,17cyclictest2008251-21sshd12:25:142
1917617996218,14cyclictest1948818-21sh10:07:372
1917606996215,42cyclictest12-21ksoftirqd/009:54:370
1917617996114,15cyclictest1923874-21diskmemload12:32:012
1917606996154,4cyclictest0-21swapper/010:35:140
1917606996152,4cyclictest1917612-21wc07:10:010
1917617996045,11cyclictest364-21snmpd10:56:012
1917617996044,13cyclictest1994415-21ssh11:53:302
1917606996026,31cyclictest12-21ksoftirqd/010:10:570
1917606996026,31cyclictest12-21ksoftirqd/010:10:570
1917617995920,12cyclictest1-21systemd10:55:012
1917617995913,15cyclictest1955183-21ssh10:22:152
1917617995913,15cyclictest1955183-21ssh10:22:152
191760699591,42cyclictest1944554-21ssh09:57:560
191762199588,5cyclictest131rcu_preempt09:15:003
1917617995841,14cyclictest1965977-21ssh10:47:302
1917617995840,14cyclictest1964740-21ssh10:44:432
1917617995839,16cyclictest1922817-21sshd08:49:142
1917617995838,17cyclictest1951755-21ssh10:14:372
1917617995838,17cyclictest1951755-21ssh10:14:372
1917617995825,28cyclictest3467602-21systemd-journal12:40:002
1917613995840,13cyclictest275-21dbus-daemon07:40:011
191760699588,40cyclictest12-21ksoftirqd/010:56:290
1917606995726,16cyclictest1941015-21ssh09:49:530
191760699571,43cyclictest0-21swapper/012:25:300
1917621995634,18cyclictest1919152-21systemctl07:40:003
1917617995643,9cyclictest1921531-21sshd08:24:142
1917617995643,10cyclictest1991558-21ssh11:46:372
1917617995619,11cyclictest1983693-21ssh11:28:352
1917606995620,31cyclictest1973570-21latency_hist11:05:010
191762199555,6cyclictest131rcu_preempt12:00:023
191762199555,6cyclictest131rcu_preempt12:00:023
1917621995511,12cyclictest37-21ksoftirqd/310:01:063
1917617995539,13cyclictest1957554-21ssh10:27:492
1917617995538,13cyclictest1922030-21sshd08:33:292
1917617995537,15cyclictest1922708-21cat08:45:032
1917617995537,15cyclictest1922708-21cat08:45:022
1917613995534,17cyclictest3467602-21systemd-journal10:40:001
1917606995534,4cyclictest16850irq/134-800000010:45:000
1917606995528,12cyclictest1988520-21ssh11:39:580
1917606995527,13cyclictest1999791-21ssh12:05:340
191760699551,48cyclictest3467602-21systemd-journal12:37:210
1917621995446,5cyclictest37-21ksoftirqd/309:35:463
1917621995430,8cyclictest37-21ksoftirqd/311:43:253
1917617995442,9cyclictest1918061-21sshd07:16:142
1917617995442,9cyclictest1918061-21sshd07:16:142
1917617995440,11cyclictest1923070-21sshd08:54:142
1917617995439,11cyclictest1918721-21cat07:30:032
1917617995439,11cyclictest1918721-21cat07:30:032
1917617995438,12cyclictest1919939-21munin-run07:55:022
1917606995429,18cyclictest275-21dbus-daemon09:40:010
1917606995427,12cyclictest1955577-21ssh10:23:230
1917606995427,12cyclictest1955577-21ssh10:23:230
191762199537,9cyclictest37-21ksoftirqd/307:55:013
1917621995313,9cyclictest131rcu_preempt10:36:123
1917617995341,9cyclictest1919635-21sshd07:45:142
1917617995337,12cyclictest1-21systemd08:00:012
191760699531,42cyclictest20105261ssh12:30:350
1917617995236,12cyclictest1918368-21sshd07:24:142
1917606995224,13cyclictest1999046-21ssh12:04:100
1917606995223,13cyclictest2006345-21ssh12:20:420
1917606995222,21cyclictest12-21ksoftirqd/009:45:000
1917621995134,12cyclictest1928096-21cron09:20:003
1917621995118,11cyclictest37-21ksoftirqd/311:13:473
1917621995110,5cyclictest131rcu_preempt10:53:023
1917617995141,6cyclictest1921262-21munin-run08:20:012
1917617995139,9cyclictest1923306-21sshd08:56:142
1917606995143,5cyclictest16950irq/224-800000007:45:010
1917606995121,21cyclictest12-21ksoftirqd/009:10:190
191762199509,11cyclictest37-21ksoftirqd/312:39:193
191762199503,42cyclictest37-21ksoftirqd/308:30:013
1917621995019,20cyclictest37-21ksoftirqd/309:48:343
1917621995014,8cyclictest131rcu_preempt11:16:563
1917621995014,8cyclictest131rcu_preempt11:16:563
1917617995037,10cyclictest364-21snmpd08:03:342
1917617995031,16cyclictest364-21snmpd07:11:462
1917613995033,12cyclictest275-21dbus-daemon12:30:001
1917606995042,4cyclictest16950irq/224-800000012:00:010
1917606995042,4cyclictest16950irq/224-800000012:00:000
1917606995029,17cyclictest1928119-21/usr/sbin/munin09:20:010
1917606995022,13cyclictest1994685-21ssh11:54:090
1917606995021,21cyclictest12-21ksoftirqd/007:50:010
1917606995016,7cyclictest0-21swapper/011:33:430
1917621994935,11cyclictest275-21dbus-daemon11:00:013
1917621994916,6cyclictest131rcu_preempt12:35:013
1917617994937,9cyclictest364-21snmpd07:39:412
1917617994929,15cyclictest1920947-21cron08:15:002
1917617994918,25cyclictest3467602-21systemd-journal08:40:012
1917613994932,13cyclictest1928096-21cron09:20:001
1917606994927,7cyclictest1977478-21ssh11:14:210
191760699492,5cyclictest1919151-21modprobe07:39:590
1917606994922,12cyclictest1949335-21ssh10:08:550
1917606994919,26cyclictest2002561-21ssh12:12:110
1917621994813,5cyclictest131rcu_preempt12:13:483
1917617994841,4cyclictest1919432-21munin-run07:45:012
1917617994831,10cyclictest1921787-21cron08:30:002
1917606994833,11cyclictest3467602-21systemd-journal09:02:140
1917606994832,13cyclictest1958829-21ssh10:30:390
1917606994828,4cyclictest12-21ksoftirqd/009:22:020
1917606994821,23cyclictest2005006-21rm12:17:560
1917606994814,29cyclictest0-21swapper/011:29:550
191762199479,5cyclictest131rcu_preempt09:31:193
1917621994734,9cyclictest37-21ksoftirqd/311:32:493
1917621994728,14cyclictest275-21dbus-daemon07:25:003
1917617994739,4cyclictest1918889-21munin-run07:35:002
1917606994739,4cyclictest16850irq/134-800000011:20:000
1917606994739,4cyclictest16850irq/134-800000011:20:000
1917606994722,10cyclictest1952706-21ssh10:16:280
1917606994716,27cyclictest12-21ksoftirqd/009:31:460
191762199468,6cyclictest131rcu_preempt12:27:323
1917621994636,6cyclictest2005822-21sendmail12:20:003
191762199462,7cyclictest37-21ksoftirqd/311:50:133
191762199461,5cyclictest131rcu_preempt10:05:243
1917617994633,9cyclictest364-21snmpd08:09:212
1917617994629,12cyclictest1917760-21sed07:10:022
1917613994632,10cyclictest1-21systemd11:15:001
1917613994631,11cyclictest3467602-21systemd-journal10:17:001
1917613994619,22cyclictest23-21ksoftirqd/110:00:001
191761399461,3cyclictest131rcu_preempt09:15:001
1917606994618,12cyclictest1932018-21ssh09:28:530
1917606994617,14cyclictest1989560-21ssh11:41:570
191760699461,38cyclictest0-21swapper/010:28:420
191762199458,21cyclictest1977953-21kworker/u8:011:25:023
1917621994519,10cyclictest37-21ksoftirqd/310:33:013
191762199451,5cyclictest131rcu_preempt10:10:283
191762199451,5cyclictest131rcu_preempt10:10:273
1917621994513,5cyclictest131rcu_preempt10:19:273
1917617994532,9cyclictest275-21dbus-daemon09:05:012
191761399453,3cyclictest131rcu_preempt09:54:521
1917606994529,12cyclictest1991244-21ssh11:45:420
1917606994526,15cyclictest1973993-21ssh11:05:540
191760699452,36cyclictest3467602-21systemd-journal08:00:000
191762199447,6cyclictest131rcu_preempt12:22:033
1917621994415,9cyclictest131rcu_preempt09:56:543
191762199441,10cyclictest0-21swapper/309:40:593
1917621994410,6cyclictest131rcu_preempt09:52:103
191761399444,3cyclictest131rcu_preempt10:56:411
1917613994431,10cyclictest1930608-21sshd09:25:141
1917613994430,10cyclictest1923350-21run-parts09:00:001
1917606994432,9cyclictest12-21ksoftirqd/008:55:020
191760699442,38cyclictest1967228-21ssh10:50:040
191760699441,35cyclictest0-21swapper/011:20:480
1917621994332,8cyclictest37-21ksoftirqd/311:35:273
1917621994330,9cyclictest37-21ksoftirqd/309:26:003
1917621994327,12cyclictest1-21systemd10:50:003
1917621994326,13cyclictest275-21dbus-daemon08:50:013
1917621994322,15cyclictest275-21dbus-daemon07:35:003
191762199431,9cyclictest0-21swapper/310:40:193
1917621994317,21cyclictest131rcu_preempt11:27:523
1917621994317,18cyclictest37-21ksoftirqd/311:07:583
1917621994312,6cyclictest131rcu_preempt12:05:203
1917613994325,9cyclictest23-21ksoftirqd/110:42:391
191761399432,3cyclictest131rcu_preempt11:48:221
1917613994323,15cyclictest275-21dbus-daemon07:45:001
191761399431,3cyclictest131rcu_preempt10:24:181
191761399431,3cyclictest131rcu_preempt10:24:181
1917606994336,4cyclictest16950irq/224-800000008:15:010
1917606994336,4cyclictest16950irq/224-800000007:15:000
1917606994331,7cyclictest16850irq/134-800000009:10:000
1917606994320,19cyclictest12-21ksoftirqd/007:34:140
1917606994319,20cyclictest12-21ksoftirqd/008:45:000
1917606994319,20cyclictest12-21ksoftirqd/008:45:000
191762199429,7cyclictest37-21ksoftirqd/309:21:073
1917621994229,8cyclictest3467602-21systemd-journal08:20:003
191762199422,7cyclictest131rcu_preempt08:15:003
191762199422,10cyclictest37-21ksoftirqd/307:15:003
191762199421,8cyclictest0-21swapper/311:02:383
191762199421,4cyclictest131rcu_preempt11:49:233
1917613994228,10cyclictest1952083-21cut10:15:021
1917613994228,10cyclictest1952083-21cut10:15:021
1917613994227,11cyclictest23-21ksoftirqd/112:06:261
1917613994226,13cyclictest275-21dbus-daemon10:05:011
1917613994225,13cyclictest275-21dbus-daemon11:20:011
1917613994225,13cyclictest275-21dbus-daemon11:20:011
1917613994225,12cyclictest1-21systemd12:20:011
1917613994215,23cyclictest1920719-21unin-run08:10:011
1917606994236,4cyclictest19350irq/34-mmc108:05:020
191762199415,8cyclictest37-21ksoftirqd/307:45:033
1917621994132,6cyclictest37-21ksoftirqd/308:35:013
191762199411,9cyclictest0-21swapper/310:20:343
191762199411,9cyclictest0-21swapper/310:20:343
191762199411,7cyclictest0-21swapper/312:03:273
1917613994131,6cyclictest1982279-21latency_hist11:25:021
1917613994131,5cyclictest309-21in:imuxsock11:30:021
1917613994126,11cyclictest275-21dbus-daemon12:40:011
191761399411,3cyclictest131rcu_preempt09:32:191
1917621994032,6cyclictest37-21ksoftirqd/310:26:283
1917621994028,8cyclictest37-21ksoftirqd/309:00:003
191761399406,8cyclictest2006955-1kworker/u9:012:25:011
1917613994028,8cyclictest23-21ksoftirqd/108:25:011
1917613994027,9cyclictest1922788-21cat08:45:041
1917613994025,11cyclictest23-21ksoftirqd/108:05:001
191761399401,3cyclictest131rcu_preempt09:46:131
1917606994033,4cyclictest16950irq/224-800000008:20:020
1917606994026,11cyclictest275-21dbus-daemon07:25:000
191760699402,5cyclictest275-21dbus-daemon08:35:010
191762199396,6cyclictest131rcu_preempt08:40:033
1917621993929,8cyclictest37-21ksoftirqd/308:00:023
1917621993910,5cyclictest131rcu_preempt08:05:013
1917613993928,8cyclictest1949046-21sshd10:08:141
1917613993928,6cyclictest309-21in:imuxsock09:05:001
191761399392,7cyclictest1966981-21cron10:50:001
1917613993927,10cyclictest1973469-21systemd-run11:05:001
1917613993926,9cyclictest309-21in:imuxsock12:05:001
1917613993922,13cyclictest1-21systemd09:45:001
1917613993920,13cyclictest1923858-21cron09:10:001
191761399391,3cyclictest131rcu_preempt09:38:501
1917606993925,9cyclictest3467602-21systemd-journal08:30:010
191760699391,5cyclictest131rcu_preempt08:40:010
1917621993822,12cyclictest1-21systemd08:45:013
1917621993822,12cyclictest1-21systemd08:45:013
1917621993810,5cyclictest131rcu_preempt09:07:053
191761399388,12cyclictest1995374-1kworker/u9:112:34:041
191761399384,3cyclictest131rcu_preempt10:51:331
1917613993831,4cyclictest1990735-21sh11:45:001
1917613993825,10cyclictest23-21ksoftirqd/110:28:231
1917613993825,10cyclictest1995068-21systemd-run11:55:001
1917613993824,11cyclictest23-21ksoftirqd/111:26:161
1917613993822,12cyclictest1918553-21cut07:25:011
1917613993821,12cyclictest275-21dbus-daemon12:15:011
1917606993823,11cyclictest275-21dbus-daemon07:30:010
1917606993823,11cyclictest275-21dbus-daemon07:30:010
191762199378,5cyclictest131rcu_preempt08:21:423
191761399373,3cyclictest131rcu_preempt11:59:361
191761399373,3cyclictest131rcu_preempt11:59:351
1917613993726,8cyclictest1988776-21sh11:40:021
1917613993725,8cyclictest3467602-21systemd-journal08:53:151
1917613993725,10cyclictest23-21ksoftirqd/110:32:131
1917613993722,11cyclictest275-21dbus-daemon08:50:011
191761399371,3cyclictest131rcu_preempt08:00:001
1917606993721,12cyclictest12-21ksoftirqd/008:24:010
191762199363,25cyclictest1920846-21grep08:10:023
1917621993628,5cyclictest1923745-21latency_hist09:05:023
1917621993626,8cyclictest37-21ksoftirqd/307:30:013
1917621993626,8cyclictest37-21ksoftirqd/307:30:013
1917621993617,4cyclictest37-21ksoftirqd/308:51:143
191761399366,25cyclictest1975736-21latency_hist11:10:021
1917613993628,5cyclictest1930085-21sh09:24:311
1917613993621,11cyclictest23-21ksoftirqd/108:15:441
1917606993629,4cyclictest16950irq/224-800000009:00:000
1917606993624,9cyclictest1920429-21sshd08:02:140
1917621993525,7cyclictest37-21ksoftirqd/307:20:013
1917621993525,7cyclictest37-21ksoftirqd/307:20:013
1917621993524,7cyclictest37-21ksoftirqd/307:50:023
1917613993519,12cyclictest3467602-21systemd-journal07:45:141
191760699355,27cyclictest0-21swapper/007:52:110
191760699351,30cyclictest0-21swapper/008:46:440
1917621993311,16cyclictest1908843-21kworker/u8:107:10:013
1917613993326,4cyclictest1-21systemd07:35:001
1917613993325,4cyclictest23-21ksoftirqd/107:20:011
1917613993325,4cyclictest23-21ksoftirqd/107:20:011
1917613993321,9cyclictest23-21ksoftirqd/108:45:011
1917613993320,10cyclictest0-21swapper/107:30:011
1917613993320,10cyclictest0-21swapper/107:30:011
1917613993313,5cyclictest131rcu_preempt07:55:031
1917613993225,4cyclictest1922492-21latency_hist08:40:031
1917613993221,9cyclictest23-21ksoftirqd/107:15:021
191761399314,3cyclictest131rcu_preempt08:30:001
1917613993117,11cyclictest1917676-21cut07:10:011
1917613993116,12cyclictest3467602-21systemd-journal08:34:001
1917613993019,9cyclictest23-21ksoftirqd/108:13:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional