You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-06 - 16:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Wed May 06, 2026 12:45:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2099504991015,14cyclictest131rcu_preempt10:55:020
2099512999655,38cyclictest30-21ksoftirqd/211:50:022
2099512997030,35cyclictest2130637-21sh10:01:412
2099504996962,2cyclictest0-21swapper/011:30:040
2099504996819,4cyclictest12-21ksoftirqd/011:39:070
209951299672,18cyclictest2150337-21ssh10:42:322
2099512996624,37cyclictest2112448-21sh09:24:092
2099512996617,44cyclictest2100147-21cat07:20:022
209951299652,17cyclictest2183832-21ssh11:52:082
209951299651,15cyclictest2158449-21ssh10:59:352
209951299651,15cyclictest2158449-21ssh10:59:352
209951299651,13cyclictest2114926-21sh09:29:212
2099512996452,6cyclictest2101530-21kworker/2:009:07:452
2099512996452,6cyclictest2101530-21kworker/2:009:07:452
209951299641,18cyclictest2165876-21munin-run11:15:022
209951299632,5cyclictest2139339-21munin-run10:20:012
209951299631,8cyclictest2124060-21ssh09:48:222
209950499638,35cyclictest2173178-21ssh11:30:020
2099504996331,11cyclictest2112896-21wc09:25:040
209951299621,9cyclictest2106047-21ssh09:10:372
209951299621,8cyclictest2153025-21ssh10:48:182
209951299621,13cyclictest2102050-21munin-run08:00:012
209951299621,10cyclictest2140648-21ssh10:22:302
209950799622,18cyclictest2205007-21ssh12:35:461
209950799622,15cyclictest2115798-21ssh09:30:501
2099504996219,22cyclictest16850irq/134-800000012:10:030
209951299612,5cyclictest2202064-21/usr/sbin/munin12:30:022
209951299612,4cyclictest2099686-21munin-run07:15:022
209951299611,9cyclictest2120518-21ssh09:40:452
209951299611,8cyclictest2171907-21ssh11:27:222
209951299611,8cyclictest2135337-21ssh10:11:282
209951299611,8cyclictest2126121-21ssh09:52:272
209951299611,7cyclictest2199281-21ssh12:24:122
209951299611,7cyclictest2192682-21ssh12:10:092
209951299611,7cyclictest2192682-21ssh12:10:082
209951299611,7cyclictest2191348-21ssh12:07:332
209951299611,7cyclictest2129566-21ssh09:59:472
209951299611,6cyclictest2146366-21ssh10:34:352
209951299611,14cyclictest2175101-21ssh11:34:152
209950799617,45cyclictest2122408-21modprobe09:45:001
2099507996121,35cyclictest2105657-21diskmemload11:06:321
2099507996121,35cyclictest2105657-21diskmemload11:06:311
209950799612,12cyclictest2117963-21sh09:35:241
2099504996116,40cyclictest12-21ksoftirqd/011:46:190
209951299602,6cyclictest2177556-21ssh11:39:242
209951299602,5cyclictest2149022-21munin-run10:40:012
209951299602,14cyclictest2186046-21ssh11:56:332
209951299602,12cyclictest2105415-21munin-run09:05:012
209951299601,9cyclictest2195764-21ssh12:16:422
209951299601,8cyclictest2180046-21ssh11:44:222
209951299601,8cyclictest2154638-21ssh10:51:182
209951299601,7cyclictest2167144-21ssh11:17:262
209951299601,7cyclictest2143432-21ssh10:28:262
209951299601,7cyclictest2132849-21ssh10:06:152
209951299601,7cyclictest2118765-21ssh09:37:152
209951299601,7cyclictest2116837-21ssh09:33:202
209950799602,16cyclictest2183792-21ssh11:52:031
209951299592,16cyclictest2206527-21sshd12:39:162
209951299591,8cyclictest2108469-21ssh09:15:322
209951299591,7cyclictest2170402-21ssh11:24:172
209951299591,7cyclictest2159365-21ssh11:01:072
209951299591,5cyclictest2101517-21munin-run07:50:012
209951299591,16cyclictest2161636-21ssh11:05:532
209951299591,16cyclictest2161636-21ssh11:05:532
209950799595,11cyclictest2104610-21modprobe08:50:011
209950799594,9cyclictest2170673-21munin-run11:25:011
2099507995920,34cyclictest2202354-21sh12:30:101
2099507995917,38cyclictest2102359-21cat08:05:021
2099507995915,39cyclictest2099610-21cut07:10:021
209950799591,15cyclictest2165289-21ssh11:13:431
209950799591,14cyclictest2159424-21ssh11:01:141
2099504995914,40cyclictest16850irq/134-800000011:55:010
209951299581,15cyclictest2188570-21ssh12:01:442
209950799583,17cyclictest2108035-21cat09:15:011
2099507995831,23cyclictest2105657-21diskmemload10:03:311
209950799582,17cyclictest2187610-21sendmail12:00:011
209950799582,15cyclictest2136241-21sh10:13:301
209950799581,19cyclictest2194292-21ssh12:13:481
209950799581,19cyclictest2194292-21ssh12:13:471
2099504995830,12cyclictest2134515-21ssh10:10:010
2099504995816,10cyclictest275-21dbus-daemon10:15:010
2099504995813,5cyclictest12-21ksoftirqd/012:37:100
2099512995718,35cyclictest30-21ksoftirqd/208:06:442
209950799578,43cyclictest2197255-21modprobe12:20:001
209950799572,7cyclictest2105389-21cron09:05:001
209950799572,17cyclictest2133728-21ssh10:08:161
209950799572,14cyclictest2200491-21ssh12:26:251
209950799571,9cyclictest2153363-21ssh10:48:591
2099507995715,37cyclictest23-21ksoftirqd/110:16:481
2099507995715,37cyclictest23-21ksoftirqd/109:50:531
2099507995714,38cyclictest23-21ksoftirqd/107:56:121
209950799571,17cyclictest2128159-21ssh09:56:391
209950799571,16cyclictest2178700-21sh11:41:261
209950799571,15cyclictest2175924-21ssh11:35:281
209950799571,15cyclictest2114343-21ssh09:28:071
209950799571,14cyclictest2180962-21rm11:45:591
209950799571,14cyclictest2109538-21ssh09:18:071
209950799571,10cyclictest2190531-21ssh12:05:411
209950799571,10cyclictest2145750-21ssh10:33:091
2099504995746,3cyclictest0-21swapper/011:10:020
2099504995746,3cyclictest0-21swapper/011:10:020
2099504995728,14cyclictest2169690-21ssh11:22:420
209951299562,5cyclictest2103586-21munin-run08:30:002
209951299561,15cyclictest2204060-21ssh12:34:032
209950799562,6cyclictest364-21snmpd08:59:011
209950799562,6cyclictest364-21snmpd08:59:011
209950799562,16cyclictest2142328-21ssh10:25:501
209950799562,16cyclictest2100720-21munin-run07:35:001
209950799562,15cyclictest2167324-21ssh11:17:581
209950799562,15cyclictest2148762-21ssh10:39:361
209950799561,9cyclictest2175274-21sh11:34:351
2099507995617,35cyclictest2105657-21diskmemload12:20:521
209950799561,19cyclictest2112615-21ssh09:24:381
209950799561,15cyclictest2157707-21ssh10:57:531
209950799561,15cyclictest2157707-21ssh10:57:531
209950799561,10cyclictest2155295-21ssh10:52:501
209951299553,5cyclictest364-21snmpd08:43:092
209951299553,4cyclictest2102299-21munin-run08:05:012
209951299552,11cyclictest2099415-21latency_hist07:10:022
209950799552,16cyclictest2124245-21ssh09:48:451
2099504995543,7cyclictest2146573-21cut10:35:000
2099504995538,13cyclictest3467602-21systemd-journal08:50:010
2099504995518,33cyclictest16850irq/134-800000009:28:110
209951499541,48cyclictest2204986-21sshd12:35:443
209951299548,40cyclictest0-21swapper/208:40:002
209951299544,13cyclictest1-21systemd09:00:012
209951299544,13cyclictest1-21systemd09:00:012
209951299543,18cyclictest3467602-21systemd-journal08:35:002
209951299543,18cyclictest3467602-21systemd-journal08:35:002
2099507995437,12cyclictest3467602-21systemd-journal07:50:001
209950799542,18cyclictest2105663-21grep09:10:001
209950799542,18cyclictest2105663-21grep09:10:001
209950799542,14cyclictest2150418-21sh10:42:421
209950799541,17cyclictest2141456-21rm10:24:191
2099504995446,4cyclictest2168265-21latency_hist11:20:020
2099504995419,6cyclictest16850irq/134-800000012:29:180
2099504995419,31cyclictest16850irq/134-800000010:40:280
2099504995419,31cyclictest16850irq/134-800000010:28:010
2099504995419,31cyclictest16850irq/134-800000010:21:150
2099504995418,7cyclictest16850irq/134-800000012:32:520
2099504995418,6cyclictest16850irq/134-800000012:19:240
2099504995418,6cyclictest16850irq/134-800000009:19:390
2099504995418,32cyclictest16850irq/134-800000010:59:230
2099504995418,32cyclictest16850irq/134-800000010:59:230
2099504995418,32cyclictest16850irq/134-800000009:45:480
2099504995418,32cyclictest16850irq/134-800000009:34:350
2099504995418,32cyclictest16850irq/134-800000009:14:300
2099504995418,31cyclictest16850irq/134-800000012:21:370
2099504995418,31cyclictest16850irq/134-800000010:47:450
2099504995418,31cyclictest16850irq/134-800000009:52:090
209950499541,37cyclictest2160457-21ssh11:03:360
209951299532,5cyclictest2101250-21munin-run07:45:002
209951299531,5cyclictest2100215-21munin-run07:25:012
209951299531,5cyclictest2100215-21munin-run07:25:012
2099512995311,37cyclictest2102776-21cron08:15:002
2099507995326,23cyclictest2172403-21sh11:28:331
209950799532,13cyclictest2189401-21rm12:03:351
2099507995321,27cyclictest275-21dbus-daemon08:15:011
209950799531,11cyclictest2101236-21cron07:45:001
2099504995319,30cyclictest16850irq/134-800000012:01:040
2099504995318,31cyclictest16850irq/134-800000010:35:500
2099504995318,31cyclictest16850irq/134-800000010:00:490
2099504995318,31cyclictest16850irq/134-800000009:56:350
2099504995317,7cyclictest16850irq/134-800000009:37:480
2099504995317,32cyclictest16850irq/134-800000009:40:520
2099504995316,32cyclictest16850irq/134-800000011:55:520
2099514995240,7cyclictest37-21ksoftirqd/310:52:353
2099514995233,13cyclictest3467602-21systemd-journal11:35:003
209951299521,16cyclictest2100940-21sshd07:36:452
209951299521,11cyclictest364-21snmpd08:47:092
209950799523,16cyclictest2100567-21grep07:30:011
2099504995218,30cyclictest16850irq/134-800000012:10:560
2099504995218,30cyclictest16850irq/134-800000012:10:560
2099504995218,30cyclictest16850irq/134-800000010:16:140
2099504995217,6cyclictest16850irq/134-800000011:11:390
209951299512,17cyclictest2103030-21sshd08:18:452
209951299512,17cyclictest2103030-21sshd08:18:452
209951299511,12cyclictest364-21snmpd07:32:372
209950799513,9cyclictest2103040-21cron08:20:001
209950799513,9cyclictest2103040-21cron08:20:001
209950799512,9cyclictest3467602-21systemd-journal08:35:011
209950799512,9cyclictest3467602-21systemd-journal08:35:011
209950799512,9cyclictest2100202-21cron07:25:001
209950799512,9cyclictest2100202-21cron07:24:591
2099507995114,33cyclictest2102525-21journal-offline08:09:551
209950799511,12cyclictest3467602-21systemd-journal08:42:051
209950499513,6cyclictest12-21ksoftirqd/008:10:010
2099504995117,5cyclictest16850irq/134-800000011:44:190
209951299502,5cyclictest364-21snmpd07:29:492
209951299502,3cyclictest0-21swapper/208:24:182
209950799502,16cyclictest2103319-21cron08:25:001
2099514994934,12cyclictest1-21systemd08:55:013
2099514994934,11cyclictest1-21systemd09:10:013
2099514994934,11cyclictest1-21systemd09:10:013
209950799494,5cyclictest2099660-21cron07:15:001
209950799492,16cyclictest2099952-21tr07:20:001
209950799492,13cyclictest2104318-21latency_hist08:40:021
2099514994810,5cyclictest131rcu_preempt10:35:013
209951299482,7cyclictest364-21snmpd07:54:142
209950799481,10cyclictest3467602-21systemd-journal08:50:271
2099504994831,13cyclictest275-21dbus-daemon09:10:010
2099504994831,13cyclictest275-21dbus-daemon09:10:010
209950499482,15cyclictest2104128-21munin-run08:40:000
2099514994730,13cyclictest1-21systemd09:50:013
2099514994730,11cyclictest37-21ksoftirqd/309:55:023
2099514994728,14cyclictest1-21systemd10:20:023
209950799472,7cyclictest1-21systemd07:55:011
2099514994632,10cyclictest3467602-21systemd-journal09:00:003
2099514994632,10cyclictest3467602-21systemd-journal09:00:003
2099514994629,10cyclictest2110378-21cron09:20:003
2099512994612,30cyclictest30-21ksoftirqd/208:55:022
209950799463,8cyclictest275-21dbus-daemon08:30:011
2099514994532,9cyclictest1-21systemd12:00:013
2099514994526,12cyclictest0-21swapper/311:00:033
2099514994526,12cyclictest0-21swapper/311:00:023
209951499451,3cyclictest2192469-21ssh12:10:023
209950799452,12cyclictest275-21dbus-daemon07:40:011
2099504994536,6cyclictest12-21ksoftirqd/008:28:520
2099504994529,12cyclictest2105131-21systemctl09:00:000
2099504994529,12cyclictest2105131-21systemctl09:00:000
2099504994521,6cyclictest12-21ksoftirqd/007:25:020
2099504994521,6cyclictest12-21ksoftirqd/007:25:020
2099514994432,8cyclictest2103065-21modprobe08:20:003
2099514994432,8cyclictest2103065-21modprobe08:20:003
2099514994425,12cyclictest1-21systemd12:35:023
209951499441,7cyclictest37-21ksoftirqd/309:21:543
209950499443,7cyclictest2101505-21munin-run07:50:010
2099504994430,10cyclictest3467602-21systemd-journal09:05:000
209951499439,5cyclictest131rcu_preempt09:43:253
2099514994332,7cyclictest2100201-21cron07:25:003
2099514994332,7cyclictest2100201-21cron07:25:003
2099514994331,8cyclictest3467602-21systemd-journal09:25:453
2099514994330,10cyclictest275-21dbus-daemon10:00:003
2099514994329,10cyclictest309-21in:imuxsock11:15:003
2099514994328,12cyclictest2173247-21cut11:30:023
2099514994320,20cyclictest37-21ksoftirqd/308:40:013
2099514994318,20cyclictest361rcuc/311:46:483
2099504994320,6cyclictest151rcuc/007:45:010
209951499426,5cyclictest131rcu_preempt12:25:003
209951499423,5cyclictest131rcu_preempt09:38:383
2099514994233,5cyclictest2099938-21sendmail07:20:003
2099514994229,7cyclictest2101491-21cron07:50:003
2099514994226,12cyclictest37-21ksoftirqd/310:36:083
2099514994219,5cyclictest37-21ksoftirqd/308:10:013
209951499421,6cyclictest131rcu_preempt10:05:013
209951499421,5cyclictest131rcu_preempt09:13:343
209951499421,5cyclictest131rcu_preempt07:15:023
2099514994215,7cyclictest131rcu_preempt11:36:193
209950499425,5cyclictest131rcu_preempt08:35:020
209950499425,5cyclictest131rcu_preempt08:35:020
209950499422,35cyclictest2104366-21munin-run08:45:010
209950499421,11cyclictest2102020-21systemctl08:00:000
2099514994128,10cyclictest37-21ksoftirqd/311:51:223
2099514994125,12cyclictest3467602-21systemd-journal11:45:003
2099514994124,13cyclictest3467602-21systemd-journal10:30:013
2099514994116,7cyclictest361rcuc/312:19:453
2099514994115,6cyclictest131rcu_preempt11:25:023
2099504994127,11cyclictest3467602-21systemd-journal07:18:310
2099504994126,11cyclictest3467718-21lldpd07:53:270
2099504994119,6cyclictest131rcu_preempt07:37:450
2099514994025,11cyclictest37-21ksoftirqd/310:12:443
209951499401,5cyclictest131rcu_preempt08:05:013
209951499401,4cyclictest131rcu_preempt12:15:003
209951499401,4cyclictest131rcu_preempt12:15:003
2099504994030,5cyclictest2102284-21munin-plugin-st08:04:590
2099504994029,7cyclictest12-21ksoftirqd/007:30:010
209950499402,7cyclictest2104856-21cron08:55:000
209951499395,6cyclictest131rcu_preempt07:35:023
2099514993929,5cyclictest2153795-21cron10:49:593
2099514993929,5cyclictest2102173-21latency_hist08:00:023
2099514993927,9cyclictest2149570-21sshd10:40:453
209951499392,6cyclictest131rcu_preempt09:30:453
209951499391,6cyclictest131rcu_preempt12:30:003
209951499391,6cyclictest131rcu_preempt12:05:023
2099514993916,6cyclictest131rcu_preempt11:17:533
2099504993923,10cyclictest0-21swapper/008:20:000
2099504993923,10cyclictest0-21swapper/008:19:590
209951499385,6cyclictest37-21ksoftirqd/311:04:453
209951499384,7cyclictest131rcu_preempt08:25:013
2099514993828,7cyclictest37-21ksoftirqd/310:21:013
2099514993827,8cyclictest275-21dbus-daemon08:35:013
2099514993827,8cyclictest275-21dbus-daemon08:35:013
209951499381,5cyclictest131rcu_preempt07:45:003
209951499381,3cyclictest131rcu_preempt11:06:213
209951499381,3cyclictest131rcu_preempt11:06:213
2099504993817,5cyclictest16850irq/134-800000008:13:530
2099514993725,10cyclictest37-21ksoftirqd/310:06:093
209951499371,5cyclictest131rcu_preempt07:10:023
209951499362,5cyclictest131rcu_preempt07:30:023
2099514993624,8cyclictest3467602-21systemd-journal07:54:313
209950499361,5cyclictest131rcu_preempt08:25:010
2099514993523,9cyclictest2100994-21basename07:40:013
2099514993517,14cyclictest275-21dbus-daemon09:05:003
209951499351,5cyclictest131rcu_preempt08:45:003
2099504993524,8cyclictest12-21ksoftirqd/007:10:020
2099504993524,6cyclictest309-21in:imuxsock07:15:000
2099504993514,4cyclictest12-21ksoftirqd/007:35:010
209951499347,5cyclictest131rcu_preempt08:25:213
209951499341,5cyclictest131rcu_preempt08:11:503
2099514993222,7cyclictest2104682-21sort08:50:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional