You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-18 - 18:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue Aug 18, 2026 12:45:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
203597099108104,2cyclictest0-21swapper/010:50:020
203597099942,61cyclictest2076460-21ssh10:21:460
2035970998831,52cyclictest2080915-21ssh10:31:080
203597099872,56cyclictest2118014-21ssh11:49:360
203597099832,41cyclictest2055723-21ssh09:38:190
203597099831,52cyclictest2108786-21modprobe11:29:590
203597099821,44cyclictest0-21swapper/011:20:250
203597099821,44cyclictest0-21swapper/011:20:250
2035970998029,47cyclictest16950irq/224-800000010:39:130
2035970998026,50cyclictest2109655-21ssh11:31:300
2035970998026,50cyclictest2109655-21ssh11:31:300
203597099802,42cyclictest2127729-21munin-run12:10:000
2035970998014,32cyclictest2141790-21rm12:39:380
203597099791,41cyclictest2136479-21ssh12:28:200
203597099784,36cyclictest2041756-21cut09:00:020
203597099784,36cyclictest2041756-21cut09:00:020
203597099772,70cyclictest2087326-21cron10:45:010
2035970997720,53cyclictest16850irq/134-800000009:49:170
203597099771,47cyclictest2042152-21diskmemload11:06:470
203597099762,43cyclictest3467602-21systemd-journal10:00:010
2035970997624,17cyclictest12-21ksoftirqd/009:55:020
203597099749,60cyclictest16850irq/134-800000012:15:440
2035970997327,40cyclictest309-21in:imuxsock09:20:010
2035970997327,40cyclictest309-21in:imuxsock09:20:000
203597099732,31cyclictest2054072-21munin-run09:35:010
203597099731,42cyclictest0-21swapper/012:12:350
203597099731,42cyclictest0-21swapper/012:12:340
2035970997228,26cyclictest2066688-21ssh10:01:100
2035970997228,26cyclictest2066688-21ssh10:01:090
203597099722,66cyclictest2057061-21ssh09:40:400
203597099721,67cyclictest48-21kcompactd011:11:070
2035970997215,39cyclictest16850irq/134-800000011:19:430
2035970997212,55cyclictest16850irq/134-800000012:20:210
2035970997211,36cyclictest2042152-21diskmemload11:52:250
2035975997130,9cyclictest3467602-21systemd-journal11:35:021
2035975997130,9cyclictest3467602-21systemd-journal11:35:011
2035970997125,30cyclictest2124867-21ssh12:03:540
2035970997125,30cyclictest2124867-21ssh12:03:540
203597099712,30cyclictest2073676-21ssh10:15:490
2035970997119,40cyclictest16850irq/134-800000010:26:410
203597099711,43cyclictest0-21swapper/007:50:000
203597099711,33cyclictest2088164-21ssh10:46:280
203597599704,62cyclictest2078084-21ssh10:25:021
2035975997027,7cyclictest2046960-21sendmail-msp09:20:011
2035975997027,7cyclictest2046960-21sendmail-msp09:20:001
203597099701,38cyclictest2069425-21ssh10:07:040
2035970997012,53cyclictest16850irq/134-800000011:38:290
203597099687,34cyclictest2042152-21diskmemload12:32:070
203597099687,34cyclictest2042152-21diskmemload12:32:060
2035970996826,39cyclictest0-21swapper/011:43:380
2035970996826,39cyclictest0-21swapper/011:43:370
2035970996822,41cyclictest16850irq/134-800000010:57:580
2035975996728,8cyclictest2059007-21latency_hist09:45:021
203597099673,33cyclictest2122657-21ssh11:59:170
203597099671,61cyclictest2039251-21sshd08:10:400
203597099671,35cyclictest2043714-21ssh09:13:080
203597099671,35cyclictest2043714-21ssh09:13:080
203597099663,30cyclictest2039642-21latency_hist08:20:010
2035970996621,29cyclictest2049702-21ssh09:25:240
2035975996519,13cyclictest0-21swapper/112:20:001
203597099651,58cyclictest48-21kcompactd007:43:030
203597099651,58cyclictest48-21kcompactd007:43:030
203597099651,41cyclictest2047520-21ssh09:20:510
2035975996430,9cyclictest120-21kswapd011:25:011
2035975996430,9cyclictest120-21kswapd011:25:011
2035970996415,44cyclictest16850irq/134-800000010:12:280
203597099641,31cyclictest2096825-21sh11:05:010
203597099641,31cyclictest2096825-21sh11:05:010
2035975996033,9cyclictest3467602-21systemd-journal08:20:011
2035975996026,12cyclictest23-21ksoftirqd/109:30:461
2035970996040,17cyclictest16950irq/224-800000008:35:010
2035970996040,17cyclictest16950irq/224-800000008:35:010
203597099601,53cyclictest48-21kcompactd009:05:440
203597099601,53cyclictest48-21kcompactd009:05:430
203597099592,53cyclictest2041330-21sshd08:50:390
203597099592,32cyclictest2039045-21munin-run08:10:010
203597099583,49cyclictest2036692-21sshd07:24:400
203597099582,52cyclictest2038751-21sshd08:01:390
203597099581,51cyclictest48-21kcompactd008:47:250
203597099581,51cyclictest48-21kcompactd007:58:360
2035979995748,5cyclictest2039047-21kworker/2:008:25:012
203597599571,38cyclictest2076105-1kworker/u9:110:25:021
203597099571,50cyclictest48-21kcompactd007:50:490
2035979995647,5cyclictest2087371-21kworker/2:311:22:092
2035979995647,5cyclictest2087371-21kworker/2:311:22:082
2035975995624,10cyclictest23-21ksoftirqd/110:41:011
203597099561,49cyclictest48-21kcompactd008:42:250
203597099552,48cyclictest364-21snmpd07:33:110
203597099551,50cyclictest364-21snmpd09:00:040
203597099551,50cyclictest2037437-21sshd07:37:400
203597099551,50cyclictest150-21pr/ttyS207:19:270
203597099551,48cyclictest48-21kcompactd007:28:380
2035979995445,5cyclictest2087371-21kworker/2:311:48:042
2035979995445,5cyclictest2087371-21kworker/2:311:05:042
2035979995436,13cyclictest275-21dbus-daemon10:35:002
203597099544,45cyclictest2035973-21grep07:10:010
203597099542,47cyclictest364-21snmpd07:14:110
203597099542,47cyclictest364-21snmpd07:14:100
2035979995345,5cyclictest2042113-21kworker/2:109:39:012
2035979995344,6cyclictest2039047-21kworker/2:009:30:512
203597099531,48cyclictest2040548-21sshd08:35:400
2035979995243,5cyclictest2042113-21kworker/2:110:04:052
2035979995243,5cyclictest2042113-21kworker/2:110:04:042
2035979995236,12cyclictest3467602-21systemd-journal12:01:392
2035979995236,12cyclictest3467602-21systemd-journal12:01:382
2035975995234,13cyclictest0-21swapper/112:05:001
2035975995234,13cyclictest0-21swapper/112:05:001
2035984995111,9cyclictest0-21swapper/310:08:203
2035979995143,5cyclictest2042113-21kworker/2:110:18:462
203597099512,45cyclictest364-21snmpd08:29:250
203597099511,47cyclictest150-21pr/ttyS208:22:110
2035984995017,7cyclictest0-21swapper/311:14:483
2035984995013,10cyclictest37-21ksoftirqd/312:00:463
2035984995013,10cyclictest37-21ksoftirqd/312:00:453
2035979995043,4cyclictest2087371-21kworker/2:311:01:262
2035979995043,4cyclictest2087371-21kworker/2:311:01:252
2035979995042,5cyclictest2039047-21kworker/2:010:49:562
2035979995042,5cyclictest2039047-21kworker/2:009:53:332
2035979995042,5cyclictest2039047-21kworker/2:009:45:582
2035979995040,6cyclictest2039047-21kworker/2:010:45:012
2035979995023,7cyclictest2065974-21sh10:00:002
2035975995033,5cyclictest2096922-21latency_hist11:05:031
2035975995033,5cyclictest2096922-21latency_hist11:05:031
2035984994935,10cyclictest1-21systemd07:15:023
2035984994935,10cyclictest1-21systemd07:15:013
2035979994941,5cyclictest2087371-21kworker/2:311:25:592
2035979994941,5cyclictest2042113-21kworker/2:110:23:112
2035979994940,6cyclictest2019896-21kworker/2:007:40:012
2035979994940,5cyclictest2134878-21kworker/2:112:38:342
2035979994940,5cyclictest2039047-21kworker/2:009:21:552
2035975994933,12cyclictest3467602-21systemd-journal09:30:011
2035975994932,13cyclictest3467602-21systemd-journal10:09:191
203598499483,4cyclictest131rcu_preempt12:08:343
2035984994834,10cyclictest1-21systemd10:45:003
2035984994832,12cyclictest1-21systemd09:15:003
2035984994832,12cyclictest1-21systemd09:15:003
2035984994810,6cyclictest131rcu_preempt12:35:023
2035984994810,6cyclictest131rcu_preempt12:35:013
2035979994840,5cyclictest2115879-21kworker/2:211:55:102
2035979994840,5cyclictest2039047-21kworker/2:009:27:412
2035979994840,5cyclictest2039047-21kworker/2:009:19:572
2035979994840,5cyclictest2039047-21kworker/2:009:19:572
2035979994839,5cyclictest2039047-21kworker/2:009:14:142
2035979994839,5cyclictest2039047-21kworker/2:009:14:132
2035979994739,5cyclictest2115879-21kworker/2:212:07:032
2035979994739,5cyclictest2087371-21kworker/2:311:11:312
2035979994739,5cyclictest2039876-21kworker/2:309:03:272
2035979994739,5cyclictest2039047-21kworker/2:010:07:182
2035979994738,6cyclictest2087371-21kworker/2:311:30:382
2035979994738,6cyclictest2087371-21kworker/2:311:30:372
2035979994726,15cyclictest3467602-21systemd-journal07:25:002
203598499463,4cyclictest131rcu_preempt11:45:283
2035984994630,13cyclictest37-21ksoftirqd/310:45:293
2035984994630,12cyclictest37-21ksoftirqd/312:20:013
203598499462,4cyclictest131rcu_preempt10:38:403
203598499462,12cyclictest2055040-21ssh09:36:433
2035984994613,6cyclictest2042152-21diskmemload09:45:433
203598499461,12cyclictest2128164-21sshd12:10:403
203598499461,12cyclictest2128164-21sshd12:10:393
2035979994639,4cyclictest2087371-21kworker/2:312:26:422
2035979994639,4cyclictest2087371-21kworker/2:311:40:022
2035979994639,4cyclictest2039047-21kworker/2:009:44:132
2035979994638,5cyclictest2039047-21kworker/2:008:55:172
2035979994638,5cyclictest2039047-21kworker/2:008:55:162
2035979994634,9cyclictest3467602-21systemd-journal08:33:402
2035979994634,9cyclictest3467602-21systemd-journal08:33:392
203597599469,3cyclictest131rcu_preempt11:55:001
2035975994632,10cyclictest0-21swapper/111:30:021
2035975994624,16cyclictest275-21dbus-daemon12:35:001
2035975994624,16cyclictest275-21dbus-daemon12:35:001
2035984994535,8cyclictest37-21ksoftirqd/312:40:023
203598499453,3cyclictest131rcu_preempt11:58:323
2035984994528,13cyclictest275-21dbus-daemon11:05:013
2035984994528,13cyclictest275-21dbus-daemon11:05:003
203598499452,13cyclictest2042152-21diskmemload11:21:363
203598499452,13cyclictest2042152-21diskmemload11:21:353
203598499451,3cyclictest131rcu_preempt12:21:363
2035979994537,5cyclictest2087371-21kworker/2:312:17:532
2035979994537,5cyclictest2026300-21kworker/2:307:17:322
2035979994536,6cyclictest2042113-21kworker/2:110:27:022
2035979994536,5cyclictest2019896-21kworker/2:007:55:002
2035979994532,10cyclictest275-21dbus-daemon10:40:012
2035975994532,10cyclictest2092279-21sed10:55:031
2035975994516,25cyclictest23-21ksoftirqd/112:00:001
203598499447,5cyclictest131rcu_preempt11:45:033
203598499447,5cyclictest131rcu_preempt11:45:023
2035984994429,11cyclictest37-21ksoftirqd/310:20:003
203598499442,7cyclictest0-21swapper/309:17:443
203598499442,7cyclictest0-21swapper/309:17:433
2035984994426,13cyclictest275-21dbus-daemon08:05:013
203598499442,38cyclictest2037330-21sed07:35:023
203597999445,5cyclictest2087371-21kworker/2:311:51:132
2035979994436,5cyclictest2087371-21kworker/2:312:30:092
2035979994436,5cyclictest2087371-21kworker/2:312:30:092
2035979994436,5cyclictest2039047-21kworker/2:008:53:252
2035975994431,9cyclictest1-21systemd09:05:001
203598499434,3cyclictest131rcu_preempt09:40:363
2035984994329,11cyclictest1-21systemd10:25:003
2035984994328,11cyclictest275-21dbus-daemon11:30:013
2035984994315,24cyclictest275-21dbus-daemon11:55:013
203598499431,3cyclictest131rcu_preempt09:52:393
2035984994312,5cyclictest131rcu_preempt12:28:263
2035979994335,5cyclictest2087371-21kworker/2:312:14:442
2035979994335,5cyclictest2087371-21kworker/2:312:14:432
2035979994334,6cyclictest2039047-21kworker/2:009:06:562
2035979994334,6cyclictest2039047-21kworker/2:009:06:562
2035979994333,8cyclictest30-21ksoftirqd/208:10:002
2035979994333,6cyclictest2019896-21kworker/2:007:48:582
203597999432,36cyclictest30-21ksoftirqd/212:23:332
203597599431,6cyclictest0-21swapper/112:37:351
2035984994232,6cyclictest2080241-21cut10:30:003
2035984994225,12cyclictest3467602-21systemd-journal11:10:013
2035984994225,12cyclictest309-21in:imuxsock10:00:023
2035979994230,9cyclictest30-21ksoftirqd/211:00:012
203597999421,5cyclictest0-21swapper/211:16:352
2035975994227,11cyclictest3467602-21systemd-journal11:40:001
2035984994127,11cyclictest37-21ksoftirqd/310:51:203
203598499411,6cyclictest2110502-21ssh11:33:363
203598499411,6cyclictest2110502-21ssh11:33:353
203598499411,4cyclictest131rcu_preempt10:57:233
203598499411,3cyclictest131rcu_preempt11:17:563
203598499411,3cyclictest131rcu_preempt09:31:263
2035979994126,11cyclictest3467602-21systemd-journal08:15:022
2035979994126,11cyclictest1-21systemd11:45:012
2035979994126,11cyclictest1-21systemd11:45:002
2035979994126,10cyclictest1-21systemd08:50:022
2035975994127,11cyclictest23-21ksoftirqd/112:25:591
2035975994127,11cyclictest23-21ksoftirqd/110:55:311
2035975994127,11cyclictest23-21ksoftirqd/110:15:211
2035975994127,10cyclictest3467602-21systemd-journal11:20:011
2035975994120,15cyclictest275-21dbus-daemon08:30:001
203598499407,5cyclictest131rcu_preempt10:12:123
203598499406,5cyclictest131rcu_preempt11:35:073
2035984994030,8cyclictest37-21ksoftirqd/308:55:003
2035984994025,11cyclictest275-21dbus-daemon08:50:013
203598499401,33cyclictest309-21in:imuxsock09:04:093
2035979994032,5cyclictest2039047-21kworker/2:010:51:212
2035979994032,5cyclictest2019896-21kworker/2:007:43:112
2035979994032,5cyclictest2019896-21kworker/2:007:43:102
2035979994026,10cyclictest275-21dbus-daemon08:05:012
2035975994026,11cyclictest23-21ksoftirqd/111:10:351
203597599402,3cyclictest131rcu_preempt10:48:381
203597599402,12cyclictest2059753-21ssh09:46:331
2035984993926,9cyclictest309-21in:imuxsock08:25:003
2035984993926,7cyclictest2041646-21sendmail-msp09:00:003
2035984993926,7cyclictest2041646-21sendmail-msp09:00:003
2035984993925,11cyclictest37-21ksoftirqd/310:33:103
2035984993925,11cyclictest37-21ksoftirqd/309:20:103
203597999397,5cyclictest131rcu_preempt07:30:002
2035979993928,9cyclictest30-21ksoftirqd/208:15:012
203597999391,4cyclictest131rcu_preempt08:30:012
203597599399,8cyclictest23-21ksoftirqd/111:47:441
2035975993926,9cyclictest23-21ksoftirqd/108:05:011
2035975993926,10cyclictest3467602-21systemd-journal07:52:391
2035975993926,10cyclictest23-21ksoftirqd/111:41:171
2035975993926,10cyclictest23-21ksoftirqd/111:41:161
2035975993925,11cyclictest23-21ksoftirqd/109:40:001
2035975993925,10cyclictest23-21ksoftirqd/112:22:211
2035975993912,6cyclictest131rcu_preempt09:10:031
2035975993912,6cyclictest131rcu_preempt09:10:021
203598499388,21cyclictest2042161-21sh09:10:003
203598499388,21cyclictest2042161-21sh09:10:003
203598499384,3cyclictest131rcu_preempt07:30:013
203598499384,3cyclictest131rcu_preempt07:10:023
2035984993827,9cyclictest37-21ksoftirqd/307:45:023
2035984993824,10cyclictest275-21dbus-daemon07:50:013
2035984993818,11cyclictest0-21swapper/308:00:003
2035979993826,9cyclictest3467602-21systemd-journal07:35:022
203597999382,3cyclictest131rcu_preempt08:45:022
203597999381,3cyclictest131rcu_preempt10:13:482
203597599385,7cyclictest23-21ksoftirqd/110:11:561
2035975993826,8cyclictest275-21dbus-daemon10:05:021
2035975993826,8cyclictest275-21dbus-daemon10:05:011
203597599382,5cyclictest131rcu_preempt10:00:021
2035975993825,9cyclictest2038008-21systemd-run07:50:001
2035975993825,10cyclictest23-21ksoftirqd/109:20:491
2035984993725,9cyclictest309-21in:imuxsock08:41:403
203598499372,4cyclictest131rcu_preempt09:28:123
2035984993724,9cyclictest0-21swapper/308:40:013
203598499372,3cyclictest131rcu_preempt10:00:103
203598499372,3cyclictest131rcu_preempt10:00:103
2035984993722,11cyclictest1-21systemd08:20:013
2035975993726,9cyclictest23-21ksoftirqd/111:10:001
2035975993724,9cyclictest23-21ksoftirqd/108:10:021
2035984993626,6cyclictest37-21ksoftirqd/307:39:133
2035984993623,11cyclictest37-21ksoftirqd/308:10:013
2035984993622,9cyclictest3467602-21systemd-journal07:20:013
203597999363,4cyclictest131rcu_preempt08:00:022
2035979993621,11cyclictest2040604-21sendmail08:40:012
203597599363,4cyclictest131rcu_preempt09:50:361
2035975993626,8cyclictest23-21ksoftirqd/112:10:471
2035975993626,8cyclictest23-21ksoftirqd/112:10:461
2035975993624,9cyclictest23-21ksoftirqd/112:06:001
2035975993624,9cyclictest23-21ksoftirqd/107:45:021
2035975993624,9cyclictest23-21ksoftirqd/107:45:011
2035975993616,15cyclictest3467602-21systemd-journal07:20:001
2035975993610,4cyclictest131rcu_preempt07:36:291
2035984993526,5cyclictest2038249-21munin-plugin-st07:55:003
2035984993521,11cyclictest37-21ksoftirqd/307:21:393
2035979993527,5cyclictest2036013-21latency_hist07:10:022
2035979993527,5cyclictest2026300-21kworker/2:307:13:402
2035979993527,5cyclictest2026300-21kworker/2:307:13:392
2035975993527,5cyclictest2080769-21sh10:30:491
2035975993526,5cyclictest2040349-21latency_hist08:35:021
2035975993526,5cyclictest2040349-21latency_hist08:35:011
2035975993524,8cyclictest23-21ksoftirqd/110:35:071
2035975993523,9cyclictest23-21ksoftirqd/108:39:041
2035975993522,9cyclictest309-21in:imuxsock07:27:401
2035975993520,11cyclictest2042139-21cron09:10:001
2035975993520,11cyclictest2042139-21cron09:10:001
2035975993519,12cyclictest23-21ksoftirqd/109:00:011
2035975993519,12cyclictest23-21ksoftirqd/109:00:001
203598499345,21cyclictest1-21systemd08:35:023
203598499345,21cyclictest1-21systemd08:35:013
2035984993425,6cyclictest2037804-21cat07:45:013
2035984993422,10cyclictest37-21ksoftirqd/308:30:013
203597599343,22cyclictest0-21swapper/108:11:421
2035975993425,7cyclictest23-21ksoftirqd/107:10:011
2035975993424,7cyclictest23-21ksoftirqd/108:45:391
2035975993420,10cyclictest3467602-21systemd-journal08:55:011
203597599341,5cyclictest2037174-21modprobe07:30:021
203598499338,4cyclictest131rcu_preempt08:11:423
2035975993320,10cyclictest23-21ksoftirqd/107:13:531
2035975993320,10cyclictest23-21ksoftirqd/107:13:531
2035975993318,11cyclictest23-21ksoftirqd/108:25:021
2035975993219,10cyclictest23-21ksoftirqd/108:41:391
2035975993123,6cyclictest23-21ksoftirqd/107:20:021
2035975993123,5cyclictest2038501-21latency_hist08:00:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional