You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-16 - 18:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Thu Jul 16, 2026 12:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
41203399912636,14cyclictest131rcu_preempt12:20:030
41203399912345,20cyclictest16850irq/134-800000011:30:020
412033999121116,3cyclictest0-21swapper/007:10:030
4120339999026,13cyclictest4180224-21ssh11:01:530
4120349998823,33cyclictest13466-21rm11:59:133
412033999851,40cyclictest4142566-21ssh09:43:070
4120339998323,12cyclictest19385-21ssh12:11:140
4120339998235,42cyclictest16850irq/134-800000011:35:010
4120339998235,42cyclictest16850irq/134-800000011:35:010
4120339998223,15cyclictest9457-21ssh11:50:250
4120339998023,14cyclictest4128325-21ssh09:13:280
4120339998023,14cyclictest4128325-21ssh09:13:280
4120339997929,46cyclictest4189369-21ssh11:21:000
4120339997927,17cyclictest4145410-21ssh09:49:270
4120339997927,10cyclictest4154576-21ssh10:08:280
4120339997927,10cyclictest4154576-21ssh10:08:280
4120339997924,14cyclictest4133324-21ssh09:23:530
4120339997813,17cyclictest16850irq/134-800000011:40:370
4120339997813,17cyclictest16850irq/134-800000011:40:370
4120339997721,12cyclictest4147292-21ssh09:53:080
4120339997628,17cyclictest4175342-21ssh10:51:420
4120339997428,14cyclictest4169962-21ssh10:40:110
412033999742,66cyclictest4186553-21modprobe11:15:020
4120344997367,4cyclictest4119851-21kworker/u8:207:25:031
4120339997334,35cyclictest4294-21munin-plugin-st11:40:000
4120339997318,12cyclictest4149682-21ssh09:58:110
4120339997312,56cyclictest275-21dbus-daemon09:20:010
4120339997215,12cyclictest32786-21ssh12:39:450
412033999717,20cyclictest0-21swapper/009:30:020
4120339997125,16cyclictest4137351-21ssh09:32:100
4120339997121,12cyclictest17606-21ssh12:07:380
4120339997038,28cyclictest16950irq/224-800000010:20:010
4120339997034,32cyclictest16850irq/134-800000012:30:220
4120339997026,40cyclictest0-21swapper/012:21:330
4120339996828,9cyclictest4186122-21ssh11:14:370
4120339996828,36cyclictest16850irq/134-800000012:00:020
412033999672,6cyclictest4125032-21munin-run08:40:010
4120349996628,34cyclictest4162936-21sh10:25:253
4120339996628,8cyclictest4156590-21ssh10:12:270
412033999661,28cyclictest0-21swapper/010:56:300
412034999652,19cyclictest4141017-21/usr/sbin/munin09:40:023
412034999652,15cyclictest4147425-21ssh09:53:243
412034999651,18cyclictest24511-21ssh12:22:063
412034999651,13cyclictest4133308-21ssh09:23:513
4120339996519,42cyclictest4165165-21ssh10:30:120
4120339996517,43cyclictest16850irq/134-800000010:45:420
4120339996517,43cyclictest16850irq/134-800000010:45:420
412034999642,20cyclictest32645-21ssh12:39:273
412034999642,18cyclictest4167522-21ssh10:35:073
412033999643,18cyclictest4126575-21modprobe09:10:000
4120339996423,4cyclictest0-21swapper/012:25:380
4120339996419,15cyclictest3467602-21systemd-journal11:50:000
412034999632,5cyclictest4120579-21munin-run07:15:013
412034999631,19cyclictest4183785-21ssh11:09:433
412034999631,10cyclictest11066-21ssh11:54:043
4120339996323,5cyclictest4123829-21latency_hist08:15:020
4120339996323,5cyclictest4123829-21latency_hist08:15:020
4120339996320,12cyclictest16117-21ssh12:04:430
4120339996319,6cyclictest4182067-21ssh11:05:350
4120349996224,33cyclictest26256-21sh12:25:323
412034999622,18cyclictest4137119-21sh09:31:413
412034999622,13cyclictest4152897-21munin-run10:05:023
412034999622,11cyclictest4173092-21rm10:47:043
412034999622,11cyclictest4173092-21rm10:47:043
412034999621,9cyclictest4187360-21ssh11:16:513
4120349996216,41cyclictest1-21systemd11:15:013
412034999621,10cyclictest4177708-21ssh10:56:373
412034999613,4cyclictest4123673-21sshd08:13:223
412034999613,4cyclictest4123673-21sshd08:13:213
412034999612,4cyclictest3467602-21systemd-journal09:12:383
412034999612,4cyclictest3467602-21systemd-journal09:12:373
412034999611,9cyclictest2285-21ssh11:35:223
412034999611,9cyclictest21500-21ssh12:15:393
412034999611,8cyclictest4154556-21ssh10:08:263
412034999611,8cyclictest4154556-21ssh10:08:253
412034999611,8cyclictest4129661-21ssh09:16:113
412034999611,8cyclictest19661-21ssh12:11:573
412034999611,8cyclictest17276-21ssh12:06:573
412034999611,7cyclictest4141509-21ssh09:40:443
412034999611,16cyclictest1240-21ssh11:33:303
412034999611,16cyclictest1240-21ssh11:33:303
4120339996122,9cyclictest4168812-21ssh10:38:100
412034999601,9cyclictest4158089-21ssh10:15:273
412034999601,8cyclictest4165911-21ssh10:31:543
412034999601,7cyclictest5522-21ssh11:42:223
412034999601,7cyclictest5522-21ssh11:42:223
412034999601,7cyclictest4174900-21ssh10:50:393
412034999601,7cyclictest4144879-21ssh09:48:113
412034999601,7cyclictest4134643-21ssh09:26:293
412034999601,7cyclictest28923-21ssh12:31:183
412034999601,6cyclictest4170836-21ssh10:42:113
412034999601,5cyclictest4180870-21sshd11:03:223
412034999601,18cyclictest4189706-21ssh11:21:423
412034999601,15cyclictest4193604-21wc11:30:033
4120339996028,8cyclictest4163697-21ssh10:27:170
4120339996022,6cyclictest4125231-21modprobe08:40:010
412034999591,6cyclictest7719-21ssh11:46:583
412034999591,6cyclictest4148564-21ssh09:55:373
412034999591,5cyclictest4155740-21ssh10:10:323
412034999591,19cyclictest14945-21ssh12:02:013
4120339995931,13cyclictest4139506-21ssh09:36:350
4120339995923,8cyclictest12-21ksoftirqd/008:25:010
4120339995923,7cyclictest4152941-21ssh10:05:030
412034999581,4cyclictest4161201-21ssh10:21:583
4120339995829,2cyclictest0-21swapper/009:02:230
4120339995829,2cyclictest0-21swapper/009:02:220
412033999581,27cyclictest0-21swapper/007:38:170
412034999574,45cyclictest0-21swapper/308:55:023
412034999571,4cyclictest4125239-21sshd08:40:373
412033999571,52cyclictest0-21swapper/008:55:010
412033999571,3cyclictest0-21swapper/010:23:410
412034999562,17cyclictest4123914-21sshd08:15:333
412034999561,6cyclictest4121083-21sshd07:20:073
412034999561,4cyclictest364-21snmpd07:27:353
4120349995611,40cyclictest0-21swapper/309:09:033
4120339995626,3cyclictest0-21swapper/008:00:010
4120339995622,12cyclictest12-21ksoftirqd/008:50:010
412034999552,5cyclictest4121594-21sshd07:31:223
412034999552,4cyclictest4126012-21modprobe08:59:213
412034999552,15cyclictest4124752-21munin-run08:35:023
412034999552,15cyclictest4124752-21munin-run08:35:023
412034999552,13cyclictest4123124-21sed08:00:033
4120346995547,5cyclictest4124247-21kworker/2:209:44:042
4120339995522,29cyclictest12-21ksoftirqd/007:20:010
412033999551,29cyclictest0-21swapper/008:35:010
412033999551,29cyclictest0-21swapper/008:35:000
412034999542,15cyclictest4123165-21sshd08:02:223
412034999542,11cyclictest4124469-21sshd08:29:233
412034999541,5cyclictest4123454-21munin-run08:10:013
4120346995445,6cyclictest4131415-21kworker/2:310:11:162
4120346995437,13cyclictest3467602-21systemd-journal09:25:012
4120346995435,14cyclictest275-21dbus-daemon12:00:012
4120339995418,6cyclictest0-21swapper/007:30:000
412034999532,4cyclictest4122135-21sshd07:43:223
412034999531,5cyclictest4126270-21sshd09:00:223
412034999531,5cyclictest4126270-21sshd09:00:223
412034999531,14cyclictest4122418-21systemctl07:50:013
412034999531,14cyclictest4122418-21systemctl07:50:003
4120346995346,4cyclictest4131415-21kworker/2:309:36:462
4120346995345,5cyclictest4155290-21kworker/2:011:11:012
4120346995345,5cyclictest4124247-21kworker/2:210:07:522
4120346995345,5cyclictest4124247-21kworker/2:210:07:522
4120346995344,4cyclictest4131415-21kworker/2:311:25:062
4120339995316,11cyclictest4122295-21wc07:45:020
412034999521,7cyclictest364-21snmpd07:16:113
4120346995244,5cyclictest4155290-21kworker/2:010:45:482
4120346995244,5cyclictest4155290-21kworker/2:010:45:472
4120346995244,5cyclictest4131415-21kworker/2:310:35:242
4120346995244,5cyclictest4131415-21kworker/2:310:24:242
4120346995244,5cyclictest4131415-21kworker/2:310:00:022
4120346995244,5cyclictest4124247-21kworker/2:209:45:152
4120346995244,5cyclictest4123405-21kworker/u8:108:45:032
4120346995244,5cyclictest4113174-21kworker/2:009:16:462
4120346995243,6cyclictest4131415-21kworker/2:310:57:342
4120346995233,14cyclictest4165510-21sshd10:30:562
412034999513,6cyclictest364-21snmpd07:52:403
412034999512,3cyclictest4125505-21sshd08:49:223
4120346995143,5cyclictest4155290-21kworker/2:011:20:552
4120346995143,5cyclictest4131415-21kworker/2:312:24:542
4120346995143,5cyclictest4131415-21kworker/2:312:01:362
4120346995143,5cyclictest4131415-21kworker/2:311:30:072
4120346995143,5cyclictest4131415-21kworker/2:311:30:062
4120346995143,5cyclictest4131415-21kworker/2:310:29:322
4120346995143,5cyclictest4124247-21kworker/2:209:55:252
4120346995142,6cyclictest4131415-21kworker/2:312:16:512
4120346995142,6cyclictest4124247-21kworker/2:209:30:092
4120346995142,6cyclictest28185-21kworker/2:212:38:432
412033999517,17cyclictest275-21dbus-daemon07:55:010
4120349995028,19cyclictest37-21ksoftirqd/308:40:023
4120346995042,5cyclictest4155290-21kworker/2:011:52:452
4120346995042,5cyclictest4155290-21kworker/2:011:48:012
4120346995042,5cyclictest4155290-21kworker/2:010:50:362
4120346995042,5cyclictest4155290-21kworker/2:010:43:452
4120346995042,5cyclictest4131415-21kworker/2:312:27:002
4120346995042,5cyclictest4131415-21kworker/2:312:10:552
4120346995042,5cyclictest4131415-21kworker/2:311:36:492
4120346995042,5cyclictest4131415-21kworker/2:311:02:122
4120346995042,5cyclictest4124247-21kworker/2:209:29:222
4120346995041,6cyclictest4131415-21kworker/2:312:08:232
4120346995041,6cyclictest4131415-21kworker/2:311:40:112
4120346995041,6cyclictest4131415-21kworker/2:311:40:102
4120346995027,16cyclictest275-21dbus-daemon08:15:012
4120346995027,16cyclictest275-21dbus-daemon08:15:012
412034499501,31cyclictest4120240-21kworker/u8:307:55:021
412034999491,13cyclictest4120456-21wc07:10:033
4120346994941,5cyclictest4155290-21kworker/2:011:15:582
4120346994941,5cyclictest4155290-21kworker/2:011:05:112
4120346994941,5cyclictest4131415-21kworker/2:312:31:452
4120346994941,5cyclictest4131415-21kworker/2:310:16:022
412034499492,36cyclictest221rcuc/112:25:021
4120339994934,10cyclictest275-21dbus-daemon08:20:000
4120339994919,11cyclictest275-21dbus-daemon09:00:010
4120346994840,5cyclictest4113174-21kworker/2:009:12:532
4120346994840,5cyclictest4113174-21kworker/2:009:12:522
4120346994840,5cyclictest4113174-21kworker/2:007:20:192
4120346994839,6cyclictest4124247-21kworker/2:209:50:402
4120346994839,6cyclictest4113174-21kworker/2:007:59:072
412034499486,24cyclictest4126562-21diskmemload11:55:021
4120339994836,9cyclictest3467602-21systemd-journal07:25:010
4120346994739,5cyclictest4113174-21kworker/2:009:04:352
4120346994739,5cyclictest4113174-21kworker/2:009:04:352
4120346994739,5cyclictest4113174-21kworker/2:008:50:582
4120346994739,5cyclictest4113174-21kworker/2:007:43:222
4120346994739,5cyclictest4113174-21kworker/2:007:27:372
4120346994739,5cyclictest4113174-21kworker/2:007:17:492
4120346994738,6cyclictest4113174-21kworker/2:008:00:282
4120346994736,7cyclictest30-21ksoftirqd/207:50:022
4120346994736,7cyclictest30-21ksoftirqd/207:50:012
412034499473,3cyclictest131rcu_preempt11:05:241
412034999461,4cyclictest364-21snmpd07:37:363
412034999461,4cyclictest1-21systemd08:25:003
412034699466,28cyclictest4124984-21modprobe08:39:592
4120346994638,5cyclictest4113174-21kworker/2:008:25:092
4120346994638,5cyclictest4113174-21kworker/2:008:17:412
4120346994638,5cyclictest4113174-21kworker/2:007:54:402
4120346994638,5cyclictest4113174-21kworker/2:007:37:182
4120346994638,5cyclictest4113174-21kworker/2:007:11:532
4120346994631,8cyclictest4126016-21cron09:00:002
4120346994537,5cyclictest4113174-21kworker/2:008:42:552
4120346994537,5cyclictest4113174-21kworker/2:008:05:102
4120346994537,5cyclictest4113174-21kworker/2:007:32:182
4120346994535,6cyclictest4113174-21kworker/2:008:32:072
4120346994535,6cyclictest4113174-21kworker/2:008:32:072
4120344994531,8cyclictest4126307-21munin-plugin-st09:05:011
4120344994531,8cyclictest4126307-21munin-plugin-st09:05:011
4120344994527,14cyclictest23-21ksoftirqd/110:40:011
4120344994520,5cyclictest23-21ksoftirqd/112:06:091
4120346994435,5cyclictest4113174-21kworker/2:009:09:152
4120344994431,10cyclictest4187988-21sshd11:18:221
4120344994429,12cyclictest3467602-21systemd-journal09:10:221
4120344994429,12cyclictest3467602-21systemd-journal09:10:221
412034499441,3cyclictest131rcu_preempt10:28:361
4120346994335,5cyclictest4113174-21kworker/2:008:20:472
412034499434,4cyclictest131rcu_preempt09:50:011
412034499433,3cyclictest131rcu_preempt10:01:381
4120344994329,9cyclictest309-21in:imuxsock08:10:301
4120344994329,9cyclictest309-21in:imuxsock08:10:301
4120344994327,11cyclictest4124978-21cron08:40:001
4120344994327,11cyclictest23-21ksoftirqd/111:30:011
4120344994327,10cyclictest275-21dbus-daemon09:30:001
4120344994326,12cyclictest23-21ksoftirqd/109:36:351
4120344994325,13cyclictest3467602-21systemd-journal10:17:001
4120344994230,9cyclictest3467602-21systemd-journal12:30:001
4120344994222,13cyclictest275-21dbus-daemon09:00:001
4120344994134,3cyclictest231ksoftirqd/111:39:591
4120344994131,7cyclictest9060-21ssh11:50:001
4120344994130,7cyclictest4179161-21sh11:00:001
4120344994129,9cyclictest275-21dbus-daemon11:45:021
4120344994129,9cyclictest275-21dbus-daemon11:45:021
4120344994129,10cyclictest23-21ksoftirqd/110:31:141
4120344994128,10cyclictest23-21ksoftirqd/110:22:421
4120344994125,12cyclictest3467602-21systemd-journal12:31:221
4120344994029,5cyclictest23-21ksoftirqd/109:09:591
4120344994027,9cyclictest309-21in:imuxsock12:20:011
4120344994027,9cyclictest309-21in:imuxsock07:29:591
4120344994027,9cyclictest23-21ksoftirqd/109:15:161
4120344994027,10cyclictest23-21ksoftirqd/111:24:591
412033999407,10cyclictest12-21ksoftirqd/008:10:010
412033999401,27cyclictest0-21swapper/008:04:330
4120344993927,9cyclictest23-21ksoftirqd/107:10:031
4120344993927,10cyclictest23-21ksoftirqd/110:07:401
4120344993927,10cyclictest23-21ksoftirqd/110:07:401
4120344993926,11cyclictest23-21ksoftirqd/111:00:191
4120344993925,9cyclictest309-21in:imuxsock10:50:031
4120344993925,10cyclictest23-21ksoftirqd/109:51:231
4120344993924,10cyclictest3467602-21systemd-journal11:35:001
4120344993924,10cyclictest3467602-21systemd-journal11:35:001
412033999394,27cyclictest4120342-21latency_hist07:10:010
4120339993923,12cyclictest3467602-21systemd-journal07:35:000
412034499383,32cyclictest4119851-21kworker/u8:207:50:031
412034499383,32cyclictest4119851-21kworker/u8:207:50:021
4120344993830,6cyclictest23-21ksoftirqd/111:55:231
4120344993829,7cyclictest23-21ksoftirqd/112:00:351
4120344993828,5cyclictest309-21in:imuxsock07:20:011
4120344993827,9cyclictest23-21ksoftirqd/110:11:591
4120344993823,11cyclictest23-21ksoftirqd/109:33:311
4120339993826,9cyclictest4122539-21date07:50:030
4120339993826,9cyclictest4122539-21date07:50:020
4120344993727,8cyclictest23-21ksoftirqd/112:12:471
4120344993727,8cyclictest23-21ksoftirqd/110:43:221
4120344993725,9cyclictest23-21ksoftirqd/109:41:341
4120344993723,11cyclictest23-21ksoftirqd/108:55:011
4120344993723,10cyclictest309-21in:imuxsock07:35:021
4120344993722,11cyclictest23-21ksoftirqd/111:12:541
4120344993717,15cyclictest275-21dbus-daemon10:00:001
412033999372,5cyclictest4124486-21munin-run08:30:000
4120344993626,8cyclictest23-21ksoftirqd/112:36:201
4120344993626,8cyclictest23-21ksoftirqd/108:50:021
4120344993624,8cyclictest23-21ksoftirqd/107:40:281
4120344993624,10cyclictest23-21ksoftirqd/110:46:111
4120344993624,10cyclictest23-21ksoftirqd/110:46:101
4120344993623,10cyclictest23-21ksoftirqd/109:24:311
4120344993620,12cyclictest4125301-21sed08:45:021
412034699351,11cyclictest4120348-21latency_hist07:10:022
4120344993524,8cyclictest23-21ksoftirqd/108:30:021
4120344993524,6cyclictest309-21in:imuxsock08:05:221
4120344993519,12cyclictest275-21dbus-daemon08:35:011
4120344993519,12cyclictest275-21dbus-daemon08:35:001
412034499351,30cyclictest23-21ksoftirqd/107:58:231
4120344993423,7cyclictest4121618-21sh07:35:011
4120344993321,9cyclictest23-21ksoftirqd/108:20:001
4120344993319,10cyclictest23-21ksoftirqd/108:25:021
4120344993218,10cyclictest23-21ksoftirqd/107:15:011
412034499291,4cyclictest131rcu_preempt08:02:211
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional