You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-04 - 21:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Sat Apr 04, 2026 12:45:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
63401099766,51cyclictest717909-21ssh12:10:011
63402299671,12cyclictest3467602-21systemd-journal09:25:413
63402299661,12cyclictest48-21kcompactd008:45:253
634022996535,25cyclictest275-21dbus-daemon10:20:013
63402299652,18cyclictest703144-21ssh11:35:313
63402299651,14cyclictest681827-21ssh10:46:143
63402299641,13cyclictest649152-21ssh09:30:403
63402299632,18cyclictest676447-21sshd10:34:163
63402299632,12cyclictest364-21snmpd10:38:583
63402299632,11cyclictest364-21snmpd11:55:483
63402299631,9cyclictest150-21pr/ttyS209:47:133
63402299621,14cyclictest642406-21munin-run09:15:013
63402299621,13cyclictest671489-21ssh10:22:273
63402299621,10cyclictest653515-21ssh09:40:433
63402299612,14cyclictest665957-21munin-run10:10:013
63402299611,9cyclictest686851-21ssh10:58:053
63402299611,13cyclictest720749-21sh12:16:323
63402299611,13cyclictest639395-21munin-run08:55:013
63402299611,13cyclictest639395-21munin-run08:55:003
63402299611,13cyclictest637036-21sshd08:06:163
63402299611,13cyclictest637036-21sshd08:06:163
63402299611,10cyclictest659115-21ssh09:54:093
634008996126,17cyclictest713533-21ssh12:00:000
634008996121,36cyclictest704886-21latency_hist11:40:010
63400899611,57cyclictest0-21swapper/009:35:020
634022996031,24cyclictest693551-21sh11:13:473
63402299602,6cyclictest635512-21munin-run07:40:003
634022996022,33cyclictest37-21ksoftirqd/312:32:163
63402299601,9cyclictest695047-21ssh11:16:553
63402299601,9cyclictest639894-21sshd09:00:373
63402299601,12cyclictest707655-21sshd11:46:063
634008996035,20cyclictest657317-21munin-run09:49:590
634008996032,12cyclictest685220-21ssh10:54:260
63402299599,46cyclictest0-21swapper/307:18:193
63402299592,20cyclictest644264-21ssh09:19:193
63402299592,13cyclictest48-21kcompactd012:27:033
63402299592,12cyclictest364-21snmpd10:14:513
63402299591,13cyclictest688329-21sshd11:01:163
634022995910,44cyclictest0-21swapper/307:43:173
63402299582,22cyclictest3467602-21systemd-journal11:05:153
63402299582,16cyclictest680719-21sh10:44:003
63402299582,15cyclictest652196-21ssh09:37:533
634022995815,37cyclictest645042-21sh09:20:423
634022995813,40cyclictest636256-21sshd07:52:003
63402299581,14cyclictest364-21snmpd08:00:593
63402299581,11cyclictest699626-21ssh11:27:303
634008995822,32cyclictest12-21ksoftirqd/009:45:010
634008995815,4cyclictest131rcu_preempt09:20:520
63402299576,4cyclictest0-21swapper/307:57:143
63402299573,9cyclictest639660-21grep09:00:003
63402299573,16cyclictest640205-21munin-run09:10:013
63402299572,17cyclictest685389-21munin-run10:55:013
63402299572,13cyclictest723054-21ssh12:21:543
63402299572,13cyclictest723054-21ssh12:21:543
63402299572,11cyclictest698416-21munin-run11:25:013
634022995712,40cyclictest719126-21journal-offline12:12:533
634010995725,21cyclictest221rcuc/110:55:021
634010995713,8cyclictest23-21ksoftirqd/112:38:381
634008995736,15cyclictest1-21systemd09:15:000
634008995723,28cyclictest12-21ksoftirqd/012:15:320
63400899571,40cyclictest699011-21ssh11:25:550
63402299567,20cyclictest634346-21cat07:15:023
63402299563,7cyclictest637849-21modprobe08:25:003
63402299562,9cyclictest364-21snmpd07:46:473
63402299562,8cyclictest364-21snmpd08:32:513
63402299561,7cyclictest635238-21cron07:35:003
634022995617,35cyclictest663534-21sh10:04:183
63400899568,30cyclictest16950irq/224-800000012:35:020
63402299553,10cyclictest661625-21munin-run10:00:003
63402299552,18cyclictest672855-21ssh10:25:293
63402299552,13cyclictest364-21snmpd12:37:573
63401699559,30cyclictest643188-21kworker/u8:009:20:032
634010995515,10cyclictest0-21swapper/111:06:311
634010995510,8cyclictest23-21ksoftirqd/111:46:501
634008995531,17cyclictest16950irq/224-800000011:50:020
634008995528,12cyclictest671697-21ssh10:23:060
634008995523,15cyclictest662073-21ssh10:00:290
634008995518,35cyclictest12-21ksoftirqd/010:10:020
63402299543,8cyclictest717159-21ssh12:08:243
63402299543,6cyclictest284-21systemd-logind08:30:013
634022995430,20cyclictest702622-21sh11:34:483
63402299542,6cyclictest637317-21munin-run08:15:003
63402299541,14cyclictest364-21snmpd11:51:483
63402299541,12cyclictest364-21snmpd12:04:013
63402299541,12cyclictest364-21snmpd12:04:003
63401099544,12cyclictest23-21ksoftirqd/110:18:221
634010995426,19cyclictest709239-21kworker/1:012:30:021
634010995420,5cyclictest0-21swapper/109:45:011
63400899543,37cyclictest131rcu_preempt12:14:120
634008995427,12cyclictest701469-21ssh11:31:420
63400899541,34cyclictest131rcu_preempt10:57:490
634022995324,23cyclictest634996-21latency_hist07:30:023
63402299531,13cyclictest364-21snmpd07:21:163
634016995342,7cyclictest30-21ksoftirqd/211:45:032
63401699533,3cyclictest131rcu_preempt11:27:152
634010995338,13cyclictest23-21ksoftirqd/112:18:281
634010995310,6cyclictest0-21swapper/110:31:081
634008995344,5cyclictest16950irq/224-800000007:50:000
634008995326,12cyclictest690020-21ssh11:05:070
634008995326,12cyclictest674586-21ssh10:29:530
634008995325,12cyclictest675763-21ssh10:32:310
63400899532,12cyclictest636533-21modprobe08:00:000
63402299521,13cyclictest638858-21sshd08:40:173
63402299521,12cyclictest364-21snmpd11:41:213
634016995244,5cyclictest639156-21kworker/2:310:30:032
634016995244,5cyclictest635261-21kworker/2:010:26:012
634016995243,5cyclictest639156-21kworker/2:311:47:402
634016995242,6cyclictest635261-21kworker/2:010:05:002
63401099524,10cyclictest23-21ksoftirqd/111:21:311
634010995230,15cyclictest3467602-21systemd-journal11:40:011
634008995223,14cyclictest689683-21ssh11:04:520
634016995143,5cyclictest635261-21kworker/2:011:08:322
634016995143,5cyclictest635261-21kworker/2:009:10:022
634016995141,6cyclictest635261-21kworker/2:012:23:292
634016995141,6cyclictest635261-21kworker/2:012:23:282
63401099518,8cyclictest723449-21ssh12:22:581
63401099518,8cyclictest723449-21ssh12:22:571
63401099517,8cyclictest23-21ksoftirqd/109:39:351
63401099515,8cyclictest23-21ksoftirqd/111:53:071
634010995133,13cyclictest713498-21systemctl12:00:001
634010995131,14cyclictest275-21dbus-daemon09:30:011
634010995110,10cyclictest672838-21rm10:25:261
634008995123,13cyclictest697115-21ssh11:21:430
634008995121,14cyclictest705552-21ssh11:41:080
63402299502,11cyclictest1-21systemd08:17:163
634016995042,5cyclictest639156-21kworker/2:311:59:302
634016995042,5cyclictest639156-21kworker/2:310:53:262
634016995042,5cyclictest635261-21kworker/2:009:45:312
634016995041,6cyclictest639156-21kworker/2:311:53:522
634016995041,6cyclictest623546-21kworker/2:207:12:362
63401099508,8cyclictest0-21swapper/110:25:021
63401099501,9cyclictest23-21ksoftirqd/111:18:381
634010995011,9cyclictest23-21ksoftirqd/107:35:011
634010995011,7cyclictest23-21ksoftirqd/111:29:081
63401099501,15cyclictest706726-21ssh11:44:171
63400899502,42cyclictest725001-21sshd12:26:150
634008995024,11cyclictest666940-21ssh10:12:040
634016994941,5cyclictest639156-21kworker/2:308:58:162
634016994941,5cyclictest635261-21kworker/2:012:05:032
634016994941,5cyclictest635261-21kworker/2:011:20:372
634016994941,5cyclictest635261-21kworker/2:009:42:472
634016994940,6cyclictest635261-21kworker/2:010:12:322
63401099496,7cyclictest23-21ksoftirqd/108:05:011
63401099494,8cyclictest23-21ksoftirqd/109:19:011
634010994931,14cyclictest1-21systemd07:40:011
634010994914,7cyclictest0-21swapper/110:46:591
63401099491,14cyclictest680309-21ssh10:42:551
634008994935,8cyclictest309-21in:imuxsock07:25:020
634008994922,12cyclictest651305-21ssh09:35:340
634008994921,25cyclictest463711-1kworker/0:1H08:55:030
634008994921,25cyclictest463711-1kworker/0:1H08:55:020
63400899491,3cyclictest150-21pr/ttyS210:40:020
634008994912,5cyclictest658818-21sh09:53:250
63402299481,10cyclictest638651-21sh08:40:013
634016994840,5cyclictest639156-21kworker/2:310:48:322
634016994840,5cyclictest639156-21kworker/2:309:56:172
634016994840,5cyclictest635261-21kworker/2:011:11:252
634016994840,5cyclictest635261-21kworker/2:010:21:462
634016994839,6cyclictest639156-21kworker/2:309:32:132
634016994837,7cyclictest30-21ksoftirqd/211:00:022
634016994831,13cyclictest30-21ksoftirqd/212:20:002
634016994831,12cyclictest275-21dbus-daemon07:55:012
63401099487,7cyclictest23-21ksoftirqd/111:12:111
63401099485,8cyclictest726820-21ssh12:30:361
63401099485,8cyclictest23-21ksoftirqd/110:37:161
634010994836,9cyclictest23-21ksoftirqd/107:55:021
63400899486,33cyclictest19350irq/34-mmc110:00:020
634008994837,6cyclictest16950irq/224-800000012:05:020
634008994837,6cyclictest16950irq/224-800000012:05:010
63400899482,17cyclictest670283-21grep10:20:000
634008994821,23cyclictest274-21cron09:17:000
634008994821,12cyclictest692296-21ssh11:10:300
634008994821,12cyclictest648445-21ssh09:29:200
634008994817,26cyclictest639635-21cron08:59:590
634016994741,3cyclictest635261-21kworker/2:012:37:132
634016994739,5cyclictest639156-21kworker/2:312:14:102
634016994739,5cyclictest639156-21kworker/2:311:03:332
634016994739,5cyclictest635261-21kworker/2:011:32:492
634016994739,5cyclictest623546-21kworker/2:207:21:142
634016994738,6cyclictest635261-21kworker/2:012:34:362
634016994738,6cyclictest635261-21kworker/2:009:05:022
634016994738,5cyclictest639156-21kworker/2:311:39:192
63401099479,9cyclictest23-21ksoftirqd/109:54:521
63401099478,5cyclictest0-21swapper/110:55:531
63401099474,9cyclictest23-21ksoftirqd/109:50:021
634010994733,10cyclictest275-21dbus-daemon10:15:001
634010994727,15cyclictest1-21systemd08:25:011
634010994714,5cyclictest0-21swapper/112:03:311
634010994714,5cyclictest0-21swapper/112:03:311
634010994711,5cyclictest0-21swapper/111:00:141
63401099471,10cyclictest23-21ksoftirqd/110:01:491
634008994728,15cyclictest723632-21ssh12:23:240
634008994728,15cyclictest723632-21ssh12:23:230
634008994728,15cyclictest638591-21kworker/u8:109:10:010
63401699464,3cyclictest131rcu_preempt12:29:032
634016994638,5cyclictest639156-21kworker/2:309:52:192
634016994635,8cyclictest30-21ksoftirqd/210:15:502
634016994632,11cyclictest3467602-21systemd-journal11:15:172
63401099467,8cyclictest23-21ksoftirqd/110:06:501
634010994613,8cyclictest0-21swapper/108:06:441
634010994613,8cyclictest0-21swapper/108:06:441
634008994638,5cyclictest16950irq/224-800000008:40:000
634008994628,14cyclictest681052-21ssh10:44:470
634008994627,15cyclictest637280-21sshd08:10:170
63400899462,13cyclictest694389-21ssh11:15:230
634008994620,11cyclictest730028-21ssh12:38:250
634008994620,10cyclictest683158-21ssh10:49:440
634016994538,4cyclictest635261-21kworker/2:009:38:312
634016994537,5cyclictest639156-21kworker/2:310:36:442
634016994537,5cyclictest623546-21kworker/2:207:20:022
63401099459,10cyclictest23-21ksoftirqd/112:10:081
63401099455,8cyclictest23-21ksoftirqd/108:55:011
63401099455,8cyclictest23-21ksoftirqd/108:55:011
63401099454,11cyclictest23-21ksoftirqd/109:55:341
634010994532,8cyclictest638112-21grep08:30:001
634010994512,6cyclictest0-21swapper/111:31:111
634016994436,5cyclictest639156-21kworker/2:309:25:432
634016994427,13cyclictest30-21ksoftirqd/208:50:012
63401099442,7cyclictest0-21swapper/109:10:061
634010994411,9cyclictest0-21swapper/107:19:581
63400899441,39cyclictest716575-21ssh12:06:530
634016994334,6cyclictest639156-21kworker/2:309:18:352
634016994323,11cyclictest3467602-21systemd-journal08:20:002
63401099435,9cyclictest23-21ksoftirqd/109:33:181
63401099435,6cyclictest23-21ksoftirqd/109:24:571
63401099434,9cyclictest23-21ksoftirqd/109:10:031
63401099431,6cyclictest0-21swapper/108:40:011
634010994312,6cyclictest0-21swapper/107:28:011
634008994331,9cyclictest16950irq/224-800000009:05:010
634016994234,5cyclictest635261-21kworker/2:010:06:512
634016994234,5cyclictest635261-21kworker/2:009:11:182
634016994230,8cyclictest1-21systemd07:35:012
63401699421,3cyclictest131rcu_preempt10:43:232
634010994233,7cyclictest221rcuc/108:00:011
63401099421,8cyclictest23-21ksoftirqd/108:10:191
63400899423,6cyclictest637571-21modprobe08:20:000
634008994236,4cyclictest19350irq/34-mmc108:25:010
634008994235,5cyclictest0-21swapper/008:50:000
63400899423,35cyclictest0-21swapper/008:35:000
634008994225,13cyclictest636258-21sshd07:51:590
63400899421,3cyclictest131rcu_preempt08:10:010
63400899421,3cyclictest131rcu_preempt08:10:010
63402299411,9cyclictest634064-21latency_hist07:10:013
634016994130,8cyclictest637842-21sshd08:24:172
634010994132,6cyclictest634745-21munin-plugin-st07:25:001
63400899412,35cyclictest0-21swapper/007:20:000
634008994123,14cyclictest3467602-21systemd-journal08:30:000
63400899411,28cyclictest709540-21ssh11:50:160
634008994031,6cyclictest19350irq/34-mmc108:43:000
634016993931,5cyclictest714563-21sh12:02:152
634016993931,5cyclictest714563-21sh12:02:142
634016993930,5cyclictest637060-21sh08:10:002
634016993930,5cyclictest637060-21sh08:10:002
634016993924,9cyclictest275-21dbus-daemon08:55:002
634016993924,9cyclictest275-21dbus-daemon08:55:002
634010993918,15cyclictest3467602-21systemd-journal08:35:011
634010993918,12cyclictest635775-21modprobe07:45:001
634008993930,6cyclictest19350irq/34-mmc108:01:380
634008993923,11cyclictest364-21snmpd07:42:420
634016993828,7cyclictest638579-21sshd08:35:292
634016993823,12cyclictest3467602-21systemd-journal08:10:022
634010993826,9cyclictest23-21ksoftirqd/108:19:431
634008993829,6cyclictest19350irq/34-mmc107:28:590
63400899382,8cyclictest3467602-21systemd-journal07:40:000
634008993826,7cyclictest634195-21cron07:15:000
634016993729,5cyclictest636032-21latency_hist07:50:032
634016993723,10cyclictest30-21ksoftirqd/207:58:592
634010993712,7cyclictest0-21swapper/108:57:351
634008993729,5cyclictest0-21swapper/007:31:450
63401699363,3cyclictest131rcu_preempt08:35:002
634016993627,7cyclictest0-21swapper/207:10:022
634016993627,6cyclictest635503-21sendmail07:40:002
634016993624,8cyclictest634734-21cron07:25:022
63401699361,31cyclictest48-21kcompactd008:00:292
63401099358,6cyclictest0-21swapper/107:45:501
63401099355,7cyclictest23-21ksoftirqd/107:15:011
63401699343,4cyclictest131rcu_preempt08:30:002
634016993427,4cyclictest638965-21latency_hist08:45:022
634016993422,10cyclictest30-21ksoftirqd/207:42:582
634010993421,10cyclictest639020-21sed08:45:021
63401099336,5cyclictest131rcu_preempt09:04:161
634010993326,5cyclictest639145-21/usr/sbin/munin08:50:001
63400899334,19cyclictest634009-21latency_hist07:10:010
634010993217,12cyclictest634083-21cat07:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional