You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-15 - 16:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue Sep 15, 2026 12:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
408886699883,3cyclictest131rcu_preempt10:55:022
4088866997325,42cyclictest275-21dbus-daemon09:20:012
4088866997027,39cyclictest30-21ksoftirqd/212:18:342
4088862996931,32cyclictest4090632-21munin-plugin-st07:45:001
4088862996928,9cyclictest4099836-21run-parts09:20:011
4088866996725,35cyclictest4124089-21sh10:11:142
4088862996724,8cyclictest23-21ksoftirqd/111:55:011
4088866996627,34cyclictest30-21ksoftirqd/212:38:042
4088866996627,34cyclictest30-21ksoftirqd/212:38:042
4088857996630,18cyclictest4177652-21ssh12:05:010
4088857996630,18cyclictest4177652-21ssh12:05:010
4088866996531,29cyclictest1-21systemd09:30:012
4088862996524,4cyclictest4143142-21sh10:51:311
4088866996321,5cyclictest4139766-21sh10:44:462
4088866996315,8cyclictest120-21kswapd010:10:002
4088857996359,2cyclictest0-21swapper/010:50:010
4088857996357,2cyclictest0-21swapper/012:15:020
4088857996357,2cyclictest0-21swapper/012:15:020
4088866996226,32cyclictest4160018-21sh11:27:252
4088866996225,8cyclictest30-21ksoftirqd/208:01:332
408886699621,21cyclictest4158775-21latency_hist11:25:012
4088862996225,11cyclictest23-21ksoftirqd/109:41:191
4088866996127,31cyclictest30-21ksoftirqd/207:40:022
4088866996125,32cyclictest4129913-21sh10:23:512
4088866996123,10cyclictest30-21ksoftirqd/209:49:152
4088866996121,6cyclictest30-21ksoftirqd/210:02:352
4088866996119,7cyclictest4092901-21date08:25:012
4088866996115,8cyclictest3467602-21systemd-journal11:11:332
4088866996022,6cyclictest4116351-21cron09:55:002
4088866996022,6cyclictest4116351-21cron09:54:592
4088862996024,7cyclictest23-21ksoftirqd/112:02:091
4088862996024,7cyclictest23-21ksoftirqd/112:02:091
4088866995928,28cyclictest30-21ksoftirqd/208:38:332
4088866995925,30cyclictest4174679-21sh11:58:472
4088862995927,4cyclictest4095039-21diskmemload10:15:071
4088862995922,10cyclictest23-21ksoftirqd/109:11:081
4088862995919,8cyclictest23-21ksoftirqd/111:10:021
4088857995922,20cyclictest4121054-21ssh10:05:010
4088866995825,29cyclictest4172431-21sh11:53:522
408886699582,18cyclictest4106892-21ssh09:35:022
4088866995815,8cyclictest4165966-21grep11:40:032
4088857995837,17cyclictest16950irq/224-800000012:40:000
4088857995837,17cyclictest16950irq/224-800000012:40:000
4088857995829,14cyclictest4135025-21ssh10:34:450
4088857995828,14cyclictest4108680-21ssh09:38:400
4088857995828,14cyclictest4104090-21ssh09:29:010
4088857995826,16cyclictest4123342-21ssh10:09:580
4088871995740,12cyclictest275-21dbus-daemon08:20:003
4088866995726,28cyclictest30-21ksoftirqd/209:21:022
4088866995726,28cyclictest30-21ksoftirqd/209:21:022
4088862995725,8cyclictest23-21ksoftirqd/109:33:371
4088862995724,6cyclictest309-21in:imuxsock10:00:001
4088862995724,5cyclictest4183991-21sh12:18:221
4088857995727,14cyclictest4113495-21ssh09:48:530
4088866995628,25cyclictest30-21ksoftirqd/212:27:202
4088866995624,7cyclictest4093859-21latency_hist08:45:022
4088866995623,8cyclictest4091914-21cron08:09:592
4088866995618,4cyclictest4155480-21sh11:17:552
408886699561,19cyclictest4094804-21cat09:05:002
4088862995622,12cyclictest23-21ksoftirqd/109:29:071
4088862995620,10cyclictest23-21ksoftirqd/110:38:211
4088857995631,10cyclictest4111466-21ssh09:44:380
4088857995627,25cyclictest4163467-21ssh11:35:000
408885799561,3cyclictest131rcu_preempt10:16:420
4088871995539,11cyclictest275-21dbus-daemon11:05:013
408886699558,17cyclictest4137637-21date10:40:022
4088866995520,7cyclictest30-21ksoftirqd/209:44:492
4088866995516,12cyclictest30-21ksoftirqd/212:08:072
4088862995519,10cyclictest23-21ksoftirqd/109:37:071
4088857995546,4cyclictest16950irq/224-800000012:20:010
4088857995528,23cyclictest4135883-21sh10:36:150
4088857995528,11cyclictest4129302-21ssh10:22:240
4088857995527,13cyclictest4163870-21ssh11:35:230
4088857995527,13cyclictest4115133-21ssh09:52:080
4088857995527,13cyclictest4115133-21ssh09:52:080
4088866995445,5cyclictest4093964-21kworker/2:210:34:522
4088866995423,4cyclictest4177853-21latency_hist12:05:022
4088866995423,4cyclictest4177853-21latency_hist12:05:022
4088862995418,10cyclictest1-21systemd11:20:021
4088862995417,12cyclictest275-21dbus-daemon12:30:011
4088857995432,16cyclictest4099830-21latency_hist09:20:010
4088857995432,15cyclictest275-21dbus-daemon08:30:010
4088857995426,12cyclictest4154369-21ssh11:15:210
4088866995344,6cyclictest4081737-21kworker/2:109:36:412
4088866995323,4cyclictest4097419-21latency_hist09:15:022
4088866995320,9cyclictest30-21ksoftirqd/212:32:412
4088866995319,8cyclictest30-21ksoftirqd/210:28:042
4088866995319,5cyclictest4187100-21latency_hist12:25:022
4088866995319,4cyclictest4180895-21sh12:11:382
4088866995319,4cyclictest4180895-21sh12:11:382
4088866995315,9cyclictest30-21ksoftirqd/211:41:442
4088862995323,8cyclictest23-21ksoftirqd/112:33:121
408885799534,20cyclictest131rcu_preempt07:55:010
4088857995335,14cyclictest1-21systemd07:35:010
4088857995328,10cyclictest4157451-21ssh11:22:000
4088857995326,11cyclictest4170837-21ssh11:50:130
4088857995326,11cyclictest4138484-21ssh10:41:440
4088857995322,15cyclictest4117395-21ssh09:56:550
4088871995234,15cyclictest3467602-21systemd-journal10:20:013
4088866995224,24cyclictest4151534-21sh11:09:482
4088866995221,8cyclictest30-21ksoftirqd/211:05:022
4088866995220,8cyclictest4089837-21cron07:30:022
4088862995223,5cyclictest4157625-21sh11:22:231
408886299521,24cyclictest4095039-21diskmemload11:15:021
4088871995133,14cyclictest37-21ksoftirqd/311:50:003
4088866995118,7cyclictest30-21ksoftirqd/211:46:492
4088857995147,2cyclictest0-21swapper/008:40:010
4088857995134,9cyclictest4089304-21cron07:19:590
4088857995124,23cyclictest12-21ksoftirqd/012:31:340
408885799511,3cyclictest131rcu_preempt12:24:030
4088871995028,18cyclictest275-21dbus-daemon09:05:013
4088866995033,13cyclictest3467602-21systemd-journal07:49:172
4088866995016,12cyclictest3467602-21systemd-journal08:00:012
4088862995025,8cyclictest23-21ksoftirqd/110:48:021
4088857995025,10cyclictest4105893-21ssh09:32:480
4088857995023,24cyclictest4144312-21ssh10:54:220
408885799502,31cyclictest3467602-21systemd-journal12:06:200
408885799501,3cyclictest131rcu_preempt09:13:400
4088871994933,11cyclictest275-21dbus-daemon07:45:013
4088871994926,17cyclictest275-21dbus-daemon11:30:013
4088871994923,6cyclictest37-21ksoftirqd/309:45:023
4088866994933,13cyclictest4092441-21systemctl08:20:002
4088866994921,8cyclictest30-21ksoftirqd/210:59:272
4088866994919,5cyclictest30-21ksoftirqd/210:45:162
4088862994932,12cyclictest3467602-21systemd-journal10:05:001
4088862994923,6cyclictest23-21ksoftirqd/112:00:021
4088857994936,7cyclictest12-21ksoftirqd/008:10:020
4088857994912,33cyclictest12-21ksoftirqd/011:47:330
4088871994834,9cyclictest3467602-21systemd-journal08:40:003
4088871994833,11cyclictest275-21dbus-daemon12:15:013
4088871994833,11cyclictest275-21dbus-daemon12:15:013
4088871994831,7cyclictest309-21in:imuxsock09:25:033
4088871994828,10cyclictest309-21in:imuxsock07:25:013
4088871994828,10cyclictest309-21in:imuxsock07:25:003
408887199482,41cyclictest275-21dbus-daemon10:00:013
4088866994842,4cyclictest4081737-21kworker/2:108:50:002
4088866994819,4cyclictest4127928-21sh10:19:452
408886299483,6cyclictest131rcu_preempt10:10:331
4088857994839,6cyclictest16950irq/224-800000008:20:000
408885799483,14cyclictest4154087-21date11:15:010
4088857994830,12cyclictest3467602-21systemd-journal07:15:010
4088857994826,7cyclictest4173685-21ssh11:56:180
4088857994824,9cyclictest4150743-21ssh11:07:560
408885799482,40cyclictest4125716-21munin-run10:15:000
4088871994734,10cyclictest4163239-21sshd11:34:333
408887199471,31cyclictest4095017-21kworker/u8:209:15:023
4088866994739,5cyclictest4130468-21kworker/2:011:33:312
4088862994730,13cyclictest275-21dbus-daemon08:40:011
4088862994712,6cyclictest131rcu_preempt10:40:101
4088857994725,7cyclictest4167978-21ssh11:44:320
4088871994630,12cyclictest275-21dbus-daemon12:10:003
4088871994625,16cyclictest3467602-21systemd-journal11:20:013
4088866994638,5cyclictest4081737-21kworker/2:107:45:002
4088866994624,5cyclictest4117094-21sh09:56:182
408886299469,6cyclictest131rcu_preempt10:57:431
4088862994631,12cyclictest4090325-21sshd07:37:331
4088857994629,13cyclictest4159298-21ssh11:25:430
4088857994629,12cyclictest3467602-21systemd-journal10:25:020
408887199457,21cyclictest3467602-21systemd-journal09:20:013
4088871994530,7cyclictest310-21in:imklog12:37:083
4088871994530,7cyclictest310-21in:imklog12:37:083
4088871994528,13cyclictest1-21systemd08:25:013
4088871994524,5cyclictest37-21ksoftirqd/311:22:583
4088871994523,16cyclictest275-21dbus-daemon07:55:003
4088871994518,20cyclictest275-21dbus-daemon07:20:013
4088862994510,5cyclictest131rcu_preempt11:47:131
4088857994538,4cyclictest16950irq/224-800000011:05:010
4088857994538,4cyclictest16950irq/224-800000008:50:010
4088857994529,12cyclictest4101486-21ssh09:23:250
4088857994529,12cyclictest4101486-21ssh09:23:250
408885799452,38cyclictest4145135-21ssh10:55:480
408887199446,29cyclictest4135154-21grep10:35:003
4088871994422,17cyclictest275-21dbus-daemon09:35:013
408886299448,24cyclictest4135339-21ssh10:35:021
408886299444,4cyclictest131rcu_preempt12:10:321
408886299444,4cyclictest131rcu_preempt12:10:321
408886299441,6cyclictest0-21swapper/109:50:351
408886299441,6cyclictest0-21swapper/109:50:351
4088857994432,9cyclictest4088861-21latency_hist07:10:020
4088857994429,12cyclictest4187463-21ssh12:25:190
4088857994423,16cyclictest3467718-21lldpd08:05:010
408887199438,9cyclictest37-21ksoftirqd/310:45:013
408887199436,10cyclictest37-21ksoftirqd/311:10:013
408887199433,3cyclictest131rcu_preempt11:57:143
4088871994329,10cyclictest37-21ksoftirqd/311:37:033
4088871994326,13cyclictest4145008-21sshd10:55:333
408886699431,9cyclictest4090016-21latency_hist07:30:022
408886299439,6cyclictest131rcu_preempt11:40:511
408886299439,6cyclictest131rcu_preempt10:28:531
408886299437,6cyclictest131rcu_preempt09:20:291
408886299437,6cyclictest131rcu_preempt09:20:291
4088862994324,12cyclictest275-21dbus-daemon10:10:011
408886299431,4cyclictest131rcu_preempt11:25:271
408885799433,7cyclictest3467602-21systemd-journal08:00:000
4088857994328,11cyclictest1-21systemd07:44:590
4088857994325,12cyclictest3467602-21systemd-journal07:40:010
4088857994325,12cyclictest12-21ksoftirqd/007:25:010
4088857994325,12cyclictest12-21ksoftirqd/007:25:010
4088871994230,6cyclictest309-21in:imuxsock08:30:003
4088871994228,10cyclictest4094506-21cron09:00:013
4088871994227,11cyclictest3467602-21systemd-journal10:08:023
408887199422,36cyclictest37-21ksoftirqd/312:34:473
408886299427,6cyclictest23-21ksoftirqd/110:24:481
408886299427,5cyclictest131rcu_preempt12:24:151
4088862994231,7cyclictest3467602-21systemd-journal07:51:331
4088862994230,9cyclictest23-21ksoftirqd/112:39:581
4088862994230,9cyclictest23-21ksoftirqd/112:39:581
408885799427,31cyclictest4095037-21sh09:10:000
4088857994232,6cyclictest19350irq/34-mmc108:10:020
408887199414,3cyclictest131rcu_preempt12:01:133
408887199414,3cyclictest131rcu_preempt12:01:133
408887199414,25cyclictest4116366-21grep09:55:003
408887199414,25cyclictest4116366-21grep09:55:003
4088871994129,9cyclictest37-21ksoftirqd/309:50:013
4088871994128,9cyclictest0-21swapper/309:25:023
4088871994128,9cyclictest0-21swapper/309:25:023
4088871994126,12cyclictest3467602-21systemd-journal10:25:003
4088871994125,12cyclictest4121172-21cat10:05:023
408886299417,6cyclictest131rcu_preempt11:32:301
4088862994132,5cyclictest4148045-21sh11:02:031
4088862994127,10cyclictest275-21dbus-daemon11:40:011
408886299411,5cyclictest131rcu_preempt09:46:291
4088857994134,4cyclictest16950irq/224-800000009:05:000
4088857994129,8cyclictest1-21systemd07:29:590
4088857994126,11cyclictest1-21systemd08:35:010
4088871994034,3cyclictest371ksoftirqd/309:40:003
4088871994031,6cyclictest4172890-21cron11:55:003
4088871994025,11cyclictest37-21ksoftirqd/310:40:003
4088871994024,12cyclictest275-21dbus-daemon11:15:013
4088871994013,20cyclictest275-21dbus-daemon07:50:013
4088866994026,11cyclictest30-21ksoftirqd/207:55:012
408886299404,5cyclictest131rcu_preempt07:47:101
408886299403,5cyclictest4095237-21ssh09:10:031
4088857994030,6cyclictest4092708-21latency_hist08:25:020
408885799402,9cyclictest4094507-21latency_hist09:00:030
4088871993924,10cyclictest0-21swapper/310:55:013
4088871993923,13cyclictest3467602-21systemd-journal11:43:413
4088871993915,17cyclictest275-21dbus-daemon12:25:013
408887199391,4cyclictest131rcu_preempt12:19:343
4088862993926,10cyclictest3467602-21systemd-journal08:04:201
408887199385,7cyclictest37-21ksoftirqd/307:35:023
408887199385,24cyclictest4142289-21modprobe10:50:013
408887199383,3cyclictest131rcu_preempt12:26:353
4088871993828,8cyclictest37-21ksoftirqd/308:35:023
408887199382,27cyclictest4091396-21cut07:59:593
4088871993817,16cyclictest275-21dbus-daemon08:10:013
4088866993829,5cyclictest4081737-21kworker/2:109:10:022
4088866993826,8cyclictest4089581-21sshd07:24:342
4088866993826,8cyclictest4089581-21sshd07:24:342
408886699382,5cyclictest131rcu_preempt08:35:012
4088866993824,10cyclictest275-21dbus-daemon07:15:012
4088862993827,8cyclictest23-21ksoftirqd/112:05:181
4088862993826,8cyclictest3467602-21systemd-journal07:25:011
4088862993826,8cyclictest3467602-21systemd-journal07:25:011
408886299382,5cyclictest131rcu_preempt08:33:271
4088862993821,12cyclictest275-21dbus-daemon08:55:011
4088871993725,9cyclictest37-21ksoftirqd/310:27:553
4088871993724,11cyclictest37-21ksoftirqd/307:40:013
408886699374,3cyclictest131rcu_preempt08:15:012
4088866993723,11cyclictest30-21ksoftirqd/209:00:012
408886299371,5cyclictest131rcu_preempt08:24:241
4088857993730,4cyclictest16950irq/224-800000007:50:010
4088871993626,7cyclictest37-21ksoftirqd/308:12:333
408887199362,3cyclictest131rcu_preempt10:11:123
408886699361,3cyclictest131rcu_preempt07:20:002
408886699361,31cyclictest48-21kcompactd008:50:122
4088862993625,9cyclictest23-21ksoftirqd/108:10:011
4088862993624,9cyclictest23-21ksoftirqd/108:58:421
4088862993624,8cyclictest23-21ksoftirqd/108:00:021
4088862993623,10cyclictest3467602-21systemd-journal08:16:321
4088862993623,10cyclictest23-21ksoftirqd/107:29:131
408886299361,3cyclictest131rcu_preempt08:45:001
408885799362,11cyclictest4093720-21sshd08:42:330
4088871993525,7cyclictest37-21ksoftirqd/307:30:023
4088871993521,10cyclictest1-21systemd08:45:013
408887199351,29cyclictest275-21dbus-daemon07:15:013
408886699354,3cyclictest131rcu_preempt08:30:022
408887199342,24cyclictest4095049-21ssh09:09:593
408886699342,4cyclictest131rcu_preempt07:10:022
4088862993425,7cyclictest23-21ksoftirqd/108:50:021
4088862993421,10cyclictest23-21ksoftirqd/108:26:511
4088862993421,10cyclictest23-21ksoftirqd/108:11:501
4088862993418,12cyclictest4088996-21cut07:10:031
408887199336,5cyclictest131rcu_preempt08:50:013
4088871993325,5cyclictest4094215-21kworker/u8:308:55:003
4088862993323,8cyclictest23-21ksoftirqd/109:05:011
4088862993322,9cyclictest23-21ksoftirqd/107:16:331
4088862993320,10cyclictest23-21ksoftirqd/107:31:591
408885799332,28cyclictest3467602-21systemd-journal08:54:190
4088871993222,4cyclictest37-21ksoftirqd/308:05:033
4088871993221,5cyclictest37-21ksoftirqd/307:10:013
4088862993220,9cyclictest23-21ksoftirqd/107:14:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional