You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-12 - 17:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Tue May 12, 2026 12:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4082638998072,5cyclictest4082014-21kworker/2:208:50:022
4082631997825,36cyclictest4093756-21grep09:20:010
408263199735,54cyclictest4170945-21run-parts12:00:000
408263199722,58cyclictest4098657-21ssh09:30:010
408263899672,16cyclictest4082845-21munin-run07:15:012
408263199661,2cyclictest0-21swapper/007:10:010
4082638996530,31cyclictest4156226-21sh11:29:252
4082638996426,34cyclictest4187406-21sh12:34:052
408263899642,16cyclictest4132861-21ssh10:40:302
408263899642,15cyclictest4087058-21/usr/sbin/munin08:35:022
4082638996421,39cyclictest30-21ksoftirqd/209:17:362
408263899641,12cyclictest4115357-21ssh10:04:452
408263899632,14cyclictest4085731-21modprobe08:09:512
408263899632,12cyclictest4169439-21sh11:56:362
408263899631,13cyclictest4188127-21ssh12:35:132
408263899623,23cyclictest4101169-21latency_hist09:35:032
408263899623,23cyclictest4101169-21latency_hist09:35:022
408263899622,8cyclictest48-21kcompactd011:10:352
408263899622,15cyclictest364-21snmpd12:10:432
408263899621,9cyclictest4157408-21sshd11:31:362
408263899621,15cyclictest4181524-21sshd12:21:372
408263899621,11cyclictest4090702-21ssh09:13:372
4082638996133,23cyclictest4112560-21sh09:58:592
4082638996132,25cyclictest4166420-21sh11:50:132
4082638996131,25cyclictest4096914-21sh09:26:082
408263899613,10cyclictest4161312-21ssh11:40:002
408263899612,7cyclictest4085256-21munin-run08:00:012
4082638996126,27cyclictest3467602-21systemd-journal07:45:002
4082638996122,34cyclictest4088865-21diskmemload10:58:282
408263899611,16cyclictest4085973-21sshd08:13:362
408263199612,32cyclictest131rcu_preempt10:23:580
408263899602,12cyclictest4163983-21ssh11:45:072
408263899602,10cyclictest4086806-21munin-run08:30:002
408263899601,14cyclictest48-21kcompactd010:20:262
408263899601,13cyclictest4179562-21ssh12:17:462
408263899601,12cyclictest4086221-21cron08:17:002
408263899601,11cyclictest4171549-21sh12:00:542
4082638996010,3cyclictest4102313-21ssh09:37:372
4082631996029,14cyclictest4183624-21ssh12:25:490
4082631996029,14cyclictest4095458-21ssh09:23:280
408263899593,16cyclictest4088891-21ssh09:10:012
408263899593,13cyclictest4139604-21munin-run10:55:002
4082638995930,26cyclictest30-21ksoftirqd/207:17:372
408263899592,8cyclictest4087563-21cron08:44:592
408263899592,16cyclictest4083944-21grep07:35:012
408263899592,13cyclictest4087502-21latency_hist08:40:022
408263899591,15cyclictest4136320-21ssh10:48:032
408263899591,15cyclictest4136320-21ssh10:48:022
408263899591,13cyclictest364-21snmpd11:20:202
408263899591,13cyclictest364-21snmpd10:29:372
408263899591,13cyclictest364-21snmpd10:29:362
408263399597,28cyclictest4120973-21kworker/u8:410:45:031
408263399597,28cyclictest4120973-21kworker/u8:410:45:021
4082638995848,6cyclictest4082014-21kworker/2:209:25:002
4082638995845,6cyclictest4139617-21kworker/2:312:30:002
408263899583,8cyclictest4122662-21cron10:20:002
408263899583,8cyclictest4122662-21cron10:20:002
408263899583,6cyclictest4120280-21ssh10:15:022
408263899582,9cyclictest4084139-21sshd07:35:362
4082638995822,32cyclictest4145324-21sh11:06:302
408263899582,13cyclictest4107511-21ssh09:48:272
408263899582,13cyclictest4088083-21munin-run08:55:012
408263899582,13cyclictest4088083-21munin-run08:55:002
408263899582,12cyclictest4088302-21modprobe08:56:362
408263899581,16cyclictest4131218-21ssh10:37:282
408263899581,16cyclictest4131218-21ssh10:37:272
408263899581,10cyclictest4084942-21sshd07:54:352
408263199589,30cyclictest131rcu_preempt11:08:050
4082631995816,37cyclictest16850irq/134-800000010:05:030
408264399571,6cyclictest131rcu_preempt12:39:593
408263899572,18cyclictest4108532-21ssh09:50:132
408263899572,12cyclictest364-21snmpd12:05:442
4082638995714,39cyclictest3467602-21systemd-journal10:32:012
4082643995635,16cyclictest275-21dbus-daemon07:20:003
408263899562,14cyclictest4162356-21sh11:41:482
408263899562,14cyclictest4162356-21sh11:41:472
408263899561,14cyclictest4151243-21ssh11:19:032
408263899561,14cyclictest4144601-21latency_hist11:05:012
408263899561,11cyclictest4105127-21ssh09:43:272
4082633995622,17cyclictest23-21ksoftirqd/112:15:031
408263199567,44cyclictest1-21systemd10:50:020
408263199567,44cyclictest1-21systemd10:50:010
4082631995627,14cyclictest4168598-21ssh11:55:010
4082631995626,14cyclictest4174853-21ssh12:08:030
408263899552,17cyclictest4116718-21ssh10:07:262
408263399555,31cyclictest4162601-21kworker/u8:011:55:021
4082631995528,12cyclictest4162692-21ssh11:42:410
4082631995528,12cyclictest4162692-21ssh11:42:400
4082631995527,13cyclictest4190196-21ssh12:39:580
4082631995526,14cyclictest4176239-21ssh12:10:400
408264399548,7cyclictest131rcu_preempt11:35:023
408264399546,33cyclictest4154162-21ssh11:25:013
4082643995431,17cyclictest4085201-21cron08:00:003
408263899542,9cyclictest4084730-21latency_hist07:50:012
4082633995418,6cyclictest131rcu_preempt12:00:001
4082633995413,6cyclictest131rcu_preempt10:30:011
4082633995413,6cyclictest131rcu_preempt10:30:001
408263199549,29cyclictest131rcu_preempt10:52:110
4082631995441,3cyclictest0-21swapper/008:45:030
4082631995427,12cyclictest4160506-21ssh11:38:100
4082631995426,13cyclictest4141935-21ssh10:59:480
4082631995419,31cyclictest4103596-21latency_hist09:40:020
408263899532,12cyclictest4086553-21munin-run08:25:012
408263899531,11cyclictest364-21snmpd07:23:512
4082631995324,13cyclictest4114884-21ssh10:03:460
4082631995318,31cyclictest4084975-21kworker/0:008:29:590
408263899523,12cyclictest3467602-21systemd-journal08:05:012
408263399527,6cyclictest131rcu_preempt09:57:441
4082633995235,10cyclictest23-21ksoftirqd/107:45:081
4082633995233,14cyclictest3467602-21systemd-journal07:35:001
4082633995215,8cyclictest23-21ksoftirqd/110:45:021
408263399521,36cyclictest4171401-21kworker/u8:112:05:031
4082633995210,6cyclictest131rcu_preempt12:35:021
408263199523,43cyclictest131rcu_preempt10:31:240
4082631995228,19cyclictest131rcu_preempt07:40:010
4082631995226,11cyclictest4155272-21ssh11:27:160
4082631995222,14cyclictest4173267-21ssh12:04:520
4082631995214,6cyclictest16850irq/134-800000010:15:010
408264399519,6cyclictest131rcu_preempt12:25:003
408264399519,6cyclictest131rcu_preempt10:24:453
408264399518,5cyclictest131rcu_preempt10:34:533
4082643995134,14cyclictest275-21dbus-daemon07:50:003
4082643995111,5cyclictest131rcu_preempt09:29:493
4082638995117,30cyclictest4088629-21latency_hist09:05:022
4082638995117,30cyclictest4088629-21latency_hist09:05:012
408263399519,6cyclictest131rcu_preempt11:27:351
408263399519,5cyclictest131rcu_preempt11:06:231
4082633995134,13cyclictest1-21systemd10:20:011
4082633995134,13cyclictest1-21systemd10:20:011
408263199519,23cyclictest131rcu_preempt11:34:440
4082631995139,9cyclictest12-21ksoftirqd/011:15:010
4082631995124,12cyclictest4104801-21ssh09:42:460
4082631995124,12cyclictest4099921-21ssh09:32:360
4082631995124,12cyclictest4099921-21ssh09:32:350
4082643995038,9cyclictest37-21ksoftirqd/307:45:013
408264399502,9cyclictest37-21ksoftirqd/310:07:233
4082643995014,19cyclictest37-21ksoftirqd/312:15:023
408263399509,6cyclictest131rcu_preempt12:30:021
408263399507,7cyclictest23-21ksoftirqd/111:32:341
4082633995011,9cyclictest4183089-21grep12:25:011
4082633995010,6cyclictest131rcu_preempt09:20:001
4082633995010,5cyclictest131rcu_preempt11:43:471
4082633995010,5cyclictest131rcu_preempt11:43:461
4082633995010,5cyclictest131rcu_preempt10:03:021
408263199509,5cyclictest131rcu_preempt11:03:360
4082631995015,30cyclictest16850irq/134-800000009:45:190
408263199501,44cyclictest131rcu_preempt08:00:010
4082631995013,32cyclictest16850irq/134-800000012:17:340
4082631995013,32cyclictest16850irq/134-800000009:11:360
4082631995010,32cyclictest12-21ksoftirqd/011:20:570
408264399498,5cyclictest131rcu_preempt10:49:033
408264399498,5cyclictest131rcu_preempt10:49:033
408264399496,6cyclictest131rcu_preempt12:08:523
408264399492,5cyclictest131rcu_preempt11:45:013
408264399492,5cyclictest131rcu_preempt11:45:003
4082643994921,23cyclictest361rcuc/308:10:013
4082643994913,6cyclictest131rcu_preempt12:00:263
408263399498,4cyclictest131rcu_preempt09:50:381
4082633994915,6cyclictest131rcu_preempt10:53:521
4082633994910,6cyclictest131rcu_preempt09:32:061
4082633994910,6cyclictest131rcu_preempt09:32:061
4082631994922,12cyclictest4180856-21ssh12:20:050
408264399489,5cyclictest131rcu_preempt12:35:003
408264399489,5cyclictest131rcu_preempt10:58:143
408264399484,6cyclictest131rcu_preempt09:23:553
4082643994833,11cyclictest275-21dbus-daemon09:35:023
4082643994833,11cyclictest275-21dbus-daemon09:35:013
4082643994813,6cyclictest131rcu_preempt10:11:063
408263399489,3cyclictest131rcu_preempt10:37:451
408263399489,3cyclictest131rcu_preempt10:37:441
408263399487,6cyclictest131rcu_preempt11:25:011
408263399487,6cyclictest131rcu_preempt09:41:181
408263399487,4cyclictest131rcu_preempt09:29:161
408263399485,7cyclictest23-21ksoftirqd/111:00:271
408263399484,8cyclictest23-21ksoftirqd/111:16:581
408263399483,8cyclictest23-21ksoftirqd/110:55:171
4082633994811,5cyclictest131rcu_preempt10:34:591
408263399481,14cyclictest0-21swapper/110:11:001
408263199484,5cyclictest131rcu_preempt07:17:370
408263199483,8cyclictest131rcu_preempt11:20:010
4082631994832,13cyclictest4122661-21cron10:20:000
4082631994832,13cyclictest4122661-21cron10:20:000
4082631994825,15cyclictest4107830-21kworker/u8:409:55:550
4082631994820,13cyclictest4108851-21ssh09:50:540
4082631994811,33cyclictest16850irq/134-800000012:32:360
4082631994811,32cyclictest16850irq/134-800000010:37:290
4082631994811,32cyclictest16850irq/134-800000010:37:280
408264399477,5cyclictest131rcu_preempt10:43:573
408264399474,5cyclictest131rcu_preempt10:04:133
408263899472,8cyclictest3467602-21systemd-journal07:30:012
408263399478,4cyclictest131rcu_preempt11:10:041
408263399476,5cyclictest131rcu_preempt12:14:541
408263399471,7cyclictest48-21kcompactd012:05:101
408263399471,4cyclictest131rcu_preempt09:35:351
4082633994712,6cyclictest131rcu_preempt10:21:141
408263199471,3cyclictest131rcu_preempt10:44:320
4082631994712,31cyclictest16850irq/134-800000011:48:310
408264399465,5cyclictest131rcu_preempt11:07:573
4082643994630,8cyclictest37-21ksoftirqd/308:18:223
4082643994629,12cyclictest275-21dbus-daemon11:55:013
4082643994627,14cyclictest275-21dbus-daemon10:55:013
408264399462,5cyclictest131rcu_preempt09:58:433
408264399462,4cyclictest131rcu_preempt09:37:553
4082643994617,10cyclictest131rcu_preempt10:37:363
4082643994617,10cyclictest131rcu_preempt10:37:363
4082643994612,6cyclictest131rcu_preempt09:45:373
4082643994611,5cyclictest131rcu_preempt11:04:093
408263399469,5cyclictest131rcu_preempt09:11:361
408263399466,8cyclictest23-21ksoftirqd/107:30:001
408263399465,8cyclictest23-21ksoftirqd/111:39:581
4082633994636,7cyclictest23-21ksoftirqd/108:40:011
408263399463,5cyclictest131rcu_preempt09:45:021
408263399462,7cyclictest23-21ksoftirqd/112:36:561
4082633994610,5cyclictest131rcu_preempt09:25:021
408264399459,6cyclictest131rcu_preempt12:25:043
408264399454,6cyclictest131rcu_preempt09:14:053
408263399459,6cyclictest131rcu_preempt11:50:021
408263399454,5cyclictest131rcu_preempt10:05:401
408263399453,6cyclictest131rcu_preempt08:29:591
408263399452,4cyclictest131rcu_preempt08:33:361
4082633994511,6cyclictest131rcu_preempt08:52:361
4082633994511,6cyclictest131rcu_preempt08:52:351
4082631994515,15cyclictest12-21ksoftirqd/010:29:240
4082631994515,15cyclictest12-21ksoftirqd/010:29:240
408264399444,7cyclictest37-21ksoftirqd/309:52:053
408264399443,4cyclictest131rcu_preempt11:12:113
4082643994415,24cyclictest361rcuc/309:45:023
408264399441,4cyclictest131rcu_preempt11:36:453
4082643994410,5cyclictest131rcu_preempt11:45:583
4082643994410,5cyclictest131rcu_preempt11:16:283
408263399446,6cyclictest131rcu_preempt08:45:011
4082633994411,5cyclictest131rcu_preempt08:25:031
4082631994432,9cyclictest4087301-21cron08:40:000
4082631994424,13cyclictest3467602-21systemd-journal07:55:010
4082631994417,7cyclictest131rcu_preempt08:35:010
408264399439,5cyclictest131rcu_preempt11:59:513
408264399438,5cyclictest131rcu_preempt09:17:543
408264399436,5cyclictest131rcu_preempt12:15:583
408264399435,4cyclictest131rcu_preempt07:15:023
408264399431,8cyclictest37-21ksoftirqd/307:34:593
4082643994310,6cyclictest131rcu_preempt10:26:323
4082643994310,6cyclictest131rcu_preempt10:26:313
408263399438,3cyclictest131rcu_preempt07:59:501
4082633994332,8cyclictest23-21ksoftirqd/108:02:361
4082633994310,5cyclictest131rcu_preempt09:05:011
4082633994310,5cyclictest131rcu_preempt09:05:001
4082631994324,15cyclictest4088828-21sshd09:05:370
408264399426,5cyclictest131rcu_preempt09:10:013
408264399424,5cyclictest131rcu_preempt11:25:593
408264399421,4cyclictest131rcu_preempt10:15:233
408264399421,4cyclictest131rcu_preempt10:15:223
408263899423,8cyclictest3467602-21systemd-journal07:10:032
408263399428,6cyclictest131rcu_preempt07:15:001
408263399424,10cyclictest23-21ksoftirqd/108:50:011
408264399419,5cyclictest131rcu_preempt08:55:023
408264399419,5cyclictest131rcu_preempt08:55:013
4082643994113,6cyclictest131rcu_preempt08:30:013
408263399414,5cyclictest131rcu_preempt08:15:021
4082633994133,5cyclictest23-21ksoftirqd/107:55:001
408263199415,7cyclictest12-21ksoftirqd/007:27:360
4082631994134,4cyclictest16950irq/224-800000009:00:010
4082631994127,11cyclictest12-21ksoftirqd/007:45:020
4082643994026,11cyclictest4088617-21sh09:05:003
4082643994026,11cyclictest4088617-21sh09:05:003
4082643994012,6cyclictest131rcu_preempt08:11:013
4082633994027,10cyclictest4083053-21sshd07:16:371
408263199403,6cyclictest131rcu_preempt08:10:020
4082631994025,11cyclictest309-21in:imuxsock08:25:000
408264399394,5cyclictest131rcu_preempt08:40:023
4082643993926,10cyclictest4088355-21basename08:59:593
408264399392,5cyclictest131rcu_preempt07:10:013
408263399398,4cyclictest131rcu_preempt08:10:001
4082633993924,10cyclictest3467602-21systemd-journal08:59:591
4082633993910,5cyclictest131rcu_preempt07:39:381
408263199396,29cyclictest4082796-21kworker/u8:207:45:510
4082631993930,6cyclictest19350irq/34-mmc108:10:050
4082631993925,10cyclictest4082814-21sshd07:14:360
408264399384,2cyclictest0-21swapper/308:45:003
408264399383,6cyclictest131rcu_preempt08:34:553
408264399383,5cyclictest131rcu_preempt08:46:493
4082643993827,7cyclictest37-21ksoftirqd/307:25:003
408263399386,7cyclictest23-21ksoftirqd/108:16:591
4082633993825,10cyclictest23-21ksoftirqd/107:25:021
408263199385,5cyclictest4084975-21kworker/0:008:41:020
4082631993827,8cyclictest12-21ksoftirqd/008:55:020
408263199382,5cyclictest1-21systemd09:05:000
408263199382,5cyclictest1-21systemd09:05:000
408263199382,13cyclictest4086301-21unin-run08:20:010
408264399376,5cyclictest131rcu_preempt08:01:363
4082643993725,8cyclictest4084945-21sshd07:54:483
408263399378,5cyclictest131rcu_preempt09:05:591
408263399376,6cyclictest23-21ksoftirqd/107:45:021
408263199376,4cyclictest4084975-21kworker/0:008:01:550
4082631993724,9cyclictest12-21ksoftirqd/007:24:360
4082643993519,11cyclictest1-21systemd08:25:023
408264399351,5cyclictest131rcu_preempt07:40:003
4082631993523,9cyclictest12-21ksoftirqd/008:55:000
408264399344,7cyclictest131rcu_preempt07:29:073
4082631993216,13cyclictest275-21dbus-daemon07:35:000
408263399295,15cyclictest4082636-21latency_hist07:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional