You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-10 - 12:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon Aug 10, 2026 00:45:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
34455049910799,6cyclictest12-21ksoftirqd/021:30:020
34455049910296,3cyclictest0-21swapper/000:00:030
3445504999483,7cyclictest3445506-21latency_hist19:10:010
344550499948,21cyclictest131rcu_preempt21:35:030
3445504998577,6cyclictest12-21ksoftirqd/022:20:020
344550999781,5cyclictest131rcu_preempt21:40:021
3445518997363,8cyclictest37-21ksoftirqd/323:55:023
3445504997338,5cyclictest131rcu_preempt19:20:000
3445513997250,8cyclictest3456510-21kworker/2:123:38:012
3445509997130,11cyclictest23-21ksoftirqd/122:40:001
344550499713,64cyclictest3482325-21ssh22:15:030
344550499702,6cyclictest0-21swapper/022:05:010
3445504996959,8cyclictest12-21ksoftirqd/023:00:030
3445504996950,13cyclictest19350irq/34-mmc121:35:020
3445504996929,9cyclictest3518763-21ssh23:32:360
3445504996929,9cyclictest3518763-21ssh23:32:360
344550499676,5cyclictest131rcu_preempt22:00:010
3445504996631,15cyclictest3503474-21ssh23:00:020
3445504996620,14cyclictest275-21dbus-daemon22:55:010
3445504996610,5cyclictest131rcu_preempt23:50:010
344550999653,39cyclictest131rcu_preempt22:20:031
3445504996550,9cyclictest16950irq/224-800000020:40:000
344550499647,6cyclictest131rcu_preempt00:30:020
3445504996436,21cyclictest275-21dbus-daemon20:55:000
344550499634,5cyclictest131rcu_preempt22:31:140
344550499633,55cyclictest0-21swapper/022:45:000
3445509996249,9cyclictest23-21ksoftirqd/100:10:001
3445509996249,9cyclictest23-21ksoftirqd/100:10:001
3445504996211,31cyclictest131rcu_preempt23:52:500
344550499616,6cyclictest131rcu_preempt00:20:500
344550499604,9cyclictest12-21ksoftirqd/021:55:010
344550499604,29cyclictest3450914-21kworker/u8:221:00:040
344551399585,6cyclictest131rcu_preempt00:00:002
3445504995837,15cyclictest1-21systemd23:40:010
3445504995812,5cyclictest131rcu_preempt00:10:450
344551899573,5cyclictest131rcu_preempt00:20:003
344550499576,38cyclictest19350irq/34-mmc119:45:020
3445504995715,32cyclictest16850irq/134-800000022:49:140
344550499569,6cyclictest131rcu_preempt21:05:000
344550499567,10cyclictest12-21ksoftirqd/020:30:000
344550499565,10cyclictest12-21ksoftirqd/020:35:000
344550499565,10cyclictest12-21ksoftirqd/020:35:000
3445504995628,12cyclictest3487294-21ssh22:25:260
344551399553,15cyclictest3482364-21ssh22:15:042
344550999551,49cyclictest131rcu_preempt23:00:031
344550499557,12cyclictest3479826-21unin-run22:10:000
3445504995546,7cyclictest12-21ksoftirqd/023:07:230
344550499551,39cyclictest3547339-21ssh00:33:280
3445513995427,18cyclictest3444840-21kworker/2:021:58:582
3445504995443,7cyclictest151rcuc/021:45:010
3445504995427,11cyclictest3510681-21ssh23:15:100
344550499542,5cyclictest131rcu_preempt00:15:430
3445504995418,18cyclictest16850irq/134-800000021:16:320
344550499541,6cyclictest3485833-21modprobe22:22:330
3445504995412,11cyclictest12-21ksoftirqd/020:05:020
3445513995345,5cyclictest3444840-21kworker/2:021:27:592
3445513995344,6cyclictest3444840-21kworker/2:022:07:362
3445513995335,13cyclictest1-21systemd23:30:012
3445513995334,15cyclictest1-21systemd21:10:002
3445513995323,7cyclictest3456510-21kworker/2:121:36:352
3445509995324,7cyclictest3487934-21sh22:26:571
3445504995328,17cyclictest275-21dbus-daemon23:15:000
3445504995324,14cyclictest3458695-21ssh21:24:440
3445504995321,28cyclictest3470495-21cron21:50:010
344551899523,4cyclictest131rcu_preempt22:05:013
3445513995242,6cyclictest3444840-21kworker/2:022:31:522
3445513995241,6cyclictest3444840-21kworker/2:022:30:012
3445509995237,11cyclictest23-21ksoftirqd/100:00:011
3445509995235,12cyclictest275-21dbus-daemon23:45:021
344550499528,5cyclictest131rcu_preempt22:40:020
344550499523,7cyclictest12-21ksoftirqd/020:45:010
3445504995215,7cyclictest12-21ksoftirqd/020:15:010
344551899513,4cyclictest131rcu_preempt00:13:513
344551899513,3cyclictest131rcu_preempt22:19:573
344551899513,3cyclictest131rcu_preempt00:09:593
344551899513,3cyclictest131rcu_preempt00:09:583
3445513995143,5cyclictest3456510-21kworker/2:100:15:532
3445513995143,5cyclictest3444840-21kworker/2:023:20:412
3445513995143,5cyclictest3444840-21kworker/2:023:20:402
3445513995135,12cyclictest3467602-21systemd-journal21:00:002
3445513995135,12cyclictest275-21dbus-daemon20:40:002
344550499519,36cyclictest0-21swapper/000:10:030
344550499519,36cyclictest0-21swapper/000:10:020
344550499513,30cyclictest12-21ksoftirqd/023:27:360
3445504995124,11cyclictest3514798-21ssh23:24:180
3445504995124,11cyclictest3514798-21ssh23:24:180
344550499511,9cyclictest12-21ksoftirqd/000:39:260
344551899507,10cyclictest37-21ksoftirqd/319:20:003
344551899505,3cyclictest131rcu_preempt22:56:123
344551899504,3cyclictest131rcu_preempt23:50:303
344551899503,3cyclictest131rcu_preempt22:00:023
344551899502,3cyclictest131rcu_preempt21:32:433
344551899501,4cyclictest131rcu_preempt22:29:303
3445513995042,5cyclictest3456510-21kworker/2:123:31:392
3445513995042,5cyclictest3456510-21kworker/2:123:31:382
3445513995042,5cyclictest3456510-21kworker/2:123:18:312
3445513995042,5cyclictest3456510-21kworker/2:123:08:342
3445513995042,5cyclictest3456510-21kworker/2:100:06:032
3445513995042,5cyclictest3456510-21kworker/2:100:06:022
3445513995042,5cyclictest3444840-21kworker/2:021:43:162
3445509995036,11cyclictest3452233-21sshd21:10:341
344550499507,9cyclictest12-21ksoftirqd/021:10:000
344551899494,5cyclictest131rcu_preempt00:40:013
344551899494,4cyclictest131rcu_preempt21:12:323
344551899494,3cyclictest131rcu_preempt23:25:023
344551899494,3cyclictest131rcu_preempt23:25:013
344551899494,3cyclictest131rcu_preempt22:13:023
344551899494,3cyclictest131rcu_preempt20:35:013
344551899494,3cyclictest131rcu_preempt20:35:003
344551899494,3cyclictest131rcu_preempt00:32:303
344551899494,3cyclictest131rcu_preempt00:04:293
344551899492,3cyclictest131rcu_preempt22:54:053
3445513994941,5cyclictest3444840-21kworker/2:021:30:542
3445513994940,5cyclictest3456510-21kworker/2:123:53:552
3445513994934,11cyclictest275-21dbus-daemon20:35:002
3445513994934,11cyclictest275-21dbus-daemon20:35:002
3445513994933,8cyclictest3456510-21kworker/2:100:40:012
344551399492,6cyclictest3496573-21ssh22:45:062
3445513994923,10cyclictest3548010-21/usr/sbin/munin00:35:022
344550999491,4cyclictest131rcu_preempt23:50:011
3445504994936,8cyclictest3446744-21munin-plugin-st19:35:000
3445504994936,8cyclictest19350irq/34-mmc120:25:020
3445504994923,12cyclictest3533314-21ssh00:03:470
344550499492,17cyclictest12-21ksoftirqd/019:55:000
344551899484,4cyclictest131rcu_preempt22:21:343
344551899484,3cyclictest131rcu_preempt00:27:433
344551899483,4cyclictest131rcu_preempt23:30:513
344551899483,4cyclictest131rcu_preempt23:30:503
344551899483,3cyclictest131rcu_preempt21:39:163
344551899482,4cyclictest131rcu_preempt23:38:373
344551899482,3cyclictest131rcu_preempt23:19:193
344551899482,3cyclictest131rcu_preempt22:06:293
344551899482,3cyclictest131rcu_preempt21:21:363
344551899481,3cyclictest131rcu_preempt00:20:043
3445513994840,5cyclictest3456510-21kworker/2:123:02:432
3445513994840,5cyclictest3456510-21kworker/2:100:22:222
3445513994840,5cyclictest3444840-21kworker/2:022:35:242
3445513994840,5cyclictest3444840-21kworker/2:022:19:362
3445513994840,5cyclictest3444840-21kworker/2:022:00:282
3445513994840,5cyclictest3444840-21kworker/2:021:21:272
3445513994840,5cyclictest3444840-21kworker/2:021:20:032
3445513994840,5cyclictest3444840-21kworker/2:021:14:432
3445513994834,10cyclictest1-21systemd21:05:012
3445513994831,13cyclictest1-21systemd20:00:012
344551399482,7cyclictest3495762-21ssh22:43:442
3445513994811,7cyclictest131rcu_preempt22:50:492
344550999481,3cyclictest131rcu_preempt21:30:001
344550499482,41cyclictest3449087-21systemctl20:20:010
344551899475,3cyclictest131rcu_preempt21:45:573
344551899473,5cyclictest131rcu_preempt21:15:513
344551899473,4cyclictest131rcu_preempt22:43:093
344551899472,3cyclictest131rcu_preempt23:26:013
344551899472,3cyclictest131rcu_preempt22:49:533
344551899472,3cyclictest131rcu_preempt22:36:263
344551899472,3cyclictest131rcu_preempt21:28:573
344551899471,4cyclictest131rcu_preempt23:12:023
344551399473,3cyclictest131rcu_preempt23:41:492
344551399473,3cyclictest131rcu_preempt00:15:012
3445504994713,8cyclictest12-21ksoftirqd/021:11:040
3445504994711,12cyclictest12-21ksoftirqd/023:43:350
344551899464,3cyclictest131rcu_preempt23:03:113
344551899463,5cyclictest131rcu_preempt21:41:553
344551899463,4cyclictest131rcu_preempt21:51:223
344551899463,4cyclictest131rcu_preempt19:35:033
344551899463,3cyclictest131rcu_preempt23:42:023
344551899461,3cyclictest131rcu_preempt23:09:503
344551899461,3cyclictest131rcu_preempt22:30:313
3445513994638,5cyclictest3456510-21kworker/2:121:52:412
3445513994638,5cyclictest3456510-21kworker/2:100:28:322
3445513994638,5cyclictest3444840-21kworker/2:022:20:452
3445513994634,9cyclictest3467602-21systemd-journal19:25:012
344550999464,3cyclictest131rcu_preempt00:40:011
3445509994627,14cyclictest23-21ksoftirqd/100:15:001
344551899454,3cyclictest131rcu_preempt20:05:003
344551899453,9cyclictest0-21swapper/321:10:003
344551399456,5cyclictest131rcu_preempt23:12:542
344551399452,3cyclictest131rcu_preempt00:05:012
3445513994511,5cyclictest131rcu_preempt19:47:332
344550999457,5cyclictest131rcu_preempt21:35:001
3445509994528,13cyclictest3467602-21systemd-journal23:10:001
3445509994525,13cyclictest3475239-21tr22:00:001
344550999451,5cyclictest131rcu_preempt19:30:021
344550999451,5cyclictest131rcu_preempt19:30:011
3445504994512,6cyclictest131rcu_preempt19:40:000
344551899444,4cyclictest131rcu_preempt19:57:323
344551899444,3cyclictest131rcu_preempt19:45:023
344551899443,4cyclictest131rcu_preempt23:46:223
344551899443,3cyclictest131rcu_preempt19:35:003
3445513994436,5cyclictest3456510-21kworker/2:122:56:072
3445513994436,5cyclictest3444840-21kworker/2:020:47:582
3445513994429,11cyclictest3447352-21sed19:45:022
344550999443,3cyclictest131rcu_preempt19:15:021
3445509994432,10cyclictest23-21ksoftirqd/122:04:271
3445509994427,13cyclictest23-21ksoftirqd/122:10:011
3445504994427,11cyclictest3467602-21systemd-journal20:00:000
344550499442,36cyclictest3445694-21modprobe19:15:010
3445504994413,5cyclictest131rcu_preempt20:50:000
344551899433,3cyclictest131rcu_preempt20:45:023
344551899433,3cyclictest131rcu_preempt19:12:323
344551899431,3cyclictest131rcu_preempt20:58:333
344551399434,7cyclictest30-21ksoftirqd/223:50:022
3445513994336,4cyclictest3444840-21kworker/2:021:48:232
3445509994328,11cyclictest23-21ksoftirqd/119:56:311
3445509994327,12cyclictest3467602-21systemd-journal20:05:001
3445509994326,13cyclictest275-21dbus-daemon23:25:011
3445509994326,13cyclictest275-21dbus-daemon23:25:001
3445509994321,15cyclictest3467602-21systemd-journal20:39:591
3445504994311,5cyclictest131rcu_preempt19:50:020
344551899423,3cyclictest131rcu_preempt20:10:033
344551899423,3cyclictest131rcu_preempt19:21:213
344551899422,4cyclictest131rcu_preempt20:29:043
344551899421,4cyclictest131rcu_preempt21:00:333
344551899421,4cyclictest131rcu_preempt20:24:333
344551399427,8cyclictest30-21ksoftirqd/220:05:012
3445509994226,12cyclictest3451482-21systemd-run21:05:001
3445509994222,15cyclictest275-21dbus-daemon21:45:011
3445509994218,6cyclictest3510525-21ssh23:15:031
344550999421,3cyclictest131rcu_preempt23:32:031
344550999421,3cyclictest131rcu_preempt23:32:021
3445504994229,9cyclictest151rcuc/019:25:020
344551899414,3cyclictest131rcu_preempt19:55:013
344551899411,3cyclictest131rcu_preempt19:45:033
3445513994133,5cyclictest3444840-21kworker/2:020:20:012
3445513994133,5cyclictest3444840-21kworker/2:019:31:072
344550999416,3cyclictest131rcu_preempt20:10:021
344550999414,3cyclictest131rcu_preempt22:35:021
3445509994131,7cyclictest3542369-21sshd00:22:511
3445509994129,10cyclictest23-21ksoftirqd/123:39:531
3445509994125,12cyclictest275-21dbus-daemon20:35:001
3445509994125,12cyclictest275-21dbus-daemon20:35:001
344550999412,4cyclictest131rcu_preempt19:50:001
3445509994124,12cyclictest23-21ksoftirqd/119:20:011
344550999411,3cyclictest131rcu_preempt22:23:371
3445504994126,11cyclictest3446472-21sshd19:28:320
3445504994126,11cyclictest3446472-21sshd19:28:320
344551899404,3cyclictest131rcu_preempt20:55:013
344551899404,3cyclictest131rcu_preempt19:26:323
344551899404,3cyclictest131rcu_preempt19:26:323
344551899401,4cyclictest131rcu_preempt20:17:003
3445513994028,9cyclictest3467602-21systemd-journal20:42:332
344550999406,5cyclictest131rcu_preempt23:17:371
344550999404,7cyclictest23-21ksoftirqd/119:10:021
344550999403,3cyclictest131rcu_preempt22:50:011
344550999403,3cyclictest131rcu_preempt00:34:591
3445509994030,6cyclictest3447961-21cat19:55:021
3445509994026,11cyclictest1-21systemd21:00:021
3445509994025,11cyclictest23-21ksoftirqd/122:13:291
3445509994024,12cyclictest23-21ksoftirqd/120:50:011
3445509994017,9cyclictest23-21ksoftirqd/100:00:131
344550999401,14cyclictest131rcu_preempt20:20:011
344550499404,32cyclictest150-21pr/ttyS220:08:320
344551399394,3cyclictest131rcu_preempt20:21:332
344550999395,3cyclictest131rcu_preempt00:20:011
344550999394,6cyclictest131rcu_preempt22:42:241
344550999394,3cyclictest131rcu_preempt23:00:031
3445509993926,9cyclictest3467602-21systemd-journal20:26:341
3445509993925,11cyclictest3467602-21systemd-journal20:42:331
3445509993925,10cyclictest23-21ksoftirqd/123:50:401
344550999391,4cyclictest131rcu_preempt19:45:031
344550999391,3cyclictest131rcu_preempt21:25:021
344550999391,13cyclictest3470194-21ssh21:49:221
344551899383,3cyclictest131rcu_preempt20:15:023
344551399384,4cyclictest131rcu_preempt19:30:012
344551399384,4cyclictest131rcu_preempt19:30:012
3445513993828,7cyclictest3447821-21sed19:55:012
3445513993827,8cyclictest30-21ksoftirqd/220:15:022
3445513993826,9cyclictest30-21ksoftirqd/219:15:002
344551399381,4cyclictest131rcu_preempt20:30:002
344550999383,5cyclictest131rcu_preempt20:55:001
344550999383,4cyclictest131rcu_preempt21:54:471
3445509993826,9cyclictest23-21ksoftirqd/121:20:011
344550999382,5cyclictest131rcu_preempt19:40:001
3445509993824,10cyclictest23-21ksoftirqd/122:50:091
344550999381,3cyclictest131rcu_preempt19:25:011
3445518993723,10cyclictest275-21dbus-daemon20:50:003
344551399374,5cyclictest131rcu_preempt19:16:492
344551399373,3cyclictest131rcu_preempt19:10:022
3445513993727,6cyclictest309-21in:imuxsock20:10:002
344550999372,3cyclictest131rcu_preempt21:10:021
344550999372,3cyclictest131rcu_preempt00:30:011
344550999371,3cyclictest131rcu_preempt23:27:361
344550999367,6cyclictest131rcu_preempt20:15:011
344551399352,3cyclictest131rcu_preempt20:50:152
3445513993520,11cyclictest3447020-21basename19:40:002
344550999357,13cyclictest23-21ksoftirqd/120:25:021
344550999352,4cyclictest131rcu_preempt19:35:011
344551899341,3cyclictest131rcu_preempt20:36:333
344551899321,4cyclictest131rcu_preempt19:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional