You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-17 - 23:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Tue Mar 17, 2026 12:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
395081099474456,15cyclictest4045702-21turbostat11:44:552
395081499465441,21cyclictest3956411-21turbostat07:25:013
395081499464457,4cyclictest4047475-21turbostat11:45:013
395081499463442,17cyclictest3975510-21turbostat08:20:013
395081099463454,6cyclictest4017707-21turbostat10:24:552
395081499462454,5cyclictest4052686-21turbostat12:00:003
395081499461454,4cyclictest3968591-21turbostat08:04:553
395081499459450,6cyclictest3982433-21turbostat08:44:553
395081499455448,4cyclictest4065419-21turbostat12:39:553
395081099455447,5cyclictest4043974-21turbostat11:39:552
395080899455448,4cyclictest3993116-21turbostat09:14:541
395081099454447,4cyclictest4040206-21turbostat11:29:552
395080499453446,4cyclictest3958152-21turbostat07:34:550
395081499450442,5cyclictest4059925-21turbostat12:20:003
395081499449442,4cyclictest4019440-21turbostat10:25:013
395081499449440,6cyclictest3998317-21turbostat09:29:563
395081499449440,6cyclictest3998317-21turbostat09:29:553
395081099445437,5cyclictest3987629-21turbostat08:59:552
395081499439431,5cyclictest4038476-21turbostat11:20:003
395081499438431,4cyclictest3963402-21turbostat07:45:003
395081499438431,4cyclictest3961670-21turbostat07:39:593
395081499438431,4cyclictest3961670-21turbostat07:39:593
395081499437430,4cyclictest4028086-21turbostat10:54:553
395081499434427,4cyclictest3994851-21turbostat09:15:003
395081099434427,4cyclictest4026361-21turbostat10:49:552
395081499432424,5cyclictest3989361-21turbostat09:04:553
395081499429421,5cyclictest3991089-21turbostat09:09:563
395081499429421,5cyclictest3991089-21turbostat09:09:553
395081499428421,4cyclictest4019440-21turbostat10:29:553
395081499428419,6cyclictest3985898-21turbostat08:54:553
395081099427405,19cyclictest3952464-21turbostat07:19:552
395081499426419,4cyclictest4005239-21turbostat09:45:003
395081099426417,6cyclictest4047475-21turbostat11:49:562
395081499425418,4cyclictest3977234-21turbostat08:29:543
395081499423415,5cyclictest4015917-21turbostat10:15:003
395081499423415,5cyclictest4015917-21turbostat10:14:593
395081499422403,16cyclictest4031553-21turbostat11:00:003
395081499421414,4cyclictest4042239-21turbostat11:30:003
395081499419411,5cyclictest4063680-21turbostat12:34:563
395081099419412,4cyclictest3994851-21turbostat09:19:552
395080499418411,4cyclictest3978976-21turbostat08:34:560
395081499417410,4cyclictest3965130-21turbostat07:50:013
395081499417409,5cyclictest3956411-21turbostat07:29:563
395080499417409,5cyclictest3975510-21turbostat08:24:550
395080499417409,5cyclictest3975510-21turbostat08:24:550
395081499415408,4cyclictest4006979-21turbostat09:54:563
395081499413402,8cyclictest4001781-21turbostat09:39:553
395081499412403,6cyclictest4010430-21turbostat10:00:003
395081499412393,16cyclictest3970322-21turbostat08:09:543
395081499410402,5cyclictest4036740-21turbostat11:15:013
395080499408401,4cyclictest4003519-21turbostat09:44:550
395081499405398,4cyclictest4058183-21turbostat12:15:003
395081099405396,6cyclictest3973784-21turbostat08:19:552
395081099404397,4cyclictest3966857-21turbostat07:59:562
395081499402395,4cyclictest4022899-21turbostat10:39:553
395081499401392,6cyclictest4050959-21turbostat11:55:003
395081499400392,5cyclictest4042239-21turbostat11:34:553
395081499398385,10cyclictest3973784-21turbostat08:15:013
395081499397390,4cyclictest4026361-21turbostat10:45:013
395081499397390,4cyclictest3980708-21turbostat08:39:563
395081499396385,8cyclictest3996581-21turbostat09:24:553
395081499395388,4cyclictest4061658-21turbostat12:29:553
395081499393386,4cyclictest3959884-21turbostat07:35:003
395081499391384,4cyclictest3952464-21turbostat07:15:003
395081499391383,5cyclictest4033281-21turbostat11:04:593
395081499391383,5cyclictest4033281-21turbostat11:04:593
395081499385377,5cyclictest4012469-21turbostat10:05:003
395081099385378,4cyclictest4014200-21turbostat10:14:552
395081099385378,4cyclictest4014200-21turbostat10:14:552
395081499383376,4cyclictest4049219-21turbostat11:50:003
395081499381374,4cyclictest4061658-21turbostat12:25:003
395081499377370,4cyclictest3980708-21turbostat08:35:013
395081499375357,15cyclictest3977234-21turbostat08:24:593
395081499375357,15cyclictest3977234-21turbostat08:24:593
395081499373366,4cyclictest4040206-21turbostat11:25:003
395081499370364,3cyclictest4035006-21turbostat11:10:013
395081499370364,3cyclictest3996581-21turbostat09:20:003
395081499369363,3cyclictest4014200-21turbostat10:10:003
395081499369362,4cyclictest4015917-21turbostat10:19:553
395081499368362,3cyclictest4021168-21turbostat10:34:553
395081499361338,20cyclictest3966857-21turbostat07:55:013
395081499355346,6cyclictest3954689-21turbostat07:19:593
395081499350343,4cyclictest4045702-21turbostat11:40:003
395081099349343,3cyclictest4005239-21turbostat09:49:552
395081099348342,3cyclictest3965130-21turbostat07:54:552
395080499348342,3cyclictest3954689-21turbostat07:24:550
395080499340332,4cyclictest4000052-21turbostat09:34:550
395081499325318,4cyclictest4028086-21turbostat10:50:013
395081099319311,5cyclictest3959884-21turbostat07:39:562
395081099319311,5cyclictest3959884-21turbostat07:39:552
395081099318312,3cyclictest3984168-21turbostat08:49:552
395081099315308,4cyclictest3950680-21turbostat07:14:552
395081099314308,3cyclictest4052686-21turbostat12:04:552
395081499313306,4cyclictest3985898-21turbostat08:50:003
395081499303296,4cyclictest4054431-21turbostat12:09:553
394964122390,5sleep00-21swapper/007:05:170
395081499215199,12cyclictest3989361-21turbostat09:00:003
395081499165151,10cyclictest4001781-21turbostat09:35:003
395081499147134,9cyclictest4006979-21turbostat09:50:013
3950483212572,43sleep10-21swapper/107:07:481
3950524210976,23sleep20-21swapper/207:08:222
39508149910795,9cyclictest3968591-21turbostat08:00:003
3950330210674,23sleep30-21swapper/307:05:493
3950814999988,8cyclictest4054431-21turbostat12:05:003
39646722910,6sleep00-21swapper/007:45:300
40370112820,6sleep04497-21PK-Backend11:15:130
197399790,21rtkit-daemon1972-21rtkit-daemon11:56:140
40026622700,4sleep10-21swapper/109:35:221
395080499640,60cyclictest1972-21rtkit-daemon09:05:270
395080499640,60cyclictest1972-21rtkit-daemon09:05:270
197399610,9rtkit-daemon1972-21rtkit-daemon11:50:550
197399600,9rtkit-daemon1972-21rtkit-daemon10:44:520
197399590,8rtkit-daemon1972-21rtkit-daemon09:25:570
197399590,8rtkit-daemon1972-21rtkit-daemon09:25:560
197399580,9rtkit-daemon1972-21rtkit-daemon09:45:590
197399580,9rtkit-daemon1972-21rtkit-daemon07:25:540
197399580,8rtkit-daemon1972-21rtkit-daemon10:45:450
197399580,8rtkit-daemon1972-21rtkit-daemon08:44:460
197399580,8rtkit-daemon1972-21rtkit-daemon08:06:060
197399580,8rtkit-daemon1972-21rtkit-daemon08:04:410
197399580,8rtkit-daemon1972-21rtkit-daemon07:51:240
197399570,9rtkit-daemon1972-21rtkit-daemon09:51:350
197399570,9rtkit-daemon1972-21rtkit-daemon07:35:430
197399570,9rtkit-daemon1972-21rtkit-daemon07:35:430
197399570,8rtkit-daemon1972-21rtkit-daemon08:49:320
40457142560,10chrt4045705-21perf11:40:010
197399560,9rtkit-daemon1972-21rtkit-daemon12:13:540
197399560,9rtkit-daemon1972-21rtkit-daemon10:01:340
197399560,9rtkit-daemon1972-21rtkit-daemon09:19:190
197399560,9rtkit-daemon1972-21rtkit-daemon08:56:280
197399560,9rtkit-daemon1972-21rtkit-daemon08:50:500
197399560,9rtkit-daemon1972-21rtkit-daemon08:37:060
197399560,9rtkit-daemon1972-21rtkit-daemon07:57:160
197399560,8rtkit-daemon1972-21rtkit-daemon08:23:121
197399560,8rtkit-daemon1972-21rtkit-daemon08:23:121
395080499551,24cyclictest13-21ksoftirqd/010:16:050
197399550,9rtkit-daemon1972-21rtkit-daemon12:26:570
197399550,9rtkit-daemon1972-21rtkit-daemon11:48:060
197399550,9rtkit-daemon1972-21rtkit-daemon11:08:490
197399550,9rtkit-daemon1972-21rtkit-daemon08:19:020
197399540,9rtkit-daemon1972-21rtkit-daemon10:29:490
197399540,8rtkit-daemon1972-21rtkit-daemon10:50:570
197399540,8rtkit-daemon1972-21rtkit-daemon09:10:440
197399540,8rtkit-daemon1972-21rtkit-daemon09:01:030
197399540,10rtkit-daemon1972-21rtkit-daemon11:40:170
395080499539,22cyclictest438-20systemd-journal12:10:010
197399530,9rtkit-daemon1972-21rtkit-daemon12:03:020
197399530,9rtkit-daemon1972-21rtkit-daemon11:03:570
197399530,9rtkit-daemon1972-21rtkit-daemon11:03:560
197399530,9rtkit-daemon1972-21rtkit-daemon10:59:580
40045332480,5sleep2331rcuc/209:40:252
3950810994725,8cyclictest34-21ksoftirqd/211:22:432
3950810994715,16cyclictest34-21ksoftirqd/207:34:482
395080499470,43cyclictest1972-21rtkit-daemon09:20:440
395081099468,16cyclictest34-21ksoftirqd/210:30:162
395080499460,42cyclictest1972-21rtkit-daemon11:32:220
39973442450,6sleep14497-21PK-Backend09:20:211
3950810994534,8cyclictest34-21ksoftirqd/209:05:192
3950810994534,8cyclictest34-21ksoftirqd/209:05:182
3950810994524,17cyclictest3962390-21/usr/sbin/munin07:40:192
3950810994514,12cyclictest34-21ksoftirqd/209:13:052
395080499440,41cyclictest1972-21rtkit-daemon09:55:480
3950810994325,13cyclictest438-20systemd-journal08:04:592
3950810994311,16cyclictest34-21ksoftirqd/210:25:462
395081099426,14cyclictest34-21ksoftirqd/210:00:172
3950810994224,15cyclictest34-21ksoftirqd/211:01:442
3950810994224,15cyclictest34-21ksoftirqd/211:01:442
395080899421,36cyclictest3974830-21proc_pri08:15:261
3950810994121,17cyclictest34-21ksoftirqd/209:25:202
3950810994121,17cyclictest34-21ksoftirqd/209:25:192
3950810994119,19cyclictest34-21ksoftirqd/207:21:332
395081099411,10cyclictest1033-21gdbus10:16:052
3950808994123,14cyclictest4000897-21memory09:30:211
3950804994115,11cyclictest141rcu_preempt10:35:000
395080499411,21cyclictest3951578-21taskset07:10:210
395081099408,11cyclictest34-21ksoftirqd/208:24:082
395081099408,11cyclictest34-21ksoftirqd/208:24:082
395081099406,14cyclictest34-21ksoftirqd/208:35:142
395081099405,32cyclictest34-21ksoftirqd/212:14:462
3950810994022,13cyclictest4064324-21df_inode12:30:152
3950810994011,15cyclictest34-21ksoftirqd/212:19:392
3950810994010,16cyclictest34-21ksoftirqd/209:35:272
3950808994014,11cyclictest4015131-21strings10:10:201
3950808994014,11cyclictest4015131-21strings10:10:201
395080899401,13cyclictest4497-21PK-Backend08:15:001
395080899401,11cyclictest3966278-21wc07:50:271
3950804994021,11cyclictest13-21ksoftirqd/012:15:040
3950804994019,17cyclictest3978998-21unin-run08:30:000
395081099397,12cyclictest34-21ksoftirqd/210:57:362
3950810993922,12cyclictest438-20systemd-journal12:07:322
3950810993919,17cyclictest34-21ksoftirqd/211:55:582
3950808993923,11cyclictest438-20systemd-journal08:34:321
395080499397,11cyclictest13-21ksoftirqd/012:30:220
3950804993921,6cyclictest13-21ksoftirqd/011:20:210
395080499390,35cyclictest1972-21rtkit-daemon10:06:030
39905392380,7chrt0-21swapper/209:00:292
395081099388,15cyclictest34-21ksoftirqd/211:13:102
395081099386,15cyclictest34-21ksoftirqd/208:40:252
395081099385,18cyclictest34-21ksoftirqd/211:08:032
3950810993823,11cyclictest4050749-21sshd11:52:332
3950808993827,7cyclictest4050955-21turbostat.cron11:55:001
395081099377,16cyclictest34-21ksoftirqd/208:34:562
395081099377,12cyclictest34-21ksoftirqd/209:30:552
3950810993727,7cyclictest34-21ksoftirqd/208:52:572
3950808993722,11cyclictest3954665-21cron07:20:001
3950808993721,12cyclictest3961468-21sshd07:38:311
3950808993721,12cyclictest3961468-21sshd07:38:311
3950808993720,13cyclictest1-21systemd09:20:011
3950808993720,12cyclictest438-20systemd-journal12:30:011
3950808993719,6cyclictest141rcu_preempt07:30:241
3950808993718,7cyclictest141rcu_preempt07:10:281
3950804993727,6cyclictest4019416-21cron10:25:000
395080499372,14cyclictest0-21swapper/008:10:260
3950804993721,11cyclictest438-20systemd-journal10:10:340
3950804993721,11cyclictest438-20systemd-journal10:10:340
395080499371,19cyclictest4061762-21grep12:25:010
395081099367,8cyclictest34-21ksoftirqd/212:25:062
3950810993621,11cyclictest4043209-21ntp_kernel_err11:30:212
3950810993612,9cyclictest34-21ksoftirqd/210:50:042
395081099361,11cyclictest3971773-21taskset08:06:442
395080899366,4cyclictest141rcu_preempt11:42:461
3950808993620,12cyclictest4031119-21sed10:55:301
3950804993619,5cyclictest13-21ksoftirqd/010:35:130
3950804993610,23cyclictest13-21ksoftirqd/011:27:180
395081099355,17cyclictest34-21ksoftirqd/208:25:182
395081099354,16cyclictest34-21ksoftirqd/210:40:342
3950810993525,7cyclictest34-21ksoftirqd/208:12:142
3950810993521,10cyclictest4060851-21strings12:20:202
3950810993518,13cyclictest4037563-21cat11:15:222
3950808993527,5cyclictest4497-21PK-Backend11:56:321
3950808993520,11cyclictest3988820-21sed08:55:311
3950808993519,12cyclictest3999284-21awk09:25:231
3950808993519,12cyclictest3999284-21awk09:25:231
395080899351,8cyclictest0-21swapper/110:49:581
3950808993517,6cyclictest141rcu_preempt12:05:031
3950808993517,6cyclictest141rcu_preempt10:50:161
3950808993516,7cyclictest141rcu_preempt09:05:171
3950808993516,7cyclictest141rcu_preempt09:05:171
3950808993516,7cyclictest141rcu_preempt07:30:001
3950808993515,8cyclictest141rcu_preempt10:20:151
395080899351,14cyclictest4497-21PK-Backend08:42:361
395080899351,10cyclictest4497-21PK-Backend12:38:281
395080899351,10cyclictest4497-21PK-Backend11:34:211
3950804993523,6cyclictest13-21ksoftirqd/007:40:130
3950804993521,6cyclictest13-21ksoftirqd/011:10:150
3950810993426,5cyclictest0-21swapper/207:45:132
3950810993420,10cyclictest4010414-21cron10:00:012
3950810993418,12cyclictest4066841-21seq12:36:142
3950810993418,12cyclictest4023904-21sh10:35:232
3950810993418,11cyclictest4012752-21cpu10:05:122
395081099341,14cyclictest3997976-21runrttasks09:20:532
395080899348,11cyclictest1137-21runrttasks08:05:241
395080899347,23cyclictest3978952-21cron08:30:001
395080899346,10cyclictest4497-21PK-Backend11:02:301
395080899346,10cyclictest4497-21PK-Backend11:02:291
3950808993427,5cyclictest0-21swapper/109:40:491
395080899341,8cyclictest0-21swapper/112:11:431
395080899341,8cyclictest0-21swapper/107:43:091
3950808993418,12cyclictest4061659-21ls12:25:001
3950808993416,6cyclictest141rcu_preempt10:40:261
395080899341,5cyclictest4497-21PK-Backend07:58:141
3950808993415,7cyclictest141rcu_preempt10:35:151
3950808993414,7cyclictest141rcu_preempt07:45:131
395080899341,19cyclictest4037166-21tr11:15:141
3950808993411,5cyclictest141rcu_preempt11:26:261
3950804993424,5cyclictest13-21ksoftirqd/009:35:110
3950804993423,6cyclictest13-21ksoftirqd/007:15:040
40204272330,8chrt0-21swapper/110:25:241
395081099335,7cyclictest34-21ksoftirqd/209:50:042
3950810993317,12cyclictest3957195-21latency07:25:212
3950808993326,4cyclictest4497-21PK-Backend12:31:141
3950808993326,4cyclictest4497-21PK-Backend11:06:111
3950808993326,4cyclictest4497-21PK-Backend09:58:561
395080899331,8cyclictest4497-21PK-Backend12:16:181
3950808993317,12cyclictest3984978-21grep08:45:211
3950808993316,13cyclictest4011836-21python310:00:331
3950808993315,6cyclictest141rcu_preempt12:00:301
395080899331,4cyclictest4497-21PK-Backend11:10:521
3950808993314,7cyclictest3955102-21tr07:20:131
3950808993314,6cyclictest141rcu_preempt10:05:241
3950808993313,8cyclictest141rcu_preempt08:35:131
3950808993313,5cyclictest141rcu_preempt08:54:361
3950808993313,5cyclictest141rcu_preempt08:02:511
395080899331,29cyclictest4497-21PK-Backend09:51:261
3950808993312,6cyclictest141rcu_preempt10:18:081
3950808993312,5cyclictest141rcu_preempt11:23:201
3950804993322,6cyclictest13-21ksoftirqd/012:35:100
3950808993226,4cyclictest0-21swapper/111:49:571
3950808993226,4cyclictest0-21swapper/109:49:521
3950808993218,10cyclictest3991091-21grep09:05:001
3950808993213,5cyclictest141rcu_preempt11:35:131
395080899321,28cyclictest4497-21PK-Backend10:33:391
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional