You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-16 - 04:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Mar 16, 2026 00:44:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
308902999522503,16cyclictest3129559-21turbostat21:09:543
308902999515502,10cyclictest3147215-21turbostat21:54:593
308902999507499,5cyclictest3134753-21turbostat21:24:563
308902999479457,19cyclictest3168288-21turbostat22:59:563
308902999479457,19cyclictest3168288-21turbostat22:59:563
308902999478456,19cyclictest3117437-21turbostat20:34:563
308902999478456,19cyclictest3117437-21turbostat20:34:553
308902999473451,19cyclictest3124365-21turbostat20:54:553
308902999465456,6cyclictest3112254-21turbostat20:15:003
308902999464455,6cyclictest3185902-21turbostat23:45:003
308902999464455,6cyclictest3185902-21turbostat23:45:003
308902399463456,4cyclictest3155866-21turbostat22:24:551
308902999461454,4cyclictest3157595-21turbostat22:24:593
308902999460453,4cyclictest3093215-21turbostat19:24:563
308902999460452,5cyclictest3134753-21turbostat21:20:013
308902999460452,5cyclictest3134753-21turbostat21:20:013
308902599460452,4cyclictest3143757-21turbostat21:49:562
308902999458451,4cyclictest3159333-21turbostat22:34:563
308902999458436,19cyclictest3150679-21turbostat22:09:563
308902999457431,4cyclictest3164520-21turbostat22:45:003
308902999455435,17cyclictest3173479-21turbostat23:09:593
308902999451443,5cyclictest3152406-21turbostat22:14:563
308902599448440,5cyclictest3191090-21turbostat00:04:562
308902999447440,4cyclictest3127818-21turbostat21:04:553
308902999447438,6cyclictest3098402-21turbostat19:35:003
308902599445435,7cyclictest3170017-21turbostat23:04:552
308902399445436,6cyclictest3096666-21turbostat19:34:541
308902999444437,4cyclictest3171750-21turbostat23:05:003
308901999444423,18cyclictest3100134-21turbostat19:44:560
308901999444423,18cyclictest3100134-21turbostat19:44:550
308902999442435,4cyclictest3187620-21turbostat23:54:553
308902999441434,4cyclictest3110518-21turbostat20:10:003
308902999441434,4cyclictest3110518-21turbostat20:10:003
308902999441433,5cyclictest3187620-21turbostat23:50:003
308902399440433,4cyclictest3164520-21turbostat22:49:551
308902999438430,5cyclictest3133009-21turbostat21:15:013
308902599438430,5cyclictest3194558-21turbostat00:14:542
308902999435412,20cyclictest3117437-21turbostat20:30:003
308902399434427,4cyclictest3088916-21turbostat19:14:551
308902999433412,18cyclictest3126102-21turbostat20:59:563
308902999433412,18cyclictest3126102-21turbostat20:59:563
308902999432425,4cyclictest3138515-21turbostat21:29:593
308902999432424,5cyclictest3198028-21turbostat00:24:563
308902999432423,6cyclictest3140251-21turbostat21:35:003
308902599430401,26cyclictest3189349-21turbostat23:59:552
308902999426419,4cyclictest3199753-21turbostat00:29:553
308902999426415,8cyclictest3196283-21turbostat00:19:553
308902999423414,5cyclictest3120906-21turbostat20:40:003
308902999422413,6cyclictest3194558-21turbostat00:10:003
308902999421414,4cyclictest3103596-21turbostat19:50:003
308902999419412,4cyclictest3178978-21turbostat23:25:003
308902999418409,6cyclictest3178978-21turbostat23:29:553
308902999418409,6cyclictest3108793-21turbostat20:05:013
308902999417410,4cyclictest3161062-21turbostat22:39:563
308902999417410,4cyclictest3113976-21turbostat20:24:553
308902599414407,4cyclictest3098402-21turbostat19:39:552
308902599411402,6cyclictest3136781-21turbostat21:29:552
308902999408401,4cyclictest3180699-21turbostat23:34:543
308902999408388,17cyclictest3175514-21turbostat23:15:003
308902999405384,18cyclictest3147215-21turbostat21:59:543
308902999401393,5cyclictest3101868-21turbostat19:44:593
308902999401393,5cyclictest3101868-21turbostat19:44:593
308902999399392,4cyclictest3203521-21turbostat00:39:553
308902999399392,4cyclictest3107053-21turbostat20:00:003
308902999399391,5cyclictest3093215-21turbostat19:20:003
308902999397388,6cyclictest3155866-21turbostat22:20:003
308902599397390,4cyclictest3112254-21turbostat20:19:552
308902399397388,6cyclictest3094946-21turbostat19:29:551
308902599393385,5cyclictest3131278-21turbostat21:14:542
308902599393384,6cyclictest3166255-21turbostat22:54:552
308902999392372,17cyclictest3159333-21turbostat22:30:013
308902999387364,20cyclictest3150679-21turbostat22:05:013
308902999385364,18cyclictest3145481-21turbostat21:50:003
308902599381374,4cyclictest3182441-21turbostat23:39:562
308902599378371,4cyclictest3103596-21turbostat19:54:542
308902999375367,5cyclictest3143757-21turbostat21:45:013
308901999374350,21cyclictest3192836-21turbostat00:09:560
308902399370363,4cyclictest3145481-21turbostat21:54:551
308902999369362,4cyclictest3122638-21turbostat20:45:003
308902999361354,4cyclictest3192836-21turbostat00:05:003
308902999360354,3cyclictest3096666-21turbostat19:30:003
308902599360352,5cyclictest3162790-21turbostat22:44:552
308902999359352,4cyclictest3177245-21turbostat23:20:013
308902999359352,4cyclictest3177245-21turbostat23:20:013
308901999355348,4cyclictest3140251-21turbostat21:39:550
308902999354349,3cyclictest3196283-21turbostat00:15:003
308902999353346,4cyclictest3203521-21turbostat00:34:593
308902999352345,4cyclictest3113976-21turbostat20:20:003
308902999347340,4cyclictest3168288-21turbostat22:55:003
308902999347340,4cyclictest3141969-21turbostat21:40:013
308902599340333,4cyclictest3105329-21turbostat19:59:552
308902999339333,3cyclictest3100134-21turbostat19:40:013
308902399334328,3cyclictest3122638-21turbostat20:49:551
308902399332324,5cyclictest3101868-21turbostat19:49:551
308902599327321,3cyclictest3148950-21turbostat22:04:542
308902599327320,4cyclictest3108793-21turbostat20:09:562
308902599327320,4cyclictest3108793-21turbostat20:09:562
308902999326318,5cyclictest3184168-21turbostat23:40:003
308902599325318,4cyclictest3175514-21turbostat23:19:552
308902599325318,4cyclictest3175514-21turbostat23:19:552
308902599323316,4cyclictest3201787-21turbostat00:34:552
308901999323316,4cyclictest3120906-21turbostat20:44:550
308902999317310,4cyclictest3090692-21turbostat19:15:013
308902999315308,4cyclictest3124365-21turbostat20:50:003
308902599314307,4cyclictest3177245-21turbostat23:24:562
308801822330,7sleep00-21swapper/019:05:200
308902999218204,10cyclictest3105329-21turbostat19:55:003
3088643211177,23sleep10-21swapper/119:06:541
3088669211078,22sleep20-21swapper/219:07:162
3088758210774,23sleep30-21swapper/319:08:253
30890259910093,4cyclictest3157595-21turbostat22:29:542
3089029999988,8cyclictest3191090-21turbostat00:00:003
31375722950,7sleep00-21swapper/021:25:200
3089029999386,4cyclictest3166255-21turbostat22:50:003
31147322820,5sleep04497-21PK-Backend20:20:200
31658362770,15chrt34-21ksoftirqd/222:45:282
197399690,5rtkit-daemon0-21swapper/023:33:500
308902399668,21cyclictest3174498-21sed23:10:211
197399590,8rtkit-daemon1972-21rtkit-daemon20:53:422
197399570,8rtkit-daemon1972-21rtkit-daemon20:03:492
197399570,7rtkit-daemon1972-21rtkit-daemon22:23:572
30973152560,16chrt34-21ksoftirqd/219:30:172
197399560,7rtkit-daemon1972-21rtkit-daemon20:36:152
197399560,7rtkit-daemon1972-21rtkit-daemon19:22:202
3089023995442,8cyclictest251rcuc/121:20:301
3089023995432,8cyclictest26-21ksoftirqd/123:03:161
3089023995415,12cyclictest26-21ksoftirqd/100:19:181
3089019995432,6cyclictest1081-21ntpd23:15:230
3089019995432,6cyclictest1081-21ntpd23:15:230
197399540,7rtkit-daemon1972-21rtkit-daemon19:42:342
197399540,7rtkit-daemon1972-21rtkit-daemon19:42:342
197399530,8rtkit-daemon1972-21rtkit-daemon21:15:402
197399530,8rtkit-daemon1972-21rtkit-daemon21:15:392
3089023995216,18cyclictest141rcu_preempt21:34:281
32029012510,8sleep13202025-21/usr/sbin/munin00:30:271
308902599511,14cyclictest3116819-21processes20:25:272
3089023995119,19cyclictest26-21ksoftirqd/121:05:251
308902599507,23cyclictest3180442-21taskset23:26:522
3089023995014,12cyclictest26-21ksoftirqd/121:43:311
3089023995011,20cyclictest26-21ksoftirqd/120:39:591
31273032490,6sleep04497-21PK-Backend20:55:270
31273032490,6sleep04497-21PK-Backend20:55:270
3089023994920,17cyclictest26-21ksoftirqd/123:49:321
308902399491,19cyclictest3116374-21taskset20:25:191
3089025994824,20cyclictest3170041-21unin-run23:00:012
3089025994824,20cyclictest3170041-21unin-run23:00:002
3089023994826,8cyclictest26-21ksoftirqd/121:58:591
3089023994819,17cyclictest26-21ksoftirqd/121:26:461
3089023994811,13cyclictest26-21ksoftirqd/119:15:111
308902599476,19cyclictest959-21udisksd23:30:222
308902599472,18cyclictest3198796-21cut00:20:212
3089023994714,18cyclictest26-21ksoftirqd/122:55:191
3089023994714,18cyclictest26-21ksoftirqd/122:55:181
308902399471,20cyclictest3201538-21taskset00:27:051
308901999471,14cyclictest13-21ksoftirqd/000:30:340
308902599469,19cyclictest3161065-21munin-run22:35:002
308902599467,9cyclictest3142972-21ntpq21:40:232
308902599462,17cyclictest3184599-21expr23:40:132
308902599462,17cyclictest3184599-21expr23:40:132
308902599461,17cyclictest3161768-21if_enp2s022:35:172
308902399465,19cyclictest0-21swapper/123:30:291
3089023994625,9cyclictest26-21ksoftirqd/120:12:591
3089023994624,9cyclictest26-21ksoftirqd/122:12:581
3089023994617,17cyclictest26-21ksoftirqd/100:38:031
308902399461,19cyclictest26-21ksoftirqd/100:25:011
308902399461,18cyclictest3127104-21seq20:55:231
308902399461,18cyclictest3127104-21seq20:55:221
308902599454,18cyclictest3127103-21ntp_states20:55:232
308902599454,18cyclictest3127103-21ntp_states20:55:222
308902599453,17cyclictest3103580-21cron19:49:592
308902599452,18cyclictest3135665-40apt-check21:20:242
308902599452,17cyclictest438-20systemd-journal20:14:592
308902399459,17cyclictest26-21ksoftirqd/120:52:481
308902399459,12cyclictest26-21ksoftirqd/123:15:291
308902399459,12cyclictest26-21ksoftirqd/123:15:291
3089023994525,8cyclictest26-21ksoftirqd/122:36:581
3089023994519,9cyclictest26-21ksoftirqd/122:07:591
3089023994518,8cyclictest26-21ksoftirqd/123:54:551
3089023994515,16cyclictest3155503-21chrt22:15:361
308902399451,23cyclictest3168258-21cat22:54:561
308902399451,21cyclictest3105561-21seq19:55:091
308901999456,9cyclictest13-21ksoftirqd/020:00:010
308901999455,11cyclictest141rcu_preempt23:46:230
3089023994416,16cyclictest26-21ksoftirqd/122:43:521
3089023994416,16cyclictest26-21ksoftirqd/100:13:191
3089019994425,8cyclictest13-21ksoftirqd/021:43:320
3089019994412,5cyclictest3119170-21perf20:39:550
308901999441,14cyclictest4497-21PK-Backend00:25:000
308902599433,16cyclictest3146616-21python321:50:272
308902599433,15cyclictest3187028-21python323:45:282
308902599431,20cyclictest3091538-21cat19:15:072
308902599430,14cyclictest3194561-21latency_hist00:10:012
3089023994334,5cyclictest251rcuc/123:23:531
3089023994318,8cyclictest26-21ksoftirqd/121:00:591
308902399431,20cyclictest26-21ksoftirqd/121:45:281
308901999439,17cyclictest13-21ksoftirqd/021:15:240
308901999439,17cyclictest13-21ksoftirqd/021:15:240
308901999437,5cyclictest141rcu_preempt23:04:500
308901999436,5cyclictest141rcu_preempt21:05:560
3089019994325,5cyclictest141rcu_preempt20:15:020
31525382420,6chrt0-21swapper/222:10:012
308902399428,17cyclictest26-21ksoftirqd/121:10:531
3089023994223,8cyclictest26-21ksoftirqd/122:27:591
3089023994219,8cyclictest26-21ksoftirqd/122:01:591
3089023994218,8cyclictest26-21ksoftirqd/123:39:051
3089023994216,9cyclictest0-21swapper/100:08:291
3089023994215,10cyclictest26-21ksoftirqd/123:09:211
3089023994214,12cyclictest26-21ksoftirqd/122:32:221
3089023994214,12cyclictest26-21ksoftirqd/120:31:321
3089023994214,12cyclictest26-21ksoftirqd/120:31:321
308901999427,5cyclictest141rcu_preempt22:59:230
308901999427,5cyclictest141rcu_preempt22:59:230
308901999427,5cyclictest141rcu_preempt22:45:140
308901999426,5cyclictest141rcu_preempt23:25:300
308901999426,5cyclictest141rcu_preempt20:50:160
308901999425,5cyclictest141rcu_preempt20:07:380
308901999425,5cyclictest141rcu_preempt20:07:370
308901999424,5cyclictest141rcu_preempt23:52:100
3089019994224,14cyclictest13-21ksoftirqd/000:40:010
3089019994213,16cyclictest13-21ksoftirqd/021:13:590
3089019994210,16cyclictest13-21ksoftirqd/020:00:200
308902599417,16cyclictest3131120-21runrttasks21:08:092
308902599417,15cyclictest3197983-21python300:19:492
308902599417,11cyclictest0-21swapper/219:10:222
308902599414,14cyclictest3152657-21acpi22:10:112
3089025994119,18cyclictest3175500-21cron23:15:012
308902599411,14cyclictest3095790-21memory19:25:202
308902399417,11cyclictest26-21ksoftirqd/120:40:261
308902399416,12cyclictest26-21ksoftirqd/121:35:251
308902399415,9cyclictest26-21ksoftirqd/120:20:311
308902399415,12cyclictest26-21ksoftirqd/121:15:201
308902399415,12cyclictest26-21ksoftirqd/121:15:191
3089023994114,12cyclictest26-21ksoftirqd/123:27:431
3089023994113,12cyclictest26-21ksoftirqd/120:06:371
3089023994113,12cyclictest26-21ksoftirqd/120:06:371
308902399411,10cyclictest2095-21JS10:49:101
308901999417,12cyclictest141rcu_preempt19:15:000
308901999415,5cyclictest141rcu_preempt23:14:350
3089019994115,15cyclictest141rcu_preempt00:15:140
3089019994115,15cyclictest13-21ksoftirqd/000:00:080
3089019994114,7cyclictest141rcu_preempt19:17:230
3089019994114,11cyclictest141rcu_preempt21:45:020
3089019994111,16cyclictest13-21ksoftirqd/023:55:240
3089019994111,16cyclictest13-21ksoftirqd/022:10:220
3089019994111,11cyclictest141rcu_preempt00:25:020
308902599409,9cyclictest2069-21gnome-shell20:46:092
308902599407,10cyclictest3154399-21apt22:15:112
308902599405,15cyclictest3187871-21cat23:50:112
308902599402,15cyclictest3172381-21df23:05:172
308902599401,17cyclictest3117870-21cpuspeed_turbos20:30:142
308902599401,17cyclictest3117870-21cpuspeed_turbos20:30:132
308902399409,10cyclictest26-21ksoftirqd/120:03:391
308902399403,22cyclictest26-21ksoftirqd/123:55:281
3089023994031,5cyclictest251rcuc/119:40:051
3089023994031,5cyclictest251rcuc/119:40:051
3089023994018,9cyclictest26-21ksoftirqd/119:21:411
308902399401,21cyclictest3100085-21taskset19:39:391
308902399401,19cyclictest3105037-21taskset19:51:311
308901999406,5cyclictest141rcu_preempt19:34:540
308901999406,5cyclictest141rcu_preempt00:12:390
308901999405,5cyclictest141rcu_preempt21:50:230
308901999403,17cyclictest3163227-21cut22:40:120
3089019994024,12cyclictest3167910-21pkexec22:50:320
308901999402,17cyclictest3093953-21irqstats19:20:190
308901999401,26cyclictest4497-21PK-Backend20:35:000
308901999401,26cyclictest4497-21PK-Backend20:35:000
3089019994011,6cyclictest141rcu_preempt21:30:220
308902599394,13cyclictest3140731-21expr21:35:132
308902599392,14cyclictest3129756-21cut21:05:022
308902599392,14cyclictest3121140-21/usr/sbin/munin20:40:182
3089023993917,10cyclictest26-21ksoftirqd/120:17:141
308901999396,4cyclictest141rcu_preempt20:25:440
308901999395,4cyclictest141rcu_preempt22:15:350
308901999395,18cyclictest3135427-21cut21:20:130
308901999393,6cyclictest3104236-21diskstats19:50:170
3089019993921,13cyclictest13-21ksoftirqd/019:45:240
308902599382,7cyclictest3204265-21irqstats00:35:202
3089025993818,16cyclictest3148012-21sed21:55:222
308902399382,21cyclictest26-21ksoftirqd/123:40:061
308902399382,21cyclictest26-21ksoftirqd/123:40:061
308901999389,12cyclictest3129156-21sed21:00:270
3089019993821,5cyclictest13-21ksoftirqd/022:25:130
308901999381,19cyclictest3185873-21runrttasks23:44:560
308901999381,19cyclictest3185873-21runrttasks23:44:560
308901999381,15cyclictest3114004-21unin-run20:20:010
3089019993811,15cyclictest13-21ksoftirqd/021:55:150
3089019993811,11cyclictest141rcu_preempt22:30:300
308902599377,10cyclictest3139941-21taskset21:31:212
3089025993720,12cyclictest438-20systemd-journal00:26:322
308902599370,15cyclictest3114829-21memory20:20:222
308901999375,16cyclictest3089013-21cyclictest22:35:220
3089019993716,8cyclictest13-21ksoftirqd/022:05:020
3089019993712,13cyclictest3156837-21netstat22:20:230
308901999371,12cyclictest0-21swapper/020:45:300
308901999371,10cyclictest3178883-21chrt23:24:060
3089019993710,11cyclictest141rcu_preempt19:30:020
3089019993620,12cyclictest13-21ksoftirqd/023:35:250
3089019993620,12cyclictest13-21ksoftirqd/022:05:120
308901999351,8cyclictest0-21swapper/019:35:190
308901999351,14cyclictest4497-21PK-Backend23:05:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional