You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-15 - 07:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun Mar 15, 2026 00:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
251624599512503,6cyclictest2556346-21turbostat21:00:013
251623599487452,21cyclictest2613022-21turbostat23:44:540
251623599466458,5cyclictest2606055-21turbostat23:24:550
251624599465456,6cyclictest2613022-21turbostat23:40:003
251623599465457,5cyclictest2538386-21turbostat20:14:540
251624599464456,5cyclictest2560109-21turbostat21:10:003
251624599464456,5cyclictest2560109-21turbostat21:09:593
251624599463456,4cyclictest2581214-21turbostat22:09:593
251623899463454,6cyclictest2621739-21turbostat00:09:551
251623899462454,5cyclictest2567031-21turbostat21:34:561
251623899462454,5cyclictest2567031-21turbostat21:34:551
251624599460453,4cyclictest2618259-21turbostat23:55:003
251623899460451,6cyclictest2588135-21turbostat22:34:541
251623599459452,4cyclictest2619992-21turbostat00:04:540
251624599458451,4cyclictest2609521-21turbostat23:34:553
251623599458451,4cyclictest2541852-21turbostat20:24:560
251623599458451,4cyclictest2541852-21turbostat20:24:560
251624599456449,4cyclictest2552578-21turbostat20:50:003
251624399456448,5cyclictest2522159-21turbostat19:29:562
251623899456449,4cyclictest2607782-21turbostat23:29:561
251623899456449,4cyclictest2607782-21turbostat23:29:561
251624399455424,28cyclictest2614796-21turbostat23:49:562
251623599455418,19cyclictest2558373-21turbostat21:09:560
251623599455418,19cyclictest2558373-21turbostat21:09:550
251624399454444,7cyclictest2572261-21turbostat21:49:562
251624599453446,4cyclictest2593327-21turbostat22:45:003
251624599453446,4cyclictest2593327-21turbostat22:44:593
251623899452423,15cyclictest2632126-21turbostat00:39:561
251624599449442,4cyclictest2607782-21turbostat23:25:013
251623899448441,4cyclictest2526241-21turbostat19:39:551
251623599447440,4cyclictest2623469-21turbostat00:14:550
251624399444415,26cyclictest2528012-21turbostat19:44:542
251623599444434,7cyclictest2540116-21turbostat20:19:550
251624599443432,8cyclictest2570524-21turbostat21:40:003
251624599441434,4cyclictest2543585-21turbostat20:29:553
251623899441434,4cyclictest2625199-21turbostat00:19:561
251624599438431,4cyclictest2524205-21turbostat19:30:013
251624399436429,4cyclictest2545314-21turbostat20:34:552
251624599434414,17cyclictest2598515-21turbostat23:04:553
251623599434427,4cyclictest2560109-21turbostat21:14:550
251624599431411,17cyclictest2589878-21turbostat22:39:553
251624599430420,6cyclictest2601984-21turbostat23:09:593
251624399430423,4cyclictest2536656-21turbostat20:09:542
251624599428420,5cyclictest2606055-21turbostat23:20:013
251624599427404,19cyclictest2531463-21turbostat19:50:013
251624599424417,4cyclictest2584683-21turbostat22:20:003
251624599424417,4cyclictest2568758-21turbostat21:35:003
251624599424417,4cyclictest2568758-21turbostat21:35:003
251624399424397,24cyclictest2570524-21turbostat21:44:552
251623899424417,4cyclictest2565295-21turbostat21:29:561
251623899424415,6cyclictest2611250-21turbostat23:39:541
251624399420412,5cyclictest2531463-21turbostat19:54:562
251624599418411,4cyclictest2579482-21turbostat22:05:003
251623899418411,4cyclictest2520431-21turbostat19:24:541
251624599417410,4cyclictest2616527-21turbostat23:50:003
251624599416409,4cyclictest2543585-21turbostat20:25:003
251624599416409,4cyclictest2543585-21turbostat20:24:593
251624399416409,4cyclictest2524205-21turbostat19:34:562
251624399416409,4cyclictest2524205-21turbostat19:34:562
251624399416407,6cyclictest2596798-21turbostat22:59:552
251624599414406,5cyclictest2526241-21turbostat19:35:003
251624599414406,5cyclictest2526241-21turbostat19:34:593
251624599410403,4cyclictest2628663-21turbostat00:29:553
251624599410391,16cyclictest2567031-21turbostat21:30:003
251624599409399,7cyclictest2614796-21turbostat23:45:013
251623599407400,4cyclictest2593327-21turbostat22:49:550
251623899402394,5cyclictest2554315-21turbostat20:59:541
251624599400393,4cyclictest2533194-21turbostat19:59:563
251624599400393,4cyclictest2533194-21turbostat19:59:553
251624599397389,5cyclictest2623469-21turbostat00:10:003
251624599392382,6cyclictest2573994-21turbostat21:50:003
251624599388381,4cyclictest2549075-21turbostat20:44:553
251624399382374,5cyclictest2604026-21turbostat23:19:542
251624599380373,4cyclictest2588135-21turbostat22:30:003
251624599378369,6cyclictest2598515-21turbostat23:00:003
251624599377355,19cyclictest2549075-21turbostat20:40:003
251624599376369,4cyclictest2595066-21turbostat22:50:003
251624599375367,5cyclictest2554315-21turbostat20:55:003
251624599373367,3cyclictest2522159-21turbostat19:25:003
251624599370363,4cyclictest2558373-21turbostat21:05:003
251623599369354,11cyclictest2630390-21idleruntime-cro00:30:000
251624599367360,4cyclictest2575724-21turbostat21:55:013
251624599367359,5cyclictest2586407-21turbostat22:25:003
251624399367360,4cyclictest2517909-21turbostat19:19:542
251624599364356,5cyclictest2601984-21turbostat23:14:553
251624599363357,3cyclictest2632126-21turbostat00:35:013
251624599363310,36cyclictest2619992-21turbostat00:00:003
251624399363355,5cyclictest2552578-21turbostat20:54:552
251624399361353,5cyclictest2595066-21turbostat22:54:552
251624599358352,3cyclictest2516134-21turbostat19:14:553
251624599358348,7cyclictest2563566-21turbostat21:20:003
251624599357351,4cyclictest2621739-21turbostat00:05:003
251624599357350,4cyclictest2520431-21turbostat19:20:003
251623599357350,4cyclictest2556346-21turbostat21:04:550
251624599350344,4cyclictest2628663-21turbostat00:25:003
251624399350342,5cyclictest2534929-21turbostat20:04:542
251624599347341,3cyclictest2565295-21turbostat21:25:003
251624599346338,5cyclictest2540116-21turbostat20:15:013
251624599345339,3cyclictest2633859-21turbostat00:40:003
251624599345337,5cyclictest2625199-21turbostat00:15:003
251624599339332,4cyclictest2581214-21turbostat22:14:543
251624599338332,3cyclictest2577453-21turbostat22:00:003
251623899335328,4cyclictest2575724-21turbostat21:59:551
251623899333325,5cyclictest2618259-21turbostat23:59:551
251624599332324,5cyclictest2536656-21turbostat20:05:003
251623899332326,3cyclictest2577453-21turbostat22:04:541
251623599326319,4cyclictest2561837-21turbostat21:19:550
251623599325318,4cyclictest2550812-21turbostat20:49:560
251624599324317,4cyclictest2589878-21turbostat22:35:013
251624599323316,4cyclictest2533194-21turbostat19:55:013
251624599322316,3cyclictest2596798-21turbostat22:55:003
251623599319312,4cyclictest2626936-21turbostat00:24:560
251624399313307,3cyclictest2579482-21turbostat22:09:552
251623899311304,4cyclictest2584683-21turbostat22:24:551
251624599262247,11cyclictest2609521-21turbostat23:29:593
251624599262247,11cyclictest2609521-21turbostat23:29:593
251624599261247,10cyclictest2538386-21turbostat20:10:003
251624599244217,23cyclictest2561837-21turbostat21:15:003
251624599242227,10cyclictest2572261-21turbostat21:45:003
251624599237223,10cyclictest2626936-21turbostat00:20:003
251624399213206,4cyclictest2563566-21turbostat21:24:552
251624599174161,9cyclictest2528012-21turbostat19:39:593
251624599170156,10cyclictest2529737-21turbostat19:45:003
2515886210975,24sleep20-21swapper/219:07:162
2515825210977,22sleep30-21swapper/319:06:283
2514621210975,24sleep00-21swapper/019:05:040
261535621080,6sleep00-21swapper/023:45:140
2516042210774,23sleep10-21swapper/119:09:161
25162459910291,8cyclictest2547348-21turbostat20:35:003
258019621010,5chrt0-21swapper/122:05:191
26070772990,5sleep10-21swapper/123:20:241
2516245999891,4cyclictest2541852-21turbostat20:20:003
2516238999891,4cyclictest2568758-21turbostat21:39:541
2516238999689,4cyclictest2547348-21turbostat20:39:551
26176612670,7chrt4497-21PK-Backend23:50:272
251623599641,37cyclictest2522658-21seq19:25:080
251623599633,37cyclictest2632370-21acpi00:35:100
251623599631,36cyclictest2533642-21cpuspeed_turbos19:55:130
251623599631,36cyclictest2533642-21cpuspeed_turbos19:55:120
251623599621,32cyclictest2555033-21cstates20:55:120
2516235996114,24cyclictest2563984-21cpuspeed_turbos21:20:100
251623599601,31cyclictest2574706-21/usr/sbin/munin21:50:190
251623599601,29cyclictest2546833-21sed20:30:290
197399600,8rtkit-daemon1972-21rtkit-daemon23:03:251
251623599591,36cyclictest2526504-21grep19:35:130
251623599591,32cyclictest2530940-21sed19:45:290
197399590,9rtkit-daemon1972-21rtkit-daemon20:25:392
197399590,9rtkit-daemon1972-21rtkit-daemon19:11:412
251623599581,31cyclictest2630933-21cat00:30:140
197399580,9rtkit-daemon1972-21rtkit-daemon20:14:022
197399580,8rtkit-daemon1972-21rtkit-daemon23:31:541
251623599571,34cyclictest2598756-21taskset23:00:040
251623599571,31cyclictest2616759-21seq23:50:100
197399570,8rtkit-daemon1972-21rtkit-daemon21:18:172
251623599562,29cyclictest2610596-21processes23:30:280
251623599561,29cyclictest2601527-21cat23:05:300
197399560,8rtkit-daemon1972-21rtkit-daemon21:32:332
197399560,8rtkit-daemon1972-21rtkit-daemon21:32:332
197399560,8rtkit-daemon1972-21rtkit-daemon20:56:122
251623599551,32cyclictest2570002-21timerandwakeup21:35:290
197399550,8rtkit-daemon1972-21rtkit-daemon21:35:232
2516235995411,21cyclictest0-21swapper/021:44:490
251623599532,26cyclictest2608415-21df23:25:140
251623599532,26cyclictest2608415-21df23:25:140
251623599532,24cyclictest2577445-21apt-get22:00:060
251623599531,28cyclictest2626332-21python300:15:280
251623599522,26cyclictest2519320-21fw_packets19:15:190
251623599522,24cyclictest2547985-21diskstats20:35:160
251623599522,23cyclictest2592263-21hddtemp_smartct22:40:160
251623599522,23cyclictest2592263-21hddtemp_smartct22:40:160
251623599521,28cyclictest2567508-21cat21:30:140
251623599521,28cyclictest2567508-21cat21:30:140
251623599511,28cyclictest2537220-21cstates20:05:130
251623599511,24cyclictest2544590-21ntpq20:25:250
251623599502,26cyclictest2528646-21diskstats19:40:170
251623599501,29cyclictest1137-21runrttasks19:30:240
251623599501,29cyclictest1137-21runrttasks19:30:240
251623599501,28cyclictest0-21swapper/023:20:010
251623599501,27cyclictest924-21NetworkManager21:46:020
2516238994920,24cyclictest2565278-21cron21:25:011
2516235994918,27cyclictest2603723-21gdbus23:11:320
251623599491,26cyclictest2587600-21cat22:25:290
251623599482,41cyclictest2612371-21gunzip23:35:270
2516235994815,28cyclictest2597679-21ls22:55:190
2516238994715,13cyclictest2616053-21sed23:45:301
2516235994719,24cyclictest1137-21runrttasks22:10:280
2516235994714,27cyclictest2589079-21cut22:30:210
197399470,31rtkit-daemon2530379-21diskstats19:45:171
2516238994614,12cyclictest141rcu_preempt19:30:021
2516235994618,24cyclictest2579721-21/usr/sbin/munin22:05:110
251623599461,27cyclictest0-21swapper/020:00:140
197399460,39rtkit-daemon26-21ksoftirqd/123:42:161
2516235994517,23cyclictest2565976-21idleruntime21:25:190
251623599441,39cyclictest1137-21runrttasks21:59:450
251623599441,38cyclictest2596218-21switchtime22:50:290
2516235994412,28cyclictest2531917-21cpuspeed_turbos19:50:120
251624399434,12cyclictest0-21swapper/220:25:012
251624399434,12cyclictest0-21swapper/220:25:002
2516238994314,12cyclictest141rcu_preempt21:05:271
2516238994314,12cyclictest141rcu_preempt21:05:271
2516238994314,12cyclictest141rcu_preempt20:50:001
251623599432,35cyclictest2585523-21memory22:20:210
2516238994212,12cyclictest141rcu_preempt20:30:011
251623599429,28cyclictest2516498-21cpuspeed19:10:140
2516235994210,27cyclictest2622875-21python300:05:270
2516235994210,27cyclictest2521332-21cat19:20:210
197399420,33rtkit-daemon2516229-21cyclictest00:28:141
2516243994124,5cyclictest34-21ksoftirqd/223:15:002
2516243994113,7cyclictest141rcu_preempt22:10:222
2516235994114,23cyclictest2550374-21timerwakeupswit20:40:280
251623599411,35cyclictest2584006-21ps22:15:280
2516235994113,24cyclictest2553561-21grep20:50:230
251623599411,17cyclictest25902782cpuspeed_turbos22:35:130
2516243994016,8cyclictest2620016-40updatedb.mlocat00:00:012
2516243994010,6cyclictest141rcu_preempt23:38:252
2516238994023,13cyclictest2603703-40lsb_release23:11:281
2516238994021,14cyclictest2540750-21df_inode20:15:161
2516238994017,20cyclictest2584703-21unin-run22:20:011
251623599408,27cyclictest2619448-21switchtime23:55:290
197399400,32rtkit-daemon4497-21PK-Backend21:10:002
197399400,32rtkit-daemon4497-21PK-Backend21:09:592
2516243993924,11cyclictest2567008-21cron21:30:002
2516243993922,5cyclictest34-21ksoftirqd/219:47:392
2516243993921,6cyclictest34-21ksoftirqd/223:44:182
2516243993921,6cyclictest34-21ksoftirqd/223:27:542
2516243993921,6cyclictest34-21ksoftirqd/223:27:532
2516243993921,6cyclictest34-21ksoftirqd/221:50:282
2516243993914,5cyclictest141rcu_preempt22:20:212
2516243993913,6cyclictest141rcu_preempt23:05:302
2516243993913,6cyclictest141rcu_preempt22:02:422
2516243993912,6cyclictest141rcu_preempt19:35:322
2516243993911,14cyclictest141rcu_preempt20:40:262
2516243993910,11cyclictest141rcu_preempt23:20:102
2516238993913,12cyclictest141rcu_preempt22:30:011
2516238993912,12cyclictest141rcu_preempt23:55:031
2516238993911,12cyclictest141rcu_preempt19:32:321
2516238993911,12cyclictest141rcu_preempt19:32:321
25538182380,5sleep14497-21PK-Backend20:50:281
2516243993812,6cyclictest141rcu_preempt00:25:002
2516243993812,6cyclictest141rcu_preempt00:10:212
2516243993811,7cyclictest141rcu_preempt20:35:232
2516243993811,6cyclictest141rcu_preempt23:30:192
2516243993811,6cyclictest141rcu_preempt22:37:592
2516243993810,7cyclictest141rcu_preempt23:00:202
2516238993822,12cyclictest2561820-21cron21:14:591
2516238993822,12cyclictest2546320-21pool-udisksd20:30:211
2516238993820,14cyclictest26-21ksoftirqd/122:49:591
251623899381,33cyclictest2590932-21which22:35:271
197399380,24rtkit-daemon0-21swapper/120:01:451
2516243993720,12cyclictest2592311-21cat22:40:162
2516243993720,12cyclictest2592311-21cat22:40:162
2516243993713,6cyclictest141rcu_preempt19:20:122
2516243993712,6cyclictest141rcu_preempt22:15:302
2516243993712,6cyclictest141rcu_preempt21:55:292
2516243993712,6cyclictest141rcu_preempt00:35:022
2516243993711,6cyclictest141rcu_preempt22:45:272
251623899377,6cyclictest141rcu_preempt00:10:141
2516238993718,14cyclictest2557234-21df_inode21:00:161
197399370,24rtkit-daemon0-21swapper/119:55:061
197399370,24rtkit-daemon0-21swapper/119:55:061
2516243993626,7cyclictest2551720-21needreboot20:45:212
2516243993619,5cyclictest34-21ksoftirqd/221:14:072
2516243993618,6cyclictest34-21ksoftirqd/200:09:362
2516243993611,6cyclictest141rcu_preempt21:00:142
2516243993610,6cyclictest141rcu_preempt00:30:012
251623899364,12cyclictest2600812-21expr23:05:151
2516238993620,12cyclictest2517678-21sshd19:12:171
2516238993619,12cyclictest2597424-21df22:55:121
197399360,24rtkit-daemon0-21swapper/119:52:321
197399360,24rtkit-daemon0-21swapper/119:40:331
25817772350,5sleep10-21swapper/122:10:141
2516243993529,4cyclictest0-21swapper/200:38:232
2516243993519,12cyclictest2626040-21load00:15:212
2516243993513,6cyclictest141rcu_preempt19:55:142
2516243993513,6cyclictest141rcu_preempt19:55:132
2516243993511,6cyclictest141rcu_preempt20:17:272
2516238993521,10cyclictest2563309-21cron21:16:591
2516238993521,10cyclictest2519874-21threads19:15:301
2516238993519,12cyclictest2574790-21cat21:50:211
2516238993512,6cyclictest141rcu_preempt20:05:111
251623899351,13cyclictest2539072-21basename20:10:181
251624399349,6cyclictest141rcu_preempt00:04:572
251623899349,6cyclictest141rcu_preempt22:54:591
2516238993419,11cyclictest2620723-21cat00:00:191
2516238993418,12cyclictest3692989-21snapd00:21:351
2516238993417,13cyclictest2549325-21find20:40:111
2516238993417,12cyclictest2571110-21expr21:40:131
2516238993410,5cyclictest141rcu_preempt20:20:221
2516238993410,5cyclictest141rcu_preempt20:20:221
197399340,27rtkit-daemon0-21swapper/123:16:001
251624399331,29cyclictest4497-21PK-Backend22:32:412
251623899332,10cyclictest2572523-21apt21:45:111
251623899331,9cyclictest0-21swapper/122:41:081
251623899331,9cyclictest0-21swapper/122:41:081
2516238993317,11cyclictest2630853-21ls00:30:131
2516243993216,12cyclictest2587623-21grep22:25:292
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional