You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-29 - 12:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 100 highest latencies:
System rack3slot3s.osadl.org (updated Fri Aug 29, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
331844122650,3sleep2411rcuc/223:30:202
29679782151107,15sleep20-21swapper/219:09:002
29680292149102,15sleep30-21swapper/319:09:413
298047821460,1sleep0221rcuc/019:20:150
29677572146103,34sleep00-21swapper/019:06:130
320282521400,2sleep00-21swapper/022:19:270
29677782130107,15sleep10-21swapper/119:06:301
333675221250,2sleep20-21swapper/223:42:552
2968186992925,2cyclictest3187025-21idleruntime-cro22:10:003
2968186992924,3cyclictest0-21swapper/322:30:003
2968186992717,6cyclictest3262650-21ls22:55:203
2968186992621,2cyclictest0-21swapper/322:20:243
2968186992619,3cyclictest0-21swapper/322:45:113
2968186992521,2cyclictest0-21swapper/323:50:003
2968186992521,2cyclictest0-21swapper/320:25:283
2968186992420,2cyclictest0-21swapper/323:35:133
2968186992420,2cyclictest0-21swapper/321:55:013
2968186992414,8cyclictest0-21swapper/323:55:113
296818699241,19cyclictest3005084-21idleruntime-cro19:45:003
296818699239,10cyclictest3171042-21awk22:00:003
296818699232,7cyclictest3128890-21sh21:31:563
2968186992318,3cyclictest3106693-21cron21:20:013
2968186992312,7cyclictest3402621-21tr00:25:193
296818699231,19cyclictest3310544-21ntpq23:25:213
296818699231,18cyclictest3378300-21kthreadcore00:10:153
296818699231,18cyclictest3044224-21/usr/sbin/munin20:20:213
296818699221,8cyclictest3336418-21sh23:42:243
2968186992218,2cyclictest943-21runrttasks21:37:333
296818699221,7cyclictest3112569-21sh21:21:533
296818699221,7cyclictest3020506-21diskstats20:00:153
296818699221,7cyclictest3009113-21ntpq19:45:203
2968186992216,3cyclictest0-21swapper/322:35:013
2968186992213,7cyclictest0-21swapper/322:40:203
2968186992213,6cyclictest3098671-21cron21:15:003
2968186992211,8cyclictest3297687-21ssh23:18:583
2968186992211,7cyclictest3230464-21cut22:35:183
2968186992210,8cyclictest3417817-21cut00:35:263
32424922211,2sleep10-21swapper/122:44:131
296818699211,7cyclictest3019679-21/usr/sbin/munin19:55:283
2968186992117,2cyclictest0-21swapper/321:00:003
2968186992112,8cyclictest3377284-21ssh00:09:403
2968186992112,7cyclictest0-21swapper/322:15:233
2968186992111,7cyclictest736-21gdbus20:30:243
2968186992110,7cyclictest3412562-21seq00:33:113
296818699210,8cyclictest3319767-21chrt23:31:223
296818699210,17cyclictest3175984-21sh22:00:323
34182312200,11sleep243-21ksoftirqd/200:35:302
296818699208,9cyclictest2989142-21ntpq19:25:213
296818699201,7cyclictest3089849-21/usr/sbin/munin21:05:243
2968186992017,2cyclictest3191493-21ntp_kernel_err22:10:193
2968186992016,3cyclictest511ktimers/323:08:263
2968186992015,3cyclictest3084916-21proc_pri21:00:243
296818699201,17cyclictest3272127-21runrttasks23:01:393
296818699201,17cyclictest0-21swapper/300:00:373
296818699201,16cyclictest3289082-21sh23:12:573
296818699201,16cyclictest3153078-21sh21:47:163
296818699201,16cyclictest3153078-21sh21:47:163
296818699201,16cyclictest3116057-21grep21:25:173
2968186992010,7cyclictest3399624-21basename00:24:593
296818699200,8cyclictest3059479-21ntpq20:35:223
296818699200,18cyclictest3256087-21runrttasks22:51:003
296818699200,17cyclictest943-21runrttasks23:52:263
296818699200,17cyclictest3302885-21sh23:20:243
296818699198,8cyclictest3387908-21cut00:15:183
2968186991913,3cyclictest31440662sleep321:40:433
2968186991913,3cyclictest31440662sleep321:40:423
2968186991912,5cyclictest3039603-21grep20:15:263
296818699191,15cyclictest3060697-21/usr/sbin/munin20:40:123
2968186991911,5cyclictest2976519-21cut19:15:173
296818699188,8cyclictest2972182-21ntpq19:10:233
2968186991813,3cyclictest2997696-21kthreadcore19:35:173
296818699181,15cyclictest3030486-21tr20:10:143
31457022170,11sleep00-21swapper/021:43:080
31457022170,11sleep00-21swapper/021:43:070
2968186991713,3cyclictest2990800-21kthreadcore19:30:153
296818699171,14cyclictest2983955-21/usr/sbin/munin19:20:193
32992422161,11sleep10-21swapper/123:20:181
31281232161,2sleep00-21swapper/021:30:500
2968186991611,3cyclictest3071051-21gltestperf20:50:173
296818699161,13cyclictest3025696-21head20:05:153
296818699161,13cyclictest0-21swapper/319:50:153
31370752151,2sleep00-21swapper/021:37:180
296818699157,6cyclictest3065712-21munin-node20:45:123
296818699157,6cyclictest3065712-21munin-node20:45:113
32637362141,7sleep03090731-21diskmemload22:55:510
32354612141,7sleep03090731-21diskmemload22:40:120
34179932131,7sleep13418230-21awk00:35:291
34051692131,7sleep23090731-21diskmemload00:27:062
33995812131,8sleep13399593-21cron00:24:591
33835882131,2sleep2191rcu_preempt00:12:502
33742262131,8sleep03374275-21ntpq00:05:230
33446332131,7sleep23090731-21diskmemload23:47:162
33365682130,1sleep10-21swapper/123:42:391
33288052131,8sleep03090731-21diskmemload23:37:560
33221382130,9sleep03322150-21sh23:34:510
32432862132,4sleep23243310-21apt-get22:45:122
32242852131,8sleep13224295-21ssh22:30:381
32211362131,7sleep00-21swapper/022:30:210
32168262131,7sleep23090731-21diskmemload22:26:312
31389012131,7sleep13138941-21munin-run21:39:591
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional