You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-07 - 02:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack3slot3s.osadl.org (updated Mon Jul 07, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
287997122280,2sleep00-21swapper/021:45:050
27013702146100,15sleep30-21swapper/319:09:143
2701162214499,16sleep00-21swapper/019:06:340
2701080214498,15sleep20-21swapper/219:05:322
2701342213395,12sleep10-21swapper/119:08:521
2701554994228,10cyclictest2895738-21grep21:55:153
2701554992723,2cyclictest0-21swapper/322:05:153
2701554992621,3cyclictest0-21swapper/322:29:593
2701554992522,2cyclictest2958880-21gltestperf22:35:153
2701554992521,3cyclictest3048082-21grep23:30:203
2701554992521,3cyclictest3048082-21grep23:30:193
2701554992521,2cyclictest2934265-21ls22:19:593
29632472241,10sleep02824483-21diskmemload22:35:500
270155499241,9cyclictest2880277-21fschecks_time21:45:143
270155499241,7cyclictest2878381-21runrttasks21:42:543
2701554992415,6cyclictest2829407-21grep21:10:303
2701554992412,8cyclictest262-21systemd-journal22:32:083
2701554992320,2cyclictest2709169-21kthreadcore19:15:173
2701554992319,2cyclictest0-21swapper/320:50:133
2701554992319,2cyclictest0-21swapper/319:25:233
2701554992319,2cyclictest0-21swapper/300:35:243
2701554992319,2cyclictest0-21swapper/300:05:283
270155499231,7cyclictest2870277-21ssh21:37:323
2701554992316,5cyclictest0-21swapper/321:15:293
2701554992312,7cyclictest2935532-21cut22:20:173
29259092221,8sleep22824483-21diskmemload22:14:112
28542262221,3sleep00-21swapper/021:26:470
28542262221,3sleep00-21swapper/021:26:460
2701554992219,2cyclictest2733496-21systemctl19:40:003
2701554992218,2cyclictest0-21swapper/322:55:193
270155499221,7cyclictest945-21snmpd21:30:123
2701554992217,3cyclictest3059552-21idleruntime-cro23:39:593
2701554992214,6cyclictest3061535-21cut23:40:183
2701554992213,6cyclictest2702065-21grep19:10:153
2701554992213,6cyclictest2702065-21grep19:10:143
2701554992212,6cyclictest3078587-21tr23:50:193
270155499221,19cyclictest3064988-21kworker/u16:1+rpciod00:13:073
2701554992211,7cyclictest3135355-21runrttasks00:28:503
2701554992211,7cyclictest3096165-21cat00:00:283
2701554992210,8cyclictest2923077-21grep22:10:223
270155499216,11cyclictest2907272-21sshd22:00:213
2701554992118,2cyclictest2798453-21needreboot20:40:213
2701554992117,2cyclictest2763803-21cron20:09:593
2701554992111,7cyclictest3090579-21sh23:58:563
2701554992110,7cyclictest2997622-21date23:00:003
270155499210,18cyclictest3013147-21ssh23:10:053
270155499210,18cyclictest2779759-21cut20:25:143
270155499206,2cyclictest2968187-21kthreadcore22:40:183
270155499205,2cyclictest3041883-21sh23:26:443
270155499202,14cyclictest945-21snmpd21:50:013
2701554992016,2cyclictest2819480-21latency_hist21:04:593
2701554992016,2cyclictest0-21swapper/323:25:003
2701554992016,2cyclictest0-21swapper/320:25:013
2701554992015,3cyclictest726-21in:imuxsock23:10:013
2701554992012,5cyclictest3021493-21ssh23:15:183
270155499199,7cyclictest2738920-21df_inode19:45:123
270155499198,7cyclictest2850888-21cut21:25:193
270155499198,7cyclictest2850888-21cut21:25:183
2701554991914,3cyclictest3140638-21unixbench_multi00:30:283
2701554991914,3cyclictest2848505-21latency_hist21:25:003
2701554991914,3cyclictest2824477-21cron21:09:593
2701554991911,6cyclictest3119667-21ssh00:15:453
270155499191,16cyclictest2824483-21diskmemload00:21:263
2701554991910,6cyclictest2974208-21/usr/sbin/munin22:45:233
2701554991910,6cyclictest2803313-21runrttasks20:45:213
270155499190,8cyclictest2794218-21apt-get20:40:013
270155499190,17cyclictest3073096-21ssh23:46:253
28551782181,2sleep10-21swapper/121:28:201
28551782181,2sleep10-21swapper/121:28:191
270155499189,6cyclictest2788161-21/usr/sbin/munin20:30:193
270155499188,7cyclictest736-21gdbus20:10:243
2701554991814,2cyclictest720-21tpm2-abrmd20:00:023
2701554991814,2cyclictest262-21systemd-journal19:45:003
2701554991814,2cyclictest0-21swapper/319:30:223
2701554991813,3cyclictest2749024-21gltestperf19:55:153
2701554991810,6cyclictest2987932-21sh22:52:283
29544952179,4sleep22954560-21ntpq22:30:232
270155499179,6cyclictest2717704-21mailstats19:20:243
270155499170,15cyclictest2814278-21taskset20:58:303
31418802161,3sleep02824483-21diskmemload00:33:070
30818722161,1sleep00-21swapper/023:52:460
28867482161,8sleep22824483-21diskmemload21:49:012
28464402163,2sleep1191rcu_preempt21:21:431
270155499168,5cyclictest262-21systemd-journal20:16:073
30489892151,3sleep22824483-21diskmemload23:30:442
30489892151,3sleep22824483-21diskmemload23:30:432
30044652151,10sleep03004478-21rm23:04:070
29642692151,8sleep22824483-21diskmemload22:37:312
29415922150,2sleep10-21swapper/122:23:481
29247392150,3sleep10-21swapper/122:12:171
27325142151,9sleep12732559-21ntpq19:35:211
270155499156,6cyclictest2744362-21cut19:50:173
30428242141,2sleep10-21swapper/123:28:171
28959092141,9sleep02897221-21tr21:55:160
28614542141,7sleep22824483-21diskmemload21:30:572
28309352141,2sleep00-21swapper/021:12:400
27033912140,3sleep20-21swapper/219:10:172
27033912140,3sleep20-21swapper/219:10:162
31476922131,8sleep23147704-21ssh00:35:372
31274852131,8sleep13127495-21ssh00:21:061
31145792131,8sleep23114773-21ssh00:15:012
30664752131,8sleep13066489-21ssh23:43:291
30650312131,8sleep02824483-21diskmemload23:41:190
30093162131,2sleep20-21swapper/223:05:262
29867472131,2sleep10-21swapper/122:50:371
29483582131,8sleep22948372-21ssh22:27:052
29479302131,5sleep00-21swapper/022:26:280
30956992120,1sleep10-21swapper/100:00:241
30278462121,3sleep10-21swapper/123:19:121
30178182121,7sleep03017824-21ssh23:10:340
29881432121,4sleep20-21swapper/222:52:472
29812362121,7sleep12981246-21ssh22:49:181
29247282121,7sleep02824483-21diskmemload22:12:170
29010052120,8sleep12901016-21sh21:56:581
28956052121,7sleep22895824-21hddtemp_smartct21:55:142
28776752121,6sleep02824483-21diskmemload21:41:460
27590502122,5sleep12759069-21apt-get20:05:191
27378782121,7sleep02733760-21/usr/sbin/munin19:40:300
31416212110,5sleep12824483-21diskmemload00:32:321
31334882110,7sleep23129681-21/usr/sbin/munin00:25:242
30967642111,3sleep00-21swapper/000:01:110
30912412111,6sleep13091379-21latency_hist00:00:011
30897672111,3sleep20-21swapper/223:57:382
30257842111,2sleep20-21swapper/223:15:572
30189542110,7sleep23018966-21ssh23:12:232
29971542111,5sleep22824483-21diskmemload22:59:452
29953862111,3sleep10-21swapper/122:56:521
29714112111,3sleep10-21swapper/122:41:191
29325922110,3sleep10-21swapper/122:17:261
29161742111,3sleep20-21swapper/222:06:052
29095192111,3sleep10-21swapper/122:02:561
28780892111,3sleep10-21swapper/121:42:271
28746892110,3sleep2411rcuc/221:40:182
28715582114,3sleep1321rcuc/121:39:301
28648352111,7sleep02865248-21cut21:35:160
28624082111,3sleep10-21swapper/121:32:361
28560012111,3sleep20-21swapper/221:29:422
28560012111,3sleep20-21swapper/221:29:412
28457982111,3sleep20-21swapper/221:20:402
28411602111,6sleep02824483-21diskmemload21:20:170
28248172111,3sleep20-21swapper/221:10:092
28234352111,5sleep22823486-21ntpq21:05:202
28151052111,6sleep22815411-21kthreadcore21:00:202
28035442111,7sleep22803592-21/usr/sbin/munin20:45:252
27834842111,6sleep02783743-21head20:25:270
27449522111,5sleep22746693-21kthreadcore19:50:182
31479532100,2sleep10-21swapper/100:36:121
31407942101,3sleep20-21swapper/200:30:412
31344552101,2sleep00-21swapper/000:26:370
31208492100,6sleep13120854-21ssh00:17:451
31128752100,1sleep00-21swapper/000:12:150
31118202101,2sleep20-21swapper/200:10:272
31061232101,3sleep22824483-21diskmemload00:09:122
30991202101,3sleep10-21swapper/100:05:021
30898802101,3sleep02824483-21diskmemload23:57:480
30646332101,3sleep20-21swapper/223:40:412
30570332101,3sleep10-21swapper/123:36:081
30503422101,3sleep10-21swapper/123:32:511
30503422101,3sleep10-21swapper/123:32:501
30429562101,3sleep00-21swapper/023:28:280
30423512101,3sleep22824483-21diskmemload23:27:312
30353632101,4sleep22824483-21diskmemload23:23:412
30338832101,3sleep10-21swapper/123:21:211
30271112101,3sleep02824483-21diskmemload23:17:580
30131442101,2sleep10-21swapper/123:10:051
30120842101,3sleep02824483-21diskmemload23:08:490
30092502101,4sleep13009315-21ssh23:05:251
30025042101,3sleep20-21swapper/223:00:522
30024522101,6sleep12824483-21diskmemload23:00:471
29790692101,3sleep22824483-21diskmemload22:45:542
29734062101,3sleep22824483-21diskmemload22:44:212
29714912101,3sleep02824483-21diskmemload22:41:260
29630382101,2sleep10-21swapper/122:35:311
29505712101,4sleep12950592-21apt-get22:30:131
29474122101,3sleep10-21swapper/122:25:411
29402572101,2sleep20-21swapper/222:21:412
29339592101,3sleep22824483-21diskmemload22:19:322
29314502101,2sleep00-21swapper/022:15:410
29172132101,4sleep02824483-21diskmemload22:07:470
29165802101,3sleep10-21swapper/122:06:441
29090222100,3sleep00-21swapper/022:02:110
28932052100,1sleep10-21swapper/121:52:021
28882482101,6sleep22889682-21kthreadcore21:50:182
28804822100,6sleep12881200-21kthreadcore21:45:151
28708692101,4sleep22824483-21diskmemload21:38:282
28392212101,2sleep20-21swapper/221:17:382
28331022101,3sleep12824483-21diskmemload21:15:091
28324002101,3sleep10-21swapper/121:14:531
27843992101,4sleep12784418-21apt-get20:30:121
27830472101,4sleep12783128-21ntpq20:25:201
27743942100,6sleep02774372-21/usr/sbin/munin20:20:130
27384892101,4sleep22738507-21munin-run19:45:012
3147993291,3sleep00-21swapper/000:36:190
3133435291,4sleep13133486-21ntpq00:25:221
3122486291,3sleep02824483-21diskmemload00:20:040
3121722291,3sleep02824483-21diskmemload00:19:150
3113433291,2sleep10-21swapper/100:13:111
3104056291,3sleep02824483-21diskmemload00:05:370
3077650291,4sleep13079375-21kthreadcore23:50:181
3074665290,3sleep10-21swapper/123:48:551
3074514290,3sleep02824483-21diskmemload23:48:420
3072652290,1sleep20-21swapper/223:45:472
3058263291,3sleep02824483-21diskmemload23:38:030
3051965291,3sleep22824483-21diskmemload23:35:052
3050761291,3sleep02824483-21diskmemload23:33:300
3050761291,3sleep02824483-21diskmemload23:33:290
3034025291,2sleep00-21swapper/023:21:360
2989959291,4sleep02990114-21df_inode22:55:140
2988595291,2sleep00-21swapper/022:53:290
2979145291,2sleep00-21swapper/022:46:000
2956813290,2sleep00-21swapper/022:33:040
2934806291,2sleep00-21swapper/022:20:020
2909330291,2sleep20-21swapper/222:02:372
2892825291,3sleep02824483-21diskmemload21:51:240
2860837291,3sleep02824483-21diskmemload21:30:280
2839880291,2sleep02824483-21diskmemload21:18:400
2819751291,3sleep02819906-21diskstats21:05:130
2809687291,3sleep12809706-21apt-get20:55:131
2767860291,4sleep22767907-21ntpq20:10:222
2738748291,3sleep12738767-21apt-get19:45:111
2733418290,1sleep00-21swapper/019:39:210
2717483291,4sleep02717536-21ntp_states19:20:220
2710959291,4sleep22712280-21memory19:15:192
2706492291,4sleep02706756-21df19:15:000
2701729290,3sleep10-21swapper/119:10:131
2701729290,3sleep10-21swapper/119:10:121
3127644281,3sleep23127649-21ssh00:21:222
3098018281,2sleep20-21swapper/200:03:142
3080034281,4sleep23080302-21grep23:50:272
2794758280,1sleep20-21swapper/220:40:112
2779380280,3sleep20-21swapper/220:25:132
2774357280,2sleep10-21swapper/120:20:021
2769569281,3sleep02771061-21kthreadcore20:15:170
2768838281,3sleep02768970-21latency_hist20:15:000
2758492280,1sleep20-21swapper/220:01:242
2743228280,1sleep00-21swapper/019:47:220
2737447281,3sleep12737498-21ntpq19:40:231
2819907270,2sleep12820116-21hddtemp_smartct21:05:141
2814412271,3sleep02814598-21sed21:00:000
2809102270,1sleep10-21swapper/120:50:571
2804716270,2sleep20-21swapper/220:50:132
2799770270,2sleep02799789-21apt-get20:45:140
2794944271,3sleep10-21swapper/120:40:181
2793906271,3sleep22793907-21sshd20:36:232
2788206270,2sleep20-21swapper/220:30:212
2762732271,3sleep02762796-21ntp_192.168.11420:05:260
2758753270,1sleep00-21swapper/020:04:430
2748693271,2sleep10-21swapper/119:55:131
2743698270,2sleep10-21swapper/119:50:131
2739450271,3sleep22741136-21kthreadcore19:45:162
2728107270,1sleep10-21swapper/119:30:371
2727477270,3sleep22727531-21ntpq19:30:212
2723886271,3sleep02724100-21smartctl19:30:160
2723120270,1sleep10-21swapper/119:25:591
2819455260,1sleep10-21swapper/121:04:551
2811310261,2sleep20-21swapper/220:55:182
2788994260,1sleep00-21swapper/020:32:390
2779119261,2sleep20-21swapper/220:25:002
2767817260,1sleep10-21swapper/120:10:211
2763742260,1sleep20-21swapper/220:09:172
2754182261,2sleep10-21swapper/120:00:151
2753267260,1sleep20-21swapper/219:57:422
2743697260,1sleep00-21swapper/019:50:120
2732789261,2sleep20-21swapper/219:35:262
2723144260,1sleep00-21swapper/019:26:180
2718700261,2sleep20-21swapper/219:25:132
2717537261,2sleep10-21swapper/119:20:221
2717375261,2sleep22717423-21ntpq19:20:192
2708691261,2sleep10-21swapper/119:15:141
2817080250,1sleep00-21swapper/021:00:220
2794787250,1sleep00-21swapper/020:40:170
2793173250,1sleep10-21swapper/120:35:221
2752352250,1sleep00-21swapper/019:55:200
2705446250,1sleep00-21swapper/019:10:200
2705446250,1sleep00-21swapper/019:10:190
2804895240,1sleep00-21swapper/020:50:160
2799761240,1sleep10-21swapper/120:45:081
2789626240,1sleep00-21swapper/020:35:170
2769277240,1sleep20-21swapper/220:15:142
2768972240,1sleep10-21swapper/120:15:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional