You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 23:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Fri May 02, 2025 12:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2705679212162,55sleep30-21swapper/307:08:483
270561529157,10sleep20-21swapper/207:07:592
270552529054,10sleep10-21swapper/107:06:501
270556028050,6sleep00-21swapper/007:07:170
2705900993225,5cyclictest2796050-21awk10:00:002
2705900993127,2cyclictest2711432-21needreboot07:20:152
2705900993124,5cyclictest2718193-40tracker-extract07:36:472
2705900993023,5cyclictest2886354-21cron11:45:002
270590099294,20cyclictest2900401-21sh11:57:402
2705900992925,2cyclictest2776509-21cat09:35:242
2705900992824,2cyclictest2725053-21cut07:55:162
2705900992823,3cyclictest2929472-21idleruntime-cro12:25:002
2705900992823,3cyclictest2929472-21idleruntime-cro12:25:002
2705900992622,3cyclictest2822949-21sshd10:30:332
2705900992622,2cyclictest0-21swapper/210:45:002
2705900992620,3cyclictest0-21swapper/208:05:152
270590099258,14cyclictest2917521-21sh12:13:542
2705900992521,2cyclictest2705895-21cyclictest12:05:112
2705900992520,3cyclictest2844134-21unixbench_singl10:55:272
2705900992513,9cyclictest0-21swapper/211:05:122
2705903992420,2cyclictest0-21swapper/312:20:153
2705903992420,2cyclictest0-21swapper/312:20:153
2705900992419,3cyclictest2787420-21latency_hist09:50:002
2705900992419,3cyclictest2752627-21kworker/u16:3-ext4-rsv-conversion10:05:012
2705900992419,3cyclictest2710246-21cron07:20:002
2705900992417,3cyclictest2858831-21kworker/u16:3+xprtiod11:30:132
2705900992412,8cyclictest264-21systemd-journal09:25:132
2705903992320,1cyclictest0-21swapper/311:55:013
2705903992319,2cyclictest0-21swapper/310:10:113
270590399231,8cyclictest2899196-21taskset11:56:053
270590099232,18cyclictest2773200-21sh09:31:342
270590099232,18cyclictest2773200-21sh09:31:342
2705900992319,2cyclictest2941271-21missed_timers12:35:192
2705900992319,2cyclictest2888109-21grep11:45:232
2705900992319,2cyclictest2722482-21munin-plugin-st07:50:002
2705900992318,3cyclictest2738876-21cpuspeed_turbos08:30:132
2705900992318,3cyclictest2738876-21cpuspeed_turbos08:30:122
2705900992318,2cyclictest0-21swapper/210:50:142
2705900992317,3cyclictest2932770-21sh12:27:182
2705900992313,8cyclictest0-21swapper/209:50:262
270590099231,19cyclictest2834880-21/usr/sbin/munin10:45:162
270590099231,19cyclictest2808061-21sh10:14:272
270590099231,19cyclictest2747745-21ntp_states08:50:192
270590099231,19cyclictest2727909-21sed08:00:242
270590099227,2cyclictest0-21swapper/211:11:072
2705900992218,2cyclictest0-21swapper/211:20:182
270590099221,7cyclictest2765003-21sh09:20:542
270590099221,7cyclictest2761141-21sh09:16:312
2705900992217,3cyclictest2878059-21latency11:35:162
2705900992217,3cyclictest2850692-21idleruntime-cro11:05:012
2705900992216,4cyclictest0-21swapper/210:25:142
2705900992214,6cyclictest2754559-21diskmemload10:19:552
2705900992212,6cyclictest2417150-21ntpd10:35:182
270590099221,18cyclictest2803542-21ssh10:08:422
270590099221,18cyclictest2782331-21sh09:43:072
2705900992211,7cyclictest2919689-21ntp_states12:15:202
2705900992210,8cyclictest2860342-21wc11:15:242
270590399219,8cyclictest2777673-21taskset09:37:103
2705903992118,1cyclictest0-21swapper/310:40:173
2705900992117,2cyclictest2891924-21gltestperf11:50:142
2705900992117,2cyclictest0-21swapper/209:10:012
270590099211,16cyclictest2757563-21sh09:12:472
270590099210,18cyclictest2724214-21seq07:52:132
2705903992017,1cyclictest0-21swapper/310:30:003
2705903992016,2cyclictest0-21swapper/312:34:263
2705903992015,2cyclictest0-21swapper/309:55:013
270590399201,16cyclictest0-21swapper/311:02:273
2705903992011,4cyclictest28222842sleep310:30:233
270590099201,7cyclictest2871664-21ssh11:28:272
270590099201,6cyclictest2935406-21/usr/sbin/munin12:30:122
2705900992016,2cyclictest2813249-21cpuspeed_turbos10:20:132
2705900992014,5cyclictest2705895-21cyclictest12:00:162
270590099201,16cyclictest1-21systemd07:35:002
270590099201,16cyclictest1-21systemd07:34:592
270590399197,1cyclictest0-21swapper/310:00:263
2705903991916,1cyclictest0-21swapper/312:25:263
2705903991916,1cyclictest0-21swapper/308:10:253
2705903991915,2cyclictest0-21swapper/311:25:323
2705903991915,2cyclictest0-21swapper/311:18:223
2705903991915,2cyclictest0-21swapper/311:13:143
2705903991915,2cyclictest0-21swapper/310:55:163
2705903991915,2cyclictest0-21swapper/308:45:143
2705903991914,3cyclictest0-21swapper/311:35:133
2705903991911,6cyclictest0-21swapper/309:47:503
270590399191,14cyclictest28767312sleep311:34:213
270590399191,14cyclictest0-21swapper/309:12:433
2705900991914,3cyclictest2734971-21tr08:20:162
270590099191,13cyclictest2744986-21runrttasks08:45:122
2705900991910,6cyclictest2713695-21/usr/sbin/munin07:25:222
28384872182,5sleep12838506-21apt-get10:50:131
27060752180,7sleep02706094-21apt-get07:10:130
270590399189,6cyclictest2812780-21turbostat10:20:003
270590399189,6cyclictest2728394-21turbostat08:05:013
270590399188,7cyclictest2742457-21turbostat08:44:543
270590399182,11cyclictest2763310-21taskset09:20:013
2705903991815,1cyclictest0-21swapper/312:35:103
2705903991815,1cyclictest0-21swapper/310:38:263
2705903991815,1cyclictest0-21swapper/309:40:193
2705903991815,1cyclictest0-21swapper/309:30:203
2705903991815,1cyclictest0-21swapper/309:30:193
2705903991814,2cyclictest0-21swapper/308:35:003
2705903991814,2cyclictest0-21swapper/308:35:003
2705903991814,1cyclictest0-21swapper/311:45:213
2705903991811,4cyclictest27616622sleep309:17:303
270590399181,14cyclictest0-21swapper/312:17:593
270590399181,13cyclictest2815983-21chrt10:23:283
270590399181,13cyclictest2769856-21chrt09:28:003
270590399181,11cyclictest2913033-21taskset12:10:003
270590399181,11cyclictest2850929-21taskset11:05:013
270590399180,14cyclictest0-21swapper/312:10:553
270590399180,13cyclictest0-21swapper/308:05:123
270590099185,8cyclictest2741438-21gcc08:35:202
2705900991814,2cyclictest2733154-21gltestperf08:15:142
2705900991814,2cyclictest2706320-21fschecks_time07:10:152
270590099181,3cyclictest2749171-21/usr/sbin/munin08:55:162
29427402171,3sleep00-21swapper/012:36:160
29237012171,4sleep02923714-21ssh12:19:560
28721492171,11sleep02754559-21diskmemload11:29:170
27780572171,2sleep00-21swapper/009:38:000
2705903991714,1cyclictest0-21swapper/310:45:143
2705903991714,1cyclictest0-21swapper/309:58:293
2705903991714,1cyclictest0-21swapper/309:02:093
2705903991714,1cyclictest0-21swapper/308:56:023
2705903991714,1cyclictest0-21swapper/308:51:553
2705903991714,1cyclictest0-21swapper/308:36:343
2705903991714,1cyclictest0-21swapper/308:27:213
2705903991714,1cyclictest0-21swapper/308:23:153
2705903991714,1cyclictest0-21swapper/308:18:083
2705903991713,2cyclictest0-21swapper/310:05:123
2705903991713,2cyclictest0-21swapper/308:00:013
2705903991712,2cyclictest0-21swapper/311:22:073
270590399171,13cyclictest0-21swapper/311:41:323
270590399171,13cyclictest0-21swapper/310:52:193
270590399171,12cyclictest0-21swapper/312:00:143
270590399170,13cyclictest0-21swapper/307:10:143
270590099178,6cyclictest2718901-21grep07:40:122
270590099176,7cyclictest2736987-21diskstats08:25:152
270590099171,13cyclictest2730888-21/usr/sbin/munin08:10:122
29382182161,2sleep00-21swapper/012:32:230
29138632161,10sleep12914042-21grep12:10:151
29126962160,11sleep00-21swapper/012:09:380
28607462161,4sleep12754559-21diskmemload11:15:351
28281252161,3sleep02754559-21diskmemload10:37:130
2705903991613,2cyclictest0-21swapper/307:20:023
2705903991613,1cyclictest0-21swapper/307:40:193
2705903991612,2cyclictest0-21swapper/307:45:283
270590399160,13cyclictest0-21swapper/307:35:193
2705900991611,3cyclictest716-21gdbus08:45:002
270590099161,10cyclictest2751637-21awk09:00:212
29422872150,9sleep12754559-21diskmemload12:35:411
28846932154,6sleep12884704-21diskmemload11:42:171
28663012151,3sleep02754559-21diskmemload11:22:190
28369182151,3sleep12754559-21diskmemload10:47:531
27983272150,1sleep10-21swapper/110:01:581
27266752150,2sleep00-21swapper/008:00:070
270590399151,9cyclictest2716486-21taskset07:35:003
270590399151,9cyclictest2716486-21taskset07:35:003
2705903991512,1cyclictest0-21swapper/307:20:003
270590399151,10cyclictest2714494-21sleep07:30:003
270590399150,11cyclictest0-21swapper/309:05:203
28758362141,2sleep00-21swapper/011:32:500
28484362140,10sleep02848445-21ssh11:00:500
28164822140,2sleep00-21swapper/010:24:260
27880802141,6sleep02788238-21df_inode09:50:150
27383162140,1sleep00-21swapper/008:27:330
27125062141,4sleep02712602-21grep07:25:000
270590399141,9cyclictest2724472-21chrt07:55:003
29042672131,8sleep02904281-21ssh12:00:400
28610202131,3sleep00-21swapper/011:16:020
28551322132,5sleep02855151-21apt-get11:10:130
28457692131,8sleep12754559-21diskmemload10:58:541
27963842130,2sleep00-21swapper/010:00:130
27917472131,6sleep12791803-21munin-run09:55:011
27407402131,7sleep02740758-21apt-get08:35:150
27198192131,8sleep02720051-21grep07:40:230
28999362121,7sleep12754559-21diskmemload11:57:041
28838612120,2sleep00-21swapper/011:40:510
28486582120,6sleep12754559-21diskmemload11:01:161
28320132120,5sleep00-21swapper/010:41:340
28188672121,6sleep12754559-21diskmemload10:25:561
28067122120,2sleep10-21swapper/110:11:571
28032942121,3sleep12803304-21ssh10:08:201
28014922120,2sleep02754559-21diskmemload10:05:260
27839742121,2sleep10-21swapper/109:45:191
27602572121,7sleep12760537-21ssh09:15:241
27580372121,5sleep00-21swapper/009:13:420
29387392110,5sleep12754559-21diskmemload12:33:041
29322502111,2sleep00-21swapper/012:26:380
29259642111,6sleep02925978-21ssh12:20:280
29259642111,6sleep02925978-21ssh12:20:280
29050912111,4sleep12754559-21diskmemload12:01:451
28953572110,2sleep00-21swapper/011:53:300
28654822111,6sleep12865493-21sh11:20:561
28238282111,6sleep02823838-21ssh10:32:040
27650212110,2sleep00-21swapper/009:20:550
27510582110,7sleep12751167-21grep09:00:171
27257082111,6sleep02725873-21timerwakeupswit07:55:270
29297042101,3sleep12754559-21diskmemload12:25:021
29239352101,4sleep12923765-21apt-get12:20:011
29239352101,4sleep12923765-21apt-get12:20:001
29099382100,1sleep10-21swapper/112:06:021
28944432101,4sleep12754559-21diskmemload11:52:191
28882072101,3sleep12754559-21diskmemload11:45:291
28804392101,3sleep12754559-21diskmemload11:37:481
28740572101,6sleep12874319-21unixbench_multi11:30:261
28576532101,3sleep12754559-21diskmemload11:12:541
28543882101,2sleep00-21swapper/011:09:120
28510142101,4sleep12851038-21apt-get11:05:121
28362632101,3sleep02754559-21diskmemload10:46:430
28320632101,3sleep12754559-21diskmemload10:41:391
28280612101,4sleep12754559-21diskmemload10:37:061
28124242100,2sleep12812433-21ssh10:19:261
27941442101,4sleep12754559-21diskmemload09:56:321
27838652101,5sleep02783973-21perl09:45:180
27817572101,3sleep12754559-21diskmemload09:42:021
27731662101,2sleep00-21swapper/009:31:300
27731662101,2sleep00-21swapper/009:31:300
27706092101,4sleep12754559-21diskmemload09:29:391
27680682100,2sleep00-21swapper/009:25:210
27591032101,3sleep02759124-21apt-get09:15:110
27574322101,5sleep12757438-21sh09:12:351
27489182101,4sleep12749074-21diskstats08:55:161
27247182101,4sleep12724864-21df07:55:151
27135452101,4sleep12713587-21ntp_states07:25:211
27118512101,5sleep12712114-21awk07:20:241
2919201291,3sleep12754559-21diskmemload12:15:151
2913351290,1sleep00-21swapper/012:10:100
2899458290,2sleep00-21swapper/011:56:270
2886639291,4sleep02886648-21ssh11:45:040
2881463290,5sleep02881472-21ssh11:39:270
2870068290,4sleep12754559-21diskmemload11:25:531
2839265291,4sleep02839316-21ntpq10:50:200
2822895290,3sleep12754559-21diskmemload10:30:291
2818972290,5sleep00-21swapper/010:26:110
2815370291,2sleep12754559-21diskmemload10:22:231
2811947290,1sleep00-21swapper/010:18:370
2804515291,2sleep00-21swapper/010:10:030
2795199291,2sleep00-21swapper/009:58:330
2782809291,5sleep02752627-21kworker/u16:3+rpciod09:44:050
2776122291,3sleep12776172-21ntpq09:35:201
2774125291,1sleep10-21swapper/109:33:281
2774125291,1sleep10-21swapper/109:33:281
2752584291,3sleep02752607-21munin-run09:05:010
2744987291,4sleep02745136-21df_inode08:45:130
2744179290,3sleep071-21kcompactd008:40:280
2741348291,4sleep12741434-21ntpq08:35:201
2739470291,3sleep12739516-21ntpq08:30:191
2739470291,3sleep12739516-21ntpq08:30:191
2734866291,4sleep02734942-21expr08:20:160
2718262290,4sleep02718212-31find07:36:550
2845667280,1sleep00-21swapper/010:58:420
2764894280,2sleep12754559-21diskmemload09:20:411
2754338280,1sleep00-21swapper/009:07:210
2749976280,1sleep00-21swapper/008:55:270
2745653280,2sleep10-21swapper/108:45:181
2733510281,3sleep02733558-21ntpq08:15:190
2728242280,1sleep10-21swapper/108:03:131
2721428281,3sleep12721476-21ntpq07:45:211
2710242281,3sleep12710418-21grep07:20:001
2747326271,3sleep12747418-21kernelversion08:50:151
2734388270,1sleep10-21swapper/108:17:521
2732507270,1sleep00-21swapper/008:13:580
2723134271,2sleep02723266-21latency07:50:130
2722258270,1sleep00-21swapper/007:47:210
2716385271,3sleep1944-21snmpd07:34:231
2716385271,3sleep1944-21snmpd07:34:231
2716128270,1sleep00-21swapper/007:31:060
2716128270,1sleep00-21swapper/007:31:060
2712762270,1sleep00-21swapper/007:25:080
2710000270,1sleep00-21swapper/007:16:590
2707490270,1sleep10-21swapper/107:11:231
2748289260,4sleep00-21swapper/008:50:380
2744600260,1sleep10-21swapper/108:43:411
2740173260,1sleep00-21swapper/008:31:100
2740173260,1sleep00-21swapper/008:31:100
2730855260,2sleep10-21swapper/108:10:021
2728641260,1sleep00-21swapper/008:05:090
2718137260,1sleep10-21swapper/107:36:401
2752868250,2sleep10-21swapper/109:05:141
2738441250,3sleep10-21swapper/108:29:101
2735502250,1sleep10-21swapper/108:20:211
2723554250,2sleep10-21swapper/107:50:181
2718823250,2sleep10-21swapper/107:40:121
2728385240,1sleep10-21swapper/108:05:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional