You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-01 - 05:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Thu Jan 01, 2026 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2671340212055,59sleep30-21swapper/319:05:393
266877228669,9sleep20-21swapper/219:05:172
267136027260,6sleep00-21swapper/019:05:510
267135127257,9sleep10-21swapper/119:05:441
2671801993533,1cyclictest0-21swapper/320:20:003
2671801992824,2cyclictest0-21swapper/323:15:013
2671801992623,1cyclictest0-21swapper/320:10:003
267180199261,20cyclictest2768569-21taskset20:36:483
2671801992522,1cyclictest0-21swapper/322:45:143
267180199242,6cyclictest2857818-21taskset22:00:303
2671801992420,2cyclictest0-21swapper/319:50:003
267180199231,11cyclictest2990802-21sleep00:15:003
267180199230,19cyclictest0-21swapper/321:36:303
2671801992219,1cyclictest0-21swapper/323:40:123
2671801992219,1cyclictest0-21swapper/323:00:003
2671801992219,1cyclictest0-21swapper/322:59:593
267180199221,12cyclictest2700314-21taskset19:35:003
267180199216,12cyclictest0-21swapper/323:30:163
2671801992017,1cyclictest0-21swapper/322:15:153
2671801992017,1cyclictest0-21swapper/320:50:003
2671801992016,2cyclictest0-21swapper/322:00:003
267180199201,16cyclictest0-21swapper/323:00:123
267180199201,15cyclictest2731967-21turbostat20:05:013
267180199192,12cyclictest2684579-21taskset19:20:123
2671801991917,1cyclictest0-21swapper/322:20:133
2671801991916,1cyclictest0-21swapper/323:20:173
2671801991916,1cyclictest0-21swapper/323:15:283
2671801991915,2cyclictest0-21swapper/322:45:003
267180199191,13cyclictest2784649-21turbostat20:54:593
267180199191,13cyclictest0-21swapper/321:45:123
27327852181,6sleep02734092-21cat20:05:160
2671801991814,2cyclictest0-21swapper/321:20:153
267180199181,14cyclictest2811045-21chrt21:19:253
267180199181,13cyclictest2763595-21turbostat20:35:003
2671801991811,3cyclictest0-21swapper/323:45:113
267180199181,13cyclictest0-21swapper/320:20:123
267180199180,11cyclictest2895083-21taskset22:39:213
267180199172,10cyclictest2883694-21taskset22:25:183
2671801991715,1cyclictest0-21swapper/320:55:193
2671801991714,1cyclictest0-21swapper/323:50:183
2671801991714,1cyclictest0-21swapper/322:10:143
2671801991714,1cyclictest0-21swapper/322:05:173
2671801991714,1cyclictest0-21swapper/321:05:273
2671801991713,2cyclictest0-21swapper/323:35:183
2671801991713,2cyclictest0-21swapper/321:40:143
2671801991713,2cyclictest0-21swapper/300:30:013
267180199171,12cyclictest2996014-21turbostat00:20:013
267180199171,10cyclictest3011785-21taskset00:35:003
267180199171,10cyclictest2819791-21taskset21:25:153
267180199171,10cyclictest2690332-21taskset19:25:133
267180199170,15cyclictest0-21swapper/321:30:263
29561822161,10sleep22957112-21memory23:35:162
28624632161,10sleep12862511-21/usr/sbin/munin22:05:211
26991392161,11sleep22699258-21modprobe19:30:192
267180199162,9cyclictest2684132-21taskset19:18:133
2671801991614,1cyclictest0-21swapper/322:50:213
2671801991613,1cyclictest0-21swapper/320:40:113
2671801991613,1cyclictest0-21swapper/300:35:113
2671801991612,2cyclictest0-21swapper/323:30:003
2671801991612,2cyclictest0-21swapper/320:10:163
267180199161,10cyclictest2705696-21taskset19:38:443
267180199161,10cyclictest2671977-21taskset19:10:063
267180199160,13cyclictest0-21swapper/321:10:223
267180199160,13cyclictest0-21swapper/320:25:153
267180199160,13cyclictest0-21swapper/319:55:113
267180199160,13cyclictest0-21swapper/319:40:233
267180199160,13cyclictest0-21swapper/300:00:123
29386522151,9sleep12940294-21cut23:20:171
26891462151,2sleep20-21swapper/219:20:292
267180199151,9cyclictest2985508-21taskset00:09:563
267180199151,9cyclictest2921944-21taskset23:05:063
2671801991512,1cyclictest0-21swapper/323:57:463
2671801991512,1cyclictest0-21swapper/322:30:133
2671801991512,1cyclictest0-21swapper/321:50:203
2671801991512,1cyclictest0-21swapper/321:00:133
2671801991511,2cyclictest0-21swapper/300:20:113
267180199151,11cyclictest2721566-21turbostat19:55:003
30168902141,8sleep03017003-21date00:39:590
29271672141,8sleep12927186-21apt-get23:10:271
28940882141,8sleep02890160-21/usr/sbin/munin22:35:190
28837512141,9sleep02883813-21pmu-power22:25:210
27309092141,9sleep22730954-21/usr/sbin/munin20:00:212
27007822140,9sleep02700282-21apt-get19:35:010
27638482132,5sleep02763866-21apt-get20:35:140
27575032131,8sleep02757629-21readlink20:25:240
26901282131,7sleep22690331-21diskstats19:25:132
26798202131,7sleep12679853-21kthreadcore19:15:141
26728832136,3sleep02674515-21cut19:10:170
29698692121,7sleep02969901-21kthreadcore23:50:170
29516522121,7sleep22951950-21needreboot23:30:202
29114142121,7sleep12911431-21apt22:55:121
29114142121,7sleep12911431-21apt22:55:111
29052032121,7sleep22905321-21cat22:45:272
28992742127,1sleep00-21swapper/022:40:190
28276492121,7sleep22828659-21cut21:35:162
28224602121,7sleep02824042-21kthreadcore21:30:180
30002252111,5sleep13000274-21ntpq00:20:201
29837812111,4sleep12983861-21ntpq00:00:201
29729662111,6sleep22973186-21needreboot23:50:192
27851752111,6sleep02785222-21/usr/sbin/munin20:55:150
27624862111,4sleep22762574-21ntpq20:30:182
27521522111,5sleep12752199-21ntpq20:20:191
27116002111,5sleep22711809-21hddtemp_smartct19:45:132
26900142111,5sleep02689516-21apt-get19:25:030
29898332101,4sleep12989885-21ntp_states00:10:181
29785322101,5sleep02978582-21sed23:55:190
29468042105,2sleep00-21swapper/023:25:210
29323822101,4sleep02932638-21df23:15:150
29101372101,4sleep22910183-21ntpq22:50:192
28994172101,4sleep22899474-21ntpq22:40:212
28796272101,4sleep22879646-21apt-get22:25:102
28518962101,4sleep12851944-21ntpq21:55:191
28205722101,4sleep12820619-21ntpq21:25:171
28154022101,5sleep02815456-21ntpq21:20:240
28011572101,4sleep12801366-21hddtemp_smartct21:10:141
27836592101,4sleep02783706-21ntpq20:50:210
27638702101,4sleep12764128-21df_inode20:35:151
27626312101,5sleep02762687-21ntpq20:30:200
27429712101,4sleep22742990-21apt-get20:15:112
27165452101,3sleep12716564-21apt-get19:50:101
27061662101,4sleep02706369-21diskstats19:40:150
27008302101,4sleep12701086-21df19:35:141
26793332101,4sleep22679536-21diskstats19:15:112
3015845291,4sleep23015901-21ld00:35:212
2991033290,3sleep10-21swapper/100:15:121
2971205291,4sleep12972964-21cut23:50:181
2964065291,3sleep22964084-21apt-get23:45:112
2953342291,4sleep12953559-21df_inode23:35:121
2937858291,4sleep22937959-21fschecks_time23:20:142
2918421291,4sleep22918762-21kthreadcore23:00:182
2916373291,3sleep0800-21snmpd22:59:160
2916373291,3sleep0800-21snmpd22:59:150
2900874291,4sleep1248-21systemd-journal22:45:021
2886299291,4sleep02887987-21cut22:30:170
2885210291,4sleep12885329-21cut22:30:141
2878053291,4sleep02878085-21ntpq22:20:190
2868857291,2sleep10-21swapper/122:15:111
2845370291,3sleep22846570-21memory21:50:162
2843732291,4sleep12845369-21tr21:50:151
2837631291,4sleep12837814-21diskstats21:45:131
2833476290,6sleep22834349-21kthreadcore21:40:182
2826607290,1sleep10-21swapper/121:32:191
2800866291,2sleep20-21swapper/221:10:112
2799355291,3sleep12799404-21ntpq21:05:221
2795377291,3sleep02795396-21apt-get21:05:120
2795116291,4sleep02795131-21sh21:05:010
2785443291,4sleep22785890-21tr20:55:172
2784506290,1sleep10-21swapper/120:53:221
2775901291,4sleep12777592-21grep20:45:181
2770283290,2sleep20-21swapper/220:40:162
2769481291,4sleep02769654-21if_err_enp1s020:40:140
2737694290,1sleep10-21swapper/120:10:051
2722096291,4sleep12722306-21hddtemp_smartct19:55:141
2711332291,4sleep02711454-21cpuspeed_turbos19:45:110
3016239281,3sleep13016470-21/usr/sbin/munin00:35:291
3011632281,3sleep23011651-21latency_hist00:35:002
3002058281,3sleep03003489-21ps00:25:160
2991606281,3sleep22992881-21kthreadcore00:15:172
2984188281,4sleep00-21swapper/000:00:270
2976618281,3sleep12978319-21sed23:55:161
2962588281,3sleep22962636-21ntpq23:40:222
2948066281,3sleep02948222-21cstates23:30:130
2942546281,3sleep02942568-21/usr/sbin/munin23:25:000
2922568281,3sleep10-21swapper/123:05:211
2916716281,3sleep02916735-21apt-get23:00:120
2915474281,3sleep22915526-21modprobe22:55:212
2915474281,3sleep22915526-21modprobe22:55:202
2888771281,3sleep22888884-21ntpq22:30:202
2883817280,1sleep10-21swapper/122:25:221
2874911281,3sleep12876537-21kthreadcore22:20:171
2842757280,2sleep00-21swapper/021:50:100
2837815281,3sleep22838024-21hddtemp_smartct21:45:142
2837547283,2sleep00-21swapper/021:45:120
2832316280,2sleep10-21swapper/121:40:131
2820626281,3sleep22820683-21ntpq21:25:192
2789751280,4sleep10-21swapper/120:58:401
2769105281,3sleep12769123-21apt-get20:40:111
2754333281,3sleep22755453-21kthreadcore20:25:182
2753420280,2sleep10-21swapper/120:25:131
2749642281,3sleep22751345-21kthreadcore20:20:172
2734094280,4sleep12735862-21cut20:05:161
2689506283,2sleep12689557-21apt-key19:25:011
3011326270,1sleep10-21swapper/100:31:071
3000920270,1sleep00-21swapper/000:21:330
2978320270,4sleep22978469-21strings23:55:172
2947766270,1sleep20-21swapper/223:29:512
2936664271,3sleep22936877-21awk23:15:282
2932098271,2sleep20-21swapper/223:15:012
2922465271,2sleep02922562-21ps23:05:210
2910915270,1sleep10-21swapper/122:51:581
2900907271,2sleep00-21swapper/022:45:130
2890731271,2sleep12891893-21kthreadcore22:35:161
2869557271,2sleep20-21swapper/222:15:162
2867713271,2sleep22867822-21/usr/sbin/munin22:10:242
2857940270,1sleep00-21swapper/022:02:000
2857228271,2sleep20-21swapper/222:00:222
2853796271,2sleep10-21swapper/122:00:171
2848564271,2sleep22849853-21cut21:55:162
2816058270,3sleep12816059-21sshd21:21:211
2811377270,2sleep20-21swapper/221:20:132
2807856271,2sleep10-21swapper/121:15:201
2805496270,1sleep00-21swapper/021:10:460
2790714271,2sleep20-21swapper/221:00:172
2768608270,1sleep20-21swapper/220:37:212
2753005270,1sleep00-21swapper/020:23:030
2747365271,3sleep12747550-21users20:15:271
2727161270,4sleep00-21swapper/020:00:140
2726600270,1sleep00-21swapper/019:58:310
2713167271,3sleep10-21swapper/119:45:161
2703484270,3sleep22704637-21/usr/sbin/munin19:35:192
2700117270,1sleep00-21swapper/019:33:060
2679248271,2sleep02679267-21apt-get19:15:090
3011603260,1sleep00-21swapper/000:34:390
3005453261,2sleep10-21swapper/100:25:221
3000853260,1sleep20-21swapper/200:20:412
2990746262,2sleep0151ktimers/000:14:420
2990539260,1sleep20-21swapper/200:12:042
2985439260,1sleep10-21swapper/100:09:031
2980407261,2sleep20-21swapper/200:00:172
2968726260,1sleep00-21swapper/023:46:090
2962492260,2sleep00-21swapper/023:40:190
2959943261,2sleep10-21swapper/123:40:161
2953291260,2sleep00-21swapper/023:35:110
2949883261,2sleep10-21swapper/123:30:181
2937113260,1sleep10-21swapper/123:17:481
2931993260,1sleep00-21swapper/023:13:420
2917286261,2sleep10-21swapper/123:00:171
2910924260,1sleep00-21swapper/022:52:050
2895436261,2sleep10-21swapper/122:40:121
2891897261,2sleep20-21swapper/222:35:172
2878752260,1sleep20-21swapper/222:20:422
2864221261,2sleep00-21swapper/022:10:170
2859797261,2sleep00-21swapper/022:05:170
2836397261,2sleep00-21swapper/021:40:240
2831796260,4sleep10-21swapper/121:36:591
2831692260,1sleep00-21swapper/021:35:420
2826603260,4sleep20-21swapper/221:32:162
2821334260,1sleep00-21swapper/021:26:480
2811019260,4sleep20-21swapper/221:19:052
2806667260,2sleep00-21swapper/021:15:180
2784389260,4sleep20-21swapper/220:51:512
2779102260,3sleep20-21swapper/220:46:152
2774951261,2sleep00-21swapper/020:45:170
2738276261,2sleep20-21swapper/220:10:192
2731625260,4sleep10-21swapper/120:00:521
2721802260,1sleep20-21swapper/219:55:112
2717661261,2sleep20-21swapper/219:50:152
2709703261,2sleep10-21swapper/119:40:201
2706080260,1sleep20-21swapper/219:40:132
2700040260,4sleep10-21swapper/119:32:071
2690599261,2sleep10-21swapper/119:25:161
2688816261,2sleep00-21swapper/019:20:270
2675812260,1sleep22675957-21strings19:10:192
2994642250,1sleep00-21swapper/000:15:190
2964035250,1sleep10-21swapper/123:45:021
2947606250,1sleep10-21swapper/123:27:511
2924157250,1sleep20-21swapper/223:05:222
2863734250,1sleep10-21swapper/122:10:151
2852659250,1sleep00-21swapper/021:56:260
2797897251,1sleep20-21swapper/221:05:192
2790644250,1sleep10-21swapper/121:00:161
2763296250,1sleep10-21swapper/120:31:221
2747612250,3sleep00-21swapper/020:16:070
2737995250,1sleep00-21swapper/020:10:160
2732502250,1sleep20-21swapper/220:05:132
2720384250,1sleep00-21swapper/019:50:160
2676067250,1sleep10-21swapper/119:10:221
3001444240,1sleep20-21swapper/200:25:072
2985229240,1sleep00-21swapper/000:06:260
2985215240,2sleep20-21swapper/200:06:162
2869156240,1sleep00-21swapper/022:15:150
2858434240,1sleep20-21swapper/222:05:102
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional