You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-10 - 23:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Sat Jan 10, 2026 12:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2662423212658,63sleep30-21swapper/307:05:113
266701928367,10sleep20-21swapper/207:05:502
266705028056,6sleep00-21swapper/007:06:110
266725827862,10sleep10-21swapper/107:08:541
27599472410,10sleep1331ktimers/108:35:141
2667471992720,3cyclictest2974699-21turbostat12:00:003
28158352251,11sleep22815880-21open_files09:25:212
28801702241,11sleep22880346-21grep10:30:152
2667471992421,1cyclictest0-21swapper/309:25:243
266747199219,8cyclictest2716921-21sleep07:55:003
2667471992118,1cyclictest0-21swapper/310:30:273
27803352201,1sleep00-21swapper/008:55:090
266747199202,12cyclictest2964501-21taskset11:50:103
266747199202,11cyclictest2720197-21taskset07:55:153
2667471992018,1cyclictest0-21swapper/307:35:123
2667471991915,2cyclictest0-21swapper/311:20:243
2667471991915,2cyclictest0-21swapper/309:45:123
2667471991915,2cyclictest0-21swapper/307:15:003
266747199191,4cyclictest2948455-21sleep11:35:003
266747199191,15cyclictest0-21swapper/308:00:123
266747199191,14cyclictest0-21swapper/312:25:163
266747199191,14cyclictest0-21swapper/308:24:013
266747199191,14cyclictest0-21swapper/307:25:123
266747199191,13cyclictest2796083-21chrt09:10:103
266747199191,11cyclictest2735037-21taskset08:10:153
2667471991910,5cyclictest28853922sleep310:35:123
27855932182,5sleep12785612-21apt-get09:00:111
266747199184,10cyclictest2928518-21chrt11:15:153
266747199182,10cyclictest2957451-21taskset11:40:163
2667471991816,1cyclictest0-21swapper/308:35:153
2667471991815,1cyclictest0-21swapper/311:45:183
2667471991815,1cyclictest0-21swapper/307:40:153
2667471991814,2cyclictest0-21swapper/311:10:163
266747199181,13cyclictest0-21swapper/308:45:183
28420102171,11sleep12842124-21rt-features09:50:221
28420102171,11sleep12842124-21rt-features09:50:211
266747199172,10cyclictest2893164-21taskset10:40:173
266747199172,10cyclictest2806276-21taskset09:20:013
2667471991715,1cyclictest0-21swapper/308:20:003
2667471991714,1cyclictest0-21swapper/310:55:163
2667471991714,1cyclictest0-21swapper/309:35:153
2667471991714,1cyclictest0-21swapper/307:15:263
2667471991713,2cyclictest0-21swapper/308:44:593
2667471991713,2cyclictest0-21swapper/308:30:003
2667471991712,3cyclictest0-21swapper/310:25:093
266747199171,10cyclictest2941174-21taskset11:25:143
266747199171,10cyclictest2848445-21taskset10:00:003
266747199170,13cyclictest0-21swapper/309:40:103
29069522161,3sleep22908046-21kthreadcore10:55:152
27381302161,3sleep10-21swapper/108:15:101
27112572161,11sleep02706632-21/usr/sbin/munin07:45:270
266747199167,6cyclictest2953700-21turbostat11:40:003
266747199161,9cyclictest2980170-21taskset12:05:063
2667471991613,2cyclictest0-21swapper/310:15:003
2667471991613,1cyclictest0-21swapper/310:50:163
2667471991613,1cyclictest0-21swapper/309:30:173
2667471991613,1cyclictest0-21swapper/307:30:143
266747199161,1cyclictest0-21swapper/309:55:213
266747199161,13cyclictest0-21swapper/312:20:213
266747199161,13cyclictest0-21swapper/312:00:183
266747199161,12cyclictest0-21swapper/308:05:193
266747199161,12cyclictest0-21swapper/307:46:223
2667471991610,4cyclictest29903832chrt12:14:353
266747199160,13cyclictest0-21swapper/310:48:543
266747199160,13cyclictest0-21swapper/309:08:553
266747199160,12cyclictest0-21swapper/312:30:213
266747199160,12cyclictest0-21swapper/308:30:213
266747199160,10cyclictest3015858-21taskset12:35:243
28474792151,9sleep22847679-21sed09:55:242
28173212151,9sleep22817531-21hddtemp_smartct09:30:152
27368412151,10sleep12736887-21ntpq08:10:171
266747199155,7cyclictest2785322-21turbostat09:00:013
266747199151,9cyclictest2995336-21taskset12:16:003
266747199151,9cyclictest2916889-21taskset11:05:053
266747199151,9cyclictest2916507-21taskset11:03:263
266747199151,9cyclictest2868828-21taskset10:15:413
2667471991512,1cyclictest0-21swapper/309:50:263
2667471991512,1cyclictest0-21swapper/309:50:253
266747199151,11cyclictest27769812sleep308:50:143
266747199150,12cyclictest0-21swapper/310:07:143
266747199150,12cyclictest0-21swapper/309:20:093
266747199150,12cyclictest0-21swapper/309:00:123
266747199150,12cyclictest0-21swapper/307:20:183
29644802142,7sleep22964498-21apt-get11:50:102
29480712140,1sleep00-21swapper/011:30:330
28691742141,4sleep22869318-21date10:20:002
28117682141,2sleep10-21swapper/109:25:121
27790292141,9sleep12779082-21ld08:50:171
27326072141,9sleep22732608-21sshd08:09:422
26793052141,8sleep02679578-21unixbench_singl07:15:250
266747199140,11cyclictest0-21swapper/310:20:183
29857052131,8sleep12985913-21hddtemp_smartct12:10:141
29279762131,7sleep12928207-21cat11:15:151
29121512131,7sleep12912377-21kthreadcore11:00:151
27809382131,7sleep12782503-21kthreadcore08:55:151
29267262121,7sleep02926882-21timerwakeupswit11:10:240
28375672121,7sleep22837570-21sshd09:48:542
27171492122,5sleep12717167-21apt-get07:55:101
29545312111,6sleep12955653-21kthreadcore11:40:151
29490922111,7sleep02949247-21/usr/sbin/munin11:35:130
28907102111,5sleep12890729-21apt-get10:40:121
28838542111,5sleep02883895-21ntpq10:30:210
28799532111,5sleep12880169-21diskstats10:30:141
28064822111,4sleep22806501-21apt-get09:20:082
27594262111,4sleep22759678-21df08:35:122
27532922110,1sleep10-21swapper/108:25:371
27224222111,5sleep12722441-21apt-get08:00:121
26855182111,3sleep00-21swapper/007:25:110
30051432101,4sleep23005196-21ntpq12:25:222
30014422101,4sleep03001652-21hddtemp_smartct12:25:150
29894832101,4sleep02989536-21ntp_states12:10:220
29544152101,4sleep02954530-21irqstats11:40:140
29421972101,4sleep22942250-21ntpq11:25:182
29370472101,5sleep22937160-21ps11:20:202
29282082101,4sleep22928517-21cut11:15:162
29169792101,4sleep12917182-21diskstats11:05:131
29014442101,3sleep12901654-21hddtemp_smartct10:50:131
28999132101,4sleep02899959-21ntpq10:45:190
28892662100,1sleep20-21swapper/210:35:192
28794742103,2sleep2581rcuop/310:28:142
28744062102,4sleep12874477-21sed10:25:011
28629242101,4sleep22862975-21ntpq10:10:182
28528392101,4sleep22853083-21users10:00:282
27928712101,5sleep22794581-21expr09:05:182
27686432101,4sleep02768692-21ntpq08:40:220
27672072101,4sleep12768455-21memory08:40:191
27546982101,5sleep02754927-21tr08:30:160
27331882101,5sleep02733398-21hddtemp_smartct08:10:120
27316652101,4sleep02731719-21ntpq08:05:190
27119632101,5sleep02712166-21diskstats07:50:130
27006822101,5sleep02700743-21gdbus07:36:300
26841002101,4sleep02684239-21ntp_192.168.11407:20:190
3016320290,4sleep20-21swapper/212:36:082
2989388291,3sleep22989429-21ntpq12:10:192
2981827291,4sleep02983611-21cut12:05:150
2969733291,3sleep12969752-21apt-get11:55:101
2953515290,1sleep10-21swapper/111:37:561
2938742291,4sleep12939458-21grep11:25:131
2930189291,4sleep02931492-21memory11:15:180
2901148291,3sleep22901170-21apt-get10:50:112
2900361291,4sleep22900526-21vmstat10:45:252
2886196291,4sleep02887877-21cut10:35:170
2879616291,4sleep02879667-21munin-run10:30:000
2843686291,4sleep02844063-21cat09:55:130
2841956291,4sleep02842009-21ntp_states09:50:210
2841956291,4sleep02842009-21ntp_states09:50:200
2811487291,3sleep12811535-21munin-run09:25:011
2796061290,3sleep20-21swapper/209:10:102
2779458291,4sleep02779702-21users08:50:240
2759729291,4sleep02759945-21if_enp1s008:35:140
2754928291,4sleep12756583-21cut08:30:171
2753403290,1sleep00-21swapper/008:27:010
2743912291,4sleep02743982-21irqstats08:20:140
2738514291,4sleep02738688-21cat08:15:140
2721177291,2sleep22721222-21ntpq07:55:192
2700744292,3sleep10-21swapper/107:36:311
2695013291,4sleep22695236-21grep07:30:262
2680807291,4sleep12681668-21tr07:20:151
2675355291,4sleep12675471-21irqstats07:15:141
3015684280,3sleep13015740-21ntp_states12:35:231
3011490280,1sleep00-21swapper/012:35:020
3011075280,3sleep10-21swapper/112:31:191
3001149280,3sleep13001168-21apt-get12:25:121
2999935281,3sleep12999989-21ntpq12:20:201
2996503281,3sleep00-21swapper/012:20:150
2994759283,2sleep02994843-21proc_pri12:15:220
2959243280,3sleep22959262-21apt-get11:45:102
2947505281,3sleep12947556-21/usr/sbin/munin11:30:211
2947266281,3sleep22947404-21nfs4_client11:30:182
2937573280,1sleep00-21swapper/011:20:270
2922427281,2sleep20-21swapper/211:10:102
2921007281,3sleep22921116-21/usr/sbin/munin11:05:212
2891501281,3sleep22893163-21cut10:40:172
2873465281,3sleep22873522-21ntp_states10:20:212
2864222281,2sleep10-21swapper/110:15:101
2863087281,3sleep12858969-21/usr/sbin/munin10:10:221
2858544280,1sleep10-21swapper/110:07:571
2849033281,3sleep02850492-21cat10:00:190
2828235281,3sleep02829545-21kthreadcore09:40:150
2822303280,5sleep10-21swapper/109:35:101
2811746281,2sleep00-21swapper/009:25:110
2775696281,3sleep22775729-21kthreadcore08:50:142
2753503280,4sleep20-21swapper/208:28:182
2747692281,3sleep22747762-21threads08:20:242
2722444281,3sleep22722688-21expr08:00:132
2712167281,3sleep12712354-21head07:50:141
2701927281,3sleep02703085-21kthreadcore07:40:150
2701646281,3sleep22701848-21expr07:40:122
2701429281,3sleep12701645-21diskstats07:40:121
2692789280,4sleep12694497-21cat07:30:181
2690446282,3sleep1331ktimers/107:29:371
2688628281,3sleep20-21swapper/207:25:172
3008101271,3sleep20-21swapper/212:30:192
3006397271,3sleep03006415-21/usr/sbin/munin12:30:120
2978757270,2sleep20-21swapper/212:00:162
2964122270,1sleep10-21swapper/111:48:491
2953599270,1sleep20-21swapper/211:39:012
2922396270,3sleep12921901-21apt-get11:10:011
2917469271,2sleep00-21swapper/011:05:160
2916463270,1sleep20-21swapper/211:02:532
2916318270,1sleep00-21swapper/011:00:590
2911272270,1sleep00-21swapper/010:58:590
2895364270,1sleep00-21swapper/010:40:590
2869459270,2sleep00-21swapper/010:20:110
2864168270,4sleep00-21swapper/010:15:010
2837320270,1sleep00-21swapper/009:45:440
2832353270,1sleep10-21swapper/109:43:331
2806480270,1sleep00-21swapper/009:20:070
2805899270,1sleep20-21swapper/209:16:162
2805268271,3sleep00-21swapper/009:15:210
2794781271,2sleep12794819-21ntpq09:05:201
2774856270,4sleep20-21swapper/208:49:552
2769310270,1sleep20-21swapper/208:41:132
2759042270,1sleep20-21swapper/208:33:472
2743433270,4sleep10-21swapper/108:20:111
2739432271,2sleep20-21swapper/208:15:152
2726540271,3sleep0553-21dbus-daemon08:00:220
2707366271,2sleep10-21swapper/107:45:161
2675472270,3sleep22676578-21cat07:15:162
2672166270,1sleep10-21swapper/107:10:251
2667645270,2sleep20-21swapper/207:10:102
3000660260,1sleep20-21swapper/212:22:102
2991244260,2sleep20-21swapper/212:15:152
2984921260,1sleep20-21swapper/212:07:062
2979917260,3sleep10-21swapper/112:05:001
2979740260,1sleep10-21swapper/112:02:491
2976204261,2sleep00-21swapper/012:00:150
2974569260,1sleep00-21swapper/011:58:340
2970991261,2sleep20-21swapper/211:55:152
2959235260,1sleep00-21swapper/011:45:030
2958598260,4sleep20-21swapper/211:40:452
2943166261,2sleep00-21swapper/011:30:010
2903291261,2sleep00-21swapper/010:50:160
2857577261,2sleep22857698-21ntpq10:05:182
2855071261,2sleep00-21swapper/010:05:150
2844064261,2sleep10-21swapper/109:55:141
2832231260,1sleep20-21swapper/209:41:582
2826872260,4sleep20-21swapper/209:35:402
2822281260,1sleep00-21swapper/009:35:090
2819166260,3sleep1321rcuc/109:30:181
2817632261,2sleep00-21swapper/009:30:170
2806020260,1sleep10-21swapper/109:17:471
2800720260,1sleep00-21swapper/009:11:220
2800136261,2sleep10-21swapper/109:10:211
2790467260,1sleep00-21swapper/009:03:510
2786516260,2sleep20-21swapper/209:00:162
2780512261,2sleep20-21swapper/208:55:122
2774742260,1sleep00-21swapper/008:48:290
2770146261,2sleep10-21swapper/108:45:131
2737555260,4sleep20-21swapper/208:11:042
2732454260,4sleep10-21swapper/108:07:451
2721006261,1sleep00-21swapper/007:55:170
2690767260,3sleep00-21swapper/007:30:130
2681669260,2sleep22683410-21kthreadcore07:20:162
2995394250,1sleep10-21swapper/112:16:471
2934709250,1sleep10-21swapper/111:20:151
2910259250,1sleep10-21swapper/110:55:181
2889505250,1sleep10-21swapper/110:35:221
2878747250,1sleep10-21swapper/110:25:201
2858961250,3sleep00-21swapper/010:10:050
2852518250,1sleep10-21swapper/110:00:231
2837921250,1sleep20-21swapper/209:50:112
2837921250,1sleep20-21swapper/209:50:102
2834425250,2sleep10-21swapper/109:45:161
2790826250,1sleep00-21swapper/009:05:110
2712355250,1sleep20-21swapper/207:50:152
2706889250,1sleep20-21swapper/207:45:132
2695980250,2sleep20-21swapper/207:35:112
2671742250,1sleep00-21swapper/007:10:200
2964478240,1sleep10-21swapper/111:50:091
2964259240,1sleep00-21swapper/011:50:010
2895719240,1sleep10-21swapper/110:45:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional