You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-09 - 20:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Sun Nov 09, 2025 12:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2672915212362,55sleep30-21swapper/307:05:273
266831528454,6sleep00-21swapper/007:05:050
267325428162,13sleep10-21swapper/107:09:471
267300027761,10sleep20-21swapper/207:06:312
2673411993633,1cyclictest0-21swapper/310:05:003
2673411993027,1cyclictest0-21swapper/310:50:223
2673411992722,3cyclictest0-21swapper/311:35:193
2673411992623,1cyclictest0-21swapper/308:20:003
2673411992618,6cyclictest66550irq/131-enp1s0-TxRx-311:19:293
267341199241,14cyclictest0-21swapper/311:05:153
267341199231,16cyclictest2802731-21sleep09:09:593
267341199226,8cyclictest3099491-21turbostat12:14:593
267341199222,14cyclictest3055391-21taskset11:50:143
2673411992219,1cyclictest0-21swapper/308:20:123
2673411992218,2cyclictest0-21swapper/309:33:353
267341199221,17cyclictest2941472-21taskset10:37:543
267341199212,12cyclictest3125893-21taskset12:29:353
2673411992115,3cyclictest0-21swapper/312:30:183
2673411992111,7cyclictest0-21swapper/310:33:303
28950142201,4sleep12802705-21diskmemload10:09:241
27763202201,7sleep02776333-21latency_hist08:45:000
2673411992018,1cyclictest0-21swapper/312:35:193
2673411992017,1cyclictest0-21swapper/311:30:163
2673411992017,1cyclictest0-21swapper/309:35:153
2673411992017,1cyclictest0-21swapper/308:50:163
2673411992017,1cyclictest0-21swapper/307:40:273
2673411992016,2cyclictest2950560-21turbostat10:45:013
2673411992016,2cyclictest0-21swapper/312:05:183
2673411992016,2cyclictest0-21swapper/311:25:003
2673411992016,2cyclictest0-21swapper/309:49:583
2673411992015,3cyclictest0-21swapper/309:15:263
267341199201,16cyclictest0-21swapper/311:55:193
267341199201,15cyclictest0-21swapper/310:20:113
267341199201,14cyclictest0-21swapper/307:13:413
2673411992010,7cyclictest2808144-21taskset09:11:003
30524022190,2sleep00-21swapper/011:47:080
267341199192,13cyclictest3080142-21taskset12:03:003
267341199192,13cyclictest2832092-21taskset09:27:553
267341199192,12cyclictest2872201-21taskset09:54:323
2673411991917,1cyclictest0-21swapper/311:40:163
2673411991916,1cyclictest0-21swapper/309:21:513
2673411991915,2cyclictest0-21swapper/310:45:153
2673411991915,2cyclictest0-21swapper/308:35:163
267341199191,15cyclictest3109133-21chrt12:20:123
2673411991911,4cyclictest29014882sleep310:12:053
267341199191,14cyclictest2893911-21chrt10:06:523
267341199191,14cyclictest0-21swapper/307:39:043
267341199191,13cyclictest0-21swapper/307:17:583
30038602180,9sleep13003875-21ssh11:17:021
29638282181,11sleep02802705-21diskmemload10:50:550
28310362181,3sleep10-21swapper/109:26:101
27710482181,3sleep12771068-21latency_hist08:40:011
27381872181,5sleep12738241-21ntpq08:05:221
267341199183,12cyclictest2995576-21taskset11:11:223
2673411991815,1cyclictest0-21swapper/311:00:193
2673411991815,1cyclictest0-21swapper/307:45:253
2673411991814,2cyclictest0-21swapper/308:25:173
2673411991814,2cyclictest0-21swapper/308:25:163
267341199181,14cyclictest0-21swapper/310:59:093
267341199181,14cyclictest0-21swapper/310:25:313
267341199181,14cyclictest0-21swapper/310:25:303
267341199181,13cyclictest0-21swapper/311:45:333
267341199181,13cyclictest0-21swapper/309:55:133
267341199181,13cyclictest0-21swapper/309:40:113
267341199181,10cyclictest2790407-21taskset08:55:163
31248132171,2sleep20-21swapper/212:28:152
30891582171,3sleep00-21swapper/012:08:110
30361512171,2sleep10-21swapper/111:38:271
29974442171,4sleep12997454-21ssh11:14:411
29901162171,4sleep22802705-21diskmemload11:10:092
267341199176,8cyclictest3105907-21taskset12:16:523
2673411991714,2cyclictest0-21swapper/309:05:013
2673411991714,1cyclictest0-21swapper/307:55:173
267341199171,13cyclictest0-21swapper/311:26:223
267341199171,13cyclictest0-21swapper/308:10:213
267341199171,12cyclictest0-21swapper/308:05:213
267341199171,10cyclictest2690237-21taskset07:20:223
31177222160,2sleep10-21swapper/112:25:121
30797192161,3sleep02802705-21diskmemload12:02:310
30772782161,8sleep23077368-21ntp_192.168.11412:00:202
30431982160,2sleep10-21swapper/111:41:391
29565422161,9sleep12802705-21diskmemload10:46:481
267341199161,12cyclictest0-21swapper/308:45:143
267341199161,10cyclictest2903472-21taskset10:15:063
31094162151,9sleep13110006-21cut12:20:161
31074322151,3sleep13107446-21ssh12:18:481
30552552151,4sleep23055269-21ssh11:50:062
29780542151,9sleep12978173-21ntpq11:00:221
29247902151,4sleep12802705-21diskmemload10:26:231
29247902151,4sleep12802705-21diskmemload10:26:231
29160512151,9sleep02916092-21ssh10:20:250
28917762152,8sleep22892266-21grep10:05:202
28719052151,10sleep22871920-21ssh09:54:042
28572432151,9sleep02678798-21munin-node09:45:130
28191622151,9sleep12819887-21cut09:20:161
267341199151,9cyclictest2718056-21taskset07:50:113
2673411991512,1cyclictest0-21swapper/308:00:443
267341199150,12cyclictest0-21swapper/308:40:153
267341199150,11cyclictest2698203-21chrt07:30:163
31168302141,9sleep03116832-21ssh12:24:240
30897522141,2sleep10-21swapper/112:08:581
30716202141,9sleep13071628-21ssh11:58:311
30608142141,9sleep13060826-21ssh11:51:091
29979042141,9sleep22997918-21ssh11:15:032
29584362141,8sleep02958447-21sh10:49:590
29426582140,2sleep0221rcuc/010:39:520
29339292145,5sleep22886569-21kworker/u16:3+nfsiod10:33:302
29315332141,9sleep12802705-21diskmemload10:30:231
29250342141,8sleep02802705-21diskmemload10:26:500
29250342141,8sleep02802705-21diskmemload10:26:490
29104422141,9sleep12910454-21ssh10:19:021
28941752141,9sleep02894188-21rm10:07:170
28343022141,3sleep22834378-21irqstats09:30:152
26777132141,8sleep12677837-21/usr/sbin/munin07:10:221
2673411991411,1cyclictest0-21swapper/307:25:183
267341199140,11cyclictest0-21swapper/308:30:143
31339322131,7sleep23133943-21ssh12:33:322
31333912130,8sleep03133406-21ssh12:32:500
30726842131,7sleep22802705-21diskmemload11:59:532
30692732131,8sleep03069287-21sh11:55:340
29890842131,8sleep02989089-21ssh11:09:200
29791042131,8sleep02979116-21ssh11:00:490
29653972131,6sleep10-21swapper/110:53:521
28857632131,6sleep02802705-21diskmemload10:01:170
28728292132,5sleep12872853-21apt-get09:55:131
28555522130,7sleep02802705-21diskmemload09:42:280
28423642130,8sleep22843151-21cut09:35:172
27007792131,7sleep22700826-21ntpq07:30:192
31180642121,3sleep00-21swapper/012:25:150
30972362121,4sleep20-21swapper/212:12:102
30964812121,6sleep13096496-21ssh12:11:141
30527762121,6sleep12802705-21diskmemload11:47:371
30124922121,3sleep13012498-21ssh11:23:191
30045342120,2sleep00-21swapper/011:18:300
29716872121,7sleep12971698-21sh10:56:341
29412832120,1sleep20-21swapper/210:37:392
29119312121,7sleep22802705-21diskmemload10:20:142
28811242121,3sleep12882208-21cat10:00:171
28790132121,7sleep22802705-21diskmemload09:57:582
28650602121,4sleep10-21swapper/109:50:111
28504312121,7sleep12852116-21kthreadcore09:40:161
28082412121,7sleep02802705-21diskmemload09:11:090
27647602121,5sleep22764806-21ntpq08:30:202
27235742120,8sleep02723787-21smartctl07:55:150
31410922111,5sleep12802705-21diskmemload12:36:191
31350402111,4sleep12802705-21diskmemload12:34:551
31056312111,3sleep20-21swapper/212:16:322
30954962111,6sleep02802705-21diskmemload12:10:250
30902412111,6sleep23090254-21rm12:09:352
30613332111,4sleep02802705-21diskmemload11:51:490
30373832111,4sleep02802705-21diskmemload11:40:060
30367852111,4sleep22802705-21diskmemload11:39:362
30344902111,5sleep02802705-21diskmemload11:35:410
30267062110,1sleep20-21swapper/211:30:372
30259292111,5sleep03025988-21ntpq11:30:240
30081602111,6sleep03009810-21latency11:20:170
29861902111,5sleep22986253-21ntp_states11:05:252
29663882111,6sleep02966395-21rm10:55:050
29647982111,3sleep20-21swapper/210:52:502
29583782111,3sleep20-21swapper/210:49:532
29492332111,3sleep20-21swapper/210:42:442
29345552111,3sleep00-21swapper/010:34:270
29017792115,2sleep02901787-21sh10:12:390
28822092111,6sleep22883949-21kthreadcore10:00:182
28556492111,6sleep22855659-21sh09:42:382
28481272111,3sleep10-21swapper/109:38:091
28472242110,1sleep00-21swapper/009:36:240
28401692111,4sleep02802705-21diskmemload09:33:020
28399782111,4sleep12802705-21diskmemload09:32:391
28074032111,3sleep20-21swapper/209:10:242
27766042111,6sleep12776865-21/usr/sbin/munin08:45:141
27543062111,4sleep12754355-21ntpq08:20:211
27127892110,1sleep20-21swapper/207:45:152
26933472112,5sleep1331ktimers/107:25:181
31412162101,4sleep22802705-21diskmemload12:36:292
31139842101,3sleep20-21swapper/212:20:492
30999122101,4sleep03100174-21df12:15:130
30544792101,2sleep20-21swapper/211:49:462
30444122101,2sleep20-21swapper/211:43:322
30289462101,3sleep10-21swapper/111:34:291
30199912101,3sleep10-21swapper/111:27:411
30142222101,4sleep03014439-21hddtemp_smartct11:25:150
30058732101,4sleep23005904-21ssh11:20:112
29874422101,2sleep10-21swapper/111:06:401
29797692100,1sleep20-21swapper/211:01:512
29714232100,3sleep22802705-21diskmemload10:56:072
29503282101,3sleep02802705-21diskmemload10:44:370
29243922101,3sleep22802705-21diskmemload10:25:442
29243922101,3sleep22802705-21diskmemload10:25:442
29133192101,5sleep12915005-21cut10:20:181
29109292101,2sleep00-21swapper/010:19:520
29035472101,4sleep22903580-21apt-get10:15:142
29021522100,1sleep20-21swapper/210:13:212
28800062100,1sleep00-21swapper/009:59:490
28713302100,1sleep02871344-21ssh09:53:020
28237712100,6sleep02823781-21ssh09:21:190
28198882101,5sleep22821616-21kthreadcore09:20:172
28178232100,4sleep02802705-21diskmemload09:19:200
28161572101,3sleep22802705-21diskmemload09:16:212
28156692101,2sleep10-21swapper/109:15:321
28088252101,2sleep10-21swapper/109:12:061
27979982101,4sleep12798122-21perl09:05:181
27911322101,4sleep12791181-21ntpq08:55:201
27716192101,2sleep20-21swapper/208:40:182
27502912101,4sleep02750310-21apt-get08:20:120
27396902101,4sleep12739793-21perl08:10:151
27381392100,1sleep00-21swapper/008:05:200
27220822101,4sleep02722139-21ntp_states07:50:210
27219792101,4sleep22722027-21ntpq07:50:192
26971632101,5sleep12697315-21interrupts07:30:141
26859232101,5sleep02685928-21cron07:20:000
3136094291,4sleep03136216-21grep12:35:170
3079211291,3sleep10-21swapper/112:01:511
2994128291,4sleep02994276-21ntp_192.168.11411:10:190
2943065291,2sleep10-21swapper/110:40:051
2862520291,2sleep20-21swapper/209:46:162
2857259291,3sleep10-21swapper/109:45:141
2832079291,2sleep20-21swapper/209:27:562
2792417290,4sleep12792436-21apt-get09:00:121
2787533291,4sleep02787688-21interrupts08:55:150
2787145291,4sleep22787406-21df_inode08:55:122
2764646291,4sleep12764759-21ntpq08:30:191
2740656291,4sleep02742401-21tr08:10:180
2716651291,4sleep02716770-21modprobe07:45:210
2707054291,4sleep02707048-31find07:38:520
2682969291,4sleep12684712-21sed07:15:161
3020037281,4sleep23020047-21rm11:27:462
2900688280,2sleep10-21swapper/110:10:441
2830872281,3sleep02830885-21ssh09:25:530
2801750281,3sleep02801806-21ntpq09:05:250
2793295281,3sleep22794955-21kthreadcore09:00:182
2781485283,3sleep2431ktimers/208:48:232
2772834281,3sleep10-21swapper/108:40:211
2755563281,3sleep22755639-21cpuspeed_turbos08:25:152
2755563281,3sleep22755639-21cpuspeed_turbos08:25:142
2755541281,3sleep12755560-21apt-get08:25:141
2755541281,3sleep12755560-21apt-get08:25:141
2749648280,1sleep20-21swapper/208:15:292
2744980281,2sleep12744999-21apt-get08:15:121
2729541281,3sleep12731210-21cut08:00:161
2728427281,3sleep22728827-21packagekitd08:00:012
2728170280,1sleep10-21swapper/107:58:291
2721789280,4sleep10-21swapper/107:50:181
2707753281,3sleep02707834-21fschecks_time07:40:150
2707102281,2sleep20-21swapper/207:39:062
2707060281,2sleep10-21swapper/107:38:531
2686778281,3sleep22687812-21kthreadcore07:20:172
2935614271,3sleep12935830-21hddtemp_smartct10:35:171
2781526270,1sleep00-21swapper/008:48:540
2770054271,3sleep22770106-21/usr/sbin/munin08:35:202
2765624270,1sleep00-21swapper/008:33:020
2744426270,1sleep20-21swapper/208:14:482
2738462271,3sleep20-21swapper/208:05:272
2729359271,3sleep02729540-21kthreadcore08:00:160
2723886271,2sleep20-21swapper/207:55:172
2707471270,2sleep20-21swapper/207:40:132
2700889271,2sleep02623694-21kworker/0:2+events07:30:210
2678312270,1sleep20-21swapper/207:11:332
2802423260,1sleep20-21swapper/209:06:312
2797266260,1sleep00-21swapper/009:02:390
2786702261,2sleep12786703-21pool-udisksd08:53:001
2782427261,2sleep20-21swapper/208:50:162
2766331261,2sleep00-21swapper/008:35:150
2760293260,1sleep00-21swapper/008:27:220
2760293260,1sleep00-21swapper/008:27:210
2717740260,1sleep10-21swapper/107:49:351
2695178260,2sleep22695440-21strings07:25:192
2691478261,2sleep00-21swapper/007:25:130
2691063260,1sleep10-21swapper/107:23:211
2681178260,2sleep22681391-21hddtemp_smartct07:15:132
2782154250,2sleep00-21swapper/008:50:130
2707835250,2sleep10-21swapper/107:40:161
2689588250,1sleep00-21swapper/007:20:190
2750550240,1sleep20-21swapper/208:20:132
2744968240,1sleep00-21swapper/008:15:020
2673565240,1sleep00-21swapper/007:10:090
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional