You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-10 - 20:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Tue Mar 10, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2915823212253,64sleep30-21swapper/307:06:183
317551821140,3sleep23175517-21unixbench-2d10:20:232
291578128656,12sleep10-21swapper/107:05:441
291577528158,17sleep00-21swapper/007:05:400
291539927656,12sleep20-21swapper/207:05:252
291622799301,19cyclictest0-21swapper/312:26:133
291622799272,18cyclictest66950irq/132-enp1s0-TxRx-310:51:443
291622799270,18cyclictest0-21swapper/310:18:523
32812892240,2sleep10-21swapper/111:20:561
291622799231,19cyclictest3258102-21turbostat11:14:553
291622799231,19cyclictest3258102-21turbostat11:14:543
291622799221,9cyclictest0-21swapper/312:18:263
2916227992214,5cyclictest0-21swapper/310:32:163
291622799221,15cyclictest0-21swapper/311:57:223
291622799220,2cyclictest0-21swapper/312:00:153
291622799220,1cyclictest0-21swapper/310:02:083
33166522211,2sleep00-21swapper/011:41:290
32288092211,4sleep23047263-21diskmemload10:51:232
291622799215,13cyclictest0-21swapper/311:28:583
2916227992114,4cyclictest0-21swapper/310:48:543
291622799211,17cyclictest0-21swapper/310:20:183
291622799211,13cyclictest3098416-21chrt09:38:043
291622799210,1cyclictest0-21swapper/312:36:203
291622799210,1cyclictest0-21swapper/311:31:273
291622799210,1cyclictest0-21swapper/311:18:253
291622799210,1cyclictest0-21swapper/311:04:533
291622799210,1cyclictest0-21swapper/310:28:453
291622799210,1cyclictest0-21swapper/309:41:253
291622799210,16cyclictest0-21swapper/311:47:443
291622799210,13cyclictest0-21swapper/309:55:143
29979532200,2sleep10-21swapper/108:20:171
291622799209,7cyclictest3333771-21taskset11:51:013
2916227992015,3cyclictest0-21swapper/311:05:033
291622799201,2cyclictest0-21swapper/310:57:563
291622799201,16cyclictest0-21swapper/309:54:573
291622799201,16cyclictest0-21swapper/307:15:383
291622799201,15cyclictest3317202-21taskset11:42:103
291622799201,15cyclictest0-21swapper/312:30:483
291622799201,15cyclictest0-21swapper/309:20:493
291622799201,11cyclictest66950irq/132-enp1s0-TxRx-310:35:113
291622799201,10cyclictest0-21swapper/310:10:113
291622799200,1cyclictest0-21swapper/312:22:533
33464792191,3sleep20-21swapper/212:00:152
33094192198,2sleep2171rcu_preempt11:38:302
32753982191,4sleep23047263-21diskmemload11:19:462
32727742191,3sleep10-21swapper/111:16:171
31581312191,3sleep00-21swapper/010:10:280
291622799191,2cyclictest0-21swapper/308:25:223
291622799191,2cyclictest0-21swapper/308:25:213
291622799191,15cyclictest0-21swapper/312:13:203
291622799191,15cyclictest0-21swapper/312:06:253
291622799191,15cyclictest0-21swapper/309:10:123
291622799191,15cyclictest0-21swapper/307:40:163
291622799191,14cyclictest3009750-21turbostat08:35:003
291622799191,14cyclictest2972388-21turbostat08:00:003
291622799191,14cyclictest2961783-21turbostat07:50:003
291622799191,14cyclictest0-21swapper/311:20:223
291622799191,14cyclictest0-21swapper/310:05:213
291622799191,14cyclictest0-21swapper/309:47:093
291622799191,14cyclictest0-21swapper/309:27:033
291622799190,1cyclictest0-21swapper/309:31:033
33697452181,4sleep00-21swapper/012:12:270
315227321811,3sleep23152287-21sh10:09:382
30640512181,13sleep03064065-21ssh09:19:060
29353142181,3sleep10-21swapper/107:25:161
291622799182,13cyclictest2977731-21turbostat08:09:543
291622799181,9cyclictest0-21swapper/311:35:133
291622799181,9cyclictest0-21swapper/309:16:013
291622799181,8cyclictest0-21swapper/310:41:353
2916227991814,2cyclictest0-21swapper/307:30:123
291622799181,13cyclictest3025721-21turbostat08:50:013
291622799181,13cyclictest2921523-21turbostat07:14:593
291622799181,13cyclictest0-21swapper/308:15:123
32634422171,3sleep10-21swapper/111:10:301
32634422171,3sleep10-21swapper/111:10:301
32050122171,3sleep20-21swapper/210:39:372
31248012171,4sleep13124812-21ssh09:53:101
291622799171,13cyclictest2934559-21turbostat07:29:563
291622799171,13cyclictest0-21swapper/308:55:023
291622799170,14cyclictest0-21swapper/308:35:123
291622799170,13cyclictest0-21swapper/309:05:203
33854722161,7sleep13385697-21users12:20:261
33623302161,9sleep20-21swapper/212:09:082
33359622161,9sleep00-21swapper/011:53:560
32567912161,3sleep00-21swapper/011:08:170
323841921611,2sleep10-21swapper/110:57:321
32137802161,11sleep23213794-21ssh10:44:392
31931692160,3sleep20-21swapper/210:30:282
31769422161,4sleep03176956-21ssh10:22:070
31131062165,5sleep23113208-21ntpq09:45:172
30977192161,3sleep00-21swapper/009:37:090
30728622161,2sleep10-21swapper/109:24:031
291622799161,9cyclictest66950irq/132-enp1s0-TxRx-307:20:213
2916227991611,4cyclictest0-21swapper/307:38:133
291622799161,12cyclictest3031024-21turbostat08:55:003
291622799160,13cyclictest0-21swapper/308:40:113
291622799160,13cyclictest0-21swapper/307:50:163
33975892151,10sleep03397603-21ssh12:29:350
33508992151,2sleep10-21swapper/112:00:331
30356142151,9sleep03035661-21ntpq08:55:200
2916227991510,3cyclictest0-21swapper/308:20:113
33869452141,7sleep00-21swapper/012:22:050
33675852141,7sleep23367650-21ntpq12:10:202
33344622141,3sleep23047263-21diskmemload11:51:572
32475932141,8sleep03047263-21diskmemload11:03:040
32144112141,7sleep00-21swapper/010:45:080
32120562141,2sleep13047263-21diskmemload10:42:211
31925432141,3sleep03192601-21ssh10:30:220
31823782141,8sleep23183537-21memory10:25:172
31697642141,3sleep10-21swapper/110:19:121
31420312141,8sleep13047263-21diskmemload10:02:431
31239972141,9sleep20-21swapper/209:52:072
31078812146,5sleep03107893-21rm09:43:590
29607262141,3sleep12960764-21ntpq07:45:191
291622799141,8cyclictest2983747-21taskset08:10:133
291622799140,2cyclictest0-21swapper/308:00:133
291622799140,11cyclictest0-21swapper/309:00:163
34145142131,1sleep10-21swapper/112:38:301
33956272131,6sleep13047263-21diskmemload12:26:591
33784902130,1sleep20-21swapper/212:17:272
33564562131,7sleep03358130-21cut12:05:180
33336492131,4sleep13333660-21sh11:50:521
33162002131,6sleep10-21swapper/111:40:541
33156302131,7sleep23315857-21users11:40:262
32635362131,5sleep00-21swapper/011:10:380
32635362131,5sleep00-21swapper/011:10:370
31677762131,5sleep00-21swapper/010:16:350
31271252131,8sleep13127207-21/usr/sbin/munin09:55:141
31135882132,6sleep03113629-21sed09:45:230
30868992131,8sleep03086958-21ntpq09:30:190
30550222131,8sleep03055033-21ssh09:13:440
29510012130,8sleep12950675-31find07:40:031
34047852121,7sleep23047263-21diskmemload12:32:092
33991212121,6sleep03399358-21cut12:30:160
33540802121,7sleep03354092-21ssh12:04:450
33198092121,5sleep03320038-21df_inode11:45:130
32985252121,7sleep23298536-21sh11:30:362
32927622121,5sleep20-21swapper/211:29:322
32878882121,6sleep13288792-21/usr/sbin/munin11:25:191
32457402120,1sleep20-21swapper/211:00:372
32370932120,1sleep00-21swapper/010:55:470
32297162121,4sleep10-21swapper/110:52:341
32010882121,6sleep13201186-21ntpq10:35:191
31863812120,1sleep10-21swapper/110:28:011
31506902121,6sleep13047263-21diskmemload10:07:341
31326612121,7sleep03132675-21ssh09:56:570
31316262120,1sleep20-21swapper/209:55:352
30726352120,1sleep00-21swapper/009:23:460
30262482121,7sleep23026336-21grep08:50:132
30208522121,7sleep03020936-21/usr/sbin/munin08:45:140
29677682121,6sleep22967884-21irqstats07:55:142
29334382121,5sleep02933530-21ntpq07:20:180
34148712110,1sleep20-21swapper/212:38:592
33790292111,7sleep13379037-21sh12:18:101
33279102111,4sleep13047263-21diskmemload11:49:471
33196122111,3sleep20-21swapper/211:45:032
33089942111,3sleep00-21swapper/011:37:550
33022572111,5sleep13302281-21apt-get11:35:121
32994252110,1sleep10-21swapper/111:31:471
32551382111,4sleep20-21swapper/211:06:072
32384332111,3sleep20-21swapper/210:57:322
32216392111,3sleep20-21swapper/210:48:262
31954712111,3sleep10-21swapper/110:33:311
31691162110,3sleep20-21swapper/210:18:212
31607132111,5sleep23047263-21diskmemload10:13:512
31506752110,2sleep00-21swapper/010:07:330
31151522111,3sleep10-21swapper/109:47:001
31044252111,4sleep23104483-21ntpq09:40:202
30975242111,4sleep13047263-21diskmemload09:36:541
30745852111,5sleep23075915-21tr09:25:172
30702582111,3sleep20-21swapper/209:20:392
30645512111,3sleep20-21swapper/209:19:432
30632392110,1sleep10-21swapper/109:18:031
30557882110,1sleep10-21swapper/109:14:451
30329092111,5sleep13034674-21cut08:55:171
30257112111,4sleep13025736-21munin-run08:50:001
30140182111,5sleep03014064-21ntpq08:35:200
30046832111,7sleep03004936-21sh08:30:110
29934102110,1sleep10-21swapper/108:16:191
29891622111,5sleep22989233-21irqstats08:15:152
29820482111,5sleep22982102-21ntpq08:05:212
29510942111,5sleep22951298-21diskstats07:40:122
29238072111,5sleep12924058-21df_inode07:15:151
29169812110,7sleep02918012-21grep07:10:160
29164632111,5sleep12916697-21df_inode07:10:131
34129902101,2sleep00-21swapper/012:36:280
33985912101,4sleep13398853-21df12:30:131
33894992101,3sleep20-21swapper/212:25:092
33807582101,4sleep23047263-21diskmemload12:20:072
33785172101,3sleep00-21swapper/012:17:290
33633032101,4sleep13047263-21diskmemload12:10:071
33592482101,2sleep10-21swapper/112:05:271
33447282101,3sleep13047263-21diskmemload11:58:571
33428932101,3sleep00-21swapper/011:56:310
33371492101,3sleep23337177-21apt-get11:55:092
32890032101,5sleep03289066-21ntpq11:25:210
32838182100,1sleep00-21swapper/011:24:160
32812062101,3sleep20-21swapper/211:20:502
32637082101,3sleep20-21swapper/211:10:512
32637082101,3sleep20-21swapper/211:10:512
32495792101,4sleep13249080-21apt-get11:05:021
32456312101,3sleep10-21swapper/111:00:281
32144742101,4sleep13214493-21apt-get10:45:131
32057202101,4sleep03205744-21apt-get10:40:110
32023512101,3sleep00-21swapper/010:36:060
31867772100,6sleep00-21swapper/010:28:320
31574782101,5sleep13157545-21ntp_states10:10:211
31232282100,1sleep00-21swapper/009:51:050
31006352101,4sleep13100855-21hddtemp_smartct09:40:161
30996832101,2sleep20-21swapper/209:39:442
30893412101,3sleep10-21swapper/109:32:411
30885882101,3sleep20-21swapper/209:31:422
30796172101,3sleep00-21swapper/009:26:270
30780422101,5sleep13078135-21/usr/sbin/munin09:25:201
30540862101,3sleep20-21swapper/209:12:302
30409832101,4sleep23041029-21ntpq09:00:202
30300432101,4sleep13030091-21ntpq08:50:211
30265242101,4sleep03026874-21cut08:50:160
30153152101,3sleep10-21swapper/108:40:101
29993322101,4sleep22999352-21apt-get08:25:122
29993322101,4sleep22999352-21apt-get08:25:122
29833382101,5sleep02983370-21cpu08:10:110
29819332101,4sleep12982046-21ntpq08:05:201
29777192101,5sleep12977746-21perf08:04:591
29777112100,2sleep00-21swapper/008:04:540
29389252101,4sleep22938972-21ntpq07:25:222
29288822101,5sleep22928904-21munin-run07:20:012
29278622101,4sleep02927900-21ntpq07:15:220
3300379291,4sleep03300390-21ssh11:33:020
3273689290,1sleep00-21swapper/011:17:280
3228283291,2sleep00-21swapper/010:50:410
3175308291,4sleep13175486-21ssh10:20:231
3140978291,3sleep23047263-21diskmemload10:01:212
3140798291,3sleep03047263-21diskmemload10:01:060
3041866290,3sleep00-21swapper/009:03:420
3040921291,4sleep13040981-21ntpq09:00:191
2994560291,4sleep22995464-21cut08:20:152
2984880290,2sleep1171rcu_preempt08:10:161
2983606291,4sleep22983744-21grep08:10:132
2966757290,1sleep10-21swapper/107:51:131
2951005290,4sleep02950675-31find07:40:060
2945200290,1sleep20-21swapper/207:34:102
2940738293,3sleep12940770-21kthreadcore07:30:171
3031518280,3sleep23031028-21apt-get08:55:022
3020643280,3sleep23020662-21apt-get08:45:122
3016337281,3sleep23017407-21kthreadcore08:40:152
3003494282,3sleep00-21swapper/008:25:230
3003494282,3sleep00-21swapper/008:25:220
2997236283,2sleep00-21swapper/008:20:170
2961668280,1sleep20-21swapper/207:48:572
2945810281,3sleep22945913-21perl07:35:142
2945116281,1sleep00-21swapper/007:33:090
2939828280,2sleep00-21swapper/007:29:030
2920611281,3sleep22920696-21ps07:10:222
3046207271,3sleep10-21swapper/109:05:191
3043883271,3sleep20-21swapper/209:05:162
3014747270,1sleep10-21swapper/108:36:261
3008813270,3sleep13008861-21/usr/sbin/munin08:30:201
3004285270,1sleep10-21swapper/108:28:371
3004285270,1sleep10-21swapper/108:28:361
2982981270,5sleep00-21swapper/008:09:000
2969041271,3sleep00-21swapper/007:55:170
2961573270,5sleep00-21swapper/007:47:470
2946687271,3sleep10-21swapper/107:35:171
2945530270,2sleep00-21swapper/007:35:120
2934380270,1sleep20-21swapper/207:23:042
2930006271,2sleep10-21swapper/107:20:151
3047232260,4sleep00-21swapper/009:09:430
3014790260,4sleep20-21swapper/208:37:002
3004985260,2sleep20-21swapper/208:30:132
2990508261,2sleep00-21swapper/008:15:160
2973227261,2sleep20-21swapper/208:00:152
2972155260,4sleep10-21swapper/107:57:141
2966912260,1sleep00-21swapper/007:53:120
2962009262,2sleep22916221-21cyclictest07:50:102
3020272250,3sleep00-21swapper/008:43:390
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional