You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-11 - 20:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Thu Dec 11, 2025 12:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
37607819914491,1443cyclictest41339042sleep311:45:483
3760490212552,68sleep30-21swapper/307:08:073
376055328553,6sleep10-21swapper/107:08:571
376036528152,6sleep00-21swapper/007:06:320
376030427862,10sleep20-21swapper/207:05:452
376078199322,13cyclictest14043-21taskset12:34:313
3760781992623,1cyclictest0-21swapper/307:55:283
3760781992522,1cyclictest0-21swapper/310:20:003
3760781992521,2cyclictest0-21swapper/309:50:013
3760781992420,2cyclictest0-21swapper/308:15:003
376078199241,18cyclictest3988463-21taskset10:10:473
40429212231,2sleep20-21swapper/210:47:272
3760781992320,1cyclictest0-21swapper/310:20:233
376078199231,18cyclictest4165203-21taskset12:06:373
376078199231,18cyclictest0-21swapper/311:33:403
376078199231,14cyclictest3912002-21taskset09:20:323
376078199231,14cyclictest3912002-21taskset09:20:313
376078199222,13cyclictest4068085-21taskset11:05:073
376078199222,13cyclictest4068085-21taskset11:05:063
3760781992220,1cyclictest0-21swapper/308:40:273
3760781992217,2cyclictest0-21swapper/310:50:173
3760781992212,7cyclictest4013714-21sleep10:30:003
3760781992211,7cyclictest4192121-21taskset12:25:173
376078199221,16cyclictest4191220-21turbostat12:24:593
376078199221,15cyclictest3891090-21turbostat09:09:593
376078199220,20cyclictest0-21swapper/308:30:253
41565422211,12sleep13891089-21diskmemload12:00:261
41565422211,12sleep13891089-21diskmemload12:00:251
376078199212,13cyclictest3942952-21taskset09:41:493
3760781992118,2cyclictest0-21swapper/307:55:013
3760781992117,2cyclictest0-21swapper/312:01:053
3760781992117,2cyclictest0-21swapper/312:01:053
3760781992117,2cyclictest0-21swapper/310:35:193
3760781992114,4cyclictest41195372sleep311:37:553
41902682200,1sleep00-21swapper/012:23:460
39762862201,7sleep13977035-21kthreadcore10:05:171
376078199207,9cyclictest3818550-21chrt08:00:163
376078199202,13cyclictest4103322-21taskset11:25:553
376078199202,13cyclictest3930283-21taskset09:35:153
3760781992017,1cyclictest0-21swapper/311:20:153
3760781992016,2cyclictest0-21swapper/309:10:233
376078199201,15cyclictest66550irq/131-enp1s0-TxRx-310:46:473
376078199201,14cyclictest4149862-21chrt11:56:423
376078199201,14cyclictest3958355-21chrt09:52:273
376078199200,14cyclictest205652sleep312:36:403
41579802191,9sleep04157983-21ssh12:02:260
41579802191,9sleep04157983-21ssh12:02:260
41282992191,11sleep04128307-21ssh11:44:370
39659192197,2sleep0171rcu_preempt09:57:100
376078199199,7cyclictest3843119-21chrt08:22:253
376078199198,6cyclictest3789520-21taskset07:32:113
376078199192,12cyclictest4088027-21taskset11:16:253
376078199192,12cyclictest4034982-21taskset10:41:343
3760781991915,2cyclictest0-21swapper/312:10:263
3760781991915,2cyclictest0-21swapper/311:00:113
3760781991915,2cyclictest0-21swapper/310:55:143
3760781991914,3cyclictest0-21swapper/312:15:153
3760781991913,3cyclictest0-21swapper/311:40:173
376078199191,15cyclictest3921351-21chrt09:28:543
376078199191,14cyclictest0-21swapper/311:51:093
39206972181,9sleep03920703-21ssh09:27:430
39047792180,2sleep20-21swapper/209:16:312
376078199182,12cyclictest4020150-21taskset10:32:483
376078199182,11cyclictest3768790-21taskset07:15:033
376078199181,14cyclictest3973000-21taskset10:00:573
376078199181,14cyclictest3904390-21taskset09:15:503
376078199181,13cyclictest4081155-21chrt11:11:253
376078199181,13cyclictest3922575-21chrt09:30:103
376078199181,13cyclictest0-21swapper/310:05:073
376078199181,13cyclictest0-21swapper/308:55:163
376078199181,13cyclictest0-21swapper/308:25:123
41911822171,10sleep24191241-21munin-run12:25:012
41658032171,4sleep03891089-21diskmemload12:07:390
41654212171,3sleep20-21swapper/212:07:012
39673642171,4sleep13891089-21diskmemload09:59:471
39232122175,7sleep13924650-21cut09:30:171
37844472171,11sleep03784481-21perf07:30:000
376078199171,2cyclictest0-21swapper/308:40:003
3760781991712,3cyclictest0-21swapper/308:45:133
376078199171,13cyclictest0-21swapper/307:27:253
376078199171,13cyclictest0-21swapper/307:20:193
376078199171,12cyclictest3960206-21chrt09:55:123
376078199171,12cyclictest3827169-21sleep08:10:003
376078199171,12cyclictest0-21swapper/308:15:213
376078199171,12cyclictest0-21swapper/307:35:213
376078199170,14cyclictest0-21swapper/307:10:133
224952171,4sleep03891089-21diskmemload12:39:170
18212171,11sleep01878-21ntpq12:25:230
41641612161,10sleep10-21swapper/112:05:271
40204032161,4sleep13891089-21diskmemload10:33:191
39743262161,2sleep00-21swapper/010:03:200
39573942161,3sleep13891089-21diskmemload09:50:381
38969322161,10sleep13896945-21sh09:11:531
38383352161,10sleep23838482-21ls08:20:122
376078199161,9cyclictest3874054-21taskset08:50:183
376078199161,12cyclictest3800507-21turbostat07:45:003
376078199161,10cyclictest3880918-21taskset09:00:123
41588212151,2sleep20-21swapper/212:03:562
41588212151,2sleep20-21swapper/212:03:562
40826182151,9sleep11-21systemd11:15:001
40676232157,4sleep14067628-21ssh11:04:421
39372902151,8sleep03891089-21diskmemload09:40:070
39246512151,9sleep23926377-21cut09:30:182
38274242151,4sleep03827683-21df_inode08:10:110
38007752151,5sleep23800975-21cstates07:45:132
41822642141,7sleep13891089-21diskmemload12:19:141
41762422141,9sleep04177855-21cut12:15:170
41418272141,8sleep24141837-21ssh11:51:252
39744932141,7sleep13891089-21diskmemload10:03:371
38709542140,3sleep1321rcuc/108:50:161
131632141,3sleep213177-21ssh12:33:192
41897572131,8sleep14189768-21ssh12:23:031
41484472131,6sleep14148496-21ntpq11:55:211
41259492135,3sleep13891089-21diskmemload11:40:301
41125742131,4sleep03891089-21diskmemload11:33:270
41065572131,3sleep24106636-21/usr/sbin/munin11:30:172
40985372130,8sleep14098593-21ssh11:25:151
40578422130,7sleep244-21ksoftirqd/210:56:092
40238132131,7sleep04025567-21cut10:35:180
39732272130,8sleep20-21swapper/210:01:252
39521682131,8sleep03952208-21perf09:50:010
38060012131,6sleep03806020-21apt-get07:50:110
376078199131,9cyclictest3805390-21chrt07:45:333
26192131,3sleep22626-21ssh12:25:372
43972121,4sleep13891089-21diskmemload12:28:041
41740512120,1sleep20-21swapper/212:13:542
41367622121,7sleep04136872-21fschecks_time11:50:140
41352862121,3sleep20-21swapper/211:48:212
41178832121,6sleep23891089-21diskmemload11:35:302
40979482121,5sleep00-21swapper/011:25:010
40503182121,5sleep13891089-21diskmemload10:51:191
40349622121,4sleep13891089-21diskmemload10:41:331
40282302121,5sleep20-21swapper/210:38:372
40127592121,7sleep14012763-21ssh10:28:221
40056912121,7sleep04005699-21ssh10:24:160
40028652121,7sleep14002922-21ssh10:20:201
39966222121,4sleep23891089-21diskmemload10:17:012
39897212121,3sleep03989732-21ssh10:13:260
39661562121,5sleep23891089-21diskmemload09:57:362
39438242120,1sleep10-21swapper/109:43:441
39308222121,6sleep03932399-21kthreadcore09:35:160
39299962123,3sleep13760772-21cyclictest09:35:131
39146452122,4sleep13914669-21apt-get09:25:121
39139742121,5sleep20-21swapper/209:24:252
39139742121,5sleep20-21swapper/209:24:252
39045712121,4sleep13891089-21diskmemload09:16:101
38934912121,7sleep23895220-21kthreadcore09:10:182
38860882121,7sleep23886300-21cat09:05:122
38810602121,5sleep03881269-21hddtemp_smartct09:00:140
38260112121,6sleep23826124-21ntpq08:05:172
38234212121,7sleep03825152-21kthreadcore08:05:150
38047202121,7sleep13804767-21ld07:45:201
37781682120,2sleep20-21swapper/207:20:202
41746862111,4sleep04174737-21munin-run12:15:000
41457512112,5sleep24147368-21cut11:55:182
41424542111,4sleep13891089-21diskmemload11:52:331
41288602111,5sleep04128879-21apt-get11:45:140
41206482111,4sleep13891089-21diskmemload11:39:541
40958852111,4sleep13891089-21diskmemload11:21:341
40880752111,4sleep13891089-21diskmemload11:16:331
40878432111,4sleep23891089-21diskmemload11:15:582
40758102111,4sleep24075831-21apt-get11:10:122
40735152111,4sleep13891089-21diskmemload11:06:381
40735152111,4sleep13891089-21diskmemload11:06:381
40722422111,3sleep00-21swapper/011:05:210
40722422111,3sleep00-21swapper/011:05:210
40664942111,4sleep23891089-21diskmemload11:02:512
40660502110,1sleep00-21swapper/011:02:040
40513132111,4sleep03891089-21diskmemload10:53:070
40431602111,4sleep13891089-21diskmemload10:47:531
40203042111,3sleep00-21swapper/010:33:060
40201392111,4sleep23891089-21diskmemload10:32:482
40118802111,3sleep20-21swapper/210:26:432
39976572111,4sleep13891089-21diskmemload10:18:521
39973372110,1sleep00-21swapper/010:18:180
39834122111,4sleep13891089-21diskmemload10:10:101
39813622111,4sleep03891089-21diskmemload10:06:550
39798212111,4sleep23891089-21diskmemload10:05:222
39596582111,4sleep03891089-21diskmemload09:54:460
39363072111,4sleep23891089-21diskmemload09:38:442
39128912111,6sleep13891089-21diskmemload09:22:201
39128912111,6sleep13891089-21diskmemload09:22:201
39123872111,4sleep03891089-21diskmemload09:21:180
39123872111,4sleep03891089-21diskmemload09:21:180
38702112111,5sleep00-21swapper/008:50:120
38581422111,4sleep03858190-21ntpq08:35:210
38335452114,3sleep03834694-21cut08:15:170
38171472111,6sleep03817319-21tail08:00:130
37954782111,6sleep23795549-21sort07:40:122
37950792110,1sleep20-21swapper/207:39:002
224132111,4sleep13891089-21diskmemload12:39:111
41727562101,3sleep13891089-21diskmemload12:11:251
41498322101,2sleep00-21swapper/011:56:400
41354802101,3sleep10-21swapper/111:48:401
41256622101,4sleep24125915-21users11:40:272
41206352101,3sleep00-21swapper/011:39:530
41118372101,3sleep13891089-21diskmemload11:32:161
40974562101,4sleep23891089-21diskmemload11:24:152
40956022101,3sleep00-21swapper/011:21:050
40880642101,3sleep00-21swapper/011:16:320
40739242101,3sleep23891089-21diskmemload11:07:202
40739242101,3sleep23891089-21diskmemload11:07:202
40577112101,3sleep10-21swapper/110:55:561
40491222101,4sleep24049177-21ntpq10:50:202
40440052101,4sleep03891089-21diskmemload10:49:220
40358402101,2sleep20-21swapper/210:43:062
40267232100,3sleep13891089-21diskmemload10:35:471
40109532101,4sleep04011207-21users10:25:270
40041862101,3sleep23891089-21diskmemload10:21:302
39839542105,2sleep23760772-21cyclictest10:10:152
39578442101,3sleep20-21swapper/209:51:312
39515882101,4sleep13891089-21diskmemload09:48:481
39507072101,3sleep20-21swapper/209:47:102
39429432100,3sleep20-21swapper/209:41:492
39214122101,3sleep20-21swapper/209:29:022
38993552102,4sleep03899546-21ls09:15:130
38795492101,4sleep03879603-21ntpq08:55:250
38685502101,5sleep13868663-21ntpq08:45:201
38646882101,4sleep23864707-21apt-get08:45:122
37950882101,4sleep13795031-31find07:39:071
37891312101,5sleep23789376-21users07:30:262
37847972101,4sleep13785000-21diskstats07:30:131
37781152101,4sleep13778166-21ntpq07:20:191
37728592101,4sleep03772905-21ntpq07:15:210
146672101,3sleep20-21swapper/212:35:032
132962101,3sleep13891089-21diskmemload12:33:301
4179057291,2sleep20-21swapper/212:15:192
4104301291,2sleep20-21swapper/211:27:402
4078289290,1sleep00-21swapper/011:10:180
4052589291,3sleep04052613-21apt-get10:55:100
4035520290,3sleep03891089-21diskmemload10:42:320
3896865291,2sleep00-21swapper/009:11:440
3879399290,1sleep20-21swapper/208:55:222
3863741291,5sleep10-21swapper/108:40:271
3855144291,4sleep13856857-21cut08:35:181
3853171291,4sleep13848834-21/usr/sbin/munin08:30:251
3847550291,3sleep23847596-21ntpq08:25:202
3843323291,4sleep13843327-21cron08:25:001
3820868291,3sleep23820917-21ntpq08:00:202
3768052290,3sleep20-21swapper/207:15:032
3765274291,3sleep03765323-21ntpq07:10:190
3761538291,4sleep23761755-21awk07:10:142
11442291,2sleep00-21swapper/012:31:000
3922589281,3sleep03922614-21apt-get09:30:120
3886066280,2sleep10-21swapper/109:05:121
3864211283,3sleep0151ktimers/008:42:190
3833264281,3sleep23833446-21gltestperf08:15:152
3833060281,2sleep13833263-21diskstats08:15:141
3822124281,3sleep13822125-21sshd08:05:081
3795087281,2sleep00-21swapper/007:39:060
3772907281,3sleep13772961-21ntpq07:15:221
3886362271,2sleep03886574-21smartctl09:05:150
3885556270,1sleep10-21swapper/109:01:511
3853760270,1sleep00-21swapper/008:33:450
3853275270,3sleep23853491-21users08:30:262
3847888271,2sleep03848043-21timerwakeupswit08:25:270
3842643271,3sleep03842878-21unixbench_multi08:20:260
3821867271,2sleep13821934-21latency_hist08:05:001
3812272271,2sleep10-21swapper/107:55:181
3810716270,1sleep20-21swapper/207:51:352
3805730270,1sleep00-21swapper/007:49:560
3795456270,2sleep10-21swapper/107:40:111
3789394270,1sleep00-21swapper/007:30:310
3779435270,4sleep10-21swapper/107:25:121
3880347260,1sleep10-21swapper/108:58:151
3874984260,1sleep20-21swapper/208:52:062
3859734261,2sleep20-21swapper/208:40:162
3858843260,1sleep20-21swapper/208:36:022
3832357261,2sleep10-21swapper/108:14:021
3816497260,1sleep20-21swapper/207:59:042
3811529260,2sleep00-21swapper/007:55:130
3806023261,2sleep10-21swapper/107:50:121
3800308260,1sleep00-21swapper/007:42:420
3781257260,2sleep23783074-21kthreadcore07:25:172
3880791250,2sleep20-21swapper/209:00:122
3868774250,1sleep00-21swapper/008:45:230
3848409250,3sleep10-21swapper/108:27:571
3778569250,1sleep00-21swapper/007:20:250
3763883250,1sleep10-21swapper/107:10:171
3827787240,1sleep20-21swapper/208:10:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional