You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-03 - 05:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Sat Jan 03, 2026 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
241922521450,1sleep20-21swapper/219:45:192
2376557212558,62sleep30-21swapper/319:06:363
23764992113104,6sleep10-21swapper/119:05:541
237650329063,19sleep00-21swapper/019:05:570
237676927851,10sleep20-21swapper/219:09:202
26422432230,11sleep134-21ksoftirqd/123:20:001
2376948992219,1cyclictest0-21swapper/321:55:253
237694899221,17cyclictest2520729-21taskset21:20:263
237694899211,15cyclictest0-21swapper/320:35:133
237694899202,12cyclictest2715030-21taskset00:25:193
2376948992018,1cyclictest0-21swapper/319:40:013
2376948992017,1cyclictest0-21swapper/300:20:133
2376948992017,1cyclictest0-21swapper/300:15:013
237694899201,15cyclictest2694383-21taskset00:05:323
2376948992010,7cyclictest2673046-21taskset23:45:273
26850822190,11sleep014-21ksoftirqd/000:00:160
237694899196,6cyclictest2494621-21taskset20:57:423
2376948991916,1cyclictest0-21swapper/319:45:183
237694899191,14cyclictest2621311-21taskset23:00:113
237694899191,14cyclictest0-21swapper/319:50:123
237694899191,11cyclictest2534688-21taskset21:35:173
25572322181,12sleep12557281-21ntpq21:55:201
2376948991815,1cyclictest0-21swapper/321:50:163
2376948991815,1cyclictest0-21swapper/321:10:163
2376948991815,1cyclictest0-21swapper/320:45:143
237694899181,14cyclictest0-21swapper/323:25:123
2376948991811,3cyclictest25624222sleep322:00:203
237694899181,13cyclictest0-21swapper/322:25:113
237694899181,13cyclictest0-21swapper/321:40:173
237694899181,11cyclictest2430281-21taskset19:55:173
25411022171,10sleep22541149-21ntpq21:40:182
24901852171,7sleep02490382-21if_enp1s020:55:150
2376948991715,1cyclictest0-21swapper/321:05:163
2376948991714,1cyclictest0-21swapper/323:15:183
2376948991714,1cyclictest0-21swapper/322:30:163
2376948991714,1cyclictest0-21swapper/300:00:143
2376948991713,2cyclictest0-21swapper/319:30:163
237694899171,13cyclictest0-21swapper/319:15:113
237694899171,10cyclictest2720440-21chrt00:30:283
237694899171,10cyclictest2604182-21taskset22:40:183
237694899171,10cyclictest2526352-21taskset21:30:003
237694899171,10cyclictest2442403-21taskset20:10:003
237694899170,10cyclictest2456714-21taskset20:20:183
25595322161,10sleep02561242-21tr22:00:160
237694899161,9cyclictest2579399-21taskset22:20:063
2376948991614,1cyclictest0-21swapper/320:10:253
2376948991614,1cyclictest0-21swapper/319:25:143
2376948991613,1cyclictest0-21swapper/323:20:173
2376948991613,1cyclictest0-21swapper/323:05:173
2376948991613,1cyclictest0-21swapper/322:55:143
2376948991613,1cyclictest0-21swapper/300:35:173
237694899161,11cyclictest2474108-21chrt20:40:103
237694899161,11cyclictest0-21swapper/322:45:103
2376948991610,4cyclictest0-21swapper/319:35:123
237694899160,14cyclictest0-21swapper/320:54:213
237694899160,13cyclictest0-21swapper/323:50:133
237694899160,13cyclictest0-21swapper/322:35:193
237694899160,13cyclictest0-21swapper/322:35:193
237694899160,13cyclictest0-21swapper/321:45:203
237694899160,13cyclictest0-21swapper/320:30:213
237694899160,13cyclictest0-21swapper/320:30:213
237694899160,13cyclictest0-21swapper/320:15:293
237694899160,13cyclictest0-21swapper/319:20:173
27108122151,8sleep02710428-21apt-get00:25:010
25321692151,10sleep02532363-21/usr/sbin/munin21:35:150
24950772151,4sleep2800-21snmpd21:00:062
237694899156,6cyclictest2662846-21taskset23:39:023
237694899156,5cyclictest23771202sleep319:10:133
237694899155,6cyclictest2684099-21taskset23:58:273
237694899151,8cyclictest2700196-21taskset00:15:013
237694899151,11cyclictest0-21swapper/323:40:143
237694899151,10cyclictest2495365-21chrt21:00:143
237694899150,2cyclictest0-21swapper/322:15:113
237694899150,13cyclictest0-21swapper/321:30:153
237694899150,13cyclictest0-21swapper/321:15:213
237694899150,12cyclictest0-21swapper/323:30:133
237694899150,12cyclictest0-21swapper/323:10:243
237694899150,12cyclictest0-21swapper/322:10:143
237694899150,12cyclictest0-21swapper/320:25:193
237694899150,11cyclictest0-21swapper/322:05:123
26317932141,5sleep12631978-21expr23:10:111
26255512141,9sleep00-21swapper/023:00:240
237694899147,4cyclictest0-21swapper/322:50:113
237694899141,8cyclictest2436792-21taskset20:04:473
27223082131,7sleep12723991-21kthreadcore00:35:161
26734142132,3sleep20-21swapper/223:50:012
25950932131,8sleep12595110-21grep22:35:131
25950932131,8sleep12595110-21grep22:35:121
25898722131,8sleep12589961-21expr22:30:111
25426802131,8sleep02542941-21df_inode21:45:130
25253462131,8sleep02521357-21/usr/sbin/munin21:25:210
24052622132,5sleep12405281-21apt-get19:35:121
26253642121,7sleep22625418-21pmu-power23:00:222
25887042121,7sleep12588826-21/usr/sbin/munin22:25:221
25271122121,7sleep22527520-21kthreadcore21:30:162
25095792120,7sleep12505586-21/usr/sbin/munin21:10:191
24794452121,5sleep02479648-21diskstats20:45:130
27159442111,3sleep22716121-21latency_hist00:30:002
26832742111,5sleep12683331-21ntp_states23:55:201
26476082111,5sleep02647811-21diskstats23:25:140
26372972111,5sleep02637501-21grep23:15:160
26265782111,5sleep02626820-21df_inode23:05:130
26095872111,4sleep22609631-21ntpq22:45:192
25966782111,2sleep20-21swapper/222:35:172
25966782111,2sleep20-21swapper/222:35:172
25735432110,1sleep00-21swapper/022:10:420
25727822111,5sleep22572871-21ntpq22:10:162
25637212110,1sleep00-21swapper/022:05:130
24571132111,4sleep22457169-21ntp_states20:20:222
24390362111,6sleep02440840-21kthreadcore20:05:170
24285762110,5sleep22430279-21latency19:55:172
23901392110,7sleep12390463-21cut19:20:171
27094132101,4sleep02709460-21ntpq00:20:210
27040022101,4sleep02704143-21ntpq00:15:190
26987902101,4sleep22698931-21ntpq00:10:202
26900402101,5sleep02690109-21cut00:05:130
26672402101,4sleep02667293-21ntp_states23:40:200
26629232101,4sleep02662995-21munin-run23:40:000
26428172101,5sleep12643357-21kthreadcore23:20:171
26095372101,4sleep12609585-21ntpq22:45:181
26009152101,5sleep02601426-21cat22:40:160
25426252102,3sleep12542126-21apt-get21:45:021
25323642104,3sleep12532991-21kthreadcore21:35:161
25265632101,4sleep12526582-21apt-get21:30:111
25201602101,4sleep12520213-21ntp_states21:20:211
25201122101,4sleep02520158-21ntpq21:20:210
25043522101,4sleep22504404-21ntpq21:05:232
25003362101,3sleep12500355-21apt-get21:05:141
24950862101,4sleep12495105-21apt-get21:00:131
24898352101,3sleep10-21swapper/120:55:121
24496842101,5sleep12451500-21cut20:15:201
24472442100,1sleep20-21swapper/220:10:482
24213262101,4sleep22421345-21apt-get19:50:122
24201092101,4sleep12420154-21ntpq19:45:221
24040052101,5sleep22404049-21ntpq19:30:202
23987572101,5sleep10-21swapper/119:25:181
23896272101,4sleep22389880-21df19:20:142
23882712101,4sleep02388319-21ntpq19:15:200
2704435291,4sleep12704452-21readlink00:15:241
2657949291,3sleep12657968-21apt-get23:35:131
2640981291,4sleep22641029-21ntpq23:15:212
2616808291,4sleep22618465-21cut22:55:172
2589963291,4sleep22590138-21diskstats22:30:122
2567763291,4sleep22567877-21awk22:05:232
2537559291,4sleep12537675-21irqstats21:40:141
2535857291,3sleep22535911-21ntpq21:35:212
2506677291,4sleep22508360-21cut21:10:162
2488633291,4sleep02488690-21ntp_states20:50:220
2474086291,3sleep22474105-21apt-get20:40:102
2463590291,4sleep02463758-21cat20:30:140
2463590291,4sleep02463758-21cat20:30:140
2458338291,4sleep02458594-21df20:25:130
2446896291,4sleep12447126-21unixbench_multi20:10:271
2446621291,5sleep00-21swapper/020:10:220
2441935290,2sleep10-21swapper/120:08:071
2435746291,4sleep02435792-21ntpq20:00:200
2433144291,4sleep12434885-21grep20:00:171
2721420281,2sleep00-21swapper/000:35:110
2721152280,1sleep00-21swapper/000:34:550
2720142281,3sleep22720186-21gcc00:30:242
2688810281,3sleep12688817-21sensors_temp00:00:251
2683222280,1sleep00-21swapper/023:55:190
2677916281,3sleep12677973-21ntpq23:50:191
2674523281,3sleep22675961-21kthreadcore23:50:162
2673920280,3sleep02673424-21apt-get23:50:020
2661927280,1sleep20-21swapper/223:35:212
2653306281,3sleep20-21swapper/223:30:162
2647522281,3sleep22647541-21apt-get23:25:112
2643360281,3sleep22645076-21cut23:20:182
2610845280,2sleep10-21swapper/122:50:111
2604655280,5sleep20-21swapper/222:40:272
2568480280,1sleep10-21swapper/122:07:511
2547945281,2sleep10-21swapper/121:50:111
2529215280,5sleep00-21swapper/021:30:180
2473805281,3sleep02473842-21perf20:40:000
2462146281,2sleep22462287-21ntpq20:25:192
2453112281,3sleep12453227-21grep20:20:131
2452085280,1sleep00-21swapper/020:15:270
2441037281,3sleep22441085-21ntpq20:05:192
2421622280,1sleep10-21swapper/119:50:151
2416633281,3sleep02417477-21tr19:45:170
2406434281,3sleep22407319-21kthreadcore19:35:172
2405001281,3sleep12405040-21perf-exec19:35:001
2394787280,3sleep02394806-21apt-get19:25:090
2388373281,2sleep22388430-21ntp_states19:15:212
2377107280,3sleep10-21swapper/119:10:031
2720655270,4sleep10-21swapper/100:30:321
2710954270,2sleep12710972-21apt-get00:25:101
2710152270,1sleep20-21swapper/200:21:382
2706866271,2sleep10-21swapper/100:20:171
2700200270,1sleep20-21swapper/200:15:052
2699867270,1sleep00-21swapper/000:13:590
2694584270,1sleep10-21swapper/100:08:071
2690309271,3sleep22690365-21kthreadcore00:05:152
2689318270,1sleep20-21swapper/200:02:432
2667888270,1sleep10-21swapper/123:41:211
2631771270,2sleep00-21swapper/023:10:100
2631291270,1sleep20-21swapper/223:07:112
2620768270,1sleep00-21swapper/022:56:430
2600083270,1sleep00-21swapper/022:40:000
2600083270,1sleep00-21swapper/022:40:000
2584659270,2sleep20-21swapper/222:25:112
2575570271,2sleep10-21swapper/122:15:161
2573882271,3sleep10-21swapper/122:15:001
2563336270,2sleep20-21swapper/222:03:432
2561243271,3sleep12558465-21/usr/sbin/munin22:00:171
2556624271,3sleep22557081-21needreboot21:55:172
2553780271,2sleep00-21swapper/021:55:150
2547618270,1sleep20-21swapper/221:49:122
2541431271,3sleep00-21swapper/021:40:240
2515826271,2sleep00-21swapper/021:20:000
2512237271,3sleep22512894-21kthreadcore21:15:172
2510858271,2sleep12510876-21apt-get21:15:121
2499889270,1sleep00-21swapper/021:02:410
2494550270,1sleep20-21swapper/220:56:492
2484900270,1sleep10-21swapper/120:50:151
2484029270,1sleep10-21swapper/120:45:581
2483549270,1sleep20-21swapper/220:45:242
2477940270,2sleep12478081-21ntpq20:40:171
2474366270,3sleep00-21swapper/020:40:120
2467868270,4sleep14122902-21kworker/1:2+events20:30:271
2467868270,4sleep14122902-21kworker/1:2+events20:30:271
2453090270,2sleep00-21swapper/020:20:130
2426535270,2sleep00-21swapper/019:55:110
2412400271,2sleep10-21swapper/119:40:171
2411388270,3sleep0221rcuc/019:40:160
2399741271,2sleep2553-21dbus-daemon19:30:002
2384893271,2sleep10-21swapper/119:15:161
2721696261,2sleep20-21swapper/200:35:132
2694985260,1sleep10-21swapper/100:10:031
2684206260,1sleep20-21swapper/223:59:502
2669175261,2sleep02500140-21kworker/0:1-events23:45:170
2663204260,1sleep20-21swapper/223:40:102
2657068261,2sleep00-21swapper/023:30:260
2652134260,4sleep10-21swapper/123:25:421
2647042260,1sleep00-21swapper/023:22:120
2636719260,3sleep20-21swapper/223:14:412
2616101260,2sleep10-21swapper/122:55:121
2615823260,1sleep00-21swapper/022:54:470
2611858261,2sleep20-21swapper/222:50:172
2601428261,2sleep10-21swapper/122:40:171
2589850260,2sleep00-21swapper/022:30:100
2580727261,2sleep20-21swapper/222:20:182
2579414261,2sleep10-21swapper/122:20:131
2578896260,1sleep20-21swapper/222:17:092
2574708261,2sleep00-21swapper/022:15:160
2552841260,1sleep00-21swapper/021:53:350
2526029260,4sleep20-21swapper/221:26:352
2510578260,1sleep00-21swapper/021:14:470
2505021260,1sleep00-21swapper/021:06:130
2473487260,1sleep10-21swapper/120:36:011
2468838261,2sleep20-21swapper/220:35:132
2462954260,1sleep10-21swapper/120:25:341
2425588261,2sleep00-21swapper/019:50:260
2415553260,1sleep20-21swapper/219:41:462
2410135260,1sleep00-21swapper/019:38:240
2404783260,4sleep00-21swapper/019:32:140
2381876260,1sleep20-21swapper/219:12:122
2378447261,2sleep00-21swapper/019:10:180
2673176250,3sleep10-21swapper/123:46:581
2631300250,3sleep10-21swapper/123:07:191
2625312250,1sleep10-21swapper/123:00:211
2605596250,3sleep00-21swapper/022:45:090
2589252250,1sleep00-21swapper/022:25:450
2547967250,2sleep20-21swapper/221:50:132
2526249250,1sleep10-21swapper/121:29:231
2520758250,1sleep20-21swapper/221:20:422
2489247250,3sleep20-21swapper/220:50:522
2452614250,3sleep20-21swapper/220:17:332
2431770251,2sleep20-21swapper/220:00:102
2430488250,2sleep10-21swapper/119:55:201
2394198250,1sleep00-21swapper/019:20:480
2652736240,1sleep10-21swapper/123:30:111
2579406240,1sleep00-21swapper/022:20:120
2463863240,1sleep20-21swapper/220:30:162
2463863240,1sleep20-21swapper/220:30:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional