You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 19:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Wed Dec 03, 2025 12:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2486575212855,68sleep30-21swapper/307:09:403
272189621160,3sleep1321rcuc/110:20:171
248627228861,17sleep10-21swapper/107:05:501
248652828451,9sleep20-21swapper/207:09:062
248629627855,6sleep00-21swapper/007:06:090
2486723992911,13cyclictest2878049-21chrt12:05:263
248672399261,21cyclictest2687723-21chrt09:57:583
248672399261,19cyclictest2546653-21taskset08:01:303
2486723992521,2cyclictest0-21swapper/312:10:593
29165332241,10sleep02615851-21diskmemload12:30:490
2486723992420,2cyclictest0-21swapper/311:05:003
28866702230,4sleep22615851-21diskmemload12:11:552
248672399232,13cyclictest2725726-21taskset10:25:033
248672399231,9cyclictest2901616-21taskset12:21:213
2486723992319,2cyclictest0-21swapper/307:39:343
2486723992317,3cyclictest0-21swapper/311:20:193
248672399230,17cyclictest2746049-21taskset10:36:223
248672399228,12cyclictest0-21swapper/311:40:193
248672399228,12cyclictest0-21swapper/311:40:193
248672399222,14cyclictest2724921-21taskset10:23:513
2486723992218,3cyclictest0-21swapper/310:50:003
2486723992218,2cyclictest0-21swapper/311:35:263
2486723992217,2cyclictest0-21swapper/312:20:013
2486723992213,4cyclictest26230852sleep309:14:453
2486723992212,2cyclictest0-21swapper/309:35:003
248672399220,14cyclictest2769487-21taskset10:53:223
248672399220,14cyclictest2769487-21taskset10:53:213
27619902210,2sleep20-21swapper/210:48:082
252037922114,4sleep22520400-40gst-plugin-scan07:39:332
248672399212,13cyclictest2716810-21taskset10:16:303
248672399212,12cyclictest2916380-21taskset12:30:313
2486723992118,1cyclictest0-21swapper/310:00:133
2486723992118,1cyclictest0-21swapper/308:40:013
2486723992118,1cyclictest0-21swapper/308:30:003
2486723992117,2cyclictest0-21swapper/311:50:573
2486723992117,2cyclictest0-21swapper/310:10:163
2486723992117,2cyclictest0-21swapper/309:40:173
248672399210,18cyclictest66550irq/131-enp1s0-TxRx-311:33:013
26904452208,8sleep22692144-21kthreadcore10:00:172
248672399204,13cyclictest0-21swapper/307:45:173
2486723992018,1cyclictest0-21swapper/311:10:013
2486723992017,2cyclictest0-21swapper/312:38:393
2486723992017,1cyclictest0-21swapper/311:20:013
2486723992016,2cyclictest0-21swapper/311:10:123
2486723992016,2cyclictest0-21swapper/310:05:013
2486723992016,2cyclictest0-21swapper/310:05:013
2486723992015,2cyclictest0-21swapper/311:45:543
2486723992015,2cyclictest0-21swapper/309:50:163
248672399201,16cyclictest2644209-21taskset09:27:433
248672399201,16cyclictest0-21swapper/309:45:013
248672399201,14cyclictest2909045-21chrt12:25:383
248672399201,13cyclictest2755503-21taskset10:45:003
248672399192,12cyclictest2739240-21taskset10:32:493
2486723991917,1cyclictest0-21swapper/312:01:563
2486723991916,1cyclictest0-21swapper/311:25:203
2486723991916,1cyclictest0-21swapper/309:20:223
2486723991916,1cyclictest0-21swapper/308:45:003
2486723991915,2cyclictest0-21swapper/309:16:303
2486723991915,2cyclictest0-21swapper/308:10:163
2486723991914,2cyclictest0-21swapper/311:55:203
28650392181,2sleep20-21swapper/211:57:482
28440982181,3sleep00-21swapper/011:43:270
28440982181,3sleep00-21swapper/011:43:270
28062462181,3sleep10-21swapper/111:17:421
27920222187,2sleep1171rcu_preempt11:08:441
27470202180,3sleep20-21swapper/210:38:192
27097062181,3sleep00-21swapper/010:12:470
248672399182,11cyclictest2775559-21taskset10:55:303
248672399181,13cyclictest2573618-21chrt08:30:113
27682622171,2sleep00-21swapper/010:50:390
27682622171,2sleep00-21swapper/010:50:390
27401692171,3sleep10-21swapper/110:34:291
26781562171,4sleep12678252-21ntpq09:50:181
26298662171,3sleep00-21swapper/009:18:310
26223072171,4sleep02615851-21diskmemload09:13:060
248672399179,5cyclictest2509881-21turbostat07:29:593
2486723991714,1cyclictest0-21swapper/308:15:163
2486723991714,1cyclictest0-21swapper/307:10:213
2486723991712,3cyclictest0-21swapper/309:05:113
2486723991712,2cyclictest0-21swapper/309:35:163
29241322161,2sleep10-21swapper/112:35:511
28715532160,1sleep10-21swapper/112:01:241
27753412160,2sleep20-21swapper/210:55:252
27684342161,4sleep22768444-21ssh10:51:012
27684342161,4sleep22768444-21ssh10:51:012
27472462161,2sleep02615851-21diskmemload10:38:490
27451692161,3sleep12615851-21diskmemload10:35:251
26652182161,2sleep00-21swapper/009:41:080
26235552161,10sleep12623584-21dpkg09:15:111
26227762161,4sleep22615851-21diskmemload09:14:002
248672399161,9cyclictest2589690-21taskset08:45:023
248672399161,4cyclictest0-21swapper/308:50:163
2486723991614,1cyclictest0-21swapper/307:15:023
2486723991613,1cyclictest0-21swapper/307:55:183
2486723991612,2cyclictest0-21swapper/308:20:233
248672399161,11cyclictest2531376-21chrt07:50:113
248672399160,14cyclictest0-21swapper/307:20:133
29041362152,7sleep12904160-21apt-get12:25:131
28963122150,3sleep12896331-21apt-get12:20:121
28937122150,3sleep20-21swapper/212:15:442
28726012151,2sleep20-21swapper/212:03:292
28286342151,4sleep02615851-21diskmemload11:32:370
28131032150,2sleep20-21swapper/211:21:162
27979262151,2sleep10-21swapper/111:10:411
27611012150,9sleep02615851-21diskmemload10:46:260
26677192154,6sleep02669366-21kthreadcore09:45:170
26428862151,9sleep22643109-21users09:25:272
26374792151,3sleep22615851-21diskmemload09:24:062
26358412151,9sleep12615851-21diskmemload09:20:361
25884192151,9sleep02588452-21ntpq08:40:190
248672399151,9cyclictest2610449-21taskset09:03:463
248672399151,2cyclictest0-21swapper/307:30:203
2486723991512,2cyclictest0-21swapper/308:05:153
2486723991512,1cyclictest0-21swapper/308:55:253
248672399151,10cyclictest2521094-21chrt07:40:133
28451642141,8sleep12845188-21apt-get11:45:121
28079012141,9sleep12808155-21cstates11:20:151
28077872141,8sleep02615851-21diskmemload11:20:060
27386262141,7sleep02615851-21diskmemload10:31:430
26989232141,8sleep12700545-21sed10:05:181
26989232141,8sleep12700545-21sed10:05:181
26427702141,9sleep12642885-21/usr/sbin/munin09:25:261
25998572140,1sleep00-21swapper/008:54:020
25670782140,6sleep00-21swapper/008:20:230
29030542130,1sleep00-21swapper/012:24:120
28963392131,5sleep22896599-21df12:20:132
28779832131,6sleep22878047-21ntpq12:05:262
28269102131,5sleep22826976-21ntp_states11:30:222
28071942131,5sleep20-21swapper/211:19:322
27999362130,2sleep20-21swapper/211:14:432
27833252131,8sleep22783334-21sh11:01:082
27762892130,8sleep02776290-21ssh10:57:010
27618792131,8sleep12761885-21ssh10:47:561
27156122131,8sleep12715663-21ntpq10:15:211
27015832131,6sleep00-21swapper/010:06:050
27015832131,6sleep00-21swapper/010:06:050
26747382131,7sleep22675761-21cut09:50:162
26526442131,7sleep12652647-21sh09:35:001
26383502131,6sleep02638610-21df09:25:120
25583852131,7sleep22560058-21kthreadcore08:15:162
29041652121,6sleep22904312-21tr12:25:142
28956052120,1sleep00-21swapper/012:19:550
28806852121,2sleep00-21swapper/012:10:010
28219222120,1sleep10-21swapper/111:29:051
27906452121,4sleep02615851-21diskmemload11:05:590
27788072121,7sleep02780465-21kthreadcore11:00:170
27764202121,5sleep12615851-21diskmemload10:57:221
27330432121,7sleep22733130-21cat10:30:012
27189922121,5sleep22615851-21diskmemload10:20:152
26934072121,7sleep02693536-21sendmail_mailqu10:00:240
26532022121,5sleep12653331-21diskstats09:35:161
29192032111,4sleep02615851-21diskmemload12:35:140
29086862111,6sleep02908932-21uptime12:25:280
28944002111,2sleep12894419-21apt-get12:17:151
28873122111,4sleep02615851-21diskmemload12:13:010
28851572111,5sleep12885211-21ntpq12:10:221
28714932111,4sleep02615851-21diskmemload12:01:150
28649782111,4sleep12615851-21diskmemload11:57:391
28645032111,4sleep02615851-21diskmemload11:56:220
28579572111,4sleep12615851-21diskmemload11:52:421
28574312111,4sleep22615851-21diskmemload11:51:352
28517372111,4sleep02615851-21diskmemload11:49:280
28498042110,7sleep22849974-21/usr/sbin/munin11:45:292
28429772111,4sleep12615851-21diskmemload11:41:071
28429772111,4sleep12615851-21diskmemload11:41:061
28361902111,4sleep02615851-21diskmemload11:37:380
28211512111,3sleep22615851-21diskmemload11:27:292
28193512113,4sleep02819404-21ssh11:25:190
27548392111,4sleep00-21swapper/010:43:540
27327652111,4sleep02615851-21diskmemload10:29:310
27317052111,3sleep10-21swapper/110:27:311
27180532111,3sleep20-21swapper/210:19:552
27095082111,3sleep10-21swapper/110:12:201
27016912111,4sleep22615851-21diskmemload10:06:242
27016912111,4sleep22615851-21diskmemload10:06:242
26948802111,4sleep12615851-21diskmemload10:02:381
26881182111,4sleep02615851-21diskmemload09:58:440
26813842111,3sleep20-21swapper/209:55:012
26797182111,3sleep02615851-21diskmemload09:51:580
26728402111,4sleep22615851-21diskmemload09:47:552
26582972111,4sleep02615851-21diskmemload09:36:270
26515132111,4sleep02615851-21diskmemload09:32:260
26363092111,4sleep02615851-21diskmemload09:21:320
26294052111,4sleep22615851-21diskmemload09:17:412
25884602111,4sleep12588509-21ntpq08:40:201
25353582112,4sleep22535404-21ntpq07:50:192
25261512111,4sleep12526369-21df_inode07:45:131
25150902110,1sleep10-21swapper/107:34:471
29241392101,3sleep20-21swapper/212:35:522
29114872101,3sleep22615851-21diskmemload12:30:072
28779272101,3sleep12615851-21diskmemload12:05:251
28521922101,4sleep02852435-21df_inode11:50:120
28426862101,3sleep20-21swapper/211:40:322
28426862101,3sleep20-21swapper/211:40:322
28362472101,3sleep20-21swapper/211:37:442
28358792101,2sleep10-21swapper/111:37:031
28291212101,3sleep12615851-21diskmemload11:33:301
28052862101,3sleep02615851-21diskmemload11:15:370
27982282101,3sleep02615851-21diskmemload11:11:200
27923932101,5sleep22792401-21ssh11:09:292
27832392101,3sleep12615851-21diskmemload11:00:591
27685752101,2sleep10-21swapper/110:51:161
27685752101,2sleep10-21swapper/110:51:161
27535032101,3sleep10-21swapper/110:41:221
27485262101,4sleep22748655-21perl10:40:152
27394932101,3sleep20-21swapper/210:33:172
27185622101,4sleep02718774-21diskstats10:20:130
27114612101,5sleep02711541-21cpuspeed_turbos10:15:110
27099492101,3sleep20-21swapper/210:13:172
26866842101,3sleep10-21swapper/109:55:521
26728322101,3sleep10-21swapper/109:47:541
26567772101,4sleep22656986-21memory09:35:212
26525822101,3sleep20-21swapper/209:34:532
25685692101,4sleep22568778-21hddtemp_smartct08:25:142
25669742101,4sleep22567022-21ntpq08:20:212
25527502101,4sleep12552959-21hddtemp_smartct08:10:141
25235582101,4sleep22524665-21memory07:40:172
25193532101,4sleep02519393-21ntpq07:35:230
25162152100,6sleep12517899-21kthreadcore07:35:191
25036192101,5sleep22503673-21ntpq07:20:222
24995302101,4sleep12499549-21apt-get07:20:131
24869212101,4sleep02487179-21df_inode07:10:140
2917592291,2sleep10-21swapper/112:32:511
2665555291,2sleep20-21swapper/209:42:302
2665186290,3sleep12615851-21diskmemload09:41:041
2621591290,3sleep12615851-21diskmemload09:11:391
2604073291,4sleep12604212-21/usr/sbin/munin08:55:191
2600199291,2sleep20-21swapper/208:55:122
2599937291,4sleep22600074-21wc08:55:002
2590941291,4sleep12592677-21cut08:45:171
2579779291,4sleep22581478-21cut08:35:162
2561677291,3sleep12561717-21ntpq08:15:191
2542784291,5sleep00-21swapper/008:00:190
2540161291,4sleep02540589-21cut07:55:180
2530895290,1sleep00-21swapper/007:47:290
2505714291,4sleep12507419-21tr07:25:181
2498914290,1sleep10-21swapper/107:15:521
2490809291,4sleep22490930-21ntpq07:10:202
2615786280,1sleep20-21swapper/209:09:232
2605515281,2sleep10-21swapper/109:00:111
2593752281,3sleep22593805-21/usr/sbin/munin08:45:222
2573596281,2sleep20-21swapper/208:30:112
2570243281,3sleep12572075-21kthreadcore08:25:171
2552460281,2sleep20-21swapper/208:10:112
2537125281,2sleep12537194-21irqstats07:55:151
2535471281,4sleep02535554-21proc_pri07:50:210
2521830281,3sleep12522852-21kthreadcore07:40:161
2521314281,3sleep02521829-21cut07:40:140
2511005281,3sleep02512167-21kthreadcore07:30:180
2509673280,1sleep20-21swapper/207:27:432
2498267281,3sleep02498321-21modprobe07:15:230
2615637270,2sleep1171rcu_preempt09:07:261
2612159271,2sleep00-21swapper/009:05:160
2605537271,2sleep20-21swapper/209:00:132
2595232271,3sleep12595349-21fschecks_time08:50:141
2589703270,2sleep00-21swapper/008:45:120
2589386271,3sleep2431ktimers/208:44:222
2578892270,2sleep10-21swapper/108:35:111
2568301270,3sleep02568365-21ls08:25:120
2551951270,1sleep20-21swapper/208:06:512
2551924270,1sleep00-21swapper/008:06:290
2546761270,1sleep20-21swapper/208:02:542
2504847270,2sleep00-21swapper/007:25:130
2503721270,4sleep00-21swapper/007:20:240
2499209270,1sleep20-21swapper/207:19:322
2578349260,1sleep10-21swapper/108:31:261
2552033260,1sleep10-21swapper/108:07:541
2536644261,2sleep20-21swapper/207:55:122
2604544250,2sleep00-21swapper/008:55:260
2562692250,1sleep00-21swapper/008:19:550
2556487250,1sleep00-21swapper/008:10:200
2546724252,1sleep10-21swapper/108:02:251
2526370250,2sleep20-21swapper/207:45:142
2491041250,1sleep10-21swapper/107:10:231
2605815240,1sleep00-21swapper/009:00:150
2579215240,1sleep00-21swapper/008:35:140
2573827240,1sleep00-21swapper/008:30:130
2563275240,1sleep10-21swapper/108:20:161
2531763240,1sleep10-21swapper/107:50:141
2510418240,1sleep20-21swapper/207:30:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional