You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 15:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Mon Dec 01, 2025 12:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2174599212759,63sleep30-21swapper/307:08:423
217463728453,10sleep10-21swapper/107:09:111
217444527960,13sleep00-21swapper/007:06:410
217455527762,9sleep20-21swapper/207:08:092
2174834993127,2cyclictest0-21swapper/309:30:183
2174834992924,2cyclictest0-21swapper/312:25:243
25281522270,3sleep20-21swapper/211:37:232
2174834992723,2cyclictest0-21swapper/311:32:023
2174834992522,1cyclictest0-21swapper/312:05:153
2174834992522,1cyclictest0-21swapper/307:39:243
2174834992520,3cyclictest0-21swapper/309:55:003
2174834992519,3cyclictest0-21swapper/310:35:203
217483499251,19cyclictest2415234-21taskset10:23:563
2174834992421,1cyclictest0-21swapper/309:00:003
2174834992420,2cyclictest0-21swapper/310:15:163
2174834992414,5cyclictest24754882sleep311:03:393
217483499241,19cyclictest2324338-21taskset09:22:033
217483499232,15cyclictest2446595-21taskset10:45:133
2174834992319,2cyclictest0-21swapper/312:16:313
2174834992319,2cyclictest0-21swapper/311:50:193
2174834992319,2cyclictest0-21swapper/311:15:233
2174834992319,2cyclictest0-21swapper/309:10:003
2174834992318,3cyclictest0-21swapper/312:35:163
2174834992318,2cyclictest0-21swapper/310:00:163
2174834992318,2cyclictest0-21swapper/309:39:433
25140102221,16sleep12304306-21diskmemload11:29:251
2174834992220,1cyclictest0-21swapper/307:50:013
2174834992219,1cyclictest0-21swapper/310:00:003
2174834992218,2cyclictest0-21swapper/312:10:223
2174834992218,2cyclictest0-21swapper/311:48:583
2174834992218,2cyclictest0-21swapper/311:25:013
2174834992218,2cyclictest0-21swapper/310:55:013
2174834992218,2cyclictest0-21swapper/310:05:003
2174834992218,2cyclictest0-21swapper/309:40:183
2174834992217,2cyclictest0-21swapper/309:10:153
2174834992216,3cyclictest0-21swapper/312:20:133
217483499221,13cyclictest2476398-21taskset11:05:013
2174834992118,2cyclictest0-21swapper/310:45:013
2174834992117,2cyclictest0-21swapper/311:35:443
2174834992117,2cyclictest0-21swapper/310:55:193
2174834992117,2cyclictest0-21swapper/310:30:243
2174834992117,2cyclictest0-21swapper/308:20:013
2174834992117,2cyclictest0-21swapper/307:30:013
2174834992116,2cyclictest0-21swapper/309:50:003
2174834992112,6cyclictest2229899-21sleep08:00:013
217483499211,17cyclictest0-21swapper/311:27:513
217483499211,15cyclictest2613545-21chrt12:31:013
2174834992110,8cyclictest2399175-21taskset10:12:223
24842092201,13sleep12484451-21expr11:10:131
22050892200,4sleep1321rcuc/107:35:151
2174834992017,1cyclictest0-21swapper/309:25:223
2174834992017,1cyclictest0-21swapper/308:40:113
2174834992017,1cyclictest0-21swapper/308:03:483
2174834992016,2cyclictest0-21swapper/308:30:013
217483499200,17cyclictest0-21swapper/310:25:133
2174834991917,1cyclictest0-21swapper/308:35:163
2174834991915,2cyclictest0-21swapper/312:05:013
2174834991915,2cyclictest0-21swapper/311:10:203
2174834991914,2cyclictest0-21swapper/311:56:163
217483499191,14cyclictest2535227-21chrt11:41:023
217483499191,14cyclictest2261779-21chrt08:30:123
26155082181,3sleep00-21swapper/012:34:230
25765102181,12sleep22576787-21cut12:10:162
24313062181,4sleep02431328-21apt-get10:35:130
24230792181,5sleep22304306-21diskmemload10:29:212
24230642181,10sleep02304306-21diskmemload10:29:200
23754022181,4sleep12304306-21diskmemload09:55:301
23109512181,2sleep10-21swapper/109:14:121
217483499182,10cyclictest2223627-21taskset07:50:203
2174834991815,1cyclictest0-21swapper/307:30:163
2174834991814,2cyclictest0-21swapper/309:15:263
217483499181,11cyclictest2287232-21taskset08:50:203
25984212171,4sleep12304306-21diskmemload12:22:491
25604232171,4sleep02561556-21kthreadcore12:00:150
25441862171,4sleep22304306-21diskmemload11:49:032
25048492171,3sleep00-21swapper/011:21:390
24975482171,4sleep12304306-21diskmemload11:17:231
24293532175,2sleep2171rcu_preempt10:32:022
23841662171,4sleep12304306-21diskmemload10:01:581
23535552171,4sleep22304306-21diskmemload09:41:552
23405422171,4sleep02340573-21ssh09:35:120
23248712171,3sleep20-21swapper/209:23:022
23099882171,4sleep22304306-21diskmemload09:11:442
2174834991714,1cyclictest0-21swapper/309:00:183
2174834991713,2cyclictest0-21swapper/308:10:243
2174834991713,2cyclictest0-21swapper/308:07:173
217483499171,11cyclictest2256255-21turbostat08:25:003
25995572161,10sleep02304306-21diskmemload12:24:320
25841832162,7sleep02584314-21diskstats12:15:140
25742792161,2sleep20-21swapper/212:07:512
25136882161,8sleep00-21swapper/011:28:520
24766302161,8sleep22476754-21diskstats11:05:132
24379192161,3sleep10-21swapper/110:38:501
24233182161,4sleep12304306-21diskmemload10:29:521
23555802161,7sleep22304306-21diskmemload09:45:112
23401332161,2sleep10-21swapper/109:34:521
23108752161,10sleep02310887-21ssh09:14:030
22937232161,4sleep02293742-21latency_hist09:00:000
217483499165,8cyclictest2282950-21turbostat08:50:003
2174834991613,1cyclictest0-21swapper/307:20:143
2174834991613,1cyclictest0-21swapper/307:15:133
217483499161,11cyclictest2214125-21turbostat07:45:003
26216702151,2sleep20-21swapper/212:36:172
25809562151,9sleep02580967-21ssh12:10:410
25166992151,9sleep22518509-21grep11:30:192
24988412150,9sleep22304306-21diskmemload11:19:482
24445052151,9sleep22304306-21diskmemload10:41:522
23847222151,3sleep22384732-21ssh10:03:032
23750462151,9sleep22375191-21grep09:55:262
23705932154,6sleep02370613-21apt-get09:55:130
217483499151,10cyclictest2179062-21taskset07:10:203
26228732140,9sleep12304306-21diskmemload12:38:171
26136942141,6sleep22304306-21diskmemload12:31:172
25884972141,7sleep12588701-21/usr/sbin/munin12:15:271
25579512140,10sleep02557960-21ssh11:56:390
25354002141,7sleep22304306-21diskmemload11:41:222
24841802142,5sleep02484203-21apt-get11:10:120
24723912141,2sleep10-21swapper/111:00:191
23838282141,9sleep02383838-21ssh10:01:210
23693912141,8sleep12369400-21sh09:52:561
23619362141,9sleep12361948-21rm09:48:161
23176682141,7sleep22304306-21diskmemload09:18:172
23175622141,9sleep12317568-21ssh09:17:411
21794412141,8sleep12179685-21users07:10:271
25912252131,6sleep22304306-21diskmemload12:19:132
25660842131,5sleep20-21swapper/212:02:432
24539302136,4sleep0603-21in:imuxsock10:50:010
24070732131,5sleep00-21swapper/010:17:570
24058512131,6sleep22304306-21diskmemload10:15:422
23496052131,7sleep12351279-21cut09:40:171
23254262131,6sleep00-21swapper/009:24:050
26123262121,4sleep12304306-21diskmemload12:30:191
26076632121,5sleep22304306-21diskmemload12:29:282
26062442122,2sleep0171rcu_preempt12:27:170
25516542121,7sleep02551664-21ssh11:53:300
25376472121,5sleep12537666-21apt-get11:45:131
25208922121,4sleep12304306-21diskmemload11:33:021
24755182121,5sleep22304306-21diskmemload11:03:422
24754562122,3sleep014-21ksoftirqd/011:03:350
24672782121,4sleep02304306-21diskmemload10:57:020
24444012121,7sleep12444413-21ssh10:41:431
24135612121,7sleep02413570-21ssh10:20:570
23464602121,4sleep12304306-21diskmemload09:37:381
23247202121,4sleep12304306-21diskmemload09:22:451
22893192121,6sleep12289352-21kthreadcore08:55:161
22086982120,6sleep02208648-31find07:39:360
21864522121,2sleep20-21swapper/207:15:222
25985672111,4sleep22304306-21diskmemload12:23:032
25754322111,3sleep10-21swapper/112:09:381
25599102111,5sleep12560126-21/usr/sbin/munin12:00:121
25532372111,6sleep12554383-21cut11:55:151
25509392110,1sleep20-21swapper/211:52:042
25375512111,4sleep02304306-21diskmemload11:45:040
25337802111,5sleep12534146-21grep11:40:201
25263472112,4sleep12526400-21ntpq11:35:191
25053102111,3sleep12304306-21diskmemload11:22:281
24818982111,3sleep10-21swapper/111:06:321
24668392111,3sleep22304306-21diskmemload10:55:512
24659532111,5sleep12466011-21ntpq10:55:221
24609452111,4sleep22304306-21diskmemload10:53:502
24604702111,3sleep12304306-21diskmemload10:52:571
24542722111,3sleep00-21swapper/010:50:120
24537432111,4sleep22304306-21diskmemload10:49:412
24517792110,2sleep12331564-21kworker/u16:0+xprtiod10:46:121
24369092111,5sleep22304306-21diskmemload10:36:512
24294582111,3sleep10-21swapper/110:32:131
24157242113,2sleep20-21swapper/210:24:482
24133462111,3sleep12304306-21diskmemload10:20:341
23985292111,3sleep22304306-21diskmemload10:11:012
23915012111,3sleep20-21swapper/210:06:522
23908432111,3sleep10-21swapper/110:05:321
23687512111,3sleep22304306-21diskmemload09:51:452
23635232111,2sleep00-21swapper/009:50:140
23455622111,4sleep22304306-21diskmemload09:35:492
23395712111,3sleep22304306-21diskmemload09:33:492
23386282111,3sleep00-21swapper/009:31:340
23318712111,4sleep22304306-21diskmemload09:27:432
22882092111,5sleep12288230-21munin-run08:55:001
22617542111,5sleep22261773-21apt-get08:30:122
22551282111,5sleep02255217-21ntpq08:20:200
22354032110,7sleep02235558-21cstates08:05:170
22155642111,6sleep12217295-21cut07:45:181
26235702101,3sleep00-21swapper/012:39:200
26072542101,3sleep12304306-21diskmemload12:28:501
25809682100,6sleep10-21swapper/112:10:421
25732912100,3sleep00-21swapper/012:06:120
25567982101,3sleep22304306-21diskmemload11:55:202
25508972100,6sleep12304306-21diskmemload11:51:581
25354562101,5sleep02535466-21ssh11:41:270
25143752101,2sleep00-21swapper/011:30:000
25070282101,3sleep22304306-21diskmemload11:25:132
25062212101,3sleep22304306-21diskmemload11:24:102
24982212101,3sleep00-21swapper/011:18:380
24890192101,3sleep20-21swapper/211:10:312
24818472100,3sleep00-21swapper/011:06:270
24446862101,2sleep00-21swapper/010:42:100
24290542100,3sleep02304306-21diskmemload10:31:270
24059702101,4sleep12304306-21diskmemload10:15:531
23933562101,3sleep10-21swapper/110:10:041
23901122101,5sleep02390175-21ntpq10:05:230
23556072100,6sleep02355623-21cpu09:45:140
23538312101,3sleep00-21swapper/009:42:250
23315072101,3sleep00-21swapper/009:26:380
23311232101,3sleep10-21swapper/109:25:461
23173712100,3sleep00-21swapper/009:17:200
23033252101,4sleep22303374-21ntpq09:05:232
23032792101,5sleep12303324-21modprobe09:05:231
22818992101,4sleep12281947-21ntpq08:45:231
22449602103,3sleep22244969-21switchtime08:10:242
22406442101,3sleep10-21swapper/108:10:111
22235782101,4sleep22223625-21ntpq07:50:202
22234372101,4sleep12223577-21ntp_192.168.11407:50:191
22223902101,4sleep02223436-21memory07:50:180
22086602100,5sleep22208648-31find07:39:272
21879092101,2sleep10-21swapper/107:20:131
2527388290,2sleep00-21swapper/011:35:580
2398582291,2sleep00-21swapper/010:11:070
2261224291,4sleep02261225-21sshd08:26:370
2255273291,3sleep22255329-21ntpq08:20:222
2246206290,1sleep00-21swapper/008:15:150
2244556291,3sleep02244645-21ntpq08:10:180
2230146290,4sleep22230165-21apt-get08:00:102
2229863291,3sleep02229890-21munin-run08:00:000
2229340291,4sleep12229505-21vmstat07:55:281
2225366291,4sleep22225542-21cut07:55:172
2214916291,4sleep02215563-21cut07:45:170
2196920291,4sleep02196967-21ntpq07:25:210
2188533291,4sleep02190237-21cut07:20:160
2287186281,2sleep20-21swapper/208:50:202
2268946281,3sleep12270774-21grep08:35:201
2255218281,2sleep10-21swapper/108:20:211
2239462280,1sleep10-21swapper/108:05:261
2202243281,3sleep22202289-21ntpq07:30:202
2197919280,1sleep10-21swapper/107:29:591
2187158281,3sleep1802-21snmpd07:17:361
2186342281,3sleep02186389-21ntpq07:15:190
2298662270,1sleep10-21swapper/109:00:291
2294030271,2sleep00-21swapper/009:00:120
2288255271,3sleep22288729-21appstreamcli08:55:022
2282672270,1sleep20-21swapper/208:46:402
2281632270,2sleep02281893-21strings08:45:210
2277368270,1sleep20-21swapper/208:44:552
2276687271,3sleep12276884-21unixbench-2d08:40:251
2267627271,2sleep00-21swapper/008:35:190
2266466270,1sleep10-21swapper/108:30:551
2214642271,2sleep22214710-21fschecks_time07:45:152
2213822270,1sleep10-21swapper/107:41:331
2209611271,2sleep02211128-21ps07:40:160
2192425270,1sleep20-21swapper/207:22:382
2174989270,1sleep00-21swapper/007:10:020
2304159260,1sleep00-21swapper/009:08:210
2299017260,4sleep20-21swapper/209:04:592
2277029260,1sleep00-21swapper/008:40:360
2260425260,2sleep20-21swapper/208:25:202
2245916260,1sleep10-21swapper/108:15:121
2240267260,1sleep20-21swapper/208:08:362
2234957260,4sleep00-21swapper/008:03:050
2230446261,2sleep10-21swapper/108:00:131
2213763260,1sleep20-21swapper/207:40:462
2202101260,1sleep10-21swapper/107:30:191
2198468261,2sleep00-21swapper/007:30:140
2175001261,1sleep20-21swapper/207:10:122
2288201250,1sleep00-21swapper/008:54:550
2267344250,1sleep20-21swapper/208:35:182
2259987250,1sleep10-21swapper/108:25:181
2250543250,1sleep20-21swapper/208:15:292
2197719251,2sleep20-21swapper/207:27:282
2261728240,1sleep00-21swapper/008:30:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional