You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 14:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot3s.osadl.org (updated Fri Nov 21, 2025 12:43:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2855025500,4sleep10-21swapper/107:10:061
1514621690,6sleep22873599cyclictest10:52:322
1994221680,1sleep30-21swapper/311:45:393
380521020,6sleep02873399cyclictest10:35:350
28733998920,50cyclictest2174-21fschecks_time07:25:210
2873399891,23cyclictest10907-21wc10:00:300
2873399871,57cyclictest28068-21apt-get12:40:060
2873399861,63cyclictest12932-21unixbench_singl11:35:350
2873399841,61cyclictest5475-21ls09:05:050
23412840,5sleep02873399cyclictest09:49:300
28733998323,41cyclictest25456-21grep09:35:310
2873399831,58cyclictest26003-21ssh11:09:430
2873399822,58cyclictest13401-21/usr/sbin/munin10:50:260
28733998123,24cyclictest0-21swapper/008:05:160
28733998123,24cyclictest0-21swapper/008:05:160
323232800,1sleep30-21swapper/312:03:063
28733998040,19cyclictest7850irq/56-dwc_otg_11:30:230
28733998022,35cyclictest0-21swapper/008:35:330
2873399801,59cyclictest32415-21wget07:20:180
2873399801,59cyclictest32415-21wget07:20:180
28733997920,42cyclictest7950irq/56-dwc_otg_10:25:270
2873399786,14cyclictest0-21swapper/007:30:180
28733997818,4cyclictest12918-21grep12:20:260
266362770,2sleep0101ktimersoftd/011:55:260
266362770,2sleep0101ktimersoftd/011:55:260
2873399752,51cyclictest0-21swapper/007:40:340
254782750,1sleep30-21swapper/311:54:453
28733997425,33cyclictest7950irq/56-dwc_otg_07:50:480
87072720,1sleep00-21swapper/012:15:110
28733997220,3cyclictest0-21swapper/009:05:270
28733997216,44cyclictest0-21swapper/008:15:210
2873399721,23cyclictest18930-21uniq11:45:240
126082720,2sleep30-21swapper/312:20:173
2873399718,12cyclictest0-21swapper/010:17:300
249612710,5sleep12873499cyclictest11:07:161
2873399702,52cyclictest0-21swapper/007:13:040
2873399702,17cyclictest0-21swapper/010:06:210
28733997016,39cyclictest0-21swapper/012:06:200
2873399701,13cyclictest1712-21rm10:33:530
28733996916,3cyclictest0-21swapper/010:14:230
2873399691,18cyclictest0-21swapper/012:01:070
28733996910,48cyclictest0-21swapper/011:24:240
28733996910,44cyclictest26555-21aten_rbpower_en11:10:190
2873399682,48cyclictest0-21swapper/007:35:290
2873399681,56cyclictest0-21swapper/009:21:390
28733996814,9cyclictest0-21swapper/011:15:260
28733996811,48cyclictest0-21swapper/008:30:340
28733996714,9cyclictest22434-21ssh11:04:370
2873399671,20cyclictest0-21swapper/009:54:570
2873399666,4cyclictest0-21swapper/010:50:070
2873399662,14cyclictest0-21swapper/008:20:300
28733996516,7cyclictest7950irq/56-dwc_otg_09:10:340
2873399651,16cyclictest9051-21/usr/sbin/munin07:45:210
2873399641,52cyclictest0-21swapper/008:51:300
2873399641,52cyclictest0-21swapper/008:51:300
2873399641,21cyclictest26491-21ssh10:22:400
2873399641,21cyclictest26491-21ssh10:22:400
2873399641,19cyclictest0-21swapper/008:46:110
28733996410,3cyclictest0-21swapper/012:30:280
282972643,12sleep20-21swapper/207:07:582
28733996315,6cyclictest30568-21apt-get07:15:170
28733996313,41cyclictest0-21swapper/009:55:190
2873399631,16cyclictest23347-21sh09:32:430
2873399631,16cyclictest23347-21sh09:32:430
2873399625,39cyclictest0-21swapper/012:10:110
2873399618,12cyclictest0-21swapper/012:28:530
2873399612,18cyclictest0-21swapper/011:52:220
28733996114,7cyclictest7950irq/56-dwc_otg_09:16:050
2873399611,12cyclictest0-21swapper/010:44:180
28733996110,15cyclictest0-21swapper/011:44:030
258602610,1sleep20-21swapper/212:36:022
2873399601,4cyclictest6713-21ssh11:27:110
2873399601,4cyclictest6713-21ssh11:27:110
28733996011,8cyclictest0-21swapper/009:43:510
2873399601,15cyclictest0-21swapper/010:55:310
2873399592,17cyclictest0-21swapper/008:56:500
28733995910,12cyclictest0-21swapper/008:11:350
2873399581,30cyclictest23954-21fschecks_count08:25:230
28733995714,7cyclictest0-21swapper/009:27:330
2873399562,32cyclictest13050-21cut07:55:220
2873399551,17cyclictest0-21swapper/008:41:120
28733995410,8cyclictest0-21swapper/008:03:380
2873699532,19cyclictest18241-21ls08:10:073
2873699532,19cyclictest18241-21ls08:10:073
2873699521,17cyclictest7850irq/56-dwc_otg_12:05:223
2873499513,14cyclictest7950irq/56-dwc_otg_07:10:081
28734995018,3cyclictest0-21swapper/108:00:071
226742500,10sleep37950irq/56-dwc_otg_08:20:323
2873599496,15cyclictest0-21swapper/211:35:192
269112473,11sleep00-21swapper/007:05:170
2873699467,25cyclictest0-21swapper/308:10:183
2873699467,25cyclictest0-21swapper/307:35:203
2873699456,24cyclictest0-21swapper/310:05:123
2873599453,25cyclictest28732-21cyclictest07:10:082
28735994418,6cyclictest32264-21sh10:30:302
2873499442,6cyclictest28732-21cyclictest09:40:361
2873599437,24cyclictest0-21swapper/209:05:232
28735994316,11cyclictest2629-21cron09:50:062
291522410,3sleep229155-21aten_rbpower_cu08:40:202
2873699416,23cyclictest0-21swapper/309:50:193
2873699403,24cyclictest0-21swapper/307:50:203
2873699403,23cyclictest0-21swapper/307:45:173
28735994020,5cyclictest18990-21ntp_states08:10:272
28735994015,4cyclictest10646-21latency_hist07:50:062
2873499404,20cyclictest7950irq/56-dwc_otg_09:10:451
2873699395,7cyclictest7750irq/56-dwc_otg10:00:063
2873699394,21cyclictest0-21swapper/308:50:193
2873699394,21cyclictest0-21swapper/308:50:193
2873699391,24cyclictest0-21swapper/309:00:193
28735993921,8cyclictest30070-21/usr/sbin/munin11:15:202
2873499397,24cyclictest0-21swapper/109:05:231
2873499392,21cyclictest0-21swapper/109:45:171
28734993916,5cyclictest27719-21sh12:39:271
28734993913,7cyclictest13753-21ssh10:05:121
2873699385,21cyclictest16377-21ssh12:25:183
2873699381,18cyclictest13479-21latency_hist10:05:063
2873599389,6cyclictest0-21swapper/210:40:232
2873599382,4cyclictest7750irq/56-dwc_otg09:15:312
2873599381,29cyclictest0-21swapper/210:20:592
2873599381,29cyclictest0-21swapper/210:20:592
2873499381,5cyclictest7850irq/56-dwc_otg_10:35:331
28736993716,13cyclictest7298-21diskmemload12:30:293
2873699371,23cyclictest0-21swapper/308:40:203
28735993723,9cyclictest7950irq/56-dwc_otg_10:10:162
28735993716,4cyclictest7950irq/56-dwc_otg_11:54:372
2873499378,12cyclictest0-21swapper/111:40:221
28734993719,9cyclictest21904-21/usr/sbin/munin08:20:301
28734993718,4cyclictest1409-21ssh11:20:221
28734993717,5cyclictest1144-21lldpd07:29:401
28734993716,4cyclictest221ktimersoftd/110:00:251
28734993715,4cyclictest657750wget09:55:211
28734993712,8cyclictest0-21swapper/108:40:171
28734993710,4cyclictest0-21swapper/110:50:181
2873699363,17cyclictest0-21swapper/307:55:223
2873699363,15cyclictest7850irq/56-dwc_otg_07:20:263
2873699363,15cyclictest7850irq/56-dwc_otg_07:20:263
28736993617,10cyclictest10717-21ssh12:16:593
2873699361,3cyclictest7850irq/56-dwc_otg_09:25:283
2873699361,28cyclictest0-21swapper/311:40:053
2873599366,5cyclictest7950irq/56-dwc_otg_10:45:322
2873599366,4cyclictest0-21swapper/211:09:162
28735993618,9cyclictest23970-21fschecks_count08:25:232
2873599361,26cyclictest0-21swapper/212:18:342
28734993619,4cyclictest568-21fschecks_time08:50:231
28734993619,4cyclictest568-21fschecks_time08:50:231
2873499361,8cyclictest5264-21fschecks_time12:10:241
28734993618,9cyclictest1120-21runrttasks12:01:291
28734993618,8cyclictest3648-21/usr/sbin/munin07:30:301
2873499361,3cyclictest7850irq/56-dwc_otg_11:25:471
2873499361,3cyclictest7850irq/56-dwc_otg_11:25:471
28734993612,4cyclictest0-21swapper/110:30:301
252022360,3sleep10-21swapper/111:54:101
325792350,2sleep30-21swapper/310:30:353
2873699358,13cyclictest20581-21awk10:15:063
2873699357,21cyclictest0-21swapper/308:25:193
28736993513,6cyclictest15004-21fschecks_time08:00:243
2873699351,16cyclictest0-21swapper/309:21:473
2873599359,3cyclictest0-21swapper/210:35:262
2873599356,4cyclictest7950irq/56-dwc_otg_07:20:292
2873599356,4cyclictest7950irq/56-dwc_otg_07:20:292
2873599353,8cyclictest11091-21tr07:50:292
28735993516,12cyclictest433-21sh12:04:172
2873499358,5cyclictest0-21swapper/112:34:021
2873499358,4cyclictest0-21swapper/111:15:171
2873499354,4cyclictest7850irq/56-dwc_otg_10:15:331
2873499352,19cyclictest0-21swapper/109:36:561
28734993516,4cyclictest16061-21latency_hist12:25:061
28734993514,4cyclictest24677-21fschecks_time10:20:251
28734993514,4cyclictest24677-21fschecks_time10:20:251
2873499351,21cyclictest5488-21wget07:35:201
28734993510,11cyclictest0-21swapper/109:25:211
223912351,4sleep20-21swapper/211:04:322
31762340,9sleep2301ktimersoftd/212:07:122
2873699344,20cyclictest28278-21rm11:57:423
2873699344,20cyclictest28278-21rm11:57:423
2873699343,4cyclictest7850irq/56-dwc_otg_08:30:213
28736993414,6cyclictest5356-21latency_hist07:35:063
28736993414,5cyclictest23050-21sh09:32:023
28736993414,5cyclictest23050-21sh09:32:023
2873699341,13cyclictest13114-21wget10:50:223
2873599349,12cyclictest0-21swapper/209:59:092
28735993414,5cyclictest4012-21basename09:00:202
2873599341,11cyclictest0-21swapper/210:15:262
2873499343,20cyclictest0-21swapper/109:33:261
2873499343,20cyclictest0-21swapper/109:33:261
28734993416,5cyclictest20229-21sh10:14:191
2873499341,5cyclictest7850irq/56-dwc_otg_08:25:371
28734993415,5cyclictest6166-21latency_hist09:55:061
2873499341,4cyclictest0-21swapper/108:13:361
2873499341,3cyclictest7950irq/56-dwc_otg_08:30:241
2873499341,23cyclictest1819-21ssh12:05:301
260792340,2sleep126067-21ssh11:55:191
260792340,2sleep126067-21ssh11:55:191
323122330,1sleep30-21swapper/311:17:493
2873699338,5cyclictest7750irq/56-dwc_otg09:10:213
2873699331,16cyclictest6876-21unixbench_singl09:05:373
2873699331,15cyclictest6334-21cut10:40:263
2873599339,6cyclictest9450irq/48-DMA09:28:482
2873599337,5cyclictest0-21swapper/210:04:052
2873599334,4cyclictest0-21swapper/211:23:572
2873599331,7cyclictest27327-21/usr/sbin/munin11:10:292
2873599331,17cyclictest0-21swapper/209:41:002
28735993310,3cyclictest0-21swapper/209:33:232
28735993310,3cyclictest0-21swapper/209:33:232
2873499339,10cyclictest0-21swapper/110:45:181
2873499338,5cyclictest9550irq/80-mmc008:20:061
2873499331,4cyclictest7850irq/56-dwc_otg_09:25:061
2873499331,29cyclictest20860-21ls11:00:441
147082330,3sleep20-21swapper/209:20:212
303032320,2sleep10-21swapper/110:27:591
2873699328,12cyclictest11731-21ntpq09:15:273
2873699327,5cyclictest29131-21grep10:25:333
2873699324,16cyclictest1939-21wget07:25:183
28736993212,10cyclictest27086-21seq09:38:143
28736993211,5cyclictest28344-21wget09:40:233
2873699321,14cyclictest0-21swapper/309:48:363
28736993210,10cyclictest0-21swapper/310:19:033
2873599329,8cyclictest301ktimersoftd/212:30:312
2873599329,5cyclictest12741-21aten_rbpower_vo12:20:232
2873599323,17cyclictest10620-21rm09:14:532
28735993213,4cyclictest17003-21missed_timers12:25:282
28735993211,5cyclictest31026-21sshd07:15:252
28735993210,6cyclictest20591-21kernelversion08:15:272
2873499327,12cyclictest8771-21apt-get12:15:191
2873499322,13cyclictest7850irq/56-dwc_otg_07:55:061
2873499321,5cyclictest0-21swapper/111:33:411
2873499321,5cyclictest0-21swapper/109:20:041
28734993215,5cyclictest17179-21/usr/sbin/munin10:55:281
2873499321,4cyclictest7850irq/56-dwc_otg_10:40:281
2873499321,3cyclictest0-21swapper/111:15:061
28734993211,4cyclictest0-21swapper/111:35:281
2873699318,11cyclictest0-21swapper/312:35:233
2873699316,12cyclictest23225-21fschecks_count11:05:243
2873699314,18cyclictest381ktimersoftd/310:55:273
2873699312,19cyclictest0-21swapper/310:49:013
2873699311,5cyclictest0-21swapper/311:10:273
2873699311,24cyclictest0-21swapper/308:35:313
2873699311,23cyclictest29722-21ls07:10:333
2873699311,21cyclictest0-21swapper/312:12:593
2873699311,20cyclictest0-21swapper/311:00:213
2873699311,20cyclictest0-21swapper/310:37:283
28736993110,13cyclictest0-21swapper/311:40:353
2873599317,11cyclictest26134-21kworker/2:209:39:102
2873599314,13cyclictest7950irq/56-dwc_otg_07:30:052
2873599314,12cyclictest7950irq/56-dwc_otg_09:50:222
2873599313,4cyclictest7950irq/56-dwc_otg_11:45:212
2873599313,14cyclictest0-21swapper/211:25:342
2873599313,14cyclictest0-21swapper/211:25:342
2873599312,6cyclictest7850irq/56-dwc_otg_11:40:202
28735993122,5cyclictest17084-21sh10:10:052
2873599311,6cyclictest32506-21munin-run08:50:062
2873599311,5cyclictest16973-21irqstats10:55:262
28735993111,4cyclictest3746-21wget07:30:192
28735993110,11cyclictest0-21swapper/212:13:072
2873499317,17cyclictest0-21swapper/112:25:401
2873499315,8cyclictest0-21swapper/108:55:541
2873499313,4cyclictest7950irq/56-dwc_otg_07:20:291
2873499313,4cyclictest7950irq/56-dwc_otg_07:20:291
2873499311,4cyclictest7750irq/56-dwc_otg08:50:061
2873499311,4cyclictest7750irq/56-dwc_otg07:58:331
2873699309,4cyclictest0-21swapper/311:24:093
2873699301,21cyclictest0-21swapper/310:24:283
2873699301,21cyclictest0-21swapper/310:24:283
2873699301,12cyclictest32082-21head08:45:363
2873599305,4cyclictest7750irq/56-dwc_otg11:55:162
2873599305,4cyclictest7750irq/56-dwc_otg11:55:162
2873599301,7cyclictest25774-21fschecks_count08:30:242
2873599301,5cyclictest28440-21irqstats10:25:242
2873599301,23cyclictest3675-21latency_hist09:00:052
2873499308,5cyclictest9450irq/48-DMA09:28:481
2873499305,13cyclictest7950irq/56-dwc_otg_07:45:211
2873499303,14cyclictest7750irq/56-dwc_otg08:08:351
2873499303,14cyclictest7750irq/56-dwc_otg08:08:351
28734993022,5cyclictest7323-21aten_rbpower_en07:40:231
2873699291,18cyclictest0-21swapper/311:30:093
2873599299,5cyclictest15359-21awk08:00:292
2873599296,6cyclictest7950irq/56-dwc_otg_08:25:052
2873599291,5cyclictest8190-21/usr/sbin/munin11:30:272
2873599291,5cyclictest18331-21irqstats09:25:242
2873499295,5cyclictest7950irq/56-dwc_otg_09:00:221
2873499295,5cyclictest7850irq/56-dwc_otg_07:15:281
2873499295,11cyclictest7950irq/56-dwc_otg_11:45:211
2873699289,6cyclictest2151-21aten_rbpower_en08:55:203
2873699286,6cyclictest39-21ksoftirqd/311:25:183
2873699286,6cyclictest39-21ksoftirqd/311:25:183
28736992813,9cyclictest8014-21proc_pri07:40:333
2873599281,9cyclictest16510-21wget08:05:172
2873599281,9cyclictest16510-21wget08:05:172
28735992811,5cyclictest26671-21kworker/u8:207:37:342
2873599281,12cyclictest7850irq/56-dwc_otg_08:50:272
2873599281,12cyclictest7850irq/56-dwc_otg_08:50:272
283752283,15sleep30-21swapper/307:09:003
2873699271,17cyclictest0-21swapper/307:17:433
2873699271,12cyclictest0-21swapper/308:16:473
2873599272,10cyclictest0-21swapper/207:55:152
28735992613,4cyclictest7254-21/usr/sbin/munin07:40:342
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional