You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-15 - 20:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot3s.osadl.org (updated Sat Nov 15, 2025 12:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1526621800,5sleep11082399cyclictest07:20:331
1526621800,5sleep11082399cyclictest07:20:331
2165721220,1sleep00-21swapper/009:55:580
89942930,3sleep00-21swapper/008:35:030
177562930,2sleep10-21swapper/111:24:061
243222800,5sleep21082499cyclictest10:00:262
201612800,5sleep11082399cyclictest10:40:311
95442760,5sleep31082599cyclictest09:40:033
74462730,2sleep20-21swapper/208:30:182
1082399724,58cyclictest7750irq/56-dwc_otg07:40:211
1082399721,52cyclictest7950irq/56-dwc_otg_08:01:321
10823997118,40cyclictest32760-21latency09:25:251
145472700,1sleep10-21swapper/110:33:021
1082399702,52cyclictest0-21swapper/107:45:181
55552690,3sleep20-21swapper/211:52:092
10823996910,42cyclictest7950irq/56-dwc_otg_07:40:021
57432680,2sleep20-21swapper/209:33:462
160912680,1sleep30-21swapper/312:07:163
1082399681,48cyclictest27177-21grep10:50:281
1082299673,16cyclictest23366-21ssh09:10:330
1082399669,39cyclictest31817-21sed10:10:361
1082399662,45cyclictest0-21swapper/110:27:401
1082399661,56cyclictest7913-21/usr/sbin/munin11:10:331
1082299656,40cyclictest0-21swapper/010:40:170
1082299656,40cyclictest0-21swapper/007:55:150
1082299656,40cyclictest0-21swapper/007:55:150
1082399643,36cyclictest5588-21diskstats10:20:221
1082299641,48cyclictest13273-21/usr/sbin/munin09:45:260
1082299641,48cyclictest13273-21/usr/sbin/munin09:45:260
1082399637,42cyclictest7950irq/56-dwc_otg_09:45:041
1082399632,51cyclictest4339-21sh12:32:311
1082399631,43cyclictest29855-21timerwakeupswit09:20:321
1082299626,10cyclictest0-21swapper/009:35:140
10822996230,15cyclictest7850irq/56-dwc_otg_11:55:200
1082399618,34cyclictest0-21swapper/111:03:231
1082399616,43cyclictest7950irq/56-dwc_otg_10:38:431
1082399615,40cyclictest7950irq/56-dwc_otg_09:37:201
1082399611,5cyclictest26857-21ssh10:04:541
10823996111,28cyclictest0-21swapper/112:05:151
1082299615,39cyclictest0-21swapper/011:45:150
1082299612,46cyclictest0-21swapper/009:10:020
1082299611,43cyclictest6625-21interrupts12:35:270
1082399604,37cyclictest7950irq/56-dwc_otg_11:30:081
1082399604,37cyclictest7950irq/56-dwc_otg_11:30:081
1082399603,47cyclictest7950irq/56-dwc_otg_11:50:271
1082399603,35cyclictest25771-21chrt11:35:261
1082399602,40cyclictest7950irq/56-dwc_otg_11:07:211
10823996019,38cyclictest23593-21rm12:16:011
10822996031,16cyclictest0-21swapper/010:50:190
1082299602,47cyclictest0-21swapper/012:25:000
1082299601,46cyclictest17569-21kworker/u8:112:10:030
43972590,2sleep30-21swapper/310:18:433
43972590,2sleep30-21swapper/310:18:433
1082399596,40cyclictest7750irq/56-dwc_otg08:12:571
1082399591,4cyclictest22121-21/usr/sbin/munin09:10:351
10822995933,15cyclictest7950irq/56-dwc_otg_12:00:270
10822995928,11cyclictest7950irq/56-dwc_otg_11:00:030
1082299591,48cyclictest0-21swapper/009:40:280
1082399583,3cyclictest0-21swapper/110:47:331
1082399583,37cyclictest26560-21head07:50:321
1082399582,4cyclictest7950irq/56-dwc_otg_09:45:251
1082399582,4cyclictest7950irq/56-dwc_otg_09:45:251
1082399581,47cyclictest13188-21sh12:04:171
1082399581,40cyclictest0-21swapper/111:44:211
1082399581,3cyclictest0-21swapper/112:35:311
10822995827,15cyclictest7950irq/56-dwc_otg_08:10:170
10825995721,8cyclictest6105-21apt-get12:35:193
1082399575,44cyclictest7950irq/56-dwc_otg_11:55:461
1082399574,43cyclictest27445-21ssh09:17:251
1082399574,41cyclictest7950irq/56-dwc_otg_08:55:321
1082399574,41cyclictest7950irq/56-dwc_otg_08:55:321
1082399571,8cyclictest3307-21date11:50:031
1082399571,4cyclictest7750irq/56-dwc_otg10:20:031
1082399571,4cyclictest7750irq/56-dwc_otg10:20:031
1082399571,39cyclictest7750irq/56-dwc_otg09:32:261
10822995711,37cyclictest0-21swapper/011:25:030
1082399563,8cyclictest0-21swapper/111:18:421
1082399561,7cyclictest679-21rm12:28:241
1082399561,44cyclictest0-21swapper/108:25:141
1082399561,38cyclictest0-21swapper/112:11:411
1082299561,30cyclictest0-21swapper/010:45:160
1082399552,37cyclictest0-21swapper/107:35:001
1082399552,34cyclictest140-21systemd-journal10:08:191
1082399551,4cyclictest23008-21ssh09:59:131
1082399551,3cyclictest0-21swapper/108:15:411
1082299557,40cyclictest0-21swapper/012:10:260
1082299556,29cyclictest0-21swapper/009:35:040
10822995525,15cyclictest7950irq/56-dwc_otg_11:35:250
1082299552,46cyclictest0-21swapper/012:17:120
10822995521,4cyclictest0-21swapper/010:30:160
1082299551,40cyclictest3942-21fschecks_time11:50:240
1082299551,25cyclictest11698-21cut11:15:210
10822995511,36cyclictest19585-21unixbench_singl09:00:320
46052540,2sleep30-21swapper/312:33:003
1082599549,24cyclictest28750-21ssh09:20:163
1082599545,3cyclictest0-21swapper/309:30:173
10825995411,23cyclictest27220-21ntp_states10:50:293
1082399541,4cyclictest31126-21rm10:55:471
1082399541,3cyclictest0-21swapper/112:20:241
1082299544,40cyclictest14224-21idleruntime-cro07:20:030
10822995433,12cyclictest2884-21ssh12:30:320
1082299541,40cyclictest0-21swapper/009:15:490
1082299541,39cyclictest0-21swapper/011:10:250
1082399535,37cyclictest205422chrt11:27:161
1082299535,39cyclictest26270-21switchtime07:50:300
1082299532,42cyclictest0-21swapper/008:30:030
1082299532,41cyclictest0-21swapper/011:30:030
1082299531,7cyclictest30505-21df_inode12:25:190
10822995315,17cyclictest0-21swapper/008:00:140
1082299531,41cyclictest0-21swapper/007:40:230
10825995211,22cyclictest17647-21timerandwakeup09:50:303
1082399522,47cyclictest11093-21fschecks_count07:10:201
1082399522,41cyclictest7950irq/56-dwc_otg_08:06:471
1082399521,5cyclictest20359-21grep09:05:191
1082399521,4cyclictest0-21swapper/109:53:221
1082399521,40cyclictest7850irq/56-dwc_otg_08:41:571
1082399521,39cyclictest14723-21latency_hist08:50:031
1082299522,43cyclictest0-21swapper/010:08:500
1082299521,38cyclictest29753-21ntp_states11:40:300
10825995116,14cyclictest17929-21ssh12:10:173
1082399512,39cyclictest0-21swapper/107:56:041
1082399512,39cyclictest0-21swapper/107:56:041
1082399511,5cyclictest13749-21unixbench_multi07:15:311
10822995124,14cyclictest7950irq/56-dwc_otg_08:55:180
10822995124,14cyclictest7950irq/56-dwc_otg_08:55:180
1082299512,36cyclictest0-21swapper/008:45:190
1082299512,35cyclictest0-21swapper/010:35:190
1082299511,5cyclictest0-21swapper/008:50:160
1082299511,38cyclictest0-21swapper/010:15:330
1082299511,38cyclictest0-21swapper/010:15:330
10825995023,20cyclictest30738-21ssh10:10:153
10825995020,15cyclictest23124-21ls12:15:313
10825995012,27cyclictest12868-21nfs4_client10:30:263
10825995010,21cyclictest1599-21sh12:30:023
1082399502,44cyclictest19509-21grep09:00:321
1082399502,3cyclictest7850irq/56-dwc_otg_08:31:021
1082399501,4cyclictest16033-21apt-get07:25:091
1082299504,33cyclictest0-21swapper/009:50:220
1082299503,33cyclictest3944-21aten_rbpower_vo08:20:440
1082299501,5cyclictest32508-21ls08:10:030
10822995011,21cyclictest24055-21seq11:32:190
10822995011,21cyclictest24055-21seq11:32:190
1082599499,18cyclictest7850irq/56-dwc_otg_08:50:033
1082599498,25cyclictest7850irq/56-dwc_otg_11:15:533
10825994915,14cyclictest14718-21latency_hist08:50:033
10825994914,17cyclictest18434-21/usr/sbin/munin09:00:343
10825994913,15cyclictest21857-21aten_rbpower_en07:40:183
10825994910,21cyclictest3935-21perl11:50:243
1082399491,41cyclictest0-21swapper/108:22:551
1082299497,30cyclictest0-21swapper/007:35:210
1082299494,33cyclictest0-21swapper/007:25:140
1082299492,32cyclictest1638-21ntpq11:00:290
1082299491,37cyclictest0-21swapper/010:25:260
1082299491,35cyclictest0-21swapper/010:23:230
1082299491,35cyclictest0-21swapper/008:40:170
10825994813,11cyclictest370-21df_inode11:45:203
1082499484,9cyclictest7850irq/56-dwc_otg_07:15:142
1082399481,5cyclictest9773-21irqstats08:35:231
1082399481,4cyclictest14489-21apt-get08:50:091
1082299484,30cyclictest0-21swapper/009:25:200
1082299482,32cyclictest0-21swapper/009:20:260
1082299482,27cyclictest0-21swapper/007:30:250
1082299481,38cyclictest0-21swapper/010:00:210
1082299481,34cyclictest7619-21ssh11:10:020
1082599477,25cyclictest0-21swapper/311:05:173
1082599477,25cyclictest0-21swapper/308:55:183
1082599477,25cyclictest0-21swapper/308:55:183
10825994715,12cyclictest29296-21ssh10:07:253
10825994715,12cyclictest13540-21/usr/sbin/munin09:45:203
10825994715,12cyclictest13540-21/usr/sbin/munin09:45:203
10825994714,18cyclictest23588-21awk07:45:153
1082499472,7cyclictest0-21swapper/210:45:162
1082299471,39cyclictest0-21swapper/007:10:220
1082299471,37cyclictest0-21swapper/010:14:210
1082599468,24cyclictest27303-21ssh09:17:083
1082599468,16cyclictest26325-21cat07:50:303
1082599467,24cyclictest0-21swapper/308:35:083
1082299461,35cyclictest0-21swapper/007:45:240
102152463,35sleep00-21swapper/007:05:460
1082599458,24cyclictest0-21swapper/309:25:173
1082599457,26cyclictest28624-21seq07:58:283
1082599457,26cyclictest28624-21seq07:58:283
1082599457,23cyclictest0-21swapper/308:01:263
10825994514,16cyclictest31141-21ssh10:55:493
10825994512,13cyclictest16754-21ssh11:21:543
10825994511,16cyclictest30666-21ssh11:41:273
1082299452,33cyclictest0-21swapper/007:20:310
1082299452,33cyclictest0-21swapper/007:20:310
1082599447,22cyclictest0-21swapper/307:25:083
10825994416,11cyclictest22322-21/usr/sbin/munin11:30:253
10825994416,11cyclictest22322-21/usr/sbin/munin11:30:253
10824994412,11cyclictest7850irq/56-dwc_otg_09:55:152
1082299441,36cyclictest1120-21runrttasks08:35:430
1082299441,35cyclictest0-21swapper/008:15:250
1082599439,16cyclictest9047-21sh11:10:323
1082599438,21cyclictest22450-21ssh10:44:463
1082599437,18cyclictest25298-21rm10:01:133
1082599432,6cyclictest7950irq/56-dwc_otg_09:10:163
10825994314,11cyclictest19861-21latency_hist07:35:033
10825994312,16cyclictest7850irq/56-dwc_otg_11:58:343
1082499435,8cyclictest7950irq/56-dwc_otg_09:45:152
1082499435,8cyclictest7950irq/56-dwc_otg_09:45:152
1082599427,22cyclictest0-21swapper/308:30:173
10825994213,17cyclictest26861-21ssh11:35:443
10825994211,5cyclictest0-21swapper/311:00:243
10825994210,13cyclictest7850irq/56-dwc_otg_09:45:033
1082499428,10cyclictest7850irq/56-dwc_otg_09:35:142
10824994220,5cyclictest5724-21cut10:20:232
1082599417,22cyclictest0-21swapper/308:25:163
10825994111,18cyclictest18457-21ssh10:38:593
10824994122,12cyclictest11245-21sh12:00:282
10824994116,8cyclictest7850irq/56-dwc_otg_10:30:172
10825994023,8cyclictest7950irq/56-dwc_otg_10:47:163
1082599397,18cyclictest13348-21ssh12:04:373
1082599397,17cyclictest21699-21ssh11:30:013
1082599395,18cyclictest7750irq/56-dwc_otg09:56:083
10825993933,4cyclictest21733-21ls07:40:033
1082499397,24cyclictest0-21swapper/208:35:092
1082499397,23cyclictest0-21swapper/208:40:172
1082499396,25cyclictest0-21swapper/211:55:162
1082499392,26cyclictest7950irq/56-dwc_otg_11:40:152
1082599387,23cyclictest0-21swapper/308:05:173
1082599384,20cyclictest10673-21ssh10:27:143
1082499386,24cyclictest0-21swapper/208:25:162
1082599379,8cyclictest28041-21ssh12:21:353
1082599377,15cyclictest1120-21runrttasks09:09:083
10825993710,12cyclictest8101-21ssh10:24:253
1082499372,8cyclictest7850irq/56-dwc_otg_08:50:162
10824993717,4cyclictest2219-21sh11:00:372
10824993716,4cyclictest6429-21sh11:07:262
1082499371,4cyclictest7750irq/56-dwc_otg07:20:162
1082499371,4cyclictest7750irq/56-dwc_otg07:20:162
10824993714,15cyclictest301ktimersoftd/210:35:242
1082599366,16cyclictest3775-21seq08:20:093
10825993611,11cyclictest3289-21fake-hwclock08:17:023
1082499368,18cyclictest0-21swapper/209:15:192
1082499366,20cyclictest0-21swapper/210:55:182
1082499365,4cyclictest7950irq/56-dwc_otg_11:20:142
10824993619,10cyclictest28266-21ssh12:22:002
10824993618,9cyclictest1032-21sh10:14:122
10824993614,6cyclictest30631-21fschecks_time12:25:212
10824993613,14cyclictest15262chrt09:26:452
1082599355,19cyclictest381ktimersoftd/307:10:163
1082499356,24cyclictest0-21swapper/207:25:082
1082499352,4cyclictest7950irq/56-dwc_otg_09:15:032
1082499351,7cyclictest7950irq/56-dwc_otg_07:50:032
10824993515,9cyclictest2772-21uname08:15:292
10824993514,5cyclictest2294-21fschecks_time12:30:232
1082599345,14cyclictest123992sleep308:41:343
1082499349,5cyclictest29920-21unixbench-2d09:20:332
10824993428,4cyclictest10804-21ls09:40:332
10824993414,5cyclictest1749-21/usr/sbin/munin10:15:322
10824993414,5cyclictest1749-21/usr/sbin/munin10:15:322
1082599337,23cyclictest0-21swapper/307:15:163
1082499335,5cyclictest7750irq/56-dwc_otg11:30:192
1082499335,5cyclictest7750irq/56-dwc_otg11:30:192
10824993313,5cyclictest18668-21ntp_kernel_err12:10:282
1082499331,16cyclictest0-21swapper/209:51:172
1082599322,21cyclictest0-21swapper/307:20:523
1082599322,21cyclictest0-21swapper/307:20:523
1082499328,5cyclictest0-21swapper/208:55:182
1082499328,5cyclictest0-21swapper/208:55:182
10824993226,4cyclictest6368-21/usr/sbin/munin12:35:242
10824993214,11cyclictest14170-21sh11:19:142
10824993212,5cyclictest22420-21cat07:40:262
10824993211,5cyclictest23279-21unixbench_singl12:15:332
1082499318,6cyclictest7850irq/56-dwc_otg_11:13:112
1082499317,6cyclictest17104-21ssh12:09:132
1082499317,4cyclictest7750irq/56-dwc_otg08:21:062
1082499313,4cyclictest7950irq/56-dwc_otg_08:45:222
10824993116,5cyclictest874-21sh11:45:272
1082499311,3cyclictest7950irq/56-dwc_otg_07:35:302
1082499311,19cyclictest27794-21ls10:05:242
104482313,20sleep10-21swapper/107:08:421
198332301,4sleep20-21swapper/209:03:072
1082499307,14cyclictest0-21swapper/210:50:182
1082499306,14cyclictest0-21swapper/209:05:182
1082499302,6cyclictest7950irq/56-dwc_otg_10:40:282
1082499302,5cyclictest25540-21/usr/sbin/munin11:35:252
10824993022,6cyclictest31042-21fschecks_count08:05:222
1082499301,6cyclictest7950irq/56-dwc_otg_08:00:182
1082499301,23cyclictest0-21swapper/210:28:202
10825992916,11cyclictest496-21ls08:10:213
1082499293,5cyclictest7950irq/56-dwc_otg_11:25:452
92102285,9sleep20-21swapper/207:05:192
1082499282,5cyclictest7850irq/56-dwc_otg_07:55:172
1082499282,5cyclictest7850irq/56-dwc_otg_07:55:172
1082499281,6cyclictest7950irq/56-dwc_otg_07:54:502
1082499281,6cyclictest7850irq/56-dwc_otg_07:30:232
1082499281,5cyclictest7950irq/56-dwc_otg_07:10:282
105432286,14sleep30-21swapper/307:09:583
1082499254,4cyclictest7850irq/56-dwc_otg_08:10:182
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional