You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 05:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Wed Dec 03, 2025 00:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1268021120,5sleep21840499cyclictest22:11:172
247372820,2sleep10-21swapper/100:35:331
296622810,5sleep21840499cyclictest21:10:262
316142800,5sleep01840099cyclictest19:45:310
138392790,2sleep313836-21grep22:55:353
18400997716,19cyclictest2962750ssh21:10:240
1840099721,34cyclictest0-21swapper/021:30:240
1840499697,26cyclictest0-21swapper/223:20:182
1840099692,37cyclictest0-21swapper/000:25:240
1840099691,30cyclictest0-21swapper/021:05:240
18400996842,5cyclictest28135-21ssh21:50:260
1840099676,8cyclictest0-21swapper/019:10:140
1840399651,6cyclictest7750irq/56-dwc_otg20:45:281
181842650,2sleep10-21swapper/100:25:471
1840399641,5cyclictest7750irq/56-dwc_otg22:00:111
1840499637,22cyclictest0-21swapper/223:55:262
1840399635,5cyclictest7750irq/56-dwc_otg19:40:111
1840099632,36cyclictest9-21ksoftirqd/022:05:220
1840399621,5cyclictest833-21tr23:20:341
18400996236,15cyclictest0-21swapper/022:20:260
222722610,2sleep30-21swapper/319:20:313
18400996124,22cyclictest9-21ksoftirqd/022:30:230
18400996122,3cyclictest0-21swapper/000:00:250
1840399606,5cyclictest7750irq/56-dwc_otg23:40:341
18400996034,4cyclictest0-21swapper/023:55:260
1840399592,11cyclictest7750irq/56-dwc_otg21:35:111
1840099599,5cyclictest9-21ksoftirqd/019:50:390
1840099596,50cyclictest0-21swapper/020:20:270
1840399583,5cyclictest7950irq/56-dwc_otg_19:40:321
1840099582,43cyclictest0-21swapper/022:15:400
210302570,6sleep123-21ksoftirqd/119:15:381
18404995716,13cyclictest16540-21apt-get21:35:232
1840399578,45cyclictest0-21swapper/122:50:281
1840399578,45cyclictest0-21swapper/122:50:281
18400995733,13cyclictest101ktimersoftd/000:05:330
1840099572,38cyclictest0-21swapper/023:05:350
1840099572,20cyclictest16816-21ssh23:40:410
1840099571,41cyclictest12968-21latency_hist22:55:130
1840099571,41cyclictest12968-21latency_hist22:55:130
1840099571,29cyclictest0-21swapper/022:25:250
1840599567,18cyclictest0-21swapper/322:20:263
1840399569,4cyclictest0-21swapper/123:12:141
1840399561,20cyclictest0-21swapper/123:29:381
18400995533,4cyclictest7950irq/56-dwc_otg_21:40:260
1840099553,26cyclictest0-21swapper/023:00:410
18400995525,13cyclictest7850irq/56-dwc_otg_21:35:260
1840099551,18cyclictest0-21swapper/023:25:290
18404995413,14cyclictest0-21swapper/219:40:242
18404995411,21cyclictest7750irq/56-dwc_otg23:40:272
1840099547,3cyclictest0-21swapper/000:10:220
1840099543,48cyclictest31986-21ssh21:55:250
1840099542,37cyclictest9200-21fschecks_time21:25:310
18405995341,9cyclictest1120-21runrttasks21:55:253
18404995314,20cyclictest19388-21sed19:10:392
18400995326,7cyclictest3487-21aten_rbpower_po22:00:260
18400995326,11cyclictest7750irq/56-dwc_otg23:10:410
18400995325,5cyclictest21560-21unixbench_singl20:45:420
18405995223,5cyclictest27934-21latency_hist21:50:133
18400995227,6cyclictest28131-21/usr/sbin/munin19:35:360
1840099522,38cyclictest0-21swapper/023:20:390
18400995223,5cyclictest8851-21grep20:10:410
18400995217,18cyclictest13-21rcuc/020:39:590
1840099521,28cyclictest31521-21unixbench_multi22:35:430
18400995124,5cyclictest26056-21fschecks_count21:00:300
1840099512,29cyclictest16850-21latency_hist23:00:120
18400995122,6cyclictest4269-21grep20:00:270
18404995025,9cyclictest28152-21apt-get23:15:162
1840399506,13cyclictest18395-21cyclictest19:10:141
1840099506,41cyclictest9-21ksoftirqd/000:30:310
1840099506,27cyclictest13-21rcuc/021:15:230
1840099506,24cyclictest0-21swapper/019:30:260
18400995026,6cyclictest7750irq/56-dwc_otg00:20:300
18400995026,14cyclictest0-21swapper/022:45:340
1840099502,40cyclictest0-21swapper/023:50:400
18400995016,18cyclictest1986-21fschecks_count22:40:300
18404994918,19cyclictest23169-21sh21:43:382
18404994911,12cyclictest1684-21ssh21:15:332
1840099497,29cyclictest0-21swapper/020:00:120
18400994935,4cyclictest7950irq/56-dwc_otg_22:10:240
18400994931,5cyclictest7850irq/56-dwc_otg_00:35:330
1840099492,28cyclictest24323-21aten_rbpower_cu21:45:260
18400994916,16cyclictest32118-21apt23:20:110
1840099491,24cyclictest12955-21sed23:35:420
1840599488,26cyclictest0-21swapper/320:45:283
1840499489,14cyclictest16986-21aten_rbpower_vo00:25:292
18404994812,21cyclictest26411-21/usr/sbin/munin22:30:372
18404994811,22cyclictest31294-21diskstats19:45:282
1840099489,17cyclictest0-21swapper/021:20:300
1840099483,21cyclictest0-21swapper/023:30:390
18400994817,18cyclictest22200-21/usr/sbin/munin20:50:380
1840599477,26cyclictest0-21swapper/322:30:253
1840599474,8cyclictest7850irq/56-dwc_otg_20:30:243
18405994731,4cyclictest7850irq/56-dwc_otg_23:15:133
1840499477,25cyclictest0-21swapper/200:05:272
18404994716,19cyclictest21098-21/usr/sbin/munin20:45:372
18404994715,14cyclictest23439-21resolvconf20:51:082
18404994715,12cyclictest1794-21ssh21:57:372
1840399472,10cyclictest0-21swapper/119:45:251
1840399471,3cyclictest7950irq/56-dwc_otg_23:00:121
1840099471,27cyclictest9588-21wc00:15:280
1840099471,12cyclictest13-21rcuc/019:45:120
18400994711,23cyclictest0-21swapper/020:05:260
1840599467,25cyclictest0-21swapper/320:20:273
1840499468,21cyclictest9933-21/usr/sbin/munin20:15:322
1840499467,25cyclictest0-21swapper/219:30:262
1840499467,24cyclictest0-21swapper/219:50:182
1840499463,28cyclictest7850irq/56-dwc_otg_23:46:252
1840499463,28cyclictest7850irq/56-dwc_otg_23:46:252
18404994614,12cyclictest30957-21apt-get00:00:232
18404994610,23cyclictest22645-21aten_rbpower_cu22:25:252
18404994610,20cyclictest11905-21rm22:53:212
18404994610,20cyclictest11905-21rm22:53:212
18403994611,8cyclictest7950irq/56-dwc_otg_21:35:231
1840099469,18cyclictest0-21swapper/023:45:270
1840099469,18cyclictest0-21swapper/023:45:270
18400994627,5cyclictest7950irq/56-dwc_otg_20:45:110
181212463,35sleep00-21swapper/019:10:060
1840599457,24cyclictest0-21swapper/319:25:263
1840599457,24cyclictest0-21swapper/319:25:263
1840599457,23cyclictest0-21swapper/319:30:263
1840599456,24cyclictest0-21swapper/319:50:173
1840599455,5cyclictest0-21swapper/300:10:223
1840599455,25cyclictest29561-21ssh21:10:193
18405994513,19cyclictest7950irq/56-dwc_otg_20:35:273
1840499453,21cyclictest18355-21latency_hist20:40:122
18404994516,13cyclictest9988-21ssh23:32:222
18404994510,20cyclictest30092-21ssh21:52:042
1840099452,24cyclictest0-21swapper/020:15:310
1840499447,25cyclictest0-21swapper/220:20:272
1840499443,24cyclictest0-21swapper/222:00:232
1840099442,23cyclictest0-21swapper/019:30:120
1840099442,23cyclictest0-21swapper/019:30:120
1840099441,40cyclictest0-21swapper/020:55:330
1840099441,28cyclictest22370-21grep19:20:320
1840599437,23cyclictest0-21swapper/323:25:243
1840599433,22cyclictest0-21swapper/319:10:143
1840499437,24cyclictest0-21swapper/200:30:252
1840499436,16cyclictest7850irq/56-dwc_otg_20:15:122
1840499434,25cyclictest0-21swapper/219:25:262
1840499434,25cyclictest0-21swapper/219:25:262
1840499434,24cyclictest7750irq/56-dwc_otg20:31:582
18404994329,11cyclictest4841-21gpgconf21:20:132
18404994312,13cyclictest4894-21sh22:44:362
18400994323,3cyclictest0-21swapper/020:25:240
18400994323,3cyclictest0-21swapper/020:25:240
18400994311,6cyclictest0-21swapper/020:30:230
1840599421,3cyclictest0-21swapper/321:20:273
1840499425,15cyclictest16878-21rm22:17:012
18404994211,13cyclictest11385-21ssh00:17:072
18404994211,12cyclictest12169-21ssh21:29:482
1840399427,25cyclictest0-21swapper/122:00:251
322902410,6sleep31840599cyclictest00:00:453
1840599413,33cyclictest7950irq/56-dwc_otg_23:20:223
18405994116,5cyclictest7750irq/56-dwc_otg00:05:333
1840599411,11cyclictest7950irq/56-dwc_otg_22:50:123
1840499419,12cyclictest28257-21apt-config23:15:122
1840499417,13cyclictest20838-21ssh22:22:132
1840499416,14cyclictest19942-21ssh23:04:102
1840499416,12cyclictest7850irq/56-dwc_otg_22:05:372
18404994111,15cyclictest7750irq/56-dwc_otg19:40:112
18404994110,12cyclictest15201-21taskset00:22:202
1840399417,24cyclictest0-21swapper/121:45:251
1840399417,23cyclictest0-21swapper/121:05:261
18403994119,4cyclictest21-21rcuc/120:15:411
18400994118,13cyclictest0-21swapper/019:15:170
27312400,4sleep12336-21/usr/sbin/munin19:55:311
1840599407,24cyclictest0-21swapper/320:40:253
1840599402,5cyclictest39-21ksoftirqd/300:30:123
1840599402,24cyclictest0-21swapper/321:00:273
1840599401,24cyclictest0-21swapper/320:05:263
1840499409,13cyclictest29402-21cat21:10:132
1840499407,22cyclictest27059-21chrt00:39:042
1840499407,17cyclictest7950irq/56-dwc_otg_22:39:032
1840499406,13cyclictest8975-21cron22:50:122
1840499404,11cyclictest0-21swapper/223:38:002
1840499402,23cyclictest0-21swapper/220:59:022
1840499402,20cyclictest7950irq/56-dwc_otg_21:48:572
18404994019,4cyclictest2084450wget23:05:272
18404994015,13cyclictest20175-21latency_hist20:45:122
181972408,15sleep30-21swapper/319:10:133
1840599392,24cyclictest0-21swapper/322:02:263
18405993918,4cyclictest37-21rcuc/321:35:123
1840499398,11cyclictest7271-21ssh00:11:112
1840499397,15cyclictest5063-21/usr/sbin/munin23:25:392
1840499394,25cyclictest26409-21ssh23:54:262
18404993913,12cyclictest15799-21ssh22:58:292
1840399395,25cyclictest1033-21ssh21:15:251
1840399395,16cyclictest5720-21ssh00:10:221
18403993919,4cyclictest6247-21fschecks_time20:05:301
1840599381,4cyclictest7950irq/56-dwc_otg_23:50:283
1840599381,23cyclictest5389-21ls22:45:133
1840599381,17cyclictest7950irq/56-dwc_otg_21:15:323
1840499387,12cyclictest14442-21seq20:26:292
1840499387,12cyclictest14442-21seq20:26:292
1840399385,5cyclictest7750irq/56-dwc_otg00:31:571
1840399384,13cyclictest0-21swapper/123:00:251
1840399381,7cyclictest2646950ssh23:54:331
18403993813,4cyclictest21-21rcuc/123:35:391
18405993715,15cyclictest37-21rcuc/320:17:503
1840599371,22cyclictest2701-21idleruntime19:55:303
1840499378,11cyclictest23533-21taskset19:24:182
1840499372,22cyclictest0-21swapper/221:32:372
1840399371,25cyclictest22412-21idleruntime19:20:331
18403993711,14cyclictest15100-21wget20:30:271
1840599368,12cyclictest436-21avahi-daemon23:43:403
1840599364,15cyclictest24552-21fschecks_time00:35:313
1840599363,6cyclictest39-21ksoftirqd/300:30:343
1840599361,4cyclictest10285-21ssh22:50:393
1840599361,4cyclictest10285-21ssh22:50:393
1840599361,17cyclictest7750irq/56-dwc_otg21:05:303
1840399369,4cyclictest0-21swapper/123:55:341
1840399366,12cyclictest7750irq/56-dwc_otg00:05:121
18403993615,5cyclictest9967-21timerwakeupswit21:25:421
1840599357,24cyclictest0-21swapper/323:55:263
1840599355,4cyclictest7850irq/56-dwc_otg_22:25:293
1840599353,9cyclictest8868-21ssh21:25:263
18405993519,9cyclictest381ktimersoftd/323:15:183
18405993510,8cyclictest17326-1kworker/u9:023:45:393
18405993510,8cyclictest17326-1kworker/u9:023:45:393
1840499354,11cyclictest7750irq/56-dwc_otg19:58:442
1840499353,13cyclictest21448-21chrt19:15:592
1840399355,11cyclictest21-21rcuc/120:40:391
1840399351,9cyclictest12825-21idleruntime-cro00:20:121
1840399351,20cyclictest5732-21ls22:45:271
139952350,2sleep30-21swapper/323:37:383
1840599347,7cyclictest39-21ksoftirqd/323:04:223
1840599346,14cyclictest7950irq/56-dwc_otg_21:40:313
1840599344,4cyclictest39-21ksoftirqd/322:05:313
18405993417,8cyclictest37-21rcuc/319:45:393
18405993410,6cyclictest13247-21aten_rbpower_en20:25:283
18405993410,6cyclictest13247-21aten_rbpower_en20:25:283
1840499347,10cyclictest7950irq/56-dwc_otg_21:01:472
1840499346,12cyclictest7477-21runrttasks20:09:462
1840399347,7cyclictest21-21rcuc/122:35:431
1840399347,15cyclictest0-21swapper/122:15:271
1840399347,13cyclictest7950irq/56-dwc_otg_23:34:331
1840399345,6cyclictest16960-21cpuspeed20:35:281
1840399344,5cyclictest7750irq/56-dwc_otg21:20:421
18403993418,5cyclictest23997-21sh23:09:361
1840399341,23cyclictest22532-21cut20:50:311
175312344,15sleep117636-21/usr/sbin/munin19:05:421
1840599337,6cyclictest0-21swapper/323:30:293
1840599335,5cyclictest22103-21ssh23:05:423
18405993314,4cyclictest29199-21diskmemload00:24:473
18404993318,5cyclictest301ktimersoftd/220:01:562
1840399331,3cyclictest0-21swapper/100:24:161
1840399331,24cyclictest11294-21ls22:10:271
18403993310,11cyclictest0-21swapper/122:35:021
1840399330,18cyclictest24169-21fschecks_time19:25:301
1840399330,18cyclictest24169-21fschecks_time19:25:301
314992320,6sleep31840599cyclictest21:54:563
1840599327,11cyclictest0-21swapper/322:15:433
18405993210,16cyclictest37-21rcuc/319:18:523
1840399324,6cyclictest30654-21cat21:10:401
1840399323,4cyclictest7750irq/56-dwc_otg22:30:121
1840399323,2cyclictest0-21swapper/119:50:281
1840399321,22cyclictest19463-21ssh22:20:341
1840399321,21cyclictest7850irq/56-dwc_otg_23:45:371
1840399321,21cyclictest7850irq/56-dwc_otg_23:45:371
1840599316,10cyclictest7850irq/56-dwc_otg_22:10:283
1840599315,15cyclictest7950irq/56-dwc_otg_19:35:423
18405993113,4cyclictest37-21rcuc/319:40:143
1840399319,4cyclictest0-21swapper/121:50:251
1840399318,5cyclictest7750irq/56-dwc_otg21:40:421
1840399316,16cyclictest0-21swapper/122:40:261
1840399313,7cyclictest5367-21ssh00:10:071
18403993117,4cyclictest23-21ksoftirqd/120:20:311
1840599307,8cyclictest7850irq/56-dwc_otg_21:35:283
1840599302,12cyclictest7750irq/56-dwc_otg00:15:343
1840399306,12cyclictest7750irq/56-dwc_otg20:10:261
1840399305,5cyclictest7750irq/56-dwc_otg22:05:361
1840399305,5cyclictest28903-21fschecks_time23:15:291
1840399304,4cyclictest7750irq/56-dwc_otg21:00:361
1840399303,5cyclictest7750irq/56-dwc_otg20:25:421
1840399303,5cyclictest7750irq/56-dwc_otg20:25:421
1840599299,6cyclictest39-21ksoftirqd/320:50:413
1840599296,14cyclictest0-21swapper/320:55:253
1840599291,8cyclictest29199-21diskmemload22:35:433
1840399294,7cyclictest0-21swapper/120:55:371
1840399291,13cyclictest7750irq/56-dwc_otg19:32:401
1840599289,11cyclictest37-21rcuc/320:01:333
1840399281,14cyclictest0-21swapper/120:05:121
18405992617,5cyclictest37-21rcuc/320:10:323
178222243,14sleep20-21swapper/219:06:192
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional