You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-03 - 03:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Mon Nov 03, 2025 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2111221730,6sleep11396999cyclictest23:10:511
321912890,2sleep10-21swapper/122:43:091
113752880,3sleep10-21swapper/100:25:441
113752880,3sleep10-21swapper/100:25:441
112882880,9sleep2301ktimersoftd/221:34:462
225322800,11sleep38050irq/56-dwc_otg_23:55:423
285072780,3sleep11396999cyclictest21:55:551
280552770,2sleep21397099cyclictest00:02:262
173952770,1sleep00-21swapper/021:41:200
175812700,2sleep00-21swapper/019:20:270
183782680,5sleep31397199cyclictest21:43:213
3832670,2sleep10-21swapper/100:10:261
1397099651,36cyclictest0-21swapper/222:00:412
16002640,20sleep00-21swapper/023:27:330
171922630,2sleep10-21swapper/123:05:521
1396899639,3cyclictest0-21swapper/022:25:440
1397099621,24cyclictest0-21swapper/223:40:322
1396899623,42cyclictest11692-21unixbench_singl20:30:560
1396899621,50cyclictest3998-21unixbench_singl22:06:000
13968996145,5cyclictest8050irq/56-dwc_otg_00:01:000
1396899599,37cyclictest1997-21unixbench_singl00:11:000
1396899592,44cyclictest17784-21unixbench_singl23:05:590
1396899591,45cyclictest5280-21fschecks_count22:50:440
13968995914,26cyclictest0-21swapper/020:45:480
271292580,3sleep10-21swapper/121:55:261
1396999587,25cyclictest0-21swapper/119:30:331
13968995839,4cyclictest7950irq/56-dwc_otg_23:20:400
1396899581,53cyclictest5189-21aten_rbpower_en20:15:400
1396899581,40cyclictest3556-21aten_rbpower_cu23:30:420
13968995736,6cyclictest7950irq/56-dwc_otg_23:05:270
13968995736,3cyclictest0-21swapper/022:11:510
13968995729,5cyclictest11610-21apt-get21:35:260
13968995632,5cyclictest7950irq/56-dwc_otg_21:25:520
1396899562,42cyclictest0-21swapper/019:25:510
1396899561,43cyclictest0-21swapper/021:21:080
1396899561,43cyclictest0-21swapper/019:30:420
1396899561,14cyclictest4889-21runrttasks22:50:270
1396899553,32cyclictest0-21swapper/022:15:410
1396899551,39cyclictest7705-21grep23:35:450
1396899551,39cyclictest7705-21grep23:35:450
1396899551,39cyclictest32370-21wc22:00:540
1396899551,38cyclictest27064-21grep19:45:460
1396899551,32cyclictest23884-21fschecks_time21:50:440
1396899544,45cyclictest101ktimersoftd/000:35:410
13968995434,6cyclictest8050irq/56-dwc_otg_22:20:420
13968995431,7cyclictest26050-21sed19:40:550
13968995429,7cyclictest29676-21netstat21:15:510
146922530,2sleep2301ktimersoftd/221:38:412
13968995333,5cyclictest7950irq/56-dwc_otg_21:10:500
13968995328,3cyclictest0-21swapper/023:55:580
1396899532,37cyclictest0-21swapper/020:50:450
1396899532,37cyclictest0-21swapper/020:50:450
1396899531,5cyclictest15512-21if_wlan023:45:490
13968995229,6cyclictest8050irq/56-dwc_otg_21:45:380
1396899521,37cyclictest22789-21missed_timers22:30:550
1396899517,7cyclictest9-21ksoftirqd/000:30:530
1396899514,35cyclictest0-21swapper/023:50:410
13968995135,4cyclictest0-21swapper/022:55:510
1396899512,35cyclictest21126-21tr23:10:520
1396899511,40cyclictest0-21swapper/000:27:470
1396899511,40cyclictest0-21swapper/000:27:470
13968995113,3cyclictest0-21swapper/023:15:420
1396899501,37cyclictest0-21swapper/000:25:000
1396899501,36cyclictest0-21swapper/020:40:490
1396899501,34cyclictest13107-21grep20:35:510
1396899501,32cyclictest0-21swapper/020:20:560
13968995010,4cyclictest0-21swapper/000:05:570
13968994936,4cyclictest7950irq/56-dwc_otg_22:40:470
1396899492,6cyclictest19852-21fschecks_count20:55:440
1396899492,43cyclictest11994-21ntp_offset23:40:490
1396899491,34cyclictest0-21swapper/020:00:450
1396899491,34cyclictest0-21swapper/020:00:450
1396899491,33cyclictest0-21swapper/021:01:530
1396899491,33cyclictest0-21swapper/019:10:440
1396899491,31cyclictest0-21swapper/021:55:500
1397099488,25cyclictest22520-21ssh23:55:422
13968994832,5cyclictest31894-21cron20:00:260
13968994831,4cyclictest8050irq/56-dwc_otg_19:50:400
1396899481,35cyclictest0-21swapper/021:40:270
1396899481,34cyclictest7496-21wc00:20:270
1396999477,25cyclictest0-21swapper/121:35:321
1396999477,19cyclictest0-21swapper/122:15:391
13968994723,3cyclictest0-21swapper/020:05:520
1396899471,32cyclictest0-21swapper/022:35:380
183092460,2sleep20-21swapper/223:50:102
1397199463,8cyclictest0-21swapper/320:10:403
13971994617,15cyclictest7850irq/56-dwc_otg22:05:453
1397099467,25cyclictest0-21swapper/219:30:402
13970994620,5cyclictest10993-21apt-key23:40:272
13970994620,5cyclictest10993-21apt-key23:40:272
1396999462,23cyclictest0-21swapper/122:30:451
13968994621,8cyclictest0-21swapper/020:10:470
1396899461,5cyclictest23506-21tune2fs21:05:470
1396899461,32cyclictest0-21swapper/020:25:490
133502463,36sleep00-21swapper/019:06:010
1397199457,6cyclictest8050irq/56-dwc_otg_19:20:393
1396999457,26cyclictest0-21swapper/123:50:411
1396999457,24cyclictest28110-21sh23:20:431
1396999457,24cyclictest0-21swapper/120:45:431
1396999456,23cyclictest0-21swapper/119:55:441
1396899457,27cyclictest24048-21timerandwakeup19:35:520
1396899452,31cyclictest0-21swapper/019:20:390
1397199442,11cyclictest0-21swapper/323:45:423
1397099447,25cyclictest0-21swapper/222:40:412
1397099447,24cyclictest0-21swapper/219:55:442
1397099446,24cyclictest0-21swapper/219:10:402
1397099444,11cyclictest14774-21ssh00:30:402
1396999447,24cyclictest0-21swapper/123:55:421
1396999447,22cyclictest0-21swapper/119:10:401
13969994420,4cyclictest6182-21sh22:10:261
1397099437,24cyclictest0-21swapper/222:50:402
1397099436,21cyclictest0-21swapper/220:45:432
48892420,3sleep224773-21diskmemload22:50:272
1397099427,25cyclictest0-21swapper/220:05:402
1396999427,22cyclictest0-21swapper/122:35:411
1396999421,17cyclictest0-21swapper/100:35:421
320382410,2sleep20-21swapper/200:08:462
1397199417,30cyclictest8050irq/56-dwc_otg_22:00:383
1397099411,25cyclictest0-21swapper/220:15:402
1397099408,21cyclictest4094-21ls23:30:492
1397099403,23cyclictest21741-21ls21:00:452
13970994014,5cyclictest744-21fschecks_time21:20:482
1397099401,25cyclictest0-21swapper/219:40:412
1396999402,5cyclictest20985-21sed21:45:561
1396999402,5cyclictest11503-21grep20:30:551
71332390,1sleep10-21swapper/122:10:481
1397099397,19cyclictest0-21swapper/223:20:432
1397099393,24cyclictest3098-21sh22:05:492
1396999397,23cyclictest0-21swapper/120:05:401
1396999393,21cyclictest7850irq/56-dwc_otg22:25:551
13969993918,5cyclictest3076-21idleruntime-cro23:30:261
13969993916,4cyclictest14667-21cron23:45:261
72492380,11sleep1221ktimersoftd/100:20:081
38672380,2sleep10-21swapper/100:15:051
1397199382,8cyclictest7850irq/56-dwc_otg22:25:413
13970993823,5cyclictest1190150ssh22:16:292
1397099381,13cyclictest0-21swapper/219:30:262
1396999382,23cyclictest7850irq/56-dwc_otg19:15:431
13969993821,5cyclictest5379-21sh22:50:451
1396999381,9cyclictest25798-21wc19:40:531
1396999381,6cyclictest69150ssh21:20:481
1397199372,7cyclictest60-21kswapd023:20:403
1397199372,3cyclictest8050irq/56-dwc_otg_22:50:393
1397199372,15cyclictest8050irq/56-dwc_otg_21:25:563
13971993720,10cyclictest0-21swapper/300:15:433
1397099375,20cyclictest7950irq/56-dwc_otg_21:05:472
13970993713,5cyclictest20526-21/usr/sbin/munin23:10:442
1396999375,22cyclictest7850irq/56-dwc_otg23:35:261
1396999374,20cyclictest7850irq/56-dwc_otg00:00:561
1396999373,5cyclictest20089-21awk19:25:511
1396999372,3cyclictest7850irq/56-dwc_otg22:20:391
13969993715,4cyclictest32043-21ntp_states22:00:501
1397199361,24cyclictest12186-21ls20:35:273
1397099361,29cyclictest0-21swapper/221:48:232
13970993612,7cyclictest27811-21fschecks_time21:55:452
1396999364,23cyclictest9124-21ssh00:21:271
1396999362,24cyclictest7950irq/56-dwc_otg_23:35:441
1396999362,24cyclictest7950irq/56-dwc_otg_23:35:441
13969993617,5cyclictest15454-21/usr/sbin/munin23:45:481
13969993615,10cyclictest8667-21latency21:30:481
13969993612,5cyclictest0-21swapper/121:40:491
106752361,30sleep30-21swapper/323:40:093
106752361,30sleep30-21swapper/323:40:093
1397199356,11cyclictest7950irq/56-dwc_otg_22:36:403
1397199355,4cyclictest8050irq/56-dwc_otg_20:40:543
13971993520,7cyclictest2270-21grep20:05:513
1397099357,8cyclictest13967-21cyclictest21:40:412
1397099356,15cyclictest16060-21fschecks_count19:15:442
1397099353,6cyclictest7850irq/56-dwc_otg23:05:562
1397099353,17cyclictest0-21swapper/219:50:402
13970993523,5cyclictest21483-21ssh23:54:002
13970993519,9cyclictest301ktimersoftd/200:11:002
1397099351,25cyclictest0-21swapper/220:39:432
13970993512,15cyclictest4882-21latency_hist20:15:262
1396999352,4cyclictest7850irq/56-dwc_otg22:55:471
1396999352,22cyclictest0-21swapper/120:56:191
13969993517,5cyclictest1111-21runrttasks21:11:391
13969993516,8cyclictest12653-21/usr/sbin/munin23:00:521
1396999351,3cyclictest0-21swapper/100:30:421
1396999351,24cyclictest18213-21ls19:20:491
1397199348,6cyclictest0-21swapper/300:00:563
1397199346,5cyclictest16260-21taskset19:15:463
1397199343,4cyclictest8050irq/56-dwc_otg_00:20:393
1397199342,11cyclictest0-21swapper/300:39:343
1397199341,24cyclictest11431-21ls23:40:413
13971993411,5cyclictest7950irq/56-dwc_otg_23:50:573
13971993410,5cyclictest28929-21/usr/sbin/munin21:15:553
1397099347,12cyclictest7850irq/56-dwc_otg22:35:472
1397099342,5cyclictest19839-21grep20:55:442
13970993416,5cyclictest9477-21missed_timers22:55:512
1397099341,24cyclictest25242-21ssh21:10:412
1397099341,24cyclictest0-21swapper/221:54:052
1396999346,17cyclictest7850irq/56-dwc_otg21:15:491
1396999341,23cyclictest29411-21ls19:50:521
264912330,5sleep31397199cyclictest19:45:263
1397199339,5cyclictest7950irq/56-dwc_otg_20:35:553
1397199339,12cyclictest0-21swapper/321:52:463
1397199335,6cyclictest7850irq/56-dwc_otg19:40:263
1397199335,5cyclictest7950irq/56-dwc_otg_23:25:443
1397199333,13cyclictest8050irq/56-dwc_otg_21:20:593
13971993316,4cyclictest16518-21/usr/sbin/munin23:05:453
1397199331,4cyclictest7850irq/56-dwc_otg19:25:563
13970993324,6cyclictest7868-21latency_hist21:30:272
1397099332,19cyclictest0-21swapper/220:20:572
13970993315,6cyclictest5769-21sh00:16:282
1397099331,25cyclictest0-21swapper/219:35:392
1397099331,23cyclictest0-21swapper/222:12:512
1397099331,20cyclictest11108-21/usr/sbin/munin20:30:492
13970993311,12cyclictest21754-21latency_hist00:40:272
1396999333,19cyclictest0-21swapper/120:50:391
1396999333,19cyclictest0-21swapper/120:50:391
13969993311,5cyclictest5975-21sh21:26:521
126292333,22sleep30-21swapper/319:05:473
1397199327,6cyclictest7950irq/56-dwc_otg_21:55:543
1397199323,11cyclictest8050irq/56-dwc_otg_22:42:563
1397199322,3cyclictest8050irq/56-dwc_otg_21:35:463
1397199322,12cyclictest7950irq/56-dwc_otg_19:30:463
13971993212,4cyclictest30969-21ls19:55:523
13971993210,4cyclictest0-21swapper/300:32:363
1397099329,14cyclictest13686-21grep23:00:542
1397099322,19cyclictest10451-21latency_hist20:30:272
1397099322,14cyclictest0-21swapper/222:21:102
13970993218,7cyclictest7950irq/56-dwc_otg_00:25:502
13970993218,7cyclictest7950irq/56-dwc_otg_00:25:502
1397099321,14cyclictest17861-21wget20:50:422
1397099321,14cyclictest17861-21wget20:50:422
1397099321,11cyclictest29213-21/usr/sbin/munin21:15:462
1397099321,11cyclictest22426-21grep22:30:502
13970993210,15cyclictest301ktimersoftd/223:15:502
1396999325,8cyclictest0-21swapper/123:15:421
1396999325,16cyclictest0-21swapper/119:40:271
1396999323,14cyclictest7850irq/56-dwc_otg22:47:081
1396999322,6cyclictest8050irq/56-dwc_otg_19:45:401
13969993224,5cyclictest221ktimersoftd/120:25:551
13969993211,12cyclictest3645-21perl20:10:441
1397199317,4cyclictest0-21swapper/323:10:463
1397199317,22cyclictest0-21swapper/322:10:413
1397199316,4cyclictest8050irq/56-dwc_otg_20:50:393
1397199316,4cyclictest8050irq/56-dwc_otg_20:50:393
1397199314,4cyclictest7950irq/56-dwc_otg_21:31:553
1397199313,4cyclictest7850irq/56-dwc_otg21:45:413
1397199312,10cyclictest8050irq/56-dwc_otg_23:15:593
1397199311,13cyclictest7850irq/56-dwc_otg22:31:003
13970993121,7cyclictest21802-21latency_hist22:30:272
1397099311,16cyclictest0-21swapper/223:29:532
1396999318,14cyclictest7850irq/56-dwc_otg21:01:371
13969993117,4cyclictest23295-21aten_rbpower_vo21:05:441
1397199309,4cyclictest7990-21sed20:20:563
1397199308,4cyclictest8050irq/56-dwc_otg_20:55:483
1397199307,4cyclictest8050irq/56-dwc_otg_23:00:483
1397199307,4cyclictest0-21swapper/300:27:413
1397199307,4cyclictest0-21swapper/300:27:413
1397199306,4cyclictest8050irq/56-dwc_otg_20:45:523
1397199305,6cyclictest7850irq/56-dwc_otg22:20:453
13971993022,5cyclictest25415-21grep21:10:443
1397199302,13cyclictest8050irq/56-dwc_otg_22:45:453
1397199301,9cyclictest8050irq/56-dwc_otg_23:00:263
13971993010,6cyclictest0-21swapper/322:18:473
1397099301,23cyclictest0-21swapper/220:40:442
1397099301,20cyclictest0-21swapper/219:45:452
1397099301,16cyclictest0-21swapper/220:00:412
1397099301,16cyclictest0-21swapper/220:00:412
1397099301,12cyclictest0-21swapper/219:23:082
1396999304,6cyclictest7850irq/56-dwc_otg20:40:481
138272301,4sleep10-21swapper/120:39:311
1397199293,5cyclictest23857-21tr21:05:513
13971992923,4cyclictest15585-21latency_hist19:15:263
13971992920,7cyclictest381ktimersoftd/319:45:423
13971992910,15cyclictest1957-21unixbench_singl00:11:003
1397099293,11cyclictest0-21swapper/200:20:442
1396999295,11cyclictest7950irq/56-dwc_otg_20:15:401
1396999292,17cyclictest0-21swapper/120:02:001
1396999292,17cyclictest0-21swapper/120:02:001
1397199286,4cyclictest8050irq/56-dwc_otg_00:05:593
1397199286,14cyclictest0-21swapper/323:30:333
1396999281,6cyclictest7703-21ps20:20:541
121392285,14sleep112151-21/usr/sbin/munin19:05:371
1397199279,4cyclictest8050irq/56-dwc_otg_21:00:503
1397199261,7cyclictest7850irq/56-dwc_otg20:29:113
134022262,15sleep20-21swapper/219:06:352
1397199252,8cyclictest5386-21grep20:15:433
13971992519,4cyclictest1469-21latency_hist20:05:273
13971992519,4cyclictest1469-21latency_hist20:05:273
1397199251,6cyclictest7950irq/56-dwc_otg_19:50:443
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional