You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-05 - 01:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Sun May 04, 2025 12:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2744122300,3sleep10-21swapper/107:10:171
673121480,5sleep32748299cyclictest11:26:513
27458991022,43cyclictest0-21swapper/009:30:310
180972910,1sleep00-21swapper/010:59:340
180972910,1sleep00-21swapper/010:59:340
2745899866,21cyclictest0-21swapper/007:30:320
2745899844,44cyclictest5518-21ssh12:05:300
2745899844,44cyclictest5518-21ssh12:05:300
2745899823,15cyclictest16852-21sshd10:10:390
2745899811,60cyclictest14333-21sh12:15:370
2745899801,57cyclictest0-21swapper/012:20:300
2745899801,56cyclictest23435-21grep10:20:350
2745899791,58cyclictest4255-21grep09:05:320
260462790,2sleep10-21swapper/112:30:181
27458997818,14cyclictest0-21swapper/009:45:320
2745899771,55cyclictest9705-21/usr/sbin/munin11:30:400
162872770,2sleep10-21swapper/110:10:321
27458997622,34cyclictest7950irq/56-dwc_otg_10:30:450
2745899761,55cyclictest19445-21cat08:15:480
2745899761,55cyclictest19445-21cat08:15:480
2745899752,12cyclictest0-21swapper/012:02:370
2745899751,10cyclictest0-21swapper/011:45:220
27458997434,5cyclictest8050irq/56-dwc_otg_12:35:300
2745899737,51cyclictest0-21swapper/011:02:130
2745899737,49cyclictest12642-21ssh09:20:040
27458997329,24cyclictest7950irq/56-dwc_otg_10:25:460
27458997318,36cyclictest16772-21ssh09:25:280
2745899731,49cyclictest0-21swapper/009:45:170
184052730,1sleep20-21swapper/211:00:162
184052730,1sleep20-21swapper/211:00:162
2747699724,15cyclictest7950irq/56-dwc_otg_12:25:292
2745899726,20cyclictest6109-21ssh09:10:230
27458997214,43cyclictest0-21swapper/011:25:450
2745899715,52cyclictest0-21swapper/010:15:290
27458997112,51cyclictest13050-21ssh12:14:420
109462700,4sleep2301ktimersoftd/209:16:092
2745899695,41cyclictest0-21swapper/010:53:190
2745899691,53cyclictest0-21swapper/010:36:020
2745899691,50cyclictest6827-21grep09:55:520
2745899691,50cyclictest6827-21grep09:55:520
2745899681,56cyclictest0-21swapper/007:19:510
27458996815,3cyclictest0-21swapper/012:25:420
2745899681,52cyclictest0-21swapper/008:45:330
27458996814,2cyclictest0-21swapper/009:50:430
27458996717,37cyclictest8050irq/56-dwc_otg_11:20:290
27458996715,37cyclictest0-21swapper/011:10:010
2745899671,52cyclictest0-21swapper/010:48:540
271642670,2sleep1221ktimersoftd/110:25:371
2746699664,24cyclictest0-21swapper/111:45:221
2745899661,47cyclictest29992-21modprobe12:35:170
2745899661,47cyclictest29992-21modprobe12:35:170
27458996613,42cyclictest16872-21/usr/sbin/munin08:10:330
27458996612,4cyclictest24659-1kworker/u9:011:14:510
27458996610,10cyclictest25554-21ssh09:36:480
2745899659,45cyclictest0-21swapper/008:02:040
2745899645,42cyclictest0-21swapper/011:38:140
2745899641,51cyclictest0-21swapper/009:20:460
27458996412,41cyclictest10351-21ps07:50:440
27458996311,42cyclictest878-21runrttasks07:11:160
2745899628,43cyclictest0-21swapper/011:19:080
27458996214,2cyclictest0-21swapper/008:30:320
27458996213,40cyclictest0-21swapper/011:54:250
27458996115,3cyclictest8050irq/56-dwc_otg_11:42:390
2745899611,14cyclictest6420-21ssh10:42:220
2745899611,11cyclictest0-21swapper/010:10:020
27458996010,8cyclictest10546-21ssh10:01:000
2745899591,41cyclictest0-21swapper/008:40:290
2745899591,15cyclictest0-21swapper/011:57:030
27482995811,27cyclictest30770-21latency_hist09:45:183
2747699584,13cyclictest7950irq/56-dwc_otg_08:35:282
323052570,2sleep30-21swapper/309:45:583
27482995717,12cyclictest0-21swapper/312:15:283
27458995713,40cyclictest0-21swapper/007:20:370
2745899571,15cyclictest1600-21wc08:55:440
2745899569,8cyclictest0-21swapper/008:21:390
2745899561,13cyclictest0-21swapper/007:55:330
27482995514,21cyclictest19605-21timerwakeupswit08:15:493
27482995514,21cyclictest19605-21timerwakeupswit08:15:493
2745899551,32cyclictest8895-21grep07:45:490
27482995418,16cyclictest15492-21ssh10:09:363
27482995417,17cyclictest8050irq/56-dwc_otg_08:30:293
2746699546,18cyclictest8050irq/56-dwc_otg_09:10:231
2746699546,15cyclictest7950irq/56-dwc_otg_09:40:341
2745899541,12cyclictest0-21swapper/008:05:420
27482995319,21cyclictest22125-21ssh12:25:313
2747699533,6cyclictest7850irq/56-dwc_otg07:50:332
27482995216,18cyclictest8050irq/56-dwc_otg_11:14:533
27482995213,22cyclictest2648-21memory09:50:413
2745899522,36cyclictest23296-21tail08:25:460
27458995222,15cyclictest613-21wget07:25:330
27458995222,15cyclictest613-21wget07:25:330
2745899521,15cyclictest0-21swapper/007:42:110
2745899521,11cyclictest0-21swapper/008:51:240
27482995114,24cyclictest15984-21ssh11:38:323
2745899517,9cyclictest0-21swapper/008:36:240
16722510,6sleep11671-21modprobe12:40:161
2748299509,22cyclictest0-21swapper/312:35:303
27482995013,23cyclictest10160-21sed07:50:413
27482995013,21cyclictest9541-21ssh12:10:223
2746699508,25cyclictest0-21swapper/108:15:351
2746699508,25cyclictest0-21swapper/108:15:351
27466995030,6cyclictest23-21ksoftirqd/110:40:551
27482994915,16cyclictest23056-21apt10:20:293
27482994915,11cyclictest7950irq/56-dwc_otg_09:30:313
27482994914,13cyclictest27419-21ssh11:51:413
2745899491,26cyclictest0-21swapper/007:35:410
2745899491,13cyclictest0-21swapper/009:02:370
27482994815,16cyclictest11913-21ssh10:04:113
27482994814,15cyclictest18308-21ssh10:12:313
27482994813,20cyclictest2684-21fschecks_time07:30:363
27482994813,11cyclictest8050irq/56-dwc_otg_08:45:313
27482994812,25cyclictest3290-21rm11:23:043
27482994811,25cyclictest15334-21seq09:22:563
2747699485,6cyclictest7950irq/56-dwc_otg_09:00:322
8112470,2sleep20-21swapper/209:48:222
2748299477,25cyclictest0-21swapper/311:05:333
27482994726,8cyclictest19259-21apt-get10:15:223
2746699477,26cyclictest0-21swapper/110:30:311
27466994720,16cyclictest0-21swapper/111:10:311
27466994716,19cyclictest0-21swapper/109:25:301
27482994619,11cyclictest5390-21ntpq10:41:033
27482994618,8cyclictest4198-21apt-get09:05:293
27482994613,23cyclictest23354-21ssh11:46:473
27482994611,19cyclictest9630-21missed_timers11:30:393
27482994610,10cyclictest31886-21ssh11:57:203
2746699468,25cyclictest0-21swapper/110:20:311
2746699467,25cyclictest0-21swapper/108:55:311
2746699467,24cyclictest0-21swapper/108:35:301
2746699464,4cyclictest0-21swapper/110:05:311
271112463,36sleep00-21swapper/007:09:320
2748299459,21cyclictest17942-21ssh10:59:133
2748299459,21cyclictest17942-21ssh10:59:133
2748299457,24cyclictest0-21swapper/307:45:343
2748299456,22cyclictest0-21swapper/309:17:163
2746699452,5cyclictest8050irq/56-dwc_otg_09:20:321
2746699451,22cyclictest1946-21latency10:35:391
2748299448,25cyclictest7950irq/56-dwc_otg_09:59:083
2748299448,25cyclictest7950irq/56-dwc_otg_09:59:083
27482994415,13cyclictest3491-21ssh12:02:123
27482994414,13cyclictest32639-21ssh11:20:063
27482994413,9cyclictest0-21swapper/308:10:293
27482994412,12cyclictest16724-21ssh09:25:233
27482994411,20cyclictest7950irq/56-dwc_otg_10:33:273
27476994418,5cyclictest12706-21latency_hist11:35:182
2746699447,23cyclictest0-21swapper/107:25:321
2746699447,23cyclictest0-21swapper/107:25:321
2746699446,6cyclictest0-21swapper/111:25:281
2746699442,26cyclictest0-21swapper/108:00:331
2748299439,13cyclictest8878-21ssh09:13:573
2748299438,19cyclictest13137-21ssh10:51:283
2748299437,27cyclictest29667-21sh12:34:423
2748299437,27cyclictest29667-21sh12:34:423
27482994317,12cyclictest2897-21ssh10:36:143
27482994314,13cyclictest21737-21rm11:04:423
27482994314,12cyclictest19507-21ssh12:21:063
2746699436,24cyclictest658-21ssh11:20:221
207932430,1sleep30-21swapper/311:44:503
2748299428,5cyclictest5517-21ssh12:05:303
2748299428,5cyclictest5517-21ssh12:05:303
2748299427,11cyclictest27142-21if_wlan010:25:373
2746699429,10cyclictest7950irq/56-dwc_otg_07:50:331
2746699427,25cyclictest0-21swapper/107:35:331
27466994212,6cyclictest23-21ksoftirqd/109:50:311
2748299418,24cyclictest0-21swapper/307:15:323
2747699417,25cyclictest0-21swapper/209:50:232
27476994112,5cyclictest14315-21unixbench_singl09:20:482
2746699413,8cyclictest8050irq/56-dwc_otg_08:05:291
27466994123,5cyclictest878-21runrttasks12:30:411
27466994123,5cyclictest878-21runrttasks12:30:411
2748299409,14cyclictest8050irq/56-dwc_otg_07:56:013
27482994010,11cyclictest9526-21ssh10:46:163
2746699407,23cyclictest0-21swapper/112:20:241
2746699406,24cyclictest0-21swapper/111:50:311
2746699406,22cyclictest0-21swapper/107:45:341
2746699402,6cyclictest8050irq/56-dwc_otg_09:00:321
2746699401,4cyclictest7850irq/56-dwc_otg11:30:301
27466994012,21cyclictest13933-21ssh11:35:421
263612408,19sleep20-21swapper/207:05:462
2748299394,11cyclictest0-21swapper/309:39:353
2747699397,22cyclictest0-21swapper/210:00:342
27476993932,5cyclictest31-21ksoftirqd/209:35:312
27476993919,5cyclictest8395-21sh09:59:152
27476993919,5cyclictest8395-21sh09:59:152
2746699393,27cyclictest0-21swapper/109:30:311
27466993912,6cyclictest7950irq/56-dwc_otg_12:15:181
2747699387,5cyclictest0-21swapper/208:40:292
2747699382,13cyclictest8050irq/56-dwc_otg_09:35:172
27476993818,6cyclictest17111-21fschecks_time09:25:352
27476993815,4cyclictest798-21/usr/sbin/munin08:55:342
2746699386,8cyclictest7850irq/56-dwc_otg07:30:321
2746699383,31cyclictest8050irq/56-dwc_otg_12:10:311
27466993819,5cyclictest27056-21latency_hist09:40:171
27466993815,6cyclictest0-21swapper/111:57:081
2746699381,24cyclictest28130-21latency08:40:381
2748299377,15cyclictest0-21swapper/308:07:383
2748299376,17cyclictest27076-21seq08:36:433
2748299371,6cyclictest31293-21aten_rbpower_vo07:20:343
27482993714,7cyclictest7950irq/56-dwc_otg_07:40:503
2747699379,14cyclictest8050irq/56-dwc_otg_09:05:442
2747699376,21cyclictest19702-21ssh10:15:322
2747699371,5cyclictest7850irq/56-dwc_otg10:21:072
27476993714,5cyclictest24564-21ls08:30:382
2746699375,20cyclictest29379-21ssh11:15:311
2746699374,5cyclictest7850irq/56-dwc_otg10:50:171
299802360,2sleep20-21swapper/210:30:172
2747699367,15cyclictest8050irq/56-dwc_otg_12:15:172
27476993617,10cyclictest27899-21if_wlan009:40:412
27476993611,5cyclictest7950irq/56-dwc_otg_10:45:292
2746699364,16cyclictest8050irq/56-dwc_otg_10:15:301
27466993621,4cyclictest21-21rcuc/109:19:541
27466993619,4cyclictest7850irq/56-dwc_otg09:45:381
2746699361,25cyclictest29608-21cut07:15:361
2748299357,10cyclictest8050irq/56-dwc_otg_08:42:143
2748299356,16cyclictest8050irq/56-dwc_otg_07:39:013
27482993519,13cyclictest22272-21/usr/sbin/munin08:25:323
2747699357,6cyclictest8050irq/56-dwc_otg_11:05:412
2747699357,24cyclictest0-21swapper/208:15:352
2747699357,24cyclictest0-21swapper/208:15:352
2747699357,23cyclictest0-21swapper/211:55:312
27476993528,4cyclictest16178-21sh10:10:232
2747699352,5cyclictest137-21systemd-journal08:20:392
2747699352,15cyclictest7950irq/56-dwc_otg_10:50:382
27476993514,4cyclictest19993-21sh12:22:012
2746699352,3cyclictest0-21swapper/111:10:071
27466993517,5cyclictest18441-21ssh11:40:461
27466993513,3cyclictest8050irq/56-dwc_otg_10:50:371
119032350,4sleep12746699cyclictest07:55:381
2748299347,10cyclictest14695-21runrttasks08:04:453
2748299341,21cyclictest2596-21ls09:00:333
2747699345,19cyclictest21513-21ssh11:45:312
2747699344,5cyclictest7950irq/56-dwc_otg_10:10:172
2747699342,23cyclictest940-21ls11:20:332
27476993416,6cyclictest29451-21aten_rbpower_po11:15:322
27476993415,11cyclictest8050irq/56-dwc_otg_10:35:392
2747699341,3cyclictest8050irq/56-dwc_otg_12:15:352
2747699341,23cyclictest29202-21ls11:15:172
2747699341,23cyclictest1523-21timerwakeupswit07:25:462
2747699341,23cyclictest1523-21timerwakeupswit07:25:462
2746699347,5cyclictest21-21rcuc/110:55:281
2746699347,5cyclictest21-21rcuc/110:55:281
27466993415,4cyclictest240372basename08:30:321
2748299338,12cyclictest2080-21taskset08:59:393
2748299332,10cyclictest8050irq/56-dwc_otg_08:24:353
27482993319,11cyclictest992-21/usr/sbin/munin07:25:383
27482993319,11cyclictest992-21/usr/sbin/munin07:25:383
2747699335,5cyclictest8050irq/56-dwc_otg_08:00:472
2747699334,5cyclictest7850irq/56-dwc_otg11:29:402
27476993314,6cyclictest31520-21fschecks_time08:50:352
2746699336,14cyclictest7950irq/56-dwc_otg_08:25:361
2746699333,7cyclictest7950irq/56-dwc_otg_12:00:351
27466993311,6cyclictest8012-21chrt09:58:241
27466993311,6cyclictest8012-21chrt09:58:241
2748299323,3cyclictest0-21swapper/307:10:413
2747699329,12cyclictest0-21swapper/211:50:252
2747699327,5cyclictest8050irq/56-dwc_otg_11:35:352
2747699326,13cyclictest7950irq/56-dwc_otg_12:40:172
2747699325,13cyclictest8050irq/56-dwc_otg_07:55:402
2747699323,5cyclictest0-21swapper/211:40:352
27476993217,5cyclictest8661-21sh12:09:092
27476993217,5cyclictest8661-21sh12:09:092
2747699321,24cyclictest0-21swapper/209:10:332
2746699328,10cyclictest0-21swapper/107:15:171
27466993215,10cyclictest221ktimersoftd/109:05:321
27466993212,5cyclictest23-21ksoftirqd/112:10:041
27466993212,5cyclictest23-21ksoftirqd/112:10:041
2748299313,13cyclictest8050irq/56-dwc_otg_08:50:453
2747699315,14cyclictest30401-21aten_rbpower_vo10:30:322
2747699314,14cyclictest8050irq/56-dwc_otg_11:03:042
2747699312,7cyclictest7950irq/56-dwc_otg_07:20:172
2747699311,19cyclictest8762-21timerwakeupswit07:45:482
2746699315,6cyclictest7850irq/56-dwc_otg11:00:331
2746699313,11cyclictest23-21ksoftirqd/107:25:171
2746699312,4cyclictest0-21swapper/108:10:481
27466993115,4cyclictest21-21rcuc/110:01:391
2746699311,14cyclictest7850irq/56-dwc_otg08:20:391
2747699309,4cyclictest0-21swapper/212:00:352
2747699308,4cyclictest8050irq/56-dwc_otg_07:10:232
2747699306,12cyclictest8050irq/56-dwc_otg_10:42:232
2747699302,7cyclictest7950irq/56-dwc_otg_12:30:362
2747699302,7cyclictest7950irq/56-dwc_otg_12:30:362
2747699302,19cyclictest0-21swapper/208:45:432
2747699301,4cyclictest17492-21ntp_states08:10:412
27476993012,6cyclictest31-21ksoftirqd/207:30:362
2746699303,5cyclictest7950irq/56-dwc_otg_08:45:351
2746699303,5cyclictest0-21swapper/107:40:301
2747699292,8cyclictest0-21swapper/207:43:222
2747699282,5cyclictest7950irq/56-dwc_otg_08:25:462
27476992812,5cyclictest5602-21seq07:39:112
2747699281,13cyclictest8050irq/56-dwc_otg_08:05:532
2746699281,16cyclictest0-21swapper/108:55:161
2747699279,5cyclictest32145-21cat07:20:462
270022274,14sleep30-21swapper/307:08:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional