You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 16:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Tue Dec 02, 2025 12:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2005822920,2sleep00-21swapper/007:05:380
2269121000,5sleep12111599cyclictest10:30:301
2269121000,5sleep12111599cyclictest10:30:301
2110399849,20cyclictest25797-21grep07:20:380
55372770,2sleep10-21swapper/109:17:321
2110399771,31cyclictest23083-21wget07:15:250
21103997624,5cyclictest278332sleep009:50:410
2110399752,28cyclictest11398-21timerandwakeup09:25:390
2110399752,28cyclictest11398-21timerandwakeup09:25:390
21103997514,28cyclictest0-21swapper/009:35:260
21103997512,52cyclictest0-21swapper/009:10:250
2110399731,23cyclictest10292-21hostname12:30:260
2110399711,20cyclictest0-21swapper/011:45:390
2110399702,30cyclictest0-21swapper/011:31:370
21103997022,4cyclictest7750irq/56-dwc_otg10:26:530
2110399701,22cyclictest11332-21ntpq08:10:350
2110399697,19cyclictest0-21swapper/012:10:340
2110399697,19cyclictest0-21swapper/012:10:340
21103996952,3cyclictest0-21swapper/009:40:250
147302690,5sleep02110399cyclictest10:19:060
2110399682,20cyclictest0-21swapper/011:05:390
64972670,1sleep00-21swapper/009:19:480
292642670,1sleep00-21swapper/009:00:320
21103996717,12cyclictest0-21swapper/011:40:270
21103996717,12cyclictest0-21swapper/011:40:270
2110399671,17cyclictest0-21swapper/011:55:120
2110399661,44cyclictest0-21swapper/012:15:230
2110399651,30cyclictest0-21swapper/010:35:180
2110399651,26cyclictest0-21swapper/012:05:120
21103996424,3cyclictest0-21swapper/010:10:590
2110399632,4cyclictest0-21swapper/012:25:120
21103996323,6cyclictest7750irq/56-dwc_otg11:00:340
21103996315,45cyclictest101ktimersoftd/012:35:270
21103996315,13cyclictest0-21swapper/009:20:300
2110399627,21cyclictest0-21swapper/008:40:270
21103995820,21cyclictest1120-21runrttasks08:40:120
21103995819,11cyclictest0-21swapper/010:58:080
21103995814,22cyclictest23151-21unixbench_singl11:15:420
199582580,5sleep219957-21latency_hist09:40:112
2110399571,9cyclictest0-21swapper/010:20:260
2110399571,9cyclictest0-21swapper/010:20:260
21103995627,7cyclictest7850irq/56-dwc_otg_11:15:120
2110399553,46cyclictest0-21swapper/008:20:240
21103995524,7cyclictest4364-21grep07:50:370
21103995524,5cyclictest5254-21tr10:05:270
21103995423,12cyclictest8052-21unixbench_multi08:00:400
286812530,2sleep30-21swapper/309:52:083
21103995322,14cyclictest677550rm11:39:590
21103995318,3cyclictest0-21swapper/011:25:250
2110399521,27cyclictest26442-21ssh09:50:110
21115995110,13cyclictest11-21rcu_preempt11:45:231
2110399517,40cyclictest0-21swapper/008:30:250
2110399517,3cyclictest0-21swapper/009:55:370
2110399517,3cyclictest0-21swapper/009:55:370
2110399511,30cyclictest4769-21cut10:50:300
2110399511,30cyclictest23703-21fschecks_time08:45:330
19682510,2sleep30-21swapper/311:31:593
2110399491,28cyclictest29619-21chrt10:40:230
2111599482,8cyclictest7750irq/56-dwc_otg11:15:251
21103994837,4cyclictest7850irq/56-dwc_otg_11:55:390
21103994825,5cyclictest6453-21wget12:25:260
2110399482,32cyclictest0-21swapper/012:09:010
2110399481,37cyclictest0-21swapper/009:05:320
2110399481,31cyclictest14161-21fschecks_time09:30:490
2110399481,27cyclictest885-21aten_rbpower_vo10:45:270
187772480,11sleep37950irq/56-dwc_otg_08:30:363
2112199477,27cyclictest0-21swapper/207:10:242
2110399471,30cyclictest0-21swapper/010:31:560
2110399471,30cyclictest0-21swapper/010:31:560
2112199466,28cyclictest0-21swapper/211:30:252
21103994621,11cyclictest0-21swapper/011:25:120
2110399461,23cyclictest21161-21apt-config07:10:120
2110399461,22cyclictest0-21swapper/008:25:370
21103994610,5cyclictest0-21swapper/007:30:320
323132450,2sleep20-21swapper/210:43:482
2112699457,26cyclictest0-21swapper/308:40:273
2112699457,24cyclictest0-21swapper/307:10:243
21103994526,8cyclictest28625-21grep09:00:110
2110399452,25cyclictest1120-21runrttasks07:40:340
21126994419,12cyclictest10835-21if_wlan009:25:313
21126994419,12cyclictest10835-21if_wlan009:25:313
21121994414,6cyclictest7950irq/56-dwc_otg_08:55:112
21121994414,6cyclictest7950irq/56-dwc_otg_08:55:112
21103994420,5cyclictest0-21swapper/010:05:110
2112699437,23cyclictest0-21swapper/310:15:263
2112199437,24cyclictest0-21swapper/210:15:262
2111599437,22cyclictest0-21swapper/108:50:261
2111599437,22cyclictest0-21swapper/108:50:261
21103994317,16cyclictest0-21swapper/008:06:430
21126994217,6cyclictest30591-21/usr/sbin/munin09:05:323
2112199426,24cyclictest0-21swapper/208:20:242
21103994220,12cyclictest0-21swapper/007:58:400
2110399421,27cyclictest0-21swapper/008:50:230
2110399421,27cyclictest0-21swapper/008:50:230
2112199418,4cyclictest7950irq/56-dwc_otg_09:45:232
21115994115,6cyclictest7950irq/56-dwc_otg_11:50:271
2110399412,5cyclictest31805-21cron07:40:120
2110399411,21cyclictest3024-21latency_hist07:50:120
2110399411,21cyclictest3024-21latency_hist07:50:120
270152400,2sleep30-21swapper/308:55:283
2112699402,3cyclictest7750irq/56-dwc_otg12:05:263
21103994016,14cyclictest0-21swapper/007:29:330
21103994014,12cyclictest0-21swapper/008:15:210
63222390,2sleep20-21swapper/210:52:052
242262390,1sleep30-21swapper/312:04:443
2112699396,20cyclictest7950irq/56-dwc_otg_11:50:253
2112699395,5cyclictest7950irq/56-dwc_otg_09:55:323
2112699395,5cyclictest7950irq/56-dwc_otg_09:55:323
2112199397,22cyclictest0-21swapper/212:30:252
2112199392,24cyclictest0-21swapper/207:49:252
2112199392,24cyclictest0-21swapper/207:49:252
2111599394,31cyclictest759-21ssh10:45:231
2111599393,18cyclictest7750irq/56-dwc_otg12:20:251
21115993919,6cyclictest13684-21ls12:35:121
21115993918,12cyclictest15859-21ps11:05:391
21115993910,8cyclictest0-21swapper/110:10:241
2112699387,5cyclictest0-21swapper/309:42:233
2112199389,4cyclictest0-21swapper/209:05:252
21121993816,11cyclictest410650ssh12:20:512
2111599387,26cyclictest0-21swapper/108:20:171
2111599385,18cyclictest7850irq/56-dwc_otg_08:10:171
2111599385,15cyclictest7850irq/56-dwc_otg_09:56:061
2111599385,15cyclictest7850irq/56-dwc_otg_09:56:061
21115993818,4cyclictest18841-21fschecks_time11:10:321
112872380,2sleep20-21swapper/209:25:382
112872380,2sleep20-21swapper/209:25:382
2112699376,3cyclictest7850irq/56-dwc_otg_10:10:253
2112699372,30cyclictest7950irq/56-dwc_otg_09:35:243
2112199374,23cyclictest19086-21ssh10:25:192
21121993729,5cyclictest32524-21ssh09:10:272
2112199372,15cyclictest7950irq/56-dwc_otg_11:15:392
2111599377,26cyclictest0-21swapper/107:35:271
2111599377,25cyclictest0-21swapper/108:25:251
2111599377,23cyclictest0-21swapper/108:45:271
2111599377,23cyclictest0-21swapper/107:45:251
2111599377,23cyclictest0-21swapper/107:45:251
53002360,1sleep30-21swapper/309:17:013
2112699365,20cyclictest7239-21ssh11:40:283
2112699365,20cyclictest7239-21ssh11:40:283
21126993615,11cyclictest17786-21sh11:09:193
2112199362,15cyclictest7950irq/56-dwc_otg_08:35:312
21121993617,4cyclictest9430-21sshd09:23:312
2112199361,28cyclictest12204-21ssh11:46:192
21121993610,11cyclictest7950irq/56-dwc_otg_08:30:112
2111599368,7cyclictest0-21swapper/111:25:241
2111599367,23cyclictest0-21swapper/110:25:271
2111599365,4cyclictest0-21swapper/110:37:201
2111599362,23cyclictest0-21swapper/110:20:251
2111599362,23cyclictest0-21swapper/110:20:251
21115993613,14cyclictest23-21ksoftirqd/112:05:261
2111599361,24cyclictest13162-21switchtime10:15:391
2111599361,23cyclictest30888-21cut09:05:291
21115993611,5cyclictest7950irq/56-dwc_otg_07:55:341
2111599361,12cyclictest0-21swapper/109:11:421
2112699356,5cyclictest7850irq/56-dwc_otg_10:35:373
21126993514,5cyclictest18918-21cat10:25:133
21126993514,5cyclictest18918-21cat10:25:133
2112199359,12cyclictest7950irq/56-dwc_otg_12:25:302
2112199356,13cyclictest7850irq/56-dwc_otg_07:20:112
2112199354,18cyclictest7950irq/56-dwc_otg_08:10:412
2112199353,4cyclictest7850irq/56-dwc_otg_12:05:252
2112199352,25cyclictest0-21swapper/211:24:412
2112199351,5cyclictest17076-21ssh11:07:352
2112199351,3cyclictest0-21swapper/212:35:332
21121993513,9cyclictest32176-21diskmemload09:57:032
21121993513,9cyclictest32176-21diskmemload09:57:032
2112199351,28cyclictest0-21swapper/210:10:002
2111599357,25cyclictest0-21swapper/108:05:281
2111599357,22cyclictest0-21swapper/107:25:251
2111599355,9cyclictest8674-21ssh09:21:311
2111599353,18cyclictest28127-21runrttasks12:10:251
2111599353,18cyclictest28127-21runrttasks12:10:251
21115993524,7cyclictest0-21swapper/107:10:221
21115993513,16cyclictest0-21swapper/112:00:221
2112699349,15cyclictest0-21swapper/310:25:193
2112699348,8cyclictest23444-21kworker/3:211:20:393
2112699345,23cyclictest4397-21ssh10:50:173
21126993428,4cyclictest6805-21ls08:00:123
21126993416,4cyclictest1569-21ssh10:00:233
2112199349,13cyclictest0-21swapper/208:40:272
2112199348,5cyclictest0-21swapper/209:30:482
2112199346,19cyclictest0-21swapper/208:30:252
2112199345,4cyclictest7850irq/56-dwc_otg_07:35:122
21121993411,8cyclictest0-21swapper/207:50:262
2111599346,13cyclictest21098-21cyclictest11:40:271
2111599346,13cyclictest21098-21cyclictest11:40:271
2111599345,16cyclictest32176-21diskmemload10:42:571
2111599345,15cyclictest7950irq/56-dwc_otg_08:40:281
2111599343,18cyclictest0-21swapper/112:26:461
21115993411,14cyclictest0-21swapper/107:20:371
234452330,2sleep30-21swapper/311:16:133
2112699334,16cyclictest7850irq/56-dwc_otg_08:48:503
2112699333,6cyclictest7950irq/56-dwc_otg_10:40:303
21126993320,4cyclictest260772chrt07:21:133
2112699331,8cyclictest585250rm11:37:523
2112699331,6cyclictest0-21swapper/312:30:123
2112699331,5cyclictest0-21swapper/310:46:503
21126993315,8cyclictest7950irq/56-dwc_otg_11:25:273
2112199339,4cyclictest0-21swapper/210:56:082
2112199338,6cyclictest7950irq/56-dwc_otg_11:50:392
2112199332,20cyclictest7486-21ssh11:40:322
2112199332,20cyclictest7486-21ssh11:40:322
2112199332,16cyclictest7950irq/56-dwc_otg_11:25:302
2112199331,5cyclictest18041-21df11:55:302
2112199331,5cyclictest0-21swapper/212:00:372
2112199331,3cyclictest0-21swapper/210:32:162
2112199331,3cyclictest0-21swapper/210:32:162
21121993313,4cyclictest8682-21sh10:10:242
21121993311,5cyclictest9900-21head08:05:432
2111599332,7cyclictest7750irq/56-dwc_otg11:04:141
2111599332,18cyclictest0-21swapper/111:35:281
21115993319,9cyclictest32092-21grep07:40:251
2111599331,22cyclictest29086-21cut09:00:301
2111599331,22cyclictest0-21swapper/111:21:381
2112699329,10cyclictest7850irq/56-dwc_otg_08:50:283
2112699329,10cyclictest7850irq/56-dwc_otg_08:50:283
2112699327,12cyclictest0-21swapper/312:36:483
2112699325,4cyclictest7850irq/56-dwc_otg_11:01:003
2112699321,6cyclictest0-21swapper/310:05:483
2112699321,4cyclictest7950irq/56-dwc_otg_12:30:243
2112699321,23cyclictest17920-21ls08:30:123
21126993211,4cyclictest1120-21runrttasks12:14:533
21126993211,4cyclictest1120-21runrttasks12:14:533
2112199328,12cyclictest7950irq/56-dwc_otg_09:43:072
2112199326,10cyclictest7950irq/56-dwc_otg_07:25:252
2112199323,4cyclictest7950irq/56-dwc_otg_09:15:312
2112199322,4cyclictest26061-21apt-get10:35:252
2112199322,13cyclictest7850irq/56-dwc_otg_08:05:122
2112199321,22cyclictest0-21swapper/208:47:332
2112199321,11cyclictest2976-21ssh10:47:452
21121993210,4cyclictest29-21rcuc/210:00:412
2111599329,4cyclictest0-21swapper/111:55:341
2111599329,12cyclictest0-21swapper/112:15:361
2111599327,15cyclictest0-21swapper/107:15:241
2111599326,6cyclictest7750irq/56-dwc_otg08:35:391
2111599326,10cyclictest7950irq/56-dwc_otg_09:45:291
2111599324,12cyclictest7750irq/56-dwc_otg10:50:301
2111599323,11cyclictest7850irq/56-dwc_otg_10:55:321
2111599322,8cyclictest13708-21latency_hist09:30:131
2111599322,8cyclictest13708-21latency_hist09:30:131
2111599322,7cyclictest18349-21ssh09:36:091
21115993222,7cyclictest2788-21grep10:00:411
21115993220,8cyclictest221ktimersoftd/109:50:251
2111599321,9cyclictest28916-21if_wlan007:30:481
2111599321,5cyclictest7750irq/56-dwc_otg10:05:411
2111599321,5cyclictest15434-21taskset09:32:041
2112699317,11cyclictest0-21swapper/311:45:283
2112699312,5cyclictest0-21swapper/309:45:373
2112699312,3cyclictest7750irq/56-dwc_otg07:35:253
2112699311,5cyclictest0-21swapper/311:14:093
2112699311,17cyclictest0-21swapper/310:34:513
2112699311,17cyclictest0-21swapper/310:34:513
2112199316,9cyclictest7850irq/56-dwc_otg_11:00:382
2112199311,6cyclictest19450-21sh11:10:412
2112199311,4cyclictest26777-21apt-get09:50:242
2111599317,18cyclictest0-21swapper/108:15:241
21115993123,5cyclictest221ktimersoftd/109:45:111
2112699308,3cyclictest0-21swapper/309:33:433
2112699307,4cyclictest8196-21tail09:20:413
2112699307,10cyclictest7950irq/56-dwc_otg_08:10:403
2112699306,5cyclictest7850irq/56-dwc_otg_12:15:293
2112699305,4cyclictest7950irq/56-dwc_otg_11:55:543
2112699305,4cyclictest7950irq/56-dwc_otg_07:40:403
2112699304,3cyclictest0-21swapper/308:05:373
21126993014,9cyclictest3585-21/usr/sbin/munin07:50:273
2112699301,18cyclictest0-21swapper/308:20:243
2112699301,17cyclictest0-21swapper/312:20:333
2112199305,5cyclictest7850irq/56-dwc_otg_12:10:272
2112199305,5cyclictest7850irq/56-dwc_otg_12:10:272
2112199303,9cyclictest25950-21tail07:20:402
2112199303,5cyclictest7950irq/56-dwc_otg_07:45:122
2112199301,6cyclictest32176-21diskmemload12:15:322
2112199301,16cyclictest7950irq/56-dwc_otg_11:35:272
2112199301,12cyclictest7950irq/56-dwc_otg_08:20:112
2111599305,14cyclictest0-21swapper/112:36:011
2111599302,3cyclictest0-21swapper/108:30:331
2111599302,13cyclictest7750irq/56-dwc_otg11:32:241
2112699297,10cyclictest7950irq/56-dwc_otg_08:03:413
2112699296,5cyclictest7850irq/56-dwc_otg_07:45:403
2112699296,5cyclictest7850irq/56-dwc_otg_07:45:403
2112699294,4cyclictest0-21swapper/309:14:433
2112699292,5cyclictest7850irq/56-dwc_otg_08:20:113
21126992920,7cyclictest9110-21ssh10:55:433
2112199292,9cyclictest0-21swapper/209:03:322
21121992912,7cyclictest5076-21apt-config07:55:122
21121992911,9cyclictest16826-21ssh10:20:442
21121992911,9cyclictest16826-21ssh10:20:442
2111599293,5cyclictest31331-21snmpd08:03:411
196112296,14sleep119688-21/usr/sbin/munin07:05:321
2112699288,5cyclictest7850irq/56-dwc_otg_07:20:113
2112199282,16cyclictest0-21swapper/208:55:402
2112199281,5cyclictest0-21swapper/207:35:372
2111599281,12cyclictest7750irq/56-dwc_otg08:55:291
2112699275,4cyclictest7850irq/56-dwc_otg_07:25:263
2112699271,10cyclictest0-21swapper/308:35:313
2111599271,4cyclictest7750irq/56-dwc_otg07:50:371
206982273,15sleep30-21swapper/307:08:373
2112699266,4cyclictest7950irq/56-dwc_otg_09:03:533
205192264,15sleep20-21swapper/207:06:182
2112699251,4cyclictest28450-21/usr/sbin/munin07:30:473
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional