You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-28 - 23:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Sat Jun 28, 2025 12:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
184352980,5sleep11269299cyclictest12:05:361
184352980,5sleep11269299cyclictest12:05:361
57192960,2sleep20-21swapper/208:20:092
148662880,2sleep10-21swapper/112:00:441
262272850,1sleep30-21swapper/310:45:463
12691998331,7cyclictest7950irq/56-dwc_otg_08:40:080
109532820,4sleep01269199cyclictest08:35:080
237172810,2sleep30-21swapper/312:11:503
289102790,2sleep10-21swapper/112:18:441
280172740,1sleep20-21swapper/212:17:082
326912730,2sleep30-21swapper/309:22:093
12692997126,13cyclictest8466-21wc10:20:331
1269199712,66cyclictest0-21swapper/011:50:260
12691997111,11cyclictest0-21swapper/010:15:240
12692996932,22cyclictest30777-21apt-get09:20:221
12692996725,27cyclictest28207-21cat07:50:421
12692996628,20cyclictest31460-21ssh12:20:511
12692996623,11cyclictest18922-21grep07:25:341
12691996620,10cyclictest0-21swapper/010:20:220
12691996619,25cyclictest30477-21cron09:20:080
121862663,15sleep30-21swapper/307:07:123
1269199652,36cyclictest0-21swapper/009:50:230
1269199652,36cyclictest0-21swapper/009:50:230
12692996431,19cyclictest14647-21ssh09:42:311
12692996431,12cyclictest2447-21fschecks_time08:10:311
12692996427,12cyclictest19969-21sh09:00:091
12692996422,13cyclictest25392-21latency_hist07:45:101
12692996421,18cyclictest8140-21memory08:25:381
1269199645,55cyclictest0-21swapper/012:20:300
12692996329,18cyclictest23606-21ssh11:30:031
12692996329,18cyclictest23606-21ssh11:30:031
12692996325,16cyclictest30138-21ssh09:19:181
12692996324,12cyclictest31289-21ssh11:40:261
12692996321,12cyclictest20019-21ssh11:24:471
162322620,1sleep00-21swapper/007:20:090
12692996233,13cyclictest15709-21ls08:45:391
12692996229,8cyclictest30336-21users07:55:381
12692996225,11cyclictest27522-21basename11:35:261
56632610,1sleep30-21swapper/312:30:093
12692996126,24cyclictest15050-21rm11:16:501
12692996124,14cyclictest11897-21sshd12:36:391
12692996030,11cyclictest25664-21df_inode07:45:281
12692996023,11cyclictest32469-21ssh10:55:341
76292590,4sleep27631-21aten_rbpower_en08:25:282
12692995934,7cyclictest20216-21apt-get09:00:231
12692995934,7cyclictest20216-21apt-get09:00:231
12692995925,12cyclictest4778-21seq10:15:281
12692995924,19cyclictest18460-21ssh09:47:561
12692995921,22cyclictest22238-21if_eth012:10:301
12691995933,4cyclictest7950irq/56-dwc_otg_12:15:090
12692995826,24cyclictest3508-21ssh11:46:001
12692995826,15cyclictest3339-21ssh12:26:101
12692995824,14cyclictest24476-21ssh10:45:011
12692995823,21cyclictest9435-21rm12:34:431
12692995823,21cyclictest9435-21rm12:34:431
12691995838,5cyclictest7850irq/56-dwc_otg07:25:090
12691995832,4cyclictest5507-21cron08:20:080
12692995729,21cyclictest28001-21ssh10:50:061
12692995728,18cyclictest6237-21ssh11:05:131
12692995727,11cyclictest12629-21ssh11:14:271
12692995726,11cyclictest10690-21ssh09:36:561
12692995724,12cyclictest29694-21ssh10:50:451
1269199572,5cyclictest3182-21fschecks_time11:00:330
1269199572,5cyclictest3182-21fschecks_time11:00:330
129162560,2sleep00-21swapper/011:15:020
12692995628,18cyclictest23121-21sshd07:36:391
12692995624,18cyclictest12187-21ssh11:58:251
12692995623,20cyclictest8567-21sh11:54:171
12692995622,9cyclictest0-21swapper/109:50:281
12692995622,9cyclictest0-21swapper/109:50:281
12692995618,16cyclictest6830-21ssh09:31:041
12692995618,13cyclictest946-21latency_hist10:10:101
1269199568,4cyclictest0-21swapper/011:30:210
12692995522,18cyclictest134-21systemd-journal11:30:101
12692995518,20cyclictest8050irq/56-dwc_otg_11:03:391
12692995518,20cyclictest8050irq/56-dwc_otg_11:03:391
1269199557,32cyclictest0-21swapper/010:55:240
1269199552,38cyclictest18668-21cut08:55:270
12692995432,11cyclictest969-21snmpd07:23:071
12692995426,11cyclictest16285-21ssh10:31:181
12692995425,11cyclictest19985-21chrt10:36:591
12692995423,21cyclictest29344-21ssh10:04:141
12692995423,18cyclictest18109-21taskset08:54:391
12692995419,15cyclictest32279-21seq08:02:121
12692995419,15cyclictest25869-21ssh09:59:001
12692995418,13cyclictest2281-21irqstats09:25:301
12692995413,10cyclictest7850irq/56-dwc_otg09:14:491
12691995433,5cyclictest7950irq/56-dwc_otg_11:25:310
12691995433,5cyclictest7950irq/56-dwc_otg_11:25:310
12691995426,14cyclictest7950irq/56-dwc_otg_08:25:260
12691995417,34cyclictest1453-21latency_hist09:25:080
1269199541,6cyclictest0-21swapper/007:45:230
12692995324,11cyclictest3233-21ssh10:12:441
12691995329,14cyclictest2987-21grep12:25:430
12691995328,5cyclictest7950irq/56-dwc_otg_09:40:090
12692995224,12cyclictest12835-21ssh10:26:151
12692995224,12cyclictest12835-21ssh10:26:151
12692995216,10cyclictest7850irq/56-dwc_otg08:35:301
12691995227,12cyclictest8050irq/56-dwc_otg_09:40:420
1269199521,32cyclictest0-21swapper/009:31:440
1269199521,27cyclictest31127-21grep11:40:230
12691995210,18cyclictest0-21swapper/007:35:090
108822520,4sleep010875-21apt07:05:220
12691995143,2cyclictest0-21swapper/007:40:360
12691995125,5cyclictest6928-21unixbench_singl08:20:430
12691995125,5cyclictest18103-21unixbench_multi11:20:430
12691995124,16cyclictest1433-21fschecks_time10:10:290
12691995124,14cyclictest0-21swapper/007:55:240
1269199512,33cyclictest0-21swapper/010:40:350
12691995115,20cyclictest13307-21latency_hist12:00:090
1269199511,39cyclictest0-21swapper/007:35:290
1269199511,30cyclictest23586-21diskmemload10:25:430
1269199511,30cyclictest23586-21diskmemload10:25:430
1269499502,10cyclictest8050irq/56-dwc_otg_11:00:283
1269499502,10cyclictest8050irq/56-dwc_otg_11:00:283
12692995020,14cyclictest7397-21sort08:25:091
12692995018,9cyclictest7850irq/56-dwc_otg08:34:291
12691995022,17cyclictest0-21swapper/010:51:050
12691995022,11cyclictest7950irq/56-dwc_otg_09:45:410
12691995014,5cyclictest0-21swapper/010:05:220
1269499494,20cyclictest8050irq/56-dwc_otg_11:30:323
12692994927,11cyclictest16075-21taskset07:20:001
1269199492,27cyclictest0-21swapper/011:16:540
1269199491,33cyclictest0-21swapper/009:13:480
81342480,8sleep2301ktimersoftd/209:34:122
233962480,6sleep21269399cyclictest10:42:272
1269499487,26cyclictest0-21swapper/311:50:283
12692994822,11cyclictest12973-21taskset08:40:161
12692994820,10cyclictest0-21swapper/108:15:241
12692994819,18cyclictest14224-21runrttasks07:14:351
12691994836,5cyclictest7950irq/56-dwc_otg_12:00:380
1269199481,4cyclictest29009-21ls07:55:100
1269199481,26cyclictest25434-21latency10:45:320
1269499475,25cyclictest19993-21runrttasks09:00:093
1269399477,25cyclictest0-21swapper/208:30:242
12691994725,3cyclictest0-21swapper/008:40:440
1269199472,31cyclictest0-21swapper/011:45:290
1269199472,25cyclictest0-21swapper/010:30:190
12691994719,19cyclictest0-21swapper/011:35:380
12691994717,21cyclictest0-21swapper/009:02:000
12691994717,21cyclictest0-21swapper/009:02:000
1269499467,25cyclictest0-21swapper/308:00:243
1269499467,25cyclictest0-21swapper/307:25:233
1269399467,22cyclictest0-21swapper/207:25:232
12692994622,11cyclictest21513-21chrt07:34:111
12691994633,4cyclictest0-21swapper/009:55:240
12691994628,3cyclictest0-21swapper/009:27:590
12691994627,5cyclictest7950irq/56-dwc_otg_12:35:410
1269199462,28cyclictest0-21swapper/011:06:100
1269199462,21cyclictest0-21swapper/012:15:370
12691994621,5cyclictest12974-21kworker/u8:412:35:090
12691994621,5cyclictest12974-21kworker/u8:412:35:090
12691994617,20cyclictest0-21swapper/008:09:230
12691994616,15cyclictest0-21swapper/007:30:080
1269199461,31cyclictest0-21swapper/010:35:330
1269199461,26cyclictest17706-21aten_rbpower_cu12:05:260
1269199461,26cyclictest17706-21aten_rbpower_cu12:05:260
1269499458,22cyclictest8050irq/56-dwc_otg_09:45:203
1269499457,24cyclictest0-21swapper/310:30:233
1269399457,22cyclictest0-21swapper/208:00:242
12692994519,10cyclictest0-21swapper/108:05:361
1269199459,31cyclictest0-21swapper/009:05:270
1269199452,34cyclictest0-21swapper/010:00:210
1269199452,31cyclictest0-21swapper/007:10:390
1269499447,10cyclictest0-21swapper/309:50:233
1269499447,10cyclictest0-21swapper/309:50:233
1269499446,24cyclictest0-21swapper/308:10:233
1269499445,10cyclictest7850irq/56-dwc_otg10:25:253
1269499445,10cyclictest7850irq/56-dwc_otg10:25:253
12694994422,10cyclictest8050irq/56-dwc_otg_12:15:523
12694994416,15cyclictest8050irq/56-dwc_otg_11:27:283
12694994416,15cyclictest8050irq/56-dwc_otg_11:27:283
12694994414,22cyclictest381ktimersoftd/310:50:293
12694994410,22cyclictest646-21sh12:23:533
1269399446,24cyclictest0-21swapper/208:10:232
1269499436,23cyclictest0-21swapper/308:15:263
1269399431,11cyclictest7850irq/56-dwc_otg09:50:292
1269399431,11cyclictest7850irq/56-dwc_otg09:50:292
12692994334,7cyclictest22580-21memory09:05:331
1269199432,33cyclictest0-21swapper/008:14:380
1269199432,29cyclictest0-21swapper/008:45:400
1269499427,26cyclictest0-21swapper/307:55:153
1269499424,21cyclictest27850-21perl11:35:303
12694994220,12cyclictest15002-21ssh12:00:593
12694994219,14cyclictest837-21rm10:56:053
1269199421,21cyclictest31086-21cut08:00:250
1269499417,22cyclictest0-21swapper/308:25:263
1269499414,18cyclictest8709-21/usr/sbin/munin10:20:373
1269499414,15cyclictest510-21rs:main3
12694994119,13cyclictest1748-21/usr/sbin/munin09:25:363
1269399417,19cyclictest8050irq/56-dwc_otg_09:00:092
12693994116,9cyclictest21002-21ssh10:39:192
1269399411,14cyclictest0-21swapper/207:50:282
114372410,6sleep31269499cyclictest11:11:273
147252400,2sleep20-21swapper/211:16:052
1269499406,16cyclictest8050irq/56-dwc_otg_09:37:053
1269499406,15cyclictest30374-21ssh09:19:543
1269499406,12cyclictest762-21munin-run10:10:093
12694994022,6cyclictest10287-21sh12:35:273
12694994019,10cyclictest8050irq/56-dwc_otg_11:57:033
1269399407,21cyclictest0-21swapper/211:50:282
1269399406,22cyclictest0-21swapper/212:25:282
1269399403,20cyclictest301ktimersoftd/208:10:102
12691994016,3cyclictest0-21swapper/008:50:340
1269499396,25cyclictest12487-21ssh09:40:153
1269499395,17cyclictest206712chrt09:00:313
1269499395,17cyclictest206712chrt09:00:313
12694993919,13cyclictest7972-21ssh12:32:023
12694993919,13cyclictest7972-21ssh12:32:023
1269499391,23cyclictest0-21swapper/308:30:243
1269399397,23cyclictest0-21swapper/208:45:252
1269399397,19cyclictest0-21swapper/207:40:242
1269499388,17cyclictest16521-21ssh11:20:083
1269499388,13cyclictest8050irq/56-dwc_otg_11:46:263
12694993820,9cyclictest3041-21ssh10:12:163
12694993819,11cyclictest8050irq/56-dwc_otg_12:05:323
12694993819,11cyclictest8050irq/56-dwc_otg_12:05:323
12694993813,10cyclictest8050irq/56-dwc_otg_09:31:103
1269399381,25cyclictest9860-21ls09:35:402
12693993811,17cyclictest301ktimersoftd/210:33:122
1269499378,4cyclictest7950irq/56-dwc_otg_11:40:503
1269499377,12cyclictest16348-21apt-get07:20:233
1269499377,11cyclictest15830-21cat08:45:403
12694993716,6cyclictest955-21runrttasks07:10:373
1269499371,24cyclictest23778-21ssh09:10:093
1269399377,15cyclictest0-21swapper/212:35:252
1269399376,17cyclictest29-21rcuc/211:27:302
1269399376,17cyclictest29-21rcuc/211:27:302
12693993720,8cyclictest0-21swapper/211:38:242
12693993716,4cyclictest955-21runrttasks10:10:332
12693993715,8cyclictest301ktimersoftd/210:50:042
1269499369,3cyclictest0-21swapper/311:05:303
1269499367,17cyclictest0-21swapper/307:40:243
12694993610,14cyclictest4774-21df_inode10:15:283
1269399367,19cyclictest0-21swapper/209:40:152
1269399366,24cyclictest301ktimersoftd/210:10:102
1269399366,14cyclictest29-21rcuc/211:56:362
1269399365,14cyclictest29-21rcuc/209:25:332
1269399363,7cyclictest7950irq/56-dwc_otg_11:25:092
1269399363,15cyclictest29-21rcuc/211:43:202
12693993629,4cyclictest5967-21aten_rbpower_en12:30:242
12693993629,4cyclictest5967-21aten_rbpower_en12:30:242
12693993614,6cyclictest6129-21latency_hist11:05:092
12693993614,6cyclictest6129-21latency_hist11:05:092
1269399361,12cyclictest2188750wget07:35:252
1269499353,16cyclictest0-21swapper/310:35:423
12694993516,9cyclictest32759-21/usr/sbin/munin08:05:363
12694993514,4cyclictest7950irq/56-dwc_otg_09:11:353
12694993514,4cyclictest381ktimersoftd/308:40:273
12694993512,4cyclictest7950irq/56-dwc_otg_09:59:053
1269399354,14cyclictest29-21rcuc/210:25:552
1269399354,14cyclictest29-21rcuc/210:25:552
12693993521,5cyclictest8798-21ssh11:08:322
12693993514,10cyclictest7950irq/56-dwc_otg_07:45:272
12693993512,6cyclictest28596-21apt.systemd.dai09:15:482
1269399351,24cyclictest134-21systemd-journal08:43:522
1269399351,17cyclictest8050irq/56-dwc_otg_07:20:092
1269499347,15cyclictest0-21swapper/307:35:253
1269499347,12cyclictest28443-21users07:50:453
1269499343,10cyclictest8050irq/56-dwc_otg_07:45:333
1269499342,22cyclictest0-21swapper/310:05:093
1269399347,16cyclictest16161-21runrttasks09:45:252
1269399345,4cyclictest18398-21/usr/sbin/munin12:05:352
1269399345,4cyclictest18398-21/usr/sbin/munin12:05:352
1269399342,21cyclictest0-21swapper/209:06:552
12693993414,10cyclictest10676-21processes11:10:402
12693993413,7cyclictest8050irq/56-dwc_otg_10:55:342
1269399341,15cyclictest0-21swapper/209:20:202
12693993410,9cyclictest7950irq/56-dwc_otg_10:50:312
12693993410,13cyclictest0-21swapper/209:10:162
12694993316,14cyclictest381ktimersoftd/307:33:093
12694993314,10cyclictest8050irq/56-dwc_otg_10:44:413
12694993312,6cyclictest381ktimersoftd/307:17:193
1269399339,15cyclictest4642-21ssh11:48:422
1269399335,13cyclictest7950irq/56-dwc_otg_10:20:312
1269399333,10cyclictest29-21rcuc/212:20:532
1269399332,25cyclictest301ktimersoftd/207:22:502
1269399332,11cyclictest8050irq/56-dwc_otg_10:15:332
1269399331,28cyclictest12894-21ls08:40:092
12693993312,11cyclictest301ktimersoftd/209:00:232
12693993312,11cyclictest301ktimersoftd/209:00:232
1269399331,14cyclictest0-21swapper/212:10:242
12694993225,5cyclictest7330-21idleruntime-cro08:25:083
1269399326,14cyclictest0-21swapper/207:55:142
1269399325,17cyclictest0-21swapper/210:00:442
1269399322,11cyclictest301ktimersoftd/211:30:262
12693993218,8cyclictest301ktimersoftd/212:02:252
1269399321,26cyclictest24228-21ls09:55:382
1269499319,11cyclictest17707-21unixbench_multi08:50:403
1269499314,12cyclictest12721-21chrt08:39:583
1269399309,5cyclictest13548-21/usr/sbin/munin07:10:402
1269399301,13cyclictest7950irq/56-dwc_otg_08:51:532
1269399297,6cyclictest301ktimersoftd/208:25:082
1269399281,17cyclictest0-21swapper/207:30:242
121182283,15sleep20-21swapper/207:06:182
121642263,14sleep10-21swapper/107:06:531
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional