You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 04:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Wed Nov 26, 2025 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10689910411,80cyclictest28073-21ls00:30:080
133002990,1sleep20-21swapper/200:08:212
106899926,56cyclictest0-21swapper/023:05:240
1068998811,64cyclictest10492-21cut23:20:300
1068998811,64cyclictest10492-21cut23:20:300
319362870,6sleep3107199cyclictest23:05:293
182142870,2sleep00-21swapper/000:15:230
182142870,2sleep00-21swapper/000:15:230
106899859,6cyclictest5485-21idleruntime-cro22:30:080
1068998518,64cyclictest17435-21sh21:15:550
183692830,5sleep3107199cyclictest22:46:093
1068998218,50cyclictest0-21swapper/020:45:220
106899814,13cyclictest0-21swapper/020:20:220
106899793,60cyclictest23613-21/usr/sbin/munin22:10:340
106899782,7cyclictest0-21swapper/021:46:050
106899781,8cyclictest581-21sed00:35:420
151852770,2sleep1221ktimersoftd/121:58:361
106899777,46cyclictest24533-21sh23:40:090
106899771,7cyclictest12475-21fschecks_count21:10:260
106899771,6cyclictest29544-21ssh23:45:390
106899771,50cyclictest30262-21modprobe21:35:080
106899752,7cyclictest10455-21idleruntime-cro00:05:080
106899752,5cyclictest23543-21ntpq20:10:340
106899752,5cyclictest0-21swapper/022:05:370
106899751,6cyclictest28001-21apt23:45:080
106899741,7cyclictest8331-21/usr/sbin/munin19:30:350
154722730,5sleep1106999cyclictest22:43:051
106899739,48cyclictest21761-21ntp_states20:05:310
106899732,7cyclictest0-21swapper/022:38:320
106899732,7cyclictest0-21swapper/021:40:310
106899732,6cyclictest0-21swapper/022:55:350
106899731,7cyclictest14521-21runrttasks21:57:040
106899731,5cyclictest10999-21grep19:35:350
106899731,53cyclictest11868-21diskmemload23:56:500
106899731,50cyclictest10591-21sort21:05:250
106899731,50cyclictest10591-21sort21:05:250
106899729,51cyclictest0-21swapper/023:01:590
106899718,45cyclictest7850irq/56-dwc_otg_00:30:190
106899712,8cyclictest0-21swapper/022:02:310
106899712,7cyclictest0-21swapper/020:27:160
1068997124,32cyclictest0-21swapper/022:30:200
106899711,7cyclictest30920-21tune2fs21:35:300
106899711,7cyclictest30920-21tune2fs21:35:300
106899709,48cyclictest494-21ssh22:22:200
106899701,8cyclictest1841-21grep20:40:290
1068997010,44cyclictest11868-21diskmemload23:12:090
1069996913,27cyclictest0-21swapper/120:20:221
106899692,4cyclictest0-21swapper/000:05:270
106899691,6cyclictest32000-21aten_rbpower_po20:35:240
106899691,6cyclictest1120-21runrttasks21:21:230
1068996916,35cyclictest7950irq/56-dwc_otg_20:35:080
106899682,7cyclictest0-21swapper/019:22:120
106899679,37cyclictest10554-21ssh21:51:180
106899671,43cyclictest21859-21diskstats00:20:250
106899662,8cyclictest0-21swapper/023:34:300
106899661,54cyclictest11868-21diskmemload23:51:400
106899661,54cyclictest0-21swapper/019:43:440
106899661,53cyclictest26441-21rm21:29:390
1068996533,6cyclictest0-21swapper/022:50:190
1068996529,26cyclictest7641-21grep19:25:380
1068996513,38cyclictest0-21swapper/000:10:280
1068996511,41cyclictest25910-21chrt20:16:070
1068996511,41cyclictest25910-21chrt20:16:070
106899642,7cyclictest0-21swapper/022:46:500
106899641,8cyclictest8265-21apt21:00:080
106899641,8cyclictest0-21swapper/019:49:010
1068996411,36cyclictest0-21swapper/023:17:480
106999632,11cyclictest27871-21ssh23:00:221
106899636,43cyclictest30323-21ssh22:19:330
106899631,7cyclictest20022-21grep20:00:340
106899625,10cyclictest16615-21ssh23:29:010
106899624,5cyclictest7850irq/56-dwc_otg_21:05:080
106899621,8cyclictest19151-21sed20:00:090
106899611,46cyclictest15258-21ssh22:42:370
106899611,38cyclictest0-21swapper/020:50:360
16982600,2sleep20-21swapper/222:25:022
106899602,7cyclictest0-21swapper/019:11:320
103142600,2sleep30-21swapper/323:20:273
103142600,2sleep30-21swapper/323:20:273
106899592,8cyclictest0-21swapper/019:54:440
283382570,2sleep10-21swapper/123:45:211
1069995714,17cyclictest32120-21df20:35:251
266272560,3sleep231-21ksoftirqd/222:58:002
1069995625,21cyclictest22868-21/usr/sbin/munin20:10:371
1069995616,24cyclictest24417-21ssh00:24:361
106999555,32cyclictest0-21swapper/100:00:251
1069995512,21cyclictest16599-21ssh23:28:591
1069995511,26cyclictest25371-21ssh23:40:301
162682540,2sleep1106999cyclictest00:11:541
107199542,6cyclictest7950irq/56-dwc_otg_23:25:203
106999546,31cyclictest0-21swapper/121:25:181
1069995418,16cyclictest20040-21apt-get22:05:211
1069995417,19cyclictest6846-21cat22:30:351
106899542,7cyclictest4506-21munin-run19:20:080
106999536,9cyclictest0-21swapper/123:21:091
106999536,9cyclictest0-21swapper/123:21:091
1069995314,36cyclictest3065250ssh00:33:191
106999526,25cyclictest12816-21ls21:10:301
1069995211,29cyclictest0-21swapper/122:25:191
1069995211,23cyclictest1976-21cat20:40:311
1069995211,22cyclictest30374-21date21:35:091
107099511,21cyclictest7950irq/56-dwc_otg_23:30:192
106999517,26cyclictest22114-21ls20:05:351
1069995113,12cyclictest24356-21munin-run23:40:081
1069995013,10cyclictest18665-21ssh22:02:551
1069995012,24cyclictest11868-21diskmemload00:05:341
107099495,8cyclictest327350ssh23:55:202
107099491,8cyclictest7750irq/56-dwc_otg00:00:252
106999497,24cyclictest7950irq/56-dwc_otg_19:15:331
1069994915,15cyclictest9526-21users19:30:381
1069994911,24cyclictest28901-21ssh22:16:221
1069994910,19cyclictest11330-21vmstat19:35:391
1071994830,14cyclictest2699150ssh21:30:223
106999487,27cyclictest0-21swapper/122:55:221
1069994819,12cyclictest18770-21ssh21:18:531
1069994814,9cyclictest0-21swapper/123:15:201
1069994814,14cyclictest1118-21ssh23:51:161
1069994812,19cyclictest26223-21ssh22:13:321
1069994810,24cyclictest17054-21chrt19:54:301
1069994810,13cyclictest2491-21ntp_offset21:40:321
323132470,2sleep20-21swapper/219:05:302
1069994713,20cyclictest31734-21ssh23:05:261
1069994712,20cyclictest6614-21rm00:00:001
5982463,35sleep00-21swapper/019:07:310
106999467,24cyclictest0-21swapper/123:30:221
106999467,24cyclictest0-21swapper/120:25:221
106999467,20cyclictest22725-21ssh21:24:301
106999467,13cyclictest14410-21netstat19:45:301
1069994611,20cyclictest4777-21ssh23:12:101
1069994611,18cyclictest10864-21ssh21:51:581
106999459,20cyclictest7850irq/56-dwc_otg_00:27:281
106999457,28cyclictest0-21swapper/119:55:141
1069994528,14cyclictest31793-21apt-get00:35:211
1069994516,20cyclictest19635-21ssh00:16:181
1069994516,20cyclictest19635-21ssh00:16:181
1069994515,16cyclictest18868-21ssh22:47:141
1069994513,23cyclictest31153-21latency21:35:321
1069994513,23cyclictest31153-21latency21:35:321
107099445,5cyclictest7950irq/56-dwc_otg_22:10:272
107099441,7cyclictest7950irq/56-dwc_otg_23:35:192
1070994417,7cyclictest446-21cron21:50:082
106999447,23cyclictest0-21swapper/119:25:241
1069994412,12cyclictest32766-21sh22:21:541
1069994411,14cyclictest6355-21taskset20:54:021
106999437,25cyclictest10792-21latency21:05:271
106999437,25cyclictest10792-21latency21:05:271
106999437,23cyclictest31060-21ls20:30:361
106999437,12cyclictest10958-21rm22:36:091
183402420,1sleep30-21swapper/323:30:373
106999427,14cyclictest21007-21latency_hist20:05:091
107099411,25cyclictest0-21swapper/222:45:212
106999417,19cyclictest7035-21ssh21:46:291
1069994114,12cyclictest22714-21ssh22:52:301
49952400,2sleep30-21swapper/321:45:023
1070994026,4cyclictest7950irq/56-dwc_otg_22:25:252
107099401,25cyclictest13361-21sh21:10:372
106999404,26cyclictest7950irq/56-dwc_otg_20:45:211
307812391,4sleep330791-21ssh23:48:143
107199391,28cyclictest9814-21ls22:35:273
107199391,25cyclictest21973-21cut00:20:273
1070993916,4cyclictest446-21cron21:10:082
1070993916,4cyclictest446-21cron21:10:082
107099391,25cyclictest7950irq/56-dwc_otg_00:20:092
107099391,25cyclictest7950irq/56-dwc_otg_00:20:092
107099391,25cyclictest0-21swapper/220:00:232
107099391,24cyclictest0-21swapper/219:55:142
107099391,23cyclictest0-21swapper/220:40:222
107099391,18cyclictest7850irq/56-dwc_otg_19:30:262
106999398,13cyclictest7984-21chrt20:56:531
324412386,16sleep10-21swapper/119:05:341
107199387,6cyclictest7850irq/56-dwc_otg_23:40:083
107199381,10cyclictest7850irq/56-dwc_otg_22:05:213
107099387,4cyclictest0-21swapper/223:02:582
107099382,24cyclictest0-21swapper/221:55:372
1070993819,9cyclictest1486550ssh22:41:452
107099381,29cyclictest7950irq/56-dwc_otg_19:15:272
107099381,24cyclictest0-21swapper/223:50:212
107099381,22cyclictest9238-21ssh21:50:272
1069993814,11cyclictest1120-21runrttasks20:16:201
1069993814,11cyclictest1120-21runrttasks20:16:201
280912370,4sleep339-21ksoftirqd/300:30:083
107099373,22cyclictest27502-21sh22:15:242
1070993719,4cyclictest17759-21latency_hist00:15:082
107099371,8cyclictest7850irq/56-dwc_otg_19:50:202
107099371,21cyclictest3901-21ls23:10:362
106999377,12cyclictest9825-21chrt21:02:201
1071993614,5cyclictest27396-21aten_rbpower_vo22:15:233
107099361,26cyclictest25732-21timerwakeupswit23:40:362
106999366,17cyclictest0-21swapper/119:43:441
106999362,26cyclictest7950irq/56-dwc_otg_19:20:191
277372350,3sleep231-21ksoftirqd/221:30:352
107199357,22cyclictest0-21swapper/322:50:193
107199356,10cyclictest0-21swapper/323:54:583
107099353,17cyclictest28318-21ssh23:45:192
1070993523,8cyclictest301ktimersoftd/200:25:202
107099352,16cyclictest0-21swapper/220:33:022
107099351,23cyclictest9836-21ls23:20:092
107099351,22cyclictest0-21swapper/223:26:182
106999357,13cyclictest7850irq/56-dwc_otg_19:10:341
107199347,6cyclictest7950irq/56-dwc_otg_00:15:263
107199347,6cyclictest7950irq/56-dwc_otg_00:15:263
107199344,7cyclictest7950irq/56-dwc_otg_19:25:083
1071993416,4cyclictest2782-21/usr/sbin/munin23:10:353
1071993411,4cyclictest0-21swapper/323:15:363
1071993410,12cyclictest16138-21cpu21:15:233
107099342,4cyclictest0-21swapper/220:45:342
1070993420,5cyclictest301ktimersoftd/222:09:532
1070993419,4cyclictest7850irq/56-dwc_otg_23:20:292
1070993419,4cyclictest7850irq/56-dwc_otg_23:20:292
1070993413,5cyclictest8913-21if_eth021:00:282
107099341,22cyclictest24548-21ls20:15:092
107099341,21cyclictest0-21swapper/219:10:382
107099341,20cyclictest30706-21wget21:35:272
107099341,20cyclictest30706-21wget21:35:272
85512330,2sleep30-21swapper/300:01:093
7302333,21sleep30-21swapper/319:09:163
107199337,4cyclictest7850irq/56-dwc_otg_23:40:213
107199333,15cyclictest7750irq/56-dwc_otg21:50:303
1071993323,8cyclictest32033-21ssh00:35:263
107199332,16cyclictest8262-21cron21:00:083
1070993321,3cyclictest301ktimersoftd/220:20:212
1070993311,13cyclictest7950irq/56-dwc_otg_23:05:342
107199329,12cyclictest0-21swapper/320:35:353
107199328,21cyclictest0-21swapper/321:20:403
107199326,12cyclictest7850irq/56-dwc_otg_23:00:353
107199325,14cyclictest0-21swapper/319:55:083
1071993214,16cyclictest2554350ssh21:27:393
1071993210,10cyclictest7950irq/56-dwc_otg_20:30:263
107099329,13cyclictest7950irq/56-dwc_otg_21:20:262
107099327,7cyclictest7950irq/56-dwc_otg_19:25:372
107099326,11cyclictest7950irq/56-dwc_otg_00:35:232
107099325,5cyclictest7950irq/56-dwc_otg_00:20:242
107099323,5cyclictest21353-21fschecks_time20:05:242
107099322,18cyclictest16784-21ssh21:15:322
107099321,16cyclictest28538-21fschecks_count20:25:262
107099321,15cyclictest0-21swapper/221:29:462
107099321,13cyclictest0-21swapper/222:50:262
1070993210,4cyclictest0-21swapper/222:30:272
107199318,4cyclictest0-21swapper/321:12:263
107199317,22cyclictest0-21swapper/322:00:223
107199317,21cyclictest0-21swapper/321:45:233
107199317,21cyclictest0-21swapper/320:50:233
107199316,12cyclictest7750irq/56-dwc_otg19:10:273
107199315,4cyclictest0-21swapper/300:10:353
1071993122,6cyclictest23964-21ssh22:10:283
107099314,12cyclictest7950irq/56-dwc_otg_20:55:262
107099311,18cyclictest28498-21aten_rbpower_vo00:30:222
107099311,17cyclictest13745-21/usr/sbin/munin19:45:282
107199309,4cyclictest0-21swapper/322:56:143
107199304,4cyclictest7850irq/56-dwc_otg_00:05:263
107199304,12cyclictest7850irq/56-dwc_otg_22:41:503
107199301,6cyclictest4148-21grep20:45:363
107199301,6cyclictest32264-21ssh21:36:323
107199301,6cyclictest32264-21ssh21:36:323
1071993011,11cyclictest0-21swapper/322:24:443
1071993010,4cyclictest7850irq/56-dwc_otg_19:40:083
107099309,4cyclictest0-21swapper/222:02:272
107099301,5cyclictest0-21swapper/221:43:252
1070993011,5cyclictest11036-1kworker/u9:122:36:322
107099301,11cyclictest0-21swapper/220:35:342
60702290,3sleep229-21rcuc/220:50:382
107199297,20cyclictest0-21swapper/320:05:213
107199295,5cyclictest7850irq/56-dwc_otg_00:30:263
107199294,13cyclictest7950irq/56-dwc_otg_23:55:233
107199292,18cyclictest0-21swapper/321:59:293
107199291,7cyclictest7950irq/56-dwc_otg_21:00:273
107199291,6cyclictest7850irq/56-dwc_otg_21:10:093
107199291,6cyclictest7850irq/56-dwc_otg_21:10:093
107199291,3cyclictest0-21swapper/322:30:503
107099296,5cyclictest31-21ksoftirqd/220:15:342
107099296,5cyclictest31-21ksoftirqd/220:15:342
107099293,13cyclictest6307-21ls19:25:082
107099291,7cyclictest13124-21users19:40:362
107099291,21cyclictest0-21swapper/219:35:342
107199287,16cyclictest0-21swapper/319:55:133
107199284,11cyclictest4750irq/17-3f00b88019:20:083
1071992820,6cyclictest14020-21fschecks_count19:45:253
1071992815,7cyclictest7950irq/56-dwc_otg_22:28:003
1071992810,9cyclictest27892-21taskset20:23:103
107199276,4cyclictest7750irq/56-dwc_otg20:15:313
107199276,4cyclictest7750irq/56-dwc_otg20:15:313
107199272,11cyclictest7950irq/56-dwc_otg_20:26:223
107199271,4cyclictest0-21swapper/319:25:253
107199264,4cyclictest7750irq/56-dwc_otg20:10:263
107199261,3cyclictest0-21swapper/320:00:243
107199252,5cyclictest7950irq/56-dwc_otg_19:40:093
107199251,5cyclictest7850irq/56-dwc_otg_20:40:303
107199251,4cyclictest7950irq/56-dwc_otg_19:30:293
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional