You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-19 - 07:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Thu Mar 19, 2026 00:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1319121780,6sleep22784499cyclictest21:17:142
3045821650,5sleep32784699cyclictest21:37:373
3045821650,5sleep32784699cyclictest21:37:373
27842999724,62cyclictest4182-21grep21:45:200
27842999424,50cyclictest20809-21/usr/sbin/munin20:20:230
27842999013,61cyclictest16112-21ssh23:55:060
277032900,7sleep20-21swapper/219:10:062
27842998820,60cyclictest0-21swapper/019:10:320
299172850,2sleep30-21swapper/322:53:143
27842998518,58cyclictest0-21swapper/021:20:250
27842998422,45cyclictest31869-21tr20:50:200
27842998418,53cyclictest29092-21switchtime23:30:320
27842998328,42cyclictest31447-21ssh00:12:340
27842998328,12cyclictest13960-21cron22:35:060
27842998328,12cyclictest13960-21cron22:35:060
27842998319,49cyclictest0-21swapper/021:56:190
27842998222,49cyclictest31293-21ssh22:16:030
27842998219,55cyclictest1404-21missed_timers00:15:260
2784299821,72cyclictest0-21swapper/023:40:340
27842998216,55cyclictest19745-21sshd00:37:320
27842998122,45cyclictest6039-21ssh23:03:190
27842998122,45cyclictest6039-21ssh23:03:190
2784299804,60cyclictest0-21swapper/022:05:170
27842998023,45cyclictest19223-21rm23:19:380
27842998020,46cyclictest15029-21netstat22:35:270
27842998018,54cyclictest0-21swapper/019:58:480
27842998017,59cyclictest21574-21diskmemload22:43:320
27842998017,49cyclictest30066-21ssh21:36:560
27842998017,49cyclictest30066-21ssh21:36:560
27842998017,49cyclictest0-21swapper/022:10:120
27842998014,56cyclictest0-21swapper/022:01:360
267122800,3sleep126715-21ntp_offset19:05:281
27842997922,41cyclictest19616-21rm23:59:010
265012790,2sleep10-21swapper/122:10:321
27842997821,47cyclictest0-21swapper/023:28:010
27842997820,41cyclictest10880-21acpi21:15:160
27842997819,49cyclictest8098-21aten_rbpower_cu19:45:190
27842997819,48cyclictest19027-21aten_rbpower_cu20:15:190
27842997819,45cyclictest13268-21seq23:11:180
27842997817,48cyclictest0-21swapper/021:34:090
27842997811,58cyclictest22860-21cut22:45:200
20052780,4sleep12001-21switchtime20:55:321
51102770,5sleep32784699cyclictest00:20:223
27844997746,25cyclictest27845-21sshd19:10:082
27842997726,41cyclictest32657-21/usr/sbin/munin23:35:250
27842997725,36cyclictest13578-21/usr/sbin/munin20:00:350
27842997725,36cyclictest13578-21/usr/sbin/munin20:00:350
27842997724,43cyclictest6611-21grep19:40:230
27842997724,43cyclictest6611-21grep19:40:230
27842997619,44cyclictest7977-21ssh22:27:050
27842997619,43cyclictest15290-21ssh00:31:570
27842997612,53cyclictest0-21swapper/000:25:160
27842997520,48cyclictest24012-21rm00:04:190
27842997520,40cyclictest0-21swapper/023:08:330
27844997433,11cyclictest15131-21/usr/sbin/munin22:35:292
27842997422,42cyclictest26994-21ssh00:07:010
27842997422,40cyclictest22167-21sh23:22:290
27842997421,50cyclictest3262-21ssh22:21:160
27842997418,41cyclictest0-21swapper/021:50:310
27842997416,48cyclictest2200-21sh21:42:300
27842997416,39cyclictest0-21swapper/019:21:370
27842997322,43cyclictest29157-21sh22:51:540
27842997319,38cyclictest9-21ksoftirqd/023:50:060
27842997318,40cyclictest0-21swapper/020:45:060
27842997316,41cyclictest25962-21runrttasks20:33:390
27842997316,41cyclictest25962-21runrttasks20:33:390
27844997241,13cyclictest25423-21apt-get22:10:182
27842997219,43cyclictest22514-21ssh21:28:380
27842997212,44cyclictest0-21swapper/000:20:540
27844997129,24cyclictest3444-21ssh00:18:112
27842997116,41cyclictest11084-21runrttasks19:50:420
27842997019,37cyclictest0-21swapper/019:18:360
27844996935,12cyclictest23939-21ssh23:25:172
2784299696,45cyclictest6424-21ssh21:10:070
27842996925,32cyclictest18744-21taskset20:15:040
27842996920,46cyclictest16688-21chrt20:06:560
27842996920,33cyclictest1455-21runrttasks22:57:460
27842996917,41cyclictest0-21swapper/020:55:450
27842996914,47cyclictest0-21swapper/020:25:490
27844996830,17cyclictest31309-21/usr/sbin/munin22:55:302
27842996819,37cyclictest0-21swapper/019:39:570
27844996736,12cyclictest31741-21munin-node20:50:162
2784399673,46cyclictest0-21swapper/119:10:081
27842996720,44cyclictest0-21swapper/021:14:520
27842996719,32cyclictest3992-21taskset19:32:160
27844996635,14cyclictest3145-21apt-get23:00:172
27844996635,14cyclictest3145-21apt-get23:00:172
27844996631,13cyclictest20192-21ssh22:41:062
2784399667,25cyclictest0-21swapper/119:40:191
2784399667,25cyclictest0-21swapper/119:40:191
2784699655,14cyclictest71150ssh00:15:173
27844996531,19cyclictest24562-21ssh22:46:202
27844996531,13cyclictest22476-21ssh00:01:302
27844996525,12cyclictest6347-21ssh00:20:542
27842996521,30cyclictest31317-21taskset20:47:530
27842996520,35cyclictest2345-21seq19:29:200
27844996332,13cyclictest32714-21ssh22:18:312
27844996328,11cyclictest25509-21rm21:31:262
27844996324,13cyclictest14006-21cat22:35:062
27844996324,13cyclictest14006-21cat22:35:062
27844996232,11cyclictest16599-21chrt21:20:452
27844996230,20cyclictest30324-21ssh21:37:232
27844996230,20cyclictest30324-21ssh21:37:232
27844996228,11cyclictest11003-21ssh00:26:512
27844996227,23cyclictest27179-21ssh00:07:222
27844996225,21cyclictest12715-21ssh23:50:232
27842996219,22cyclictest27820-21seq20:39:200
27844996135,12cyclictest29949-21apt-get20:45:162
27844996130,13cyclictest6024-21ssh23:42:002
27844996129,12cyclictest21329-21sed20:20:252
27842996120,24cyclictest4186-21runrttasks21:01:340
27844996037,11cyclictest22717-21ssh22:05:582
27844996033,15cyclictest6823-21ls19:40:252
27844996033,15cyclictest6823-21ls19:40:252
27844996031,12cyclictest10927-21ssh23:09:352
27844996028,13cyclictest19727-21ssh00:37:302
27844996024,15cyclictest29-21rcuc/221:55:182
2784399607,34cyclictest0-21swapper/122:25:201
27844995932,17cyclictest3262-21ssh22:21:172
27844995930,13cyclictest9085-21ssh21:12:322
27844995928,13cyclictest2735-21ssh23:38:332
27844995925,19cyclictest32287-21aten_rbpower_en21:40:192
27844995925,12cyclictest18765-21apt20:15:052
27844995920,15cyclictest10051-21rm21:51:082
27844995828,11cyclictest17888-21ssh23:17:122
27844995826,11cyclictest7096-21ssh21:48:252
27844995727,20cyclictest21214-21ssh21:26:212
27844995727,12cyclictest8463-21cat19:45:232
27844995726,23cyclictest27648-21taskset20:37:052
27844995725,11cyclictest29746-21sh23:31:112
27844995719,12cyclictest22146-21ssh23:22:252
27844995630,15cyclictest14997-21ssh23:14:302
27844995630,11cyclictest29141-21sh22:51:532
27844995629,12cyclictest18130-21rm23:56:182
88082550,2sleep30-21swapper/321:12:043
27844995528,12cyclictest18806-21ssh22:01:332
27844995526,21cyclictest25809-21seq20:31:402
27844995526,21cyclictest25809-21seq20:31:402
27844995526,18cyclictest15348-21taskset00:32:032
27844995526,12cyclictest10455-21ssh23:47:332
27844995524,13cyclictest31152-21taskset19:18:582
27844995523,14cyclictest19103-21cut20:15:202
27844995523,12cyclictest7847-21rm22:26:522
275642553,44sleep00-21swapper/019:10:010
27844995429,14cyclictest4205-21chrt19:35:022
27844995425,15cyclictest31462-21ssh00:12:362
2784699522,7cyclictest0-21swapper/320:00:193
2784699522,7cyclictest0-21swapper/320:00:193
27844995226,12cyclictest6217-21taskset21:09:252
27844995132,15cyclictest15392-21apt-get20:05:162
27844995123,11cyclictest2330-21taskset20:55:472
27844995021,12cyclictest29672-21chrt20:44:502
2784499497,7cyclictest7950irq/56-dwc_otg_21:00:162
27844994927,11cyclictest24042-21sshd20:28:312
2784399498,2cyclictest0-21swapper/120:40:161
27844994837,7cyclictest9886-21apt-get19:50:172
27844994819,10cyclictest0-21swapper/219:40:052
27844994732,13cyclictest15225-21latency_hist20:05:062
27844994732,13cyclictest15225-21latency_hist20:05:062
27844994716,13cyclictest0-21swapper/219:26:422
2784399478,25cyclictest0-21swapper/120:10:191
2784399478,24cyclictest0-21swapper/120:25:191
2784399473,5cyclictest7850irq/56-dwc_otg_22:20:181
27843994714,5cyclictest22423-21unixbench_multi22:05:341
27843994710,9cyclictest7850irq/56-dwc_otg_00:25:161
2784399468,23cyclictest0-21swapper/100:30:121
2784399467,24cyclictest0-21swapper/119:35:181
27844994523,11cyclictest306-21chrt19:21:422
27844994521,11cyclictest0-21swapper/219:59:142
2784399457,15cyclictest0-21swapper/120:00:191
2784399457,15cyclictest0-21swapper/120:00:191
2784399447,22cyclictest0-21swapper/119:50:191
2784399437,22cyclictest0-21swapper/120:30:111
2784399437,22cyclictest0-21swapper/120:30:111
2784399427,24cyclictest0-21swapper/123:20:191
211552420,2sleep10-21swapper/121:26:151
38362410,2sleep10-21swapper/123:40:101
27846994115,5cyclictest6296-21diskmemload00:13:523
27843994126,5cyclictest23-21ksoftirqd/123:00:181
27843994126,5cyclictest23-21ksoftirqd/123:00:181
27843994119,13cyclictest908-21runrttasks21:13:531
27843994118,16cyclictest23-21ksoftirqd/122:30:161
27843994118,16cyclictest23-21ksoftirqd/122:30:161
23942400,9sleep1221ktimersoftd/123:37:571
27846993919,5cyclictest2072950wget00:00:213
2784699391,5cyclictest7850irq/56-dwc_otg_20:40:203
185072390,2sleep10-21swapper/123:18:201
2784699382,9cyclictest0-21swapper/320:20:183
2784699381,3cyclictest7750irq/56-dwc_otg00:25:163
2784699381,23cyclictest29360-21latency_hist22:15:063
2784399383,25cyclictest13001-21rm23:10:481
2784699379,13cyclictest0-21swapper/319:10:083
2784699377,24cyclictest0-21swapper/321:45:203
27846993718,4cyclictest908-21runrttasks23:18:243
27846993716,8cyclictest12185-21grep21:15:333
2784699371,23cyclictest1876-21timerwakeupswit19:25:333
2784399377,6cyclictest7750irq/56-dwc_otg00:40:061
2784399377,19cyclictest24811-21ssh00:05:111
2784399375,23cyclictest724-21ssh00:15:181
2784399371,25cyclictest27340-21timerwakeupswit20:35:331
27846993623,5cyclictest1148-21aten_rbpower_cu20:55:203
27846993622,8cyclictest7950irq/56-dwc_otg_21:30:163
27846993611,5cyclictest7850irq/56-dwc_otg_19:20:063
2784399361,7cyclictest32541-21grep22:55:341
2784399361,23cyclictest21107-21grep20:20:221
261582360,5sleep32784699cyclictest00:05:323
2784699357,5cyclictest7750irq/56-dwc_otg22:00:063
2784699356,21cyclictest7950irq/56-dwc_otg_22:20:183
27846993511,7cyclictest6694-21ssh22:25:313
2784399352,7cyclictest1938750ssh22:40:291
27843993520,11cyclictest0-21swapper/121:15:221
2784399351,24cyclictest0-21swapper/122:45:201
2784399351,22cyclictest17395-21ls23:55:321
27843993510,4cyclictest0-21swapper/100:20:211
2784699347,5cyclictest7850irq/56-dwc_otg_22:30:333
2784699347,5cyclictest7850irq/56-dwc_otg_22:30:333
2784699347,14cyclictest0-21swapper/321:52:443
2784699342,12cyclictest7750irq/56-dwc_otg23:05:073
2784699342,12cyclictest7750irq/56-dwc_otg23:05:073
27846993416,5cyclictest21294-21sh00:40:073
2784399349,5cyclictest1229-21latency_hist22:20:061
2784399349,4cyclictest0-21swapper/121:40:191
2784399346,5cyclictest7750irq/56-dwc_otg21:50:271
2784399346,16cyclictest0-21swapper/119:30:181
2784399346,15cyclictest0-21swapper/121:45:111
2784399344,11cyclictest7750irq/56-dwc_otg20:15:231
2784399342,5cyclictest7750irq/56-dwc_otg23:45:331
27843993419,11cyclictest0-21swapper/122:35:171
2784399341,22cyclictest8045-21sh23:05:271
2784399341,22cyclictest24530-21latency23:25:261
27843993410,13cyclictest0-21swapper/121:35:281
27843993410,13cyclictest0-21swapper/121:35:281
2784699338,4cyclictest0-21swapper/323:10:313
2784699335,6cyclictest7850irq/56-dwc_otg_21:00:333
2784699332,6cyclictest7750irq/56-dwc_otg22:10:053
27846993316,5cyclictest18378-21vmstat20:10:343
27846993311,4cyclictest7750irq/56-dwc_otg22:35:283
2784399337,16cyclictest5392-21ntpq21:05:281
2784399335,6cyclictest29975-21fschecks_time19:15:231
27843993316,6cyclictest13624-21ssh23:50:351
27843993316,5cyclictest19369-21sh22:02:321
2784399331,22cyclictest15584-21ls21:20:241
2784399331,14cyclictest0-21swapper/123:30:191
212572331,29sleep30-21swapper/323:20:483
317472320,2sleep10-21swapper/120:50:161
2784699329,5cyclictest39-21ksoftirqd/323:50:323
2784699327,7cyclictest7850irq/56-dwc_otg_22:05:063
2784699325,14cyclictest7950irq/56-dwc_otg_00:00:063
2784699321,6cyclictest908-21runrttasks00:30:123
2784699321,6cyclictest0-21swapper/323:45:323
27846993212,4cyclictest0-21swapper/322:59:243
2784699321,17cyclictest22938-21cut20:25:233
27846993211,7cyclictest0-21swapper/322:46:063
2784399326,6cyclictest7950irq/56-dwc_otg_21:00:201
2784399325,13cyclictest7850irq/56-dwc_otg_22:50:331
27843993214,5cyclictest908-21runrttasks00:14:351
2784699318,10cyclictest0-21swapper/323:40:203
2784699316,6cyclictest7850irq/56-dwc_otg_21:05:283
2784699314,5cyclictest31119-21/usr/sbin/munin20:45:333
2784699314,14cyclictest7850irq/56-dwc_otg_20:35:273
2784699313,13cyclictest7950irq/56-dwc_otg_23:40:063
2784699312,6cyclictest7950irq/56-dwc_otg_22:20:063
27846993120,8cyclictest0-21swapper/319:30:223
27846993114,9cyclictest915-21lldpd19:43:003
27846993114,9cyclictest915-21lldpd19:43:003
27846993111,5cyclictest28349-21forks23:30:233
27846993111,4cyclictest19783-21cut21:25:223
2784699311,14cyclictest19771-21ssh22:40:333
27846993110,4cyclictest0-21swapper/323:30:063
2784399318,12cyclictest0-21swapper/121:33:021
27843993124,5cyclictest8310-21fschecks_count19:45:221
27843993110,10cyclictest12677-21ssh21:55:181
2784699307,14cyclictest0-21swapper/320:50:193
2784699303,13cyclictest7850irq/56-dwc_otg_19:50:233
2784699302,14cyclictest0-21swapper/320:05:093
2784699301,7cyclictest16354-21ssh21:20:343
2784699301,4cyclictest9640-21sh19:50:053
27846993010,6cyclictest5407-21timerandwakeup19:35:313
2784399305,5cyclictest7750irq/56-dwc_otg19:55:221
2784399302,18cyclictest0-21swapper/119:25:091
2784699295,12cyclictest7950irq/56-dwc_otg_20:35:063
2784699295,12cyclictest7950irq/56-dwc_otg_20:35:063
2784699292,15cyclictest8414-21ssh23:05:323
2784699291,14cyclictest7750irq/56-dwc_otg21:40:303
2784399292,8cyclictest0-21swapper/119:21:221
2784399291,20cyclictest15977-21missed_timers20:05:261
2784399291,19cyclictest20211-21apt-get00:00:091
2784699285,10cyclictest7950irq/56-dwc_otg_20:00:063
2784399283,5cyclictest7950irq/56-dwc_otg_20:45:291
27846992721,4cyclictest908-21runrttasks19:20:203
2784699271,4cyclictest7950irq/56-dwc_otg_20:15:303
260092274,16sleep30-21swapper/319:05:153
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional