You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-11 - 12:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Tue Aug 11, 2026 00:43:40)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2635025640,4sleep10-21swapper/119:05:191
27953991556,104cyclictest1080-21nfs4_client23:00:260
27953991333,108cyclictest32409-21fschecks_time22:20:220
27953991272,58cyclictest7850irq/56-dwc_otg_22:50:320
27953991272,103cyclictest31430-21ssh21:40:050
27953991182,84cyclictest5846-21taskset23:45:310
27953991172,85cyclictest10429-21cat21:15:030
27953991132,85cyclictest17429-21/usr/sbin/munin22:40:320
279539911315,55cyclictest8282-21expr21:50:290
27955991122,12cyclictest25815-21unixbench_singl22:50:352
27953991061,86cyclictest32726-21unixbench_singl19:20:340
27953991053,90cyclictest0-21swapper/020:10:220
27953991041,84cyclictest3978-21ls21:00:300
27953991023,33cyclictest15571-21ssh22:00:150
27953991022,74cyclictest0-21swapper/021:15:050
27955991014,21cyclictest0-21swapper/200:15:272
27955991014,21cyclictest0-21swapper/200:15:272
27953991003,39cyclictest24441-21memory22:10:240
2795399981,79cyclictest5137-21grep21:05:220
2795399976,60cyclictest30701-21netstat00:15:260
2795399976,60cyclictest30701-21netstat00:15:260
2795399971,83cyclictest0-21swapper/000:00:330
2795399971,79cyclictest16159-21ntpq20:05:270
2795399966,58cyclictest28553-21ls22:15:230
2795599953,16cyclictest8593-21grep21:50:342
2795399953,85cyclictest11711-21diskstats21:55:180
2795399951,78cyclictest0-21swapper/019:15:230
2795399951,78cyclictest0-21swapper/019:15:230
27953999426,43cyclictest0-21swapper/023:40:200
2795399941,72cyclictest0-21swapper/020:36:370
2795599933,17cyclictest7950irq/56-dwc_otg_21:40:282
2795499935,60cyclictest7750irq/56-dwc_otg21:55:201
2795399923,80cyclictest0-21swapper/021:25:130
2795399923,80cyclictest0-21swapper/021:25:130
27954999151,25cyclictest7850irq/56-dwc_otg_22:20:191
2795399911,71cyclictest28988-21latency_hist23:35:020
2795399902,74cyclictest0-21swapper/019:36:090
2795399902,74cyclictest0-21swapper/019:36:090
2795399902,72cyclictest15635-21sh22:39:410
2795399901,42cyclictest15410-21seq20:05:030
2795499897,53cyclictest0-21swapper/121:25:151
2795499897,53cyclictest0-21swapper/121:25:151
2795399891,62cyclictest0-21swapper/019:57:110
2795399891,62cyclictest0-21swapper/019:57:110
2795399881,68cyclictest20522-21chrt20:18:160
2795399881,66cyclictest5091-21vmstat22:25:330
2795399877,52cyclictest0-21swapper/020:40:170
2795399872,67cyclictest1473-21grep20:55:190
2795599864,19cyclictest5809-21grep23:45:312
27953998639,27cyclictest0-21swapper/000:30:060
27954998544,18cyclictest7950irq/56-dwc_otg_22:50:331
2795399851,73cyclictest0-21swapper/023:12:190
2795399851,63cyclictest0-21swapper/023:29:200
2795399851,63cyclictest0-21swapper/023:29:200
2795399841,69cyclictest2173-21taskset19:25:450
2795399841,63cyclictest0-21swapper/023:55:140
2795399841,45cyclictest31626-21munin-run20:50:030
2795599833,10cyclictest9612-21apt-get00:30:082
2795599832,19cyclictest23786-21/usr/sbin/munin22:10:252
27955998315,13cyclictest1180-21runrttasks23:00:242
2795399831,67cyclictest0-21swapper/022:31:020
2795399822,63cyclictest0-21swapper/023:15:090
2795399822,62cyclictest0-21swapper/021:21:150
2795399821,69cyclictest0-21swapper/000:37:300
2795399821,66cyclictest0-21swapper/023:09:310
2795399821,66cyclictest0-21swapper/021:45:270
2795399821,66cyclictest0-21swapper/020:26:080
2795399821,66cyclictest0-21swapper/020:26:080
237792820,2sleep20-21swapper/200:06:552
2795599813,10cyclictest24387-21kworker/u8:321:15:032
2795399811,65cyclictest32216-21chrt20:50:220
2795399811,65cyclictest32216-21chrt20:50:220
2795399811,61cyclictest1180-21runrttasks19:41:240
2795399811,60cyclictest0-21swapper/000:09:010
27956998033,32cyclictest7750irq/56-dwc_otg22:15:303
2795599802,20cyclictest16786-21ssh22:00:332
2795599802,16cyclictest7950irq/56-dwc_otg_22:20:202
27954998033,20cyclictest7850irq/56-dwc_otg_21:50:291
2795399791,66cyclictest0-21swapper/000:23:050
2795399791,62cyclictest26603-21ssh21:34:270
43142780,6sleep32795699cyclictest21:01:483
310122780,3sleep10-21swapper/122:18:441
2795599786,10cyclictest17427-21ssh22:40:322
2795399781,50cyclictest27652-21ssh21:35:210
283642770,6sleep02795399cyclictest00:12:440
27954997741,9cyclictest7750irq/56-dwc_otg21:40:061
2795399772,63cyclictest0-21swapper/023:35:280
2795399771,72cyclictest0-21swapper/019:10:180
2795399771,66cyclictest0-21swapper/023:54:550
2795399771,64cyclictest0-21swapper/022:06:260
2795399761,62cyclictest0-21swapper/023:20:520
2795399761,62cyclictest0-21swapper/022:55:520
2795399761,60cyclictest0-21swapper/022:47:360
2795699752,30cyclictest37-21rcuc/322:20:213
27956997434,17cyclictest381ktimersoftd/321:50:343
2795599749,7cyclictest0-21swapper/221:55:182
27955997412,14cyclictest28852-21/usr/sbin/munin22:15:302
2795499746,41cyclictest7750irq/56-dwc_otg21:15:041
27954997410,43cyclictest7750irq/56-dwc_otg23:45:241
2795399741,57cyclictest0-21swapper/020:31:210
27954997313,32cyclictest17449-21unixbench_singl22:40:321
2795399731,68cyclictest0-21swapper/020:20:550
2795399731,61cyclictest0-21swapper/019:51:570
2795399731,58cyclictest0-21swapper/019:49:190
2795399731,53cyclictest0-21swapper/019:30:580
109772730,2sleep10-21swapper/123:51:201
27954997218,13cyclictest10374-21cron21:15:051
2795399721,54cyclictest0-21swapper/000:26:000
2795499713,46cyclictest7750irq/56-dwc_otg23:40:191
2795699708,30cyclictest381ktimersoftd/322:50:323
2795699703,43cyclictest0-21swapper/321:55:183
2795699693,25cyclictest0-21swapper/321:15:053
27956996918,14cyclictest0-21swapper/321:40:053
2795699688,35cyclictest0-21swapper/320:25:143
2795699688,35cyclictest0-21swapper/320:25:143
2795699666,42cyclictest7950irq/56-dwc_otg_23:45:233
27956996619,25cyclictest24454-21cat22:10:253
27954996611,28cyclictest16737-21ssh22:00:321
2795699645,30cyclictest0-21swapper/322:00:363
27954996416,23cyclictest30762-21ssh00:15:291
27954996416,23cyclictest30762-21ssh00:15:291
2795699624,37cyclictest0-21swapper/322:40:333
2795699623,42cyclictest975-21ssh23:00:243
27956996211,32cyclictest30738-21ssh00:15:283
27956996211,32cyclictest30738-21ssh00:15:283
2795499623,43cyclictest1052-21ssh23:00:261
27954996217,19cyclictest0-21swapper/122:10:241
27956995916,25cyclictest0-21swapper/323:40:193
2795699587,33cyclictest1180-21runrttasks20:05:093
2795699577,33cyclictest0-21swapper/300:30:063
27955995724,11cyclictest7850irq/56-dwc_otg_23:40:222
130432570,2sleep20-21swapper/223:55:032
27955995613,13cyclictest7850irq/56-dwc_otg_21:35:152
2795499569,33cyclictest7950irq/56-dwc_otg_00:30:071
2795499567,6cyclictest0-21swapper/120:40:161
27954995610,30cyclictest0-21swapper/121:05:181
2795699555,9cyclictest7850irq/56-dwc_otg_21:25:123
2795699555,9cyclictest7850irq/56-dwc_otg_21:25:123
2795699548,30cyclictest0-21swapper/319:55:163
2795699548,30cyclictest0-21swapper/319:55:163
2795599542,10cyclictest0-21swapper/219:20:172
2795599538,27cyclictest0-21swapper/220:45:172
27955995228,9cyclictest7950irq/56-dwc_otg_21:25:152
27955995228,9cyclictest7950irq/56-dwc_otg_21:25:152
2795599513,30cyclictest0-21swapper/221:15:062
2795499517,29cyclictest0-21swapper/120:00:171
2795699508,28cyclictest0-21swapper/321:05:183
2795499507,28cyclictest0-21swapper/120:05:081
2795599498,27cyclictest0-21swapper/220:20:172
2795499495,4cyclictest14202-21ssh00:35:091
240892490,3sleep324091-21modprobe22:50:023
275352473,14sleep00-21swapper/019:08:140
27956994610,17cyclictest7850irq/56-dwc_otg_21:15:063
2795499467,14cyclictest21-21rcuc/120:10:201
2795499463,29cyclictest0-21swapper/120:35:171
2795499462,24cyclictest6908-21latency19:40:241
36342450,2sleep30-21swapper/319:30:323
2795499458,27cyclictest1711-21sh00:20:201
294362440,3sleep20-21swapper/200:14:442
2795699448,21cyclictest0-21swapper/320:15:183
2795699446,23cyclictest0-21swapper/320:55:153
27956994418,8cyclictest39-21ksoftirqd/319:15:333
27956994418,8cyclictest39-21ksoftirqd/319:15:333
2795499446,26cyclictest3795-21ssh22:25:171
27954994412,18cyclictest221ktimersoftd/121:00:291
2795699437,23cyclictest0-21swapper/323:25:173
2795699437,23cyclictest0-21swapper/323:25:173
2795699436,5cyclictest7750irq/56-dwc_otg22:05:143
2795699434,24cyclictest14901-21ssh21:20:183
2795699428,29cyclictest0-21swapper/319:40:173
2795699428,18cyclictest0-21swapper/320:30:153
2795699423,4cyclictest7850irq/56-dwc_otg_00:10:213
2795599428,24cyclictest0-21swapper/219:50:172
2795499426,17cyclictest221ktimersoftd/120:50:031
27954994216,6cyclictest20883-21apt-key20:20:031
2795499421,28cyclictest0-21swapper/119:20:331
2795699411,10cyclictest0-21swapper/323:35:203
2795599419,7cyclictest7950irq/56-dwc_otg_19:55:202
2795599419,7cyclictest7950irq/56-dwc_otg_19:55:202
27955994125,3cyclictest7850irq/56-dwc_otg_00:35:262
2795599412,25cyclictest0-21swapper/220:35:332
2795599411,27cyclictest24922-21cut20:30:202
2795599411,10cyclictest7750irq/56-dwc_otg23:30:282
27955994110,8cyclictest4750irq/17-3f00b88019:45:132
2795499417,24cyclictest0-21swapper/123:15:191
2795699408,17cyclictest0-21swapper/320:35:173
2795599407,19cyclictest28481-21grep20:40:212
2795599401,33cyclictest17561-21cut20:10:212
2795499408,14cyclictest7950irq/56-dwc_otg_22:55:181
2795499403,12cyclictest0-21swapper/119:25:141
27954994023,4cyclictest7750irq/56-dwc_otg21:45:191
27954994022,6cyclictest20394-21basename22:45:191
2795599393,27cyclictest7950irq/56-dwc_otg_21:50:032
2795499396,23cyclictest7750irq/56-dwc_otg20:55:161
2795499396,17cyclictest7850irq/56-dwc_otg_19:15:271
2795499396,17cyclictest7850irq/56-dwc_otg_19:15:271
27954993923,10cyclictest7750irq/56-dwc_otg23:35:261
2795499392,23cyclictest7950irq/56-dwc_otg_22:05:141
2795499392,10cyclictest11584-21latency_hist19:55:031
2795699383,23cyclictest7950irq/56-dwc_otg_23:59:123
27956993817,5cyclictest39-21ksoftirqd/320:10:333
27956993813,6cyclictest0-21swapper/321:31:593
2795699381,29cyclictest9180-21ls19:45:313
2795599383,24cyclictest0-21swapper/219:39:292
2795599383,24cyclictest0-21swapper/219:39:292
27955993816,4cyclictest8154-21diskstats22:30:182
2795599381,5cyclictest7750irq/56-dwc_otg19:40:332
2795499387,18cyclictest221ktimersoftd/123:55:161
2795499383,8cyclictest9347-21sed19:45:321
2795499382,6cyclictest2589350ssh23:30:261
2795499382,11cyclictest12927-21grep19:55:331
2795499382,11cyclictest12927-21grep19:55:331
27954993818,8cyclictest7750irq/56-dwc_otg22:33:481
2795499381,6cyclictest18538-21cat00:00:311
2795699376,4cyclictest7850irq/56-dwc_otg_00:05:193
2795599378,20cyclictest948-21runrttasks19:25:132
27955993718,4cyclictest2035350wget22:45:192
2795599371,22cyclictest8485-21ls23:10:032
2795499377,16cyclictest21-21rcuc/121:34:321
2795499372,8cyclictest0-21swapper/119:10:151
27954993720,10cyclictest221ktimersoftd/121:39:451
2795699365,22cyclictest25165-21ssh23:30:173
2795699361,26cyclictest16691-21ls23:20:033
2795699361,15cyclictest0-21swapper/323:05:303
27955993616,4cyclictest21146-21sh00:04:382
27955993614,5cyclictest20900-21unixbench_multi22:05:312
27955993614,10cyclictest6402-21diskmemload00:20:192
2795599361,27cyclictest20021-21timerandwakeup20:15:332
2795599361,24cyclictest32568-21ls23:00:032
2795499368,5cyclictest23-21ksoftirqd/120:15:341
2795499366,6cyclictest23-21ksoftirqd/100:25:271
2795499363,6cyclictest3095-21fschecks_time19:30:231
2795499363,19cyclictest8367-21latency_hist23:10:031
2795699358,12cyclictest37-21rcuc/320:05:023
2795699355,4cyclictest7950irq/56-dwc_otg_21:40:023
27956993523,8cyclictest12298-21fschecks_count22:35:183
27956993513,5cyclictest29534-21latency_hist19:15:033
2795699351,24cyclictest17499-21latency23:20:243
27956993510,11cyclictest14282-21ssh00:35:153
2795599359,6cyclictest7950irq/56-dwc_otg_23:55:092
2795599356,6cyclictest7750irq/56-dwc_otg23:10:172
2795599353,19cyclictest17059-21latency_hist20:10:032
27955993514,13cyclictest30633-21ssh23:35:352
2795499355,19cyclictest7750irq/56-dwc_otg20:26:061
2795499355,19cyclictest7750irq/56-dwc_otg20:26:061
2795499355,17cyclictest221ktimersoftd/123:12:221
27954993515,4cyclictest15276-21ssh22:39:011
215712350,2sleep10-21swapper/123:25:231
215712350,2sleep10-21swapper/123:25:231
2795699346,16cyclictest7950irq/56-dwc_otg_20:45:323
2795699346,13cyclictest7850irq/56-dwc_otg_23:10:313
2795699344,9cyclictest7950irq/56-dwc_otg_23:53:173
2795699344,13cyclictest7850irq/56-dwc_otg_22:30:243
27956993415,7cyclictest28663-21ssh22:55:153
2795699341,21cyclictest7601-21ls22:30:033
27956993411,9cyclictest0-21swapper/320:40:263
2795599347,16cyclictest0-21swapper/219:30:182
2795599344,6cyclictest3324-21fschecks_time21:00:202
2795599342,15cyclictest7850irq/56-dwc_otg_20:25:172
2795599342,15cyclictest7850irq/56-dwc_otg_20:25:172
2795499346,15cyclictest21-21rcuc/121:23:551
2795499345,13cyclictest7950irq/56-dwc_otg_23:20:491
2795499343,16cyclictest221ktimersoftd/100:12:071
27954993421,4cyclictest7750irq/56-dwc_otg20:35:021
2795499341,3cyclictest0-21swapper/100:05:271
267022347,12sleep20-21swapper/219:05:252
307512330,2sleep20-21swapper/219:15:332
307512330,2sleep20-21swapper/219:15:332
2795699332,5cyclictest7750irq/56-dwc_otg19:50:153
2795699332,17cyclictest0-21swapper/321:46:523
2795699331,4cyclictest7950irq/56-dwc_otg_00:20:283
2795599334,6cyclictest0-21swapper/223:15:232
2795599333,5cyclictest0-21swapper/223:20:152
27955993314,6cyclictest29610-21cut19:15:032
2795499336,7cyclictest21-21rcuc/119:36:451
2795499336,7cyclictest21-21rcuc/119:36:451
2795499334,7cyclictest23-21ksoftirqd/120:50:181
2795499334,7cyclictest23-21ksoftirqd/120:50:181
323782320,2sleep20-21swapper/220:50:242
323782320,2sleep20-21swapper/220:50:242
2795699328,5cyclictest5723-21grep19:35:303
2795699328,5cyclictest5723-21grep19:35:303
2795699322,20cyclictest0-21swapper/319:26:193
2795699321,16cyclictest37-21rcuc/319:20:283
2795599326,4cyclictest7950irq/56-dwc_otg_00:25:252
2795599322,7cyclictest23721-21ls21:30:262
2795599322,6cyclictest0-21swapper/221:20:232
27955993219,7cyclictest7850irq/56-dwc_otg_22:26:182
27955993215,5cyclictest6402-21diskmemload23:25:182
27955993215,5cyclictest6402-21diskmemload23:25:182
276002322,23sleep30-21swapper/319:09:063
27956993113,4cyclictest18119-21ssh00:00:253
2795599314,12cyclictest7850irq/56-dwc_otg_22:39:322
2795599312,5cyclictest0-21swapper/221:05:222
2795599311,4cyclictest0-21swapper/220:00:242
2795599303,11cyclictest7750irq/56-dwc_otg20:55:272
2795699299,5cyclictest7750irq/56-dwc_otg00:25:163
2795699286,5cyclictest7850irq/56-dwc_otg_20:20:213
2795699271,17cyclictest0-21swapper/320:50:053
2795699271,17cyclictest0-21swapper/320:50:053
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional