You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-12 - 13:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Thu Mar 12, 2026 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2171421140,2sleep10-21swapper/122:17:591
18547991031,41cyclictest22570-21egrep21:40:170
18547991012,43cyclictest11587-21wget20:20:160
18547991012,43cyclictest11587-21wget20:20:160
1854899893,4cyclictest0-21swapper/100:35:151
1854799891,40cyclictest1336-21ps19:50:260
1854799871,38cyclictest26139-21grep23:40:300
1854799871,38cyclictest26139-21grep23:40:300
1854799831,39cyclictest32747-21ssh22:30:290
1854799812,36cyclictest13538-21sed23:25:290
1854799811,42cyclictest0-21swapper/021:15:170
1854799811,34cyclictest19498-21timerandwakeup19:10:270
1854799803,36cyclictest15115-21/usr/sbin/munin20:30:220
1854799802,38cyclictest0-21swapper/022:30:020
1854799801,36cyclictest0-21swapper/000:00:260
1854799792,41cyclictest0-21swapper/022:00:030
1854799782,29cyclictest0-21swapper/022:50:020
18547997717,29cyclictest16519-21netstat22:50:240
1854799757,28cyclictest0-21swapper/021:34:060
1854799756,32cyclictest0-21swapper/000:20:140
1854799752,34cyclictest0-21swapper/022:05:150
176472740,6sleep20-21swapper/219:05:282
139122730,1sleep00-21swapper/021:30:030
1854899722,8cyclictest16605-21ssh23:30:121
1854799722,38cyclictest0-21swapper/023:05:280
18547997127,5cyclictest7950irq/56-dwc_otg_00:30:020
1854799711,34cyclictest0-21swapper/022:44:270
171982710,1sleep2301ktimersoftd/222:12:202
1854799682,34cyclictest0-21swapper/021:12:310
1854799672,50cyclictest18497-21fschecks_count21:35:180
1854799672,50cyclictest18497-21fschecks_count21:35:180
1854799671,56cyclictest0-21swapper/021:20:180
52632660,2sleep00-21swapper/000:35:010
18550996613,8cyclictest7750irq/56-dwc_otg22:50:173
1854999668,12cyclictest7750irq/56-dwc_otg22:15:132
1854899667,29cyclictest0-21swapper/122:30:141
18547996612,8cyclictest0-21swapper/022:15:130
1855099652,3cyclictest0-21swapper/321:15:163
18547996554,4cyclictest101ktimersoftd/023:45:320
1854799653,47cyclictest21632-21grep20:45:290
1854799651,49cyclictest23645-21tune2fs22:20:200
133802650,3sleep337-21rcuc/323:25:283
13082640,2sleep20-21swapper/223:10:372
1854899634,10cyclictest7950irq/56-dwc_otg_19:55:131
1854799634,43cyclictest0-21swapper/023:55:020
1854799631,54cyclictest5926-21fschecks_count00:35:200
1855099625,11cyclictest7850irq/56-dwc_otg_23:40:293
1855099625,11cyclictest7850irq/56-dwc_otg_23:40:293
1855099622,14cyclictest0-21swapper/300:22:263
1855099621,5cyclictest7950irq/56-dwc_otg_20:45:033
18547996218,34cyclictest0-21swapper/021:55:020
1854799621,46cyclictest28677-21unixbench-2d19:35:270
1854799621,28cyclictest0-21swapper/023:23:320
1854799612,48cyclictest0-21swapper/023:34:260
1855099605,13cyclictest30274-21grep23:45:323
1855099601,14cyclictest0-21swapper/322:00:073
1854899609,34cyclictest0-21swapper/123:55:151
18547996031,14cyclictest6620-21wget22:00:160
1854799602,44cyclictest0-21swapper/021:05:200
1854799602,42cyclictest15165-21fschecks_count22:10:190
1855099597,4cyclictest0-21swapper/300:12:343
1855099592,12cyclictest0-21swapper/300:30:273
1854799591,48cyclictest0-21swapper/023:55:290
1854799591,43cyclictest0-21swapper/021:00:230
1854799591,42cyclictest20242-21fschecks_count22:55:170
1854799591,42cyclictest20242-21fschecks_count22:55:170
1855099589,3cyclictest0-21swapper/300:02:003
1855099582,4cyclictest4074-21apt-get23:15:153
18547995734,6cyclictest24324-21grep19:25:200
1854799571,42cyclictest3889-21latency_hist23:15:030
1854799563,36cyclictest4130-21grep23:15:170
18547995627,6cyclictest26817-21uniq21:45:190
1854799561,45cyclictest22645-21unixbench-2d00:15:310
1854799561,45cyclictest0-21swapper/019:55:210
1854799561,42cyclictest10022-21fschecks_time20:15:220
1855099553,18cyclictest22677-21fschecks_count21:40:203
1855099552,11cyclictest0-21swapper/323:12:313
1855099551,19cyclictest0-21swapper/322:30:523
1854799553,43cyclictest9-21ksoftirqd/000:10:180
1854799551,43cyclictest8668-21proc_pri20:10:270
1855099542,15cyclictest0-21swapper/323:06:543
1855099541,20cyclictest0-21swapper/322:39:163
18547995429,15cyclictest4199-21grep22:35:260
1854799541,41cyclictest21609-21grep23:35:270
1854799541,41cyclictest0-21swapper/020:01:560
1855099532,3cyclictest0-21swapper/323:01:163
1854799535,38cyclictest14174-21timerandwakeup00:05:270
1854799532,40cyclictest0-21swapper/020:50:270
1854799524,37cyclictest908-21runrttasks20:55:250
1854799522,43cyclictest0-21swapper/020:06:580
1854799521,40cyclictest0-21swapper/023:04:360
1854799521,39cyclictest0-21swapper/020:38:380
1854899515,5cyclictest0-21swapper/120:35:121
18547995134,4cyclictest7950irq/56-dwc_otg_19:15:140
18547995127,15cyclictest0-21swapper/020:25:130
1854799511,39cyclictest23376-21/usr/sbin/munin19:20:280
1855099505,17cyclictest15217-21ssh22:49:373
18547995034,7cyclictest7850irq/56-dwc_otg_19:40:180
1854799503,37cyclictest0-21swapper/019:30:270
1854799503,37cyclictest0-21swapper/019:30:270
1854799502,36cyclictest0-21swapper/020:40:280
179562503,12sleep00-21swapper/019:05:550
1854899492,4cyclictest0-21swapper/123:35:151
1855099482,21cyclictest0-21swapper/322:10:553
18548994712,7cyclictest0-21swapper/121:50:161
1855099466,25cyclictest0-21swapper/319:30:153
1855099466,25cyclictest0-21swapper/319:30:153
1854899468,25cyclictest0-21swapper/120:05:081
1854899467,23cyclictest0-21swapper/120:55:151
1855099457,25cyclictest0-21swapper/319:50:153
18549994522,5cyclictest29993-21ssh21:10:182
1854899457,24cyclictest0-21swapper/119:35:131
244472440,1sleep30-21swapper/322:20:303
1854899447,24cyclictest0-21swapper/120:20:161
1854899447,24cyclictest0-21swapper/120:20:161
1854899442,8cyclictest3738-21munin-plugin-st23:15:021
18548994421,4cyclictest7950irq/56-dwc_otg_19:10:211
18548994319,5cyclictest30408-21sh23:45:361
1854799432,31cyclictest0-21swapper/019:47:450
1854999421,13cyclictest25266-21latency_hist00:20:022
1854899427,25cyclictest0-21swapper/100:10:151
1854899426,12cyclictest7750irq/56-dwc_otg22:25:161
18548994220,13cyclictest0-21swapper/122:10:081
5242410,2sleep20-21swapper/222:30:442
1855099417,22cyclictest0-21swapper/320:20:163
1855099417,22cyclictest0-21swapper/320:20:163
1855099417,19cyclictest0-21swapper/319:40:143
1854899417,24cyclictest0-21swapper/121:00:141
1854899417,24cyclictest0-21swapper/120:15:171
1854899417,24cyclictest0-21swapper/100:00:141
1854899416,11cyclictest7850irq/56-dwc_otg_23:25:121
1854899412,10cyclictest0-21swapper/119:50:141
18548994116,5cyclictest26285-21sh21:45:021
18548994110,5cyclictest7950irq/56-dwc_otg_22:50:141
101572410,5sleep11854899cyclictest21:25:171
1855099401,25cyclictest0-21swapper/321:05:163
1855099401,24cyclictest0-21swapper/320:50:153
1855099401,24cyclictest0-21swapper/320:30:143
1854999405,6cyclictest0-21swapper/222:50:132
18549994017,5cyclictest7903-21/usr/sbin/munin20:10:252
1854899407,23cyclictest0-21swapper/123:40:151
1854899407,23cyclictest0-21swapper/123:40:151
1854899402,18cyclictest7950irq/56-dwc_otg_21:30:281
18548994018,5cyclictest26056-21sh00:20:191
18548994017,8cyclictest10500-21sh22:05:031
18548994013,14cyclictest7750irq/56-dwc_otg21:37:421
18548994013,14cyclictest7750irq/56-dwc_otg21:37:421
18550993919,11cyclictest2983250ssh21:49:083
1855099391,7cyclictest0-21swapper/321:20:193
1854999392,4cyclictest7750irq/56-dwc_otg00:35:132
18549993916,4cyclictest4191-21aten_rbpower_vo23:15:172
1854899397,21cyclictest0-21swapper/100:15:161
1854899392,6cyclictest29336-21diskmemload23:00:261
18548993920,8cyclictest7850irq/56-dwc_otg_21:17:551
1854899391,5cyclictest7888-21perl22:40:201
18548993912,13cyclictest7850irq/56-dwc_otg_23:17:561
47682380,2sleep34770-21latency_hist23:55:033
18550993825,4cyclictest16584-21/usr/sbin/munin23:30:193
18550993815,12cyclictest24322-21fschecks_time19:25:203
1855099381,23cyclictest2684-21cut21:55:213
1854999387,13cyclictest7850irq/56-dwc_otg_23:50:032
1854999383,3cyclictest7850irq/56-dwc_otg_21:28:352
18549993820,8cyclictest16822-21sh21:33:052
18549993818,14cyclictest29336-21diskmemload23:26:322
18549993815,11cyclictest9624-21ssh23:20:402
1854899387,23cyclictest0-21swapper/123:50:151
1854899382,11cyclictest16973-21idleruntime-cro00:10:021
18548993816,9cyclictest7750irq/56-dwc_otg21:59:531
18548993815,6cyclictest29499-21unixbench_singl23:05:301
18548993815,4cyclictest7950irq/56-dwc_otg_20:50:201
265242370,2sleep10-21swapper/119:30:251
265242370,2sleep10-21swapper/119:30:251
1855099371,25cyclictest24480-21cut20:55:193
1855099371,22cyclictest908-21runrttasks00:35:533
1855099371,20cyclictest20580-21aten_rbpower_cu20:45:153
18549993719,5cyclictest301ktimersoftd/222:05:152
1854899372,9cyclictest3711-21fschecks_time22:35:191
1854899372,5cyclictest0-21swapper/120:10:141
18548993721,6cyclictest0-21swapper/119:15:161
18548993711,12cyclictest7850irq/56-dwc_otg_22:47:241
18550993624,5cyclictest10201-21ssh23:21:433
1855099362,22cyclictest0-21swapper/323:55:223
1855099362,20cyclictest0-21swapper/321:00:143
1855099362,14cyclictest0-21swapper/320:06:033
1855099361,23cyclictest14355-21ls20:25:293
18549993616,6cyclictest18691-21if_eth021:35:192
18549993616,6cyclictest18691-21if_eth021:35:192
18549993616,4cyclictest22286-21sh21:40:042
18549993615,5cyclictest24190-21aten_rbpower_cu20:55:152
1854999361,16cyclictest26811-21ssh21:45:192
1854899368,15cyclictest0-21swapper/121:45:151
1854899362,21cyclictest0-21swapper/122:59:031
1854899362,21cyclictest0-21swapper/122:59:031
18548993615,10cyclictest7750irq/56-dwc_otg00:25:081
18548993611,13cyclictest221ktimersoftd/120:25:311
1855099358,12cyclictest8970-21grep20:10:303
18550993514,9cyclictest21986-21latency_hist19:20:023
1855099351,19cyclictest0-21swapper/319:14:403
1854999357,13cyclictest7950irq/56-dwc_otg_23:35:302
18549993518,4cyclictest7950irq/56-dwc_otg_21:50:172
18549993518,4cyclictest20881-21swap22:55:262
18549993518,4cyclictest20881-21swap22:55:262
18549993514,4cyclictest25362-21unixbench_singl23:00:302
1854899358,6cyclictest23-21ksoftirqd/122:21:501
18548993516,4cyclictest7750irq/56-dwc_otg22:05:231
18548993515,8cyclictest7750irq/56-dwc_otg00:33:311
1854899351,4cyclictest0-21swapper/121:20:131
18548993510,14cyclictest221ktimersoftd/123:20:431
1855099349,4cyclictest0-21swapper/300:25:333
18550993415,4cyclictest7750irq/56-dwc_otg22:55:273
18550993415,4cyclictest7750irq/56-dwc_otg22:55:273
18550993413,8cyclictest21436-21/usr/sbin/munin00:15:293
18550993412,7cyclictest0-21swapper/321:50:183
18550993411,5cyclictest10725-21cut22:05:153
1854999345,4cyclictest7950irq/56-dwc_otg_22:25:292
18549993413,4cyclictest24722-21sh22:20:452
18549993412,9cyclictest0-21swapper/200:30:212
1854999341,12cyclictest21116-21modprobe00:15:022
1854899348,10cyclictest7750irq/56-dwc_otg19:23:531
1854899344,6cyclictest7750irq/56-dwc_otg19:25:261
18548993412,7cyclictest7750irq/56-dwc_otg21:12:421
1855099336,21cyclictest23591-21rm23:38:163
1855099331,24cyclictest0-21swapper/322:41:393
1855099331,14cyclictest18541-21fschecks_count21:35:183
1855099331,14cyclictest18541-21fschecks_count21:35:183
1854999339,13cyclictest0-21swapper/221:21:122
1854999337,5cyclictest7850irq/56-dwc_otg_22:40:022
1854999331,7cyclictest7950irq/56-dwc_otg_19:35:032
1854999331,7cyclictest7950irq/56-dwc_otg_19:35:032
1854999331,7cyclictest2823-21users21:15:312
1854899331,6cyclictest221ktimersoftd/119:45:171
1854899331,2cyclictest0-21swapper/120:00:151
18550993218,7cyclictest381ktimersoftd/300:07:223
1855099321,6cyclictest0-21swapper/321:10:183
1855099321,5cyclictest0-21swapper/320:18:023
18550993212,12cyclictest0-21swapper/322:25:313
18550993211,6cyclictest14170-21aten_rbpower_po21:30:163
1854999327,8cyclictest503-21in:imuxsock23:51:092
1854999327,7cyclictest7950irq/56-dwc_otg_23:59:362
1854999327,12cyclictest7950irq/56-dwc_otg_00:03:302
1854999326,4cyclictest7850irq/56-dwc_otg_19:20:022
1854999321,7cyclictest7950irq/56-dwc_otg_20:50:032
18549993216,6cyclictest301ktimersoftd/219:10:142
18549993214,5cyclictest6179-21latency_hist22:00:032
18549993213,7cyclictest0-21swapper/223:05:382
18549993213,4cyclictest19900-21grep20:40:282
1854899322,7cyclictest16291-21unixbench_singl20:30:281
1854899321,25cyclictest18915-21ls20:40:151
1855099316,17cyclictest5771-21munin-run20:05:023
1855099314,4cyclictest28569-21wc19:35:263
1855099312,4cyclictest0-21swapper/319:55:153
1855099311,21cyclictest0-21swapper/322:15:243
1854999313,4cyclictest7850irq/56-dwc_otg_21:00:252
1854999311,6cyclictest7950irq/56-dwc_otg_20:00:022
1854999311,6cyclictest16699-21modprobe20:35:022
1854999311,5cyclictest7950irq/56-dwc_otg_21:05:302
1854999311,5cyclictest7903-21idleruntime22:40:202
18549993113,6cyclictest6782-21/usr/sbin/munin22:00:192
18549993112,7cyclictest10160-21interrupts20:15:232
1854999311,20cyclictest4692-21latency20:00:242
1854899314,5cyclictest7850irq/56-dwc_otg_19:40:251
1855099309,14cyclictest1462-1kworker/u9:221:25:193
1855099308,12cyclictest16920-21/usr/sbin/munin20:35:233
1854999307,3cyclictest0-21swapper/200:21:202
1854999306,14cyclictest7750irq/56-dwc_otg19:20:212
1854999305,5cyclictest7750irq/56-dwc_otg23:40:232
1854999305,5cyclictest7750irq/56-dwc_otg23:40:232
1854999304,12cyclictest7950irq/56-dwc_otg_22:45:252
1854999303,6cyclictest7850irq/56-dwc_otg_19:50:302
18549993024,4cyclictest16183-21apt-get23:30:082
1854999301,5cyclictest7950irq/56-dwc_otg_20:25:162
1854999301,5cyclictest7950irq/56-dwc_otg_20:10:022
1854999301,5cyclictest23486-21sed20:50:292
1854999301,3cyclictest0-21swapper/219:25:202
18548993017,10cyclictest7950irq/56-dwc_otg_21:10:031
1855099294,4cyclictest7750irq/56-dwc_otg19:22:273
1855099291,18cyclictest0-21swapper/319:45:283
1854999296,4cyclictest7750irq/56-dwc_otg00:05:122
1854999293,5cyclictest7750irq/56-dwc_otg00:25:182
1854999292,5cyclictest7950irq/56-dwc_otg_19:45:142
18549992919,6cyclictest29469-21aten_rbpower_en19:40:142
18548992923,4cyclictest20779-21fschecks_count20:45:181
181942293,16sleep10-21swapper/119:09:041
18549992820,6cyclictest11814-21cut20:20:192
18549992820,6cyclictest11814-21cut20:20:192
1854999281,6cyclictest7950irq/56-dwc_otg_19:35:202
1854999281,14cyclictest7850irq/56-dwc_otg_20:37:352
181452273,15sleep30-21swapper/319:08:253
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional