You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-23 - 08:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Thu Jul 23, 2026 00:43:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2108321780,4sleep32700799cyclictest21:28:583
270049913621,61cyclictest0-21swapper/023:45:530
2992721170,9sleep32700799cyclictest20:45:453
27004991012,71cyclictest0-21swapper/021:10:440
27004991012,71cyclictest0-21swapper/021:10:440
302102990,2sleep20-21swapper/222:20:122
27004999237,23cyclictest23358-21ssh23:32:200
27004999237,23cyclictest23358-21ssh23:32:200
27004999018,52cyclictest29437-21cat19:15:440
27004998918,58cyclictest11881-21/usr/sbin/munin19:55:490
144342890,2sleep10-21swapper/122:42:051
2700499888,63cyclictest9783-21grep00:35:530
27004998818,59cyclictest8183-21rm23:53:540
2700499869,58cyclictest18579-21/usr/sbin/munin21:25:420
27005998535,29cyclictest8050irq/56-dwc_otg_23:32:181
27005998535,29cyclictest8050irq/56-dwc_otg_23:32:181
2700499857,61cyclictest0-21swapper/021:45:360
27004998413,52cyclictest4262-21munin-run00:30:220
27004998413,52cyclictest4262-21munin-run00:30:220
27004998322,48cyclictest0-21swapper/019:10:230
27004998243,8cyclictest30676-21sh22:20:360
27004998213,44cyclictest8594-21sh23:15:220
27004998213,44cyclictest8594-21sh23:15:220
27004998116,40cyclictest21315-21latency22:50:430
2700499806,66cyclictest0-21swapper/021:40:330
27004998017,46cyclictest25863-21grep00:15:490
27007997923,28cyclictest23346-21seq23:32:173
27007997923,28cyclictest23346-21seq23:32:173
2700499797,57cyclictest14367-21grep23:20:520
27004997917,47cyclictest11213-21taskset23:57:000
27004997917,47cyclictest11213-21taskset23:57:000
27004997912,50cyclictest17237-21cat00:05:220
2700599785,24cyclictest2224-21ssh23:45:511
27004997814,52cyclictest2680-21fschecks_time22:25:390
27004997814,51cyclictest16326-21latency_hist22:45:220
27004997813,44cyclictest22484-21aten_rbpower_en22:10:370
2700499778,53cyclictest0-21swapper/020:00:500
27004997716,47cyclictest25700-21awk20:35:380
27004997615,41cyclictest21277-21awk00:10:220
172812740,2sleep00-21swapper/022:04:110
27007997325,24cyclictest2224-21ssh23:45:523
2700599732,9cyclictest6545-21ssh21:10:441
2700599732,9cyclictest6545-21ssh21:10:441
27004997318,41cyclictest3327-21taskset19:32:100
27004997311,45cyclictest0-21swapper/020:30:230
27004997217,37cyclictest5012-21/usr/sbin/munin00:30:390
27004997214,46cyclictest20713-21ssh22:07:540
27004997214,25cyclictest9545-21fschecks_time19:50:390
27004997213,41cyclictest1224-21unixbench_multi19:25:510
27004997213,41cyclictest1224-21unixbench_multi19:25:510
27004997212,44cyclictest23280-21sh00:13:410
27004997211,58cyclictest27479-21taskset23:37:250
27004997118,43cyclictest9790-21grep23:15:490
27004997114,13cyclictest23978-21grep20:30:400
27007997019,23cyclictest6545-21ssh21:10:443
27007997019,23cyclictest6545-21ssh21:10:443
27004997019,40cyclictest27903-21ssh21:36:330
27004997016,39cyclictest26329-21ssh22:56:090
27004997010,44cyclictest30720-21ssh23:01:390
27004996919,35cyclictest11980-21ssh22:40:160
27004996916,36cyclictest7989-21ssh21:51:590
27006996819,29cyclictest8050irq/56-dwc_otg_21:10:442
27006996819,29cyclictest8050irq/56-dwc_otg_21:10:442
2700499689,30cyclictest15051-21/usr/sbin/munin21:20:490
27004996812,43cyclictest31224-21missed_timers19:20:430
2700699673,39cyclictest2265-21ls23:45:522
2700499676,47cyclictest11461-21runrttasks21:16:100
2700499676,35cyclictest5561-21diskmemload22:34:390
27004996719,27cyclictest29151-21fschecks_time00:20:400
27004996711,40cyclictest521-21munin-node20:40:540
2700499667,47cyclictest0-21swapper/023:27:530
27004996611,42cyclictest4994-21wc21:05:560
109082660,2sleep30-21swapper/300:37:313
2700499658,35cyclictest0-21swapper/019:37:070
2700499654,11cyclictest0-21swapper/020:45:330
27004996416,38cyclictest16478-21ssh22:45:230
2700499638,45cyclictest0-21swapper/023:09:560
2700499632,10cyclictest17318-21grep20:10:490
27004996310,40cyclictest0-21swapper/021:30:540
27004996310,40cyclictest0-21swapper/021:30:540
2700499627,34cyclictest0-21swapper/022:00:220
2700499622,32cyclictest9102-21latency_hist19:50:220
2700799613,8cyclictest8050irq/56-dwc_otg_21:40:343
2700499612,45cyclictest789-21latency20:55:430
293522600,3sleep10-21swapper/100:20:421
270102600,1sleep30-21swapper/300:17:423
2700499607,34cyclictest0-21swapper/022:15:430
2700699597,35cyclictest0-21swapper/223:32:172
2700699597,35cyclictest0-21swapper/223:32:172
2700699592,7cyclictest7850irq/56-dwc_otg23:55:342
2700699592,7cyclictest7850irq/56-dwc_otg23:55:342
2700499599,32cyclictest0-21swapper/020:54:310
2700499592,44cyclictest18551-21fschecks_time20:15:390
27004995912,35cyclictest29529-21aten_rbpower_po23:40:380
2700599581,19cyclictest2532-21ls19:30:421
2700499587,37cyclictest0-21swapper/020:20:350
2700599575,33cyclictest0-21swapper/120:30:351
101202570,3sleep30-21swapper/323:55:483
101202570,3sleep30-21swapper/323:55:483
2700799566,20cyclictest14087-21ssh21:20:363
27004995611,42cyclictest36822chrt21:04:390
141552560,2sleep30-21swapper/300:00:433
35832550,30sleep38050irq/56-dwc_otg_00:29:103
35832550,30sleep38050irq/56-dwc_otg_00:29:103
252612544,11sleep00-21swapper/019:05:360
66242520,1sleep30-21swapper/323:51:053
2700799521,6cyclictest8050irq/56-dwc_otg_23:15:353
2700699528,27cyclictest0-21swapper/219:35:362
2700799517,26cyclictest0-21swapper/321:36:163
27007995113,16cyclictest24503-21/usr/sbin/munin20:30:483
2700699512,8cyclictest7850irq/56-dwc_otg20:25:352
2700599517,29cyclictest0-21swapper/119:10:351
2700599515,22cyclictest7850irq/56-dwc_otg23:55:441
2700599515,22cyclictest7850irq/56-dwc_otg23:55:441
27005995128,7cyclictest7950irq/56-dwc_otg_20:15:351
27004995111,10cyclictest0-21swapper/019:42:270
2700599508,26cyclictest0-21swapper/121:00:281
2700599503,30cyclictest0-21swapper/119:20:331
2700499507,28cyclictest13-21rcuc/020:05:520
2700799498,27cyclictest0-21swapper/321:45:363
2700799497,7cyclictest0-21swapper/323:35:363
27007994913,8cyclictest25718-21diskstats20:35:403
2700699497,8cyclictest7950irq/56-dwc_otg_20:00:332
2700599489,25cyclictest7850irq/56-dwc_otg23:10:331
2700599489,25cyclictest7850irq/56-dwc_otg23:10:331
2700799471,28cyclictest0-21swapper/319:25:293
2700799471,28cyclictest0-21swapper/319:25:293
2700599471,32cyclictest7850irq/56-dwc_otg23:20:401
2700799468,26cyclictest16563-21ssh22:02:473
2700799468,24cyclictest1554-21runrttasks20:56:513
2700799461,4cyclictest8050irq/56-dwc_otg_20:20:313
2700799461,29cyclictest0-21swapper/320:51:343
27007994611,21cyclictest12107-21apt-config22:40:233
2700599465,6cyclictest0-21swapper/122:05:331
2700599464,19cyclictest21-21rcuc/120:35:531
2700599462,10cyclictest17876-21missed_timers23:25:451
2700799458,4cyclictest13105-21ssh23:20:373
2700799457,27cyclictest0-21swapper/322:40:353
2700699451,27cyclictest0-21swapper/219:15:362
2700599457,25cyclictest0-21swapper/100:15:291
2700599456,7cyclictest8050irq/56-dwc_otg_22:10:351
27005994521,9cyclictest7850irq/56-dwc_otg21:33:551
27005994521,9cyclictest7850irq/56-dwc_otg21:33:551
27005994511,7cyclictest0-21swapper/121:45:501
110022450,8sleep1221ktimersoftd/100:37:421
2700799447,22cyclictest7950irq/56-dwc_otg_19:50:353
27007994412,13cyclictest29786-21/usr/sbin/munin19:15:503
2700699442,7cyclictest0-21swapper/221:55:352
2700699441,9cyclictest5558-21sh21:10:222
2700699441,5cyclictest12210-21ssh22:40:232
2700599448,20cyclictest21-21rcuc/121:44:071
2700599445,27cyclictest4947-21tail19:35:531
2700799435,19cyclictest7850irq/56-dwc_otg19:20:473
27007994321,12cyclictest11531-21ssh21:16:193
2700699437,23cyclictest0-21swapper/219:40:362
2700599439,23cyclictest7850irq/56-dwc_otg23:00:221
2700599433,27cyclictest7850irq/56-dwc_otg19:40:401
2700599432,26cyclictest0-21swapper/120:45:431
2700599432,13cyclictest0-21swapper/121:50:341
2700799427,23cyclictest0-21swapper/322:10:373
2700799427,20cyclictest21861-21ssh00:11:033
2700799426,24cyclictest0-21swapper/321:50:353
27007994224,9cyclictest8050irq/56-dwc_otg_23:00:333
27007994213,5cyclictest381ktimersoftd/323:43:243
2700699427,21cyclictest24594-21ssh00:15:292
2700599426,25cyclictest7850irq/56-dwc_otg00:25:351
2700599426,25cyclictest7850irq/56-dwc_otg00:25:351
2700599425,25cyclictest221ktimersoftd/122:40:231
2700599423,8cyclictest0-21swapper/123:15:371
2700599423,19cyclictest8050irq/56-dwc_otg_22:20:441
2700599423,11cyclictest0-21swapper/122:15:331
27005994225,6cyclictest23-21ksoftirqd/121:25:341
27005994214,17cyclictest221ktimersoftd/119:26:181
27005994214,17cyclictest221ktimersoftd/119:26:181
2700599421,33cyclictest221ktimersoftd/119:15:511
2700799419,13cyclictest1406-21ssh22:24:173
2700799418,3cyclictest0-21swapper/322:45:473
2700799418,19cyclictest0-21swapper/321:00:353
2700699416,23cyclictest24859-21ssh22:55:352
2700599416,7cyclictest21-21rcuc/122:29:141
27005994119,8cyclictest7850irq/56-dwc_otg19:55:491
27005994112,9cyclictest7850irq/56-dwc_otg21:20:471
2700799409,13cyclictest5719-21ssh22:29:363
2700799403,25cyclictest17433-21ssh00:05:283
2700799402,4cyclictest0-21swapper/319:35:353
2700699403,17cyclictest0-21swapper/221:35:372
2700599407,14cyclictest221ktimersoftd/123:40:221
2700599405,22cyclictest7850irq/56-dwc_otg00:10:221
27005994015,12cyclictest7950irq/56-dwc_otg_20:25:401
2700599401,25cyclictest29196-21ls23:00:391
2700799398,17cyclictest0-21swapper/320:40:363
2700799398,16cyclictest0-21swapper/319:55:373
2700799398,11cyclictest27767-21/usr/sbin/munin19:10:463
27007993924,8cyclictest8050irq/56-dwc_otg_22:07:583
2700799392,11cyclictest7422-21ssh22:32:193
2700799390,15cyclictest21919-21grep22:50:513
2700699392,7cyclictest17518-21ssh00:05:362
27006993913,16cyclictest8786-21wget00:35:382
2700599398,27cyclictest0-21swapper/119:50:351
2700599396,23cyclictest31809-21ls20:50:511
2700599395,17cyclictest21-21rcuc/122:05:101
2700599392,9cyclictest10387-21fschecks_count21:55:411
27005993915,9cyclictest221ktimersoftd/122:50:461
27005993914,10cyclictest23-21ksoftirqd/100:03:021
2700599391,15cyclictest0-21swapper/120:00:331
2700799388,18cyclictest30000-21ssh22:19:483
2700799383,10cyclictest0-21swapper/323:10:333
2700799383,10cyclictest0-21swapper/323:10:333
27007993824,4cyclictest7950irq/56-dwc_otg_00:23:313
27007993822,12cyclictest7133-21sh00:33:053
27007993812,4cyclictest381ktimersoftd/323:05:333
2700699382,17cyclictest7950irq/56-dwc_otg_19:30:362
2700699382,11cyclictest13613-21ssh23:20:432
2700599385,17cyclictest17561-21sed20:10:511
2700599384,23cyclictest7850irq/56-dwc_otg20:25:071
2700599383,23cyclictest7850irq/56-dwc_otg21:15:531
27005993822,9cyclictest221ktimersoftd/120:41:221
137532380,2sleep20-21swapper/200:00:382
2700799378,16cyclictest0-21swapper/323:25:373
2700799375,11cyclictest8050irq/56-dwc_otg_21:33:413
2700799375,11cyclictest8050irq/56-dwc_otg_21:33:413
2700799374,14cyclictest7444-21/usr/sbin/munin19:45:353
2700799374,13cyclictest19818-21taskset20:20:153
2700699379,14cyclictest8050irq/56-dwc_otg_21:20:522
27006993722,4cyclictest29103-21acpi20:45:302
27006993717,5cyclictest332-21latency_hist23:05:222
27006993716,4cyclictest2798-21if_eth022:25:402
27006993713,5cyclictest6014-21grep23:10:512
27006993713,5cyclictest6014-21grep23:10:512
2700699371,26cyclictest6626-21rm22:30:522
2700599374,16cyclictest21-21rcuc/121:36:141
2700599372,25cyclictest7850irq/56-dwc_otg23:50:391
27005993710,5cyclictest23-21ksoftirqd/121:10:231
267342375,19sleep126772-21ls19:10:231
2700799368,17cyclictest0-21swapper/319:30:283
27007993618,8cyclictest5639-21apt-get19:40:343
27007993614,5cyclictest28685-21sh23:00:223
27007993613,13cyclictest23651-21latency_hist20:30:233
2700699369,13cyclictest5559-21grep00:30:472
2700699368,17cyclictest0-21swapper/219:20:352
2700699367,4cyclictest7850irq/56-dwc_otg21:25:362
2700699364,29cyclictest8050irq/56-dwc_otg_23:40:362
2700699363,20cyclictest8050irq/56-dwc_otg_23:25:412
2700699362,16cyclictest7950irq/56-dwc_otg_23:54:062
27006993618,14cyclictest8050irq/56-dwc_otg_23:35:342
27006993615,6cyclictest25404-21sh21:34:252
27006993615,6cyclictest25404-21sh21:34:252
2700599367,11cyclictest21-21rcuc/120:06:291
2700599364,16cyclictest21-21rcuc/123:40:371
2700599363,18cyclictest23-21ksoftirqd/119:50:221
2700599361,9cyclictest24192-21ssh00:15:221
27005993613,4cyclictest0-21swapper/123:05:411
2700799355,12cyclictest5607-21cut21:10:223
2700699353,20cyclictest32647-21aten_rbpower_po19:25:362
2700699353,20cyclictest32647-21aten_rbpower_po19:25:362
2700699352,8cyclictest3302-21unixbench_multi21:00:512
2700699352,15cyclictest7950irq/56-dwc_otg_21:15:262
2700699352,10cyclictest0-21swapper/220:55:332
27006993516,5cyclictest11250-21df19:55:382
2700599355,16cyclictest23-21ksoftirqd/122:34:541
2700599354,20cyclictest7850irq/56-dwc_otg20:55:421
27005993515,9cyclictest221ktimersoftd/122:45:481
27005993511,10cyclictest23-21ksoftirqd/100:30:371
2700799348,3cyclictest0-21swapper/322:00:223
2700799348,13cyclictest17949-21taskset20:14:383
2700699344,16cyclictest7950irq/56-dwc_otg_19:50:222
2700699342,18cyclictest0-21swapper/222:00:422
2700699341,8cyclictest28005-21ntp_states20:40:462
2700699337,6cyclictest7950irq/56-dwc_otg_19:10:502
2700699336,9cyclictest31-21ksoftirqd/222:10:492
2700699332,9cyclictest0-21swapper/220:36:332
27006993316,5cyclictest7889-21sh21:51:482
2700699331,22cyclictest2016-21latency_hist22:25:232
2700699331,22cyclictest14945-21cut20:05:412
2700699331,15cyclictest7850irq/56-dwc_otg20:10:432
2700799327,4cyclictest8050irq/56-dwc_otg_20:09:413
2700699328,6cyclictest0-21swapper/222:45:412
2700699325,12cyclictest7950irq/56-dwc_otg_23:15:482
2700699324,5cyclictest8050irq/56-dwc_otg_00:20:362
2700699324,14cyclictest7950irq/56-dwc_otg_22:05:412
2700699321,7cyclictest13570-21timerwakeupswit22:40:502
27006993210,4cyclictest0-21swapper/222:51:422
27006993210,4cyclictest0-21swapper/221:40:502
2700799313,11cyclictest0-21swapper/320:04:163
2700699315,5cyclictest7850irq/56-dwc_otg23:05:392
2700699313,7cyclictest8050irq/56-dwc_otg_21:45:522
2700699311,8cyclictest31-21ksoftirqd/219:50:422
2700699311,13cyclictest8050irq/56-dwc_otg_20:30:372
2700699307,13cyclictest8050irq/56-dwc_otg_20:20:372
2700699306,14cyclictest8050irq/56-dwc_otg_20:55:222
2700699304,6cyclictest7850irq/56-dwc_otg00:25:462
2700699304,6cyclictest7850irq/56-dwc_otg00:25:462
27006993010,11cyclictest0-21swapper/200:10:582
27006992816,8cyclictest18271-21aten_rbpower_cu20:15:352
266232288,13sleep30-21swapper/319:08:553
251712273,14sleep20-21swapper/219:05:302
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional