You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-27 - 07:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Mon Jul 27, 2026 00:43:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3739914113,46cyclictest26635-21timerwakeupswit22:45:520
3739913123,35cyclictest0-21swapper/000:00:490
375991272,103cyclictest13233-21/usr/sbin/munin21:50:472
373991273,41cyclictest0-21swapper/021:50:470
373991263,49cyclictest0-21swapper/022:35:480
373991263,40cyclictest0-21swapper/000:20:510
375991253,86cyclictest22929-21diskstats00:00:422
3739911936,7cyclictest0-21swapper/022:40:450
375991163,85cyclictest2681-21memory23:35:492
373991163,41cyclictest2022-21/usr/sbin/munin23:35:490
373991124,64cyclictest0-21swapper/023:25:550
375991089,71cyclictest7950irq/56-dwc_otg_22:45:542
373991067,24cyclictest22085-21ssh23:20:380
373991067,24cyclictest22085-21ssh23:20:380
375991056,77cyclictest7590-21runrttasks00:20:502
373991052,64cyclictest21555-21ntp_states22:00:490
373991052,64cyclictest21555-21ntp_states22:00:490
3739910470,11cyclictest7950irq/56-dwc_otg_23:45:430
373999960,16cyclictest518-21rs:main0
37599973,70cyclictest0-21swapper/222:36:002
37399964,68cyclictest0-21swapper/000:10:490
374999512,30cyclictest23375-21sed00:00:511
37599923,66cyclictest21813-21grep22:00:562
37599923,66cyclictest21813-21grep22:00:562
373999264,8cyclictest7850irq/56-dwc_otg21:45:370
373999215,52cyclictest0-21swapper/021:15:540
374999032,23cyclictest11274-21diskmemload21:22:251
373999012,40cyclictest6800-21unixbench_singl23:00:550
375998914,68cyclictest31592-21ssh00:10:492
374998817,27cyclictest21612-21pluginstate22:00:521
374998817,27cyclictest21612-21pluginstate22:00:521
37399883,69cyclictest0-21swapper/022:05:440
37599873,77cyclictest1922-21grep22:15:552
37699865,56cyclictest8050irq/56-dwc_otg_22:05:433
37599863,11cyclictest25333-21awk22:05:452
37599853,75cyclictest6790-21grep23:00:552
375998526,53cyclictest14466-21sh22:30:562
374998517,47cyclictest2612-21latency23:35:471
37599844,74cyclictest0-21swapper/223:45:432
37399842,62cyclictest0-21swapper/021:22:270
37599834,72cyclictest27472-21grep23:25:552
374998311,49cyclictest0-21swapper/100:20:501
37399832,46cyclictest0-21swapper/022:55:470
37399832,46cyclictest0-21swapper/022:55:470
37599823,72cyclictest2129-21grep22:55:482
37599823,72cyclictest2129-21grep22:55:482
37599823,55cyclictest0-21swapper/200:29:452
49542810,5sleep037399cyclictest20:50:460
37599813,72cyclictest22976-21ssh23:20:502
37599813,72cyclictest22976-21ssh23:20:502
48562800,2sleep00-21swapper/000:17:300
48562800,2sleep00-21swapper/000:17:300
37599804,69cyclictest16988-21timerandwakeup21:15:542
374998032,32cyclictest0-21swapper/122:35:301
37599783,9cyclictest8346-21apt-get21:45:382
374997820,25cyclictest667-21/usr/sbin/munin22:15:561
37599774,66cyclictest21990-21ssh22:40:462
37399773,57cyclictest0-21swapper/022:30:580
374997615,40cyclictest26807-21unixbench_singl22:45:551
374997613,29cyclictest14451-21ssh22:30:561
219712760,5sleep337699cyclictest21:21:533
374997517,39cyclictest0-21swapper/123:25:541
374997414,39cyclictest8365-21ssh21:45:391
139152740,2sleep30-21swapper/319:45:493
37499733,45cyclictest0-21swapper/123:45:431
374997117,26cyclictest0-21swapper/123:20:521
374997117,26cyclictest0-21swapper/123:20:521
305572710,2sleep10-21swapper/120:30:511
37599703,7cyclictest0-21swapper/221:22:262
37499702,45cyclictest25271-21ls22:05:441
37699692,45cyclictest2635-21ls23:35:473
376996913,25cyclictest23010-21ssh00:00:453
37699682,51cyclictest2317-21ssh22:55:553
37699682,51cyclictest2317-21ssh22:55:553
374996838,11cyclictest22105-21ssh22:40:481
37499677,42cyclictest0-21swapper/100:10:491
374996715,33cyclictest13185-21chrt21:50:461
376996610,30cyclictest8050irq/56-dwc_otg_23:45:443
37499668,26cyclictest2142-21ntpq22:55:481
37499668,26cyclictest2142-21ntpq22:55:481
168482660,3sleep10-21swapper/123:14:061
374996533,10cyclictest11274-21diskmemload23:00:511
374996527,21cyclictest15855-21ssh21:15:281
319582650,2sleep10-21swapper/119:05:511
37699633,24cyclictest1897-21sed22:15:553
37599637,45cyclictest0-21swapper/221:10:322
37399635,31cyclictest0-21swapper/021:05:370
140522630,4sleep229-21rcuc/223:10:402
376996211,24cyclictest37-21rcuc/322:00:553
376996211,24cyclictest37-21rcuc/322:00:553
37599623,46cyclictest8050irq/56-dwc_otg_19:11:212
37599621,4cyclictest9022-21ssh23:43:442
373996249,3cyclictest0-21swapper/023:15:370
33922620,2sleep20-21swapper/221:39:322
318602624,14sleep031957-21sed19:05:510
37699615,13cyclictest0-21swapper/323:25:553
373996127,20cyclictest0-21swapper/000:29:460
37699604,14cyclictest8050irq/56-dwc_otg_22:40:463
375996015,32cyclictest14470-21seq19:47:192
37599601,10cyclictest8050irq/56-dwc_otg_22:25:422
37399602,46cyclictest0-21swapper/019:20:370
37599592,12cyclictest0-21swapper/200:30:282
37599591,6cyclictest7262-21ssh22:22:542
37499592,35cyclictest0-21swapper/121:30:341
37499592,35cyclictest0-21swapper/121:30:341
374995910,11cyclictest7850irq/56-dwc_otg20:35:361
376995814,29cyclictest31656-21grep00:10:503
37599584,9cyclictest8050irq/56-dwc_otg_23:35:242
37399581,32cyclictest0-21swapper/022:50:350
376995712,25cyclictest8050irq/56-dwc_otg_22:30:563
376995712,19cyclictest8050irq/56-dwc_otg_21:15:553
37599576,8cyclictest7950irq/56-dwc_otg_23:50:362
37599576,8cyclictest7950irq/56-dwc_otg_23:50:362
37599572,51cyclictest23044-21ps20:10:502
373995714,22cyclictest7490-21latency_hist19:30:250
373995714,22cyclictest7490-21latency_hist19:30:250
376995614,36cyclictest6653-21ls23:00:533
37599566,5cyclictest8050irq/56-dwc_otg_00:20:242
37599566,5cyclictest8050irq/56-dwc_otg_00:20:242
37599561,3cyclictest0-21swapper/220:00:492
374995625,15cyclictest7850irq/56-dwc_otg22:50:451
373995621,15cyclictest7850irq/56-dwc_otg20:25:520
37699554,18cyclictest26701-21ssh22:45:543
37699552,6cyclictest0-21swapper/319:50:353
37699552,13cyclictest8050irq/56-dwc_otg_22:35:583
375995529,14cyclictest28310-21ls20:25:452
37599551,5cyclictest20917-21ssh23:57:522
37599551,4cyclictest8050irq/56-dwc_otg_21:33:502
37599551,4cyclictest8050irq/56-dwc_otg_21:33:502
37399552,30cyclictest30467-21wget23:30:390
373995519,21cyclictest0-21swapper/020:00:380
37399551,14cyclictest20584-21ls22:00:250
37699545,22cyclictest8050irq/56-dwc_otg_00:20:483
37599545,39cyclictest0-21swapper/221:42:132
37599541,5cyclictest19329-21ssh21:58:242
37599541,4cyclictest0-21swapper/200:08:482
37599541,43cyclictest0-21swapper/220:55:532
374995415,23cyclictest221ktimersoftd/100:25:511
373995435,5cyclictest0-21swapper/023:50:370
373995435,5cyclictest0-21swapper/023:50:370
373995411,26cyclictest0-21swapper/020:40:360
37599532,8cyclictest25849-21latency_hist20:20:252
37599532,8cyclictest25849-21latency_hist20:20:252
375995317,33cyclictest4555-21fschecks_count20:50:422
37599531,5cyclictest11957-21ssh23:07:432
37699524,33cyclictest0-21swapper/321:45:383
37599527,4cyclictest7950irq/56-dwc_otg_23:15:442
37599524,7cyclictest8050irq/56-dwc_otg_22:53:492
37599524,5cyclictest24220-21ssh21:25:362
37599522,9cyclictest8050irq/56-dwc_otg_19:51:022
37599522,8cyclictest6909-21unixbench-2d19:25:532
37599522,8cyclictest6909-21unixbench-2d19:25:532
37599521,9cyclictest29567-21aten_rbpower_en20:30:382
37599521,8cyclictest9030-21unixbench_singl21:00:502
37599521,7cyclictest10660-21unixbench_multi19:35:542
37599521,6cyclictest20841-21taskset20:05:452
37599521,3cyclictest0-21swapper/200:35:422
37599521,3cyclictest0-21swapper/200:35:422
376995113,13cyclictest7850irq/56-dwc_otg23:20:503
376995113,13cyclictest7850irq/56-dwc_otg23:20:503
37599512,7cyclictest5205-21unixbench_singl19:20:522
375995113,35cyclictest30158-21sh22:10:522
375995112,36cyclictest16773-21acpi19:55:342
37399511,29cyclictest15007-21wget19:50:380
37399511,26cyclictest22999-21cat00:40:250
37399511,26cyclictest22999-21cat00:40:250
37599508,26cyclictest0-21swapper/219:30:302
37599501,10cyclictest0-21swapper/219:16:432
374995017,18cyclictest7850irq/56-dwc_otg23:18:431
37399507,28cyclictest0-21swapper/019:15:390
37699493,12cyclictest13110-21missed_timers21:50:453
375994911,3cyclictest2407-21cat20:45:252
374994910,11cyclictest0-21swapper/119:35:371
373994924,5cyclictest15165-21aten_rbpower_vo00:30:400
37399492,32cyclictest0-21swapper/021:30:340
37399492,32cyclictest0-21swapper/021:30:340
37399492,28cyclictest0-21swapper/020:15:490
37399492,28cyclictest0-21swapper/020:15:490
373994920,6cyclictest7850irq/56-dwc_otg00:05:540
37399491,38cyclictest31011-21kworker/0:220:50:240
37399491,29cyclictest0-21swapper/023:10:470
37599481,9cyclictest0-21swapper/220:24:082
374994814,10cyclictest25256-21ls21:25:521
37399486,23cyclictest4801-21/usr/sbin/munin21:40:450
37399484,12cyclictest17276-21ls19:55:430
37399481,4cyclictest28105-21ssh21:30:250
373994813,6cyclictest0-21swapper/020:30:350
37399481,25cyclictest125-21systemd-journal21:11:220
59972470,5sleep137499cyclictest21:41:441
37699479,22cyclictest0-21swapper/300:29:463
37599471,6cyclictest0-21swapper/220:37:162
37499477,24cyclictest17710-21ls21:55:511
374994712,28cyclictest0-21swapper/120:10:391
37399476,39cyclictest0-21swapper/020:10:300
373994731,10cyclictest101ktimersoftd/023:07:160
37399471,35cyclictest0-21swapper/021:38:000
37699462,5cyclictest8050irq/56-dwc_otg_00:05:373
37599462,8cyclictest0-21swapper/221:05:472
37499468,17cyclictest13595-21ssh23:10:251
374994621,9cyclictest7850irq/56-dwc_otg23:54:581
374994621,9cyclictest7850irq/56-dwc_otg23:54:581
373994623,6cyclictest7950irq/56-dwc_otg_19:40:460
373994623,6cyclictest7950irq/56-dwc_otg_19:40:460
373994621,14cyclictest0-21swapper/020:05:510
37399461,24cyclictest7880-21/usr/sbin/munin21:00:470
373994612,14cyclictest13656-21/usr/sbin/munin19:45:440
37499457,23cyclictest0-21swapper/119:40:381
37499457,23cyclictest0-21swapper/119:40:381
374994522,9cyclictest7850irq/56-dwc_otg20:55:461
373994527,4cyclictest7950irq/56-dwc_otg_22:25:480
373994521,5cyclictest7850irq/56-dwc_otg23:55:470
373994521,13cyclictest4819-21/usr/sbin/munin22:20:510
37399451,5cyclictest30224-21sed22:10:530
373994513,22cyclictest0-21swapper/023:44:490
37599442,3cyclictest0-21swapper/220:48:122
374994418,14cyclictest7850irq/56-dwc_otg22:25:341
374994410,3cyclictest7850irq/56-dwc_otg19:30:321
37399447,25cyclictest0-21swapper/019:30:370
37399442,25cyclictest0-21swapper/019:14:290
373994417,18cyclictest0-21swapper/019:37:390
37699435,11cyclictest4232-21ssh21:40:373
37599431,9cyclictest0-21swapper/219:42:322
37599431,9cyclictest0-21swapper/219:42:322
374994322,8cyclictest7850irq/56-dwc_otg21:39:451
374994314,16cyclictest7850irq/56-dwc_otg23:41:101
374994310,15cyclictest6389-21ssh00:20:181
374994310,15cyclictest6389-21ssh00:20:181
37499429,22cyclictest0-21swapper/122:24:021
37499422,18cyclictest19169-21/usr/sbin/munin20:00:461
373994218,13cyclictest0-21swapper/020:57:030
9412410,2sleep3381ktimersoftd/323:34:053
37699413,29cyclictest8050irq/56-dwc_otg_20:35:363
37499417,25cyclictest29016-21ssh22:10:381
37499417,23cyclictest0-21swapper/120:50:381
374994123,5cyclictest8050irq/56-dwc_otg_20:05:411
374994114,16cyclictest7950irq/56-dwc_otg_19:16:431
37399411,21cyclictest0-21swapper/020:35:360
37499405,7cyclictest0-21swapper/119:55:371
373994013,16cyclictest0-21swapper/020:24:160
79022390,2sleep30-21swapper/321:00:363
37699392,10cyclictest19151-21ssh00:35:353
37699392,10cyclictest19151-21ssh00:35:353
376993910,20cyclictest39-21ksoftirqd/322:50:363
374993919,10cyclictest8050irq/56-dwc_otg_23:35:241
374993918,6cyclictest221ktimersoftd/121:10:261
376993820,5cyclictest9930-21sh22:25:513
37499382,20cyclictest18406-21ssh00:34:401
374993816,11cyclictest8050irq/56-dwc_otg_23:57:471
374993814,14cyclictest9033-21grep21:00:501
37499381,11cyclictest11828-21aten_rbpower_po21:10:391
37499378,13cyclictest7950irq/56-dwc_otg_00:40:241
37499378,13cyclictest7950irq/56-dwc_otg_00:40:241
376993616,7cyclictest29636-21head21:30:533
376993616,7cyclictest29636-21head21:30:533
37499368,10cyclictest28986-21taskset20:27:001
37499366,5cyclictest7850irq/56-dwc_otg19:15:241
37499363,3cyclictest7850irq/56-dwc_otg00:05:381
374993616,9cyclictest7850irq/56-dwc_otg20:45:251
374993616,4cyclictest7850irq/56-dwc_otg19:45:451
37699359,6cyclictest0-21swapper/322:24:283
37699357,10cyclictest8050irq/56-dwc_otg_21:55:503
37699354,15cyclictest18252-21cut23:15:413
37699351,22cyclictest0-21swapper/320:00:513
37499351,22cyclictest26760-21ls20:20:501
37499351,18cyclictest4199-21ls20:50:251
37699349,18cyclictest8050irq/56-dwc_otg_21:35:383
37699349,12cyclictest0-21swapper/300:34:183
37699348,4cyclictest7850irq/56-dwc_otg23:50:363
37699348,4cyclictest7850irq/56-dwc_otg23:50:363
376993416,10cyclictest16986-21sh23:14:213
37499347,24cyclictest0-21swapper/119:50:381
37499344,12cyclictest0-21swapper/119:20:361
37499342,11cyclictest25551-21taskset20:19:081
37499342,11cyclictest25551-21taskset20:19:081
91252330,2sleep3381ktimersoftd/323:43:553
37699337,15cyclictest0-21swapper/320:10:303
37699333,16cyclictest8050irq/56-dwc_otg_23:58:063
37699331,5cyclictest11272-21sh21:10:243
37699331,10cyclictest0-21swapper/300:18:323
37699331,10cyclictest0-21swapper/300:18:323
250332330,3sleep325034-21timerandwakeup20:15:523
250332330,3sleep325034-21timerandwakeup20:15:523
37699322,7cyclictest8050irq/56-dwc_otg_19:10:383
37699322,5cyclictest30222-21grep22:10:533
376993212,12cyclictest0-21swapper/321:13:223
37699317,3cyclictest0-21swapper/320:05:473
37699311,6cyclictest2456-21latency_hist20:45:253
37699311,20cyclictest9327-21latency_hist19:35:253
37499313,6cyclictest0-21swapper/119:25:481
37499313,6cyclictest0-21swapper/119:25:481
37699301,5cyclictest9627-21/usr/sbin/munin23:05:433
37699301,15cyclictest0-21swapper/321:27:073
113272300,1sleep30-21swapper/319:40:253
37699298,13cyclictest5907-21hostname19:25:383
37699298,13cyclictest5907-21hostname19:25:383
37699291,9cyclictest11486-21aten_rbpower_vo19:40:393
37699291,9cyclictest11486-21aten_rbpower_vo19:40:393
37699291,13cyclictest75050snmpd19:15:553
37699287,16cyclictest39-21ksoftirqd/320:25:353
37699286,6cyclictest8050irq/56-dwc_otg_20:30:413
37699285,4cyclictest8050irq/56-dwc_otg_20:20:423
37699284,6cyclictest7850irq/56-dwc_otg19:55:483
37699281,3cyclictest8050irq/56-dwc_otg_20:55:383
37699281,15cyclictest8050irq/56-dwc_otg_20:45:253
37699271,18cyclictest0-21swapper/320:52:093
324012273,17sleep20-21swapper/219:08:122
37699266,5cyclictest8050irq/56-dwc_otg_19:20:443
322292264,15sleep30-21swapper/319:06:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional