You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-05 - 16:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Sun Apr 05, 2026 12:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
360422230,2sleep10-21swapper/116:26:051
759521720,4sleep3446499cyclictest20:16:523
321232800,5sleep0445599cyclictest20:07:350
93042790,4sleep09291-21ssh19:01:050
261462780,5sleep3446499cyclictest20:40:153
165972780,6sleep0445599cyclictest18:31:010
165972780,6sleep0445599cyclictest18:31:010
92712770,4sleep1221ktimersoftd/120:58:131
260342750,2sleep30-21swapper/321:57:053
446399732,6cyclictest0-21swapper/219:05:392
104722730,3sleep3381ktimersoftd/319:41:053
446499675,20cyclictest15783-21ssh18:30:443
446499675,20cyclictest15783-21ssh18:30:443
245522670,3sleep34447-21cyclictest18:40:553
445999665,11cyclictest7950irq/56-dwc_otg_19:25:381
445599667,31cyclictest0-21swapper/016:50:520
4463996541,19cyclictest7950irq/56-dwc_otg_20:35:512
446399653,4cyclictest10409-21fschecks_time18:15:562
29162650,2sleep20-21swapper/217:55:532
446399641,8cyclictest0-21swapper/220:45:472
445599646,45cyclictest0-21swapper/019:10:500
445999632,5cyclictest30686-21ls17:41:041
445999631,13cyclictest14970-21dpkg21:05:391
445999631,13cyclictest14970-21dpkg21:05:391
446399624,13cyclictest7950irq/56-dwc_otg_17:40:562
445999622,6cyclictest0-21swapper/120:45:471
445999622,4cyclictest0-21swapper/118:46:051
445999621,6cyclictest24483-21grep21:55:541
445999617,11cyclictest7750irq/56-dwc_otg18:15:531
445999612,4cyclictest0-21swapper/118:55:391
4459996113,42cyclictest0-21swapper/120:10:521
445999611,13cyclictest0-21swapper/119:00:021
97442600,5sleep2446399cyclictest19:40:562
446399602,5cyclictest7850irq/56-dwc_otg_16:51:062
446399601,5cyclictest28276-21fschecks_time17:35:542
445999608,20cyclictest0-21swapper/120:50:511
445999607,9cyclictest7950irq/56-dwc_otg_20:17:091
445999606,9cyclictest7950irq/56-dwc_otg_18:40:521
445999601,13cyclictest15312-21/usr/sbin/munin17:00:571
445999601,13cyclictest15312-21/usr/sbin/munin17:00:571
445999596,4cyclictest0-21swapper/119:59:411
445999594,10cyclictest7950irq/56-dwc_otg_19:26:001
4459995913,5cyclictest18700-21modprobe17:10:381
445999591,10cyclictest17796-21ssh21:08:301
445599591,39cyclictest25577-21systemd-journal17:05:390
445599591,39cyclictest25577-21systemd-journal17:05:390
446399587,3cyclictest0-21swapper/218:34:002
446399587,3cyclictest0-21swapper/218:34:002
446399585,50cyclictest0-21swapper/220:00:502
446399584,6cyclictest0-21swapper/221:30:562
446399582,15cyclictest0-21swapper/220:21:022
446399581,10cyclictest10024-21aten_rbpower_vo16:45:522
445999582,5cyclictest10797-21ls20:20:591
445999581,9cyclictest16563-21latency_hist19:10:391
4459995815,40cyclictest19371-21aten_rbpower_cu21:10:531
445999581,4cyclictest7750irq/56-dwc_otg20:06:011
445999581,3cyclictest0-21swapper/116:51:031
445999581,14cyclictest6606-21tune2fs16:35:561
445999581,11cyclictest7950irq/56-dwc_otg_19:17:051
445999581,11cyclictest7950irq/56-dwc_otg_19:17:051
445599583,34cyclictest0-21swapper/016:40:500
4455995828,7cyclictest23932-21kworker/u8:320:36:070
445599582,27cyclictest9-21ksoftirqd/021:55:490
446399578,4cyclictest0-21swapper/221:21:252
446399576,4cyclictest7750irq/56-dwc_otg18:46:072
445999575,12cyclictest7950irq/56-dwc_otg_21:44:541
445999571,9cyclictest21491-21ssh20:34:211
445999571,9cyclictest18242-21rm19:11:181
4459995719,35cyclictest25912-21idleruntime-cro17:30:381
445999571,6cyclictest3979-21ssh19:34:121
445999571,5cyclictest3709-21df_inode21:30:541
4459995713,4cyclictest9786-21idleruntime-cro18:15:381
445999571,13cyclictest0-21swapper/118:25:511
445999571,12cyclictest2621-21latency_hist17:55:381
445999571,12cyclictest2621-21latency_hist17:55:381
445999571,12cyclictest0-21swapper/117:35:521
445999571,10cyclictest10702-21ssh21:39:181
445599579,24cyclictest0-21swapper/019:45:520
445599579,24cyclictest0-21swapper/019:45:520
446399567,12cyclictest15410-21diskmemload21:20:062
446399563,9cyclictest7950irq/56-dwc_otg_19:46:032
446399563,9cyclictest7950irq/56-dwc_otg_19:46:032
446399563,6cyclictest7950irq/56-dwc_otg_21:55:582
446399563,11cyclictest3295-21rm18:54:292
446399561,5cyclictest7850irq/56-dwc_otg_20:43:192
445999562,14cyclictest7950irq/56-dwc_otg_19:45:391
445999561,17cyclictest7850irq/56-dwc_otg_19:05:391
4455995627,3cyclictest0-21swapper/021:36:040
445599562,36cyclictest0-21swapper/019:50:500
446399555,5cyclictest7750irq/56-dwc_otg21:05:452
4463995521,30cyclictest8719-21ls21:36:032
445999553,10cyclictest7850irq/56-dwc_otg_20:40:261
445999551,8cyclictest0-21swapper/121:19:421
445999551,5cyclictest31622-21grep21:25:391
445999551,4cyclictest13436-21apt-get19:45:511
445999551,4cyclictest13436-21apt-get19:45:511
4455995517,22cyclictest8953-21idleruntime-cro19:40:380
40322553,12sleep00-21swapper/016:28:360
446399546,13cyclictest0-21swapper/219:11:432
445999542,9cyclictest0-21swapper/121:51:061
445999542,5cyclictest15379-21wc20:26:051
445999541,6cyclictest5319-21ssh19:35:521
445999541,3cyclictest7850irq/56-dwc_otg_16:45:521
445999541,2cyclictest0-21swapper/121:46:171
445999541,12cyclictest29199-21ssh20:43:161
4455995435,7cyclictest7850irq/56-dwc_otg_20:50:550
4455995434,9cyclictest0-21swapper/017:10:530
445599541,5cyclictest15114-21runrttasks21:05:450
4463995321,18cyclictest0-21swapper/219:05:552
445999534,11cyclictest7950irq/56-dwc_otg_20:05:391
445999531,9cyclictest7950irq/56-dwc_otg_16:30:521
445999531,9cyclictest0-21swapper/118:37:421
4459995313,4cyclictest6547-21cpu18:05:531
445599532,4cyclictest0-21swapper/020:20:490
445599532,28cyclictest0-21swapper/020:25:510
445599531,4cyclictest2553-21proc_pri20:11:020
4455995314,23cyclictest23633-21cpu21:15:520
4455995314,22cyclictest16816-21zcat21:46:040
445599531,32cyclictest0-21swapper/017:30:520
445599531,19cyclictest22026-21cut19:55:540
446399527,9cyclictest7950irq/56-dwc_otg_18:40:532
446399522,4cyclictest7850irq/56-dwc_otg_20:06:042
445999524,8cyclictest0-21swapper/118:02:431
445999524,7cyclictest7950irq/56-dwc_otg_16:41:031
445999522,11cyclictest15410-21diskmemload19:54:321
445999521,4cyclictest7950irq/56-dwc_otg_21:25:421
445999521,3cyclictest7950irq/56-dwc_otg_17:22:221
445599522,29cyclictest20910-21ps21:51:030
445599521,36cyclictest2006-21grep19:31:060
445599521,30cyclictest28101-21latency_hist18:45:380
445599521,27cyclictest21703-21ntp_kernel_err19:16:000
445599521,27cyclictest21703-21ntp_kernel_err19:16:000
446499512,6cyclictest0-21swapper/320:25:493
446499512,5cyclictest7750irq/56-dwc_otg20:45:493
446399511,25cyclictest22683-21ssh21:53:322
445999513,12cyclictest0-21swapper/118:34:531
445999513,12cyclictest0-21swapper/118:34:531
4455995129,13cyclictest505-21rs:main0
4455995115,19cyclictest32017-21ssh21:25:560
445599511,31cyclictest12582-21ssh19:05:420
446499508,29cyclictest21723-21ssh19:55:503
446499503,26cyclictest0-21swapper/319:30:503
446399504,3cyclictest7750irq/56-dwc_otg19:25:382
445999502,13cyclictest0-21swapper/116:56:311
4455995025,4cyclictest19238-21proc_pri20:31:030
445599502,25cyclictest10230-21switchtime19:41:030
445599502,18cyclictest23932-21kworker/u8:321:21:040
445599501,46cyclictest22606-21aten_rbpower_en17:20:520
445599501,35cyclictest26068-21grep19:21:040
445599501,28cyclictest27097-21grep20:01:070
446399492,10cyclictest0-21swapper/219:52:212
446399491,14cyclictest0-21swapper/221:42:502
445999495,5cyclictest0-21swapper/118:21:181
445599492,33cyclictest0-21swapper/016:45:570
4455994918,22cyclictest0-21swapper/018:50:410
446399483,9cyclictest7950irq/56-dwc_otg_19:28:332
446399481,6cyclictest1935-21df20:10:532
445999481,4cyclictest7950irq/56-dwc_otg_17:48:561
4455994826,13cyclictest0-21swapper/018:20:510
445599482,43cyclictest0-21swapper/017:16:070
4455994823,23cyclictest101ktimersoftd/020:15:510
4455994823,14cyclictest10110-21/usr/sbin/munin18:15:590
4455994822,6cyclictest7850irq/56-dwc_otg_18:30:390
445599482,24cyclictest4447-21cyclictest17:26:080
4455994818,18cyclictest0-21swapper/020:45:530
445599481,27cyclictest0-21swapper/021:00:390
445599481,25cyclictest10936-21taskset21:00:490
445599481,25cyclictest10936-21taskset21:00:490
446399471,10cyclictest23863-21ssh18:40:312
445999473,6cyclictest7950irq/56-dwc_otg_17:10:491
445999471,11cyclictest27513-21chrt17:33:141
4455994723,17cyclictest0-21swapper/017:45:510
445599472,27cyclictest0-21swapper/019:25:590
4455994721,6cyclictest13055-21unixbench_singl21:41:070
4455994720,8cyclictest19407-21aten_rbpower_en21:10:530
4455994719,18cyclictest0-21swapper/016:57:510
446499467,25cyclictest0-21swapper/319:35:503
446499467,25cyclictest0-21swapper/317:30:533
445599466,5cyclictest0-21swapper/017:05:490
445599462,21cyclictest0-21swapper/021:34:000
4455994620,5cyclictest4632-21wget18:00:520
4455994619,6cyclictest23956-21ssh18:40:380
446499457,24cyclictest0-21swapper/316:50:523
446499457,23cyclictest0-21swapper/317:15:523
446399456,10cyclictest7950irq/56-dwc_otg_21:46:032
445999451,3cyclictest7850irq/56-dwc_otg_17:57:181
4455994528,3cyclictest0-21swapper/018:55:580
4455994520,11cyclictest13-21rcuc/018:46:010
445599451,4cyclictest0-21swapper/017:50:540
445599451,4cyclictest0-21swapper/017:50:540
445599451,29cyclictest0-21swapper/016:31:060
446499447,22cyclictest0-21swapper/317:00:533
446499447,22cyclictest0-21swapper/317:00:533
4464994430,6cyclictest10039-21ssh20:20:453
4455994414,20cyclictest0-21swapper/017:45:160
445599441,23cyclictest9352-21grep18:11:060
446499437,23cyclictest0-21swapper/321:20:523
446499437,23cyclictest0-21swapper/318:15:433
446499437,22cyclictest0-21swapper/317:50:513
446499437,22cyclictest0-21swapper/317:50:513
446399437,12cyclictest0-21swapper/220:20:252
446399435,24cyclictest2674-21ssh20:50:512
4455994329,4cyclictest8012-21latency_hist16:40:390
445599432,22cyclictest0-21swapper/018:06:010
4455994318,16cyclictest3285-21cut17:55:590
37082437,23sleep20-21swapper/216:26:072
446499426,22cyclictest15116-21ssh21:05:453
446399427,22cyclictest0-21swapper/218:00:512
446399423,19cyclictest0-21swapper/219:15:492
446399423,19cyclictest0-21swapper/219:15:492
446399421,6cyclictest7950irq/56-dwc_otg_19:35:492
445999421,10cyclictest0-21swapper/117:19:421
446499417,24cyclictest0-21swapper/316:35:523
446499412,8cyclictest7750irq/56-dwc_otg18:50:503
4464994012,5cyclictest0-21swapper/321:02:523
4464994012,5cyclictest0-21swapper/321:02:523
446499391,21cyclictest7116-21ssh20:55:583
4464993811,14cyclictest26778-21/usr/sbin/munin20:01:043
446399383,6cyclictest7850irq/56-dwc_otg_19:00:392
4463993829,6cyclictest9900-21latency_hist18:15:392
4463993810,8cyclictest501-21unixbench-2d21:26:082
4464993718,5cyclictest908-21runrttasks21:31:073
445599371,19cyclictest0-21swapper/017:37:130
446499367,24cyclictest0-21swapper/317:45:513
446499362,4cyclictest7950irq/56-dwc_otg_18:20:493
4463993616,4cyclictest7439-21unixbench_singl16:36:062
446499357,7cyclictest27975-21grep20:41:053
446499356,7cyclictest7850irq/56-dwc_otg_18:00:583
446499355,13cyclictest7950irq/56-dwc_otg_21:26:073
446499351,5cyclictest19346-21ssh21:10:523
4464993515,9cyclictest2805-21seq17:55:513
4464993513,6cyclictest20027-21/usr/sbin/munin21:50:553
446499351,17cyclictest5571-21ssh18:56:063
446399354,11cyclictest7950irq/56-dwc_otg_18:25:392
446399353,12cyclictest7950irq/56-dwc_otg_21:00:382
4463993518,5cyclictest22900-21sh21:15:142
4463993513,4cyclictest18058-21sh20:30:392
4463993510,7cyclictest7850irq/56-dwc_otg_20:30:582
446499348,16cyclictest0-21swapper/319:15:523
446499348,16cyclictest0-21swapper/319:15:523
446499347,23cyclictest12607-21ssh19:05:443
446499346,18cyclictest7950irq/56-dwc_otg_19:50:533
446499344,6cyclictest7750irq/56-dwc_otg21:16:033
4464993411,4cyclictest0-21swapper/318:46:003
446399348,10cyclictest7950irq/56-dwc_otg_19:30:502
446399341,17cyclictest17446-21fschecks_time17:05:552
446499337,14cyclictest0-21swapper/318:25:443
4464993325,6cyclictest5121-21rm20:53:113
4464993310,16cyclictest0-21swapper/318:10:493
446399339,5cyclictest23655-21unixbench_multi17:21:062
446399338,6cyclictest7850irq/56-dwc_otg_17:10:492
446399338,15cyclictest16888-21munin-plugin-st17:05:392
446399338,15cyclictest16888-21munin-plugin-st17:05:392
446399337,5cyclictest7850irq/56-dwc_otg_16:31:032
446399336,14cyclictest0-21swapper/218:25:452
446399333,11cyclictest7950irq/56-dwc_otg_17:25:532
446399331,21cyclictest21918-21ls19:55:512
446499327,15cyclictest0-21swapper/316:40:523
446499323,19cyclictest0-21swapper/316:47:063
446499321,5cyclictest28912-21idleruntime-cro19:25:383
446499321,21cyclictest18197-21sed17:06:053
4464993211,5cyclictest18380-21tr20:30:493
446399325,12cyclictest7950irq/56-dwc_otg_17:46:042
446399322,19cyclictest0-21swapper/217:54:492
446399322,19cyclictest0-21swapper/217:54:492
446399321,23cyclictest14833-21ls21:05:392
446399321,23cyclictest14833-21ls21:05:392
446499318,4cyclictest7750irq/56-dwc_otg20:05:523
446499317,14cyclictest0-21swapper/319:45:543
446499317,14cyclictest0-21swapper/319:45:543
446499317,14cyclictest0-21swapper/319:25:523
446499314,6cyclictest7850irq/56-dwc_otg_20:10:523
446499314,5cyclictest7750irq/56-dwc_otg21:41:023
4464993123,6cyclictest20022-21ssh18:35:423
4464993121,7cyclictest8557-21sh19:00:543
446499311,19cyclictest29943-21ls17:40:533
446399317,5cyclictest7850irq/56-dwc_otg_16:40:522
446499309,4cyclictest0-21swapper/319:12:513
446499306,12cyclictest7950irq/56-dwc_otg_16:31:023
446499304,14cyclictest7950irq/56-dwc_otg_17:11:013
446399305,5cyclictest7950irq/56-dwc_otg_16:56:032
446399304,6cyclictest7850irq/56-dwc_otg_17:30:572
446499294,7cyclictest7850irq/56-dwc_otg_17:35:483
4464992925,2cyclictest0-21swapper/317:30:393
4464992919,8cyclictest16007-21aten_rbpower_po21:45:533
446499291,6cyclictest7750irq/56-dwc_otg16:55:553
446499291,13cyclictest7950irq/56-dwc_otg_17:21:033
4464992910,4cyclictest0-21swapper/321:36:573
446399295,10cyclictest7850irq/56-dwc_otg_17:15:542
41042292,19sleep30-21swapper/316:29:323
446399281,11cyclictest0-21swapper/218:05:572
446499271,9cyclictest7072-21ntpq18:06:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional