You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-15 - 05:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Wed Apr 15, 2026 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1059722720,3sleep30-21swapper/304:26:003
2988021640,3sleep31220499cyclictest07:55:593
2399021000,9sleep00-21swapper/007:48:480
205492990,2sleep20-21swapper/208:23:322
205492990,2sleep20-21swapper/208:23:322
12201999465,20cyclictest7950irq/56-dwc_otg_06:55:570
1220199941,70cyclictest27142-21sh07:13:450
1220199931,74cyclictest31179-21aten_rbpower_en09:15:580
1220199931,74cyclictest19730-21ssh07:05:220
12201999018,53cyclictest14141-21grep07:36:120
1220199892,73cyclictest0-21swapper/007:24:580
1220199891,78cyclictest14123-21aten_rbpower_po04:35:590
1220199891,76cyclictest0-21swapper/008:16:100
1220199891,69cyclictest30500-21aten_rbpower_po08:36:000
1220199881,75cyclictest21836-21ssh09:04:330
1220199881,69cyclictest28849-21fschecks_count05:16:000
1220199872,69cyclictest0-21swapper/006:15:540
1220199871,71cyclictest10443-21ssh09:30:090
1220199871,70cyclictest3373-21idleruntime-cro06:45:440
1220199871,70cyclictest14844-21fschecks_count08:56:000
1220199871,67cyclictest19873-21cut06:20:580
1220199871,60cyclictest0-21swapper/007:55:480
218822860,5sleep21220399cyclictest09:04:392
1220199861,72cyclictest4380-21ls06:46:050
1220199861,69cyclictest0-21swapper/007:44:280
1220199861,65cyclictest3705-21rm08:41:430
1220199852,66cyclictest0-21swapper/005:45:590
1220199851,72cyclictest4550-21ssh09:21:580
1220199851,68cyclictest0-21swapper/008:11:130
1220199841,72cyclictest0-21swapper/009:36:100
1220199841,69cyclictest9380-21wc05:51:020
1220199841,67cyclictest4990-21chrt05:40:140
1220199841,67cyclictest4990-21chrt05:40:140
1220199831,68cyclictest15690-21taskset06:07:210
1220199831,68cyclictest0-21swapper/009:13:290
1220199831,67cyclictest25577-21systemd-journal07:31:590
1220199831,59cyclictest0-21swapper/005:10:430
1220199831,59cyclictest0-21swapper/005:10:430
1220199821,78cyclictest0-21swapper/009:57:370
1220199821,67cyclictest25575-21ssh08:30:090
1220199821,65cyclictest27666-21ssh07:52:570
1220199811,65cyclictest8237-21ssh08:47:260
1220199801,67cyclictest29740-21ssh06:37:200
1220199791,68cyclictest0-21swapper/009:52:390
1220199791,65cyclictest31612-21ssh07:19:200
1220199791,61cyclictest25447-21ssh06:32:120
1220199791,54cyclictest2552150ssh09:47:230
1220199781,66cyclictest4434-21ssh08:04:260
1220199781,63cyclictest0-21swapper/007:27:450
1220199781,58cyclictest28533-21ssh08:32:590
1220199775,48cyclictest6986-21latency_hist05:45:440
1220199771,65cyclictest8625-21ntp_states06:51:090
1220199771,65cyclictest27091-21cut05:11:000
1220199771,65cyclictest22379-21ssh09:44:130
1220199771,65cyclictest13623-21ssh09:33:220
1220199771,59cyclictest24794-21ssh09:07:240
1220199761,63cyclictest0-21swapper/008:21:040
1220199761,63cyclictest0-21swapper/008:21:040
1220199751,70cyclictest0-21swapper/008:51:050
1220199751,70cyclictest0-21swapper/008:51:050
1220199751,63cyclictest8955-21ssh08:10:070
1220199751,60cyclictest0-21swapper/004:31:000
1220199751,60cyclictest0-21swapper/004:31:000
1220199742,64cyclictest0-21swapper/004:45:570
1220199731,66cyclictest0-21swapper/006:15:340
1220199731,62cyclictest0-21swapper/005:56:260
1220199731,59cyclictest0-21swapper/005:31:350
12201997311,19cyclictest0-21swapper/006:00:560
1220199711,61cyclictest0-21swapper/005:02:420
212872700,5sleep31220499cyclictest07:45:573
1220199701,58cyclictest17058-21runrttasks04:41:240
1220199701,52cyclictest0-21swapper/005:26:430
1220199691,61cyclictest0-21swapper/007:07:300
1220199691,58cyclictest20931-21runrttasks04:54:450
1220199681,37cyclictest30843-21irqstats05:21:030
10562680,2sleep10-21swapper/109:18:111
1220199672,49cyclictest0-21swapper/006:26:470
1220199671,46cyclictest0-21swapper/004:57:240
99592650,2sleep10-21swapper/108:50:331
140872650,5sleep11220299cyclictest04:35:581
12202995832,19cyclictest7950irq/56-dwc_otg_08:35:571
39452560,1sleep30-21swapper/308:03:323
1220499567,30cyclictest0-21swapper/304:50:573
1220299542,3cyclictest7750irq/56-dwc_otg06:35:581
163342530,2sleep30-21swapper/308:18:263
1220299532,24cyclictest0-21swapper/105:35:561
1220299532,24cyclictest0-21swapper/105:35:561
31482520,2sleep10-21swapper/108:02:051
1220399525,18cyclictest7750irq/56-dwc_otg04:35:582
1220299513,5cyclictest7750irq/56-dwc_otg09:25:581
12202995012,4cyclictest7950irq/56-dwc_otg_06:50:581
1220299501,13cyclictest10670-21timerandwakeup08:11:101
1220499492,7cyclictest7850irq/56-dwc_otg_06:35:593
12203994911,22cyclictest0-21swapper/205:35:492
12203994911,22cyclictest0-21swapper/205:35:492
1220299494,8cyclictest0-21swapper/104:55:551
116292485,36sleep00-21swapper/004:26:550
94792470,2sleep2301ktimersoftd/208:10:442
1220399471,2cyclictest0-21swapper/205:00:552
1220299476,5cyclictest7950irq/56-dwc_otg_07:30:531
1220299473,8cyclictest0-21swapper/108:15:551
12202994622,5cyclictest21026-21idleruntime-cro04:55:431
1220399457,25cyclictest0-21swapper/209:20:582
1220299457,25cyclictest0-21swapper/105:45:581
1220399447,22cyclictest0-21swapper/205:40:572
1220299443,11cyclictest0-21swapper/106:00:561
12202994421,4cyclictest314-21/usr/sbin/munin06:41:081
1220299441,4cyclictest7750irq/56-dwc_otg07:00:551
301262431,37sleep10-21swapper/105:20:431
1220499438,16cyclictest7950irq/56-dwc_otg_09:21:013
1220499437,24cyclictest0-21swapper/306:05:593
12204994313,17cyclictest39-21ksoftirqd/305:10:433
12204994313,17cyclictest39-21ksoftirqd/305:10:433
1220399438,25cyclictest0-21swapper/204:45:572
294082420,4sleep2301ktimersoftd/206:36:432
1220499428,25cyclictest0-21swapper/308:30:573
1220499427,25cyclictest0-21swapper/307:10:573
1220399427,22cyclictest0-21swapper/204:40:572
1220299427,20cyclictest0-21swapper/108:30:571
1220299426,27cyclictest0-21swapper/108:50:571
1220299426,27cyclictest0-21swapper/108:50:571
12202994222,5cyclictest2363050wget09:45:581
12202994220,6cyclictest23-21ksoftirqd/107:55:431
12202994216,5cyclictest21-21rcuc/107:35:591
12202994213,11cyclictest21-21rcuc/105:23:571
1220499417,26cyclictest0-21swapper/306:00:583
1220499417,24cyclictest0-21swapper/305:45:583
1220499417,22cyclictest0-21swapper/305:00:503
1220499417,20cyclictest0-21swapper/305:35:483
1220499417,20cyclictest0-21swapper/305:35:483
1220499411,14cyclictest0-21swapper/308:11:103
12203994120,4cyclictest27446-21cron06:35:432
12203994120,12cyclictest0-21swapper/209:17:122
1220299416,23cyclictest23-21ksoftirqd/108:20:571
1220299416,23cyclictest23-21ksoftirqd/108:20:571
1220299413,7cyclictest21010-21kworker/u8:209:50:561
1220499405,26cyclictest39-21ksoftirqd/308:20:573
1220499405,26cyclictest39-21ksoftirqd/308:20:573
12204994015,12cyclictest39-21ksoftirqd/307:10:433
1220399407,21cyclictest0-21swapper/207:00:562
1220399405,25cyclictest0-21swapper/207:20:492
1220299407,21cyclictest0-21swapper/105:05:571
1220299407,21cyclictest0-21swapper/105:05:571
1220299407,14cyclictest7850irq/56-dwc_otg_06:45:551
12202994020,5cyclictest554150wget08:05:581
12202994018,14cyclictest221ktimersoftd/104:30:541
12202994018,14cyclictest221ktimersoftd/104:30:541
1220299401,5cyclictest7950irq/56-dwc_otg_07:40:571
12202994013,5cyclictest23-21ksoftirqd/107:11:101
1220499394,16cyclictest7750irq/56-dwc_otg05:51:063
1220499393,15cyclictest7950irq/56-dwc_otg_07:30:553
12204993915,5cyclictest23624-21aten_rbpower_en06:30:583
12204993915,4cyclictest16916-21sh07:01:063
1220299398,17cyclictest221ktimersoftd/107:50:311
1220299397,23cyclictest0-21swapper/106:05:591
1220299397,20cyclictest7950irq/56-dwc_otg_09:21:011
12202993916,6cyclictest7850irq/56-dwc_otg_08:26:011
153292381,3sleep30-21swapper/307:38:093
1220399387,18cyclictest0-21swapper/206:10:572
1220399385,15cyclictest26898-21cut05:10:572
1220399384,14cyclictest7850irq/56-dwc_otg_07:45:432
12203993815,5cyclictest1307450wget07:35:582
1220299386,5cyclictest23-21ksoftirqd/107:56:081
12202993820,6cyclictest23-21ksoftirqd/109:36:031
1220299381,4cyclictest0-21swapper/109:41:011
12202993814,10cyclictest8898-21aten_rbpower_cu05:50:551
1220499375,23cyclictest10415-21ssh08:50:573
1220499375,23cyclictest10415-21ssh08:50:573
1220499375,15cyclictest7950irq/56-dwc_otg_06:21:013
1220499374,6cyclictest4530-21ssh07:25:543
12204993717,6cyclictest39-21ksoftirqd/308:10:203
12204993713,6cyclictest39-21ksoftirqd/309:36:053
12204993712,10cyclictest39-21ksoftirqd/308:55:593
1220399377,23cyclictest0-21swapper/205:55:582
12203993719,5cyclictest717050wget05:45:582
12203993717,5cyclictest29617-21sshd07:55:542
1220299376,3cyclictest0-21swapper/107:25:541
1220299375,20cyclictest2128-21ssh08:40:571
12202993716,10cyclictest21-21rcuc/109:35:431
12204993621,9cyclictest37-21rcuc/305:42:553
12204993619,5cyclictest39-21ksoftirqd/306:45:433
12204993617,10cyclictest21010-21kworker/u8:207:22:073
1220499361,29cyclictest19360-21ls04:50:443
12203993625,6cyclictest301ktimersoftd/209:10:522
1220399361,4cyclictest7750irq/56-dwc_otg06:50:562
1220299369,6cyclictest23-21ksoftirqd/106:34:491
1220299364,16cyclictest7950irq/56-dwc_otg_05:01:041
1220299361,8cyclictest3209150ssh09:55:591
1220299361,25cyclictest0-21swapper/106:56:051
12202993611,11cyclictest0-21swapper/109:06:061
1220499357,4cyclictest7950irq/56-dwc_otg_08:35:593
1220499354,8cyclictest7950irq/56-dwc_otg_09:26:133
1220499352,6cyclictest2476-21sort08:41:023
12204993516,4cyclictest37-21rcuc/306:15:453
1220499351,3cyclictest7750irq/56-dwc_otg07:15:563
1220499351,23cyclictest13182-21ls04:31:103
1220499351,23cyclictest13182-21ls04:31:103
12204993510,5cyclictest7850irq/56-dwc_otg_06:30:433
1220399351,24cyclictest14903-21latency06:06:052
1220399351,24cyclictest0-21swapper/207:45:572
1220299355,12cyclictest7950irq/56-dwc_otg_05:31:001
12202993521,5cyclictest28569-21apt-get07:15:551
12202993518,7cyclictest23-21ksoftirqd/105:10:551
1220299351,6cyclictest23-21ksoftirqd/107:08:361
1220299351,23cyclictest1309-21latency_hist05:30:431
1220299351,22cyclictest18400-21ls09:00:431
12202993511,6cyclictest21-21rcuc/104:44:171
1220499344,4cyclictest7950irq/56-dwc_otg_09:55:553
1220499342,6cyclictest6608-21fschecks_time08:46:013
12204993424,8cyclictest7950irq/56-dwc_otg_09:10:593
1220499341,6cyclictest31520-21modprobe09:55:443
12204993416,4cyclictest12630-21missed_timers06:56:053
12204993413,4cyclictest22834-21sh09:05:573
1220399348,13cyclictest7950irq/56-dwc_otg_05:26:012
12203993427,5cyclictest301ktimersoftd/209:25:582
12203993424,7cyclictest31675-21sh09:55:452
1220399341,23cyclictest9824-21timerandwakeup07:31:082
1220299346,19cyclictest221ktimersoftd/106:13:081
1220299345,11cyclictest7950irq/56-dwc_otg_09:01:061
1220299342,5cyclictest7750irq/56-dwc_otg06:26:101
1220299342,19cyclictest23-21ksoftirqd/106:15:541
1220299341,9cyclictest10134-21kworker/u8:105:55:491
1220299341,9cyclictest0-21swapper/109:10:531
12202993412,15cyclictest25577-21systemd-journal04:46:261
1220499334,6cyclictest18302-21cut06:16:013
1220499334,21cyclictest7850irq/56-dwc_otg_05:16:003
1220499334,11cyclictest7750irq/56-dwc_otg06:46:093
1220499333,16cyclictest7950irq/56-dwc_otg_04:36:043
12204993318,9cyclictest26818-21aten_rbpower_en05:10:563
12204993312,9cyclictest20100-21sh09:01:243
1220499331,15cyclictest0-21swapper/309:20:443
1220399338,5cyclictest7950irq/56-dwc_otg_06:26:132
1220399337,16cyclictest0-21swapper/208:55:562
1220399334,15cyclictest7950irq/56-dwc_otg_05:35:442
1220399332,6cyclictest15502-21rm09:36:002
1220399331,7cyclictest7850irq/56-dwc_otg_08:20:442
1220399331,22cyclictest29241-21missed_timers07:16:052
1220399331,17cyclictest15101-21ssh06:59:302
12203993310,7cyclictest25577-21systemd-journal09:32:242
1220299337,4cyclictest5540-21cut05:41:001
12202993310,17cyclictest6101-21kworker/1:007:24:541
1220499324,17cyclictest37-21rcuc/307:53:193
1220499322,4cyclictest7850irq/56-dwc_otg_05:56:033
1220399328,11cyclictest7950irq/56-dwc_otg_07:10:442
1220399326,6cyclictest7750irq/56-dwc_otg08:40:582
1220399326,5cyclictest22941-21ls05:00:442
1220399323,5cyclictest10500-21runrttasks05:55:432
1220399322,24cyclictest5111-21latency07:26:032
12203993219,4cyclictest23921-21fschecks_time09:46:022
12203993213,6cyclictest908-21runrttasks09:58:322
1220399321,21cyclictest19742-21uniq09:41:012
110392326,11sleep10-21swapper/104:26:071
1220499318,16cyclictest0-21swapper/309:42:133
1220499316,7cyclictest7950irq/56-dwc_otg_08:26:033
1220499315,12cyclictest7950irq/56-dwc_otg_06:54:133
1220499313,12cyclictest7950irq/56-dwc_otg_09:46:013
1220499312,6cyclictest0-21swapper/309:31:083
1220499312,21cyclictest0-21swapper/304:57:243
12204993120,9cyclictest7950irq/56-dwc_otg_07:40:553
12204993113,4cyclictest1390-21latency_hist05:30:443
12204993113,3cyclictest7950irq/56-dwc_otg_05:21:103
1220399319,4cyclictest0-21swapper/207:10:552
1220399319,11cyclictest0-21swapper/206:47:032
1220399317,6cyclictest7950irq/56-dwc_otg_07:53:542
1220399317,15cyclictest7950irq/56-dwc_otg_08:28:462
1220399317,14cyclictest0-21swapper/208:10:572
1220399313,6cyclictest7750irq/56-dwc_otg06:16:122
1220399311,6cyclictest7850irq/56-dwc_otg_08:51:012
1220399311,6cyclictest7850irq/56-dwc_otg_08:51:012
1220399311,3cyclictest0-21swapper/206:00:562
12203993112,5cyclictest26501-21entropy08:31:002
1220299312,9cyclictest7950irq/56-dwc_otg_06:20:551
1220399308,10cyclictest7950irq/56-dwc_otg_05:21:112
1220399307,5cyclictest7850irq/56-dwc_otg_08:01:002
1220399307,14cyclictest0-21swapper/204:30:562
1220399307,14cyclictest0-21swapper/204:30:562
1220399301,7cyclictest6836-21ls08:46:032
1220399301,15cyclictest0-21swapper/209:06:062
1220399294,5cyclictest0-21swapper/208:36:102
1220399292,4cyclictest7750irq/56-dwc_otg06:43:062
12203992923,4cyclictest29677-21unixbench_multi05:16:112
1220399282,6cyclictest7850irq/56-dwc_otg_05:06:012
1220399282,6cyclictest7850irq/56-dwc_otg_05:06:012
116512286,14sleep20-21swapper/204:27:052
1220499276,5cyclictest1669-21cut05:30:573
1220499275,4cyclictest7950irq/56-dwc_otg_04:41:073
1220399273,2cyclictest0-21swapper/206:21:062
1220399272,4cyclictest7850irq/56-dwc_otg_04:51:102
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional