You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-24 - 05:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Wed Dec 24, 2025 00:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1973991290,8rtkit-daemon1972-21rtkit-daemon23:25:203
1973991290,8rtkit-daemon1972-21rtkit-daemon23:25:203
22943622930,6sleep14497-21PK-Backend19:45:271
197399880,73rtkit-daemon0-21swapper/119:06:581
197399770,6rtkit-daemon0-21swapper/000:19:460
197399760,5rtkit-daemon0-21swapper/023:56:220
197399750,6rtkit-daemon4497-21PK-Backend20:02:330
197399750,5rtkit-daemon0-21swapper/022:40:300
197399750,5rtkit-daemon0-21swapper/020:51:550
197399750,5rtkit-daemon0-21swapper/020:22:520
197399750,5rtkit-daemon0-21swapper/019:19:560
197399740,6rtkit-daemon4497-21PK-Backend21:17:260
197399740,5rtkit-daemon4497-21PK-Backend20:28:130
197399740,5rtkit-daemon0-21swapper/021:27:410
197399740,5rtkit-daemon0-21swapper/019:56:470
197399730,6rtkit-daemon0-21swapper/021:57:540
197399720,5rtkit-daemon0-21swapper/021:03:340
197399710,6rtkit-daemon4497-21PK-Backend22:21:320
197399700,5rtkit-daemon4497-21PK-Backend19:49:120
197399670,5rtkit-daemon0-21swapper/021:41:500
228068099640,59cyclictest1972-21rtkit-daemon23:25:202
228068099640,59cyclictest1972-21rtkit-daemon23:25:202
227987426342,13sleep00-21swapper/019:05:270
227897326341,15sleep20-21swapper/219:05:052
228033326242,13sleep30-21swapper/319:08:283
197399600,9rtkit-daemon1972-21rtkit-daemon23:39:572
197399600,9rtkit-daemon1972-21rtkit-daemon23:19:032
22814222580,20sleep342-21ksoftirqd/319:10:163
2280669995720,12cyclictest2378613-21python323:45:290
2280669995720,12cyclictest2378613-21python323:45:290
197399570,9rtkit-daemon1972-21rtkit-daemon22:50:463
2280686995623,7cyclictest2377477-21munin-run23:45:003
2280686995618,11cyclictest2379794-21expr23:50:143
228066999561,31cyclictest0-21swapper/000:10:150
2280669995518,11cyclictest2369073-21find23:20:110
23927902540,5chrt0-21swapper/200:25:222
2280686995416,13cyclictest2360977-21memory22:55:213
2280686995412,13cyclictest438-20systemd-journal23:15:003
228068099540,50cyclictest1972-21rtkit-daemon23:42:572
228068099540,50cyclictest1972-21rtkit-daemon22:48:392
197399540,9rtkit-daemon1972-21rtkit-daemon23:21:573
197399540,8rtkit-daemon1972-21rtkit-daemon23:17:373
197399540,8rtkit-daemon1972-21rtkit-daemon23:00:072
2280686995318,12cyclictest2391085-21cat00:20:223
2280686995318,10cyclictest2373737-21seq23:31:403
23534672520,7sleep24497-21PK-Backend22:35:152
2280686995219,12cyclictest2283539-21cstates19:15:163
2280686995217,9cyclictest2310151-21python320:30:293
2280686995120,11cyclictest2322617-21grep21:05:293
2280686995120,10cyclictest42-21ksoftirqd/320:20:013
2280686995118,6cyclictest2321429-21turbostat.cron21:05:003
2280686995115,11cyclictest2285647-21ls19:20:213
2280686995114,12cyclictest2352645-21seq22:31:233
2280686995114,12cyclictest2287831-21wc19:25:333
2280686995113,12cyclictest2331537-21seq21:30:543
228068099510,47cyclictest1972-21rtkit-daemon23:07:252
23836072500,5sleep30-21swapper/300:00:163
228068699502,11cyclictest2301256-21cat20:05:223
2280686995015,11cyclictest2293826-21cpuspeed_turbos19:45:193
228068099500,46cyclictest1972-21rtkit-daemon23:34:062
23603952490,8sleep20-21swapper/222:55:112
2280686994916,9cyclictest2324297-21processes21:10:283
2280686994915,10cyclictest2362363-21tr23:00:173
2280669994916,13cyclictest2386520-40apt-check00:07:560
23349712480,16chrt42-21ksoftirqd/321:40:313
228068699482,24cyclictest2349272-40lsb_release22:23:003
2280686994817,7cyclictest2389456-21ntp_kernel_err00:15:233
2280686994817,6cyclictest2337066-21latency_hist21:50:023
2280686994812,11cyclictest2346024-21cpu22:15:143
2280686994812,11cyclictest2346024-21cpu22:15:143
197399480,33rtkit-daemon4497-21PK-Backend00:23:351
197399480,24rtkit-daemon0-21swapper/123:53:001
228068699472,9cyclictest2337729-21fw_packets21:50:183
2280686994716,10cyclictest2339615-21grep21:55:223
23840042460,7chrt4497-21PK-Backend00:00:261
228068699462,25cyclictest2303848-21munin-run20:15:013
2280669994617,10cyclictest923-21dbus-daemon22:39:560
2280669994614,9cyclictest13-21ksoftirqd/000:00:320
197399460,5rtkit-daemon4497-21PK-Backend19:56:061
228068699452,7cyclictest2307308-21munin-run20:25:003
228068699451,7cyclictest2327040-21cpuspeed_turbos21:20:133
2280686994517,7cyclictest2299261-21df20:00:163
2280686994516,9cyclictest42-21ksoftirqd/322:45:113
228068699451,23cyclictest2312872-21expr20:40:163
228068699451,10cyclictest2354845-21latency_hist22:40:023
2280686994417,7cyclictest2307602-21cpuspeed20:25:143
228068699441,5cyclictest2298657-21systemd-run20:00:003
228068699432,8cyclictest2386520-40apt-check00:08:043
228068699432,5cyclictest23438952sleep322:09:043
2280686994324,4cyclictest0-21swapper/300:14:153
2280686994315,9cyclictest42-21ksoftirqd/323:45:313
2280686994315,9cyclictest42-21ksoftirqd/323:45:313
2280686994216,7cyclictest924-21NetworkManager23:58:433
2280686994211,7cyclictest0-21swapper/300:25:153
23760472410,6sleep1251rcuc/123:40:141
23447732410,11chrt2344230-21/usr/sbin/munin22:10:231
2280674994131,6cyclictest26-21ksoftirqd/122:15:141
2280674994131,6cyclictest26-21ksoftirqd/122:15:141
2280669994123,6cyclictest13-21ksoftirqd/000:23:100
2280669994123,13cyclictest2372919-21df_inode23:30:160
2280669994122,14cyclictest2331163-21ntp_states21:30:270
2280669994122,14cyclictest13-21ksoftirqd/021:50:100
228066999412,10cyclictest2069-21gnome-shell22:56:080
197399410,23rtkit-daemon0-21swapper/022:04:290
23446502400,5sleep2331rcuc/222:10:202
2280669994023,13cyclictest923-21dbus-daemon20:50:000
2280669994021,14cyclictest13-21ksoftirqd/022:31:070
197399400,26rtkit-daemon4497-21PK-Backend23:50:021
197399400,26rtkit-daemon4497-21PK-Backend23:50:021
23026382390,6chrt13-21ksoftirqd/020:10:170
2280680993924,11cyclictest2391149-21tail00:20:232
2280680993923,12cyclictest923-21dbus-daemon21:35:012
2280674993928,7cyclictest2317213-21ntpq20:50:251
2280674993925,10cyclictest26-21ksoftirqd/121:07:441
228067499391,34cyclictest2300103-21chrt20:01:571
228066999396,11cyclictest13-21ksoftirqd/021:46:030
228066999396,11cyclictest13-21ksoftirqd/020:15:080
2280669993923,12cyclictest1-21systemd23:05:010
2280669993920,14cyclictest13-21ksoftirqd/023:13:230
197399390,27rtkit-daemon4497-21PK-Backend21:51:501
197399390,27rtkit-daemon4497-21PK-Backend21:21:071
197399390,27rtkit-daemon4497-21PK-Backend19:40:440
197399390,26rtkit-daemon4497-21PK-Backend22:28:150
23147322380,5chrt0-21swapper/320:45:143
23135632380,7sleep14497-21PK-Backend20:40:291
2280680993822,11cyclictest438-20systemd-journal20:11:182
2280680993815,19cyclictest2305581-21unin-run20:20:012
228066999381,14cyclictest23945722cat00:30:220
197399380,26rtkit-daemon4497-21PK-Backend22:14:300
197399380,25rtkit-daemon4497-21PK-Backend20:47:061
197399380,25rtkit-daemon0-21swapper/123:59:431
197399380,25rtkit-daemon0-21swapper/119:31:321
197399380,25rtkit-daemon0-21swapper/119:31:311
197399380,25rtkit-daemon0-21swapper/021:11:120
197399380,24rtkit-daemon0-21swapper/121:14:281
2280686993721,12cyclictest2289235-21grep19:30:243
2280686993721,12cyclictest2289235-21grep19:30:243
2280680993721,12cyclictest2315942-21apt-get20:50:002
2280680993720,12cyclictest438-20systemd-journal23:50:002
2280680993720,12cyclictest438-20systemd-journal23:50:002
2280680993719,13cyclictest438-20systemd-journal22:30:002
2280674993720,12cyclictest438-20systemd-journal00:15:171
2280674993720,12cyclictest2363232-21sed23:00:341
228066999376,11cyclictest13-21ksoftirqd/019:38:550
2280669993720,13cyclictest2309649-21df_inode20:30:160
2280669993719,13cyclictest13-21ksoftirqd/019:20:590
197399370,25rtkit-daemon0-21swapper/120:13:551
197399370,25rtkit-daemon0-21swapper/020:59:040
197399370,25rtkit-daemon0-21swapper/000:26:370
197399370,24rtkit-daemon0-21swapper/122:25:431
2280686993620,12cyclictest2395199-21seq00:32:363
2280686993620,12cyclictest2355500-21cat22:40:213
2280686993620,11cyclictest2396047-21diskstats00:35:163
2280686993620,11cyclictest2396047-21diskstats00:35:163
2280680993620,12cyclictest2384485-21sshd00:02:172
2280680993618,14cyclictest34-21ksoftirqd/221:07:452
2280680993615,9cyclictest2284779-21perf19:20:002
2280680993614,19cyclictest2383004-40mandb00:00:012
2280669993620,12cyclictest2364056-21tr23:05:130
2280669993620,11cyclictest2380966-21gpgv23:55:000
2280669993619,12cyclictest2288514-21apt19:30:130
2280669993619,12cyclictest2288514-21apt19:30:130
2280669993618,6cyclictest13-21ksoftirqd/000:38:080
2280669993618,6cyclictest13-21ksoftirqd/000:38:080
2280669993618,5cyclictest13-21ksoftirqd/023:40:570
2280669993618,14cyclictest13-21ksoftirqd/020:40:160
2280669993618,14cyclictest13-21ksoftirqd/019:12:390
2280669993610,5cyclictest141rcu_preempt21:37:350
197399360,25rtkit-daemon0-21swapper/019:29:100
23662322350,5sleep1251rcuc/123:10:241
2280686993523,7cyclictest42-21ksoftirqd/321:15:123
2280686993521,10cyclictest42-21ksoftirqd/319:45:003
228068699351,9cyclictest0-21swapper/321:37:003
228068699351,9cyclictest0-21swapper/320:39:493
2280686993519,12cyclictest2329062-21cat21:25:173
2280680993524,6cyclictest438-20systemd-journal21:19:552
2280680993521,9cyclictest2069-21gnome-shell22:05:092
2280680993520,11cyclictest2317955-21cron20:55:002
2280680993518,12cyclictest2365768-21expr23:10:152
2280674993520,11cyclictest2343832-21sshd22:08:171
2280674993518,12cyclictest438-20systemd-journal00:40:011
2280674993518,12cyclictest438-20systemd-journal00:40:011
2280674993517,13cyclictest438-20systemd-journal21:17:001
228067499351,29cyclictest2385298-21awk00:05:151
228066999353,11cyclictest13-21ksoftirqd/022:47:410
2280669993525,7cyclictest2359056-21needreboot22:50:220
2280669993521,10cyclictest13-21ksoftirqd/020:35:170
197399350,29rtkit-daemon0-21swapper/223:22:042
197399350,24rtkit-daemon0-21swapper/022:06:260
197399350,24rtkit-daemon0-21swapper/021:09:040
23392092340,13sleep12339210-21cpuspeed_turbos21:55:141
2280686993419,11cyclictest2319182-21find20:55:283
2280686993418,12cyclictest2296161-21awk19:50:253
2280686993417,13cyclictest42-21ksoftirqd/322:27:113
2280680993427,4cyclictest0-21swapper/200:12:082
2280680993424,7cyclictest1137-21runrttasks21:35:542
2280680993422,8cyclictest2297143-21find19:55:112
2280680993421,8cyclictest2286768-21find19:25:132
2280680993419,11cyclictest2299275-21diskstats20:00:182
2280680993418,12cyclictest2385971-21awk00:05:262
2280680993417,13cyclictest2193-21gdbus00:19:572
2280680993417,12cyclictest2328603-21wc21:25:022
2280680993414,10cyclictest2301333-21netstat20:05:232
2280680993413,11cyclictest2328749-21cat21:25:132
2280674993425,6cyclictest2331867-21dump-pmu-power21:35:001
2280674993420,10cyclictest26-21ksoftirqd/121:40:181
2280674993419,11cyclictest2347936-21mktemp22:20:041
2280674993418,11cyclictest438-20systemd-journal20:35:001
228066999341,11cyclictest13-21ksoftirqd/023:26:040
228066999341,11cyclictest13-21ksoftirqd/023:26:040
197399340,26rtkit-daemon1054-21snmpd23:07:363
2280686993325,5cyclictest4497-21PK-Backend20:50:413
2280686993319,10cyclictest2341197-21smartctl22:00:193
228068699331,11cyclictest2291919-21cut19:40:023
2280680993322,7cyclictest2293635-21ls19:45:022
2280680993322,7cyclictest1137-21runrttasks19:20:122
2280680993320,9cyclictest34-21ksoftirqd/222:22:172
2280680993318,11cyclictest2310063-21pluginstate20:30:272
2280680993318,11cyclictest2294861-21sshd19:46:172
2280680993318,10cyclictest2396727-21cat00:35:282
2280680993318,10cyclictest2396727-21cat00:35:272
2280680993318,10cyclictest2312462-21cron20:40:022
228068099331,29cyclictest0-21swapper/219:39:272
2280674993323,6cyclictest26-21ksoftirqd/123:33:091
2280674993323,5cyclictest26-21ksoftirqd/122:55:021
2280674993322,6cyclictest26-21ksoftirqd/120:05:121
2280674993322,6cyclictest26-21ksoftirqd/100:25:131
2280674993320,9cyclictest26-21ksoftirqd/123:35:221
2280674993318,11cyclictest2388325-21sshd00:13:171
2280674993318,11cyclictest2286500-21cron19:25:001
228067499331,28cyclictest0-21swapper/119:19:251
2280669993325,5cyclictest0-21swapper/023:35:130
2280669993319,10cyclictest2296379-21cat19:50:310
2280669993317,12cyclictest2368819-21/usr/sbin/munin23:20:010
2280669993316,5cyclictest13-21ksoftirqd/022:17:510
2280669993316,5cyclictest13-21ksoftirqd/022:17:510
197399330,27rtkit-daemon0-21swapper/222:53:412
197399330,25rtkit-daemon4497-21PK-Backend20:15:091
23607962320,7sleep14497-21PK-Backend22:55:181
23410382320,7chrt2341003-21cstates22:00:142
228068699322,25cyclictest2374643-21df23:35:163
228068699322,25cyclictest2345744-21munin-run22:15:013
2280680993219,8cyclictest2069-21gnome-shell20:25:092
2280680993218,10cyclictest2281794-21cat19:10:242
2280680993217,11cyclictest2394465-21grep00:30:212
2280680993217,11cyclictest2346399-21basename22:15:202
2280680993217,11cyclictest2346399-21basename22:15:202
2280680993217,11cyclictest2324082-21cat21:10:242
2280680993217,11cyclictest2306691-21python320:20:292
228068099321,28cyclictest0-21swapper/221:44:512
228068099321,26cyclictest438-20systemd-journal21:00:002
2280680993211,17cyclictest2296909-21grep19:55:022
2280674993223,5cyclictest4497-21PK-Backend19:10:591
2280674993223,5cyclictest26-21ksoftirqd/119:40:281
2280674993222,6cyclictest26-21ksoftirqd/122:05:001
2280674993222,6cyclictest26-21ksoftirqd/121:03:141
2280674993222,6cyclictest26-21ksoftirqd/119:29:001
2280674993222,5cyclictest26-21ksoftirqd/100:30:201
2280674993221,6cyclictest26-21ksoftirqd/123:22:091
2280674993220,8cyclictest2371677-21grep23:25:251
2280674993220,8cyclictest2371677-21grep23:25:251
2280674993218,10cyclictest2295585-21grep19:50:161
2280674993217,11cyclictest2357167-21cut22:45:221
2280674993217,11cyclictest2354294-21awk22:35:311
2280674993217,10cyclictest2356522-21grep22:45:021
2280674993217,10cyclictest2352048-21cat22:30:221
2280674993216,12cyclictest2311653-21cat20:35:231
228067499321,4cyclictest4497-21PK-Backend21:35:271
228066999326,5cyclictest141rcu_preempt21:20:240
2280680993120,7cyclictest2337827-21kernelversion21:50:212
2280680993120,7cyclictest2336082-21interrupts21:45:192
228068099311,7cyclictest2380929-21apt-get23:55:002
2280680993115,11cyclictest2355040-21wc22:40:132
228068099311,4cyclictest0-21swapper/222:33:572
2280674993120,7cyclictest2291027-21proc_pri19:35:251
2280674993115,5cyclictest26-21ksoftirqd/121:45:281
197399310,19rtkit-daemon13-21ksoftirqd/020:05:090
228068099301,9cyclictest0-21swapper/221:01:012
228068099301,8cyclictest924-21NetworkManager20:35:142
2280680993014,12cyclictest2340295-21sshd21:57:172
228067499301,6cyclictest4497-21PK-Backend20:25:261
228068099298,4cyclictest141rcu_preempt19:30:192
228068099298,4cyclictest141rcu_preempt19:30:192
2280674992919,6cyclictest26-21ksoftirqd/123:19:081
2280674992919,6cyclictest26-21ksoftirqd/123:05:191
2280674992917,8cyclictest2306419-21sort20:20:241
2280674992916,10cyclictest1-21systemd21:00:011
2280674992810,5cyclictest141rcu_preempt21:25:481
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional