You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-26 - 12:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Fri Dec 26, 2025 00:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
351651921010,8sleep10-21swapper/123:00:231
343401128955,24sleep00-21swapper/019:07:550
35246592820,3sleep10-21swapper/123:25:151
34811272740,4chrt3480623-21/usr/sbin/munin21:20:202
197399740,60rtkit-daemon0-21swapper/219:08:092
343269127346,17sleep10-21swapper/119:05:071
197399690,5rtkit-daemon0-21swapper/220:58:282
197399690,5rtkit-daemon0-21swapper/220:58:282
197399690,5rtkit-daemon0-21swapper/219:44:262
197399670,8rtkit-daemon1972-21rtkit-daemon19:57:170
197399660,8rtkit-daemon1972-21rtkit-daemon00:08:100
343410926443,14sleep30-21swapper/319:08:443
34603132620,6sleep20-21swapper/220:20:242
197399600,9rtkit-daemon1972-21rtkit-daemon22:05:481
197399600,9rtkit-daemon1972-21rtkit-daemon22:05:481
34711402590,7sleep14497-21PK-Backend20:50:291
3434342995946,10cyclictest13-21ksoftirqd/023:41:500
3434342995946,10cyclictest13-21ksoftirqd/023:41:500
197399590,9rtkit-daemon1972-21rtkit-daemon22:35:401
343434699580,53cyclictest1972-21rtkit-daemon21:23:321
197399580,9rtkit-daemon1972-21rtkit-daemon22:24:040
197399580,8rtkit-daemon1972-21rtkit-daemon20:50:450
197399580,7rtkit-daemon1972-21rtkit-daemon22:00:061
35021672570,5sleep3411rcuc/322:20:253
197399570,8rtkit-daemon1972-21rtkit-daemon21:30:181
3434342995617,35cyclictest1972-21rtkit-daemon23:36:320
197399560,9rtkit-daemon1972-21rtkit-daemon21:42:490
197399560,9rtkit-daemon1972-21rtkit-daemon20:01:020
197399560,8rtkit-daemon1972-21rtkit-daemon21:47:570
197399550,9rtkit-daemon1972-21rtkit-daemon23:24:450
197399550,9rtkit-daemon1972-21rtkit-daemon23:19:150
197399550,9rtkit-daemon1972-21rtkit-daemon23:03:410
197399550,9rtkit-daemon1972-21rtkit-daemon21:03:410
197399550,9rtkit-daemon1972-21rtkit-daemon20:37:330
197399550,9rtkit-daemon1972-21rtkit-daemon20:24:290
197399550,8rtkit-daemon1972-21rtkit-daemon22:49:320
3434346995417,14cyclictest923-21dbus-daemon22:42:211
197399540,9rtkit-daemon1972-21rtkit-daemon23:13:300
197399540,9rtkit-daemon1972-21rtkit-daemon20:08:190
197399540,8rtkit-daemon1972-21rtkit-daemon22:25:391
197399540,8rtkit-daemon1972-21rtkit-daemon00:01:070
3434346995220,12cyclictest3543893-21expr00:20:151
343434299500,47cyclictest1972-21rtkit-daemon20:12:530
197399500,43rtkit-daemon2069-21gnome-shell21:05:090
34754472490,6sleep20-21swapper/221:05:122
343434699490,45cyclictest1972-21rtkit-daemon20:41:331
3434352994820,23cyclictest3442823-21grep19:30:203
3434346994821,6cyclictest3489037-21idleruntime-cro21:44:591
3434346994718,13cyclictest3447200-21/usr/sbin/munin19:45:011
3434346994716,11cyclictest3436658-21grep19:15:051
34791522460,4sleep20-21swapper/221:15:172
343435299451,26cyclictest3440281-21cut19:24:593
343435299451,26cyclictest3440281-21cut19:24:593
35319472440,14chrt331rcuc/223:45:202
34350022440,7sleep34497-21PK-Backend19:10:183
3434352994419,7cyclictest3512661-21cstates22:50:173
343434699441,9cyclictest0-21swapper/122:58:091
3434342994422,18cyclictest3549732-21python300:35:300
3434342994421,8cyclictest13-21ksoftirqd/021:15:280
3434342994421,8cyclictest13-21ksoftirqd/019:50:290
197399440,36rtkit-daemon2481-40tracker-miner-f22:00:320
197399440,31rtkit-daemon0-21swapper/319:35:183
34725262430,7sleep04497-21PK-Backend20:55:240
34725262430,7sleep04497-21PK-Backend20:55:230
3434348994322,17cyclictest3456450-21cat20:10:162
3434348994320,20cyclictest3487934-21df21:40:162
3434342994335,6cyclictest161rcuc/021:37:030
35303502420,6sleep30-21swapper/323:40:223
35303502420,6sleep30-21swapper/323:40:223
3434352994223,14cyclictest3477558-21df_inode21:10:183
3434342994214,14cyclictest1137-21runrttasks19:40:190
35127332410,7sleep04497-21PK-Backend22:50:160
3434342994132,6cyclictest161rcuc/020:28:450
197399410,35rtkit-daemon927-21gmain23:33:480
197399410,32rtkit-daemon3434336-21cyclictest21:10:050
34491402400,18sleep234-21ksoftirqd/219:50:022
3434352994016,19cyclictest3459548-21/usr/sbin/munin20:20:303
343434899404,17cyclictest34529922cat20:00:152
343434899401,14cyclictest438-20systemd-journal22:40:002
3434346994027,8cyclictest2093-21JS10:49:091
343434699402,16cyclictest35353912awk23:55:181
343434699401,16cyclictest34616432cat20:25:151
343434299405,19cyclictest2069-21gnome-shell22:58:090
343434299404,20cyclictest13-21ksoftirqd/022:40:090
3434342994023,12cyclictest438-20systemd-journal23:10:000
3434342994018,9cyclictest13-21ksoftirqd/022:29:130
197399400,33rtkit-daemon2221-21packagekitd23:57:490
343435299392,11cyclictest0-21swapper/321:08:493
3434352993920,13cyclictest3510191-21systemd-udevd22:42:183
343435299391,34cyclictest3448562-21runrttasks19:45:333
3434348993930,6cyclictest34-21ksoftirqd/222:42:182
3434348993923,11cyclictest438-20systemd-journal20:35:192
3434346993918,16cyclictest3463697-21grep20:30:241
3434346993918,16cyclictest3463697-21grep20:30:241
343434299396,17cyclictest13-21ksoftirqd/020:50:010
343434299394,8cyclictest13-21ksoftirqd/019:15:020
3434342993918,8cyclictest13-21ksoftirqd/022:08:200
3434342993918,8cyclictest13-21ksoftirqd/022:08:200
3434342993917,8cyclictest13-21ksoftirqd/023:55:020
3434352993822,12cyclictest1-21systemd20:15:003
343435299382,13cyclictest0-21swapper/322:18:233
3434352993821,13cyclictest3469019-21awk20:45:233
3434352993820,14cyclictest3463789-21ntp_states20:30:283
3434352993820,14cyclictest3463789-21ntp_states20:30:283
3434348993820,6cyclictest34-21ksoftirqd/220:00:012
343434899381,16cyclictest34497402latency19:50:202
343434299387,17cyclictest13-21ksoftirqd/000:15:000
343434299385,13cyclictest13-21ksoftirqd/020:40:090
343434299384,15cyclictest13-21ksoftirqd/023:45:320
197399380,26rtkit-daemon0-21swapper/320:56:023
197399380,26rtkit-daemon0-21swapper/320:56:013
34701322370,6chrt0-21swapper/220:50:022
343435299372,31cyclictest3479279-21df21:15:183
3434352993721,12cyclictest3542669-21ntpq00:15:253
3434352993721,12cyclictest3524864-21expr23:25:163
3434352993714,19cyclictest3536499-40mandb00:00:023
3434348993721,11cyclictest438-20systemd-journal20:16:182
3434348993719,6cyclictest34-21ksoftirqd/219:18:472
3434348993711,22cyclictest3536499-40mandb00:00:022
3434346993720,13cyclictest3550067-21seq00:36:571
343434299377,16cyclictest13-21ksoftirqd/019:20:080
343434299377,16cyclictest13-21ksoftirqd/019:20:080
3434342993716,8cyclictest13-21ksoftirqd/022:11:350
3434342993716,8cyclictest13-21ksoftirqd/020:30:550
3434342993716,8cyclictest13-21ksoftirqd/020:30:540
3434342993715,6cyclictest141rcu_preempt22:39:490
343434299371,21cyclictest3547508-21runrttasks00:30:170
197399370,30rtkit-daemon0-21swapper/121:45:571
197399370,24rtkit-daemon0-21swapper/319:15:233
35051202360,7sleep2331rcuc/222:30:142
34493792360,17sleep126-21ksoftirqd/119:50:121
3434352993624,7cyclictest3474098-21df_inode21:00:183
343435299362,13cyclictest3488317-21ntp_states21:40:253
3434352993621,11cyclictest3539784-21sshd00:08:173
3434352993620,11cyclictest3464735-21seq20:35:033
3434352993619,13cyclictest3523575-21grep23:20:253
343435299361,31cyclictest3467620-21rt-features20:40:273
3434348993621,11cyclictest923-21dbus-daemon00:25:022
3434348993619,13cyclictest976-21rs:main2
3434348993619,12cyclictest438-20systemd-journal23:30:002
3434348993614,19cyclictest3543389-21cron00:20:012
3434346993619,13cyclictest937-21polkitd23:41:521
3434346993619,13cyclictest937-21polkitd23:41:521
343434299362,17cyclictest13-21ksoftirqd/019:35:300
3434342993620,12cyclictest3544389-21ntp_kernel_pll_00:20:250
3434342993616,8cyclictest13-21ksoftirqd/022:16:150
3434342993616,8cyclictest13-21ksoftirqd/019:48:460
197399360,25rtkit-daemon0-21swapper/320:53:453
35505312350,6chrt0-21swapper/300:40:023
343435299359,22cyclictest3487133-21sshd21:38:023
3434352993521,10cyclictest3526724-21sort23:30:223
3434352993519,12cyclictest3452701-21cpu20:00:133
3434352993518,13cyclictest3514617-21if_enp2s022:55:223
3434352993518,13cyclictest3455113-21ntpq20:05:233
3434352993518,12cyclictest3458382-21sed20:15:243
3434352993518,12cyclictest1074-21lldpd22:10:313
3434352993517,13cyclictest3540793-21memory00:10:233
3434352993516,14cyclictest3528765-21ntpq23:35:243
343435299351,13cyclictest3491410-21diskstats21:50:163
343434899358,7cyclictest141rcu_preempt23:55:012
343434899352,8cyclictest438-20systemd-journal19:35:002
343434899352,6cyclictest141rcu_preempt22:50:152
3434348993523,7cyclictest34-21ksoftirqd/221:10:312
343434899352,17cyclictest3497717-21munin-run22:10:012
343434899352,17cyclictest3497717-21munin-run22:10:012
3434348993521,10cyclictest3538236-21systemctl00:05:012
3434348993521,10cyclictest3494213-21cron22:00:002
3434348993520,11cyclictest3529867-21cat23:40:142
3434348993520,11cyclictest3529867-21cat23:40:142
3434348993519,12cyclictest3464284-21seq20:32:292
3434348993519,12cyclictest3464284-21seq20:32:292
3434348993518,13cyclictest34-21ksoftirqd/222:45:152
3434346993520,11cyclictest3454590-21cat20:05:141
343434699351,9cyclictest4497-21PK-Backend19:45:331
3434346993517,13cyclictest3505242-21df22:30:151
343434699351,14cyclictest4497-21PK-Backend22:20:591
343434699351,14cyclictest4497-21PK-Backend21:12:251
343434699351,10cyclictest4497-21PK-Backend23:37:251
343434299357,12cyclictest4497-21PK-Backend21:55:050
343434299353,12cyclictest13-21ksoftirqd/019:10:140
343434299351,16cyclictest3481606-21wc21:20:290
197399350,27rtkit-daemon13-21ksoftirqd/021:52:000
343435299348,22cyclictest3504385-21taskset22:27:203
343435299348,22cyclictest0-21swapper/323:15:113
3434352993419,11cyclictest3533529-21cat23:50:173
3434352993419,11cyclictest3446199-21cat19:40:213
3434352993418,12cyclictest923-21dbus-daemon19:55:013
3434352993418,12cyclictest3545411-21grep00:25:143
3434352993418,12cyclictest3545411-21grep00:25:143
3434352993417,13cyclictest3693027-21snapd20:26:093
343435299341,30cyclictest4497-21PK-Backend21:58:273
3434348993424,6cyclictest34-21ksoftirqd/200:10:002
343434899342,11cyclictest141rcu_preempt20:51:372
3434348993419,11cyclictest3439105-21cat19:20:152
3434348993419,11cyclictest3439105-21cat19:20:142
3434348993419,10cyclictest3487505-21ls21:40:022
3434348993418,11cyclictest438-20systemd-journal23:01:172
3434348993417,13cyclictest3547875-21ntp_states00:30:252
3434348993417,13cyclictest3514547-21sshd22:55:172
3434348993417,13cyclictest3474472-21uname21:00:262
3434348993417,13cyclictest34-21ksoftirqd/223:25:012
3434346993423,6cyclictest26-21ksoftirqd/120:15:191
343434699342,10cyclictest3441282-21ntp_states19:25:251
343434699341,9cyclictest4497-21PK-Backend23:16:041
343434699341,9cyclictest4497-21PK-Backend22:46:261
343434699341,9cyclictest4497-21PK-Backend21:19:351
3434346993419,11cyclictest3542081-21cut00:15:151
343434699341,8cyclictest4497-21PK-Backend00:14:431
3434346993418,12cyclictest3435266-21awk19:10:221
3434346993418,11cyclictest438-20systemd-journal00:30:171
3434346993416,14cyclictest976-21rs:main1
3434346993415,15cyclictest3469281-21grep20:45:281
343434699341,30cyclictest4497-21PK-Backend22:15:191
343434699341,15cyclictest4497-21PK-Backend23:52:141
343434699341,10cyclictest4497-21PK-Backend21:05:451
343434299349,8cyclictest13-21ksoftirqd/021:30:160
343434299349,13cyclictest13-21ksoftirqd/019:31:110
343434299341,17cyclictest3546751-21taskset00:28:470
343434299341,17cyclictest3546751-21taskset00:28:470
197399340,26rtkit-daemon1972-21rtkit-daemon20:16:400
343435299333,8cyclictest3548005-21python300:30:283
3434352993322,7cyclictest3545130-21apt00:25:003
3434352993322,7cyclictest3507248-21diskstats22:35:173
3434352993322,7cyclictest3484302-21tr21:30:153
3434352993319,10cyclictest3512241-21cat22:50:023
3434352993318,11cyclictest3451943-21expr19:55:283
3434352993317,12cyclictest42-21ksoftirqd/322:32:093
3434352993317,12cyclictest3517297-21sshd23:04:173
343435299331,12cyclictest3532400-21gzip23:45:313
3434348993323,6cyclictest34-21ksoftirqd/223:35:292
3434348993318,11cyclictest3467154-21forks20:40:182
3434348993318,11cyclictest3436054-21systemd19:15:012
3434348993317,12cyclictest3504518-21seq22:29:042
343434899331,4cyclictest0-21swapper/200:39:592
343434899331,29cyclictest0-21swapper/219:38:502
3434348993312,5cyclictest141rcu_preempt23:05:242
3434346993326,4cyclictest4497-21PK-Backend22:12:311
3434346993321,8cyclictest3456281-21tr20:10:151
343434699331,9cyclictest4497-21PK-Backend20:56:561
343434699331,9cyclictest4497-21PK-Backend20:56:561
3434346993319,10cyclictest3483990-21sed21:30:011
343434699331,8cyclictest4497-21PK-Backend23:48:281
343434699331,8cyclictest4497-21PK-Backend21:01:311
3434346993318,11cyclictest3527016-21ntpq23:30:261
3434346993318,11cyclictest3519767-21awk23:10:211
343434699331,29cyclictest0-21swapper/119:37:231
343434699331,13cyclictest4497-21PK-Backend21:36:481
343434299334,17cyclictest0-21swapper/022:31:470
3434342993320,10cyclictest13-21ksoftirqd/019:25:210
343434299331,6cyclictest3483307-21ls21:25:320
197399330,25rtkit-daemon4497-21PK-Backend20:00:061
3434352993226,4cyclictest0-21swapper/323:11:003
3434352993220,8cyclictest3441090-21grep19:25:213
3434352993218,10cyclictest3495300-21grep22:00:273
3434352993216,12cyclictest42-21ksoftirqd/321:50:003
3434352993214,13cyclictest3482322-21sed21:25:013
343435299321,28cyclictest4497-21PK-Backend23:07:523
343435299321,28cyclictest0-21swapper/322:06:543
343435299321,28cyclictest0-21swapper/322:06:543
3434348993223,5cyclictest34-21ksoftirqd/221:45:112
3434348993222,6cyclictest34-21ksoftirqd/221:25:182
3434348993221,6cyclictest34-21ksoftirqd/223:15:282
343434899321,28cyclictest0-21swapper/221:51:262
343434899321,28cyclictest0-21swapper/220:25:072
3434346993225,4cyclictest4497-21PK-Backend00:01:201
3434346993221,8cyclictest3459752-21cpuspeed_turbos20:20:131
343434699321,8cyclictest4497-21PK-Backend22:50:121
3434346993217,11cyclictest3465502-21ntp_kernel_pll_20:35:271
3434346993217,11cyclictest3439893-21seq19:20:551
3434346993217,11cyclictest3439893-21seq19:20:541
343434699321,28cyclictest4497-21PK-Backend00:08:501
343434699321,28cyclictest0-21swapper/100:29:471
343434699321,28cyclictest0-21swapper/100:29:471
343434299322,9cyclictest13-21ksoftirqd/000:15:170
3434342993218,10cyclictest3525380-21cut23:25:280
35180382310,6chrt0-21swapper/123:05:181
343435299312,25cyclictest3482966-21memory21:25:243
3434348993122,5cyclictest34-21ksoftirqd/223:11:392
3434348993121,5cyclictest34-21ksoftirqd/221:30:232
343434899311,9cyclictest141rcu_preempt19:25:192
3434348993118,9cyclictest3498435-21seq22:10:202
3434348993117,10cyclictest3545585-21grep00:25:152
3434348993117,10cyclictest3545585-21grep00:25:152
3434348993117,10cyclictest3541486-21seq00:12:542
3434348993117,10cyclictest3526512-21cstates23:30:182
343434899311,4cyclictest4497-21PK-Backend22:01:072
343434899311,4cyclictest0-21swapper/222:18:012
3434346993117,10cyclictest3442526-21cat19:30:141
343434699311,4cyclictest4497-21PK-Backend21:51:331
343434699311,27cyclictest4497-21PK-Backend19:59:371
343435299302,24cyclictest3536719-21/usr/sbin/munin00:00:243
3434348992920,5cyclictest34-21ksoftirqd/220:09:532
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional