You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-08 - 12:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Dec 08, 2025 00:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
278655827959,14sleep20-21swapper/219:08:562
28784522780,7sleep31753-21gnome-shell23:30:023
278638227548,17sleep30-21swapper/319:06:433
278660927140,25sleep10-21swapper/119:09:361
28545922660,24sleep30-21swapper/322:20:233
27965712660,7sleep20-21swapper/219:35:192
278635926543,15sleep00-21swapper/019:06:250
162399640,9rtkit-daemon1622-21rtkit-daemon22:37:030
162399640,8rtkit-daemon1622-21rtkit-daemon00:36:241
162399640,8rtkit-daemon1622-21rtkit-daemon00:36:241
162399610,8rtkit-daemon1622-21rtkit-daemon20:34:201
162399610,8rtkit-daemon1622-21rtkit-daemon20:34:191
28801672600,10chrt2880154-21perf23:35:002
2786775996017,10cyclictest2789322-21ls19:15:160
162399600,8rtkit-daemon1622-21rtkit-daemon22:45:401
162399600,8rtkit-daemon1622-21rtkit-daemon20:50:421
162399590,8rtkit-daemon1622-21rtkit-daemon19:33:551
162399590,8rtkit-daemon1622-21rtkit-daemon19:18:131
27961752580,20sleep342-21ksoftirqd/319:35:133
162399580,8rtkit-daemon1622-21rtkit-daemon21:16:041
28373922570,4sleep00-21swapper/021:30:330
278677599572,8cyclictest1753-21gnome-shell22:06:540
162399550,8rtkit-daemon1622-21rtkit-daemon22:39:451
2786775995423,8cyclictest1753-21gnome-shell20:29:010
162399530,8rtkit-daemon1622-21rtkit-daemon00:00:361
2786775995218,12cyclictest854-21avahi-daemon00:00:050
278677599520,47cyclictest1622-21rtkit-daemon00:39:290
278677599520,47cyclictest1622-21rtkit-daemon00:39:280
28811362510,6sleep32197014-21PK-Backend23:35:253
28780002500,3sleep10-21swapper/123:25:321
28047412490,7chrt0-21swapper/120:00:021
28933202470,6sleep01753-21gnome-shell00:10:230
2786779994719,12cyclictest2894053-21/usr/sbin/munin00:15:001
2786775994715,10cyclictest1753-21gnome-shell21:28:110
278677599470,42cyclictest1622-21rtkit-daemon19:33:280
28561332460,7sleep1251rcuc/122:25:151
278678499459,10cyclictest42-21ksoftirqd/321:33:233
2786784994517,23cyclictest42-21ksoftirqd/320:05:203
2786784994517,10cyclictest1753-21gnome-shell19:16:453
278678499451,39cyclictest1-21systemd23:00:013
2786781994534,7cyclictest1069-21runrttasks22:13:562
2786781994529,12cyclictest2885365-21munin-run23:50:012
2786781994529,12cyclictest2885365-21munin-run23:50:002
2786781994514,8cyclictest28933312ntp_states00:10:252
2786775994536,6cyclictest181irq_work/021:48:450
162399450,33rtkit-daemon0-21swapper/319:10:473
28246472440,5sleep00-21swapper/020:55:220
28073342440,6sleep22786769-21cyclictest20:05:262
278678499449,16cyclictest1753-21gnome-shell19:55:463
29010622430,7chrt0-21swapper/200:35:012
28844202430,7sleep12197014-21PK-Backend23:45:211
28844202430,7sleep12197014-21PK-Backend23:45:211
278678499436,10cyclictest42-21ksoftirqd/321:21:043
2786784994327,12cyclictest42-21ksoftirqd/321:42:303
2786784994314,16cyclictest1753-21gnome-shell21:36:033
2786781994324,14cyclictest2847699-40lsb_release22:00:252
2786781994321,18cyclictest2898450-21grep00:25:262
2786781994321,11cyclictest2869877-21cut23:05:022
2786781994318,20cyclictest1-21systemd00:00:032
278678199431,13cyclictest2820414-21cat20:45:012
2786779994326,12cyclictest2836748-21anacron21:30:181
278678199428,6cyclictest141rcu_preempt00:24:122
278678199425,8cyclictest34-21ksoftirqd/219:27:532
278678199422,12cyclictest859-21NetworkManager22:37:472
2786781994210,5cyclictest141rcu_preempt22:45:312
28760212410,5sleep32197014-21PK-Backend23:20:283
278678499416,16cyclictest1753-21gnome-shell22:18:023
278678499416,16cyclictest1753-21gnome-shell22:18:023
2786784994122,14cyclictest2814617-21users20:25:333
2786784994114,9cyclictest141rcu_preempt00:17:203
2786781994123,13cyclictest2866552-21apt22:55:152
2786781994123,13cyclictest2789594-21entropy19:15:202
2786781994114,6cyclictest141rcu_preempt19:31:192
2786779994121,16cyclictest3242253-21nautilus23:00:001
278677999411,35cyclictest2835376-21tr21:25:251
2786775994126,11cyclictest2880157-21/usr/sbin/munin23:35:000
2786775994122,14cyclictest2838858-21ntp_states21:35:260
28456452400,5sleep20-21swapper/221:55:222
2786784994026,11cyclictest42-21ksoftirqd/322:53:233
2786784994025,11cyclictest42-21ksoftirqd/320:59:243
2786784994024,12cyclictest2813036-21sshd20:22:193
2786784994023,12cyclictest438-20systemd-journal22:49:353
2786784994011,24cyclictest42-21ksoftirqd/319:40:323
2786781994023,12cyclictest438-20systemd-journal20:02:152
2786781994023,12cyclictest2792415-21/usr/sbin/munin19:25:012
2786781994022,13cyclictest2903051-21latency_hist00:40:032
2786781994022,13cyclictest2903051-21latency_hist00:40:022
2786781994022,13cyclictest2827422-21cron21:04:592
2786779994023,13cyclictest855-21dbus-daemon20:15:011
2786779994022,13cyclictest2847500-21sh22:00:221
162399400,3rtkit-daemon1622-21rtkit-daemon19:38:511
28910752390,6sleep20-21swapper/200:05:142
28910752390,6sleep20-21swapper/200:05:142
28652252390,7sleep2331rcuc/222:50:202
2786784993921,6cyclictest42-21ksoftirqd/322:10:213
2786784993921,13cyclictest2901353-21ls00:35:143
2786784993921,13cyclictest2901353-21ls00:35:143
2786784993919,17cyclictest42-21ksoftirqd/321:13:293
2786784993910,16cyclictest1753-21gnome-shell00:14:523
278678199394,9cyclictest34-21ksoftirqd/219:13:242
278678199392,8cyclictest1753-21gnome-shell21:44:362
278678199392,33cyclictest2825452-40lsb_release20:57:232
2786781993922,13cyclictest34-21ksoftirqd/220:13:392
278678199391,33cyclictest2873089-21sshd23:13:342
2786779993922,13cyclictest26-21ksoftirqd/120:06:541
2786779993915,20cyclictest2817676-21needreboot20:35:261
278677599392,9cyclictest0-21swapper/023:35:270
278677599392,8cyclictest1753-21gnome-shell00:17:360
278677599392,32cyclictest1753-21gnome-shell19:38:030
2786775993917,17cyclictest1753-21gnome-shell19:45:080
162399390,31rtkit-daemon2786769-21cyclictest22:30:371
28884402380,5sleep30-21swapper/323:55:333
28160112380,4sleep22816017-21ntp_states20:30:272
28160112380,4sleep22816017-21ntp_states20:30:272
278678499387,5cyclictest141rcu_preempt21:15:293
278678499387,11cyclictest42-21ksoftirqd/323:03:253
2786784993828,7cyclictest2855917-21cpuspeed_turbos22:25:153
278678499382,7cyclictest1753-21gnome-shell00:05:133
278678499382,7cyclictest1753-21gnome-shell00:05:133
2786784993826,8cyclictest1753-21gnome-shell19:46:553
2786784993825,9cyclictest2792405-21perf19:25:013
2786784993823,9cyclictest2793060-21systemd-udevd19:25:203
2786784993822,12cyclictest42-21ksoftirqd/321:57:423
2786784993821,13cyclictest42-21ksoftirqd/320:39:553
2786784993821,13cyclictest42-21ksoftirqd/300:22:173
2786784993819,14cyclictest2873854-21df_inode23:15:173
2786784993818,14cyclictest42-21ksoftirqd/323:50:163
2786784993815,19cyclictest391irq_work/322:01:273
278678499381,32cyclictest2844853-21munin-run21:55:003
278678199385,11cyclictest34-21ksoftirqd/221:16:592
278678199384,10cyclictest34-21ksoftirqd/221:54:182
2786781993825,8cyclictest2799814-21tr19:45:142
278678199382,32cyclictest1753-21gnome-shell20:18:172
278678199382,32cyclictest1753-21gnome-shell00:16:162
2786781993822,11cyclictest438-20systemd-journal22:35:002
278678199382,12cyclictest141rcu_preempt22:09:192
2786781993814,20cyclictest2836111-21latency_hist21:30:022
2786781993811,24cyclictest2888855-40.mlocate00:00:012
2786781993810,6cyclictest141rcu_preempt22:16:452
2786781993810,6cyclictest141rcu_preempt22:16:442
278677999386,9cyclictest26-21ksoftirqd/121:44:591
2786779993827,7cyclictest251rcuc/121:05:511
2786779993821,13cyclictest26-21ksoftirqd/123:40:011
2786779993821,13cyclictest26-21ksoftirqd/121:15:001
2786779993821,12cyclictest2898515-21ntp_kernel_pll_00:25:281
2786779993821,12cyclictest2895088-21open_inodes00:15:291
278677999381,34cyclictest0-21swapper/121:38:321
278677599388,6cyclictest141rcu_preempt22:30:000
278677599382,31cyclictest2804207-21users19:55:330
2786775993823,11cyclictest1-21systemd00:25:000
2786775993819,14cyclictest2863045-40apt-check22:45:190
162399380,30rtkit-daemon0-21swapper/123:01:091
2786784993724,9cyclictest42-21ksoftirqd/320:15:183
2786784993723,10cyclictest42-21ksoftirqd/323:44:493
2786784993721,11cyclictest2857861-21expr22:30:163
2786784993720,12cyclictest2849261-21fw_packets22:05:213
2786784993719,14cyclictest2871040-21sed23:05:363
278678499371,6cyclictest2197014-21PK-Backend22:40:013
2786784993716,17cyclictest1753-21gnome-shell19:50:333
2786784993716,17cyclictest1753-21gnome-shell19:50:323
2786784993716,16cyclictest3242253-21nautilus21:50:023
278678199377,27cyclictest1069-21runrttasks21:08:432
278678199374,10cyclictest34-21ksoftirqd/221:13:162
278678199372,9cyclictest34-21ksoftirqd/223:54:242
2786781993723,10cyclictest2822495-21expr20:50:172
278678199371,7cyclictest2197014-21PK-Backend19:52:202
278678199371,7cyclictest2197014-21PK-Backend19:52:192
278677999374,10cyclictest26-21ksoftirqd/121:02:251
2786779993727,6cyclictest251rcuc/120:16:201
2786779993725,8cyclictest2852505-21cpuspeed_turbos22:15:151
2786779993725,8cyclictest2852505-21cpuspeed_turbos22:15:151
2786779993725,8cyclictest2802064-21strings19:50:261
2786779993725,8cyclictest2802064-21strings19:50:261
2786779993724,9cyclictest26-21ksoftirqd/122:00:021
278677999372,31cyclictest1753-21gnome-shell19:26:541
2786779993722,11cyclictest2899525-21apt00:30:121
2786779993720,12cyclictest2888113-21awk23:55:291
2786779993720,12cyclictest2825469-21sshd20:57:261
2786779993719,14cyclictest2871493-21apt23:10:001
2786779993717,7cyclictest1069-21runrttasks19:21:581
278677599372,8cyclictest1753-21gnome-shell23:55:440
2786775993721,11cyclictest2847008-21sed22:00:040
2786775993720,12cyclictest438-20systemd-journal20:39:180
2786775993719,13cyclictest2793064-21systemd-udevd19:25:200
2786775993716,17cyclictest2846563-21apt-get22:00:010
162399370,8rtkit-daemon707415-21gmain19:40:181
278678499362,6cyclictest141rcu_preempt00:30:203
2786784993621,11cyclictest42-21ksoftirqd/323:13:093
2786784993621,11cyclictest2823694-21gpgv20:55:013
2786784993619,13cyclictest42-21ksoftirqd/321:25:043
278678499361,31cyclictest2805253-21hddtemp_smartct20:00:203
2786784993612,19cyclictest2827220-21sshd21:02:263
2786784993611,21cyclictest1753-21gnome-shell19:34:033
2786781993626,7cyclictest1753-21gnome-shell23:43:252
2786781993621,11cyclictest2874196-21ntpq23:15:242
2786781993621,11cyclictest2838796-21cut21:35:232
2786781993621,11cyclictest2798466-21sed19:40:222
2786781993620,11cyclictest438-20systemd-journal23:35:012
2786781993617,15cyclictest1753-21gnome-shell22:25:152
2786779993627,6cyclictest2813247-21cron20:25:001
2786779993627,6cyclictest251rcuc/120:40:491
2786779993626,7cyclictest1753-21gnome-shell20:48:531
2786779993625,7cyclictest2891040-21cpuspeed_turbos00:05:151
2786779993625,7cyclictest2891040-21cpuspeed_turbos00:05:141
2786779993623,9cyclictest1753-21gnome-shell21:53:191
2786779993622,11cyclictest2897558-21systemd00:25:021
2786779993622,10cyclictest855-21dbus-daemon20:30:001
2786779993621,11cyclictest2842335-21sh21:45:281
2786779993620,12cyclictest2851465-21python322:10:281
2786779993620,11cyclictest2873578-21cpuspeed_turbos23:15:131
2786779993619,13cyclictest1632-21dbus-daemon22:45:031
2786779993618,13cyclictest2849928-21cat22:05:321
278677599362,8cyclictest0-21swapper/022:22:100
2786775993623,8cyclictest2872502-21sh23:10:260
2786775993621,11cyclictest2840073-21expr21:40:150
2786775993620,12cyclictest2886378-21ntpq23:50:270
2786775993619,12cyclictest904-21systemd-logind00:31:500
2786775993618,13cyclictest1-21systemd23:05:000
28337882350,8chrt1753-21gnome-shell21:20:300
278678499353,5cyclictest2197014-21PK-Backend23:31:563
2786784993524,7cyclictest2809777-21sh20:15:003
2786784993523,8cyclictest1753-21gnome-shell00:02:243
2786784993519,12cyclictest2819951-21chrt20:42:063
2786784993519,12cyclictest2816007-21ntpq20:30:253
2786784993519,12cyclictest2816007-21ntpq20:30:243
2786784993518,12cyclictest438-20systemd-journal21:06:273
2786784993517,13cyclictest2898292-21kernelversion00:25:243
2786781993523,8cyclictest2842477-21grep21:45:322
2786781993521,10cyclictest2813837-21expr20:25:152
2786781993518,6cyclictest34-21ksoftirqd/223:21:282
2786779993526,6cyclictest251rcuc/120:00:501
2786779993521,10cyclictest1753-21gnome-shell19:45:191
2786779993520,11cyclictest1-21systemd19:15:011
2786779993519,12cyclictest2834322-21sshd21:24:321
2786775993523,8cyclictest2831344-21tr21:15:140
278677599352,30cyclictest0-21swapper/020:54:570
278677599352,13cyclictest0-21swapper/021:15:030
278677599352,10cyclictest1753-21gnome-shell20:16:390
278677599351,9cyclictest2197014-21PK-Backend21:01:440
278677599351,9cyclictest0-21swapper/022:10:060
2786775993519,12cyclictest2812022-21expr20:20:170
2786775993518,12cyclictest2862509-21apt-get22:45:010
2786775993517,13cyclictest2820914-21awk20:45:220
278677599351,30cyclictest0-21swapper/019:11:030
278677599351,14cyclictest2853452-21taskset22:15:490
278677599351,14cyclictest2853452-21taskset22:15:480
278677599351,14cyclictest0-21swapper/023:26:250
162399350,29rtkit-daemon0-21swapper/122:21:391
278678499344,5cyclictest141rcu_preempt22:40:263
278678499342,12cyclictest2884639-21ntp_states23:45:263
278678499342,12cyclictest2884639-21ntp_states23:45:263
278678199347,16cyclictest1753-21gnome-shell19:55:372
278678199342,6cyclictest141rcu_preempt23:25:492
2786781993419,11cyclictest2962348-21rs:main2
2786781993418,12cyclictest2833088-21cut21:20:162
2786781993417,12cyclictest994-21lldpd22:40:382
2786781993416,14cyclictest2855000-21cat22:20:312
2786779993420,10cyclictest2886297-21strings23:50:241
2786779993420,10cyclictest2865914-21awk22:50:321
2786779993416,14cyclictest26-21ksoftirqd/123:20:241
2786779993415,15cyclictest2871960-21cstates23:10:141
278677599343,7cyclictest141rcu_preempt20:44:300
278677599342,29cyclictest0-21swapper/023:17:150
2786775993419,11cyclictest2808592-21expr20:10:170
2786775993417,13cyclictest1753-21gnome-shell21:05:580
278677599341,5cyclictest141rcu_preempt19:20:280
2786775993412,11cyclictest2798896-21sed19:40:310
2786775993410,20cyclictest1738-21gdbus23:21:570
2786784993318,11cyclictest2820983-21grep20:45:233
2786781993321,8cyclictest2870881-21rt-features23:05:352
2786781993319,10cyclictest2817815-21cut20:35:302
2786775993322,7cyclictest2868016-21idleruntime-cro23:00:010
2786775993318,11cyclictest2882328-21grep23:40:160
2786775993317,12cyclictest2891918-21cat00:05:320
2786775993317,12cyclictest2891918-21cat00:05:320
2786775993316,13cyclictest2898277-21awk00:25:240
278677599331,5cyclictest2197014-21PK-Backend20:05:330
2786781993224,5cyclictest2197014-21PK-Backend20:45:492
2786781993218,10cyclictest2836524-21grep21:30:152
2786779993221,7cyclictest64850irq/126-i91523:40:231
2786779993217,11cyclictest2878885-21cstates23:30:161
2786775993218,10cyclictest2806008-21sshd20:01:160
2786775993216,12cyclictest2843355-21awk21:50:110
2786775993216,12cyclictest1-21systemd22:55:010
2786775993120,7cyclictest2801363-21/usr/sbin/munin19:50:240
2786775993120,7cyclictest2801363-21/usr/sbin/munin19:50:230
2786775993119,8cyclictest1753-21gnome-shell20:30:300
2786775993119,8cyclictest1753-21gnome-shell20:30:290
278677599311,4cyclictest2197014-21PK-Backend22:30:200
2786775993114,13cyclictest2870186-21grep23:05:190
278677599311,10cyclictest1358-1kworker/u9:123:49:540
278677599311,10cyclictest1358-1kworker/u9:123:49:540
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional