You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-02 - 20:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Fri Jan 02, 2026 12:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
356880322880,8sleep00-21swapper/007:05:190
362235221170,5sleep14497-21PK-Backend09:35:181
360496521080,5sleep20-21swapper/208:45:252
36680202830,7sleep2331rcuc/211:45:212
36762992780,7sleep23676298-21turbostat12:10:002
356955427740,30sleep20-21swapper/207:07:152
36557862760,7sleep14497-21PK-Backend11:10:151
356960027439,13sleep10-21swapper/107:07:501
36715722650,10sleep0161rcuc/011:55:210
356959726444,14sleep30-21swapper/307:07:483
3569957996213,22cyclictest3654823-21taskset11:05:292
3569957996021,34cyclictest3673125-21expr12:00:152
3569957995813,18cyclictest3590604-21cat08:05:212
3569957995621,30cyclictest3625546-21grep09:45:142
3569957995518,33cyclictest3597725-21open_files08:25:252
3569957995512,15cyclictest3571635-21idleruntime-cro07:15:002
3569957995412,17cyclictest3615582-21cat09:15:292
3569957995219,28cyclictest3618415-21sort09:25:022
3569957995218,29cyclictest438-20systemd-journal12:13:392
3569957995218,29cyclictest3683020-21timerandwakeup12:25:302
3569957995120,27cyclictest3613279-21expr09:10:142
3569957995120,26cyclictest3634044-21latency_hist10:10:032
3569957995114,15cyclictest3596033-21ps08:20:282
3569957995018,28cyclictest3681283-21ls12:20:302
3569957995013,15cyclictest3585060-21cstates07:50:142
3569957994913,15cyclictest3584012-21cat07:45:322
197399490,34rtkit-daemon3672705-21cat12:00:023
35869702480,6sleep0161rcuc/007:55:160
3569957994833,12cyclictest0-21swapper/210:20:112
3569957994824,20cyclictest3685286-21unin-run12:35:012
3569957994824,20cyclictest3685286-21unin-run12:35:012
3569957994820,24cyclictest3648308-21basename10:50:002
3569957994820,24cyclictest3611069-21latency_hist09:05:022
3569957994819,25cyclictest923-21dbus-daemon08:56:282
3569957994819,24cyclictest3664455-21expr11:35:152
3569957994818,26cyclictest3659577-21missed_timers11:20:222
197399480,31rtkit-daemon3600159-40apt-check08:31:540
36289812470,7sleep04497-21PK-Backend09:55:140
3569957994720,23cyclictest3621756-21grep09:35:012
3569957994720,22cyclictest1137-21runrttasks10:50:252
3569957994718,24cyclictest3606151-21cpuspeed_turbos08:50:142
3569957994621,21cyclictest3641212-21cut10:30:142
3569957994621,20cyclictest3629403-21cat09:55:202
3569957994621,20cyclictest3603052-21grep08:40:222
3569957994621,20cyclictest3599039-21/usr/sbin/munin08:30:122
3569957994620,22cyclictest3650704-21/usr/sbin/munin10:55:182
3569957994619,23cyclictest3619080-21missed_timers09:25:232
3569957994617,26cyclictest3692992-21snapd07:15:252
3569957994614,28cyclictest0-21swapper/210:40:162
3569957994611,32cyclictest0-21swapper/208:03:022
3569957994611,31cyclictest0-21swapper/212:16:232
197399460,31rtkit-daemon3654493-21ntp_states11:05:243
356995799454,22cyclictest0-21swapper/211:01:542
3569957994522,18cyclictest3636169-21expr10:15:162
3569957994519,21cyclictest3669686-21expr11:50:142
36446232440,5sleep1251rcuc/110:40:011
3569957994422,17cyclictest3666719-21awk11:40:262
3569957994422,17cyclictest3574495-21/usr/sbin/munin07:20:172
3569957994421,19cyclictest3634695-21expr10:10:212
3569957994421,18cyclictest3591952-21grep08:10:122
3569957994420,20cyclictest3655845-21/usr/sbin/munin11:10:162
3569957994420,20cyclictest3587292-21ntpq07:55:222
3569957994420,19cyclictest438-20systemd-journal10:39:452
3569957994419,20cyclictest3611587-21chrt09:05:172
35929542430,6sleep10-21swapper/108:11:391
3569961994316,23cyclictest3605690-21perf08:50:003
3569957994321,17cyclictest438-20systemd-journal09:40:012
3569957994321,17cyclictest3580347-21wc07:35:312
3569957994320,18cyclictest3594491-21wc08:15:352
3569957994320,18cyclictest3594491-21wc08:15:342
3569957994319,20cyclictest3660939-21expr11:25:142
3569957994313,27cyclictest0-21swapper/209:50:292
3569957994313,26cyclictest0-21swapper/212:35:192
3569957994312,27cyclictest0-21swapper/211:30:142
35768112420,6sleep30-21swapper/307:25:293
3569957994222,16cyclictest3623989-21cstates09:40:142
3569957994220,17cyclictest3631198-21cat10:00:192
3569957994212,27cyclictest0-21swapper/208:35:152
3569951994227,11cyclictest923-21dbus-daemon08:56:260
197399420,28rtkit-daemon4497-21PK-Backend11:15:273
197399420,28rtkit-daemon4497-21PK-Backend09:24:091
3569961994113,24cyclictest3636205-21df_inode10:15:173
3569957994122,15cyclictest3582008-21cat07:40:282
3569957994120,17cyclictest3671076-21grep11:55:142
3569957994120,17cyclictest3576149-21cat07:25:142
3569957994120,16cyclictest3639272-21cat10:25:092
3569957994119,18cyclictest3658537-21chrt11:18:292
356995499416,10cyclictest0-21swapper/109:15:271
3569951994120,16cyclictest3667931-21cstates11:45:160
356996199409,27cyclictest1036-21gdbus08:19:563
356996199409,27cyclictest1036-21gdbus08:19:563
3569961994019,17cyclictest3676322-21unin-run12:10:013
3569957994021,14cyclictest3578156-21taskset07:30:232
3569954994027,9cyclictest3579448-21tr07:35:121
3569951994024,11cyclictest3665804-21sed11:40:020
197399400,29rtkit-daemon4497-21PK-Backend10:23:090
197399400,27rtkit-daemon4497-21PK-Backend08:37:341
3569961993928,8cyclictest2180-21evolution-addre10:09:473
356996199392,16cyclictest0-21swapper/308:25:003
3569961993916,19cyclictest42-21ksoftirqd/308:10:103
3569961993912,23cyclictest3648671-21cpuspeed_turbos10:50:143
3569954993922,13cyclictest3609823-21df09:00:151
3569951993927,7cyclictest3655884-21hddtemp_smartct11:10:200
3569951993915,12cyclictest3603712-21systemd-udevd08:42:330
197399390,27rtkit-daemon4497-21PK-Backend07:29:251
197399390,26rtkit-daemon4497-21PK-Backend08:05:441
36465612380,7sleep13646565-21sh10:45:001
3569961993828,7cyclictest1-21systemd08:32:393
3569961993824,10cyclictest3579017-21cron07:35:003
3569961993823,11cyclictest3580154-21awk07:35:283
3569961993822,11cyclictest3608869-21systemd-udevd08:56:283
356996199381,9cyclictest3603720-21systemd-udevd08:42:333
3569961993819,13cyclictest3606489-21cat08:50:213
3569954993812,5cyclictest141rcu_preempt09:50:211
356995199389,13cyclictest3685257-21cron12:35:010
356995199389,13cyclictest3685257-21cron12:35:010
3569951993827,7cyclictest3686980-21sh12:40:000
356995199382,20cyclictest3653546-21latency_hist11:05:010
356995199381,15cyclictest36269112latency_hist09:50:020
197399380,27rtkit-daemon4497-21PK-Backend10:16:480
197399380,27rtkit-daemon4497-21PK-Backend10:02:480
197399380,26rtkit-daemon4497-21PK-Backend09:58:291
197399380,26rtkit-daemon4497-21PK-Backend09:32:470
197399380,26rtkit-daemon4497-21PK-Backend07:37:330
197399380,26rtkit-daemon0-21swapper/008:05:520
197399380,25rtkit-daemon0-21swapper/009:14:290
356996199376,27cyclictest3643127-21cat10:35:143
3569961993726,7cyclictest3627272-21tr09:50:143
3569961993724,8cyclictest3650459-21grep10:55:133
356996199372,31cyclictest3617527-21apt-get09:20:383
3569961993723,10cyclictest3632049-21cron10:05:003
356996199372,30cyclictest3684545-21ntp_states12:30:253
356996199372,30cyclictest3684545-21ntp_states12:30:243
3569961993722,11cyclictest3681339-21cat12:20:313
3569961993721,11cyclictest438-20systemd-journal10:45:003
3569961993720,13cyclictest3682813-21ntp_states12:25:263
3569961993720,12cyclictest3582540-21apt-get07:45:003
3569961993719,14cyclictest3667962-21df11:45:173
3569961993719,13cyclictest438-20systemd-journal09:10:003
3569961993719,13cyclictest3666591-21ntp_kernel_pll_11:40:233
3569954993720,13cyclictest3620966-21ntp_states09:30:261
356995499371,32cyclictest3597923-21wc08:25:291
3569951993720,13cyclictest3675554-21awk12:05:250
197399370,26rtkit-daemon4497-21PK-Backend10:47:400
197399370,26rtkit-daemon4497-21PK-Backend07:43:460
197399370,26rtkit-daemon0-21swapper/108:47:001
197399370,25rtkit-daemon4497-21PK-Backend10:58:411
197399370,25rtkit-daemon4497-21PK-Backend09:13:001
197399370,25rtkit-daemon4497-21PK-Backend07:33:181
197399370,25rtkit-daemon0-21swapper/010:28:160
197399370,25rtkit-daemon0-21swapper/010:06:200
197399370,24rtkit-daemon4497-21PK-Backend08:51:301
197399370,24rtkit-daemon0-21swapper/109:44:401
197399370,24rtkit-daemon0-21swapper/010:10:210
197399370,24rtkit-daemon0-21swapper/007:45:320
197399370,23rtkit-daemon0-21swapper/009:07:010
35780192360,7sleep04497-21PK-Backend07:30:210
356996199363,12cyclictest3673382-21memory12:00:223
356996199362,12cyclictest438-20systemd-journal09:38:393
3569961993620,12cyclictest3623809-21expr09:40:133
3569961993620,11cyclictest3598545-21gpgv08:30:013
3569961993619,13cyclictest3659530-21memory11:20:223
3569961993619,12cyclictest3601174-21awk08:35:173
356996199361,16cyclictest3638346-21/usr/sbin/munin10:20:263
3569954993629,5cyclictest0-21swapper/107:42:381
3569954993621,11cyclictest923-21dbus-daemon10:33:311
3569954993620,12cyclictest3670062-21ntpq11:50:241
356995499361,9cyclictest4497-21PK-Backend07:18:141
356995499361,31cyclictest36197722sleep109:28:011
3569951993619,12cyclictest3595287-21awk08:20:140
197399360,25rtkit-daemon4497-21PK-Backend07:55:481
197399360,24rtkit-daemon4497-21PK-Backend08:43:171
197399360,24rtkit-daemon0-21swapper/110:20:051
197399360,24rtkit-daemon0-21swapper/107:51:411
197399360,24rtkit-daemon0-21swapper/008:11:350
35891942350,9sleep04497-21PK-Backend08:00:280
356996199358,23cyclictest3634383-21expr10:10:143
3569961993523,8cyclictest3642740-21latency_hist10:35:023
356996199352,29cyclictest3679372-21users12:15:323
3569961993519,12cyclictest3662518-21cat11:30:133
3569961993519,12cyclictest3661576-21sed11:25:293
3569961993519,12cyclictest3616125-21/usr/sbin/munin09:20:013
3569961993517,13cyclictest438-20systemd-journal08:05:003
356996199351,29cyclictest3585719-21python307:50:293
356995499351,9cyclictest4497-21PK-Backend11:19:081
356995499351,9cyclictest4497-21PK-Backend07:22:571
3569954993519,12cyclictest3686862-21sshd12:38:391
3569954993518,13cyclictest3664868-21ntpq11:35:251
3569954993518,12cyclictest3625780-21diskstats09:45:181
3569951993519,12cyclictest3650679-21forks10:55:170
356995199351,30cyclictest3619947-21turbostat09:30:020
197399350,24rtkit-daemon4497-21PK-Backend07:10:491
197399350,24rtkit-daemon0-21swapper/010:39:350
197399350,24rtkit-daemon0-21swapper/009:22:230
197399350,24rtkit-daemon0-21swapper/008:55:000
197399350,22rtkit-daemon0-21swapper/007:19:400
3569961993426,5cyclictest0-21swapper/309:46:543
3569961993422,8cyclictest3670071-21ntp_states11:50:253
356996199342,28cyclictest3656251-21/usr/sbin/munin11:10:273
3569961993422,8cyclictest3647200-21df_inode10:45:163
3569961993422,8cyclictest3570255-21tr07:10:153
356996199342,26cyclictest3664539-21hddtemp_smartct11:35:203
3569961993419,12cyclictest3609790-21expr09:00:143
3569961993419,11cyclictest42-21ksoftirqd/310:25:113
3569961993419,11cyclictest3677340-21awk12:10:273
3569961993419,11cyclictest3588028-21cron08:00:003
3569961993418,11cyclictest3583068-21sort07:45:133
3569961993416,14cyclictest3575189-21users07:20:303
3569961993414,16cyclictest42-21ksoftirqd/309:30:143
356996199341,12cyclictest3629729-21sensors_temp09:55:273
3569954993426,5cyclictest4497-21PK-Backend10:54:011
3569954993424,6cyclictest3659088-21tr11:20:141
356995499342,27cyclictest3595603-21df_inode08:20:161
356995499341,9cyclictest4497-21PK-Backend12:07:091
356995499341,9cyclictest4497-21PK-Backend10:06:321
356995499341,9cyclictest4497-21PK-Backend09:08:371
3569954993418,11cyclictest3588901-21grep08:00:221
3569954993416,13cyclictest3666581-21ntpq11:40:231
356995499341,29cyclictest4497-21PK-Backend12:32:321
356995499341,29cyclictest4497-21PK-Backend12:32:321
356995499341,29cyclictest4497-21PK-Backend12:27:501
356995499341,29cyclictest4497-21PK-Backend08:16:251
356995499341,29cyclictest4497-21PK-Backend08:16:251
356995499341,10cyclictest4497-21PK-Backend11:57:391
356995499341,10cyclictest4497-21PK-Backend07:46:351
3569951993419,11cyclictest3648977-21sort10:50:200
3569951993418,12cyclictest3657607-21fw_packets11:15:190
3569951993418,12cyclictest3654135-21cat11:05:170
3569951993417,12cyclictest3585601-21ntp_states07:50:260
356995199341,30cyclictest4497-21PK-Backend11:26:350
356995199341,29cyclictest4497-21PK-Backend12:26:190
197399340,23rtkit-daemon0-21swapper/007:22:200
3569961993325,5cyclictest0-21swapper/311:00:053
3569961993324,6cyclictest3692992-21snapd09:25:403
3569961993318,11cyclictest42-21ksoftirqd/309:10:163
356996199331,11cyclictest3573228-21rt-features07:15:283
356995499331,9cyclictest4497-21PK-Backend10:48:571
356995499331,8cyclictest4497-21PK-Backend12:18:151
3569954993318,11cyclictest3681790-21cron12:25:001
356995499331,29cyclictest0-21swapper/108:32:031
356995499331,28cyclictest4497-21PK-Backend08:56:081
356995199331,5cyclictest4497-21PK-Backend11:31:350
356995199331,29cyclictest4497-21PK-Backend11:23:090
356995199331,29cyclictest4497-21PK-Backend10:32:290
356995199331,29cyclictest0-21swapper/007:12:580
3569961993220,8cyclictest3686108-21memory12:35:203
3569954993219,9cyclictest3640887-21sed10:30:021
356995499321,4cyclictest0-21swapper/110:03:031
3569954993213,5cyclictest141rcu_preempt11:00:201
3569951993225,4cyclictest4497-21PK-Backend12:20:040
3569951993218,10cyclictest3645925-21smartctl10:40:280
3569951993217,11cyclictest3672992-21cstates12:00:140
356995199321,4cyclictest0-21swapper/012:13:300
356995199321,4cyclictest0-21swapper/009:40:160
356995199321,4cyclictest0-21swapper/007:25:100
3569961993115,12cyclictest3590387-21expr08:05:143
356995499312,9cyclictest26-21ksoftirqd/112:10:271
3569954993125,4cyclictest0-21swapper/110:19:011
356995499311,4cyclictest4497-21PK-Backend10:10:331
356995499311,28cyclictest0-21swapper/111:33:061
3569951993117,10cyclictest3627756-21grep09:50:220
3569951993114,14cyclictest3622562-21cat09:35:210
3569951993114,13cyclictest2345-21gdbus08:39:560
356995199311,11cyclictest3670784-21cron11:55:000
356995499301,11cyclictest26-21ksoftirqd/111:26:251
356995199306,6cyclictest141rcu_preempt11:45:000
3569951993020,5cyclictest13-21ksoftirqd/008:26:310
3569951993015,11cyclictest3593906-21hddtemp_smartct08:15:230
3569951993015,11cyclictest3593906-21hddtemp_smartct08:15:230
356995199301,11cyclictest438-20systemd-journal09:20:000
3569954992918,8cyclictest4497-21PK-Backend11:07:501
3569954992916,9cyclictest3669063-21turbostat.cron11:50:011
356995199298,5cyclictest36109392awk09:05:000
3569951992919,6cyclictest3679750-21latency_hist12:20:010
356995199281,4cyclictest4497-21PK-Backend08:46:500
356995499271,12cyclictest4497-21PK-Backend12:02:271
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional