You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-10 - 22:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Sat Jan 10, 2026 12:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
197399840,4rtkit-daemon4497-21PK-Backend11:19:481
197399790,10rtkit-daemon2746-21gmain10:09:101
197399750,8rtkit-daemon1039-21gmain09:16:131
197399750,10rtkit-daemon2078-21gmain09:02:381
39974732730,5sleep00-21swapper/007:45:020
398465399643,21cyclictest4088587-21awk12:00:010
398450026442,15sleep30-21swapper/307:09:573
398445226444,14sleep20-21swapper/207:09:192
398438626345,13sleep00-21swapper/007:08:290
398420826343,14sleep10-21swapper/107:06:091
197399600,9rtkit-daemon1972-21rtkit-daemon10:35:581
197399590,9rtkit-daemon1972-21rtkit-daemon12:24:531
197399590,9rtkit-daemon1972-21rtkit-daemon11:38:081
197399590,9rtkit-daemon1972-21rtkit-daemon08:19:171
197399590,9rtkit-daemon1972-21rtkit-daemon07:55:381
40661712580,7sleep34497-21PK-Backend10:55:203
40661712580,7sleep34497-21PK-Backend10:55:203
3984665995724,11cyclictest4018305-21cron08:45:002
3984665995724,11cyclictest4018305-21cron08:45:002
197399570,8rtkit-daemon1972-21rtkit-daemon10:17:121
197399570,8rtkit-daemon1972-21rtkit-daemon08:54:391
197399570,7rtkit-daemon1972-21rtkit-daemon12:38:010
197399570,7rtkit-daemon1972-21rtkit-daemon07:25:480
197399560,9rtkit-daemon1972-21rtkit-daemon10:41:100
197399560,8rtkit-daemon1972-21rtkit-daemon11:00:430
197399560,8rtkit-daemon1972-21rtkit-daemon07:16:080
197399560,7rtkit-daemon1972-21rtkit-daemon09:17:330
197399560,7rtkit-daemon1972-21rtkit-daemon08:43:160
197399560,7rtkit-daemon1972-21rtkit-daemon08:43:160
197399550,8rtkit-daemon1972-21rtkit-daemon12:26:200
197399550,8rtkit-daemon1972-21rtkit-daemon12:26:190
197399550,8rtkit-daemon1972-21rtkit-daemon11:40:480
197399550,8rtkit-daemon1972-21rtkit-daemon09:10:530
197399550,8rtkit-daemon1972-21rtkit-daemon08:24:570
197399550,7rtkit-daemon1972-21rtkit-daemon12:19:580
197399540,8rtkit-daemon1972-21rtkit-daemon10:49:590
197399540,8rtkit-daemon1972-21rtkit-daemon07:10:251
197399540,7rtkit-daemon1972-21rtkit-daemon09:35:370
197399540,7rtkit-daemon1972-21rtkit-daemon07:50:080
197399540,7rtkit-daemon1972-21rtkit-daemon07:50:080
197399530,9rtkit-daemon1972-21rtkit-daemon11:25:351
197399530,9rtkit-daemon1972-21rtkit-daemon09:10:401
197399530,8rtkit-daemon1972-21rtkit-daemon07:46:151
197399520,8rtkit-daemon1972-21rtkit-daemon10:55:411
197399520,8rtkit-daemon1972-21rtkit-daemon10:55:411
197399520,6rtkit-daemon1972-21rtkit-daemon07:18:461
197399510,8rtkit-daemon1972-21rtkit-daemon08:25:491
197399510,6rtkit-daemon1972-21rtkit-daemon09:05:410
398465399504,19cyclictest3997578-21cpuspeed07:45:140
3984653995011,17cyclictest4065207-21apt-key10:55:010
3984653995010,18cyclictest4059447-21gpgv10:40:020
40316122490,8sleep234-21ksoftirqd/209:20:252
398465399498,19cyclictest3994029-21cpu07:35:130
40008112480,6sleep2331rcuc/207:55:032
40008112480,6sleep2331rcuc/207:55:032
398465399473,20cyclictest4078824-21cut11:30:230
398465399472,19cyclictest4046441-21cut10:05:120
398465399471,15cyclictest4101391-21sed12:35:010
40712932460,8sleep10-21swapper/111:10:151
398465399469,20cyclictest4034217-21cpu09:30:130
398465399468,16cyclictest4090685-21cpuspeed_turbos12:05:120
398465399464,16cyclictest4014273-21python308:30:300
398465399463,26cyclictest3993728-21munin-run07:35:010
398465399463,20cyclictest4049547-21find10:15:010
3984653994614,14cyclictest4015415-21cat08:35:130
398465399461,27cyclictest4031282-21memory09:20:190
398465399454,17cyclictest3986442-21grep07:15:020
398465399453,20cyclictest4037785-21cpuspeed_turbos09:40:110
398465399453,19cyclictest4050640-21sed10:15:210
40785632440,6sleep14497-21PK-Backend11:30:191
40103292440,18sleep234-21ksoftirqd/208:20:172
398465399445,17cyclictest4039355-21/usr/sbin/munin09:45:110
398465399445,17cyclictest4039355-21/usr/sbin/munin09:45:110
398465399445,15cyclictest4043359-21sed09:55:180
398465399443,16cyclictest4066421-21ntp_states10:55:240
398465399443,16cyclictest4066421-21ntp_states10:55:240
398465399443,15cyclictest4084388-21ntp_states11:45:230
398465399443,15cyclictest4080536-21hddtemp_smartct11:35:180
398465399442,20cyclictest4076537-21cut11:25:110
398465399440,27cyclictest4042572-21munin-run09:54:590
398465399440,27cyclictest4018987-21hddtemp_smartct08:45:180
40212712430,6chrt4497-21PK-Backend08:50:272
3984665994331,8cyclictest4008999-21wc08:15:262
398465399439,12cyclictest4023613-21latency_hist09:00:020
398465399435,9cyclictest4497-21PK-Backend09:05:010
398465399433,18cyclictest4057228-21anacron10:30:370
398465399433,18cyclictest4011873-21cstates08:25:150
398465399433,17cyclictest4093004-21chrt12:10:220
398465399433,17cyclictest3990357-21date07:25:010
398465399432,19cyclictest4004173-21apt08:05:000
398465399432,17cyclictest4032429-21awk09:25:120
398465399431,16cyclictest40012862expr07:55:140
197399430,35rtkit-daemon1941-21systemd08:00:431
40947162420,8chrt4497-21PK-Backend12:15:231
40552642420,6chrt0-21swapper/310:26:073
40552642420,6chrt0-21swapper/310:26:063
3984665994225,12cyclictest438-20systemd-journal07:28:572
3984658994230,7cyclictest26-21ksoftirqd/110:25:011
398465399427,14cyclictest4004409-21/usr/sbin/munin08:05:100
398465399424,17cyclictest4090531-21latency_hist12:05:020
398465399423,16cyclictest4044758-21tr10:00:160
398465399423,15cyclictest4071790-21python311:10:280
3984653994228,11cyclictest4074409-21cron11:20:000
197399420,35rtkit-daemon1941-21systemd08:32:211
197399420,34rtkit-daemon1941-21systemd08:13:471
197399420,33rtkit-daemon1941-21systemd09:39:441
3984665994124,12cyclictest438-20systemd-journal09:55:562
398465399414,15cyclictest4009676-21munin-run08:20:000
398465399412,16cyclictest4052748-21netstat10:20:220
197399410,34rtkit-daemon1941-21systemd10:14:131
197399410,34rtkit-daemon1941-21systemd08:45:551
197399410,34rtkit-daemon1941-21systemd07:44:081
398466899404,10cyclictest42-21ksoftirqd/311:05:153
3984668994024,11cyclictest438-20systemd-journal12:09:553
3984668994024,11cyclictest438-20systemd-journal10:15:573
3984668994024,11cyclictest3999384-21tr07:50:133
3984668994024,11cyclictest3999384-21tr07:50:133
3984668994023,12cyclictest438-20systemd-journal08:25:183
3984668994018,18cyclictest1054-21snmpd12:38:383
3984665994022,13cyclictest438-20systemd-journal09:27:552
398465399409,27cyclictest4097284-21chrt12:22:350
398465399405,13cyclictest4070322-21runrttasks11:06:000
197399400,33rtkit-daemon1941-21systemd12:07:401
398466899397,6cyclictest141rcu_preempt08:10:063
3984668993920,15cyclictest4087512-21seq11:55:183
398466599395,29cyclictest4052215-21grep10:20:132
3984665993928,7cyclictest4013137-21idleruntime-cro08:29:592
3984665993921,14cyclictest4059368-21dpkg10:39:592
3984665993921,14cyclictest4050382-21df_inode10:15:142
3984658993923,12cyclictest4082493-21memory11:40:211
3984668993823,11cyclictest4093663-21sshd12:13:543
3984668993820,6cyclictest42-21ksoftirqd/311:35:293
398466899381,16cyclictest4071287-21df_inode11:10:153
398466599382,32cyclictest4101030-40lsb_release12:32:032
398466599382,32cyclictest4064395-21ntp_states10:50:252
3984665993821,12cyclictest4094678-21cat12:15:212
3984658993823,5cyclictest26-21ksoftirqd/107:20:131
3984658993820,6cyclictest26-21ksoftirqd/112:30:151
398465399381,15cyclictest4006867-21if_enp2s008:10:170
3984668993721,13cyclictest4094715-21/usr/sbin/munin12:15:223
3984668993720,6cyclictest42-21ksoftirqd/310:51:463
3984668993718,15cyclictest365357-21update-manager09:39:273
3984665993721,12cyclictest4093174-21python312:10:282
3984665993720,13cyclictest4007553-40apt-check08:10:582
3984665993719,13cyclictest438-20systemd-journal09:50:312
398466599371,14cyclictest40469582head10:05:202
3984658993723,10cyclictest26-21ksoftirqd/112:25:321
3984658993723,10cyclictest26-21ksoftirqd/112:25:321
3984658993719,13cyclictest438-20systemd-journal11:05:001
398465399373,10cyclictest4076367-21cat11:25:010
197399370,29rtkit-daemon0-21swapper/107:35:491
40610302360,7chrt0-21swapper/310:41:003
398466899369,23cyclictest4044750-21grep10:00:163
3984668993625,7cyclictest3994182-21cpuspeed_turbos07:35:133
398466899362,11cyclictest3993140-21python307:30:293
3984668993621,11cyclictest42-21ksoftirqd/308:35:203
3984668993620,12cyclictest3990085-21sshd07:22:563
3984668993620,11cyclictest438-20systemd-journal11:30:003
398466899361,9cyclictest0-21swapper/309:04:383
3984668993619,12cyclictest4020319-21dpkg08:50:113
3984668993615,17cyclictest3997740-21expr07:45:153
398466899361,15cyclictest4017267-21/usr/sbin/munin08:40:213
398466899361,15cyclictest4017267-21/usr/sbin/munin08:40:213
398466599366,5cyclictest141rcu_preempt08:05:272
398466599362,6cyclictest4059345-21apt-get10:40:052
3984665993626,6cyclictest4040830-21sh09:49:592
3984665993626,6cyclictest4040830-21sh09:49:592
3984665993621,11cyclictest3989744-21sed07:20:302
3984665993620,12cyclictest3986210-21sshd07:13:562
3984665993620,11cyclictest438-20systemd-journal10:48:552
3984658993622,10cyclictest26-21ksoftirqd/109:45:301
3984658993622,10cyclictest26-21ksoftirqd/109:45:301
398465899361,8cyclictest0-21swapper/111:20:451
3984658993618,13cyclictest26-21ksoftirqd/112:10:141
398465399368,24cyclictest4086544-21chrt11:51:270
398465399365,9cyclictest13-21ksoftirqd/010:25:240
398465399365,9cyclictest13-21ksoftirqd/010:25:240
398465399361,10cyclictest4021791-21turbostat08:55:000
398466899355,26cyclictest4090323-21taskset12:05:013
398466899355,11cyclictest42-21ksoftirqd/307:42:203
3984668993525,7cyclictest0-21swapper/310:47:073
3984668993524,7cyclictest4051994-21cnf-update-db10:20:033
398466899351,9cyclictest0-21swapper/311:00:043
398466899351,9cyclictest0-21swapper/308:04:503
3984668993519,12cyclictest4002258-21sshd07:57:563
3984668993519,11cyclictest438-20systemd-journal10:07:563
3984668993518,12cyclictest3991047-21cut07:25:203
398466899351,30cyclictest4005567-21sshd08:05:563
398466899351,13cyclictest4032558-21tr09:25:143
398466599355,5cyclictest141rcu_preempt10:30:032
398466599355,5cyclictest141rcu_preempt10:30:032
398466599352,5cyclictest4497-21PK-Backend11:50:022
3984665993521,10cyclictest3986738-21ntp_kernel_pll_07:15:032
3984665993520,11cyclictest4049079-21cat10:10:282
3984665993520,11cyclictest4045097-21cat10:00:242
3984665993520,11cyclictest3992648-21entropy07:30:172
3984665993519,12cyclictest2193-21gdbus09:09:572
3984665993518,13cyclictest4019170-21memory08:45:202
3984665993517,14cyclictest4023883-21cat09:00:142
398466599351,19cyclictest40911992missed_timers12:05:212
3984658993519,12cyclictest4022397-21expr08:55:121
3984658993518,13cyclictest4026527-21cat09:05:321
197399350,28rtkit-daemon0-21swapper/111:57:091
3984668993423,7cyclictest4043311-21irqstats09:55:183
3984668993423,7cyclictest4022206-21tr08:55:133
3984668993422,8cyclictest4049634-21apt-config10:15:023
3984668993418,12cyclictest923-21dbus-daemon09:49:483
3984668993418,12cyclictest923-21dbus-daemon09:49:483
3984668993418,12cyclictest4009052-21awk08:15:253
3984668993417,12cyclictest438-20systemd-journal08:34:563
3984668993417,12cyclictest4099755-21mktemp12:30:043
3984668993416,14cyclictest4026270-21ntpq09:05:273
398466899341,13cyclictest40563302cstates10:30:153
398466899341,11cyclictest4096518-21cut12:20:193
3984668993411,12cyclictest4042098-21cat09:50:293
3984665993422,8cyclictest4015124-21apt08:35:112
3984665993421,10cyclictest4101727-21cpuspeed_turbos12:35:142
3984665993420,10cyclictest4087374-21expr11:55:132
3984665993420,10cyclictest34-21ksoftirqd/208:00:242
3984665993419,11cyclictest4038019-21entropy09:40:152
3984665993419,11cyclictest4034470-21expr09:30:152
3984665993419,11cyclictest3996090-21expr07:40:122
3984665993417,13cyclictest4067464-21grep11:00:132
3984665993416,13cyclictest4035977-21cat09:35:122
398465899341,9cyclictest4497-21PK-Backend11:48:461
3984658993419,11cyclictest4006176-21gpg-connect-age08:10:011
3984658993419,10cyclictest4102416-21cut12:35:281
3984658993418,12cyclictest4069906-21ntpq11:05:231
3984658993417,13cyclictest4086400-21cat11:50:291
3984658993417,13cyclictest3037-21gmain10:55:011
398465899341,14cyclictest4497-21PK-Backend09:33:351
398465899341,11cyclictest4497-21PK-Backend07:51:381
398465899341,11cyclictest4497-21PK-Backend07:51:381
3984668993326,4cyclictest4497-21PK-Backend11:44:013
3984668993318,11cyclictest4086263-21grep11:50:273
3984668993318,11cyclictest4074446-21pluginstate11:20:013
3984668993318,10cyclictest4030682-21/usr/sbin/munin09:20:093
3984668993316,12cyclictest466-21systemd-udevd10:37:403
398466899331,4cyclictest4497-21PK-Backend08:20:083
398466899331,28cyclictest4078351-21cat11:30:153
398466899331,27cyclictest4030621-21sed09:20:023
398466899331,12cyclictest0-21swapper/309:31:553
398466599336,5cyclictest141rcu_preempt12:00:212
398466599332,6cyclictest141rcu_preempt11:35:192
3984665993322,8cyclictest4070643-21ls11:10:002
3984665993321,8cyclictest4097306-40apt-check12:23:022
3984665993319,10cyclictest4071089-21grep11:10:142
3984665993319,10cyclictest4014843-21cron08:35:002
3984665993318,11cyclictest4023331-21sshd08:57:572
3984665993316,13cyclictest4056991-21python310:30:282
398466599331,5cyclictest141rcu_preempt11:15:242
3984665993315,14cyclictest4098531-21ntpq12:25:252
3984665993315,14cyclictest4098531-21ntpq12:25:252
398466599331,29cyclictest0-21swapper/209:15:392
398466599331,29cyclictest0-21swapper/207:58:062
3984658993323,5cyclictest26-21ksoftirqd/110:05:021
3984658993322,6cyclictest26-21ksoftirqd/109:20:101
398465899332,16cyclictest4497-21PK-Backend09:27:481
398465899332,16cyclictest4497-21PK-Backend08:36:411
398465899332,10cyclictest4055592-21apt-get10:30:061
398465899331,8cyclictest4497-21PK-Backend07:33:271
3984658993318,11cyclictest4062259-21sed10:45:211
3984658993318,11cyclictest4060175-21expr10:40:141
3984658993317,12cyclictest4088991-21tr12:00:111
3984658993317,12cyclictest4010656-21ntpq08:20:231
398465899331,29cyclictest4497-21PK-Backend09:50:561
197399330,26rtkit-daemon1051-21thermald09:56:341
398466899322,26cyclictest4019336-21ntp_states08:45:243
3984668993221,7cyclictest4099262-21latency_hist12:30:033
3984668993221,7cyclictest4099262-21latency_hist12:30:033
3984668993218,10cyclictest42-21ksoftirqd/309:40:103
3984668993218,10cyclictest42-21ksoftirqd/307:15:053
3984665993219,9cyclictest2069-21gnome-shell07:47:102
3984665993218,10cyclictest4078397-21expr11:30:142
3984665993218,10cyclictest4076390-21cat11:25:082
398466599321,29cyclictest0-21swapper/207:39:392
3984658993224,5cyclictest4497-21PK-Backend10:29:421
3984658993224,5cyclictest4497-21PK-Backend10:29:421
3984658993223,5cyclictest26-21ksoftirqd/109:40:221
40854982310,7sleep24497-21PK-Backend11:50:122
398466899312,9cyclictest3986355-21munin-node07:15:023
3984668993125,4cyclictest0-21swapper/311:49:393
3984668993117,10cyclictest4027829-21grep09:10:213
398466899311,26cyclictest4075161-21irqstats11:20:183
398466599313,5cyclictest141rcu_preempt11:00:012
398466599313,5cyclictest141rcu_preempt11:00:002
3984665993124,5cyclictest0-21swapper/211:40:082
3984665993119,8cyclictest1137-21runrttasks09:12:102
3984658993110,12cyclictest26-21ksoftirqd/108:40:121
3984658993110,12cyclictest26-21ksoftirqd/108:40:111
3984665993014,12cyclictest1054-21snmpd11:21:222
398465899291,9cyclictest141rcu_preempt07:25:301
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional