You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-02 - 08:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu Apr 02, 2026 00:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
139099970,80rtkit-daemon0-21swapper/219:05:252
139099970,80rtkit-daemon0-21swapper/219:05:252
391944427840,30sleep10-21swapper/119:05:171
391944427840,30sleep10-21swapper/119:05:161
391904226542,14sleep00-21swapper/019:05:120
391904226542,14sleep00-21swapper/019:05:110
391947226442,14sleep30-21swapper/319:05:193
391947226442,14sleep30-21swapper/319:05:183
40134912420,3sleep23056-21PK-Backend23:30:182
139099360,3rtkit-daemon0-21swapper/022:51:010
139099360,3rtkit-daemon0-21swapper/020:49:530
139099360,3rtkit-daemon0-21swapper/000:36:040
392067199316,23cyclictest3959637-21grep21:00:013
392067199311,7cyclictest3962953-21awk21:10:003
392066199305,11cyclictest13-21ksoftirqd/021:00:000
3920661993010,10cyclictest13-21ksoftirqd/022:25:000
392067199291,3cyclictest3979946-21python321:55:203
3920671992912,14cyclictest4024748-21cron00:05:003
3920661992917,8cyclictest1095-21runrttasks22:59:580
39424002280,4sleep234-21ksoftirqd/220:10:092
392067199289,17cyclictest3971626-21latency_hist21:35:003
392067199282,21cyclictest0-21swapper/319:35:093
392067199281,25cyclictest3941044-21df20:05:143
392067199280,6cyclictest3978848-21unin-run21:55:013
392067199280,25cyclictest3981185-21df_inode22:00:133
3920661992824,3cyclictest161rcuc/021:39:110
39634502270,3sleep30-21swapper/321:10:133
392067199279,15cyclictest3998318-21latency_hist22:50:013
392067199278,17cyclictest4005316-21date23:10:013
392067199271,3cyclictest4007306-21cnf-update-db23:15:023
3920671992711,15cyclictest3969450-21sed21:25:213
3920671992710,16cyclictest3989257-21ls22:25:013
3920671992710,15cyclictest3932706-21wc19:40:213
392066199277,7cyclictest13-21ksoftirqd/023:15:020
392066199271,3cyclictest13-21ksoftirqd/020:05:010
139099270,18rtkit-daemon0-21swapper/000:25:160
392067199267,17cyclictest4018498-21cat23:45:143
392067199267,17cyclictest3975622-21expr21:45:123
392067199266,18cyclictest3944354-21cat20:15:133
3920671992614,10cyclictest4025400-21diskstats00:05:123
3920671992610,14cyclictest4010368-21grep23:20:213
392066199265,7cyclictest13-21ksoftirqd/000:30:180
392066199261,3cyclictest13-21ksoftirqd/021:50:010
139099260,4rtkit-daemon1389-21rtkit-daemon20:04:001
392067199259,14cyclictest4034176-21cstates00:30:143
392067199258,16cyclictest3947183-21seq20:23:073
392067199257,16cyclictest4032549-21expr00:25:103
392067199252,4cyclictest0-21swapper/321:35:113
392067199251,4cyclictest0-21swapper/322:50:113
392067199251,11cyclictest3952815-21apt20:40:113
3920671992510,13cyclictest3928016-21sh19:29:593
392067199250,23cyclictest3990061-21taskset22:25:163
392067199250,22cyclictest4015423-21chrt23:35:203
139099250,4rtkit-daemon1389-21rtkit-daemon19:49:251
392067199249,14cyclictest3922572-21munin-node19:15:043
392067199248,14cyclictest42-21ksoftirqd/322:40:133
392067199248,14cyclictest0-21swapper/300:15:143
392067199248,14cyclictest0-21swapper/300:15:143
392067199241,4cyclictest436-20systemd-journal20:55:013
392067199240,3cyclictest4023051-21/usr/sbin/munin00:00:003
392066199247,4cyclictest13-21ksoftirqd/000:00:000
392066199241,4cyclictest13-21ksoftirqd/019:50:000
139099240,16rtkit-daemon0-21swapper/219:36:452
39478072230,4sleep13056-21PK-Backend20:25:121
392067199237,14cyclictest4026967-21cstates00:10:113
392067199237,14cyclictest3944086-21latency_hist20:15:013
392067199236,16cyclictest0-21swapper/322:19:503
392067199236,15cyclictest3947626-21cpu20:25:113
392067199234,3cyclictest0-21swapper/323:15:003
392067199233,6cyclictest3967007-21expr21:20:133
392067199231,8cyclictest3964949-21apt21:15:133
392067199231,6cyclictest4017827-21awk23:45:013
392067199231,4cyclictest3951947-21python320:35:213
392067199231,20cyclictest3934369-21ls19:45:183
3920671992310,11cyclictest4013010-21tr23:30:133
3920671992310,11cyclictest1095-21runrttasks20:30:123
392067199230,6cyclictest4037435-21apt-config00:40:023
392067199230,6cyclictest3954697-21cpuspeed_turbos20:45:113
392067199230,5cyclictest3922366-21taskset19:15:023
3920667992321,1cyclictest0-21swapper/221:36:182
3920667992320,2cyclictest3056-21PK-Backend20:15:202
3920667992320,2cyclictest0-21swapper/221:50:242
3920667992315,6cyclictest4027618-21python300:10:172
3920661992310,2cyclictest141rcu_preempt21:18:500
392066199230,22cyclictest0-21swapper/020:19:330
139099230,15rtkit-daemon0-21swapper/021:28:310
392067199229,11cyclictest3056-21PK-Backend22:35:443
392067199225,15cyclictest0-21swapper/319:50:223
392067199221,7cyclictest436-20systemd-journal21:45:003
3920671992217,4cyclictest3056-21PK-Backend20:02:433
392067199221,5cyclictest3959862-21cpuspeed_turbos21:00:123
392067199221,4cyclictest0-21swapper/322:10:113
392067199220,7cyclictest3984042-21systemd-run22:10:003
392067199220,7cyclictest3929498-21chrt19:31:533
392067199220,4cyclictest4003372-21munin-run23:05:003
392067199220,21cyclictest978-21snmpd23:50:353
3920667992220,1cyclictest0-21swapper/222:16:272
3920667992220,1cyclictest0-21swapper/221:43:212
3920667992220,1cyclictest0-21swapper/221:29:512
3920667992220,1cyclictest0-21swapper/221:03:482
3920667992220,1cyclictest0-21swapper/220:30:432
3920667992220,1cyclictest0-21swapper/200:21:322
392066799220,21cyclictest0-21swapper/220:39:472
3920665992215,5cyclictest436-20systemd-journal23:15:001
3920665992214,6cyclictest3920871-21apt19:10:121
392066199229,2cyclictest141rcu_preempt19:41:300
392066199225,4cyclictest13-21ksoftirqd/021:20:120
3920661992220,1cyclictest0-21swapper/023:25:350
3920661992218,3cyclictest161rcuc/022:32:490
3920661992218,3cyclictest161rcuc/022:32:490
3920661992218,3cyclictest161rcuc/021:01:110
392066199220,6cyclictest3056-21PK-Backend23:18:260
392066199220,2cyclictest3963614-21smartctl21:10:150
392066199220,21cyclictest0-21swapper/023:30:140
39933692210,3sleep0161rcuc/022:35:150
39468262210,5sleep03946829-21awk20:20:220
39310032210,1sleep00-21swapper/019:35:200
392067199216,13cyclictest42-21ksoftirqd/300:20:173
392067199215,3cyclictest3937428-21cstates19:55:133
392067199211,7cyclictest3926181-21runrttasks19:23:523
392067199211,5cyclictest4001607-21update-manager22:59:573
3920667992119,1cyclictest0-21swapper/223:42:522
3920667992119,1cyclictest0-21swapper/223:39:002
3920667992119,1cyclictest0-21swapper/223:11:572
3920667992119,1cyclictest0-21swapper/223:07:032
3920667992119,1cyclictest0-21swapper/222:51:362
3920667992119,1cyclictest0-21swapper/222:22:082
3920667992119,1cyclictest0-21swapper/221:21:452
3920667992119,1cyclictest0-21swapper/220:59:222
3920667992119,1cyclictest0-21swapper/220:48:052
3920667992119,1cyclictest0-21swapper/220:09:532
3920667992119,1cyclictest0-21swapper/220:02:292
3920667992119,1cyclictest0-21swapper/200:06:042
3920667992119,1cyclictest0-21swapper/200:01:112
3920667992118,2cyclictest3056-21PK-Backend22:05:322
392066799210,20cyclictest0-21swapper/222:40:132
392066799210,20cyclictest0-21swapper/221:48:352
392066799210,20cyclictest0-21swapper/221:16:182
392066799210,1cyclictest0-21swapper/219:47:562
3920665992119,1cyclictest3056-21PK-Backend00:38:301
3920665992118,2cyclictest3056-21PK-Backend00:05:181
3920665992117,3cyclictest26-21ksoftirqd/100:27:361
3920665992114,6cyclictest3938830-21sed20:00:011
3920665992110,9cyclictest3950811-21cron20:34:591
392066599210,20cyclictest3056-21PK-Backend23:24:081
392066199216,4cyclictest13-21ksoftirqd/021:30:170
392066199214,5cyclictest13-21ksoftirqd/022:20:010
3920661992119,1cyclictest0-21swapper/023:45:150
3920661992119,1cyclictest0-21swapper/023:24:330
3920661992119,1cyclictest0-21swapper/022:09:230
3920661992119,1cyclictest0-21swapper/022:01:320
3920661992119,1cyclictest0-21swapper/020:10:070
3920661992119,1cyclictest0-21swapper/019:56:480
3920661992119,1cyclictest0-21swapper/019:24:370
392066199210,6cyclictest4026502-21munin-run00:10:010
392066199210,20cyclictest0-21swapper/020:41:570
392066199210,20cyclictest0-21swapper/000:04:550
39321532200,2sleep23056-21PK-Backend19:40:142
392067199201,4cyclictest0-21swapper/322:30:173
392067199201,4cyclictest0-21swapper/322:30:173
392067199200,3cyclictest4011949-21python323:25:193
392066799208,10cyclictest3970351-21tr21:30:142
3920667992018,1cyclictest0-21swapper/223:46:442
3920667992018,1cyclictest0-21swapper/200:37:482
3920667992013,5cyclictest4008073-21ntp_kernel_pll_23:15:192
392066799200,19cyclictest0-21swapper/223:53:282
392066799200,19cyclictest0-21swapper/222:39:122
392066799200,19cyclictest0-21swapper/222:33:312
392066799200,19cyclictest0-21swapper/222:33:302
392066799200,19cyclictest0-21swapper/222:01:552
392066599208,2cyclictest141rcu_preempt23:35:091
3920665992018,1cyclictest0-21swapper/121:21:471
3920665992018,1cyclictest0-21swapper/119:17:021
3920665992017,2cyclictest1788598-21PressureMonitor00:02:271
3920665992013,5cyclictest436-20systemd-journal22:45:001
3920665992012,6cyclictest3989210-21cron22:24:591
3920665992011,7cyclictest3976817-21apt-get21:50:001
392066599200,1cyclictest3056-21PK-Backend00:33:031
392066599200,1cyclictest0-21swapper/121:41:551
392066599200,1cyclictest0-21swapper/100:16:271
392066599200,1cyclictest0-21swapper/100:16:271
392066599200,19cyclictest3056-21PK-Backend22:04:591
392066599200,19cyclictest3056-21PK-Backend21:19:101
392066599200,19cyclictest3056-21PK-Backend20:07:461
392066599200,19cyclictest0-21swapper/123:05:291
392066599200,19cyclictest0-21swapper/122:47:381
392066599200,19cyclictest0-21swapper/121:01:051
392066599200,19cyclictest0-21swapper/120:46:521
392066599200,19cyclictest0-21swapper/119:53:251
392066599200,19cyclictest0-21swapper/119:29:131
392066599200,18cyclictest3056-21PK-Backend20:15:011
3920661992018,1cyclictest0-21swapper/023:09:340
3920661992018,1cyclictest0-21swapper/022:26:470
3920661992018,1cyclictest0-21swapper/021:55:310
3920661992018,1cyclictest0-21swapper/021:06:510
3920661992018,1cyclictest0-21swapper/020:54:000
3920661992018,1cyclictest0-21swapper/020:25:480
3920661992018,1cyclictest0-21swapper/019:18:080
3920661992018,1cyclictest0-21swapper/000:17:190
3920661992018,1cyclictest0-21swapper/000:17:190
3920661992016,3cyclictest161rcuc/019:25:190
3920661992012,6cyclictest3936224-21ntp_states19:50:200
392066199201,13cyclictest1095-21runrttasks19:10:210
392066199200,1cyclictest0-21swapper/019:30:180
392066199200,19cyclictest0-21swapper/023:02:020
392066199200,19cyclictest0-21swapper/022:41:350
392066199200,11cyclictest13-21ksoftirqd/020:05:210
139099200,13rtkit-daemon0-21swapper/000:24:280
392066799192,15cyclictest3998187-21grep22:50:002
3920667991917,1cyclictest0-21swapper/223:28:262
3920667991917,1cyclictest0-21swapper/200:25:242
3920667991910,7cyclictest3922976-21df19:15:042
392066799190,1cyclictest3056-21PK-Backend21:10:022
392066799190,1cyclictest0-21swapper/223:02:102
392066799190,1cyclictest0-21swapper/222:27:492
392066799190,1cyclictest0-21swapper/220:42:382
392066799190,1cyclictest0-21swapper/219:29:582
392066799190,18cyclictest0-21swapper/223:57:192
392066799190,18cyclictest0-21swapper/223:23:332
392066799190,18cyclictest0-21swapper/222:55:272
392066799190,18cyclictest0-21swapper/219:22:152
3920665991917,1cyclictest3056-21PK-Backend23:59:371
3920665991917,1cyclictest3056-21PK-Backend20:52:401
3920665991917,1cyclictest3056-21PK-Backend20:42:261
3920665991917,1cyclictest3056-21PK-Backend20:36:591
3920665991917,1cyclictest0-21swapper/120:10:491
3920665991911,6cyclictest3983110-21cat22:05:201
392066599190,1cyclictest3056-21PK-Backend22:18:181
392066599190,1cyclictest3056-21PK-Backend19:33:051
392066599190,1cyclictest3056-21PK-Backend00:22:561
392066599190,1cyclictest0-21swapper/122:33:521
392066599190,1cyclictest0-21swapper/122:33:521
392066599190,1cyclictest0-21swapper/119:41:361
392066599190,18cyclictest3056-21PK-Backend22:12:501
392066599190,18cyclictest3056-21PK-Backend20:23:401
392066599190,18cyclictest0-21swapper/123:26:591
392066599190,18cyclictest0-21swapper/123:04:291
392066599190,18cyclictest0-21swapper/122:55:081
392066599190,18cyclictest0-21swapper/122:54:071
392066599190,18cyclictest0-21swapper/121:58:581
392066599190,18cyclictest0-21swapper/121:31:261
392066599190,18cyclictest0-21swapper/121:10:311
392066199197,5cyclictest13-21ksoftirqd/000:11:270
392066199192,6cyclictest13-21ksoftirqd/022:45:120
3920661991917,1cyclictest0-21swapper/023:42:240
3920661991917,1cyclictest0-21swapper/022:11:120
3920661991917,1cyclictest0-21swapper/021:52:550
3920661991917,1cyclictest0-21swapper/021:42:400
3920661991917,1cyclictest0-21swapper/020:38:180
3920661991917,1cyclictest0-21swapper/020:30:140
139099190,12rtkit-daemon3056-21PK-Backend22:38:511
3920667991816,1cyclictest0-21swapper/221:57:282
3920667991812,5cyclictest34-21ksoftirqd/222:10:112
3920665991816,1cyclictest3056-21PK-Backend23:41:591
3920665991816,1cyclictest3056-21PK-Backend21:52:081
3920665991816,1cyclictest0-21swapper/123:48:281
3920665991816,1cyclictest0-21swapper/121:35:051
3920665991811,6cyclictest3962190-21ntp_kernel_pll_21:05:181
392066199188,3cyclictest141rcu_preempt23:52:500
3920667991714,2cyclictest3935214-21dump-pmu-power19:54:562
3920667991713,2cyclictest4033972-21cpu00:30:132
3920667991712,4cyclictest2019-21gmain20:55:002
3920665991715,1cyclictest0-21swapper/123:17:391
3920665991711,4cyclictest3925617-21ps19:20:181
3920665991710,6cyclictest3931679-21date19:40:001
392066199170,4cyclictest3056-21PK-Backend23:35:090
3920667991614,1cyclictest3056-21PK-Backend20:25:152
3920667991612,2cyclictest34-21ksoftirqd/200:15:212
3920667991612,2cyclictest34-21ksoftirqd/200:15:202
392066799160,5cyclictest0-21swapper/219:30:112
3920665991614,1cyclictest3056-21PK-Backend23:32:411
3920665991611,3cyclictest4020181-21sh23:50:131
3920665991611,3cyclictest1095-21runrttasks22:28:261
392066599161,12cyclictest3968426-21apt21:25:111
3920667991513,1cyclictest0-21swapper/220:21:362
392066799150,5cyclictest3056-21PK-Backend20:00:012
3920665991513,1cyclictest0-21swapper/120:56:521
3920667991410,2cyclictest34-21ksoftirqd/221:05:182
392066799140,3cyclictest3056-21PK-Backend19:10:202
392066599139,3cyclictest3056-21PK-Backend00:14:081
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional