You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-02 - 12:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Fri Jan 02, 2026 00:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
327975227840,13sleep00-21swapper/019:05:070
33150102750,6sleep342-21ksoftirqd/320:40:303
328026827147,17sleep30-21swapper/319:05:223
33937312670,5chrt0-21swapper/300:25:223
33937312670,5chrt0-21swapper/300:25:223
328108226640,20sleep20-21swapper/219:08:062
328089426444,14sleep10-21swapper/119:05:391
328136199600,56cyclictest1972-21rtkit-daemon23:13:490
3281367995819,19cyclictest34-21ksoftirqd/220:20:002
3281365995814,31cyclictest1033-21gdbus21:24:401
3281365995814,31cyclictest1033-21gdbus21:24:391
3281365995812,38cyclictest3360643-21seq22:51:061
33865572560,17sleep234-21ksoftirqd/200:05:172
3281365995620,30cyclictest3289764-21awk19:30:261
3281365995613,39cyclictest3389370-21perf00:15:001
3281367995528,15cyclictest34-21ksoftirqd/223:50:232
328136599559,38cyclictest3382335-21seq23:53:561
3281365995521,28cyclictest3373776-21systemd-udevd23:30:131
3281365995512,36cyclictest3311985-21seq20:33:261
3281365995512,36cyclictest3311985-21seq20:33:261
197399550,9rtkit-daemon1972-21rtkit-daemon21:43:131
197399550,9rtkit-daemon1972-21rtkit-daemon21:22:080
197399550,9rtkit-daemon1972-21rtkit-daemon21:22:080
197399550,9rtkit-daemon1972-21rtkit-daemon19:52:151
197399540,9rtkit-daemon1972-21rtkit-daemon22:35:101
197399540,8rtkit-daemon1972-21rtkit-daemon19:37:030
3281367995327,14cyclictest34-21ksoftirqd/223:55:322
3281367995313,22cyclictest34-21ksoftirqd/223:30:162
3281365995315,32cyclictest3316314-21cat20:45:201
3281361995345,6cyclictest161rcuc/020:00:000
197399530,8rtkit-daemon1972-21rtkit-daemon21:08:541
197399530,8rtkit-daemon1972-21rtkit-daemon20:20:291
197399530,8rtkit-daemon1972-21rtkit-daemon00:25:451
197399530,8rtkit-daemon1972-21rtkit-daemon00:25:451
3281365995216,29cyclictest1137-21runrttasks23:06:561
3281365995215,29cyclictest3364775-21latency_hist23:05:011
197399520,8rtkit-daemon1972-21rtkit-daemon21:11:491
197399520,8rtkit-daemon1972-21rtkit-daemon20:42:381
197399520,8rtkit-daemon1972-21rtkit-daemon20:35:071
197399520,8rtkit-daemon1972-21rtkit-daemon19:56:351
3281367995116,18cyclictest34-21ksoftirqd/222:40:232
3281361995142,6cyclictest161rcuc/020:50:030
197399510,6rtkit-daemon1972-21rtkit-daemon21:56:230
197399510,6rtkit-daemon1972-21rtkit-daemon21:00:460
3281371995041,6cyclictest411rcuc/321:40:273
3281371995017,6cyclictest141rcu_preempt00:20:273
3281367995019,10cyclictest34-21ksoftirqd/221:45:322
3281365995015,29cyclictest3349569-21kernelversion22:20:211
3281361995041,6cyclictest161rcuc/023:57:360
328136199502,12cyclictest3396939-21seq00:35:180
3281371994940,6cyclictest411rcuc/323:40:033
3281371994927,7cyclictest42-21ksoftirqd/323:55:113
328136599493,40cyclictest3389699-21sort00:15:121
328136199499,14cyclictest3294792-21expr19:45:160
328136199498,11cyclictest13-21ksoftirqd/020:10:290
33648412480,6sleep34497-21PK-Backend23:05:013
3281367994816,16cyclictest34-21ksoftirqd/222:55:252
3281367994815,18cyclictest34-21ksoftirqd/219:15:042
3281367994812,19cyclictest34-21ksoftirqd/219:29:592
3281365994824,20cyclictest3327767-21unin-run21:20:001
3281365994819,25cyclictest3338960-21sed21:50:211
3281365994814,30cyclictest3331152-21seq21:29:091
3281365994813,29cyclictest3397253-21netstat00:35:251
328136199487,12cyclictest13-21ksoftirqd/021:30:000
328136199486,18cyclictest13-21ksoftirqd/023:00:250
3281361994816,8cyclictest0-21swapper/022:05:190
3281361994816,8cyclictest0-21swapper/022:05:180
3281361994812,18cyclictest13-21ksoftirqd/021:40:020
33424962470,14sleep3411rcuc/322:00:233
328137199475,17cyclictest42-21ksoftirqd/323:39:093
3281371994726,7cyclictest42-21ksoftirqd/320:24:083
328136799478,14cyclictest34-21ksoftirqd/220:25:252
328136799477,12cyclictest34-21ksoftirqd/220:05:162
3281367994710,13cyclictest34-21ksoftirqd/223:50:002
328136599479,32cyclictest438-20systemd-journal20:55:001
328136599478,34cyclictest3305558-21expr20:15:151
3281365994715,26cyclictest3344370-21sh22:05:281
3281365994715,26cyclictest3344370-21sh22:05:281
3281365994715,26cyclictest3283256-21fw_packets19:15:021
3281365994712,31cyclictest3391444-21cpuspeed00:20:141
3281365994712,30cyclictest3340368-21cat21:55:151
3281371994621,22cyclictest3369994-21idleruntime-cro23:20:003
3281367994618,9cyclictest34-21ksoftirqd/219:23:232
3281367994613,18cyclictest34-21ksoftirqd/222:19:562
3281367994613,17cyclictest34-21ksoftirqd/220:00:392
3281367994612,15cyclictest34-21ksoftirqd/200:12:202
3281367994611,19cyclictest34-21ksoftirqd/219:35:242
328136599468,32cyclictest3342497-21cat22:00:221
3281365994615,27cyclictest3292421-21/usr/sbin/munin19:40:131
3281365994613,29cyclictest3321973-21switchtime21:00:331
3281365994611,30cyclictest3303098-21latency_hist20:10:021
3281365994610,32cyclictest3295971-21systemd-run19:50:011
328136199465,14cyclictest13-21ksoftirqd/020:15:020
3281361994614,7cyclictest13-21ksoftirqd/021:45:160
3281361994613,11cyclictest13-21ksoftirqd/020:00:300
3281361994611,19cyclictest13-21ksoftirqd/023:35:240
3281361994610,9cyclictest13-21ksoftirqd/022:20:040
3281367994519,8cyclictest3390738-21taskset00:15:322
3281367994513,15cyclictest34-21ksoftirqd/223:40:132
3281367994510,19cyclictest34-21ksoftirqd/200:35:232
328136599458,33cyclictest3384874-21basename00:00:221
328136599458,33cyclictest3384874-21basename00:00:221
328136599456,11cyclictest3334290-21ls21:35:321
328136599455,36cyclictest1137-21runrttasks21:34:531
3281365994518,23cyclictest3387041-21gunzip00:05:271
3281365994513,26cyclictest3377580-21df23:40:161
3281365994510,31cyclictest3286697-21seq19:21:321
328136199456,9cyclictest13-21ksoftirqd/023:00:010
328136199451,21cyclictest13-21ksoftirqd/021:44:590
3281361994512,12cyclictest13-21ksoftirqd/020:20:310
328136199451,12cyclictest3348720-21apt-config22:20:010
3281361994511,10cyclictest13-21ksoftirqd/020:45:010
328137199442,20cyclictest42-21ksoftirqd/323:20:033
3281367994410,15cyclictest34-21ksoftirqd/222:30:002
3281367994410,15cyclictest34-21ksoftirqd/222:29:592
328136599449,30cyclictest3366882-21cut23:10:151
328136599448,31cyclictest3372789-21processes23:25:301
328136599448,31cyclictest3290752-21cpu19:35:131
328136599447,32cyclictest3371833-21sed23:25:011
3281365994415,24cyclictest26-21ksoftirqd/123:15:291
3281365994414,26cyclictest1137-21runrttasks22:19:481
3281365994414,25cyclictest3287697-21if_enp2s019:25:191
3281365994413,27cyclictest3394867-21cpu00:30:141
3281365994412,27cyclictest3337663-21timerandwakeup21:45:301
3281365994411,29cyclictest3383419-21ntpq23:55:261
3281365994410,30cyclictest3376206-21ntpq23:35:251
328136199449,22cyclictest13-21ksoftirqd/019:15:290
328136199447,20cyclictest13-21ksoftirqd/020:05:170
328136199447,18cyclictest13-21ksoftirqd/023:15:270
328136199444,10cyclictest924-21NetworkManager23:40:110
328136199442,17cyclictest3309918-21cat20:25:330
3281361994417,8cyclictest13-21ksoftirqd/020:45:210
3281361994416,8cyclictest13-21ksoftirqd/023:47:060
3281361994414,9cyclictest13-21ksoftirqd/000:30:000
3281361994414,9cyclictest13-21ksoftirqd/000:30:000
3281361994413,8cyclictest13-21ksoftirqd/020:55:260
3281361994413,8cyclictest13-21ksoftirqd/020:55:250
328136199441,12cyclictest13-21ksoftirqd/022:10:190
328136799433,20cyclictest0-21swapper/220:58:472
328136799433,20cyclictest0-21swapper/220:58:472
3281367994313,6cyclictest141rcu_preempt23:20:002
3281367994311,16cyclictest34-21ksoftirqd/221:20:532
3281367994311,16cyclictest34-21ksoftirqd/221:20:532
3281367994310,17cyclictest34-21ksoftirqd/221:40:242
3281367994310,15cyclictest34-21ksoftirqd/222:46:402
328136599439,29cyclictest3350881-21cpu22:25:141
328136599439,29cyclictest3350881-21cpu22:25:141
328136599439,29cyclictest3300129-21if_err_enp2s020:00:191
3281365994316,22cyclictest3319561-21cstates20:55:151
3281365994316,22cyclictest3319561-21cstates20:55:151
3281365994311,27cyclictest3356236-21cstates22:40:141
328136199437,9cyclictest13-21ksoftirqd/023:30:270
328136199436,9cyclictest13-21ksoftirqd/023:10:010
328136199432,15cyclictest3281543-21runrttasks19:10:100
3281361994317,7cyclictest141rcu_preempt23:25:020
328136199431,18cyclictest13-21ksoftirqd/021:05:270
328136199431,14cyclictest3387886-21/usr/sbin/munin00:10:110
328136199431,13cyclictest13-21ksoftirqd/000:35:020
328136199431,12cyclictest13-21ksoftirqd/022:32:380
328136199431,11cyclictest13-21ksoftirqd/019:34:390
3281361994311,11cyclictest13-21ksoftirqd/022:45:280
328137199426,18cyclictest42-21ksoftirqd/321:05:393
328137199425,7cyclictest42-21ksoftirqd/323:00:013
3281371994217,7cyclictest42-21ksoftirqd/319:18:073
328137199421,19cyclictest42-21ksoftirqd/323:07:343
328137199421,11cyclictest42-21ksoftirqd/319:50:233
3281367994232,7cyclictest331rcuc/221:10:172
3281367994215,5cyclictest141rcu_preempt19:42:162
3281367994214,13cyclictest34-21ksoftirqd/219:10:192
3281367994212,5cyclictest141rcu_preempt21:57:582
3281367994212,5cyclictest141rcu_preempt21:32:422
328136799421,23cyclictest3360778-21taskset22:52:492
3281367994211,15cyclictest34-21ksoftirqd/223:20:022
3281367994211,15cyclictest34-21ksoftirqd/222:13:492
3281367994211,15cyclictest34-21ksoftirqd/220:50:422
3281365994214,25cyclictest1137-21runrttasks20:26:231
3281365994214,24cyclictest3345562-21cstates22:10:141
3281365994213,26cyclictest1137-21runrttasks22:46:521
3281365994213,24cyclictest26-21ksoftirqd/122:55:271
328136199427,9cyclictest13-21ksoftirqd/022:50:280
328136199427,11cyclictest13-21ksoftirqd/000:19:560
328136199425,8cyclictest13-21ksoftirqd/019:20:130
328136199424,20cyclictest13-21ksoftirqd/019:30:010
3281361994215,9cyclictest13-21ksoftirqd/022:40:160
328136199421,22cyclictest13-21ksoftirqd/000:23:090
3281361994212,11cyclictest13-21ksoftirqd/000:00:230
3281361994212,11cyclictest13-21ksoftirqd/000:00:230
328136199421,10cyclictest13-21ksoftirqd/000:05:220
328137199413,13cyclictest3316443-21latency20:45:223
328137199413,12cyclictest4497-21PK-Backend00:11:543
328137199413,12cyclictest42-21ksoftirqd/319:20:083
328137199413,11cyclictest42-21ksoftirqd/322:49:563
328137199412,12cyclictest4497-21PK-Backend21:36:443
328137199411,13cyclictest3373394-21taskset23:29:453
328137199411,11cyclictest42-21ksoftirqd/322:21:223
328137199411,11cyclictest42-21ksoftirqd/322:16:393
3281367994131,7cyclictest331rcuc/223:35:272
328136799413,14cyclictest0-21swapper/220:43:332
3281367994122,8cyclictest34-21ksoftirqd/219:30:132
3281367994112,5cyclictest141rcu_preempt19:47:152
328136799411,22cyclictest3308303-21taskset20:20:362
328136799411,21cyclictest3352550-21taskset22:30:082
328136599419,28cyclictest3283662-21ls19:15:051
328136599418,29cyclictest3304286-21wc20:10:301
328136199416,8cyclictest13-21ksoftirqd/020:30:190
328136199416,8cyclictest13-21ksoftirqd/020:30:180
328136199414,20cyclictest13-21ksoftirqd/022:00:170
328136199412,9cyclictest13-21ksoftirqd/019:40:200
328136199412,14cyclictest3381224-21cstates23:50:130
3281361994113,9cyclictest13-21ksoftirqd/021:20:020
328136199411,18cyclictest13-21ksoftirqd/021:30:130
328136199411,10cyclictest3372453-21/usr/sbin/munin23:25:230
328137199405,9cyclictest42-21ksoftirqd/320:17:093
328137199405,8cyclictest42-21ksoftirqd/322:53:103
328137199405,8cyclictest42-21ksoftirqd/321:21:263
328137199405,8cyclictest42-21ksoftirqd/321:21:253
328137199405,8cyclictest42-21ksoftirqd/320:11:003
328137199404,9cyclictest42-21ksoftirqd/300:05:243
328137199404,8cyclictest42-21ksoftirqd/321:50:243
328137199403,8cyclictest42-21ksoftirqd/319:10:103
328137199403,7cyclictest42-21ksoftirqd/320:04:483
328137199403,11cyclictest42-21ksoftirqd/319:38:363
328137199402,7cyclictest42-21ksoftirqd/322:40:103
328137199401,9cyclictest3290083-21runrttasks19:30:323
328137199401,17cyclictest42-21ksoftirqd/321:13:173
328137199401,12cyclictest42-21ksoftirqd/320:56:113
328137199401,12cyclictest42-21ksoftirqd/320:56:113
328137199401,12cyclictest42-21ksoftirqd/320:05:363
328137199401,11cyclictest42-21ksoftirqd/323:10:033
328137199401,11cyclictest42-21ksoftirqd/322:36:033
328137199401,11cyclictest42-21ksoftirqd/322:25:053
328137199401,11cyclictest42-21ksoftirqd/322:25:053
328136799404,14cyclictest0-21swapper/220:12:392
3281367994015,11cyclictest141rcu_preempt21:00:262
328136799401,21cyclictest3334573-21runrttasks21:38:362
328136799401,20cyclictest3393682-21runrttasks00:25:222
328136799401,20cyclictest3393682-21runrttasks00:25:212
3281367994011,5cyclictest141rcu_preempt22:35:132
328136599402,17cyclictest3380453-40lsb_release23:47:061
328136199408,10cyclictest13-21ksoftirqd/021:10:130
328136199405,8cyclictest13-21ksoftirqd/019:50:190
328136199404,19cyclictest13-21ksoftirqd/022:35:260
328136199401,18cyclictest13-21ksoftirqd/022:25:190
328136199401,18cyclictest13-21ksoftirqd/022:25:190
328136199401,12cyclictest13-21ksoftirqd/021:50:150
328136199401,11cyclictest13-21ksoftirqd/020:40:000
33458452390,5sleep342-21ksoftirqd/322:10:203
328137199397,7cyclictest42-21ksoftirqd/319:46:173
328137199394,7cyclictest42-21ksoftirqd/321:55:203
328137199394,7cyclictest42-21ksoftirqd/300:15:223
328137199393,8cyclictest42-21ksoftirqd/300:37:553
328137199393,15cyclictest42-21ksoftirqd/323:54:353
3281371993923,12cyclictest3332614-21sshd21:30:393
3281371993913,7cyclictest42-21ksoftirqd/300:30:243
328137199391,17cyclictest42-21ksoftirqd/321:01:293
328137199391,16cyclictest42-21ksoftirqd/300:00:253
328137199391,16cyclictest42-21ksoftirqd/300:00:253
328137199391,13cyclictest42-21ksoftirqd/320:25:243
328137199391,12cyclictest42-21ksoftirqd/323:47:193
328137199391,12cyclictest42-21ksoftirqd/321:45:323
328137199391,11cyclictest42-21ksoftirqd/320:54:023
328137199391,11cyclictest42-21ksoftirqd/319:27:533
328136799396,9cyclictest34-21ksoftirqd/219:50:232
328136799394,16cyclictest0-21swapper/223:26:412
3281367993923,11cyclictest3313361-21sort20:35:342
328136799392,14cyclictest0-21swapper/223:05:082
3281367993915,5cyclictest141rcu_preempt20:35:012
3281367993915,5cyclictest141rcu_preempt20:35:012
3281367993913,6cyclictest141rcu_preempt22:03:572
3281367993912,5cyclictest141rcu_preempt21:26:222
328136799391,18cyclictest3392477-21runrttasks00:20:372
3281367993911,5cyclictest141rcu_preempt21:51:392
328136799391,12cyclictest3327409-21runrttasks21:16:032
3281367993911,14cyclictest141rcu_preempt23:00:132
33242502380,13chrt34-21ksoftirqd/221:09:542
328137199388,8cyclictest42-21ksoftirqd/323:31:093
328137199385,17cyclictest42-21ksoftirqd/319:55:493
328137199383,7cyclictest42-21ksoftirqd/319:40:403
328137199381,19cyclictest42-21ksoftirqd/320:36:303
328136799386,10cyclictest438-20systemd-journal20:47:382
328136799385,16cyclictest34-21ksoftirqd/219:58:232
328136799383,16cyclictest0-21swapper/223:10:542
3281367993812,6cyclictest141rcu_preempt00:01:582
3281367993812,6cyclictest141rcu_preempt00:01:572
3281367993811,5cyclictest141rcu_preempt00:32:512
3281367993810,7cyclictest34-21ksoftirqd/222:20:122
328136599381,14cyclictest3353723-21taskset22:31:061
328137199374,7cyclictest42-21ksoftirqd/320:30:183
328137199374,7cyclictest42-21ksoftirqd/320:30:183
328137199372,9cyclictest42-21ksoftirqd/322:30:563
328137199371,17cyclictest42-21ksoftirqd/322:05:093
328137199371,17cyclictest42-21ksoftirqd/322:05:093
328137199371,10cyclictest42-21ksoftirqd/321:27:573
3281367993710,5cyclictest141rcu_preempt22:06:562
3281367993710,5cyclictest141rcu_preempt22:06:562
328137199361,9cyclictest42-21ksoftirqd/321:15:273
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional