You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-26 - 15:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Fri Dec 26, 2025 12:44:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
372105722580,21sleep342-21ksoftirqd/307:05:213
372188322070,9sleep24497-21PK-Backend07:08:002
379155421020,14sleep013-21ksoftirqd/010:22:530
197399980,5rtkit-daemon0-21swapper/312:05:173
38196942910,8sleep04497-21PK-Backend11:45:030
37493232830,6sleep04497-21PK-Backend08:25:020
37880942770,7sleep24497-21PK-Backend10:15:012
37510312760,3sleep03751032-21sleep08:30:010
372185827447,17sleep10-21swapper/107:07:571
37318282650,19sleep126-21ksoftirqd/107:35:161
372185726544,13sleep00-21swapper/007:07:560
38257222590,15chrt26-21ksoftirqd/112:00:251
3722210995922,12cyclictest438-20systemd-journal11:17:013
3722210995617,13cyclictest438-20systemd-journal08:08:073
3722210995516,11cyclictest3777368-21grep09:45:013
3722210995319,10cyclictest3731289-21perf07:34:593
3722210995316,11cyclictest438-20systemd-journal12:00:213
3722210995315,11cyclictest3724829-21cstates07:15:143
3722205995313,19cyclictest34-21ksoftirqd/212:25:022
37556512520,19sleep013-21ksoftirqd/008:40:300
3722210995218,13cyclictest3800757-21awk10:50:143
3722210995216,11cyclictest3796567-21cat10:35:303
3722210995116,11cyclictest3768371-21apt-get09:20:003
372220299511,16cyclictest3729546-21sh07:30:001
37341002500,5sleep30-21swapper/307:40:253
3722210995020,7cyclictest3784885-21sh10:05:153
3722210995018,12cyclictest3755292-21cat08:40:223
3722210994921,9cyclictest3729642-21latency_hist07:30:003
3722210994917,12cyclictest3834241-21df12:25:163
3722210994914,11cyclictest3836887-21seq12:32:193
3722210994816,8cyclictest3807833-21cstates11:10:133
3722202994825,19cyclictest3770417-21awk09:25:011
38204932470,5chrt0-21swapper/011:45:230
3722210994714,10cyclictest3816900-21sed11:35:223
3722210994713,10cyclictest3724056-21date07:15:023
372221099471,26cyclictest3824436-21chrt11:57:003
3722210994712,10cyclictest438-20systemd-journal11:35:003
372221099462,23cyclictest3779324-21apt09:50:113
3722210994619,6cyclictest3795241-21perf10:35:003
3722210994616,7cyclictest3799108-21tr10:45:143
372221099452,7cyclictest3769269-21df_inode09:20:143
3722210994525,4cyclictest0-21swapper/311:44:123
3722210994518,7cyclictest3778076-21interrupts09:45:183
372221099442,22cyclictest3749951-21fw_packets08:25:193
3722210994418,7cyclictest3820219-21irqstats11:45:193
372221099441,7cyclictest3788214-21/usr/sbin/munin10:15:273
372221099441,28cyclictest0-21swapper/310:20:053
372221099441,23cyclictest3805310-21taskset11:01:073
372221099441,11cyclictest3738126-21taskset07:53:323
372221099441,10cyclictest3821634-21cpuspeed_turbos11:50:173
3722205994417,8cyclictest34-21ksoftirqd/208:53:052
3722205994416,14cyclictest141rcu_preempt07:11:472
3722205994415,16cyclictest34-21ksoftirqd/211:38:112
3722205994413,17cyclictest34-21ksoftirqd/211:01:222
3722202994434,7cyclictest251rcuc/108:15:331
3722202994426,8cyclictest26-21ksoftirqd/109:49:371
3722202994410,8cyclictest26-21ksoftirqd/107:15:101
37342502430,7sleep24497-21PK-Backend07:40:282
372221099431,8cyclictest2190-21gvfs-udisks2-vo11:09:563
3722210994317,7cyclictest3740534-21/usr/sbin/munin08:00:233
3722205994322,13cyclictest34-21ksoftirqd/210:56:042
3722205994322,13cyclictest34-21ksoftirqd/210:56:042
3722205994313,12cyclictest141rcu_preempt12:30:162
3722205994312,15cyclictest34-21ksoftirqd/208:12:092
3722205994312,15cyclictest34-21ksoftirqd/208:12:092
372220299438,6cyclictest141rcu_preempt12:39:591
372220299433,10cyclictest26-21ksoftirqd/111:54:591
3722202994316,8cyclictest26-21ksoftirqd/109:41:121
3722202994314,10cyclictest26-21ksoftirqd/111:15:281
372220299431,15cyclictest3751004-21cron08:30:011
3722210994224,4cyclictest4497-21PK-Backend08:21:323
3722210994219,5cyclictest42-21ksoftirqd/311:27:183
3722210994217,7cyclictest3731524-21/usr/sbin/munin07:35:243
3722205994215,8cyclictest34-21ksoftirqd/211:25:262
3722205994215,7cyclictest141rcu_preempt08:05:212
3722205994210,16cyclictest34-21ksoftirqd/208:59:252
3722202994217,13cyclictest141rcu_preempt12:17:381
3722202994217,13cyclictest141rcu_preempt09:37:131
3722202994216,10cyclictest26-21ksoftirqd/112:24:561
3722202994214,8cyclictest26-21ksoftirqd/110:20:231
372220299421,19cyclictest26-21ksoftirqd/109:51:181
3722202994210,18cyclictest26-21ksoftirqd/107:20:091
3722199994215,14cyclictest13-21ksoftirqd/007:29:590
197399420,28rtkit-daemon4497-21PK-Backend10:10:032
37723592410,6sleep10-21swapper/109:30:011
372221099412,7cyclictest3832966-21users12:20:293
3722210994117,7cyclictest1054-21snmpd09:03:503
3722210994115,10cyclictest3781255-21ls09:55:143
372220599418,12cyclictest3822189-21netstat11:50:262
3722205994125,12cyclictest34-21ksoftirqd/208:21:482
3722205994123,14cyclictest34-21ksoftirqd/212:35:192
3722205994121,13cyclictest34-21ksoftirqd/210:44:572
372220299419,8cyclictest26-21ksoftirqd/108:40:001
372220299419,8cyclictest26-21ksoftirqd/108:39:591
372220299412,21cyclictest26-21ksoftirqd/108:30:181
3722202994122,14cyclictest438-20systemd-journal10:10:011
372220299411,6cyclictest4497-21PK-Backend10:05:011
3722202994113,16cyclictest26-21ksoftirqd/109:18:341
3722202994112,9cyclictest26-21ksoftirqd/112:10:211
3722199994124,6cyclictest13-21ksoftirqd/010:00:020
3722199994121,12cyclictest13-21ksoftirqd/012:25:020
372219999411,36cyclictest3806149-21cstates11:05:120
3722205994014,14cyclictest34-21ksoftirqd/211:08:572
3722205994013,8cyclictest34-21ksoftirqd/208:30:072
3722205994012,9cyclictest34-21ksoftirqd/209:18:212
372220599401,20cyclictest34-21ksoftirqd/211:40:182
372220299406,11cyclictest26-21ksoftirqd/107:45:271
3722202994023,12cyclictest438-20systemd-journal09:11:171
372220299402,17cyclictest0-21swapper/111:35:211
372220299402,15cyclictest26-21ksoftirqd/111:55:201
3722202994016,8cyclictest26-21ksoftirqd/111:24:541
3722202994015,6cyclictest141rcu_preempt12:10:011
3722202994013,7cyclictest141rcu_preempt10:12:271
3722202994013,6cyclictest141rcu_preempt08:15:001
3722202994013,6cyclictest141rcu_preempt08:14:591
3722199994030,6cyclictest1137-21runrttasks12:35:010
3722199994023,13cyclictest3746637-21memory08:15:230
37325292390,6sleep00-21swapper/007:35:310
3722210993923,4cyclictest4497-21PK-Backend12:37:033
372221099392,33cyclictest923-21dbus-daemon10:15:003
372220599394,18cyclictest0-21swapper/208:40:312
3722205993922,13cyclictest34-21ksoftirqd/211:57:332
3722205993914,8cyclictest34-21ksoftirqd/210:03:382
3722205993914,6cyclictest141rcu_preempt07:59:552
3722205993913,8cyclictest34-21ksoftirqd/211:14:222
3722205993913,6cyclictest141rcu_preempt07:25:012
3722205993912,8cyclictest34-21ksoftirqd/207:20:022
3722205993912,14cyclictest141rcu_preempt12:25:292
372220299398,6cyclictest141rcu_preempt12:32:151
372220299397,9cyclictest26-21ksoftirqd/112:25:221
372220299396,16cyclictest3812803-21acpi11:25:121
372220299392,20cyclictest3788600-21df10:15:161
3722202993913,6cyclictest141rcu_preempt11:10:001
372220299391,18cyclictest26-21ksoftirqd/108:20:311
3722199993926,9cyclictest1137-21runrttasks07:55:560
3722199993922,13cyclictest3823205-21/usr/sbin/munin11:55:110
197399390,27rtkit-daemon4497-21PK-Backend11:31:492
372221099389,8cyclictest0-21swapper/308:45:223
3722210993816,8cyclictest3747165-21vmstat08:15:333
3722210993812,7cyclictest0-21swapper/308:56:103
3722205993822,13cyclictest34-21ksoftirqd/212:16:582
3722205993822,12cyclictest34-21ksoftirqd/209:45:462
3722205993822,12cyclictest34-21ksoftirqd/209:29:462
3722205993822,12cyclictest34-21ksoftirqd/208:03:162
372220599382,17cyclictest3826669-21sort12:05:102
3722205993821,13cyclictest34-21ksoftirqd/212:03:072
3722205993821,13cyclictest34-21ksoftirqd/210:36:032
3722205993821,13cyclictest34-21ksoftirqd/209:11:122
3722205993821,13cyclictest34-21ksoftirqd/207:51:132
3722205993821,13cyclictest34-21ksoftirqd/207:38:532
3722205993821,12cyclictest34-21ksoftirqd/209:55:072
372220599381,8cyclictest3801065-21df10:50:152
3722205993813,8cyclictest34-21ksoftirqd/209:23:572
3722205993813,8cyclictest34-21ksoftirqd/207:28:122
372220599381,12cyclictest438-20systemd-journal09:54:592
372220299388,6cyclictest141rcu_preempt08:45:171
372220299385,14cyclictest0-21swapper/109:05:251
372220299384,20cyclictest0-21swapper/109:05:001
372220299384,19cyclictest0-21swapper/108:55:261
3722202993822,11cyclictest438-20systemd-journal10:40:011
372220299382,16cyclictest923-21dbus-daemon11:30:311
372220299381,15cyclictest26-21ksoftirqd/108:09:491
372220299381,15cyclictest0-21swapper/107:33:261
3722202993810,7cyclictest141rcu_preempt10:35:031
3722199993822,12cyclictest976-21rs:main0
3722199993821,6cyclictest13-21ksoftirqd/010:43:390
3722199993821,12cyclictest438-20systemd-journal07:45:010
197399380,26rtkit-daemon4497-21PK-Backend11:21:042
3722210993716,7cyclictest3829165-21ntp_states12:10:253
372220599376,6cyclictest141rcu_preempt11:45:102
372220599374,16cyclictest0-21swapper/208:25:302
3722205993722,12cyclictest34-21ksoftirqd/209:37:082
3722205993722,12cyclictest34-21ksoftirqd/209:08:212
3722205993721,13cyclictest34-21ksoftirqd/210:31:352
3722205993721,13cyclictest34-21ksoftirqd/208:47:012
3722205993721,12cyclictest34-21ksoftirqd/210:20:122
3722205993721,12cyclictest34-21ksoftirqd/209:31:582
3722205993720,13cyclictest34-21ksoftirqd/210:47:382
3722205993719,14cyclictest3776620-21ntp_states09:40:272
3722205993714,6cyclictest141rcu_preempt07:35:012
3722205993711,8cyclictest34-21ksoftirqd/209:03:302
372220299375,16cyclictest0-21swapper/108:50:271
3722202993721,12cyclictest26-21ksoftirqd/110:43:221
3722202993720,13cyclictest26-21ksoftirqd/111:10:121
3722202993720,13cyclictest26-21ksoftirqd/108:01:221
372220299371,6cyclictest4497-21PK-Backend07:10:281
3722202993713,7cyclictest26-21ksoftirqd/107:40:161
372220299371,18cyclictest26-21ksoftirqd/108:00:001
372219999379,13cyclictest141rcu_preempt09:55:010
3722199993723,10cyclictest13-21ksoftirqd/010:10:250
372219999371,32cyclictest3757929-21cron08:50:000
3722199993712,21cyclictest2193-21gdbus07:49:570
197399370,26rtkit-daemon4497-21PK-Backend11:22:303
197399370,25rtkit-daemon0-21swapper/210:28:072
197399370,24rtkit-daemon0-21swapper/210:11:472
3722210993625,7cyclictest3770867-21tr09:25:143
3722210993620,12cyclictest3744915-21wc08:10:213
3722210993620,12cyclictest3744915-21wc08:10:213
3722205993621,12cyclictest34-21ksoftirqd/208:17:242
3722205993620,12cyclictest34-21ksoftirqd/211:15:052
3722205993620,12cyclictest34-21ksoftirqd/208:37:252
3722205993620,12cyclictest34-21ksoftirqd/208:37:252
3722205993619,13cyclictest34-21ksoftirqd/212:12:082
3722205993619,13cyclictest34-21ksoftirqd/207:48:052
372220299369,6cyclictest141rcu_preempt11:00:301
372220299364,15cyclictest0-21swapper/109:55:251
3722202993620,5cyclictest26-21ksoftirqd/110:29:491
3722202993620,12cyclictest26-21ksoftirqd/111:46:271
3722202993620,12cyclictest26-21ksoftirqd/111:42:181
3722202993620,12cyclictest26-21ksoftirqd/109:32:081
3722202993619,13cyclictest26-21ksoftirqd/107:51:381
3722199993630,4cyclictest0-21swapper/011:37:060
3722199993619,6cyclictest13-21ksoftirqd/011:54:050
3722199993619,6cyclictest13-21ksoftirqd/007:15:240
372219999361,7cyclictest3801082-21chrt10:50:180
3722199993615,8cyclictest141rcu_preempt12:10:020
372219999361,12cyclictest37623342/usr/sbin/munin09:00:220
197399360,24rtkit-daemon4497-21PK-Backend07:58:473
372221099356,13cyclictest2069-21gnome-shell07:22:593
372220299352,19cyclictest26-21ksoftirqd/110:45:231
372220299351,9cyclictest4497-21PK-Backend10:50:251
3722199993520,11cyclictest3814700-21expr11:30:120
3722199993519,12cyclictest3766878-21seq09:15:080
3722199993519,12cyclictest3751480-21cpuspeed_turbos08:30:130
3722199993518,13cyclictest3774500-21df09:35:150
3722199993517,14cyclictest3758613-21which08:50:190
3722210993419,11cyclictest3735439-21df07:45:153
372220299343,15cyclictest0-21swapper/110:55:121
372220299343,15cyclictest0-21swapper/110:55:121
3722202993419,11cyclictest26-21ksoftirqd/108:40:111
3722199993424,6cyclictest3785968-21sh10:10:000
3722199993424,6cyclictest13-21ksoftirqd/010:30:260
3722199993424,6cyclictest13-21ksoftirqd/009:20:040
3722199993420,10cyclictest3731308-21systemctl07:35:010
3722199993419,11cyclictest3753663-21cat08:35:230
3722199993419,11cyclictest3753663-21cat08:35:220
3722199993418,12cyclictest3727355-21sed07:20:290
3722199993418,11cyclictest438-20systemd-journal08:05:110
3722199993418,11cyclictest3839007-21cut12:40:020
3722199993412,9cyclictest141rcu_preempt11:01:560
372219999340,15cyclictest3789778-21unin-run10:20:010
38138422330,15sleep0161rcuc/011:25:300
3722210993319,10cyclictest3783781-21cat10:00:303
372221099331,29cyclictest0-21swapper/309:33:293
3722210993310,19cyclictest42-21ksoftirqd/310:27:353
372219999334,6cyclictest141rcu_preempt08:15:000
372219999334,6cyclictest141rcu_preempt08:15:000
3722199993323,6cyclictest13-21ksoftirqd/012:25:560
3722199993322,7cyclictest3811211-21grep11:20:140
3722199993322,6cyclictest13-21ksoftirqd/011:15:490
3722199993321,7cyclictest438-20systemd-journal10:25:010
3722199993319,10cyclictest3826523-21sed12:05:010
3722199993318,11cyclictest3764513-21sed09:05:300
3722199993318,11cyclictest3738262-21apt-get07:55:010
3722199993317,11cyclictest3722785-21sshd07:10:170
3722199993316,13cyclictest3783301-21cut10:00:210
3722199993316,13cyclictest1-21systemd10:40:000
372219999331,29cyclictest0-21swapper/012:10:520
372219999331,29cyclictest0-21swapper/009:45:270
3722210993218,10cyclictest3764474-21expr09:05:313
3722210993217,11cyclictest3774662-21grep09:35:223
372221099321,28cyclictest0-21swapper/312:18:173
3722199993221,6cyclictest3761450-21sh09:00:010
3722199993217,11cyclictest923-21dbus-daemon09:30:010
3722199993215,5cyclictest13-21ksoftirqd/011:10:130
3722199993214,15cyclictest13-21ksoftirqd/009:15:010
3722199993124,4cyclictest4497-21PK-Backend10:47:230
3722199993121,6cyclictest13-21ksoftirqd/009:41:480
3722199993117,10cyclictest3742010-21cron08:05:000
372219999311,4cyclictest0-21swapper/010:57:420
372219999311,4cyclictest0-21swapper/010:57:410
3722210993024,4cyclictest0-21swapper/308:50:383
3722210993019,7cyclictest1137-21runrttasks08:35:273
3722210993019,7cyclictest1137-21runrttasks08:35:273
3722210993013,12cyclictest438-20systemd-journal11:00:003
3722210993013,12cyclictest438-20systemd-journal11:00:003
372221099301,10cyclictest3798457-21taskset10:42:033
3722199993024,4cyclictest0-21swapper/009:32:380
3722210992918,8cyclictest4497-21PK-Backend09:13:013
3722210992918,8cyclictest4497-21PK-Backend08:32:373
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional