You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-19 - 22:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Fri Dec 19, 2025 12:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
397887421140,10sleep13978876-21processes11:30:281
400123521110,5sleep3411rcuc/312:30:303
397905921070,4sleep00-21swapper/011:30:310
39133752950,8sleep342-21ksoftirqd/308:25:193
39842802930,4sleep30-21swapper/311:45:243
39842802930,4sleep30-21swapper/311:45:233
38879402930,6sleep10-21swapper/107:15:061
39332962920,6sleep30-21swapper/309:20:243
39332962920,6sleep30-21swapper/309:20:243
197399830,5rtkit-daemon0-21swapper/009:55:390
197399830,5rtkit-daemon0-21swapper/007:40:430
197399820,6rtkit-daemon4497-21PK-Backend07:55:250
197399820,5rtkit-daemon0-21swapper/008:56:210
197399810,4rtkit-daemon0-21swapper/008:13:520
197399800,6rtkit-daemon4497-21PK-Backend09:24:201
197399800,6rtkit-daemon4497-21PK-Backend09:24:201
197399800,6rtkit-daemon4497-21PK-Backend07:10:150
197399790,6rtkit-daemon0-21swapper/009:41:080
197399790,5rtkit-daemon0-21swapper/007:33:520
197399780,5rtkit-daemon0-21swapper/008:32:120
197399770,5rtkit-daemon4497-21PK-Backend07:50:160
197399750,60rtkit-daemon0-21swapper/007:08:520
388405527347,16sleep30-21swapper/307:05:123
197399730,5rtkit-daemon0-21swapper/009:48:540
197399720,4rtkit-daemon4497-21PK-Backend10:46:101
388552926546,12sleep20-21swapper/207:10:012
39063492630,5sleep2331rcuc/208:05:212
388538126342,14sleep10-21swapper/107:08:101
197399610,9rtkit-daemon1972-21rtkit-daemon10:27:200
197399610,9rtkit-daemon1972-21rtkit-daemon10:20:101
3885674996020,35cyclictest3972782-21expr11:15:131
3885674995918,11cyclictest3912231-21seq08:22:501
39388502580,7sleep00-21swapper/009:35:290
197399580,9rtkit-daemon1972-21rtkit-daemon10:10:440
3885674995714,13cyclictest2372-21snap-store12:27:391
3885674995619,32cyclictest3993677-21grep12:10:251
3885674995619,32cyclictest3993677-21grep12:10:251
3885674995515,11cyclictest3996179-21date12:20:011
197399550,8rtkit-daemon1972-21rtkit-daemon11:58:291
3885679995446,5cyclictest341ksoftirqd/211:21:022
3885679995446,5cyclictest341ksoftirqd/211:21:022
3885674995413,13cyclictest3996770-21basename12:20:191
388567499529,13cyclictest3930995-21hddtemp_smartct09:15:181
388567499523,20cyclictest3925842-21ntpq09:00:241
3885674995219,6cyclictest3887469-21latency_hist07:15:011
3885674995215,12cyclictest3943078-21awk09:50:111
3885674995212,11cyclictest3952549-21swap10:15:281
39995102510,4chrt0-21swapper/212:26:062
3885674995118,8cyclictest4497-21PK-Backend09:25:121
3885674995117,7cyclictest3940154-21missed_timers09:40:231
3885674995116,8cyclictest3947221-21ntp_offset10:00:231
3885674995114,11cyclictest3989564-21expr12:00:151
388567499511,10cyclictest3977764-21cut11:30:001
3885674995018,8cyclictest3950687-21ntp_states10:10:241
3885674995018,7cyclictest4497-21PK-Backend09:56:151
3885674995017,8cyclictest4497-21PK-Backend11:01:261
3885674995017,29cyclictest3969414-21if_enp2s011:05:191
3885674995015,7cyclictest3982429-21missed_timers11:40:221
3885674995015,7cyclictest3971161-21if_enp2s011:10:211
3885674995014,7cyclictest3905652-21latency_hist08:05:011
3885674995010,10cyclictest3929086-21cat09:10:151
3885674994918,7cyclictest4497-21PK-Backend08:35:541
3885674994918,7cyclictest4497-21PK-Backend08:33:491
3885674994916,6cyclictest3898347-21latency_hist07:45:011
3885674994915,7cyclictest3938788-21timerandwakeup09:35:301
3885674994915,7cyclictest3892049-21kernelversion07:25:211
3885674994910,12cyclictest3975318-21systemd-udevd11:21:001
3885674994910,12cyclictest3975318-21systemd-udevd11:21:001
3885674994910,10cyclictest3941349-21cpu09:45:131
388567499486,12cyclictest3986759-21apt-get11:55:011
3885674994817,7cyclictest4497-21PK-Backend08:40:421
3885674994817,10cyclictest3890285-21if_err_enp2s007:20:181
3885674994814,7cyclictest1137-21runrttasks09:32:031
3885674994813,8cyclictest3979810-21ls11:35:131
3885674994813,7cyclictest3913563-21missed_timers08:25:211
3885674994716,7cyclictest3949256-21timerwakeupswit10:05:301
3885674994711,9cyclictest3909657-21if_enp2s008:15:181
388567499462,17cyclictest3903416-40lsb_release07:56:021
3885674994613,6cyclictest3992613-21latency_hist12:10:011
3885674994610,7cyclictest0-21swapper/108:55:201
39039582450,7sleep20-21swapper/208:00:012
388567499452,16cyclictest3956611-21munin-run10:30:011
3885674994514,7cyclictest4497-21PK-Backend07:36:061
3885679994426,14cyclictest34-21ksoftirqd/209:15:002
3885679994418,15cyclictest34-21ksoftirqd/212:00:592
388567499442,17cyclictest3957676-21grep10:30:271
388567499442,17cyclictest3957676-21grep10:30:271
388567499442,15cyclictest3894161-21python307:30:261
388567499432,8cyclictest3927588-21ntp_states09:05:251
3885669994323,16cyclictest3975719-21apt-get11:25:040
197399430,31rtkit-daemon4497-21PK-Backend09:34:480
39695262420,14sleep013-21ksoftirqd/011:05:190
197399420,33rtkit-daemon13-21ksoftirqd/011:59:550
3885683994122,6cyclictest42-21ksoftirqd/311:30:103
3885683994120,10cyclictest3981006-21systemd-udevd11:37:303
388568399411,4cyclictest4497-21PK-Backend12:05:143
3885679994127,10cyclictest3986759-21apt-get11:55:012
3885674994111,9cyclictest3906438-21grep08:05:221
3885669994126,11cyclictest3893005-21cron07:29:590
3885683994022,13cyclictest438-20systemd-journal07:37:113
3885683994021,6cyclictest42-21ksoftirqd/308:55:053
3885679994030,6cyclictest3919888-21/usr/sbin/munin08:45:122
388567499405,6cyclictest0-21swapper/108:55:011
3885674994013,7cyclictest26-21ksoftirqd/107:52:091
3885669994025,11cyclictest3985007-21cron11:50:000
3885669994025,11cyclictest3985007-21cron11:49:590
197399400,31rtkit-daemon3885664-21cyclictest10:39:531
197399400,27rtkit-daemon4497-21PK-Backend11:02:432
40020862390,5sleep30-21swapper/312:35:143
39672042390,7chrt4497-21PK-Backend11:00:011
39113192390,7sleep34497-21PK-Backend08:20:153
3885683993923,11cyclictest438-20systemd-journal09:57:103
3885683993921,14cyclictest3964374-21memory10:50:213
3885679993929,6cyclictest1081-21ntpd08:30:242
3885679993929,5cyclictest34-21ksoftirqd/209:30:012
3885679993923,12cyclictest3965236-21cron10:55:002
3885679993914,11cyclictest1-21systemd10:35:002
3885679993914,11cyclictest1-21systemd10:35:002
3885669993924,11cyclictest3914439-21cron08:30:010
197399390,26rtkit-daemon0-21swapper/008:48:380
38988722380,6sleep3411rcuc/307:45:133
3885683993824,11cyclictest42-21ksoftirqd/308:45:273
3885679993827,7cyclictest3984358-21wc11:45:252
3885679993827,7cyclictest3984358-21wc11:45:252
3885679993821,6cyclictest34-21ksoftirqd/208:11:592
3885679993821,5cyclictest34-21ksoftirqd/210:00:012
3885679993820,13cyclictest3943768-21ntp_states09:50:242
3885669993827,8cyclictest13-21ksoftirqd/012:27:370
3885669993825,9cyclictest2092-21JS10:49:090
388566999382,32cyclictest3959341-21ntp_states10:35:230
3885669993821,12cyclictest3975340-21systemd-udevd11:21:030
3885669993821,12cyclictest3975340-21systemd-udevd11:21:020
388566999381,33cyclictest3996752-21idleruntime12:20:190
197399380,26rtkit-daemon4497-21PK-Backend08:12:331
197399380,26rtkit-daemon4497-21PK-Backend07:48:251
3885683993723,10cyclictest42-21ksoftirqd/308:50:223
3885683993722,11cyclictest42-21ksoftirqd/307:18:553
3885683993722,11cyclictest3994763-21grep12:15:133
3885683993721,12cyclictest3915119-21fw_packets08:30:173
3885683993720,13cyclictest3956010-21python310:25:283
3885683993720,12cyclictest3954001-21expr10:20:213
388568399371,9cyclictest3930329-21turbostat.cron09:15:003
3885683993719,14cyclictest3962428-21df_inode10:45:153
3885679993720,5cyclictest34-21ksoftirqd/211:40:032
388566999372,6cyclictest0-21swapper/009:15:010
3885669993721,12cyclictest3935675-21seq09:28:170
3885669993720,12cyclictest438-20systemd-journal09:05:090
3885669993712,11cyclictest4002996-21systemd-udevd12:35:420
388568399362,13cyclictest3886271-21hddtemp_smartct07:10:183
3885683993621,11cyclictest923-21dbus-daemon11:00:013
3885683993621,11cyclictest923-21dbus-daemon11:00:003
3885683993621,10cyclictest3890422-21expr07:20:203
3885683993620,12cyclictest3973686-21seq11:17:233
3885683993620,11cyclictest438-20systemd-journal10:09:553
3885683993620,11cyclictest438-20systemd-journal09:10:003
3885683993616,6cyclictest42-21ksoftirqd/309:00:343
3885683993614,18cyclictest3987968-21head11:55:233
3885679993625,6cyclictest34-21ksoftirqd/207:24:592
3885679993620,12cyclictest3889478-21sshd07:19:112
3885679993619,12cyclictest438-20systemd-journal10:14:592
3885679993617,14cyclictest34-21ksoftirqd/209:40:282
3885679993612,7cyclictest39483972cpuspeed_turbos10:05:132
388567499362,14cyclictest4000155-21taskset12:30:121
388566999362,5cyclictest3988795-21apt-get12:00:080
3885669993621,11cyclictest923-21dbus-daemon11:37:320
3885669993620,12cyclictest3985864-21cat11:50:210
197399360,25rtkit-daemon0-21swapper/211:14:262
3885683993525,6cyclictest42-21ksoftirqd/309:29:413
3885683993522,9cyclictest3977408-40lsb_release11:26:033
3885683993520,11cyclictest3932087-21awk09:20:003
3885683993519,12cyclictest1137-21runrttasks11:40:523
3885683993518,12cyclictest976-21rs:main3
388568399351,29cyclictest3918770-21memory08:40:213
388568399351,14cyclictest3950888-21sed10:10:293
388567999358,8cyclictest34-21ksoftirqd/208:05:002
3885679993526,5cyclictest1054-21snmpd12:19:502
3885679993521,10cyclictest438-20systemd-journal10:24:112
3885679993520,11cyclictest3919332-21sshd08:41:102
3885679993520,11cyclictest3895217-21expr07:35:142
3885679993519,5cyclictest34-21ksoftirqd/208:53:582
3885679993519,12cyclictest4000857-21ntp_kernel_err12:30:232
3885679993519,12cyclictest3894157-21python307:30:262
3885679993519,12cyclictest1-21systemd11:00:002
3885679993519,12cyclictest1-21systemd11:00:002
3885679993518,13cyclictest3960456-21cpuspeed_turbos10:40:142
388567499351,9cyclictest4497-21PK-Backend10:58:351
388566999352,12cyclictest3964462-21strings10:50:210
3885669993520,11cyclictest3952067-21forks10:15:170
3885669993519,13cyclictest4001102-21grep12:30:280
3885669993519,12cyclictest1033-21gdbus11:41:070
3885669993518,13cyclictest3895752-21ntpq07:35:250
39414482340,6sleep34497-21PK-Backend09:45:153
388568399344,26cyclictest3971507-21proc_pri11:10:273
388568399342,9cyclictest3903387-31apt-check07:55:523
3885683993421,9cyclictest42-21ksoftirqd/311:05:253
3885683993420,10cyclictest3904000-21seq08:00:043
3885683993419,11cyclictest3960931-21sed10:40:213
3885683993419,11cyclictest3901303-21cat07:50:293
3885683993419,11cyclictest3897418-21cat07:40:243
3885683993419,10cyclictest3976037-21cut11:25:033
3885683993419,10cyclictest3976037-21cut11:25:033
3885683993418,12cyclictest3958756-21expr10:35:133
3885683993415,14cyclictest42-21ksoftirqd/312:25:013
3885683993413,7cyclictest141rcu_preempt12:25:143
3885679993424,6cyclictest3897212-21hddtemp_smartct07:40:212
3885679993424,6cyclictest34-21ksoftirqd/209:00:132
3885679993424,5cyclictest34-21ksoftirqd/209:05:122
3885679993423,7cyclictest3960060-21grep10:39:592
3885679993423,6cyclictest34-21ksoftirqd/211:55:202
3885679993423,6cyclictest34-21ksoftirqd/211:25:222
3885679993423,6cyclictest34-21ksoftirqd/211:05:142
3885679993423,5cyclictest34-21ksoftirqd/210:05:012
3885679993419,11cyclictest3956722-21ls10:30:022
3885679993419,11cyclictest3933535-21sed09:20:292
3885679993419,11cyclictest3933535-21sed09:20:292
3885679993417,13cyclictest976-21rs:main2
388567999341,30cyclictest0-21swapper/212:13:012
388567999341,30cyclictest0-21swapper/212:13:012
388567999341,29cyclictest3909488-21cat08:15:132
388567999341,11cyclictest141rcu_preempt09:15:252
3885669993421,9cyclictest2069-21gnome-shell11:13:090
3885669993419,11cyclictest923-21dbus-daemon09:55:010
3885669993419,11cyclictest3992851-21sshd12:10:110
3885669993419,11cyclictest3992851-21sshd12:10:100
3885669993419,11cyclictest3960828-21sed10:40:200
3885669993418,12cyclictest976-21rs:main0
3885669993417,13cyclictest3931343-21ntp_states09:15:250
3885669993417,13cyclictest3888845-21ntp_states07:15:240
39656272330,7sleep00-21swapper/010:55:140
39656272330,7sleep00-21swapper/010:55:140
39167912330,15sleep234-21ksoftirqd/208:35:162
388568399336,23cyclictest411rcuc/307:33:113
3885683993322,7cyclictest3937985-21cpuspeed_turbos09:35:133
388568399331,9cyclictest4497-21PK-Backend11:54:513
3885683993319,10cyclictest3917480-21sort08:35:303
3885683993318,11cyclictest3957438-21sed10:30:223
3885683993318,11cyclictest3957438-21sed10:30:213
3885683993318,11cyclictest1-21systemd09:35:003
3885683993317,12cyclictest3909374-21expr08:15:113
388568399331,29cyclictest0-21swapper/309:52:203
388568399331,28cyclictest4497-21PK-Backend09:42:203
388568399331,27cyclictest3907496-21apt08:10:113
388568399331,11cyclictest3967963-21ntpq11:00:243
3885683993311,10cyclictest141rcu_preempt12:00:093
3885679993326,4cyclictest0-21swapper/210:18:162
3885679993324,6cyclictest1054-21snmpd11:33:232
3885679993323,7cyclictest1137-21runrttasks07:10:432
3885679993323,6cyclictest34-21ksoftirqd/208:58:542
3885679993322,6cyclictest34-21ksoftirqd/207:25:132
3885679993321,9cyclictest34-21ksoftirqd/209:35:012
3885679993320,9cyclictest3996061-21cron12:24:562
3885679993317,5cyclictest34-21ksoftirqd/211:37:322
3885679993317,13cyclictest34-21ksoftirqd/210:49:572
3885674993318,11cyclictest3961796-21ls10:45:001
388566999335,11cyclictest13-21ksoftirqd/010:24:590
3885669993323,7cyclictest1137-21runrttasks08:17:460
3885669993321,8cyclictest3891276-21cron07:24:590
3885669993318,11cyclictest3948364-21tr10:05:130
3885669993315,13cyclictest438-20systemd-journal08:23:100
39231392321,4sleep00-21swapper/008:55:030
388568399321,8cyclictest4497-21PK-Backend12:11:593
388568399321,8cyclictest4497-21PK-Backend12:11:583
388568399321,28cyclictest4497-21PK-Backend10:18:553
3885679993222,7cyclictest3898699-21cpuspeed_turbos07:45:142
3885679993222,6cyclictest34-21ksoftirqd/209:45:052
3885679993222,5cyclictest34-21ksoftirqd/208:20:162
388567999321,7cyclictest438-20systemd-journal12:10:002
3885679993214,14cyclictest3914074-40apt-check08:25:562
388567999321,28cyclictest0-21swapper/207:53:092
388567499321,4cyclictest4497-21PK-Backend11:46:221
388567499321,4cyclictest4497-21PK-Backend11:46:221
388566999325,11cyclictest13-21ksoftirqd/010:46:480
3885669993221,6cyclictest13-21ksoftirqd/007:45:390
3885669993218,10cyclictest3933323-21ntpq09:20:250
3885669993218,10cyclictest3933323-21ntpq09:20:240
3885669993217,11cyclictest3919059-21seq08:40:280
3885669993216,12cyclictest3946473-21awk10:00:090
3885669993213,8cyclictest3991308-21cpuspeed_turbos12:05:140
388566999321,28cyclictest0-21swapper/009:05:010
3885683993116,11cyclictest3947004-21grep10:00:193
3885679993122,5cyclictest34-21ksoftirqd/209:35:142
388567499315,21cyclictest4003020-21libinput-device12:35:411
388567499311,27cyclictest4497-21PK-Backend08:49:051
388566999316,10cyclictest13-21ksoftirqd/010:31:240
388566999316,10cyclictest13-21ksoftirqd/010:31:240
388566999312,9cyclictest13-21ksoftirqd/008:05:230
3885669993119,8cyclictest2069-21gnome-shell11:03:090
3885669993115,12cyclictest3917409-21timerandwakeup08:35:300
197399310,5rtkit-daemon0-21swapper/211:15:052
3885683993016,10cyclictest3892138-21load07:25:203
3885669993013,12cyclictest2345-21gdbus12:19:560
3885674992922,4cyclictest4497-21PK-Backend10:50:431
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional