You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-27 - 12:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Sat Dec 27, 2025 00:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
409205921180,16sleep234-21ksoftirqd/223:00:222
409205921180,16sleep234-21ksoftirqd/223:00:222
401251221000,7sleep14497-21PK-Backend19:15:141
197399790,61rtkit-daemon0-21swapper/119:05:161
197399760,61rtkit-daemon0-21swapper/019:09:400
197399730,5rtkit-daemon0-21swapper/023:01:540
197399730,5rtkit-daemon0-21swapper/023:01:540
197399730,5rtkit-daemon0-21swapper/022:43:570
197399720,5rtkit-daemon0-21swapper/022:54:210
197399720,5rtkit-daemon0-21swapper/019:53:440
400951027139,9sleep30-21swapper/319:05:423
197399680,5rtkit-daemon0-21swapper/021:20:580
400967126445,13sleep20-21swapper/219:07:472
4009985995915,38cyclictest4062397-21missed_timers21:35:253
197399590,8rtkit-daemon1972-21rtkit-daemon00:05:343
41036092580,5sleep0161rcuc/023:35:090
40394602580,7chrt4497-21PK-Backend20:30:210
4009978995817,36cyclictest4123310-21basename00:30:001
4009985995616,35cyclictest4123665-21ls00:30:153
4009985995419,31cyclictest4108565-21systemd23:50:023
4009985995419,31cyclictest4108565-21systemd23:50:013
400997899544,46cyclictest4047758-21tr20:55:131
400997899544,20cyclictest4052702-21ls21:10:011
4009978995417,11cyclictest4090314-21grep22:55:211
41033382530,7sleep3411rcuc/323:35:003
4009985995316,32cyclictest4038285-21users20:25:313
400998299530,50cyclictest1972-21rtkit-daemon00:03:322
4009978995317,32cyclictest4497-21PK-Backend23:58:531
4009978995317,32cyclictest4497-21PK-Backend23:15:421
4009978995317,31cyclictest4089694-21wc22:55:021
41023042520,15sleep126-21ksoftirqd/123:30:191
40224712520,5sleep14497-21PK-Backend19:41:011
4009985995218,29cyclictest4118765-21ntpq00:15:253
4009985995217,31cyclictest4087846-21grep22:50:013
4009985995217,30cyclictest4065806-21sed21:45:223
4009985995217,30cyclictest4027077-21cat19:55:213
4009985995215,31cyclictest4058370-21sort21:25:123
4009978995212,11cyclictest4040000-21expr20:30:301
4009978995210,14cyclictest4033601-21cpuspeed_turbos20:15:131
400998599512,45cyclictest4081858-21ntp_states22:30:263
4009985995120,27cyclictest4016584-21ntp_kernel_err19:25:233
4009978995118,30cyclictest4497-21PK-Backend22:37:071
4009978995118,30cyclictest4497-21PK-Backend22:37:071
4009978995117,7cyclictest4054124-21timerwakeupswit21:10:311
4009978995113,12cyclictest923-21dbus-daemon23:25:011
4009978995113,12cyclictest923-21dbus-daemon23:25:011
4009978995112,10cyclictest4069857-21seq21:56:211
4009985995023,22cyclictest4048005-21diskstats20:55:173
4009985995021,25cyclictest4068421-21dump-pmu-power21:59:553
4009985995015,31cyclictest4077415-21systemd22:20:013
4009978995014,9cyclictest4094682-21turbostat.cron23:10:001
4009978995012,11cyclictest4085060-21uname22:40:201
400998599495,10cyclictest4070666-21cat22:00:113
4009985994925,20cyclictest4033600-21cpuspeed_turbos20:15:133
400998599492,43cyclictest4021693-21df_inode19:40:153
400998599492,42cyclictest4089928-21cpuspeed_turbos22:55:143
400998599492,42cyclictest4043023-21memory20:40:213
400998599492,42cyclictest4039847-21python320:30:283
400998599492,42cyclictest4029158-21python320:00:283
400997899492,17cyclictest4038355-40lsb_release20:26:021
4009978994915,7cyclictest4094928-21/usr/sbin/munin23:10:301
4009978994913,8cyclictest4060633-21missed_timers21:30:231
4009974994941,6cyclictest161rcuc/019:40:060
400998599482,41cyclictest4015099-21cat19:20:303
400998599482,41cyclictest4015099-21cat19:20:293
4009985994818,25cyclictest4119967-21grep00:20:123
4009985994817,27cyclictest4125305-21cpu00:35:113
4009985994816,28cyclictest4049640-21expr21:00:153
4009985994815,29cyclictest4100073-21latency_hist23:25:023
4009985994815,29cyclictest4100073-21latency_hist23:25:023
400998599481,43cyclictest42-21ksoftirqd/323:05:223
4009985994813,10cyclictest4019650-21expr19:35:133
4009978994816,9cyclictest4025660-21processes19:50:251
400997899481,15cyclictest4059789-21munin-run21:30:001
400998599477,11cyclictest4122448-21needreboot00:25:233
4009985994719,24cyclictest4101616-21latency_hist23:30:023
4009985994715,28cyclictest4104147-21latency23:35:183
4009985994715,27cyclictest4033241-21systemd20:15:013
400998599471,41cyclictest4108803-21apt23:50:133
4009978994715,7cyclictest4497-21PK-Backend19:11:021
4009978994710,8cyclictest4073324-21grep22:05:281
4009978994710,11cyclictest4115125-21processes00:05:281
4009974994723,20cyclictest4112334-40.mlocate00:00:010
41075242460,16sleep126-21ksoftirqd/123:45:201
41075242460,16sleep126-21ksoftirqd/123:45:201
400998599468,11cyclictest4011291-21seq19:10:373
400998599462,39cyclictest4047370-21munin-run20:55:003
4009985994622,20cyclictest4091867-21df_inode23:00:183
4009985994622,20cyclictest4091867-21df_inode23:00:173
4009985994620,22cyclictest4031093-21uptime20:05:283
4009985994616,27cyclictest1137-21runrttasks22:42:563
4009985994616,25cyclictest42-21ksoftirqd/322:25:003
400998599461,41cyclictest4088009-21/usr/sbin/munin22:50:233
4009985994614,11cyclictest4072862-21hddtemp_smartct22:05:203
4009985994610,10cyclictest4079366-21acpi22:25:123
400997899467,9cyclictest4068627-21ls21:55:011
4009978994615,7cyclictest4497-21PK-Backend23:50:561
4009978994615,7cyclictest4497-21PK-Backend20:14:491
4009978994615,7cyclictest4497-21PK-Backend00:38:401
400997899461,13cyclictest438-20systemd-journal20:35:031
400997499461,21cyclictest4070349-21date22:00:020
197399460,33rtkit-daemon4497-21PK-Backend20:04:181
400998599452,9cyclictest4068456-21munin-run21:55:013
4009985994525,16cyclictest3692989-21snapd21:33:093
4009985994517,25cyclictest4036826-21latency_hist20:25:023
4009985994517,24cyclictest1137-21runrttasks21:44:053
4009985994516,25cyclictest42-21ksoftirqd/321:05:203
4009985994516,24cyclictest4012001-21sh19:15:033
4009985994514,27cyclictest4097923-21sshd23:17:193
400998599451,39cyclictest4113471-21awk00:00:283
400998599451,39cyclictest4083393-21latency22:35:233
400998599451,39cyclictest4083393-21latency22:35:223
400998599451,39cyclictest4055002-21grep21:15:133
4009985994513,10cyclictest4074689-21kernelversion22:10:213
4009985994513,10cyclictest4074689-21kernelversion22:10:203
4009985994511,11cyclictest4017706-21cat19:30:123
4009985994510,11cyclictest4105600-21expr23:40:143
4009978994517,24cyclictest4082577-21turbostat22:35:001
4009978994513,7cyclictest4497-21PK-Backend22:45:481
400997899451,15cyclictest4113668-21taskset00:00:301
4009978994510,6cyclictest40278542chrt19:58:071
400997499459,19cyclictest13-21ksoftirqd/019:20:150
400997499459,19cyclictest13-21ksoftirqd/019:20:140
4009974994534,8cyclictest13-21ksoftirqd/000:25:030
4009974994524,8cyclictest13-21ksoftirqd/020:36:390
4009974994524,7cyclictest13-21ksoftirqd/023:11:120
4009974994522,19cyclictest4017364-21unin-run19:30:000
4009974994520,9cyclictest13-21ksoftirqd/000:05:020
4009985994417,23cyclictest4023157-21sort19:45:153
4009985994416,24cyclictest4044376-21cstates20:45:153
4009985994412,7cyclictest4116231-21tr00:10:153
4009982994414,14cyclictest141rcu_preempt23:36:492
400997899442,13cyclictest4078523-21python322:20:301
4009978994412,7cyclictest0-21swapper/121:45:301
400997899441,13cyclictest4100821-21grep23:25:221
400997899441,13cyclictest4047324-21chrt20:54:451
4009978994410,6cyclictest4043361-21wc20:40:281
4009974994413,17cyclictest13-21ksoftirqd/019:47:150
400998599431,23cyclictest4110767-21/usr/sbin/munin23:55:203
400997899438,8cyclictest4497-21PK-Backend00:20:141
4009978994313,6cyclictest40313112chrt20:08:061
400997499439,17cyclictest13-21ksoftirqd/021:10:020
4009974994323,7cyclictest13-21ksoftirqd/022:27:070
4009974994322,8cyclictest13-21ksoftirqd/021:47:460
4009974994322,8cyclictest13-21ksoftirqd/021:26:530
4009985994212,7cyclictest4040680-21/usr/sbin/munin20:35:253
4009985994211,10cyclictest4057972-21seq21:24:183
4009982994223,14cyclictest34-21ksoftirqd/219:10:032
400997899428,9cyclictest4015067-21switchtime19:20:291
400997899428,9cyclictest4015067-21switchtime19:20:281
4009978994211,7cyclictest4497-21PK-Backend21:35:501
400997899421,13cyclictest4124989-21chrt00:34:381
400997899421,11cyclictest0-21swapper/119:30:451
400997499426,16cyclictest4497-21PK-Backend21:50:140
4009974994233,6cyclictest161rcuc/021:12:210
4009974994222,7cyclictest13-21ksoftirqd/023:21:370
4009974994222,7cyclictest13-21ksoftirqd/023:21:360
4009974994222,7cyclictest13-21ksoftirqd/022:20:380
4009974994221,8cyclictest13-21ksoftirqd/022:55:080
4009974994220,8cyclictest13-21ksoftirqd/000:15:440
4009974994211,18cyclictest13-21ksoftirqd/019:33:340
400998599419,10cyclictest4025069-21cat19:50:143
4009982994124,5cyclictest34-21ksoftirqd/200:10:162
4009982994112,14cyclictest141rcu_preempt19:39:452
4009978994122,16cyclictest4497-21PK-Backend22:26:341
4009978994122,14cyclictest26-21ksoftirqd/120:21:251
400997499418,13cyclictest13-21ksoftirqd/000:15:000
400997499416,12cyclictest13-21ksoftirqd/021:04:190
4009974994134,5cyclictest161rcuc/022:18:050
4009974994121,8cyclictest13-21ksoftirqd/021:16:090
4009974994121,7cyclictest13-21ksoftirqd/020:18:540
400997499411,14cyclictest4086834-21taskset22:45:240
400998599406,9cyclictest0-21swapper/321:12:403
4009982994023,6cyclictest34-21ksoftirqd/221:16:312
4009982994022,14cyclictest34-21ksoftirqd/223:45:492
4009982994022,14cyclictest34-21ksoftirqd/223:45:492
4009982994022,14cyclictest34-21ksoftirqd/200:05:462
4009982994022,13cyclictest438-20systemd-journal22:14:592
4009982994022,13cyclictest438-20systemd-journal22:14:592
400997899401,35cyclictest4497-21PK-Backend21:42:091
400997899401,14cyclictest4071808-21chrt22:00:491
400997899401,13cyclictest4106886-21sort23:45:021
400997899401,13cyclictest4017181-21chrt19:28:161
400997499406,15cyclictest13-21ksoftirqd/021:30:260
400997499401,17cyclictest13-21ksoftirqd/020:55:010
400997499401,16cyclictest4093430-21cat23:05:120
4009985993911,24cyclictest0-21swapper/323:12:033
4009982993928,7cyclictest4045877-21sort20:50:122
4009982993923,13cyclictest34-21ksoftirqd/200:16:142
4009982993922,5cyclictest34-21ksoftirqd/220:25:092
4009982993922,13cyclictest34-21ksoftirqd/223:20:222
4009982993922,13cyclictest34-21ksoftirqd/223:20:222
4009982993922,13cyclictest34-21ksoftirqd/223:10:112
4009982993921,6cyclictest34-21ksoftirqd/220:21:052
400998299391,34cyclictest4045629-21latency_hist20:50:022
400997899391,13cyclictest4050620-21chrt21:02:391
400997899391,13cyclictest4045531-21chrt20:48:571
400997899391,13cyclictest4020529-21chrt19:36:471
400997499398,17cyclictest13-21ksoftirqd/020:12:490
400997499398,12cyclictest13-21ksoftirqd/020:25:150
400997499397,12cyclictest13-21ksoftirqd/020:25:000
400997499396,17cyclictest13-21ksoftirqd/020:48:080
4009974993931,5cyclictest161rcuc/023:41:130
4009974993930,6cyclictest161rcuc/019:14:220
400997499391,21cyclictest4108452-21taskset23:49:110
400997499391,21cyclictest4108452-21taskset23:49:110
400997499391,18cyclictest13-21ksoftirqd/023:15:290
400997499391,15cyclictest4048927-21taskset20:58:080
400997499391,14cyclictest13-21ksoftirqd/020:05:150
197399390,25rtkit-daemon0-21swapper/021:35:410
400998299385,28cyclictest4122962-40lsb_release00:26:042
400998299382,7cyclictest4042920-21irqstats20:40:202
4009982993823,12cyclictest4050750-21sshd21:04:192
4009982993822,13cyclictest34-21ksoftirqd/219:59:562
4009982993821,5cyclictest34-21ksoftirqd/221:35:002
4009982993821,13cyclictest34-21ksoftirqd/222:22:052
4009982993821,12cyclictest438-20systemd-journal21:12:192
4009982993820,14cyclictest4012726-21expr19:15:142
400997899381,13cyclictest4075573-21chrt22:14:131
400997899381,13cyclictest4075573-21chrt22:14:131
400997499389,16cyclictest13-21ksoftirqd/022:38:510
400997499389,16cyclictest13-21ksoftirqd/022:38:510
400997499386,10cyclictest4497-21PK-Backend23:25:160
400997499382,13cyclictest0-21swapper/019:39:520
4009974993821,13cyclictest4012148-21df_inode19:15:040
197399380,27rtkit-daemon4497-21PK-Backend23:01:161
197399380,27rtkit-daemon4497-21PK-Backend23:01:161
40580302370,3sleep24058075-21munin-run21:25:022
400998299372,31cyclictest4023472-21diskstats19:45:182
4009982993721,5cyclictest34-21ksoftirqd/223:50:242
4009982993721,13cyclictest34-21ksoftirqd/220:35:012
4009982993721,12cyclictest923-21dbus-daemon00:00:012
4009982993721,12cyclictest34-21ksoftirqd/221:50:262
4009982993720,6cyclictest34-21ksoftirqd/222:01:572
4009982993719,14cyclictest34-21ksoftirqd/222:54:552
4009982993717,15cyclictest4086676-21df_inode22:45:182
400998299371,33cyclictest4100121-21/usr/sbin/munin23:25:282
400997899371,13cyclictest4077247-21chrt22:18:271
400997499376,16cyclictest13-21ksoftirqd/000:05:140
400997499376,15cyclictest13-21ksoftirqd/022:30:160
400997499374,16cyclictest13-21ksoftirqd/023:50:150
4009974993713,7cyclictest13-21ksoftirqd/000:24:570
197399370,25rtkit-daemon0-21swapper/000:30:450
400998299369,7cyclictest4048729-21taskset20:55:342
400998299362,18cyclictest4083222-21diskstats22:35:182
400998299362,18cyclictest4083222-21diskstats22:35:182
4009982993620,12cyclictest4068691-21apt21:55:112
4009982993620,12cyclictest4016014-21expr19:25:132
4009982993620,12cyclictest34-21ksoftirqd/222:33:112
4009982993620,11cyclictest438-20systemd-journal19:40:192
4009982993619,14cyclictest34-21ksoftirqd/222:40:542
4009982993619,13cyclictest4090507-21ntp_states22:55:242
4009982993619,13cyclictest34-21ksoftirqd/220:13:212
4009982993618,13cyclictest438-20systemd-journal20:20:002
4009982993618,13cyclictest34-21ksoftirqd/223:40:292
4009982993613,19cyclictest4077413-21munin-run22:20:012
400997899364,8cyclictest0-21swapper/100:15:231
4009978993619,13cyclictest26-21ksoftirqd/100:12:291
400997499363,16cyclictest13-21ksoftirqd/021:40:140
4009974993627,6cyclictest161rcuc/020:41:220
197399360,23rtkit-daemon4497-21PK-Backend21:21:341
4009982993525,6cyclictest34-21ksoftirqd/221:35:592
4009982993524,6cyclictest34-21ksoftirqd/221:45:202
4009982993521,10cyclictest4018310-21ntp_kernel_err19:30:232
4009982993519,6cyclictest34-21ksoftirqd/222:26:562
4009982993519,5cyclictest34-21ksoftirqd/222:07:572
4009982993514,5cyclictest141rcu_preempt20:05:152
400998299351,30cyclictest4025039-21seq19:50:132
400997899358,5cyclictest141rcu_preempt21:15:171
400997499351,16cyclictest4027107-21latency19:55:210
400997499351,13cyclictest13-21ksoftirqd/022:00:250
4009982993424,6cyclictest34-21ksoftirqd/223:30:152
4009982993424,6cyclictest34-21ksoftirqd/221:40:182
4009982993423,6cyclictest34-21ksoftirqd/221:25:242
4009982993422,7cyclictest34-21ksoftirqd/223:10:002
4009982993419,11cyclictest4126749-21cron00:40:002
400998299341,30cyclictest0-21swapper/223:19:122
400997499345,12cyclictest0-21swapper/022:08:140
4009974993424,7cyclictest13-21ksoftirqd/022:10:170
4009974993424,7cyclictest13-21ksoftirqd/022:10:160
4009974993424,7cyclictest13-21ksoftirqd/000:36:090
4009982993318,11cyclictest4123731-21grep00:30:142
4009982993318,11cyclictest4051592-21cut21:05:212
4009982993316,12cyclictest4015207-21sed19:20:292
4009982993316,12cyclictest4015207-21sed19:20:292
400998299331,5cyclictest4028685-21hddtemp_smartct20:00:192
4009982993314,7cyclictest141rcu_preempt00:20:102
400998299331,29cyclictest0-21swapper/220:36:162
4009978993326,4cyclictest4497-21PK-Backend19:48:381
400997499331,16cyclictest4497-21PK-Backend20:00:290
400997899321,4cyclictest4497-21PK-Backend23:35:401
400997499322,12cyclictest13-21ksoftirqd/023:30:040
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional