You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-29 - 08:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Mon Dec 29, 2025 00:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
96464322960,8sleep0964528-21cstates19:05:130
1973991170,9rtkit-daemon1972-21rtkit-daemon21:11:430
1973991170,9rtkit-daemon1972-21rtkit-daemon21:11:430
107689421120,12sleep01076899-21cpuspeed_turbos00:20:150
197399920,8rtkit-daemon1972-21rtkit-daemon19:35:273
10147732840,7sleep2331rcuc/221:25:162
10356872830,8sleep34497-21PK-Backend22:20:263
96573227740,14sleep30-21swapper/319:09:353
9646962770,17sleep234-21ksoftirqd/219:05:162
10337952770,5sleep30-21swapper/322:15:243
10386972710,8sleep14497-21PK-Backend22:30:151
197399670,9rtkit-daemon1972-21rtkit-daemon22:19:512
197399670,4rtkit-daemon4497-21PK-Backend23:08:443
197399670,4rtkit-daemon4497-21PK-Backend23:08:443
197399670,4rtkit-daemon4497-21PK-Backend22:59:313
96550026540,20sleep10-21swapper/119:06:371
197399650,8rtkit-daemon1972-21rtkit-daemon19:18:122
197399650,8rtkit-daemon1972-21rtkit-daemon00:28:562
197399650,4rtkit-daemon0-21swapper/323:33:243
197399650,4rtkit-daemon0-21swapper/323:33:243
96599299640,60cyclictest1972-21rtkit-daemon22:12:422
197399640,9rtkit-daemon1972-21rtkit-daemon20:56:322
197399640,8rtkit-daemon1972-21rtkit-daemon20:12:012
197399630,9rtkit-daemon1972-21rtkit-daemon23:57:312
197399630,8rtkit-daemon1972-21rtkit-daemon19:56:252
197399630,8rtkit-daemon1972-21rtkit-daemon00:38:342
96599299620,58cyclictest1972-21rtkit-daemon21:09:112
197399620,8rtkit-daemon1972-21rtkit-daemon23:16:462
197399610,9rtkit-daemon1972-21rtkit-daemon22:41:563
197399610,9rtkit-daemon1972-21rtkit-daemon22:41:553
197399610,8rtkit-daemon1972-21rtkit-daemon22:20:272
197399600,8rtkit-daemon1972-21rtkit-daemon23:38:222
197399600,8rtkit-daemon1972-21rtkit-daemon23:24:522
197399600,8rtkit-daemon1972-21rtkit-daemon00:05:402
197399590,9rtkit-daemon1972-21rtkit-daemon22:46:032
197399590,9rtkit-daemon1972-21rtkit-daemon20:25:503
197399590,9rtkit-daemon1972-21rtkit-daemon19:29:482
197399590,8rtkit-daemon1972-21rtkit-daemon21:39:353
197399580,8rtkit-daemon1972-21rtkit-daemon23:25:533
197399570,9rtkit-daemon1972-21rtkit-daemon20:18:492
197399570,9rtkit-daemon1972-21rtkit-daemon19:36:382
197399570,9rtkit-daemon1972-21rtkit-daemon00:23:112
197399570,8rtkit-daemon1972-21rtkit-daemon19:25:113
197399560,6rtkit-daemon1972-21rtkit-daemon23:39:593
197399550,8rtkit-daemon1972-21rtkit-daemon22:08:582
10616252550,5sleep1251rcuc/123:35:261
197399540,9rtkit-daemon1972-21rtkit-daemon22:32:173
197399540,8rtkit-daemon1972-21rtkit-daemon22:56:462
197399540,8rtkit-daemon1972-21rtkit-daemon21:47:382
197399540,8rtkit-daemon1972-21rtkit-daemon21:23:113
197399540,6rtkit-daemon1972-21rtkit-daemon20:06:182
197399530,8rtkit-daemon1972-21rtkit-daemon19:10:362
197399530,8rtkit-daemon1972-21rtkit-daemon19:10:362
96599299500,46cyclictest1972-21rtkit-daemon21:11:442
96599299500,46cyclictest1972-21rtkit-daemon21:11:432
965981994919,14cyclictest1024436-21missed_timers21:50:230
10508652470,15sleep1261ksoftirqd/123:05:191
10508652470,15sleep1261ksoftirqd/123:05:191
9910132460,17sleep126-21ksoftirqd/120:15:321
197399450,36rtkit-daemon466-21systemd-udevd23:28:402
10369462450,6sleep10-21swapper/122:25:151
96599799440,40cyclictest1972-21rtkit-daemon21:56:393
197399440,28rtkit-daemon0-21swapper/220:01:542
197399440,28rtkit-daemon0-21swapper/220:01:542
9742132430,6chrt4497-21PK-Backend19:30:141
965992994312,13cyclictest141rcu_preempt22:30:192
965981994323,16cyclictest1032914-21unin-run22:15:020
965981994313,26cyclictest1017848-40lsb_release21:32:000
965981994312,13cyclictest1009018-21systemd-udevd21:08:460
10772822430,6sleep3411rcuc/300:20:223
96599799423,22cyclictest141rcu_preempt20:10:223
96599299420,39cyclictest1972-21rtkit-daemon21:31:062
965988994224,13cyclictest438-20systemd-journal20:33:301
965988994222,6cyclictest26-21ksoftirqd/121:55:001
965997994123,13cyclictest1031965-21memory22:10:233
96598899416,9cyclictest26-21ksoftirqd/120:41:521
965981994120,17cyclictest975664-21latency_hist19:35:010
965981994110,27cyclictest1076231-21fuser00:17:070
96599799402,15cyclictest141rcu_preempt20:49:593
96599799402,15cyclictest141rcu_preempt20:49:593
965988994025,11cyclictest923-21dbus-daemon22:39:561
965988994021,6cyclictest26-21ksoftirqd/122:05:521
965981994023,12cyclictest438-20systemd-journal19:22:180
965981994022,6cyclictest13-21ksoftirqd/022:22:040
965981994022,14cyclictest997304-21df_inode20:35:180
965981994022,14cyclictest1049820-21users23:00:360
10766712400,5chrt4497-21PK-Backend00:20:033
9683442390,7chrt4497-21PK-Backend19:15:031
96599799398,16cyclictest1016856-21expr21:30:143
965997993913,22cyclictest968518-21processes19:15:073
96599799390,35cyclictest1972-21rtkit-daemon23:20:223
965992993924,11cyclictest1025338-21cron21:55:012
96598899399,6cyclictest141rcu_preempt21:29:571
96598899395,5cyclictest141rcu_preempt00:28:251
965988993922,13cyclictest1083346-21sshd00:39:321
96598899392,10cyclictest26-21ksoftirqd/123:20:161
965981993927,8cyclictest1041552-21grep22:40:010
965981993911,24cyclictest1046761-21unin-run22:55:000
965997993817,18cyclictest991165-40apt-check20:16:533
965997993810,24cyclictest1-21systemd20:00:013
96599799380,35cyclictest1972-21rtkit-daemon20:44:293
96599799380,34cyclictest1972-21rtkit-daemon20:50:233
96599299386,10cyclictest34-21ksoftirqd/222:25:122
965992993826,7cyclictest994120-21tail20:25:222
965992993823,11cyclictest975621-21systemd-run19:35:022
965992993820,14cyclictest1975-21dbus-daemon20:41:372
965992993820,13cyclictest1020803-21cut21:40:222
965988993821,12cyclictest1023328-21/usr/sbin/munin21:50:001
965988993819,14cyclictest1063303-21awk23:40:241
96598199389,5cyclictest141rcu_preempt20:10:230
965981993822,11cyclictest438-20systemd-journal21:25:000
965981993820,6cyclictest13-21ksoftirqd/019:28:550
197399380,30rtkit-daemon965975-21cyclictest22:53:562
197399380,23rtkit-daemon4497-21PK-Backend19:23:252
96599799372,31cyclictest974503-21diskstats19:30:153
965997993723,11cyclictest42-21ksoftirqd/300:10:313
965997993723,10cyclictest1008927-21taskset21:08:003
965997993720,12cyclictest1006384-21ntpq21:00:283
965997993712,7cyclictest10827022open_files00:35:293
965992993721,12cyclictest977735-21grep19:40:162
965988993723,10cyclictest1065756-21cron23:50:001
965988993722,11cyclictest923-21dbus-daemon19:55:001
965988993722,11cyclictest1044986-21cron22:50:011
96598899371,13cyclictest9923712cat20:20:261
965981993728,5cyclictest991456-21cron20:20:000
965981993726,7cyclictest161rcuc/020:48:550
965981993726,7cyclictest161rcuc/020:48:550
965981993722,11cyclictest923-21dbus-daemon20:29:560
965981993720,5cyclictest13-21ksoftirqd/021:16:050
965981993720,13cyclictest1064895-21memory23:45:240
965981993717,17cyclictest1061009-21tr23:35:130
96598199371,5cyclictest141rcu_preempt23:40:170
96598199371,5cyclictest141rcu_preempt22:06:190
96598199371,5cyclictest141rcu_preempt00:30:300
96598199371,14cyclictest0-21swapper/019:45:290
197399370,30rtkit-daemon968110-21munin-node19:51:292
965997993622,10cyclictest1012377-21cron21:16:593
96599299368,12cyclictest141rcu_preempt21:05:002
965992993618,14cyclictest34-21ksoftirqd/200:17:082
965988993625,7cyclictest1004317-21sort20:55:201
965988993621,11cyclictest26-21ksoftirqd/100:16:541
965988993620,11cyclictest438-20systemd-journal23:02:311
965988993619,13cyclictest923-21dbus-daemon22:20:011
965981993626,7cyclictest161rcuc/019:38:370
965981993626,7cyclictest161rcuc/000:26:150
965981993621,11cyclictest992818-21uptime20:20:330
965981993620,12cyclictest978240-21needreboot19:40:250
965981993619,13cyclictest1-21systemd00:40:010
96598199361,5cyclictest141rcu_preempt21:48:590
96598199361,5cyclictest141rcu_preempt20:10:000
965981993615,17cyclictest1018382-21acpi21:35:120
96598199361,4cyclictest141rcu_preempt23:10:270
96598199361,4cyclictest141rcu_preempt22:55:170
197399360,27rtkit-daemon4497-21PK-Backend23:58:003
965997993520,11cyclictest995495-21cat20:30:153
965997993518,12cyclictest1009952-21latency21:10:223
965997993518,12cyclictest1009952-21latency21:10:223
965997993516,15cyclictest997635-21netstat20:35:253
965992993520,11cyclictest923-21dbus-daemon23:10:022
965992993520,11cyclictest923-21dbus-daemon23:10:022
96599299351,7cyclictest924-21NetworkManager23:51:232
965992993513,6cyclictest141rcu_preempt20:45:162
965992993513,6cyclictest141rcu_preempt20:45:152
96599299351,14cyclictest924-21NetworkManager23:10:022
965988993519,12cyclictest1036295-21seq22:24:331
965988993519,11cyclictest438-20systemd-journal22:50:311
965988993518,12cyclictest1018797-21which21:35:201
965988993517,13cyclictest1020252-21tr21:40:131
965988993515,16cyclictest1013873-21sed21:20:291
96598899351,14cyclictest1137-21runrttasks20:10:361
96598199356,10cyclictest13-21ksoftirqd/021:02:230
96598199355,7cyclictest13-21ksoftirqd/019:15:330
965981993526,6cyclictest161rcuc/023:16:370
965981993525,7cyclictest1137-21runrttasks19:55:270
965981993525,6cyclictest0-21swapper/020:50:240
965981993522,9cyclictest1039815-21turbostat22:35:000
965981993520,11cyclictest976-21rs:main0
965981993519,12cyclictest984912-21hddtemp_smartct20:00:200
965981993519,12cyclictest984912-21hddtemp_smartct20:00:190
96598199351,5cyclictest141rcu_preempt19:10:240
96598199351,5cyclictest141rcu_preempt19:10:230
96598199351,4cyclictest141rcu_preempt23:50:260
96598199351,4cyclictest141rcu_preempt21:25:200
965981993513,6cyclictest141rcu_preempt23:05:190
965981993513,6cyclictest141rcu_preempt23:05:190
965981993513,13cyclictest13-21ksoftirqd/023:30:090
965981993513,13cyclictest13-21ksoftirqd/023:30:090
197399350,27rtkit-daemon4497-21PK-Backend20:21:283
96599799349,21cyclictest42-21ksoftirqd/320:06:253
965997993419,12cyclictest970624-21apt19:20:113
965997993419,11cyclictest1080285-21cat00:30:143
965997993419,10cyclictest1030846-21apt-get22:10:023
965997993418,12cyclictest1054568-21needreboot23:15:223
965997993418,12cyclictest1037946-21seq22:28:323
965997993417,13cyclictest967234-21python319:10:283
965997993417,13cyclictest967234-21python319:10:273
96599799341,30cyclictest4497-21PK-Backend21:48:113
96599299347,11cyclictest34-21ksoftirqd/200:14:012
965992993418,12cyclictest923-21dbus-daemon23:35:012
965992993418,12cyclictest923-21dbus-daemon23:35:002
96599299341,16cyclictest924-21NetworkManager23:40:312
96598899343,10cyclictest26-21ksoftirqd/120:46:361
96598899343,10cyclictest26-21ksoftirqd/120:46:351
96598899342,11cyclictest1054094-21grep23:15:131
965988993420,10cyclictest997798-21/usr/sbin/munin20:35:281
965988993419,12cyclictest1058792-21sshd23:29:311
965988993419,11cyclictest1053065-21seq23:10:301
965988993419,11cyclictest1027795-21grep22:00:151
96598899341,7cyclictest971424-21grep19:20:251
965988993417,12cyclictest1077469-21ntpq00:20:251
965988993412,11cyclictest1031372-21acpi22:10:131
96598899341,13cyclictest4497-21PK-Backend21:07:541
965981993424,6cyclictest982502-21cron19:55:000
965981993419,11cyclictest1058833-21cron23:30:000
965981993419,11cyclictest1036603-21cat22:25:110
965981993419,11cyclictest1033888-21cat22:15:250
965981993419,11cyclictest1020475-21cat21:40:150
965981993417,13cyclictest996453-21sshd20:32:310
965981993417,12cyclictest466-21systemd-udevd20:41:360
965981993417,12cyclictest438-20systemd-journal00:00:000
965981993412,13cyclictest13-21ksoftirqd/021:55:110
965981993410,5cyclictest141rcu_preempt00:00:230
197399340,28rtkit-daemon0-21swapper/200:34:212
197399340,27rtkit-daemon1074-21lldpd22:36:042
965997993326,4cyclictest4497-21PK-Backend22:50:433
965997993319,10cyclictest923-21dbus-daemon21:00:013
965997993318,11cyclictest1063304-21ntp_kernel_pll_23:40:243
965997993318,11cyclictest1016092-21apt-get21:30:003
965997993317,12cyclictest986049-21wc20:05:013
965997993317,12cyclictest986049-21wc20:05:013
965997993317,12cyclictest923-21dbus-daemon00:10:003
96599799331,29cyclictest4497-21PK-Backend22:37:353
965992993321,8cyclictest1012187-21grep21:15:312
96599299332,12cyclictest141rcu_preempt23:00:352
965992993319,10cyclictest979312-21awk19:45:112
965992993318,11cyclictest1042390-21load22:40:222
965992993318,11cyclictest1042390-21load22:40:212
965992993318,11cyclictest1019356-21cat21:35:292
965992993317,11cyclictest438-20systemd-journal20:54:592
965992993315,13cyclictest34-21ksoftirqd/222:00:082
96599299331,29cyclictest4497-21PK-Backend21:23:052
965988993320,5cyclictest231irq_work/100:08:581
965988993317,5cyclictest26-21ksoftirqd/122:40:131
965988993317,5cyclictest26-21ksoftirqd/122:40:131
965988993317,11cyclictest1068496-21grep23:55:241
965988993316,12cyclictest923-21dbus-daemon19:10:041
965988993316,12cyclictest923-21dbus-daemon19:10:031
96598899331,29cyclictest0-21swapper/119:42:151
96598899331,28cyclictest4497-21PK-Backend20:06:381
96598899331,15cyclictest4497-21PK-Backend21:15:131
965981993319,11cyclictest1056906-21sshd23:22:310
965981993319,10cyclictest1004453-21cat20:55:230
965981993316,12cyclictest1044426-21switchtime22:45:290
965981993316,11cyclictest1071284-21cpu00:05:120
96598199331,29cyclictest4497-21PK-Backend00:11:110
197399330,9rtkit-daemon931-21irqbalance20:33:072
96599799322,11cyclictest1044655-21chrt22:45:453
965997993216,12cyclictest1079974-21pluginstate00:30:013
96599799321,28cyclictest4497-21PK-Backend23:50:263
96599799321,28cyclictest4497-21PK-Backend19:53:243
96599799321,28cyclictest4497-21PK-Backend19:41:123
96599799321,28cyclictest0-21swapper/300:01:233
96599799321,27cyclictest1025260-21chrt21:54:023
96599799321,12cyclictest1028688-21sed22:00:303
965992993217,11cyclictest1025802-21grep21:55:142
96599299321,29cyclictest0-21swapper/223:48:112
965988993226,4cyclictest0-21swapper/119:39:261
965988993216,12cyclictest980188-21python319:45:281
965988993215,12cyclictest923-21dbus-daemon20:00:011
96598899321,4cyclictest4497-21PK-Backend21:55:451
96598899321,4cyclictest0-21swapper/119:26:541
96598899321,28cyclictest4497-21PK-Backend21:01:591
96598899321,13cyclictest4497-21PK-Backend00:30:161
96598199324,5cyclictest141rcu_preempt22:40:140
96598199324,5cyclictest141rcu_preempt22:40:130
96599799316,21cyclictest980346-21timerwakeupswit19:45:313
965997993117,10cyclictest1064811-21sed23:45:243
965997993115,5cyclictest141rcu_preempt23:05:023
96599799311,25cyclictest1053819-21sed23:15:013
965992993117,10cyclictest1069890-21seq00:00:152
965992993115,5cyclictest34-21ksoftirqd/220:20:242
965988993120,7cyclictest1066403-21taskset23:50:171
965988993119,8cyclictest2069-21gnome-shell21:31:581
965988993117,10cyclictest985222-21ntpq20:00:241
965988993117,10cyclictest985222-21ntpq20:00:231
965988993115,12cyclictest1047784-21awk22:55:301
96598899311,4cyclictest4497-21PK-Backend23:34:351
96598899311,4cyclictest4497-21PK-Backend23:34:341
965988993114,12cyclictest438-20systemd-journal00:15:001
965992993020,5cyclictest34-21ksoftirqd/220:35:292
965988993014,12cyclictest923-21dbus-daemon00:00:021
965988993014,11cyclictest438-20systemd-journal21:11:321
965988993014,11cyclictest438-20systemd-journal21:11:311
965988992914,11cyclictest994360-21grep20:25:291
965997992818,7cyclictest1137-21runrttasks21:40:023
965988992817,7cyclictest1002257-21cpuspeed_turbos20:50:131
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional