You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-25 - 14:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu Dec 25, 2025 00:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
285507622650,9sleep00-21swapper/019:05:120
197399920,4rtkit-daemon4497-21PK-Backend00:19:191
29050862910,5sleep10-21swapper/121:25:131
197399900,24rtkit-daemon0-21swapper/019:18:220
197399900,24rtkit-daemon0-21swapper/019:18:210
2856431997816,35cyclictest2960894-21cat00:05:012
2856431997816,35cyclictest2960894-21cat00:05:012
2856431997610,37cyclictest2968922-21missed_timers00:25:222
197399750,60rtkit-daemon0-21swapper/119:05:051
2856431997414,41cyclictest2893038-21ntp_kernel_err20:50:252
2856431997414,41cyclictest2893038-21ntp_kernel_err20:50:252
2856431997019,31cyclictest1137-21runrttasks21:39:572
2856431997019,31cyclictest1137-21runrttasks21:39:572
197399670,9rtkit-daemon1972-21rtkit-daemon23:52:441
197399670,9rtkit-daemon1972-21rtkit-daemon00:35:011
285642299650,61cyclictest1972-21rtkit-daemon23:55:290
285605426546,13sleep20-21swapper/219:07:452
29494602640,5sleep00-21swapper/023:30:280
285589326343,14sleep30-21swapper/319:05:403
197399610,9rtkit-daemon1972-21rtkit-daemon21:56:391
197399600,9rtkit-daemon1972-21rtkit-daemon22:03:041
197399600,9rtkit-daemon1972-21rtkit-daemon21:52:590
2856431995920,34cyclictest2921080-21cat22:10:222
197399590,8rtkit-daemon1972-21rtkit-daemon22:40:271
285643199582,33cyclictest0-21swapper/219:30:122
2856431995817,36cyclictest2923952-21expr22:20:132
2856431995814,39cyclictest2920245-21cat22:10:012
197399580,9rtkit-daemon1972-21rtkit-daemon23:56:121
197399580,9rtkit-daemon1972-21rtkit-daemon23:51:210
197399580,9rtkit-daemon1972-21rtkit-daemon21:23:131
197399580,8rtkit-daemon1972-21rtkit-daemon23:21:183
2856431995719,33cyclictest2959269-21date00:00:022
197399570,8rtkit-daemon1972-21rtkit-daemon23:25:170
2856441995617,35cyclictest2920105-21unin-run22:10:003
2856431995619,32cyclictest2963441-21sh00:10:162
2856431995616,36cyclictest2858468-21ls19:15:022
2856431995616,36cyclictest2858468-21ls19:15:022
197399560,9rtkit-daemon1972-21rtkit-daemon23:41:341
197399560,8rtkit-daemon1972-21rtkit-daemon21:04:161
197399560,10rtkit-daemon1972-21rtkit-daemon23:06:093
2856441995520,30cyclictest2859090-21/usr/sbin/munin19:15:123
2856441995520,30cyclictest2859090-21/usr/sbin/munin19:15:113
2856431995518,32cyclictest2894650-21expr20:55:232
197399550,9rtkit-daemon1972-21rtkit-daemon20:15:483
2856441995418,11cyclictest2944022-21cat23:15:223
2856431995415,33cyclictest2938159-21cat23:00:142
2856431995412,37cyclictest2947444-21cat23:25:232
197399540,9rtkit-daemon1972-21rtkit-daemon21:39:021
197399540,9rtkit-daemon1972-21rtkit-daemon21:39:021
197399540,8rtkit-daemon1972-21rtkit-daemon21:41:540
2856441995328,21cyclictest2895513-21perf21:00:003
285643199531,9cyclictest2193-21gdbus21:59:572
2856431995317,31cyclictest2866697-21expr19:35:142
2856431995313,35cyclictest2885685-21cat20:30:152
2856431995313,34cyclictest2917102-21cat22:00:152
197399530,8rtkit-daemon1972-21rtkit-daemon23:24:051
2856441995214,12cyclictest2959923-21cat00:00:243
2856441995214,12cyclictest2959923-21cat00:00:243
2856431995213,34cyclictest2927927-21expr22:30:172
2856431995213,34cyclictest2897622-21cat21:05:132
2856431995212,35cyclictest2950278-21date23:35:022
197399520,44rtkit-daemon3692979-21snapd21:56:100
2856441995114,11cyclictest2913055-21seq21:49:153
2856441995114,11cyclictest2899746-21expr21:10:143
2856431995118,28cyclictest2913183-21date21:50:012
2856431995117,30cyclictest2858376-21munin-node19:15:062
2856431995117,30cyclictest2858376-21munin-node19:15:062
2856431995116,30cyclictest2922735-21needreboot22:15:222
2856431995115,31cyclictest2882170-21expr20:20:142
2856431995111,35cyclictest2931171-21tr22:40:132
285644199502,44cyclictest2938332-21diskstats23:00:163
285644199502,43cyclictest2933159-21hddtemp_smartct22:45:183
2856441995015,10cyclictest2882550-21cat20:20:213
2856441995014,10cyclictest2893586-21seq20:52:373
2856441995014,10cyclictest2893586-21seq20:52:373
2856441995013,11cyclictest2902670-21seq21:19:173
2856441995012,11cyclictest2954260-21cat23:45:183
285643199502,10cyclictest2863985-40lsb_release19:26:052
2856431995015,30cyclictest2878519-21cpu20:10:132
2856431995011,34cyclictest2961969-21wc00:05:292
2856441994920,8cyclictest2964245-40lsb_release00:11:063
2856441994919,11cyclictest2956735-21sed23:50:293
2856441994916,8cyclictest2966307-21perf00:20:003
2856441994914,11cyclictest2928541-21sort22:30:323
2856441994911,11cyclictest2920735-21awk22:10:173
2856431994917,29cyclictest4497-21PK-Backend20:26:112
2856431994917,29cyclictest4497-21PK-Backend20:26:112
2856431994915,30cyclictest2967191-21missed_timers00:20:212
2856431994915,30cyclictest2952192-21cpuspeed23:40:142
2856431994915,29cyclictest2871765-21ls19:50:132
2856431994910,35cyclictest2900491-21timerandwakeup21:10:302
2856422994916,10cyclictest13-21ksoftirqd/020:50:160
2856422994916,10cyclictest13-21ksoftirqd/020:50:150
197399490,42rtkit-daemon1051-21thermald22:35:211
285644199486,17cyclictest2864799-21cstates19:30:123
2856441994818,8cyclictest2926758-21grep22:25:283
2856441994817,6cyclictest2917094-21cstates22:00:153
2856441994813,10cyclictest2923327-21seq22:17:283
285644199481,12cyclictest2951625-21taskset23:37:283
285644199481,12cyclictest2951625-21taskset23:37:283
285643199489,34cyclictest2965760-21switchtime00:15:292
2856431994816,28cyclictest4497-21PK-Backend20:44:302
2856431994813,31cyclictest2972384-21missed_timers00:35:202
2856431994811,32cyclictest2954235-21sh23:45:182
2856441994717,7cyclictest1137-21runrttasks20:02:583
2856441994717,7cyclictest1137-21runrttasks20:02:583
2856441994716,7cyclictest2883864-21cstates20:25:153
2856441994716,7cyclictest2883864-21cstates20:25:153
2856441994715,7cyclictest2871762-21cstates19:50:143
2856441994712,10cyclictest2967130-21ls00:20:193
285643199472,7cyclictest976-21rs:main2
285643199472,7cyclictest976-21rs:main2
285643199472,14cyclictest2946361-21taskset23:21:342
2856431994716,27cyclictest2913406-21cpu21:50:132
2856431994714,29cyclictest2943599-21tr23:15:142
29127532460,7chrt0-21swapper/021:45:290
285644199469,12cyclictest2969247-21cat00:25:293
2856441994617,8cyclictest4497-21PK-Backend21:28:253
2856441994615,9cyclictest2915863-21grep21:55:223
2856441994615,6cyclictest2866335-21/usr/sbin/munin19:35:133
2856441994614,8cyclictest2856948-21cpuspeed_turbos19:10:123
2856441994614,8cyclictest2856948-21cpuspeed_turbos19:10:123
2856441994613,12cyclictest1989-21gdbus21:53:013
285644199461,22cyclictest2923782-21/usr/sbin/munin22:20:173
2856441994610,11cyclictest976-21rs:main3
2856441994610,11cyclictest976-21rs:main3
285643199467,11cyclictest2901559-21expr21:15:152
285643199462,14cyclictest2971284-21taskset00:32:252
2856431994618,25cyclictest1137-21runrttasks20:48:242
2856431994617,25cyclictest2907932-21cron21:35:002
2856431994615,27cyclictest4497-21PK-Backend22:51:022
2856431994612,30cyclictest4497-21PK-Backend20:15:412
285643199461,11cyclictest2878203-21apt20:10:002
197399460,5rtkit-daemon4497-21PK-Backend19:49:430
197399460,4rtkit-daemon0-21swapper/120:14:451
285644199459,11cyclictest2937670-21seq22:59:583
2856441994515,11cyclictest2889158-21expr20:40:123
2856441994515,10cyclictest42-21ksoftirqd/320:30:233
2856441994513,8cyclictest2880057-21ls20:15:013
2856441994513,10cyclictest2932551-21sed22:45:023
285644199451,23cyclictest2888123-21taskset20:35:293
2856441994512,10cyclictest2873002-21cron19:59:553
2856441994511,13cyclictest2952588-21awk23:40:203
2856441994511,13cyclictest2907931-21cron21:35:003
2856441994510,11cyclictest2864187-21seq19:28:303
285643199457,33cyclictest2896061-21cstates21:00:142
2856431994513,29cyclictest4497-21PK-Backend22:47:522
2856431994513,28cyclictest1137-21runrttasks22:56:012
2856431994511,31cyclictest2926104-21expr22:25:132
28689422440,5sleep24497-21PK-Backend19:40:272
2856441994419,7cyclictest924-21NetworkManager00:09:443
2856441994411,10cyclictest2959094-21sh00:00:013
285643199441,17cyclictest2069-21gnome-shell23:53:082
2856422994422,18cyclictest2944929-21perf23:20:000
285644199432,23cyclictest2878236-21munin-run20:10:023
2856441994316,7cyclictest2970016-21/usr/sbin/munin00:30:263
2856441994315,7cyclictest2910150-21cstates21:40:153
2856441994311,7cyclictest0-21swapper/323:32:253
285643199431,14cyclictest2910128-21expr21:40:142
285643199431,12cyclictest2941333-21latency_hist23:10:022
285643199431,11cyclictest1137-21runrttasks20:01:222
285643199431,11cyclictest1137-21runrttasks20:01:222
285643199431,10cyclictest2905744-21timerandwakeup21:25:292
2856427994325,14cyclictest2941884-21cat23:10:141
285642299436,15cyclictest2878851-21diskstats20:10:170
2856422994332,8cyclictest13-21ksoftirqd/019:31:180
2856422994329,10cyclictest2909682-21ls21:40:000
2856422994329,10cyclictest2909682-21ls21:40:000
2856422994315,6cyclictest141rcu_preempt21:15:290
29615952420,7sleep00-21swapper/000:05:210
29469192420,7sleep14497-21PK-Backend23:25:141
285644199426,14cyclictest2896147-21df21:00:153
2856441994214,12cyclictest1-21systemd23:30:013
285643199421,13cyclictest2930460-21chrt22:36:172
285643199421,12cyclictest2943030-21chrt23:13:162
285643199421,12cyclictest2903019-21grep21:20:122
285643199421,12cyclictest2862046-21seq19:22:062
285643199421,11cyclictest2871505-21taskset19:50:022
2856427994216,8cyclictest4497-21PK-Backend23:37:411
2856427994216,8cyclictest4497-21PK-Backend23:37:411
197399420,35rtkit-daemon4497-21PK-Backend22:18:440
197399420,32rtkit-daemon1051-21thermald21:32:530
285644199412,17cyclictest2868468-21diskstats19:40:173
2856441994110,7cyclictest0-21swapper/321:20:023
285643199411,11cyclictest2888520-21chrt20:39:142
197399410,28rtkit-daemon13-21ksoftirqd/023:20:020
197399410,27rtkit-daemon4497-21PK-Backend19:23:390
285644199407,7cyclictest0-21swapper/320:45:293
2856427994026,10cyclictest26-21ksoftirqd/123:15:221
2856422994023,12cyclictest438-20systemd-journal22:49:550
197399400,29rtkit-daemon4497-21PK-Backend19:43:300
197399400,27rtkit-daemon4497-21PK-Backend20:30:150
197399400,27rtkit-daemon0-21swapper/300:35:283
285644199392,15cyclictest2897890-21diskstats21:05:163
2856441993914,7cyclictest924-21NetworkManager19:23:423
285644199391,21cyclictest2942354-21needreboot23:10:223
2856427993926,9cyclictest231irq_work/120:05:581
2856427993922,6cyclictest26-21ksoftirqd/121:40:201
2856422993928,7cyclictest1137-21runrttasks19:55:140
285642299391,34cyclictest4497-21PK-Backend19:50:020
285643199382,8cyclictest2874503-21chrt19:57:052
2856427993825,9cyclictest2890966-21df_inode20:45:161
2856427993822,6cyclictest26-21ksoftirqd/120:20:171
2856427993822,12cyclictest2955307-21/usr/sbin/munin23:50:011
2856427993822,12cyclictest1-21systemd19:15:021
2856427993822,12cyclictest1-21systemd19:15:011
285642799382,13cyclictest141rcu_preempt19:29:181
285642799381,34cyclictest1972-21rtkit-daemon20:56:141
2856427993813,21cyclictest2867854-21unin-run19:40:011
2856422993821,13cyclictest2929012-21chrt22:34:540
197399380,26rtkit-daemon4497-21PK-Backend20:02:070
197399380,26rtkit-daemon4497-21PK-Backend20:02:070
197399380,25rtkit-daemon4497-21PK-Backend19:29:060
197399380,25rtkit-daemon0-21swapper/020:43:070
285642799379,11cyclictest26-21ksoftirqd/120:05:011
285642799379,11cyclictest26-21ksoftirqd/120:05:001
2856427993721,11cyclictest438-20systemd-journal20:18:171
2856427993720,6cyclictest26-21ksoftirqd/119:20:201
2856427993720,13cyclictest2914684-21check-new-relea21:52:531
2856427993719,14cyclictest2926507-21ntp_kernel_err22:25:221
285642799371,33cyclictest1972-21rtkit-daemon22:12:531
2856422993720,12cyclictest438-20systemd-journal20:25:000
2856422993717,16cyclictest2973263-21unin-run00:40:000
2856422993716,6cyclictest141rcu_preempt23:35:170
2856422993716,6cyclictest141rcu_preempt23:35:170
2856422993715,7cyclictest141rcu_preempt19:10:180
2856422993715,7cyclictest141rcu_preempt19:10:180
2856422993715,19cyclictest13-21ksoftirqd/022:40:010
285642299371,18cyclictest2921838-21systemd22:15:010
285642299371,13cyclictest0-21swapper/023:45:150
197399370,25rtkit-daemon4497-21PK-Backend23:03:050
197399370,25rtkit-daemon0-21swapper/000:30:090
197399370,24rtkit-daemon0-21swapper/019:35:200
2856441993620,12cyclictest2935997-21grep22:55:013
2856427993619,13cyclictest2906866-21sshd21:30:181
285642299368,13cyclictest141rcu_preempt00:25:000
2856422993626,7cyclictest2924893-21taskset22:20:300
2856422993626,7cyclictest13-21ksoftirqd/021:10:020
2856422993625,8cyclictest13-21ksoftirqd/020:50:000
2856422993625,7cyclictest2940339-21tail23:05:220
285642299361,32cyclictest2898985-21apt-get21:10:110
285642299361,32cyclictest0-21swapper/022:41:450
285642299360,33cyclictest1972-21rtkit-daemon21:23:410
197399360,24rtkit-daemon4497-21PK-Backend00:27:040
2856427993520,11cyclictest2940042-21cat23:05:141
2856427993519,12cyclictest2884260-21wc20:25:221
2856427993519,12cyclictest2884260-21wc20:25:221
2856427993519,12cyclictest2859459-21df19:15:151
2856427993519,12cyclictest2859459-21df19:15:151
2856427993519,12cyclictest26-21ksoftirqd/119:50:171
285642799351,31cyclictest1972-21rtkit-daemon22:23:361
2856422993522,10cyclictest13-21ksoftirqd/020:35:180
2856422993518,12cyclictest438-20systemd-journal21:00:000
285642299351,30cyclictest2885118-21cron20:30:010
285642299351,30cyclictest2885118-21cron20:30:000
285642299351,15cyclictest4497-21PK-Backend00:10:170
197399350,28rtkit-daemon1941-21systemd22:06:590
197399350,24rtkit-daemon0-21swapper/022:57:190
285644199341,12cyclictest2871293-21munin-plugin-st19:50:013
285642799348,7cyclictest2069-21gnome-shell20:52:581
285642799348,7cyclictest2069-21gnome-shell20:52:581
2856427993423,6cyclictest26-21ksoftirqd/121:11:091
2856427993419,11cyclictest2963519-21awk00:10:201
2856427993419,11cyclictest2869798-21find19:45:111
2856427993418,12cyclictest2935810-21sshd22:53:181
2856427993418,12cyclictest2874542-21seq19:57:351
2856422993422,9cyclictest13-21ksoftirqd/020:20:000
285642299342,28cyclictest2877496-21ntp_states20:05:270
2856422993418,12cyclictest2917408-21grep22:00:220
2856422993415,14cyclictest2069-21gnome-shell22:26:080
285642299341,30cyclictest0-21swapper/022:52:540
285642299341,14cyclictest4497-21PK-Backend00:15:280
197399340,28rtkit-daemon1941-21systemd21:27:550
197399340,25rtkit-daemon0-21swapper/021:03:150
2856441993318,11cyclictest2929973-21grep22:35:223
285642799337,9cyclictest26-21ksoftirqd/123:30:191
285642799337,4cyclictest141rcu_preempt19:31:181
2856427993324,6cyclictest2936976-21ntpq22:55:251
2856427993319,10cyclictest2970002-21date00:30:021
2856427993319,10cyclictest2869570-21seq19:45:001
2856427993318,11cyclictest2939560-21cut23:05:021
2856427993317,12cyclictest2889342-21sshd20:40:191
285642799331,29cyclictest4497-21PK-Backend22:16:351
285642799331,29cyclictest0-21swapper/100:36:221
285642799331,10cyclictest4497-21PK-Backend20:35:011
285642299337,6cyclictest29432252ls23:15:020
2856422993323,6cyclictest13-21ksoftirqd/023:40:140
2856422993313,6cyclictest141rcu_preempt00:00:170
2856422993313,6cyclictest141rcu_preempt00:00:170
285642799329,7cyclictest4497-21PK-Backend00:03:421
285642799329,7cyclictest4497-21PK-Backend00:03:421
285642799324,11cyclictest26-21ksoftirqd/120:35:211
2856427993225,5cyclictest0-21swapper/100:09:291
2856427993223,5cyclictest26-21ksoftirqd/121:15:091
2856427993221,6cyclictest26-21ksoftirqd/121:05:151
2856427993217,11cyclictest2933641-21swap22:45:281
2856427993217,11cyclictest2911649-21basename21:45:091
285642799321,5cyclictest141rcu_preempt00:20:291
285642799321,4cyclictest4497-21PK-Backend22:06:531
2856427993013,13cyclictest26-21ksoftirqd/122:30:181
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional