You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-22 - 08:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Dec 22, 2025 00:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11442002870,5sleep30-21swapper/319:45:263
197399820,64rtkit-daemon0-21swapper/119:05:131
12074832810,9sleep04497-21PK-Backend22:45:240
12460792800,7sleep34497-21PK-Backend00:35:303
11975762780,7sleep14497-21PK-Backend22:17:001
113026327741,13sleep20-21swapper/219:07:072
112905526848,13sleep00-21swapper/019:05:160
113036126444,14sleep30-21swapper/319:08:213
197399590,9rtkit-daemon1972-21rtkit-daemon21:40:352
197399590,8rtkit-daemon1972-21rtkit-daemon21:20:503
197399590,8rtkit-daemon1972-21rtkit-daemon21:20:503
197399580,9rtkit-daemon1972-21rtkit-daemon23:13:501
197399570,9rtkit-daemon1972-21rtkit-daemon22:32:151
12108102560,8sleep20-21swapper/222:55:242
197399540,8rtkit-daemon1972-21rtkit-daemon21:26:292
197399530,8rtkit-daemon1972-21rtkit-daemon21:32:132
1130643995315,7cyclictest34-21ksoftirqd/223:49:542
12309722510,7sleep234-21ksoftirqd/223:54:402
113064399501,10cyclictest4497-21PK-Backend20:16:012
1130643994922,22cyclictest1239695-21munin-run00:20:002
197399480,41rtkit-daemon2753-21gmain21:41:493
11939972470,6chrt0-21swapper/022:05:320
11950052460,7sleep24497-21PK-Backend22:10:172
11950052460,7sleep24497-21PK-Backend22:10:172
11665402460,5chrt0-21swapper/020:50:020
1130640994635,5cyclictest26-21ksoftirqd/100:05:011
11704152450,5chrt0-21swapper/321:00:153
197399440,37rtkit-daemon3037-21gmain21:15:362
113064399441,12cyclictest34-21ksoftirqd/200:25:022
1130640994421,19cyclictest1159425-21unin-run20:30:011
197399430,24rtkit-daemon0-21swapper/019:24:370
197399430,24rtkit-daemon0-21swapper/019:24:370
1130640994331,9cyclictest26-21ksoftirqd/120:16:011
1130640994324,14cyclictest26-21ksoftirqd/121:55:001
1130627994329,10cyclictest1201279-21cron22:30:000
11686122420,8sleep30-21swapper/320:55:163
1130640994223,6cyclictest26-21ksoftirqd/123:20:001
1130640994223,14cyclictest26-21ksoftirqd/119:25:001
1130640994223,14cyclictest26-21ksoftirqd/119:25:001
1130640994223,14cyclictest26-21ksoftirqd/100:40:021
1130640994222,15cyclictest26-21ksoftirqd/100:15:151
12200862410,6chrt0-21swapper/323:20:283
12200862410,6chrt0-21swapper/323:20:283
1130647994111,25cyclictest1188589-21smartctl21:50:293
113064399419,8cyclictest34-21ksoftirqd/219:42:512
113064399418,16cyclictest34-21ksoftirqd/200:00:022
113064399418,16cyclictest34-21ksoftirqd/200:00:012
113064399418,16cyclictest1137-21runrttasks20:48:052
1130643994130,8cyclictest34-21ksoftirqd/200:29:412
1130643994121,8cyclictest34-21ksoftirqd/200:33:322
1130643994118,19cyclictest1131440-21sed19:10:222
113064399411,21cyclictest34-21ksoftirqd/222:25:002
1130640994124,13cyclictest26-21ksoftirqd/119:20:001
1130640994123,6cyclictest26-21ksoftirqd/121:55:091
1130640994123,14cyclictest26-21ksoftirqd/123:05:001
1130640994123,14cyclictest26-21ksoftirqd/121:15:011
1130640994122,6cyclictest26-21ksoftirqd/121:35:011
1130640994122,6cyclictest26-21ksoftirqd/120:50:001
113062799412,35cyclictest1157652-21cron20:25:000
197399400,28rtkit-daemon4497-21PK-Backend21:55:293
11755262400,5sleep0161rcuc/021:15:090
1130647994013,23cyclictest1153151-21kernelversion20:10:253
1130643994020,17cyclictest1215249-21apt-config23:10:012
1130640994023,5cyclictest26-21ksoftirqd/123:35:301
1130640994023,5cyclictest26-21ksoftirqd/121:50:001
1130640994023,5cyclictest26-21ksoftirqd/120:00:091
1130640994023,13cyclictest26-21ksoftirqd/123:25:231
1130640994023,13cyclictest26-21ksoftirqd/122:50:141
1130640994022,5cyclictest26-21ksoftirqd/122:50:001
1130640994022,5cyclictest26-21ksoftirqd/120:55:001
1130640994022,14cyclictest26-21ksoftirqd/121:25:231
1130640994022,14cyclictest26-21ksoftirqd/121:15:221
1130640994020,7cyclictest26-21ksoftirqd/122:10:311
1130640994020,7cyclictest26-21ksoftirqd/122:10:311
1130640994020,16cyclictest26-21ksoftirqd/120:10:181
12106232390,14chrt42-21ksoftirqd/322:55:213
11526172390,6chrt0-21swapper/220:10:042
1130647993920,14cyclictest2143-21upowerd20:16:003
1130647993914,8cyclictest12236072cat23:30:293
113064399395,18cyclictest34-21ksoftirqd/222:45:012
113064399394,31cyclictest1204590-21sshd22:38:082
1130643993919,8cyclictest34-21ksoftirqd/221:05:142
1130643993917,8cyclictest34-21ksoftirqd/219:45:392
1130643993916,9cyclictest34-21ksoftirqd/222:05:012
113064399391,32cyclictest438-20systemd-journal23:34:592
113064399391,21cyclictest1148857-21taskset20:00:062
1130640993923,12cyclictest26-21ksoftirqd/122:30:011
1130640993922,5cyclictest26-21ksoftirqd/121:05:001
1130640993922,5cyclictest26-21ksoftirqd/120:10:001
1130640993922,13cyclictest26-21ksoftirqd/123:41:341
1130640993921,14cyclictest26-21ksoftirqd/120:55:231
1130640993921,13cyclictest26-21ksoftirqd/123:35:001
1130640993921,13cyclictest26-21ksoftirqd/100:10:191
1130627993921,14cyclictest1244007-21memory00:30:230
11314362380,6chrt0-21swapper/019:10:220
1130647993822,11cyclictest438-20systemd-journal23:14:093
1130647993810,19cyclictest42-21ksoftirqd/300:16:133
113064399387,15cyclictest34-21ksoftirqd/221:10:432
113064399384,16cyclictest34-21ksoftirqd/223:10:202
113064399382,18cyclictest34-21ksoftirqd/222:10:022
1130643993820,13cyclictest1151026-21df20:05:142
1130643993817,8cyclictest34-21ksoftirqd/221:35:462
1130643993817,8cyclictest34-21ksoftirqd/200:39:562
113064399381,20cyclictest34-21ksoftirqd/221:25:012
113064399381,20cyclictest34-21ksoftirqd/221:25:012
113064399381,20cyclictest34-21ksoftirqd/220:45:012
113064399381,20cyclictest34-21ksoftirqd/220:30:022
113064399381,20cyclictest34-21ksoftirqd/219:30:312
113064099385,11cyclictest0-21swapper/122:55:251
1130640993828,6cyclictest26-21ksoftirqd/120:40:021
1130640993821,5cyclictest26-21ksoftirqd/121:05:151
1130640993821,5cyclictest26-21ksoftirqd/119:50:011
1130640993821,5cyclictest26-21ksoftirqd/100:10:001
1130640993821,5cyclictest26-21ksoftirqd/100:10:001
1130640993821,14cyclictest26-21ksoftirqd/122:35:071
1130640993821,13cyclictest26-21ksoftirqd/123:45:271
1130640993821,13cyclictest26-21ksoftirqd/119:40:251
1130640993820,6cyclictest26-21ksoftirqd/121:45:021
1130640993819,6cyclictest26-21ksoftirqd/122:00:171
113064099381,15cyclictest1231045-21unin-run23:55:011
1130627993828,6cyclictest3693027-21snapd22:43:380
1130627993821,12cyclictest1218816-21sshd23:19:080
1130627993816,6cyclictest141rcu_preempt19:55:000
113062799381,22cyclictest4497-21PK-Backend21:15:010
113062799381,13cyclictest1144916-21perf19:50:010
11468532370,7sleep14497-21PK-Backend19:55:021
113064799379,8cyclictest1241047-21vmstat00:20:333
113064799375,5cyclictest141rcu_preempt20:54:023
1130647993726,7cyclictest1182275-21chrt21:35:003
1130647993719,13cyclictest42-21ksoftirqd/319:16:163
1130643993720,12cyclictest1209042-21cat22:50:202
113064399371,19cyclictest34-21ksoftirqd/220:35:022
113064399371,10cyclictest4497-21PK-Backend19:25:232
1130640993721,5cyclictest26-21ksoftirqd/119:55:281
1130640993721,13cyclictest26-21ksoftirqd/119:25:141
1130640993720,5cyclictest26-21ksoftirqd/120:22:291
1130640993719,6cyclictest26-21ksoftirqd/119:15:001
1130640993719,5cyclictest26-21ksoftirqd/121:20:211
1130640993719,5cyclictest26-21ksoftirqd/121:20:211
1130640993719,5cyclictest26-21ksoftirqd/100:20:021
1130640993719,14cyclictest26-21ksoftirqd/122:41:511
1130640993718,14cyclictest26-21ksoftirqd/122:05:231
1130627993726,7cyclictest1225814-21cron23:40:000
1130627993723,10cyclictest13-21ksoftirqd/019:30:120
1130627993721,12cyclictest1158639-21ntp_kernel_err20:25:230
1130627993720,12cyclictest923-21dbus-daemon20:11:070
11699452360,17sleep234-21ksoftirqd/221:00:022
1130647993621,11cyclictest1229311-21systemd-run23:49:593
1130647993620,12cyclictest1229868-21expr23:50:163
113064799361,15cyclictest4497-21PK-Backend23:02:443
113064399369,14cyclictest34-21ksoftirqd/221:59:382
113064399364,18cyclictest34-21ksoftirqd/200:05:232
113064399364,18cyclictest34-21ksoftirqd/200:05:232
1130643993621,11cyclictest1157650-21cron20:25:002
1130643993612,9cyclictest34-21ksoftirqd/221:50:192
1130640993621,11cyclictest26-21ksoftirqd/100:25:331
1130640993620,5cyclictest26-21ksoftirqd/123:20:211
1130640993620,5cyclictest26-21ksoftirqd/123:20:211
1130640993620,5cyclictest26-21ksoftirqd/122:20:281
1130640993620,5cyclictest26-21ksoftirqd/100:30:201
1130640993619,6cyclictest26-21ksoftirqd/120:30:101
1130640993619,13cyclictest26-21ksoftirqd/119:39:401
1130640993618,14cyclictest26-21ksoftirqd/123:55:211
1130640993618,14cyclictest26-21ksoftirqd/123:55:201
113064099361,14cyclictest1163106-21find20:40:121
1130627993621,11cyclictest1173293-21turbostat.cron21:10:000
1130627993620,12cyclictest1202040-21awk22:30:200
1130627993618,13cyclictest438-20systemd-journal20:00:080
113062799361,7cyclictest1213411-21turbostat23:04:590
197399350,24rtkit-daemon0-21swapper/019:37:290
197399350,23rtkit-daemon0-21swapper/020:07:490
12316022350,5sleep0161rcuc/023:55:140
12316022350,5sleep0161rcuc/023:55:140
113064799352,29cyclictest1200573-21ntp_states22:25:253
1130647993521,10cyclictest1244867-21cron00:35:003
1130647993520,11cyclictest1207468-21ntp_kernel_pll_22:45:243
1130647993520,11cyclictest1147369-21cat19:55:213
1130647993519,12cyclictest1-21systemd23:40:013
1130647993519,12cyclictest1142911-21seq19:41:533
113064799351,14cyclictest11741262latency21:10:223
113064399354,16cyclictest34-21ksoftirqd/200:02:522
1130643993525,7cyclictest1221574-21needreboot23:25:222
1130643993525,7cyclictest1219325-21cpuspeed_turbos23:20:132
1130643993525,7cyclictest1219325-21cpuspeed_turbos23:20:132
1130643993521,10cyclictest1168086-21cron20:55:002
1130643993514,8cyclictest34-21ksoftirqd/223:39:082
1130643993514,8cyclictest34-21ksoftirqd/222:48:092
1130643993514,8cyclictest34-21ksoftirqd/221:00:402
1130643993510,7cyclictest34-21ksoftirqd/219:53:092
1130640993518,13cyclictest26-21ksoftirqd/119:33:251
1130627993520,11cyclictest1226991-21expr23:40:300
1130627993519,11cyclictest1199765-21sed22:25:020
1130627993518,13cyclictest1228228-21which23:45:210
197399340,27rtkit-daemon0-21swapper/321:45:553
197399340,27rtkit-daemon0-21swapper/221:48:282
197399340,23rtkit-daemon0-21swapper/019:19:040
197399340,22rtkit-daemon0-21swapper/020:54:050
11622852340,17sleep342-21ksoftirqd/320:35:323
113064799346,11cyclictest42-21ksoftirqd/322:12:593
113064799346,11cyclictest42-21ksoftirqd/322:12:583
1130647993426,5cyclictest0-21swapper/321:09:103
1130647993421,8cyclictest1199126-21sort22:20:313
1130647993419,11cyclictest1241922-21cat00:25:163
1130647993419,11cyclictest1164235-21taskset20:40:403
1130647993419,11cyclictest1135914-21seq19:20:563
1130647993419,11cyclictest1135914-21seq19:20:563
1130647993418,12cyclictest1196380-21grep22:15:163
1130647993417,12cyclictest956-21systemd-logind22:53:413
1130647993415,14cyclictest1191545-21df_inode22:00:163
113064799341,30cyclictest4497-21PK-Backend20:02:153
113064799341,29cyclictest4497-21PK-Backend21:15:453
113064399346,17cyclictest34-21ksoftirqd/222:25:042
113064399346,15cyclictest0-21swapper/220:38:242
113064399342,18cyclictest34-21ksoftirqd/200:10:242
113064399341,9cyclictest4497-21PK-Backend19:20:162
113064399341,9cyclictest4497-21PK-Backend19:20:162
1130643993419,11cyclictest1147225-21expr19:55:162
1130643993411,7cyclictest34-21ksoftirqd/223:40:132
113064099344,18cyclictest1215129-21apt-get23:10:001
113062799346,10cyclictest13-21ksoftirqd/023:09:080
113062799345,12cyclictest13-21ksoftirqd/022:00:490
1130627993426,5cyclictest0-21swapper/020:55:060
1130627993424,7cyclictest1137-21runrttasks22:55:140
1130627993424,6cyclictest1204763-21turbostat22:39:590
1130627993422,8cyclictest1164589-21sh20:45:000
1130627993419,11cyclictest1189323-21sed21:55:020
1130627993419,11cyclictest1137963-21cron19:30:000
1130627993418,12cyclictest1246484-21seq00:38:360
1130627993418,11cyclictest438-20systemd-journal23:53:080
1130627993417,12cyclictest1155568-21systemd-udevd20:15:590
1130627993416,14cyclictest1170464-21df21:00:160
113064799332,9cyclictest1180554-21munin-run21:30:013
113064799332,9cyclictest1131417-21memory19:10:213
1130647993323,7cyclictest1137-21runrttasks23:05:273
1130647993322,6cyclictest42-21ksoftirqd/322:38:023
1130647993319,10cyclictest1221584-21/usr/sbin/munin23:25:233
1130647993318,11cyclictest1158853-21grep20:25:283
1130647993317,12cyclictest1233780-21ntp_states00:00:273
1130647993316,12cyclictest438-20systemd-journal22:34:083
1130647993313,6cyclictest141rcu_preempt23:40:153
113064799331,29cyclictest4497-21PK-Backend23:57:253
113064799331,29cyclictest4497-21PK-Backend23:57:253
1130643993314,6cyclictest141rcu_preempt23:02:472
1130643993313,5cyclictest141rcu_preempt19:35:582
113064399331,29cyclictest4497-21PK-Backend22:34:242
1130640993316,5cyclictest26-21ksoftirqd/121:40:001
1130627993323,6cyclictest13-21ksoftirqd/000:01:460
1130627993323,6cyclictest1196097-21latency_hist22:15:000
1130627993323,6cyclictest1196097-21latency_hist22:14:590
1130627993319,10cyclictest1219578-21smartctl23:20:180
1130627993319,10cyclictest1219578-21smartctl23:20:170
1130627993318,11cyclictest13-21ksoftirqd/021:30:180
1130627993317,12cyclictest1220880-21cat23:25:110
1130627993316,13cyclictest1182760-21cat21:35:170
1130647993225,4cyclictest0-21swapper/321:36:423
1130647993221,7cyclictest1156339-21tr20:20:143
1130647993221,6cyclictest42-21ksoftirqd/319:50:183
1130647993220,8cyclictest1140713-21ntp_offset19:35:243
1130647993217,11cyclictest1236817-21expr00:10:163
1130647993217,11cyclictest1165479-21grep20:45:233
113064799321,28cyclictest4497-21PK-Backend22:42:223
113064799321,27cyclictest1150640-21/usr/sbin/munin20:05:103
113064399325,14cyclictest0-21swapper/222:15:152
113064399321,9cyclictest4497-21PK-Backend23:18:512
113064399321,16cyclictest1134263-21cron19:17:002
1130627993222,7cyclictest1243142-21cron00:30:010
1130627993221,7cyclictest1185710-21sh21:45:010
1130627993218,11cyclictest1222824-21tr23:30:140
1130627993218,10cyclictest1197017-21seq22:15:260
1130627993218,10cyclictest1187439-21grep21:50:000
1130627993217,11cyclictest1189992-21cat21:55:200
1130627993216,12cyclictest1234756-21awk00:05:120
1130627993216,12cyclictest1234756-21awk00:05:120
1130627993216,12cyclictest1162427-21sed20:35:330
113062799321,4cyclictest0-21swapper/021:22:180
113062799321,4cyclictest0-21swapper/021:22:180
11930002310,5sleep3411rcuc/322:05:153
113064799312,7cyclictest1137267-21ntp_states19:25:273
1130647993124,4cyclictest4497-21PK-Backend00:05:523
1130647993124,4cyclictest4497-21PK-Backend00:05:513
113064799311,28cyclictest0-21swapper/323:15:313
1130627993121,7cyclictest1148361-21dump-pmu-power20:00:000
1130627993119,8cyclictest1142634-21sort19:40:280
1130627993116,11cyclictest1209075-21grep22:50:210
1130627993116,10cyclictest1160790-21seq20:30:540
1130627993114,7cyclictest1137-21runrttasks21:26:240
1130627993114,12cyclictest1238713-21kernelversion00:15:230
1130627993113,15cyclictest1241103-21sshd00:21:080
113062799311,28cyclictest0-21swapper/000:12:250
1130647993020,5cyclictest42-21ksoftirqd/320:32:143
1130627993017,9cyclictest1215917-21expr23:10:150
113064799291,6cyclictest11394002sleep319:31:133
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional