You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-31 - 08:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: ondemand
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot4s.osadl.org (updated Tue Mar 31, 2026 00:44:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
97979922010,3sleep375-21kworker/3:1-events19:21:343
10898042810,2chrt281rcuc/121:11:351
11372772750,2sleep30-21swapper/322:01:303
965390996452,9cyclictest280-21systemd-journal19:15:011
965390996452,9cyclictest280-21systemd-journal19:15:011
965386996353,7cyclictest1116512-21latency_hist21:40:000
965390996155,4cyclictest1210509-21apt-get23:11:421
965390996150,9cyclictest1245901-21latency_hist23:50:001
965386995847,7cyclictest1017005-21latency_hist20:00:010
965386995645,8cyclictest1026861-21latency_hist20:10:010
12507712550,2chrt161rcu_preempt23:51:442
12507712550,2chrt161rcu_preempt23:51:442
965393995332,17cyclictest1061492-21cp20:45:002
965390995347,4cyclictest1071981-21kworker/u8:1+flush-8:020:56:291
965386995344,6cyclictest1251069-21latency_hist23:55:000
965386995344,6cyclictest1251069-21latency_hist23:55:000
965386995341,8cyclictest280-21systemd-journal23:45:000
965390995140,9cyclictest1200414-21ntpq23:01:391
965393995028,19cyclictest1046579-21/usr/sbin/munin20:30:002
965390995045,4cyclictest933311-21kworker/u8:0+flush-8:019:41:421
965390995045,4cyclictest1007332-21kworker/u8:1+flush-8:020:26:281
965390995044,4cyclictest4065205-21kworker/u8:4+events_unbound21:26:421
965390995039,8cyclictest280-21systemd-journal20:40:001
965386995041,6cyclictest1007105-21latency_hist19:50:010
965390994943,5cyclictest1007332-21kworker/u8:1+flush-8:019:51:331
965386994942,5cyclictest933311-21kworker/u8:0+events_unbound20:26:370
965390994843,4cyclictest4065205-21kworker/u8:4+flush-8:020:41:291
965390994838,7cyclictest1031801-21latency_hist20:15:011
965390994741,4cyclictest280-21systemd-journal23:27:401
965390994741,4cyclictest1200962-21kworker/u8:0+flush-8:000:21:281
965390994739,5cyclictest1266035-21latency_hist00:10:001
965390994738,6cyclictest280-21systemd-journal00:40:011
965390994640,4cyclictest933311-21kworker/u8:0+events_unbound21:01:331
965390994639,5cyclictest1245898-21apt-get23:50:011
965390994639,5cyclictest1245898-21apt-get23:50:011
965386994642,3cyclictest1210900-21kworker/u8:1+events_unbound23:26:300
965390994534,8cyclictest1180591-21ntpq22:41:381
965386994538,5cyclictest1200962-21kworker/u8:0+events_unbound23:36:290
965386994536,6cyclictest280-21systemd-journal19:16:320
965386994536,6cyclictest280-21systemd-journal19:16:320
9646152450,6sleep3451irq_work/319:06:413
965390994336,5cyclictest280-21systemd-journal00:02:391
965390994336,5cyclictest280-21systemd-journal00:02:391
965386994329,11cyclictest280-21systemd-journal21:59:590
96538699430,40cyclictest1-21systemd00:00:000
965390994237,4cyclictest4065205-21kworker/u8:4+flush-8:019:16:281
965390994237,4cyclictest4065205-21kworker/u8:4+flush-8:019:16:281
965390994236,5cyclictest1166310-21kworker/u8:1+events_unbound22:51:421
965386994234,6cyclictest280-21systemd-journal21:01:340
965386994114,16cyclictest589-21runrttasks20:46:250
965390994036,3cyclictest1200962-21kworker/u8:0+flush-8:000:16:281
965390994035,4cyclictest1210900-21kworker/u8:1+flush-8:023:16:451
965386994037,2cyclictest280-21systemd-journal19:51:320
965393993931,6cyclictest1131663-21gpgv21:55:012
965390993935,3cyclictest1240569-21kworker/u8:3+events_unbound23:56:411
965390993934,4cyclictest1141804-21kworker/u8:3+flush-8:022:31:291
965390993931,5cyclictest280-21systemd-journal19:36:321
965386993930,6cyclictest280-21systemd-journal23:20:400
965393993827,8cyclictest1076712-21awk21:00:002
965390993831,5cyclictest1071981-21kworker/u8:1+flush-8:021:16:371
965390993829,6cyclictest1231023-21latency_hist23:35:011
965386993814,13cyclictest589-21runrttasks23:11:240
12226162380,2chrt75-21kworker/3:1+events23:26:293
965393993719,16cyclictest1176942-21aten2_r5power_p22:41:272
965390993731,4cyclictest1022280-21apt-get20:06:231
965390993730,5cyclictest1206199-21latency_hist23:10:011
96538699376,19cyclictest1186984-21df22:51:260
965386993727,7cyclictest280-21systemd-journal21:20:000
965393993626,8cyclictest1280875-21awk00:25:002
965390993632,3cyclictest1166310-21kworker/u8:1+events_unbound22:46:401
965390993632,3cyclictest1007332-21kworker/u8:1+flush-8:020:31:431
965390993629,5cyclictest1101663-21kworker/u8:2+flush-8:021:56:291
96538699367,8cyclictest1099477-21kthreadcore21:21:350
96538699367,18cyclictest992035-21munin-run19:35:000
965386993629,5cyclictest996103-21ntpq19:36:370
965386993614,11cyclictest1105786-21ntpq21:26:380
965386993613,14cyclictest1256692-21/usr/sbin/munin00:01:300
965386993613,14cyclictest1256692-21/usr/sbin/munin00:01:300
965386993613,11cyclictest1294824-21ntpq00:36:360
965386993612,14cyclictest589-21runrttasks23:05:100
965386993612,13cyclictest1243463-21ps23:46:330
965386993611,15cyclictest986342-21ntpq19:26:390
965386993611,13cyclictest1182039-21diskstats22:46:280
965386993610,15cyclictest1275160-21ntpq00:16:390
965386993610,13cyclictest1088109-21ps21:11:340
965393993523,10cyclictest1132100-21aten2_r5power_v21:56:262
965390993530,3cyclictest1071981-21kworker/u8:1+flush-8:022:03:401
965390993529,4cyclictest1240569-21kworker/u8:3+events_unbound00:31:261
965390993529,4cyclictest1141804-21kworker/u8:3+events_unbound22:36:401
965390993529,4cyclictest1021930-21latency_hist20:05:011
965390993528,6cyclictest1071981-21kworker/u8:1+flush-8:021:51:391
96538699359,15cyclictest1030985-21ntpq20:11:380
96538699359,14cyclictest1071052-21proc_pri20:51:400
96538699359,12cyclictest1266392-21apt-get00:11:280
96538699358,9cyclictest1128953-21ps21:51:330
96538699357,8cyclictest1156299-21tail22:16:460
96538699357,17cyclictest1122073-21forks21:46:280
965386993513,11cyclictest1212230-21ps23:16:330
965386993513,10cyclictest1180602-21ntpq22:41:380
965386993512,10cyclictest1191959-21/usr/sbin/munin22:56:280
965386993511,14cyclictest1150841-21ntpq22:11:370
965386993511,12cyclictest1072123-21/usr/sbin/munin20:56:260
965386993510,7cyclictest0-21swapper/020:01:390
965386993510,14cyclictest1050519-21memory20:31:360
965393993417,15cyclictest1295674-21idleruntime-cro00:40:002
965390993429,4cyclictest4065205-21kworker/u8:4+events_unbound19:21:331
965390993427,5cyclictest1221117-21latency_hist23:25:011
965390993427,5cyclictest1061956-21/usr/sbin/munin20:46:361
96538699349,15cyclictest1170782-21ntp_states22:31:400
96538699349,14cyclictest1200938-21users23:01:450
96538699348,15cyclictest1262294-21irqstats00:06:310
96538699347,16cyclictest1226644-21df23:31:280
96538699347,16cyclictest1110667-21netstat21:31:350
96538699347,14cyclictest1280099-21ntp_states00:21:390
96538699347,12cyclictest1032191-21aten2_r5power_e20:16:260
965386993427,5cyclictest280-21systemd-journal20:23:160
965386993417,10cyclictest583-21snmpd22:01:260
965386993412,10cyclictest1286540-21ls00:31:290
965386993411,12cyclictest1142361-21/usr/sbin/munin22:06:290
965386993411,11cyclictest1160787-21ntpq22:21:370
965393993316,15cyclictest1246159-21gpgv23:50:002
965390993328,4cyclictest1141804-21kworker/u8:3+flush-8:022:11:291
965390993327,5cyclictest1101663-21kworker/u8:2+events_unbound22:16:391
96538699338,15cyclictest981196-21memory19:21:360
96538699337,16cyclictest965864-21df19:11:270
96538699337,16cyclictest965864-21df19:11:270
96538699337,15cyclictest1172128-21diskstats22:36:280
96538699337,15cyclictest1052097-21df20:36:270
96538699337,14cyclictest997841-21/usr/sbin/munin19:41:300
96538699336,12cyclictest1284424-21cut00:26:350
965386993311,12cyclictest1120732-21ntpq21:41:410
965386993311,12cyclictest1120732-21ntpq21:41:410
965386993311,10cyclictest1165763-21ntpq22:26:370
965386993310,8cyclictest589-21runrttasks21:06:240
965386993310,11cyclictest1060569-21ntpq20:41:380
96539399321,11cyclictest986307-21ntpq19:26:372
965390993228,3cyclictest1200962-21kworker/u8:0+flush-8:023:36:421
965390993228,3cyclictest1071981-21kworker/u8:1+flush-8:021:36:281
965390993227,4cyclictest1200962-21kworker/u8:0+flush-8:023:41:431
965390993227,3cyclictest933311-21kworker/u8:0+events_unbound19:31:341
965393993124,5cyclictest1066418-21journal-offline20:48:202
965393993112,3cyclictest0-21swapper/219:46:302
965390993125,5cyclictest1071981-21kworker/u8:1+flush-8:022:06:371
965390993125,4cyclictest1071981-21kworker/u8:1+events_unbound21:46:291
965390993120,9cyclictest280-21systemd-journal19:25:321
9605842314,8sleep10-21swapper/119:05:471
96539399303,7cyclictest1097055-21apt-get21:21:262
965393993014,2cyclictest0-21swapper/222:11:362
965393993014,2cyclictest0-21swapper/220:00:002
965393993012,3cyclictest0-21swapper/223:41:362
96539399301,11cyclictest1046926-21apt-get20:31:272
965390993026,3cyclictest1101663-21kworker/u8:2+flush-8:022:21:291
965390993026,3cyclictest1061427-21kworker/u8:2+events_unbound20:51:431
965390993025,3cyclictest1256113-21kworker/u8:1+flush-8:000:11:311
965390993025,3cyclictest1099213-21kthreadcore21:21:341
96539399293,7cyclictest1261411-21apt-get00:06:242
96539399292,7cyclictest1160770-21ntpq22:21:372
965393992914,2cyclictest1201787-21cpuspeed_turbos23:06:262
96539399291,11cyclictest1211508-21apt-get23:16:272
965390992924,4cyclictest1071981-21kworker/u8:1+flush-8:021:41:421
965390992924,4cyclictest1071981-21kworker/u8:1+flush-8:021:41:421
96539399284,6cyclictest1067146-21apt-get20:51:232
965393992823,3cyclictest1111241-21unixbench_singl21:31:442
965393992820,5cyclictest280-21systemd-journal20:20:002
965393992818,7cyclictest970285-21awk19:15:012
965393992818,7cyclictest970285-21awk19:15:012
965393992814,4cyclictest1096388-21unixbench_singl21:16:452
965390992823,4cyclictest933311-21kworker/u8:0+events_unbound21:06:361
965390992823,4cyclictest1107181-21cpuspeed_turbos21:31:261
965390992823,3cyclictest1283371-21tail00:26:341
9646972284,15sleep0964918-21/usr/sbin/munin19:06:430
96539399277,6cyclictest1261210-21ls00:05:002
96539399277,6cyclictest1261210-21ls00:04:592
965393992723,2cyclictest1152135-21aten2_r5power_p22:16:282
965393992716,9cyclictest0-21swapper/220:11:442
965393992714,2cyclictest413-21systemd-logind19:25:002
965393992710,15cyclictest1226049-21cron23:30:002
96539399270,3cyclictest12830772cut00:26:342
965390992722,4cyclictest1141804-21kworker/u8:3+flush-8:022:56:441
965390992722,3cyclictest1101663-21kworker/u8:2+flush-8:022:26:361
96539399266,9cyclictest977201-21sendmail19:20:002
96539399266,9cyclictest977201-21sendmail19:20:002
965393992622,2cyclictest1102214-21cpuspeed_turbos21:26:272
965393992613,2cyclictest1167272-21fschecks_time22:31:292
965390992621,4cyclictest1007332-21kworker/u8:1+flush-8:020:19:391
965390992619,5cyclictest280-21systemd-journal19:48:081
965393992519,4cyclictest1220795-21unixbench_singl23:21:442
965390992521,3cyclictest1007332-21kworker/u8:1+flush-8:020:20:311
965393992420,2cyclictest0-21swapper/222:10:002
965393992415,7cyclictest1290754-21cat00:35:002
96539399241,10cyclictest1021233-21ps20:01:422
96539399240,3cyclictest11372812sort22:01:302
965393992319,2cyclictest1271359-21aten2_r5power_e00:16:262
965393992319,2cyclictest1082598-21fschecks_time21:06:282
965393992315,6cyclictest1001019-21tail19:41:352
965393992314,7cyclictest1166567-21awk22:30:002
965390992317,3cyclictest0-21swapper/119:58:101
965393992219,2cyclictest1187036-21fschecks_count22:51:282
965393992219,2cyclictest1007489-21aten2_r5power_c19:51:262
965393992218,2cyclictest1171965-21aten2_r5power_p22:36:262
965393992016,2cyclictest0-21swapper/222:56:362
965393992015,3cyclictest1201204-21cron23:05:002
9605632204,9sleep20-21swapper/219:05:322
965393991914,3cyclictest0-21swapper/219:40:002
12523062190,2sleep3451irq_work/323:56:313
96539399189,7cyclictest1055250-21tr20:36:342
965393991814,2cyclictest0-21swapper/223:36:362
965393991810,6cyclictest1022447-21sort20:06:262
96539399178,7cyclictest1117701-21sed21:41:312
96539399178,7cyclictest1117701-21sed21:41:312
965393991713,2cyclictest0-21swapper/223:11:262
965393991713,2cyclictest0-21swapper/221:40:002
12680592178,5sleep30-21swapper/300:11:373
96539399162,13cyclictest0-21swapper/221:45:112
96539399162,13cyclictest0-21swapper/221:04:162
965393991613,2cyclictest1087538-21fschecks_time21:11:302
965393991613,2cyclictest0-21swapper/223:31:502
965393991612,2cyclictest0-21swapper/223:56:222
965393991612,2cyclictest0-21swapper/220:25:012
965393991611,3cyclictest0-21swapper/219:31:332
96539399161,11cyclictest1185558-21ntpq22:46:372
11319902162,9sleep30-21swapper/321:56:253
96539399156,7cyclictest1268302-21tr00:11:362
10671472152,8sleep30-21swapper/320:51:243
10392312145,5sleep30-21swapper/320:21:343
9928762132,7sleep30-21swapper/319:36:323
9663432130,2sleep3471rcuc/319:11:313
9663432130,2sleep3471rcuc/319:11:313
12586932132,7sleep30-21swapper/300:01:373
12586932132,7sleep30-21swapper/300:01:373
12102132132,7sleep30-21swapper/323:11:373
11953252132,7sleep30-21swapper/322:56:363
11175232132,7sleep30-21swapper/321:41:323
11175232132,7sleep30-21swapper/321:41:323
11083642132,7sleep30-21swapper/321:31:333
10908552132,7sleep30-21swapper/321:11:383
10260232135,5sleep30-21swapper/320:06:373
9874502122,6sleep30-21swapper/319:31:263
9868162122,6sleep30-21swapper/319:26:443
9741042122,6sleep30-21swapper/319:16:333
9741042122,6sleep30-21swapper/319:16:333
12866202122,6sleep30-21swapper/300:31:313
12855112122,6sleep30-21swapper/300:26:463
12762772122,6sleep30-21swapper/300:21:243
12659002122,6sleep30-21swapper/300:08:283
12483002122,6sleep30-21swapper/323:51:343
12483002122,6sleep30-21swapper/323:51:343
12324492123,5sleep30-21swapper/323:36:333
12290962122,6sleep30-21swapper/323:31:363
12113762122,5sleep30-21swapper/323:15:023
12018172122,6sleep30-21swapper/323:06:283
11963282122,5sleep30-21swapper/323:00:023
11772942122,6sleep30-21swapper/322:41:323
11718832123,6sleep30-21swapper/322:36:263
11618212122,6sleep30-21swapper/322:25:023
11563532122,6sleep30-21swapper/322:16:483
11256902122,6sleep30-21swapper/321:46:423
10922602122,6sleep30-21swapper/321:16:283
10717662122,5sleep30-21swapper/320:55:013
10618902122,6sleep30-21swapper/320:45:343
10465562122,5sleep30-21swapper/320:30:013
10205102122,6sleep30-21swapper/320:01:363
10162042123,5sleep30-21swapper/319:56:383
12956112112,6sleep30-21swapper/300:39:343
12709722111,6sleep30-21swapper/300:15:023
12412512112,6sleep30-21swapper/323:46:083
12363442112,5sleep30-21swapper/323:41:273
12164412112,6sleep30-21swapper/323:21:083
11866162112,5sleep30-21swapper/322:50:023
11820762112,5sleep30-21swapper/322:46:293
11668952112,5sleep30-21swapper/322:31:243
11572542112,5sleep30-21swapper/322:21:293
11470282112,5sleep30-21swapper/322:10:383
11418372112,6sleep30-21swapper/322:05:363
11272522112,5sleep30-21swapper/321:51:313
11119642112,5sleep30-21swapper/321:36:263
11020222112,5sleep30-21swapper/321:26:273
10967252112,5sleep30-21swapper/321:20:013
10821362112,6sleep30-21swapper/321:05:163
10774542112,5sleep30-21swapper/321:01:273
10567002112,5sleep30-21swapper/320:40:163
10518322112,5sleep30-21swapper/320:36:233
10419862112,5sleep30-21swapper/320:26:243
10320332112,6sleep30-21swapper/320:15:163
10276532112,5sleep30-21swapper/320:11:313
10073972112,6sleep30-21swapper/319:50:563
10023942112,5sleep30-21swapper/319:45:023
10017542112,6sleep30-21swapper/319:42:503
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional