You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-26 - 18:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun Apr 26, 2026 12:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
106281321170,4sleep13056-21PK-Backend08:05:161
10595972980,4sleep13056-21PK-Backend07:55:181
139099870,52rtkit-daemon0-21swapper/107:08:101
104076926444,13sleep00-21swapper/007:05:150
104190226343,14sleep30-21swapper/307:09:243
104171726243,13sleep20-21swapper/207:07:032
11187652410,2sleep21118760-21gpgv10:45:012
10493602390,5sleep13056-21PK-Backend07:25:221
11049942360,6sleep30-21swapper/310:05:163
104218499361,20cyclictest0-21swapper/208:28:312
104218499341,19cyclictest0-21swapper/212:25:262
104218499341,18cyclictest0-21swapper/211:07:232
104218499331,19cyclictest0-21swapper/210:16:102
104218499322,18cyclictest0-21swapper/208:22:542
104218499321,19cyclictest0-21swapper/208:34:042
104218499311,18cyclictest0-21swapper/210:32:462
104218499311,18cyclictest0-21swapper/209:31:182
104218499311,18cyclictest0-21swapper/207:41:512
104218499311,18cyclictest0-21swapper/207:41:512
104219099308,19cyclictest1062137-21cron08:04:593
104219099301,3cyclictest0-21swapper/309:15:123
104219099291,26cyclictest1067593-21/usr/sbin/munin08:20:143
104218499290,3cyclictest1107494-21chrt10:11:252
11457922280,4sleep13056-21PK-Backend12:00:191
1042190992812,14cyclictest1101968-21gzip09:55:203
1042190992810,16cyclictest1134559-21dpkg11:30:023
104219099280,26cyclictest1080644-21cat08:55:223
11456292270,2sleep30-21swapper/312:00:163
104219099277,4cyclictest1111921-21tr10:25:133
104219099277,17cyclictest1122940-21kernelversion10:55:173
1042190992715,9cyclictest1132596-21munin-plugin-st11:25:013
1042184992713,12cyclictest1125734-21cut11:05:012
1042184992711,14cyclictest1139858-21latency_hist11:45:002
1042184992711,14cyclictest1121241-21cat10:50:172
1042184992711,14cyclictest1050634-21smartctl07:30:152
1042184992710,15cyclictest1143814-21cat11:55:132
104219099268,16cyclictest1113724-21seq10:30:133
104219099264,3cyclictest1118457-21idleruntime-cro10:44:593
104219099262,8cyclictest1083216-21apt09:05:113
104219099261,9cyclictest1139162-21awk11:40:203
104219099261,8cyclictest1155434-21cat12:30:003
104218499267,17cyclictest1062620-21cat08:05:132
1042184992611,13cyclictest1130944-21sed11:20:002
1042184992611,13cyclictest1048808-21expr07:25:152
104218099269,15cyclictest1117197-21cstates10:40:121
104218099267,18cyclictest1091881-21latency_hist09:30:011
104218099266,18cyclictest1136840-21cat11:35:141
104218099266,18cyclictest1136840-21cat11:35:141
104218099266,18cyclictest1042551-21apt07:10:121
104218099265,19cyclictest1093998-21sort09:35:121
104219099259,14cyclictest1093256-21sshd09:30:433
104219099259,14cyclictest1066256-21df08:15:153
104219099259,14cyclictest1057047-21sed07:50:013
104219099259,14cyclictest0-21swapper/308:05:183
104219099258,15cyclictest1079696-21date08:55:003
104219099257,16cyclictest1047811-21timerwakeupswit07:20:223
104219099256,4cyclictest1050394-21grep07:30:123
104219099252,4cyclictest0-21swapper/310:35:123
104219099252,19cyclictest1074553-21apt08:40:133
104219099251,8cyclictest1-21systemd09:53:353
104219099251,8cyclictest1074323-21latency_hist08:40:003
104219099251,8cyclictest1057275-21/usr/sbin/munin07:50:133
104219099251,7cyclictest1052759-21proc_pri07:35:183
104219099251,5cyclictest1110029-21/usr/sbin/munin10:20:143
104219099250,8cyclictest1059744-21wc07:55:193
104219099250,7cyclictest1097072-21idleruntime-cro09:45:003
104219099250,23cyclictest1155664-21apt12:30:113
104219099250,23cyclictest1065007-21sensors_temp08:10:203
104219099250,23cyclictest0-21swapper/312:15:243
104218499259,14cyclictest1157156-21systemd-run12:35:002
104218499259,14cyclictest1146972-21find12:05:102
1042184992513,11cyclictest1158921-21apt-config12:40:002
1042184992513,11cyclictest1158921-21apt-config12:40:002
1042184992511,12cyclictest436-20systemd-journal12:05:012
1042184992511,12cyclictest1055609-21cpuspeed_turbos07:45:122
1042184992510,13cyclictest1064468-21expr08:10:122
104218099259,14cyclictest926-21udisksd08:48:031
104218099259,14cyclictest926-21udisksd08:48:021
104218099259,14cyclictest3056-21PK-Backend09:12:041
104218099258,15cyclictest1153902-21find12:25:111
104218099251,4cyclictest1090382-21/usr/sbin/munin09:25:241
104218099250,1cyclictest3056-21PK-Backend07:36:111
104219099246,16cyclictest1107266-21/usr/sbin/munin10:10:213
104219099243,19cyclictest42-21ksoftirqd/307:25:193
104219099243,18cyclictest1134339-21apt-get11:30:043
104219099243,17cyclictest1140054-21apt11:45:133
104219099242,17cyclictest1090408-21apt09:25:113
104219099242,15cyclictest1118451-21apt-get10:45:033
104219099241,8cyclictest1148018-21/usr/sbin/munin12:05:213
104219099241,8cyclictest1124157-21/usr/sbin/munin11:00:153
104219099241,7cyclictest1157942-21cat12:35:173
104219099241,7cyclictest1157942-21cat12:35:163
104219099241,7cyclictest1127803-21cpuspeed_turbos11:10:123
104219099241,7cyclictest1103006-21cpuspeed_turbos10:00:123
104219099241,7cyclictest1097319-21/usr/sbin/munin09:45:223
104219099241,6cyclictest1088457-21date09:20:003
104219099241,5cyclictest1149449-21grep12:10:203
104219099241,5cyclictest1129365-21/usr/sbin/munin11:15:203
104219099241,5cyclictest1126102-21grep11:05:123
104219099241,3cyclictest0-21swapper/307:10:203
1042190992412,10cyclictest978-21snmpd11:51:033
104219099240,8cyclictest1054427-21grep07:40:193
104219099240,8cyclictest1054427-21grep07:40:193
104219099240,7cyclictest1145020-21wc12:00:003
104219099240,6cyclictest1044774-21munin-node07:15:033
104219099240,22cyclictest0-21swapper/310:15:193
104218499249,13cyclictest2156-21gnome-shell08:40:092
104218499249,13cyclictest1084219-21timerwakeupswit09:05:222
104218499249,13cyclictest1066227-21cat08:15:142
104218499246,16cyclictest2317-21gsd-power07:24:562
104218499245,17cyclictest1132928-21cat11:25:122
104218499241,21cyclictest1073681-21python308:35:202
1042184992412,10cyclictest1150500-21cpu12:15:132
1042184992412,10cyclictest1094948-21uptime09:35:232
104218499241,20cyclictest1102853-21apt10:00:112
104218499240,4cyclictest978-21snmpd11:39:002
104218499240,4cyclictest978-21snmpd11:38:592
104218099247,15cyclictest1116716-21latency_hist10:40:001
104218099243,4cyclictest1153812-21latency_hist12:25:011
104218099242,3cyclictest1044802-21python307:15:031
104218099241,7cyclictest1099298-21cstates09:50:131
104218099241,7cyclictest0-21swapper/111:40:181
1042176992412,11cyclictest1120478-21unin-run10:50:000
104219099239,12cyclictest1120691-21/usr/sbin/munin10:50:123
104219099237,4cyclictest1137105-21wc11:35:173
104219099237,4cyclictest1137105-21wc11:35:173
104219099231,8cyclictest1095338-21perf09:40:003
104219099231,11cyclictest1082944-21perf09:05:003
104219099230,7cyclictest1070857-21wc08:30:003
104218499239,12cyclictest0-21swapper/208:00:222
104218499236,15cyclictest1142359-21load11:50:172
104218499236,15cyclictest1128729-21awk11:10:222
104218499235,16cyclictest1135830-21wc11:30:232
104218499235,16cyclictest1096201-21ls09:40:182
1042184992318,4cyclictest0-21swapper/209:03:382
104218499231,18cyclictest1101122-21apt09:55:112
104218499231,18cyclictest1043409-21python307:10:202
1042184992310,11cyclictest1119193-21cstates10:45:132
104218499230,5cyclictest883-21dbus-daemon09:53:342
104218499230,22cyclictest0-21swapper/209:47:472
104218099238,14cyclictest3056-21PK-Backend12:32:271
104218099238,13cyclictest3056-21PK-Backend11:46:381
104218099235,16cyclictest1133032-21cpuspeed_turbos11:25:121
104218099231,8cyclictest1061045-21df08:00:131
104218099231,7cyclictest1065015-21/usr/sbin/munin08:10:211
104218099230,5cyclictest1129111-21cron11:15:011
1042176992321,1cyclictest0-21swapper/010:29:590
1042176992319,2cyclictest3056-21PK-Backend12:25:230
104217699230,3cyclictest1074927-21diskstats08:40:150
11513172220,4sleep00-21swapper/012:15:210
11400372220,6sleep20-21swapper/211:45:112
104219099224,3cyclictest0-21swapper/309:10:113
104219099222,4cyclictest1152675-21kernelversion12:20:183
104219099221,5cyclictest1071207-21grep08:30:133
104219099220,18cyclictest3056-21PK-Backend08:49:413
104219099220,18cyclictest3056-21PK-Backend08:49:403
104218499226,13cyclictest1052233-21cat07:35:122
104218499223,17cyclictest1148994-21expr12:10:132
104218499221,4cyclictest1115924-21strings10:35:182
1042184992211,9cyclictest1095-21runrttasks08:55:352
104218499221,19cyclictest1087054-21diskstats09:15:152
104218499221,18cyclictest1058082-21python307:50:222
104218499221,18cyclictest0-21swapper/209:25:102
104218499221,18cyclictest0-21swapper/207:55:122
104218499221,17cyclictest1084951-21apt09:10:122
104218499221,16cyclictest1122941-21kernelversion10:55:172
1042184992211,10cyclictest1095-21runrttasks11:22:232
1042184992211,10cyclictest1095-21runrttasks10:06:052
104218499220,20cyclictest1111664-21cat10:25:002
104218099228,12cyclictest11588392chrt12:39:481
104218099228,12cyclictest11588392chrt12:39:481
104218099227,14cyclictest3056-21PK-Backend08:28:261
104218099224,16cyclictest0-21swapper/109:50:011
104218099221,6cyclictest1078494-21irqstats08:50:181
1042176992220,1cyclictest0-21swapper/011:20:490
1042176992220,1cyclictest0-21swapper/009:50:250
1042176992220,1cyclictest0-21swapper/009:35:440
1042176992220,1cyclictest0-21swapper/009:03:200
1042176992220,1cyclictest0-21swapper/008:30:280
1042176992220,1cyclictest0-21swapper/007:27:350
104217699220,20cyclictest0-21swapper/009:16:170
11438632210,3sleep13056-21PK-Backend11:55:131
11322142210,3sleep13056-21PK-Backend11:20:251
104218499219,10cyclictest1152236-21cpu12:20:132
104218499212,15cyclictest1044012-21apt-get07:15:012
104218499211,5cyclictest1111922-21cut10:25:122
104218499211,17cyclictest0-21swapper/209:20:122
104218099212,17cyclictest1103792-21chrt10:00:201
104218099211,7cyclictest1076045-21munin-run08:44:591
104218099211,4cyclictest1126036-21grep11:05:131
104218099210,6cyclictest1125764-21grep11:05:001
104218099210,5cyclictest1097192-21grep09:45:001
104218099210,19cyclictest3056-21PK-Backend10:16:041
104218099210,19cyclictest3056-21PK-Backend09:04:251
104218099210,19cyclictest3056-21PK-Backend07:43:011
104218099210,19cyclictest3056-21PK-Backend07:43:011
1042176992119,1cyclictest0-21swapper/009:55:380
1042176992119,1cyclictest0-21swapper/009:22:530
1042176992119,1cyclictest0-21swapper/008:59:210
1042176992118,2cyclictest0-21swapper/007:36:120
1042176992117,2cyclictest1109784-21sh10:20:000
1042176992115,5cyclictest1144975-21perf12:00:000
1042176992113,6cyclictest1091479-21awk09:25:220
104217699210,20cyclictest0-21swapper/012:39:580
104217699210,20cyclictest0-21swapper/012:39:580
104217699210,20cyclictest0-21swapper/011:50:570
104217699210,20cyclictest0-21swapper/010:09:430
104217699210,20cyclictest0-21swapper/007:57:350
104217699210,20cyclictest0-21swapper/007:20:590
1042184992018,1cyclictest0-21swapper/208:54:542
104218099203,5cyclictest26-21ksoftirqd/110:45:241
104218099202,4cyclictest3056-21PK-Backend08:38:331
1042180992018,1cyclictest3056-21PK-Backend09:19:471
1042180992017,2cyclictest3056-21PK-Backend10:50:251
104218099200,7cyclictest1072343-21seq08:32:381
104218099200,6cyclictest1143002-21chrt11:52:041
104218099200,4cyclictest3056-21PK-Backend10:25:131
104218099200,4cyclictest10689382sleep108:23:251
104218099200,1cyclictest3056-21PK-Backend12:05:421
104218099200,1cyclictest3056-21PK-Backend09:09:121
104218099200,1cyclictest3056-21PK-Backend07:50:121
104218099200,19cyclictest3056-21PK-Backend10:58:561
104218099200,19cyclictest3056-21PK-Backend09:56:301
104218099200,18cyclictest3056-21PK-Backend08:17:411
104218099200,18cyclictest3056-21PK-Backend07:47:261
1042176992018,1cyclictest0-21swapper/012:32:330
1042176992018,1cyclictest0-21swapper/012:06:320
1042176992018,1cyclictest0-21swapper/011:46:510
1042176992018,1cyclictest0-21swapper/011:40:430
1042176992018,1cyclictest0-21swapper/011:04:590
1042176992018,1cyclictest0-21swapper/010:54:260
1042176992018,1cyclictest0-21swapper/010:02:410
1042176992018,1cyclictest0-21swapper/009:33:070
1042176992018,1cyclictest0-21swapper/008:21:420
1042176992018,1cyclictest0-21swapper/007:54:460
1042176992018,1cyclictest0-21swapper/007:17:000
1042176992017,2cyclictest0-21swapper/008:51:360
1042176992013,5cyclictest1065340-21sshd08:11:430
104217699200,19cyclictest0-21swapper/008:06:220
104217699200,19cyclictest0-21swapper/008:04:190
104217699200,19cyclictest0-21swapper/007:30:590
10771462190,17sleep01077205-21sed08:45:220
10771462190,17sleep01077205-21sed08:45:210
10477022190,4sleep13056-21PK-Backend07:20:211
1042180991917,1cyclictest3056-21PK-Backend08:55:061
1042180991916,2cyclictest3056-21PK-Backend10:12:261
104218099190,6cyclictest1111296-21chrt10:22:221
104218099190,6cyclictest1089584-21chrt09:20:231
104218099190,3cyclictest11057082sleep110:05:411
104218099190,3cyclictest1051583-21chrt07:33:451
104217699192,13cyclictest1116990-21apt10:40:100
1042176991917,1cyclictest0-21swapper/010:10:450
1042176991917,1cyclictest0-21swapper/007:49:180
1042176991917,1cyclictest0-21swapper/007:42:150
1042176991917,1cyclictest0-21swapper/007:42:150
1042176991917,1cyclictest0-21swapper/007:12:270
1042176991913,4cyclictest883-21dbus-daemon11:09:120
1042176991913,4cyclictest1115405-21tr10:35:120
104217699191,16cyclictest1085617-21grep09:10:180
104217699190,1cyclictest0-21swapper/012:00:090
104217699190,1cyclictest0-21swapper/009:45:100
104217699190,1cyclictest0-21swapper/008:19:190
104217699190,18cyclictest0-21swapper/010:23:300
104218499180,4cyclictest10773772sleep208:45:272
104218499180,4cyclictest10773772sleep208:45:272
104218099181,3cyclictest3056-21PK-Backend11:30:541
104218099180,4cyclictest1151916-21dump-pmu-power12:20:001
104218099180,4cyclictest1149559-21processes12:10:201
104218099180,2cyclictest11306622sleep111:17:441
1042176991816,1cyclictest0-21swapper/010:31:000
1042176991816,1cyclictest0-21swapper/009:41:450
1042176991816,1cyclictest0-21swapper/008:39:270
1042176991811,5cyclictest1153721-21date12:25:000
104218099170,3cyclictest1114369-21chrt10:30:191
104217699179,6cyclictest1122909-21cat10:55:160
1042176991714,2cyclictest0-21swapper/008:25:400
1042176991710,5cyclictest1137324-21ntp_states11:35:200
1042176991710,5cyclictest1137324-21ntp_states11:35:190
104217699168,6cyclictest1133970-21sshd11:25:430
1042176991612,3cyclictest1149378-21head12:10:190
1042176991611,4cyclictest1095-21runrttasks09:10:000
104217699157,6cyclictest1130038-21tr11:15:180
1042176991513,1cyclictest0-21swapper/011:14:260
1042176991411,2cyclictest1-21systemd11:34:590
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional