You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-19 - 04:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Tue May 19, 2026 00:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
125627621390,4sleep20-21swapper/222:25:122
124130521310,3sleep33056-21PK-Backend21:40:183
124814121230,2sleep33056-21PK-Backend22:00:193
120503921160,3sleep053-21kcompactd019:56:580
128164621100,3sleep20-21swapper/223:35:182
118706627439,29sleep20-21swapper/219:09:332
118702727339,28sleep30-21swapper/319:09:023
118676226244,13sleep00-21swapper/019:05:370
118542226141,14sleep10-21swapper/119:05:021
12560592480,2sleep33056-21PK-Backend22:25:013
12446902420,3sleep3411rcuc/321:50:173
13036062410,3sleep10-21swapper/100:40:011
12229012380,2sleep00-21swapper/020:50:110
118723399343,13cyclictest0-21swapper/021:35:000
118723399343,13cyclictest0-21swapper/021:35:000
118723399319,9cyclictest1271797-21runrttasks23:08:090
118723399318,5cyclictest1214204-21/usr/sbin/munin20:25:200
118723399298,7cyclictest1209037-21wc20:10:120
118723399294,14cyclictest1302334-21df00:35:130
12686852280,2sleep20-21swapper/223:00:012
118723399286,8cyclictest1273364-21chrt23:11:020
118723399285,13cyclictest0-21swapper/021:10:520
118723399285,11cyclictest1251266-21df22:10:130
118723399285,10cyclictest1233678-21diskstats21:20:130
118723399284,8cyclictest1289310-21munin-run00:00:000
1187238992722,4cyclictest26-21ksoftirqd/100:30:001
118723399277,11cyclictest1297804-21chrt00:22:550
118723399274,13cyclictest0-21swapper/023:45:310
118723399273,15cyclictest0-21swapper/022:55:130
118723399273,14cyclictest0-21swapper/000:25:010
118723399272,6cyclictest0-21swapper/020:05:110
1187233992715,7cyclictest1187446-21apt19:10:120
118723399266,7cyclictest1213-21kerneloops23:52:430
118723399266,10cyclictest1191042-21chrt19:19:000
118723399265,8cyclictest1247837-21hddtemp_smartct22:00:160
118723399265,7cyclictest1220644-21sshd20:41:540
118723399265,12cyclictest0-21swapper/000:08:110
118723399264,14cyclictest0-21swapper/021:29:280
118723399263,7cyclictest1-21systemd20:40:000
118723399263,7cyclictest1-21systemd20:40:000
1187238992520,4cyclictest26-21ksoftirqd/120:55:001
118723399258,5cyclictest1221161-21acpi20:45:110
118723399258,5cyclictest1194418-21chrt19:27:340
118723399256,7cyclictest0-21swapper/022:48:240
118723399255,8cyclictest0-21swapper/019:40:040
118723399254,9cyclictest1280948-21cpu23:35:110
118723399254,8cyclictest1233035-21latency_hist21:20:010
118723399253,9cyclictest886-21NetworkManager19:50:000
118723399252,9cyclictest0-21swapper/022:35:130
118723399252,9cyclictest0-21swapper/022:35:120
118723399252,14cyclictest0-21swapper/000:15:180
1187233992514,8cyclictest1266495-21perf22:55:000
139099240,3rtkit-daemon1389-21rtkit-daemon21:20:532
118723899240,22cyclictest1256208-21cat22:25:121
118723399245,8cyclictest1095-21runrttasks21:07:270
118723399245,7cyclictest0-21swapper/022:43:230
118723399244,8cyclictest0-21swapper/020:04:200
118723399242,14cyclictest0-21swapper/022:25:200
1187233992415,6cyclictest2156-21gnome-shell23:15:080
1187243992313,9cyclictest1216678-21ntp_kernel_err20:30:183
1187240992321,1cyclictest0-21swapper/222:07:312
1187240992321,1cyclictest0-21swapper/219:34:412
1187240992315,6cyclictest1209387-21df20:10:132
118723899235,5cyclictest26-21ksoftirqd/121:35:011
118723899235,5cyclictest26-21ksoftirqd/121:35:001
118723899230,5cyclictest1198403-21apt-config19:40:011
118723399235,5cyclictest0-21swapper/022:21:080
118723399234,7cyclictest0-21swapper/020:33:510
118723399234,6cyclictest0-21swapper/000:00:220
118723399233,12cyclictest0-21swapper/021:50:100
1187233992318,3cyclictest1276433-21/usr/sbin/munin23:20:190
1187233992316,4cyclictest1-21systemd21:35:070
1187233992315,4cyclictest1246563-21python321:55:200
1187233992315,3cyclictest1268762-21apt23:00:120
1187233992313,7cyclictest1195306-21/usr/sbin/munin19:30:160
1187243992219,2cyclictest3056-21PK-Backend00:07:183
1187243992218,2cyclictest3056-21PK-Backend23:25:003
1187243992214,6cyclictest1289979-21fw_packets00:00:163
1187243992213,7cyclictest1259293-21cron22:35:003
118724399220,20cyclictest3056-21PK-Backend20:58:593
1187240992220,1cyclictest0-21swapper/223:34:512
1187240992220,1cyclictest0-21swapper/223:34:512
1187240992220,1cyclictest0-21swapper/222:43:072
1187240992220,1cyclictest0-21swapper/222:31:312
1187240992220,1cyclictest0-21swapper/222:12:592
1187240992220,1cyclictest0-21swapper/220:52:552
1187240992220,1cyclictest0-21swapper/219:11:092
118724099220,21cyclictest0-21swapper/220:55:332
118723899222,5cyclictest26-21ksoftirqd/100:00:001
1187238992219,2cyclictest3056-21PK-Backend20:09:471
1187238992214,6cyclictest1229464-21sshd21:08:531
118723399227,13cyclictest1241212-21grep21:40:170
1187233992213,7cyclictest1284118-21idleruntime-cro23:45:000
1187233992213,6cyclictest1227845-21ls21:05:000
1187233992213,5cyclictest1293054-21apt00:10:130
1187233992212,8cyclictest2399-21gdbus19:38:030
1187233992212,6cyclictest1224613-21/usr/sbin/munin20:55:240
139099210,4rtkit-daemon1389-21rtkit-daemon23:42:412
118724399210,1cyclictest3056-21PK-Backend22:38:363
118724399210,1cyclictest3056-21PK-Backend22:38:363
118724399210,19cyclictest3056-21PK-Backend19:25:513
118724399210,19cyclictest3056-21PK-Backend00:36:383
118724399210,19cyclictest3056-21PK-Backend00:25:293
1187240992119,1cyclictest3056-21PK-Backend23:05:382
1187240992119,1cyclictest0-21swapper/223:26:132
1187240992119,1cyclictest0-21swapper/222:53:072
1187240992119,1cyclictest0-21swapper/222:46:592
1187240992119,1cyclictest0-21swapper/222:17:522
1187240992119,1cyclictest0-21swapper/222:03:532
1187240992119,1cyclictest0-21swapper/221:35:132
1187240992119,1cyclictest0-21swapper/221:00:462
1187240992119,1cyclictest0-21swapper/219:39:342
1187240992119,1cyclictest0-21swapper/200:29:472
1187240992119,1cyclictest0-21swapper/200:10:062
1187240992118,2cyclictest3056-21PK-Backend21:06:002
1187240992118,2cyclictest3056-21PK-Backend20:17:472
1187240992118,2cyclictest3056-21PK-Backend19:19:402
118724099210,20cyclictest0-21swapper/221:42:162
118724099210,20cyclictest0-21swapper/220:32:202
118724099210,20cyclictest0-21swapper/220:24:022
118723899212,2cyclictest141rcu_preempt21:15:011
1187238992119,1cyclictest0-21swapper/123:35:151
1187238992119,1cyclictest0-21swapper/122:58:451
1187238992119,1cyclictest0-21swapper/122:20:321
1187238992119,1cyclictest0-21swapper/122:19:311
1187238992119,1cyclictest0-21swapper/121:03:401
1187238992119,1cyclictest0-21swapper/120:01:081
1187238992119,1cyclictest0-21swapper/100:16:111
1187238992116,4cyclictest26-21ksoftirqd/119:30:001
1187238992114,5cyclictest883-21dbus-daemon23:15:081
118723899210,20cyclictest0-21swapper/122:12:011
118723899210,20cyclictest0-21swapper/121:35:101
118723899210,20cyclictest0-21swapper/121:26:311
118723899210,20cyclictest0-21swapper/120:40:011
118723899210,20cyclictest0-21swapper/119:16:121
118723899210,20cyclictest0-21swapper/119:14:231
118723399218,11cyclictest1242452-21cstates21:45:150
1187233992113,6cyclictest1250034-21python322:05:200
1187233992112,7cyclictest1212245-21awk20:20:010
139099200,3rtkit-daemon1389-21rtkit-daemon19:41:012
1187243992018,1cyclictest3056-21PK-Backend22:58:093
1187243992018,1cyclictest3056-21PK-Backend22:09:363
1187243992018,1cyclictest0-21swapper/320:26:483
1187243992017,2cyclictest3056-21PK-Backend23:50:013
1187243992017,2cyclictest3056-21PK-Backend19:34:093
1187243992017,2cyclictest3056-21PK-Backend00:14:073
1187243992012,6cyclictest42-21ksoftirqd/300:15:133
118724399200,19cyclictest3056-21PK-Backend21:33:333
118724399200,19cyclictest3056-21PK-Backend21:33:333
118724399200,19cyclictest0-21swapper/321:16:173
118724399200,19cyclictest0-21swapper/321:14:273
118724399200,19cyclictest0-21swapper/321:09:133
118724399200,19cyclictest0-21swapper/319:43:423
118724399200,17cyclictest1233546-21cstates21:20:113
1187240992018,1cyclictest0-21swapper/223:22:552
1187240992018,1cyclictest0-21swapper/223:17:342
1187240992018,1cyclictest0-21swapper/222:39:152
1187240992018,1cyclictest0-21swapper/222:39:152
1187240992018,1cyclictest0-21swapper/222:22:452
1187240992018,1cyclictest0-21swapper/219:50:362
1187240992018,1cyclictest0-21swapper/200:31:022
118724099200,1cyclictest0-21swapper/200:05:202
118724099200,19cyclictest0-21swapper/221:52:102
118724099200,19cyclictest0-21swapper/221:27:102
118724099200,19cyclictest0-21swapper/221:13:512
118724099200,19cyclictest0-21swapper/220:35:112
118724099200,19cyclictest0-21swapper/220:35:112
118724099200,19cyclictest0-21swapper/220:02:392
118724099200,19cyclictest0-21swapper/219:57:262
118724099200,19cyclictest0-21swapper/200:23:592
118724099200,19cyclictest0-21swapper/200:15:542
118723899209,2cyclictest141rcu_preempt00:00:221
118723899206,3cyclictest26-21ksoftirqd/123:20:191
118723899202,8cyclictest0-21swapper/119:55:201
1187238992018,1cyclictest3056-21PK-Backend21:16:311
1187238992018,1cyclictest0-21swapper/121:45:381
1187238992018,1cyclictest0-21swapper/121:40:241
1187238992018,1cyclictest0-21swapper/120:30:551
1187238992018,1cyclictest0-21swapper/120:12:231
1187238992018,1cyclictest0-21swapper/119:22:271
1187238992018,1cyclictest0-21swapper/100:34:091
1187238992012,6cyclictest1272460-21ls23:10:131
1187238992012,6cyclictest1272019-21sed23:10:011
1187238992012,2cyclictest26-21ksoftirqd/121:59:071
1187238992011,7cyclictest1284132-21systemd23:45:001
118723899200,19cyclictest0-21swapper/121:53:281
118723899200,19cyclictest0-21swapper/119:49:041
118723899200,19cyclictest0-21swapper/100:20:441
118723899200,19cyclictest0-21swapper/100:11:381
118723899200,18cyclictest1203562-21/usr/sbin/munin19:55:001
118723399208,10cyclictest1258046-21cstates22:30:130
1187233992018,1cyclictest0-21swapper/019:50:450
1187233992014,4cyclictest1280055-21grep23:30:210
1187233992014,4cyclictest1280055-21grep23:30:210
1187233992011,7cyclictest1212652-21cpuspeed_turbos20:20:100
1187233992011,7cyclictest1191546-21cut19:20:110
1187233992011,6cyclictest1301101-21python300:30:200
1187233992010,8cyclictest1278924-21awk23:30:000
118724399195,3cyclictest141rcu_preempt20:45:003
1187243991917,1cyclictest3056-21PK-Backend23:31:223
1187243991917,1cyclictest3056-21PK-Backend23:31:223
1187243991917,1cyclictest3056-21PK-Backend23:01:143
1187243991917,1cyclictest3056-21PK-Backend22:47:083
1187243991917,1cyclictest3056-21PK-Backend22:16:063
1187243991917,1cyclictest3056-21PK-Backend21:46:383
1187243991917,1cyclictest3056-21PK-Backend20:48:593
1187243991917,1cyclictest3056-21PK-Backend20:35:543
1187243991917,1cyclictest3056-21PK-Backend20:35:543
1187243991917,1cyclictest3056-21PK-Backend19:15:113
1187243991917,1cyclictest0-21swapper/321:55:303
1187243991916,1cyclictest3056-21PK-Backend19:53:023
1187243991915,2cyclictest3056-21PK-Backend23:55:083
1187243991914,4cyclictest1227834-21munin-plugin-st21:05:003
1187243991914,3cyclictest42-21ksoftirqd/320:01:383
1187243991913,4cyclictest1198121-21idleruntime-cro19:40:003
1187243991910,7cyclictest1224379-21cut20:55:003
118724399190,1cyclictest3056-21PK-Backend23:10:263
118724399190,1cyclictest3056-21PK-Backend22:54:043
118724399190,1cyclictest3056-21PK-Backend21:27:313
118724399190,1cyclictest3056-21PK-Backend00:34:223
118724399190,18cyclictest3056-21PK-Backend21:36:103
118724399190,18cyclictest3056-21PK-Backend19:49:383
118724399190,18cyclictest3056-21PK-Backend19:22:133
1187240991917,1cyclictest0-21swapper/223:14:162
1187240991917,1cyclictest0-21swapper/221:33:242
1187240991917,1cyclictest0-21swapper/221:33:242
1187240991917,1cyclictest0-21swapper/220:06:182
1187240991911,6cyclictest1233022-21cron21:20:002
118724099190,1cyclictest3056-21PK-Backend23:51:542
118724099190,1cyclictest3056-21PK-Backend00:00:332
118724099190,1cyclictest0-21swapper/219:26:102
118724099190,18cyclictest3056-21PK-Backend21:55:482
118724099190,18cyclictest0-21swapper/200:35:352
118723899197,4cyclictest26-21ksoftirqd/123:45:311
1187238991913,4cyclictest26-21ksoftirqd/123:50:161
1187238991911,6cyclictest436-20systemd-journal20:29:591
1187238991911,6cyclictest1263832-21sed22:45:171
1187238991911,2cyclictest26-21ksoftirqd/122:37:291
1187238991911,2cyclictest26-21ksoftirqd/122:37:291
1187238991910,7cyclictest1249912-21ntp_states22:05:191
118723899190,1cyclictest0-21swapper/123:04:401
118723899190,1cyclictest0-21swapper/122:30:541
118723899190,1cyclictest0-21swapper/122:03:081
118723899190,1cyclictest0-21swapper/121:23:541
118723899190,1cyclictest0-21swapper/120:39:001
118723899190,1cyclictest0-21swapper/120:38:591
118723899190,1cyclictest0-21swapper/120:18:241
118723899190,1cyclictest0-21swapper/119:30:461
1187233991911,6cyclictest1253341-21tr22:15:190
1187243991816,1cyclictest3056-21PK-Backend22:11:253
1187243991816,1cyclictest3056-21PK-Backend20:06:403
1187243991816,1cyclictest0-21swapper/323:40:483
1187243991815,2cyclictest3056-21PK-Backend19:57:143
1187243991811,5cyclictest1209181-21cut20:10:133
1187243991810,2cyclictest42-21ksoftirqd/300:25:003
118724399180,5cyclictest42-21ksoftirqd/322:45:003
1187240991816,1cyclictest0-21swapper/220:49:302
1187240991816,1cyclictest0-21swapper/220:40:522
1187240991810,6cyclictest883-21dbus-daemon19:25:012
118723899184,12cyclictest1292044-21ntpq00:05:191
118723899181,4cyclictest0-21swapper/123:25:121
1187238991813,3cyclictest1225096-21if_err_enp2s020:55:161
1187238991812,4cyclictest1213959-21latency_hist20:25:011
1187238991810,2cyclictest26-21ksoftirqd/119:44:351
118723899180,1cyclictest0-21swapper/122:54:391
118724399179,6cyclictest42-21ksoftirqd/323:15:233
118724399178,7cyclictest1270954-21cat23:05:153
1187243991715,1cyclictest3056-21PK-Backend23:51:023
1187243991715,1cyclictest3056-21PK-Backend20:17:083
1187243991714,2cyclictest3056-21PK-Backend22:29:163
1187243991710,5cyclictest42-21ksoftirqd/320:21:393
1187240991715,1cyclictest3056-21PK-Backend20:26:532
1187240991715,1cyclictest0-21swapper/221:47:442
118724099170,6cyclictest883-21dbus-daemon00:00:012
118724099170,5cyclictest0-21swapper/223:45:392
118723899170,7cyclictest26-21ksoftirqd/120:45:201
118724399166,3cyclictest42-21ksoftirqd/319:12:413
1187243991611,2cyclictest42-21ksoftirqd/323:25:213
1187240991614,1cyclictest0-21swapper/223:00:312
1187240991611,4cyclictest1201032-21grep19:45:182
118723899169,6cyclictest1-21systemd23:34:131
118723899169,6cyclictest1-21systemd23:34:131
1187243991511,2cyclictest42-21ksoftirqd/323:35:113
1187238991513,1cyclictest0-21swapper/122:43:031
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional