You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-25 - 00:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Fri Jul 24, 2026 12:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
115949721310,6sleep01159493-21latency_hist08:15:010
121204921290,3sleep21212029-21apt-get10:45:002
113904821160,4sleep33056-21PK-Backend07:15:213
124704021080,3sleep03056-21PK-Backend12:20:180
1390991040,90rtkit-daemon0-21swapper/207:09:272
120064721030,4sleep23056-21PK-Backend10:10:202
113560127941,14sleep00-21swapper/007:08:080
113557627740,14sleep10-21swapper/107:07:501
113469427555,14sleep30-21swapper/307:05:183
11901052680,4sleep03056-21PK-Backend09:40:180
11901052680,4sleep03056-21PK-Backend09:40:180
12038232480,4sleep13056-21PK-Backend10:20:131
11948132480,9sleep234-21ksoftirqd/209:55:162
12155532390,2sleep30-21swapper/310:50:223
11415162300,4chrt1141519-21munin-plugin-st07:25:010
113589399263,15cyclictest42-21ksoftirqd/309:30:003
1135893992621,3cyclictest411rcuc/310:15:023
113589399261,24cyclictest1243855-21/usr/sbin/munin12:10:213
113588699269,7cyclictest1171946-21tr08:50:130
1135888992520,3cyclictest1240787-21cron12:05:011
1135888992510,10cyclictest2267-21packagekitd11:10:041
139099240,16rtkit-daemon0-21swapper/208:30:212
1135893992410,12cyclictest1151012-21latency07:50:183
11817212230,7sleep3411rcuc/309:15:233
1135891992319,3cyclictest1223192-21chrt11:11:392
1135888992321,1cyclictest0-21swapper/112:25:041
1135886992314,7cyclictest1136448-21df07:10:140
1135886992310,11cyclictest918-21rs:main0
1135886992310,11cyclictest1237365-21unin-run11:55:010
11867482220,7sleep2341ksoftirqd/209:30:182
113589399229,11cyclictest42-21ksoftirqd/311:00:123
113589399220,21cyclictest3056-21PK-Backend11:47:033
113589399220,21cyclictest3056-21PK-Backend09:45:023
113589399220,20cyclictest1165185-21expr08:30:123
113589199220,20cyclictest1197903-21munin-run10:05:002
1135888992220,1cyclictest3056-21PK-Backend09:35:381
1135888992220,1cyclictest3056-21PK-Backend09:34:371
1135888992220,1cyclictest3056-21PK-Backend07:43:041
1135888992220,1cyclictest0-21swapper/109:25:381
113588899220,21cyclictest0-21swapper/108:55:091
1135886992220,1cyclictest0-21swapper/009:57:120
1135886992220,1cyclictest0-21swapper/009:17:590
1135886992219,2cyclictest3056-21PK-Backend10:13:560
113588699220,21cyclictest0-21swapper/010:38:230
139099210,15rtkit-daemon3056-21PK-Backend11:56:473
1135893992113,6cyclictest1209535-21ntp_kernel_err10:35:173
113589399210,20cyclictest0-21swapper/308:15:393
113589399210,19cyclictest3056-21PK-Backend11:40:133
1135891992119,1cyclictest0-21swapper/210:36:382
113589199210,20cyclictest0-21swapper/212:35:202
113589199210,20cyclictest0-21swapper/207:44:282
113589199210,1cyclictest0-21swapper/212:16:412
113589199210,19cyclictest0-21swapper/211:06:122
113589199210,19cyclictest0-21swapper/209:43:302
113589199210,19cyclictest0-21swapper/209:43:302
113589199210,19cyclictest0-21swapper/207:14:252
1135888992119,1cyclictest3056-21PK-Backend12:17:331
1135888992119,1cyclictest3056-21PK-Backend09:01:381
1135888992119,1cyclictest0-21swapper/111:20:361
1135888992119,1cyclictest0-21swapper/110:11:541
1135888992119,1cyclictest0-21swapper/110:08:021
1135888992119,1cyclictest0-21swapper/109:47:131
1135888992119,1cyclictest0-21swapper/109:22:331
1135888992119,1cyclictest0-21swapper/107:24:171
1135888992119,1cyclictest0-21swapper/107:17:221
1135888992117,3cyclictest1195645-21timerandwakeup09:55:261
113588899210,20cyclictest3056-21PK-Backend10:31:011
113588899210,20cyclictest3056-21PK-Backend08:24:331
113588899210,19cyclictest3056-21PK-Backend08:06:161
1135886992119,1cyclictest0-21swapper/012:16:040
1135886992119,1cyclictest0-21swapper/012:09:070
1135886992119,1cyclictest0-21swapper/012:04:340
1135886992119,1cyclictest0-21swapper/011:05:530
1135886992119,1cyclictest0-21swapper/011:01:280
1135886992119,1cyclictest0-21swapper/010:33:300
1135886992119,1cyclictest0-21swapper/010:28:360
1135886992119,1cyclictest0-21swapper/010:23:420
1135886992119,1cyclictest0-21swapper/009:46:110
1135886992119,1cyclictest0-21swapper/009:32:040
1135886992119,1cyclictest0-21swapper/008:57:100
1135886992119,1cyclictest0-21swapper/008:05:390
1135886992119,1cyclictest0-21swapper/007:56:480
1135886992119,1cyclictest0-21swapper/007:54:110
1135886992119,1cyclictest0-21swapper/007:40:460
1135886992118,2cyclictest3056-21PK-Backend10:09:030
113588699210,8cyclictest1182541-21sort09:20:110
139099200,14rtkit-daemon3056-21PK-Backend11:18:313
12399802200,4chrt0-21swapper/212:00:192
12356112200,8chrt436-20systemd-journal11:50:002
12240362200,3sleep00-21swapper/011:15:130
1135893992018,1cyclictest3056-21PK-Backend07:23:553
1135893992018,1cyclictest0-21swapper/312:17:183
1135893992018,1cyclictest0-21swapper/307:44:493
1135893992015,3cyclictest1154963-21ntp_states08:00:183
113589399200,1cyclictest3056-21PK-Backend10:30:183
113589399200,1cyclictest0-21swapper/311:33:243
113589399200,1cyclictest0-21swapper/311:33:233
113589399200,1cyclictest0-21swapper/309:12:443
113589399200,19cyclictest3056-21PK-Backend12:00:413
113589399200,19cyclictest3056-21PK-Backend11:36:483
113589399200,19cyclictest3056-21PK-Backend08:37:283
113589399200,19cyclictest0-21swapper/312:39:423
113589399200,19cyclictest0-21swapper/312:25:493
113589399200,19cyclictest0-21swapper/309:41:583
113589399200,19cyclictest0-21swapper/309:41:583
113589399200,19cyclictest0-21swapper/309:38:533
113589399200,19cyclictest0-21swapper/308:58:043
113589399200,19cyclictest0-21swapper/308:14:373
113589399200,18cyclictest3056-21PK-Backend10:26:253
113589399200,18cyclictest3056-21PK-Backend07:13:533
113589199209,9cyclictest1229101-21cut11:30:122
113589199209,9cyclictest1229101-21cut11:30:122
1135891992018,1cyclictest0-21swapper/212:33:452
1135891992018,1cyclictest0-21swapper/212:33:442
1135891992018,1cyclictest0-21swapper/211:43:562
1135891992018,1cyclictest0-21swapper/211:23:282
1135891992018,1cyclictest0-21swapper/208:08:482
1135891992018,1cyclictest0-21swapper/207:51:172
1135891992016,2cyclictest1177091-40unattended-upgr09:05:042
1135891992015,4cyclictest1160158-21awk08:15:162
1135891992014,4cyclictest1170736-21cut08:45:172
1135891992010,7cyclictest1176772-21apt-get09:04:562
113589199200,19cyclictest0-21swapper/212:24:252
113589199200,19cyclictest0-21swapper/210:45:082
113589199200,19cyclictest0-21swapper/210:30:422
113589199200,19cyclictest0-21swapper/209:51:422
113589199200,19cyclictest0-21swapper/208:44:152
113589199200,19cyclictest0-21swapper/208:01:312
113589199200,19cyclictest0-21swapper/207:59:422
113589199200,19cyclictest0-21swapper/207:16:212
1135888992018,1cyclictest3056-21PK-Backend10:51:021
1135888992018,1cyclictest3056-21PK-Backend10:01:201
1135888992018,1cyclictest3056-21PK-Backend09:14:361
1135888992018,1cyclictest3056-21PK-Backend08:16:421
1135888992018,1cyclictest0-21swapper/111:57:261
1135888992018,1cyclictest0-21swapper/111:05:071
1135888992018,1cyclictest0-21swapper/110:27:091
1135888992018,1cyclictest0-21swapper/108:36:031
1135888992015,3cyclictest1169829-21turbostat.cron08:45:001
113588899200,1cyclictest3056-21PK-Backend11:25:221
113588899200,1cyclictest0-21swapper/111:40:571
113588899200,19cyclictest3056-21PK-Backend12:11:401
113588899200,19cyclictest3056-21PK-Backend10:35:401
113588899200,19cyclictest0-21swapper/111:46:441
113588899200,19cyclictest0-21swapper/107:30:401
1135886992018,1cyclictest0-21swapper/009:54:080
1135886992018,1cyclictest0-21swapper/009:11:510
1135886992018,1cyclictest0-21swapper/008:36:350
1135886992018,1cyclictest0-21swapper/008:22:500
1135886992015,4cyclictest13-21ksoftirqd/011:55:110
1135886992014,5cyclictest1214019-21apt-get10:50:000
1135886992012,6cyclictest436-20systemd-journal09:29:550
1135886992012,6cyclictest1211275-21ntpq10:40:170
1135886992012,6cyclictest1168716-21df08:40:140
1135886992012,2cyclictest13-21ksoftirqd/008:03:080
1135886992010,3cyclictest13-21ksoftirqd/011:45:130
113588699200,1cyclictest0-21swapper/008:33:440
113588699200,19cyclictest0-21swapper/012:34:280
113588699200,19cyclictest0-21swapper/012:34:280
113588699200,19cyclictest0-21swapper/012:27:320
113588699200,19cyclictest0-21swapper/011:10:190
113588699200,19cyclictest0-21swapper/010:57:020
113588699200,19cyclictest0-21swapper/009:09:470
113588699200,19cyclictest0-21swapper/009:02:030
1135893991917,1cyclictest3056-21PK-Backend11:51:023
1135893991917,1cyclictest3056-21PK-Backend11:11:473
1135893991917,1cyclictest3056-21PK-Backend09:31:573
1135893991917,1cyclictest3056-21PK-Backend09:21:573
1135893991917,1cyclictest0-21swapper/311:23:233
1135893991917,1cyclictest0-21swapper/310:41:533
1135893991917,1cyclictest0-21swapper/310:05:033
1135893991917,1cyclictest0-21swapper/307:39:493
1135893991914,4cyclictest42-21ksoftirqd/309:05:213
1135893991914,3cyclictest1246916-21diskstats12:20:153
1135893991914,3cyclictest1156433-21irqstats08:05:153
1135893991912,5cyclictest1250394-21diskstats12:30:133
1135893991912,5cyclictest1250394-21diskstats12:30:133
1135893991912,5cyclictest1152710-21seq07:55:153
113589399191,16cyclictest1177064-21munin-run09:05:003
113589399190,1cyclictest3056-21PK-Backend09:50:563
113589399190,1cyclictest3056-21PK-Backend08:42:223
113589399190,1cyclictest0-21swapper/312:07:243
113589399190,1cyclictest0-21swapper/310:17:393
113589399190,1cyclictest0-21swapper/307:49:513
113589399190,18cyclictest0-21swapper/311:29:123
113589399190,18cyclictest0-21swapper/310:55:533
113589399190,18cyclictest0-21swapper/310:48:223
113589399190,18cyclictest0-21swapper/309:59:553
113589399190,18cyclictest0-21swapper/308:52:103
113589399190,18cyclictest0-21swapper/308:47:163
113589399190,18cyclictest0-21swapper/308:26:343
113589399190,18cyclictest0-21swapper/308:21:073
113589199199,9cyclictest1167169-21wc08:35:162
1135891991917,1cyclictest0-21swapper/212:11:202
1135891991917,1cyclictest0-21swapper/210:29:402
1135891991917,1cyclictest0-21swapper/210:23:472
1135891991917,1cyclictest0-21swapper/210:17:522
1135891991917,1cyclictest0-21swapper/210:09:532
1135891991917,1cyclictest0-21swapper/209:26:212
1135891991917,1cyclictest0-21swapper/208:20:442
1135891991916,2cyclictest0-21swapper/207:39:272
1135891991912,6cyclictest1188537-21awk09:35:192
1135891991912,5cyclictest883-21dbus-daemon07:28:032
1135891991912,5cyclictest34-21ksoftirqd/209:15:112
113589199190,1cyclictest0-21swapper/212:26:012
113589199190,1cyclictest0-21swapper/211:57:292
113589199190,1cyclictest0-21swapper/211:28:282
113589199190,1cyclictest0-21swapper/209:45:342
113589199190,1cyclictest0-21swapper/209:14:512
113589199190,18cyclictest0-21swapper/211:19:152
113589199190,18cyclictest0-21swapper/210:58:062
1135888991917,1cyclictest3056-21PK-Backend08:03:391
1135888991917,1cyclictest0-21swapper/111:51:311
1135888991917,1cyclictest0-21swapper/111:35:571
1135888991917,1cyclictest0-21swapper/108:32:251
1135888991917,1cyclictest0-21swapper/107:37:161
1135888991916,2cyclictest3056-21PK-Backend12:30:591
1135888991916,2cyclictest3056-21PK-Backend12:30:591
1135888991913,5cyclictest1193429-21interrupts09:50:151
113588899190,1cyclictest3056-21PK-Backend12:24:161
113588899190,1cyclictest3056-21PK-Backend09:18:271
113588899190,1cyclictest3056-21PK-Backend07:55:131
113588899190,1cyclictest0-21swapper/111:01:291
113588899190,1cyclictest0-21swapper/110:55:271
113588699193,4cyclictest13-21ksoftirqd/010:15:270
1135886991917,1cyclictest0-21swapper/009:35:090
1135886991917,1cyclictest0-21swapper/008:16:350
1135886991917,1cyclictest0-21swapper/007:49:580
1135886991913,5cyclictest1232836-21if_enp2s011:40:150
1135886991912,5cyclictest1243687-21sort12:10:200
1135886991911,6cyclictest1226188-21ntpq11:20:190
113588699190,1cyclictest0-21swapper/010:50:330
113588699190,1cyclictest0-21swapper/008:46:220
1135893991815,2cyclictest3056-21PK-Backend11:07:223
1135893991815,2cyclictest3056-21PK-Backend07:28:283
1135893991811,5cyclictest436-20systemd-journal10:20:273
1135893991810,6cyclictest1198004-21date10:05:003
1135891991816,1cyclictest0-21swapper/211:53:432
1135891991816,1cyclictest0-21swapper/208:28:012
1135891991816,1cyclictest0-21swapper/207:33:392
1135891991814,3cyclictest978-21snmpd10:54:382
1135891991814,3cyclictest1184012-21dump-pmu-power09:25:012
1135891991811,5cyclictest1219734-21cron11:05:002
113589199180,1cyclictest0-21swapper/207:22:432
1135888991816,1cyclictest3056-21PK-Backend09:42:211
1135888991816,1cyclictest3056-21PK-Backend09:42:201
1135888991815,2cyclictest3056-21PK-Backend08:10:421
1135888991810,6cyclictest1164244-21awk08:25:231
113588899180,1cyclictest3056-21PK-Backend07:47:031
1135886991816,1cyclictest0-21swapper/012:37:400
1135886991816,1cyclictest0-21swapper/008:25:400
1135886991810,6cyclictest1231251-21memory11:35:160
1135886991810,2cyclictest13-21ksoftirqd/010:01:080
1135886991810,2cyclictest13-21ksoftirqd/007:25:230
113588699180,1cyclictest0-21swapper/007:30:440
1135891991715,1cyclictest0-21swapper/212:06:472
1135891991715,1cyclictest0-21swapper/208:55:182
1135891991715,1cyclictest0-21swapper/207:47:042
113589199171,12cyclictest1157916-21runrttasks08:10:032
1135888991715,1cyclictest3056-21PK-Backend10:42:571
1135888991715,1cyclictest3056-21PK-Backend10:17:351
1135888991715,1cyclictest3056-21PK-Backend09:06:181
1135888991715,1cyclictest3056-21PK-Backend08:51:301
1135888991715,1cyclictest0-21swapper/107:11:131
1135888991711,5cyclictest1171041-21expr08:45:221
1135888991710,5cyclictest1241747-21ls12:05:171
1135888991710,5cyclictest1224976-21seq11:17:181
1135886991711,4cyclictest1229075-21tr11:30:110
1135886991711,4cyclictest1229075-21tr11:30:110
1135891991611,4cyclictest1172894-21awk08:50:242
113589199158,6cyclictest1230865-21expr11:35:132
1135888991512,2cyclictest3056-21PK-Backend12:38:291
1135888991511,3cyclictest1141753-21/usr/sbin/munin07:25:171
1135888991510,3cyclictest1095-21runrttasks10:45:181
113588699158,6cyclictest1145467-21cat07:35:140
113588699156,3cyclictest141rcu_preempt11:25:090
113589399141,5cyclictest141rcu_preempt07:32:583
113588899139,3cyclictest3056-21PK-Backend11:33:381
113588899139,3cyclictest3056-21PK-Backend11:33:381
1135888991311,1cyclictest3056-21PK-Backend07:53:371
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional