You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-31 - 10:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Aug 31, 2026 00:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
135634521400,2sleep00-21swapper/000:00:120
134246221210,4sleep33056-21PK-Backend23:20:133
126853321180,4sleep2341ksoftirqd/219:50:132
130673021040,3sleep33056-21PK-Backend21:40:003
13321332920,1sleep10-21swapper/122:50:171
139099910,70rtkit-daemon0-21swapper/019:05:130
125175228670,9sleep20-21swapper/219:05:142
125221126544,13sleep10-21swapper/119:05:201
125287426344,13sleep30-21swapper/319:05:553
13155242490,1sleep00-21swapper/022:05:000
12828992420,4sleep10-21swapper/120:30:211
12585162410,4sleep20-21swapper/219:20:262
13308132380,1chrt1329842-21/usr/sbin/munin22:45:212
12898772240,3sleep03056-21PK-Backend20:50:220
1253351992415,8cyclictest1355956-40.mlocate23:59:591
1253351992415,8cyclictest1355956-40.mlocate23:59:591
125335099240,4cyclictest1261415-21tr19:30:130
125335199231,20cyclictest1264517-21latency_hist19:40:001
1253350992321,1cyclictest0-21swapper/021:21:260
125335099231,9cyclictest1361881-21irqstats00:15:190
13530502220,2sleep00-21swapper/023:50:150
13250472220,4sleep20-21swapper/222:30:162
1253356992220,1cyclictest3056-21PK-Backend20:00:043
1253356992220,1cyclictest3056-21PK-Backend20:00:043
1253356992211,9cyclictest1337453-21/usr/sbin/munin23:05:163
125335699220,20cyclictest3056-21PK-Backend21:49:483
125335699220,20cyclictest3056-21PK-Backend20:32:363
125335699220,20cyclictest3056-21PK-Backend00:36:303
1253354992212,7cyclictest1295621-21apt-get21:10:002
1253351992216,4cyclictest1281761-21dump-pmu-power20:30:001
1253350992220,1cyclictest0-21swapper/021:41:270
1253350992220,1cyclictest0-21swapper/021:08:270
1253350992220,1cyclictest0-21swapper/020:30:150
1253350992212,7cyclictest1268159-21apt19:50:110
139099210,14rtkit-daemon0-21swapper/021:30:390
13618822210,11sleep234-21ksoftirqd/200:15:182
1253356992119,1cyclictest3056-21PK-Backend23:41:423
1253356992119,1cyclictest3056-21PK-Backend23:28:033
1253356992119,1cyclictest3056-21PK-Backend22:58:013
1253356992119,1cyclictest3056-21PK-Backend22:32:403
1253356992119,1cyclictest3056-21PK-Backend22:23:273
1253356992119,1cyclictest3056-21PK-Backend22:18:353
1253356992119,1cyclictest3056-21PK-Backend20:49:453
1253356992119,1cyclictest3056-21PK-Backend20:22:413
1253356992119,1cyclictest3056-21PK-Backend19:22:593
1253356992119,1cyclictest0-21swapper/323:49:123
1253356992119,1cyclictest0-21swapper/322:39:573
1253356992119,1cyclictest0-21swapper/321:08:033
1253356992118,2cyclictest3056-21PK-Backend23:59:063
1253356992118,2cyclictest3056-21PK-Backend23:59:063
1253356992118,2cyclictest3056-21PK-Backend22:45:043
1253356992118,2cyclictest3056-21PK-Backend22:03:063
1253356992118,2cyclictest3056-21PK-Backend20:16:053
1253356992118,2cyclictest1340528-21apt23:15:113
125335699210,1cyclictest1335069-21apt23:00:123
125335699210,19cyclictest3056-21PK-Backend23:53:443
125335699210,19cyclictest3056-21PK-Backend22:28:203
125335699210,19cyclictest3056-21PK-Backend20:14:443
125335699210,19cyclictest3056-21PK-Backend00:08:113
125335699210,19cyclictest1303831-21irqstats21:30:183
1253354992112,6cyclictest1293297-21python321:00:202
125335499210,20cyclictest0-21swapper/220:15:342
125335499210,20cyclictest0-21swapper/219:39:172
1253351992119,1cyclictest3056-21PK-Backend21:58:031
1253351992119,1cyclictest0-21swapper/123:42:401
1253351992119,1cyclictest0-21swapper/123:25:171
1253351992119,1cyclictest0-21swapper/123:20:451
1253351992119,1cyclictest0-21swapper/121:22:281
1253351992119,1cyclictest0-21swapper/121:19:111
1253351992119,1cyclictest0-21swapper/120:22:471
1253351992119,1cyclictest0-21swapper/120:03:131
1253351992119,1cyclictest0-21swapper/120:03:121
1253351992119,1cyclictest0-21swapper/119:18:451
1253351992119,1cyclictest0-21swapper/100:07:021
1253351992110,9cyclictest1322697-21unin-run22:25:011
125335199210,20cyclictest3056-21PK-Backend00:33:311
125335199210,20cyclictest0-21swapper/123:19:091
125335199210,20cyclictest0-21swapper/120:10:571
125335199210,20cyclictest0-21swapper/119:41:371
125335199210,20cyclictest0-21swapper/100:23:371
1253350992119,1cyclictest0-21swapper/023:37:460
1253350992119,1cyclictest0-21swapper/023:13:250
1253350992119,1cyclictest0-21swapper/022:52:440
1253350992119,1cyclictest0-21swapper/022:36:490
1253350992119,1cyclictest0-21swapper/021:04:010
1253350992119,1cyclictest0-21swapper/020:36:160
1253350992119,1cyclictest0-21swapper/020:22:180
1253350992119,1cyclictest0-21swapper/019:25:540
1253350992113,6cyclictest883-21dbus-daemon19:21:130
1253350992113,6cyclictest1320497-21cat22:15:220
125335099210,20cyclictest0-21swapper/023:59:430
125335099210,20cyclictest0-21swapper/023:59:430
125335099210,20cyclictest0-21swapper/022:26:000
125335099210,20cyclictest0-21swapper/022:23:510
125335099210,20cyclictest0-21swapper/021:48:160
125335099210,20cyclictest0-21swapper/021:18:080
139099200,14rtkit-daemon0-21swapper/021:51:580
139099200,14rtkit-daemon0-21swapper/000:37:110
139099200,13rtkit-daemon0-21swapper/000:33:580
13446292200,4sleep20-21swapper/223:25:172
1253356992018,1cyclictest3056-21PK-Backend23:37:573
1253356992018,1cyclictest3056-21PK-Backend00:26:373
1253356992018,1cyclictest0-21swapper/322:12:193
1253356992018,1cyclictest0-21swapper/322:06:383
1253356992018,1cyclictest0-21swapper/322:06:373
1253356992018,1cyclictest0-21swapper/321:00:343
1253356992018,1cyclictest0-21swapper/319:55:043
125335699201,7cyclictest1363740-21sed00:20:183
1253356992017,2cyclictest3056-21PK-Backend23:11:273
1253356992013,5cyclictest42-21ksoftirqd/321:50:223
1253356992012,6cyclictest1267292-21python319:45:203
125335699200,7cyclictest1361398-21awk00:15:143
125335699200,2cyclictest3056-21PK-Backend19:38:353
125335699200,19cyclictest3056-21PK-Backend20:51:553
125335699200,19cyclictest0-21swapper/319:25:103
125335699200,19cyclictest0-21swapper/300:12:443
125335699200,18cyclictest3056-21PK-Backend20:59:463
1253354992018,1cyclictest0-21swapper/223:30:482
1253354992018,1cyclictest0-21swapper/221:51:192
1253354992018,1cyclictest0-21swapper/221:37:272
1253354992018,1cyclictest0-21swapper/220:07:022
1253354992018,1cyclictest0-21swapper/200:21:112
1253354992013,5cyclictest1352380-21cron23:50:012
1253354992013,5cyclictest1340244-21cron23:15:002
125335499200,8cyclictest13563472cpuspeed_turbos00:00:132
125335499200,1cyclictest0-21swapper/222:27:352
125335499200,19cyclictest0-21swapper/223:36:572
125335499200,19cyclictest0-21swapper/222:58:302
125335499200,19cyclictest0-21swapper/222:37:492
125335499200,19cyclictest0-21swapper/221:27:532
125335499200,19cyclictest0-21swapper/220:57:252
125335499200,19cyclictest0-21swapper/219:17:002
125335499200,19cyclictest0-21swapper/200:36:392
125335499200,19cyclictest0-21swapper/200:34:162
1253351992018,1cyclictest3056-21PK-Backend22:36:371
1253351992018,1cyclictest0-21swapper/123:33:351
1253351992018,1cyclictest0-21swapper/123:12:251
1253351992018,1cyclictest0-21swapper/122:26:371
1253351992018,1cyclictest0-21swapper/120:49:161
1253351992015,4cyclictest26-21ksoftirqd/121:10:011
1253351992015,3cyclictest1095-21runrttasks20:35:141
1253351992012,6cyclictest1356870-21head00:00:181
1253351992012,6cyclictest1297229-21awk21:10:231
125335199200,1cyclictest3056-21PK-Backend20:05:081
125335199200,1cyclictest0-21swapper/100:35:071
125335199200,19cyclictest3056-21PK-Backend22:09:471
125335199200,19cyclictest3056-21PK-Backend22:09:471
125335199200,19cyclictest0-21swapper/123:50:591
125335199200,19cyclictest0-21swapper/122:56:041
125335199200,19cyclictest0-21swapper/122:32:181
125335199200,19cyclictest0-21swapper/121:43:381
125335199200,19cyclictest0-21swapper/121:36:141
125335199200,19cyclictest0-21swapper/100:25:121
1253350992018,1cyclictest3056-21PK-Backend23:15:010
1253350992018,1cyclictest0-21swapper/023:42:180
1253350992018,1cyclictest0-21swapper/021:29:570
1253350992018,1cyclictest0-21swapper/019:36:210
125335099200,19cyclictest3056-21PK-Backend19:11:060
125335099200,19cyclictest0-21swapper/023:46:040
125335099200,19cyclictest0-21swapper/022:05:400
125335099200,19cyclictest0-21swapper/022:05:400
125335099200,19cyclictest0-21swapper/020:41:440
125335099200,19cyclictest0-21swapper/020:10:270
125335099200,19cyclictest0-21swapper/019:19:240
125335699195,12cyclictest0-21swapper/320:05:133
1253356991917,1cyclictest3056-21PK-Backend21:21:363
1253356991917,1cyclictest3056-21PK-Backend21:11:553
1253356991917,1cyclictest3056-21PK-Backend20:35:193
1253356991917,1cyclictest3056-21PK-Backend20:29:183
1253356991916,2cyclictest3056-21PK-Backend19:31:043
1253356991911,6cyclictest1254271-21ntp_states19:10:193
125335699190,1cyclictest3056-21PK-Backend22:52:543
125335699190,1cyclictest3056-21PK-Backend22:44:173
125335699190,1cyclictest3056-21PK-Backend21:44:203
125335699190,1cyclictest3056-21PK-Backend21:18:183
125335699190,1cyclictest3056-21PK-Backend19:40:583
125335699190,1cyclictest3056-21PK-Backend19:16:443
125335699190,1cyclictest3056-21PK-Backend00:34:543
125335699190,18cyclictest0-21swapper/321:26:503
125335499199,8cyclictest1280023-21cron20:25:002
1253354991917,1cyclictest0-21swapper/223:44:412
1253354991917,1cyclictest0-21swapper/222:17:352
1253354991917,1cyclictest0-21swapper/221:21:512
1253354991917,1cyclictest0-21swapper/221:17:112
1253354991917,1cyclictest0-21swapper/220:54:262
1253354991912,5cyclictest1342250-21find23:20:132
1253354991912,5cyclictest1335561-21sed23:00:162
1253354991912,5cyclictest1328497-21df22:40:112
1253354991911,6cyclictest1341981-21cron23:20:002
125335499190,1cyclictest0-21swapper/223:58:342
125335499190,1cyclictest0-21swapper/223:58:332
125335499190,1cyclictest0-21swapper/223:06:482
125335499190,1cyclictest0-21swapper/222:07:342
125335499190,1cyclictest0-21swapper/222:07:342
125335499190,1cyclictest0-21swapper/221:56:122
125335499190,1cyclictest0-21swapper/219:12:272
125335499190,1cyclictest0-21swapper/200:27:202
125335499190,1cyclictest0-21swapper/200:11:512
125335499190,18cyclictest0-21swapper/222:11:542
125335499190,18cyclictest0-21swapper/222:01:532
125335499190,18cyclictest0-21swapper/221:49:082
125335499190,18cyclictest0-21swapper/220:31:092
125335499190,18cyclictest0-21swapper/219:31:332
125335199199,8cyclictest883-21dbus-daemon19:55:001
1253351991917,1cyclictest3056-21PK-Backend21:32:551
1253351991917,1cyclictest0-21swapper/123:38:071
1253351991917,1cyclictest0-21swapper/123:01:111
1253351991917,1cyclictest0-21swapper/122:48:461
1253351991917,1cyclictest0-21swapper/122:42:171
1253351991917,1cyclictest0-21swapper/120:55:441
1253351991917,1cyclictest0-21swapper/120:44:231
1253351991915,3cyclictest26-21ksoftirqd/120:55:001
1253351991914,3cyclictest1260949-21cron19:30:001
1253351991912,5cyclictest1257861-21sshd19:20:161
1253351991911,5cyclictest1318787-21sshd22:13:161
125335199190,1cyclictest0-21swapper/121:49:051
125335199190,1cyclictest0-21swapper/119:33:531
125335199190,18cyclictest0-21swapper/122:16:011
125335199190,18cyclictest0-21swapper/121:00:041
125335199190,18cyclictest0-21swapper/119:46:231
1253350991917,1cyclictest0-21swapper/022:14:050
1253350991917,1cyclictest0-21swapper/021:38:420
1253350991917,1cyclictest0-21swapper/020:25:020
1253350991917,1cyclictest0-21swapper/019:58:520
1253350991916,2cyclictest0-21swapper/022:30:200
1253350991916,2cyclictest0-21swapper/020:08:320
1253350991912,6cyclictest1272168-21sed20:00:180
1253350991912,6cyclictest1272168-21sed20:00:180
1253350991912,5cyclictest1344187-21expr23:25:130
1253350991912,5cyclictest13-21ksoftirqd/022:55:130
1253350991911,3cyclictest1296903-21sh21:10:200
125335099190,1cyclictest0-21swapper/023:02:110
125335099190,1cyclictest0-21swapper/022:49:460
125335099190,1cyclictest0-21swapper/020:46:380
125335099190,18cyclictest0-21swapper/022:43:180
125335099190,18cyclictest0-21swapper/000:25:520
1253356991816,1cyclictest3056-21PK-Backend20:42:083
1253356991813,3cyclictest1313363-21vmstat21:55:233
1253354991816,1cyclictest0-21swapper/222:21:542
1253354991816,1cyclictest0-21swapper/221:33:212
1253354991816,1cyclictest0-21swapper/220:47:582
1253354991816,1cyclictest0-21swapper/220:12:162
1253354991816,1cyclictest0-21swapper/219:25:382
125335199189,7cyclictest1269893-21apt19:55:131
125335199182,11cyclictest1336524-21apt-get23:05:031
1253351991816,1cyclictest0-21swapper/123:47:141
1253351991816,1cyclictest0-21swapper/121:27:421
1253351991816,1cyclictest0-21swapper/120:18:071
1253351991812,3cyclictest1362870-21turbostat00:19:591
1253350991817,1cyclictest0-21swapper/020:56:380
1253350991816,1cyclictest0-21swapper/000:21:200
1253350991816,1cyclictest0-21swapper/000:13:360
1253350991811,5cyclictest1358022-21ls00:05:120
1253350991810,6cyclictest436-20systemd-journal20:19:590
125335099180,1cyclictest0-21swapper/023:05:080
139099170,3rtkit-daemon0-21swapper/021:59:150
125335499179,7cyclictest883-21dbus-daemon23:51:132
1253354991715,1cyclictest0-21swapper/222:50:272
1253354991715,1cyclictest0-21swapper/220:43:392
1253354991715,1cyclictest0-21swapper/219:55:592
1253354991710,5cyclictest1307365-21grep21:40:162
125335199176,8cyclictest1255027-21unin-run19:15:001
1253351991715,1cyclictest3056-21PK-Backend21:51:141
125335099179,6cyclictest1264997-21cat19:40:120
1253350991715,1cyclictest0-21swapper/023:33:140
1253350991712,3cyclictest1267892-21latency_hist19:50:010
1253350991711,5cyclictest1342792-21sed23:20:180
1253356991614,1cyclictest3056-21PK-Backend23:33:243
1253356991614,1cyclictest3056-21PK-Backend19:51:593
1253356991614,1cyclictest3056-21PK-Backend00:04:263
1253354991614,1cyclictest3056-21PK-Backend21:10:022
1253354991614,1cyclictest0-21swapper/220:29:462
1253354991614,1cyclictest0-21swapper/200:05:442
1253354991611,3cyclictest1095-21runrttasks20:00:022
1253354991611,3cyclictest1095-21runrttasks20:00:012
125335499150,6cyclictest3056-21PK-Backend19:45:182
125335199159,5cyclictest1360739-21uptime00:10:231
125335199153,8cyclictest2174-21JS10:27:481
125335499149,3cyclictest1264826-21cpuspeed_turbos19:40:132
1253354991412,1cyclictest0-21swapper/220:38:462
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional