You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-02 - 16:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu Jul 02, 2026 12:44:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
128853721250,3sleep23056-21PK-Backend10:30:132
139099810,72rtkit-daemon0-21swapper/307:06:383
121626927137,28sleep10-21swapper/107:09:021
121623526544,14sleep20-21swapper/207:08:342
121621826343,14sleep00-21swapper/007:08:210
12261392470,3sleep21225848-21latency_hist07:35:002
13197592420,3sleep00-21swapper/012:00:000
12960492410,3sleep30-21swapper/310:50:233
12216992410,4sleep33056-21PK-Backend07:20:223
13275102290,9sleep126-21ksoftirqd/112:20:141
139099250,4rtkit-daemon1389-21rtkit-daemon11:14:572
1216479992517,5cyclictest3056-21PK-Backend10:15:193
1216476992512,9cyclictest883-21dbus-daemon11:30:012
121647099259,14cyclictest13-21ksoftirqd/007:26:350
121647099250,23cyclictest1262748-21/usr/sbin/munin09:15:160
139099230,3rtkit-daemon1389-21rtkit-daemon12:25:373
12285622230,4sleep33056-21PK-Backend07:40:163
1216470992321,1cyclictest0-21swapper/010:07:340
1216470992311,4cyclictest13-21ksoftirqd/012:20:000
139099220,4rtkit-daemon1389-21rtkit-daemon11:02:012
12523252220,4sleep33056-21PK-Backend08:45:193
121647999228,12cyclictest1281100-21latency_hist10:10:003
1216479992214,6cyclictest1219270-21df07:15:133
1216479992214,2cyclictest141rcu_preempt11:45:013
121647999220,20cyclictest3056-21PK-Backend11:15:163
121647999220,20cyclictest3056-21PK-Backend09:49:303
1216476992220,1cyclictest0-21swapper/208:43:032
1216476992220,1cyclictest0-21swapper/208:22:072
1216476992220,1cyclictest0-21swapper/207:15:292
121647699220,5cyclictest3056-21PK-Backend11:45:012
121647699220,21cyclictest0-21swapper/211:36:132
121647699220,21cyclictest0-21swapper/209:07:492
1216474992220,1cyclictest0-21swapper/111:38:411
1216474992214,6cyclictest1223988-21sshd07:28:341
1216474992210,9cyclictest1314200-21apt-get11:45:001
121647099228,12cyclictest1261971-21unin-run09:15:000
121647099225,4cyclictest13-21ksoftirqd/010:21:440
1216470992220,1cyclictest0-21swapper/010:34:590
1216470992220,1cyclictest0-21swapper/009:28:280
1216470992220,1cyclictest0-21swapper/009:00:160
1216470992220,1cyclictest0-21swapper/008:42:310
1216470992220,1cyclictest0-21swapper/007:55:060
1216470992214,6cyclictest1252027-21awk08:45:150
139099210,3rtkit-daemon1389-21rtkit-daemon12:29:062
12775492210,3sleep21277547-21sh10:00:002
12396222210,3chrt0-21swapper/108:10:161
1216479992119,1cyclictest3056-21PK-Backend08:08:323
121647999210,1cyclictest3056-21PK-Backend09:56:413
121647999210,19cyclictest3056-21PK-Backend12:34:373
121647999210,19cyclictest3056-21PK-Backend12:12:063
121647999210,19cyclictest3056-21PK-Backend12:09:363
121647999210,19cyclictest3056-21PK-Backend11:35:433
1216476992119,1cyclictest0-21swapper/211:54:472
1216476992119,1cyclictest0-21swapper/211:34:172
1216476992119,1cyclictest0-21swapper/211:09:102
1216476992119,1cyclictest0-21swapper/209:48:252
1216476992119,1cyclictest0-21swapper/209:17:362
1216476992119,1cyclictest0-21swapper/209:14:062
1216476992119,1cyclictest0-21swapper/208:38:302
1216476992119,1cyclictest0-21swapper/208:28:012
1216476992118,2cyclictest3056-21PK-Backend09:30:482
1216476992117,3cyclictest1895000-21snapd07:55:512
121647699210,20cyclictest0-21swapper/212:31:362
121647699210,20cyclictest0-21swapper/212:07:302
121647699210,20cyclictest0-21swapper/210:57:342
121647699210,20cyclictest0-21swapper/207:35:362
1216474992119,1cyclictest3056-21PK-Backend11:12:581
1216474992119,1cyclictest0-21swapper/111:58:411
1216474992119,1cyclictest0-21swapper/111:27:451
1216474992119,1cyclictest0-21swapper/111:02:581
1216474992119,1cyclictest0-21swapper/110:57:581
1216474992119,1cyclictest0-21swapper/110:37:241
1216474992119,1cyclictest0-21swapper/110:00:391
1216474992119,1cyclictest0-21swapper/109:45:261
1216474992119,1cyclictest0-21swapper/109:10:161
1216474992119,1cyclictest0-21swapper/108:43:531
1216474992119,1cyclictest0-21swapper/108:35:151
1216474992119,1cyclictest0-21swapper/108:15:281
1216474992119,1cyclictest0-21swapper/108:15:281
1216474992119,1cyclictest0-21swapper/107:44:591
1216474992119,1cyclictest0-21swapper/107:13:561
1216474992114,5cyclictest1330372-21systemctl12:30:001
121647499210,20cyclictest0-21swapper/112:16:321
121647499210,20cyclictest0-21swapper/109:31:321
121647499210,20cyclictest0-21swapper/107:24:041
121647099214,9cyclictest13-21ksoftirqd/008:15:150
121647099214,9cyclictest13-21ksoftirqd/008:15:150
1216470992119,1cyclictest0-21swapper/012:26:520
1216470992119,1cyclictest0-21swapper/011:43:200
1216470992119,1cyclictest0-21swapper/010:02:550
1216470992119,1cyclictest0-21swapper/009:34:160
1216470992119,1cyclictest0-21swapper/008:51:300
1216470992119,1cyclictest0-21swapper/008:35:480
1216470992119,1cyclictest0-21swapper/008:27:440
1216470992119,1cyclictest0-21swapper/008:13:590
1216470992119,1cyclictest0-21swapper/007:14:100
1216470992113,6cyclictest1264136-21expr09:20:110
1216470992112,2cyclictest141rcu_preempt11:31:130
121647099210,6cyclictest1235425-21apt08:00:110
121647099210,20cyclictest3056-21PK-Backend11:26:570
121647099210,20cyclictest161rcuc/011:20:010
121647099210,20cyclictest0-21swapper/011:13:250
121647099210,20cyclictest0-21swapper/010:57:230
121647099210,20cyclictest0-21swapper/010:40:470
121647099210,20cyclictest0-21swapper/010:36:550
121647099210,20cyclictest0-21swapper/009:47:470
121647099210,20cyclictest0-21swapper/008:22:370
121647099210,20cyclictest0-21swapper/008:08:520
121647099210,20cyclictest0-21swapper/007:20:590
12664362200,4sleep23056-21PK-Backend09:25:202
1216479992018,1cyclictest3056-21PK-Backend12:00:583
1216479992018,1cyclictest3056-21PK-Backend11:05:013
1216479992018,1cyclictest3056-21PK-Backend10:24:403
1216479992018,1cyclictest3056-21PK-Backend08:26:363
1216479992017,2cyclictest3056-21PK-Backend11:45:103
121647999200,19cyclictest3056-21PK-Backend08:23:393
121647999200,18cyclictest3056-21PK-Backend10:26:013
1216476992018,1cyclictest0-21swapper/212:35:282
1216476992018,1cyclictest0-21swapper/210:51:122
1216476992018,1cyclictest0-21swapper/210:45:242
1216476992018,1cyclictest0-21swapper/210:22:532
1216476992018,1cyclictest0-21swapper/210:06:312
1216476992018,1cyclictest0-21swapper/208:11:532
1216476992017,2cyclictest0-21swapper/210:10:362
121647699200,1cyclictest0-21swapper/210:29:422
121647699200,1cyclictest0-21swapper/210:02:252
121647699200,19cyclictest0-21swapper/212:18:522
121647699200,19cyclictest0-21swapper/209:25:002
121647699200,19cyclictest0-21swapper/207:27:522
121647699200,19cyclictest0-21swapper/207:20:232
121647699200,18cyclictest1319979-21cat12:00:002
1216474992018,1cyclictest0-21swapper/110:42:241
1216474992018,1cyclictest0-21swapper/109:35:581
1216474992018,1cyclictest0-21swapper/108:21:441
1216474992018,1cyclictest0-21swapper/107:58:521
1216474992018,1cyclictest0-21swapper/107:46:351
1216474992015,3cyclictest1266359-21grep09:25:181
1216474992013,5cyclictest1331541-21cat12:30:201
1216474992012,6cyclictest1224388-21apt07:30:121
1216474992011,7cyclictest1289289-21users10:30:211
121647499200,1cyclictest0-21swapper/110:15:541
121647499200,19cyclictest0-21swapper/111:51:451
121647499200,19cyclictest0-21swapper/111:46:451
121647499200,18cyclictest0-21swapper/110:14:321
121647099209,4cyclictest13-21ksoftirqd/010:19:340
121647099209,10cyclictest1258745-21apt09:05:120
1216470992018,1cyclictest0-21swapper/012:03:000
1216470992018,1cyclictest0-21swapper/011:54:070
1216470992018,1cyclictest0-21swapper/011:36:230
1216470992018,1cyclictest0-21swapper/011:20:350
1216470992018,1cyclictest0-21swapper/011:01:150
1216470992018,1cyclictest0-21swapper/010:52:580
1216470992018,1cyclictest0-21swapper/010:12:140
1216470992018,1cyclictest0-21swapper/009:51:050
1216470992018,1cyclictest0-21swapper/009:43:550
1216470992012,6cyclictest1332733-21awk12:35:150
1216470992012,6cyclictest1269465-21expr09:35:130
121647099200,1cyclictest0-21swapper/011:07:370
121647099200,19cyclictest0-21swapper/012:33:490
121647099200,19cyclictest0-21swapper/012:05:300
121647099200,19cyclictest0-21swapper/010:28:230
121647099200,19cyclictest0-21swapper/007:16:390
121647999194,13cyclictest1282752-21dump-pmu-power10:14:593
1216479991917,1cyclictest3056-21PK-Backend11:56:323
1216479991917,1cyclictest3056-21PK-Backend10:49:123
1216479991917,1cyclictest3056-21PK-Backend07:30:463
1216479991917,1cyclictest0-21swapper/311:20:303
1216479991916,2cyclictest3056-21PK-Backend07:35:203
121647999190,1cyclictest3056-21PK-Backend12:36:333
121647999190,1cyclictest3056-21PK-Backend10:33:583
121647999190,1cyclictest3056-21PK-Backend09:43:423
121647999190,1cyclictest3056-21PK-Backend09:27:413
121647999190,1cyclictest3056-21PK-Backend09:00:173
121647999190,1cyclictest3056-21PK-Backend08:58:543
121647999190,1cyclictest3056-21PK-Backend08:50:563
121647999190,1cyclictest3056-21PK-Backend08:44:553
121647999190,1cyclictest3056-21PK-Backend08:13:393
121647999190,1cyclictest3056-21PK-Backend07:29:113
121647999190,1cyclictest0-21swapper/311:26:513
121647999190,1cyclictest0-21swapper/307:10:403
121647999190,17cyclictest1327134-21apt12:20:113
1216476991917,1cyclictest3056-21PK-Backend07:45:102
1216476991917,1cyclictest0-21swapper/210:16:032
1216476991917,1cyclictest0-21swapper/208:51:272
1216476991916,2cyclictest0-21swapper/209:53:052
1216476991916,2cyclictest0-21swapper/209:43:122
1216476991912,5cyclictest1292409-21awk10:40:182
1216476991911,6cyclictest1321254-21timerwakeupswit12:00:212
121647699190,1cyclictest0-21swapper/212:23:182
121647699190,1cyclictest0-21swapper/208:58:502
121647699190,1cyclictest0-21swapper/208:47:222
121647699190,1cyclictest0-21swapper/208:33:562
121647699190,17cyclictest1306745-21threads11:20:202
121647499196,11cyclictest1227931-21unin-run07:40:001
1216474991917,1cyclictest3056-21PK-Backend11:21:031
1216474991917,1cyclictest0-21swapper/110:29:051
1216474991917,1cyclictest0-21swapper/108:51:521
1216474991917,1cyclictest0-21swapper/108:46:041
1216474991914,3cyclictest1271557-21strings09:40:181
1216474991911,6cyclictest1293980-21cat10:45:161
121647499190,1cyclictest0-21swapper/112:37:291
121647499190,1cyclictest0-21swapper/111:07:591
121647499190,1cyclictest0-21swapper/108:55:441
121647499190,1cyclictest0-21swapper/108:34:271
121647499190,1cyclictest0-21swapper/108:25:151
121647499190,18cyclictest0-21swapper/109:20:591
1216470991917,1cyclictest0-21swapper/007:40:470
1216470991917,1cyclictest0-21swapper/007:37:360
1216470991917,1cyclictest0-21swapper/007:32:150
1216470991916,2cyclictest161rcuc/012:22:010
1216470991912,6cyclictest1256240-21sed08:55:230
121647099190,18cyclictest0-21swapper/011:45:150
1216479991816,1cyclictest3056-21PK-Backend12:17:073
1216479991816,1cyclictest3056-21PK-Backend11:54:363
1216479991816,1cyclictest3056-21PK-Backend10:40:203
1216479991816,1cyclictest3056-21PK-Backend10:38:243
1216479991816,1cyclictest3056-21PK-Backend10:03:173
1216479991816,1cyclictest3056-21PK-Backend08:39:013
1216479991815,2cyclictest3056-21PK-Backend11:03:053
1216479991815,1cyclictest3056-21PK-Backend09:37:203
1216479991812,5cyclictest1261547-21awk09:10:223
1216479991811,5cyclictest1229617-21cron07:45:023
121647999180,1cyclictest0-21swapper/311:13:533
1216476991817,1cyclictest0-21swapper/207:40:232
1216476991816,1cyclictest0-21swapper/209:39:552
1216476991811,6cyclictest883-21dbus-daemon08:16:372
1216476991811,6cyclictest883-21dbus-daemon08:16:372
1216474991816,1cyclictest0-21swapper/110:52:231
1216474991813,3cyclictest1258086-21users09:00:231
1216474991811,5cyclictest26-21ksoftirqd/111:15:121
1216474991811,5cyclictest1325349-21sed12:15:001
1216474991811,5cyclictest1280155-21cat10:05:171
1216474991811,5cyclictest1237859-21ntpq08:05:161
1216474991811,5cyclictest1232372-21ntpq07:50:181
121647499180,17cyclictest3056-21PK-Backend11:32:321
121647099185,4cyclictest1277392-21chrt09:58:110
1216470991816,1cyclictest0-21swapper/012:12:270
1216470991816,1cyclictest0-21swapper/010:46:360
1216470991812,5cyclictest13-21ksoftirqd/007:50:010
1216470991811,5cyclictest1232535-21cat07:50:200
1216470991810,6cyclictest1246096-21apt08:30:120
121647999177,8cyclictest3056-21PK-Backend09:08:053
1216479991715,1cyclictest3056-21PK-Backend10:58:393
1216479991715,1cyclictest3056-21PK-Backend09:50:523
1216479991714,2cyclictest3056-21PK-Backend09:16:383
1216476991715,1cyclictest0-21swapper/211:48:242
1216476991715,1cyclictest0-21swapper/210:35:442
1216476991711,5cyclictest1235615-21cut08:00:122
121647499172,13cyclictest1219034-21cpuspeed_turbos07:15:111
121647499172,11cyclictest1320194-21apt12:00:121
1216474991715,1cyclictest0-21swapper/110:22:291
1216474991715,1cyclictest0-21swapper/109:52:211
1216474991712,3cyclictest1263279-21uptime09:15:221
1216474991711,5cyclictest1259109-21df09:05:121
1216474991710,5cyclictest1276565-21runrttasks09:55:171
121647499170,16cyclictest0-21swapper/108:01:161
1216479991614,1cyclictest3056-21PK-Backend09:22:403
1216479991614,1cyclictest0-21swapper/311:30:433
1216479991613,2cyclictest3056-21PK-Backend08:02:093
1216479991612,3cyclictest1232422-21which07:50:193
121647699169,5cyclictest1323850-21tr12:10:122
1216476991614,1cyclictest0-21swapper/207:12:582
1216476991612,3cyclictest1231606-21/usr/sbin/munin07:50:162
1216474991610,4cyclictest1322072-21grep12:05:121
121647999159,4cyclictest1242217-21grep08:20:013
121647999159,4cyclictest1242217-21grep08:20:003
1216479991513,1cyclictest3056-21PK-Backend09:33:293
1216479991513,1cyclictest3056-21PK-Backend08:31:443
121647699159,4cyclictest978-21snmpd09:00:182
121647699149,3cyclictest1095-21runrttasks08:05:102
121647699147,5cyclictest1304718-21tr11:15:162
121647999139,3cyclictest3056-21PK-Backend07:56:353
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional