You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-25 - 05:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu Jun 25, 2026 00:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
115642522370,7sleep11156429-21ntp_kernel_err19:05:201
126332621090,3sleep20-21swapper/200:05:162
126332621090,3sleep20-21swapper/200:05:162
118231921060,3sleep13056-21PK-Backend20:15:221
12329812970,2sleep30-21swapper/322:40:123
115706528037,37sleep30-21swapper/319:08:453
115694927540,30sleep20-21swapper/219:07:152
115656826944,16sleep00-21swapper/019:05:250
12148562400,3sleep0161rcuc/021:50:000
139099290,3rtkit-daemon1389-21rtkit-daemon00:28:460
12476162290,3sleep342-21ksoftirqd/323:20:423
139099280,4rtkit-daemon1389-21rtkit-daemon22:35:311
11873732280,3sleep0161rcuc/020:30:220
115730599289,10cyclictest13-21ksoftirqd/021:45:000
12324322270,3sleep013-21ksoftirqd/022:40:010
139099260,4rtkit-daemon1389-21rtkit-daemon20:29:320
139099260,3rtkit-daemon1389-21rtkit-daemon20:10:570
115730599268,9cyclictest13-21ksoftirqd/021:35:010
139099250,4rtkit-daemon1389-21rtkit-daemon23:21:320
11898282250,4sleep13056-21PK-Backend20:40:001
115730599259,4cyclictest13-21ksoftirqd/021:14:590
115730599257,9cyclictest13-21ksoftirqd/020:25:000
115730599256,8cyclictest13-21ksoftirqd/021:55:010
115730599252,6cyclictest13-21ksoftirqd/019:55:190
1157311992419,3cyclictest1242363-21users23:05:222
115731199240,21cyclictest1215014-21apt21:50:112
115730599245,4cyclictest1245619-21python323:15:200
1157305992421,2cyclictest161rcuc/023:09:190
139099230,3rtkit-daemon1389-21rtkit-daemon20:50:040
12455022230,4sleep13056-21PK-Backend23:15:191
12299562230,4sleep30-21swapper/322:30:223
12273092230,4sleep30-21swapper/322:25:133
1157315992314,7cyclictest1257570-21df23:50:123
115731199230,22cyclictest1095-21runrttasks22:52:162
1157309992318,3cyclictest1265959-21latency_hist00:15:001
115730599234,7cyclictest13-21ksoftirqd/020:15:190
115730599234,4cyclictest1250832-21python323:30:200
115730599233,7cyclictest13-21ksoftirqd/021:55:140
115730599233,7cyclictest1226277-21python322:20:200
115730599233,6cyclictest13-21ksoftirqd/019:50:000
115730599232,7cyclictest1159507-21diskstats19:15:010
115730599232,6cyclictest13-21ksoftirqd/021:40:000
1157315992220,1cyclictest0-21swapper/323:45:173
1157315992220,1cyclictest0-21swapper/319:33:033
1157311992220,1cyclictest0-21swapper/221:27:332
1157311992220,1cyclictest0-21swapper/220:42:322
1157311992220,1cyclictest0-21swapper/220:35:412
1157311992220,1cyclictest0-21swapper/219:45:542
1157311992220,1cyclictest0-21swapper/200:12:192
115731199220,21cyclictest0-21swapper/200:15:592
1157309992220,1cyclictest3056-21PK-Backend22:41:121
1157309992220,1cyclictest3056-21PK-Backend22:17:331
1157309992220,1cyclictest0-21swapper/100:06:351
1157309992220,1cyclictest0-21swapper/100:06:351
115730599228,7cyclictest13-21ksoftirqd/021:20:100
115730599227,7cyclictest13-21ksoftirqd/000:25:000
115730599224,16cyclictest1267622-21turbostat00:20:000
115730599223,6cyclictest13-21ksoftirqd/022:00:130
115730599222,8cyclictest1235453-21sshd22:45:400
1157305992220,1cyclictest0-21swapper/021:29:020
1157305992220,1cyclictest0-21swapper/020:44:060
1157305992220,1cyclictest0-21swapper/019:25:580
1157305992219,2cyclictest161rcuc/022:26:530
115730599221,7cyclictest13-21ksoftirqd/019:35:240
115730599220,21cyclictest0-21swapper/019:23:350
12368712210,4chrt3056-21PK-Backend22:50:201
115731599215,7cyclictest141rcu_preempt22:35:383
1157315992119,1cyclictest3056-21PK-Backend20:54:213
1157315992119,1cyclictest0-21swapper/323:37:453
1157315992119,1cyclictest0-21swapper/322:07:563
1157315992119,1cyclictest0-21swapper/320:47:453
1157315992119,1cyclictest0-21swapper/320:42:053
1157315992119,1cyclictest0-21swapper/320:23:453
1157315992119,1cyclictest0-21swapper/319:26:473
1157315992118,2cyclictest0-21swapper/320:00:133
1157315992118,2cyclictest0-21swapper/319:45:073
1157315992113,6cyclictest1270679-21sed00:25:213
1157311992119,1cyclictest0-21swapper/223:23:332
1157311992119,1cyclictest0-21swapper/223:18:392
1157311992119,1cyclictest0-21swapper/223:11:082
1157311992119,1cyclictest0-21swapper/223:11:082
1157311992119,1cyclictest0-21swapper/222:56:212
1157311992119,1cyclictest0-21swapper/221:48:422
1157311992119,1cyclictest0-21swapper/220:05:202
1157311992119,1cyclictest0-21swapper/219:26:542
1157311992119,1cyclictest0-21swapper/219:17:562
1157311992119,1cyclictest0-21swapper/200:31:392
1157311992114,5cyclictest34-21ksoftirqd/222:35:112
115731199211,19cyclictest34-21ksoftirqd/220:22:582
1157311992110,9cyclictest1177598-21unin-run20:04:592
115731199210,20cyclictest0-21swapper/223:45:492
115731199210,20cyclictest0-21swapper/221:13:072
115731199210,20cyclictest0-21swapper/219:57:442
115731199210,20cyclictest0-21swapper/200:36:262
115731199210,20cyclictest0-21swapper/200:01:172
1157309992119,1cyclictest3056-21PK-Backend23:23:291
1157309992119,1cyclictest3056-21PK-Backend23:12:491
1157309992119,1cyclictest3056-21PK-Backend23:12:481
1157309992119,1cyclictest3056-21PK-Backend20:26:201
1157309992119,1cyclictest3056-21PK-Backend20:00:521
1157309992119,1cyclictest3056-21PK-Backend19:52:461
1157309992119,1cyclictest3056-21PK-Backend19:37:391
1157309992119,1cyclictest0-21swapper/100:20:211
1157309992118,2cyclictest3056-21PK-Backend23:57:571
1157309992118,2cyclictest3056-21PK-Backend22:31:451
1157309992118,2cyclictest3056-21PK-Backend19:22:531
115730999210,20cyclictest3056-21PK-Backend23:38:101
115730999210,20cyclictest3056-21PK-Backend21:48:541
115730999210,20cyclictest0-21swapper/120:51:551
115730999210,19cyclictest3056-21PK-Backend23:52:291
115730999210,19cyclictest3056-21PK-Backend22:22:201
115730999210,19cyclictest3056-21PK-Backend21:36:021
115730999210,19cyclictest3056-21PK-Backend19:30:371
115730599214,8cyclictest1271726-21diskstats00:30:130
115730599213,9cyclictest13-21ksoftirqd/022:35:110
115730599213,5cyclictest13-21ksoftirqd/021:20:000
115730599213,16cyclictest1198634-21sh21:05:010
115730599212,6cyclictest13-21ksoftirqd/022:10:210
115730599212,6cyclictest13-21ksoftirqd/022:10:210
115730599212,6cyclictest13-21ksoftirqd/019:40:130
115730599212,5cyclictest161rcuc/022:06:220
1157305992119,1cyclictest0-21swapper/023:44:140
1157305992119,1cyclictest0-21swapper/022:56:480
1157305992119,1cyclictest0-21swapper/022:50:200
1157305992119,1cyclictest0-21swapper/020:09:260
115730599210,8cyclictest1255864-21needreboot23:45:170
115730599210,20cyclictest13-21ksoftirqd/023:29:530
115730599210,20cyclictest0-21swapper/023:58:190
115730599210,20cyclictest0-21swapper/020:36:570
1157315992018,1cyclictest3056-21PK-Backend23:05:413
1157315992018,1cyclictest0-21swapper/322:15:463
1157315992018,1cyclictest0-21swapper/320:27:583
1157315992018,1cyclictest0-21swapper/319:57:023
1157315992018,1cyclictest0-21swapper/319:24:243
1157315992018,1cyclictest0-21swapper/319:12:493
115731599201,6cyclictest1261172-21irqstats00:00:173
115731599201,5cyclictest1158960-21apt-get19:15:043
1157315992015,3cyclictest978-21snmpd21:55:133
115731599201,17cyclictest1169396-21irqstats19:40:163
115731599200,1cyclictest3056-21PK-Backend23:10:083
115731599200,1cyclictest3056-21PK-Backend23:10:073
115731599200,1cyclictest0-21swapper/321:32:073
115731599200,1cyclictest0-21swapper/300:30:183
115731599200,19cyclictest3056-21PK-Backend21:27:343
115731599200,19cyclictest0-21swapper/323:55:233
115731599200,19cyclictest0-21swapper/323:16:173
115731599200,19cyclictest0-21swapper/322:01:553
115731599200,19cyclictest0-21swapper/320:32:103
1157311992018,1cyclictest0-21swapper/223:02:102
1157311992018,1cyclictest0-21swapper/222:12:292
1157311992018,1cyclictest0-21swapper/222:12:292
1157311992018,1cyclictest0-21swapper/221:37:072
1157311992018,1cyclictest0-21swapper/221:06:382
1157311992018,1cyclictest0-21swapper/220:46:582
1157311992018,1cyclictest0-21swapper/219:35:592
1157311992014,4cyclictest28650irq/123-ahci[0000:20:212
1157311992013,5cyclictest1208946-21python321:30:192
1157311992013,5cyclictest1162684-21ntp_kernel_err19:20:172
1157311992012,6cyclictest1234333-21basename22:45:102
115731199200,8cyclictest11823162cat20:15:232
115731199200,1cyclictest0-21swapper/220:12:502
115731199200,19cyclictest0-21swapper/223:53:342
115731199200,19cyclictest0-21swapper/223:43:332
115731199200,19cyclictest0-21swapper/222:34:042
115731199200,19cyclictest0-21swapper/222:07:412
115731199200,19cyclictest0-21swapper/221:00:292
1157309992018,1cyclictest3056-21PK-Backend22:10:021
1157309992018,1cyclictest3056-21PK-Backend22:10:021
1157309992018,1cyclictest3056-21PK-Backend21:32:441
1157309992018,1cyclictest3056-21PK-Backend20:08:211
1157309992018,1cyclictest3056-21PK-Backend00:38:051
1157309992018,1cyclictest3056-21PK-Backend00:26:171
1157309992018,1cyclictest0-21swapper/121:06:551
1157309992018,1cyclictest0-21swapper/100:31:371
1157309992017,2cyclictest3056-21PK-Backend22:05:491
115730999200,2cyclictest3056-21PK-Backend20:31:071
115730999200,1cyclictest3056-21PK-Backend23:29:171
115730999200,1cyclictest3056-21PK-Backend20:45:051
115730999200,1cyclictest3056-21PK-Backend20:13:281
115730999200,1cyclictest3056-21PK-Backend19:59:441
115730999200,1cyclictest0-21swapper/123:05:321
115730999200,1cyclictest0-21swapper/121:12:501
115730999200,19cyclictest3056-21PK-Backend22:03:261
115730999200,19cyclictest3056-21PK-Backend20:24:511
115730999200,19cyclictest3056-21PK-Backend19:46:241
115730999200,19cyclictest3056-21PK-Backend19:43:471
115730999200,19cyclictest0-21swapper/100:15:341
115730599209,2cyclictest141rcu_preempt19:10:120
115730599203,5cyclictest13-21ksoftirqd/022:15:130
115730599203,4cyclictest1230360-21munin-run22:35:000
115730599202,7cyclictest13-21ksoftirqd/019:50:120
1157305992018,1cyclictest0-21swapper/021:07:240
115730599201,6cyclictest1165674-21cstates19:30:130
115730599200,7cyclictest13-21ksoftirqd/023:35:170
115730599200,19cyclictest0-21swapper/023:53:260
115730599200,19cyclictest0-21swapper/023:11:220
115730599200,19cyclictest0-21swapper/023:11:210
115730599200,19cyclictest0-21swapper/020:45:340
115730599200,19cyclictest0-21swapper/000:10:020
115731599199,8cyclictest978-21snmpd23:30:043
115731599199,7cyclictest42-21ksoftirqd/322:10:113
115731599199,7cyclictest42-21ksoftirqd/322:10:113
1157315991917,1cyclictest0-21swapper/323:26:103
1157315991917,1cyclictest0-21swapper/322:21:483
1157315991916,2cyclictest0-21swapper/321:22:403
1157315991914,3cyclictest42-21ksoftirqd/319:35:183
1157315991913,4cyclictest42-21ksoftirqd/321:45:193
1157315991912,6cyclictest1204079-21apt-get21:19:593
1157315991912,5cyclictest1182495-21sshd20:16:423
1157315991911,6cyclictest42-21ksoftirqd/322:50:003
115731599190,7cyclictest1210892-21users21:35:233
115731599190,1cyclictest0-21swapper/320:39:423
115731599190,18cyclictest3056-21PK-Backend21:03:003
115731199199,9cyclictest2176-21JS10:27:482
1157311991917,1cyclictest0-21swapper/223:55:502
1157311991917,1cyclictest0-21swapper/222:40:132
1157311991917,1cyclictest0-21swapper/222:29:042
1157311991917,1cyclictest0-21swapper/222:17:142
1157311991917,1cyclictest0-21swapper/222:04:232
1157311991917,1cyclictest0-21swapper/221:43:222
1157311991917,1cyclictest0-21swapper/221:19:092
1157311991917,1cyclictest0-21swapper/220:56:572
1157311991917,1cyclictest0-21swapper/220:31:152
1157311991917,1cyclictest0-21swapper/219:53:242
1157311991917,1cyclictest0-21swapper/219:10:382
1157311991912,6cyclictest34-21ksoftirqd/221:55:122
1157311991912,5cyclictest1185482-21ntp_kernel_pll_20:25:162
115731199190,1cyclictest0-21swapper/219:31:542
115731199190,17cyclictest1269986-21df00:25:132
1157309991917,1cyclictest3056-21PK-Backend23:30:261
1157309991917,1cyclictest3056-21PK-Backend19:26:111
1157309991917,1cyclictest0-21swapper/123:00:511
1157309991917,1cyclictest0-21swapper/121:04:521
1157309991911,2cyclictest26-21ksoftirqd/119:14:591
115730999190,1cyclictest3056-21PK-Backend23:46:281
115730999190,1cyclictest3056-21PK-Backend22:29:421
115730999190,1cyclictest3056-21PK-Backend21:56:371
115730999190,1cyclictest3056-21PK-Backend21:50:221
115730999190,1cyclictest3056-21PK-Backend21:25:131
115730999190,1cyclictest3056-21PK-Backend21:19:131
1157305991917,1cyclictest0-21swapper/000:39:370
1157305991916,2cyclictest161rcuc/020:05:000
115730599190,7cyclictest13-21ksoftirqd/023:00:020
115730599190,18cyclictest0-21swapper/000:02:380
115731599189,7cyclictest1265889-21sh00:15:013
115731599186,4cyclictest141rcu_preempt23:03:413
1157315991816,1cyclictest3056-21PK-Backend21:09:163
1157315991816,1cyclictest3056-21PK-Backend00:06:263
1157315991816,1cyclictest3056-21PK-Backend00:06:253
1157315991816,1cyclictest0-21swapper/322:55:213
1157315991816,1cyclictest0-21swapper/321:52:553
1157315991812,5cyclictest1174136-21munin-run19:55:003
115731599180,16cyclictest1253771-21cstates23:40:133
1157311991816,1cyclictest0-21swapper/220:54:552
1157311991816,1cyclictest0-21swapper/219:40:402
1157311991812,4cyclictest34-21ksoftirqd/223:30:182
115731199180,1cyclictest0-21swapper/223:27:522
115730999189,7cyclictest26-21ksoftirqd/100:00:221
1157309991816,1cyclictest3056-21PK-Backend20:40:181
1157305991815,2cyclictest161rcuc/020:55:310
115731599179,6cyclictest42-21ksoftirqd/320:10:113
1157315991715,1cyclictest3056-21PK-Backend20:58:133
1157315991710,5cyclictest1237186-21sshd22:50:413
115731599170,5cyclictest1177953-21cpuspeed_turbos20:05:123
115731599170,4cyclictest1212146-21sed21:40:163
115731199179,6cyclictest1225414-21apt22:20:112
1157311991715,1cyclictest0-21swapper/221:24:152
1157309991715,1cyclictest3056-21PK-Backend23:40:261
1157309991714,2cyclictest3056-21PK-Backend21:24:201
1157309991713,3cyclictest1237852-21cpu22:55:111
1157309991710,5cyclictest26-21ksoftirqd/121:00:011
115730599171,12cyclictest1262157-21apt-get00:05:040
115730599171,12cyclictest1262157-21apt-get00:05:040
115731599168,6cyclictest42-21ksoftirqd/300:35:193
1157315991611,3cyclictest1268362-21irqstats00:20:153
1157309991612,2cyclictest26-21ksoftirqd/122:45:121
115730999160,2cyclictest141rcu_preempt19:15:021
115731599150,3cyclictest3056-21PK-Backend21:10:183
115730999159,5cyclictest26-21ksoftirqd/121:42:461
115731599139,2cyclictest42-21ksoftirqd/300:15:223
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional