You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-04 - 19:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Sat Apr 04, 2026 12:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
114014821000,3sleep10-21swapper/108:20:131
12142222980,4sleep03056-21PK-Backend11:50:160
12142222980,4sleep03056-21PK-Backend11:50:160
11790202780,8sleep234-21ksoftirqd/210:10:132
111445727839,14sleep20-21swapper/207:09:182
111442827840,12sleep30-21swapper/307:08:543
111447026444,14sleep00-21swapper/007:09:280
111433126242,14sleep10-21swapper/107:07:401
11284282410,21sleep234-21ksoftirqd/207:45:222
1114665993819,18cyclictest13-21ksoftirqd/008:20:280
139099370,3rtkit-daemon0-21swapper/107:55:311
111467599290,28cyclictest1389-21rtkit-daemon07:42:143
11965102280,4chrt3056-21PK-Backend11:00:131
1114665992713,5cyclictest13-21ksoftirqd/007:41:120
139099260,4rtkit-daemon1389-21rtkit-daemon09:37:240
1114665992611,4cyclictest13-21ksoftirqd/007:46:240
12197792250,3chrt0-21swapper/012:05:200
1114675992514,8cyclictest1174313-21/usr/sbin/munin09:55:223
1114675992416,7cyclictest1147510-21diskstats08:40:153
1114675992416,6cyclictest436-20systemd-journal07:30:003
111466599249,4cyclictest13-21ksoftirqd/007:57:160
111466599248,5cyclictest13-21ksoftirqd/010:01:420
1114665992410,5cyclictest13-21ksoftirqd/009:15:100
1114665992410,4cyclictest13-21ksoftirqd/008:47:550
11872242230,3chrt1187223-21apt-config10:35:010
111467599234,16cyclictest1180068-21perf10:14:593
1114665992310,5cyclictest13-21ksoftirqd/009:07:450
139099220,4rtkit-daemon1389-21rtkit-daemon10:15:410
11226172220,3sleep10-21swapper/107:30:131
111467599224,2cyclictest141rcu_preempt09:05:013
1114675992214,6cyclictest1132452-21cron08:00:003
1114675992210,8cyclictest883-21dbus-daemon10:05:013
111467599220,2cyclictest1228354-21strings12:30:173
111467599220,20cyclictest3056-21PK-Backend07:18:433
1114671992218,3cyclictest1137328-21smartctl08:10:212
1114671992214,6cyclictest1191392-21fw_packets10:45:142
111466599227,4cyclictest13-21ksoftirqd/011:48:160
1114665992210,4cyclictest13-21ksoftirqd/008:42:310
139099210,15rtkit-daemon3056-21PK-Backend09:23:441
139099210,15rtkit-daemon3056-21PK-Backend08:12:230
11750052210,3sleep0161rcuc/010:00:010
1114675992115,4cyclictest1195935-21dump-pmu-power11:00:003
1114675992114,5cyclictest883-21dbus-daemon10:30:013
1114675992113,6cyclictest436-20systemd-journal08:40:013
1114675992113,6cyclictest1227177-21sshd12:27:153
1114675992111,8cyclictest42-21ksoftirqd/311:25:193
1114675992110,10cyclictest978-21snmpd07:47:423
111467599210,20cyclictest3056-21PK-Backend08:30:013
111467599210,19cyclictest3056-21PK-Backend08:52:313
1114671992119,1cyclictest0-21swapper/210:24:522
1114671992118,2cyclictest0-21swapper/207:50:202
1114671992115,5cyclictest34-21ksoftirqd/209:50:002
1114671992113,6cyclictest1173752-21expr09:55:132
111467199210,20cyclictest0-21swapper/211:38:342
1114666992119,1cyclictest0-21swapper/110:25:151
1114666992114,5cyclictest883-21dbus-daemon11:35:011
1114666992114,5cyclictest1171140-21cron09:50:001
1114666992113,6cyclictest1208609-21apt11:35:111
111466699210,20cyclictest3056-21PK-Backend12:27:581
111466699210,20cyclictest3056-21PK-Backend07:49:421
111466699210,1cyclictest3056-21PK-Backend12:13:381
111466699210,19cyclictest3056-21PK-Backend11:58:581
111466699210,19cyclictest3056-21PK-Backend08:11:051
111466599219,5cyclictest13-21ksoftirqd/009:26:220
111466599219,4cyclictest13-21ksoftirqd/007:51:140
111466599214,3cyclictest141rcu_preempt09:51:160
1114665992119,1cyclictest0-21swapper/011:36:400
1114665992119,1cyclictest0-21swapper/007:12:380
1114665992111,8cyclictest436-20systemd-journal07:35:000
111466599210,20cyclictest0-21swapper/009:43:050
139099200,4rtkit-daemon1389-21rtkit-daemon10:07:330
11241032200,3sleep33056-21PK-Backend07:35:123
1114675992018,1cyclictest3056-21PK-Backend12:18:143
111467599201,4cyclictest1165342-21python309:30:203
1114675992013,5cyclictest1215539-21apt11:55:113
1114675992013,5cyclictest1212960-21cat11:45:213
1114675992012,6cyclictest1168125-21cut09:40:133
1114675992012,6cyclictest1134153-21sshd08:01:173
1114675992012,6cyclictest1134153-21sshd08:01:173
1114675992012,5cyclictest1197141-21strings11:00:173
1114675992012,5cyclictest1191863-21python310:45:203
1114675992011,6cyclictest436-20systemd-journal11:55:003
1114675992011,6cyclictest436-20systemd-journal11:55:003
111467599200,1cyclictest3056-21PK-Backend11:21:373
111467599200,19cyclictest0-21swapper/311:40:553
111467599200,19cyclictest0-21swapper/311:07:363
111467599200,19cyclictest0-21swapper/310:43:573
111467599200,19cyclictest0-21swapper/310:09:173
111467599200,19cyclictest0-21swapper/309:27:543
111467599200,19cyclictest0-21swapper/309:16:103
111467599200,19cyclictest0-21swapper/308:45:353
111467599200,19cyclictest0-21swapper/308:08:243
111467599200,18cyclictest1157916-21diskstats09:10:143
1114671992018,1cyclictest0-21swapper/211:05:292
1114671992018,1cyclictest0-21swapper/210:52:312
1114671992018,1cyclictest0-21swapper/210:52:312
1114671992018,1cyclictest0-21swapper/210:29:532
1114671992018,1cyclictest0-21swapper/207:26:412
1114671992014,5cyclictest1157276-21ls09:10:002
1114671992013,6cyclictest1177633-21awk10:05:202
111467199200,1cyclictest0-21swapper/211:14:062
111467199200,1cyclictest0-21swapper/209:02:322
111467199200,1cyclictest0-21swapper/208:18:192
111467199200,19cyclictest0-21swapper/212:17:062
111467199200,19cyclictest0-21swapper/212:00:452
111467199200,19cyclictest0-21swapper/211:31:582
111467199200,19cyclictest0-21swapper/210:36:222
111467199200,19cyclictest0-21swapper/210:17:432
111467199200,19cyclictest0-21swapper/210:01:412
111467199200,19cyclictest0-21swapper/209:44:442
111467199200,19cyclictest0-21swapper/209:12:062
111467199200,19cyclictest0-21swapper/208:40:572
111467199200,19cyclictest0-21swapper/208:35:502
111467199200,19cyclictest0-21swapper/208:29:202
111467199200,19cyclictest0-21swapper/207:40:062
1114666992018,1cyclictest0-21swapper/110:06:361
1114666992018,1cyclictest0-21swapper/109:30:201
1114666992018,1cyclictest0-21swapper/107:26:371
1114666992014,5cyclictest26-21ksoftirqd/111:40:161
111466699200,1cyclictest3056-21PK-Backend09:09:171
111466699200,19cyclictest3056-21PK-Backend09:13:171
111466699200,19cyclictest0-21swapper/108:37:331
111466699200,19cyclictest0-21swapper/108:30:181
111466699200,19cyclictest0-21swapper/107:13:041
111466699200,18cyclictest3056-21PK-Backend12:07:501
111466699200,18cyclictest3056-21PK-Backend10:50:501
111466699200,18cyclictest3056-21PK-Backend10:50:501
111466599209,4cyclictest13-21ksoftirqd/009:49:300
111466599209,4cyclictest13-21ksoftirqd/009:13:460
111466599208,4cyclictest13-21ksoftirqd/011:14:440
111466599208,4cyclictest13-21ksoftirqd/010:21:550
111466599207,4cyclictest13-21ksoftirqd/012:25:290
1114665992018,1cyclictest3056-21PK-Backend08:30:520
1114665992018,1cyclictest0-21swapper/011:20:050
1114665992013,5cyclictest1222212-21perf12:15:000
111466599200,19cyclictest0-21swapper/011:29:100
139099190,13rtkit-daemon0-21swapper/109:04:521
111467599192,13cyclictest1171636-21runrttasks09:50:033
1114675991917,1cyclictest3056-21PK-Backend08:32:103
1114675991917,1cyclictest3056-21PK-Backend07:33:303
1114675991917,1cyclictest0-21swapper/312:01:513
1114675991917,1cyclictest0-21swapper/309:48:143
1114675991916,2cyclictest3056-21PK-Backend07:12:343
1114675991914,3cyclictest1224207-21apt12:20:113
1114675991912,5cyclictest42-21ksoftirqd/310:35:193
111467599191,15cyclictest1199911-21/usr/sbin/munin11:10:113
1114675991910,7cyclictest436-20systemd-journal09:10:003
1114675991910,6cyclictest1193382-21grep10:50:183
1114675991910,6cyclictest1193382-21grep10:50:183
111467599190,6cyclictest1136655-21cstates08:10:133
111467599190,1cyclictest3056-21PK-Backend07:22:403
111467599190,1cyclictest0-21swapper/312:10:583
111467599190,1cyclictest0-21swapper/310:19:383
111467599190,18cyclictest0-21swapper/312:07:253
111467599190,18cyclictest0-21swapper/311:36:093
1114671991917,1cyclictest0-21swapper/212:24:242
1114671991917,1cyclictest0-21swapper/211:26:512
1114671991917,1cyclictest0-21swapper/211:03:122
1114671991917,1cyclictest0-21swapper/209:30:582
1114671991917,1cyclictest0-21swapper/209:24:372
1114671991917,1cyclictest0-21swapper/208:30:082
1114671991917,1cyclictest0-21swapper/208:05:352
1114671991916,2cyclictest0-21swapper/212:05:162
1114671991912,5cyclictest1173292-21wc09:55:002
1114671991910,7cyclictest1214884-21users11:50:222
1114671991910,7cyclictest1214884-21users11:50:212
111467199190,7cyclictest11226182cstates07:30:132
111467199190,1cyclictest0-21swapper/212:27:552
111467199190,1cyclictest0-21swapper/212:13:212
111467199190,1cyclictest0-21swapper/211:59:292
111467199190,1cyclictest0-21swapper/211:43:202
111467199190,1cyclictest0-21swapper/208:46:172
111467199190,1cyclictest0-21swapper/207:38:172
1114666991917,1cyclictest3056-21PK-Backend12:22:091
1114666991917,1cyclictest3056-21PK-Backend12:00:131
1114666991917,1cyclictest3056-21PK-Backend09:15:401
1114666991917,1cyclictest0-21swapper/111:28:491
1114666991917,1cyclictest0-21swapper/107:15:281
1114666991916,2cyclictest3056-21PK-Backend11:45:331
111466699190,1cyclictest3056-21PK-Backend10:42:261
111466699190,1cyclictest3056-21PK-Backend10:04:471
111466699190,1cyclictest3056-21PK-Backend09:27:571
111466699190,1cyclictest0-21swapper/112:31:161
111466699190,1cyclictest0-21swapper/111:05:381
111466699190,1cyclictest0-21swapper/108:45:051
111466699190,1cyclictest0-21swapper/108:27:201
111466699190,1cyclictest0-21swapper/107:36:501
111466699190,18cyclictest3056-21PK-Backend08:07:531
111466699190,18cyclictest3056-21PK-Backend08:00:571
111466699190,18cyclictest3056-21PK-Backend08:00:571
111466699190,18cyclictest3056-21PK-Backend07:41:161
111466699190,18cyclictest0-21swapper/112:36:501
111466699190,18cyclictest0-21swapper/108:51:131
111466599199,7cyclictest1121994-21cron07:30:000
111466599198,4cyclictest13-21ksoftirqd/010:35:420
111466599198,4cyclictest13-21ksoftirqd/009:21:320
111466599198,4cyclictest13-21ksoftirqd/008:04:170
111466599198,4cyclictest13-21ksoftirqd/008:04:170
111466599198,3cyclictest13-21ksoftirqd/008:39:340
111466599197,4cyclictest13-21ksoftirqd/012:20:440
111466599197,10cyclictest1125566-21unin-run07:40:010
111466599196,4cyclictest13-21ksoftirqd/010:11:120
111466599195,4cyclictest13-21ksoftirqd/010:55:220
1114665991916,2cyclictest0-21swapper/011:04:290
1114665991915,3cyclictest13-21ksoftirqd/008:55:000
1114665991913,5cyclictest13-21ksoftirqd/010:50:320
1114665991913,5cyclictest13-21ksoftirqd/010:50:320
1114665991913,5cyclictest1210428-21ls11:40:120
1114665991912,5cyclictest1229868-21cat12:35:150
1114665991912,5cyclictest1198921-21tr11:05:180
1114665991911,6cyclictest1222869-21which12:15:160
1114665991911,6cyclictest1138370-21tr08:15:120
111466599190,18cyclictest0-21swapper/011:19:030
11731922180,4chrt3056-21PK-Backend09:55:011
11347872180,8chrt13-21ksoftirqd/008:05:130
1114675991816,1cyclictest0-21swapper/307:52:553
1114675991812,4cyclictest1182535-21cat10:20:163
1114675991812,4cyclictest11393112sleep308:15:453
1114675991811,5cyclictest1166688-21awk09:35:173
1114675991811,4cyclictest1187044-21apt-key10:35:013
1114675991810,6cyclictest1202059-21awk11:15:153
1114675991810,6cyclictest1161747-21ntp_states09:20:163
111467199186,2cyclictest141rcu_preempt11:20:162
1114671991816,1cyclictest0-21swapper/211:48:062
1114671991816,1cyclictest0-21swapper/210:30:542
1114671991816,1cyclictest0-21swapper/209:17:402
1114671991816,1cyclictest0-21swapper/208:53:132
1114671991813,4cyclictest34-21ksoftirqd/212:30:162
1114671991811,5cyclictest1133845-21rt-features08:00:232
1114671991811,5cyclictest1133845-21rt-features08:00:222
111467199180,17cyclictest0-21swapper/207:15:322
1114666991816,1cyclictest3056-21PK-Backend10:36:111
1114666991816,1cyclictest3056-21PK-Backend10:14:071
1114666991816,1cyclictest3056-21PK-Backend09:43:041
1114666991816,1cyclictest3056-21PK-Backend08:41:181
1114666991816,1cyclictest0-21swapper/112:17:221
1114666991810,6cyclictest1138584-21fw_packets08:15:161
111466599187,4cyclictest13-21ksoftirqd/012:30:060
111466599187,4cyclictest13-21ksoftirqd/012:01:190
111466599187,4cyclictest13-21ksoftirqd/010:41:040
111466599187,4cyclictest13-21ksoftirqd/009:34:520
111466599186,8cyclictest883-21dbus-daemon07:20:000
1114665991816,1cyclictest0-21swapper/011:56:130
1114665991816,1cyclictest0-21swapper/010:45:440
1114665991813,3cyclictest1141878-21cstates08:25:120
111466599181,15cyclictest1119552-21ntp_states07:20:200
111466599181,12cyclictest1206874-21apt11:30:100
1114665991810,6cyclictest1152326-21find08:55:110
139099170,14rtkit-daemon1389-21rtkit-daemon11:17:441
111467599170,15cyclictest11403092sleep308:20:153
111467199179,6cyclictest1132470-21apt-get08:00:002
111467199176,9cyclictest1164241-21systemd09:30:002
1114671991715,1cyclictest0-21swapper/207:20:192
1114671991712,4cyclictest978-21snmpd12:38:482
1114671991711,5cyclictest1195085-21sed10:55:192
1114671991711,5cyclictest1153200-21sensors08:55:192
1114671991710,6cyclictest1167895-21sed09:40:012
1114671991710,5cyclictest1202626-21ls11:15:202
1114666991715,1cyclictest3056-21PK-Backend11:50:051
1114666991715,1cyclictest3056-21PK-Backend11:50:051
1114666991715,1cyclictest3056-21PK-Backend10:48:481
1114666991715,1cyclictest3056-21PK-Backend10:32:311
1114666991715,1cyclictest3056-21PK-Backend10:18:471
1114666991715,1cyclictest0-21swapper/109:56:431
1114666991712,3cyclictest1095-21runrttasks07:20:131
1114665991714,2cyclictest0-21swapper/009:03:440
111467199161,6cyclictest1116327-21munin-run07:15:002
1114671991614,1cyclictest0-21swapper/210:41:022
111467199161,3cyclictest1141409-21munin-run08:25:002
1114666991614,1cyclictest0-21swapper/108:55:121
1114666991611,3cyclictest1204154-21sh11:20:171
111466599165,4cyclictest13-21ksoftirqd/010:27:410
1114675991513,1cyclictest3056-21PK-Backend08:55:433
111466699159,5cyclictest1129326-21cpuspeed07:50:131
1114666991513,1cyclictest0-21swapper/109:35:061
1114666991510,4cyclictest3056-21PK-Backend11:13:001
111467599143,2cyclictest141rcu_preempt12:35:203
1114675991412,1cyclictest0-21swapper/311:31:023
1114666991412,1cyclictest3056-21PK-Backend10:21:371
111466699130,2cyclictest3056-21PK-Backend11:00:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional