You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-08 - 20:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Fri May 08, 2026 12:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
225502153130,16sleep20-21swapper/207:06:582
224642142106,26sleep30-21swapper/307:05:523
226832140104,25sleep10-21swapper/107:08:421
227172139102,26sleep00-21swapper/007:09:100
269372700,1sleep10-21swapper/109:21:421
13812690,3sleep31378-21ssh11:50:143
257602660,2sleep20-21swapper/211:20:142
89362650,1sleep20-21swapper/210:20:162
50932640,2sleep00-21swapper/011:29:000
102992620,1sleep20-21swapper/211:32:272
102992620,1sleep20-21swapper/211:32:262
89862590,2sleep00-21swapper/012:19:570
89862590,2sleep00-21swapper/012:19:560
207882570,1sleep30-21swapper/309:17:243
4172550,2sleep20-21swapper/211:49:402
232722540,1sleep10-21swapper/108:25:131
264442530,2sleep10-21swapper/110:56:291
114282530,2sleep20-21swapper/210:45:242
114282530,2sleep20-21swapper/210:45:232
23962510,1sleep20-21swapper/208:50:222
154242500,9sleep012-21ksoftirqd/012:24:330
193292480,1sleep2271rcuc/211:15:172
120322480,1sleep30-21swapper/309:10:293
55842470,2sleep00-21swapper/010:17:410
198922470,1sleep219894-21sh10:28:212
118782470,2sleep00-21swapper/009:35:100
106462470,2sleep00-21swapper/009:34:090
8252450,2chrt0-21swapper/310:14:423
10422450,4sleep20-21swapper/209:26:252
235412400,1sleep00-21swapper/008:25:170
234532400,3chrt0-21swapper/010:30:240
262722360,2sleep10-21swapper/111:44:491
8132310,2chrt0-21swapper/211:01:282
109502300,2chrt0-21swapper/111:33:061
109502300,2chrt0-21swapper/111:33:051
2306099290,12cyclictest13638-21head09:35:252
2305899298,12cyclictest12-21ksoftirqd/009:10:290
23058992912,4cyclictest12-21ksoftirqd/010:46:100
23058992912,4cyclictest12-21ksoftirqd/010:46:090
2306099284,7cyclictest28-21ksoftirqd/212:30:162
2306099284,5cyclictest6158-21ntpq09:30:192
2306099283,9cyclictest2718-21ssh10:39:582
2306099282,8cyclictest28-21ksoftirqd/210:10:452
2306099281,17cyclictest5220-21sh11:29:082
2305899288,5cyclictest12-21ksoftirqd/010:20:230
100652280,2sleep20-21swapper/209:56:592
2306099272,6cyclictest0-21swapper/211:05:202
2305899273,8cyclictest12-21ksoftirqd/010:44:250
23058992712,5cyclictest12-21ksoftirqd/011:50:040
2306099269,6cyclictest28-21ksoftirqd/210:59:152
2306099262,7cyclictest28-21ksoftirqd/210:43:572
2306099260,14cyclictest28-21ksoftirqd/209:15:512
2305999261,1cyclictest0-21swapper/107:12:241
23058992617,4cyclictest131rcuc/011:10:230
23058992611,4cyclictest12-21ksoftirqd/011:40:560
23058992610,3cyclictest12-21ksoftirqd/012:03:220
2305899260,11cyclictest12-21ksoftirqd/009:00:010
2306199251,23cyclictest0-21swapper/307:12:383
2306099258,6cyclictest28-21ksoftirqd/212:01:512
2306099254,8cyclictest28-21ksoftirqd/212:25:272
2306099254,7cyclictest28-21ksoftirqd/209:53:542
2306099253,7cyclictest28-21ksoftirqd/212:36:432
2306099253,6cyclictest4593-21ssh10:16:422
2306099252,6cyclictest22460-21taskset10:05:452
23060992521,3cyclictest271rcuc/209:10:072
2306099251,8cyclictest9037-21ssh11:55:182
2306099251,23cyclictest0-21swapper/207:12:392
2305899259,8cyclictest141rcu_preempt10:09:290
2305899259,5cyclictest12-21ksoftirqd/010:01:190
2305899259,5cyclictest12-21ksoftirqd/010:01:180
2305899259,4cyclictest12-21ksoftirqd/009:23:420
2305899258,3cyclictest12-21ksoftirqd/009:48:250
2305899257,9cyclictest12-21ksoftirqd/011:45:150
2305899257,5cyclictest12-21ksoftirqd/011:03:270
2305899256,8cyclictest12-21ksoftirqd/012:36:170
2305899253,15cyclictest18706-21ssh09:40:120
2305899253,12cyclictest0-21swapper/011:08:520
23058992523,1cyclictest0-21swapper/007:12:410
23058992510,4cyclictest12-21ksoftirqd/010:58:340
299632240,4sleep3331rcuc/310:35:213
2306099244,6cyclictest28-21ksoftirqd/212:10:002
2306099243,7cyclictest28-21ksoftirqd/212:05:272
2306099242,7cyclictest28-21ksoftirqd/211:41:052
2306099242,6cyclictest961-21taskset08:49:402
2306099242,6cyclictest6601-21ssh11:54:022
2306099242,6cyclictest16733-21ssh11:13:382
23060992411,3cyclictest28-21ksoftirqd/211:36:122
2306099240,9cyclictest28827-21rm09:46:572
2306099240,10cyclictest27070-21ssh09:21:512
23059992414,7cyclictest8921-21fw_packets10:20:141
2305899249,5cyclictest141rcu_preempt10:10:040
2305899249,5cyclictest12-21ksoftirqd/010:35:420
2305899248,4cyclictest12-21ksoftirqd/012:06:210
2305899247,4cyclictest12-21ksoftirqd/011:56:050
2305899247,4cyclictest12-21ksoftirqd/011:21:130
168992240,2sleep00-21swapper/009:15:030
32632230,2sleep30-21swapper/309:28:363
31852230,3sleep30-21swapper/309:51:543
23061992313,7cyclictest27740-21diskstats07:20:133
2306099236,6cyclictest28-21ksoftirqd/209:41:182
2306099234,7cyclictest28-21ksoftirqd/212:17:412
2306099234,7cyclictest28-21ksoftirqd/212:17:402
2306099234,7cyclictest28-21ksoftirqd/208:25:012
2306099233,6cyclictest28-21ksoftirqd/212:24:542
2306099232,7cyclictest28-21ksoftirqd/210:50:582
23060992318,3cyclictest271rcuc/210:00:312
23060992318,3cyclictest271rcuc/210:00:302
23060992312,8cyclictest1322-21date07:35:012
2305999231,7cyclictest7768-21ntp_states09:55:191
2305999231,15cyclictest21986-21ssh10:30:111
2305899239,5cyclictest12-21ksoftirqd/009:26:010
2305899234,12cyclictest141rcu_preempt12:10:220
2305899232,14cyclictest16319-21ssh12:25:120
23058992312,2cyclictest131321cron09:59:590
23058992311,2cyclictest141rcu_preempt10:28:050
2305899231,11cyclictest6521-21apt-get11:30:110
2305899231,11cyclictest6521-21apt-get11:30:100
2306199222,14cyclictest9494-21apt-get12:20:093
23061992214,6cyclictest21151-21cron12:05:003
2306099223,7cyclictest28-21ksoftirqd/208:08:392
2306099223,6cyclictest24906-21ssh10:31:352
2306099222,6cyclictest28-21ksoftirqd/208:16:592
2306099220,7cyclictest28-21ksoftirqd/208:22:092
2305999221,18cyclictest8275-21seq07:50:141
2305899228,5cyclictest12-21ksoftirqd/009:51:180
2305899227,8cyclictest141rcu_preempt12:30:160
2305899223,5cyclictest12-21ksoftirqd/011:35:180
23058992217,3cyclictest12-21ksoftirqd/008:15:170
23058992216,4cyclictest12-21ksoftirqd/008:10:000
23058992213,7cyclictest10543-21fw_packets07:55:150
2305899220,5cyclictest17436-21ssh10:50:150
23061992113,6cyclictest12821-21seq11:58:293
23061992112,7cyclictest4554-21python308:55:233
23061992112,7cyclictest3915-21processes12:15:223
23061992112,7cyclictest3915-21processes12:15:213
2306099214,10cyclictest28-21ksoftirqd/208:00:182
2306099212,6cyclictest28-21ksoftirqd/209:01:372
2306099211,7cyclictest28-21ksoftirqd/208:41:312
2306099211,7cyclictest28-21ksoftirqd/207:22:072
23060992117,3cyclictest271rcuc/207:55:002
23060992111,8cyclictest25564-21perl08:30:152
2305999213,6cyclictest10660-21proc_pri12:20:201
23059992113,6cyclictest30153-21seq10:11:391
23059992113,6cyclictest14569-21seq09:12:591
23059992113,6cyclictest11224-21grep07:55:211
23059992112,7cyclictest5234-21seq07:41:071
23059992112,6cyclictest14604-21ntp_kernel_pll_10:00:201
23059992112,6cyclictest14604-21ntp_kernel_pll_10:00:191
2305899219,4cyclictest12-21ksoftirqd/011:15:180
2305899212,13cyclictest640-21snmpd07:20:110
23058992115,5cyclictest12-21ksoftirqd/007:30:000
23058992113,6cyclictest16981-21fw_packets08:10:150
117332210,2sleep30-21swapper/310:22:003
2306199201,9cyclictest19606-21ntp_states11:15:213
2306099208,2cyclictest141rcu_preempt08:10:112
2306099202,6cyclictest28-21ksoftirqd/208:55:112
2306099202,12cyclictest5801-21apt-get07:45:112
2306099201,7cyclictest28-21ksoftirqd/208:38:242
2306099201,6cyclictest3108-21taskset07:36:302
2306099201,6cyclictest3108-21taskset07:36:292
23060992011,6cyclictest10746-21sed07:55:162
2306099200,8cyclictest5361-21taskset07:42:432
2306099200,8cyclictest31339-21taskset07:29:042
2305999208,3cyclictest2299-21ntpq11:50:201
2305999201,7cyclictest9589-21diskmemload10:49:051
2305999201,7cyclictest9589-21diskmemload10:49:041
23059992014,4cyclictest22-21ksoftirqd/112:40:011
23059992012,6cyclictest31051-21seq08:42:101
23059992012,6cyclictest30923-21grep07:25:241
23059992012,6cyclictest28866-21awk10:35:111
23059992012,6cyclictest22330-21ssh10:05:371
2305899208,5cyclictest12-21ksoftirqd/008:45:120
2305899207,7cyclictest141rcu_preempt08:50:170
23058992012,6cyclictest7230-21grep07:45:250
23058992011,6cyclictest25886-21sed08:30:170
23058992010,4cyclictest12-21ksoftirqd/007:45:000
237662190,8sleep334-21ksoftirqd/309:20:043
2306199197,5cyclictest26322-21latency09:45:183
2306199192,10cyclictest26918-21ssh11:45:123
2306199191,8cyclictest24688-21ntp_states10:55:213
23061991914,3cyclictest7955-21sendmail10:20:013
23061991913,4cyclictest25823-21sh10:32:283
23061991912,5cyclictest9630-21seq09:56:333
23061991912,5cyclictest636-21runrttasks12:05:113
23061991912,5cyclictest26583-21sed11:45:013
23061991912,5cyclictest23888-21ssh10:07:113
23061991912,5cyclictest16678-21diskstats12:25:143
23061991912,5cyclictest12162-21date08:00:023
23061991912,4cyclictest0-21swapper/311:10:113
23061991911,6cyclictest4961-21seq11:28:513
23061991911,5cyclictest52850irq/140-enp3s0-10:40:113
23061991910,6cyclictest30863-21cut11:00:153
2306199190,4cyclictest0-21swapper/308:13:343
2306099191,6cyclictest28-21ksoftirqd/207:19:052
2305999192,8cyclictest9050-21ntp_states11:55:201
2305999191,8cyclictest17038-21ssh09:15:121
2305999191,7cyclictest22065-21perl12:05:151
2305999191,6cyclictest0-21swapper/112:31:141
2305999191,5cyclictest640-21snmpd11:35:211
2305999191,5cyclictest0-21swapper/109:32:321
2305999191,4cyclictest0-21swapper/109:00:151
23059991913,4cyclictest4104-21sh11:27:591
23059991913,4cyclictest30177-21sh11:46:591
23059991913,4cyclictest21405-21idleruntime08:20:161
23059991913,4cyclictest17747-21sh10:26:141
23059991912,5cyclictest3907-21sh09:52:361
23059991912,5cyclictest18345-21cat11:15:021
23059991911,6cyclictest7783-21expr09:05:131
23059991911,6cyclictest29436-21ls07:25:001
23059991911,6cyclictest14241-21cut08:05:011
23059991911,5cyclictest18107-21mailstats10:50:221
23059991910,6cyclictest52950irq/141-enp3s0-10:15:101
23059991910,6cyclictest22-21ksoftirqd/111:00:111
23059991910,6cyclictest20058-21seq08:15:251
23059991910,6cyclictest16982-21fw_packets08:10:161
2305899197,5cyclictest12-21ksoftirqd/007:50:180
23058991913,4cyclictest12-21ksoftirqd/007:40:000
23058991913,4cyclictest12-21ksoftirqd/007:40:000
23058991911,6cyclictest560-21dbus-daemon08:25:000
260892180,1sleep30-21swapper/312:31:393
2306199189,7cyclictest14255-21cut08:05:003
2306199189,7cyclictest1233-21cat08:50:093
2306199189,6cyclictest6709-21ntp_offset07:45:203
2306199186,3cyclictest141rcu_preempt11:20:133
2306199181,8cyclictest2456-21ssh12:38:263
23061991812,4cyclictest9589-21diskmemload09:35:233
23061991812,4cyclictest6862-21vmstat09:30:243
23061991812,4cyclictest2998-21savelog07:36:033
23061991812,4cyclictest2998-21savelog07:36:023
23061991812,4cyclictest21554-21sh09:41:303
23061991812,4cyclictest11012-21sh10:45:193
23061991812,4cyclictest11012-21sh10:45:183
23061991811,5cyclictest5812-21expr09:00:113
23061991811,5cyclictest25329-21expr08:30:123
23061991811,5cyclictest20992-21expr08:20:123
23061991811,5cyclictest13703-21grep10:00:123
23061991811,5cyclictest13703-21grep10:00:123
23061991811,4cyclictest9589-21diskmemload10:27:443
23061991811,4cyclictest26269-21processes07:15:213
2306199181,12cyclictest14166-21ssh11:35:173
23061991810,6cyclictest5809-21ntpq11:05:183
23061991810,6cyclictest31973-21seq12:12:183
23061991810,6cyclictest29650-21grep07:25:133
23061991810,6cyclictest19608-21pluginstate08:15:223
23061991810,6cyclictest15664-21cat08:05:253
2306099180,7cyclictest7661-21runrttasks09:05:062
2305999187,9cyclictest0-21swapper/112:13:591
2305999187,9cyclictest0-21swapper/112:01:511
2305999185,11cyclictest22-21ksoftirqd/107:15:231
2305999182,11cyclictest18695-21apt-get09:40:111
2305999182,11cyclictest16301-21apt-get12:25:121
2305999181,8cyclictest18905-21df_inode11:15:131
2305999181,8cyclictest14773-21rm09:36:321
2305999181,6cyclictest0-21swapper/109:25:181
23059991813,3cyclictest9019-21sh10:44:261
23059991813,3cyclictest3506-21latency12:15:181
23059991813,3cyclictest3506-21latency12:15:171
23059991812,4cyclictest27319-21sh09:45:291
23059991811,5cyclictest6685-21ntp_kernel_err07:45:191
23059991811,5cyclictest3859-21perl08:55:141
23059991811,5cyclictest16196-21seq08:09:191
23059991811,4cyclictest0-21swapper/111:20:131
23059991810,6cyclictest623-21ls07:30:241
23059991810,6cyclictest26881-21seq08:33:461
2305899181,5cyclictest0-21swapper/009:05:170
2305899181,5cyclictest0-21swapper/008:40:120
2306199179,6cyclictest8158-21fw_packets09:05:163
23061991712,3cyclictest3640-21/usr/sbin/munin07:40:203
23061991711,4cyclictest18738-21sh10:50:383
2306199171,11cyclictest31679-21apt-get07:30:123
23061991710,5cyclictest7252-21cat11:30:153
23061991710,5cyclictest7252-21cat11:30:153
23061991710,5cyclictest31829-21expr08:45:143
2305999179,6cyclictest2500-21which07:35:231
2305999179,6cyclictest2500-21which07:35:221
23059991710,5cyclictest4754-21cat11:05:101
23059991710,5cyclictest342-21cat08:45:231
23059991710,5cyclictest28825-21seq08:36:151
2305899175,4cyclictest12-21ksoftirqd/008:35:230
23058991712,3cyclictest32592-21ntp_offset07:30:190
23058991712,3cyclictest25216-21/usr/sbin/munin07:15:170
23058991710,5cyclictest6139-21basename09:00:140
23058991710,5cyclictest12438-21expr08:00:120
2306199169,5cyclictest8096-21cat07:50:113
2306199169,5cyclictest29781-21grep08:40:143
2306199168,6cyclictest22933-21/usr/sbin/munin08:25:223
2305999169,5cyclictest2330-21ps08:50:221
23061991510,3cyclictest27217-21/usr/sbin/munin08:35:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional