You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-22 - 17:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Jul 22, 2026 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
269022236163,24sleep20-21swapper/207:06:202
270432202164,25sleep10-21swapper/107:08:081
268782202165,24sleep30-21swapper/307:06:013
252252201164,25sleep00-21swapper/007:05:050
27278999491,2cyclictest0-21swapper/209:49:282
246342600,1sleep10-21swapper/108:15:141
190332560,1sleep10-21swapper/111:40:251
286952550,1sleep20-21swapper/210:50:172
27278995135,3cyclictest27924-21switchtime08:20:252
252382490,1sleep30-21swapper/311:55:243
27285994127,3cyclictest0-21swapper/309:04:413
27272994111,20cyclictest16695-21munin-run10:25:011
27285994026,7cyclictest324442sleep312:11:533
27278994026,6cyclictest0-21swapper/211:35:452
2727299402,15cyclictest87550irq/128-enp1s0-09:50:161
2727299401,36cyclictest18276-21idleruntime-cro09:15:011
2727299401,36cyclictest18276-21idleruntime-cro09:15:011
2727299401,27cyclictest11681-21sleep09:00:011
2727299401,18cyclictest6445-21grep12:25:251
2727299391,24cyclictest29597-21cut09:40:111
2727299387,22cyclictest32070-21latency_hist11:00:021
2727299381,26cyclictest20464-21latency_hist09:20:011
2727299381,23cyclictest18880-21latency_hist10:30:011
2727299381,22cyclictest2836-21munin-run08:40:011
2726299389,13cyclictest11671-21munin-run09:00:010
2727299377,21cyclictest4731-21latency_hist12:25:011
2727299371,25cyclictest9178-21fschecks_count11:20:141
2728599364,23cyclictest0-21swapper/309:33:293
27285993627,4cyclictest0-21swapper/308:05:573
2728599361,21cyclictest0-21swapper/309:41:213
2727299362,24cyclictest29588-21munin-node09:20:291
2727299361,31cyclictest219482sleep108:10:011
2727299361,29cyclictest9788-21kworker/u8:010:06:551
2727299361,25cyclictest7160-21/usr/sbin/munin12:30:151
2727299361,25cyclictest18692-21load08:00:191
2727299361,24cyclictest27495-21cstates09:35:131
2727299361,23cyclictest24866-21turbostat09:30:011
2727299360,29cyclictest20974-21kworker/u8:111:00:211
2726299368,13cyclictest32092-21idleruntime-cro11:00:010
2728599351,27cyclictest0-21swapper/311:07:253
2728599351,19cyclictest0-21swapper/307:22:053
27278993524,3cyclictest0-21swapper/210:10:252
27278993524,3cyclictest0-21swapper/208:27:092
27278993515,7cyclictest0-21swapper/209:04:412
2727299358,18cyclictest12412-21cat10:15:011
2727299354,19cyclictest0-21swapper/107:50:011
2727299354,18cyclictest16068-21diskmemload10:30:141
2727299352,21cyclictest87550irq/128-enp1s0-11:05:181
2727299351,26cyclictest21922-21kworker/u8:211:54:321
2727299351,25cyclictest27906-21/usr/sbin/munin10:50:241
2727299351,25cyclictest15326-21/usr/sbin/munin11:35:231
2727299351,24cyclictest6800-21/usr/sbin/munin07:35:131
2727299351,21cyclictest5097-21if_enp2s011:10:171
2726299359,13cyclictest23284-21munin-run10:40:010
27285993422,7cyclictest0-21swapper/309:54:133
27285993422,7cyclictest0-21swapper/308:51:213
2728599341,31cyclictest0-21swapper/308:27:093
2728599341,23cyclictest0-21swapper/307:32:333
2728599341,12cyclictest0-21swapper/310:55:413
2728599341,12cyclictest0-21swapper/310:32:333
27278993423,3cyclictest0-21swapper/207:27:572
27278993421,3cyclictest0-21swapper/208:59:222
27278993418,7cyclictest0-21swapper/212:26:492
27278993418,10cyclictest0-21swapper/207:42:532
27278993416,8cyclictest0-21swapper/207:39:492
2727299345,20cyclictest87550irq/128-enp1s0-08:25:131
2727299344,16cyclictest28533-21/usr/sbin/munin12:05:201
2727299343,20cyclictest87550irq/128-enp1s0-07:40:261
2727299341,29cyclictest12899-21latency_hist11:30:011
2727299341,24cyclictest23507-21/usr/sbin/munin10:40:241
2727299341,24cyclictest11318-21latency_hist12:40:011
2727299341,21cyclictest24212-21cpuspeed_turbos11:55:141
2727299341,20cyclictest26469-21tr12:00:141
2727299340,30cyclictest0-21swapper/108:50:191
2728599331,21cyclictest0-21swapper/308:22:093
27278993318,10cyclictest0-21swapper/207:20:092
27278993315,8cyclictest0-21swapper/211:25:292
2727299334,20cyclictest16068-21diskmemload10:36:541
2727299332,26cyclictest31436-21ntp_states07:15:221
2727299331,29cyclictest24155-21latency_hist08:15:011
2727299331,29cyclictest24155-21latency_hist08:15:011
2727299331,29cyclictest14115-21if_enp1s007:50:181
2727299331,21cyclictest25943-21ls09:30:191
2727299331,20cyclictest0-21swapper/111:30:141
2727299331,18cyclictest14364-21kworker/u8:210:46:361
27285993218,3cyclictest0-21swapper/308:33:133
2728599321,22cyclictest0-21swapper/307:57:493
2728599321,21cyclictest0-21swapper/307:39:013
2728599321,11cyclictest0-21swapper/309:20:213
2727899329,8cyclictest0-21swapper/208:53:412
27278993218,8cyclictest0-21swapper/207:33:332
27278993218,3cyclictest0-21swapper/208:06:252
27278993210,11cyclictest0-21swapper/208:40:532
2727299324,18cyclictest5700-21cut08:45:161
2727299323,19cyclictest87650irq/129-enp1s0-12:10:141
2727299322,23cyclictest20974-21kworker/u8:111:16:021
2727299321,23cyclictest7743-21seq10:03:121
2726299324,13cyclictest12919-21basename11:30:010
27285993118,3cyclictest0-21swapper/308:03:133
27285993117,3cyclictest0-21swapper/307:40:533
2728599311,14cyclictest0-21swapper/309:14:093
2728599311,14cyclictest0-21swapper/309:14:093
2728599311,13cyclictest0-21swapper/308:18:133
2728599311,12cyclictest0-21swapper/312:01:573
2727899319,8cyclictest0-21swapper/208:37:572
27278993119,3cyclictest0-21swapper/208:04:092
27278993116,10cyclictest0-21swapper/209:23:012
27278993113,11cyclictest0-21swapper/209:59:532
2727899311,14cyclictest0-21swapper/211:47:252
2727899311,13cyclictest0-21swapper/209:42:292
2727899310,29cyclictest0-21swapper/209:36:332
2727299315,17cyclictest0-21swapper/107:30:021
2727299311,24cyclictest9788-21kworker/u8:010:15:071
27272993110,18cyclictest879-21idleruntime07:20:161
2727299310,25cyclictest17017-21kworker/u8:011:46:271
2726299315,13cyclictest27060-21munin-run09:35:010
2728599301,20cyclictest0-21swapper/307:16:333
2728599301,14cyclictest0-21swapper/310:00:293
2728599301,13cyclictest0-21swapper/310:23:293
2728599301,11cyclictest0-21swapper/311:37:373
27278993016,10cyclictest0-21swapper/210:41:582
2727299308,12cyclictest0-21swapper/108:30:161
2727299303,22cyclictest485-21smart_sda09:45:241
2727299302,10cyclictest87650irq/129-enp1s0-08:22:261
2727299301,21cyclictest27472-21/usr/sbin/munin07:10:191
2727299301,20cyclictest0-21swapper/107:55:211
2727299301,18cyclictest1521-21missed_timers12:15:201
2727299301,15cyclictest3558-21fschecks_time08:40:141
2727299300,21cyclictest495-21jbd2/dm-0-807:34:561
2727299300,20cyclictest16078-21perf09:10:011
2726299304,13cyclictest18886-21munin-run10:30:010
2726299303,12cyclictest28615-21awk08:25:010
27285992919,3cyclictest0-21swapper/307:14:213
2728599291,3cyclictest0-21swapper/309:45:293
2728599291,20cyclictest0-21swapper/307:51:533
2727899299,8cyclictest0-21swapper/208:12:332
2727899299,8cyclictest0-21swapper/208:12:332
27278992916,8cyclictest0-21swapper/211:01:212
27278992916,3cyclictest0-21swapper/209:31:292
2727899291,2cyclictest0-21swapper/212:37:452
27278992912,3cyclictest0-21swapper/209:05:332
2727899291,12cyclictest0-21swapper/211:22:532
2727899291,12cyclictest0-21swapper/209:51:492
27278992910,8cyclictest0-21swapper/210:31:142
2727299291,21cyclictest87550irq/128-enp1s0-09:55:191
27285992817,3cyclictest0-21swapper/310:35:493
2728599281,25cyclictest0-21swapper/311:10:413
2728599281,21cyclictest0-21swapper/307:26:493
2728599281,15cyclictest0-21swapper/307:47:573
2728599281,14cyclictest0-21swapper/308:46:593
2728599281,12cyclictest0-21swapper/312:23:133
2728599281,12cyclictest0-21swapper/308:35:453
2728599281,11cyclictest0-21swapper/311:34:013
2728599281,11cyclictest0-21swapper/309:06:493
2728599281,10cyclictest0-21swapper/311:40:113
2727899289,6cyclictest0-21swapper/211:31:532
2727899289,3cyclictest0-21swapper/210:06:172
2727899288,6cyclictest5414-21ntp_states11:10:212
2727899288,11cyclictest0-21swapper/212:04:252
2727899287,8cyclictest0-21swapper/209:13:452
2727899287,8cyclictest0-21swapper/209:13:452
2727899287,3cyclictest0-21swapper/211:05:252
2727899286,8cyclictest0-21swapper/207:57:052
2727899285,3cyclictest0-21swapper/212:23:132
27278992813,3cyclictest0-21swapper/207:12:292
2727299281,24cyclictest12238-21/usr/sbin/munin09:00:151
2728599271,20cyclictest0-21swapper/308:41:373
2728599271,12cyclictest0-21swapper/310:12:493
2728599271,11cyclictest0-21swapper/310:48:173
2728599270,11cyclictest0-21swapper/311:25:233
2727899279,3cyclictest0-21swapper/209:15:172
27278992717,3cyclictest0-21swapper/208:46:372
27278992717,3cyclictest0-21swapper/207:52:052
27278992714,7cyclictest0-21swapper/210:58:292
27278992713,3cyclictest30453-21cat07:15:132
2727899271,17cyclictest0-21swapper/211:53:572
2727899271,12cyclictest0-21swapper/210:37:572
2727899271,11cyclictest0-21swapper/212:32:012
2726299270,12cyclictest16075-21munin-run09:10:010
2728599261,6cyclictest0-21swapper/311:52:403
2728599261,13cyclictest0-21swapper/308:56:333
2728599261,12cyclictest0-21swapper/311:00:533
2728599261,12cyclictest0-21swapper/310:08:253
2728599261,11cyclictest22188-21/usr/sbin/munin08:10:183
2728599261,11cyclictest22188-21/usr/sbin/munin08:10:183
2728599261,10cyclictest0-21swapper/312:07:133
2727899268,8cyclictest0-21swapper/207:48:412
27278992611,8cyclictest0-21swapper/212:11:052
2727899261,18cyclictest0-21swapper/210:00:172
2727899261,12cyclictest26509-21irqstats10:45:172
27278992610,3cyclictest0-21swapper/209:29:252
2727899260,14cyclictest0-21swapper/210:20:332
2728599251,9cyclictest0-21swapper/310:42:093
2728599251,23cyclictest0-21swapper/310:54:573
2728599251,11cyclictest0-21swapper/311:45:293
2727899259,8cyclictest0-21swapper/208:31:052
2727899259,3cyclictest0-21swapper/208:16:492
27278992511,2cyclictest6819-21cat11:15:132
2726299256,16cyclictest9133-21munin-run12:35:010
2726299251,12cyclictest5045-21perf08:45:010
2728599241,9cyclictest0-21swapper/310:19:293
2728599241,12cyclictest0-21swapper/309:29:533
2728599241,11cyclictest4961-21/usr/sbin/munin12:25:113
2728599241,11cyclictest0-21swapper/309:16:253
2727899248,3cyclictest0-21swapper/212:08:212
2727899245,7cyclictest0-21swapper/210:28:412
2726299241,3cyclictest0-21swapper/011:06:130
2726299241,12cyclictest18275-21munin-run09:15:010
2726299241,12cyclictest18275-21munin-run09:15:010
2728599231,21cyclictest0-21swapper/312:31:333
2728599231,10cyclictest0-21swapper/311:15:293
2727899238,3cyclictest0-21swapper/211:55:252
2727899237,6cyclictest0-21swapper/210:16:452
27278992310,3cyclictest0-21swapper/211:44:212
2726299231,11cyclictest16068-21diskmemload09:55:010
2726299230,4cyclictest0-21swapper/011:53:390
2728599222,4cyclictest0-21swapper/312:19:363
2728599221,8cyclictest0-21swapper/309:57:333
2728599221,5cyclictest0-21swapper/311:21:093
2728599221,14cyclictest0-21swapper/310:28:293
2728599221,13cyclictest0-21swapper/309:36:213
2726299223,16cyclictest4378-21munin-run07:30:010
2726299221,13cyclictest31028-21/usr/sbin/munin08:30:190
2726299219,4cyclictest0-21swapper/011:18:260
2726299211,3cyclictest0-21swapper/012:25:130
2728599204,2cyclictest0-21swapper/312:35:393
2727899204,3cyclictest0-21swapper/212:18:172
2726299207,3cyclictest17407-21cat11:40:010
2726299204,9cyclictest11-21rcuc/009:20:180
2726299203,14cyclictest19767-21cut10:30:160
2726299203,10cyclictest7-21ksoftirqd/010:05:180
2726299203,10cyclictest5168-21kernelversion11:10:180
2726299202,16cyclictest14494-21munin-run10:20:010
2726299202,10cyclictest11-21rcuc/009:25:290
2726299201,4cyclictest0-21swapper/012:18:170
27262992011,3cyclictest87550irq/128-enp1s0-10:10:400
2726299201,11cyclictest17526-21/usr/sbin/munin11:40:170
2726299201,10cyclictest0-21swapper/010:00:260
27262992010,3cyclictest87550irq/128-enp1s0-12:12:090
27262992010,3cyclictest870-21users09:45:260
2726299199,3cyclictest87550irq/128-enp1s0-10:44:150
2726299196,2cyclictest286962if_err_enp2s010:50:170
2726299195,3cyclictest2456-21vmstat08:35:270
2726299193,7cyclictest11-21rcuc/012:05:010
2726299193,14cyclictest27694-21perf10:50:010
2726299192,9cyclictest28205-21memory09:35:190
2726299192,15cyclictest31471-21awk09:45:010
2726299191,5cyclictest29603-21ntp_states12:05:220
2726299191,3cyclictest9651-21memory11:20:180
27262991912,4cyclictest18454-21gltestperf08:00:170
27262991911,5cyclictest1-21systemd07:40:200
2726299191,10cyclictest9356-21/usr/sbin/munin12:35:170
2726299191,10cyclictest32689-21fschecks_count11:00:150
2726299191,10cyclictest19310-21irqstats09:15:190
2726299190,11cyclictest1360-21ps07:20:220
2726299187,10cyclictest0-21swapper/011:50:010
2726299184,8cyclictest0-21swapper/007:54:440
2726299181,9cyclictest0-21swapper/007:19:260
2726299181,8cyclictest24880-21if_enp2s011:55:180
2726299181,3cyclictest0-21swapper/012:22:000
2726299174,5cyclictest7-21ksoftirqd/011:30:120
2726299171,9cyclictest0-21swapper/007:35:250
2726299171,6cyclictest16831-21chrt07:55:260
2726299171,6cyclictest11-21rcuc/008:10:260
2726299171,6cyclictest11-21rcuc/008:10:260
2726299171,6cyclictest0-21swapper/010:00:010
27262991711,3cyclictest8590-21pmu-power08:50:200
2726299171,10cyclictest0-21swapper/008:47:030
2726299171,10cyclictest0-21swapper/007:45:210
2726299165,2cyclictest15475-21if_enp2s010:20:170
2726299161,6cyclictest0-21swapper/007:10:180
2726299161,5cyclictest19993-21/usr/sbin/munin08:05:220
27262991611,3cyclictest4602-21/usr/sbin/munin07:30:240
27262991610,4cyclictest12997-21/usr/sbin/munin09:00:230
27262991610,3cyclictest29092-21expr08:25:130
27262991510,3cyclictest25011-21gltestperf08:15:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional