You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-22 - 15:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackaslot8s.osadl.org (updated Sun Mar 22, 2026 12:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
244272172149,15sleep00-21swapper/007:05:240
246572143106,26sleep30-21swapper/307:07:323
247032141104,26sleep20-21swapper/207:08:072
227172139101,25sleep10-21swapper/107:05:011
304672720,2sleep20-21swapper/212:10:272
147432710,8sleep122-21ksoftirqd/108:00:141
128092650,1sleep20-21swapper/212:21:392
31582640,2sleep228-21ksoftirqd/211:02:532
149512630,2sleep30-21swapper/308:00:163
21512610,4sleep3331rcuc/311:50:143
143732570,1sleep20-21swapper/210:23:362
146412560,1sleep10-21swapper/111:10:511
58002550,1sleep10-21swapper/111:52:361
274272540,1sleep10-21swapper/112:09:101
206492540,5sleep012-21ksoftirqd/011:40:030
257652530,2sleep10-21swapper/111:43:381
160682520,2sleep00-21swapper/012:24:540
88792510,2sleep00-21swapper/011:55:120
106192510,1sleep10-21swapper/109:56:231
238472480,1sleep10-21swapper/109:18:291
106882480,3sleep10-21swapper/111:08:381
5612470,2chrt0-21swapper/209:25:132
88942460,1sleep20-21swapper/211:06:512
302402460,1sleep00-21swapper/008:35:200
268072450,1sleep20-21swapper/210:55:552
264132440,1sleep00-21swapper/012:31:450
108092440,1sleep30-21swapper/310:45:003
108092440,1sleep30-21swapper/310:45:003
16942420,2sleep00-21swapper/009:50:160
2532999398,15cyclictest0-21swapper/311:44:233
2532999395,23cyclictest16481-21ssh12:25:063
2532999389,16cyclictest31972-21sh09:24:543
25329993810,11cyclictest14458-21taskset11:10:403
12612380,3chrt0-21swapper/110:37:241
2532999368,17cyclictest591-21sh12:36:403
2532999359,8cyclictest11856-21diskmemload12:06:273
2532999358,9cyclictest0-21swapper/312:20:433
2532999338,10cyclictest11824-21ssh10:21:053
2532999337,11cyclictest28903-21smartctl10:10:163
2532999336,13cyclictest17785-21/usr/sbin/munin10:50:153
2532999335,11cyclictest27102-21taskset10:56:123
2532999329,9cyclictest11856-21diskmemload10:45:263
2532999328,9cyclictest21692-21unixbench_singl09:40:253
2532999327,8cyclictest0-21swapper/311:28:263
2532999327,8cyclictest0-21swapper/309:11:533
2532999327,8cyclictest0-21swapper/309:11:533
2532999326,9cyclictest1482-21ls09:25:223
2532999325,12cyclictest467-21ssh12:12:553
2532999325,10cyclictest13801-21/usr/sbin/munin11:35:183
2532999319,6cyclictest1330-21rm11:01:033
2532999318,10cyclictest13030-21ssh09:35:023
2532999314,18cyclictest4244-21ssh09:51:473
25329993110,10cyclictest19789-21sh12:03:043
10752310,2sleep10-21swapper/111:49:351
2532999307,9cyclictest11856-21diskmemload09:48:583
2532999306,8cyclictest10124-21ssh09:55:543
2532999306,6cyclictest9583-21ls11:55:193
2532999305,8cyclictest640-21snmpd10:19:393
2532999305,10cyclictest0-21swapper/310:35:123
25329993011,16cyclictest28019-21fschecks_count11:45:153
214272300,2chrt0-21swapper/310:05:073
2532999297,9cyclictest10285-21ssh11:31:543
2532999296,9cyclictest31123-21awk07:20:243
2532999296,7cyclictest16217-21apt-get10:25:113
2532999296,10cyclictest10114-21ssh11:08:043
2532999295,9cyclictest29264-21ssh12:34:343
2532999295,9cyclictest11856-21diskmemload12:18:423
2532999294,9cyclictest9973-21apt-get09:05:153
2532799299,7cyclictest22-21ksoftirqd/109:15:001
2532799299,7cyclictest22-21ksoftirqd/109:15:001
2532799297,10cyclictest22-21ksoftirqd/109:51:241
2532999286,8cyclictest11856-21diskmemload11:22:023
2532999285,11cyclictest27960-21ssh10:33:233
2532799285,9cyclictest22-21ksoftirqd/111:29:181
297842270,2sleep00-21swapper/011:45:530
2532999276,7cyclictest0-21swapper/310:00:063
2532999276,6cyclictest21369-21perl08:15:163
2532999275,7cyclictest19246-21apt-get11:15:123
2532999275,10cyclictest2889-21awk08:45:273
2532999274,9cyclictest0-21swapper/309:31:393
2532899271,10cyclictest20756-21/usr/sbin/munin11:40:152
2532799278,9cyclictest22-21ksoftirqd/110:32:321
2532799277,9cyclictest22-21ksoftirqd/110:25:101
2532799275,11cyclictest22-21ksoftirqd/110:01:011
2532799274,12cyclictest22-21ksoftirqd/109:41:021
25327992711,10cyclictest22-21ksoftirqd/111:22:141
2532699271,6cyclictest20441-21apt-key11:40:010
25326992714,8cyclictest52950irq/141-enp3s0-12:05:120
2532699271,24cyclictest0-21swapper/007:10:060
139472270,2chrt0-21swapper/211:59:002
2532999269,14cyclictest624-21gdbus07:45:253
25329992611,12cyclictest11050-21grep07:50:203
25329992611,11cyclictest20189-21grep09:15:213
2532899261,22cyclictest1275-21cron07:30:002
2532799269,3cyclictest22-21ksoftirqd/112:00:451
2532799269,3cyclictest22-21ksoftirqd/110:53:081
2532799267,4cyclictest22-21ksoftirqd/112:10:361
2532799266,8cyclictest22-21ksoftirqd/109:30:211
2532799264,10cyclictest22-21ksoftirqd/109:36:021
25327992616,4cyclictest22-21ksoftirqd/112:22:071
25327992613,5cyclictest22-21ksoftirqd/108:20:471
2532799261,24cyclictest0-21swapper/107:10:061
21482260,3sleep20-21swapper/210:14:582
206222260,2sleep00-21swapper/011:15:230
137502260,1chrt0-21swapper/110:46:231
25329992512,11cyclictest29416-21cron07:20:003
25329992512,10cyclictest8847-21ps09:00:213
2532899257,12cyclictest19266-21sh11:15:122
25328992513,10cyclictest7973-21sh10:19:012
2532799259,4cyclictest22-21ksoftirqd/112:32:441
2532799257,1cyclictest53050irq/142-enp3s0-10:07:401
2532799255,11cyclictest22-21ksoftirqd/111:57:111
2532799254,6cyclictest3406-21df12:15:141
2532799254,11cyclictest22-21ksoftirqd/109:46:241
2532799254,10cyclictest22-21ksoftirqd/111:17:071
2532799253,11cyclictest22-21ksoftirqd/109:23:141
25327992522,2cyclictest211rcuc/110:55:221
25327992511,4cyclictest22-21ksoftirqd/112:36:591
25327992510,8cyclictest22-21ksoftirqd/110:21:311
174362250,3chrt0-21swapper/211:13:372
312032240,2sleep00-21swapper/011:23:500
2532999246,12cyclictest0-21swapper/308:55:113
25329992411,10cyclictest5438-21awk08:55:003
25329992411,10cyclictest1471-21grep07:30:003
25329992410,11cyclictest1299-21sed08:45:013
2532799248,3cyclictest141rcu_preempt10:10:311
2532799246,11cyclictest22-21ksoftirqd/111:04:371
2532799242,5cyclictest22-21ksoftirqd/110:45:001
2532799242,5cyclictest22-21ksoftirqd/110:45:001
2532799241,6cyclictest22-21ksoftirqd/108:10:191
25327992412,4cyclictest22-21ksoftirqd/108:29:021
25327992412,4cyclictest22-21ksoftirqd/107:17:181
25327992410,7cyclictest141rcu_preempt11:35:071
25327992410,7cyclictest141rcu_preempt09:25:541
2532699244,10cyclictest11856-21diskmemload10:35:420
25329992313,8cyclictest27258-21sh07:15:003
25329992313,7cyclictest19307-21gltestperf08:10:153
25329992311,9cyclictest16830-21/usr/sbin/munin08:05:143
25329992311,8cyclictest0-21swapper/308:35:133
25329992310,10cyclictest27864-21gltestperf08:30:153
25328992316,4cyclictest30784-21sh12:35:172
25328992311,8cyclictest10636-21basename10:45:002
25328992311,8cyclictest10636-21basename10:45:002
2532799239,4cyclictest22-21ksoftirqd/112:25:011
2532799234,12cyclictest22-21ksoftirqd/110:15:091
25327992318,3cyclictest211rcuc/111:35:001
2532699231,17cyclictest52950irq/141-enp3s0-12:00:120
2532699231,10cyclictest14259-21ntp_states09:35:210
2532999229,10cyclictest14322-21latency_hist08:00:003
25329992212,7cyclictest4062-21fschecks_count07:35:143
25329992211,9cyclictest24502-21/usr/sbin/munin08:20:233
25329992210,9cyclictest25319-21cpuspeed_turbos08:25:133
2532899224,16cyclictest20098-21ntp_states09:15:202
2532899221,10cyclictest5676-21taskset08:55:122
25328992210,3cyclictest15370-21idleruntime10:00:152
2532799223,9cyclictest22-21ksoftirqd/109:05:531
25327992216,4cyclictest22-21ksoftirqd/108:39:591
25327992214,6cyclictest30916-21ls07:20:221
2532699222,10cyclictest2618-21munin-run12:15:000
2532699221,6cyclictest12742-21sed10:45:280
25326992216,4cyclictest25355-21sendmail09:20:000
25326992214,6cyclictest22806-21seq08:19:590
25326992211,9cyclictest26977-21ssh10:56:050
147732220,2sleep00-21swapper/011:10:580
2532899219,8cyclictest29521-21apt-get08:35:142
2532899216,3cyclictest279112expr11:45:132
2532899213,4cyclictest11856-21diskmemload10:35:072
25328992115,4cyclictest19489-21grep10:50:252
25328992113,6cyclictest17110-21sed10:25:182
25328992111,7cyclictest23220-21apt-get10:30:112
2532899211,17cyclictest14565-21irqstats11:35:172
25328992111,6cyclictest52750irq/139-enp3s0-09:45:122
2532799212,7cyclictest453-21threads08:40:231
2532799212,5cyclictest0-21swapper/107:25:231
2532799211,7cyclictest22-21ksoftirqd/108:05:231
25327992117,3cyclictest211rcuc/108:20:001
25327992117,3cyclictest211rcuc/107:49:591
25327992113,6cyclictest9511-21seq09:02:321
2532699219,3cyclictest141rcu_preempt12:15:230
2532699216,3cyclictest141rcu_preempt09:40:200
2532699214,4cyclictest11856-21diskmemload09:48:050
2532699214,3cyclictest11856-21diskmemload10:03:190
2532699211,8cyclictest9225-21python309:55:210
2532699211,8cyclictest0-21swapper/010:50:130
25326992113,6cyclictest593-21smartctl09:25:140
25326992113,6cyclictest4438-21memory07:35:170
25326992113,6cyclictest24179-21seq10:06:210
25326992113,6cyclictest11327-21seq09:33:340
25326992112,7cyclictest11785-21seq09:09:180
25326992111,7cyclictest27024-21sendmail_mailst09:20:210
25329992013,5cyclictest1633-21grep07:30:123
25329992013,5cyclictest1633-21grep07:30:123
25329992012,6cyclictest6057-21grep07:40:143
2532899208,10cyclictest24181-21latency12:30:182
2532899204,3cyclictest141rcu_preempt11:27:032
2532899201,5cyclictest0-21swapper/212:05:162
25328992014,4cyclictest31041-21sh09:23:582
25328992012,6cyclictest13289-21cat09:35:132
25327992012,6cyclictest3316-21seq07:33:441
25327992012,6cyclictest3316-21seq07:33:441
25327992011,7cyclictest22-21ksoftirqd/108:55:001
2532799201,10cyclictest0-21swapper/108:45:211
2532799200,6cyclictest12096-21cron07:55:001
2532699204,3cyclictest11856-21diskmemload11:01:220
2532699204,3cyclictest11856-21diskmemload09:10:060
2532699204,3cyclictest11856-21diskmemload09:10:060
2532699203,4cyclictest11856-21diskmemload10:43:510
2532699203,4cyclictest11856-21diskmemload10:43:510
25326992013,5cyclictest29880-21grep10:10:250
25326992013,5cyclictest17008-21diskstats12:25:150
25326992012,6cyclictest5578-21sed08:55:000
25326992012,6cyclictest31800-21find07:25:130
25326992012,6cyclictest26367-21run-parts08:25:220
25326992012,6cyclictest13236-21awk07:55:210
2532699201,10cyclictest0-21swapper/010:20:130
25326992010,8cyclictest16401-21cron08:05:000
27962190,2sleep00-21swapper/011:50:190
2532899198,9cyclictest52750irq/139-enp3s0-11:51:052
2532899193,3cyclictest11856-21diskmemload10:08:092
2532899192,3cyclictest11856-21diskmemload09:32:352
2532899192,12cyclictest75-21kswapd012:25:122
25328991912,5cyclictest25234-21apt-get08:25:112
2532899191,16cyclictest3042-21apt-get12:15:122
25328991911,6cyclictest28627-21grep08:30:232
2532799199,4cyclictest22-21ksoftirqd/107:40:251
2532799193,9cyclictest22-21ksoftirqd/107:56:471
25327991911,6cyclictest27722-21cat08:30:141
25327991910,6cyclictest6016-21cat08:55:151
2532699191,6cyclictest0-21swapper/011:09:020
2532699191,5cyclictest0-21swapper/011:30:040
25326991913,4cyclictest636-21runrttasks10:15:120
25326991912,5cyclictest7495-21seq08:59:230
25326991912,5cyclictest17021-21sed10:25:170
25326991911,6cyclictest9930-21date09:05:000
25326991911,6cyclictest30368-21expr12:35:140
25326991911,6cyclictest23819-21sed08:20:180
25326991911,6cyclictest1389-21netstat11:25:210
25326991910,7cyclictest3518-21cut07:35:000
25326991910,7cyclictest3518-21cut07:35:000
226442190,3sleep20-21swapper/208:17:552
11672190,3sleep20-21swapper/208:45:012
2532899183,7cyclictest13211-21ssh11:34:492
2532899182,11cyclictest1126-21apt-get09:50:122
2532899181,8cyclictest14855-21diskstats08:00:152
2532899181,4cyclictest141rcu_preempt11:20:122
2532899181,4cyclictest0-21swapper/212:00:032
25328991814,2cyclictest30835-21sensors_temp07:20:222
2532899181,3cyclictest11856-21diskmemload09:58:412
25328991813,3cyclictest3654-21/usr/sbin/munin07:35:122
25328991813,3cyclictest19886-21sh09:40:062
25328991811,5cyclictest8074-21expr09:00:132
25328991811,5cyclictest2277-21sed07:30:172
25328991811,5cyclictest2277-21sed07:30:172
25328991811,5cyclictest1640-21expr08:45:152
25328991811,5cyclictest13289-21awk09:10:192
25328991811,5cyclictest13289-21awk09:10:192
25328991811,5cyclictest11615-21uniq10:45:192
25328991811,5cyclictest10594-21if_err_enp3s009:05:192
25328991811,5cyclictest10314-21expr07:50:132
2532899181,14cyclictest9100-21wc07:45:242
25328991810,6cyclictest6540-21basename07:40:172
25328991810,6cyclictest3871-21df08:50:152
2532899180,16cyclictest25123-21wc08:25:002
25327991810,6cyclictest4662-21awk07:35:201
2532699189,7cyclictest31362-21seq07:22:330
2532699182,11cyclictest27521-21apt-get07:15:140
2532699181,5cyclictest0-21swapper/010:31:130
25326991810,6cyclictest8299-21expr07:45:150
25326991810,6cyclictest6659-21wc07:40:170
15662180,2sleep00-21swapper/008:45:150
2532899179,5cyclictest28241-21latency07:15:192
25328991712,3cyclictest18769-21/usr/sbin/munin08:10:172
25328991710,5cyclictest26090-21wc07:10:192
2532699179,5cyclictest18879-21grep08:10:120
2532699179,5cyclictest17173-21sed08:05:150
25326991712,3cyclictest31652-21/usr/sbin/munin08:40:200
2532699171,14cyclictest10616-21tune2fs07:50:160
2532899168,6cyclictest17378-21grep08:05:192
2532699162,4cyclictest12-21ksoftirqd/008:33:560
2532899147,5cyclictest12939-21basename07:55:172
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional