You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-17 - 02:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Wed Dec 17, 2025 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
265422238163,24sleep20-21swapper/219:05:372
265132214183,21sleep00-21swapper/019:05:260
266882204167,24sleep10-21swapper/119:07:281
266032200164,24sleep30-21swapper/319:06:223
2237321620,3sleep12697399cyclictest21:25:021
338721550,4sleep12697399cyclictest20:40:191
1342121140,3sleep02696799cyclictest19:50:140
26973991070,105cyclictest0-21swapper/121:01:371
2696799579,13cyclictest6880-21munin-run20:50:010
295142300,4sleep229517-21fschecks_count21:40:162
2696799293,12cyclictest0-21swapper/019:30:000
118052280,2sleep07-21ksoftirqd/019:45:200
2698499271,10cyclictest28406-21tail21:35:273
26973992710,6cyclictest717-40dnf21:48:581
2697399261,17cyclictest23195-21chrt23:51:131
2697399261,17cyclictest10086-21pmu-power00:35:231
2696799261,12cyclictest32330-21munin-run00:15:010
2698499251,12cyclictest14317-21sshd22:20:093
2697899255,6cyclictest770-21rngd20:45:202
297832240,2sleep20-21swapper/222:55:122
2698499249,3cyclictest25888-21sshd20:18:083
2698499242,9cyclictest0-21swapper/323:41:103
2698499241,9cyclictest3106-21perf21:55:013
26973992416,5cyclictest30988-21switchtime22:55:241
2696799245,16cyclictest26662-21munin-run21:35:010
2696799242,12cyclictest800-21snmpd22:45:020
26984992314,7cyclictest22132-21kworker/u8:121:11:293
2698499231,11cyclictest4202-21sshd00:23:093
2698499231,11cyclictest29714-21/usr/sbin/munin22:55:193
2698499231,11cyclictest2323-21sshd20:38:083
2698499231,10cyclictest17518-21needreboot22:25:193
2698499231,10cyclictest12776-21sshd19:49:093
26973992317,3cyclictest28255-21munin-run20:25:021
26973992314,3cyclictest0-21swapper/123:55:291
26973992314,3cyclictest0-21swapper/123:15:181
26973992314,3cyclictest0-21swapper/123:15:171
26973992314,3cyclictest0-21swapper/122:05:161
26973992314,3cyclictest0-21swapper/122:05:161
2697399230,16cyclictest0-21swapper/122:20:261
2697399230,16cyclictest0-21swapper/119:30:311
2696799236,9cyclictest19409-21kworker/0:020:12:480
302942221,17sleep230408-21df_inode19:15:132
2698499221,10cyclictest31913-21/usr/sbin/munin23:00:143
2698499221,10cyclictest13401-21sshd21:04:083
2697399223,10cyclictest679-21cat20:35:111
2697399221,3cyclictest269152sleep100:00:201
26973992213,3cyclictest0-21swapper/123:46:181
26973992213,3cyclictest0-21swapper/119:55:191
26973992213,3cyclictest0-21swapper/119:20:181
2697399221,18cyclictest31058-21latency_hist21:45:001
2697399221,15cyclictest0-21swapper/122:41:271
2697399221,15cyclictest0-21swapper/121:57:401
2697399221,15cyclictest0-21swapper/121:35:051
26973992210,3cyclictest24415-21df20:15:151
2697399220,16cyclictest0-21swapper/123:36:551
2697399220,16cyclictest0-21swapper/119:11:091
12372221,3sleep21260-21ntp_states00:15:182
2698499218,3cyclictest0-21swapper/323:39:583
2698499218,1cyclictest0-21swapper/320:40:193
2698499211,18cyclictest0-21swapper/323:30:223
2698499211,17cyclictest6717-21sshd20:48:083
2698499211,10cyclictest21904-21needreboot22:35:193
2698499211,10cyclictest18723-21/usr/sbin/munin22:30:203
2698499211,10cyclictest10964-21sshd20:56:093
26984992110,3cyclictest32040-21awk19:20:013
2697899212,6cyclictest21885-21/usr/sbin/munin20:10:212
26978992110,3cyclictest26674-21awk21:35:022
2697399219,3cyclictest2266-21cstates19:25:141
2697399219,3cyclictest18439-21df21:15:131
2697399212,11cyclictest14091-21latency_hist22:20:021
2697399211,3cyclictest0-21swapper/121:25:141
26973992113,5cyclictest7449-21head00:30:171
26973992112,3cyclictest0-21swapper/123:06:011
2697399211,15cyclictest0-21swapper/121:06:071
2697399211,15cyclictest0-21swapper/120:30:421
2697399211,15cyclictest0-21swapper/120:13:191
2697399211,14cyclictest0-21swapper/121:31:281
2697399211,14cyclictest0-21swapper/120:51:121
2697399211,14cyclictest0-21swapper/120:25:461
2697399211,14cyclictest0-21swapper/119:40:021
2697399211,14cyclictest0-21swapper/119:35:121
26973992111,3cyclictest0-21swapper/123:29:341
2697399211,11cyclictest6871-21latency_hist20:50:011
2697399211,11cyclictest16588-21hddtemp_smartct21:10:161
2697399210,16cyclictest0-21swapper/123:10:571
2697399210,15cyclictest0-21swapper/122:51:081
2696799214,15cyclictest28268-21basename20:25:020
2696799212,11cyclictest9935-21/usr/sbin/munin22:10:180
2696799211,3cyclictest0-21swapper/021:20:010
26967992110,3cyclictest1579-21ntp_states20:35:200
2698499208,3cyclictest0-21swapper/300:00:563
2698499206,3cyclictest87450irq/129-enp1s0-23:13:103
2698499205,4cyclictest27127-21sshd22:48:093
2698499204,3cyclictest29278-21cstates21:40:143
2698499204,3cyclictest15049-21dump-pmu-power19:59:563
2698499203,3cyclictest19766-21cpuspeed_turbos20:05:123
2698499202,4cyclictest12432-21sshd23:29:093
2698499201,8cyclictest15232-21cat19:55:023
2698499201,6cyclictest3330-21/usr/sbin/munin21:55:203
2698499201,6cyclictest0-21swapper/322:00:183
2698499201,5cyclictest4937-21diskstats00:25:143
2698499201,5cyclictest19377-21sshd20:04:083
2698499201,16cyclictest3172-21ntp_states19:25:243
2698499201,16cyclictest22522-21head23:50:183
2698499201,16cyclictest1744-21unixbench_singl00:15:243
2698499201,10cyclictest0-21swapper/319:45:013
2698499200,1cyclictest0-21swapper/323:15:123
2698499200,1cyclictest0-21swapper/323:15:123
2698499200,10cyclictest0-21swapper/323:08:303
2698499200,10cyclictest0-21swapper/300:33:293
2697899206,3cyclictest0-21swapper/200:30:132
2697899201,9cyclictest3095-21latency_hist21:55:012
2697899201,7cyclictest3748-21/usr/sbin/munin21:55:162
2697399209,3cyclictest10937-21cpu19:45:111
2697399208,3cyclictest5876-21diskstats22:00:131
2697399207,4cyclictest29962-21fw_packets19:15:051
2697399207,4cyclictest10281-21diskstats22:10:141
2697399204,6cyclictest21-21rcuc/100:20:191
2697399201,7cyclictest0-21swapper/123:40:121
26973992016,3cyclictest0-21swapper/100:15:451
26973992013,1cyclictest0-21swapper/120:05:141
2697399201,2cyclictest0-21swapper/100:27:421
2697399201,14cyclictest0-21swapper/123:22:001
2697399201,14cyclictest0-21swapper/123:02:431
2697399201,14cyclictest0-21swapper/121:50:421
2697399201,11cyclictest31228-21sed00:10:181
2697399201,11cyclictest27342-21cat22:50:021
2697399200,14cyclictest0-21swapper/122:29:551
2696799209,3cyclictest22498-21/usr/sbin/munin21:25:150
2696799209,3cyclictest14103-21awk22:20:010
2696799207,4cyclictest87150irq/126-enp1s0-23:50:100
2696799204,4cyclictest770-21rngd22:57:360
2696799203,15cyclictest29500-21perf22:55:000
2696799201,9cyclictest800-21snmpd21:38:560
26967992014,4cyclictest17877-21ls21:15:000
2696799201,11cyclictest5528-21/usr/sbin/munin22:00:160
2696799201,11cyclictest32263-21/usr/sbin/munin19:20:210
2696799201,11cyclictest28151-21/usr/sbin/munin00:05:230
2696799201,11cyclictest1758-21/usr/sbin/munin23:05:160
2696799201,10cyclictest15681-21diskmemload23:20:190
2698499191,6cyclictest21885-21/usr/sbin/munin20:10:213
2698499191,5cyclictest29120-21gltestperf20:25:153
2698499191,15cyclictest27163-21sshd19:10:083
2698499191,15cyclictest26268-21sshd20:20:083
2698499191,15cyclictest12128-21/usr/sbin/munin22:15:113
2698499190,5cyclictest0-21swapper/322:09:093
2698499190,5cyclictest0-21swapper/322:09:093
2697899197,9cyclictest9093-21awk20:55:012
2697899191,9cyclictest13811-21ntp_states23:30:222
2697899191,6cyclictest8801-21ntp_states22:05:222
2697899191,6cyclictest8801-21ntp_states22:05:222
2697899191,6cyclictest16790-21grep19:55:242
2697899191,4cyclictest0-21swapper/223:27:092
2697899191,15cyclictest13712-21users22:15:242
2697399191,2cyclictest0-21swapper/120:55:271
26973991911,2cyclictest0-21swapper/122:34:121
2697399191,10cyclictest29105-21ls00:05:181
2697399190,13cyclictest0-21swapper/122:35:261
2697399190,13cyclictest0-21swapper/119:51:351
2696799199,4cyclictest10952-21munin-run00:40:010
2696799192,4cyclictest7-21ksoftirqd/000:30:260
2696799191,8cyclictest0-21swapper/021:48:580
2696799191,10cyclictest17479-21/usr/sbin/munin20:00:110
2696799191,10cyclictest15280-21/usr/sbin/munin19:55:250
2698499186,2cyclictest0-21swapper/300:10:123
2698499181,4cyclictest8349-21/usr/sbin/munin23:20:173
26984991812,4cyclictest31389-21cut21:45:123
26984991812,3cyclictest4401-21tr19:30:123
2698499181,15cyclictest0-21swapper/323:56:263
2697899182,8cyclictest7517-21awk22:05:012
2697899182,13cyclictest22189-21grep23:50:152
2697899181,8cyclictest0-21swapper/222:39:572
2697899181,6cyclictest19332-21/usr/sbin/munin23:45:162
2697899181,6cyclictest10086-21sshd23:22:102
2697899181,6cyclictest0-21swapper/222:20:192
2697899181,4cyclictest0-21swapper/200:05:202
26978991812,3cyclictest15295-21users21:05:252
2697899181,15cyclictest0-21swapper/219:45:272
2697899180,5cyclictest0-21swapper/219:44:302
2697899180,4cyclictest0-21swapper/219:33:082
2697399181,2cyclictest0-21swapper/123:34:131
26973991810,3cyclictest0-21swapper/120:02:591
2696799187,3cyclictest25937-21/usr/sbin/munin00:00:200
2696799185,2cyclictest7-21ksoftirqd/020:50:220
2696799184,2cyclictest25508-21kworker/0:000:17:040
2696799183,5cyclictest11-21rcuc/000:25:020
2696799182,7cyclictest16300-21munin-run22:25:010
2696799182,6cyclictest31065-21munin-run21:45:000
2696799182,5cyclictest16913-21perf23:40:020
2696799181,9cyclictest800-21snmpd19:39:280
2696799181,7cyclictest1727-21cat23:05:010
26967991813,3cyclictest9703-21munin-run22:10:010
26967991813,3cyclictest9703-21munin-run22:10:010
26967991811,4cyclictest32235-21sed20:30:260
2696799181,14cyclictest29715-21find20:25:230
26967991811,4cyclictest14943-21fgrep21:05:210
2696799181,11cyclictest0-21swapper/021:20:150
2696799181,10cyclictest800-21snmpd22:40:020
281162170,4sleep3401ktimersoftd/322:50:153
2698499172,3cyclictest8982-21/usr/sbin/munin00:35:233
2698499171,3cyclictest22498-21/usr/sbin/munin21:25:253
26984991712,3cyclictest29645-21unixbench_multi00:05:263
26984991712,3cyclictest11736-21chrt22:13:043
2698499171,14cyclictest0-21swapper/321:30:213
26984991711,3cyclictest9175-21wc20:55:003
26984991711,3cyclictest19570-21cat23:45:133
2697899178,7cyclictest22132-21kworker/u8:120:55:272
2697899172,12cyclictest18517-21cat22:30:012
2697899171,5cyclictest14148-21ntp_states19:50:222
2697899171,5cyclictest0-21swapper/200:00:192
2697899171,4cyclictest0-21swapper/222:50:232
26978991712,3cyclictest32265-21users20:30:262
26978991712,3cyclictest26993-21grep21:35:132
2697899171,15cyclictest0-21swapper/222:33:092
2697899171,14cyclictest0-21swapper/219:10:192
2697899170,4cyclictest0-21swapper/221:24:492
2697899170,15cyclictest0-21swapper/223:40:102
2697899170,15cyclictest0-21swapper/222:46:022
2696799177,2cyclictest12050irq/123-ahci[0020:10:010
2696799176,3cyclictest9307-21/usr/sbin/munin20:55:180
2696799174,2cyclictest11-21rcuc/023:15:190
2696799174,2cyclictest11-21rcuc/023:15:190
2696799172,7cyclictest87350irq/128-enp1s0-23:45:140
2696799171,9cyclictest0-21swapper/023:14:410
2696799171,3cyclictest0-21swapper/000:28:420
26967991713,3cyclictest7-21ksoftirqd/020:40:090
26967991713,3cyclictest0-21swapper/023:58:050
26967991713,2cyclictest15681-21diskmemload23:30:080
2696799171,13cyclictest3330-21/usr/sbin/munin21:55:160
26967991710,4cyclictest29166-21service19:15:020
2696799170,6cyclictest0-21swapper/021:01:100
2696799170,4cyclictest81ktimersoftd/019:40:080
256742171,2sleep20-21swapper/223:59:532
188562171,2sleep00-21swapper/023:41:590
2698499162,3cyclictest1560-21sshd19:21:093
2698499161,2cyclictest0-21swapper/321:15:233
2698499161,1cyclictest0-21swapper/321:24:003
26984991611,3cyclictest30985-21expr20:30:123
26984991610,3cyclictest23186-21cpuspeed_turbos22:40:113
26984991610,3cyclictest14049-21sh21:05:113
2697899169,5cyclictest12739-21sendmail_mailqu21:00:242
2697899166,9cyclictest32337-21awk00:15:002
2697899166,3cyclictest87450irq/129-enp1s0-00:35:232
2697899164,2cyclictest87450irq/129-enp1s0-21:25:172
2697899162,9cyclictest87150irq/126-enp1s0-20:25:172
2697899161,5cyclictest0-21swapper/222:40:242
2697899161,5cyclictest0-21swapper/200:25:172
2697899161,3cyclictest0-21swapper/223:00:172
2697899161,3cyclictest0-21swapper/220:35:172
2697899161,2cyclictest0-21swapper/219:22:582
26978991612,3cyclictest0-21swapper/220:00:402
26978991612,3cyclictest0-21swapper/219:36:172
2697899161,14cyclictest0-21swapper/220:21:052
26978991611,3cyclictest6151-21/usr/sbin/munin23:15:132
26978991611,3cyclictest6151-21/usr/sbin/munin23:15:132
26978991611,3cyclictest3203-21open_inodes19:25:252
26978991611,3cyclictest16407-21grep23:35:252
26978991611,3cyclictest11453-21unixbench_multi22:10:242
26978991610,4cyclictest0-21swapper/223:10:222
26978991610,3cyclictest19704-21cpu20:05:112
2696799166,3cyclictest87350irq/128-enp1s0-21:50:150
2696799163,2cyclictest11-21rcuc/023:25:230
2696799161,6cyclictest4296-21/usr/sbin/munin19:30:100
2696799161,6cyclictest24081-21/usr/sbin/munin20:15:170
26967991611,3cyclictest27231-21seq22:49:280
26984991510,3cyclictest7367-21sed19:35:203
2697899159,3cyclictest16462-21fschecks_time21:10:162
2697899158,4cyclictest3692-21/usr/sbin/munin20:40:232
2697899158,4cyclictest31754-21tune2fs21:45:152
2697899151,3cyclictest87450irq/129-enp1s0-21:15:242
26978991510,3cyclictest3070-21hddtemp_smartct00:20:162
2697899150,6cyclictest0-21swapper/223:05:162
2696799151,5cyclictest19761-21ntp_kernel_pll_22:30:200
2696799150,4cyclictest0-21swapper/019:15:120
2696799150,14cyclictest0-21swapper/022:25:210
2697899138,3cyclictest24990-21load20:15:192
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional