You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-16 - 12:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Tue Dec 16, 2025 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
72952236162,24sleep30-21swapper/319:09:313
71122236162,24sleep10-21swapper/119:07:071
71202217164,41sleep20-21swapper/219:07:132
71952202169,22sleep00-21swapper/019:08:110
7421999490,3cyclictest0-21swapper/121:35:231
7421996557,2cyclictest0-21swapper/122:02:471
7421996258,3cyclictest0-21swapper/119:55:361
217932600,1sleep20-21swapper/200:35:122
72942590,2sleep20-21swapper/200:00:212
217242520,2sleep20-21swapper/222:05:232
217242520,2sleep20-21swapper/222:05:232
155032470,1sleep10-21swapper/100:20:141
202872450,5sleep3743399cyclictest19:35:193
209902440,4sleep011-21rcuc/023:20:000
209902440,4sleep011-21rcuc/023:20:000
742199400,16cyclictest0-21swapper/123:55:181
742199361,3cyclictest13100-21grep20:35:011
742199361,11cyclictest21126-21head00:30:231
742199361,11cyclictest21126-21head00:30:231
7421993513,3cyclictest1088-21df21:20:151
741499359,12cyclictest3948-21munin-run23:55:020
741499348,11cyclictest14975-21awk00:20:010
741499346,13cyclictest432-21idleruntime-cro21:20:010
54362340,4sleep341-21ksoftirqd/321:30:133
742199338,4cyclictest23161-21cat23:25:011
742199338,4cyclictest15055-40dnf20:37:091
742199332,11cyclictest22338-21perl00:35:161
742199332,11cyclictest0-21swapper/100:03:081
742199321,4cyclictest9531-21unixbench_singl22:50:251
741499315,12cyclictest28506-21perf21:10:010
741499313,12cyclictest29731-21idleruntime-cro23:40:010
149522310,2sleep30-21swapper/323:05:163
742199304,3cyclictest0-21swapper/119:27:161
741499291,13cyclictest5536-21munin-run22:45:020
7414992910,16cyclictest6498-21munin-run20:20:020
741499282,13cyclictest3340-21munin-run22:40:020
741499281,12cyclictest31313-21awk22:30:020
161512280,3sleep341-21ksoftirqd/321:55:133
742199279,3cyclictest12555-21irqstats21:45:181
7414992710,14cyclictest10589-21perf00:10:010
742199261,7cyclictest0-21swapper/123:14:081
7414992622,2cyclictest7-21ksoftirqd/020:50:190
43942260,1sleep00-21swapper/023:55:130
125622260,1sleep20-21swapper/219:20:012
109462260,1sleep10-21swapper/121:40:261
743399251,9cyclictest27233-21cpuspeed_turbos22:20:123
742199251,12cyclictest21796-21taskset20:54:041
7414992520,2cyclictest11-21rcuc/022:10:240
316662250,2sleep011-21rcuc/022:30:130
265772250,2sleep30-21swapper/323:30:203
743399241,20cyclictest800-21snmpd22:05:273
743399241,20cyclictest800-21snmpd22:05:273
743399241,16cyclictest87150irq/126-enp1s0-20:25:333
743399240,21cyclictest5670-21pmu-power20:15:223
7427992410,5cyclictest311ktimersoftd/219:15:012
742199249,3cyclictest1364-21chrt22:35:121
742199245,2cyclictest87350irq/128-enp1s0-19:50:131
742199241,13cyclictest0-21swapper/120:56:161
742199241,13cyclictest0-21swapper/120:56:161
289782240,3sleep1742199cyclictest23:35:231
743399239,3cyclictest17106-21/usr/sbin/munin19:30:223
743399231,20cyclictest0-21swapper/319:40:183
743399231,19cyclictest869-21sssd_nss23:39:093
743399231,11cyclictest821-21lldpd21:47:303
743399231,10cyclictest8064-21basename21:35:163
743399230,21cyclictest0-21swapper/323:20:173
742199239,12cyclictest2690-21kworker/u8:021:32:411
742199231,18cyclictest19357-21munin-run00:30:011
742199231,18cyclictest0-21swapper/119:46:071
742199231,18cyclictest0-21swapper/119:46:071
742199231,16cyclictest0-21swapper/120:18:001
742199231,10cyclictest0-21swapper/121:27:561
741499238,2cyclictest8289-21sed20:20:260
7414992319,2cyclictest11-21rcuc/000:10:200
743399223,16cyclictest41-21ksoftirqd/322:30:203
743399221,10cyclictest29946-21/usr/sbin/munin23:40:133
743399221,10cyclictest23439-21sshd00:36:083
743399221,10cyclictest16514-21head20:40:193
743399221,10cyclictest14254-21sshd23:04:083
743399220,20cyclictest0-21swapper/323:45:133
7427992213,6cyclictest28263-21systemd-journal21:07:172
742799220,3cyclictest87450irq/129-enp1s0-22:10:232
742199228,4cyclictest27134-21users23:30:261
742199226,3cyclictest0-21swapper/122:58:321
742199226,3cyclictest0-21swapper/122:34:281
742199226,3cyclictest0-21swapper/119:11:401
742199225,3cyclictest0-21swapper/121:56:511
742199223,7cyclictest11486-21cat19:15:201
742199223,2cyclictest0-21swapper/123:45:311
7421992217,2cyclictest103782sleep100:07:381
742199221,16cyclictest0-21swapper/120:08:021
742199221,11cyclictest0-21swapper/121:50:521
742199221,10cyclictest0-21swapper/122:48:241
742199221,10cyclictest0-21swapper/121:06:361
742199220,15cyclictest0-21swapper/119:37:091
741499224,15cyclictest11-21rcuc/019:35:240
741499221,11cyclictest800-21snmpd19:55:010
741499221,10cyclictest0-21swapper/019:15:190
7414992210,8cyclictest0-21swapper/021:10:220
741499220,15cyclictest0-21swapper/021:20:260
117182221,2sleep10-21swapper/100:10:201
743399214,5cyclictest770-21rngd00:10:253
743399213,15cyclictest39-21rcuc/323:55:273
743399212,3cyclictest9105-21grep19:10:263
743399211,9cyclictest800-21snmpd23:10:403
743399211,9cyclictest39-21rcuc/322:55:243
743399211,9cyclictest28490-21diskmemload22:15:223
743399211,17cyclictest19908-21sshd20:50:083
743399210,19cyclictest0-21swapper/322:38:083
743399210,18cyclictest0-21swapper/321:40:283
742799213,8cyclictest311ktimersoftd/221:50:012
742199216,3cyclictest0-21swapper/122:18:561
742199215,4cyclictest0-21swapper/123:42:071
742199215,3cyclictest0-21swapper/121:19:521
7421992117,3cyclictest0-21swapper/122:14:111
7421992113,3cyclictest0-21swapper/119:21:251
742199211,17cyclictest8484-21seq20:22:231
742199211,17cyclictest310922sleep122:27:291
742199211,14cyclictest0-21swapper/123:52:091
742199211,14cyclictest0-21swapper/122:40:201
742199211,14cyclictest0-21swapper/120:01:111
742199211,11cyclictest2479-21tr20:10:121
741499218,2cyclictest0-21swapper/023:30:150
741499217,3cyclictest800-21snmpd20:35:090
741499217,3cyclictest13473-21sshd21:47:080
741499217,3cyclictest10462-21memory21:40:180
741499211,7cyclictest0-21swapper/021:37:580
7414992117,2cyclictest11-21rcuc/020:55:170
7414992117,2cyclictest11-21rcuc/020:55:170
741499211,11cyclictest15373-21irqstats23:05:180
741499211,11cyclictest0-21swapper/022:00:160
741499211,10cyclictest0-21swapper/021:50:230
741499210,9cyclictest25204-21grep19:45:240
741499210,9cyclictest25204-21grep19:45:230
743399207,3cyclictest821-21lldpd20:59:583
743399207,3cyclictest821-21lldpd20:59:583
743399206,4cyclictest12050irq/123-ahci[0019:50:013
743399206,4cyclictest12050irq/123-ahci[0019:50:013
743399203,7cyclictest39-21rcuc/322:45:253
743399202,10cyclictest19585-21/usr/sbin/munin00:30:173
743399202,10cyclictest19585-21/usr/sbin/munin00:30:163
743399201,9cyclictest0-21swapper/320:30:253
743399201,6cyclictest14958-21latency_hist00:20:013
743399201,12cyclictest0-21swapper/321:21:093
743399201,10cyclictest0-21swapper/321:09:583
743399201,10cyclictest0-21swapper/300:29:313
743399200,14cyclictest0-21swapper/319:15:333
742799209,3cyclictest87350irq/128-enp1s0-20:05:092
742799208,3cyclictest12050irq/123-ahci[0022:55:012
742799204,8cyclictest87150irq/126-enp1s0-20:15:132
742799203,7cyclictest18334-21missed_timers00:25:182
7427992010,2cyclictest87150irq/126-enp1s0-23:20:152
742199208,4cyclictest25487-21ps21:00:231
742199204,3cyclictest0-21swapper/119:30:561
742199202,3cyclictest26391-21kworker/u8:120:45:321
742199201,9cyclictest0-21swapper/120:28:041
7421992012,2cyclictest0-21swapper/100:15:071
742199201,10cyclictest12356-21/usr/sbin/munin23:00:271
7421992010,3cyclictest20917-21latency_hist23:20:021
7421992010,3cyclictest20917-21latency_hist23:20:021
742199200,9cyclictest0-21swapper/119:40:481
741499209,3cyclictest54372expr21:30:120
741499208,4cyclictest21356-21irqstats22:05:180
741499208,4cyclictest21356-21irqstats22:05:180
741499206,3cyclictest87350irq/128-enp1s0-23:45:190
741499205,4cyclictest0-21swapper/000:00:260
741499204,3cyclictest0-21swapper/020:49:140
741499201,9cyclictest0-21swapper/022:20:220
741499201,16cyclictest1126-21head20:05:190
741499201,11cyclictest2951-21/usr/sbin/munin21:25:200
741499200,18cyclictest0-21swapper/020:14:090
99182190,15sleep30-21swapper/322:55:003
743399196,3cyclictest87150irq/126-enp1s0-00:05:143
743399196,2cyclictest24316-21/usr/sbin/munin21:00:173
743399195,3cyclictest12050irq/123-ahci[0000:05:013
743399194,3cyclictest12050irq/123-ahci[0019:55:233
743399192,13cyclictest22860-21tr22:10:143
743399191,7cyclictest0-21swapper/321:53:313
743399191,6cyclictest3829-21cstates22:40:113
743399191,6cyclictest3142-21cstates21:25:133
743399191,6cyclictest30815-21cut20:00:153
743399191,6cyclictest20318-21munin-run22:05:013
743399191,6cyclictest12731-21egrep19:20:093
743399191,6cyclictest0-21swapper/321:10:183
743399191,5cyclictest24937-21users23:25:273
743399191,5cyclictest19049-21cpuspeed_turbos23:15:143
743399191,5cyclictest19049-21cpuspeed_turbos23:15:143
743399191,10cyclictest0-21swapper/319:29:403
743399190,10cyclictest0-21swapper/320:23:593
742799198,3cyclictest87150irq/126-enp1s0-20:49:252
742799195,3cyclictest311ktimersoftd/221:15:172
742799194,7cyclictest12050irq/123-ahci[0023:55:022
742799194,4cyclictest32-21ksoftirqd/222:20:152
742799193,3cyclictest7740-21perf22:50:022
742799192,8cyclictest87150irq/126-enp1s0-21:40:012
7427991914,4cyclictest32-21ksoftirqd/220:35:012
742799191,12cyclictest0-21swapper/219:25:252
742199199,3cyclictest17486-21latency_hist20:45:011
742199194,3cyclictest0-21swapper/121:13:581
742199191,10cyclictest29100-21latency_hist22:25:021
742199191,10cyclictest20564-21egrep22:05:121
742199191,10cyclictest20564-21egrep22:05:121
742199190,17cyclictest0-21swapper/123:06:381
741499199,2cyclictest28091-21/usr/sbin/munin19:55:160
741499197,1cyclictest0-21swapper/019:10:220
741499196,3cyclictest87350irq/128-enp1s0-23:20:170
741499196,2cyclictest800-21snmpd19:29:550
741499195,3cyclictest87350irq/128-enp1s0-20:00:190
741499194,4cyclictest20612-21awk00:30:180
741499194,4cyclictest20612-21awk00:30:180
741499191,6cyclictest30009-21awk23:40:120
741499191,5cyclictest22105-21head19:40:140
741499191,5cyclictest19087-21awk19:35:000
741499191,4cyclictest26057-21sshd21:02:090
743399187,3cyclictest0-21swapper/321:15:083
743399184,3cyclictest87150irq/126-enp1s0-19:50:263
743399184,3cyclictest32573-21cpuspeed_turbos20:05:123
743399181,5cyclictest19681-21latency_hist20:50:003
742799186,4cyclictest12050irq/123-ahci[0020:55:012
742799183,9cyclictest30-21rcuc/223:35:132
742799183,4cyclictest25541-21sshd23:30:082
742799182,13cyclictest16184-21needreboot00:20:192
742799181,9cyclictest311ktimersoftd/220:30:152
742799181,8cyclictest0-21swapper/222:35:232
742799181,7cyclictest22113-21/usr/sbin/munin20:55:132
742799181,7cyclictest22113-21/usr/sbin/munin20:55:132
742799181,6cyclictest12708-21sa119:20:092
742799181,6cyclictest0-21swapper/200:10:242
7427991811,4cyclictest4516-21df_inode23:55:142
742799181,11cyclictest0-21swapper/222:30:162
7427991810,5cyclictest3842-21grep20:10:262
7421991814,3cyclictest0-21swapper/123:25:321
741499184,3cyclictest10971-21irqstats22:55:190
741499182,4cyclictest10892-21latency_hist20:30:020
741499181,9cyclictest800-21snmpd22:18:560
741499181,7cyclictest11129-21/usr/sbin/munin20:30:150
7414991816,1cyclictest0-21swapper/000:25:150
741499181,5cyclictest0-21swapper/022:50:210
7414991813,3cyclictest16956-21kernelversion21:55:190
741499181,14cyclictest23064-21grep00:35:240
741499181,14cyclictest15195-21/usr/sbin/munin00:20:260
741499180,16cyclictest0-21swapper/022:45:190
299822180,1sleep20-21swapper/221:10:262
155192181,2sleep00-21swapper/020:40:100
140192181,2sleep00-21swapper/023:01:080
743399172,3cyclictest2741-21if_enp2s023:50:173
743399171,5cyclictest0-21swapper/300:21:323
743399171,4cyclictest30400-21ntp_states22:25:213
7433991711,3cyclictest4345-21latency_hist20:15:023
742799179,2cyclictest87150irq/126-enp1s0-22:15:212
742799176,8cyclictest20724-21pmu-power00:30:192
742799176,8cyclictest20724-21pmu-power00:30:192
742799173,6cyclictest30-21rcuc/221:40:162
742799171,8cyclictest12050irq/123-ahci[0023:30:012
742799171,7cyclictest635-21missed_timers23:45:192
742799171,7cyclictest3565-21/usr/sbin/munin22:40:232
742799171,7cyclictest19299-21/usr/sbin/munin19:35:162
742799171,6cyclictest2730-21perf21:25:022
742799171,5cyclictest0-21swapper/222:00:202
7427991713,3cyclictest32-21ksoftirqd/223:40:252
7427991713,3cyclictest0-21swapper/219:47:172
7427991713,3cyclictest0-21swapper/219:47:172
7427991711,3cyclictest29676-21df_inode22:25:122
7427991711,3cyclictest28155-21cpuspeed_turbos19:55:122
7427991711,3cyclictest24316-21/usr/sbin/munin21:00:142
742799171,10cyclictest0-21swapper/223:02:272
7427991710,4cyclictest0-21swapper/223:10:132
742799170,7cyclictest15204-21smart_sda21:50:242
741499171,6cyclictest821-21lldpd19:21:580
741499171,3cyclictest18492-21sshd23:12:080
7414991712,3cyclictest23351-21/usr/sbin/munin23:25:240
7433991613,2cyclictest0-21swapper/320:35:063
742799162,4cyclictest8777-21/usr/sbin/munin00:05:122
742799162,3cyclictest311ktimersoftd/220:40:142
742799161,9cyclictest0-21swapper/221:30:152
742799161,6cyclictest18958-21/usr/sbin/munin23:15:252
742799161,6cyclictest18958-21/usr/sbin/munin23:15:252
742799161,6cyclictest0-21swapper/219:43:092
742799161,2cyclictest0-21swapper/200:19:142
7427991612,3cyclictest0-21swapper/220:28:582
7427991612,3cyclictest0-21swapper/219:30:162
7427991611,3cyclictest32-21ksoftirqd/219:50:202
7427991610,3cyclictest10840-21sh22:55:172
45742160,2sleep20-21swapper/221:25:472
742799152,4cyclictest15118-21perl23:05:162
742799151,6cyclictest0-21swapper/220:00:062
7427991510,3cyclictest7611-21wc20:20:182
742799150,6cyclictest0-21swapper/221:55:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional