You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-01 - 02:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Fri May 01, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
326052219150,56sleep20-21swapper/219:07:342
326882203166,25sleep30-21swapper/319:08:413
326462202165,24sleep10-21swapper/119:08:061
327572201163,26sleep00-21swapper/019:09:340
2870221670,4sleep142299cyclictest20:10:221
162421640,3sleep041499cyclictest00:05:010
1010621140,5sleep110109-21unixbench_multi20:40:231
173522540,3sleep017356-21latency_hist21:00:020
43299484,31cyclictest7047-40dnf19:20:243
42299431,34cyclictest3202-21cut21:40:161
42299431,34cyclictest26043-21irqstats23:45:191
42299431,33cyclictest30400-21sed21:30:021
42299431,33cyclictest14662-21ntpq22:05:191
42299431,24cyclictest19987-21latency_hist22:20:011
43299419,3cyclictest27783-21/usr/sbin/munin20:10:273
42299411,31cyclictest12755-21chrt20:49:501
42299411,29cyclictest15982-21cpuspeed_turbos22:10:121
87902400,1sleep30-21swapper/319:25:183
42299401,26cyclictest14972-21latency_hist20:55:011
42299401,23cyclictest9641-21basename23:10:011
43299387,3cyclictest11904-21NetworkManager19:31:443
43299385,23cyclictest87350irq/126-enp1s0-00:00:163
42699382,16cyclictest0-21swapper/223:35:262
42699382,16cyclictest0-21swapper/223:35:262
42299388,18cyclictest8065-21cpu19:25:121
42299382,32cyclictest909-21sssd_nss19:55:191
42299381,34cyclictest4288-21awk00:15:021
42299381,33cyclictest13642-21mailstats19:35:211
42299374,22cyclictest21579-21diskmemload23:30:131
42299372,25cyclictest3659-21cpu19:15:121
42299371,28cyclictest20839-21/usr/sbin/munin23:35:161
42299371,28cyclictest20839-21/usr/sbin/munin23:35:151
42299371,26cyclictest8903-21/usr/sbin/munin00:25:261
42299371,24cyclictest31209-21/usr/sbin/munin22:45:201
42299371,23cyclictest30329-21diskstats20:15:131
42299371,23cyclictest0-21swapper/123:40:171
42299368,18cyclictest19172-21cstates19:50:121
42299366,20cyclictest0-21swapper/122:35:011
42299361,32cyclictest16581-21awk19:45:001
42299361,31cyclictest8669-21latency_hist00:25:021
42299361,31cyclictest19379-21latency_hist21:05:021
42299361,26cyclictest19610-21/usr/sbin/munin21:05:281
42299361,23cyclictest4552-21cat22:55:251
42299361,23cyclictest12267-21grep22:00:181
42299361,18cyclictest14556-21/usr/sbin/munin00:35:251
42299360,28cyclictest0-21swapper/120:05:011
61242350,2sleep00-21swapper/023:00:160
42299353,22cyclictest87550irq/128-enp1s0-23:25:111
42299352,28cyclictest27873-40dnf21:21:231
42299351,23cyclictest0-21swapper/123:50:191
42299351,21cyclictest24395-21munin-run22:30:011
42299351,21cyclictest24395-21munin-run22:30:011
42299351,16cyclictest4852-21sh00:15:151
43299344,26cyclictest24990-21cut22:30:173
43299342,5cyclictest3959-21diskstats19:15:133
432993412,19cyclictest8186-21proc_pri21:50:233
42299342,28cyclictest28675-21sshd22:38:471
42299341,29cyclictest790-21in:imjournal19:10:211
42299341,29cyclictest11588-21fschecks_time00:30:161
42299341,28cyclictest87650irq/129-enp1s0-00:08:071
42299341,23cyclictest0-21swapper/119:25:011
42299341,22cyclictest13422-21grep23:15:241
42299340,28cyclictest5659-21tr21:45:181
42299340,26cyclictest29625-21/usr/sbin/munin23:55:141
42299340,24cyclictest0-21swapper/121:15:021
41499346,14cyclictest752-21munin-run22:50:010
43299338,4cyclictest3286-21gltestperf21:40:163
432993312,18cyclictest22863-21ntp_offset21:10:193
432993310,20cyclictest1420-21netstat21:35:183
42699338,14cyclictest16596-21if_err_enp1s022:10:182
42699337,14cyclictest5892-21ntp_states21:45:212
426993312,10cyclictest29968-21df_inode23:55:132
426993311,12cyclictest14591-21diskstats23:20:132
42299332,28cyclictest11013-21seq21:57:481
42299332,24cyclictest25621-21kworker/u8:021:19:511
42299332,24cyclictest25621-21kworker/u8:021:19:511
42299331,26cyclictest4443-21kworker/u8:122:23:591
42299331,21cyclictest6316-21cat20:35:011
42299331,20cyclictest6761-21diskstats20:35:121
42299331,18cyclictest31822-21/usr/sbin/munin00:00:261
42299331,17cyclictest7591-21fschecks_time21:50:161
43299323,6cyclictest22448-21smart_sda19:55:253
43299321,9cyclictest808-21snmpd19:54:583
43299321,28cyclictest808-21snmpd00:32:473
43299321,27cyclictest12815-21fschecks_count19:35:143
432993212,17cyclictest2305-21/usr/sbin/munin00:10:123
42699329,9cyclictest19189-21proc_pri22:15:242
42699328,12cyclictest7036-21users23:00:272
426993210,6cyclictest21579-21diskmemload21:35:002
42299325,14cyclictest3336-21switchtime20:25:231
42299324,18cyclictest87650irq/129-enp1s0-22:50:131
42299321,28cyclictest18882-21cat19:50:011
42299321,27cyclictest11816-21crond23:15:001
42299321,22cyclictest29010-21/usr/sbin/munin22:40:261
41499323,15cyclictest27562-21munin-run20:10:010
43299319,18cyclictest1870-21cut22:50:183
43299318,3cyclictest26223-21needreboot23:45:193
43299317,3cyclictest14270-21smart_sda20:50:243
43299314,4cyclictest30897-21/usr/sbin/munin23:55:243
432993112,16cyclictest30701-21cat20:15:173
432993112,16cyclictest17924-21grep23:25:253
43299311,12cyclictest3093-21ntp_states20:25:213
432993110,3cyclictest15259-21latency_hist00:40:013
42699319,9cyclictest9004-21munin-run21:55:022
42299312,19cyclictest0-21swapper/123:00:161
42299311,23cyclictest25588-21/usr/sbin/munin20:05:101
42299311,22cyclictest0-21swapper/120:57:281
42299310,28cyclictest10203-21/usr/sbin/munin19:30:241
42299310,20cyclictest0-21swapper/121:35:161
43299309,3cyclictest7839-21cstates23:05:163
43299309,3cyclictest17157-21forks19:45:153
43299308,3cyclictest23146-21grep23:40:143
43299307,3cyclictest18983-21df_inode23:30:143
43299306,3cyclictest0-21swapper/322:24:483
43299306,3cyclictest0-21swapper/300:20:193
43299305,4cyclictest9170-21fschecks_time20:40:133
43299303,3cyclictest87350irq/126-enp1s0-22:45:023
43299301,6cyclictest5209-21needreboot20:30:203
43299301,4cyclictest21579-21diskmemload22:10:263
43299301,12cyclictest32223-21sshd21:30:473
432993011,16cyclictest14897-21expr19:40:133
432993010,3cyclictest827-21runrttasks21:01:403
432993010,3cyclictest25361-21dump-pmu-power20:05:013
432993010,17cyclictest26816-21/usr/sbin/munin22:35:183
432993010,17cyclictest24725-21cat21:15:183
432993010,17cyclictest24725-21cat21:15:183
42699308,11cyclictest30975-21latency_hist22:45:012
42699302,13cyclictest32644-21kernelversion00:00:192
426993013,13cyclictest7319-21grep19:20:252
42299301,26cyclictest31273-21hddtemp_smartct21:30:171
42299301,22cyclictest0-21swapper/123:21:241
43299299,4cyclictest827-21runrttasks21:20:143
43299299,2cyclictest827-21runrttasks21:28:143
43299297,3cyclictest22422-21/usr/sbin/munin22:25:173
43299297,3cyclictest22422-21/usr/sbin/munin22:25:173
43299297,2cyclictest827-21runrttasks20:59:483
43299296,3cyclictest827-21runrttasks22:08:063
43299292,5cyclictest9718-21irqstats00:25:213
43299291,8cyclictest735-21needreboot20:20:193
432992911,15cyclictest10170-21cat23:10:133
432992911,15cyclictest10159-21cat21:55:193
432992910,3cyclictest827-21runrttasks21:09:393
42699298,8cyclictest19379-21crond21:05:002
42699298,14cyclictest21579-21diskmemload22:20:242
42699290,18cyclictest0-21swapper/200:20:382
41499291,13cyclictest22813-21munin-run23:40:020
41499291,13cyclictest22813-21munin-run23:40:020
43299288,3cyclictest26057-21fschecks_time20:05:143
43299288,3cyclictest2095-21cat19:10:253
43299287,4cyclictest6537-21cut20:35:103
43299287,3cyclictest827-21runrttasks23:01:223
43299287,2cyclictest21505-21grep23:35:163
43299287,2cyclictest21505-21grep23:35:163
43299285,4cyclictest0-21swapper/323:50:273
43299284,4cyclictest4567-21cpu00:15:123
43299284,3cyclictest87350irq/126-enp1s0-23:00:023
43299281,5cyclictest21579-21diskmemload23:20:133
42699289,5cyclictest7656-21/usr/sbin/munin23:05:192
42699289,11cyclictest30304-21chrt21:29:212
42699288,3cyclictest0-21swapper/220:19:122
42699286,10cyclictest24301-21smart_sda23:40:262
42699285,14cyclictest18640-21/usr/sbin/munin23:30:252
426992810,4cyclictest12050irq/123-ahci[0019:45:012
42299280,26cyclictest0-21swapper/120:20:111
54352270,5sleep011-21rcuc/020:30:240
43299277,3cyclictest12449-21cut23:15:153
43299273,5cyclictest18372-21diskstats22:15:143
43299273,4cyclictest13404-21munin-run22:05:003
43299272,5cyclictest2089-21munin-run00:10:013
43299272,5cyclictest0-21swapper/320:45:233
432992711,13cyclictest5696-21cat21:45:173
42699278,5cyclictest21579-21diskmemload22:53:532
42699278,10cyclictest17356-21/usr/sbin/munin23:25:172
42699277,11cyclictest0-21swapper/220:45:212
42699275,10cyclictest87350irq/126-enp1s0-23:48:402
42699274,9cyclictest1016-21latency_hist22:50:012
42699272,14cyclictest0-21swapper/222:35:272
426992712,2cyclictest0-21swapper/220:25:142
426992710,6cyclictest15553-21diskstats20:55:142
42699270,16cyclictest0-21swapper/200:37:112
41499277,17cyclictest6808-21munin-run21:50:010
43299266,3cyclictest485-21seq22:47:053
42699267,7cyclictest755-21rngd23:54:412
42699266,7cyclictest28123-21df20:10:162
42699266,7cyclictest14533-21cat19:40:012
42699266,5cyclictest11689-21tail00:30:162
42699266,5cyclictest0-21swapper/221:35:172
42699266,11cyclictest25588-21/usr/sbin/munin20:05:182
42699264,13cyclictest16793-21/usr/sbin/munin19:45:202
42699262,10cyclictest13609-21latency_hist22:05:022
42699261,8cyclictest87350irq/126-enp1s0-00:18:502
426992610,8cyclictest9421-21irqstats20:40:162
41499261,13cyclictest4284-21munin-run00:15:010
42699257,7cyclictest5211-21needreboot20:30:202
42699257,6cyclictest25781-21taskset21:17:412
42699257,6cyclictest25781-21taskset21:17:412
42699257,5cyclictest87650irq/129-enp1s0-21:55:222
42699256,9cyclictest12050irq/123-ahci[0023:12:462
42699256,16cyclictest827-21runrttasks23:19:182
42699256,12cyclictest3994-21taskset00:11:302
42699256,11cyclictest0-21swapper/220:55:002
42699256,10cyclictest14543-21memory22:05:192
42699252,14cyclictest0-21swapper/221:40:502
42699251,9cyclictest739-21cpuspeed_turbos19:10:122
42699251,10cyclictest21986-21cstates21:10:112
42699250,19cyclictest22305-1kworker/u9:222:55:312
41499256,16cyclictest30373-21munin-run21:30:010
41499251,5cyclictest0-21swapper/021:38:570
41499251,11cyclictest28797-21awk22:40:010
86852240,1sleep20-21swapper/219:25:172
42699247,9cyclictest12050irq/123-ahci[0021:05:152
42699246,5cyclictest87650irq/129-enp1s0-22:30:012
42699246,5cyclictest87650irq/129-enp1s0-22:30:012
42699244,12cyclictest0-21swapper/200:08:382
42699244,11cyclictest87350irq/126-enp1s0-19:55:162
42699244,10cyclictest87650irq/129-enp1s0-22:33:362
42699243,6cyclictest808-21snmpd21:20:142
42699239,11cyclictest20965-21munin-run19:55:002
42699235,10cyclictest87650irq/129-enp1s0-19:30:212
41499231,12cyclictest13042-21cut23:15:200
41499230,11cyclictest16578-21awk19:45:000
42699224,4cyclictest12050irq/123-ahci[0019:20:012
426992213,6cyclictest833-21/usr/sbin/munin20:20:192
426992213,6cyclictest24105-21/usr/sbin/munin20:00:162
426992212,7cyclictest10414-21grep00:25:282
426992211,4cyclictest87350irq/126-enp1s0-20:35:192
41499221,5cyclictest0-21swapper/021:59:540
41499221,16cyclictest0-21swapper/023:47:110
41499221,12cyclictest10139-21sendmail_mailqu00:25:260
41499221,11cyclictest0-21swapper/023:25:120
41499219,3cyclictest0-21swapper/021:09:530
41499216,6cyclictest0-21swapper/021:51:150
41499214,3cyclictest0-21swapper/022:16:060
41499212,16cyclictest15603-21munin-run22:10:000
414992114,4cyclictest11827-21ls23:15:010
41499211,2cyclictest87650irq/129-enp1s0-23:31:520
41499211,11cyclictest24919-21memory21:15:200
41499211,11cyclictest24919-21memory21:15:200
41499210,5cyclictest0-21swapper/022:03:270
41499209,3cyclictest7962-21smart_sda00:20:230
41499201,3cyclictest0-21swapper/023:22:510
41499201,11cyclictest30596-21/usr/sbin/munin21:30:180
41499201,11cyclictest21188-21/usr/sbin/munin19:55:240
41499201,10cyclictest30971-21grep20:15:200
41499201,10cyclictest22762-21df_inode22:25:120
41499201,10cyclictest22762-21df_inode22:25:120
41499201,10cyclictest0-21swapper/022:40:120
41499195,7cyclictest0-21swapper/000:33:180
41499194,6cyclictest0-21swapper/022:20:300
41499194,3cyclictest12050irq/123-ahci[0022:55:010
41499191,9cyclictest4397-21pmu-power22:55:230
41499191,7cyclictest18250-21kworker/u8:200:08:510
41499191,7cyclictest0-21swapper/023:44:560
414991914,3cyclictest23772-21idleruntime-cro21:15:000
41499191,15cyclictest2229-21grep19:10:270
41499191,10cyclictest4506-21/usr/sbin/munin00:15:120
41499191,10cyclictest13007-21/usr/sbin/munin20:50:130
154412190,5sleep011-21rcuc/000:40:010
41499185,11cyclictest9-21rcu_preempt21:40:170
41499181,7cyclictest0-21swapper/019:20:260
41499181,5cyclictest0-21swapper/022:31:390
41499181,3cyclictest24144-21if_enp2s020:00:180
41499181,3cyclictest0-21swapper/022:10:400
41499181,15cyclictest0-21swapper/020:41:090
41499181,10cyclictest0-21swapper/023:51:120
41499181,10cyclictest0-21swapper/019:45:250
41499181,10cyclictest0-21swapper/019:30:250
41499180,4cyclictest0-21swapper/021:20:040
41499180,10cyclictest0-21swapper/023:55:130
41499171,7cyclictest11-21rcuc/020:10:190
41499171,6cyclictest8582-21fschecks_time19:25:150
41499171,6cyclictest0-21swapper/020:20:140
41499170,9cyclictest0-21swapper/019:52:010
41499170,5cyclictest0-21swapper/023:05:110
41499163,11cyclictest11-21rcuc/019:15:120
41499162,5cyclictest0-21swapper/020:35:210
41499161,7cyclictest827-21runrttasks20:26:010
414991612,3cyclictest0-21swapper/019:38:250
190662161,2sleep00-21swapper/021:01:060
124562151,3sleep00-21swapper/020:46:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional