You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-20 - 03:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri Mar 20, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
200412236163,23sleep20-21swapper/219:07:112
201712220190,20sleep00-21swapper/019:08:490
201072203165,25sleep30-21swapper/319:08:003
200782203166,24sleep10-21swapper/119:07:381
2037099106102,3cyclictest0-21swapper/300:01:463
2037099105102,2cyclictest0-21swapper/321:02:223
20370999592,2cyclictest0-21swapper/322:43:313
20370998986,2cyclictest0-21swapper/320:38:383
20370998784,2cyclictest0-21swapper/323:46:163
34722490,3sleep30-21swapper/323:22:103
2035399316,12cyclictest854-21munin-run22:05:010
228882300,4sleep22036399cyclictest22:55:002
2035399302,13cyclictest4037-21idleruntime-cro19:45:010
20370992915,5cyclictest41-21ksoftirqd/323:00:003
2035399293,12cyclictest20063-21perf21:35:000
2035799271,19cyclictest0-21swapper/122:39:311
2035399275,10cyclictest27622-21perf19:25:010
2035399271,11cyclictest23851-21idleruntime-cro20:30:000
2035799261,17cyclictest9470-21taskset22:22:271
2035399261,12cyclictest16897-21munin-run23:55:000
25032250,4sleep12035799cyclictest23:20:171
2035799256,9cyclictest18458-21fschecks_count21:30:151
2035799254,11cyclictest31017-21fschecks_count20:45:151
2035799254,11cyclictest31017-21fschecks_count20:45:141
2035799251,23cyclictest0-21swapper/120:05:131
2035799251,17cyclictest0-21swapper/123:25:251
2035799251,16cyclictest9065-21diskmemload23:35:111
2035399251,10cyclictest0-21swapper/021:00:270
199702250,4sleep07-21ksoftirqd/020:20:120
137522251,20sleep013864-21/usr/sbin/munin20:05:160
2035799241,17cyclictest0-21swapper/122:28:111
2035799240,18cyclictest0-21swapper/122:53:411
2035399245,16cyclictest14092-21awk22:35:020
2037099231,9cyclictest12050irq/123-ahci[0019:45:013
2037099231,13cyclictest0-21swapper/323:05:253
2037099231,13cyclictest0-21swapper/323:05:253
20357992314,3cyclictest0-21swapper/100:03:101
2035799231,16cyclictest0-21swapper/123:16:401
2035799231,15cyclictest0-21swapper/121:55:211
2035799231,15cyclictest0-21swapper/121:55:211
20357992311,3cyclictest18044-21unixbench_singl22:40:251
2035799231,11cyclictest7492-21munin-run22:20:011
2035799231,11cyclictest26076-21cpuspeed_turbos00:15:121
2035399231,10cyclictest284392ls00:20:130
2035399231,10cyclictest284392ls00:20:130
2037099229,3cyclictest41-21ksoftirqd/300:40:013
2037099225,9cyclictest86550irq/126-enp1s0-00:15:143
2037099221,9cyclictest17297-21cstates23:55:143
2037099221,6cyclictest19145-21ntp_states21:30:223
2037099221,12cyclictest39-21rcuc/322:10:153
2037099221,10cyclictest0-21swapper/321:20:513
20363992210,3cyclictest17757-21kworker/u8:122:28:572
2035799221,16cyclictest0-21swapper/123:54:271
2035799221,16cyclictest0-21swapper/122:45:591
2035799221,16cyclictest0-21swapper/121:21:391
20357992210,3cyclictest23787-21diskstats19:15:131
2035799220,16cyclictest0-21swapper/100:35:221
2035799220,15cyclictest0-21swapper/123:58:191
2035399226,5cyclictest0-21swapper/023:27:250
130162221,1sleep20-21swapper/223:45:152
273032210,2sleep30-21swapper/323:05:013
269822210,3sleep22036399cyclictest21:50:122
2037099219,2cyclictest401ktimersoftd/319:50:213
2037099217,3cyclictest20903-21/usr/sbin/munin22:50:253
2037099215,9cyclictest5719-1kworker/u9:223:18:593
2037099212,6cyclictest401ktimersoftd/300:05:213
2036399211,6cyclictest24388-21grep00:10:142
20363992115,3cyclictest8979-21gltestperf23:35:162
2036399210,5cyclictest24813-21kworker/u8:023:06:082
2036399210,5cyclictest24813-21kworker/u8:023:06:082
2035799219,3cyclictest30419-21latency_hist20:45:011
2035799217,6cyclictest0-21swapper/122:55:091
2035799213,7cyclictest25442-21cpuspeed_turbos23:00:121
20357992117,3cyclictest0-21swapper/119:10:061
20357992113,3cyclictest0-21swapper/123:10:031
20357992113,3cyclictest0-21swapper/122:33:581
20357992113,2cyclictest0-21swapper/122:05:071
2035799211,15cyclictest0-21swapper/123:49:041
2035799211,15cyclictest0-21swapper/123:44:541
2035799211,15cyclictest0-21swapper/122:14:001
2035799211,15cyclictest0-21swapper/121:44:511
2035799211,15cyclictest0-21swapper/100:25:401
2035799211,14cyclictest0-21swapper/100:09:061
2035799211,12cyclictest0-21swapper/121:51:231
2035799211,11cyclictest28083-21perl23:05:151
2035799211,11cyclictest28083-21perl23:05:151
2035799211,11cyclictest1264-21if_err_enp2s020:50:181
2035399217,4cyclictest1177-21sort20:50:160
2035399215,6cyclictest0-21swapper/023:33:350
2035399214,6cyclictest0-21swapper/021:39:540
2035399214,5cyclictest6093-21perl22:15:140
2035399212,11cyclictest9910-21/usr/sbin/munin22:25:200
2035399211,6cyclictest0-21swapper/021:18:030
2035399211,6cyclictest0-21swapper/000:09:180
2035399211,10cyclictest0-21swapper/023:12:090
2037099209,5cyclictest401ktimersoftd/321:35:163
2037099201,7cyclictest0-21swapper/321:27:413
2037099201,6cyclictest401ktimersoftd/323:55:023
20370992015,3cyclictest25523-21grep20:30:263
2037099201,3cyclictest41-21ksoftirqd/323:30:013
2037099201,11cyclictest8661-21/usr/sbin/munin19:55:153
2037099201,10cyclictest0-21swapper/322:15:203
2036399203,5cyclictest31012-21sshd21:59:482
2036399203,5cyclictest31012-21sshd21:59:482
2036399203,4cyclictest86550irq/126-enp1s0-20:15:242
2036399201,8cyclictest0-21swapper/219:35:102
2036399201,15cyclictest28174-21/usr/sbin/munin19:25:132
2036399201,12cyclictest0-21swapper/223:33:532
2036399201,12cyclictest0-21swapper/220:20:192
2036399201,10cyclictest0-21swapper/222:05:402
20357992016,3cyclictest0-21swapper/119:56:131
20357992014,3cyclictest16826-21unixbench_singl20:10:261
2035799201,2cyclictest0-21swapper/121:00:061
2035799201,15cyclictest0-21swapper/122:02:191
2035799201,14cyclictest0-21swapper/121:39:501
2035799201,13cyclictest86850irq/129-enp1s0-21:16:261
2035799201,11cyclictest3024-21missed_timers19:40:181
2035799201,10cyclictest11257-21latency_hist21:15:021
2035799200,2cyclictest86850irq/129-enp1s0-00:20:441
2035799200,2cyclictest86850irq/129-enp1s0-00:20:441
2035799200,18cyclictest0-21swapper/121:25:191
2035399206,3cyclictest0-21swapper/023:56:330
2035399205,6cyclictest0-21swapper/021:58:540
2035399205,6cyclictest0-21swapper/021:58:540
2035399204,14cyclictest29492-21perf23:10:020
2035399204,14cyclictest29492-21perf23:10:010
2035399202,10cyclictest14304-21/usr/sbin/munin22:35:250
2035399201,9cyclictest32450-21grep00:30:020
20353992014,3cyclictest13347-21grep23:45:160
2035399201,11cyclictest13688-21/usr/sbin/munin21:20:250
2035399201,11cyclictest1099-21/usr/sbin/munin22:05:240
211432191,2sleep00-21swapper/000:03:020
2037099198,3cyclictest28454-21/usr/sbin/munin20:40:193
2037099191,7cyclictest0-21swapper/323:35:193
2037099191,6cyclictest0-21swapper/322:27:363
2037099191,5cyclictest39-21rcuc/323:40:133
20370991914,3cyclictest29221-21tr21:55:133
20370991914,3cyclictest29221-21tr21:55:133
2037099191,12cyclictest0-21swapper/321:44:023
2037099191,10cyclictest18700-21/usr/sbin/munin22:45:263
2037099191,10cyclictest15268-21/usr/sbin/munin20:10:213
2037099190,10cyclictest401ktimersoftd/322:00:163
2036399195,7cyclictest20655-21runrttasks22:49:552
2036399195,3cyclictest86550irq/126-enp1s0-21:20:142
2036399192,5cyclictest21606-21chrt19:10:202
2036399191,8cyclictest23847-21awk20:30:002
2036399191,7cyclictest32425-21/usr/sbin/munin23:15:142
2036399191,7cyclictest1147-21head00:30:202
20363991913,3cyclictest21536-21/usr/sbin/munin00:05:172
2036399191,11cyclictest0-21swapper/200:00:102
2035799199,2cyclictest5497-21sendmail_mailqu19:45:231
2035799197,3cyclictest812-21snmpd20:55:081
2035799192,9cyclictest0-21swapper/120:25:221
20357991915,3cyclictest0-21swapper/121:08:431
20357991915,3cyclictest0-21swapper/119:51:261
2035799191,2cyclictest9065-21diskmemload00:33:231
2035799191,17cyclictest0-21swapper/120:36:191
2035799191,17cyclictest0-21swapper/119:23:471
2035799191,16cyclictest0-21swapper/119:30:471
2035799191,16cyclictest0-21swapper/119:30:471
2035799191,15cyclictest24404-21df_inode20:30:141
2035799191,14cyclictest0-21swapper/100:13:211
2035799191,10cyclictest24896-21expr21:45:121
2035799191,10cyclictest11899-21/usr/sbin/munin20:00:211
2035799190,2cyclictest0-21swapper/123:30:541
2035399197,3cyclictest0-21swapper/022:52:270
2035399196,4cyclictest0-21swapper/022:56:590
2035399196,4cyclictest0-21swapper/022:22:170
2035399191,15cyclictest8040-21grep19:50:240
2035399191,11cyclictest0-21swapper/023:00:590
2035399191,10cyclictest0-21swapper/021:52:100
2035399191,10cyclictest0-21swapper/021:43:120
2035399191,10cyclictest0-21swapper/000:14:320
2035399190,9cyclictest0-21swapper/020:31:320
2035399190,6cyclictest0-21swapper/023:36:420
56252181,2sleep20-21swapper/223:26:412
5152181,2sleep20-21swapper/222:01:372
316512181,2sleep20-21swapper/223:14:562
2037099189,3cyclictest2218-21kworker/3:121:13:113
2037099184,6cyclictest771-21rngd21:52:103
2037099183,6cyclictest30650-21/usr/sbin/munin20:45:223
2037099183,6cyclictest30650-21/usr/sbin/munin20:45:223
2037099181,8cyclictest24902-21grep19:15:253
2037099181,8cyclictest18855-21cat20:15:273
2037099181,7cyclictest0-21swapper/300:30:133
2037099181,11cyclictest0-21swapper/322:30:223
2037099181,11cyclictest0-21swapper/321:18:143
2036399184,3cyclictest88350irq/133-enp2s0-20:49:292
2036399184,3cyclictest88350irq/133-enp2s0-20:49:292
2036399183,9cyclictest0-21swapper/223:41:072
2036399183,6cyclictest32-21ksoftirqd/220:10:122
2036399183,4cyclictest24353-21which22:55:232
2036399182,7cyclictest22480-21/usr/sbin/munin21:40:262
2036399182,6cyclictest812-21snmpd21:30:222
2036399181,9cyclictest31196-21ps19:30:202
2036399181,9cyclictest31196-21ps19:30:202
2036399181,8cyclictest4489-21ls00:40:012
20363991813,3cyclictest14985-21smartctl22:35:152
20363991812,4cyclictest5663-21grep22:15:112
20363991812,3cyclictest4022-21idleruntime22:10:152
2036399181,15cyclictest0-21swapper/221:35:592
2036399181,13cyclictest0-21swapper/220:35:092
2036399181,11cyclictest0-21swapper/221:16:572
2036399181,10cyclictest0-21swapper/221:09:102
20357991812,3cyclictest32226-21/usr/sbin/munin19:35:111
2035799181,16cyclictest0-21swapper/120:21:531
2035799181,16cyclictest0-21swapper/120:15:371
2035399186,3cyclictest0-21swapper/000:30:180
2035399184,8cyclictest86550irq/126-enp1s0-21:10:210
2035399181,4cyclictest15883-21/usr/sbin/munin21:25:190
20353991813,3cyclictest7-21ksoftirqd/021:00:010
20353991813,3cyclictest17298-21if_err_enp2s022:40:180
20353991812,4cyclictest27341-21/usr/sbin/munin00:15:260
20353991812,3cyclictest3497-21cat22:10:110
20353991811,5cyclictest0-21swapper/022:45:160
2035399181,10cyclictest0-21swapper/023:23:130
2035399181,10cyclictest0-21swapper/021:46:140
2035399181,10cyclictest0-21swapper/000:37:250
65982171,2sleep20-21swapper/221:01:442
292042170,2sleep20-21swapper/200:20:212
292042170,2sleep20-21swapper/200:20:212
2037099176,3cyclictest7713-21/usr/sbin/munin22:20:123
2037099175,3cyclictest86550irq/126-enp1s0-20:25:243
2037099172,8cyclictest0-21swapper/319:13:513
2037099172,7cyclictest86550irq/126-enp1s0-19:45:083
2037099172,7cyclictest401ktimersoftd/320:05:013
2037099171,7cyclictest30030-21/usr/sbin/munin19:30:183
2037099171,7cyclictest30030-21/usr/sbin/munin19:30:183
2037099171,7cyclictest0-21swapper/321:45:093
2037099171,6cyclictest0-21swapper/300:25:213
2037099171,4cyclictest7094-21/usr/sbin/munin21:05:213
2037099171,3cyclictest0-21swapper/319:35:523
20370991713,3cyclictest0-21swapper/319:23:533
20370991712,3cyclictest1099-21/usr/sbin/munin22:05:153
20370991711,5cyclictest16053-21kworker/u8:120:06:183
20370991711,4cyclictest28472-21sed19:25:163
20370991711,3cyclictest6768-21expr23:30:153
20370991711,3cyclictest20953-21sort20:20:263
2037099171,10cyclictest0-21swapper/322:39:233
2037099171,10cyclictest0-21swapper/320:55:183
20370991710,4cyclictest0-21swapper/323:10:213
2037099170,9cyclictest0-21swapper/300:20:223
2037099170,9cyclictest0-21swapper/300:20:223
2036399175,3cyclictest86550irq/126-enp1s0-23:00:112
2036399171,7cyclictest9770-21cat22:25:012
2036399171,7cyclictest2207-21grep19:40:112
2036399171,6cyclictest0-21swapper/223:50:022
2036399171,4cyclictest24248-21irqstats19:15:162
20363991712,3cyclictest8782-21tr19:55:122
20363991712,3cyclictest24358-21ls20:30:132
20363991711,3cyclictest11676-21irqstats20:00:202
2036399171,10cyclictest0-21swapper/221:48:222
20363991710,4cyclictest17783-21head23:55:172
20363991710,4cyclictest0-21swapper/200:25:272
2035799172,6cyclictest812-21snmpd19:27:221
2035399172,12cyclictest10527-21/usr/sbin/munin23:40:100
2035399171,7cyclictest10863-21/usr/sbin/munin20:00:240
2035399171,6cyclictest28232-21munin-run20:40:010
2035399171,15cyclictest0-21swapper/019:34:530
2035399171,15cyclictest0-21swapper/019:34:530
2035399171,10cyclictest0-21swapper/023:16:350
2037099165,3cyclictest86550irq/126-enp1s0-00:10:113
2037099161,12cyclictest2046-21sed20:50:263
2036399165,3cyclictest86850irq/129-enp1s0-20:50:152
2036399163,7cyclictest86550irq/126-enp1s0-23:20:122
2036399163,2cyclictest86850irq/129-enp1s0-21:27:582
2036399162,5cyclictest10283-21strings21:10:212
2036399161,9cyclictest0-21swapper/220:59:202
2036399161,6cyclictest0-21swapper/222:40:032
2036399161,4cyclictest0-21swapper/200:15:022
20363991611,3cyclictest7047-21fw_packets19:50:152
20363991611,3cyclictest6238-21latency_hist19:50:022
20363991611,3cyclictest26237-21gltestperf19:20:142
20363991611,3cyclictest13267-21expr20:05:122
2036399161,12cyclictest13022-21memory22:30:172
2035399169,4cyclictest7714-21gltestperf21:05:150
2035399165,3cyclictest30750-21expr20:45:120
2035399165,3cyclictest30750-21expr20:45:120
2035399161,6cyclictest28462-21acpi20:40:120
2035399161,6cyclictest28159-21cstates19:25:140
20353991611,3cyclictest5745-21sed19:45:240
20353991611,3cyclictest32226-21/usr/sbin/munin19:35:140
20353991611,3cyclictest23688-21expr19:15:110
20353991611,3cyclictest20967-21fschecks_count19:10:130
20353991611,3cyclictest10047-21cat19:55:230
2035399161,12cyclictest17463-21/usr/sbin/munin20:15:120
2036399158,4cyclictest28819-21/usr/sbin/munin20:40:142
2035399159,3cyclictest15927-21gltestperf20:10:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional