You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-29 - 02:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Jun 29, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
64322640,4sleep230-21rcuc/219:05:212
13812239163,25sleep00-21swapper/019:06:400
14632236164,59sleep30-21swapper/319:07:443
16042233164,56sleep10-21swapper/119:09:231
1275221460,5sleep0174499cyclictest22:00:130
1922421340,5sleep0174499cyclictest23:25:250
1757997572,2cyclictest0-21swapper/221:31:122
1757995047,2cyclictest0-21swapper/219:13:462
85162480,3sleep230-21rcuc/221:50:122
175799444,33cyclictest6968-40dnf00:13:252
175799433,33cyclictest24793-40dnf21:12:242
174499431,32cyclictest17195-21ntp_states00:35:240
174499431,32cyclictest17195-21ntp_states00:35:240
175799411,28cyclictest19883-21/usr/sbin/munin23:30:172
175799411,28cyclictest19883-21/usr/sbin/munin23:30:172
1744994111,20cyclictest19051-21awk22:15:010
175799403,32cyclictest18827-40dnf22:12:282
175799393,24cyclictest12050irq/123-ahci[0023:25:012
175799393,16cyclictest87550irq/128-enp1s0-22:00:012
175799391,26cyclictest30954-21idleruntime22:40:162
175799383,23cyclictest0-21swapper/223:50:132
175799381,28cyclictest4812-21sed21:40:202
174499384,21cyclictest12050irq/123-ahci[0021:10:020
174499383,29cyclictest6968-40dnf00:13:240
174499381,26cyclictest12042-21/usr/sbin/munin20:45:160
175799371,21cyclictest6457-21latency_hist23:00:022
174499373,23cyclictest4064-21munin-node20:25:280
174499371,33cyclictest810-21sshd21:12:270
174499371,27cyclictest2929-21/usr/sbin/munin00:05:140
175799367,19cyclictest0-21swapper/200:20:152
175799365,21cyclictest12050irq/123-ahci[0023:50:022
175799362,29cyclictest30699-40dnf20:11:282
175799362,26cyclictest12163-21ntp_states23:10:222
175799361,28cyclictest0-21swapper/222:15:192
175799361,28cyclictest0-21swapper/222:15:192
175799361,27cyclictest3714-21if_err_enp2s000:05:172
175799361,25cyclictest0-21swapper/221:45:132
175799361,24cyclictest27438-21/usr/sbin/munin21:20:272
175799361,24cyclictest0-21swapper/223:05:132
175799361,23cyclictest87350irq/126-enp1s0-19:50:132
174499367,21cyclictest0-21swapper/023:00:250
174499361,24cyclictest1028-21/usr/sbin/munin20:20:270
174499361,18cyclictest0-21swapper/023:45:180
175799354,22cyclictest29516-21cat22:35:272
175799353,24cyclictest13178-21taskset19:32:192
175799351,5cyclictest7338-21/usr/sbin/munin00:15:222
175799351,25cyclictest31214-21/usr/sbin/munin20:15:142
175799351,24cyclictest13292-21/usr/sbin/munin23:15:162
174499354,22cyclictest762-21awk20:20:010
174499352,8cyclictest4840-21/usr/sbin/munin19:15:220
174499351,24cyclictest27206-21latency_hist21:20:010
175799342,26cyclictest0-21swapper/220:24:562
175799341,23cyclictest87550irq/128-enp1s0-20:45:132
175799341,21cyclictest7036-21/usr/sbin/munin19:20:232
175799341,20cyclictest6691-21/usr/sbin/munin23:00:162
175799336,19cyclictest87350irq/126-enp1s0-23:40:122
175799333,21cyclictest87350irq/126-enp1s0-21:35:122
175799333,20cyclictest16128-21acpi00:35:142
175799333,20cyclictest16128-21acpi00:35:142
175799332,24cyclictest15941-21ntp_states22:05:232
175799330,2cyclictest0-21swapper/222:30:022
174499334,17cyclictest87350irq/126-enp1s0-21:20:140
174499333,3cyclictest87350irq/126-enp1s0-21:45:140
174499332,3cyclictest2293-21/usr/sbin/munin22:50:230
174499332,28cyclictest2290-21grep00:00:270
174499332,27cyclictest23785-21cpuspeed_turbos22:25:130
174499331,21cyclictest22813-21diskmemload21:50:450
1744993312,18cyclictest5848-21munin-run21:45:010
175799324,21cyclictest0-21swapper/200:28:332
175799322,26cyclictest21193-21diskstats21:05:152
175799321,27cyclictest3356-21ntp_states22:50:202
175799321,23cyclictest0-21swapper/222:45:082
175799321,22cyclictest18753-21ntp_states23:25:212
175799321,20cyclictest31511-21tail23:55:162
175799321,17cyclictest29654-21cpu21:25:132
174499324,18cyclictest12050irq/123-ahci[0022:40:020
174499323,3cyclictest87550irq/128-enp1s0-21:35:220
174499322,18cyclictest909-21sssd_nss23:30:180
174499322,18cyclictest909-21sssd_nss23:30:180
174499321,5cyclictest16846-21perf22:10:010
174499321,3cyclictest87550irq/128-enp1s0-22:30:220
174499321,3cyclictest22813-21diskmemload00:26:270
175799312,25cyclictest28080-21smart_sda20:05:242
175799311,26cyclictest19233-21gltestperf21:00:162
175799311,22cyclictest0-21swapper/200:30:052
175799311,21cyclictest87650irq/129-enp1s0-20:40:172
175799311,20cyclictest87350irq/126-enp1s0-22:30:312
175799311,20cyclictest0-21swapper/219:15:112
175799311,16cyclictest0-21swapper/222:00:182
1757993110,18cyclictest22591-21pmu-power22:20:232
175799310,5cyclictest0-21swapper/221:20:012
174499312,3cyclictest87550irq/128-enp1s0-00:30:160
174499312,25cyclictest6521-21ntp_states20:30:210
174499312,25cyclictest24962-21diskstats20:00:140
174499311,3cyclictest87550irq/128-enp1s0-23:24:270
174499311,18cyclictest30250-21/usr/sbin/munin22:40:190
174499310,3cyclictest87350irq/126-enp1s0-23:55:250
174499310,3cyclictest13629-21/usr/sbin/munin19:35:170
175799308,19cyclictest17411-21users19:40:252
175799306,21cyclictest763-21dbus-daemon20:54:272
175799301,26cyclictest19282-21smart_sda19:45:252
175799301,26cyclictest18137-21chrt20:56:422
175799301,15cyclictest0-21swapper/200:00:122
174499308,19cyclictest12233-21head23:10:230
174499303,3cyclictest87550irq/128-enp1s0-21:30:230
174499303,15cyclictest87350irq/126-enp1s0-20:45:010
174499302,2cyclictest87350irq/126-enp1s0-22:15:120
174499302,2cyclictest87350irq/126-enp1s0-22:15:120
174499302,24cyclictest3624-40dnf19:11:230
174499301,6cyclictest30897-21smart_sda21:25:260
174499301,3cyclictest10468-21/usr/sbin/munin21:55:200
1744993012,15cyclictest7036-21/usr/sbin/munin19:20:170
174499301,20cyclictest8895-21/usr/sbin/munin23:05:140
174499301,16cyclictest909-21sssd_nss23:50:190
176199291,12cyclictest4452-21ntp_states19:15:033
175799299,17cyclictest24797-21cstates20:00:132
175799292,24cyclictest23486-21ntp_states19:55:212
175799292,24cyclictest23486-21ntp_states19:55:212
175799291,25cyclictest9617-21munin-run20:40:002
175799291,25cyclictest10495-21smart_sda19:25:242
1757992912,14cyclictest3857-21grep20:25:162
175799291,16cyclictest22945-21sed23:35:202
175799290,4cyclictest0-21swapper/219:40:012
174499295,21cyclictest11545-21tr19:30:130
174499293,2cyclictest87350irq/126-enp1s0-22:45:170
174499292,2cyclictest87350irq/126-enp1s0-23:35:050
174499292,23cyclictest5559-21ntp_states22:55:210
174499292,22cyclictest18042-21runrttasks20:55:280
174499291,4cyclictest9582-21diskstats19:25:140
174499291,3cyclictest7338-21/usr/sbin/munin00:15:240
174499291,25cyclictest18635-21gltestperf19:45:160
174499291,24cyclictest14354-21ntp_states23:15:220
1744992912,14cyclictest28132-21sort20:05:240
1744992910,16cyclictest21773-21expr22:20:130
174499290,5cyclictest0-21swapper/019:50:130
174499281,3cyclictest87350irq/126-enp1s0-00:20:300
174499281,3cyclictest20620-21/usr/sbin/munin21:05:010
174499281,24cyclictest4064-21munin-node19:55:110
174499281,24cyclictest4064-21munin-node19:55:110
174499280,17cyclictest0-21swapper/019:40:510
175799277,17cyclictest5636-21cstates20:30:122
175099272,17cyclictest13013-21diskstats22:00:151
174499275,18cyclictest14897-21gltestperf20:50:160
174499271,5cyclictest0-21swapper/020:10:130
174499271,2cyclictest8369-21/usr/sbin/munin20:35:160
1744992711,13cyclictest24497-21expr23:40:130
175099261,17cyclictest7504-21cat20:35:021
175099252,3cyclictest26256-21munin-run23:45:011
175099250,16cyclictest87650irq/129-enp1s0-20:36:271
176199242,12cyclictest2015-21df21:35:133
176199241,9cyclictest0-21swapper/321:05:473
175099244,3cyclictest773-21NetworkManager21:02:511
176199231,16cyclictest0-21swapper/323:52:553
175099232,11cyclictest29412-21munin-run21:25:011
175099231,5cyclictest32746-21memory21:30:191
175099231,3cyclictest18100-21munin-run00:40:011
175099231,3cyclictest18100-21munin-run00:40:011
175099231,16cyclictest0-21swapper/121:54:521
176199221,4cyclictest1939-21ps00:00:233
1761992212,3cyclictest41-21ksoftirqd/321:45:023
175099228,4cyclictest0-21swapper/123:04:241
175099222,3cyclictest32088-21rt-features23:55:231
1750992215,4cyclictest27183-21forks20:05:151
175099221,4cyclictest12646-21memory00:25:191
1750992212,2cyclictest19172-21latency_hist22:15:011
175099221,18cyclictest14345-21fschecks_count00:30:151
175099221,16cyclictest0-21swapper/120:20:031
175099221,14cyclictest0-21swapper/123:51:351
175099221,11cyclictest24379-21idleruntime-cro20:00:011
175099221,11cyclictest24379-21idleruntime-cro20:00:011
175099220,15cyclictest0-21swapper/120:55:011
176199218,4cyclictest13968-21hddtemp_smartct23:15:173
176199217,2cyclictest39-21rcuc/322:55:173
176199211,12cyclictest0-21swapper/321:25:253
176199211,11cyclictest23377-21df21:10:133
176199210,19cyclictest0-21swapper/320:03:003
175099218,4cyclictest31662-21date21:30:011
175099218,3cyclictest29647-21grep20:10:171
175099217,2cyclictest0-21swapper/123:25:071
175099212,4cyclictest6968-40dnf00:13:241
1750992114,1cyclictest0-21swapper/122:50:171
175099211,3cyclictest18415-21munin-run21:00:011
175099211,3cyclictest0-21swapper/100:05:121
175099211,3cyclictest0-21swapper/100:01:191
175099211,2cyclictest0-21swapper/123:15:271
175099211,17cyclictest11050-21gltestperf21:55:161
175099211,15cyclictest0-21swapper/122:59:521
175099211,15cyclictest0-21swapper/122:17:291
175099211,15cyclictest0-21swapper/122:17:291
175099211,14cyclictest0-21swapper/123:32:391
175099211,14cyclictest0-21swapper/123:32:391
175099211,14cyclictest0-21swapper/122:41:001
175099211,14cyclictest0-21swapper/121:48:031
175099211,12cyclictest0-21swapper/123:37:561
175099211,11cyclictest2145-21cat19:10:121
175099211,10cyclictest22199-21perf19:55:011
175099210,16cyclictest0-21swapper/122:39:101
175099210,11cyclictest21-21rcuc/122:30:211
176199206,3cyclictest22813-21diskmemload22:17:553
176199206,3cyclictest22813-21diskmemload22:17:553
176199205,7cyclictest0-21swapper/321:16:333
176199205,4cyclictest755-21rngd20:10:263
176199202,15cyclictest0-21swapper/322:45:163
176199202,10cyclictest24276-21/usr/sbin/munin23:40:233
176199201,9cyclictest0-21swapper/300:06:263
176199201,6cyclictest32531-21idleruntime21:30:173
1761992014,3cyclictest7383-21diskstats19:20:153
1761992014,3cyclictest2896-21tail22:50:153
176199201,11cyclictest18637-21/usr/sbin/munin21:00:173
176199201,11cyclictest0-21swapper/300:15:223
176199201,10cyclictest7055-21sshd00:14:273
176199201,10cyclictest28680-21gltestperf22:35:183
176199201,10cyclictest0-21swapper/319:52:553
176199200,7cyclictest0-21swapper/322:10:383
176199200,6cyclictest0-21swapper/323:39:173
176199200,18cyclictest0-21swapper/321:47:263
176199200,10cyclictest15485-21/usr/sbin/munin23:20:123
175099209,3cyclictest31552-21df20:15:131
175099209,3cyclictest17502-21cut23:25:011
175099204,6cyclictest0-21swapper/121:40:051
175099201,9cyclictest0-21swapper/123:09:471
175099201,6cyclictest31323-21kworker/u8:222:05:281
1750992016,3cyclictest0-21swapper/120:03:231
1750992013,2cyclictest0-21swapper/123:48:031
1750992013,2cyclictest0-21swapper/121:17:051
175099201,2cyclictest808-21snmpd21:09:421
175099201,2cyclictest22813-21diskmemload00:21:341
1750992012,3cyclictest0-21swapper/122:20:531
1750992012,3cyclictest0-21swapper/120:48:381
1750992012,3cyclictest0-21swapper/120:25:211
175099201,14cyclictest0-21swapper/122:25:561
175099201,11cyclictest2010-21df21:35:131
175099201,10cyclictest0-21swapper/100:15:131
176199199,2cyclictest13645-21sshd20:45:273
176199196,4cyclictest3916-21crond19:15:013
176199195,4cyclictest579-21systemd-journal19:30:013
176199191,9cyclictest0-21swapper/322:09:263
1761991913,3cyclictest12239-21tr23:10:233
176199191,10cyclictest9808-21memory23:05:183
176199191,10cyclictest87350irq/126-enp1s0-22:21:113
176199191,10cyclictest10507-21cat00:20:193
176199191,10cyclictest0-21swapper/323:57:523
176199191,10cyclictest0-21swapper/323:32:133
176199191,10cyclictest0-21swapper/323:32:133
176199191,10cyclictest0-21swapper/322:44:373
176199191,10cyclictest0-21swapper/300:30:443
176199190,6cyclictest0-21swapper/322:02:363
1750991913,2cyclictest0-21swapper/121:13:531
175099191,17cyclictest0-21swapper/122:49:091
175099191,17cyclictest0-21swapper/119:48:261
175099191,10cyclictest15966-21cpuspeed_turbos19:40:121
175099191,10cyclictest12052-21missed_timers23:10:191
175099190,17cyclictest0-21swapper/120:42:231
95542180,2sleep30-21swapper/320:39:233
176199186,6cyclictest39-21rcuc/319:43:493
176199182,9cyclictest28353-21memory21:20:203
176199181,9cyclictest0-21swapper/321:58:323
176199181,7cyclictest6610-21proc_pri20:30:223
176199181,11cyclictest0-21swapper/323:49:493
176199181,11cyclictest0-21swapper/323:02:153
176199181,11cyclictest0-21swapper/321:53:153
176199181,10cyclictest0-21swapper/322:25:303
175099188,2cyclictest12690-21sensors19:30:241
1750991815,2cyclictest0-21swapper/119:20:241
1750991815,2cyclictest0-21swapper/119:16:541
175099181,16cyclictest0-21swapper/119:37:261
176199171,8cyclictest0-21swapper/322:30:483
176199171,6cyclictest17189-21ntp_states00:35:233
176199171,6cyclictest17189-21ntp_states00:35:233
176199171,5cyclictest0-21swapper/323:25:143
176199171,3cyclictest17351-21memory20:55:193
1761991713,3cyclictest0-21swapper/320:16:533
1761991713,2cyclictest22518-21cpuspeed_turbos19:55:123
1761991713,2cyclictest22518-21cpuspeed_turbos19:55:123
1761991711,3cyclictest827-21runrttasks20:50:093
1761991711,3cyclictest11732-21/usr/sbin/munin00:25:253
176199171,11cyclictest87350irq/126-enp1s0-20:22:343
176199171,10cyclictest0-21swapper/320:42:163
1750991712,3cyclictest9921-21hwlatdetect19:25:161
176199169,2cyclictest87350irq/126-enp1s0-20:05:033
176199163,4cyclictest18038-21egrep19:45:123
176199161,6cyclictest0-21swapper/319:37:163
176199161,5cyclictest12693-21smart_sda19:30:243
176199151,3cyclictest808-21snmpd20:25:583
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional