You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-12 - 08:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Mar 12, 2026 00:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
297072184116,59sleep00-21swapper/019:05:220
30405217392,24sleep20-21swapper/219:08:262
289112155125,20sleep30-21swapper/319:05:143
303412152116,24sleep10-21swapper/119:07:371
252192820,3sleep33063199cyclictest23:50:113
237552620,3sleep081ktimersoftd/023:45:200
3063199401,23cyclictest2019-21hddtemp_smartct19:15:153
3063199371,12cyclictest0-21swapper/320:56:293
30631993524,3cyclictest0-21swapper/322:44:343
3063199311,13cyclictest22549-21irqstats21:15:183
3061399316,12cyclictest7978-21munin-run00:25:000
3061399315,12cyclictest15546-21perf22:15:010
3061399314,12cyclictest674-21awk22:55:010
3063199301,16cyclictest0-21swapper/322:55:163
3063199301,12cyclictest0-21swapper/319:47:443
3063199301,12cyclictest0-21swapper/319:42:583
3063199301,10cyclictest19313-21diskmemload21:10:003
3063199301,10cyclictest19313-21diskmemload21:10:003
3061899307,3cyclictest31838-21kworker/1:119:40:121
3061399301,14cyclictest330-21perf19:15:020
3061399301,14cyclictest16144-21munin-run23:30:010
3061399301,14cyclictest16144-21munin-run23:30:010
3063199292,14cyclictest0-21swapper/323:57:213
3063199291,13cyclictest0-21swapper/320:11:193
3063199291,12cyclictest31589-21ntp_states21:35:203
3063199291,12cyclictest0-21swapper/319:34:283
3063199290,13cyclictest0-21swapper/320:35:033
3061899292,14cyclictest14587-40dnf20:56:091
3061399291,23cyclictest0-21swapper/020:56:080
3063199281,13cyclictest0-21swapper/319:54:563
3063199281,13cyclictest0-21swapper/300:23:103
3063199281,12cyclictest0-21swapper/320:21:253
3063199280,13cyclictest0-21swapper/320:05:153
3062599284,15cyclictest28230-21latency_hist21:30:022
3062599284,12cyclictest1388-21basename00:10:022
3061899289,3cyclictest31838-21kworker/1:119:20:231
3061899283,17cyclictest88350irq/128-enp1s0-23:25:131
3061899283,17cyclictest88350irq/128-enp1s0-23:25:131
3063199271,13cyclictest0-21swapper/323:42:493
3063199271,13cyclictest0-21swapper/321:25:043
3063199271,13cyclictest0-21swapper/300:38:443
3063199271,12cyclictest0-21swapper/321:59:063
3063199271,12cyclictest0-21swapper/319:10:033
3063199271,12cyclictest0-21swapper/300:32:363
3063199271,11cyclictest0-21swapper/320:04:433
3063199270,12cyclictest0-21swapper/320:53:013
3063199270,11cyclictest0-21swapper/320:41:133
3062599277,6cyclictest30904-21cpuspeed_turbos19:10:102
3062599275,11cyclictest88150irq/126-enp1s0-21:45:252
3062599272,11cyclictest12942-21/usr/sbin/munin00:35:152
3062599272,10cyclictest0-21swapper/222:57:582
3063199261,14cyclictest0-21swapper/319:24:323
3063199261,12cyclictest0-21swapper/323:46:133
3063199261,12cyclictest0-21swapper/323:18:063
3063199261,12cyclictest0-21swapper/321:11:553
3063199261,12cyclictest0-21swapper/300:05:393
3063199261,11cyclictest19313-21diskmemload21:46:143
3063199261,10cyclictest0-21swapper/319:56:563
3063199260,13cyclictest0-21swapper/321:00:353
3062599269,3cyclictest22083-21df_inode21:15:142
3062599265,8cyclictest6503-21grep20:40:102
3062599261,6cyclictest0-21swapper/222:48:332
30625992616,6cyclictest29096-40dnf23:57:232
3061899260,20cyclictest22475-21kworker/1:023:04:321
3061399268,15cyclictest28111-21awk21:30:010
3063199251,13cyclictest0-21swapper/322:46:593
3063199251,13cyclictest0-21swapper/321:53:283
3063199251,12cyclictest0-21swapper/323:07:383
3063199251,12cyclictest0-21swapper/323:03:003
3062599253,12cyclictest88150irq/126-enp1s0-23:20:002
3062599253,12cyclictest28720-21fschecks_count21:30:152
3062599252,12cyclictest20401-21grep19:55:242
3061899256,8cyclictest23-21ksoftirqd/121:57:561
3061899256,10cyclictest23-21ksoftirqd/120:41:491
3061899251,19cyclictest22475-21kworker/1:022:52:451
3063199241,10cyclictest0-21swapper/322:28:573
3063199241,10cyclictest0-21swapper/322:14:053
3062599243,4cyclictest0-21swapper/223:35:012
3062599242,13cyclictest19313-21diskmemload22:30:132
3062599241,8cyclictest13669-21grep22:10:112
3062599241,10cyclictest7091-21cpuspeed_turbos21:55:132
3062599240,15cyclictest10505-21latency_hist20:50:002
3061899246,8cyclictest23-21ksoftirqd/100:37:101
3061899245,6cyclictest0-21swapper/122:55:111
3061899243,9cyclictest21-21rcuc/120:11:471
3061899242,12cyclictest31838-21kworker/1:121:20:141
30618992417,4cyclictest976-21/usr/sbin/munin00:05:251
30618992410,3cyclictest31838-21kworker/1:120:20:191
3061399248,14cyclictest0-21swapper/023:00:010
3061399241,12cyclictest31555-21awk00:05:020
3061399241,11cyclictest23107-21basename20:05:010
3063199239,4cyclictest1503-21kworker/u8:021:30:393
3063199231,14cyclictest0-21swapper/300:26:263
3062599237,3cyclictest2654-21sort20:30:142
3062599236,8cyclictest0-21swapper/220:00:202
3062599236,5cyclictest823-21snmpd21:53:292
3062599235,5cyclictest20760-21/usr/sbin/munin23:40:132
3062599234,10cyclictest13326-21latency_hist22:10:012
3062599233,7cyclictest19313-21diskmemload22:26:092
3062599233,17cyclictest17385-21sed19:50:152
3062599232,13cyclictest30234-21cat20:20:122
3062599232,13cyclictest2296-21taskset21:44:372
3062599232,12cyclictest10199-21sa122:00:202
3062599231,12cyclictest0-21swapper/222:35:122
3062599231,11cyclictest0-21swapper/223:06:592
3062599230,9cyclictest1-21systemd00:30:192
3061899238,3cyclictest22475-21kworker/1:000:25:261
3061899237,8cyclictest23-21ksoftirqd/123:43:551
3061899235,3cyclictest23-21ksoftirqd/119:58:241
3061899234,8cyclictest23-21ksoftirqd/119:39:261
3061899232,10cyclictest23-21ksoftirqd/120:29:511
3063199221,9cyclictest0-21swapper/320:30:113
3063199221,10cyclictest0-21swapper/323:11:373
3063199220,11cyclictest0-21swapper/322:19:533
3062599226,8cyclictest1051-21switchtime20:25:242
3062599225,14cyclictest26301-21awk23:50:212
3062599224,2cyclictest88450irq/129-enp1s0-20:15:212
3062599223,11cyclictest2560-21/usr/sbin/munin00:10:192
3062599222,8cyclictest8496-21cat23:10:192
3062599221,9cyclictest11559-21timerwakeupswit19:35:232
3062599220,13cyclictest19313-21diskmemload00:00:102
3061899228,4cyclictest23416-21ntp_states22:30:211
3061899227,12cyclictest10043-21cat22:00:181
3061899222,8cyclictest23-21ksoftirqd/100:23:401
30618992215,3cyclictest31838-21kworker/1:122:28:351
3061899221,14cyclictest0-21swapper/121:42:511
3061399228,13cyclictest0-21swapper/020:40:020
3061399227,13cyclictest0-21swapper/020:20:020
3063199211,9cyclictest0-21swapper/300:15:583
3063199211,15cyclictest8541-21/usr/sbin/munin20:45:153
3063199211,15cyclictest1419-21ntp_states21:40:223
3063199211,13cyclictest0-21swapper/322:20:253
3063199210,12cyclictest0-21swapper/300:10:123
3062599216,3cyclictest88450irq/129-enp1s0-21:24:222
3062599213,9cyclictest22959-21/usr/sbin/munin23:45:262
3062599212,7cyclictest19653-21expr21:10:122
3062599211,7cyclictest3368-21latency_hist19:20:012
3062599211,7cyclictest0-21swapper/223:20:172
3062599211,6cyclictest0-21swapper/200:25:162
3062599211,13cyclictest0-21swapper/221:36:512
3062599211,12cyclictest18982-21sort23:35:142
3061899219,2cyclictest88350irq/128-enp1s0-20:51:211
3061899217,8cyclictest21-21rcuc/121:54:521
3061899215,8cyclictest23-21ksoftirqd/119:26:521
3061899212,11cyclictest21608-21tune2fs20:00:161
3061899211,11cyclictest1297-21ntp_states19:15:051
3061899211,11cyclictest0-21swapper/122:40:231
30618992110,3cyclictest25282-21latency_hist20:10:011
3061399219,4cyclictest4760-21/usr/sbin/munin21:50:160
3061399217,13cyclictest0-21swapper/023:10:000
3061399216,6cyclictest0-21swapper/022:28:190
3061399214,6cyclictest0-21swapper/022:16:440
3061399213,15cyclictest31884-21munin-run20:25:020
3061399212,5cyclictest16495-21munin-run19:50:000
3063199201,15cyclictest0-21swapper/321:20:203
3063199201,14cyclictest12143-21if_err_enp2s022:05:153
3063199201,13cyclictest0-21swapper/320:16:113
30631992010,2cyclictest0-21swapper/323:20:113
3062599203,9cyclictest5297-21sed00:15:272
3062599203,9cyclictest31144-21acpi22:50:122
30625992011,5cyclictest17344-21/usr/sbin/munin21:05:292
30625992011,5cyclictest17344-21/usr/sbin/munin21:05:292
3062599201,12cyclictest0-21swapper/222:44:522
3062599201,10cyclictest913-21sssd_nss23:25:202
3062599201,10cyclictest913-21sssd_nss23:25:202
3062599201,10cyclictest11067-21expr20:50:132
3062599200,7cyclictest88450irq/129-enp1s0-22:25:002
3061899209,3cyclictest21508-21latency_hist21:15:021
3061899208,2cyclictest23-21ksoftirqd/121:48:041
3061899207,3cyclictest25610-21ntp_states22:35:211
3061899206,2cyclictest31838-21kworker/1:121:30:041
3061899205,9cyclictest31838-21kworker/1:119:11:101
3061899205,9cyclictest23-21ksoftirqd/121:19:561
3061899205,3cyclictest88350irq/128-enp1s0-22:15:231
3061899205,3cyclictest23-21ksoftirqd/120:49:351
3061899204,3cyclictest88350irq/128-enp1s0-20:17:271
3061899204,3cyclictest88250irq/127-enp1s0-19:33:341
3061899202,10cyclictest21-21rcuc/119:48:401
30618992014,3cyclictest7354-21munin-run23:10:001
3061899201,13cyclictest21-21rcuc/123:17:221
3061899201,12cyclictest0-21swapper/121:35:111
3061899201,11cyclictest27118-21latency_hist23:55:021
3061899201,11cyclictest26941-21kernelversion21:25:161
3061899201,11cyclictest24548-21users23:45:291
3061899201,10cyclictest580-21systemd-journal21:05:011
3061899201,10cyclictest5053-21load20:35:181
3061899201,10cyclictest12307-21df_inode23:20:121
3061899201,10cyclictest11692-21expr22:05:111
3061899200,5cyclictest0-21swapper/100:01:171
3061399207,7cyclictest0-21swapper/023:50:170
3061399201,6cyclictest0-21swapper/022:05:530
3061399201,11cyclictest8207-21/usr/sbin/munin00:25:240
3061399201,11cyclictest23931-21/usr/sbin/munin21:20:140
3061399201,11cyclictest22715-21sshd21:15:190
184992201,3sleep10-21swapper/121:05:251
184992201,3sleep10-21swapper/121:05:251
3063199191,14cyclictest0-21swapper/319:28:323
3063199191,10cyclictest306-21sshd22:50:523
3063199191,10cyclictest0-21swapper/323:38:043
3063199190,12cyclictest0-21swapper/323:29:043
3063199190,12cyclictest0-21swapper/323:29:043
3062599191,5cyclictest0-21swapper/221:02:152
30625991912,4cyclictest4607-21ntp_states19:20:202
30625991911,5cyclictest5409-21smart_sda20:35:222
30625991911,5cyclictest3520-21diskstats23:00:142
30625991911,5cyclictest16819-21ntp_states22:15:202
3061899197,2cyclictest23-21ksoftirqd/123:31:441
3061899196,7cyclictest23-21ksoftirqd/119:51:581
3061899195,7cyclictest23-21ksoftirqd/100:11:351
3061899194,4cyclictest23-21ksoftirqd/120:30:411
3061899193,2cyclictest22475-21kworker/1:000:19:191
3061899191,15cyclictest20319-21chrt23:37:211
3061899191,10cyclictest29136-21expr22:45:131
3061899191,10cyclictest14532-21/usr/sbin/munin22:10:191
3061399192,14cyclictest31160-21tail21:35:140
3061399191,3cyclictest580-21systemd-journal21:14:190
30613991912,4cyclictest843-21runrttasks23:40:140
30613991912,4cyclictest843-21runrttasks23:15:220
30613991912,4cyclictest5151-21latency_hist23:05:000
3061399191,10cyclictest9340-21chrt23:12:280
3061399191,10cyclictest5785-21munin-run00:20:000
3061399191,10cyclictest32724-21/usr/sbin/munin21:40:200
3061399191,10cyclictest2564-21/usr/sbin/munin21:45:250
3061399191,10cyclictest10735-21/usr/sbin/munin20:50:180
3061399191,10cyclictest0-21swapper/023:37:340
3063199181,6cyclictest0-21swapper/319:35:103
30631991811,4cyclictest25721-21processes22:35:223
3063199181,14cyclictest0-21swapper/322:00:253
3063199181,14cyclictest0-21swapper/300:00:153
3062599189,6cyclictest913-21sssd_nss19:45:162
3062599189,6cyclictest6690-21/usr/sbin/munin00:20:152
3062599189,6cyclictest5738-21/usr/sbin/munin19:25:262
3062599183,7cyclictest26575-21ntpq20:10:222
3062599181,8cyclictest14521-21users20:55:252
3062599181,6cyclictest0-21swapper/220:06:322
30625991810,5cyclictest9764-21sshd19:33:202
3061899186,2cyclictest23-21ksoftirqd/123:56:471
3061899181,8cyclictest0-21swapper/122:23:121
3061899181,10cyclictest8318-21if_enp2s023:10:151
3061899181,10cyclictest0-21swapper/100:30:131
3061399189,2cyclictest0-21swapper/000:05:210
3061399188,3cyclictest22346-21/usr/sbin/munin22:30:140
3061399188,2cyclictest28324-21/usr/sbin/munin21:30:130
3061399183,8cyclictest0-21swapper/021:59:300
3061399181,6cyclictest0-21swapper/023:58:120
3061399181,5cyclictest0-21swapper/023:30:220
3061399181,4cyclictest13018-21kworker/0:221:05:000
30613991812,4cyclictest6981-21cut20:40:150
30613991812,3cyclictest9573-21cut22:00:150
30613991811,4cyclictest24937-21fschecks_count22:35:130
3063199171,6cyclictest24259-21chrt22:34:223
3063199171,11cyclictest802-21pmu-power20:25:213
30631991710,4cyclictest0-21swapper/323:30:203
30625991710,4cyclictest12670-21diskstats19:40:132
3061399178,3cyclictest2184-21expr20:30:110
3061399178,2cyclictest17344-21/usr/sbin/munin21:05:170
3061399178,2cyclictest17344-21/usr/sbin/munin21:05:170
3061399171,8cyclictest2902-21wc00:10:230
3061399171,3cyclictest12598-21/usr/sbin/munin00:35:170
3061399171,3cyclictest0-21swapper/019:43:220
30613991712,3cyclictest5506-21latency_hist19:25:010
30613991712,3cyclictest27550-21irqstats22:40:170
3061399171,15cyclictest0-21swapper/022:20:340
30613991711,4cyclictest11530-21cat19:35:220
3061399171,13cyclictest7285-21grep19:25:250
3061399171,10cyclictest0-21swapper/020:10:260
30613991710,4cyclictest29098-21grep22:45:120
3061399170,6cyclictest0-21swapper/019:30:100
3061399169,4cyclictest20023-21pmu-power19:55:220
3061399166,3cyclictest11-21rcuc/020:45:220
30613991611,3cyclictest16718-21/usr/sbin/munin19:50:190
30613991611,3cyclictest1325-21/usr/sbin/munin19:15:250
30613991611,3cyclictest12881-21missed_timers23:20:170
30613991611,3cyclictest1086-21sed20:25:250
3061399161,10cyclictest0-21swapper/000:33:370
3061399160,6cyclictest0-21swapper/020:05:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional