You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-28 - 01:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed May 27, 2026 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
158612239162,25sleep00-21swapper/007:09:030
156062238163,24sleep30-21swapper/307:05:493
156562232164,55sleep10-21swapper/107:06:271
156302232163,21sleep20-21swapper/207:06:072
2482121570,3sleep31603399cyclictest09:55:123
16026996462,1cyclictest0-21swapper/209:51:282
283492490,2sleep20-21swapper/208:50:022
16026994643,2cyclictest0-21swapper/212:05:232
16013994012,14cyclictest20114-21munin-run09:45:010
16013993412,12cyclictest8379-21/usr/sbin/munin11:45:130
1601399334,25cyclictest23558-21munin-run12:20:010
162542320,4sleep11602199cyclictest12:00:231
16026993117,7cyclictest20722-21sed12:10:242
1601399319,5cyclictest12161-21/usr/sbin/munin10:40:170
1601399319,12cyclictest22990-21sed11:05:020
1601399315,17cyclictest31338-21/usr/sbin/munin10:10:180
1601399306,13cyclictest1-21systemd12:10:250
16013993014,7cyclictest4643-40dnf09:09:130
16013993010,10cyclictest28269-21sort12:30:130
1601399299,16cyclictest20717-21cut08:30:200
1601399297,12cyclictest15000-21sh09:30:220
1601399297,10cyclictest16729-21runrttasks12:02:120
1601399295,9cyclictest15924-21unixbench_singl10:45:240
1601399295,11cyclictest6299-21munin-run08:00:010
1601399294,12cyclictest5961-21munin-run11:40:010
16021992817,9cyclictest0-21swapper/108:00:011
1601399283,14cyclictest0-21swapper/008:15:130
1601399283,12cyclictest22313-21munin-run09:50:010
16013992810,7cyclictest23775-21wc09:50:220
1602699272,8cyclictest27950-21munin-run12:30:012
16026992710,3cyclictest18012-21cat09:40:012
1601399278,16cyclictest23899-21grep07:25:160
1601399274,11cyclictest25766-21awk12:25:010
1601399274,10cyclictest18598-21timerwakeupswit12:05:250
1601399273,13cyclictest0-21swapper/011:25:220
16013992715,10cyclictest11332-21awk09:25:010
1601399271,16cyclictest0-21swapper/012:35:140
16021992614,3cyclictest18831-21cpuspeed_turbos10:55:121
1601399268,15cyclictest12095-21grep11:50:240
1601399267,11cyclictest87350irq/126-enp1s0-09:35:140
1601399266,9cyclictest19254-21grep10:55:150
1601399266,12cyclictest15321-21/usr/sbin/munin08:20:160
1601399263,14cyclictest0-21swapper/007:50:230
1601399261,13cyclictest9753-21idleruntime-cro10:35:010
1601399261,12cyclictest26709-21munin-run10:00:000
16021992518,4cyclictest14364-21users11:55:251
1602199250,18cyclictest0-21swapper/111:26:361
1601399258,5cyclictest30911-21proc_pri11:20:220
1601399254,7cyclictest0-21swapper/010:35:230
1601399254,6cyclictest87350irq/126-enp1s0-11:10:210
1601399254,11cyclictest3554-21cstates10:20:120
1601399252,16cyclictest0-21swapper/011:42:250
1601399252,11cyclictest1959-21kworker/u8:110:17:480
16013992518,4cyclictest22981-21ntp_states08:35:210
16013992518,4cyclictest18441-21memory08:25:190
16013992517,5cyclictest29489-21diskstats10:05:130
16013992512,6cyclictest8215-21ntp_states09:15:200
16013992510,12cyclictest6324-21timerandwakeup09:10:250
1603399243,7cyclictest13479-21idleruntime11:55:163
1602199241,19cyclictest19498-21tr12:10:111
1602199241,17cyclictest0-21swapper/110:05:151
1602199241,17cyclictest0-21swapper/110:00:171
16021992410,2cyclictest0-21swapper/109:18:491
1602199240,18cyclictest0-21swapper/110:29:131
1602199240,18cyclictest0-21swapper/110:29:131
1602199240,15cyclictest87550irq/128-enp1s0-09:09:131
1601399247,8cyclictest27902-21/usr/sbin/munin07:35:120
1601399247,8cyclictest14076-21kworker/u8:111:15:120
1601399246,7cyclictest3132-21timerandwakeup11:30:230
16013992416,5cyclictest17630-21ntp_states10:50:200
16013992415,6cyclictest23953-21/usr/sbin/munin11:05:150
1603399231,9cyclictest0-21swapper/312:12:053
1603399231,5cyclictest0-21swapper/309:20:153
1602699232,7cyclictest31047-40dnf10:09:232
1602699232,3cyclictest0-21swapper/210:20:232
1602199239,6cyclictest29088-21memory12:30:181
1602199233,17cyclictest0-21swapper/109:10:241
16021992318,3cyclictest31721-21sh11:25:011
16021992317,3cyclictest32362-21/usr/sbin/munin10:10:181
1602199231,19cyclictest2129-21df_inode11:30:121
1602199231,17cyclictest0-21swapper/109:27:501
16021992311,2cyclictest87550irq/128-enp1s0-07:19:231
1601399237,9cyclictest27054-21/usr/sbin/munin12:25:220
1601399236,6cyclictest0-21swapper/009:29:270
1601399232,12cyclictest13584-21irqstats11:55:160
1601399232,12cyclictest0-21swapper/008:45:140
16013992316,4cyclictest6640-21ntp_states10:25:220
16013992316,4cyclictest6640-21ntp_states10:25:220
16013992316,4cyclictest1173-21smart_sda07:45:230
16013992315,5cyclictest7339-21irqstats08:00:180
16013992315,5cyclictest7339-21irqstats08:00:170
16013992314,6cyclictest29583-21ntp_states08:50:210
16013992314,6cyclictest28146-21/usr/sbin/munin10:00:220
16013992314,6cyclictest13081-21cat08:15:020
16013992310,5cyclictest808-21snmpd07:41:230
16013992310,10cyclictest26368-21head07:30:150
1603399220,12cyclictest0-21swapper/308:40:173
1602199221,18cyclictest28680-21cstates08:50:121
1602199221,17cyclictest0-21swapper/111:19:541
1602199221,12cyclictest17857-21head12:05:161
1602199220,4cyclictest0-21swapper/107:27:221
1602199220,16cyclictest0-21swapper/111:40:201
1602199220,16cyclictest0-21swapper/110:51:061
1601399229,5cyclictest755-21rngd07:20:160
16013992215,4cyclictest1809-21smart_sda09:00:250
16013992214,5cyclictest31802-21/usr/sbin/munin08:55:230
16013992214,5cyclictest24928-21irqstats08:40:170
16013992212,6cyclictest16560-21/usr/sbin/munin07:10:120
46082210,1sleep10-21swapper/111:35:161
1603399217,3cyclictest0-21swapper/310:08:333
1603399212,10cyclictest24752-21users11:05:243
1603399211,6cyclictest17692-21sshd09:37:213
16033992115,3cyclictest6217-21cut10:25:153
16033992115,3cyclictest6217-21cut10:25:153
1603399211,11cyclictest31774-21ntp_states08:55:223
1603399210,9cyclictest0-21swapper/311:19:333
1603399210,10cyclictest0-21swapper/312:37:333
1602699212,7cyclictest23161-21/usr/sbin/munin11:05:172
1602699212,6cyclictest808-21snmpd09:20:232
1602699211,7cyclictest8197-21ntpq09:15:192
1602699211,10cyclictest4727-21diskmemload09:45:442
1602199218,3cyclictest18245-1kworker/u9:210:23:051
1602199212,10cyclictest31084-21strings07:40:181
1602199211,5cyclictest0-21swapper/109:40:191
1602199211,3cyclictest21261-21unixbench_singl08:30:271
1602199211,18cyclictest0-21swapper/107:20:211
1602199211,14cyclictest0-21swapper/111:52:241
1602199211,14cyclictest0-21swapper/111:12:361
1602199210,15cyclictest0-21swapper/111:08:311
16013992113,5cyclictest19465-21/usr/sbin/munin07:15:150
16013992112,6cyclictest8734-21/usr/sbin/munin08:05:160
259852201,3sleep20-21swapper/209:55:242
1603399209,3cyclictest19115-21/usr/sbin/munin07:15:223
1603399207,3cyclictest0-21swapper/312:33:373
1603399201,11cyclictest29754-21/usr/sbin/munin11:20:223
1603399201,10cyclictest0-21swapper/308:30:133
16033992010,3cyclictest87650irq/129-enp1s0-10:50:033
16033992010,3cyclictest87650irq/129-enp1s0-10:15:183
1602699205,5cyclictest23508-21/usr/sbin/munin07:25:152
1602699204,4cyclictest31907-21ls11:25:012
1602699204,4cyclictest18654-21cat10:55:022
1602699201,5cyclictest4727-21diskmemload10:55:182
1602699201,16cyclictest4107-21munin-run07:55:012
1602699201,11cyclictest0-21swapper/212:32:442
1602199208,7cyclictest0-21swapper/112:36:021
1602199203,4cyclictest0-21swapper/109:52:011
1602199202,2cyclictest87650irq/129-enp1s0-09:45:181
1602199201,7cyclictest16839-21grep08:20:251
16021992014,3cyclictest27677-21munin-run07:35:011
1602199201,3cyclictest9675-21cat11:45:231
1602199201,18cyclictest0-21swapper/108:58:391
1602199201,18cyclictest0-21swapper/107:39:351
1602199201,17cyclictest0-21swapper/108:02:211
1602199201,17cyclictest0-21swapper/108:02:211
1602199201,14cyclictest0-21swapper/112:16:491
1602199201,14cyclictest0-21swapper/110:35:031
1602199201,14cyclictest0-21swapper/110:15:341
1602199201,14cyclictest0-21swapper/109:22:271
1602199201,13cyclictest0-21swapper/111:02:001
1602199201,13cyclictest0-21swapper/110:40:111
1602199200,3cyclictest26495-21kworker/u8:109:59:341
1602199200,15cyclictest0-21swapper/110:31:241
1602199200,14cyclictest0-21swapper/109:35:581
46032190,2sleep30-21swapper/309:08:423
321382191,3sleep20-21swapper/212:37:332
294342191,2sleep20-21swapper/211:19:002
1603399195,2cyclictest0-21swapper/310:14:213
1603399192,4cyclictest15716-21munin-run09:35:013
16033991914,3cyclictest28012-21tr07:35:123
16033991914,3cyclictest25289-21grep12:20:263
1603399191,10cyclictest827-21runrttasks08:05:203
1603399191,10cyclictest28511-21/usr/sbin/munin08:50:123
1603399190,7cyclictest0-21swapper/311:53:243
1602699199,2cyclictest30-21rcuc/211:25:142
1602699196,3cyclictest87350irq/126-enp1s0-10:42:142
1602699195,4cyclictest32672-21dump-pmu-power09:00:002
1602699195,4cyclictest0-21swapper/210:35:232
1602699192,7cyclictest579-21systemd-journal12:00:012
1602699192,7cyclictest10934-21chrt11:50:132
1602699192,5cyclictest4459-21fschecks_time11:35:142
1602699192,10cyclictest0-21swapper/210:03:112
1602699191,5cyclictest27292-21awk07:30:252
1602699191,4cyclictest0-21swapper/212:00:122
1602699191,2cyclictest0-21swapper/207:44:552
1602199191,6cyclictest3280-21tr07:50:221
16021991915,3cyclictest0-21swapper/109:30:071
16021991915,3cyclictest0-21swapper/107:14:301
16021991913,3cyclictest11265-21df08:10:121
16021991912,4cyclictest22501-21/usr/sbin/munin08:35:171
1602199191,17cyclictest0-21swapper/108:15:561
1602199191,10cyclictest9804-21ntp_states08:05:211
1602199190,9cyclictest0-21swapper/110:45:191
1602199190,4cyclictest0-21swapper/112:20:251
1602199190,18cyclictest0-21swapper/108:25:531
66812181,3sleep20-21swapper/209:12:062
5112180,2sleep10-21swapper/107:45:161
258592181,2sleep20-21swapper/208:42:162
189672180,2sleep30-21swapper/312:07:453
1603399181,7cyclictest12402-21grep08:10:253
1603399181,7cyclictest0-21swapper/309:47:333
1603399181,6cyclictest19753-21head10:55:193
1603399181,4cyclictest9966-21/usr/sbin/munin10:35:113
1603399181,4cyclictest14358-21/usr/sbin/munin10:45:173
1603399181,3cyclictest3720-21diskstats10:20:143
1603399181,15cyclictest0-21swapper/307:41:243
1603399181,11cyclictest27807-21sshd12:28:223
1603399181,10cyclictest20957-21/usr/sbin/munin11:00:133
1602699185,3cyclictest87350irq/126-enp1s0-07:20:102
1602699184,3cyclictest87350irq/126-enp1s0-10:25:122
1602699184,3cyclictest87350irq/126-enp1s0-10:25:122
1602699184,3cyclictest19247-21cpuspeed_turbos07:15:132
1602699181,7cyclictest4727-21diskmemload11:43:042
1602699181,6cyclictest0-21swapper/208:25:062
1602699181,5cyclictest8030-21cat10:30:102
1602699181,5cyclictest4758-21cat09:10:022
1602699181,5cyclictest21695-21munin-run08:35:012
1602699181,5cyclictest0-21swapper/208:05:262
16026991813,3cyclictest0-21swapper/207:10:122
16026991811,4cyclictest5596-21smart_sda07:55:262
16026991811,4cyclictest15705-21wc10:45:232
1602699181,10cyclictest0-21swapper/211:32:202
1602199181,2cyclictest0-21swapper/112:25:131
16021991811,4cyclictest27566-21sendmail_mailtr08:45:231
16021991810,4cyclictest24462-21diskstats08:40:131
1602199180,17cyclictest0-21swapper/109:02:571
1603399177,3cyclictest87350irq/126-enp1s0-09:50:203
1603399176,4cyclictest0-21swapper/309:25:193
1603399176,3cyclictest773-21NetworkManager09:42:173
1603399172,12cyclictest1786-21/usr/sbin/munin11:30:213
1603399171,9cyclictest0-21swapper/308:20:133
1603399171,6cyclictest24578-21ntp_states07:25:223
1603399171,6cyclictest1813-21taskset07:49:013
1603399171,6cyclictest12502-21df_inode10:40:123
1603399171,6cyclictest0-21swapper/311:29:133
1603399171,6cyclictest0-21swapper/310:01:173
1603399171,3cyclictest23600-21taskset08:36:273
16033991712,3cyclictest3279-21head07:50:223
16033991712,3cyclictest15678-21sort12:00:163
1603399171,14cyclictest0-21swapper/309:01:423
16033991711,3cyclictest9614-21find11:45:223
16033991711,3cyclictest3985-21/usr/sbin/munin11:35:183
16033991711,3cyclictest16219-21/usr/sbin/munin07:10:193
1603399171,12cyclictest0-21swapper/311:14:053
1603399170,9cyclictest0-21swapper/310:30:333
1603399170,2cyclictest0-21swapper/309:19:243
1602699173,3cyclictest87650irq/129-enp1s0-12:20:162
1602699172,4cyclictest14354-21taskset08:15:222
1602699172,12cyclictest26187-21kernelversion11:10:172
1602699171,7cyclictest538-21chrt10:11:082
1602699171,7cyclictest0-21swapper/210:15:152
1602699171,4cyclictest22302-21basename12:15:152
1602699171,4cyclictest22102-21cstates08:35:132
1602699171,4cyclictest1984-21cat07:50:012
1602699171,4cyclictest13755-21cpu09:30:112
16026991713,3cyclictest0-21swapper/209:42:582
16026991712,3cyclictest10350-21latency_hist11:50:012
1602699171,15cyclictest0-21swapper/209:02:272
1602699171,14cyclictest0-21swapper/208:10:242
1602699171,14cyclictest0-21swapper/207:37:422
1602699171,10cyclictest0-21swapper/209:29:222
1603399166,3cyclictest87650irq/129-enp1s0-12:15:203
1603399161,9cyclictest0-21swapper/311:44:103
1603399161,6cyclictest5721-21if_err_enp1s009:10:183
1603399161,5cyclictest4674-21df_inode07:55:163
1603399161,5cyclictest0-21swapper/308:00:133
1603399161,5cyclictest0-21swapper/308:00:133
1603399161,14cyclictest0-21swapper/308:29:103
1603399161,12cyclictest0-21swapper/307:23:293
1603399161,11cyclictest0-21swapper/308:16:333
1602699162,5cyclictest20957-21/usr/sbin/munin11:00:182
16026991612,3cyclictest0-21swapper/208:50:372
16026991612,3cyclictest0-21swapper/208:02:422
16026991612,3cyclictest0-21swapper/208:02:422
1603399155,3cyclictest87350irq/126-enp1s0-08:45:123
16033991510,3cyclictest27739-21wc07:35:023
1602699159,3cyclictest15930-21gltestperf08:20:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional