You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-23 - 00:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Wed Apr 22, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
135392237161,25sleep00-21swapper/007:08:120
136252204167,25sleep20-21swapper/207:09:202
134022201166,23sleep30-21swapper/307:06:263
134102200164,24sleep10-21swapper/107:06:331
2653621860,4sleep31378399cyclictest08:50:133
13778996461,2cyclictest0-21swapper/211:57:562
299272520,4sleep21377899cyclictest12:38:062
13767994211,28cyclictest19110-21munin-run12:15:020
13767994211,28cyclictest19110-21munin-run12:15:020
13767994010,14cyclictest6256-21munin-run08:05:010
1376799386,15cyclictest27616-21munin-run07:40:010
1376799385,3cyclictest12050irq/123-ahci[0011:15:020
1376799382,22cyclictest396-21/usr/sbin/munin09:05:160
1376799371,4cyclictest28120-21/usr/sbin/munin12:35:170
1376799369,24cyclictest11884-21basename10:45:000
1376799358,3cyclictest0-21swapper/012:15:160
1376799352,5cyclictest26926-21tail08:50:160
13767993513,19cyclictest12105-21/usr/sbin/munin10:45:150
1376799351,10cyclictest25309-21/usr/sbin/munin11:15:160
1376799349,2cyclictest10992-21kworker/u8:109:45:220
1376799349,2cyclictest10992-21kworker/u8:109:45:220
1376799347,12cyclictest24473-21idleruntime-cro10:00:020
1376799342,28cyclictest21188-21grep08:35:250
1376799342,28cyclictest21188-21grep08:35:240
13767993411,20cyclictest27841-21/usr/sbin/munin07:40:200
13767993411,20cyclictest23448-21/usr/sbin/munin07:30:220
13767993410,21cyclictest9294-21/usr/sbin/munin09:25:250
13767993410,20cyclictest827-21runrttasks08:30:230
1376799336,4cyclictest29157-21sshd11:21:070
1376799333,26cyclictest18868-21cpuspeed_turbos11:00:120
1376799331,9cyclictest0-21swapper/011:58:070
1376799331,6cyclictest6132-21/usr/sbin/munin11:45:230
1376799331,2cyclictest87350irq/126-enp1s0-09:50:170
13767993311,18cyclictest23504-21munin-run12:25:010
13767993310,20cyclictest3705-21sh11:40:020
102262330,4sleep011-21rcuc/008:10:240
1376799328,12cyclictest12859-21perf08:20:000
1376799324,25cyclictest5909-21munin-run11:45:010
1376799322,27cyclictest32639-21ls09:05:020
1376799321,28cyclictest5517-21/usr/sbin/munin10:30:120
13767993211,18cyclictest827-21runrttasks08:28:070
13767993211,18cyclictest23731-21/usr/sbin/munin12:25:160
13767993210,19cyclictest827-21runrttasks07:28:250
13767993210,19cyclictest827-21runrttasks07:28:240
1376799318,3cyclictest1203-21cat07:50:230
1376799314,12cyclictest9696-21idleruntime-cro10:40:000
1376799311,8cyclictest7258-21kworker/u8:211:50:190
1376799311,27cyclictest23331-21grep08:40:260
13767993112,16cyclictest4672-21sh09:15:020
13767993110,18cyclictest827-21runrttasks09:44:380
13767993110,18cyclictest31899-21/usr/sbin/munin11:30:230
143762300,2sleep20-21swapper/207:10:152
1376799306,6cyclictest0-21swapper/012:05:220
1376799302,4cyclictest31770-40dnf07:47:070
1376799301,8cyclictest2476-21diskmemload12:30:370
1376799301,4cyclictest1100-21/usr/sbin/munin10:20:120
13767993012,15cyclictest25773-21open_files10:00:210
13767993011,16cyclictest17277-21cat10:55:170
13767993011,16cyclictest17277-21cat10:55:170
13767993010,17cyclictest827-21runrttasks11:10:010
13767993010,17cyclictest827-21runrttasks07:10:440
13767993010,17cyclictest7097-21/usr/sbin/munin09:20:170
1376799299,17cyclictest14640-21missed_timers09:35:190
1376799297,2cyclictest24062-21/usr/sbin/munin08:45:120
1376799294,3cyclictest0-21swapper/010:19:110
1376799293,3cyclictest13071-21df_inode12:00:140
1376799291,9cyclictest0-21swapper/010:07:520
1376799291,4cyclictest6481-21/usr/sbin/munin08:05:240
1376799291,13cyclictest30444-21awk09:00:020
243622280,2sleep30-21swapper/309:58:583
1376799287,3cyclictest16299-21unixbench_singl07:15:010
1376799287,3cyclictest11867-21fschecks_count09:30:160
1376799280,11cyclictest0-21swapper/010:10:210
1378399279,10cyclictest0-21swapper/308:32:443
13778992710,3cyclictest0-21swapper/209:42:152
13772992710,3cyclictest32247-21diskstats11:30:131
1376799276,3cyclictest14465-21grep10:50:110
1376799276,3cyclictest13217-21cpuspeed_turbos08:20:120
1376799272,4cyclictest21023-21munin-run07:25:010
1376799272,22cyclictest29702-21/usr/sbin/munin11:25:180
1376799271,4cyclictest3141-21ntp_states07:55:200
1376799271,4cyclictest2476-21diskmemload10:25:130
13767992710,14cyclictest6847-21seq09:19:550
1377899261,6cyclictest0-21swapper/212:06:542
1377299252,16cyclictest87550irq/128-enp1s0-12:09:111
13772992515,3cyclictest5143-21expr09:15:131
1378399247,2cyclictest0-21swapper/312:25:253
1378399244,3cyclictest87350irq/126-enp1s0-12:00:003
1378399241,8cyclictest0-21swapper/311:45:113
1377899248,3cyclictest19290-21ls12:15:012
1377899248,3cyclictest19290-21ls12:15:012
1377899247,8cyclictest87350irq/126-enp1s0-10:40:022
1377899241,12cyclictest32-21ksoftirqd/212:20:112
13772992413,6cyclictest89250irq/135-enp2s0-10:14:461
1377299241,17cyclictest0-21swapper/111:43:291
13772992411,5cyclictest27100-21kworker/u8:111:19:331
1377299240,6cyclictest0-21swapper/110:46:361
1378399231,8cyclictest0-21swapper/307:42:353
1377899236,3cyclictest15731-21fschecks_count08:25:152
1377899235,8cyclictest87350irq/126-enp1s0-07:30:142
1377899233,8cyclictest22680-21taskset11:07:322
1377899233,7cyclictest32019-21awk07:50:022
1377899232,8cyclictest0-21swapper/209:01:472
1377299237,5cyclictest0-21swapper/111:20:161
1377299231,6cyclictest22261-21munin-run09:55:021
13772992313,2cyclictest31465-21cstates10:15:131
13772992313,2cyclictest15039-21crond08:25:001
1377299231,16cyclictest0-21swapper/107:51:351
1377299230,18cyclictest87650irq/129-enp1s0-10:51:051
1378399222,2cyclictest87350irq/126-enp1s0-12:20:113
1378399221,9cyclictest98592cat10:40:013
1377899226,3cyclictest87550irq/128-enp1s0-09:45:242
1377899226,3cyclictest87550irq/128-enp1s0-09:45:242
13778992211,3cyclictest87650irq/129-enp1s0-08:50:212
13778992211,2cyclictest87650irq/129-enp1s0-07:35:222
1377899220,12cyclictest311ktimersoftd/211:15:212
1377299221,17cyclictest7269-21if_err_enp2s008:05:171
1377299221,16cyclictest0-21swapper/110:58:531
1377299221,16cyclictest0-21swapper/110:58:531
1377299221,16cyclictest0-21swapper/109:40:531
1377299221,15cyclictest0-21swapper/110:01:101
1377299220,16cyclictest0-21swapper/109:26:091
1378399219,8cyclictest41-21ksoftirqd/307:15:023
1378399215,4cyclictest0-21swapper/312:00:043
1378399214,3cyclictest30025-21seq07:45:033
13783992114,4cyclictest9347-21gltestperf08:10:153
1377899217,5cyclictest24538-21interrupts12:25:182
1377899216,4cyclictest0-21swapper/210:13:362
1377899216,3cyclictest5163-21rt-features11:40:232
1377899214,9cyclictest21358-21ntp_states09:50:212
1377899211,10cyclictest0-21swapper/212:32:232
1377899211,10cyclictest0-21swapper/209:55:252
1377899211,10cyclictest0-21swapper/209:36:002
1377299218,6cyclictest0-21swapper/112:16:301
13772992117,2cyclictest19139-21ntpq09:45:191
13772992117,2cyclictest19139-21ntpq09:45:191
13772992116,3cyclictest29630-21unixbench_multi12:35:241
1377299211,3cyclictest21867-21ls11:05:181
13772992113,3cyclictest0-21swapper/108:52:291
1377299211,2cyclictest0-21swapper/108:25:241
1377299211,15cyclictest0-21swapper/111:47:341
1377299211,15cyclictest0-21swapper/111:14:531
1377299211,15cyclictest0-21swapper/110:07:311
1377299211,15cyclictest0-21swapper/109:56:421
1377299211,15cyclictest0-21swapper/108:36:211
1377299211,15cyclictest0-21swapper/108:36:211
1377299211,15cyclictest0-21swapper/108:30:291
1377299211,14cyclictest0-21swapper/109:24:151
13772992110,9cyclictest221ktimersoftd/109:11:171
13772992110,3cyclictest18255-21timerwakeupswit07:15:231
1377299210,15cyclictest0-21swapper/107:25:131
1377299210,15cyclictest0-21swapper/107:25:131
118672211,4sleep111880-21munin-run10:45:001
250522201,2sleep10-21swapper/107:30:301
1378399207,4cyclictest39-21rcuc/309:25:143
1378399203,3cyclictest24708-21smart_sda07:30:233
1378399203,2cyclictest41-21ksoftirqd/310:00:193
1378399201,9cyclictest15382-21ntp_states10:50:213
1378399201,2cyclictest87650irq/129-enp1s0-10:43:163
1378399201,11cyclictest23075-21sshd07:28:183
1378399201,11cyclictest23075-21sshd07:28:183
1378399201,11cyclictest20447-21sshd11:02:073
1378399201,11cyclictest13690-21/usr/sbin/munin09:35:213
1378399200,11cyclictest0-21swapper/310:22:553
1377899209,3cyclictest87350irq/126-enp1s0-08:00:132
1377899207,7cyclictest87650irq/129-enp1s0-11:10:152
1377899207,5cyclictest87350irq/126-enp1s0-10:20:232
1377899206,3cyclictest87350irq/126-enp1s0-09:15:242
1377899205,2cyclictest0-21swapper/211:00:592
1377899204,8cyclictest28381-21unixbench_multi10:05:242
1377899204,6cyclictest24742-21cpu10:00:122
1377899204,4cyclictest32-21ksoftirqd/209:25:172
1377899203,8cyclictest8558-21expr11:50:122
1377899203,8cyclictest20248-21memory12:15:172
1377899201,5cyclictest0-21swapper/211:48:202
13778992014,3cyclictest22747-21grep07:25:262
13778992014,3cyclictest22747-21grep07:25:262
13778992013,4cyclictest827-21runrttasks08:05:182
1377899200,10cyclictest0-21swapper/210:41:042
1377299209,3cyclictest19053-21diskstats11:00:131
1377299207,5cyclictest0-21swapper/112:00:191
1377299201,8cyclictest0-21swapper/110:38:501
1377299201,7cyclictest808-21snmpd11:57:221
13772992014,3cyclictest27512-21users12:30:261
1377299201,18cyclictest0-21swapper/107:24:021
1377299201,14cyclictest0-21swapper/108:18:181
1377299201,14cyclictest0-21swapper/107:56:071
1377299201,14cyclictest0-21swapper/107:43:301
1377299201,14cyclictest0-21swapper/107:37:061
1377299201,11cyclictest30941-21expr09:00:121
1377299201,11cyclictest16906-21dump-pmu-power12:14:561
1377299201,11cyclictest16906-21dump-pmu-power12:14:561
1377299201,11cyclictest1279-21cat09:05:161
1377299201,11cyclictest10688-21cat08:15:001
1377299201,10cyclictest4572-21ls08:00:121
271822190,2sleep30-21swapper/307:35:233
1378399198,3cyclictest16856-21/usr/sbin/munin07:15:183
1378399197,5cyclictest87650irq/129-enp1s0-10:55:253
1378399197,5cyclictest87650irq/129-enp1s0-10:55:253
1378399195,4cyclictest41-21ksoftirqd/312:35:133
1378399192,9cyclictest9922-21users11:50:253
1378399191,9cyclictest0-21swapper/310:07:573
1378399191,4cyclictest0-21swapper/312:06:093
13783991913,3cyclictest2334-21gltestperf11:35:163
13783991912,4cyclictest29427-21df_inode10:10:123
1378399191,10cyclictest2077-21/usr/sbin/munin07:55:213
1378399191,10cyclictest20288-21/usr/sbin/munin09:50:223
1378399191,10cyclictest18085-21/usr/sbin/munin09:45:233
1378399191,10cyclictest18085-21/usr/sbin/munin09:45:233
1377899199,3cyclictest87650irq/129-enp1s0-09:10:012
1377899196,3cyclictest14523-21cat10:50:122
1377899195,3cyclictest87650irq/129-enp1s0-11:25:212
1377899193,8cyclictest31911-21grep10:15:162
1377899192,4cyclictest12778-21hddtemp_smartct10:45:152
1377899191,8cyclictest13187-21grep12:00:152
13778991913,3cyclictest28969-21grep11:20:242
1377899191,11cyclictest0-21swapper/211:37:442
1377899191,10cyclictest0-21swapper/209:10:172
1377899191,10cyclictest0-21swapper/207:20:172
1377299193,7cyclictest0-21swapper/112:23:551
1377299192,14cyclictest11628-21tr09:30:121
13772991912,2cyclictest0-21swapper/108:42:201
1377299191,14cyclictest0-21swapper/110:22:001
1377299191,14cyclictest0-21swapper/108:55:591
1377299191,10cyclictest5520-21acpi10:30:111
1377299191,10cyclictest31093-21ntpq07:45:221
1377299191,10cyclictest24658-21gltestperf08:45:141
1377299191,10cyclictest14946-21needreboot07:10:191
1377299190,14cyclictest0-21swapper/110:29:241
1378399184,7cyclictest28719-21expr08:55:143
1378399184,4cyclictest12374-21cat09:30:183
1378399184,3cyclictest87650irq/129-enp1s0-09:15:123
1378399184,3cyclictest39-21rcuc/311:25:123
1378399183,3cyclictest87650irq/129-enp1s0-11:40:233
1378399181,9cyclictest808-21snmpd12:12:243
1378399181,9cyclictest808-21snmpd12:12:243
1378399181,9cyclictest0-21swapper/311:21:403
1378399181,7cyclictest26871-21missed_timers12:30:183
1378399181,4cyclictest0-21swapper/309:10:253
1378399181,3cyclictest87350irq/126-enp1s0-09:42:263
13783991811,4cyclictest8373-21switchtime09:20:233
1378399181,14cyclictest4746-21fschecks_time08:00:143
13783991811,4cyclictest25432-21cut11:15:133
1378399181,10cyclictest20905-21sshd11:05:073
1377899183,12cyclictest24170-21cpuspeed_turbos08:45:112
1377899182,14cyclictest0-21swapper/207:50:042
1377899182,10cyclictest0-21swapper/211:30:112
1377899181,9cyclictest0-21swapper/209:20:242
1377899181,7cyclictest0-21swapper/210:34:352
1377899181,15cyclictest0-21swapper/208:14:372
1377899181,11cyclictest0-21swapper/209:32:512
1377899181,10cyclictest0-21swapper/208:15:142
1377899180,9cyclictest0-21swapper/210:25:132
1377899180,8cyclictest29631-21ps08:55:232
1377899180,10cyclictest16511-21/usr/sbin/munin10:55:252
1377899180,10cyclictest16511-21/usr/sbin/munin10:55:252
1377299182,2cyclictest2476-21diskmemload11:35:161
1377299181,9cyclictest9249-21memory11:50:191
1377299181,3cyclictest14650-21sort09:35:191
1377299181,2cyclictest0-21swapper/111:26:201
1378399179,4cyclictest0-21swapper/308:25:133
1378399178,3cyclictest0-21swapper/311:10:233
1378399177,2cyclictest19668-21/usr/sbin/munin08:35:173
1378399177,2cyclictest19668-21/usr/sbin/munin08:35:173
1378399175,2cyclictest3555-21cstates10:25:123
1378399174,8cyclictest25816-40dnf08:47:253
1378399174,3cyclictest39-21rcuc/311:30:273
1378399172,12cyclictest32178-21grep09:00:263
1378399171,7cyclictest20094-21awk12:15:153
1378399171,7cyclictest0-21swapper/310:45:153
1378399171,5cyclictest11219-21df_inode08:15:143
1378399171,4cyclictest0-21swapper/310:33:353
13783991713,3cyclictest0-21swapper/309:09:583
1378399171,14cyclictest0-21swapper/307:52:003
13783991711,3cyclictest31283-21/usr/sbin/munin10:15:223
1377899178,3cyclictest0-21swapper/207:59:132
1377899176,3cyclictest87350irq/126-enp1s0-08:35:262
1377899176,3cyclictest87350irq/126-enp1s0-08:35:262
1377899173,11cyclictest14069-21needreboot08:20:202
1377899171,13cyclictest18963-21cat07:20:012
1377299171,8cyclictest0-21swapper/112:25:091
297062161,2sleep20-21swapper/207:43:512
1378399163,4cyclictest7945-21unixbench-2d08:05:243
1378399161,2cyclictest0-21swapper/307:20:303
1378399160,14cyclictest0-21swapper/308:22:033
1377899161,9cyclictest0-21swapper/208:40:252
1377899161,6cyclictest0-21swapper/208:30:142
13783991510,3cyclictest22227-21forks08:40:153
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional