You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-16 - 15:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Tue Jun 16, 2026 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
194452237160,25sleep00-21swapper/007:09:150
192512237163,24sleep30-21swapper/307:06:453
193782233158,24sleep20-21swapper/207:08:252
194362232163,56sleep10-21swapper/107:09:091
639721770,4sleep01958799cyclictest10:16:080
19598997471,2cyclictest0-21swapper/208:45:072
174032490,3sleep10-21swapper/110:41:251
1960499428,24cyclictest26078-21unixbench_singl11:00:253
19598994138,2cyclictest0-21swapper/212:35:522
19593994011,8cyclictest14818-21hddtemp_smartct11:50:171
19604993815,15cyclictest29262-21df11:10:123
1959399383,22cyclictest1285-21/usr/sbin/munin07:40:281
19604993710,13cyclictest10404-21sshd09:14:023
19593993712,16cyclictest938-21/usr/sbin/munin11:20:301
1959399369,13cyclictest16339-21/usr/sbin/munin11:55:191
19593993611,12cyclictest25265-21cpuspeed_turbos12:15:121
19593993610,15cyclictest23296-21/usr/sbin/munin08:30:151
19593993610,13cyclictest3334-21timerwakeupswit08:55:241
1958799367,13cyclictest18680-21basename08:20:000
19604993513,14cyclictest24582-21hddtemp_smartct09:45:143
1959399358,16cyclictest4118-21/usr/sbin/munin09:00:221
1959399358,12cyclictest8294-21munin-run09:10:021
19593993514,12cyclictest1914-21df12:35:141
1959399351,26cyclictest0-21swapper/109:56:111
19593993510,15cyclictest819-21lldpd11:37:311
19593993510,12cyclictest7666-21latency_hist07:55:021
1958799358,12cyclictest2947-21awk11:25:010
19587993510,12cyclictest28074-21munin-run09:55:010
19604993411,9cyclictest14352-21/usr/sbin/munin10:35:183
19604993410,11cyclictest23792-21grep10:55:253
19604993410,11cyclictest23792-21grep10:55:253
1959399348,20cyclictest12357-21kworker/u8:209:25:151
19593993414,11cyclictest32068-21/usr/sbin/munin12:30:121
19593993411,12cyclictest29662-21perf08:45:011
1960499337,16cyclictest21306-21perl12:05:163
1960499337,16cyclictest21306-21perl12:05:163
19604993312,11cyclictest29541-21/usr/sbin/munin12:25:173
19604993311,6cyclictest8290-21diskmemload11:38:093
1959399339,11cyclictest24731-21cat07:20:011
1959399339,10cyclictest9030-21cat10:25:011
1959399338,13cyclictest4511-21latency_hist10:15:011
1959399337,11cyclictest827-21runrttasks08:45:121
1959399335,7cyclictest28423-21kworker/u8:011:07:221
1959399333,24cyclictest0-21swapper/110:55:241
1959399333,24cyclictest0-21swapper/110:55:241
19593993313,16cyclictest25695-40dnf11:00:241
19593993312,8cyclictest87550irq/128-enp1s0-09:20:261
19593993312,6cyclictest773-21NetworkManager10:52:381
19593993311,13cyclictest31023-21cat11:15:021
1960499328,8cyclictest1-21systemd09:30:003
19604993221,6cyclictest30278-40dnf10:00:043
1959399328,12cyclictest11684-21diskstats10:30:131
1959399328,11cyclictest5528-21chrt07:50:021
1959399327,12cyclictest21626-21latency12:05:181
1959399327,12cyclictest21626-21latency12:05:181
1959399326,13cyclictest808-21snmpd07:12:181
1959399325,12cyclictest0-21swapper/109:17:041
19593993215,7cyclictest9333-21irqstats09:10:181
1959399321,24cyclictest0-21swapper/112:20:101
19593993211,10cyclictest808-21snmpd10:37:581
19593993210,19cyclictest26845-21latency_hist07:25:001
19593993210,19cyclictest25838-21diskstats08:35:141
19593993210,19cyclictest25838-21diskstats08:35:141
19593993210,15cyclictest808-21snmpd11:28:081
1960499317,14cyclictest8868-21taskset10:24:373
1960499316,13cyclictest23371-21cat08:30:133
1960499314,15cyclictest27683-21/usr/sbin/munin08:40:143
19604993111,17cyclictest14244-21grep11:50:133
19604993110,12cyclictest24636-21sshd12:11:313
1959399319,19cyclictest10329-21cut11:40:161
1959399317,18cyclictest19539-21kworker/u8:010:48:061
1959399317,12cyclictest6031-21switchtime10:15:221
1959399317,12cyclictest24377-21sort09:45:131
19593993115,13cyclictest21092-21sed09:35:271
19593993114,5cyclictest20152-40dnf12:00:301
1959399311,23cyclictest0-21swapper/110:07:171
19593993112,12cyclictest26916-21irqstats09:50:171
19593993111,12cyclictest0-21swapper/110:25:181
19593993110,11cyclictest87550irq/128-enp1s0-07:30:191
1960499307,13cyclictest11335-21/usr/sbin/munin10:30:163
1960499307,10cyclictest14270-21latency_hist08:10:003
1960499306,9cyclictest0-21swapper/309:24:103
1960499306,8cyclictest16522-21cstates11:55:143
1960499306,13cyclictest31468-21/usr/sbin/munin07:35:263
1960499306,13cyclictest0-21swapper/311:15:123
1960499304,23cyclictest26732-21/usr/sbin/munin11:05:253
19604993011,6cyclictest18714-21/usr/sbin/munin10:45:173
19598993011,9cyclictest21095-21/usr/sbin/munin08:25:272
1959399309,11cyclictest17308-21/usr/sbin/munin09:30:231
1959399308,12cyclictest21707-21/usr/sbin/munin09:40:181
1959399306,14cyclictest0-21swapper/108:00:291
19593993020,6cyclictest9803-40dnf07:59:241
1959399301,23cyclictest87650irq/129-enp1s0-10:01:091
19593993010,6cyclictest31468-21/usr/sbin/munin07:35:251
19593993010,17cyclictest19041-21cpuspeed_turbos08:20:121
19593993010,10cyclictest12050irq/123-ahci[0011:35:021
1960499298,12cyclictest8290-21diskmemload09:37:463
1960499296,10cyclictest25820-21missed_timers07:20:193
19604992910,9cyclictest2880-21df10:10:153
19604992910,15cyclictest5699-21/usr/sbin/munin07:50:203
1959899293,15cyclictest8607-21/usr/sbin/munin09:10:112
19598992915,5cyclictest9864-21latency_hist08:00:022
1959399299,17cyclictest13218-21/usr/sbin/munin11:45:231
1959399298,13cyclictest808-21snmpd08:17:311
1959399297,12cyclictest0-21swapper/112:11:501
19593992910,8cyclictest32377-21awk11:15:241
1960499289,16cyclictest26012-21cut08:35:163
1960499289,16cyclictest26012-21cut08:35:163
1960499287,13cyclictest8290-21diskmemload10:26:583
1960499286,18cyclictest9751-21/usr/sbin/munin11:40:163
1960499284,9cyclictest31915-21cstates12:30:123
1959899285,8cyclictest31755-21gltestperf11:15:152
19598992814,5cyclictest5714-21bc07:50:122
19598992811,3cyclictest0-21swapper/211:45:122
1959399289,9cyclictest29665-21expr12:25:131
1960499277,17cyclictest1570-21/usr/sbin/munin12:35:273
1960499277,10cyclictest0-21swapper/309:15:113
1960499276,12cyclictest0-21swapper/308:00:123
19604992720,4cyclictest2894-21needreboot08:55:183
19604992710,5cyclictest9964-21cat08:00:013
1959399278,17cyclictest0-21swapper/108:25:141
1959399275,7cyclictest0-21swapper/108:10:251
19593992710,4cyclictest28547-21grep07:25:271
1958799273,11cyclictest1691-21munin-run08:55:020
1960499269,14cyclictest7835-21unixbench_multi09:05:263
1960499267,9cyclictest87350irq/126-enp1s0-10:40:193
1960499266,9cyclictest5353-21/usr/sbin/munin11:30:113
19604992618,5cyclictest30531-21smart_sda07:30:233
19604992610,12cyclictest4730-21gltestperf09:00:153
1959899262,11cyclictest1-21systemd10:39:302
19598992612,6cyclictest87350irq/126-enp1s0-12:15:012
19598992612,5cyclictest808-21snmpd09:52:502
1959899261,16cyclictest0-21swapper/209:15:182
1960499256,7cyclictest25976-21sed12:15:203
1960499256,16cyclictest20128-21/usr/sbin/munin10:50:233
1960499256,15cyclictest32710-21/usr/sbin/munin10:05:143
19604992518,4cyclictest27018-21/usr/sbin/munin09:50:183
19604992518,4cyclictest1289-21diskstats11:20:153
19604992517,5cyclictest3504-21diskstats11:25:133
19604992515,7cyclictest4266-21/usr/sbin/munin07:45:163
19604992515,7cyclictest19944-21/usr/sbin/munin12:00:253
19604992515,7cyclictest12678-21iwlist11:45:153
1959899251,15cyclictest0-21swapper/210:00:042
19598992510,4cyclictest19488-21gltestperf08:20:162
1959899250,15cyclictest790-21in:imjournal12:20:012
19593992517,5cyclictest379-21/usr/sbin/munin08:50:181
19593992515,7cyclictest12652-21diskstats08:05:151
1958799251,13cyclictest25882-21munin-run09:50:010
1958799251,13cyclictest24298-21munin-run11:00:020
1958799251,13cyclictest24298-21munin-run11:00:020
1958799251,12cyclictest8915-21munin-run10:25:010
255132241,3sleep025758-21head12:15:160
1960499249,12cyclictest22304-21grep09:40:163
1960499248,8cyclictest87350irq/126-enp1s0-09:59:583
19604992416,5cyclictest22165-21ntp_states08:25:223
19604992415,6cyclictest18891-21/usr/sbin/munin08:20:193
19604992415,6cyclictest18371-21/usr/sbin/munin09:30:203
19604992415,6cyclictest15631-21grep08:10:223
19604992415,5cyclictest5088-21diskstats10:15:133
19604992410,11cyclictest28337-21/usr/sbin/munin07:25:243
1959899247,9cyclictest18245-1kworker/u9:210:34:322
1959899247,3cyclictest10362-21gunzip10:25:232
1959899244,12cyclictest755-21rngd09:45:222
1959899241,16cyclictest0-21swapper/208:50:132
1959899241,12cyclictest0-21swapper/209:20:172
1959899241,10cyclictest0-21swapper/211:01:172
19604992316,4cyclictest704-21ntp_states08:50:223
19604992316,4cyclictest28118-21/usr/sbin/munin12:20:173
19604992316,4cyclictest1663-21/usr/sbin/munin07:40:153
19604992315,5cyclictest17046-21/usr/sbin/munin08:15:143
19604992314,6cyclictest29941-21/usr/sbin/munin08:45:113
19604992314,6cyclictest22681-21/usr/sbin/munin07:15:113
19604992314,6cyclictest20617-21sed07:10:183
1959899239,7cyclictest87550irq/128-enp1s0-09:01:302
1959899236,3cyclictest0-21swapper/211:35:322
1959899234,9cyclictest87650irq/129-enp1s0-10:59:572
1959899234,9cyclictest87650irq/129-enp1s0-10:59:572
1959899231,9cyclictest0-21swapper/211:07:062
19598992317,3cyclictest17671-21/usr/sbin/munin09:30:152
1959899231,10cyclictest0-21swapper/211:29:522
1958799233,4cyclictest0-21swapper/011:57:500
1958799231,11cyclictest14288-21idleruntime-cro08:10:000
1959899223,9cyclictest1182-21latency_hist07:40:012
1959899222,12cyclictest29276-21/usr/sbin/munin09:55:202
1959899221,7cyclictest0-21swapper/207:45:252
1959899221,3cyclictest0-21swapper/210:22:312
1959899221,3cyclictest0-21swapper/207:22:192
1959899221,10cyclictest0-21swapper/211:42:152
1959899221,10cyclictest0-21swapper/211:12:552
1959899220,19cyclictest4512-21sh10:15:022
1958799229,4cyclictest13013-21ntp_states11:45:200
1958799227,3cyclictest0-21swapper/012:27:230
1958799221,5cyclictest0-21swapper/012:07:300
1958799221,5cyclictest0-21swapper/012:07:300
1958799221,11cyclictest22384-21kworker/0:211:10:010
1958799220,5cyclictest0-21swapper/009:26:400
1959899215,9cyclictest87550irq/128-enp1s0-07:42:382
1959899211,9cyclictest827-21runrttasks11:57:252
1959899211,8cyclictest0-21swapper/211:53:092
1959899211,14cyclictest0-21swapper/210:09:352
1959899210,11cyclictest0-21swapper/207:31:032
1958799217,5cyclictest0-21swapper/009:44:140
1958799217,5cyclictest0-21swapper/009:30:350
1958799216,6cyclictest0-21swapper/012:22:330
1958799211,6cyclictest0-21swapper/011:04:150
1958799211,5cyclictest29970-21kworker/0:110:01:130
1958799211,4cyclictest0-21swapper/011:51:420
1958799211,4cyclictest0-21swapper/009:13:450
1958799211,13cyclictest0-21swapper/008:55:190
1958799210,5cyclictest0-21swapper/010:40:570
1958799210,4cyclictest0-21swapper/010:53:230
200972200,4sleep230-21rcuc/210:50:012
1959899205,7cyclictest12304-21/usr/sbin/munin08:05:142
1959899203,9cyclictest19207-21hddtemp_smartct12:00:152
1959899202,5cyclictest20576-21ntp_states09:35:232
1959899201,9cyclictest0-21swapper/211:20:272
1959899201,8cyclictest0-21swapper/210:53:472
1959899201,5cyclictest0-21swapper/212:28:242
1959899201,3cyclictest23302-21vmstat09:40:292
19598992012,5cyclictest32071-21df12:30:122
1959899201,12cyclictest0-21swapper/208:55:032
1959899201,11cyclictest0-21swapper/212:09:442
1959899201,11cyclictest0-21swapper/212:09:442
1959899201,11cyclictest0-21swapper/208:40:162
1959899201,11cyclictest0-21swapper/208:15:182
1959899201,10cyclictest27688-21sh12:20:132
1958799209,3cyclictest1570-21/usr/sbin/munin12:35:150
1958799204,9cyclictest11-21rcuc/012:30:150
1958799203,9cyclictest11-21rcuc/007:35:170
1958799203,10cyclictest11-21rcuc/011:25:170
19587992014,4cyclictest10206-21grep08:00:120
19587992014,4cyclictest0-21swapper/010:12:530
1958799201,13cyclictest0-21swapper/011:41:100
1958799201,11cyclictest5703-21diskstats11:30:120
1958799201,11cyclictest3500-21/usr/sbin/munin07:45:160
1958799201,11cyclictest30704-21kernelversion08:45:180
1958799201,11cyclictest21095-21/usr/sbin/munin08:25:250
1958799201,11cyclictest11335-21/usr/sbin/munin10:30:240
1958799201,11cyclictest0-21swapper/010:35:150
1958799201,10cyclictest24189-21tail07:15:230
1958799201,10cyclictest0-21swapper/007:40:180
1958799200,4cyclictest13722-21kworker/u8:112:01:250
155672200,4sleep230-21rcuc/208:10:222
1959899194,2cyclictest6261-21load11:30:182
1959899192,7cyclictest32-21ksoftirqd/208:05:022
1959899192,5cyclictest27078-21users08:35:282
1959899192,5cyclictest27078-21users08:35:282
1959899191,9cyclictest15444-21df09:25:132
1959899191,9cyclictest0-21swapper/207:19:282
1959899191,6cyclictest0-21swapper/208:30:122
1959899191,10cyclictest6225-21unixbench_multi10:15:242
1958799199,3cyclictest9547-21grep10:25:140
1958799199,3cyclictest4118-21/usr/sbin/munin09:00:210
1958799197,4cyclictest0-21swapper/011:16:580
1958799193,2cyclictest87350irq/126-enp1s0-10:07:420
1958799192,15cyclictest8285-21idleruntime-cro09:10:020
1958799191,6cyclictest0-21swapper/009:23:200
1958799191,6cyclictest0-21swapper/008:40:160
1958799191,2cyclictest0-21swapper/010:47:170
1958799191,10cyclictest9501-21sshd07:55:280
1958799191,10cyclictest28333-21cpu09:55:120
1958799191,10cyclictest27077-21/usr/sbin/munin07:25:250
1958799191,10cyclictest19952-21ntpq08:20:200
1958799191,10cyclictest11709-21cut09:15:180
1958799191,10cyclictest0-21swapper/011:12:100
1958799191,10cyclictest0-21swapper/009:38:080
245952181,2sleep00-21swapper/012:10:590
215072180,2sleep20-21swapper/207:11:562
1959899181,9cyclictest808-21snmpd10:40:092
1958799188,2cyclictest7551-21/usr/sbin/munin11:35:210
1958799187,3cyclictest24350-21ntpq08:30:210
1958799186,7cyclictest0-21swapper/008:36:290
1958799186,7cyclictest0-21swapper/008:36:290
74722171,2sleep00-21swapper/007:52:380
1959899171,9cyclictest0-21swapper/207:25:242
1958799174,6cyclictest11-21rcuc/007:20:200
197792160,2sleep00-21swapper/007:10:040
1959899161,9cyclictest0-21swapper/209:08:092
1958799166,3cyclictest87550irq/128-enp1s0-08:10:160
19587991612,3cyclictest0-21swapper/007:34:370
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional