You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-06 - 22:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Sep 06, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13652237163,23sleep20-21swapper/207:07:202
15632203166,24sleep30-21swapper/307:09:423
13272202166,24sleep10-21swapper/107:06:501
14602201164,25sleep00-21swapper/007:08:340
1689996958,6cyclictest0-21swapper/210:04:312
75222560,5sleep2168999cyclictest12:15:122
120872470,2sleep00-21swapper/011:10:210
121492440,1sleep10-21swapper/111:10:231
169599431,31cyclictest28558-21tune2fs10:35:153
167899400,14cyclictest2627-21idleruntime-cro12:05:020
169599392,26cyclictest32148-21munin-run10:45:013
169599391,29cyclictest8822-21/usr/sbin/munin11:05:133
169599391,29cyclictest25168-21/usr/sbin/munin09:15:183
169599391,28cyclictest19915-21latency_hist07:50:023
169599391,25cyclictest0-21swapper/311:10:213
169599391,24cyclictest1-21systemd08:20:083
167899394,7cyclictest13303-40dnf12:26:240
169599381,28cyclictest25786-21/usr/sbin/munin10:30:143
167899384,6cyclictest29556-40dnf09:25:100
169599374,21cyclictest23018-21cpu09:10:143
169599372,22cyclictest22454-21ntp_states10:20:213
1695993716,16cyclictest29742-21cstates09:25:103
169599371,5cyclictest20500-21grep11:30:163
169599364,21cyclictest14797-21/usr/sbin/munin10:05:173
169599361,7cyclictest0-21swapper/311:16:123
169599361,26cyclictest12574-21missed_timers12:25:203
169599361,25cyclictest14721-21cat10:05:023
169599357,14cyclictest22740-21diskmemload12:31:533
169599357,14cyclictest22740-21diskmemload12:31:533
169599357,10cyclictest87350irq/126-enp1s0-12:00:213
169599356,15cyclictest0-21swapper/308:57:083
169599354,15cyclictest25568-21idleruntime-cro10:30:023
169599351,2cyclictest17493-21cut11:25:013
169599351,2cyclictest17493-21cut11:25:013
169599351,25cyclictest0-21swapper/312:10:173
169599351,20cyclictest909-21sssd_nss08:45:143
169599351,19cyclictest0-21swapper/309:50:133
169599347,15cyclictest13333-21awk07:35:013
169599341,29cyclictest7287-21hddtemp_smartct11:00:163
169599341,24cyclictest87650irq/129-enp1s0-10:50:163
169599341,24cyclictest20774-21/usr/sbin/munin09:05:123
169599341,22cyclictest13864-21expr07:35:133
169599339,12cyclictest31760-21/usr/sbin/munin09:30:143
169599335,14cyclictest87650irq/129-enp1s0-10:15:263
169599332,22cyclictest26370-21cat11:45:023
169599331,23cyclictest1591-21/usr/sbin/munin09:35:113
169599331,19cyclictest3600-21if_enp2s012:05:173
169599331,17cyclictest0-21swapper/309:20:143
167899339,3cyclictest8537-21df_inode09:50:130
167899333,4cyclictest22740-21diskmemload10:22:230
167899333,12cyclictest9547-21munin-run08:40:010
167899331,6cyclictest23068-21sort09:10:150
167899331,5cyclictest24845-40dnf10:25:240
167899331,5cyclictest13399-21interrupts10:00:170
1678993312,3cyclictest27710-21df_inode09:20:130
1678993310,3cyclictest17605-21/usr/sbin/munin11:25:230
169599323,25cyclictest5731-21diskstats08:30:143
169599322,26cyclictest17287-21smart_sda12:35:273
169599321,27cyclictest808-21snmpd07:52:103
167899329,4cyclictest16083-21/usr/sbin/munin11:20:160
167899329,4cyclictest16083-21/usr/sbin/munin11:20:160
167899321,6cyclictest31127-21df11:55:130
167899321,5cyclictest7234-21/usr/sbin/munin12:15:230
1678993212,3cyclictest26365-21cat11:45:010
1678993212,3cyclictest17951-21/usr/sbin/munin07:45:170
1678993212,3cyclictest16676-21expr08:55:150
1678993211,17cyclictest1353-21munin-run09:35:010
169599314,19cyclictest87350irq/126-enp1s0-11:35:143
169599311,5cyclictest0-21swapper/309:55:273
169599311,27cyclictest31489-21diskstats08:15:143
169599311,26cyclictest808-21snmpd11:47:563
169599310,4cyclictest0-21swapper/307:15:183
167899319,3cyclictest2220-21/usr/sbin/munin10:50:220
167899315,3cyclictest16026-21/usr/sbin/munin12:35:220
167899311,4cyclictest3904-21ntp_states12:05:200
1678993110,3cyclictest32371-21/usr/sbin/munin10:45:150
169599303,24cyclictest18258-21smart_sda10:10:263
169599303,24cyclictest18258-21smart_sda10:10:263
169599303,24cyclictest10432-21/usr/sbin/munin12:20:203
169599302,16cyclictest87350irq/126-enp1s0-11:55:223
169599301,16cyclictest0-21swapper/310:45:123
1695993011,16cyclictest4729-21cat09:40:193
168399301,7cyclictest31535-21awk09:30:021
167899309,3cyclictest19997-21wc09:00:250
167899309,3cyclictest19997-21wc09:00:250
167899308,4cyclictest14593-21fschecks_count08:50:150
167899303,5cyclictest10729-21df09:55:130
167899302,13cyclictest15181-21munin-run11:20:000
167899301,6cyclictest27419-21runrttasks10:30:530
167899301,4cyclictest27981-21/usr/sbin/munin10:35:250
167899301,4cyclictest21155-21latency_hist10:20:020
1678993011,3cyclictest8669-21cut11:05:010
1678993010,3cyclictest27083-21df08:05:120
1678993010,3cyclictest16144-21expr10:05:240
1678993010,3cyclictest14874-21cat07:35:250
169599298,18cyclictest7234-21/usr/sbin/munin12:15:133
169599291,4cyclictest10754-21/usr/sbin/munin08:40:203
169599291,2cyclictest87550irq/128-enp1s0-10:55:173
169599291,25cyclictest7925-21df_inode08:35:173
169599291,25cyclictest3843-21crond07:15:013
169599291,24cyclictest28935-21chrt11:50:143
169599291,24cyclictest24893-21/usr/sbin/munin08:00:153
1695992911,15cyclictest29014-21wc08:10:123
169599290,1cyclictest0-21swapper/309:00:163
169599290,1cyclictest0-21swapper/309:00:163
167899299,3cyclictest766-21ntpd08:45:210
167899293,3cyclictest22344-21/usr/sbin/munin07:55:170
167899291,4cyclictest808-21snmpd11:47:550
167899291,3cyclictest28938-21sshd08:10:080
1678992913,3cyclictest8032-21ntp_states07:20:210
167899291,12cyclictest15758-21/usr/sbin/munin07:40:190
1678992910,3cyclictest827-21runrttasks07:17:170
1678992910,3cyclictest5265-21wc10:55:180
169599288,17cyclictest27118-21fschecks_count08:05:153
169599282,23cyclictest3530-21/usr/sbin/munin08:25:143
169599282,22cyclictest16105-21diskstats07:40:153
169599281,3cyclictest6362-21/usr/sbin/munin09:45:163
169599281,3cyclictest22927-21/usr/sbin/munin07:55:163
169599280,4cyclictest17605-21/usr/sbin/munin11:25:253
1683992814,3cyclictest4611-21irqstats09:40:181
168399280,21cyclictest0-21swapper/110:55:081
167899284,4cyclictest773-21NetworkManager07:33:470
167899282,3cyclictest20149-21/usr/sbin/munin07:50:210
167899281,5cyclictest9773-21df_inode12:20:140
167899281,4cyclictest0-21swapper/012:30:340
167899281,4cyclictest0-21swapper/012:30:340
1678992810,3cyclictest2833-21seq08:23:390
1678992810,3cyclictest22112-21timerandwakeup09:05:240
1678992810,3cyclictest19805-21/usr/sbin/munin11:30:230
169599272,22cyclictest14355-21/usr/sbin/munin08:50:123
169599271,2cyclictest9167-21/usr/sbin/munin07:25:153
169599271,23cyclictest7497-21/usr/sbin/munin07:20:163
168399272,17cyclictest6450-21ntp_states08:30:201
1683992712,7cyclictest8882-21systemd-journal07:24:281
168399271,16cyclictest0-21swapper/107:50:201
168399270,14cyclictest0-21swapper/107:55:111
167899279,3cyclictest87550irq/128-enp1s0-12:13:300
167899275,3cyclictest0-21swapper/008:25:160
167899271,11cyclictest9765-21sshd08:40:080
1678992710,3cyclictest3480-21vmstat07:10:260
168399261,17cyclictest31138-21diskstats11:55:141
168399261,16cyclictest0-21swapper/112:25:231
168399261,15cyclictest9-21rcu_preempt10:50:131
167899269,4cyclictest18200-21processes10:10:250
167899269,4cyclictest18200-21processes10:10:250
167899263,11cyclictest10800-21perf11:10:000
167899261,6cyclictest0-21swapper/009:38:060
167899261,11cyclictest9167-21/usr/sbin/munin07:25:110
168399252,11cyclictest24561-21users09:10:291
168399251,9cyclictest3420-21kworker/u8:110:30:201
1683992514,3cyclictest0-21swapper/111:08:281
1683992513,3cyclictest0-21swapper/108:16:151
168399251,15cyclictest0-21swapper/110:46:471
168399251,15cyclictest0-21swapper/110:10:081
168399251,15cyclictest0-21swapper/110:10:081
168399251,15cyclictest0-21swapper/110:01:391
1683992511,3cyclictest4704-21grep08:25:241
1683992511,3cyclictest24845-40dnf10:25:221
168399251,13cyclictest0-21swapper/111:19:311
168399251,12cyclictest28045-21sed08:05:231
168399251,12cyclictest17434-21ntp_states08:55:231
168399251,11cyclictest31007-21sed10:40:191
168399251,11cyclictest15454-21gltestperf10:05:151
168399251,11cyclictest11694-21switchtime09:55:241
168399251,11cyclictest0-21swapper/109:22:061
168399251,10cyclictest2618-21munin-run12:05:011
167899251,9cyclictest0-21swapper/009:17:570
167899251,3cyclictest22740-21diskmemload09:40:140
147172250,5sleep230-21rcuc/211:15:252
168999242,5cyclictest3575-21munin-run09:40:022
168399249,4cyclictest7034-21ntp_kernel_pll_09:45:211
1683992421,2cyclictest0-21swapper/111:53:361
168399241,6cyclictest0-21swapper/110:24:591
168399241,3cyclictest10495-21ntp_states12:20:201
168399241,20cyclictest25980-21irqstats09:15:181
168399241,15cyclictest0-21swapper/111:49:501
168399241,15cyclictest0-21swapper/108:35:291
168399241,15cyclictest0-21swapper/107:37:281
168399241,13cyclictest10734-21kworker/u8:011:35:081
168399241,11cyclictest7905-21sed11:00:261
168399241,11cyclictest356-21ntp_states09:30:221
168399241,11cyclictest2866-21smart_sda09:35:251
1683992410,3cyclictest14638-21irqstats12:30:171
1683992410,3cyclictest14638-21irqstats12:30:171
168399240,16cyclictest0-21swapper/112:06:451
168399240,16cyclictest0-21swapper/107:34:281
167899241,7cyclictest0-21swapper/011:54:130
167899241,12cyclictest23969-21munin-run11:40:010
168399237,3cyclictest25434-21fgrep11:40:241
168399236,15cyclictest8979-21kworker/u8:212:35:171
168399231,2cyclictest0-21swapper/108:46:231
168399231,20cyclictest0-21swapper/111:34:081
168399231,14cyclictest0-21swapper/107:44:141
1683992311,3cyclictest0-21swapper/112:18:081
168399231,11cyclictest6099-21ntpq12:10:201
168399231,11cyclictest28326-21df_inode10:35:151
168399231,11cyclictest0-21swapper/110:15:231
168399231,10cyclictest29698-21iwlist08:10:151
168399230,14cyclictest0-21swapper/108:40:281
168999221,8cyclictest4149-21diskstats09:40:142
1689992210,2cyclictest22130-21awk07:55:012
168399228,4cyclictest9316-21grep07:25:111
168399228,2cyclictest1-21systemd11:30:011
168399221,19cyclictest0-21swapper/107:11:351
168399221,14cyclictest0-21swapper/109:50:441
168399221,14cyclictest0-21swapper/108:00:321
168399221,10cyclictest22183-21sort09:05:261
168399220,14cyclictest0-21swapper/108:52:231
168999212,4cyclictest30-21rcuc/211:42:572
168999211,8cyclictest0-21swapper/210:09:232
168999211,12cyclictest0-21swapper/207:39:112
168999211,11cyclictest20451-21smart_sda10:15:232
168999211,10cyclictest0-21swapper/208:13:092
168399217,3cyclictest0-21swapper/107:15:031
1683992111,2cyclictest0-21swapper/109:04:271
1683992111,2cyclictest0-21swapper/109:04:271
167899218,3cyclictest26303-21kworker/0:208:03:340
167899214,14cyclictest32157-21awk10:45:020
313022200,3sleep20-21swapper/209:27:152
168999206,3cyclictest87350irq/126-enp1s0-09:20:172
168999202,4cyclictest30699-21cat11:55:022
168999202,2cyclictest87550irq/128-enp1s0-10:15:022
168999202,2cyclictest87550irq/128-enp1s0-10:15:022
168999201,6cyclictest6807-21irqstats09:45:192
168999201,6cyclictest31438-21smart_sda10:40:242
168999201,6cyclictest27141-21munin-run09:20:022
168999201,6cyclictest25303-21awk08:00:172
168999201,6cyclictest15758-21users08:50:252
168999201,6cyclictest0-21swapper/212:03:322
168999201,6cyclictest0-21swapper/211:00:172
168999201,5cyclictest14960-21pmu-power12:30:222
168999201,5cyclictest14960-21pmu-power12:30:222
168999201,3cyclictest0-21swapper/210:48:112
168999201,10cyclictest808-21snmpd11:22:512
168999201,10cyclictest808-21snmpd11:22:512
168399206,11cyclictest1768-21cat08:20:161
168399201,3cyclictest16224-21irqstats11:20:191
168399201,3cyclictest16224-21irqstats11:20:191
1683992010,2cyclictest0-21swapper/107:45:161
168999194,4cyclictest10835-21timerwakeupswit12:20:242
168999192,9cyclictest20774-21/usr/sbin/munin09:05:132
168999192,10cyclictest0-21swapper/209:56:392
168999191,9cyclictest8822-21/usr/sbin/munin11:05:242
168999191,7cyclictest0-21swapper/211:34:302
168999191,6cyclictest470-21ps09:30:232
168999191,6cyclictest3409-21ls08:25:122
168999191,6cyclictest2230-21diskstats07:10:122
168999191,5cyclictest0-21swapper/209:10:282
168999191,3cyclictest0-21swapper/209:52:272
1689991912,4cyclictest4145-21switchtime12:05:242
1689991912,4cyclictest31476-21sh08:15:122
168999191,12cyclictest0-21swapper/212:11:402
168999191,12cyclictest0-21swapper/210:55:222
168999191,10cyclictest26742-21diskstats11:45:132
168999191,10cyclictest17951-21/usr/sbin/munin07:45:232
168999191,10cyclictest17613-21users12:35:282
168999191,10cyclictest0-21swapper/212:25:072
167899197,2cyclictest5417-21kworker/0:309:45:300
129202190,3sleep20-21swapper/211:14:222
274842181,3sleep20-21swapper/210:31:432
168999185,3cyclictest87350irq/126-enp1s0-11:25:242
168999184,7cyclictest87650irq/129-enp1s0-07:15:182
168999184,4cyclictest12518-1kworker/2:0H07:31:142
168999181,6cyclictest24810-21gzip10:25:212
168999181,6cyclictest22992-21strings11:35:202
168999181,6cyclictest0-21swapper/210:21:062
168999181,5cyclictest6606-21gunzip08:30:212
168999181,5cyclictest3988-21munin-node07:20:282
168999181,5cyclictest29241-21smart_sda10:35:242
168999181,5cyclictest28008-21smart_sda08:05:242
168999181,5cyclictest23604-21smart_sda07:55:242
168999181,5cyclictest13994-21cat08:50:002
168999181,5cyclictest0-21swapper/210:52:072
168999181,4cyclictest0-21swapper/207:25:582
1689991811,4cyclictest19412-21/usr/sbin/munin09:00:182
1689991811,4cyclictest19412-21/usr/sbin/munin09:00:182
168999181,14cyclictest1079-21tr08:20:112
1678991812,3cyclictest31140-21/usr/sbin/munin08:15:250
168999170,9cyclictest31495-21idleruntime11:55:182
168999170,6cyclictest8163-21gltestperf08:35:182
168999170,5cyclictest0-21swapper/208:55:272
1678991710,3cyclictest4872-21kworker/0:008:30:230
168999161,6cyclictest0-21swapper/207:40:242
1689991610,3cyclictest10661-21sort08:40:182
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional