You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-20 - 12:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Sat Dec 20, 2025 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
174112239163,25sleep20-21swapper/219:08:182
174112239163,25sleep20-21swapper/219:08:182
172402238163,23sleep10-21swapper/119:06:031
172402238163,23sleep10-21swapper/119:06:031
174092201163,27sleep00-21swapper/019:08:160
174092201163,27sleep00-21swapper/019:08:160
174122200164,24sleep30-21swapper/319:08:193
174122200164,24sleep30-21swapper/319:08:183
96721820,5sleep01762999cyclictest20:55:210
17639999996,2cyclictest0-21swapper/200:04:052
125982990,7sleep230-21rcuc/221:20:562
297362570,1sleep20-21swapper/200:30:012
17639995753,3cyclictest0-21swapper/222:17:482
300352560,6sleep130037-21unixbench_multi22:00:231
250312480,8sleep341-21ksoftirqd/320:35:273
125502450,1sleep10-21swapper/120:10:121
17639993936,2cyclictest0-21swapper/219:35:292
1764499301,19cyclictest28011-21ntp_kernel_pll_23:10:213
1763499301,9cyclictest0-21swapper/123:30:221
1763499301,14cyclictest0-21swapper/100:22:581
317802290,4sleep230-21rcuc/223:20:152
1764499291,25cyclictest31695-21df23:20:143
1764499281,21cyclictest0-21swapper/323:25:473
1764499271,16cyclictest11587-21acpi22:35:123
1764499271,13cyclictest0-21swapper/322:59:353
1763499271,12cyclictest0-21swapper/122:58:261
1763499271,11cyclictest0-21swapper/121:42:251
1763499270,25cyclictest0-21swapper/100:07:481
297502260,4sleep339-21rcuc/319:35:143
1764499261,16cyclictest801-21snmpd23:01:463
1763999261,6cyclictest0-21swapper/200:39:052
1763999261,4cyclictest0-21swapper/223:59:122
1763499261,13cyclictest0-21swapper/122:50:121
1762999263,16cyclictest87150irq/128-enp1s0-20:40:240
1762999263,15cyclictest11-21rcuc/021:00:170
34622250,1sleep00-21swapper/023:30:120
17644992512,3cyclictest8688-21tail22:25:253
17644992510,9cyclictest0-21swapper/323:57:273
1763499255,18cyclictest0-21swapper/120:25:171
17634992520,2cyclictest87150irq/128-enp1s0-21:10:001
1763499251,6cyclictest86950irq/126-enp1s0-21:55:021
17634992515,3cyclictest801-21snmpd19:23:281
1763499251,17cyclictest0-21swapper/123:15:271
1763499251,11cyclictest0-21swapper/123:49:401
1763499251,10cyclictest22672-21awk20:30:261
1763499250,17cyclictest13059-21latency23:50:171
1762999255,11cyclictest17876-21switchtime00:00:260
1762999253,6cyclictest0-21swapper/021:46:580
1762999251,5cyclictest0-21swapper/023:53:140
17644992411,3cyclictest27509-21ps21:55:223
17644992411,2cyclictest19156-21needreboot22:50:173
1764499241,11cyclictest7181-21users23:35:263
1763499249,3cyclictest25435-21grep23:05:161
1763499248,4cyclictest0-21swapper/121:18:221
1763499242,9cyclictest10430-21cpuspeed_turbos20:05:141
1763499242,10cyclictest751-21rngd19:50:021
1763499241,16cyclictest0-21swapper/120:00:181
1763499241,10cyclictest0-21swapper/122:26:141
1762999241,16cyclictest0-21swapper/023:17:240
1762999241,11cyclictest32105-21awk20:55:010
1762999241,11cyclictest11548-21sed23:45:280
1762999241,11cyclictest0-21swapper/022:39:530
17629992410,3cyclictest986-1kworker/u9:122:06:090
231992230,2sleep230-21rcuc/223:00:172
1764499239,2cyclictest0-21swapper/320:05:263
1764499237,6cyclictest0-21swapper/322:00:113
1764499237,2cyclictest19431-21gltestperf00:05:173
1764499231,20cyclictest0-21swapper/322:45:133
1764499231,20cyclictest0-21swapper/322:20:203
1764499231,20cyclictest0-21swapper/322:10:163
1764499231,18cyclictest0-21swapper/300:00:133
17644992310,2cyclictest0-21swapper/323:30:163
1763999237,5cyclictest18987-21kernelversion22:50:172
1763999231,6cyclictest0-21swapper/200:17:202
1763499237,10cyclictest0-21swapper/100:31:511
1763499231,6cyclictest0-21swapper/123:27:221
1763499231,6cyclictest0-21swapper/122:35:151
17634992314,3cyclictest0-21swapper/119:36:521
1763499231,2cyclictest32723-21perf22:10:021
1763499231,16cyclictest0-21swapper/122:33:071
1763499230,8cyclictest0-21swapper/100:39:261
1762999238,3cyclictest11901-21grep20:05:260
1762999237,10cyclictest0-21swapper/000:20:170
1762999236,7cyclictest5941-21diskstats23:35:140
1762999235,9cyclictest0-21swapper/000:30:160
1762999233,10cyclictest4866-21cat22:20:010
1762999232,9cyclictest11-21rcuc/000:10:200
17629992317,3cyclictest826-21lldpd20:01:410
1762999231,6cyclictest0-21swapper/021:38:040
1762999231,6cyclictest0-21swapper/021:27:110
1762999231,5cyclictest0-21swapper/022:41:080
1762999231,5cyclictest0-21swapper/022:41:080
1762999231,4cyclictest0-21swapper/023:20:130
1762999231,3cyclictest0-21swapper/022:26:420
1762999231,16cyclictest0-21swapper/023:43:170
17629992311,2cyclictest11-21rcuc/023:25:200
1762999231,11cyclictest24906-21awk19:25:010
1762999231,10cyclictest26750-21awk23:10:010
1762999230,14cyclictest17658-21kworker/u8:122:51:390
1762999230,10cyclictest20139-21timerandwakeup00:05:260
1764499229,3cyclictest2901-21if_enp2s021:00:183
1764499227,3cyclictest14738-21/usr/sbin/munin20:15:143
1764499227,3cyclictest14738-21/usr/sbin/munin20:15:143
1764499221,9cyclictest18540-21sshd20:20:283
1764499221,2cyclictest0-21swapper/321:20:003
1764499221,2cyclictest0-21swapper/300:17:013
1764499221,17cyclictest0-21swapper/300:13:433
17644992211,3cyclictest31504-21if_enp2s022:05:163
17644992211,3cyclictest0-21swapper/300:31:263
1764499221,10cyclictest32013-21cpu00:35:123
17644992210,2cyclictest30391-21gzip23:15:263
1764499220,18cyclictest0-21swapper/322:44:233
1764499220,18cyclictest0-21swapper/322:44:233
1764499220,17cyclictest22237-21kworker/u8:223:08:503
1763999221,5cyclictest0-21swapper/221:17:192
1763499221,9cyclictest0-21swapper/123:56:171
1763499221,9cyclictest0-21swapper/119:43:181
1763499221,6cyclictest0-21swapper/121:12:121
17634992214,2cyclictest23-21ksoftirqd/122:17:021
1763499221,16cyclictest0-21swapper/120:51:301
1763499221,16cyclictest0-21swapper/119:13:551
1763499221,15cyclictest0-21swapper/100:04:291
1763499221,13cyclictest0-21swapper/123:23:481
1763499221,12cyclictest0-21swapper/123:01:401
1763499221,10cyclictest0-21swapper/122:14:171
1763499221,10cyclictest0-21swapper/121:32:071
1763499221,10cyclictest0-21swapper/121:32:071
1763499220,15cyclictest3333-21processes21:00:231
1763499220,14cyclictest21608-21kworker/u8:122:22:411
1762999227,8cyclictest801-21snmpd21:56:340
1762999226,9cyclictest0-21swapper/021:50:210
1762999221,9cyclictest0-21swapper/021:12:280
1762999221,6cyclictest0-21swapper/022:47:220
1762999221,5cyclictest986-1kworker/u9:122:02:120
1762999221,5cyclictest844-21expr22:10:140
1762999221,4cyclictest0-21swapper/021:20:000
1762999221,4cyclictest0-21swapper/000:17:010
1762999221,12cyclictest15447-21ntp_states23:55:200
1762999221,10cyclictest0-21swapper/022:30:280
254752211,3sleep225698-21fw_packets19:25:152
1764499219,3cyclictest0-21swapper/321:32:273
1764499219,3cyclictest0-21swapper/321:32:273
1764499218,3cyclictest0-21swapper/323:50:173
1764499217,3cyclictest10954-21/usr/sbin/munin21:20:193
1764499213,15cyclictest5938-21/usr/sbin/munin19:55:183
17644992112,2cyclictest22595-21smart_sda20:30:253
1764499211,14cyclictest28859-21smartctl00:25:243
1764499211,10cyclictest0-21swapper/321:15:023
1764499210,19cyclictest0-21swapper/321:45:233
1764499210,19cyclictest0-21swapper/321:25:153
1763999219,3cyclictest11755-21cstates22:35:142
1763999211,5cyclictest0-21swapper/222:25:452
1763999211,5cyclictest0-21swapper/221:31:292
1763999211,5cyclictest0-21swapper/221:31:292
1763999211,4cyclictest0-21swapper/222:13:422
1763999211,4cyclictest0-21swapper/221:26:212
1763999211,3cyclictest0-21swapper/200:11:102
1763499218,2cyclictest9738-21kworker/1:121:36:211
1763499216,3cyclictest506-21grep20:55:171
1763499216,3cyclictest0-21swapper/120:35:191
1763499213,3cyclictest0-21swapper/122:49:481
1763499211,8cyclictest0-21swapper/123:40:131
1763499211,6cyclictest0-21swapper/123:37:201
1763499211,5cyclictest0-21swapper/121:55:531
17634992114,3cyclictest0-21swapper/120:46:211
1763499211,15cyclictest0-21swapper/120:42:481
1763499211,15cyclictest0-21swapper/120:15:491
1763499211,15cyclictest0-21swapper/120:15:491
1763499211,15cyclictest0-21swapper/119:19:151
1763499211,14cyclictest0-21swapper/100:25:041
1763499211,11cyclictest0-21swapper/121:21:121
1763499211,10cyclictest0-21swapper/100:10:081
17634992110,1cyclictest0-21swapper/123:10:261
1763499210,15cyclictest0-21swapper/119:33:381
1762999216,3cyclictest21278-21memory22:55:200
1762999215,7cyclictest0-21swapper/000:36:380
1762999214,4cyclictest23300-21dump-pmu-power20:35:000
1762999212,11cyclictest27311-21/usr/sbin/munin19:30:130
1762999211,9cyclictest0-21swapper/019:35:270
1762999211,5cyclictest0-21swapper/000:28:070
1762999211,11cyclictest4083-21df_inode19:50:130
1762999211,11cyclictest19484-21diskstats20:25:130
1762999211,10cyclictest21331-21users21:40:250
1762999211,10cyclictest18447-21sed20:20:250
145452211,2sleep214631-21sed22:40:192
145452211,2sleep214631-21sed22:40:192
1764499209,3cyclictest0-21swapper/319:33:073
1764499209,2cyclictest0-21swapper/322:30:253
1764499207,3cyclictest10620-21gltestperf23:45:173
1764499207,3cyclictest0-21swapper/320:01:413
1764499201,6cyclictest26000-21gltestperf00:20:183
1764499201,16cyclictest8140-21diskstats23:40:163
17644992010,2cyclictest0-21swapper/320:25:123
17644992010,1cyclictest0-21swapper/322:15:213
1764499200,18cyclictest0-21swapper/320:55:203
1763999205,9cyclictest18543-40dnf20:20:312
1763999202,2cyclictest0-21swapper/221:09:232
1763999201,6cyclictest21717-21latency_hist21:45:012
1763999201,6cyclictest0-21swapper/222:47:392
1763999201,5cyclictest0-21swapper/222:20:142
1763999201,5cyclictest0-21swapper/221:14:072
17639992014,4cyclictest9636-21grep20:00:262
1763999201,3cyclictest0-21swapper/223:46:142
1763999201,3cyclictest0-21swapper/223:18:392
1763499206,4cyclictest0-21swapper/100:15:041
1763499205,3cyclictest0-21swapper/119:51:421
1763499204,11cyclictest87150irq/128-enp1s0-21:48:291
17634992016,3cyclictest0-21swapper/119:59:001
17634992015,3cyclictest17607-21sh20:20:171
17634992014,1cyclictest0-21swapper/121:25:161
1763499201,10cyclictest27082-21latency_hist19:30:021
1763499200,5cyclictest0-21swapper/122:41:511
1763499200,5cyclictest0-21swapper/122:41:511
1762999209,3cyclictest17829-21/usr/sbin/munin19:10:180
1762999206,8cyclictest0-21swapper/019:27:180
1762999205,3cyclictest5676-21sort22:20:150
1762999203,2cyclictest0-21swapper/021:34:030
1762999203,2cyclictest0-21swapper/021:34:030
1762999201,7cyclictest0-21swapper/023:14:000
1762999201,6cyclictest0-21swapper/023:00:490
1762999201,11cyclictest12538-21/usr/sbin/munin20:10:240
1762999201,10cyclictest21327-21head19:15:170
1762999201,10cyclictest1462-21cat19:45:020
17629992010,3cyclictest0-21swapper/020:45:200
1762999200,12cyclictest10954-21/usr/sbin/munin21:20:140
248662190,4sleep230-21rcuc/220:35:262
1764499197,3cyclictest0-21swapper/320:47:283
1764499196,3cyclictest19746-21/usr/sbin/munin21:40:143
1764499196,3cyclictest0-21swapper/319:19:073
1764499195,7cyclictest41-21ksoftirqd/319:50:253
1764499194,8cyclictest0-21swapper/321:39:163
1764499192,14cyclictest32286-21gltestperf19:40:153
1764499191,6cyclictest23258-21df_inode19:20:123
1764499191,3cyclictest14095-21sed20:10:263
1764499191,11cyclictest0-21swapper/320:43:173
1763999193,6cyclictest14738-21/usr/sbin/munin20:15:192
1763999193,6cyclictest14738-21/usr/sbin/munin20:15:192
1763999193,3cyclictest86950irq/126-enp1s0-23:35:142
1763999191,6cyclictest0-21swapper/223:43:052
1763999191,5cyclictest22918-21/usr/sbin/munin19:20:192
1763999191,5cyclictest0-21swapper/223:31:352
1763999191,4cyclictest0-21swapper/223:54:112
1763999191,3cyclictest0-21swapper/223:28:542
1763999191,3cyclictest0-21swapper/223:09:092
1763999191,3cyclictest0-21swapper/200:32:352
17639991912,4cyclictest19924-21latency_hist19:15:022
1763999191,15cyclictest27475-21tr19:30:122
1763999191,10cyclictest0-21swapper/223:10:042
1762999198,2cyclictest6339-21awk21:10:000
1762999192,14cyclictest904-21sssd_nss20:35:210
1762999191,9cyclictest1919-21fschecks_count19:45:160
1762999191,9cyclictest16882-21latency_hist20:20:010
1762999191,9cyclictest16882-21latency_hist20:20:010
1762999191,7cyclictest0-21swapper/019:55:210
1764499186,3cyclictest0-21swapper/321:50:113
1764499181,9cyclictest0-21swapper/319:46:203
1763999184,5cyclictest28982-21fschecks_count22:00:132
1763999183,3cyclictest87250irq/129-enp1s0-20:05:152
1763999182,7cyclictest25400-21/usr/sbin/munin00:20:182
1763999182,6cyclictest12538-21/usr/sbin/munin20:10:122
1763999181,6cyclictest0-21swapper/221:55:522
1763999181,6cyclictest0-21swapper/221:50:122
1763999181,5cyclictest0-21swapper/221:48:102
1763999181,5cyclictest0-21swapper/221:35:482
1763999181,4cyclictest27922-21/usr/sbin/munin20:45:122
1763999181,3cyclictest0-21swapper/222:33:152
17639991812,3cyclictest30840-21cpuspeed_turbos22:05:112
1763999181,11cyclictest0-21swapper/200:06:192
1763999181,10cyclictest0-21swapper/222:59:092
1763999181,10cyclictest0-21swapper/220:50:182
1764499172,7cyclictest0-21swapper/319:25:073
1764499172,12cyclictest4503-21cpuspeed_turbos21:05:133
1764499171,6cyclictest86950irq/126-enp1s0-19:10:163
17644991710,4cyclictest30961-21latency20:50:173
1763999175,3cyclictest86950irq/126-enp1s0-20:30:262
1763999175,3cyclictest22698-21awk19:20:002
1763999171,9cyclictest0-21swapper/219:43:322
1763999171,6cyclictest4343-21head19:50:152
1763999171,5cyclictest0-21swapper/221:00:192
17639991712,3cyclictest6283-21sh19:55:152
1763999170,4cyclictest0-21swapper/220:41:182
1763999162,5cyclictest32318-21/usr/sbin/munin20:55:282
1763999161,6cyclictest0-21swapper/220:25:172
17639991610,3cyclictest1915-21grep19:45:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional