You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-19 - 06:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Wed Aug 19, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
263742238163,24sleep20-21swapper/219:07:232
262572222192,19sleep30-21swapper/319:05:513
265762202165,25sleep00-21swapper/019:10:000
243852202167,23sleep10-21swapper/119:05:021
63021440,2sleep10-21swapper/119:20:221
2122721340,2sleep20-21swapper/221:20:272
2122721340,2sleep20-21swapper/221:20:262
266769910399,3cyclictest0-21swapper/219:50:182
161272710,5sleep316131-21latency_hist22:25:013
62562500,2sleep10-21swapper/120:46:111
298922500,2sleep22667699cyclictest21:40:212
2666499432,25cyclictest10244-21/usr/sbin/munin23:25:190
310862410,1sleep10-21swapper/100:10:221
2666499411,30cyclictest8044-21/usr/sbin/munin23:20:130
2666499397,29cyclictest31726-21munin-run19:20:000
2666499390,28cyclictest0-21swapper/020:00:140
26682993825,8cyclictest0-21swapper/323:18:563
2666499384,31cyclictest7218-21idleruntime-cro22:05:010
2666499372,24cyclictest909-21sssd_nss21:50:390
2666499361,7cyclictest14201-21cstates22:20:150
2668299352,21cyclictest10976-40dnf21:00:043
2668299351,23cyclictest0-21swapper/319:16:313
2666499351,4cyclictest27856-21/usr/sbin/munin00:05:320
2668299341,22cyclictest0-21swapper/320:45:523
2668299341,21cyclictest0-21swapper/319:32:043
2666499348,13cyclictest30143-21munin-run20:30:000
2666499345,2cyclictest87350irq/126-enp1s0-23:30:200
2666499344,24cyclictest27298-40dnf00:01:030
2666499341,29cyclictest21344-21grep23:50:130
2666499331,2cyclictest0-21swapper/000:20:250
2666499331,29cyclictest13559-21tr21:05:120
26682993217,3cyclictest0-21swapper/320:35:123
2668299321,21cyclictest0-21swapper/319:48:083
2668299321,14cyclictest0-21swapper/300:26:513
2668299320,16cyclictest0-21swapper/300:23:513
2668299320,15cyclictest0-21swapper/323:13:563
2666499329,20cyclictest26863-21/usr/sbin/munin19:10:150
2666499322,27cyclictest17797-21/usr/sbin/munin21:15:120
2666499321,5cyclictest10976-40dnf21:00:000
2666499321,17cyclictest1437-21/usr/sbin/munin23:05:190
2666499320,24cyclictest0-21swapper/000:15:190
2666499320,21cyclictest0-21swapper/022:05:180
2668299311,20cyclictest0-21swapper/320:12:153
2668299311,20cyclictest0-21swapper/320:12:153
2668299311,18cyclictest0-21swapper/300:05:473
2668299311,15cyclictest0-21swapper/321:17:363
2666499319,19cyclictest8060-21/usr/sbin/munin20:50:240
2666499319,19cyclictest7824-21ls23:20:000
2666499318,19cyclictest16880-21tail22:25:170
2666499312,6cyclictest16860-21smart_sda21:10:240
26664993116,12cyclictest29567-21grep22:55:120
2666499311,4cyclictest17013-21cat23:40:120
2666499311,4cyclictest17013-21cat23:40:120
2666499311,17cyclictest909-21sssd_nss23:10:210
2666499311,14cyclictest1556-21munin-run19:25:010
26664993111,17cyclictest23667-21expr23:55:130
26664993110,17cyclictest4597-21/usr/sbin/munin20:45:190
2666499310,19cyclictest0-21swapper/022:30:180
2666499310,19cyclictest0-21swapper/022:30:180
2668299301,20cyclictest0-21swapper/320:23:153
2668299301,13cyclictest0-21swapper/323:52:083
2668299301,13cyclictest0-21swapper/322:57:323
2668299301,10cyclictest0-21swapper/321:23:323
2668299301,10cyclictest0-21swapper/321:23:323
2668299300,20cyclictest0-21swapper/322:25:233
2668299300,15cyclictest0-21swapper/300:16:273
2668299300,13cyclictest0-21swapper/322:46:163
2666499308,20cyclictest14031-21sensors_temp19:50:250
2666499306,22cyclictest16606-21kworker/0:220:05:310
2666499305,21cyclictest27585-21head21:35:210
2666499301,6cyclictest16612-21munin-run23:40:010
2666499301,6cyclictest14969-21/usr/sbin/munin19:55:210
2666499301,3cyclictest4986-21/usr/sbin/munin00:25:180
2666499301,3cyclictest12006-21/usr/sbin/munin22:15:130
2666499301,16cyclictest19171-21grep23:45:140
26664993011,16cyclictest8705-21cpu00:35:120
26664993011,16cyclictest32226-21expr21:45:250
26682992917,3cyclictest0-21swapper/320:26:233
26682992917,3cyclictest0-21swapper/319:41:083
2668299291,20cyclictest0-21swapper/321:05:163
2668299291,14cyclictest0-21swapper/321:26:323
2668299291,14cyclictest0-21swapper/300:30:033
2668299291,13cyclictest0-21swapper/322:02:393
2668299291,13cyclictest0-21swapper/321:47:073
2668299291,13cyclictest0-21swapper/321:13:563
2668299291,12cyclictest0-21swapper/323:01:123
2668299291,11cyclictest0-21swapper/322:36:113
2668299291,11cyclictest0-21swapper/320:43:453
2668299290,2cyclictest0-21swapper/323:35:023
2668299290,13cyclictest0-21swapper/323:40:523
2668299290,13cyclictest0-21swapper/323:40:523
2668299290,13cyclictest0-21swapper/323:25:243
2668299290,13cyclictest0-21swapper/322:33:193
2668299290,13cyclictest0-21swapper/322:33:193
2668299290,12cyclictest0-21swapper/323:22:363
2668299290,12cyclictest0-21swapper/300:03:003
2666499299,17cyclictest827-21runrttasks22:47:390
2666499299,17cyclictest23528-21threads21:25:250
2666499298,18cyclictest827-21runrttasks20:23:330
2666499298,18cyclictest32564-21/usr/sbin/munin20:35:220
2666499298,17cyclictest22953-21cpuspeed_turbos22:40:120
2666499297,19cyclictest4058-21cpuspeed_turbos19:30:130
2666499292,4cyclictest30704-21df20:30:130
2666499292,4cyclictest3025-21/usr/sbin/munin21:55:300
2666499292,4cyclictest21919-21diskstats20:10:160
2666499292,4cyclictest21919-21diskstats20:10:160
2666499291,4cyclictest22075-21/usr/sbin/munin22:35:260
2666499291,4cyclictest10976-40dnf21:00:010
2666499291,3cyclictest87350irq/126-enp1s0-22:50:240
26664992910,16cyclictest32031-21munin-run00:15:020
26664992910,16cyclictest10579-21/usr/sbin/munin19:45:220
26682992818,3cyclictest0-21swapper/319:13:023
2668299281,14cyclictest0-21swapper/319:55:243
2668299281,13cyclictest0-21swapper/300:37:233
2668299281,12cyclictest0-21swapper/323:56:003
2668299280,19cyclictest0-21swapper/320:02:443
2668299280,19cyclictest0-21swapper/319:39:043
2668299280,11cyclictest0-21swapper/322:07:353
2666499288,18cyclictest23897-21grep20:15:140
2666499287,18cyclictest24953-21cut21:30:160
2666499286,19cyclictest3447-21ntpq20:40:210
2666499286,18cyclictest808-21snmpd19:44:450
2666499284,3cyclictest8203-21sshd00:31:550
2666499282,5cyclictest30050-21smart_sda21:40:240
2666499282,3cyclictest909-21sssd_nss22:10:250
2666499281,5cyclictest0-21swapper/023:00:170
2666499281,4cyclictest20329-21df21:20:170
2666499281,4cyclictest20329-21df21:20:170
2666499281,3cyclictest7643-21/usr/sbin/munin19:35:260
2668299271,14cyclictest0-21swapper/322:15:393
2668299271,12cyclictest0-21swapper/321:50:273
2668299271,10cyclictest0-21swapper/321:35:443
2668299270,19cyclictest0-21swapper/319:27:083
2668299270,14cyclictest0-21swapper/321:55:033
2668299270,13cyclictest0-21swapper/319:54:483
2668299270,13cyclictest0-21swapper/300:10:353
2668299270,12cyclictest0-21swapper/323:36:403
2668299270,12cyclictest0-21swapper/323:07:483
2668299270,10cyclictest0-21swapper/322:10:513
2666499271,5cyclictest2633-21/usr/sbin/munin19:25:160
69952260,4sleep230-21rcuc/220:50:142
2668299261,19cyclictest0-21swapper/319:21:083
2668299261,12cyclictest0-21swapper/323:45:323
2668299260,12cyclictest0-21swapper/322:51:403
2668299260,12cyclictest0-21swapper/320:30:243
2668299260,11cyclictest0-21swapper/321:43:113
2668299260,11cyclictest0-21swapper/321:33:283
2668299260,11cyclictest0-21swapper/320:16:073
253652260,1sleep20-21swapper/222:45:152
30562250,3sleep123-21ksoftirqd/120:40:171
2668299251,23cyclictest0-21swapper/320:55:203
2668299251,11cyclictest0-21swapper/320:08:593
2668299250,10cyclictest0-21swapper/322:42:043
2667199251,17cyclictest0-21swapper/122:25:281
2668299249,3cyclictest0-21swapper/320:50:123
2667699243,7cyclictest27298-40dnf00:01:022
26676992415,8cyclictest0-21swapper/220:58:282
2667199249,5cyclictest29333-21sed00:05:301
326832230,4sleep232-21ksoftirqd/200:15:162
2667199231,6cyclictest909-21sssd_nss21:50:201
2667199231,4cyclictest8255-21kernelversion22:05:191
26671992314,3cyclictest0-21swapper/122:12:241
26671992311,5cyclictest21172-21kworker/u8:019:14:121
2667199231,15cyclictest0-21swapper/122:44:061
174632230,4sleep230-21rcuc/222:25:252
2667199228,3cyclictest5639-21grep22:00:161
2667199223,11cyclictest0-21swapper/122:20:211
2667199221,18cyclictest6143-21cstates23:15:121
2667199221,18cyclictest17978-21cstates21:15:121
2667199221,18cyclictest13073-21seq21:03:511
2667199221,16cyclictest0-21swapper/122:47:271
2667199221,15cyclictest0-21swapper/120:19:011
2667199221,11cyclictest9658-21needreboot00:35:201
2667199221,10cyclictest0-21swapper/123:25:391
2667199220,14cyclictest87650irq/129-enp1s0-23:12:381
2667699214,7cyclictest16797-21processes21:10:242
2667699213,16cyclictest579-21systemd-journal19:30:022
2667699211,4cyclictest0-21swapper/222:50:562
26671992115,3cyclictest28761-21tail20:25:161
2667199211,4cyclictest0-21swapper/122:57:251
2667199211,18cyclictest0-21swapper/100:30:051
2667199211,17cyclictest9082-21ntp_kernel_pll_23:20:181
2667199211,17cyclictest0-21swapper/121:25:211
2667199211,15cyclictest0-21swapper/120:02:051
2667199211,12cyclictest2527-21ntp_states23:05:201
2667199211,12cyclictest0-21swapper/123:48:541
2667199211,11cyclictest20395-21munin-run22:35:011
2667199211,11cyclictest20395-21munin-run22:35:011
2667199211,11cyclictest0-21swapper/122:53:501
26671992110,3cyclictest23-21ksoftirqd/119:15:481
2667199210,5cyclictest0-21swapper/121:20:261
2667199210,5cyclictest0-21swapper/121:20:261
2667699205,2cyclictest87550irq/128-enp1s0-22:23:042
2667199201,9cyclictest0-21swapper/122:38:161
26671992016,2cyclictest17049-21cat23:40:111
26671992016,2cyclictest17049-21cat23:40:111
26671992012,3cyclictest0-21swapper/119:31:051
2667199201,18cyclictest0-21swapper/123:02:541
2667199201,17cyclictest0-21swapper/120:30:241
2667199201,17cyclictest0-21swapper/100:00:101
2667199201,14cyclictest0-21swapper/123:39:571
2667199201,14cyclictest0-21swapper/120:52:331
2667199201,14cyclictest0-21swapper/120:12:461
2667199201,14cyclictest0-21swapper/120:12:461
2667199201,14cyclictest0-21swapper/119:38:151
2667199201,11cyclictest21324-21cat20:09:561
2667199201,11cyclictest1866-21munin-plugin-st00:20:021
2667199200,14cyclictest0-21swapper/119:45:191
2667699198,3cyclictest8164-21awk19:40:012
2667699196,4cyclictest30133-21crond20:30:012
2667699191,9cyclictest6557-21runrttasks20:50:022
2667699191,8cyclictest32025-21latency_hist00:15:012
2667699191,6cyclictest0-21swapper/200:39:282
2667699191,12cyclictest0-21swapper/221:00:202
2667199199,2cyclictest8894-21fschecks_time19:40:161
2667199198,10cyclictest27029-21kworker/u8:020:35:101
26671991915,3cyclictest0-21swapper/123:51:281
26671991915,3cyclictest0-21swapper/100:20:201
2667199191,2cyclictest15368-21diskmemload23:59:501
2667199191,2cyclictest13942-21ps19:50:241
26671991912,2cyclictest0-21swapper/122:15:281
2667199191,17cyclictest0-21swapper/121:40:101
2667199191,17cyclictest0-21swapper/120:59:031
2667199191,14cyclictest0-21swapper/119:28:161
2667199191,11cyclictest0-21swapper/121:56:141
2667199191,10cyclictest5334-21/usr/sbin/munin00:25:221
2667199191,10cyclictest25496-21pmu-power21:30:231
2667199191,10cyclictest0-21swapper/121:37:361
2667199191,10cyclictest0-21swapper/121:12:211
2667199190,5cyclictest32606-21kworker/u8:121:46:261
2667199190,14cyclictest0-21swapper/120:23:091
2667199190,14cyclictest0-21swapper/119:59:331
2667699185,6cyclictest30-21rcuc/222:37:442
2667699185,11cyclictest25390-21idleruntime-cro00:00:012
2667699184,6cyclictest12015-21sshd23:27:392
2667699182,5cyclictest28565-21munin-run21:40:012
2667699181,9cyclictest0-21swapper/219:45:152
2667699181,7cyclictest8382-21/usr/sbin/munin19:40:132
2667699181,7cyclictest0-21swapper/200:08:362
2667699181,6cyclictest0-21swapper/222:41:532
2667699181,6cyclictest0-21swapper/220:00:122
2667699181,3cyclictest13352-21unixbench_multi22:15:262
2667699181,3cyclictest0-21swapper/222:10:352
26676991813,3cyclictest7429-21/usr/sbin/munin22:05:222
26676991813,3cyclictest13401-21/usr/sbin/munin21:05:272
2667699181,15cyclictest0-21swapper/223:31:372
2667699181,10cyclictest0-21swapper/221:50:162
26671991815,2cyclictest0-21swapper/123:31:541
2667199181,15cyclictest0-21swapper/121:05:071
2667699179,5cyclictest21702-21grep20:10:142
2667699179,5cyclictest21702-21grep20:10:142
2667699174,3cyclictest26713-21kworker/u8:223:07:442
2667699173,4cyclictest26364-21munin-run21:35:002
2667699172,7cyclictest29756-21/usr/sbin/munin19:15:232
2667699172,2cyclictest87550irq/128-enp1s0-23:44:442
2667699172,2cyclictest87550irq/128-enp1s0-23:44:432
2667699171,6cyclictest4505-21cat20:45:022
2667699171,6cyclictest3191-21open_inodes00:20:222
2667699171,6cyclictest0-21swapper/200:25:152
2667699171,3cyclictest0-21swapper/222:03:412
2667699171,3cyclictest0-21swapper/220:07:482
2667699171,2cyclictest0-21swapper/221:56:312
26676991712,3cyclictest1794-21users20:35:282
2667699171,15cyclictest87550irq/128-enp1s0-21:16:532
2667699171,14cyclictest20267-21find23:45:242
26676991711,3cyclictest32138-21grep23:00:172
2667699171,10cyclictest0-21swapper/222:30:242
2667699171,10cyclictest0-21swapper/222:30:242
2667699171,10cyclictest0-21swapper/219:55:172
26676991710,4cyclictest0-21swapper/223:50:232
2667699163,4cyclictest6274-21fschecks_count23:15:152
2667699162,5cyclictest0-21swapper/219:10:152
2667699161,7cyclictest3654-21/usr/sbin/munin23:10:252
2667699161,6cyclictest0-21swapper/223:35:212
2667699161,4cyclictest0-21swapper/200:30:262
26676991612,3cyclictest0-21swapper/219:30:142
2667699161,14cyclictest0-21swapper/220:21:072
2667699161,14cyclictest0-21swapper/220:16:412
26676991611,3cyclictest32-21ksoftirqd/221:25:242
26676991611,3cyclictest29496-21ls22:55:122
2667699161,10cyclictest0-21swapper/220:31:462
26676991610,4cyclictest0-21swapper/221:45:242
26676991610,3cyclictest8044-21/usr/sbin/munin23:20:112
26676991510,3cyclictest30-21rcuc/219:24:222
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional