You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-30 - 06:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue Dec 30, 2025 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
208742223155,21sleep10-21swapper/119:05:441
192462203166,25sleep20-21swapper/219:05:092
211832202165,24sleep30-21swapper/319:09:413
211482202164,26sleep00-21swapper/019:09:140
3155021550,4sleep12129999cyclictest20:45:011
428221300,4sleep02129499cyclictest22:10:120
271002680,3sleep011-21rcuc/020:35:010
317342620,4sleep32131199cyclictest00:25:153
21304996255,2cyclictest0-21swapper/219:25:082
93032540,2sleep39305-21gltestperf22:20:193
21304994845,2cyclictest0-21swapper/219:55:522
312682460,4sleep230-21rcuc/219:30:142
150412440,2sleep10-21swapper/120:05:201
21304994138,2cyclictest0-21swapper/219:52:052
21304994137,3cyclictest0-21swapper/222:23:102
21294994011,19cyclictest13642-21grep22:30:170
2129999385,22cyclictest751-21rngd00:01:411
2129999375,23cyclictest751-21rngd19:28:081
21299993725,1cyclictest0-21swapper/120:20:101
2129999371,27cyclictest0-21swapper/120:34:061
21294993712,12cyclictest31058-21munin-run00:25:010
2129999365,22cyclictest751-21rngd19:38:571
2129499367,14cyclictest23207-21awk21:40:010
21294993615,18cyclictest20033-21perf00:00:020
2129999359,17cyclictest0-21swapper/120:47:581
2129999352,26cyclictest0-21swapper/119:44:011
21299993510,8cyclictest87050irq/127-enp1s0-21:20:261
2129499357,13cyclictest27602-21basename21:50:000
21294993511,13cyclictest9501-21expr23:35:130
2129499349,12cyclictest31990-21munin-run22:00:000
2129499346,13cyclictest32617-21munin-run23:15:010
21294993415,9cyclictest19727-21/usr/sbin/munin22:45:130
2129499335,13cyclictest26660-21munin-run00:15:010
2129499321,29cyclictest0-21swapper/020:22:020
21294993212,16cyclictest24141-21cpuspeed_turbos22:55:130
2129999311,21cyclictest0-21swapper/119:55:591
2129499317,9cyclictest0-21swapper/021:10:230
2129499315,17cyclictest751-21rngd23:49:270
2129499313,9cyclictest751-21rngd23:08:480
21294993124,4cyclictest1839-21munin-run22:05:000
21294993114,14cyclictest12831-21perf22:30:010
2129499311,28cyclictest0-21swapper/020:51:460
2131199301,10cyclictest0-21swapper/322:17:543
2129999303,19cyclictest751-21rngd22:38:341
2129999301,22cyclictest751-21rngd20:59:461
2129999301,22cyclictest751-21rngd20:59:461
2129999301,21cyclictest0-21swapper/123:53:111
2129499308,10cyclictest28783-21/usr/sbin/munin19:25:160
21294993013,14cyclictest24495-21sh21:40:220
21294993011,9cyclictest9662-21netstat22:20:220
2131199291,8cyclictest8575-21fschecks_time21:05:173
2131199290,8cyclictest0-21swapper/321:30:023
2129999293,17cyclictest751-21rngd22:20:011
2129999293,16cyclictest0-21swapper/100:24:301
2129999293,12cyclictest751-21rngd20:17:421
2129999292,9cyclictest751-21rngd00:17:441
2129999292,17cyclictest0-21swapper/100:38:161
2129999291,10cyclictest0-21swapper/123:29:061
21299992910,4cyclictest751-21rngd23:18:241
2129499294,8cyclictest0-21swapper/000:18:550
2129499293,16cyclictest12000-21kworker/0:321:20:060
2129499293,14cyclictest751-21rngd22:40:220
2129499292,20cyclictest0-21swapper/023:24:460
2129499292,20cyclictest0-21swapper/023:24:460
21294992921,5cyclictest11589-21tr23:40:120
21294992913,13cyclictest13186-21if_enp2s021:15:180
21294992912,9cyclictest6465-21acpi22:15:110
2129499291,12cyclictest0-21swapper/019:44:000
2129499291,12cyclictest0-21swapper/019:21:260
21294992910,13cyclictest0-21swapper/000:36:450
2129999282,10cyclictest0-21swapper/119:12:381
21299992815,5cyclictest31773-21systemd-journal21:57:291
2129999281,26cyclictest0-21swapper/121:13:021
2129999281,16cyclictest0-21swapper/119:46:051
2129999281,16cyclictest0-21swapper/119:46:051
2129999281,14cyclictest0-21swapper/121:49:541
2129999281,12cyclictest0-21swapper/121:04:581
2129999281,11cyclictest0-21swapper/122:13:191
2129999281,10cyclictest0-21swapper/122:46:181
2129999281,10cyclictest0-21swapper/120:50:101
21294992812,6cyclictest16232-21fschecks_count23:50:140
2129499281,14cyclictest0-21swapper/022:51:100
2129499281,14cyclictest0-21swapper/020:44:420
2129499281,13cyclictest0-21swapper/020:56:220
2129499281,13cyclictest0-21swapper/020:56:220
2129499281,13cyclictest0-21swapper/019:11:100
2129499281,11cyclictest0-21swapper/019:39:180
21294992810,15cyclictest22729-21expr00:05:130
2129999279,3cyclictest0-21swapper/121:25:201
2129999274,13cyclictest751-21rngd19:32:151
2129999271,13cyclictest0-21swapper/119:50:331
2129999271,11cyclictest0-21swapper/122:42:341
2129999271,10cyclictest0-21swapper/122:23:011
2129999271,10cyclictest0-21swapper/120:40:001
2129999270,24cyclictest0-21swapper/119:20:301
2129999270,11cyclictest0-21swapper/121:34:401
2129499279,6cyclictest2978-21memory22:05:210
2129499279,6cyclictest2978-21memory22:05:210
2129499279,15cyclictest2465-21idleruntime-cro23:20:010
2129499278,9cyclictest835-21runrttasks21:34:500
21294992719,5cyclictest8669-21smart_sda19:50:230
21294992716,8cyclictest26908-21hddtemp_smartct23:00:160
2129499271,13cyclictest0-21swapper/019:31:580
2129499271,13cyclictest0-21swapper/019:19:580
21294992710,9cyclictest0-21swapper/000:33:220
21294992710,9cyclictest0-21swapper/000:33:220
2131199267,4cyclictest39-21rcuc/321:10:243
2129999265,3cyclictest751-21rngd22:54:261
2129999263,13cyclictest751-21rngd23:06:181
2129999263,13cyclictest751-21rngd19:19:541
2129999262,11cyclictest0-21swapper/123:33:441
2129999261,7cyclictest0-21swapper/120:00:091
2129999261,6cyclictest0-21swapper/123:45:301
2129999261,17cyclictest87250irq/129-enp1s0-00:05:231
2129999261,13cyclictest0-21swapper/121:42:501
21299992611,1cyclictest0-21swapper/121:39:341
2129999261,11cyclictest0-21swapper/100:32:421
2129999261,11cyclictest0-21swapper/100:32:421
2129499269,14cyclictest835-21runrttasks21:27:380
2129499269,10cyclictest0-21swapper/020:35:520
2129499262,17cyclictest0-21swapper/023:33:250
21294992618,5cyclictest9831-40dnf21:07:560
21294992618,5cyclictest6473-21smart_sda19:45:230
21294992618,5cyclictest6473-21smart_sda19:45:230
21294992618,5cyclictest17224-21awk22:40:020
21294992618,5cyclictest15987-21munin-run20:10:010
21294992616,7cyclictest20389-21awk20:20:020
21294992616,6cyclictest12148-21chrt20:00:140
21294992613,4cyclictest751-21rngd20:27:140
21294992613,11cyclictest7824-21awk21:05:010
2129499261,16cyclictest0-21swapper/019:58:140
2129499261,11cyclictest0-21swapper/021:50:230
2129499261,10cyclictest0-21swapper/000:29:020
21294992610,8cyclictest0-21swapper/023:25:220
284402250,5sleep230-21rcuc/221:50:162
2131199251,6cyclictest0-21swapper/300:14:183
2131199251,10cyclictest0-21swapper/321:20:003
2130499251,6cyclictest0-21swapper/223:22:172
2130499251,6cyclictest0-21swapper/223:22:172
2129999257,3cyclictest0-21swapper/123:41:361
2129999255,3cyclictest0-21swapper/120:28:141
2129999253,12cyclictest13498-21ntp_states21:15:221
2129999252,5cyclictest0-21swapper/122:07:031
2129999252,5cyclictest0-21swapper/122:07:031
2129999251,13cyclictest0-21swapper/123:14:401
2129999251,10cyclictest0-21swapper/122:25:101
2129999250,10cyclictest87250irq/129-enp1s0-22:01:281
21294992516,5cyclictest782-21in:imjournal00:00:250
21294992513,6cyclictest0-21swapper/020:45:120
21304992410,4cyclictest18150-21sort22:40:172
2129999246,5cyclictest751-21rngd22:57:261
2129999245,3cyclictest13963-21load22:30:201
2129999241,16cyclictest0-21swapper/123:55:211
2129999241,12cyclictest0-21swapper/123:21:421
2129999241,12cyclictest0-21swapper/123:21:421
2129999241,10cyclictest0-21swapper/121:53:581
2129999241,10cyclictest0-21swapper/120:11:141
2129999240,16cyclictest25035-21diskstats00:10:151
21294992413,8cyclictest16684-21fschecks_time20:10:170
2131199238,4cyclictest0-21swapper/321:31:583
2131199231,10cyclictest21855-21users00:00:273
2130499238,4cyclictest264782expr23:00:132
2130499231,6cyclictest0-21swapper/200:07:502
2129999238,3cyclictest0-21swapper/123:02:301
2129999231,3cyclictest0-21swapper/100:28:251
2131199221,5cyclictest0-21swapper/321:58:033
2131199221,15cyclictest0-21swapper/320:20:283
2131199221,12cyclictest0-21swapper/323:42:573
2130499225,5cyclictest751-21rngd00:00:252
2129999221,3cyclictest0-21swapper/121:07:101
2131199213,5cyclictest30421-1kworker/u9:222:58:533
2131199212,11cyclictest13663-21/usr/sbin/munin23:45:253
2131199211,4cyclictest0-21swapper/319:55:253
2131199211,3cyclictest2281-21unixbench-2d19:35:273
2130499215,6cyclictest19187-21cut21:30:132
2130499212,3cyclictest0-21swapper/200:13:502
2130499211,6cyclictest0-21swapper/221:41:522
2130499211,4cyclictest0-21swapper/221:47:402
2130499211,3cyclictest25921-21kworker/u8:223:06:042
2131199207,4cyclictest17981-21latency_hist23:55:023
2131199204,8cyclictest0-21swapper/319:25:173
2131199203,10cyclictest0-21swapper/322:30:213
2131199202,10cyclictest23421-21/usr/sbin/munin21:40:143
2131199202,10cyclictest19641-21/usr/sbin/munin22:45:193
2131199202,10cyclictest1408-21/usr/sbin/munin20:50:213
2131199201,6cyclictest0-21swapper/323:19:033
2131199201,5cyclictest0-21swapper/322:10:293
2131199201,4cyclictest19719-21kworker/u8:223:57:023
21311992013,4cyclictest28138-21grep19:20:283
2131199201,16cyclictest22336-21kernelversion19:10:193
2131199201,14cyclictest0-21swapper/320:25:253
2131199201,11cyclictest26591-21users20:30:253
2131199201,11cyclictest0-21swapper/319:45:203
2131199201,11cyclictest0-21swapper/319:45:203
21311992010,2cyclictest19997-21users20:15:283
2130499206,4cyclictest32291-21cat22:00:132
2130499202,3cyclictest0-21swapper/223:27:342
2130499202,3cyclictest0-21swapper/223:18:532
2130499201,6cyclictest0-21swapper/222:20:022
2130499201,4cyclictest0-21swapper/222:56:192
2130499201,4cyclictest0-21swapper/222:37:062
2130499201,11cyclictest0-21swapper/222:10:142
2130499200,4cyclictest25921-21kworker/u8:223:37:552
2129999206,3cyclictest751-21rngd23:37:541
2131199196,7cyclictest0-21swapper/319:30:573
2131199192,9cyclictest3010-21/usr/sbin/munin19:40:223
2131199192,7cyclictest21498-21expr21:35:143
2131199191,7cyclictest25921-21kworker/u8:223:05:183
2131199191,6cyclictest0-21swapper/300:05:503
2131199191,5cyclictest0-21swapper/321:23:243
2131199191,4cyclictest0-21swapper/322:41:443
2131199191,2cyclictest0-21swapper/322:27:493
2131199191,11cyclictest801-21snmpd23:12:323
2131199191,10cyclictest473-21idleruntime22:00:183
2131199191,10cyclictest4675-21cat00:35:293
2131199191,10cyclictest10859-21users23:35:283
2131199191,10cyclictest0-21swapper/323:25:163
2131199191,10cyclictest0-21swapper/320:40:283
21311991910,3cyclictest0-21swapper/320:49:303
2130499195,6cyclictest86950irq/126-enp1s0-21:10:252
2130499193,12cyclictest87250irq/129-enp1s0-00:35:262
2130499192,14cyclictest2419-21strings20:50:202
2130499191,7cyclictest31706-21grep23:10:212
2130499191,7cyclictest19194-21pmu-power23:55:252
2130499191,7cyclictest0-21swapper/221:25:182
2130499191,6cyclictest801-21snmpd00:21:442
2130499191,6cyclictest0-21swapper/200:17:522
2130499191,3cyclictest0-21swapper/222:05:042
2130499191,3cyclictest0-21swapper/222:05:042
2130499191,16cyclictest0-21swapper/221:00:252
2130499190,4cyclictest32-21ksoftirqd/222:51:252
2131199189,2cyclictest86950irq/126-enp1s0-21:00:243
2131199188,2cyclictest801-21snmpd20:05:593
2131199188,2cyclictest4892-21smart_sda20:55:243
2131199188,2cyclictest4892-21smart_sda20:55:243
2131199187,3cyclictest7073-21/usr/sbin/munin23:30:193
2131199184,9cyclictest3898-40dnf22:08:273
2131199184,9cyclictest3898-40dnf22:08:273
2131199183,5cyclictest30421-1kworker/u9:221:46:023
2131199181,7cyclictest30421-1kworker/u9:221:54:593
2131199181,6cyclictest0-21swapper/323:05:003
2131199181,6cyclictest0-21swapper/300:24:023
2131199181,6cyclictest0-21swapper/300:16:393
2131199181,3cyclictest27803-21gltestperf20:35:173
2131199181,2cyclictest16415-21ps22:35:233
2131199181,10cyclictest24429-21kworker/3:100:30:263
2131199181,10cyclictest24429-21kworker/3:100:30:263
2131199181,10cyclictest11804-21/usr/sbin/munin20:00:143
2131199180,5cyclictest30421-1kworker/u9:222:51:283
2130499189,3cyclictest87250irq/129-enp1s0-22:25:262
2130499186,4cyclictest10007-21diskmemload21:15:032
2130499181,8cyclictest9075-21latency_hist23:35:012
2130499181,6cyclictest31423-21grep00:25:122
2130499181,5cyclictest0-21swapper/222:30:182
21304991812,4cyclictest9150-21sed21:05:252
21304991812,3cyclictest835-21runrttasks20:45:162
21304991812,3cyclictest16277-21fschecks_count23:50:152
2130499181,11cyclictest0-21swapper/223:41:322
2130499181,10cyclictest4113-21open_inodes19:40:212
2131199178,2cyclictest0-21swapper/319:50:123
2131199171,6cyclictest17280-21ntpq20:10:233
2131199171,6cyclictest0-21swapper/323:24:353
2131199171,6cyclictest0-21swapper/323:24:353
2130499173,5cyclictest12217-21tune2fs20:00:152
2130499173,5cyclictest0-21swapper/221:22:142
2130499173,4cyclictest7176-21sh19:50:012
2130499173,4cyclictest7176-21sh19:50:012
2130499172,6cyclictest29397-21/usr/sbin/munin20:40:192
2130499171,7cyclictest23055-21grep19:10:262
2130499171,7cyclictest0-21swapper/222:45:372
2130499171,6cyclictest27756-21proc_pri19:20:252
2130499171,6cyclictest0-21swapper/200:34:202
2130499171,6cyclictest0-21swapper/200:34:202
2130499171,5cyclictest0-21swapper/221:56:372
2130499171,4cyclictest0-21swapper/220:35:282
21304991712,3cyclictest16320-21grep20:10:132
21304991711,3cyclictest24769-21latency_hist20:30:002
2130499171,11cyclictest0-21swapper/221:36:472
21304991710,5cyclictest0-21swapper/220:55:252
21304991710,5cyclictest0-21swapper/220:55:252
2130499170,7cyclictest25921-21kworker/u8:223:47:502
21311991612,3cyclictest0-21swapper/319:18:533
2130499161,6cyclictest24389-21/usr/sbin/munin19:15:132
2130499161,6cyclictest22574-21latency_hist20:25:012
2130499161,14cyclictest0-21swapper/220:34:312
21304991611,3cyclictest20414-21cut20:20:012
2130499151,12cyclictest880-21cpuspeed_turbos19:35:142
21304991510,3cyclictest14013-21/usr/sbin/munin20:05:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional