You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-09 - 18:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Mar 09, 2026 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
586522880,3sleep011-21rcuc/007:05:160
71042202164,25sleep30-21swapper/307:08:263
69972202166,24sleep10-21swapper/107:07:021
69582202165,25sleep20-21swapper/207:06:312
2740721550,5sleep2732099cyclictest11:34:592
7326999693,2cyclictest0-21swapper/308:01:413
7326996057,2cyclictest0-21swapper/308:13:043
51392440,4sleep35140-21cut09:30:123
198142420,2sleep30-21swapper/310:00:263
7326993726,3cyclictest0-21swapper/311:36:153
732699371,17cyclictest20855-21fschecks_count10:05:153
7326993630,2cyclictest0-21swapper/311:18:073
7326993622,8cyclictest0-21swapper/307:55:093
732699361,26cyclictest0-21swapper/310:33:083
730899368,13cyclictest1656-21munin-run11:50:020
730899366,14cyclictest12656-21munin-run12:15:010
7326993519,9cyclictest0-21swapper/307:33:433
732699351,14cyclictest21111-21switchtime08:50:253
7326993419,10cyclictest0-21swapper/308:21:383
732699340,27cyclictest0-21swapper/310:28:543
7326993320,3cyclictest0-21swapper/307:39:093
732699331,14cyclictest0-21swapper/312:10:123
732699331,13cyclictest25558-21munin-run07:50:013
730899337,12cyclictest21457-21perf12:35:010
730899337,12cyclictest21457-21perf12:35:010
7326993218,8cyclictest0-21swapper/308:09:493
732699321,15cyclictest0-21swapper/312:16:303
732699320,19cyclictest0-21swapper/310:51:253
7326993118,8cyclictest0-21swapper/308:28:263
7326993117,9cyclictest0-21swapper/308:15:423
7326993117,3cyclictest0-21swapper/308:58:473
732699311,19cyclictest0-21swapper/307:54:413
732699311,19cyclictest0-21swapper/307:54:413
732699311,15cyclictest0-21swapper/312:21:363
731699313,5cyclictest19603-21kworker/u8:309:27:291
7326993020,3cyclictest0-21swapper/309:05:333
7326993019,3cyclictest0-21swapper/311:54:353
7326993017,8cyclictest0-21swapper/307:21:293
732699301,20cyclictest6623-21diskstats12:00:153
732699301,20cyclictest0-21swapper/309:14:433
732699300,16cyclictest0-21swapper/310:39:183
7326992919,3cyclictest0-21swapper/308:39:123
7326992918,3cyclictest0-21swapper/309:17:113
7326992912,2cyclictest314692sleep311:40:433
732699291,13cyclictest0-21swapper/310:23:063
732699291,12cyclictest0-21swapper/311:12:593
732699291,12cyclictest0-21swapper/310:11:583
732699291,12cyclictest0-21swapper/310:11:583
732699291,11cyclictest28377-21diskmemload11:31:413
732699290,13cyclictest0-21swapper/312:35:123
7326992816,7cyclictest0-21swapper/307:29:313
732699281,13cyclictest0-21swapper/311:09:473
732699281,13cyclictest0-21swapper/310:57:253
732699281,13cyclictest0-21swapper/309:48:113
732699281,12cyclictest0-21swapper/309:53:313
732699280,11cyclictest0-21swapper/309:44:453
730899286,10cyclictest16785-21perf07:30:000
7308992811,15cyclictest13007-21perf08:35:020
730899280,13cyclictest14231-21idleruntime-cro11:05:000
243392280,5sleep124340-21cpuspeed_turbos09:00:121
243392280,5sleep124340-21cpuspeed_turbos09:00:121
732699271,9cyclictest0-21swapper/312:32:383
732699271,9cyclictest0-21swapper/312:32:383
7326992712,3cyclictest0-21swapper/311:58:593
732699271,17cyclictest4453-21strings10:40:193
732699271,13cyclictest0-21swapper/312:25:123
732699270,13cyclictest0-21swapper/311:20:533
732699270,13cyclictest0-21swapper/311:20:533
732699261,12cyclictest0-21swapper/311:46:173
732699261,11cyclictest0-21swapper/312:09:593
731699261,16cyclictest0-21swapper/111:25:591
730899263,11cyclictest0-21swapper/008:50:010
732699259,9cyclictest0-21swapper/307:18:323
732699251,11cyclictest0-21swapper/308:44:283
732699250,11cyclictest0-21swapper/310:16:263
732699250,11cyclictest0-21swapper/309:26:053
732099259,4cyclictest2509-21tr11:50:152
731699251,16cyclictest0-21swapper/112:19:231
730899251,12cyclictest21673-21/usr/sbin/munin12:35:260
730899251,11cyclictest25676-21grep10:15:180
730899250,5cyclictest19603-21kworker/u8:309:31:280
732699249,10cyclictest0-21swapper/307:11:543
732699241,9cyclictest28377-21diskmemload11:04:073
732699241,15cyclictest0-21swapper/310:45:213
732699241,15cyclictest0-21swapper/309:55:133
732699241,15cyclictest0-21swapper/308:32:443
7326992410,3cyclictest0-21swapper/309:39:073
731699244,16cyclictest27102-40dnf11:31:081
7316992417,4cyclictest8846-21/usr/sbin/munin12:05:141
731699241,5cyclictest312-21if_enp1s011:45:171
731699241,5cyclictest11740-21ntpq12:10:201
731699241,13cyclictest28612-21/usr/sbin/munin09:10:251
730899242,10cyclictest25450-21/usr/sbin/munin11:30:150
730899241,21cyclictest0-21swapper/012:20:200
730899241,21cyclictest0-21swapper/011:55:230
730899241,11cyclictest17007-21users08:40:260
7308992410,2cyclictest9024-21grep07:10:250
730899240,21cyclictest11-21rcuc/007:50:250
730899240,21cyclictest11-21rcuc/007:50:250
732699239,9cyclictest0-21swapper/307:42:253
732699236,3cyclictest0-21swapper/309:23:173
7326992310,3cyclictest0-21swapper/309:03:513
7326992310,3cyclictest0-21swapper/309:03:513
732099231,8cyclictest1648-21sh11:50:012
731699232,2cyclictest88350irq/128-enp1s0-09:52:231
731699231,4cyclictest5323-21latency08:15:191
731699231,20cyclictest0-21swapper/110:02:341
731699231,19cyclictest28450-21fschecks_time07:55:131
731699231,14cyclictest0-21swapper/110:40:221
731699231,12cyclictest27767-21grep11:35:111
731699231,11cyclictest0-21swapper/109:57:531
731699230,14cyclictest0-21swapper/109:36:061
730899231,20cyclictest0-21swapper/009:46:090
730899231,11cyclictest30181-21/usr/sbin/munin08:00:200
730899231,11cyclictest23588-21/usr/sbin/munin07:45:120
730899230,11cyclictest23627-21strings10:10:200
730899230,11cyclictest23627-21strings10:10:200
732699224,3cyclictest0-21swapper/308:45:163
732099226,9cyclictest28377-21diskmemload10:44:252
732099226,3cyclictest31405-21head11:40:272
732099223,9cyclictest0-21swapper/210:05:242
732099223,10cyclictest14402-21cat11:05:012
732099222,11cyclictest19590-21munin-run08:50:012
732099222,11cyclictest15276-21irqstats11:05:172
7320992216,3cyclictest18032-21timerandwakeup11:10:272
732099221,5cyclictest0-21swapper/209:35:242
732099221,10cyclictest0-21swapper/210:58:062
731699229,4cyclictest88350irq/128-enp1s0-12:20:231
731699228,9cyclictest0-21swapper/111:18:041
731699228,12cyclictest29392-21kworker/u8:211:54:141
731699222,11cyclictest88350irq/128-enp1s0-10:55:181
7316992218,3cyclictest0-21swapper/111:02:211
7316992214,3cyclictest0-21swapper/112:02:151
731699221,16cyclictest0-21swapper/111:08:241
7316992211,3cyclictest18877-21smart_sda08:45:241
731699221,10cyclictest0-21swapper/110:36:121
731699221,10cyclictest0-21swapper/109:43:231
7316992210,3cyclictest0-21swapper/110:05:171
730899229,3cyclictest88350irq/128-enp1s0-11:36:240
730899222,8cyclictest31427-21/usr/sbin/munin10:30:230
730899221,11cyclictest0-21swapper/012:15:260
730899221,10cyclictest22008-21/usr/sbin/munin08:55:200
730899220,20cyclictest0-21swapper/007:55:190
732699218,3cyclictest0-21swapper/311:25:113
732099215,9cyclictest88150irq/126-enp1s0-11:20:132
732099215,9cyclictest88150irq/126-enp1s0-11:20:132
732099214,9cyclictest10415-21cat09:40:192
732099211,8cyclictest0-21swapper/210:25:142
732099211,5cyclictest12897-21sensors09:45:232
732099211,5cyclictest12011-21strings08:30:172
732099211,10cyclictest22520-21cut07:40:232
731699218,3cyclictest27031-21/usr/sbin/munin10:20:161
731699211,9cyclictest0-21swapper/111:11:381
731699211,4cyclictest0-21swapper/112:25:201
731699211,15cyclictest0-21swapper/110:46:541
731699211,14cyclictest0-21swapper/111:40:331
731699211,11cyclictest32281-21cat08:05:021
731699211,11cyclictest29576-21diskstats10:25:141
731699210,15cyclictest0-21swapper/112:37:131
730899217,3cyclictest8487-21/usr/sbin/munin12:05:140
730899216,3cyclictest9446-21/usr/sbin/munin09:40:250
730899216,14cyclictest0-21swapper/009:15:000
730899211,6cyclictest0-21swapper/011:07:030
7308992112,4cyclictest88350irq/128-enp1s0-10:20:410
730899211,17cyclictest823-21snmpd08:22:510
92092201,2sleep20-21swapper/207:11:372
732099201,7cyclictest4632-21taskset11:55:142
732099201,7cyclictest29616-21latency_hist11:40:012
732099201,7cyclictest16764-21latency_hist07:30:012
732099201,7cyclictest13556-21hddtemp_smartct12:15:152
732099201,7cyclictest0-21swapper/210:49:002
732099201,6cyclictest16951-21memory09:55:182
732099201,6cyclictest12370-21sh07:20:022
732099201,4cyclictest0-21swapper/210:30:142
732099201,13cyclictest0-21swapper/212:00:082
732099201,10cyclictest23941-21sed11:25:172
732099201,10cyclictest2227-21/usr/sbin/munin08:10:192
731699209,3cyclictest25895-21ntp_states10:15:211
731699209,3cyclictest23590-21acpi07:45:111
731699209,3cyclictest21744-21diskstats07:40:141
731699202,17cyclictest0-21swapper/107:12:251
731699201,7cyclictest0-21swapper/111:20:371
731699201,7cyclictest0-21swapper/111:20:371
7316992017,2cyclictest0-21swapper/107:35:191
7316992016,3cyclictest0-21swapper/108:08:051
7316992012,3cyclictest0-21swapper/111:57:461
731699201,18cyclictest0-21swapper/108:12:311
731699201,13cyclictest0-21swapper/112:32:541
731699201,13cyclictest0-21swapper/112:32:541
731699201,13cyclictest0-21swapper/109:47:571
731699201,11cyclictest10757-21diskstats07:15:141
731699201,10cyclictest22133-21cpuspeed_turbos08:55:131
731699200,18cyclictest0-21swapper/108:35:531
731699200,12cyclictest88350irq/128-enp1s0-09:22:241
730899207,2cyclictest823-21snmpd11:13:230
730899206,3cyclictest20441-21/usr/sbin/munin10:05:170
730899204,6cyclictest0-21swapper/011:40:140
730899201,6cyclictest0-21swapper/011:53:300
730899201,6cyclictest0-21swapper/010:53:360
730899201,5cyclictest13838-21/usr/sbin/munin09:50:200
7308992014,3cyclictest21047-21find08:50:230
730899201,18cyclictest0-21swapper/007:15:070
730899201,10cyclictest0-21swapper/008:25:180
730899200,19cyclictest0-21swapper/010:45:250
732099195,3cyclictest88450irq/129-enp1s0-12:10:142
732099194,9cyclictest0-21swapper/212:35:102
732099194,5cyclictest6612-21sshd08:20:082
732099194,3cyclictest30-21rcuc/207:55:212
732099193,8cyclictest12050irq/123-ahci[0009:00:022
732099193,3cyclictest15608-21ls08:40:122
732099193,10cyclictest0-21swapper/209:19:172
732099192,5cyclictest1751-21pmu-power09:20:222
732099191,9cyclictest19745-21unixbench_multi10:00:252
732099191,7cyclictest842-21lldpd08:07:132
732099191,6cyclictest823-21snmpd07:37:422
732099191,6cyclictest2840-21/usr/sbin/munin09:25:172
732099191,6cyclictest27748-21latency_hist07:55:022
732099191,6cyclictest27748-21latency_hist07:55:012
732099191,6cyclictest20162-21diskstats08:50:152
732099191,6cyclictest18964-21latency_hist07:35:012
732099191,6cyclictest0-21swapper/209:34:422
7320991916,2cyclictest0-21swapper/210:52:572
732099191,5cyclictest19474-21/usr/sbin/munin12:30:212
732099191,5cyclictest19474-21/usr/sbin/munin12:30:212
732099191,5cyclictest15183-21latency_hist08:40:012
732099191,5cyclictest13599-21cut07:20:192
7320991914,3cyclictest28609-21grep10:20:262
732099191,11cyclictest0-21swapper/209:52:412
732099191,10cyclictest3240-21perf10:40:022
7316991916,2cyclictest0-21swapper/109:16:041
7316991915,3cyclictest0-21swapper/108:25:421
7316991915,3cyclictest0-21swapper/107:20:261
7316991913,3cyclictest8680-21irqstats10:50:191
7316991913,3cyclictest6625-40dnf09:30:261
731699191,17cyclictest0-21swapper/110:13:271
731699191,17cyclictest0-21swapper/110:13:271
731699191,16cyclictest0-21swapper/107:50:041
731699191,16cyclictest0-21swapper/107:50:041
731699191,15cyclictest14801-21sshd07:25:101
731699191,10cyclictest32118-21cat10:30:161
730899198,1cyclictest0-21swapper/007:35:120
730899196,3cyclictest18293-21switchtime07:30:230
730899195,4cyclictest28377-21diskmemload10:26:220
730899194,5cyclictest2534-21sshd09:24:100
730899194,3cyclictest2886-21sshd10:35:400
730899194,2cyclictest88350irq/128-enp1s0-12:02:440
730899191,9cyclictest0-21swapper/008:18:090
730899191,7cyclictest3464-21/usr/sbin/munin10:40:210
730899191,6cyclictest4104-21chrt09:25:230
730899191,6cyclictest28383-21munin-run09:10:000
730899191,6cyclictest0-21swapper/010:58:520
7308991914,3cyclictest9225-21awk09:40:020
732099185,3cyclictest88450irq/129-enp1s0-10:15:232
732099184,8cyclictest6910-21kworker/u8:211:16:042
732099181,5cyclictest17274-21/usr/sbin/munin12:25:122
732099181,5cyclictest0-21swapper/212:05:172
732099181,4cyclictest15234-21cpuspeed_turbos12:20:132
732099181,3cyclictest29873-21smart_sda09:10:252
7320991812,3cyclictest26544-21tr09:05:132
7320991810,5cyclictest10085-21smart_sda08:25:242
731699181,9cyclictest17750-21if_enp2s007:30:151
7316991815,2cyclictest0-21swapper/109:06:261
7316991813,3cyclictest20341-21/usr/sbin/munin08:50:161
731699181,16cyclictest0-21swapper/108:40:281
731699181,16cyclictest0-21swapper/108:30:491
731699181,16cyclictest0-21swapper/108:23:261
730899184,3cyclictest88350irq/128-enp1s0-08:07:480
730899184,3cyclictest88350irq/128-enp1s0-07:20:180
730899184,3cyclictest28377-21diskmemload10:00:180
730899183,5cyclictest6910-21kworker/u8:211:17:500
730899181,6cyclictest241702wc11:25:200
730899181,6cyclictest22540-21head07:40:230
730899181,4cyclictest14441-21ps08:35:230
40302181,2sleep00-21swapper/008:13:050
188862181,2sleep00-21swapper/012:25:360
160332181,2sleep00-21swapper/009:55:100
732099171,5cyclictest0-21swapper/210:10:132
732099171,5cyclictest0-21swapper/210:10:132
732099171,4cyclictest30181-21/usr/sbin/munin08:00:272
732099171,4cyclictest25186-21vmstat07:45:262
732099171,4cyclictest24206-21/usr/sbin/munin09:00:122
732099171,4cyclictest24206-21/usr/sbin/munin09:00:122
730899174,2cyclictest88350irq/128-enp1s0-09:15:110
730899171,4cyclictest21054-21/usr/sbin/munin11:20:130
730899171,4cyclictest21054-21/usr/sbin/munin11:20:130
730899171,3cyclictest24206-21/usr/sbin/munin09:00:210
730899171,3cyclictest24206-21/usr/sbin/munin09:00:210
7320991610,3cyclictest4562-21tr08:15:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional