You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-02 - 23:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Wed Sep 02, 2026 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
253402238164,24sleep30-21swapper/307:09:053
251142236163,24sleep20-21swapper/207:06:112
252782235163,58sleep10-21swapper/107:08:191
253222204166,25sleep00-21swapper/007:08:510
1603021740,4sleep22550999cyclictest10:25:202
250521600,3sleep02549999cyclictest08:40:240
1750121460,4sleep02549999cyclictest11:43:240
1750121460,4sleep02549999cyclictest11:43:240
134021440,4sleep12550399cyclictest07:25:161
25509991131,2cyclictest0-21swapper/212:24:012
25509998273,3cyclictest0-21swapper/208:50:062
102212560,7sleep3401ktimersoftd/309:00:133
206662510,1sleep30-21swapper/308:10:133
206662510,1sleep30-21swapper/308:10:133
25509994947,1cyclictest0-21swapper/210:55:362
25042490,4sleep07-21ksoftirqd/011:10:120
143822460,2sleep10-21swapper/111:35:181
275502450,1sleep00-21swapper/010:50:260
167112410,4sleep20-21swapper/211:40:202
167112410,4sleep20-21swapper/211:40:202
25509994037,2cyclictest0-21swapper/212:29:422
2549999397,22cyclictest14436-21/usr/sbin/munin09:10:230
2549999391,29cyclictest6878-21/usr/sbin/munin11:20:210
2549999384,22cyclictest28244-21/usr/sbin/munin10:55:170
2549999381,20cyclictest0-21swapper/007:55:120
2549999375,22cyclictest87350irq/126-enp1s0-08:40:010
2549999372,27cyclictest7215-21/usr/sbin/munin07:40:200
2549999371,26cyclictest12850-21/usr/sbin/munin10:20:180
2549999371,22cyclictest9104-21cut10:10:150
2549999364,22cyclictest0-21swapper/011:15:280
2549999362,29cyclictest29297-40dnf09:41:230
2549999361,20cyclictest17781-21head09:15:220
2551699350,33cyclictest0-21swapper/310:55:213
2549999356,21cyclictest5301-21/usr/sbin/munin12:30:160
2549999355,20cyclictest87550irq/128-enp1s0-08:30:230
2549999351,8cyclictest18212-21/usr/sbin/munin08:05:250
2549999351,25cyclictest9076-21/usr/sbin/munin11:25:230
2549999351,23cyclictest13023-21sshd11:32:230
2549999349,12cyclictest28979-21munin-run08:30:000
2549999343,27cyclictest1968-21ntp_states12:20:220
2549999341,24cyclictest32019-21/usr/sbin/munin09:50:230
2549999340,26cyclictest0-21swapper/009:35:260
2551699330,15cyclictest0-21swapper/309:54:433
2549999338,12cyclictest19224-21munin-run10:35:010
2549999331,5cyclictest0-21swapper/012:25:110
2549999331,22cyclictest32753-21latency_hist07:25:020
2549999331,18cyclictest0-21swapper/012:35:210
2551699321,20cyclictest0-21swapper/309:07:433
2551699321,20cyclictest0-21swapper/308:43:193
2551699321,20cyclictest0-21swapper/308:20:513
2551699320,21cyclictest0-21swapper/308:28:513
2551699320,16cyclictest0-21swapper/309:30:433
2549999326,19cyclictest87550irq/128-enp1s0-08:00:220
2549999322,3cyclictest87550irq/128-enp1s0-09:22:210
2549999322,17cyclictest17864-21/usr/sbin/munin11:45:110
2549999322,16cyclictest26821-21pmu-power07:10:220
2549999321,5cyclictest11895-21fgrep10:15:240
2549999321,4cyclictest0-21swapper/010:45:190
25499993213,16cyclictest24229-21sadc12:00:010
2551699311,14cyclictest0-21swapper/310:05:193
2549999317,17cyclictest30560-21awk07:20:010
2549999315,18cyclictest808-21snmpd07:46:420
2549999313,16cyclictest0-21swapper/009:07:050
2549999312,25cyclictest23870-21smart_sda08:15:240
2549999311,26cyclictest790-21in:imjournal08:50:020
25499993110,18cyclictest31814-21if_enp2s012:15:180
2551699301,14cyclictest0-21swapper/310:28:553
2549999302,24cyclictest23393-40dnf10:42:230
2549999302,16cyclictest26672-21/usr/sbin/munin12:05:260
2549999301,3cyclictest87550irq/128-enp1s0-12:00:250
2549999301,3cyclictest87350irq/126-enp1s0-11:50:230
2549999301,16cyclictest0-21swapper/008:20:120
25499993010,17cyclictest6251-21grep07:35:230
2549999300,3cyclictest8564-1kworker/u9:110:35:120
2551699291,19cyclictest0-21swapper/311:42:393
2551699291,19cyclictest0-21swapper/311:42:393
2551699291,13cyclictest0-21swapper/311:08:153
2551699291,13cyclictest0-21swapper/310:52:473
2551699291,12cyclictest0-21swapper/312:36:353
2551699291,12cyclictest0-21swapper/311:12:353
2551699291,12cyclictest0-21swapper/309:25:113
2551699290,13cyclictest0-21swapper/311:57:313
2551699290,13cyclictest0-21swapper/309:18:193
2549999299,17cyclictest13468-21/usr/sbin/munin11:35:230
2549999292,22cyclictest535-21diskstats11:05:120
2549999291,25cyclictest1272-21gltestperf07:25:150
2549999290,6cyclictest0-21swapper/009:25:200
2549999290,4cyclictest4252-21timerwakeupswit07:30:260
2551699281,25cyclictest0-21swapper/307:13:433
2551699281,19cyclictest0-21swapper/308:18:293
2551699281,13cyclictest0-21swapper/307:39:153
2551699281,12cyclictest0-21swapper/307:27:193
2549999289,16cyclictest2048-21expr09:55:140
2549999288,17cyclictest10399-21fschecks_count09:00:150
2549999283,13cyclictest0-21swapper/009:30:480
2549999281,7cyclictest14206-21diskmemload10:28:000
2549999281,24cyclictest12640-21/usr/sbin/munin07:50:210
2549999281,23cyclictest9333-21unixbench_multi08:55:270
2551699271,12cyclictest0-21swapper/311:52:473
2551699271,11cyclictest0-21swapper/310:43:133
2551699271,10cyclictest0-21swapper/311:33:473
2551699270,25cyclictest0-21swapper/308:05:113
2549999271,3cyclictest4056-21/usr/sbin/munin10:00:190
2549999271,3cyclictest32029-21/usr/sbin/munin11:00:260
2549999271,3cyclictest30175-21/usr/sbin/munin09:45:150
2549999271,3cyclictest29531-21grep12:10:150
2549999271,3cyclictest21464-21/usr/sbin/munin08:10:210
2549999271,3cyclictest21464-21/usr/sbin/munin08:10:210
2549999271,2cyclictest5634-21/usr/sbin/munin08:50:130
2549999271,22cyclictest6262-21/usr/sbin/munin10:05:130
2551699267,3cyclictest0-21swapper/311:00:153
2551699261,9cyclictest0-21swapper/311:21:353
2551699261,17cyclictest0-21swapper/308:54:533
2551699261,13cyclictest31349-21cstates12:15:133
2551699261,12cyclictest0-21swapper/309:23:233
2551699261,12cyclictest0-21swapper/307:30:033
2551699261,11cyclictest0-21swapper/312:33:393
2551699261,11cyclictest0-21swapper/312:04:233
2551699261,10cyclictest15527-21latency_hist11:40:013
2551699260,12cyclictest0-21swapper/311:29:313
2551699251,17cyclictest0-21swapper/307:45:263
2551699251,11cyclictest0-21swapper/307:53:423
2551699251,10cyclictest0-21swapper/311:45:473
2551699250,12cyclictest0-21swapper/310:31:503
2551699250,11cyclictest0-21swapper/310:45:133
2550999257,10cyclictest0-21swapper/212:35:212
2550399251,17cyclictest0-21swapper/112:32:081
2551699241,9cyclictest0-21swapper/309:10:553
2551699241,12cyclictest0-21swapper/310:17:273
2551699240,11cyclictest0-21swapper/310:20:023
2550399242,14cyclictest7839-21df_inode12:35:141
25503992412,10cyclictest31672-21kworker/u8:010:01:271
2551699231,8cyclictest0-21swapper/311:18:313
2551699230,9cyclictest0-21swapper/309:44:493
2551699230,11cyclictest0-21swapper/307:55:403
2550999231,9cyclictest20507-21ntp_states10:35:212
2550399233,12cyclictest0-21swapper/111:03:101
2550399231,11cyclictest0-21swapper/111:42:241
2550399231,11cyclictest0-21swapper/111:42:241
2551699226,3cyclictest0-21swapper/308:48:133
2551699221,9cyclictest0-21swapper/312:05:553
2551699221,9cyclictest0-21swapper/309:36:593
2551699221,9cyclictest0-21swapper/308:37:453
2551699221,9cyclictest0-21swapper/307:17:453
2551699221,8cyclictest0-21swapper/312:10:463
2551699221,8cyclictest0-21swapper/309:45:033
2551699221,2cyclictest0-21swapper/307:40:393
2551699221,13cyclictest0-21swapper/309:55:473
2551699221,12cyclictest0-21swapper/310:36:543
2551699221,10cyclictest0-21swapper/308:04:103
2550999228,3cyclictest224862expr11:55:122
2550999225,10cyclictest2508-21smartctl08:40:242
2550999221,2cyclictest0-21swapper/211:39:332
2550399221,6cyclictest0-21swapper/111:30:441
25503992216,3cyclictest29538-21df08:30:131
25503992215,4cyclictest4245-21irqstats08:45:171
2550399221,3cyclictest29627-21cut09:45:021
2550399221,3cyclictest24266-21tr09:30:201
2550399221,14cyclictest0-21swapper/112:27:221
2550399221,12cyclictest10494-21ls10:15:011
2550399220,16cyclictest0-21swapper/110:20:501
2550399220,16cyclictest0-21swapper/107:42:501
262132210,2sleep20-21swapper/212:02:132
2551699216,5cyclictest0-21swapper/310:10:183
2551699215,3cyclictest0-21swapper/307:20:293
2551699213,2cyclictest87350irq/126-enp1s0-10:00:113
2551699211,9cyclictest0-21swapper/308:56:493
2551699211,13cyclictest0-21swapper/308:30:513
2551699211,10cyclictest0-21swapper/312:25:393
25509992115,3cyclictest2594-21dump-pmu-power07:30:002
2550399217,12cyclictest19709-21kworker/u8:212:18:531
2550399213,6cyclictest8564-1kworker/u9:110:37:511
2550399212,11cyclictest24817-21/usr/sbin/munin08:20:121
25503992115,3cyclictest16683-21hddtemp_smartct08:00:161
25503992112,3cyclictest0-21swapper/107:38:401
2550399211,18cyclictest0-21swapper/112:10:101
2550399211,18cyclictest0-21swapper/109:21:371
2550399211,15cyclictest0-21swapper/112:20:331
2550399210,5cyclictest16742-21kworker/u8:210:40:531
2550399210,19cyclictest0-21swapper/110:05:191
2550999201,6cyclictest8564-1kworker/u9:111:47:322
2550399206,13cyclictest23-21ksoftirqd/110:27:531
2550399205,7cyclictest21-21rcuc/109:15:101
2550399202,9cyclictest0-21swapper/111:50:091
2550399201,9cyclictest13574-21sh07:55:011
25503992016,2cyclictest26520-21open_inodes09:35:231
25503992013,2cyclictest0-21swapper/112:06:511
2550399201,2cyclictest87550irq/128-enp1s0-10:56:301
25503992012,3cyclictest0-21swapper/107:19:481
2550399201,15cyclictest0-21swapper/108:06:401
2550399201,14cyclictest0-21swapper/112:01:111
2550399201,14cyclictest0-21swapper/111:56:381
2550399201,14cyclictest0-21swapper/108:53:421
2550399201,14cyclictest0-21swapper/108:39:041
2550399201,14cyclictest0-21swapper/107:24:471
2550399201,11cyclictest25907-21expr07:10:121
2550399201,11cyclictest23861-21egrep10:45:121
2550399201,11cyclictest20744-21df08:10:131
2550399201,11cyclictest20744-21df08:10:131
2550399201,11cyclictest0-21swapper/109:14:071
2550399200,14cyclictest0-21swapper/108:25:051
2551699191,9cyclictest0-21swapper/312:20:373
2550999191,8cyclictest0-21swapper/211:02:082
2550999191,7cyclictest0-21swapper/211:50:432
25509991912,4cyclictest28293-21hddtemp_smartct09:40:162
2550999191,12cyclictest0-21swapper/211:11:382
2550399198,3cyclictest2931-21ntp_states09:55:231
2550399193,5cyclictest14206-21diskmemload11:12:591
2550399192,2cyclictest87550irq/128-enp1s0-11:05:181
2550399191,9cyclictest0-21swapper/111:29:421
2550399191,7cyclictest7953-21ntpq11:20:221
2550399191,2cyclictest0-21swapper/110:15:181
2550399191,17cyclictest0-21swapper/110:50:191
2550399191,16cyclictest0-21swapper/107:48:191
2550399191,14cyclictest0-21swapper/108:57:191
2550399191,14cyclictest0-21swapper/108:42:071
25503991911,3cyclictest0-21swapper/109:01:201
25503991911,3cyclictest0-21swapper/107:59:081
2550399191,12cyclictest0-21swapper/111:18:481
2550399191,10cyclictest23945-21grep08:15:241
244782191,3sleep20-21swapper/208:18:562
2550999186,2cyclictest32-21ksoftirqd/210:04:212
2550999185,3cyclictest87350irq/126-enp1s0-11:15:232
2550999181,8cyclictest32019-21/usr/sbin/munin09:50:112
2550999181,7cyclictest0-21swapper/210:45:002
25509991812,3cyclictest5301-21/usr/sbin/munin12:30:202
25509991811,4cyclictest31457-21fschecks_count12:15:152
2550999181,11cyclictest0-21swapper/209:20:592
2550399181,9cyclictest32725-21idleruntime09:50:171
2550399181,4cyclictest18304-21ntp_offset10:30:211
25503991813,3cyclictest12237-21/usr/sbin/munin09:05:201
2550399181,2cyclictest0-21swapper/107:34:031
2550399181,16cyclictest0-21swapper/111:49:431
2550399181,16cyclictest0-21swapper/109:46:311
2550399181,16cyclictest0-21swapper/109:26:331
20852181,3sleep20-21swapper/211:07:592
204102180,4sleep230-21rcuc/208:10:122
204102180,4sleep230-21rcuc/208:10:122
173632180,3sleep230-21rcuc/208:00:252
82062171,2sleep20-21swapper/210:09:512
311082171,2sleep20-21swapper/208:34:212
2550999179,5cyclictest910-21systemd-logind09:05:222
2550999176,2cyclictest32-21ksoftirqd/211:26:452
2550999173,4cyclictest13576-21sh07:55:012
2550999171,7cyclictest32214-21irqstats08:35:192
2550999171,7cyclictest0-21swapper/207:55:212
2550999171,6cyclictest14323-21grep10:20:262
2550999171,6cyclictest11611-21df11:30:132
2550999171,6cyclictest0-21swapper/210:47:452
2550999171,6cyclictest0-21swapper/210:33:372
25509991714,2cyclictest32-21ksoftirqd/209:30:172
2550999171,3cyclictest0-21swapper/207:32:282
25509991713,3cyclictest32-21ksoftirqd/208:05:242
2550999171,2cyclictest0-21swapper/210:12:382
2550999171,15cyclictest0-21swapper/209:49:252
2550999171,15cyclictest0-21swapper/208:58:412
2550999170,9cyclictest30553-21sh07:20:012
2550999169,4cyclictest0-21swapper/208:25:152
2550999162,3cyclictest22171-21pmu-power09:25:232
2550999161,6cyclictest6280-21smart_sda07:35:242
2550999161,6cyclictest3430-21taskset08:45:062
2550999161,6cyclictest30017-21grep12:10:222
2550999161,6cyclictest0-21swapper/209:10:212
2550999161,5cyclictest26624-21perl10:50:162
2550999161,2cyclictest0-21swapper/209:38:222
25509991611,3cyclictest6878-21/usr/sbin/munin11:20:162
25509991611,3cyclictest24801-21/usr/sbin/munin08:20:262
25509991611,3cyclictest17036-21cut09:15:142
25509991611,3cyclictest10033-21/usr/sbin/munin09:00:132
2550999161,10cyclictest0-21swapper/207:43:552
2550999160,10cyclictest0-21swapper/207:10:022
2550999155,3cyclictest2110-21cstates09:55:142
2550999151,5cyclictest10486-21ntp_states07:45:232
2550999151,3cyclictest10765-21cpuspeed_turbos10:15:132
25509991511,3cyclictest0-21swapper/207:20:462
2550999150,13cyclictest0-21swapper/212:05:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional