You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-07 - 21:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Dec 07, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
72602239163,24sleep20-21swapper/207:07:202
72442238163,24sleep30-21swapper/307:07:083
71582235159,24sleep00-21swapper/007:06:000
73092204170,22sleep10-21swapper/107:07:581
38172590,1sleep00-21swapper/010:40:130
175892520,3sleep230-21rcuc/211:10:172
7559993519,6cyclictest0-21swapper/010:06:340
7565993420,4cyclictest10346-21unixbench_singl12:05:271
7570993128,2cyclictest0-21swapper/211:41:062
756599311,8cyclictest0-21swapper/110:09:051
7565993118,9cyclictest0-21swapper/107:20:151
7565993023,2cyclictest0-21swapper/111:00:261
7565993016,9cyclictest0-21swapper/110:19:071
755999303,21cyclictest0-21swapper/011:49:190
756599299,2cyclictest0-21swapper/110:00:271
7565992916,8cyclictest0-21swapper/110:42:501
7565992915,9cyclictest0-21swapper/108:48:171
7565992915,9cyclictest0-21swapper/108:48:161
7565992914,9cyclictest0-21swapper/107:44:421
7565992914,9cyclictest0-21swapper/107:44:421
756599291,26cyclictest0-21swapper/111:50:291
7559992917,2cyclictest0-21swapper/009:40:170
755999291,20cyclictest0-21swapper/010:36:330
164312291,11sleep10-21swapper/112:20:221
95752281,2sleep29576-21sshd07:13:142
757099287,2cyclictest29674-21kworker/2:012:19:252
756599283,3cyclictest836-21runrttasks09:02:431
7565992815,8cyclictest0-21swapper/112:36:141
7565992815,8cyclictest0-21swapper/112:05:001
7565992815,8cyclictest0-21swapper/109:34:411
7565992815,8cyclictest0-21swapper/108:36:361
7565992814,9cyclictest0-21swapper/111:32:361
7565992814,9cyclictest0-21swapper/107:15:171
756599281,24cyclictest26037-21/usr/sbin/munin07:50:151
756599281,24cyclictest25518-21awk11:30:021
755999280,6cyclictest0-21swapper/010:32:310
23502280,4sleep02352-21timerwakeupswit09:20:280
757699272,8cyclictest29221-21df_inode09:10:143
757099272,14cyclictest0-21swapper/211:20:192
756599274,3cyclictest0-21swapper/111:48:451
7565992721,3cyclictest14634-21snmpd09:59:131
7565992716,6cyclictest0-21swapper/107:13:401
7565992714,8cyclictest0-21swapper/112:32:371
7565992714,8cyclictest0-21swapper/112:16:401
7565992714,8cyclictest0-21swapper/107:48:391
7565992714,8cyclictest0-21swapper/107:39:321
756599271,3cyclictest18917-21perf11:15:011
7565992713,8cyclictest0-21swapper/108:05:201
7565992712,4cyclictest1980-21ntp_states09:20:241
756599271,20cyclictest0-21swapper/110:28:001
756599271,12cyclictest10106-21munin-run10:55:011
755999271,9cyclictest0-21swapper/009:13:180
755999271,3cyclictest22599-21sensors11:20:250
7559992712,8cyclictest14634-21snmpd11:39:300
755999271,20cyclictest0-21swapper/010:20:300
755999271,19cyclictest0-21swapper/011:54:460
757699266,9cyclictest41-21ksoftirqd/311:20:023
757699265,6cyclictest848-21sssd_nss11:40:183
757699260,13cyclictest4075-21kworker/u8:011:55:163
757099265,5cyclictest29674-21kworker/2:011:46:252
7570992611,9cyclictest0-21swapper/209:44:312
756599262,21cyclictest3786-21hddtemp_smartct09:25:171
7565992615,7cyclictest0-21swapper/109:53:261
7565992615,6cyclictest0-21swapper/108:41:521
756599261,4cyclictest0-21swapper/108:28:571
7565992614,8cyclictest0-21swapper/108:24:181
7565992614,8cyclictest0-21swapper/108:00:121
7565992614,7cyclictest0-21swapper/111:55:251
7565992614,7cyclictest0-21swapper/111:15:291
756599261,22cyclictest2482-21/usr/sbin/munin08:10:151
7565992611,4cyclictest6261-21df10:45:131
755999267,5cyclictest0-21swapper/012:11:080
755999264,18cyclictest0-21swapper/012:24:050
755999263,2cyclictest0-21swapper/009:35:280
7559992618,2cyclictest28644-21diskmemload10:49:210
7559992618,1cyclictest0-21swapper/007:45:190
755999261,5cyclictest7-21ksoftirqd/012:36:180
755999261,19cyclictest0-21swapper/011:41:160
755999261,10cyclictest0-21swapper/012:33:440
755999260,5cyclictest0-21swapper/011:33:550
757699254,11cyclictest26085-21diskstats11:30:163
757699253,3cyclictest0-21swapper/310:50:153
757099259,3cyclictest22552-1kworker/u9:109:23:312
757099251,13cyclictest29951-21ntp_states09:10:232
7570992511,3cyclictest0-21swapper/208:40:152
757099251,11cyclictest0-21swapper/210:30:202
7565992517,3cyclictest0-21swapper/109:07:111
7565992515,6cyclictest0-21swapper/111:35:201
7565992515,3cyclictest0-21swapper/111:08:181
7565992514,7cyclictest0-21swapper/112:25:061
7565992514,7cyclictest0-21swapper/110:32:301
7565992514,6cyclictest0-21swapper/109:43:391
7565992513,3cyclictest0-21swapper/109:45:051
7565992513,3cyclictest0-21swapper/109:36:011
7565992513,3cyclictest0-21swapper/109:17:201
7565992512,7cyclictest0-21swapper/107:55:571
7565992512,2cyclictest0-21swapper/110:10:201
756599251,21cyclictest1885-21diskstats10:35:161
7565992511,8cyclictest0-21swapper/108:58:261
7565992511,8cyclictest0-21swapper/108:30:071
7565992511,4cyclictest32095-21latency_hist11:45:021
7565992511,3cyclictest0-21swapper/111:20:121
7565992511,3cyclictest0-21swapper/108:15:251
755999256,3cyclictest0-21swapper/010:55:260
755999256,3cyclictest0-21swapper/010:55:260
755999254,9cyclictest0-21swapper/012:03:140
755999252,18cyclictest0-21swapper/011:03:570
7559992517,3cyclictest0-21swapper/008:33:570
755999251,5cyclictest0-21swapper/011:11:030
755999251,4cyclictest0-21swapper/012:25:200
755999251,4cyclictest0-21swapper/012:05:240
755999251,4cyclictest0-21swapper/011:59:580
755999251,4cyclictest0-21swapper/009:27:430
755999251,3cyclictest0-21swapper/010:50:480
755999251,3cyclictest0-21swapper/009:31:560
755999251,17cyclictest0-21swapper/011:29:080
755999251,17cyclictest0-21swapper/011:16:210
757699247,2cyclictest8817-21ntpq07:10:223
757699247,11cyclictest0-21swapper/310:05:233
757699245,8cyclictest1282-21fschecks_count09:20:173
757699245,6cyclictest0-21swapper/310:20:273
757699243,6cyclictest0-21swapper/308:51:473
757699241,12cyclictest30773-21smart_sda10:25:283
7576992410,8cyclictest0-21swapper/308:55:233
757099247,4cyclictest32-21ksoftirqd/211:40:012
757099244,5cyclictest0-21swapper/212:30:152
757099241,9cyclictest0-21swapper/212:12:132
7570992411,8cyclictest0-21swapper/212:22:452
757099241,17cyclictest0-21swapper/209:32:232
757099241,13cyclictest0-21swapper/210:47:062
757099241,10cyclictest19350-21chrt12:27:442
7565992420,3cyclictest0-21swapper/108:52:071
7565992414,6cyclictest0-21swapper/112:13:431
756599241,3cyclictest22336-21kworker/u8:210:59:061
756599241,3cyclictest22336-21kworker/u8:210:59:061
7565992413,3cyclictest0-21swapper/107:32:001
756599241,21cyclictest0-21swapper/110:21:311
756599241,20cyclictest0-21swapper/109:10:161
756599241,11cyclictest16775-21taskset07:27:041
755999244,3cyclictest87950irq/132-enp2s0-12:17:000
755999242,10cyclictest16301-21/usr/sbin/munin09:55:170
755999241,8cyclictest0-21swapper/010:26:040
755999241,7cyclictest0-21swapper/009:50:200
7559992417,2cyclictest0-21swapper/008:00:260
7559992416,3cyclictest0-21swapper/008:17:280
755999241,5cyclictest0-21swapper/010:12:490
755999241,4cyclictest0-21swapper/009:15:560
755999241,21cyclictest22336-21kworker/u8:211:05:130
755999241,18cyclictest0-21swapper/010:03:050
755999241,11cyclictest17872-21/usr/sbin/munin08:45:170
755999241,11cyclictest17872-21/usr/sbin/munin08:45:170
7559992410,3cyclictest7761-21/usr/sbin/munin07:10:150
7559992410,3cyclictest13476-21/usr/sbin/munin08:35:130
757699238,10cyclictest0-21swapper/312:10:233
757699234,2cyclictest784-21NetworkManager11:10:333
757699232,11cyclictest9-21rcu_preempt12:07:313
757699231,5cyclictest0-21swapper/310:31:153
7576992312,3cyclictest0-21swapper/311:25:173
757699231,12cyclictest22311-21diskstats12:35:143
757699231,10cyclictest0-21swapper/310:59:143
757699231,10cyclictest0-21swapper/310:59:143
7576992310,3cyclictest14117-21users11:00:273
757099235,4cyclictest22552-1kworker/u9:111:25:202
757099233,11cyclictest17479-21expr07:30:132
757099232,9cyclictest0-21swapper/209:45:582
757099232,4cyclictest0-21swapper/211:00:142
757099232,12cyclictest19688-21sshd08:48:142
757099232,12cyclictest19688-21sshd08:48:142
757099232,11cyclictest14736-21/usr/sbin/munin11:05:132
757099231,6cyclictest16910-21grep09:55:172
757099231,14cyclictest0-21swapper/210:19:352
757099231,13cyclictest0-21swapper/212:02:212
757099231,13cyclictest0-21swapper/210:52:052
757099231,13cyclictest0-21swapper/209:51:242
757099231,12cyclictest0-21swapper/210:05:332
7570992310,7cyclictest0-21swapper/207:36:202
755999231,4cyclictest0-21swapper/010:17:030
757699229,3cyclictest41-21ksoftirqd/308:05:263
757699227,8cyclictest0-21swapper/309:45:183
757699225,8cyclictest86850irq/129-enp1s0-11:09:383
757699225,10cyclictest779-21ntpd08:20:593
757699224,8cyclictest39-21rcuc/309:15:173
757699223,9cyclictest7500-21/usr/sbin/munin09:35:173
757699222,7cyclictest14441-21df_inode09:50:143
757699221,13cyclictest0-21swapper/310:43:353
757699221,11cyclictest3645-21grep11:50:283
757699221,10cyclictest0-21swapper/308:32:143
757699220,10cyclictest0-21swapper/312:20:223
757099225,9cyclictest86550irq/126-enp1s0-08:10:142
757099225,9cyclictest32-21ksoftirqd/207:51:192
757099221,6cyclictest27761-21latency_hist11:35:012
757099221,6cyclictest0-21swapper/209:16:552
757099221,5cyclictest0-21swapper/208:55:142
757099221,4cyclictest0-21swapper/211:17:342
757099221,15cyclictest0-21swapper/209:38:212
757099221,12cyclictest5033-21hddtemp_smartct11:55:162
757099221,12cyclictest4066-21df10:40:132
757099221,12cyclictest0-21swapper/210:57:482
757099221,12cyclictest0-21swapper/210:57:482
7570992210,7cyclictest0-21swapper/208:52:002
7570992210,7cyclictest0-21swapper/207:59:472
755999229,2cyclictest28234-21/usr/sbin/munin07:55:220
755999221,2cyclictest0-21swapper/009:45:180
755999221,10cyclictest14634-21snmpd07:53:490
259372220,4sleep025941-21unixbench-2d09:00:250
757699218,8cyclictest0-21swapper/310:15:253
757699217,8cyclictest5928-21/usr/sbin/munin10:45:263
757699215,8cyclictest39-21rcuc/311:20:193
757699215,4cyclictest17026-21awk07:30:003
757699214,8cyclictest1516-21/usr/sbin/munin10:35:193
757699214,4cyclictest41-21ksoftirqd/309:40:163
757699212,8cyclictest0-21swapper/308:28:233
7576992117,3cyclictest0-21swapper/309:00:143
757699211,6cyclictest17234-21sed09:55:213
7576992113,3cyclictest0-21swapper/307:20:273
757699211,11cyclictest23822-21memory10:10:183
757699211,11cyclictest17921-21hddtemp_smartct07:30:163
7576992110,4cyclictest17568-21/usr/sbin/munin12:25:183
7576992110,3cyclictest0-21swapper/312:30:303
757699210,9cyclictest0-21swapper/312:15:143
757099218,3cyclictest86850irq/129-enp1s0-09:25:232
757099217,9cyclictest0-21swapper/207:48:292
757099214,4cyclictest0-21swapper/210:35:252
757099212,9cyclictest0-21swapper/209:03:372
757099211,7cyclictest28337-40dnf09:06:142
757099211,4cyclictest0-21swapper/210:00:252
7570992113,6cyclictest22880-21/usr/sbin/munin12:35:192
757099211,13cyclictest0-21swapper/210:28:012
757099211,10cyclictest0-21swapper/210:10:172
755999219,3cyclictest14634-21snmpd07:24:530
755999214,3cyclictest10583-21unixbench_multi08:25:290
755999213,16cyclictest86550irq/126-enp1s0-08:10:160
755999211,10cyclictest0-21swapper/007:25:130
755999210,9cyclictest0-21swapper/009:05:580
757699201,6cyclictest39-21rcuc/309:30:263
757699201,6cyclictest26371-21df07:50:133
757699201,6cyclictest16014-21df08:40:153
757699201,4cyclictest0-21swapper/308:47:143
757699201,4cyclictest0-21swapper/308:47:143
7576992014,3cyclictest673-21if_err_enp2s011:45:183
757699201,11cyclictest19171-21hddtemp_smartct10:00:173
757699200,15cyclictest0-21swapper/309:26:253
757099209,3cyclictest0-21swapper/208:31:302
757099202,9cyclictest9688-21memory12:05:192
757099201,6cyclictest32023-21users08:00:272
757099201,6cyclictest0-21swapper/207:20:212
757099201,10cyclictest28568-21smart_sda10:20:262
757099201,10cyclictest13826-21diskstats08:35:152
755999206,3cyclictest0-21swapper/008:05:220
755999201,6cyclictest0-21swapper/007:40:190
755999201,6cyclictest0-21swapper/007:40:190
755999201,5cyclictest17339-21seq08:41:140
755999201,5cyclictest10413-21ntp_states07:15:040
755999200,3cyclictest86550irq/126-enp1s0-08:50:190
79442190,5sleep2757099cyclictest08:20:252
757699198,2cyclictest0-21swapper/307:35:143
757699191,8cyclictest25580-21sshd07:47:133
757699191,6cyclictest3293-21irqstats08:10:193
7576991912,3cyclictest0-21swapper/311:35:303
757699191,11cyclictest0-21swapper/312:01:023
757699191,10cyclictest26670-21/usr/sbin/munin09:05:193
757699190,18cyclictest0-21swapper/307:40:253
757699190,18cyclictest0-21swapper/307:40:253
757099193,4cyclictest21990-21diskstats07:40:142
757099193,4cyclictest21990-21diskstats07:40:142
757099192,9cyclictest0-21swapper/211:52:492
757099191,6cyclictest15047-21/usr/sbin/munin07:25:122
757099191,5cyclictest0-21swapper/208:05:142
755999192,3cyclictest21418-21munin-run07:40:010
757699181,5cyclictest5896-21ps08:15:243
757699181,5cyclictest13476-21/usr/sbin/munin08:35:153
757699181,14cyclictest29791-21unixbench_singl07:55:263
757099182,14cyclictest86550irq/126-enp1s0-07:15:152
757099180,5cyclictest0-21swapper/208:25:202
755999181,7cyclictest0-21swapper/008:58:400
755999181,6cyclictest17245-21/usr/sbin/munin07:30:160
7576991712,3cyclictest12054-21timerwakeupswit07:15:273
7576991711,3cyclictest31083-21gltestperf08:00:163
7570991712,3cyclictest5148-21fschecks_time08:15:162
7559991711,3cyclictest8066-21grep08:20:270
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional