You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-02 - 19:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Thu Apr 02, 2026 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
123672240164,24sleep30-21swapper/307:06:483
124752238166,24sleep20-21swapper/207:08:122
123122228155,60sleep00-21swapper/007:06:040
123552203165,25sleep10-21swapper/107:06:381
1703121780,4sleep01275899cyclictest08:50:230
2838221560,5sleep21277099cyclictest09:15:252
3205021480,3sleep01275899cyclictest09:25:180
2666321460,4sleep01275899cyclictest11:40:180
1428021450,4sleep01275899cyclictest11:10:570
127709910990,18cyclictest0-21swapper/208:41:002
1277099105102,2cyclictest0-21swapper/209:38:422
12770998582,2cyclictest0-21swapper/212:19:332
12770996562,2cyclictest0-21swapper/207:51:232
28302570,2sleep10-21swapper/110:45:211
277272520,1sleep30-21swapper/308:00:013
1277099403,23cyclictest14366-40dnf11:12:052
12774993611,4cyclictest0-21swapper/311:00:153
1277499352,8cyclictest0-21swapper/311:12:053
1277099351,23cyclictest5623-21grep12:05:242
1277499341,11cyclictest22604-21kworker/u8:110:45:133
12770993410,20cyclictest16117-21systemd07:18:152
1277499332,5cyclictest13563-21sort07:15:023
12774993311,19cyclictest8681-21sh09:45:203
1275899338,12cyclictest24373-21diskmemload12:30:010
1275899335,13cyclictest24373-21diskmemload11:25:010
1277499328,3cyclictest0-21swapper/312:31:153
1277499321,5cyclictest27060-21expr09:15:123
1277499321,3cyclictest6335-21memory09:40:173
12774993212,17cyclictest30255-21smart_sda09:20:233
1277099320,4cyclictest0-21swapper/208:45:162
1277499317,3cyclictest16201-21awk10:05:003
1277499313,26cyclictest87350irq/126-enp1s0-10:11:113
1277499311,3cyclictest6029-21/usr/sbin/munin10:55:163
12774993111,3cyclictest4501-21wc09:35:233
1277099318,20cyclictest29277-21smart_sda11:45:232
1277099318,19cyclictest19051-21ntpq08:55:192
1277099314,23cyclictest26527-21/usr/sbin/munin11:40:172
1277099312,25cyclictest1383-21crond10:45:012
1277099311,5cyclictest0-21swapper/212:22:112
1277099311,27cyclictest27525-21latency_hist08:00:012
1277099311,18cyclictest2311-21hddtemp_smartct10:45:152
1275899312,13cyclictest27187-21awk10:30:010
1275899312,13cyclictest27187-21awk10:30:010
1277499309,3cyclictest23667-21gltestperf10:20:153
1277499302,6cyclictest15577-21munin-run08:50:013
1277499302,5cyclictest20869-21hddtemp_smartct09:00:163
1277499302,3cyclictest87350irq/126-enp1s0-11:05:193
1277499301,6cyclictest2993-21latency_hist09:35:013
12774993013,14cyclictest5096-21smart_sda10:50:233
12774993013,14cyclictest13246-21/usr/sbin/munin12:25:173
12774993010,3cyclictest16286-21latency_hist07:20:023
12774993010,2cyclictest2266-21/usr/sbin/munin12:00:113
1277499300,11cyclictest0-21swapper/308:20:133
1277099307,20cyclictest307-21/usr/sbin/munin08:15:162
1277099307,20cyclictest21426-21/usr/sbin/munin11:30:152
1277099305,3cyclictest32618-21wc09:25:252
1277099303,24cyclictest0-21swapper/207:35:112
1277099301,5cyclictest0-21swapper/211:26:002
1277099301,4cyclictest30559-21df_inode11:50:122
1277099301,4cyclictest16499-21/usr/sbin/munin12:30:202
1277099301,4cyclictest14379-21/usr/sbin/munin10:00:112
1277099301,3cyclictest17757-21/usr/sbin/munin11:20:162
12770993011,16cyclictest30153-21tune2fs10:35:142
12770993010,17cyclictest24362-21sh09:10:012
1276599302,3cyclictest28070-21switchtime09:15:231
1275899306,11cyclictest11178-21munin-run08:40:010
1275899301,5cyclictest0-21swapper/010:05:250
1277499299,3cyclictest7342-21expr08:30:133
1277499299,3cyclictest31338-21cpuspeed_turbos09:25:123
1277499298,4cyclictest17034-21grep08:50:233
1277499298,3cyclictest6780-21latency_hist08:30:013
1277499298,3cyclictest30106-21seq08:07:003
1277499298,3cyclictest30106-21seq08:07:003
1277499297,3cyclictest30479-21latency10:35:173
1277499292,9cyclictest26120-21unixbench_multi09:10:273
1277499292,5cyclictest1394-21munin-run10:45:013
1277499291,5cyclictest29980-21idleruntime-cro11:50:013
1277499291,4cyclictest87350irq/126-enp1s0-10:05:123
1277499291,4cyclictest12006-21/usr/sbin/munin09:55:123
12774992912,14cyclictest673-21sh08:15:123
12774992912,14cyclictest19935-21/usr/sbin/munin07:35:103
12774992911,3cyclictest827-21runrttasks07:31:003
1277499291,12cyclictest25208-21/usr/sbin/munin10:25:213
1277499291,12cyclictest25208-21/usr/sbin/munin10:25:213
1277499291,12cyclictest20879-21sshd11:26:023
12774992910,3cyclictest827-21runrttasks08:01:143
12774992910,3cyclictest8027-21sort12:10:223
12774992910,3cyclictest17641-21/usr/sbin/munin12:35:223
1277499290,3cyclictest9032-21cstates12:15:103
1277099299,17cyclictest7289-21grep12:10:132
1277099299,17cyclictest1-21systemd07:15:012
1277099298,18cyclictest24816-21sshd10:23:012
1277099297,19cyclictest18381-21latency_hist10:10:012
1277099296,20cyclictest23625-21cat07:45:142
1277099292,4cyclictest5153-21df_inode08:25:132
1277099292,4cyclictest19128-21/usr/sbin/munin12:35:252
1277099291,5cyclictest0-21swapper/211:00:262
1277099291,5cyclictest0-21swapper/210:50:242
1277099291,4cyclictest2925-21hddtemp_smartct12:00:152
1277099291,4cyclictest28316-21memory10:30:172
1277099291,4cyclictest20870-21chrt09:00:162
1277099291,3cyclictest18615-21/usr/sbin/munin10:10:152
1277099291,19cyclictest11399-21users09:50:252
1277499289,2cyclictest827-21runrttasks12:22:153
1277499287,3cyclictest0-21swapper/309:06:143
1277499286,5cyclictest1373-21timerandwakeup11:55:253
1277499284,5cyclictest0-21swapper/309:50:143
1277499282,5cyclictest25595-21munin-run11:40:013
1277499282,5cyclictest25595-21munin-run11:40:013
1277499282,5cyclictest11747-21diskstats08:40:133
1277499282,23cyclictest5048-21gltestperf12:05:173
1277499281,8cyclictest15017-21cstates11:15:123
12774992811,14cyclictest27953-21date11:45:013
12774992810,3cyclictest31414-21processes11:50:213
12774992810,3cyclictest17996-21/usr/sbin/munin08:55:113
1277499280,14cyclictest0-21swapper/308:11:453
1277099288,17cyclictest827-21runrttasks07:31:532
1277099288,17cyclictest6029-21/usr/sbin/munin10:55:132
1277099288,17cyclictest3120-21cat09:35:012
1277099287,18cyclictest766-21ntpd08:10:202
1277099282,4cyclictest25786-21perl10:25:152
1277099282,4cyclictest25786-21perl10:25:152
1277099282,3cyclictest28724-21diskstats08:05:092
1277099282,3cyclictest28724-21diskstats08:05:092
1277099282,3cyclictest2752-21grep08:20:112
1277099282,3cyclictest25830-21/usr/sbin/munin09:10:242
1277099281,4cyclictest9548-21diskstats08:35:132
1277099281,4cyclictest536-21/usr/sbin/munin11:55:162
1277099281,4cyclictest29345-21entropy09:20:132
1277099281,4cyclictest13922-21hddtemp_smartct12:25:142
1277099281,3cyclictest8429-21/usr/sbin/munin09:45:172
1277099281,3cyclictest8109-21open_inodes08:30:212
1277099281,3cyclictest20813-21/usr/sbin/munin10:15:162
1277099281,3cyclictest19207-21ls07:30:022
1277099281,3cyclictest11462-21/usr/sbin/munin11:05:202
1277099281,3cyclictest0-21swapper/211:35:232
1277099281,3cyclictest0-21swapper/211:35:232
12770992810,15cyclictest827-21runrttasks09:41:302
1276599283,6cyclictest0-21swapper/112:23:191
1277499279,3cyclictest27522-21grep10:30:103
1277499279,3cyclictest21872-21ntpq10:15:203
1277499278,3cyclictest18171-21ntpq07:25:163
1277499278,3cyclictest0-21swapper/307:45:203
1277499277,3cyclictest22406-21sed11:30:193
1277499277,3cyclictest18079-21ntp_kernel_pll_11:20:193
1277499277,2cyclictest827-21runrttasks07:44:163
1277099272,4cyclictest12353-21diskstats09:55:122
1277099271,4cyclictest15818-21egrep08:50:122
1277099271,3cyclictest15714-21wc11:15:172
1275899279,9cyclictest0-21swapper/009:40:240
1275899272,6cyclictest27607-21irqstats09:15:170
1277499268,3cyclictest9614-21fschecks_count08:35:143
1276599262,4cyclictest32144-21diskstats10:40:131
1276599261,8cyclictest0-21swapper/110:23:471
1275899260,12cyclictest25606-21perf11:40:010
1275899260,12cyclictest25606-21perf11:40:010
1277499259,3cyclictest0-21swapper/307:20:433
1277499257,3cyclictest251482chrt07:52:473
1277099251,3cyclictest22624-21wc07:45:012
1277099251,3cyclictest16568-21sshd07:20:492
1276599251,8cyclictest18763-21kworker/u8:109:57:281
1276599251,12cyclictest0-21swapper/112:31:521
12765992511,2cyclictest0-21swapper/110:15:211
12765992510,5cyclictest14366-40dnf11:12:051
1275899258,14cyclictest23410-21idleruntime-cro11:35:010
1275899252,6cyclictest3616-21perf10:50:010
1275899252,11cyclictest5204-21awk09:40:010
1275899251,5cyclictest0-21swapper/009:48:520
1277099241,4cyclictest27807-21sshd08:00:502
1276599241,6cyclictest0-21swapper/109:13:311
1276599241,5cyclictest0-21swapper/111:35:061
1276599241,5cyclictest0-21swapper/111:35:061
1276599241,3cyclictest14235-21strings12:25:181
12765992410,11cyclictest16432-21vmstat11:15:251
1276599240,8cyclictest3545-21kworker/u8:210:58:041
1275899241,7cyclictest21876-21ntp_states10:15:210
1276599231,4cyclictest0-21swapper/109:42:331
1276599231,3cyclictest11690-21chrt09:54:001
1276599231,16cyclictest0-21swapper/108:54:131
1276599231,16cyclictest0-21swapper/107:22:071
1276599231,11cyclictest14603-21grep10:00:131
1276599231,11cyclictest0-21swapper/111:07:361
1276599231,11cyclictest0-21swapper/110:09:461
1276599230,7cyclictest0-21swapper/112:39:331
1276599230,17cyclictest0-21swapper/109:32:461
1275899238,13cyclictest23348-21grep07:45:110
1275899231,5cyclictest0-21swapper/012:38:320
1275899231,11cyclictest29380-21munin-run10:35:010
1276599227,3cyclictest0-21swapper/111:24:131
1276599225,9cyclictest0-21swapper/111:50:161
12765992213,8cyclictest0-21swapper/112:13:441
12765992212,3cyclictest87550irq/128-enp1s0-10:13:131
1276599221,13cyclictest8971-21latency_hist08:35:011
1276599221,10cyclictest0-21swapper/111:03:071
1276599220,6cyclictest0-21swapper/109:26:531
1276599220,3cyclictest0-21swapper/111:59:341
1276599220,15cyclictest28425-21kworker/u8:209:35:531
1275899225,15cyclictest16206-21basename10:05:000
1275899224,9cyclictest3546-21sed12:00:240
1275899221,11cyclictest17128-21/usr/sbin/munin07:25:210
1275899221,11cyclictest15445-21/usr/sbin/munin12:30:220
1275899221,11cyclictest12006-21/usr/sbin/munin09:55:190
256302211,16sleep025664-21ntp_states09:10:220
1276599214,2cyclictest87650irq/129-enp1s0-07:35:141
1276599213,3cyclictest87550irq/128-enp1s0-08:55:251
1276599211,3cyclictest9227-21sshd09:45:341
1276599211,18cyclictest0-21swapper/109:05:141
1276599211,15cyclictest0-21swapper/111:46:571
1276599211,15cyclictest0-21swapper/111:26:511
1276599211,15cyclictest0-21swapper/110:26:091
1276599211,15cyclictest0-21swapper/110:26:091
1276599211,15cyclictest0-21swapper/107:44:511
1276599211,14cyclictest0-21swapper/108:27:321
1276599211,11cyclictest26619-21ntp_states07:55:161
1276599210,19cyclictest0-21swapper/111:42:421
1276599210,16cyclictest0-21swapper/108:18:561
1275899211,6cyclictest0-21swapper/007:55:110
1275899211,5cyclictest0-21swapper/010:40:010
1275899211,11cyclictest24301-21switchtime10:20:230
1276599206,6cyclictest0-21swapper/108:01:201
1276599206,3cyclictest22604-21kworker/u8:110:50:151
12765992013,4cyclictest9810-21grep08:35:151
1276599201,14cyclictest0-21swapper/107:46:031
1276599201,14cyclictest0-21swapper/107:29:011
1276599201,11cyclictest5369-21memory12:05:201
1276599201,11cyclictest28140-21cat10:30:151
1276599201,11cyclictest21163-21cat09:00:191
1276599201,10cyclictest0-21swapper/112:04:361
1276599200,15cyclictest0-21swapper/108:42:061
1276599200,15cyclictest0-21swapper/107:13:521
1275899209,3cyclictest808-21snmpd08:48:010
1275899206,3cyclictest9218-21strings11:00:200
1275899203,9cyclictest0-21swapper/008:24:010
1275899202,6cyclictest0-21swapper/009:53:290
1275899201,11cyclictest4455-21/usr/sbin/munin12:05:270
1275899201,11cyclictest30217-21/usr/sbin/munin11:50:240
1276599199,2cyclictest25329-21latency_hist07:55:011
1276599194,9cyclictest22604-21kworker/u8:110:36:281
1276599194,3cyclictest0-21swapper/112:16:231
12765991916,2cyclictest0-21swapper/107:30:051
1276599191,4cyclictest21772-21diskstats11:30:141
1276599191,2cyclictest87550irq/128-enp1s0-09:21:201
1276599191,14cyclictest0-21swapper/108:45:321
1276599191,14cyclictest0-21swapper/108:11:091
1276599191,10cyclictest2952-21sh08:20:121
1276599190,17cyclictest0-21swapper/107:15:031
1276599190,14cyclictest0-21swapper/108:06:531
1276599190,14cyclictest0-21swapper/108:06:531
1275899199,3cyclictest87550irq/128-enp1s0-09:24:280
1275899195,8cyclictest0-21swapper/007:37:270
1275899194,7cyclictest11-21rcuc/010:50:140
1275899193,4cyclictest200350kworker/u8:112:10:050
1275899192,6cyclictest405-1kworker/u9:009:10:010
1275899192,4cyclictest0-21swapper/011:05:280
1275899191,10cyclictest8844-21/usr/sbin/munin12:15:120
1275899191,10cyclictest0-21swapper/010:55:260
1275899190,9cyclictest12391-21needreboot08:40:190
1275899190,9cyclictest0-21swapper/011:15:100
1275899190,5cyclictest405-1kworker/u9:009:30:330
1275899188,3cyclictest1856-21sshd11:58:020
1275899185,7cyclictest0-21swapper/007:30:310
1275899183,9cyclictest909-21sssd_nss07:15:010
1275899181,5cyclictest0-21swapper/010:40:250
1275899181,5cyclictest0-21swapper/010:10:260
1275899181,4cyclictest0-21swapper/012:23:510
1275899181,4cyclictest0-21swapper/008:15:010
1275899180,5cyclictest405-1kworker/u9:011:29:440
1275899174,3cyclictest773-21NetworkManager08:01:200
1275899173,3cyclictest87350irq/126-enp1s0-07:55:010
1275899171,6cyclictest9-21rcu_preempt08:15:100
1275899171,6cyclictest29039-21netstat11:45:190
1275899171,14cyclictest22167-21sh09:05:010
12758991711,3cyclictest28741-21fschecks_count08:05:100
12758991711,3cyclictest28741-21fschecks_count08:05:100
1275899162,6cyclictest17996-21/usr/sbin/munin08:55:250
12758991611,3cyclictest7004-21seq08:30:080
12758991611,3cyclictest6444-21seq08:25:510
1275899152,5cyclictest839-21lldpd07:40:060
12758991510,3cyclictest16673-21seq07:22:100
12758991510,3cyclictest15756-21unixbench_multi07:15:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional