You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-04 - 16:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sat Apr 04, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
213322231162,56sleep00-21swapper/007:08:060
214362203166,24sleep20-21swapper/207:09:292
213402202166,24sleep30-21swapper/307:08:143
212132201163,25sleep10-21swapper/107:06:351
1749621710,4sleep12157199cyclictest08:10:191
21578997875,2cyclictest0-21swapper/209:05:372
72102720,2sleep20-21swapper/209:00:212
21578996763,3cyclictest0-21swapper/211:35:492
325612540,3sleep00-21swapper/012:25:220
68412530,5sleep339-21rcuc/307:45:253
228022470,3sleep121-21rcuc/112:05:131
259942440,1sleep20-21swapper/209:45:112
21578993936,2cyclictest0-21swapper/208:01:542
2156699389,13cyclictest5527-21munin-run12:40:010
21566993812,13cyclictest25040-21munin-run08:30:010
21578993734,2cyclictest0-21swapper/209:25:562
2156699375,29cyclictest30061-21idleruntime-cro09:55:010
21566993713,21cyclictest31573-21hddtemp_smartct11:10:150
2156699368,13cyclictest17483-21awk10:40:010
2156699365,28cyclictest4910-21munin-run11:25:010
2156699363,3cyclictest87350irq/126-enp1s0-11:45:190
193802360,4sleep02156699cyclictest11:55:200
2156699351,31cyclictest820-21munin-run07:35:010
2157199341,13cyclictest0-21swapper/108:25:231
21571993410,15cyclictest0-21swapper/109:20:011
21571993410,15cyclictest0-21swapper/109:20:011
21566993411,3cyclictest24924-21/usr/sbin/munin12:10:190
223032330,3sleep02156699cyclictest07:10:150
2156699333,11cyclictest19382-21tr09:30:110
21566993312,18cyclictest24345-21grep07:15:030
21566993311,19cyclictest21477-21/usr/sbin/munin09:35:230
21566993310,3cyclictest26496-21/usr/sbin/munin11:00:180
2156699329,3cyclictest9526-21/usr/sbin/munin11:35:230
2156699327,3cyclictest0-21swapper/012:33:360
2156699326,4cyclictest909-21sssd_nss09:10:230
2156699322,5cyclictest7973-21smart_sda10:15:230
2156699321,11cyclictest0-21swapper/010:05:020
21566993210,3cyclictest29003-21expr11:05:130
2156699319,3cyclictest7783-21cpuspeed_turbos07:50:130
2156699315,3cyclictest0-21swapper/009:16:040
2156699315,3cyclictest0-21swapper/009:16:040
2156699311,4cyclictest4512-21/usr/sbin/munin10:10:140
2156699311,3cyclictest1864-21if_err_enp1s007:35:170
2156699311,10cyclictest0-21swapper/010:50:040
2156699308,4cyclictest19331-21gltestperf08:15:160
2156699308,4cyclictest0-21swapper/010:20:140
2156699307,3cyclictest27930-21fschecks_time08:35:140
2156699306,6cyclictest0-21swapper/009:55:230
2156699304,4cyclictest0-21swapper/012:07:430
2156699302,24cyclictest3892-21/usr/sbin/munin08:55:270
2156699301,5cyclictest12408-21df08:00:140
2156699301,5cyclictest0-21swapper/009:23:550
21566993012,15cyclictest4318-21grep07:40:200
21566993012,15cyclictest32065-21expr08:45:130
21566993011,16cyclictest31625-21dump-pmu-power08:45:020
21566993010,4cyclictest0-21swapper/007:25:180
21566993010,3cyclictest882-21cat11:15:120
21566993010,3cyclictest8293-21/usr/sbin/munin09:05:250
21566993010,3cyclictest14982-21sshd08:05:170
21566993010,3cyclictest14982-21sshd08:05:170
21566993010,3cyclictest10260-21diskmemload11:50:230
273132290,4sleep339-21rcuc/308:35:013
2156699299,3cyclictest827-21runrttasks08:52:310
2156699298,3cyclictest22174-21cat12:01:010
2156699297,4cyclictest0-21swapper/009:25:210
2156699297,3cyclictest7191-21cat11:30:020
2156699297,2cyclictest13304-21/usr/sbin/munin10:30:210
2156699297,2cyclictest13304-21/usr/sbin/munin10:30:210
2156699292,5cyclictest28116-21smart_sda07:20:240
2156699291,3cyclictest87350irq/126-enp1s0-09:45:170
2157199284,14cyclictest0-21swapper/111:35:151
2156699289,3cyclictest11726-21/usr/sbin/munin11:40:120
2156699289,3cyclictest11726-21/usr/sbin/munin11:40:120
2156699287,3cyclictest7449-21latency_hist07:50:010
2156699284,3cyclictest87550irq/128-enp1s0-09:00:180
2156699284,3cyclictest11899-21if_err_enp2s010:25:160
2156699281,8cyclictest0-21swapper/012:18:330
2156699281,11cyclictest10260-21diskmemload10:41:150
21566992810,3cyclictest24416-21cut10:55:130
2156699280,9cyclictest0-21swapper/011:33:250
2156699280,9cyclictest0-21swapper/010:47:430
2156699280,6cyclictest0-21swapper/010:01:220
2156699280,2cyclictest0-21swapper/012:22:170
21582992711,3cyclictest22511-21perf12:05:013
2157199278,10cyclictest11923-21expr11:40:121
2157199278,10cyclictest11923-21expr11:40:121
2156699279,2cyclictest827-21runrttasks09:43:170
2156699278,2cyclictest827-21runrttasks07:55:360
2156699277,3cyclictest25402-21expr08:30:120
2156699276,3cyclictest0-21swapper/008:10:140
2158299266,10cyclictest0-21swapper/311:10:133
2157199251,12cyclictest0-21swapper/110:47:051
2157199251,11cyclictest0-21swapper/112:26:331
2157199251,11cyclictest0-21swapper/111:06:191
2157199248,4cyclictest3969-21kworker/u8:210:56:211
2157199241,12cyclictest0-21swapper/111:49:291
2157199241,11cyclictest0-21swapper/110:00:171
2157199241,11cyclictest0-21swapper/108:16:171
2156699241,4cyclictest21132-21expr08:20:130
2157899231,6cyclictest0-21swapper/212:36:132
21571992312,4cyclictest0-21swapper/108:00:141
2157199231,11cyclictest0-21swapper/112:02:421
2157199231,10cyclictest0-21swapper/111:59:381
2157199231,10cyclictest0-21swapper/111:13:451
2157199231,10cyclictest0-21swapper/110:50:251
2157199231,10cyclictest0-21swapper/110:32:571
2157199231,10cyclictest0-21swapper/110:32:571
2157199231,10cyclictest0-21swapper/109:58:091
2157199231,10cyclictest0-21swapper/109:36:341
2157199230,12cyclictest0-21swapper/109:05:131
2158299228,3cyclictest0-21swapper/311:55:533
2158299221,10cyclictest0-21swapper/311:22:193
2157899221,6cyclictest0-21swapper/212:33:042
2157199227,2cyclictest0-21swapper/109:50:211
2157199221,9cyclictest30993-40dnf07:29:531
2157199221,3cyclictest3672-21sleep08:55:001
2157199221,3cyclictest10207-21df07:55:131
2157199221,14cyclictest0-21swapper/112:24:131
2157199221,11cyclictest25729-21ntp_states07:15:221
2157199221,10cyclictest0-21swapper/111:26:291
2157199221,10cyclictest0-21swapper/111:23:251
2157199221,10cyclictest0-21swapper/111:17:411
2157899211,6cyclictest0-21swapper/210:57:452
2157899211,6cyclictest0-21swapper/209:50:582
2157899211,6cyclictest0-21swapper/209:38:062
2157899211,5cyclictest0-21swapper/211:11:272
2157899211,13cyclictest0-21swapper/211:59:502
2157199219,3cyclictest0-21swapper/111:32:531
2157199218,4cyclictest7572-21cat07:50:011
2157199218,11cyclictest4618-21kworker/u8:207:20:061
2157199216,3cyclictest0-21swapper/112:35:531
2157199215,4cyclictest0-21swapper/110:20:271
2157199215,3cyclictest87450irq/127-enp1s0-09:00:231
2157199211,9cyclictest0-21swapper/110:12:211
2157199211,7cyclictest0-21swapper/110:29:421
2157199211,18cyclictest0-21swapper/107:53:411
2157199211,15cyclictest0-21swapper/108:57:521
2157199211,11cyclictest28804-21timerandwakeup08:35:241
2157199211,11cyclictest26332-21ntp_states08:30:201
2157199211,10cyclictest16851-21dump-pmu-power09:25:011
2157199211,10cyclictest0-21swapper/111:01:051
2157199211,10cyclictest0-21swapper/110:16:251
2157199211,10cyclictest0-21swapper/110:09:091
21571992110,3cyclictest459-21ntp_states08:45:211
2158299205,2cyclictest39-21rcuc/309:45:013
2158299203,9cyclictest39-21rcuc/311:00:253
2158299202,10cyclictest22098-21/usr/sbin/munin10:50:243
2158299202,10cyclictest12072-21/usr/sbin/munin08:00:193
2158299201,6cyclictest0-21swapper/309:27:113
2158299201,12cyclictest0-21swapper/310:15:133
2158299201,10cyclictest17186-21ntp_states11:50:213
2158299201,10cyclictest0-21swapper/307:30:153
2157899208,4cyclictest0-21swapper/211:31:252
2157899207,10cyclictest9641-21latency_hist07:55:012
2157899205,3cyclictest12050irq/123-ahci[0011:20:002
2157899202,7cyclictest14835-21memory11:45:182
2157899201,6cyclictest0-21swapper/212:03:512
2157899201,16cyclictest10946-21cut09:10:152
2157199207,1cyclictest0-21swapper/107:30:171
2157199205,3cyclictest30830-21ps08:40:231
2157199205,3cyclictest0-21swapper/109:49:021
2157199201,3cyclictest0-21swapper/108:24:531
2157199201,18cyclictest0-21swapper/112:12:531
2157199201,18cyclictest0-21swapper/110:38:231
2157199201,17cyclictest0-21swapper/107:43:301
2157199201,15cyclictest0-21swapper/109:12:111
2157199201,14cyclictest0-21swapper/108:08:521
2157199201,14cyclictest0-21swapper/108:08:521
234822190,7sleep339-21rcuc/308:25:143
2158299199,2cyclictest87350irq/126-enp1s0-11:40:183
2158299199,2cyclictest87350irq/126-enp1s0-11:40:183
2158299198,3cyclictest19893-21/usr/sbin/munin10:45:253
2158299196,2cyclictest0-21swapper/310:55:173
2158299195,8cyclictest9006-40dnf11:31:253
2158299193,7cyclictest0-21swapper/311:07:493
2158299191,9cyclictest0-21swapper/307:10:183
2158299191,7cyclictest41-21ksoftirqd/312:40:013
2158299191,7cyclictest0-21swapper/311:38:073
2158299191,4cyclictest0-21swapper/312:09:313
2158299191,15cyclictest1813-21cut08:50:133
2158299191,11cyclictest0-21swapper/311:48:463
2158299191,10cyclictest3287-21/usr/sbin/munin07:40:193
2157899191,7cyclictest23878-21cat07:15:012
2157899191,7cyclictest0-21swapper/212:16:322
2157899191,7cyclictest0-21swapper/211:50:572
2157899191,6cyclictest0-21swapper/212:28:142
2157899191,6cyclictest0-21swapper/212:20:542
2157899191,5cyclictest0-21swapper/212:11:272
2157899191,5cyclictest0-21swapper/210:45:152
2157899191,5cyclictest0-21swapper/210:35:232
2157899191,5cyclictest0-21swapper/210:34:002
2157899191,5cyclictest0-21swapper/210:34:002
2157899191,5cyclictest0-21swapper/210:26:322
2157899191,4cyclictest19278-21/usr/sbin/munin09:30:192
2157899191,4cyclictest0-21swapper/209:42:282
2157899191,4cyclictest0-21swapper/209:24:032
2157199197,3cyclictest0-21swapper/111:50:171
2157199191,2cyclictest0-21swapper/107:39:551
2157199191,17cyclictest0-21swapper/112:31:011
2157199191,16cyclictest0-21swapper/109:30:281
2157199191,13cyclictest23742-21munin-run07:15:011
2157199191,10cyclictest17701-21basename10:40:111
2157199191,10cyclictest0-21swapper/109:25:181
2158299187,3cyclictest14276-21/usr/sbin/munin08:05:153
2158299187,3cyclictest14276-21/usr/sbin/munin08:05:153
2158299182,7cyclictest29319-21/usr/sbin/munin12:20:223
2158299182,5cyclictest755-21rngd10:23:533
2158299181,9cyclictest0-21swapper/310:35:163
2158299181,7cyclictest0-21swapper/307:51:053
21582991814,3cyclictest0-21swapper/308:16:303
2158299181,14cyclictest1886-21grep11:15:223
2158299181,11cyclictest839-21lldpd10:10:313
2158299181,11cyclictest0-21swapper/309:53:223
2158299181,10cyclictest31855-21/usr/sbin/munin08:45:243
21582991810,5cyclictest0-21swapper/310:30:173
21582991810,5cyclictest0-21swapper/310:30:173
2158299180,6cyclictest0-21swapper/309:30:213
2157899185,4cyclictest3892-21/usr/sbin/munin08:55:122
2157899184,5cyclictest15864-21kworker/u8:212:05:182
2157899183,4cyclictest518-21cut08:45:222
2157899181,9cyclictest0-21swapper/211:21:152
2157899181,8cyclictest32-21ksoftirqd/211:25:192
2157899181,7cyclictest30091-21sed11:05:252
2157899181,6cyclictest0-21swapper/210:42:082
2157899181,11cyclictest0-21swapper/210:21:572
2157899180,4cyclictest0-21swapper/210:00:572
2157199181,9cyclictest0-21swapper/112:15:231
2157199181,10cyclictest0-21swapper/109:45:021
2158299177,2cyclictest87550irq/128-enp1s0-12:25:263
2158299175,6cyclictest0-21swapper/310:02:533
2158299174,5cyclictest41-21ksoftirqd/308:35:193
2158299172,6cyclictest41-21ksoftirqd/309:05:113
2158299172,14cyclictest0-21swapper/309:36:293
2158299171,7cyclictest2665-21df_inode10:05:153
2158299171,7cyclictest16804-21expr08:10:143
2158299171,7cyclictest0-21swapper/312:34:573
2158299171,7cyclictest0-21swapper/309:15:253
2158299171,7cyclictest0-21swapper/309:15:253
2158299171,7cyclictest0-21swapper/309:12:483
2158299171,6cyclictest25871-21runrttasks09:45:073
2158299171,6cyclictest0-21swapper/312:11:503
21582991713,3cyclictest41-21ksoftirqd/311:30:013
21582991713,3cyclictest0-21swapper/310:25:093
21582991712,3cyclictest31018-21idleruntime-cro07:30:013
2158299170,9cyclictest39-21rcuc/309:20:163
2158299170,2cyclictest0-21swapper/307:38:433
2157899176,3cyclictest87350irq/126-enp1s0-08:50:132
2157899172,2cyclictest87650irq/129-enp1s0-09:57:002
2157899171,9cyclictest0-21swapper/208:18:472
2157899171,8cyclictest87650irq/129-enp1s0-10:15:232
2157899171,8cyclictest14276-21/usr/sbin/munin08:05:182
2157899171,8cyclictest14276-21/usr/sbin/munin08:05:182
2157899171,6cyclictest5862-21grep10:10:232
2157899171,6cyclictest0-21swapper/209:18:502
2157899171,6cyclictest0-21swapper/209:18:502
2157899171,6cyclictest0-21swapper/208:10:162
2157899171,5cyclictest0-21swapper/211:00:542
2157899171,3cyclictest87350irq/126-enp1s0-07:35:142
2157899171,3cyclictest0-21swapper/210:08:502
21578991712,3cyclictest23318-21/usr/sbin/munin10:50:232
2157899171,10cyclictest0-21swapper/207:45:382
21578991710,4cyclictest30154-21/usr/sbin/munin07:25:232
2158299167,7cyclictest39-21rcuc/307:15:043
2158299167,2cyclictest87650irq/129-enp1s0-09:55:133
2158299164,3cyclictest0-21swapper/310:40:253
2158299163,10cyclictest27539-21cut12:15:153
2158299162,12cyclictest21418-21cut08:20:173
2158299161,6cyclictest0-21swapper/307:55:123
2158299161,10cyclictest0-21swapper/308:55:263
21582991610,3cyclictest30350-21hwlatdetect08:40:173
2158299160,10cyclictest0-21swapper/309:00:163
2157899163,8cyclictest26078-40dnf08:30:192
2157899161,6cyclictest30009-21dnf08:40:152
2157899161,6cyclictest29430-21latency_hist08:40:022
2157899161,6cyclictest12401-21hddtemp_smartct11:40:162
2157899161,6cyclictest12401-21hddtemp_smartct11:40:162
21578991612,3cyclictest0-21swapper/207:55:162
21578991612,3cyclictest0-21swapper/207:15:472
21578991611,3cyclictest3953-21sed07:40:172
21578991611,3cyclictest23975-21load08:25:182
2157899160,5cyclictest27458-21gltestperf07:20:172
2157899160,15cyclictest0-21swapper/207:32:562
2157899160,10cyclictest0-21swapper/208:20:172
21582991510,2cyclictest27159-21cat07:20:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional