You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-07 - 14:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri Aug 07, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
62892236163,23sleep20-21swapper/207:06:392
62892236163,23sleep20-21swapper/207:06:392
62602236164,21sleep30-21swapper/307:06:153
62602236164,21sleep30-21swapper/307:06:153
62472236162,25sleep00-21swapper/007:06:050
62472236162,25sleep00-21swapper/007:06:040
63872233165,55sleep10-21swapper/107:07:551
63872233165,55sleep10-21swapper/107:07:541
89452700,2sleep00-21swapper/012:05:230
6648995046,3cyclictest0-21swapper/209:15:482
663599389,13cyclictest17312-21idleruntime-cro10:00:010
663599379,12cyclictest26130-21awk10:20:010
6635993711,12cyclictest7578-21munin-run12:05:010
664899349,12cyclictest14502-21latency_hist08:40:012
665299338,15cyclictest17580-21strings12:25:193
664199324,7cyclictest19491-40dnf10:04:441
663599324,13cyclictest2554-21munin-run10:40:010
664899319,12cyclictest23299-21perf09:00:022
6648993110,9cyclictest14394-21/usr/sbin/munin12:20:182
6641993125,3cyclictest19589-21ntp_states07:35:221
663599303,12cyclictest8531-21munin-run09:40:010
664899299,7cyclictest24149-21/usr/sbin/munin10:15:162
664899298,8cyclictest13232-21ntpq12:15:222
664899293,9cyclictest21742-21awk10:10:012
6648992910,7cyclictest12019-21ntp_states09:45:222
6648992910,10cyclictest6320-21latency_hist09:35:012
124222290,2sleep20-21swapper/207:20:152
664899287,11cyclictest9998-21/usr/sbin/munin12:10:142
664899284,14cyclictest7051-21latency_hist10:50:012
664199282,18cyclictest15480-21cut09:55:121
664199281,25cyclictest0-21swapper/110:35:151
664199281,23cyclictest579-21systemd-journal09:30:011
664199281,21cyclictest0-21swapper/112:02:521
177102280,2sleep00-21swapper/008:45:170
665299271,6cyclictest27708-21perf09:10:003
665299271,18cyclictest0-21swapper/312:15:583
665299271,18cyclictest0-21swapper/311:43:203
664899276,9cyclictest22740-21if_err_enp2s010:10:162
664899274,7cyclictest19528-21latency_hist10:05:012
664899273,14cyclictest20071-21chrt11:19:142
664199271,23cyclictest23186-21tail11:25:161
287012270,4sleep123-21ksoftirqd/107:55:271
111002270,5sleep121-21rcuc/107:15:241
665299264,15cyclictest0-21swapper/309:39:333
665299261,18cyclictest0-21swapper/311:31:343
664899267,6cyclictest15733-21cat11:09:572
664899267,10cyclictest26573-21cstates10:20:112
664899265,9cyclictest19128-21/usr/sbin/munin08:50:202
664899265,12cyclictest29979-21cat09:15:012
664899261,11cyclictest0-21swapper/212:38:332
6641992610,3cyclictest15750-21turbostat11:10:011
663599261,6cyclictest12483-21cut11:00:180
665299252,17cyclictest0-21swapper/311:18:223
665299251,18cyclictest0-21swapper/312:14:173
665299251,18cyclictest0-21swapper/309:50:363
665299251,18cyclictest0-21swapper/309:45:063
665299251,18cyclictest0-21swapper/309:28:573
665299251,17cyclictest20672-21memory10:05:193
665299251,15cyclictest0-21swapper/310:28:123
665299251,14cyclictest0-21swapper/312:01:353
664899257,7cyclictest18562-1kworker/2:0H12:00:112
664899255,7cyclictest17529-21gltestperf08:45:162
664899254,8cyclictest16790-21cat08:45:022
664899253,8cyclictest1176-21/usr/sbin/munin11:50:122
664899253,10cyclictest27704-21diskmemload09:10:012
664899252,14cyclictest31373-21cpu11:45:122
6648992511,6cyclictest25113-21diskstats11:30:152
6648992510,7cyclictest17567-21users11:10:262
664199259,3cyclictest32100-21munin-run09:20:001
664199252,11cyclictest28934-21munin-run11:40:001
664199251,5cyclictest0-21swapper/111:58:491
664199251,5cyclictest0-21swapper/111:58:491
664199251,21cyclictest18517-21users08:45:261
664199251,21cyclictest17235-21memory07:30:191
664199251,16cyclictest0-21swapper/111:49:191
664199251,15cyclictest0-21swapper/111:53:551
664199251,12cyclictest8616-21irqstats12:05:181
664199251,11cyclictest20178-21fschecks_count10:05:161
664199250,3cyclictest0-21swapper/112:26:471
663599251,12cyclictest16711-21munin-run08:45:010
665299249,7cyclictest0-21swapper/310:15:503
665299241,16cyclictest0-21swapper/312:08:143
665299241,16cyclictest0-21swapper/311:53:083
665299241,16cyclictest0-21swapper/310:59:573
665299241,15cyclictest0-21swapper/311:56:443
665299241,15cyclictest0-21swapper/311:56:443
665299240,17cyclictest0-21swapper/312:20:203
665299240,12cyclictest0-21swapper/311:27:203
664899247,4cyclictest8140-21sshd12:05:132
664899247,4cyclictest2778-21/usr/sbin/munin10:40:162
664899246,9cyclictest6575-21grep09:35:132
664899243,8cyclictest25106-21/usr/sbin/munin07:50:242
664899242,8cyclictest1928-21latency_hist09:25:012
6648992415,5cyclictest23867-21diskstats09:00:142
664199248,2cyclictest21055-21grep12:35:111
664199241,2cyclictest16717-21awk08:45:021
664199241,2cyclictest0-21swapper/107:15:021
664199241,21cyclictest0-21swapper/110:23:501
664199241,17cyclictest0-21swapper/112:23:261
664199241,15cyclictest0-21swapper/111:40:341
664199241,11cyclictest27090-21awk07:55:001
664199241,11cyclictest19437-21smart_sda11:15:231
664199241,11cyclictest12939-21basename09:50:011
664199241,11cyclictest0-21swapper/111:30:551
664199240,8cyclictest0-21swapper/109:42:431
664199240,15cyclictest0-21swapper/112:10:091
663599248,14cyclictest1949-21perf09:25:010
226822240,7sleep339-21rcuc/308:55:233
665299238,8cyclictest0-21swapper/309:55:093
665299237,10cyclictest0-21swapper/312:31:353
665299231,6cyclictest0-21swapper/310:49:563
6652992312,5cyclictest0-21swapper/310:11:233
665299231,17cyclictest0-21swapper/311:46:053
665299231,16cyclictest0-21swapper/309:41:583
665299230,15cyclictest0-21swapper/309:20:153
664899235,6cyclictest22937-21fschecks_count11:25:152
664899233,9cyclictest30539-21timerandwakeup11:40:242
6648992315,5cyclictest2807-21smart_sda08:10:232
6648992312,7cyclictest27299-21df_inode11:35:132
664199232,11cyclictest0-21swapper/109:32:031
664199232,11cyclictest0-21swapper/109:32:031
664199231,20cyclictest0-21swapper/107:26:161
664199231,19cyclictest23291-21latency_hist09:00:011
664199231,14cyclictest0-21swapper/111:11:581
664199231,11cyclictest7459-21sed09:35:201
664199231,11cyclictest13354-21expr09:50:141
664199231,11cyclictest12229-21cpu12:15:111
664199231,10cyclictest23213-21smart_sda10:10:231
663599231,14cyclictest22761-21sshd12:37:440
665299228,2cyclictest8689-21sed10:50:243
665299227,3cyclictest21362-21grep11:20:203
665299221,6cyclictest0-21swapper/310:02:263
665299221,16cyclictest0-21swapper/310:31:163
665299221,15cyclictest0-21swapper/311:00:203
665299221,10cyclictest31012-21sed08:00:243
664899227,7cyclictest4465-21ntp_states11:55:232
664899227,7cyclictest4465-21ntp_states11:55:232
664899226,8cyclictest8143-21/usr/sbin/munin08:25:142
664899226,5cyclictest16322-21/usr/sbin/munin07:30:122
664899225,5cyclictest16589-21/usr/sbin/munin12:25:132
664899224,5cyclictest0-21swapper/210:25:152
664899222,12cyclictest0-21swapper/212:35:012
6648992214,5cyclictest5422-21ntp_states09:30:192
6648992214,5cyclictest30762-21smart_sda08:00:232
6648992214,5cyclictest20374-21/usr/sbin/munin11:20:292
6648992213,6cyclictest6618-21/usr/sbin/munin08:20:152
6648992213,6cyclictest6618-21/usr/sbin/munin08:20:152
664199228,3cyclictest12396-21cat08:35:021
664199227,9cyclictest0-21swapper/110:30:151
664199221,19cyclictest0-21swapper/108:24:531
664199221,19cyclictest0-21swapper/108:24:531
664199221,19cyclictest0-21swapper/108:08:171
664199221,10cyclictest2809-21smartctl08:10:241
664199221,10cyclictest12719-21cstates08:35:151
664199221,10cyclictest10237-21cat08:30:021
664199221,10cyclictest0-21swapper/110:55:201
664199220,20cyclictest0-21swapper/109:11:441
664199220,20cyclictest0-21swapper/109:05:441
664199220,20cyclictest0-21swapper/108:01:391
663599221,6cyclictest15454-21ntp_offset12:20:220
663599221,12cyclictest25727-21/usr/sbin/munin09:05:210
665299218,2cyclictest14775-21needreboot11:05:203
665299217,8cyclictest0-21swapper/308:35:253
665299216,3cyclictest87350irq/126-enp1s0-09:30:143
665299216,3cyclictest87350irq/126-enp1s0-09:30:143
665299214,10cyclictest41-21ksoftirqd/308:05:143
665299211,9cyclictest0-21swapper/309:15:243
6652992117,3cyclictest0-21swapper/307:30:183
665299211,15cyclictest0-21swapper/310:22:193
665299211,14cyclictest27566-21head11:35:163
665299211,14cyclictest0-21swapper/312:37:163
6652992111,4cyclictest0-21swapper/310:40:123
665299211,12cyclictest20713-21/usr/sbin/munin07:40:193
664899217,6cyclictest13503-21diskstats09:50:152
664899215,5cyclictest87350irq/126-enp1s0-11:00:222
664899212,10cyclictest87650irq/129-enp1s0-09:40:142
6648992115,4cyclictest1872-21switchtime10:35:242
6648992114,4cyclictest31653-21memory10:30:182
6648992112,6cyclictest8639-21/usr/sbin/munin10:50:242
6648992112,6cyclictest4535-21/usr/sbin/munin08:15:182
6648992112,6cyclictest10897-21/usr/sbin/munin07:15:222
6648992111,7cyclictest31714-21/usr/sbin/munin08:05:172
664199219,3cyclictest0-21swapper/107:22:091
664199218,3cyclictest19858-21ntp_states12:30:221
664199218,11cyclictest0-21swapper/110:45:181
664199211,9cyclictest0-21swapper/111:03:451
664199211,4cyclictest0-21swapper/110:16:041
664199211,18cyclictest0-21swapper/110:29:471
664199211,18cyclictest0-21swapper/109:03:101
664199211,11cyclictest0-21swapper/109:20:161
664199211,11cyclictest0-21swapper/107:45:031
664199211,10cyclictest0-21swapper/108:17:571
664199210,10cyclictest0-21swapper/110:50:221
663599215,3cyclictest21476-21if_enp2s007:40:170
663599212,11cyclictest17546-21/usr/sbin/munin10:00:130
663599212,11cyclictest15978-21/usr/sbin/munin11:10:130
217422211,16sleep121936-21sed11:20:261
665299209,4cyclictest0-21swapper/310:35:253
665299204,9cyclictest0-21swapper/308:16:463
665299203,9cyclictest0-21swapper/308:20:223
665299203,9cyclictest0-21swapper/308:20:223
665299203,9cyclictest0-21swapper/307:45:273
6652992017,2cyclictest0-21swapper/308:25:153
6652992013,4cyclictest19461-21df08:50:123
665299201,14cyclictest0-21swapper/311:10:253
665299201,11cyclictest0-21swapper/309:10:023
665299201,10cyclictest27307-21/usr/sbin/munin07:55:173
664899201,3cyclictest10632-21sensors_temp10:55:232
6648992013,4cyclictest7185-21diskstats07:10:142
6648992013,4cyclictest23582-21hddtemp_smartct07:45:152
6648992013,4cyclictest21772-21/usr/sbin/munin07:40:192
6648992012,5cyclictest10687-21diskstats08:30:152
6648992011,6cyclictest27654-21diskstats07:55:152
6648992011,6cyclictest14126-21/usr/sbin/munin07:25:142
664199201,3cyclictest4246-21unixbench-2d10:40:271
664199201,18cyclictest0-21swapper/108:54:551
664199201,11cyclictest0-21swapper/107:40:241
663599209,3cyclictest4972-21/usr/sbin/munin10:45:200
663599202,10cyclictest27930-21/usr/sbin/munin09:10:250
663599201,4cyclictest20721-21diskstats11:20:140
663599201,11cyclictest21951-21/usr/sbin/munin10:10:150
665299193,8cyclictest12050irq/123-ahci[0009:05:003
6652991915,3cyclictest0-21swapper/308:10:113
665299191,10cyclictest25106-21/usr/sbin/munin07:50:223
664899199,7cyclictest15995-21sed09:55:152
6648991911,5cyclictest2160-21munin-node09:25:102
6648991911,5cyclictest19436-21memory07:35:192
663599199,2cyclictest87350irq/126-enp1s0-11:30:260
663599199,2cyclictest22911-21/usr/sbin/munin07:45:220
663599198,3cyclictest9375-21/usr/sbin/munin10:55:220
663599198,3cyclictest31357-21/usr/sbin/munin11:45:230
663599196,3cyclictest31269-21tr09:15:190
663599196,3cyclictest2773-21gzip08:10:220
663599191,7cyclictest0-21swapper/011:59:140
663599191,7cyclictest0-21swapper/011:59:140
663599191,7cyclictest0-21swapper/011:52:250
6635991913,4cyclictest11650-21/usr/sbin/munin09:45:170
663599191,10cyclictest2160-21/usr/sbin/munin09:25:100
165472191,2sleep30-21swapper/308:43:123
134632191,3sleep013501-21diskstats09:50:150
665299183,13cyclictest41-21ksoftirqd/307:10:153
665299182,4cyclictest11372-21ntp_kernel_err08:30:213
665299181,9cyclictest9731-21/usr/sbin/munin07:15:183
6652991814,3cyclictest0-21swapper/307:25:253
665299181,14cyclictest401ktimersoftd/307:35:123
663599188,2cyclictest5935-21/usr/sbin/munin08:20:160
663599188,2cyclictest5935-21/usr/sbin/munin08:20:160
663599187,3cyclictest26960-21/usr/sbin/munin11:35:140
663599186,3cyclictest18099-21unixbench_multi12:25:250
663599181,7cyclictest3122-21df_inode10:40:150
663599181,7cyclictest0-21swapper/007:25:160
663599181,6cyclictest0-21swapper/008:15:260
663599181,10cyclictest18176-21/usr/sbin/munin11:15:150
663599180,10cyclictest27704-21diskmemload10:32:400
216842181,2sleep00-21swapper/010:09:430
154842180,2sleep00-21swapper/011:06:440
665299177,1cyclictest0-21swapper/307:20:273
665299171,6cyclictest0-21swapper/308:45:243
663599178,2cyclictest28546-21/usr/sbin/munin10:25:160
663599171,7cyclictest0-21swapper/010:50:430
663599171,6cyclictest579-21systemd-journal08:35:010
663599171,5cyclictest0-21swapper/011:27:360
663599171,5cyclictest0-21swapper/009:34:590
663599171,5cyclictest0-21swapper/009:34:590
6635991712,3cyclictest26952-21gltestperf10:20:150
6635991712,3cyclictest12726-21grep12:15:150
6635991712,3cyclictest11928-21/usr/sbin/munin07:20:150
6635991712,3cyclictest11199-21grep07:15:250
663599171,14cyclictest25474-21grep07:50:140
663599170,6cyclictest0-21swapper/007:30:230
663599162,12cyclictest18787-21/usr/sbin/munin12:30:250
663599161,7cyclictest808-21snmpd07:12:010
663599161,5cyclictest0-21swapper/012:12:560
663599161,5cyclictest0-21swapper/011:40:120
663599161,4cyclictest9641-21unixbench_multi08:25:260
6635991611,3cyclictest12538-21/usr/sbin/munin08:35:180
6635991611,3cyclictest10165-21expr09:40:240
663599155,3cyclictest87350irq/126-enp1s0-08:55:170
663599151,12cyclictest27307-21/usr/sbin/munin07:55:190
6635991511,2cyclictest20199-21ntp_states08:50:210
6635991510,3cyclictest31714-21/usr/sbin/munin08:05:150
6635991510,3cyclictest29500-21/usr/sbin/munin08:00:140
6635991510,3cyclictest23520-21/usr/sbin/munin09:00:200
6635991510,3cyclictest19345-21kernelversion07:35:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional