You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-17 - 13:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Aug 17, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
86752238162,24sleep20-21swapper/219:07:342
86342238163,24sleep30-21swapper/319:07:033
85152237161,25sleep00-21swapper/019:05:300
86172203165,25sleep10-21swapper/119:06:481
2112921570,4sleep3896699cyclictest20:49:033
8961998784,2cyclictest0-21swapper/223:17:272
8961997975,3cyclictest0-21swapper/219:59:102
98752750,1sleep30-21swapper/322:50:153
252332640,4sleep225229-21acpi19:45:142
895499462,5cyclictest0-21swapper/123:40:241
299752460,1sleep10-21swapper/119:55:151
895499431,32cyclictest368-21munin-run22:30:011
895499401,25cyclictest13501-21proc_pri00:10:231
8954994011,26cyclictest27243-21tr23:30:141
895499398,3cyclictest18388-21grep23:10:151
895499398,3cyclictest18388-21grep23:10:141
895499381,10cyclictest25355-21awk22:10:221
895499371,2cyclictest0-21swapper/122:18:341
895499371,26cyclictest0-21swapper/121:05:131
895499368,1cyclictest0-21swapper/119:25:021
895499365,3cyclictest87550irq/128-enp1s0-23:50:151
895499361,8cyclictest10556-21chrt21:36:281
895499361,32cyclictest24991-21latency_hist19:45:011
895499361,29cyclictest27813-21latency_hist21:05:021
895499359,3cyclictest13654-21latency_hist23:00:021
895499358,2cyclictest790-21in:imjournal21:55:011
895499357,4cyclictest0-21swapper/120:30:271
895499357,3cyclictest16565-21chrt20:36:531
895499351,3cyclictest14328-21ntp_offset21:45:201
894899356,14cyclictest20898-21munin-run00:30:020
896199341,23cyclictest0-21swapper/221:17:062
895499348,3cyclictest0-21swapper/100:17:291
895499346,3cyclictest9202-21missed_timers20:20:191
895499342,24cyclictest23-21ksoftirqd/121:58:001
895499341,7cyclictest7287-21/usr/sbin/munin22:45:221
8954993413,18cyclictest12660-21users21:40:251
8954993411,20cyclictest13890-21/usr/sbin/munin23:00:271
895499341,10cyclictest31086-21chrt23:38:081
8954993410,20cyclictest22681-21/usr/sbin/munin23:20:161
895499340,4cyclictest26535-21kworker/u8:121:16:331
895499331,5cyclictest0-21swapper/123:26:461
895499331,28cyclictest29395-21perf19:55:011
895499331,24cyclictest4658-21cstates21:25:151
8954993312,18cyclictest17703-21tail20:40:161
8954993311,19cyclictest20489-21/usr/sbin/munin23:15:201
895499328,3cyclictest4880-21latency_hist22:40:011
895499325,24cyclictest25207-21grep20:55:281
8954993211,18cyclictest763-21dbus-daemon20:30:001
8954993211,18cyclictest19862-21/usr/sbin/munin22:00:251
8954993211,18cyclictest19862-21/usr/sbin/munin22:00:251
895499321,10cyclictest31863-21cpu20:00:141
895499320,30cyclictest0-21swapper/119:15:041
88282320,3sleep121-21rcuc/100:00:201
213442320,5sleep0894899cyclictest23:15:200
896699311,20cyclictest0-21swapper/319:25:353
896699311,15cyclictest0-21swapper/322:20:133
896699310,11cyclictest0-21swapper/320:10:183
895499318,3cyclictest4892-21ntp_kernel_err20:10:201
895499316,2cyclictest20077-21/usr/sbin/munin00:25:241
895499311,2cyclictest9492-21/usr/sbin/munin22:50:261
895499311,2cyclictest7931-21smart_sda21:30:251
895499311,27cyclictest23086-21latency_hist00:35:021
8954993112,16cyclictest10334-21ls00:05:141
8954993111,17cyclictest6309-21tail23:55:171
895499310,10cyclictest0-21swapper/123:48:501
75212310,4sleep011-21rcuc/020:15:250
8966993012,9cyclictest320682sleep321:13:193
896699300,20cyclictest0-21swapper/320:55:233
895499308,19cyclictest23659-21df_inode00:35:161
895499303,3cyclictest4005-21kworker/u8:019:30:341
895499301,26cyclictest4242-21latency_hist21:25:011
8954993010,17cyclictest22143-21cat22:05:121
8954993010,17cyclictest17717-21strings00:20:221
8954993010,17cyclictest17717-21strings00:20:211
895499300,9cyclictest0-21swapper/123:06:181
895499300,8cyclictest0-21swapper/120:05:231
894899307,11cyclictest25304-21idleruntime-cro00:40:010
894899303,12cyclictest16512-21awk00:20:020
896699291,12cyclictest0-21swapper/322:35:473
896699290,24cyclictest0-21swapper/300:24:273
896699290,24cyclictest0-21swapper/300:24:273
896199291,13cyclictest0-21swapper/200:12:452
895499299,3cyclictest31158-21missed_timers21:10:211
895499298,4cyclictest1-21systemd19:15:011
895499298,3cyclictest20825-21/usr/sbin/munin19:35:191
895499298,2cyclictest0-21swapper/122:41:161
895499291,25cyclictest21263-21cat20:50:011
8954992910,2cyclictest1798-21open_inodes22:30:231
896699281,14cyclictest0-21swapper/321:47:293
8961992819,8cyclictest0-21swapper/223:00:292
896199281,12cyclictest0-21swapper/219:44:342
896199281,11cyclictest0-21swapper/219:21:412
896199281,11cyclictest0-21swapper/200:28:352
895499288,3cyclictest827-21runrttasks20:50:101
895499287,3cyclictest16434-21/usr/sbin/munin19:25:251
896699270,17cyclictest0-21swapper/323:22:323
896699270,10cyclictest0-21swapper/321:05:193
896699270,10cyclictest0-21swapper/320:24:003
896199271,12cyclictest0-21swapper/220:11:552
896199271,10cyclictest0-21swapper/222:33:372
895499278,3cyclictest8072-21latency_hist20:20:011
895499278,3cyclictest28838-21cstates22:20:131
894899271,13cyclictest23090-21munin-run00:35:010
894899271,12cyclictest30010-21diskmemload22:00:010
122972270,6sleep2311ktimersoftd/219:15:112
896699267,2cyclictest0-21swapper/320:40:013
896699261,8cyclictest0-21swapper/321:30:193
896699261,10cyclictest0-21swapper/319:22:513
896199261,11cyclictest0-21swapper/222:21:172
896199261,11cyclictest0-21swapper/219:12:332
896199260,12cyclictest0-21swapper/200:38:472
896199260,12cyclictest0-21swapper/200:17:392
895499265,3cyclictest827-21runrttasks19:48:571
75042260,1sleep30-21swapper/322:45:133
896699251,8cyclictest0-21swapper/322:32:153
896699251,10cyclictest909-21sssd_nss19:10:243
896699251,10cyclictest0-21swapper/319:17:313
896699250,17cyclictest0-21swapper/322:05:103
896199251,9cyclictest0-21swapper/223:32:492
896199251,9cyclictest0-21swapper/222:06:252
896199251,12cyclictest0-21swapper/223:37:172
896199251,11cyclictest0-21swapper/222:48:512
896199250,12cyclictest0-21swapper/221:33:422
896699248,3cyclictest25044-21if_err_enp1s022:10:183
896699247,7cyclictest0-21swapper/323:40:263
896699241,11cyclictest2407-21if_enp2s020:05:183
896699241,10cyclictest0-21swapper/319:30:153
896199241,10cyclictest0-21swapper/220:29:132
894899240,11cyclictest18404-21munin-run19:30:000
896699239,3cyclictest0-21swapper/321:00:193
896699232,9cyclictest27974-21perl19:50:163
896699231,12cyclictest0-21swapper/323:01:433
896699231,12cyclictest0-21swapper/300:12:033
896699230,8cyclictest0-21swapper/300:25:193
896699230,11cyclictest0-21swapper/322:18:303
896199231,9cyclictest30010-21diskmemload23:42:572
896199231,13cyclictest0-21swapper/220:42:072
896199231,11cyclictest0-21swapper/221:29:342
896199230,9cyclictest0-21swapper/222:02:452
896199230,9cyclictest0-21swapper/222:02:452
896199230,10cyclictest0-21swapper/222:55:492
894899232,3cyclictest0-21swapper/023:30:120
8948992313,7cyclictest26535-21kworker/u8:121:10:230
894899231,11cyclictest11478-21awk22:55:000
894899231,11cyclictest0-21swapper/022:50:000
327172230,2sleep00-21swapper/021:15:130
896699228,2cyclictest0-21swapper/321:37:433
896699221,8cyclictest0-21swapper/320:30:193
896699221,8cyclictest0-21swapper/300:17:133
896699221,10cyclictest32617-21sshd22:27:293
896699220,9cyclictest0-21swapper/323:35:363
896699220,9cyclictest0-21swapper/319:38:413
896699220,10cyclictest0-21swapper/319:50:003
896199221,9cyclictest0-21swapper/223:58:492
896199221,9cyclictest0-21swapper/219:29:582
236392221,3sleep223646-21acpi20:55:132
896699219,3cyclictest0-21swapper/321:19:313
896699219,2cyclictest0-21swapper/322:40:313
896699217,3cyclictest5722-21/usr/sbin/munin23:55:133
896699217,3cyclictest0-21swapper/320:27:173
896699216,8cyclictest0-21swapper/319:55:483
896699211,9cyclictest0-21swapper/323:29:423
896699211,9cyclictest0-21swapper/320:44:193
896699210,9cyclictest0-21swapper/323:09:433
896699210,9cyclictest0-21swapper/321:55:043
896199212,4cyclictest18791-21pmu-power21:55:212
896199211,11cyclictest0-21swapper/200:23:112
896199211,11cyclictest0-21swapper/200:23:112
894899218,3cyclictest11750-21kworker/0:200:10:080
894899211,12cyclictest3270-21sshd20:05:290
224652211,2sleep30-21swapper/320:50:203
896699208,11cyclictest12050irq/123-ahci[0000:35:113
896699205,3cyclictest0-21swapper/323:13:473
896699205,3cyclictest0-21swapper/323:13:473
896699204,2cyclictest0-21swapper/300:06:253
896699201,7cyclictest20220-21/usr/sbin/munin22:00:163
896699201,7cyclictest20220-21/usr/sbin/munin22:00:163
896699201,11cyclictest21464-21df_inode00:30:163
896699200,9cyclictest247152sleep319:41:333
896199202,6cyclictest9425-21unixbench_multi00:00:272
8961992014,4cyclictest19912-21sed20:45:162
8961992012,5cyclictest30359-21cut21:10:132
896199201,10cyclictest0-21swapper/219:38:302
896199200,10cyclictest0-21swapper/200:32:052
8948992013,5cyclictest26535-21kworker/u8:120:25:110
894899201,12cyclictest0-21swapper/019:30:220
894899201,10cyclictest808-21snmpd21:49:510
896699199,2cyclictest11694-21/usr/sbin/munin22:55:263
896699197,6cyclictest0-21swapper/321:54:533
896699196,3cyclictest0-21swapper/323:50:203
896699196,3cyclictest0-21swapper/320:04:403
896699195,3cyclictest0-21swapper/321:26:133
896699191,4cyclictest3341-21ntp_states21:20:213
896699191,11cyclictest0-21swapper/323:18:083
896699191,10cyclictest30010-21diskmemload00:04:553
896699191,10cyclictest27080-21/usr/sbin/munin23:30:293
896699191,10cyclictest0-21swapper/321:40:153
896699190,8cyclictest0-21swapper/323:48:043
896199195,8cyclictest16756-40dnf20:39:242
896199194,6cyclictest87550irq/128-enp1s0-20:20:202
896199191,11cyclictest0-21swapper/221:47:342
894899198,3cyclictest808-21snmpd22:45:020
894899198,3cyclictest28656-21/usr/sbin/munin22:20:250
894899191,6cyclictest0-21swapper/021:54:340
894899191,6cyclictest0-21swapper/021:38:260
894899191,4cyclictest0-21swapper/021:21:090
8948991914,3cyclictest30263-21sed19:55:170
8948991910,3cyclictest16959-21expr00:20:150
8948991910,3cyclictest16959-21expr00:20:150
894899190,5cyclictest8564-1kworker/u9:122:05:030
896699186,3cyclictest0-21swapper/320:16:423
896199181,7cyclictest9684-21irqstats21:35:192
896199181,5cyclictest0-21swapper/222:16:112
8961991813,2cyclictest2120-21irqstats23:45:162
8961991812,3cyclictest10127-21/usr/sbin/munin00:05:162
896199181,15cyclictest0-21swapper/221:50:242
896199181,12cyclictest0-21swapper/220:20:012
896199181,10cyclictest0-21swapper/219:34:282
896199180,9cyclictest0-21swapper/223:20:222
894899187,3cyclictest87550irq/128-enp1s0-20:55:210
894899185,8cyclictest32694-40dnf23:40:250
894899183,2cyclictest87550irq/128-enp1s0-23:05:480
894899181,7cyclictest0-21swapper/021:05:160
894899181,5cyclictest0-21swapper/023:12:020
894899181,5cyclictest0-21swapper/023:12:020
894899181,5cyclictest0-21swapper/021:43:410
894899181,3cyclictest87350irq/126-enp1s0-23:50:310
8948991813,3cyclictest8900-21sed20:20:160
8948991812,4cyclictest11126-21munin-run19:15:010
8948991812,3cyclictest1936-21gltestperf23:45:140
894899181,10cyclictest0-21swapper/021:34:030
268742181,3sleep00-21swapper/019:46:040
256872181,2sleep20-21swapper/223:25:222
218272181,2sleep00-21swapper/022:04:580
218272181,2sleep00-21swapper/022:04:580
896199172,5cyclictest20266-21munin-run23:15:022
896199172,5cyclictest20266-21munin-run23:15:022
896199171,6cyclictest10470-21needreboot22:50:192
896199171,15cyclictest0-21swapper/221:01:272
896199171,14cyclictest0-21swapper/222:35:482
896199171,14cyclictest0-21swapper/221:07:572
8961991711,3cyclictest32501-21hwlatdetect20:00:182
896199171,11cyclictest0-21swapper/221:24:492
896199171,11cyclictest0-21swapper/220:50:552
896199171,11cyclictest0-21swapper/220:09:012
896199171,10cyclictest0-21swapper/223:05:182
894899176,3cyclictest808-21snmpd23:35:110
894899175,3cyclictest13646-21strings20:30:210
894899172,12cyclictest23025-21/usr/sbin/munin19:40:120
894899171,8cyclictest11371-21sendmail_mailst00:05:260
894899171,8cyclictest0-21swapper/023:23:380
894899171,7cyclictest13890-21/usr/sbin/munin23:00:140
8948991712,3cyclictest2891-21/usr/sbin/munin22:35:170
8948991712,3cyclictest27342-21unixbench_multi21:00:270
894899171,15cyclictest0-21swapper/022:15:410
8948991711,3cyclictest21617-21if_err_enp2s019:35:160
8948991711,3cyclictest14851-21/usr/sbin/munin20:35:170
894899171,10cyclictest0-21swapper/022:59:290
894899171,10cyclictest0-21swapper/020:40:250
12712170,2sleep00-21swapper/020:03:180
896199161,6cyclictest0-21swapper/222:10:122
8961991612,3cyclictest0-21swapper/222:42:452
8961991612,2cyclictest31670-21irqstats22:25:202
8961991611,3cyclictest4916-21cat23:50:262
8961991611,3cyclictest28397-21needreboot19:50:202
8961991611,3cyclictest12733-21cat20:30:132
8961991611,3cyclictest11541-21fschecks_time21:40:152
894899169,4cyclictest0-21swapper/000:00:170
894899161,6cyclictest14586-21diskstats19:20:160
894899161,6cyclictest0-21swapper/023:27:170
894899161,5cyclictest0-21swapper/022:11:240
894899161,14cyclictest0-21swapper/020:51:500
8948991611,3cyclictest6358-21gltestperf23:55:180
8948991611,3cyclictest4476-21/usr/sbin/munin21:25:200
8948991611,3cyclictest32120-21smart_sda22:25:270
8948991611,3cyclictest11417-21netstat19:15:040
894899161,12cyclictest3859-21/usr/sbin/munin20:10:170
894899154,5cyclictest0-21swapper/020:45:150
894899153,4cyclictest0-21swapper/019:54:260
894899151,8cyclictest87350irq/126-enp1s0-22:30:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional