You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-10 - 08:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Wed Jun 10, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
139052234162,24sleep20-21swapper/219:09:412
136072207132,23sleep10-21swapper/119:05:481
136732206169,24sleep30-21swapper/319:06:403
136312203167,25sleep00-21swapper/019:06:080
1956421760,5sleep31403199cyclictest21:45:233
14026991091,106cyclictest0-21swapper/200:22:102
14026991070,105cyclictest0-21swapper/220:06:012
140269910197,3cyclictest0-21swapper/222:06:432
14026999491,2cyclictest0-21swapper/220:54:442
14026996158,2cyclictest0-21swapper/200:20:002
110752510,1sleep00-21swapper/021:25:260
224782500,4sleep21402699cyclictest21:54:452
14031993916,7cyclictest17680-21memory22:55:203
14031993824,9cyclictest25920-40dnf20:45:333
14031993810,15cyclictest2725-21diskmemload23:45:133
1403199379,16cyclictest755-21rngd00:10:133
14031993611,15cyclictest5240-21ntpq23:40:213
1403199349,14cyclictest3637-21egrep22:25:113
14031993414,11cyclictest32276-21/usr/sbin/munin22:15:163
14031993414,11cyclictest32276-21/usr/sbin/munin22:15:163
14031993412,9cyclictest18258-21sort21:45:023
14031993410,8cyclictest6416-21smart_sda21:15:273
1403199338,11cyclictest10808-21grep22:40:173
14031993313,17cyclictest27747-21gltestperf22:05:163
14031993310,11cyclictest20358-21ntp_kernel_pll_19:20:213
1403199329,10cyclictest30766-21irqstats23:25:173
1403199329,10cyclictest30766-21irqstats23:25:173
1403199328,15cyclictest11760-21cpu21:30:123
1403199328,11cyclictest29520-21cstates22:10:123
14031993216,7cyclictest21508-21df_inode23:05:143
14031993215,9cyclictest87550irq/128-enp1s0-20:40:133
14031993214,6cyclictest29311-21memory00:35:183
14031993214,6cyclictest29311-21memory00:35:183
14031993212,12cyclictest2725-21diskmemload22:39:013
1401299328,11cyclictest13106-21awk20:20:010
1401299326,12cyclictest6176-21idleruntime-cro23:45:010
1403199319,8cyclictest3483-21cat22:25:013
14031993114,8cyclictest28354-21sshd20:53:303
14031993113,14cyclictest21163-21grep00:15:313
1401299312,21cyclictest13632-21kworker/0:021:48:560
1403199309,12cyclictest24004-21/usr/sbin/munin00:25:263
1403199308,10cyclictest5790-21cpu22:30:123
14031993020,6cyclictest23997-21smart_sda21:55:253
14031993011,6cyclictest4295-21sh20:00:023
14031993010,7cyclictest28756-21cut23:20:203
1403199298,13cyclictest26202-21/usr/sbin/munin00:30:223
1403199298,12cyclictest9544-21/usr/sbin/munin21:25:263
14031992921,5cyclictest15636-21ntp_states22:50:223
14031992917,7cyclictest14855-21/usr/sbin/munin21:35:183
14031992915,5cyclictest26964-21ntp_states19:35:223
14031992913,13cyclictest3764-21sshd23:37:343
14031992913,13cyclictest21604-21sshd20:36:313
14031992912,4cyclictest19689-21latency_hist20:35:023
1401299292,16cyclictest4819-21kworker/0:221:14:490
1401299291,21cyclictest31125-21kworker/0:322:20:020
1403199287,12cyclictest19043-21cat23:00:113
1403199285,14cyclictest0-21swapper/322:45:143
14031992820,5cyclictest8613-21/usr/sbin/munin21:20:233
14031992816,4cyclictest9276-21diskstats20:10:153
1401299282,13cyclictest25687-21idleruntime-cro19:35:000
1401299281,12cyclictest16555-21perf22:55:010
1403199278,7cyclictest0-21swapper/321:10:233
1403199278,12cyclictest21566-21awk19:25:103
1403199275,9cyclictest0-21swapper/323:55:163
14031992721,4cyclictest17193-21awk00:10:003
14031992719,5cyclictest16913-21smart_sda19:15:043
14031992718,6cyclictest22152-21diskstats00:20:133
14031992718,6cyclictest21798-21smart_sda21:50:223
14031992718,5cyclictest25909-21diskstats23:15:153
14031992718,5cyclictest24050-21diskstats19:30:143
14031992713,5cyclictest1266-21/usr/sbin/munin19:50:223
14031992713,5cyclictest1266-21/usr/sbin/munin19:50:223
14031992711,3cyclictest1987-21find21:05:243
14031992711,13cyclictest32134-21sshd19:48:293
268832260,3sleep121-21rcuc/120:50:151
1403199269,14cyclictest808-21snmpd20:09:303
14031992619,4cyclictest4873-21diskstats20:00:143
14031992619,4cyclictest28440-21diskstats19:40:143
14031992619,4cyclictest14137-21irqstats20:20:173
14031992618,5cyclictest15750-21/usr/sbin/munin19:10:243
14031992617,6cyclictest9500-21/usr/sbin/munin23:50:193
14031992617,6cyclictest30049-21/usr/sbin/munin20:55:243
14031992617,6cyclictest25958-21/usr/sbin/munin22:00:193
1401999262,15cyclictest0-21swapper/122:06:231
14012992610,4cyclictest0-21swapper/000:07:100
238322250,5sleep3909-21sssd_nss23:10:153
1403199258,8cyclictest87650irq/129-enp1s0-00:00:133
14031992518,4cyclictest508-21irqstats23:30:173
14031992517,5cyclictest12380-21smart_sda20:15:223
14031992510,12cyclictest16361-21kernelversion20:25:173
1402699252,7cyclictest2100-21latency_hist19:55:012
1402699252,7cyclictest2100-21latency_hist19:55:012
1401299253,10cyclictest16204-21perf19:15:010
53552240,19sleep15443-21mailstats23:40:251
14031992417,4cyclictest30923-21/usr/sbin/munin21:00:123
1401999249,7cyclictest0-21swapper/123:13:101
1401999243,16cyclictest0-21swapper/121:09:111
1401999242,12cyclictest31521-21sendmail_mailqu19:45:231
1401999241,16cyclictest19002-21/usr/sbin/munin00:10:261
1401299244,3cyclictest1935-21kworker/0:119:55:120
1402699230,8cyclictest0-21swapper/222:11:272
1401999231,10cyclictest200962tr23:00:221
1401299235,10cyclictest0-21swapper/023:15:180
1401299231,9cyclictest21486-21kworker/0:100:28:180
1401299231,9cyclictest16421-21kworker/0:123:08:050
1401299231,10cyclictest19212-21kworker/0:000:16:220
243392220,2sleep10-21swapper/100:25:141
1401999228,12cyclictest30403-21kworker/u8:123:28:371
1401999228,12cyclictest30403-21kworker/u8:123:28:361
1401999224,3cyclictest25617-21sshd19:34:281
1401999221,9cyclictest0-21swapper/122:34:381
1401999221,7cyclictest0-21swapper/123:06:101
1401999221,3cyclictest10501-21df22:40:131
1401999221,19cyclictest101952sleep123:50:311
1401999221,18cyclictest29423-21switchtime19:40:241
14019992211,3cyclictest14475-21unixbench-2d00:00:281
1401999221,12cyclictest16249-21cat19:15:011
1401299226,4cyclictest0-21swapper/023:30:180
1401299223,11cyclictest28709-21/usr/sbin/munin20:55:250
1401299222,9cyclictest4146-21kworker/0:020:03:280
1401299222,11cyclictest11204-21kworker/0:121:31:380
1401299221,8cyclictest8422-21kworker/0:320:23:380
1401299221,8cyclictest8217-21kworker/0:100:01:520
1401299221,8cyclictest16109-21kworker/0:119:24:240
1401299221,5cyclictest8564-1kworker/u9:122:29:200
105372220,1sleep10-21swapper/120:10:301
1402699211,10cyclictest0-21swapper/222:45:182
1401999219,3cyclictest579-21systemd-journal20:10:011
1401999212,11cyclictest909-21sssd_nss22:10:141
1401999211,9cyclictest0-21swapper/121:52:441
14019992117,2cyclictest14499-21perl21:35:161
1401999211,3cyclictest4736-21cstates20:00:121
1401999211,17cyclictest14679-21timerandwakeup20:20:241
1401999211,15cyclictest0-21swapper/123:56:451
1401299212,9cyclictest28821-21kworker/0:122:08:300
1401299212,8cyclictest22207-21kworker/0:221:57:520
1401299211,8cyclictest27597-21kworker/0:119:44:380
1401299211,8cyclictest20928-21kworker/0:223:11:530
1401299211,8cyclictest15071-21kworker/0:120:35:190
1401299211,8cyclictest10155-21kworker/0:222:40:520
1401299211,7cyclictest27384-21kworker/0:123:27:310
1401299211,7cyclictest27384-21kworker/0:123:27:310
14012992115,4cyclictest7118-21dump-pmu-power21:20:020
1401299210,9cyclictest7957-21kworker/0:022:36:540
1402699208,3cyclictest22488-21cat23:05:242
1402699202,8cyclictest0-21swapper/222:51:452
14026992014,4cyclictest31481-21processes19:45:232
1402699201,12cyclictest0-21swapper/223:27:462
1402699201,12cyclictest0-21swapper/223:27:462
1402699201,10cyclictest0-21swapper/200:35:182
1402699201,10cyclictest0-21swapper/200:35:182
1401999209,3cyclictest25746-21irqstats22:00:171
14019992016,3cyclictest0-21swapper/120:35:261
1401999201,4cyclictest25998-21kworker/u8:221:45:161
14019992013,3cyclictest0-21swapper/123:46:411
14019992013,2cyclictest0-21swapper/123:38:031
1401999201,2cyclictest20567-21sensors19:20:251
1401999201,18cyclictest0-21swapper/122:59:561
1401999201,18cyclictest0-21swapper/121:58:181
1401999201,18cyclictest0-21swapper/120:42:391
1401999201,15cyclictest0-21swapper/100:19:061
1401999201,14cyclictest0-21swapper/122:54:201
1401999201,11cyclictest30672-21latency_hist21:00:011
1401999201,11cyclictest26281-21latency_hist20:50:011
1401999201,11cyclictest21958-21expr00:20:121
1401999201,11cyclictest16334-21irqstats20:25:171
1401999201,11cyclictest0-21swapper/122:45:421
1401999201,10cyclictest3344-21tune2fs21:10:141
1401999201,10cyclictest2943-21sed22:20:271
1401999201,10cyclictest28176-21fschecks_count23:20:151
1401999201,10cyclictest27265-21ntp_states00:30:201
1401299202,9cyclictest31022-21kworker/0:222:11:440
1401299202,9cyclictest12723-21kworker/0:200:11:480
1401299202,8cyclictest31125-21kworker/0:322:46:300
1401299202,8cyclictest22207-21kworker/0:222:02:000
1401299202,8cyclictest10155-21kworker/0:222:57:500
1401299202,11cyclictest13632-21kworker/0:021:44:060
1401299201,9cyclictest8217-21kworker/0:123:59:340
1401299201,9cyclictest32145-21kworker/0:023:45:100
1401299201,9cyclictest15071-21kworker/0:120:34:250
1401299201,8cyclictest28821-21kworker/0:122:18:380
1401299201,8cyclictest28821-21kworker/0:122:18:380
1401299201,8cyclictest16421-21kworker/0:123:00:350
1401299201,8cyclictest13632-21kworker/0:021:37:460
1401299201,8cyclictest12723-21kworker/0:200:21:240
1401299201,6cyclictest16109-21kworker/0:119:19:090
1401299201,11cyclictest7688-21df_inode21:20:140
1401299200,8cyclictest27384-21kworker/0:123:24:090
3672190,2sleep00-21swapper/021:04:320
1402699196,4cyclictest30-21rcuc/222:22:212
1402699196,2cyclictest32-21ksoftirqd/221:25:262
14026991914,3cyclictest10795-21sed22:40:162
14026991912,6cyclictest32-21ksoftirqd/219:15:252
1402699190,5cyclictest30-21rcuc/221:42:372
1401999191,9cyclictest23663-21sed19:30:011
1401999191,8cyclictest0-21swapper/121:17:591
1401999191,17cyclictest0-21swapper/122:27:241
1401999191,17cyclictest0-21swapper/121:28:551
1401999191,16cyclictest0-21swapper/121:03:261
1401999191,16cyclictest0-21swapper/100:08:121
1401999191,10cyclictest9932-21crond22:40:011
1401999191,10cyclictest8883-21unixbench_singl21:20:251
1401999191,10cyclictest333-21smartctl22:15:231
1401999191,10cyclictest333-21smartctl22:15:231
1401999191,10cyclictest29310-21memory00:35:171
1401999191,10cyclictest29310-21memory00:35:171
1401999191,10cyclictest16674-21gltestperf19:15:021
1401999190,4cyclictest0-21swapper/123:31:021
1401299192,10cyclictest21501-21/usr/sbin/munin19:25:190
1401299191,9cyclictest13632-21kworker/0:021:50:060
1401299191,10cyclictest32491-21/usr/sbin/munin19:50:170
1401299191,10cyclictest32491-21/usr/sbin/munin19:50:170
1401299191,10cyclictest25899-21/usr/sbin/munin19:35:200
1401299190,8cyclictest25947-21kworker/0:200:32:180
276682180,7sleep2311ktimersoftd/223:20:012
253092180,2sleep20-21swapper/219:30:322
151852180,2sleep20-21swapper/220:23:442
1402699187,9cyclictest2737-21awk21:10:012
1402699181,8cyclictest32-21ksoftirqd/223:35:122
1402699181,7cyclictest27860-21idleruntime-cro19:40:012
1402699181,6cyclictest0-21swapper/222:35:112
1402699181,6cyclictest0-21swapper/200:33:152
14026991812,4cyclictest13233-21grep21:30:322
14026991812,3cyclictest821-21ntp_states23:30:212
14026991812,3cyclictest5600-21ntpq20:00:222
1402699181,11cyclictest0-21swapper/221:46:202
1401999186,3cyclictest3454-21cut19:55:211
14019991814,3cyclictest0-21swapper/119:50:391
14019991814,3cyclictest0-21swapper/119:50:391
1401999181,2cyclictest12733-21strings21:30:201
1401999181,16cyclictest0-21swapper/121:40:511
1401999181,16cyclictest0-21swapper/120:18:141
1401999181,16cyclictest0-21swapper/119:36:021
1401999181,10cyclictest26150-21gltestperf23:15:161
1401299188,2cyclictest24310-21/usr/sbin/munin20:45:170
1401299186,2cyclictest8422-21kworker/0:320:26:270
1401299181,9cyclictest32145-21kworker/0:023:35:240
1401299181,7cyclictest8217-21kworker/0:123:53:440
1401299181,10cyclictest0-21swapper/000:37:340
1401299181,10cyclictest0-21swapper/000:37:340
54842171,2sleep20-21swapper/222:29:262
1402699179,7cyclictest32-21ksoftirqd/221:15:172
1402699175,3cyclictest30-21rcuc/222:55:192
1402699173,4cyclictest0-21swapper/221:20:192
1402699172,7cyclictest30-21rcuc/222:30:132
1402699172,6cyclictest22112-21/usr/sbin/munin20:40:122
1402699171,9cyclictest0-21swapper/219:30:022
1402699171,7cyclictest0-21swapper/220:35:162
1402699171,6cyclictest0-21swapper/222:15:072
1402699171,6cyclictest0-21swapper/222:15:072
1402699171,5cyclictest32-21ksoftirqd/221:14:012
1402699171,5cyclictest10463-21chrt23:53:562
14026991712,3cyclictest3113-21if_err_enp2s019:55:162
14026991712,3cyclictest28093-21/usr/sbin/munin19:40:152
14026991712,3cyclictest12988-21/usr/sbin/munin00:00:152
1402699171,15cyclictest0-21swapper/200:28:542
14026991711,3cyclictest20817-21unixbench_multi19:20:262
14026991711,3cyclictest0-21swapper/220:45:332
1402699171,10cyclictest0-21swapper/222:03:032
1401999171,10cyclictest0-21swapper/120:30:071
1401299172,4cyclictest31466-40dnf19:45:230
1401299171,9cyclictest0-21swapper/020:40:170
1401299171,11cyclictest0-21swapper/022:31:440
1402699169,4cyclictest29952-21sendmail_mailst20:55:232
1402699166,3cyclictest87650irq/129-enp1s0-23:55:252
1402699165,3cyclictest87650irq/129-enp1s0-21:35:202
1402699165,3cyclictest5750irq/9-acpi20:15:482
1402699162,2cyclictest87650irq/129-enp1s0-23:10:172
1402699162,11cyclictest18319-21gltestperf20:30:162
1402699161,7cyclictest0-21swapper/223:40:242
1402699161,4cyclictest8564-1kworker/u9:121:59:302
14026991612,3cyclictest0-21swapper/223:23:482
1402699161,14cyclictest0-21swapper/223:02:482
14026991611,3cyclictest8927-21/usr/sbin/munin20:10:122
14026991611,3cyclictest6387-21acpi23:45:112
14026991611,3cyclictest17658-21sed20:30:022
14026991611,3cyclictest14216-21/usr/sbin/munin19:10:142
1402699160,5cyclictest15364-21cpuspeed_turbos00:05:172
14012991612,3cyclictest0-21swapper/021:05:050
14012991611,3cyclictest8927-21/usr/sbin/munin20:10:180
14012991611,3cyclictest7644-21memory20:05:170
14012991610,3cyclictest26577-21cpuspeed_turbos20:50:130
1402699156,2cyclictest87650irq/129-enp1s0-00:10:202
14026991511,2cyclictest32198-21switchtime21:00:242
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional