You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-29 - 02:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Aug 29, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
14002238162,24sleep10-21swapper/119:07:551
13252238163,24sleep30-21swapper/319:06:573
13892233161,60sleep00-21swapper/019:07:450
12622202165,25sleep20-21swapper/219:06:062
392221400,4sleep2168099cyclictest00:05:212
168099106103,2cyclictest0-21swapper/219:41:302
164752950,3sleep123-21ksoftirqd/123:20:211
1680998683,2cyclictest0-21swapper/223:34:012
1680998178,2cyclictest0-21swapper/221:06:022
1680998178,2cyclictest0-21swapper/221:06:022
211732560,1sleep00-21swapper/019:50:190
311542520,1sleep30-21swapper/322:40:183
91682510,5sleep3168499cyclictest23:05:143
1667993810,13cyclictest25568-21awk22:30:020
168499375,8cyclictest12050irq/123-ahci[0021:30:013
168499373,7cyclictest1772-40dnf22:47:243
167399371,31cyclictest0-21swapper/122:40:421
168499364,3cyclictest827-21runrttasks23:23:123
1684993616,17cyclictest4138-21diskstats21:40:133
168499348,2cyclictest30928-21awk20:15:023
1684993418,13cyclictest22737-21diskmemload23:57:103
108872340,1sleep00-21swapper/000:20:250
168499339,3cyclictest8026-21ntp_states19:20:213
168499338,8cyclictest10042-21gltestperf00:20:153
168499331,7cyclictest1-21systemd21:00:003
1684993313,2cyclictest14759-21unixbench_singl23:15:253
168499331,10cyclictest5061-21/usr/sbin/munin00:10:153
168499313,12cyclictest7257-21/usr/sbin/munin00:15:253
168499312,13cyclictest23583-21/usr/sbin/munin22:25:173
168499312,12cyclictest27079-21df20:05:123
168499311,5cyclictest3860-21chrt22:50:533
168499311,4cyclictest17434-21sshd19:41:303
167399311,20cyclictest0-21swapper/100:33:491
168499301,8cyclictest808-21snmpd00:01:073
168499301,6cyclictest9001-21irqstats21:50:183
168499301,5cyclictest22366-21taskset19:55:133
168499301,5cyclictest22366-21taskset19:55:133
168499301,5cyclictest18556-21unixbench_singl22:10:243
168499301,4cyclictest11998-21diskstats00:25:153
1684993014,3cyclictest21112-21switchtime23:30:243
1684993012,3cyclictest3385-21unixbench_multi19:10:383
1684993012,3cyclictest15563-21cat20:50:243
168499301,10cyclictest579-21systemd-journal21:05:003
167399301,11cyclictest0-21swapper/122:33:581
166799305,12cyclictest5779-21idleruntime-cro21:45:010
168499298,11cyclictest0-21swapper/321:45:123
168499293,6cyclictest0-21swapper/322:06:023
166799294,11cyclictest6403-21perf23:00:010
166799290,13cyclictest22744-21munin-run21:10:010
166799290,13cyclictest22744-21munin-run21:10:010
104632290,3sleep1167399cyclictest20:40:161
168499282,10cyclictest0-21swapper/323:13:543
168499281,5cyclictest1860-21grep20:20:183
168499281,13cyclictest0-21swapper/320:18:523
168499281,12cyclictest0-21swapper/322:04:083
167399287,18cyclictest14257-21idleruntime19:35:171
1673992811,3cyclictest0-21swapper/119:21:541
166799284,11cyclictest21171-21basename22:20:020
166799284,11cyclictest21171-21basename22:20:020
166799282,14cyclictest10793-21munin-run23:10:000
204282270,2sleep10-21swapper/119:50:121
168499272,3cyclictest0-21swapper/321:57:523
168499271,13cyclictest0-21swapper/319:29:473
168499271,12cyclictest0-21swapper/323:03:153
168499271,11cyclictest0-21swapper/321:35:313
1684992710,3cyclictest0-21swapper/300:38:313
167399272,11cyclictest827-21runrttasks20:06:321
167399271,17cyclictest0-21swapper/120:36:371
168499269,3cyclictest0-21swapper/319:17:193
168499261,9cyclictest0-21swapper/322:32:283
168499261,13cyclictest0-21swapper/300:32:123
168499261,12cyclictest0-21swapper/321:32:313
168499261,12cyclictest0-21swapper/321:19:353
168499261,11cyclictest0-21swapper/321:11:393
167399266,2cyclictest0-21swapper/119:55:251
167399266,2cyclictest0-21swapper/119:55:251
1673992619,5cyclictest9167-21diskstats23:05:151
1673992611,3cyclictest13767-21latency_hist00:30:011
65102250,3sleep10-21swapper/120:30:211
63572250,1sleep00-21swapper/000:10:250
168499259,3cyclictest0-21swapper/322:37:593
168499251,13cyclictest0-21swapper/323:36:443
168499251,11cyclictest4768-21users20:25:273
168499251,11cyclictest0-21swapper/323:42:563
168499251,10cyclictest0-21swapper/320:37:353
1684992510,3cyclictest0-21swapper/323:26:233
1684992510,3cyclictest0-21swapper/323:26:233
1684992510,3cyclictest0-21swapper/321:06:273
1684992510,3cyclictest0-21swapper/321:06:273
1684992510,3cyclictest0-21swapper/319:48:363
1684992510,3cyclictest0-21swapper/300:06:403
168499249,3cyclictest0-21swapper/323:48:403
168499248,3cyclictest0-21swapper/322:15:313
168499248,3cyclictest0-21swapper/322:15:313
168499248,3cyclictest0-21swapper/320:42:113
168499241,10cyclictest0-21swapper/321:24:073
1684992410,3cyclictest0-21swapper/322:23:553
167399246,4cyclictest0-21swapper/123:15:211
167399242,4cyclictest12636-21hddtemp_smartct20:45:151
1673992417,4cyclictest5228-21irqstats22:55:191
166799241,11cyclictest0-21swapper/023:50:010
168499239,3cyclictest0-21swapper/323:50:123
168499239,3cyclictest0-21swapper/319:36:403
168499238,3cyclictest0-21swapper/320:47:513
168499237,3cyclictest0-21swapper/322:57:313
168099231,5cyclictest0-21swapper/223:15:032
167399232,9cyclictest0-21swapper/121:38:141
167399232,3cyclictest24556-21users21:10:261
1673992313,3cyclictest0-21swapper/121:05:241
1673992313,3cyclictest0-21swapper/121:05:241
167399231,10cyclictest0-21swapper/120:00:171
167399230,17cyclictest0-21swapper/122:53:391
167399230,17cyclictest0-21swapper/122:07:551
136062230,4sleep2168099cyclictest19:35:132
168499226,3cyclictest0-21swapper/319:34:323
168499221,13cyclictest0-21swapper/320:00:143
168099221,5cyclictest0-21swapper/223:50:122
167399229,4cyclictest0-21swapper/120:25:231
167399228,4cyclictest19553-40dnf19:45:301
167399228,3cyclictest22674-21hddtemp_smartct23:35:151
167399228,2cyclictest22737-21diskmemload23:01:181
167399221,15cyclictest0-21swapper/122:25:511
1673992210,3cyclictest7459-21grep21:45:241
1673992210,3cyclictest22297-21memory22:20:191
166799221,11cyclictest8609-21perf23:05:000
168499218,3cyclictest0-21swapper/320:34:353
168499217,3cyclictest0-21swapper/319:50:323
167399216,9cyclictest0-21swapper/122:01:591
167399216,4cyclictest0-21swapper/121:20:111
1673992113,2cyclictest0-21swapper/121:50:391
1673992113,2cyclictest0-21swapper/121:33:161
167399211,2cyclictest0-21swapper/123:32:551
167399211,18cyclictest6730-21turbostat19:20:021
167399211,11cyclictest1699-21ntpq00:00:241
167399211,11cyclictest12981-21latency_hist23:15:021
167399211,11cyclictest1-21systemd21:40:201
167399211,10cyclictest665-21perf20:20:011
167399211,10cyclictest18962-21munin-run22:15:001
167399210,5cyclictest0-21swapper/100:35:491
166799216,6cyclictest0-21swapper/000:36:210
166799213,10cyclictest7-21ksoftirqd/000:15:530
166799212,11cyclictest11646-21smart_sda21:55:230
166799211,11cyclictest25784-21users23:40:250
324582201,15sleep132708-21df22:45:161
168099209,2cyclictest87650irq/129-enp1s0-00:30:242
168099205,3cyclictest663-21basename20:20:012
168099201,12cyclictest0-21swapper/222:40:112
168099200,3cyclictest12234-21kworker/u8:022:26:102
167399209,3cyclictest13497-21cat19:35:011
167399208,3cyclictest30233-21hddtemp_smartct21:25:161
167399205,9cyclictest0-21swapper/100:22:331
167399202,14cyclictest14325-21tr20:50:131
167399201,9cyclictest0-21swapper/100:07:581
167399201,6cyclictest8153-21sed00:15:181
1673992016,2cyclictest0-21swapper/123:25:121
1673992016,2cyclictest0-21swapper/123:25:121
167399201,4cyclictest29312-21threads22:35:251
1673992013,4cyclictest11907-21grep21:55:241
167399201,17cyclictest22737-21diskmemload23:49:511
167399201,10cyclictest3031-21latency_hist20:25:011
167399200,14cyclictest0-21swapper/122:15:501
167399200,14cyclictest0-21swapper/122:15:501
167399200,12cyclictest0-21swapper/119:27:171
166799209,6cyclictest0-21swapper/021:49:560
166799202,10cyclictest11965-21/usr/sbin/munin20:45:160
166799201,11cyclictest3279-21sshd21:36:290
166799201,11cyclictest30794-21/usr/sbin/munin23:55:190
166799201,11cyclictest2221-21/usr/sbin/munin22:50:210
166799201,11cyclictest14168-21/usr/sbin/munin20:50:140
166799201,11cyclictest0-21swapper/023:54:190
312072191,3sleep022737-21diskmemload21:26:030
260242191,14sleep226127-21vmstat20:00:252
168099191,9cyclictest23164-21expr21:10:132
168099191,7cyclictest0-21swapper/222:45:182
168099191,4cyclictest0-21swapper/223:03:012
168099191,12cyclictest0-21swapper/222:20:042
167399192,4cyclictest18902-21df_inode21:00:131
167399191,8cyclictest0-21swapper/123:44:331
167399191,5cyclictest30010-21ntp_states20:10:211
1673991915,3cyclictest0-21swapper/123:57:381
1673991914,3cyclictest17032-21cut20:55:171
167399191,10cyclictest29128-21fschecks_time23:50:151
166799194,13cyclictest1423-21timerwakeupswit22:45:260
166799193,2cyclictest87350irq/126-enp1s0-21:18:080
166799193,10cyclictest0-21swapper/021:24:020
166799191,9cyclictest16763-21munin-run22:10:000
166799191,6cyclictest0-21swapper/020:55:170
166799191,11cyclictest7-21ksoftirqd/019:40:190
166799191,11cyclictest4082-21sendmail_mailqu00:05:220
166799191,11cyclictest0-21swapper/022:30:380
166799191,10cyclictest27977-21/usr/sbin/munin22:35:120
166799191,10cyclictest18562-21/usr/sbin/munin21:00:190
166799191,10cyclictest0-21swapper/021:50:120
166799190,6cyclictest0-21swapper/023:33:120
49482181,2sleep10-21swapper/100:10:021
188152181,2sleep00-21swapper/022:13:200
168099189,5cyclictest9410-21grep23:05:172
168099185,2cyclictest10882-21kworker/u8:120:29:332
168099184,8cyclictest0-21swapper/223:26:412
168099184,8cyclictest0-21swapper/223:26:412
168099183,2cyclictest87550irq/128-enp1s0-19:20:272
168099182,8cyclictest10385-21/usr/sbin/munin21:55:162
168099181,9cyclictest0-21swapper/222:31:272
168099181,7cyclictest16264-21cat20:55:012
168099181,7cyclictest0-21swapper/221:43:402
168099181,7cyclictest0-21swapper/221:20:422
168099181,6cyclictest2587-21basename19:10:152
168099181,6cyclictest0-21swapper/221:31:462
1680991814,3cyclictest1334-21crond21:35:022
1680991813,3cyclictest9794-21df_inode00:20:122
168099181,15cyclictest0-21swapper/223:55:072
1680991811,4cyclictest9542-21grep20:40:012
167399182,13cyclictest25319-21grep21:15:121
167399182,13cyclictest17148-21timerwakeupswit19:40:231
167399181,16cyclictest0-21swapper/119:10:191
166799189,3cyclictest7135-21sshd20:32:290
166799186,3cyclictest0-21swapper/023:35:360
166799184,3cyclictest87550irq/128-enp1s0-19:24:210
166799183,8cyclictest21709-21expr22:20:130
166799183,3cyclictest7-21ksoftirqd/019:15:130
166799183,13cyclictest11-21rcuc/023:15:000
166799182,9cyclictest7-21ksoftirqd/022:00:120
166799181,9cyclictest0-21swapper/021:14:310
166799181,6cyclictest0-21swapper/022:43:460
166799181,4cyclictest13846-21/usr/sbin/munin00:30:190
1667991813,3cyclictest1343-21munin-run21:35:010
1667991812,3cyclictest11885-21fschecks_time19:30:150
166799181,12cyclictest0-21swapper/000:25:030
166799181,10cyclictest0-21swapper/023:25:210
166799181,10cyclictest0-21swapper/023:25:210
28632170,2sleep00-21swapper/019:10:200
168099176,3cyclictest87350irq/126-enp1s0-21:50:122
168099174,7cyclictest1655-21cyclictest19:50:172
168099173,5cyclictest30-21rcuc/200:15:152
168099172,6cyclictest808-21snmpd23:10:052
168099171,9cyclictest0-21swapper/222:14:272
168099171,9cyclictest0-21swapper/221:45:342
168099171,9cyclictest0-21swapper/200:28:092
168099171,7cyclictest0-21swapper/223:40:372
168099171,6cyclictest14171-21users22:00:252
168099171,6cyclictest12862-21unixbench_multi19:30:282
168099171,6cyclictest0-21swapper/222:56:082
168099171,6cyclictest0-21swapper/219:25:252
168099171,6cyclictest0-21swapper/200:11:362
168099171,5cyclictest0-21swapper/223:20:122
168099171,5cyclictest0-21swapper/222:54:262
1680991713,3cyclictest0-21swapper/220:45:132
1680991712,3cyclictest28233-21unixbench_multi20:05:232
168099171,15cyclictest0-21swapper/221:17:552
168099171,15cyclictest0-21swapper/220:22:532
168099171,14cyclictest26988-21grep23:45:152
168099171,14cyclictest0-21swapper/222:35:062
166799173,9cyclictest0-21swapper/000:00:440
166799172,12cyclictest15407-21/usr/sbin/munin23:20:120
166799171,6cyclictest1481-21grep20:20:150
1667991711,3cyclictest5357-21cat20:30:020
1667991711,3cyclictest31101-21cut20:15:020
1667991711,3cyclictest18046-21grep19:45:110
166799171,10cyclictest0-21swapper/023:16:530
168099164,3cyclictest893-21cat00:00:132
168099164,3cyclictest30970-21latency_hist20:15:012
168099163,8cyclictest19553-40dnf19:45:312
168099163,5cyclictest30-21rcuc/219:15:172
168099162,8cyclictest87350irq/126-enp1s0-22:15:192
168099162,8cyclictest87350irq/126-enp1s0-22:15:192
168099161,6cyclictest32-21ksoftirqd/200:35:182
168099161,6cyclictest16659-21expr20:55:132
168099161,6cyclictest16056-21smartctl22:05:242
168099161,5cyclictest0-21swapper/221:25:262
168099161,3cyclictest0-21swapper/219:57:282
168099161,3cyclictest0-21swapper/219:57:282
1680991611,3cyclictest21997-21/usr/sbin/munin23:35:122
168099161,12cyclictest19303-21if_enp2s021:00:162
1680991610,3cyclictest827-21runrttasks20:42:082
166799161,6cyclictest27844-21/usr/sbin/munin20:05:200
1667991613,2cyclictest7-21ksoftirqd/020:40:400
1667991612,3cyclictest7-21ksoftirqd/020:15:070
1667991612,2cyclictest7-21ksoftirqd/020:00:010
1667991612,2cyclictest7-21ksoftirqd/020:00:010
1667991611,3cyclictest24883-21df_inode20:00:130
151612161,2sleep00-21swapper/019:35:310
168099151,7cyclictest0-21swapper/220:32:542
166799151,3cyclictest10760-21sshd19:25:290
1667991510,3cyclictest7982-21tune2fs20:35:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional