You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-15 - 18:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Fri May 15, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1818129584,6sleep00-21swapper/007:09:130
1801629380,9sleep20-21swapper/207:07:062
1800729265,9sleep30-21swapper/307:06:593
1810228265,13sleep10-21swapper/107:08:131
18317992523,1cyclictest32-21ksoftirqd/210:33:292
18321992322,1cyclictest0-21swapper/311:01:063
18316992322,1cyclictest0-21swapper/110:57:111
18317992220,1cyclictest32-21ksoftirqd/212:28:252
1832199210,20cyclictest0-21swapper/309:55:003
18317992119,1cyclictest32-21ksoftirqd/210:00:292
1832199201,17cyclictest0-21swapper/312:08:243
1832199200,1cyclictest0-21swapper/309:12:413
18317992016,4cyclictest32-21ksoftirqd/210:47:172
18317992014,1cyclictest32-21ksoftirqd/211:10:582
1831799200,3cyclictest0-21swapper/210:19:122
18316992013,4cyclictest221irq_work/112:12:531
1831599206,7cyclictest5201-21munin-run07:50:010
1831799193,1cyclictest141rcu_preempt10:28:232
18317991916,2cyclictest32-21ksoftirqd/210:45:002
18317991913,2cyclictest32-21ksoftirqd/211:20:332
18317991911,4cyclictest141rcu_preempt12:10:192
18316991917,2cyclictest0-21swapper/110:47:541
1831699191,0cyclictest141rcu_preempt12:30:071
1831699190,9cyclictest225-21systemd-journal09:04:591
1831699190,4cyclictest20871-21cpu09:15:141
1831699190,1cyclictest0-21swapper/109:11:081
1831699190,18cyclictest0-21swapper/110:01:341
18315991919,0cyclictest0-21swapper/010:03:270
1831599190,18cyclictest0-21swapper/012:31:230
1832199182,1cyclictest0-21swapper/311:24:283
1832199182,14cyclictest13270-21apt-get09:30:133
1832199180,1cyclictest0-21swapper/312:21:133
1832199180,1cyclictest0-21swapper/311:59:063
18317991815,2cyclictest32-21ksoftirqd/211:50:212
18317991811,3cyclictest32-21ksoftirqd/209:37:062
1831799180,17cyclictest0-21swapper/211:39:412
1831699184,1cyclictest11564-21diskmemload10:10:561
18316991817,1cyclictest25-21ksoftirqd/109:25:121
18316991816,1cyclictest25-21ksoftirqd/111:22:191
18315991818,0cyclictest0-21swapper/012:37:580
1831599180,5cyclictest0-21swapper/010:12:430
18321991717,0cyclictest0-21swapper/309:19:523
1831799177,6cyclictest32-21ksoftirqd/211:48:262
1831799172,15cyclictest0-21swapper/211:04:542
1831699174,12cyclictest11564-21diskmemload09:30:021
18316991717,0cyclictest0-21swapper/110:39:041
18316991716,1cyclictest0-21swapper/111:53:531
18316991715,1cyclictest25-21ksoftirqd/111:07:221
18316991713,2cyclictest0-21swapper/112:24:031
18316991711,4cyclictest712-21sed10:20:161
1831699171,0cyclictest141rcu_preempt11:42:371
1831699170,16cyclictest0-21swapper/110:25:141
1831699170,12cyclictest0-21swapper/110:15:121
1832199162,11cyclictest0-21swapper/309:35:133
1831799169,2cyclictest32-21ksoftirqd/209:41:292
18317991613,2cyclictest32-21ksoftirqd/208:15:012
18317991612,2cyclictest32-21ksoftirqd/209:18:192
1831799161,1cyclictest141rcu_preempt09:26:572
1831699162,2cyclictest25-21ksoftirqd/111:01:401
1831699160,16cyclictest0-21swapper/109:48:531
1832199150,14cyclictest0-21swapper/309:25:153
18317991513,2cyclictest32-21ksoftirqd/210:54:232
1831799150,6cyclictest0-21swapper/207:45:142
1831799150,14cyclictest0-21swapper/212:36:112
18316991514,1cyclictest11564-21diskmemload10:34:541
1831699150,1cyclictest0-21swapper/111:27:311
1831699150,15cyclictest0-21swapper/110:41:301
18315991511,3cyclictest19392-21turbostat.cron11:30:010
1831599150,1cyclictest0-21swapper/010:15:280
189942140,0chrt0-21swapper/310:50:193
1832199144,4cyclictest5339-21ssh12:00:133
18321991412,1cyclictest24377-21kworker/u8:112:15:373
18321991411,2cyclictest13087-21tr08:05:153
1831799149,2cyclictest32-21ksoftirqd/211:26:132
1831799149,2cyclictest32-21ksoftirqd/209:53:482
1831799146,7cyclictest31720-21apache209:56:272
1831799142,2cyclictest0-21swapper/209:11:262
18317991410,3cyclictest31789-21wc07:35:202
1831799140,7cyclictest0-21swapper/209:32:112
1831799140,12cyclictest0-21swapper/210:22:442
1831699142,4cyclictest18761-21/usr/sbin/munin11:10:181
18316991410,3cyclictest38150irq/131-snd_hda11:49:591
1831599148,4cyclictest13-21ksoftirqd/011:19:580
1831599142,6cyclictest21075-21munin-run09:34:590
1831599142,10cyclictest0-21swapper/010:25:000
18315991410,3cyclictest12853-21grep11:45:140
1832199139,3cyclictest15990-21ls10:10:153
1832199139,2cyclictest30014-21cut08:40:133
1832199136,2cyclictest26625-21head10:35:233
1832199131,2cyclictest28330-21ssh12:33:013
18321991310,2cyclictest29846-21head09:20:183
18321991310,2cyclictest26985-21grep10:55:163
18321991310,2cyclictest24305-21grep07:20:183
18321991310,2cyclictest23538-21cat08:25:193
18321991310,2cyclictest18812-21ssh09:52:143
18321991310,2cyclictest1176-21wc08:45:213
18321991310,2cyclictest1118-21grep10:20:183
1832199130,2cyclictest0-21swapper/311:43:113
1832199130,13cyclictest0-21swapper/312:25:473
1832199130,13cyclictest0-21swapper/311:36:483
1831799131,8cyclictest225-21systemd-journal08:19:432
18317991312,1cyclictest0-21swapper/212:06:492
18317991312,1cyclictest0-21swapper/212:02:042
18317991312,1cyclictest0-21swapper/210:57:082
18317991310,2cyclictest32-21ksoftirqd/207:20:232
18317991310,2cyclictest30138-21grep11:55:172
18317991310,2cyclictest14456-21grep08:05:252
18317991310,2cyclictest12203-21wc11:05:252
1831699139,3cyclictest5523-21grep07:50:131
1831699139,3cyclictest15423-21cut08:10:121
1831699139,3cyclictest14916-21cron08:10:001
1831699138,4cyclictest6904-21lockfile-remove12:40:001
1831699137,1cyclictest141rcu_preempt12:16:371
1831699131,1cyclictest141rcu_preempt12:25:221
18316991311,2cyclictest11564-21diskmemload10:54:491
18316991311,2cyclictest11564-21diskmemload09:43:021
18316991311,1cyclictest25-21ksoftirqd/111:35:471
18316991310,2cyclictest6779-21dump-pmu-power09:00:011
18316991310,2cyclictest30657-21grep08:40:161
18316991310,2cyclictest30152-21sed11:55:171
1831699130,0cyclictest0-21swapper/109:54:411
1831599139,3cyclictest17900-21tr10:30:200
18315991311,2cyclictest30563-21strings11:55:190
18315991311,2cyclictest13-21ksoftirqd/012:29:000
18315991310,2cyclictest4591-21cat08:55:000
18315991310,2cyclictest32397-21grep12:15:280
18315991310,2cyclictest30961-21grep07:35:150
18315991310,2cyclictest26078-21head10:35:170
18315991310,2cyclictest23299-21grep08:25:170
18315991310,2cyclictest22412-21wc11:50:230
18315991310,2cyclictest10138-21dpkg08:00:000
1831599130,0cyclictest0-21swapper/010:06:270
201372120,0sleep00-21swapper/010:50:390
1832199129,2cyclictest32044-21cut10:00:173
1832199129,2cyclictest31600-21grep12:35:133
1832199129,2cyclictest31305-21pmu-power09:40:213
1832199129,2cyclictest28932-21wc08:35:243
1832199129,2cyclictest20457-21ls11:50:003
1832199129,2cyclictest15372-21chrt11:07:413
1832199129,2cyclictest12569-21basename08:05:003
1832199129,2cyclictest10922-21tr10:45:213
1832199128,3cyclictest20964-21sed08:20:183
1832199127,1cyclictest0-21swapper/309:48:443
1832199120,11cyclictest7117-21sed07:50:253
1831799129,2cyclictest32264-21wc08:45:002
1831799129,2cyclictest29907-21sed07:30:262
1831799129,2cyclictest29609-21grep09:20:172
1831799129,2cyclictest2836-21wc11:20:002
1831799129,2cyclictest19661-21grep07:10:192
1831799129,2cyclictest11192-21sed08:00:162
1831799129,1cyclictest141rcu_preempt11:32:352
1831799128,4cyclictest32-21ksoftirqd/210:05:572
1831799128,3cyclictest7555-21cron07:55:002
1831799128,3cyclictest15653-21cut10:10:142
1831799127,4cyclictest32-21ksoftirqd/211:40:152
1831799127,3cyclictest25270-21grep10:35:142
1831799126,2cyclictest141rcu_preempt12:20:122
1831799124,4cyclictest32-21ksoftirqd/209:45:192
1831799121,7cyclictest9175-21idleruntime-cro09:04:592
18317991211,1cyclictest32-21ksoftirqd/212:32:012
18317991210,1cyclictest8600-21cut07:55:172
1831699129,2cyclictest7500-21grep10:05:141
1831699129,2cyclictest29017-21cat11:15:271
1831699128,3cyclictest22407-21grep09:35:161
1831699121,1cyclictest0-21swapper/107:15:141
1831599129,2cyclictest6163-21head07:50:150
1831599129,2cyclictest4495-21grep11:40:150
1831599129,2cyclictest31878-21sed09:40:240
1831599129,2cyclictest28299-21sed08:35:170
1831599129,2cyclictest27161-21wc08:35:000
1831599129,2cyclictest26675-21head07:25:170
1831599129,2cyclictest2079-21grep10:40:190
1831599129,2cyclictest20163-21tr11:30:140
1831599129,2cyclictest19613-21cat07:10:190
1831599129,2cyclictest1428-21grep07:40:150
1831599129,2cyclictest13544-21grep08:05:160
1831599128,2cyclictest2218-21wc08:50:010
1831599124,6cyclictest0-21swapper/010:25:120
1831599122,2cyclictest0-21swapper/009:45:140
1831599120,3cyclictest0-21swapper/009:37:010
1831599120,10cyclictest0-21swapper/009:28:000
1832199119,1cyclictest7047-21sh10:05:073
1832199118,2cyclictest8568-21tail07:55:173
1832199118,2cyclictest6284-21grep08:55:253
1832199118,2cyclictest4162-21cat07:45:193
1832199118,2cyclictest28074-21ssh12:13:373
1832199118,2cyclictest27600-21cat11:15:153
1832199118,2cyclictest26501-21ssh11:34:033
1832199118,2cyclictest26146-21grep07:25:133
1832199118,2cyclictest24754-21head10:15:173
1832199118,2cyclictest23574-21rm11:50:543
1832199118,2cyclictest23339-21ssh11:12:253
1832199118,2cyclictest19619-21cat07:10:193
1832199118,2cyclictest19468-21sed08:15:263
1832199118,2cyclictest17411-21grep10:30:163
1832199118,2cyclictest12420-21head11:25:173
1832199118,2cyclictest11170-21rm10:26:093
1832199111,7cyclictest30675-21/usr/sbin/munin07:35:123
1832199110,8cyclictest9172-21cron09:04:593
1832199110,10cyclictest17279-21seq08:12:113
1831799118,2cyclictest26543-21cut07:25:152
1831799118,1cyclictest32-21ksoftirqd/208:35:162
1831799113,7cyclictest32-21ksoftirqd/212:15:172
1831799113,4cyclictest225-21systemd-journal07:19:592
1831799110,10cyclictest6318-21sed08:55:252
1831699119,1cyclictest21770-21grep11:30:261
1831699118,2cyclictest9994-21ls08:00:011
1831699118,2cyclictest7497-21ssh12:00:351
1831699118,2cyclictest30812-21cut07:35:151
1831699118,2cyclictest30294-21taskset07:33:411
1831699118,2cyclictest27982-21ssh09:57:541
1831699118,2cyclictest26673-21cut07:25:161
1831699118,2cyclictest25552-21cut08:30:151
1831699118,2cyclictest23437-21if_err_enp0s31f08:25:181
1831699118,2cyclictest20632-21ssh12:09:191
1831699118,2cyclictest19206-21cut07:10:161
1831699116,3cyclictest9579-21grep09:05:151
1831599119,1cyclictest24739-21sed09:55:260
1831599118,2cyclictest9832-21expr09:05:150
1831599118,2cyclictest9359-21ssh12:21:110
1831599118,2cyclictest9045-21ssh11:04:160
1831599118,2cyclictest6060-21tune2fs12:00:170
1831599118,2cyclictest4611-21wc11:20:210
1831599118,2cyclictest31864-21ssh09:21:050
1831599118,2cyclictest30095-21grep11:35:240
1831599118,2cyclictest30013-21grep08:40:130
1831599118,2cyclictest26087-21ssh10:55:090
1831599118,2cyclictest24167-21cut07:20:160
1831599118,2cyclictest23528-21ssh09:16:070
1831599118,2cyclictest21802-21grep07:15:170
1831599118,2cyclictest21776-21ssh12:10:020
1831599118,2cyclictest20261-21cut08:20:140
1831599118,2cyclictest19013-21grep08:15:210
1831599118,2cyclictest18960-21ssh09:14:160
1831599118,2cyclictest17491-21ssh09:51:130
1831599118,2cyclictest14839-21wc12:05:190
1831599118,2cyclictest11327-21grep11:05:170
1831599118,2cyclictest10453-21ssh10:45:170
1831599117,3cyclictest7292-21tr09:00:150
1831599110,1cyclictest0-21swapper/011:10:510
1832199108,1cyclictest9908-21expr09:05:163
1832199108,1cyclictest815-21cut07:40:133
1832199108,1cyclictest25229-21cat08:30:153
1832199108,1cyclictest1049-21ssh10:40:083
1832199107,2cyclictest29664-21taskset07:30:243
1832199107,2cyclictest2677-21expr08:50:143
1832199107,2cyclictest21433-21cut07:15:153
1831799108,1cyclictest24377-21taskset08:26:282
1831799107,2cyclictest26183-21grep08:30:222
1831799107,2cyclictest21192-21cat08:20:202
1831799107,2cyclictest1480-21grep07:40:152
1831799107,2cyclictest1296-21grep08:45:232
1831799102,4cyclictest225-21systemd-journal08:54:032
1831799100,6cyclictest11514-21sshd09:09:422
1831699108,1cyclictest4917-21taskset07:46:411
1831699108,1cyclictest25472sleep107:41:431
1831699108,1cyclictest20998-21basename08:20:181
1831699108,1cyclictest196702sleep108:17:171
1831699107,2cyclictest314-21ssh09:21:481
1831699107,2cyclictest28670-21sh08:35:211
1831699107,2cyclictest2785-21df08:50:141
1831699107,2cyclictest11432-21sed08:00:181
1831699100,9cyclictest24481-21cut07:20:191
1831599108,1cyclictest16995-21date08:10:250
1831599107,2cyclictest28548-21grep07:30:150
1831599107,2cyclictest10663-21cut08:00:140
183169990,8cyclictest0-21swapper/108:45:181
183159998,1cyclictest4723-21grep08:55:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional