You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-13 - 00:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Wed Aug 12, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1505629468,8sleep20-21swapper/207:06:302
1499929465,9sleep00-21swapper/007:05:450
1515928569,12sleep30-21swapper/307:07:483
1503028272,6sleep10-21swapper/107:06:101
309042660,0sleep00-21swapper/011:37:090
15419992827,1cyclictest0-21swapper/311:31:593
15419992826,1cyclictest39-21ksoftirqd/311:52:043
1541399280,27cyclictest0-21swapper/109:17:261
1541699260,26cyclictest0-21swapper/211:44:012
1541999240,2cyclictest0-21swapper/310:09:033
15413992423,1cyclictest38150irq/131-snd_hda12:16:471
15413992422,1cyclictest0-21swapper/111:21:571
15413992422,1cyclictest0-21swapper/111:21:561
1541999232,19cyclictest846-21seq09:43:443
1541699230,23cyclictest0-21swapper/209:48:142
1541399235,18cyclictest0-21swapper/110:58:071
1541399233,19cyclictest0-21swapper/109:46:251
15413992321,1cyclictest0-21swapper/111:15:131
15413992319,4cyclictest0-21swapper/110:30:001
1541999223,0cyclictest141rcu_preempt11:28:123
15419992221,1cyclictest0-21swapper/309:14:483
15419992219,2cyclictest39-21ksoftirqd/309:15:193
1541999220,21cyclictest0-21swapper/310:45:023
1541699220,22cyclictest0-21swapper/211:49:052
1541699220,22cyclictest0-21swapper/209:36:052
1541399229,9cyclictest225-21systemd-journal09:25:431
1541399229,10cyclictest0-21swapper/110:54:021
15413992214,8cyclictest0-21swapper/111:40:161
1541399220,2cyclictest0-21swapper/112:20:181
15419992117,1cyclictest39-21ksoftirqd/310:15:433
15413992119,1cyclictest0-21swapper/111:50:381
15413992115,3cyclictest225-21systemd-journal10:34:221
15413992112,7cyclictest13492-21df_abs10:10:151
15413992110,11cyclictest0-21swapper/111:27:141
15409992118,2cyclictest29959-21ssh12:35:130
120032210,0sleep00-21swapper/010:47:360
1541999201,18cyclictest0-21swapper/310:20:263
1541999200,1cyclictest141rcu_preempt12:10:293
1541399201,8cyclictest0-21swapper/108:30:131
15413992018,1cyclictest8429-21diskmemload11:06:571
15413992018,1cyclictest8429-21diskmemload10:17:261
15413992016,4cyclictest8429-21diskmemload11:58:291
15413992015,4cyclictest12256-21ssh12:05:161
15413992012,8cyclictest0-21swapper/109:38:511
15413992012,4cyclictest8429-21diskmemload12:11:531
1541399201,18cyclictest31056-21ntp_kernel_err10:20:231
1541699190,0cyclictest141rcu_preempt09:55:052
1541399199,10cyclictest0-21swapper/109:33:381
15413991916,2cyclictest8429-21diskmemload09:59:281
15413991915,3cyclictest19097-21date11:50:011
15413991911,8cyclictest8429-21diskmemload12:27:021
1541399190,1cyclictest0-21swapper/111:03:141
15409991915,1cyclictest18338-21ssh11:30:130
15419991814,3cyclictest39-21ksoftirqd/311:35:133
1541999180,0cyclictest0-21swapper/309:58:473
15416991817,1cyclictest0-21swapper/208:25:522
1541699180,17cyclictest0-21swapper/212:19:352
1541399183,12cyclictest25-21ksoftirqd/111:39:171
1540999185,7cyclictest8007-21apache210:36:110
15419991716,1cyclictest39-21ksoftirqd/311:45:273
15416991717,0cyclictest0-21swapper/209:16:322
15416991712,4cyclictest16722-21sh09:52:562
1541699171,1cyclictest141rcu_preempt09:25:452
15416991711,2cyclictest32-21ksoftirqd/211:37:402
1541399178,7cyclictest0-21swapper/110:43:501
15413991715,1cyclictest0-21swapper/111:33:081
15413991714,2cyclictest0-21swapper/109:50:111
15413991714,2cyclictest0-21swapper/109:40:491
15413991712,4cyclictest8796-21awk10:45:231
15413991711,5cyclictest8662-21ssh10:07:031
1540999170,3cyclictest141rcu_preempt10:10:320
1541999168,1cyclictest141rcu_preempt11:02:173
1541999164,1cyclictest141rcu_preempt12:03:273
15419991611,3cyclictest39-21ksoftirqd/312:15:353
1541999161,12cyclictest39-21ksoftirqd/309:22:183
1541999160,4cyclictest0-21swapper/311:21:023
1541999160,4cyclictest0-21swapper/311:21:023
1541699161,6cyclictest14911-21cpuspeed_turbos08:15:162
15416991614,2cyclictest32-21ksoftirqd/209:30:152
15416991614,1cyclictest32-21ksoftirqd/211:50:122
1541699161,0cyclictest141rcu_preempt11:19:182
1541699160,15cyclictest26733-21sh12:13:452
1541399165,10cyclictest0-21swapper/107:20:011
15413991614,2cyclictest25-21ksoftirqd/107:39:441
15413991614,2cyclictest0-21swapper/112:31:251
15413991614,1cyclictest31062-21perf08:50:011
15413991610,5cyclictest5667-21unixbench-2d12:00:281
1540999162,7cyclictest1458-21munin-run09:25:000
15409991610,4cyclictest22047-21turbostat10:34:590
1540999160,14cyclictest0-21swapper/011:40:140
1541999157,5cyclictest39-21ksoftirqd/310:27:533
15419991511,3cyclictest381rcuc/310:50:183
1541699156,6cyclictest32-21ksoftirqd/212:35:192
15416991513,2cyclictest0-21swapper/210:02:382
1541699151,2cyclictest0-21swapper/210:29:502
1541699150,1cyclictest0-21swapper/210:10:372
1541699150,0cyclictest0-21swapper/212:22:402
1541399158,6cyclictest0-21swapper/108:00:191
1541399157,7cyclictest0-21swapper/110:04:011
1541399157,7cyclictest0-21swapper/109:10:361
1541399152,4cyclictest25-21ksoftirqd/112:39:491
15413991513,2cyclictest8429-21diskmemload11:10:471
15413991513,2cyclictest0-21swapper/108:20:141
15413991512,1cyclictest8429-21diskmemload10:36:531
1540999153,11cyclictest13230-21apt-get12:25:120
1540999150,9cyclictest738-21perf11:19:590
1540999150,9cyclictest1181-21/usr/sbin/munin11:20:160
1540999150,9cyclictest1181-21/usr/sbin/munin11:20:150
1541999142,11cyclictest0-21swapper/310:45:003
1541999142,10cyclictest7181-21ssh12:20:433
1541999141,6cyclictest39-21ksoftirqd/312:25:083
1541999141,4cyclictest381rcuc/311:42:143
15419991412,1cyclictest381rcuc/310:55:213
1541999140,1cyclictest25516-21ssh09:39:123
1541999140,1cyclictest141rcu_preempt11:19:513
1541699146,3cyclictest537-21snmpd11:09:292
15416991413,1cyclictest0-21swapper/209:06:382
15416991412,1cyclictest32-21ksoftirqd/209:13:252
15416991412,1cyclictest18945-21irqstats07:15:182
1541699140,3cyclictest0-21swapper/207:55:112
1541699140,1cyclictest141rcu_preempt11:59:592
1541699140,1cyclictest0-21swapper/210:24:042
1541699140,1cyclictest0-21swapper/210:19:382
15413991412,1cyclictest16780-21munin-run08:20:001
15413991411,2cyclictest13310-21grep08:10:221
15409991414,0cyclictest0-21swapper/011:00:150
15409991411,2cyclictest18196-21ls09:35:000
15409991410,2cyclictest31596-21cut08:50:140
1540999140,13cyclictest0-21swapper/011:50:490
1541999139,3cyclictest7919-21grep09:05:253
1541999131,4cyclictest0-21swapper/309:28:113
15419991311,2cyclictest39-21ksoftirqd/308:20:013
15419991310,2cyclictest39-21ksoftirqd/308:40:013
15419991310,2cyclictest31603-21sed07:40:253
15419991310,2cyclictest30595-21sed08:45:253
15419991310,2cyclictest28652-21tr10:00:143
1541999130,13cyclictest0-21swapper/311:08:253
1541699134,4cyclictest225-21systemd-journal07:50:002
15416991312,1cyclictest0-21swapper/211:30:122
15416991310,2cyclictest9369-21ssh10:45:332
15416991310,2cyclictest6053-21cat09:05:012
15416991310,2cyclictest32-21ksoftirqd/211:30:012
15416991310,2cyclictest23897-21cron08:34:592
1541399139,3cyclictest27918-21grep08:40:241
1541399139,3cyclictest27918-21grep08:40:231
1541399137,5cyclictest27394-21latency_hist07:34:591
15413991311,1cyclictest3584-21cron09:00:011
15413991310,2cyclictest27202-21grep09:20:201
15413991310,2cyclictest24495-21grep07:25:271
1540999139,3cyclictest31047-21idleruntime-cro08:50:010
1540999139,3cyclictest18272-21grep07:15:140
1540999139,3cyclictest11013-21grep09:10:260
1540999139,2cyclictest16758-21cron08:20:000
1540999139,2cyclictest11797-21cut09:50:130
1540999138,3cyclictest1526-21taskset11:58:400
1540999132,9cyclictest537-21snmpd12:05:160
15409991310,2cyclictest5516-21grep09:00:250
15409991310,2cyclictest3871-21grep09:25:250
15409991310,2cyclictest20185-21tail08:25:190
15409991310,2cyclictest17218-21wc07:10:260
15409991310,2cyclictest14182-21perf08:15:000
1540999130,13cyclictest0-21swapper/009:38:250
1541999129,2cyclictest5141-21grep07:55:143
1541999129,2cyclictest24508-21grep07:25:273
1541999129,2cyclictest17263-21sed07:10:273
1541999129,2cyclictest17124-21tr11:10:133
1541999128,2cyclictest20295-21sh10:33:393
1541999127,4cyclictest12957-21apt-get10:10:133
1541999124,1cyclictest141rcu_preempt07:38:443
1541999122,4cyclictest39-21ksoftirqd/310:36:213
1541999120,6cyclictest0-21swapper/309:45:063
1541699129,2cyclictest28298-21head07:35:162
1541699129,2cyclictest22891-21tr10:35:152
1541699129,2cyclictest14919-21grep10:30:162
1541699128,2cyclictest32588-21sed10:40:202
15416991210,2cyclictest32-21ksoftirqd/209:23:582
15416991210,1cyclictest311rcuc/211:00:152
1541699120,11cyclictest14285-21uname12:25:182
1541399129,2cyclictest25807-21grep08:35:261
1541399129,2cyclictest21081-21grep08:25:281
1541399126,5cyclictest0-21swapper/107:45:001
1541399124,6cyclictest0-21swapper/107:20:151
15413991210,1cyclictest0-21swapper/107:10:141
1540999129,2cyclictest4955-21grep12:20:140
1540999129,2cyclictest31706-21tail10:20:290
1540999129,2cyclictest25902-21head07:30:170
1540999129,2cyclictest25307-21cut10:55:200
1540999129,2cyclictest21795-21ssh12:10:220
1540999129,2cyclictest18438-21sh09:15:160
1540999129,2cyclictest17596-21cron11:30:010
1540999129,2cyclictest10824-21tr08:05:220
1540999128,2cyclictest20714-21grep09:55:170
1540999121,6cyclictest6674-21perf10:44:590
15409991210,1cyclictest29669-21head12:15:170
1541999119,1cyclictest10941-21grep08:05:223
1541999118,3cyclictest9083-21kworker/u8:008:34:413
1541999118,2cyclictest30117-21ssh11:55:413
1541999118,2cyclictest27114-21taskset07:32:583
1541999118,2cyclictest2251-21sed08:55:183
1541999118,2cyclictest21524-21cron08:30:003
1541999118,2cyclictest19122-21ssh09:54:483
1541999118,2cyclictest1513-21grep07:45:283
1541999118,2cyclictest13472-21cat08:10:253
1541999118,2cyclictest12953-21cut12:05:203
1541999117,3cyclictest39-21ksoftirqd/312:31:023
1541999110,10cyclictest1039-21taskset08:52:583
1541699118,2cyclictest8960-21sed08:00:282
1541699118,2cyclictest8723-21ssh12:02:422
1541699118,2cyclictest4552-21ssh10:05:152
1541699118,2cyclictest32052-21munin-plugin-st07:45:002
1541699118,2cyclictest31911-21expr08:50:142
1541699118,2cyclictest30335-21python08:45:242
1541699118,2cyclictest25397-21grep08:35:212
1541699118,2cyclictest24879-21grep10:55:172
1541699118,2cyclictest24767-21taskset07:28:222
1541699118,2cyclictest1840-21ssh09:44:302
1541699118,2cyclictest17223-21ssh12:07:482
1541699118,2cyclictest1097-21ssh11:20:072
1541699118,2cyclictest1097-21ssh11:20:062
1541699118,2cyclictest10032-21cat08:05:152
1541699117,3cyclictest17540-21fschecks_count11:10:172
1541699112,1cyclictest141rcu_preempt10:50:432
1541699110,1cyclictest141rcu_preempt12:30:212
1541399118,2cyclictest9905-21sort08:05:161
1541399118,2cyclictest5083-21tr07:55:131
1541399118,2cyclictest4656-21cut09:00:161
1541399118,2cyclictest1147-21wc07:45:231
1540999118,2cyclictest8944-21ssh09:48:260
1540999118,2cyclictest6823-21wc10:25:210
1540999118,2cyclictest3288-21taskset08:56:180
1540999118,2cyclictest30233-21cstates07:40:140
1540999118,2cyclictest29649-21cron07:40:000
1540999118,2cyclictest28668-21cron08:45:000
1540999118,2cyclictest28668-21cron08:44:590
1540999118,2cyclictest27908-21sed09:40:170
1540999118,2cyclictest23731-21uname07:25:200
1540999118,2cyclictest22841-21cut10:15:220
1540999118,2cyclictest22497-21head12:30:180
1540999118,2cyclictest17462-21sh10:50:270
1540999118,2cyclictest12303-21ssh10:09:530
1540999118,2cyclictest10139-21ssh11:05:250
1540999117,3cyclictest18740-21sed11:10:260
1540999114,2cyclictest28661-21aten2_r7power_p10:00:150
1540999110,10cyclictest15168-21ssh11:47:020
1541999108,2cyclictest5586-21awk09:00:253
1541999108,2cyclictest4518-21taskset07:54:153
1541999108,1cyclictest9330-21taskset08:04:013
1541999108,1cyclictest32279-21taskset12:35:413
1541999108,1cyclictest286252sleep308:44:333
1541999108,1cyclictest286252sleep308:44:333
1541999107,2cyclictest11865-21wc09:30:233
1541999106,3cyclictest381rcuc/308:20:183
1541999103,5cyclictest39-21ksoftirqd/307:20:183
1541699108,1cyclictest3444-21chrt08:58:182
1541699108,1cyclictest28037-21threads08:40:272
1541699108,1cyclictest28037-21threads08:40:262
1541699108,1cyclictest25365-21cpuspeed_turbos07:30:142
1541699108,1cyclictest21417-21cat07:20:202
1541699107,2cyclictest16779-21tr07:10:212
1541699106,3cyclictest18675-21grep08:20:282
1541399108,1cyclictest9882sleep108:52:241
1541399108,1cyclictest6758-21sed09:05:151
1540999108,1cyclictest23539-21chrt08:30:270
1540999108,1cyclictest21737-21grep07:20:220
1540999108,1cyclictest1880-21chrt07:48:350
1540999107,2cyclictest8080-21taskset09:05:480
1540999107,2cyclictest7340-21aten2_r7power_p08:00:160
1540999107,2cyclictest5941-21grep07:55:190
1540999107,2cyclictest4419-21ssh12:00:170
1540999107,2cyclictest25277-21head08:35:190
1540999107,2cyclictest18667-21grep08:20:270
154199998,1cyclictest19049-21missed_timers07:15:183
154169997,1cyclictest12875-21basename08:10:182
154169990,5cyclictest141rcu_preempt07:50:142
154139996,2cyclictest4604-21awk07:55:001
154099996,2cyclictest3184-21grep07:50:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional