You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-07 - 07:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackaslot8s.osadl.org (updated Tue Jul 07, 2026 00:45:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
119322250,6sleep21192-21unixbench_multi19:05:202
1410221696,96sleep10-21swapper/119:06:561
1483217395,24sleep00-21swapper/019:07:520
1448217198,23sleep30-21swapper/319:07:273
226892770,2sleep30-21swapper/323:35:153
45432720,2sleep10-21swapper/121:22:021
113362610,4sleep011332-21df_inode22:15:150
76652590,1sleep20-21swapper/223:24:242
76652590,1sleep20-21swapper/223:24:232
253002520,1sleep10-21swapper/123:13:061
296102510,1sleep30-21swapper/300:28:573
83652500,3sleep30-21swapper/323:25:003
83652500,3sleep30-21swapper/323:25:003
291162500,1sleep10-21swapper/122:04:411
323692460,1sleep10-21swapper/100:30:231
15972430,2chrt0-21swapper/022:31:310
11322350,3sleep228-21ksoftirqd/200:07:182
247342340,2chrt0-21swapper/122:48:581
209272320,3chrt0-21swapper/021:10:220
191999318,2cyclictest141rcu_preempt22:36:020
269342300,10sleep122-21ksoftirqd/122:26:121
1921993017,9cyclictest22851-21python321:35:272
225752290,2sleep30-21swapper/322:46:473
1919992917,7cyclictest19452-21diskmemload23:05:040
1919992915,5cyclictest12-21ksoftirqd/023:37:200
313722280,11sleep122-21ksoftirqd/122:53:531
191999288,13cyclictest12-21ksoftirqd/000:00:160
1919992822,4cyclictest16090-21apt23:55:000
1919992815,7cyclictest141rcu_preempt22:04:240
213972270,2sleep00-21swapper/000:22:240
192099271,2cyclictest0-21swapper/119:13:111
191999272,4cyclictest12-21ksoftirqd/023:30:010
1919992710,4cyclictest12-21ksoftirqd/021:47:410
192299263,17cyclictest17746-21taskset22:20:113
191999269,4cyclictest12-21ksoftirqd/022:40:540
191999269,10cyclictest12-21ksoftirqd/023:13:490
191999264,5cyclictest12-21ksoftirqd/022:49:390
1919992619,4cyclictest22820-40ogrotate23:59:590
1919992613,5cyclictest12-21ksoftirqd/000:40:000
191999261,24cyclictest0-21swapper/019:12:420
1919992612,3cyclictest12-21ksoftirqd/021:19:260
1919992611,4cyclictest12-21ksoftirqd/021:36:080
1919992610,9cyclictest12-21ksoftirqd/023:40:430
1922992523,1cyclictest0-21swapper/319:12:423
192199251,23cyclictest0-21swapper/219:13:242
1920992514,7cyclictest31036-21ntpq00:05:201
191999255,12cyclictest12-21ksoftirqd/000:09:220
191999254,8cyclictest12-21ksoftirqd/000:14:520
1919992513,4cyclictest12-21ksoftirqd/021:22:000
1919992511,7cyclictest26343-21tr22:50:120
1919992510,4cyclictest12-21ksoftirqd/022:14:330
191999250,8cyclictest3323-21/usr/sbin/munin23:20:240
191999250,8cyclictest3323-21/usr/sbin/munin23:20:230
192199242,16cyclictest16430-21ssh21:55:152
1921992415,5cyclictest1559-21sendmail23:19:592
1921992413,7cyclictest25864-21sed22:25:252
1920992415,6cyclictest23745-21perl00:00:141
191999249,5cyclictest141rcu_preempt21:52:150
191999249,5cyclictest141rcu_preempt21:52:140
191999248,9cyclictest12-21ksoftirqd/020:03:070
191999248,10cyclictest12-21ksoftirqd/021:26:300
191999247,4cyclictest12-21ksoftirqd/023:15:310
191999247,4cyclictest12-21ksoftirqd/022:29:560
191999245,5cyclictest12-21ksoftirqd/021:30:160
191999242,7cyclictest12-21ksoftirqd/021:40:140
191999242,11cyclictest12-21ksoftirqd/023:01:450
1919992417,2cyclictest141rcu_preempt23:31:430
1919992411,2cyclictest141rcu_preempt21:57:430
23262230,1sleep30-21swapper/322:32:153
1920992314,7cyclictest12035-21ntp_states22:15:211
191999239,3cyclictest12-21ksoftirqd/000:15:090
191999238,8cyclictest12-21ksoftirqd/020:56:190
191999238,4cyclictest12-21ksoftirqd/020:13:140
191999238,2cyclictest141rcu_preempt00:34:050
191999237,9cyclictest12-21ksoftirqd/019:27:480
191999235,10cyclictest12-21ksoftirqd/023:48:000
191999235,10cyclictest12-21ksoftirqd/022:55:520
1919992311,9cyclictest30497-21sh22:05:160
1919992311,4cyclictest12-21ksoftirqd/019:50:110
192199223,12cyclictest4294-21ssh00:10:122
192199223,12cyclictest29860-21apt-get22:05:112
192199222,14cyclictest13986-21sh23:05:032
1921992212,6cyclictest25669-21grep20:05:252
192099221,7cyclictest0-21swapper/123:30:131
1920992213,7cyclictest4511-21awk21:45:251
1920992212,7cyclictest14781-21smartctl23:05:151
191999229,5cyclictest12-21ksoftirqd/022:21:230
191999228,4cyclictest12-21ksoftirqd/020:18:160
191999227,8cyclictest12-21ksoftirqd/020:22:420
191999226,9cyclictest12-21ksoftirqd/020:41:230
1919992211,2cyclictest141rcu_preempt20:50:050
192299216,11cyclictest28384-1kworker/u9:122:05:113
1922992113,4cyclictest0-21swapper/321:45:043
1922992112,6cyclictest17974-21ls00:20:123
192199218,4cyclictest11060-21timerandwakeup21:50:242
192199218,4cyclictest11060-21timerandwakeup21:50:232
1921992115,4cyclictest28-21ksoftirqd/220:40:012
1921992114,3cyclictest23215-21/usr/sbin/munin00:00:172
1921992112,7cyclictest6529-21seq19:21:122
192199211,18cyclictest30470-21ssh00:29:492
1921992110,8cyclictest19318-21sed22:20:252
1921992110,3cyclictest29017-21cut23:40:132
192099211,5cyclictest16526-21apt-get23:55:121
1920992114,5cyclictest22190-21basename21:35:191
1920992113,6cyclictest7919-21ls19:25:201
1920992113,6cyclictest29653-21seq23:15:431
1920992113,6cyclictest15313-21ls21:00:001
1920992112,7cyclictest29738-21ls21:40:251
1920992112,6cyclictest14744-21apt-get21:30:131
1920992111,7cyclictest27652-21ntp_kernel_err21:15:201
191999217,4cyclictest12-21ksoftirqd/020:38:180
191999217,4cyclictest12-21ksoftirqd/019:43:370
191999217,4cyclictest12-21ksoftirqd/000:25:400
191999217,3cyclictest12-21ksoftirqd/020:09:420
191999217,2cyclictest141rcu_preempt20:45:160
191999215,9cyclictest12-21ksoftirqd/019:58:550
191999211,11cyclictest12-21ksoftirqd/019:45:380
121192210,11sleep334-21ksoftirqd/323:51:013
225462200,8sleep334-21ksoftirqd/323:59:433
192299209,5cyclictest3771-21cat20:30:243
1922992014,4cyclictest16009-21grep23:30:163
1922992014,4cyclictest13585-21sh23:04:593
1922992012,6cyclictest30252-21expr00:05:143
1922992012,6cyclictest29783-21seq23:15:513
1922992012,6cyclictest11283-21expr00:15:143
1922992011,7cyclictest4439-21sensors_temp23:45:223
1922992011,7cyclictest3246-21tune2fs19:15:153
1922992011,7cyclictest14873-21wc19:40:253
1922992011,5cyclictest1917-21cyclictest21:57:093
1922992010,8cyclictest9292-21ls20:45:153
1922992010,7cyclictest11187-21cat20:50:133
192199202,7cyclictest10983-21ssh22:15:122
192199202,13cyclictest2755-21chrt19:15:032
192199202,12cyclictest26637-21apt-get21:15:112
192199201,4cyclictest0-21swapper/222:04:302
1921992014,4cyclictest28-21ksoftirqd/220:30:012
1921992014,4cyclictest17322-21idleruntime-cro21:05:012
1921992012,6cyclictest9995-21ls23:50:132
1921992012,6cyclictest20707-21cat21:10:192
1921992012,5cyclictest4927-21latency_hist00:35:012
1921992011,6cyclictest27857-21grep22:50:242
1921992011,6cyclictest19705-21basename19:55:002
1921992010,6cyclictest18260-21vmstat23:55:292
192199200,8cyclictest28548-21cat23:40:002
192199200,4cyclictest0-21swapper/220:42:072
192199200,18cyclictest18000-21ssh21:31:582
192099201,5cyclictest0-21swapper/122:38:471
1920992013,5cyclictest4368-21seq22:57:321
1920992013,5cyclictest4313-21seq20:33:381
1920992012,6cyclictest8854-21seq00:13:181
1920992012,6cyclictest30202-21seq20:17:331
1920992012,6cyclictest17041-21ls19:45:221
1920992012,6cyclictest10937-21seq19:32:501
1920992012,5cyclictest278-21systemd-journal22:15:011
1920992011,7cyclictest7607-21memory20:40:191
1920992011,7cyclictest24847-21tune2fs00:25:171
1920992011,6cyclictest29845-21awk23:40:191
191999209,5cyclictest141rcu_preempt19:22:170
191999203,8cyclictest12-21ksoftirqd/021:02:120
191999202,4cyclictest0-21swapper/019:15:170
81652190,3sleep228-21ksoftirqd/222:36:072
192299199,5cyclictest24294-21proc_pri22:00:213
192299192,13cyclictest28517-21apt-get20:15:113
1922991914,3cyclictest11903-21sh21:27:373
1922991913,4cyclictest28706-21sh23:40:043
1922991913,4cyclictest19009-21sh23:08:343
1922991912,5cyclictest53050irq/142-enp3s0-21:30:283
1922991912,4cyclictest5536-21sh22:35:013
1922991911,6cyclictest9376-21grep19:30:133
1922991911,6cyclictest6851-21pluginstate19:25:003
1922991911,6cyclictest4961-21expr20:35:133
1922991911,6cyclictest26625-21cat21:15:103
1922991911,6cyclictest25584-21wc20:05:253
1922991911,6cyclictest17650-21cut19:50:003
1922991911,6cyclictest16168-21ls21:00:183
1922991911,6cyclictest12003-21ntpq22:15:213
1922991911,6cyclictest11661-21expr19:35:133
1922991911,6cyclictest10994-21cat21:50:243
1922991911,6cyclictest10994-21cat21:50:233
1922991910,7cyclictest5050-21cut00:10:183
1922991910,7cyclictest4498-21seq22:57:403
1922991910,7cyclictest22439-21diskstats20:00:143
1922991910,6cyclictest7119-21cat20:40:133
192199199,7cyclictest560-21dbus-daemon22:35:002
192199197,9cyclictest560-21dbus-daemon20:59:592
192199192,13cyclictest17688-21ssh00:20:112
192199192,12cyclictest2017-21ssh21:20:192
192199192,11cyclictest3101-21apt-get23:45:112
1921991911,5cyclictest20433-21grep22:45:202
1921991910,6cyclictest29835-21timerwakeupswit20:15:232
1921991910,6cyclictest13912-21grep19:40:152
192099197,9cyclictest13846-21cat23:05:011
192099192,12cyclictest9623-21sh21:50:121
192099192,12cyclictest9623-21sh21:50:121
192099192,12cyclictest2005-21ssh23:20:131
192099192,12cyclictest2005-21ssh23:20:121
1920991914,3cyclictest24412-21cstates20:05:121
1920991914,3cyclictest11943-21hwlatdetect19:35:161
1920991913,4cyclictest25563-21sh23:37:081
1920991913,4cyclictest23522-21sh21:12:191
1920991913,4cyclictest19987-21sh00:20:591
1920991913,4cyclictest16371-21/usr/sbin/munin21:55:141
1920991912,5cyclictest32661-21sed20:25:001
1920991912,5cyclictest28214-21seq20:14:311
1920991912,5cyclictest19403-21seq21:09:421
1920991912,4cyclictest636-21runrttasks23:45:141
1920991912,4cyclictest10297-21sh21:26:021
1920991911,5cyclictest52950irq/141-enp3s0-23:25:141
1920991911,5cyclictest278-21systemd-journal22:34:591
192099191,14cyclictest12543-21ssh22:40:121
1920991910,7cyclictest10915-21pmu-power23:50:211
191999197,4cyclictest12-21ksoftirqd/021:07:250
191999197,4cyclictest12-21ksoftirqd/019:39:280
191999196,2cyclictest141rcu_preempt19:31:390
191999193,6cyclictest12-21ksoftirqd/020:30:320
1919991910,6cyclictest986-21idleruntime20:25:170
192299189,7cyclictest21060-21grep23:10:123
192299189,6cyclictest31668-21expr00:30:173
192299182,6cyclictest4154-21ssh22:10:083
1922991813,3cyclictest5183-21sh00:35:023
1922991813,3cyclictest21259-21sh21:35:003
1922991813,3cyclictest15992-21sh22:42:113
1922991812,4cyclictest8697-21sh23:25:123
1922991812,4cyclictest30832-21sh21:41:173
1922991812,4cyclictest27826-21sh22:27:053
1922991811,5cyclictest7329-21cat19:25:143
1922991811,5cyclictest32004-21ls20:20:263
1922991810,6cyclictest561-21cat20:25:143
1922991810,6cyclictest32541-21basename22:55:013
1922991810,6cyclictest19999-21awk19:55:153
1922991810,6cyclictest19897-21apt-get21:10:133
1922991810,6cyclictest17539-21cat21:05:113
1922991810,6cyclictest1748-21grep21:20:183
1922991810,5cyclictest19452-21diskmemload00:00:103
192199189,7cyclictest13323-21cut19:40:012
192199181,7cyclictest14141-21unixbench_multi22:40:252
1921991813,3cyclictest8835-21sh22:13:192
1921991812,4cyclictest8312-21sh21:25:142
1921991812,4cyclictest20867-21/usr/sbin/munin23:10:242
1921991812,4cyclictest1644-21uname22:55:252
1921991811,5cyclictest23982-21sh20:04:592
1921991811,5cyclictest20152-21expr19:55:142
1921991811,5cyclictest17024-21sed19:45:232
192199181,13cyclictest11377-21ssh00:15:142
1921991810,6cyclictest18974-21awk21:05:252
192099181,7cyclictest683-21ssh22:06:551
1920991812,4cyclictest10860-21sh00:15:041
1920991811,5cyclictest3368-21idleruntime19:15:151
1920991811,5cyclictest22010-21date20:00:001
1920991811,5cyclictest18057-21expr22:20:121
1920991811,5cyclictest14685-21sensors19:40:221
192099181,15cyclictest0-21swapper/100:35:161
192099181,12cyclictest22108-21taskset20:00:131
1920991810,6cyclictest16112-21expr21:00:181
1920991810,6cyclictest13143-21cut20:55:011
1920991810,6cyclictest1313-21ntp_states20:25:211
1920991810,6cyclictest10492-21awk20:45:261
1922991710,5cyclictest18825-21ps19:50:213
1922991710,5cyclictest14089-21needreboot20:55:213
192199179,5cyclictest12470-21timerwakeupswit20:50:252
192199177,7cyclictest15853-21cat23:30:162
192199173,13cyclictest9373-21head23:25:182
192199172,10cyclictest2651-21apt-get21:45:032
1921991712,3cyclictest7067-21/usr/sbin/munin19:25:192
1921991712,3cyclictest19452-21diskmemload21:43:422
1921991712,3cyclictest10944-21sh23:02:262
1921991711,4cyclictest636-21runrttasks00:38:282
1921991710,5cyclictest4524-21sed20:35:002
1920991712,3cyclictest6265-21timerwakeupswit19:20:261
1920991710,5cyclictest19361-21seq19:50:571
192199169,5cyclictest28-21ksoftirqd/220:45:242
192199168,6cyclictest9807-21cat19:30:162
1921991611,3cyclictest30656-21/usr/sbin/munin20:20:212
192299157,6cyclictest26546-21cat20:10:123
192199156,7cyclictest26376-21apt20:10:112
192099157,5cyclictest5506-21cat20:35:191
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional