You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-20 - 02:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Fri Mar 20, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
21508210057,39sleep30-21swapper/319:08:093
2148028774,9sleep00-21swapper/019:07:450
2155428371,8sleep20-21swapper/219:08:452
2159828159,6sleep10-21swapper/119:09:191
148382550,0sleep10-21swapper/121:48:381
2174199270,12cyclictest0-21swapper/123:11:441
2174699260,1cyclictest0-21swapper/323:20:213
21741992625,1cyclictest0-21swapper/120:35:121
21744992318,4cyclictest0-21swapper/221:49:122
2174199230,2cyclictest4516-21missed_timers22:40:191
2174199220,20cyclictest0-21swapper/122:11:151
21744992110,7cyclictest24090-21dump-pmu-power19:15:002
2174499210,21cyclictest0-21swapper/200:17:022
2174499210,20cyclictest0-21swapper/220:55:122
2173999210,1cyclictest0-21swapper/023:35:130
2174699202,2cyclictest0-21swapper/323:14:233
2174199202,3cyclictest14323-21diskmemload23:01:471
2174199201,19cyclictest0-21swapper/122:08:581
2174199200,3cyclictest0-21swapper/123:20:191
2173999203,15cyclictest537-21snmpd23:28:080
2174699193,5cyclictest14323-21diskmemload22:55:133
2174699190,19cyclictest0-21swapper/322:19:563
2174199192,16cyclictest0-21swapper/100:00:151
2174199190,2cyclictest0-21swapper/121:36:211
2173999190,2cyclictest0-21swapper/021:13:480
2174699187,2cyclictest141rcu_preempt23:38:053
21741991811,6cyclictest0-21swapper/100:18:191
2173999182,12cyclictest11738-21ssh22:45:130
21739991817,1cyclictest0-21swapper/022:15:370
2174699170,17cyclictest0-21swapper/323:52:393
21744991716,1cyclictest0-21swapper/222:25:332
2174199177,5cyclictest225-21systemd-journal22:55:001
2173999170,17cyclictest0-21swapper/022:58:380
2174699162,7cyclictest0-21swapper/300:10:133
2174199162,8cyclictest38150irq/131-snd_hda00:35:111
2174199162,1cyclictest14323-21diskmemload23:27:481
2174199162,13cyclictest0-21swapper/123:55:191
2174199161,15cyclictest1942-21cstates22:00:141
2174199161,15cyclictest0-21swapper/121:55:261
2174199161,13cyclictest0-21swapper/100:14:381
2174199160,14cyclictest0-21swapper/121:13:491
2173999164,6cyclictest17332-21perf00:25:000
2173999162,10cyclictest0-21swapper/021:40:140
2173999162,10cyclictest0-21swapper/021:40:130
21739991611,3cyclictest14682-21unixbench_multi23:05:260
2173999160,0cyclictest0-21swapper/021:18:470
2174699153,8cyclictest18632-21sendmail_mailtr21:50:223
2174699152,9cyclictest0-21swapper/320:00:143
21746991513,2cyclictest0-21swapper/322:05:213
2174499151,2cyclictest0-21swapper/223:45:142
21744991512,1cyclictest0-21swapper/223:18:022
21744991512,1cyclictest0-21swapper/223:18:012
21744991510,4cyclictest11227-21sh22:45:002
2174499150,13cyclictest0-21swapper/200:27:042
2174499150,13cyclictest0-21swapper/200:27:032
21741991512,1cyclictest0-21swapper/122:18:331
2174199151,11cyclictest14403-21sh22:45:571
2173999151,2cyclictest141rcu_preempt00:12:300
2173999150,2cyclictest0-21swapper/023:41:010
21746991413,1cyclictest0-21swapper/322:22:393
2174699141,10cyclictest0-21swapper/300:33:493
2174499141,3cyclictest0-21swapper/219:35:122
21744991413,1cyclictest0-21swapper/223:55:412
21744991411,3cyclictest0-21swapper/221:30:232
21744991410,3cyclictest22105-21cat23:30:002
21744991410,3cyclictest18228-21wc20:10:012
21744991410,2cyclictest8556-21head21:25:182
2174199143,10cyclictest14323-21diskmemload22:35:241
2174199142,9cyclictest0-21swapper/120:45:121
21741991413,1cyclictest0-21swapper/122:26:001
21741991413,1cyclictest0-21swapper/100:20:461
21741991411,2cyclictest15545-21turbostat21:30:001
2174199140,14cyclictest0-21swapper/100:28:201
2174199140,14cyclictest0-21swapper/100:28:191
2174199140,13cyclictest0-21swapper/123:33:551
2173999148,4cyclictest94-21kswapd000:15:120
21739991411,3cyclictest6652-21apache221:33:520
21739991410,2cyclictest26770-21apache200:39:540
2173999140,14cyclictest0-21swapper/022:36:470
2174699138,2cyclictest537-21snmpd21:10:153
2174699136,5cyclictest31218-21munin-node21:42:483
2174699136,5cyclictest31218-21munin-node21:42:483
2174699132,6cyclictest16054-21ssh21:30:133
2174699132,5cyclictest0-21swapper/300:00:143
21746991310,2cyclictest10515-21grep19:50:253
2174499139,3cyclictest31643-21cron23:54:592
2174499139,2cyclictest32356-21grep21:20:162
2174499132,2cyclictest0-21swapper/219:15:122
21744991312,1cyclictest0-21swapper/223:02:232
21744991311,1cyclictest0-21swapper/200:32:342
21744991310,2cyclictest27640-21cron20:30:002
21744991310,2cyclictest20587-21sed22:50:142
2174199138,4cyclictest25051-21ssh22:33:281
2174199137,3cyclictest537-21snmpd23:50:481
2174199135,5cyclictest537-21snmpd23:07:181
2174199132,1cyclictest14323-21diskmemload22:55:091
21741991310,3cyclictest0-21swapper/123:19:011
21741991310,3cyclictest0-21swapper/123:19:001
2174199130,1cyclictest0-21swapper/121:20:131
2174199130,11cyclictest0-21swapper/120:25:141
2173999131,7cyclictest14323-21diskmemload23:45:260
21739991311,1cyclictest18857-21cat21:50:250
2173999130,10cyclictest14335-21sh21:10:000
2174699129,2cyclictest9398-21ssh23:40:503
2174699129,2cyclictest7430-21ssh22:03:323
2174699129,2cyclictest5772-21head19:40:253
2174699129,2cyclictest5734-21ssh00:36:413
2174699129,2cyclictest32087-21sh21:59:023
2174699129,2cyclictest29216-21ssh21:18:363
2174699129,2cyclictest26956-21grep20:25:273
2174699129,2cyclictest19897-21ssh22:50:063
2174699129,2cyclictest12546-21sed22:25:273
2174699129,1cyclictest18157-21ssh22:48:543
2174699128,1cyclictest26685-21apt19:19:593
2174699126,3cyclictest14323-21diskmemload22:35:033
2174699125,6cyclictest537-21snmpd22:41:123
2174699123,5cyclictest225-21systemd-journal19:23:343
2174699122,9cyclictest0-21swapper/323:03:273
2174699122,6cyclictest225-21systemd-journal00:08:573
2174699121,6cyclictest20817-21switchtime22:30:243
21746991211,1cyclictest0-21swapper/323:45:473
21746991210,2cyclictest39-21ksoftirqd/323:15:333
21746991210,2cyclictest39-21ksoftirqd/323:15:323
21746991210,1cyclictest24687-21tr21:35:133
2174699120,9cyclictest0-21swapper/323:06:123
2174699120,5cyclictest0-21swapper/321:20:143
2174699120,11cyclictest7831-21cat20:55:153
2174699120,11cyclictest22373-21ssh22:12:103
2174699120,11cyclictest17912-21cut00:25:143
2174699120,11cyclictest17912-21cut00:25:133
2174499129,2cyclictest669-21cut21:40:142
2174499129,2cyclictest669-21cut21:40:142
2174499129,2cyclictest30390-21cut20:35:132
2174499129,2cyclictest22453-21tr23:10:202
2174499129,2cyclictest21461-21grep20:15:182
2174499129,2cyclictest20547-21grep22:30:212
2174499129,2cyclictest18635-21grep20:10:142
2174499129,2cyclictest11137-21wc00:20:252
2174499129,2cyclictest0-21swapper/221:36:472
2174499128,3cyclictest779-21cat00:15:012
2174499128,3cyclictest17404-21grep00:05:152
2174499128,3cyclictest15324-21grep20:00:272
2174499128,2cyclictest28511-21grep22:55:132
2174499125,5cyclictest2370-21ls20:45:002
21744991212,0cyclictest0-21swapper/221:05:122
21744991210,1cyclictest0-21swapper/219:55:212
2174499120,5cyclictest0-21swapper/222:01:372
2174499120,3cyclictest0-21swapper/223:33:562
2174499120,3cyclictest0-21swapper/200:00:162
2174499120,1cyclictest0-21swapper/222:15:372
2174199129,2cyclictest571-21expr20:40:151
2174199129,2cyclictest30644-21ssh00:32:221
2174199129,2cyclictest26613-21chrt19:19:201
2174199129,2cyclictest24314-21cat20:20:201
2174199129,2cyclictest24246-21cat19:15:001
2174199129,2cyclictest2233-21grep21:40:261
2174199129,2cyclictest2233-21grep21:40:251
2174199129,2cyclictest16840-21tr00:05:131
2174199126,5cyclictest6602-21apache219:35:111
2174199121,7cyclictest0-21swapper/123:45:261
21741991211,1cyclictest0-21swapper/121:15:201
2174199120,11cyclictest18653-21grep20:10:141
2174199120,11cyclictest0-21swapper/121:32:211
2173999129,2cyclictest9349-21pluginstate22:05:000
2173999129,2cyclictest9144-21rm21:45:140
2173999129,2cyclictest31129-21ssh23:15:240
2173999129,2cyclictest31129-21ssh23:15:230
2173999129,2cyclictest27241-21grep20:25:270
2173999129,2cyclictest27027-21grep00:30:180
2173999129,2cyclictest25631-21cat23:50:250
2173999129,2cyclictest24686-21tr21:35:140
2173999129,2cyclictest2374-21grep19:35:160
2173999129,2cyclictest22618-21wget23:30:130
2173999129,2cyclictest21226-21tr23:10:120
2173999129,2cyclictest1972-21grep20:40:280
2173999129,2cyclictest17105-21sed20:05:190
2173999129,2cyclictest16854-21cut00:05:130
2173999128,3cyclictest7730-21ssh21:25:130
2173999128,3cyclictest28943-21wc20:30:210
2173999128,3cyclictest1411-21perf19:34:590
2173999124,4cyclictest24071-21cron19:15:000
2173999124,4cyclictest225-21systemd-journal20:49:590
2173999123,5cyclictest2761-21cstates22:20:150
2173999120,0cyclictest0-21swapper/022:30:120
2174699119,1cyclictest696-21ls19:30:233
2174699119,1cyclictest32636-21chrt20:40:003
2174699119,1cyclictest24487-21wc20:20:223
2174699119,1cyclictest13953-21ssh23:25:063
2174699119,1cyclictest11193-21chrt19:55:003
2174699119,1cyclictest10697-21tr00:20:203
2174699118,2cyclictest8890-21ssh21:25:203
2174699118,2cyclictest3448-21taskset19:35:593
2174699118,2cyclictest29407-21cut19:25:143
2174699118,2cyclictest2892-21grep20:45:143
2174699118,2cyclictest23289-21find19:10:213
2174699118,2cyclictest16249-21tr20:05:143
2174699118,2cyclictest14534-21sed21:10:003
2174699118,2cyclictest1397-21tr00:15:143
2174699118,2cyclictest10479-21wc21:45:243
2174699116,4cyclictest32527-21grep23:55:163
2174699115,2cyclictest20763-21apt-get20:15:143
2174699110,11cyclictest29884-21chrt20:33:233
2174699110,10cyclictest22220-21ssh23:30:013
2174699110,10cyclictest12024-21sed21:05:003
2174499118,2cyclictest6731-21sh23:40:072
2174499118,2cyclictest5626-21basename23:20:102
2174499118,2cyclictest31849-21grep19:30:122
2174499118,2cyclictest30476-21tr19:25:202
2174499118,2cyclictest29472-21unixbench_singl22:35:252
2174499118,2cyclictest25385-21ssh21:55:092
2174499118,2cyclictest2397-21tr00:35:142
2174499118,2cyclictest18895-21grep22:10:152
2174499118,2cyclictest1851-21ssh23:36:332
2174499118,2cyclictest18157-21grep21:50:182
2174499118,2cyclictest15980-21ssh22:47:112
2174499118,2cyclictest15123-21ssh23:05:412
2174499118,2cyclictest14603-21ssh21:10:032
2174499118,2cyclictest10547-21tail21:00:152
2174499118,2cyclictest10320-21munin-run22:24:592
2174499117,3cyclictest4305-21grep20:45:262
2174499117,3cyclictest29082-21basename19:25:002
2174499113,4cyclictest225-21systemd-journal22:05:262
2174499113,4cyclictest225-21systemd-journal20:25:002
2174499113,4cyclictest225-21systemd-journal19:54:592
2174199119,1cyclictest29602-21tr19:25:141
2174199119,1cyclictest27373-21expr19:20:141
2174199119,1cyclictest2727-21grep22:20:151
2174199119,1cyclictest180332sleep120:09:051
2174199119,1cyclictest12536-21expr21:05:141
2174199119,1cyclictest10248-21expr21:00:131
2174199118,2cyclictest30747-21chrt23:35:101
2174199118,2cyclictest21956-21ssh21:52:401
2174199118,2cyclictest11835-21grep19:55:161
2174199117,3cyclictest7149-21taskset20:54:481
2174199111,3cyclictest0-21swapper/123:40:121
2173999118,2cyclictest9545-21sh20:59:590
2173999118,2cyclictest8901-21grep00:00:180
2173999118,2cyclictest27849-21ssh21:55:440
2173999118,2cyclictest24245-21ssh22:13:370
2173999118,2cyclictest23898-21ssh22:51:520
2173999118,2cyclictest23869-21grep20:20:170
2173999118,2cyclictest1970-21ssh23:55:360
2173999118,2cyclictest18100-21tr00:25:150
2173999118,2cyclictest18100-21tr00:25:140
2173999118,2cyclictest1527-21ssh21:20:340
2173999118,2cyclictest12716-21cat19:55:250
2173999118,2cyclictest10215-21ssh22:44:130
2173999118,2cyclictest10114-21tr22:05:150
2173999117,3cyclictest19154-21sed20:10:180
2173999112,4cyclictest11518-21fschecks_count22:25:170
2173999112,3cyclictest10991-21awk19:55:000
2174699108,1cyclictest70902sleep320:54:033
2174699107,2cyclictest30522-21tr20:35:143
2174699107,2cyclictest19016-21cut20:10:163
2174499108,1cyclictest6474-21awk19:45:132
2174499108,1cyclictest6474-21awk19:45:122
2174499108,1cyclictest5011-21apt20:50:132
2174499108,1cyclictest24364-21head21:15:182
2174499107,2cyclictest4781-21grep19:40:152
2174199109,1cyclictest7890-21expr20:55:141
2174199109,1cyclictest5209-21taskset19:40:181
2174199108,2cyclictest6843-21ls19:45:141
2174199108,2cyclictest6843-21ls19:45:131
2174199108,1cyclictest9853-21sort19:50:181
2174199108,1cyclictest28635-21head20:30:181
2174199107,2cyclictest22200-21cat20:15:261
2173999108,1cyclictest6468-21cat20:50:260
2173999108,1cyclictest5548-21ssh23:20:070
2173999108,1cyclictest30397-21aten2_r7power_v20:35:130
2173999108,1cyclictest207132sleep020:15:010
2173999108,1cyclictest15879-21cat20:05:000
2173999107,2cyclictest8191-21head19:45:250
2173999107,2cyclictest8191-21head19:45:250
2173999107,2cyclictest30014-21sort19:25:170
2173999107,2cyclictest27557-21fschecks_count19:20:150
2173999107,2cyclictest24770-21cat19:15:130
2173999103,5cyclictest0-21swapper/019:40:200
2173999100,9cyclictest98022sleep021:00:090
2173999100,2cyclictest0-21swapper/023:00:020
217469996,2cyclictest6696-21cat19:45:143
217469996,2cyclictest6696-21cat19:45:133
217449991,4cyclictest225-21systemd-journal20:30:072
217419997,2cyclictest32286-21diskstats19:30:141
217419997,1cyclictest13654-21seq20:00:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional