You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-02 - 03:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sat May 02, 2026 00:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5573220995,90sleep00-21swapper/019:09:140
5465214597,40sleep10-21swapper/119:07:521
53842142106,25sleep20-21swapper/219:06:472
54272139103,25sleep30-21swapper/319:07:223
303052820,4sleep00-21swapper/000:25:140
119802680,2sleep30-21swapper/323:24:023
247412650,2sleep10-21swapper/100:20:261
111452630,8sleep122-21ksoftirqd/123:23:121
177052580,3sleep35903-21cyclictest22:40:123
38692550,1sleep20-21swapper/223:41:232
321592550,2sleep30-21swapper/300:02:263
221102530,9sleep012-21ksoftirqd/022:19:180
203192530,1sleep10-21swapper/122:41:221
326592520,2sleep10-21swapper/100:02:571
116562490,1sleep20-21swapper/221:46:582
180032480,2sleep018001-21ssh22:40:130
295522470,2sleep10-21swapper/121:12:251
5905994610,19cyclictest25720-21sh23:58:020
318022460,1sleep00-21swapper/023:38:400
51352400,1sleep20-21swapper/221:42:112
69792380,2chrt0-21swapper/322:31:253
590599361,7cyclictest32151-21sh22:25:520
100472360,2sleep10-21swapper/100:10:161
5905993513,19cyclictest30299-21sh23:13:260
5905993410,21cyclictest30146-21expr20:05:140
5905993312,18cyclictest11380-21seq22:59:440
590599330,4cyclictest8415-21ssh22:08:530
90122320,4sleep0121ksoftirqd/021:20:430
590599329,21cyclictest6132-21expr19:10:130
590599327,23cyclictest0-21swapper/023:04:000
590599320,5cyclictest8980-21ssh22:33:260
590599320,5cyclictest11246-21ssh00:10:260
590599320,3cyclictest9680-21mailstats21:45:210
33312320,2sleep10-21swapper/122:53:231
590599319,19cyclictest6906-21cat20:25:180
590599319,19cyclictest6906-21cat20:25:170
590599319,19cyclictest12744-21seq20:40:070
590599311,27cyclictest7985-21/usr/sbin/munin23:20:230
5905993110,19cyclictest1414-21seq20:13:130
5905993110,18cyclictest24559-21cat22:45:120
5905993110,18cyclictest24559-21cat22:45:110
5905993110,18cyclictest10484-21cpuspeed_turbos20:35:110
590599310,7cyclictest26088-21ssh21:34:160
590599310,5cyclictest32660-21sh21:39:050
590599310,5cyclictest25375-21diskmemload23:41:010
590599306,20cyclictest9223-21sort00:10:010
5905993014,13cyclictest32048-21netstat22:50:180
590599301,3cyclictest0-21swapper/000:35:090
5905993013,15cyclictest26173-21diskstats21:10:140
590599301,2cyclictest0-21swapper/021:25:110
590599300,5cyclictest4289-21rm21:41:220
590599300,3cyclictest982-21ssh23:15:220
590599300,3cyclictest11025-21ntpq22:10:220
590599298,18cyclictest17093-21expr19:35:130
590599298,18cyclictest17093-21expr19:35:120
590599297,19cyclictest24874-21cat19:50:240
5905992912,15cyclictest32585-21sh21:15:110
590599290,8cyclictest28776-21diskstats00:00:130
590599290,6cyclictest6485-21sh00:31:060
590599290,5cyclictest19087-21ssh23:53:090
590599290,4cyclictest21581-21ssh23:06:260
590599290,3cyclictest8167-21expr23:45:140
590599290,3cyclictest24086-21ssh22:20:180
590599290,3cyclictest13754-21sensors_temp19:25:210
156112290,3chrt331rcuc/322:38:193
590599289,16cyclictest20773-21seq19:43:120
590599287,18cyclictest19429-21expr20:55:130
590599285,5cyclictest13766-21sh23:25:150
590599282,5cyclictest11497-21grep22:35:140
5905992811,15cyclictest18831-21sh21:52:220
5905992811,14cyclictest15800-21ntpq20:45:190
5905992810,15cyclictest28889-21sh00:24:360
590599280,7cyclictest22689-21cut21:55:150
590599280,3cyclictest9035-21ls20:30:180
590599280,3cyclictest28472-21irqstats20:00:170
590599280,3cyclictest20485-21df_inode23:30:130
79782270,2sleep20-21swapper/200:32:352
590599279,15cyclictest16565-21sh00:15:150
590599279,15cyclictest16565-21sh00:15:140
590599278,17cyclictest29154-21expr22:00:130
590599275,5cyclictest21453-21expr19:45:130
5905992710,14cyclictest21815-21gltestperf21:00:160
590599270,3cyclictest4313-21df_inode20:20:130
5906992624,1cyclictest0-21swapper/119:11:151
590599265,5cyclictest18870-21seq20:54:190
590599263,5cyclictest2162-21forks20:15:140
590599262,22cyclictest0-21swapper/021:05:120
590599261,8cyclictest15629-21/usr/sbin/munin19:30:200
590599261,7cyclictest12399-21date19:25:010
54112260,1sleep10-21swapper/123:19:141
5908992523,1cyclictest0-21swapper/319:11:503
5907992523,1cyclictest0-21swapper/219:11:292
590599256,3cyclictest8223-21tr19:15:110
590599250,7cyclictest25651-21find19:55:110
266092250,2chrt0-21swapper/123:58:551
23522250,11sleep228-21ksoftirqd/222:28:322
5907992417,5cyclictest24402-21ssh21:32:342
5906992416,5cyclictest21107-21sendmail21:00:001
5906992415,6cyclictest560-21dbus-daemon19:40:011
5906992415,6cyclictest560-21dbus-daemon19:40:001
5906992414,7cyclictest14478-21cron19:30:001
5906992413,7cyclictest4647-21grep00:30:151
5906992412,8cyclictest20923-21cat23:30:181
590899232,19cyclictest18929-21cron20:55:013
5908992313,7cyclictest3730-21perl22:05:143
5908992313,7cyclictest10577-21df_inode19:20:103
5908992311,7cyclictest25575-21ssh23:10:113
590699232,15cyclictest7927-21ssh23:45:121
590699232,15cyclictest21947-21chrt21:55:041
5908992213,7cyclictest17662-21sed19:35:173
5908992213,7cyclictest17662-21sed19:35:173
5908992213,7cyclictest1013-21ls21:15:173
5907992214,6cyclictest31241-21ls00:25:242
5907992212,7cyclictest23615-21df_inode22:20:132
590699222,14cyclictest16303-21ssh00:15:141
590699222,14cyclictest16303-21ssh00:15:131
590699222,14cyclictest14615-21ssh23:50:111
5906992215,4cyclictest25375-21diskmemload22:30:011
5906992211,7cyclictest8089-21grep21:45:011
322882220,2sleep10-21swapper/123:39:081
291552220,2sleep229271-21ssh00:00:182
284782220,2sleep10-21swapper/122:23:561
82572210,2sleep30-21swapper/320:30:103
5908992114,5cyclictest1700-21sed20:15:003
5908992113,6cyclictest24045-21smartctl21:05:153
5908992110,6cyclictest0-21swapper/321:45:103
590799212,16cyclictest22838-21grep22:20:012
590799212,16cyclictest11432-21grep22:10:242
5907992113,6cyclictest22002-21ls19:45:192
590799210,5cyclictest0-21swapper/222:48:552
590799210,5cyclictest0-21swapper/222:48:542
590699217,4cyclictest7495-21gltestperf21:20:141
5906992113,6cyclictest3483-21expr22:05:131
5906992112,7cyclictest23012-21seq21:02:171
5906992112,5cyclictest20063-21mailstats23:05:191
5906992111,7cyclictest16806-21grep21:50:251
45162210,2sleep10-21swapper/123:42:021
590899207,4cyclictest34-21ksoftirqd/321:10:193
590899202,13cyclictest16625-21ssh22:15:113
5908992016,3cyclictest17395-21latency_hist00:40:003
590899201,5cyclictest0-21swapper/323:02:593
590899201,4cyclictest0-21swapper/322:57:253
5908992012,6cyclictest22053-21sed21:00:183
5908992011,7cyclictest17320-21ntpq00:15:223
5908992011,7cyclictest17320-21ntpq00:15:213
5908992011,5cyclictest25375-21diskmemload23:59:053
590899201,13cyclictest28909-21ssh22:00:123
590899200,17cyclictest19794-21cat20:55:163
590799208,10cyclictest27907-21pmu-power23:35:202
590799203,9cyclictest640-21snmpd23:00:142
590799202,12cyclictest16881-21apt-get19:35:122
590799202,12cyclictest16881-21apt-get19:35:112
590799202,12cyclictest10247-21apt-get19:20:102
590799201,8cyclictest13380-21apt-get23:25:122
5907992013,4cyclictest4030-21cpu20:20:122
5907992013,4cyclictest20063-21sh22:41:072
5907992012,6cyclictest13745-21grep19:25:212
5907992012,6cyclictest12063-21awk22:35:202
5907992011,6cyclictest8460-21expr23:45:152
5907992011,6cyclictest11433-21smartctl00:35:152
590799200,9cyclictest72212which20:25:222
590799200,9cyclictest72212which20:25:212
590699202,12cyclictest25665-21apt-get19:55:101
590699202,12cyclictest19038-21apt-get19:40:111
590699201,7cyclictest8186-21ssh22:56:291
5906992014,3cyclictest30305-21threads22:00:231
5906992013,5cyclictest31359-21grep00:25:251
5906992012,6cyclictest22078-21ls21:30:251
5906992012,5cyclictest278-21systemd-journal23:29:541
5906992011,6cyclictest53050irq/142-enp3s0-00:05:171
5906992010,7cyclictest28297-21grep20:00:171
590899193,11cyclictest6127-21apt-get20:25:133
590899193,11cyclictest6127-21apt-get20:25:123
590899192,5cyclictest0-21swapper/323:25:143
590899192,12cyclictest14741-21apt-get19:30:113
590899192,11cyclictest640-21snmpd23:05:103
590899191,6cyclictest0-21swapper/323:40:433
5908991914,3cyclictest11326-21ntp_states20:35:193
5908991913,4cyclictest3852-21dump-pmu-power22:30:003
5908991912,5cyclictest27168-21sh21:58:443
5908991912,5cyclictest26612-21date23:35:003
5908991912,5cyclictest26567-21ntpq19:55:203
5908991912,5cyclictest13381-21cat19:25:173
5908991912,4cyclictest8396-21grep23:45:163
5908991912,4cyclictest25375-21diskmemload00:09:413
5908991911,6cyclictest7651-21cat21:20:163
5908991911,6cyclictest5251-21sed20:20:233
5908991911,6cyclictest32434-21expr23:15:143
5908991911,6cyclictest10754-21cat22:10:193
5908991910,7cyclictest30873-21seq23:37:443
590899190,8cyclictest31007-21/usr/sbin/munin22:50:143
590799192,11cyclictest19192-21apt-get20:55:122
5907991913,4cyclictest8630-21sh22:33:052
5907991913,4cyclictest16053-21timerwakeupswit19:30:242
5907991913,4cyclictest14988-21sh23:50:132
5907991913,3cyclictest25375-21diskmemload21:54:592
5907991913,3cyclictest21503-21grep23:55:132
5907991912,4cyclictest29668-21sh21:36:052
5907991912,4cyclictest17845-21sh21:27:482
5907991911,6cyclictest9003-21cat20:30:172
5907991911,6cyclictest32392-21expr20:10:122
5907991911,6cyclictest25348-21ls23:10:002
5907991911,6cyclictest22804-21cat21:55:172
5907991911,6cyclictest10131-21smartctl00:10:162
5907991911,5cyclictest8408-21expr19:15:122
5907991911,5cyclictest26420-21load19:55:182
5907991910,7cyclictest27037-21cat23:10:242
5907991910,6cyclictest7042-21sort21:20:112
5907991910,6cyclictest2538-21ls20:15:172
5907991910,6cyclictest16492-21cat00:15:152
5907991910,6cyclictest16492-21cat00:15:152
5907991910,6cyclictest1204-21ntpq21:15:202
5907991910,6cyclictest11702-21cat20:35:222
590699199,7cyclictest278-21systemd-journal19:55:011
590699199,6cyclictest25499-21sed21:10:011
590699192,5cyclictest10791-21apt-get00:35:111
5906991915,3cyclictest53050irq/142-enp3s0-21:35:191
5906991914,3cyclictest14841-21ls21:25:221
5906991913,3cyclictest5367-21sh21:18:511
5906991912,4cyclictest26089-21gltestperf23:10:151
5906991911,6cyclictest25582-21tr22:45:211
5906991911,6cyclictest25582-21tr22:45:201
5906991911,6cyclictest10621-21seq21:45:551
5906991910,7cyclictest9228-21tr19:15:211
5906991910,6cyclictest278-21systemd-journal20:50:011
191272190,4sleep334-21ksoftirqd/321:29:043
590899189,7cyclictest23971-21ls00:20:203
590899182,11cyclictest9499-21ssh00:10:113
590899182,11cyclictest29722-21apt-get00:25:113
590899182,11cyclictest24310-21apt-get22:45:113
590899182,11cyclictest24310-21apt-get22:45:103
590899181,6cyclictest0-21swapper/322:21:033
5908991812,3cyclictest28687-21sh21:35:223
5908991811,5cyclictest30226-21expr20:05:143
5908991811,5cyclictest2731-21cat20:15:193
590899181,13cyclictest20521-21ssh21:30:123
5908991810,6cyclictest4513-21cut00:30:153
5908991810,6cyclictest15418-21expr21:50:133
590799189,6cyclictest29275-21forks22:00:142
590799188,7cyclictest21679-21df_inode21:00:132
590799186,10cyclictest0-21swapper/221:12:282
5907991813,3cyclictest8995-21sh22:57:182
5907991813,3cyclictest6188-21sh00:07:082
5907991813,3cyclictest4967-21latency_hist22:55:012
5907991813,3cyclictest25844-21sh23:34:282
5907991812,4cyclictest3627-21sh23:17:272
5907991811,5cyclictest8362-21vmstat23:20:272
5907991811,5cyclictest23743-21expr21:05:132
5907991811,4cyclictest25805-21sh00:21:302
5907991810,5cyclictest30542-21idleruntime20:05:162
5907991810,5cyclictest19770-21cat19:40:162
5907991810,5cyclictest16355-21grep20:45:252
590699188,7cyclictest7047-21ntp_states20:25:211
590699188,7cyclictest7047-21ntp_states20:25:201
5906991812,4cyclictest17414-21latency22:15:171
5906991811,5cyclictest10675-21expr20:35:121
590699181,14cyclictest4394-21kworker/u8:122:30:171
5906991810,6cyclictest18572-21wc23:05:011
5906991810,5cyclictest278-21systemd-journal20:09:591
5906991810,5cyclictest10579-21df19:20:101
590699180,15cyclictest17516-21gltestperf20:50:131
590899179,6cyclictest24036-21grep19:50:153
590899175,3cyclictest141rcu_preempt23:50:163
5908991711,5cyclictest34-21ksoftirqd/320:50:003
5908991710,5cyclictest20342-21date19:40:233
5908991710,5cyclictest1902-21expr21:40:133
590799179,5cyclictest3969-21sed22:05:172
590799179,5cyclictest12754-21find20:40:112
590799178,6cyclictest28973-21mailstats20:00:222
5907991712,3cyclictest25410-21sh19:55:002
590699179,6cyclictest32107-21grep20:10:121
590699179,6cyclictest22840-21seq19:47:171
590699179,6cyclictest16624-21cron19:34:591
5906991712,3cyclictest11680-21users22:10:261
5906991710,5cyclictest12886-21grep20:40:121
590899169,5cyclictest8177-21expr19:15:113
5908991611,3cyclictest13400-21if_err_enp3s020:40:173
590899161,11cyclictest27825-21apt-get20:00:133
5907991610,4cyclictest17221-21cstates20:50:102
590699169,5cyclictest11735-21irqstats22:35:171
590699169,4cyclictest1871-21cpuspeed_turbos20:15:121
5906991611,3cyclictest8239-21/usr/sbin/munin20:30:141
5906991611,3cyclictest4305-21sh20:20:131
590899157,6cyclictest22478-21sed19:45:233
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional