You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-26 - 03:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Tue May 26, 2026 00:45:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2962222230,3sleep122-21ksoftirqd/119:05:141
308092222101,113sleep20-21swapper/219:06:442
291542181101,27sleep00-21swapper/019:05:010
309092177102,29sleep30-21swapper/319:08:023
113232620,2sleep00-21swapper/023:29:380
59632610,2sleep00-21swapper/022:12:250
59632610,2sleep00-21swapper/022:12:240
169922590,2sleep20-21swapper/221:56:012
91842530,2sleep30-21swapper/322:15:123
229342530,2sleep10-21swapper/120:05:001
225172530,1sleep10-21swapper/123:37:251
123402520,2sleep10-21swapper/121:53:031
291162510,1sleep129117-21sh23:42:191
291162510,1sleep129117-21sh23:42:181
214792500,2sleep10-21swapper/122:24:371
33072450,1sleep30-21swapper/321:21:393
321732450,2sleep10-21swapper/120:25:151
49852430,2sleep10-21swapper/123:24:571
291772420,1sleep20-21swapper/223:18:242
129612410,1sleep00-21swapper/000:19:100
284832400,3chrt0-21swapper/222:30:002
176712400,2sleep30-21swapper/322:45:233
31335993910,26cyclictest1291-21rm22:09:281
196642360,2chrt0-21swapper/323:10:433
191502350,1sleep00-21swapper/023:10:240
17982340,2chrt0-21swapper/223:45:282
31336993322,3cyclictest28-21ksoftirqd/221:16:572
301092330,2chrt0-21swapper/222:55:132
231792330,2sleep20-21swapper/222:00:312
3133599320,29cyclictest7952-21ssh22:14:271
3133599320,29cyclictest7952-21ssh22:14:261
3133799301,26cyclictest8104-21ssh22:14:363
3133799301,26cyclictest8104-21ssh22:14:353
3133699301,7cyclictest28-21ksoftirqd/222:35:092
31335993017,9cyclictest10244-21df22:40:151
3133599301,27cyclictest13949-21ssh22:18:461
3133599301,26cyclictest23818-21ssh22:50:161
31334993022,5cyclictest354-21ssh22:08:330
31334992914,4cyclictest12-21ksoftirqd/021:40:180
96152280,2chrt0-21swapper/020:45:250
3133799281,3cyclictest26118-21ssh22:03:293
3133799280,25cyclictest3872-21if_enp3s022:35:163
3133499281,25cyclictest0-21swapper/022:50:150
109402280,2sleep30-21swapper/323:53:003
3133699277,11cyclictest28-21ksoftirqd/223:31:232
31336992720,4cyclictest13420-21sh22:42:162
3133599271,25cyclictest0-21swapper/122:04:411
3133599271,24cyclictest0-21swapper/122:25:271
3133499278,5cyclictest12-21ksoftirqd/000:09:560
3133499278,10cyclictest12-21ksoftirqd/021:52:420
3133499274,13cyclictest12-21ksoftirqd/022:21:430
3133499271,6cyclictest12-21ksoftirqd/023:50:120
3133499271,24cyclictest0-21swapper/022:35:190
3133499271,1cyclictest0-21swapper/022:17:470
31334992711,4cyclictest12-21ksoftirqd/023:40:500
31334992711,4cyclictest12-21ksoftirqd/023:40:490
280452270,8sleep122-21ksoftirqd/123:17:151
84182260,1sleep20-21swapper/200:15:192
49762260,1sleep00-21swapper/021:23:210
3133799261,23cyclictest0-21swapper/322:50:253
3133799260,24cyclictest32714-21ssh22:08:273
3133699266,4cyclictest28-21ksoftirqd/221:27:182
3133699266,4cyclictest28-21ksoftirqd/221:27:182
3133699266,11cyclictest28-21ksoftirqd/221:51:162
3133699261,2cyclictest0-21swapper/222:35:012
3133699261,2cyclictest0-21swapper/222:17:512
3133699261,24cyclictest0-21swapper/222:11:502
3133699261,24cyclictest0-21swapper/222:11:492
3133599261,1cyclictest0-21swapper/122:35:111
31334992624,1cyclictest0-21swapper/022:02:390
3133499261,24cyclictest0-21swapper/019:23:420
31334992610,4cyclictest12-21ksoftirqd/023:34:540
31337992514,8cyclictest14602-21df00:20:143
31337992514,8cyclictest14602-21df00:20:143
3133699252,17cyclictest27545-21ssh00:30:102
3133699251,23cyclictest0-21swapper/222:09:352
3133699250,23cyclictest0-21swapper/219:23:092
3133599252,15cyclictest7077-21ssh21:25:141
3133599252,15cyclictest7077-21ssh21:25:131
3133599251,1cyclictest0-21swapper/119:23:431
31334992517,6cyclictest12-21ksoftirqd/023:00:000
31334992511,4cyclictest12-21ksoftirqd/023:39:500
178082250,8sleep012-21ksoftirqd/021:56:510
91102240,2chrt0-21swapper/319:30:243
43132240,20sleep319033-21diskmemload23:00:133
31337992422,1cyclictest0-21swapper/319:23:063
3133699248,4cyclictest28-21ksoftirqd/223:01:562
3133699248,4cyclictest28-21ksoftirqd/200:13:202
3133699244,11cyclictest28-21ksoftirqd/221:37:482
31336992420,2cyclictest14462chrt21:20:212
31336992414,7cyclictest11359-21df_inode23:05:142
3133699241,1cyclictest0-21swapper/222:50:182
3133499248,14cyclictest12-21ksoftirqd/023:21:370
3133499244,5cyclictest12-21ksoftirqd/000:14:560
31334992422,1cyclictest0-21swapper/022:25:260
31334992412,3cyclictest12-21ksoftirqd/021:05:240
31337992321,1cyclictest0-21swapper/322:40:193
3133699237,14cyclictest277551rm00:05:212
3133699234,9cyclictest0-21swapper/223:10:112
3133699234,2cyclictest28-21ksoftirqd/223:43:092
3133699234,2cyclictest28-21ksoftirqd/223:43:082
3133699232,15cyclictest16550-21ssh22:45:132
3133699232,14cyclictest13360-21ssh23:55:122
31335992321,1cyclictest0-21swapper/122:34:471
31335992316,5cyclictest23487-21rm21:12:451
3133499239,4cyclictest12-21ksoftirqd/023:49:060
3133499239,3cyclictest141rcu_preempt00:38:270
3133499238,2cyclictest141rcu_preempt22:40:200
3133499238,2cyclictest141rcu_preempt00:23:050
3133499238,2cyclictest141rcu_preempt00:23:040
3133499237,4cyclictest12-21ksoftirqd/023:55:110
3133499236,4cyclictest12-21ksoftirqd/000:30:220
3133499235,9cyclictest12-21ksoftirqd/021:33:300
3133499233,10cyclictest12-21ksoftirqd/023:09:310
31334992316,4cyclictest30803-21awk23:20:000
31334992311,2cyclictest141rcu_preempt21:15:000
31334992311,2cyclictest141rcu_preempt00:01:550
31334992310,9cyclictest4682-21df_inode23:00:140
27602230,2sleep228-21ksoftirqd/219:15:262
201062230,2sleep334-21ksoftirqd/321:59:123
71482220,2sleep20-21swapper/220:40:212
38892220,2sleep30-21swapper/323:23:513
31337992215,5cyclictest15769-21sh23:56:093
3133699227,13cyclictest141rcu_preempt23:54:472
3133699226,7cyclictest141rcu_preempt21:31:022
3133699225,3cyclictest141rcu_preempt20:10:122
3133699225,12cyclictest141rcu_preempt19:30:102
31336992211,8cyclictest281ksoftirqd/223:25:012
3133499229,4cyclictest12-21ksoftirqd/021:05:000
3133499228,4cyclictest12-21ksoftirqd/022:30:200
3133499226,3cyclictest141rcu_preempt21:46:240
31334992213,5cyclictest19033-21diskmemload21:35:120
303922220,8sleep334-21ksoftirqd/321:17:573
64472210,2sleep30-21swapper/320:40:133
3133799219,8cyclictest352-21df_inode23:45:153
3133799213,13cyclictest27542-21apt-get00:30:123
31337992115,4cyclictest34-21ksoftirqd/320:40:003
31337992115,4cyclictest34-21ksoftirqd/320:39:593
31337992114,4cyclictest31510-21sh20:24:593
31337992113,6cyclictest32300-21df20:25:163
31337992113,5cyclictest25835-21cat23:40:133
31337992113,5cyclictest25835-21cat23:40:133
31337992112,6cyclictest560-21dbus-daemon19:39:593
31337992110,9cyclictest29462-21sh23:18:423
3133699215,9cyclictest28-21ksoftirqd/223:25:122
3133699215,8cyclictest28-21ksoftirqd/200:04:232
3133699213,11cyclictest141rcu_preempt21:13:412
3133699212,13cyclictest27088-21apt-get21:40:112
31336992112,5cyclictest15750-21tr22:20:142
3133699211,11cyclictest141rcu_preempt23:37:162
3133699211,11cyclictest141rcu_preempt00:24:462
3133699211,11cyclictest141rcu_preempt00:24:452
3133599212,13cyclictest20374-21apt-get21:35:111
31335992112,7cyclictest29104-21seq21:40:431
3133499218,3cyclictest0-21swapper/020:34:590
3133499217,3cyclictest141rcu_preempt20:35:130
3133499217,3cyclictest141rcu_preempt20:35:130
3133499216,5cyclictest12-21ksoftirqd/021:26:020
3133499216,5cyclictest12-21ksoftirqd/021:26:010
3133499216,3cyclictest141rcu_preempt22:46:420
3133499213,13cyclictest141rcu_preempt21:13:520
3133499212,6cyclictest12-21ksoftirqd/019:40:140
31334992115,4cyclictest12-21ksoftirqd/020:55:020
31334992115,4cyclictest12-21ksoftirqd/020:45:000
187412210,2sleep30-21swapper/322:21:513
106382210,3sleep0131rcuc/019:35:150
3133799203,5cyclictest8029-21apt-get21:50:123
3133799202,12cyclictest12833-21apt-get20:55:123
31337992014,4cyclictest11726-21threads20:50:253
3133799201,3cyclictest0-21swapper/323:25:153
31337992012,6cyclictest22616-21sh23:37:313
31337992012,6cyclictest1659-21forks19:15:163
31337992012,5cyclictest19033-21diskmemload23:30:293
31337992011,7cyclictest30180-21df22:55:123
31337992011,6cyclictest8612-21ntpq00:15:203
31337992011,6cyclictest22081-21sed21:35:253
31337992011,6cyclictest21642-21cat20:00:193
31337992010,7cyclictest20006-21cat19:55:233
31337992010,6cyclictest52950irq/141-enp3s0-00:35:123
3133699206,3cyclictest141rcu_preempt20:30:192
3133699205,4cyclictest141rcu_preempt21:46:112
31336992012,6cyclictest14863-21fw_packets19:45:152
31336992011,7cyclictest863-21date20:25:272
31335992014,3cyclictest14225-21/usr/sbin/munin00:20:161
31335992014,3cyclictest14225-21/usr/sbin/munin00:20:151
31335992013,4cyclictest9225-21tail00:15:261
31335992012,6cyclictest7152-21perl23:50:141
31335992012,6cyclictest13388-21seq23:06:051
31335992012,6cyclictest1125-21pluginstate23:45:221
3133499207,11cyclictest0-21swapper/020:00:000
3133499206,3cyclictest141rcu_preempt00:25:210
31334992014,4cyclictest12-21ksoftirqd/020:15:010
165152200,2sleep10-21swapper/121:55:321
42442190,3sleep334-21ksoftirqd/300:12:033
3133799192,12cyclictest20048-21apt-get00:00:133
31337991913,4cyclictest31838-21sh21:43:293
31337991913,4cyclictest25611-21sh22:27:053
31337991912,5cyclictest28367-21ntpq20:15:213
31337991912,4cyclictest636-21runrttasks21:34:493
31337991911,6cyclictest2409-21sed20:30:183
31337991911,6cyclictest23899-21ntp_states20:05:223
31337991911,6cyclictest17264-21sed19:50:183
31337991911,6cyclictest12686-21sed23:05:243
3133699199,8cyclictest640-21snmpd19:39:442
3133699199,7cyclictest11959-21grep20:50:272
3133699199,2cyclictest28-21ksoftirqd/200:35:032
3133699197,3cyclictest28-21ksoftirqd/220:20:142
3133699194,3cyclictest141rcu_preempt20:15:162
31336991911,6cyclictest15329-21diskstats21:00:142
31336991911,6cyclictest12989-21sed19:40:162
3133599198,8cyclictest15327-21unixbench_multi21:30:251
3133599192,12cyclictest640-21snmpd20:35:131
3133599192,12cyclictest640-21snmpd20:35:121
3133599192,12cyclictest26730-21ssh00:05:121
3133599192,11cyclictest45-21kcompactd023:00:041
31335991914,3cyclictest8230-21head19:30:151
31335991913,4cyclictest22-21ksoftirqd/120:35:001
31335991912,5cyclictest1936-21cat19:15:181
31335991912,5cyclictest15330-21awk19:45:201
31335991911,6cyclictest32172-21seq00:33:231
31335991911,6cyclictest14394-21seq20:55:591
31335991911,5cyclictest28302-21grep20:15:211
3133599191,13cyclictest12117-21apt-get23:30:121
31335991910,5cyclictest18962-21cut23:10:221
3133499199,2cyclictest12-21ksoftirqd/019:53:410
3133499198,3cyclictest12-21ksoftirqd/019:30:160
31334991912,5cyclictest16021-21seq19:46:460
31334991911,6cyclictest31151-21seq20:20:320
31334991911,6cyclictest2424-21tr19:15:240
31334991911,5cyclictest6754-21tr19:25:240
31334991911,5cyclictest1196-21seq20:27:030
31334991910,6cyclictest28118-21cut20:15:190
3133799189,6cyclictest31384-21cpu19:10:113
31337991813,3cyclictest22701-21sh21:11:583
31337991813,3cyclictest15912-21ntp_states21:00:223
31337991813,3cyclictest15279-21netstat19:45:183
31337991812,4cyclictest28470-21sh00:05:293
31337991811,5cyclictest6707-21which19:25:233
31337991811,5cyclictest2177-21latency21:45:183
31337991810,6cyclictest29517-21expr22:30:163
31337991810,6cyclictest21155-21cat00:25:143
31337991810,6cyclictest13477-21cat19:40:233
3133699189,6cyclictest17113-21cat21:05:122
3133699181,3cyclictest141rcu_preempt19:10:172
31336991811,5cyclictest4094-21expr20:35:142
31336991811,5cyclictest4094-21expr20:35:132
31336991811,5cyclictest21133-21expr20:00:152
31336991811,5cyclictest17982-21awk19:50:252
31336991811,3cyclictest28-21ksoftirqd/200:25:252
3133699181,12cyclictest5614-21apt-get19:25:142
31336991810,6cyclictest14863-21date21:00:012
3133599188,7cyclictest27246-21ntp_states21:15:211
3133599182,11cyclictest8344-21apt-get20:45:111
31335991813,3cyclictest22950-21acpi20:05:121
31335991813,3cyclictest19576-21cron00:00:011
31335991812,4cyclictest8762-21sh23:27:021
31335991812,4cyclictest3442-21sh21:45:471
31335991811,5cyclictest26776-21seq20:12:181
31335991811,5cyclictest20456-21seq19:58:501
31335991811,5cyclictest17765-21cat21:05:171
31335991811,5cyclictest12263-21seq20:53:481
3133599181,14cyclictest17302-21missed_timers19:50:181
31335991810,6cyclictest30337-21cut22:55:141
31335991810,6cyclictest10038-21sed19:35:001
3133599180,16cyclictest16222-21switchtime21:00:251
31334991811,5cyclictest32521-21grep19:10:210
31334991811,5cyclictest24759-21seq20:08:530
31334991810,5cyclictest11964-21grep20:50:260
3133799179,6cyclictest7682-21expr21:25:183
3133799179,6cyclictest7682-21expr21:25:173
31337991712,3cyclictest25086-21/usr/sbin/munin20:10:253
31337991710,5cyclictest8399-21ls20:45:143
31337991710,5cyclictest18666-21seq21:05:383
3133699179,5cyclictest83072sleep220:45:032
3133699177,2cyclictest28-21ksoftirqd/219:55:102
31336991712,3cyclictest24059-21sendmail_mailqu20:05:222
3133599178,6cyclictest868-21gpgv19:15:001
3133599172,10cyclictest20920-21apt-get00:25:131
31335991712,3cyclictest3069-21sh00:35:221
31335991712,3cyclictest2855-21sh00:10:381
31335991712,3cyclictest22086-21sh00:00:461
31335991710,5cyclictest6227-21expr20:40:121
31335991710,5cyclictest16724-21cat22:45:131
31335991710,4cyclictest19033-21diskmemload21:20:031
3133499170,15cyclictest21580-21cat20:00:190
3133599169,5cyclictest6765-21grep19:25:231
3133599169,5cyclictest30715-21which20:20:231
3133599152,8cyclictest12260-21apt-get19:40:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional