You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-25 - 11:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Mon May 25, 2026 00:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
9507998849,25cyclictest35966irq/58-eth019:35:143
9507996048,6cyclictest35966irq/58-eth023:05:113
950799467,20cyclictest35966irq/58-eth000:15:143
9507994628,9cyclictest35966irq/58-eth021:55:123
950799454,28cyclictest35966irq/58-eth021:25:133
950799451,31cyclictest35966irq/58-eth019:25:133
950799445,25cyclictest35966irq/58-eth023:45:123
950799433,25cyclictest35966irq/58-eth022:50:153
950799423,23cyclictest35966irq/58-eth000:20:133
950799423,21cyclictest35966irq/58-eth021:05:133
950799412,24cyclictest35966irq/58-eth021:30:123
949699389,9cyclictest5294-21latency_hist22:45:022
949699389,9cyclictest5294-21latency_hist22:45:012
949699371,9cyclictest15519-21fschecks_time00:20:182
949699370,13cyclictest3153666ssh23:51:242
948899371,29cyclictest31912-21rm21:30:121
948899371,22cyclictest28083-21ssh21:10:151
948899370,30cyclictest2534-21ssh23:55:121
947699371,13cyclictest6255-21apt-get00:15:130
948899361,17cyclictest10917-21apt-get19:45:121
950799354,21cyclictest35966irq/58-eth023:40:133
948899351,6cyclictest29022-21apt-get22:40:121
948899351,6cyclictest29022-21apt-get22:40:121
948899351,12cyclictest26491-21ntp_192.168.11421:25:241
948899350,10cyclictest5827-21ssh00:30:361
947699359,5cyclictest0-21swapper/020:45:270
948899348,23cyclictest13773-21basename00:35:301
948899348,23cyclictest13773-21basename00:35:291
948899341,9cyclictest9880-21ntp_192.168.11400:15:251
948899341,7cyclictest21989-21missed_timers23:10:231
948899341,11cyclictest31360-21/usr/sbin/munin23:35:181
948899341,10cyclictest4825-21df_inode21:15:171
948899340,6cyclictest18859-21kthreadcore19:50:231
948899340,6cyclictest18859-21kthreadcore19:50:221
950799334,21cyclictest35966irq/58-eth019:55:143
9507993319,8cyclictest35966irq/58-eth022:20:143
948899336,4cyclictest0-21swapper/123:50:221
948899331,8cyclictest27133-21ssh22:38:121
948899331,8cyclictest15914-21cut19:15:191
948899331,7cyclictest31178-21ntp_states23:15:271
948899331,7cyclictest18211-21ssh22:33:181
948899331,7cyclictest12029-21ntp_states20:50:261
948899331,5cyclictest16079-21ntp_states21:55:261
948899331,3cyclictest12157-21apt-get20:20:141
9476993314,3cyclictest131rcu_preempt00:10:110
947699331,13cyclictest6230-21ntp_states23:55:260
947699331,10cyclictest16868-21ssh23:45:120
950799322,18cyclictest35966irq/58-eth020:40:133
950799321,21cyclictest35966irq/58-eth023:25:123
949699320,11cyclictest930-21snmpd19:30:132
948899328,21cyclictest27636-21diskmemload22:00:271
948899321,9cyclictest26539-21kthreadcore22:55:211
948899321,9cyclictest23169-21df21:05:161
948899321,8cyclictest27580-21ntp_states19:25:261
948899321,8cyclictest24331-21ntp_states00:05:301
948899321,8cyclictest22391-21ssh23:46:161
948899321,8cyclictest17528-21cpu21:40:131
948899321,7cyclictest9841-21kthreadcore21:35:201
948899321,7cyclictest7824-21proc_pri23:20:261
948899321,7cyclictest21683-21nfs_client21:00:231
948899321,6cyclictest6483-21kthreadcore22:45:201
948899321,6cyclictest3072-21cut19:35:211
948899321,6cyclictest24635-21cut23:30:221
948899321,6cyclictest24331-21unixbench_multi19:55:281
948899321,6cyclictest18243-21tr22:15:201
948899321,6cyclictest14187-21cut21:20:181
948899321,6cyclictest14187-21cut21:20:171
948899321,5cyclictest9204-21cat23:05:011
948899321,5cyclictest7049-21memory20:45:231
948899321,4cyclictest1866-21ntp_192.168.11422:05:251
947699328,5cyclictest14326-21tr21:55:210
9476993211,3cyclictest16965-21apt20:25:120
950799313,13cyclictest35966irq/58-eth000:05:163
950799311,15cyclictest35966irq/58-eth020:10:123
948899311,8cyclictest26623-21df_inode20:35:151
948899311,7cyclictest4909-21tr21:50:221
948899311,6cyclictest7150-21/usr/sbin/munin22:10:121
948899311,6cyclictest32137-21ssh21:46:201
948899311,6cyclictest27297-21cut00:25:191
948899311,6cyclictest2604-21cut22:25:181
948899311,6cyclictest21254-21ssh00:21:521
948899311,6cyclictest15469-21ssh22:50:211
948899311,6cyclictest13502-21grep23:05:271
948899311,6cyclictest13417-21cpuspeed20:55:141
948899311,6cyclictest11585-21ssh00:00:151
948899311,6cyclictest11585-21ssh00:00:141
948899311,5cyclictest9616-21kthreadcore23:40:201
948899310,7cyclictest21782-21apt-get20:30:141
9476993111,3cyclictest12707-21latency_hist23:25:020
947699311,11cyclictest2528766ssh22:20:130
947699311,10cyclictest12091-21ntp_192.168.11421:35:250
9496993012,13cyclictest628-21tcf-agent23:10:122
948899301,8cyclictest29652-21cpu00:10:141
948899301,7cyclictest9641-21awk19:10:131
948899301,7cyclictest3497-21idleruntime-cro20:45:011
948899301,6cyclictest7127-21cut20:15:011
948899301,6cyclictest30569-21ssh22:21:001
948899301,6cyclictest19872-21cut19:20:181
948899301,6cyclictest19872-21cut19:20:171
948899301,6cyclictest11924-21cut20:20:011
948899301,6cyclictest11924-21cut20:20:001
948899301,6cyclictest1174-21latency_hist19:35:011
948899301,24cyclictest13108-21ssh23:25:141
947699307,6cyclictest26081-21cut21:25:230
947699307,6cyclictest2357-21latency_hist23:55:010
947699307,6cyclictest10057-21fschecks_count00:35:180
947699307,6cyclictest10057-21fschecks_count00:35:180
947699302,9cyclictest11366-21ssh23:40:240
947699301,2cyclictest131rcu_preempt21:20:130
947699301,2cyclictest131rcu_preempt21:20:120
950799292,15cyclictest35966irq/58-eth021:15:133
948899291,7cyclictest9603-21date19:10:011
948899291,6cyclictest30190-21grep20:05:151
948899291,6cyclictest17504-21sed20:25:171
950799281,15cyclictest35966irq/58-eth019:50:143
950799281,15cyclictest35966irq/58-eth019:50:143
948899281,6cyclictest29924-21idleruntime-cro20:05:011
948899280,6cyclictest10713-21latency_hist19:45:011
947699288,5cyclictest17403-21cut22:50:230
947699288,4cyclictest22609-21kthreadcore00:05:260
947699286,6cyclictest13514-21fschecks_time20:55:150
947699281,9cyclictest28783-21irqstats21:10:210
947699281,12cyclictest11564-21ssh00:00:140
947699281,12cyclictest11564-21ssh00:00:130
947699281,11cyclictest15306-21ssh00:20:140
947699277,14cyclictest17515-21ssh21:40:120
947699271,10cyclictest27063-21munin-run23:15:020
950799262,17cyclictest35966irq/58-eth000:10:133
950799262,15cyclictest35966irq/58-eth020:15:143
950799262,15cyclictest35966irq/58-eth020:15:133
9507992614,7cyclictest35966irq/58-eth022:35:133
950799261,14cyclictest0-21swapper/322:30:143
949699261,8cyclictest1513266ssh23:06:272
9496992617,6cyclictest536-21usb-storage22:55:112
947699261,5cyclictest895166kworker/u9:223:30:140
950799258,10cyclictest35966irq/58-eth022:55:133
9507992512,10cyclictest35966irq/58-eth000:30:133
949699251,7cyclictest2849166ssh23:32:142
949699251,7cyclictest2077266rm23:28:512
949699251,7cyclictest1639266ssh22:15:162
949699251,6cyclictest761766ssh23:55:462
949699251,6cyclictest651066ssh23:01:552
947699251,7cyclictest13162-21ntp_states23:05:250
947699250,2cyclictest131rcu_preempt23:15:130
950799244,12cyclictest35966irq/58-eth023:35:163
949699241,6cyclictest2137966ssh21:40:272
949699241,6cyclictest2017366ssh21:59:002
949699241,6cyclictest1467166ssh00:19:472
947699246,4cyclictest27636-21diskmemload23:39:430
9507992317,4cyclictest321rcuc/321:23:583
9507992317,4cyclictest321rcuc/321:23:573
9507992317,4cyclictest321rcuc/300:38:043
9507992317,4cyclictest321rcuc/300:38:033
949699239,11cyclictest11553-21sshd19:45:172
949699231,6cyclictest1482066ssh21:37:192
949699231,6cyclictest1246866ssh22:47:462
9496992315,4cyclictest930-21snmpd19:35:142
9496992312,9cyclictest2587-21ssh00:11:042
9496992312,8cyclictest9501-21latency_hist00:35:012
949699231,18cyclictest13748-21apt-get21:20:122
949699231,18cyclictest13748-21apt-get21:20:112
9496992311,5cyclictest22938-21ssh23:46:532
9496992310,9cyclictest32087-21sshd21:46:172
949699230,6cyclictest941866ssh23:21:062
9507992215,5cyclictest321rcuc/300:00:373
9507992215,5cyclictest321rcuc/300:00:373
950799221,12cyclictest0-21swapper/320:50:133
949699221,9cyclictest93066snmpd20:20:232
949699221,6cyclictest1966566ssh22:50:362
9496992211,9cyclictest4723-21ssh21:31:032
949699221,13cyclictest1463166ssh22:13:362
949699221,12cyclictest10195-21kthreadcore23:40:222
947699223,3cyclictest131rcu_preempt21:15:270
9507992116,3cyclictest321rcuc/300:27:363
9507992115,4cyclictest321rcuc/323:22:503
9507992115,4cyclictest0-21swapper/323:30:493
9507992115,4cyclictest0-21swapper/321:52:163
950799211,14cyclictest0-21swapper/322:10:143
949699218,10cyclictest261rcuc/222:30:202
949699217,6cyclictest11025-21sshd21:17:182
949699212,10cyclictest20032-21ssh22:35:142
949699211,6cyclictest629466ssh22:09:222
949699211,6cyclictest2570266ssh22:00:292
949699211,13cyclictest394766ssh23:35:522
949699211,11cyclictest15086-21memory00:00:252
949699211,11cyclictest15086-21memory00:00:252
949699211,10cyclictest11925-21sendmail20:20:012
949699211,10cyclictest11925-21sendmail20:20:002
9496992110,9cyclictest24122-21ssh00:25:042
947699213,4cyclictest12-21ksoftirqd/022:25:410
9476992112,6cyclictest29897-21sshd20:01:180
947699211,16cyclictest20511-21apt-get19:55:140
9476992111,6cyclictest0-21swapper/000:30:140
947699211,15cyclictest5703-21ssh22:45:150
947699211,14cyclictest24234-21ssh00:25:110
9507992015,3cyclictest321rcuc/322:42:063
9507992015,3cyclictest321rcuc/322:42:053
9507992015,3cyclictest321rcuc/321:39:593
9507992014,4cyclictest321rcuc/321:48:223
9507992014,4cyclictest0-21swapper/323:55:073
9507992014,4cyclictest0-21swapper/323:00:283
9507992014,4cyclictest0-21swapper/322:06:363
9507992014,4cyclictest0-21swapper/322:04:413
9507992014,4cyclictest0-21swapper/321:11:363
9507992013,5cyclictest0-21swapper/323:10:023
949699209,8cyclictest0-21swapper/220:35:242
949699208,10cyclictest9963-21ps19:10:172
949699208,10cyclictest616-21systemd-logind00:35:382
949699208,10cyclictest616-21systemd-logind00:35:372
949699206,6cyclictest17661-21wc19:15:222
949699205,12cyclictest15828-21fschecks_count19:50:172
949699205,12cyclictest15828-21fschecks_count19:50:162
949699202,9cyclictest5721-21tr21:50:222
949699201,11cyclictest25306-21ssh22:20:132
949699201,10cyclictest8546-21apt-get20:50:142
949699201,10cyclictest30809-21grep23:15:222
949699201,10cyclictest25970-21kthreadcore21:25:212
947699203,4cyclictest12-21ksoftirqd/022:40:290
947699203,4cyclictest12-21ksoftirqd/022:40:280
9476992010,7cyclictest25216-21ntpq22:00:260
950799197,8cyclictest35966irq/58-eth021:40:133
9507991914,3cyclictest0-21swapper/322:25:313
9507991913,4cyclictest0-21swapper/323:15:053
9507991913,4cyclictest0-21swapper/322:45:133
9507991913,4cyclictest0-21swapper/322:16:453
9507991912,5cyclictest0-21swapper/323:50:223
950799191,11cyclictest0-21swapper/319:45:023
949699199,7cyclictest1906-21/usr/sbin/munin22:25:172
949699198,8cyclictest28033-21/usr/sbin/munin21:10:282
949699193,8cyclictest24999-21kthreadcore21:05:222
949699193,8cyclictest24780-21kernelversion19:25:182
949699191,9cyclictest8428-21kthreadcore19:40:202
949699191,9cyclictest29111-21timerwakeupswit20:00:262
949699191,9cyclictest17206-21unixbench_singl20:55:282
9496991913,3cyclictest20693-21cut19:55:172
949699191,15cyclictest18476-21hwlatdetect21:00:172
949699191,10cyclictest30866-21kthreadcore20:05:192
949699191,10cyclictest2720-21fschecks_time20:10:162
949699191,10cyclictest25183-21ntp_kernel_pll_20:30:242
949699191,10cyclictest23076-21unixbench_multi19:20:282
949699191,10cyclictest23076-21unixbench_multi19:20:272
949699191,10cyclictest20797-21fschecks_count00:05:192
947699194,11cyclictest23921-21df22:55:140
9476991913,4cyclictest0-21swapper/023:25:260
9507991812,4cyclictest0-21swapper/319:10:233
950799181,12cyclictest0-21swapper/320:30:143
950799181,10cyclictest0-21swapper/321:00:173
949699182,9cyclictest9536-21latency_hist19:10:022
949699181,9cyclictest4115-21irqcore20:45:182
949699181,9cyclictest32296-21kthreadcore20:40:192
949699181,9cyclictest17291-21sort20:25:182
947699182,12cyclictest30653-21ssh22:05:140
947699181,3cyclictest0-21swapper/023:00:510
9476991813,3cyclictest12-21ksoftirqd/022:15:200
947699181,2cyclictest0-21swapper/021:53:010
9476991812,3cyclictest12-21ksoftirqd/020:15:140
9476991812,3cyclictest12-21ksoftirqd/020:15:130
9476991811,4cyclictest0-21swapper/019:20:160
9476991811,4cyclictest0-21swapper/019:20:150
9476991811,3cyclictest12-21ksoftirqd/019:40:130
9476991810,6cyclictest11223-21cat22:10:290
950799172,10cyclictest0-21swapper/320:00:133
9507991711,4cyclictest0-21swapper/320:45:153
950799171,10cyclictest0-21swapper/320:25:163
947699179,6cyclictest25372-21sed20:30:260
947699178,7cyclictest18979-21cron19:20:010
947699177,8cyclictest2274-21users20:05:310
947699177,7cyclictest24382-21cat22:35:300
947699172,3cyclictest131rcu_preempt21:05:020
947699171,9cyclictest27389-21grep21:45:180
947699171,3cyclictest131rcu_preempt22:35:010
947699171,3cyclictest131rcu_preempt21:30:360
950799162,10cyclictest0-21swapper/320:05:143
9507991611,3cyclictest321rcuc/319:20:303
9507991611,3cyclictest321rcuc/319:20:293
9507991611,3cyclictest0-21swapper/320:55:113
9507991611,3cyclictest0-21swapper/319:45:173
947699168,6cyclictest9744-21forks19:10:150
947699167,6cyclictest8794-21basename20:50:170
947699162,11cyclictest26533-21ntp_states21:05:260
947699161,7cyclictest4028-21kthreadcore19:35:210
947699161,12cyclictest31454-21diskstats20:40:160
9476991610,4cyclictest2844-21if_eth120:10:170
9476991610,4cyclictest15969-21grep20:20:270
9476991610,3cyclictest11350-21cut19:45:160
950799152,9cyclictest0-21swapper/319:15:143
950799151,11cyclictest0-21swapper/320:20:143
9507991510,3cyclictest0-21swapper/320:35:133
947699158,5cyclictest28992-21/usr/sbin/munin19:30:140
947699157,6cyclictest27749-21sed19:25:260
947699157,6cyclictest20132-21cat19:50:320
947699157,6cyclictest20132-21cat19:50:310
9476991510,3cyclictest26865-21irqcore20:35:190
950799121,9cyclictest0-21swapper/319:30:173
947699121,8cyclictest9515-21latency_hist19:10:020
950799111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional