You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-12 - 23:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sun Jul 12, 2026 12:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30364996652,10cyclictest35966irq/58-eth012:05:133
30364996126,24cyclictest35966irq/58-eth010:50:153
30346995416,5cyclictest131rcu_preempt10:55:131
3036499455,26cyclictest35966irq/58-eth009:35:143
3036499455,26cyclictest35966irq/58-eth009:35:133
3036499445,24cyclictest35966irq/58-eth010:10:133
3036499433,27cyclictest35966irq/58-eth012:15:123
3033599412,24cyclictest27340-21ssh10:10:130
3036499403,24cyclictest35966irq/58-eth012:10:133
3036499403,21cyclictest35966irq/58-eth010:05:143
3036499402,25cyclictest35966irq/58-eth007:45:123
3036499384,23cyclictest35966irq/58-eth011:40:123
3036499381,24cyclictest35966irq/58-eth011:55:123
3036499381,24cyclictest35966irq/58-eth011:55:123
30346993816,5cyclictest131rcu_preempt07:40:121
3033599381,22cyclictest16439-21ssh10:40:130
3036499374,18cyclictest35966irq/58-eth009:50:133
3036499362,22cyclictest35966irq/58-eth009:20:143
3036499351,21cyclictest35966irq/58-eth012:35:143
3036499343,18cyclictest35966irq/58-eth008:05:133
3036499334,17cyclictest35966irq/58-eth009:10:123
3036499324,20cyclictest35966irq/58-eth010:20:153
30364993210,9cyclictest0-21swapper/311:35:133
30364993210,13cyclictest35966irq/58-eth010:45:143
3036499304,15cyclictest35966irq/58-eth009:15:133
30335993019,3cyclictest12-21ksoftirqd/012:30:120
3035499293,17cyclictest0-21swapper/212:15:012
3035499290,22cyclictest7497-21ssh10:35:152
30346992913,3cyclictest21-21ksoftirqd/110:30:151
3033599293,14cyclictest12-21ksoftirqd/010:30:150
3033599291,4cyclictest131rcu_preempt08:40:120
30354992614,4cyclictest32694-21munin-plugin-st08:20:022
3035499261,3cyclictest131rcu_preempt08:00:012
30354992611,6cyclictest0-21swapper/207:10:282
3035499261,14cyclictest0-21swapper/212:33:062
3036499255,12cyclictest35966irq/58-eth009:40:123
3036499252,16cyclictest35966irq/58-eth011:30:133
3035499258,9cyclictest1423-21awk12:00:282
3035499254,8cyclictest25045-21vmstat09:30:342
3035499251,5cyclictest1439-21latency_hist10:50:012
3035499251,17cyclictest16089-21diskmemload10:54:262
3035499251,13cyclictest16089-21diskmemload11:10:122
3035499251,13cyclictest16089-21diskmemload10:30:022
3035499251,13cyclictest16089-21diskmemload09:50:432
3035499251,11cyclictest16089-21diskmemload12:26:322
30346992520,3cyclictest1953-21ssh10:50:151
3033599252,2cyclictest131rcu_preempt09:25:130
3036499245,9cyclictest0-21swapper/311:50:123
3036499242,15cyclictest35966irq/58-eth012:00:133
3036499242,14cyclictest35966irq/58-eth011:00:143
30364992418,4cyclictest0-21swapper/310:55:323
30364992416,6cyclictest0-21swapper/311:20:013
3036499241,15cyclictest0-21swapper/310:25:103
3035499244,13cyclictest27-21ksoftirqd/207:25:122
30354992413,9cyclictest2267-21ssh11:44:262
30354992413,3cyclictest11806-21irqcore09:05:172
3035499241,17cyclictest16089-21diskmemload11:15:582
3035499241,10cyclictest19751-21ls09:30:012
3035499241,10cyclictest16089-21diskmemload12:08:102
3036499233,14cyclictest0-21swapper/310:40:143
3036499232,15cyclictest35966irq/58-eth011:25:123
3035499235,5cyclictest30723-21awk07:40:292
3035499231,9cyclictest18341-21kthreadcore11:35:192
3035499231,9cyclictest16089-21diskmemload11:54:222
3035499231,4cyclictest5862-21kthreadcore11:45:222
30354992312,9cyclictest26206-21ssh11:02:232
3035499231,19cyclictest2145-21ssh09:20:162
3035499231,15cyclictest2923-21ls07:15:142
3035499231,14cyclictest16089-21diskmemload10:55:202
30354992311,3cyclictest16568-21df09:10:132
3035499231,10cyclictest16089-21diskmemload09:19:232
3035499230,14cyclictest31080-21ls12:35:232
3033599237,3cyclictest131rcu_preempt12:20:010
3033599232,5cyclictest31023-21irqrtprio07:10:200
3033599232,2cyclictest131rcu_preempt12:05:120
3033599232,2cyclictest131rcu_preempt11:40:130
3033599231,17cyclictest9454-21ssh10:00:150
3033599230,2cyclictest131rcu_preempt09:20:150
3036499223,13cyclictest0-21swapper/310:15:133
3036499222,11cyclictest35966irq/58-eth008:50:153
30364992216,4cyclictest0-21swapper/309:25:263
30364992213,7cyclictest0-21swapper/312:30:163
30364992213,6cyclictest0-21swapper/308:50:013
30364992213,6cyclictest0-21swapper/308:50:013
3035499228,3cyclictest18043-21kthreadcore09:45:232
3035499227,7cyclictest16089-21diskmemload10:35:012
3035499225,9cyclictest27956-21wc08:15:012
3035499225,9cyclictest20009-21ssh10:05:222
3035499221,9cyclictest16089-21diskmemload10:44:462
3035499221,8cyclictest16089-21diskmemload10:20:182
3035499221,7cyclictest0-21swapper/211:21:222
3035499221,15cyclictest16089-21diskmemload10:19:052
3035499221,15cyclictest16089-21diskmemload09:57:092
3035499221,14cyclictest8670-21apt-get11:30:152
3035499221,14cyclictest16089-21diskmemload09:41:572
3035499221,13cyclictest16089-21diskmemload12:20:032
3034699221,7cyclictest25046-21awk09:30:331
3034699221,17cyclictest17582-21ssh11:35:121
3033599226,3cyclictest131rcu_preempt11:15:140
3033599222,3cyclictest131rcu_preempt10:25:290
30335992214,5cyclictest22946-21ssh12:13:520
3033599221,16cyclictest25476-21apt-get09:15:120
30364992116,3cyclictest0-21swapper/307:35:113
30364992115,4cyclictest321rcuc/307:20:313
30364992115,4cyclictest0-21swapper/312:20:543
30364992115,4cyclictest0-21swapper/311:20:193
30364992115,4cyclictest0-21swapper/310:35:153
30364992115,4cyclictest0-21swapper/309:30:123
30364992114,5cyclictest0-21swapper/311:10:123
30364992114,5cyclictest0-21swapper/311:05:133
30364992114,5cyclictest0-21swapper/309:45:133
30364992113,5cyclictest0-21swapper/308:25:143
30364992112,6cyclictest0-21swapper/307:50:183
3035499215,9cyclictest27-21ksoftirqd/211:55:142
3035499215,9cyclictest27-21ksoftirqd/211:55:132
3035499211,6cyclictest0-21swapper/209:35:302
3035499211,6cyclictest0-21swapper/209:35:302
30354992111,8cyclictest0-21swapper/207:45:142
3035499211,14cyclictest16089-21diskmemload11:26:242
30354992110,9cyclictest0-21swapper/208:30:252
3035499210,6cyclictest167-21systemd-network12:18:442
3035499210,4cyclictest28774-21ssh11:05:132
3034699215,2cyclictest131rcu_preempt11:06:381
3034699214,11cyclictest819-21ssh12:20:121
30346992111,7cyclictest9014-21cron10:00:011
3033599212,2cyclictest131rcu_preempt11:30:190
3033599211,7cyclictest11679-21wc11:50:010
3033599211,7cyclictest10508-21ps12:25:200
3033599211,4cyclictest0-21swapper/012:35:200
3033599211,3cyclictest131rcu_preempt10:45:120
3033599211,2cyclictest131rcu_preempt10:15:130
3033599211,15cyclictest14499-21ssh10:55:250
30364992014,4cyclictest0-21swapper/312:26:473
30364992014,4cyclictest0-21swapper/310:00:323
30364992014,4cyclictest0-21swapper/309:55:083
30364992014,4cyclictest0-21swapper/307:55:173
30364992013,5cyclictest0-21swapper/310:30:163
30364992013,5cyclictest0-21swapper/308:10:153
30364992012,5cyclictest0-21swapper/311:45:323
30364992012,5cyclictest0-21swapper/307:30:263
30364992010,7cyclictest33-21ksoftirqd/309:00:273
3035499206,4cyclictest14133-21irqcore08:00:172
3035499206,4cyclictest14133-21irqcore08:00:172
3035499205,12cyclictest556-21sendmail_mailqu08:50:272
3035499202,7cyclictest27-21ksoftirqd/207:23:022
3035499201,7cyclictest0-21swapper/210:11:072
3035499201,6cyclictest19496-21munin-run08:40:022
30354992013,4cyclictest0-21swapper/208:05:042
3035499200,9cyclictest930-21snmpd08:27:142
3034699203,15cyclictest14832-21ssh09:45:131
3034699202,14cyclictest25337-21ssh10:45:141
3034699202,14cyclictest12083-21ssh11:50:141
30346992013,4cyclictest2144-21sh09:20:151
3034699201,2cyclictest131rcu_preempt07:20:121
30346992012,5cyclictest0-21swapper/108:45:021
30346992011,6cyclictest688-21sshd09:18:501
3033599202,9cyclictest0-21swapper/011:51:140
3033599202,4cyclictest12-21ksoftirqd/011:35:170
3033599202,3cyclictest29880-21ssh09:51:550
3033599201,6cyclictest21955-21/usr/sbin/munin10:05:240
30335992013,5cyclictest1909-21cat10:50:140
3033599200,3cyclictest131rcu_preempt11:20:140
3033599200,2cyclictest131rcu_preempt11:55:130
3033599200,2cyclictest131rcu_preempt11:55:130
3036499194,9cyclictest321rcuc/307:15:123
30364991914,3cyclictest0-21swapper/308:45:033
30364991913,4cyclictest0-21swapper/308:55:233
30364991913,4cyclictest0-21swapper/308:35:143
30364991913,4cyclictest0-21swapper/307:25:163
3036499191,12cyclictest0-21swapper/308:00:163
3036499191,12cyclictest0-21swapper/308:00:153
3035499191,9cyclictest9408-21/usr/sbin/munin10:00:292
3035499191,8cyclictest23232-21awk08:40:242
3035499191,7cyclictest930-21snmpd07:39:422
3035499191,12cyclictest24692-21apt-get08:45:132
3035499191,12cyclictest24692-21apt-get08:45:122
3034699199,7cyclictest4055-21df_inode10:15:141
3034699199,7cyclictest32484-21kthreadcore09:35:241
3034699199,7cyclictest32484-21kthreadcore09:35:231
3034699199,7cyclictest10123-21netstat12:05:251
3034699192,8cyclictest21003-21sshd11:55:131
3034699192,8cyclictest21003-21sshd11:55:121
3034699191,9cyclictest131rcu_preempt07:55:161
3034699191,7cyclictest14307-21grep10:20:221
30346991913,4cyclictest0-21swapper/112:35:281
30346991912,4cyclictest22293-21irqcore10:25:161
30346991912,4cyclictest10644-21kthreadcore11:30:231
30346991911,6cyclictest4245-21cat11:25:311
30346991911,5cyclictest11795-21usage_Anybus10:35:311
3033599196,2cyclictest131rcu_preempt09:30:490
3033599194,3cyclictest131rcu_preempt08:05:020
3033599194,3cyclictest131rcu_preempt08:05:010
3033599192,2cyclictest131rcu_preempt12:22:510
3033599191,9cyclictest0-21swapper/012:02:060
3033599191,7cyclictest6787-21/usr/sbin/munin09:00:160
3033599191,4cyclictest12-21ksoftirqd/009:36:320
3033599191,4cyclictest12-21ksoftirqd/009:36:310
30335991912,5cyclictest28399-21sed11:05:020
30335991911,6cyclictest6445-21cron09:00:010
30335991911,6cyclictest11555-21awk10:35:300
3033599191,10cyclictest0-21swapper/011:10:570
3033599191,10cyclictest0-21swapper/010:21:280
30335991910,6cyclictest31734-21tr08:50:210
3036499182,10cyclictest0-21swapper/309:05:133
30364991812,4cyclictest0-21swapper/307:40:183
30364991812,4cyclictest0-21swapper/307:10:023
30364991812,4cyclictest0-21swapper/307:10:013
3036499181,11cyclictest0-21swapper/308:20:163
30364991810,4cyclictest0-21swapper/307:10:153
3035499187,9cyclictest17485-21hwlatdetect07:30:162
3035499186,6cyclictest7133-21irqcore09:00:192
3035499185,10cyclictest0-21swapper/208:55:212
3035499181,9cyclictest5586-21kthreadcore07:50:202
3035499181,4cyclictest0-21swapper/208:20:112
3035499181,13cyclictest30458-21latency_hist07:10:012
3034699189,6cyclictest27186-21grep08:10:311
3034699188,7cyclictest9431-21apt-get10:00:141
3034699184,11cyclictest31297-21cat10:10:291
3034699182,12cyclictest23880-21ssh09:50:131
3034699181,9cyclictest15183-21kthreadcore08:00:201
3034699181,9cyclictest15183-21kthreadcore08:00:201
3034699181,9cyclictest131rcu_preempt09:40:151
30346991814,2cyclictest0-21swapper/107:35:131
3034699181,3cyclictest0-21swapper/112:11:481
30346991813,3cyclictest0-21swapper/107:50:251
3034699181,13cyclictest1849-21apt-get08:55:131
30346991810,6cyclictest7401-21sshd07:15:451
30346991810,6cyclictest20239-21sed10:40:281
3034699180,9cyclictest2089966ssh11:17:491
3034699180,10cyclictest7843-21tr09:00:211
3033599184,3cyclictest131rcu_preempt09:45:200
3033599184,3cyclictest131rcu_preempt07:35:020
3033599182,2cyclictest131rcu_preempt09:58:370
3033599181,4cyclictest7703-21ls07:50:290
3033599181,4cyclictest12-21ksoftirqd/011:05:130
3033599181,2cyclictest131rcu_preempt09:41:360
30335991812,3cyclictest30326-21cyclictest08:45:130
30335991812,3cyclictest30326-21cyclictest08:45:120
30335991812,3cyclictest0-21swapper/011:25:130
30335991811,4cyclictest0-21swapper/008:30:170
3036499172,9cyclictest0-21swapper/308:15:133
30364991711,4cyclictest0-21swapper/308:30:183
3034699179,5cyclictest18770-21/usr/sbin/munin12:30:141
3034699178,6cyclictest1-21systemd08:29:311
3034699177,7cyclictest19489-21cron08:40:011
3034699177,2cyclictest0-21swapper/109:10:311
3034699172,3cyclictest131rcu_preempt11:45:261
3034699171,3cyclictest0-21swapper/107:25:391
3034699171,2cyclictest131rcu_preempt12:15:111
30346991712,3cyclictest30326-21cyclictest11:20:131
30346991712,3cyclictest0-21swapper/112:25:131
30346991711,3cyclictest7225-21sh11:10:211
3034699171,13cyclictest29980-21ssh12:00:121
3034699171,13cyclictest26721-21diskstats11:40:141
30346991710,4cyclictest28480-21unixbench_multi08:45:291
30346991710,4cyclictest28480-21unixbench_multi08:45:281
30346991710,4cyclictest14490-21cut09:05:211
3034699170,6cyclictest315-21latency_hist08:20:011
3034699170,2cyclictest131rcu_preempt10:05:241
3034699170,2cyclictest131rcu_preempt09:25:131
3033599179,6cyclictest614-21dbus-daemon08:24:280
3033599179,6cyclictest30904-21cut08:15:210
3033599177,2cyclictest12-21ksoftirqd/007:25:180
3033599172,4cyclictest12-21ksoftirqd/009:11:130
30335991712,3cyclictest22709-21kernelversion07:35:180
3035499161,12cyclictest30424-21latency_hist07:10:012
3034699166,2cyclictest0-21swapper/107:45:201
3034699165,8cyclictest18547-21apt-get08:05:121
3034699163,2cyclictest131rcu_preempt11:02:541
3034699163,10cyclictest11317-21kthreadcore08:30:211
3034699161,4cyclictest0-21swapper/108:50:211
3034699161,12cyclictest1356-21memory07:10:231
3033599168,6cyclictest19152-21sed08:05:180
3033599168,6cyclictest12157-21tail09:05:180
3033599168,6cyclictest12049-21kthreadcore07:55:220
3033599167,7cyclictest3784-21users07:45:300
3033599163,10cyclictest3783-21kthreadcore07:15:180
3033599161,9cyclictest27538-21usage_EthcTask08:10:310
3033599161,6cyclictest16842-21kthreadcore08:35:200
3034699152,10cyclictest18136-21kthreadcore07:30:191
3034699151,11cyclictest471-21apt-get08:20:141
3033599151,11cyclictest7985-21if_eth107:20:180
3033599151,11cyclictest5394-21diskstats08:25:150
3033599151,11cyclictest26873-21apt-get07:40:140
3034699141,10cyclictest30245-21latency_hist07:10:011
3034699131,9cyclictest30245-21latency_hist07:10:011
3033599121,8cyclictest30326-21cyclictest07:10:020
3033599121,8cyclictest30326-21cyclictest07:10:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional