You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-23 - 21:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Mon Mar 23, 2026 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17248994916,24cyclictest27-21ksoftirqd/210:35:162
1724899471,38cyclictest28246-21sh10:00:142
1723899475,33cyclictest31503-21ssh10:20:141
1723899471,16cyclictest30699-21apt-get11:50:141
1723899465,4cyclictest131rcu_preempt11:45:151
1723899453,40cyclictest30302-21ssh09:25:141
17248994411,15cyclictest261rcuc/212:10:112
1723899442,8cyclictest0-21swapper/109:40:131
1723899442,37cyclictest28232-21ssh10:00:131
1723899442,34cyclictest1311-21ssh09:45:121
1723899441,41cyclictest19276-21ssh12:20:111
1723899411,36cyclictest18334-21rm11:25:131
1724899401,32cyclictest3885-21ssh11:35:132
1723899405,31cyclictest0-21swapper/112:25:121
1723899403,27cyclictest24882-21ssh12:05:101
17229994026,8cyclictest0-21swapper/010:25:140
17248993918,17cyclictest27-21ksoftirqd/208:05:152
1723899391,4cyclictest131rcu_preempt12:15:131
1724899382,18cyclictest8248-21ntp_192.168.11412:30:242
17256993616,13cyclictest0-21swapper/309:50:143
1723899364,22cyclictest6861-21ssh09:30:131
1723899361,11cyclictest10181-21sendmail_mailqu08:40:271
1723899361,10cyclictest2351-21tr08:35:191
17256993520,8cyclictest28366irq/58-eth011:15:123
17256993520,8cyclictest28366irq/58-eth011:15:113
1723899359,4cyclictest24538-21kthreadcore11:10:191
1723899356,5cyclictest5498-21kthreadcore11:35:201
1723899352,25cyclictest13565-21ssh10:10:121
1723899352,25cyclictest13565-21ssh10:10:121
1723899352,10cyclictest18977-21unixbench_singl11:05:311
1723899351,8cyclictest10061-21latency_hist08:10:001
1723899351,10cyclictest20006-21ps09:55:181
1723899341,25cyclictest0-21swapper/111:20:141
1723899341,14cyclictest6085-21ntp_192.168.11410:40:241
1723899341,11cyclictest15868-21df_inode09:35:141
1723899341,11cyclictest13852-21/usr/sbin/munin12:35:171
1723899341,10cyclictest996-21irqstats11:15:181
1723899341,10cyclictest996-21irqstats11:15:181
17248993318,9cyclictest2166-21ssh12:27:242
1723899331,11cyclictest9896-21ls11:00:291
1723899331,11cyclictest512-21proc_pri08:30:271
1723899331,11cyclictest27830-21tr08:25:201
1723899331,11cyclictest24683-21kthreadcore07:50:201
1723899331,11cyclictest16045-21kthreadcore08:15:181
1723899331,10cyclictest1722-21ntp_kernel_err09:05:241
17256993220,9cyclictest0-21swapper/309:25:143
1724899323,19cyclictest10331-21ssh09:50:142
1723899323,9cyclictest1-21systemd12:01:231
1723899322,9cyclictest30500-21apt-config09:05:021
1723899322,8cyclictest0-21swapper/112:10:131
1723899321,9cyclictest5846-21cut12:30:191
1723899321,9cyclictest28461-21ssh11:30:181
1723899321,12cyclictest7341-21diskstats11:55:141
1723899321,10cyclictest11483-21awk10:45:121
17229993220,4cyclictest0-21swapper/008:35:140
1722999321,21cyclictest12-21ksoftirqd/008:15:120
1724899315,5cyclictest13597-21ssh12:35:132
1723899314,10cyclictest27228-21grep07:20:151
1723899312,8cyclictest12345-21cpuspeed09:15:141
1723899311,9cyclictest30098-21ps07:55:191
1723899311,8cyclictest12411-21apt-get11:40:021
1723899311,7cyclictest20561-21ntp_192.168.11410:30:241
1723899311,11cyclictest20339-21kthreadcore07:45:201
1723899311,10cyclictest25457-21kthreadcore10:15:231
1723899311,10cyclictest25457-21kthreadcore10:15:221
1723899311,10cyclictest23392-21ps08:20:261
1724899308,14cyclictest27-21ksoftirqd/209:20:112
1724899304,7cyclictest0-21swapper/210:25:152
1724899303,18cyclictest22346irq/40-dwc2_hso07:55:132
1723899305,7cyclictest24008-21cut07:15:201
1723899304,22cyclictest21-21ksoftirqd/107:35:141
1723899302,7cyclictest11667-21ntp_states10:25:281
1723899301,9cyclictest6896-21ntp_192.168.11409:10:241
1723899301,9cyclictest2035-21latency_hist10:40:021
1723899301,11cyclictest760-21wc10:55:281
1723899301,11cyclictest17701-21head07:40:341
1723899301,10cyclictest32481-21sed07:25:191
1722999304,21cyclictest4676-21ssh10:05:150
17256992917,9cyclictest0-21swapper/310:45:103
1724899295,15cyclictest22346irq/40-dwc2_hso10:40:132
1724899292,21cyclictest0-21swapper/209:00:142
1724899292,20cyclictest0-21swapper/208:30:122
1724899292,18cyclictest22346irq/40-dwc2_hso11:40:132
1723899292,10cyclictest20885-21grep08:55:011
1723899292,10cyclictest20885-21grep08:55:011
1723899291,11cyclictest8262-21ntp_192.168.11410:05:271
1722999293,19cyclictest5808-21ssh11:00:120
1722999292,21cyclictest29795-21ssh11:49:240
1722999292,20cyclictest12765-21ssh11:40:050
1722999291,20cyclictest22515-21ssh10:15:140
1722999291,20cyclictest22515-21ssh10:15:130
1725699283,18cyclictest0-21swapper/310:35:243
17256992822,3cyclictest28366irq/58-eth009:35:133
1725699281,15cyclictest28366irq/58-eth008:00:153
1724899288,8cyclictest27-21ksoftirqd/208:55:302
1724899282,19cyclictest10569-21ssh09:30:262
1724899282,17cyclictest16178-21ssh12:00:132
1724899282,11cyclictest0-21swapper/209:10:122
1723899285,6cyclictest25688-21wc09:00:011
1723899283,7cyclictest0-21swapper/109:20:131
1723899281,9cyclictest5500-21kthreadcore07:30:201
1722999285,16cyclictest1337-21ssh09:45:140
1722999282,7cyclictest12-21ksoftirqd/010:40:240
1725699273,12cyclictest28366irq/58-eth012:00:133
17256992721,4cyclictest0-21swapper/311:05:133
1724899272,19cyclictest0-21swapper/207:10:132
1724899272,18cyclictest0-21swapper/208:45:142
17248992710,12cyclictest131rcu_preempt12:23:242
1722999278,8cyclictest12-21ksoftirqd/009:38:010
1722999277,8cyclictest12-21ksoftirqd/011:21:360
1722999273,7cyclictest12-21ksoftirqd/011:26:060
1722999271,21cyclictest16920-21ssh10:30:130
17229992710,11cyclictest151rcuc/011:15:220
17229992710,11cyclictest151rcuc/011:15:220
1725699263,17cyclictest0-21swapper/311:49:253
17256992616,6cyclictest0-21swapper/308:10:113
17256992610,10cyclictest0-21swapper/311:14:233
1724899261,5cyclictest23789-21ssh11:10:132
1722999263,6cyclictest12-21ksoftirqd/012:18:160
1722999262,5cyclictest12-21ksoftirqd/012:05:010
1725699257,12cyclictest28366irq/58-eth010:30:133
1725699253,15cyclictest28366irq/58-eth009:30:133
17256992519,3cyclictest0-21swapper/312:15:123
17256992518,3cyclictest0-21swapper/309:40:133
17256992517,6cyclictest0-21swapper/311:30:293
17256992517,6cyclictest0-21swapper/311:00:123
1724899256,10cyclictest131rcu_preempt11:58:342
1724899252,9cyclictest0-21swapper/208:20:132
1724899251,8cyclictest24811-21ssh09:40:122
17248992514,7cyclictest27-21ksoftirqd/212:05:112
17248992512,10cyclictest26767-21cron07:20:012
1724899251,18cyclictest22346irq/40-dwc2_hso11:50:142
1724899251,15cyclictest19267-21ssh09:55:132
1724899251,12cyclictest19992-21kworker/u8:211:15:132
1724899251,12cyclictest19992-21kworker/u8:211:15:122
17248992510,8cyclictest27-21ksoftirqd/210:46:212
1723899251,6cyclictest21940-21munin-run07:15:011
1723899251,12cyclictest20336-21ssh10:50:121
1722999257,8cyclictest12-21ksoftirqd/010:37:540
1722999257,7cyclictest12-21ksoftirqd/010:48:160
1722999256,6cyclictest12-21ksoftirqd/009:16:240
1722999253,15cyclictest3900-21ssh11:35:130
1722999251,21cyclictest4648-21ssh12:30:130
1722999251,20cyclictest28221-21apt-get10:00:130
17229992511,5cyclictest131rcu_preempt11:53:370
1722999250,8cyclictest12-21ksoftirqd/010:55:140
17256992418,2cyclictest0-21swapper/310:00:133
17256992417,4cyclictest0-21swapper/312:05:123
17256992416,3cyclictest0-21swapper/312:30:123
17256992415,5cyclictest0-21swapper/308:45:123
17256992414,7cyclictest0-21swapper/311:50:233
1724899249,2cyclictest131rcu_preempt11:00:232
1724899242,16cyclictest0-21swapper/210:50:252
1724899241,9cyclictest22505-21ssh10:15:142
1724899241,9cyclictest22505-21ssh10:15:142
17248992412,9cyclictest22047-21ssh10:30:472
1724899241,18cyclictest27349-21ssh11:30:122
1724899241,11cyclictest721-21ssh10:55:272
1722999247,8cyclictest12-21ksoftirqd/010:24:460
1722999246,7cyclictest12-21ksoftirqd/012:37:160
1722999246,7cyclictest12-21ksoftirqd/012:25:590
1722999245,8cyclictest131rcu_preempt12:07:360
1722999245,11cyclictest12-21ksoftirqd/009:59:250
1722999241,7cyclictest24087-21/usr/sbin/munin10:50:270
17229992415,6cyclictest27237-21ssh11:30:050
17229992413,3cyclictest12-21ksoftirqd/009:40:450
17229992412,6cyclictest0-21swapper/010:14:140
17229992412,6cyclictest0-21swapper/010:14:130
1725699239,6cyclictest0-21swapper/310:20:243
1725699239,11cyclictest0-21swapper/312:20:113
1725699233,13cyclictest0-21swapper/310:55:243
17256992319,2cyclictest0-21swapper/310:10:123
17256992319,2cyclictest0-21swapper/310:10:123
17256992318,2cyclictest0-21swapper/310:05:143
17256992318,2cyclictest0-21swapper/309:00:113
17256992318,2cyclictest0-21swapper/307:15:123
17256992316,3cyclictest0-21swapper/307:25:143
17256992313,7cyclictest0-21swapper/309:56:223
17256992313,7cyclictest0-21swapper/308:40:283
1724899239,4cyclictest27-21ksoftirqd/209:05:192
1724899239,4cyclictest131rcu_preempt09:29:412
1724899232,6cyclictest0-21swapper/211:46:242
1724899232,6cyclictest0-21swapper/211:24:232
1724899231,17cyclictest8353-21ssh10:05:292
1724899231,12cyclictest22346irq/40-dwc2_hso08:35:142
1724899231,10cyclictest2211166ssh10:15:012
1724899231,10cyclictest2211166ssh10:15:012
1722999238,4cyclictest12-21ksoftirqd/011:05:580
1722999237,8cyclictest12-21ksoftirqd/009:53:440
1722999237,3cyclictest131rcu_preempt08:50:280
1722999237,3cyclictest131rcu_preempt08:50:280
1722999236,8cyclictest12-21ksoftirqd/009:21:510
17229992317,3cyclictest151rcuc/009:10:590
17229992311,8cyclictest7223-21ssh11:55:120
1722999231,18cyclictest27919-21ls08:25:220
1722999231,15cyclictest16123-21ls08:50:020
17229992310,7cyclictest12-21ksoftirqd/007:20:220
17229992310,3cyclictest131rcu_preempt11:13:000
1725699227,9cyclictest28466irq/59-eth011:25:123
1725699225,10cyclictest0-21swapper/309:45:143
17256992216,3cyclictest0-21swapper/312:10:123
17256992216,3cyclictest0-21swapper/311:40:133
17256992216,3cyclictest0-21swapper/308:55:133
17256992216,3cyclictest0-21swapper/307:50:123
17256992215,3cyclictest0-21swapper/311:20:143
17256992214,5cyclictest28366irq/58-eth010:50:133
17256992213,6cyclictest0-21swapper/312:25:263
17256992213,6cyclictest0-21swapper/310:40:223
17256992212,7cyclictest0-21swapper/312:35:163
17256992212,7cyclictest0-21swapper/308:30:163
1725699221,14cyclictest28366irq/58-eth010:25:153
1725699221,12cyclictest28366irq/58-eth007:45:143
17256992210,8cyclictest0-21swapper/308:25:283
1724899228,3cyclictest0-21swapper/209:38:062
1724899227,9cyclictest131rcu_preempt12:16:142
1724899227,4cyclictest27-21ksoftirqd/207:50:132
1724899227,12cyclictest18520-21fschecks_time11:25:162
1724899224,12cyclictest924-21snmpd08:10:112
1722999226,8cyclictest12-21ksoftirqd/009:30:490
1722999221,4cyclictest0-21swapper/012:11:060
17229992214,6cyclictest23986-21ssh12:20:300
1722999221,15cyclictest30300-21sh09:25:140
17229992211,3cyclictest131rcu_preempt07:30:240
1725699219,9cyclictest0-21swapper/309:20:113
1725699219,8cyclictest28366irq/58-eth011:35:133
1725699219,6cyclictest0-21swapper/309:15:293
1725699214,10cyclictest28366irq/58-eth007:40:123
17256992112,7cyclictest0-21swapper/310:15:123
17256992112,7cyclictest0-21swapper/310:15:123
17256992111,8cyclictest0-21swapper/307:35:293
17256992111,7cyclictest0-21swapper/311:56:133
17256992111,7cyclictest0-21swapper/307:10:143
1724899219,3cyclictest27-21ksoftirqd/207:45:152
1724899218,3cyclictest131rcu_preempt09:15:022
1724899216,12cyclictest27-21ksoftirqd/209:45:282
1724899216,10cyclictest131rcu_preempt11:09:392
1724899211,3cyclictest0-21swapper/210:20:132
1722999213,5cyclictest12-21ksoftirqd/007:50:270
1722999213,5cyclictest12-21ksoftirqd/007:35:200
1722999211,16cyclictest23335-21ls07:50:020
1725699209,8cyclictest0-21swapper/307:55:243
17256992016,2cyclictest0-21swapper/309:05:133
17256992015,3cyclictest0-21swapper/309:10:123
17256992014,3cyclictest0-21swapper/308:35:133
17256992014,3cyclictest0-21swapper/308:15:123
17256992011,6cyclictest0-21swapper/308:50:203
17256992011,6cyclictest0-21swapper/308:50:193
1724899202,14cyclictest31676-21latency_hist07:25:022
1724899201,9cyclictest7097-21kthreadcore07:30:212
1724899201,16cyclictest4371-21timerwakeupswit08:00:312
1724899201,10cyclictest7288-21cut08:40:182
17248992010,8cyclictest0-21swapper/207:25:152
1722999205,3cyclictest131rcu_preempt09:05:200
1722999202,5cyclictest12-21ksoftirqd/008:05:150
17229992011,6cyclictest0-21swapper/007:55:240
1725699199,7cyclictest0-21swapper/307:20:173
17256991910,7cyclictest0-21swapper/307:30:163
17256991910,6cyclictest0-21swapper/308:20:153
17256991910,6cyclictest0-21swapper/308:05:203
1724899194,8cyclictest13823-21sort07:40:112
1724899191,9cyclictest18576-21sendmail_mailqu08:15:272
1724899191,9cyclictest17949-21kthreadcore08:50:212
1724899191,9cyclictest17949-21kthreadcore08:50:212
1724899191,14cyclictest8975-21apt-get07:35:142
1724899191,10cyclictest24790-21fschecks_count08:25:152
1722999194,5cyclictest12-21ksoftirqd/008:00:290
1722999193,5cyclictest12-21ksoftirqd/009:00:230
1722999193,10cyclictest12-21ksoftirqd/008:20:220
17229991912,4cyclictest0-21swapper/008:30:130
1722999191,14cyclictest23451-21kthreadcore07:15:200
17229991910,7cyclictest13831-21apt-get07:40:120
1723899181,9cyclictest12748-21tr08:45:201
1723899181,9cyclictest10306-21/usr/sbin/munin09:50:311
1723899181,3cyclictest0-21swapper/108:00:231
1722999187,3cyclictest131rcu_preempt07:10:100
1722999181,9cyclictest12378-21kthreadcore08:10:190
1722999181,6cyclictest12-21ksoftirqd/008:40:250
17229991811,5cyclictest12-21ksoftirqd/009:00:010
1724899171,7cyclictest17341-21sort07:10:022
17229991710,5cyclictest0-21swapper/007:25:300
1723899162,11cyclictest13902-21threads08:10:261
1723899151,11cyclictest17146-21latency_hist07:10:021
1722999151,11cyclictest17298-21grep07:10:020
1725699139,2cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional