You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-09 - 03:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sun Aug 09, 2026 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
824799867,9cyclictest131rcu_preempt20:15:150
824799683,48cyclictest25511-21ssh21:45:120
824999603,45cyclictest15689-21ssh23:45:151
824999585,47cyclictest15076-21ssh22:15:141
824799542,39cyclictest20293-21ssh22:00:140
824799533,10cyclictest131rcu_preempt22:55:130
824799533,10cyclictest131rcu_preempt22:55:120
824799511,37cyclictest24688-21ssh23:50:110
824799505,35cyclictest3349-21ssh21:15:140
824799502,40cyclictest26639-21ssh21:10:130
826499462,27cyclictest35966irq/58-eth022:25:153
824999461,37cyclictest17396-21ssh23:10:121
826499433,25cyclictest35966irq/58-eth021:30:133
824999435,31cyclictest30781-21ssh21:30:141
824799431,26cyclictest13570-21ssh22:50:120
824799421,30cyclictest16477-21ssh21:40:150
824799414,4cyclictest131rcu_preempt00:15:150
824999393,31cyclictest26227-21diskmemload22:25:141
8264993819,11cyclictest35966irq/58-eth000:25:143
825699382,20cyclictest26621-21ssh00:08:112
824999385,20cyclictest3005-21ssh23:20:101
826499372,22cyclictest35966irq/58-eth023:05:153
824999371,28cyclictest25498-21ssh21:45:121
824799371,28cyclictest23365-21ssh00:25:140
8264993512,16cyclictest321rcuc/322:15:133
825699352,27cyclictest6930-21sh00:32:552
825699352,17cyclictest930-21snmpd00:37:112
8256993516,14cyclictest27-21ksoftirqd/222:15:132
824799352,23cyclictest24214-21ssh22:20:160
8247993519,11cyclictest0-21swapper/000:00:130
8249993415,13cyclictest26227-21diskmemload00:32:561
826499333,20cyclictest35966irq/58-eth021:40:133
8256993312,15cyclictest0-21swapper/200:10:132
824799332,25cyclictest31790-21ssh23:00:140
8256993223,4cyclictest3338-21sh21:15:132
824799322,2cyclictest131rcu_preempt23:25:130
825699314,11cyclictest261rcuc/200:20:132
824999314,21cyclictest21-21ksoftirqd/100:20:131
824999311,22cyclictest24704-21ssh23:50:131
825699303,18cyclictest27346irq/40-dwc2_hso19:15:132
825699301,19cyclictest27346irq/40-dwc2_hso20:40:132
825699293,15cyclictest8332-21ssh23:05:152
825699292,11cyclictest5990-21ssh22:10:162
825699291,17cyclictest17401-21ssh23:10:132
825699281,20cyclictest0-21swapper/220:00:162
825699280,14cyclictest18647-21ls22:35:022
824999281,21cyclictest12487-21diskmemload21:20:141
824999281,20cyclictest26430-21ssh23:15:131
825699272,20cyclictest28746-21ssh00:26:112
825699272,14cyclictest26445-21ssh23:15:142
825699271,18cyclictest20375-21ls22:00:142
824799272,19cyclictest8115-21rm23:57:520
8247992715,3cyclictest12-21ksoftirqd/000:35:130
826499266,10cyclictest35966irq/58-eth023:10:133
826499265,15cyclictest0-21swapper/300:20:133
825699266,7cyclictest16463-21ssh21:40:132
825699263,15cyclictest21080-21ssh23:30:162
825699261,9cyclictest261rcuc/223:35:152
825699261,18cyclictest0-21swapper/221:45:132
826499258,12cyclictest35966irq/58-eth021:20:143
825699252,13cyclictest1270-21ssh23:55:132
824999253,16cyclictest0-21swapper/123:05:151
8249992512,3cyclictest0-21swapper/120:25:281
8247992519,5cyclictest12-21ksoftirqd/021:35:150
826499242,16cyclictest35966irq/58-eth023:20:113
825699242,13cyclictest0-21swapper/219:50:142
8249992416,4cyclictest21-21ksoftirqd/100:10:001
824799243,14cyclictest2079-21ssh21:50:090
824799243,14cyclictest2079-21ssh21:50:090
824799242,16cyclictest30126-21sh23:35:150
8247992411,9cyclictest12623-21kworker/0:022:30:130
826499231,16cyclictest35966irq/58-eth023:50:123
825699234,3cyclictest131rcu_preempt20:45:282
825699233,14cyclictest27346irq/40-dwc2_hso23:45:142
825699231,9cyclictest0-21swapper/219:20:142
8256992315,5cyclictest13939-21ssh23:25:212
8256992311,9cyclictest26798-21ssh21:25:542
825699230,14cyclictest131rcu_preempt22:25:122
824999236,11cyclictest16452-21sh21:40:131
824999234,13cyclictest27915-21ssh22:40:121
8249992312,8cyclictest7891-21cron23:05:011
824999231,18cyclictest991-21ls20:40:221
8249992311,10cyclictest0-21swapper/123:31:051
824799234,11cyclictest21679-21ssh21:25:150
824799231,19cyclictest14481-21fschecks_time00:20:160
826499226,14cyclictest33-21ksoftirqd/322:02:273
826499226,13cyclictest33-21ksoftirqd/323:02:143
826499226,13cyclictest33-21ksoftirqd/322:43:313
826499226,13cyclictest33-21ksoftirqd/321:15:073
8264992215,5cyclictest33-21ksoftirqd/322:55:133
8264992215,5cyclictest33-21ksoftirqd/322:55:133
825699227,6cyclictest131rcu_preempt19:10:312
825699222,15cyclictest0-21swapper/200:19:112
825699222,15cyclictest0-21swapper/200:04:572
825699222,12cyclictest131rcu_preempt21:35:112
8256992216,4cyclictest0-21swapper/222:55:262
8256992216,4cyclictest0-21swapper/222:55:262
8256992212,8cyclictest0-21swapper/223:21:362
8256992212,8cyclictest0-21swapper/220:30:232
8256992211,9cyclictest7992-21ssh21:51:432
8256992211,9cyclictest7992-21ssh21:51:422
8256992211,9cyclictest2062-21ssh22:42:362
8256992211,9cyclictest0-21swapper/222:50:492
8256992211,9cyclictest0-21swapper/220:05:262
825699221,16cyclictest12480-21ssh21:20:142
825699221,15cyclictest6757-21ssh23:40:122
825699220,17cyclictest27346irq/40-dwc2_hso22:35:162
824999224,3cyclictest131rcu_preempt00:40:011
824999221,5cyclictest21-21ksoftirqd/120:45:161
824999221,15cyclictest12885-21kworker/1:222:49:521
824999221,15cyclictest12885-21kworker/1:222:49:521
8249992210,10cyclictest0-21swapper/122:31:151
824799223,11cyclictest28582-21sh00:10:140
824799222,16cyclictest5980-21ssh22:10:160
824799221,16cyclictest28883-21find20:05:120
8247992210,10cyclictest8240-21cyclictest21:05:130
826499219,9cyclictest0-21swapper/321:35:343
826499218,8cyclictest35966irq/58-eth023:30:153
826499216,13cyclictest33-21ksoftirqd/323:47:463
826499216,13cyclictest33-21ksoftirqd/322:23:043
826499216,13cyclictest33-21ksoftirqd/300:34:043
826499215,14cyclictest33-21ksoftirqd/321:59:033
826499215,14cyclictest33-21ksoftirqd/300:01:363
826499215,13cyclictest33-21ksoftirqd/321:49:113
826499215,13cyclictest33-21ksoftirqd/300:16:393
826499212,12cyclictest0-21swapper/323:25:133
826499211,16cyclictest35966irq/58-eth023:15:133
825699219,6cyclictest0-21swapper/221:59:132
825699218,6cyclictest10698-21ntpq20:50:262
825699212,2cyclictest131rcu_preempt19:55:182
825699211,16cyclictest10948-21ls20:20:142
825699211,14cyclictest27346irq/40-dwc2_hso23:00:142
8256992111,4cyclictest0-21swapper/221:30:362
825699211,12cyclictest131rcu_preempt22:23:062
825699211,10cyclictest29309-21ssh22:05:122
824999219,3cyclictest21-21ksoftirqd/100:13:521
824999212,12cyclictest30211-21ssh00:27:551
824999211,3cyclictest131rcu_preempt22:55:411
824999211,3cyclictest131rcu_preempt22:55:411
8249992113,5cyclictest20382-21df22:00:151
824999211,2cyclictest131rcu_preempt21:10:521
8249992112,5cyclictest9942-21usage_EthcTask00:15:321
824999210,15cyclictest14817-21ls22:15:021
824799210,17cyclictest9841-21fschecks_count19:45:170
826499205,13cyclictest33-21ksoftirqd/322:09:183
826499205,13cyclictest33-21ksoftirqd/321:51:303
826499205,13cyclictest33-21ksoftirqd/321:51:303
826499205,13cyclictest33-21ksoftirqd/321:26:053
8264992012,5cyclictest35966irq/58-eth000:35:143
826499201,13cyclictest0-21swapper/319:55:023
825699209,9cyclictest2442-21ssh21:14:262
825699208,9cyclictest27085-21kthreadcore20:35:202
825699201,9cyclictest28074-21sed19:30:172
825699201,9cyclictest21509-21latency_hist21:05:002
825699201,9cyclictest13640-21kthreadcore20:55:202
825699201,8cyclictest8311-21latency_hist19:10:012
825699201,2cyclictest131rcu_preempt23:50:052
825699201,16cyclictest5549-21kthreadcore22:45:202
825699201,16cyclictest5549-21kthreadcore22:45:202
825699201,13cyclictest0-21swapper/219:40:302
825699201,13cyclictest0-21swapper/219:25:022
825699201,10cyclictest13531-21grep19:45:302
825699200,15cyclictest27346irq/40-dwc2_hso19:35:152
824999209,9cyclictest0-21swapper/121:51:461
824999209,9cyclictest0-21swapper/121:51:451
824999208,9cyclictest21-21ksoftirqd/123:58:481
824999202,15cyclictest12069-21ssh23:25:141
824999201,3cyclictest131rcu_preempt23:38:421
8249992013,4cyclictest21-21ksoftirqd/119:50:221
8247992013,4cyclictest2618-21latency_hist23:20:020
824799201,16cyclictest29024-21timerwakeupswit20:35:280
8247992011,6cyclictest28657-21cron20:05:010
8247992011,6cyclictest22610-21cron19:25:010
826499199,7cyclictest0-21swapper/322:46:163
826499199,7cyclictest0-21swapper/322:46:163
8264991912,5cyclictest0-21swapper/300:10:233
8264991911,4cyclictest35966irq/58-eth020:20:133
8264991910,4cyclictest0-21swapper/300:09:113
825699193,9cyclictest7701-21kthreadcore20:15:222
825699191,15cyclictest15672-21cpu20:25:142
825699191,10cyclictest1459-21grep20:10:172
824999199,7cyclictest1006-21awk22:05:281
824999198,3cyclictest21-21ksoftirqd/121:18:111
824999198,2cyclictest131rcu_preempt21:37:171
824999193,4cyclictest21-21ksoftirqd/122:22:091
824999192,13cyclictest11099-21kernelversion00:00:201
824999191,2cyclictest131rcu_preempt23:40:131
8249991911,6cyclictest21817-21cut21:25:171
8249991911,6cyclictest1-21systemd21:02:221
8249991910,6cyclictest22583-21ntpq22:35:291
824999190,2cyclictest131rcu_preempt20:00:281
824799199,7cyclictest14507-21sshd19:50:150
824799197,9cyclictest15804-21kthreadcore22:15:210
824799193,13cyclictest32388-21ssh00:30:120
824799192,15cyclictest2295-21kthreadcore22:25:210
824799191,5cyclictest19993-21irqcore00:05:210
8247991912,4cyclictest8717-21irqcore23:05:190
824799191,13cyclictest6621-21ntp_192.168.11423:20:240
824799191,13cyclictest29443-21ssh22:05:150
824799191,11cyclictest15772-21cut20:25:170
8247991910,6cyclictest9473-21cron19:45:010
826499188,8cyclictest0-21swapper/322:11:043
826499188,7cyclictest0-21swapper/323:55:263
826499182,10cyclictest0-21swapper/320:45:153
826499182,10cyclictest0-21swapper/319:15:143
8264991812,4cyclictest33-21ksoftirqd/319:20:143
8264991810,6cyclictest0-21swapper/322:51:013
8264991810,6cyclictest0-21swapper/321:10:073
8264991810,5cyclictest35966irq/58-eth022:30:133
825699181,7cyclictest22344-21kthreadcore21:05:192
824999187,2cyclictest21-21ksoftirqd/119:15:251
824999183,11cyclictest21-21ksoftirqd/119:20:111
824999182,2cyclictest131rcu_preempt20:10:261
824999181,9cyclictest10036-21irqcore19:45:191
824999181,2cyclictest131rcu_preempt21:58:101
8249991811,5cyclictest0-21swapper/119:40:271
8249991811,4cyclictest21-21ksoftirqd/119:35:281
824999181,14cyclictest13257-21tr20:55:191
824999181,11cyclictest17127-21ssh22:50:231
8249991810,5cyclictest29832-21sh20:40:011
824799188,7cyclictest20816-21grep23:10:230
824799187,3cyclictest131rcu_preempt19:10:130
824799181,8cyclictest21176-21df_inode23:30:170
824799181,7cyclictest11162-21/usr/sbin/munin21:55:160
8247991812,3cyclictest12-21ksoftirqd/019:25:130
826499179,6cyclictest0-21swapper/323:41:533
826499178,6cyclictest0-21swapper/322:35:073
826499171,13cyclictest0-21swapper/323:35:293
826499171,12cyclictest0-21swapper/319:25:243
824999178,7cyclictest20553-21df20:30:151
824999174,10cyclictest22874-21kthreadcore21:05:201
824999172,12cyclictest9010-21kthreadcore20:15:251
8249991711,4cyclictest0-21swapper/119:30:231
824999171,13cyclictest32727-21head20:05:291
824999171,13cyclictest21008-21kthreadcore19:55:211
8249991710,4cyclictest11225-21irqcore20:20:181
824799178,6cyclictest9006-21cut23:40:210
824799175,6cyclictest0-21swapper/022:35:150
824799174,9cyclictest0-21swapper/021:00:130
824799172,13cyclictest31836-21kthreadcore21:30:200
824799172,11cyclictest0-21swapper/019:35:150
824799172,10cyclictest0-21swapper/020:30:150
824799171,7cyclictest29932-21kthreadcore19:30:200
824799171,13cyclictest4229-21kthreadcore20:45:210
824799171,12cyclictest19474-21grep23:45:290
824799171,12cyclictest17845-21sendmail-msp19:20:000
8247991710,4cyclictest28663-21kthreadcore22:40:210
824799170,6cyclictest4504-21ssh22:45:130
824799170,6cyclictest4504-21ssh22:45:120
826499167,4cyclictest0-21swapper/320:10:133
826499162,7cyclictest0-21swapper/320:40:133
826499161,10cyclictest0-21swapper/319:35:023
826499161,10cyclictest0-21swapper/319:10:013
824999162,10cyclictest8557-21cut19:10:171
824999161,11cyclictest7906-21kernelversion20:50:191
824999160,5cyclictest53646usb-storage19:25:041
824799163,11cyclictest16642-21sendmail-msp21:00:010
824799162,5cyclictest12-21ksoftirqd/021:20:200
824799162,11cyclictest30135-21cpuspeed20:40:140
824799162,10cyclictest0-21swapper/020:20:140
824799162,10cyclictest0-21swapper/020:10:150
824799162,10cyclictest0-21swapper/019:55:140
8247991611,3cyclictest7259-21cpu20:50:140
826499151,8cyclictest0-21swapper/320:50:173
826499151,11cyclictest0-21swapper/320:30:203
826499151,11cyclictest0-21swapper/319:45:173
8264991510,3cyclictest0-21swapper/319:37:043
826499149,3cyclictest0-21swapper/319:41:003
826499141,10cyclictest0-21swapper/320:35:183
824999141,9cyclictest8167-21latency_hist19:10:011
826499139,1cyclictest0-21swapper/320:05:143
826499132,8cyclictest321rcuc/320:25:143
826499131,7cyclictest0-21swapper/320:55:173
826499131,7cyclictest0-21swapper/319:55:183
826499131,6cyclictest0-21swapper/321:05:273
824799131,9cyclictest8312-21latency_hist19:10:010
826499127,3cyclictest0-21swapper/320:00:273
826499121,9cyclictest0-21swapper/321:00:223
826499121,5cyclictest0-21swapper/319:10:163
826499111,7cyclictest0-21swapper/320:15:243
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional