You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-28 - 06:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sun Dec 28, 2025 00:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3056899663,39cyclictest21846-21ssh21:50:152
3055999626,12cyclictest131rcu_preempt19:10:151
3055999554,11cyclictest131rcu_preempt23:20:141
3055099523,38cyclictest30604-21ssh00:05:140
3055099522,44cyclictest29372-21ssh22:50:150
30568995119,23cyclictest0-21swapper/221:55:152
3056899511,37cyclictest29632-21apt-get20:50:132
3055999515,8cyclictest131rcu_preempt21:20:161
3055099514,8cyclictest131rcu_preempt20:25:150
30576995032,15cyclictest0-21swapper/322:00:163
3056899501,44cyclictest25856-21apt-get23:25:172
3055999502,40cyclictest21855-21ssh21:50:161
3055099502,34cyclictest10789-21ssh23:35:130
3056899491,37cyclictest15168-21apt-get20:35:142
3055999483,38cyclictest32027-21rm23:10:131
3056899471,39cyclictest0-21swapper/222:05:142
3055999463,37cyclictest4558-21ssh23:50:131
3056899452,39cyclictest4375-21apt-get19:50:142
3056899451,32cyclictest0-21swapper/221:45:152
3056899450,39cyclictest532-21apt-get22:15:142
30568994435,4cyclictest22346irq/40-dwc2_hso22:20:152
3056899442,35cyclictest5719-21apt-get22:55:142
30559994417,21cyclictest21-21ksoftirqd/119:50:131
30559994410,25cyclictest21-21ksoftirqd/119:35:141
3056899431,37cyclictest0-21swapper/223:00:122
3056899430,39cyclictest0-21swapper/220:00:162
30568994130,8cyclictest261rcuc/221:25:142
3056899411,33cyclictest6855-21apt-get22:00:162
3055999415,30cyclictest11776-21ssh22:40:141
3055099413,27cyclictest14461-21ssh23:00:120
3056899402,29cyclictest0-21swapper/223:10:122
3056899398,26cyclictest9264-21diskmemload00:30:142
3056899392,31cyclictest31902-21apt-get19:45:142
3056899390,31cyclictest0-21swapper/220:55:142
3055099392,4cyclictest131rcu_preempt22:40:140
30568993819,16cyclictest0-21swapper/220:45:152
3056899380,32cyclictest0-21swapper/223:15:142
3055099381,27cyclictest30540-21ssh21:55:150
30576993719,11cyclictest28366irq/58-eth023:40:143
30576993718,13cyclictest28366irq/58-eth022:10:153
3056899371,34cyclictest8073-21ssh00:29:002
3056899371,32cyclictest3632-21kthreadcore21:20:262
3056899371,32cyclictest2835-21users19:10:332
3056899371,32cyclictest21900-21/usr/sbin/munin00:00:252
3056899371,32cyclictest0-21swapper/219:55:122
3056899371,30cyclictest15506-21apt-get00:15:122
3056899371,22cyclictest0-21swapper/223:09:082
3056899370,26cyclictest24333-21apt-get22:10:142
3055999375,23cyclictest16164-21diskmemload21:40:121
3055999373,25cyclictest18115-21ssh22:25:131
3057699364,19cyclictest28366irq/58-eth023:20:123
3057699364,19cyclictest28366irq/58-eth019:25:123
3057699363,20cyclictest28366irq/58-eth021:25:143
3056899362,27cyclictest25154-21kthreadcore20:10:222
3056899362,25cyclictest3213-21fschecks_count19:15:172
3056899362,25cyclictest3213-21fschecks_count19:15:172
3056899361,26cyclictest20569-21cpuspeed22:45:132
3056899360,22cyclictest0-21swapper/221:19:572
3056899360,21cyclictest0-21swapper/223:51:002
3056899351,30cyclictest10870-21ps20:30:202
3056899351,25cyclictest7189-21kthreadcore20:25:232
3056899351,25cyclictest4153-21/usr/sbin/munin20:20:232
3056899350,31cyclictest25680-21nfs_client19:35:232
30550993516,12cyclictest15524-21diskmemload22:05:150
30576993427,5cyclictest28366irq/58-eth023:35:133
30568993415,15cyclictest0-21swapper/200:10:142
3056899341,27cyclictest0-21swapper/222:30:112
3056899341,25cyclictest26555-21ssh21:33:502
3056899340,32cyclictest22053-21ssh22:25:302
3056899340,25cyclictest10176-21sed21:00:232
3056899340,25cyclictest10176-21sed21:00:232
3055999342,22cyclictest16560-21ssh21:10:141
3055099344,27cyclictest552-21ssh22:15:130
3055099342,28cyclictest5751-21ssh22:55:140
3056899338,22cyclictest16164-21diskmemload22:35:332
3056899336,25cyclictest31310-21ssh00:23:192
3056899333,22cyclictest14873-21unixbench_multi23:35:282
3056899331,29cyclictest20164-21ntp_192.168.11421:10:252
3056899331,28cyclictest31855-21ps20:15:262
3056899331,28cyclictest30412-21kthreadcore21:35:232
3056899331,24cyclictest19655-21df23:40:152
3056899331,19cyclictest0-21swapper/200:09:292
3056899330,29cyclictest23296-21/usr/sbin/munin20:05:342
3056899330,23cyclictest30685-21date19:10:022
30568993215,9cyclictest0-21swapper/220:40:122
3056899321,28cyclictest23039-21ssh00:35:522
3056899321,28cyclictest10323-21ssh23:34:552
3056899321,28cyclictest10323-21ssh23:34:552
3056899321,27cyclictest11279-21nfs_client19:20:252
3056899321,26cyclictest0-21swapper/219:40:142
3056899321,26cyclictest0-21swapper/219:40:142
30568993212,18cyclictest11742-21/usr/sbin/munin22:40:342
3056899320,29cyclictest21958-21sshd23:20:332
3056899320,29cyclictest21418-21ssh23:59:542
3056899320,28cyclictest7635-21kthreadcore21:40:232
3056899320,27cyclictest0-21swapper/219:25:112
3055999321,24cyclictest73-21kswapd022:45:111
3057699312,14cyclictest28366irq/58-eth000:35:143
3056899312,17cyclictest29230-21sh22:50:072
3056899311,27cyclictest31973-21ntp_states23:45:302
3055999312,23cyclictest32577-21sh23:28:011
3055999312,20cyclictest573-21ssh00:25:151
3056899301,26cyclictest18451-21cut19:30:202
3055999302,19cyclictest21947-21ssh00:00:151
3055099304,14cyclictest27887-21ssh21:35:150
3057699291,20cyclictest28366irq/58-eth022:55:133
3056899290,25cyclictest15189-21mailstats21:05:272
30576992821,4cyclictest0-21swapper/323:45:163
30576992820,4cyclictest28366irq/58-eth021:50:143
3057699281,16cyclictest28366irq/58-eth021:55:153
3055099282,19cyclictest17923-21ssh00:35:150
3057699274,14cyclictest28366irq/58-eth022:05:153
3055999273,22cyclictest2111-21ssh23:30:151
3055999273,22cyclictest2111-21ssh23:30:141
3055999272,19cyclictest23455-21ssh00:19:341
3055999271,21cyclictest25302-21ssh21:15:131
3055099272,19cyclictest29846-21sh00:04:330
3055099271,20cyclictest24362-21ssh22:10:150
3057699263,15cyclictest0-21swapper/323:28:013
3055999262,19cyclictest8387-21sh23:15:151
3055999262,18cyclictest28287-21ssh23:45:161
3055099263,17cyclictest17079-21ssh23:20:130
3057699253,14cyclictest0-21swapper/322:17:553
3057699251,13cyclictest28366irq/58-eth023:30:143
3057699251,13cyclictest28366irq/58-eth023:30:143
3055999254,13cyclictest562-21ssh22:15:141
3055999253,13cyclictest24344-21sh22:10:141
3055999252,15cyclictest26804-21ssh22:30:141
3055099253,19cyclictest16038-1kworker/u9:121:30:150
3055099252,18cyclictest25887-21ssh23:25:170
30559992412,6cyclictest131rcu_preempt21:35:151
30559992410,4cyclictest131rcu_preempt00:05:231
30550992413,6cyclictest0-21swapper/022:00:150
3057699232,16cyclictest28366irq/58-eth023:00:133
3055999234,16cyclictest0-21swapper/122:00:171
3055999232,16cyclictest5588-21apt-get20:25:141
3055999231,17cyclictest16164-21diskmemload23:06:341
3055999231,16cyclictest21-21ksoftirqd/119:20:141
30559992310,4cyclictest131rcu_preempt19:45:151
3055999230,19cyclictest25115-21ls20:45:171
30550992315,4cyclictest0-21swapper/023:40:140
3057699225,10cyclictest0-21swapper/322:50:153
3057699221,13cyclictest28366irq/58-eth021:15:143
3055999226,12cyclictest6898-21ssh00:10:151
3055999221,18cyclictest18001-21ls00:35:161
3055999221,16cyclictest73-21kswapd000:30:131
3055099222,14cyclictest633-21Async12:11:530
3055099222,14cyclictest17605-21ssh21:28:320
3057699219,9cyclictest0-21swapper/322:45:293
3057699213,11cyclictest0-21swapper/321:30:153
3057699211,13cyclictest28366irq/58-eth000:05:153
3055999212,14cyclictest10471-21kworker/u8:022:21:331
30559992114,4cyclictest6582-21sendmail21:00:011
3055999211,17cyclictest10360-21apt-get21:25:131
3055999211,16cyclictest18773-21apt-get20:05:151
30559992110,8cyclictest30522-21apt-get21:55:141
3055099218,11cyclictest12-21ksoftirqd/022:45:100
3055099212,17cyclictest17318-21ssh21:10:180
30550992115,4cyclictest0-21swapper/023:55:150
30550992112,7cyclictest12615-21ssh21:44:590
3055099211,17cyclictest2326-21fschecks_time23:30:180
3055099211,17cyclictest2326-21fschecks_time23:30:170
3055099211,10cyclictest419766ssh23:50:020
3055099210,18cyclictest8521-21cut23:15:170
3057699202,10cyclictest28366irq/58-eth020:35:143
30576992014,4cyclictest28366irq/58-eth022:35:133
3055999209,8cyclictest19744-21fschecks_time23:40:171
3055999205,9cyclictest0-21swapper/122:55:151
30559992011,6cyclictest16132-21kthreadcore23:00:181
3055999201,14cyclictest19168-21ssh21:30:141
3055999201,14cyclictest16164-21diskmemload23:55:131
3055999201,12cyclictest16608-21ls21:45:251
3055999200,14cyclictest883-21timerandwakeup22:50:301
30550992012,6cyclictest5008-21ssh23:11:140
30550992012,6cyclictest32140-21ssh21:18:070
30550992012,6cyclictest1439-21ssh21:20:030
30550992011,7cyclictest17249-21ssh00:34:410
3057699193,12cyclictest28366irq/58-eth019:10:153
3057699193,11cyclictest0-21swapper/322:20:333
3057699193,11cyclictest0-21swapper/321:37:343
3057699193,11cyclictest0-21swapper/300:01:343
30576991911,5cyclictest28366irq/58-eth022:25:133
30576991911,5cyclictest28366irq/58-eth000:25:153
3057699191,12cyclictest0-21swapper/321:05:283
3057699191,12cyclictest0-21swapper/300:30:303
30576991910,6cyclictest0-21swapper/323:15:173
3055999198,8cyclictest18104-21cut22:05:221
3055999192,12cyclictest0-21swapper/119:55:151
3055999191,4cyclictest0-21swapper/100:20:131
30559991912,4cyclictest11205-21sed23:35:181
30559991911,6cyclictest0-21swapper/120:30:201
3055999190,16cyclictest30800-21timerandwakeup19:40:301
3055999190,16cyclictest30800-21timerandwakeup19:40:291
3055099197,3cyclictest131rcu_preempt22:25:260
3055099197,3cyclictest131rcu_preempt21:45:040
3055099197,3cyclictest131rcu_preempt00:14:200
3055099195,4cyclictest131rcu_preempt23:05:180
3055099191,9cyclictest7570-21kthreadcore19:50:230
3055099191,9cyclictest13700-21kthreadcore21:05:210
3055099191,8cyclictest3221-21df22:35:150
30550991912,5cyclictest15461-21ssh00:15:080
30550991912,5cyclictest12582-21ssh23:54:420
30550991911,6cyclictest492-21ssh22:32:140
30550991911,6cyclictest31722-21ssh00:23:510
30550991911,6cyclictest28203-21ssh21:52:310
3055099191,15cyclictest550-21apt-get00:25:130
3055099190,14cyclictest6400-21ls19:15:240
3055099190,14cyclictest6400-21ls19:15:230
3057699189,6cyclictest0-21swapper/321:14:253
3057699186,6cyclictest0-21swapper/323:50:113
3057699182,11cyclictest0-21swapper/322:40:163
3057699182,10cyclictest0-21swapper/319:50:153
3057699182,10cyclictest0-21swapper/300:15:133
30576991812,3cyclictest28366irq/58-eth021:45:143
30576991811,4cyclictest0-21swapper/323:55:273
3057699181,13cyclictest28366irq/58-eth000:10:143
3057699181,13cyclictest0-21swapper/319:35:173
3055999184,6cyclictest15804-21kthreadcore19:25:211
3055999183,10cyclictest0-21swapper/120:50:141
3055999181,8cyclictest6118-21kthreadcore22:35:231
30559991811,5cyclictest0-21swapper/119:30:181
3055999181,14cyclictest966-21cut20:20:171
30559991810,5cyclictest18887-21grep20:35:291
30559991810,5cyclictest0-21swapper/119:15:331
30559991810,5cyclictest0-21swapper/119:15:321
3055099183,13cyclictest29194-21kthreadcore19:40:220
3055099183,13cyclictest29194-21kthreadcore19:40:210
3055099181,9cyclictest31359-21tr20:50:200
3055099181,8cyclictest32268-21kthreadcore19:10:210
3055099181,10cyclictest21018-21cut20:05:240
3057699179,6cyclictest0-21swapper/323:10:093
3057699179,6cyclictest0-21swapper/322:31:463
3057699179,6cyclictest0-21swapper/321:40:363
3057699172,11cyclictest0-21swapper/321:20:223
3057699172,10cyclictest0-21swapper/320:00:163
30576991711,4cyclictest0-21swapper/323:05:313
3057699171,10cyclictest0-21swapper/319:45:253
3057699171,10cyclictest0-21swapper/300:20:163
3055999172,13cyclictest9068-21kthreadcore21:00:211
3055999172,13cyclictest9068-21kthreadcore21:00:211
30559991710,4cyclictest29134-21kthreadcore20:15:181
3055999170,11cyclictest28132-21latency_hist20:15:021
3055999170,11cyclictest17248-21ls20:00:261
3055099179,5cyclictest10172-21kthreadcore22:20:240
3055099175,9cyclictest4991-21kthreadcore20:55:230
3055099172,10cyclictest0-21swapper/020:00:170
3055099171,14cyclictest25114-21kthreadcore19:35:220
3055099171,14cyclictest17823-21irqcore19:30:180
30550991711,4cyclictest13731-21cron20:00:010
3055099171,13cyclictest24570-21apt-get20:45:040
3057699161,12cyclictest0-21swapper/320:20:163
3055999164,5cyclictest20820-21cut20:40:191
3055999161,11cyclictest11835-21grep21:05:171
3055099169,5cyclictest1252-21grep20:20:180
3055099162,11cyclictest8528-21irqrtprio19:20:200
3055099162,11cyclictest8008-21kthreadcore21:00:200
3055099162,11cyclictest8008-21kthreadcore21:00:200
3055099162,11cyclictest14262-21kthreadcore19:25:190
3055099161,6cyclictest20286-21grep20:40:180
3055099161,12cyclictest17056-21kthreadcore20:35:220
3055099161,12cyclictest1243-21kthreadcore19:45:230
3055099161,11cyclictest23653-21diskstats20:10:160
3055099161,11cyclictest10864-21grep20:30:210
3057699152,9cyclictest0-21swapper/320:05:173
3057699152,10cyclictest28366irq/58-eth019:15:143
3057699152,10cyclictest28366irq/58-eth019:15:143
3057699149,3cyclictest0-21swapper/320:50:043
3057699149,3cyclictest0-21swapper/320:10:183
3057699146,3cyclictest0-21swapper/319:20:283
3057699141,6cyclictest0-21swapper/320:15:263
3055999141,10cyclictest30661-21latency_hist19:10:011
3055099141,10cyclictest30582-21latency_hist19:10:020
3057699132,9cyclictest0-21swapper/321:00:143
3057699132,9cyclictest0-21swapper/321:00:133
3057699132,9cyclictest0-21swapper/320:55:153
3057699132,9cyclictest0-21swapper/320:45:153
3057699132,9cyclictest0-21swapper/319:55:153
3057699131,9cyclictest0-21swapper/320:30:183
3057699122,7cyclictest0-21swapper/319:30:163
3057699122,6cyclictest0-21swapper/320:25:163
3057699121,9cyclictest0-21swapper/319:40:233
3057699121,9cyclictest0-21swapper/319:40:233
3057699112,5cyclictest0-21swapper/320:40:143
3057699111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional