You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-27 - 13:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sat Jun 27, 2026 00:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32725997418,9cyclictest131rcu_preempt22:20:120
3274599691,48cyclictest0-21swapper/222:35:112
32745996737,26cyclictest0-21swapper/222:50:112
3272599626,45cyclictest20859-21ssh22:05:120
32725996149,6cyclictest13277-21ssh21:25:130
3274599602,51cyclictest5278-21apt-get19:15:132
3272599544,8cyclictest131rcu_preempt00:00:120
3272599544,8cyclictest131rcu_preempt00:00:120
3274599511,37cyclictest6700-21apt-get22:15:112
3273399512,4cyclictest0-21swapper/120:50:111
3273399503,38cyclictest26100-21sh21:50:111
3273399492,5cyclictest0-21swapper/119:15:141
3275399485,25cyclictest35966irq/58-eth023:20:123
3275399485,25cyclictest35966irq/58-eth023:20:113
3274599481,43cyclictest0-21swapper/220:40:122
3274599481,32cyclictest28064-21cut22:25:222
3273399481,11cyclictest26786-21apt-get23:20:121
3273399481,11cyclictest26786-21apt-get23:20:111
3274599472,40cyclictest12295-21apt-get23:30:122
3274599471,30cyclictest8479-21ssh23:10:012
3272599471,39cyclictest14668-21ssh00:25:150
32725994636,5cyclictest18325-21diskmemload00:35:120
32725994612,24cyclictest12-21ksoftirqd/021:45:140
3275399454,27cyclictest35966irq/58-eth022:45:133
3274599451,37cyclictest0-21swapper/221:55:132
3274599451,31cyclictest0-21swapper/223:35:112
3273399452,4cyclictest0-21swapper/120:10:121
3274599441,33cyclictest19678-21apt-get19:30:122
3274599441,32cyclictest0-21swapper/221:50:112
3274599441,25cyclictest32119-21ssh22:10:232
3273399441,4cyclictest0-21swapper/119:30:121
3272599442,7cyclictest131rcu_preempt21:20:130
3275399431,25cyclictest35966irq/58-eth000:35:123
3274599431,39cyclictest0-21swapper/219:45:122
3274599431,37cyclictest8736-21apt-get23:10:122
3274599431,37cyclictest8736-21apt-get23:10:122
3274599431,21cyclictest18325-21diskmemload00:20:122
3273399432,29cyclictest17055-21ssh21:45:131
3272599433,26cyclictest12-21ksoftirqd/023:05:130
3272599431,34cyclictest31339-21ssh21:35:130
3275399422,24cyclictest35966irq/58-eth022:30:123
3274599423,34cyclictest10085-21apt-get19:20:142
3274599421,33cyclictest0-21swapper/221:05:112
3274599421,33cyclictest0-21swapper/219:50:112
3274599421,26cyclictest23638-21cut20:05:202
3274599420,36cyclictest0-21swapper/222:30:132
3273399422,2cyclictest0-21swapper/122:35:111
32745994114,17cyclictest0-21swapper/223:20:112
32745994114,17cyclictest0-21swapper/223:20:112
3274599411,36cyclictest20138-21grep23:50:282
3274599411,22cyclictest17247-21cat22:20:212
3274599410,25cyclictest1492-21ntp_192.168.11423:40:212
3273399411,6cyclictest0-21swapper/122:50:111
3275399402,23cyclictest35966irq/58-eth020:40:143
3274599402,34cyclictest0-21swapper/221:00:122
3274599401,36cyclictest0-21swapper/200:15:132
3274599401,34cyclictest16132-21apt-get20:00:132
3274599401,33cyclictest0-21swapper/200:00:132
3274599401,33cyclictest0-21swapper/200:00:122
3274599400,23cyclictest19450-21date00:10:022
3273399401,4cyclictest0-21swapper/122:00:141
3273399401,4cyclictest0-21swapper/122:00:131
3273399401,4cyclictest0-21swapper/120:20:121
3273399401,2cyclictest0-21swapper/100:10:121
3275399393,18cyclictest35966irq/58-eth023:05:123
3275399392,26cyclictest35966irq/58-eth021:30:123
3274599391,35cyclictest7968-21apt-get21:40:132
3274599390,35cyclictest6917-21apt-get23:45:132
3274599390,35cyclictest22328-21ssh21:30:142
3274599390,35cyclictest0-21swapper/223:00:132
3273399392,4cyclictest0-21swapper/123:35:111
3273399392,25cyclictest5561-21sh00:20:111
3275399387,25cyclictest35966irq/58-eth000:25:153
3275399384,19cyclictest35966irq/58-eth023:40:093
3274599383,23cyclictest0-21swapper/220:50:202
3274599381,33cyclictest7733-21apt-get20:25:132
3274599381,22cyclictest29974-21cat23:55:322
3274599381,22cyclictest0-21swapper/221:48:312
3274599380,27cyclictest14636-21apt-get00:25:132
3273399381,5cyclictest0-21swapper/119:45:121
3272599387,9cyclictest131rcu_preempt23:50:120
3272599382,32cyclictest28980-21ssh00:15:130
3274599372,30cyclictest0-21swapper/219:55:132
3274599371,32cyclictest4139-21ntp_192.168.11400:35:222
3274599371,31cyclictest32247-21ntpq22:45:262
3274599371,27cyclictest27633-21/usr/sbin/munin21:15:172
3274599370,33cyclictest27346irq/40-dwc2_hso19:35:132
3273399373,2cyclictest0-21swapper/123:30:111
3274599361,30cyclictest26468-21latency_hist23:20:012
3274599361,28cyclictest0-21swapper/221:20:132
3274599360,27cyclictest24580-21ntp_states22:05:272
3273399363,2cyclictest20848-21apt-get22:05:121
3273399362,4cyclictest0-21swapper/121:40:121
3273399361,9cyclictest21036-21diskstats20:05:151
3273399360,4cyclictest28574-21apt-get22:45:141
3275399351,18cyclictest35966irq/58-eth000:30:133
3274599357,20cyclictest0-21swapper/222:55:242
3274599351,31cyclictest0-21swapper/220:30:142
3274599351,31cyclictest0-21swapper/220:30:132
3274599351,30cyclictest29369-21apt-get19:40:122
3274599351,30cyclictest18563-21cut20:35:192
3274599351,29cyclictest0-21swapper/220:45:122
3274599350,32cyclictest27346irq/40-dwc2_hso22:00:122
3274599350,32cyclictest27346irq/40-dwc2_hso22:00:112
3274599350,27cyclictest23040-21/usr/sbin/munin22:40:232
3274599350,21cyclictest18325-21diskmemload23:26:262
3273399351,8cyclictest10726-21/usr/sbin/munin00:05:271
3273399351,4cyclictest0-21swapper/122:55:111
3273399351,12cyclictest894-21grep22:10:231
3273399350,4cyclictest1722-21apt-get00:00:111
3273399350,4cyclictest1722-21apt-get00:00:111
32753993415,12cyclictest35966irq/58-eth023:55:123
3274599341,30cyclictest25629-21ssh21:13:042
3274599341,29cyclictest1434-21memory20:15:222
3274599341,26cyclictest31085-21ssh00:33:262
3273399341,9cyclictest3530-21fschecks_count23:25:161
3273399341,9cyclictest26739-21ssh23:55:201
3273399341,7cyclictest31350-21grep21:15:261
3273399341,3cyclictest0-21swapper/121:00:111
3273399341,3cyclictest0-21swapper/120:25:131
3272599341,29cyclictest25052-21ssh23:55:120
3274599331,29cyclictest25733-21latency_hist20:10:022
3274599331,29cyclictest13282-21ssh21:25:132
3274599331,28cyclictest343-21cut19:10:022
3274599330,30cyclictest27346irq/40-dwc2_hso20:20:122
3274599330,30cyclictest18682-21grep19:25:292
3273399339,21cyclictest3093-21if_err_eth021:55:181
3273399338,21cyclictest6889-21sh21:20:201
3273399331,8cyclictest3909-21kthreadcore00:35:211
3273399331,8cyclictest16504-21irqcore23:50:181
3273399331,8cyclictest10838-21kthreadcore19:20:191
3273399331,7cyclictest24776-21ssh00:30:191
3273399331,10cyclictest4117-21apt20:55:131
3273399330,6cyclictest21758-21find23:15:281
3272599336,23cyclictest28585-21ssh22:45:130
3275399322,21cyclictest35966irq/58-eth021:10:123
3275399322,16cyclictest35966irq/58-eth000:00:123
3275399322,16cyclictest35966irq/58-eth000:00:113
3274599321,28cyclictest26598-21sh00:12:422
3274599321,27cyclictest4697-21irqrtprio20:55:182
3274599321,27cyclictest31304-21munin-node21:35:122
3273399323,26cyclictest13292-21ssh21:25:151
3273399322,26cyclictest19550-21ssh22:40:111
3273399321,8cyclictest7582-21kthreadcore22:15:191
3273399321,8cyclictest3345-21ntpq23:05:251
3273399321,8cyclictest12467-21ssh23:10:281
3273399321,8cyclictest12467-21ssh23:10:271
3273399321,7cyclictest24666-21grep19:35:161
3273399321,7cyclictest13552-21kthreadcore19:55:221
3273399320,8cyclictest802-21irqcore19:10:171
3273399320,12cyclictest18358-21ssh00:25:281
3272599324,20cyclictest12-21ksoftirqd/020:40:140
32725993216,10cyclictest12-21ksoftirqd/021:55:140
3274599311,27cyclictest135-21systemd-journal19:10:362
3273399311,8cyclictest32177-21cut20:15:211
3273399311,8cyclictest300-21grep00:15:281
3273399311,8cyclictest1871-21ssh23:40:271
3273399311,8cyclictest15932-21df22:20:141
3273399311,7cyclictest31267-21kthreadcore19:40:201
3273399311,7cyclictest13829-21diskstats21:05:141
3273399311,6cyclictest28555-21ntpq22:25:261
3273399311,6cyclictest18812-21ssh21:10:141
3273399311,6cyclictest12845-21if_eth020:30:171
3273399311,6cyclictest12845-21if_eth020:30:171
3273399311,5cyclictest19781-21switchtime20:00:271
3273399310,8cyclictest14982-21df_inode19:25:161
3273399310,7cyclictest9645-21cut19:50:201
3273399310,6cyclictest22880-21cut21:30:171
3273399310,6cyclictest10534-21netstat23:45:221
3272599312,2cyclictest131rcu_preempt22:55:120
3273399301,9cyclictest1608-21df22:30:131
3273399301,7cyclictest7554-21cron21:40:011
3273399301,7cyclictest29790-21kthreadcore20:45:201
3273399301,5cyclictest22236-21sort20:40:141
3273399301,4cyclictest0-21swapper/120:35:121
32725993019,3cyclictest23201-21ssh23:00:140
3273399291,6cyclictest27113-21wc23:00:301
3272599292,20cyclictest22332-21ssh21:30:130
3272599291,20cyclictest26808-21ssh23:20:130
3272599291,20cyclictest26808-21ssh23:20:120
3275399272,18cyclictest0-21swapper/322:05:133
3272599272,19cyclictest17794-21ssh23:15:110
3272599271,20cyclictest19818-21ssh00:10:140
3275399263,15cyclictest35966irq/58-eth023:15:113
3275399262,14cyclictest35966irq/58-eth000:05:133
32753992614,8cyclictest35966irq/58-eth023:30:133
32725992614,8cyclictest12-21ksoftirqd/021:40:140
3272599261,18cyclictest24859-21ssh22:25:140
3273399251,7cyclictest356-21sed19:10:031
3272599253,16cyclictest12327-21rm23:30:130
3272599251,21cyclictest8751-21ssh23:10:140
3272599251,21cyclictest8751-21ssh23:10:130
3275399241,13cyclictest0-21swapper/323:50:123
3272599242,6cyclictest131rcu_preempt19:35:120
3272599241,19cyclictest10775-21ssh00:05:140
3275399239,8cyclictest0-21swapper/321:35:153
3275399231,11cyclictest35966irq/58-eth021:15:143
32753992310,6cyclictest0-21swapper/323:25:313
3272599235,9cyclictest0-21swapper/022:40:110
32725992310,9cyclictest630-21Async09:36:310
3275399222,17cyclictest35966irq/58-eth022:15:133
3275399222,14cyclictest0-21swapper/323:00:133
32753992213,6cyclictest35966irq/58-eth022:35:133
3272599224,4cyclictest131rcu_preempt23:40:110
3275399212,13cyclictest0-21swapper/321:20:133
32753992111,7cyclictest0-21swapper/323:39:013
32753992111,5cyclictest0-21swapper/300:15:143
3272599211,16cyclictest7085-21cut23:45:150
3272599211,16cyclictest7008-21ssh23:25:280
3272599211,16cyclictest6414-21ls22:15:010
3272599211,15cyclictest15879-21ls22:00:300
3272599211,15cyclictest15879-21ls22:00:290
3272599211,14cyclictest4070-21ls20:55:020
3272599211,10cyclictest8705-21sendmail21:00:000
3272599211,10cyclictest29741-21sed21:50:240
32753992011,6cyclictest35966irq/58-eth000:20:123
3272599208,8cyclictest12-21ksoftirqd/021:10:130
3272599204,4cyclictest131rcu_preempt20:10:120
3272599202,15cyclictest27690-21ssh21:15:150
3272599202,12cyclictest12-21ksoftirqd/022:30:120
3272599201,9cyclictest13368-21kthreadcore22:35:220
32725992014,4cyclictest32405-21sed20:15:200
32725992012,6cyclictest16923-21sshd20:30:320
32725992012,6cyclictest16923-21sshd20:30:320
3275399198,6cyclictest0-21swapper/321:55:273
3275399192,12cyclictest0-21swapper/323:45:133
3275399192,12cyclictest0-21swapper/322:25:153
3275399191,13cyclictest0-21swapper/322:20:143
3275399191,12cyclictest0-21swapper/321:25:193
3272599199,7cyclictest9273-21sed22:50:280
3272599195,4cyclictest131rcu_preempt19:55:170
3272599194,10cyclictest0-21swapper/019:20:130
3272599192,9cyclictest29351-21/usr/sbin/munin19:40:180
3272599192,9cyclictest15236-21irqcore19:25:190
32725991912,4cyclictest2081-21irqcore19:45:160
32725991912,4cyclictest12-21ksoftirqd/000:25:010
3275399189,6cyclictest35966irq/58-eth023:00:023
3275399189,6cyclictest0-21swapper/322:43:283
3275399188,8cyclictest0-21swapper/322:54:483
3275399186,9cyclictest0-21swapper/321:50:193
3275399184,11cyclictest0-21swapper/321:45:223
32753991810,6cyclictest0-21swapper/321:40:063
3272599188,7cyclictest22909-21kthreadcore20:05:190
3272599185,10cyclictest19493-21kthreadcore20:35:200
3272599181,11cyclictest29920-21cron23:40:020
3275399179,6cyclictest0-21swapper/322:00:033
3275399179,6cyclictest0-21swapper/322:00:033
3275399179,5cyclictest35966irq/58-eth022:10:143
3275399178,4cyclictest0-21swapper/323:10:303
3275399178,4cyclictest0-21swapper/323:10:303
3275399172,9cyclictest0-21swapper/319:30:133
32753991711,4cyclictest0-21swapper/300:11:383
3275399171,13cyclictest0-21swapper/320:25:153
3272599179,3cyclictest32717-21cyclictest20:25:120
3272599178,6cyclictest6546-21apt-get19:50:120
3272599177,7cyclictest12-21ksoftirqd/020:00:210
3272599174,2cyclictest131rcu_preempt19:10:120
3272599171,14cyclictest21821-21kthreadcore19:30:210
3272599171,13cyclictest9830-21cron19:20:010
3272599171,12cyclictest6719-21ssh22:15:110
3272599171,11cyclictest366-21latency_hist19:10:020
32725991710,3cyclictest12-21ksoftirqd/021:05:110
3275399161,12cyclictest0-21swapper/321:05:233
3275399161,10cyclictest0-21swapper/320:10:203
3272599161,12cyclictest3703-21kthreadcore20:20:170
3272599161,11cyclictest28833-21kthreadcore20:45:190
3272599161,10cyclictest9717-21kthreadcore21:00:180
3275399157,6cyclictest0-21swapper/319:58:323
3275399151,11cyclictest33-21ksoftirqd/320:55:273
3275399149,3cyclictest0-21swapper/320:24:413
3275399149,3cyclictest0-21swapper/319:10:173
3275399147,4cyclictest33-21ksoftirqd/320:15:263
3275399141,9cyclictest0-21swapper/319:15:143
3275399141,8cyclictest0-21swapper/319:50:153
3275399139,2cyclictest33-21ksoftirqd/321:00:253
3275399139,2cyclictest33-21ksoftirqd/320:36:423
3275399139,2cyclictest33-21ksoftirqd/319:40:283
3275399135,3cyclictest33-21ksoftirqd/319:49:443
3275399133,6cyclictest0-21swapper/320:30:153
3275399133,6cyclictest0-21swapper/320:30:143
3275399132,8cyclictest0-21swapper/320:00:213
3275399128,2cyclictest33-21ksoftirqd/320:50:263
3275399125,5cyclictest0-21swapper/320:45:193
3275399123,7cyclictest0-21swapper/319:35:153
3275399121,9cyclictest0-21swapper/319:25:213
3275399121,8cyclictest0-21swapper/320:05:133
3275399121,8cyclictest0-21swapper/319:20:153
3275399111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional