You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-10 - 09:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Tue Mar 10, 2026 00:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3184299584,38cyclictest31601-21ssh23:05:121
31859995427,20cyclictest28366irq/58-eth023:20:133
3185199544,32cyclictest22552-21ssh23:00:132
31842995412,31cyclictest21-21ksoftirqd/120:45:131
31842995130,9cyclictest21-21ksoftirqd/119:10:131
3184299481,38cyclictest29347-21ssh22:10:131
3184299473,32cyclictest17248-21ssh23:15:121
31851994626,15cyclictest0-21swapper/223:55:132
3183199456,7cyclictest131rcu_preempt21:30:120
3185999433,24cyclictest28366irq/58-eth023:00:143
3185999432,24cyclictest28366irq/58-eth000:20:123
3183199435,35cyclictest5972-21ssh22:15:130
3184299420,30cyclictest20322-21ssh22:05:131
3185999413,9cyclictest28366irq/58-eth023:25:123
3185999413,20cyclictest28366irq/58-eth021:10:143
3183199406,29cyclictest15020-21ssh22:20:120
3184299390,25cyclictest7450-21ssh21:40:131
3185999383,24cyclictest28366irq/58-eth022:20:123
3185999383,18cyclictest28366irq/58-eth021:40:133
3185999381,19cyclictest28366irq/58-eth019:30:113
3185199381,25cyclictest24077-21ssh22:25:132
3183199381,28cyclictest17231-21rm23:15:110
3185999374,23cyclictest28366irq/58-eth021:20:123
3185999374,23cyclictest28366irq/58-eth021:20:113
3185999372,12cyclictest0-21swapper/323:15:123
3185199372,28cyclictest5154-21grep21:20:192
3185199372,28cyclictest5154-21grep21:20:182
31842993714,7cyclictest21-21ksoftirqd/123:00:151
31831993718,12cyclictest131rcu_preempt21:20:120
31831993718,12cyclictest131rcu_preempt21:20:110
3185999365,19cyclictest28366irq/58-eth023:55:133
3185999365,15cyclictest28366irq/58-eth022:00:123
3185999363,20cyclictest28366irq/58-eth022:25:133
31859993621,10cyclictest28366irq/58-eth023:45:143
3184299366,19cyclictest4437-21ssh22:50:121
3184299365,21cyclictest21-21ksoftirqd/120:40:131
3184299362,19cyclictest11160-21ssh22:00:141
3184299361,26cyclictest28515-21ssh00:15:131
3183199353,20cyclictest32214-21ssh00:35:130
3185999342,17cyclictest28366irq/58-eth020:30:133
3184299344,23cyclictest26452-21ssh23:20:131
3184299341,3cyclictest131rcu_preempt23:10:131
3183199343,28cyclictest30805-21ssh21:35:140
3183199342,28cyclictest20316-21ssh22:05:140
3183199342,26cyclictest0-21swapper/021:55:130
3185999332,18cyclictest28366irq/58-eth019:10:123
3185999323,15cyclictest28366irq/58-eth021:25:113
3185999322,20cyclictest28366irq/58-eth022:35:123
3185199322,22cyclictest14096-21df_inode19:25:142
3183199326,22cyclictest15919-21ssh23:50:130
3183199323,26cyclictest30255-21ssh23:40:130
3185999313,16cyclictest28366irq/58-eth020:25:143
31851993110,3cyclictest131rcu_preempt19:20:132
3185199301,17cyclictest23270-21ssh00:30:142
3183199304,23cyclictest1502-21ssh00:00:130
3185999292,11cyclictest28366irq/58-eth019:45:133
3185199293,11cyclictest0-21swapper/222:39:182
3185199292,17cyclictest6924-21ssh23:45:132
3184299291,22cyclictest24865-21ssh23:55:141
3183199297,10cyclictest633-21Async12:11:530
3183199291,10cyclictest12-21ksoftirqd/020:30:140
3185999288,7cyclictest28366irq/58-eth000:15:143
3185999283,10cyclictest0-21swapper/322:09:203
3185999282,9cyclictest0-21swapper/322:55:133
3185999282,9cyclictest0-21swapper/322:55:133
31859992812,10cyclictest28366irq/58-eth023:05:143
3185199284,17cyclictest22346irq/40-dwc2_hso19:50:142
3185199282,22cyclictest27-21ksoftirqd/200:25:122
3184299282,20cyclictest5132-21ssh00:00:241
3184299282,20cyclictest18015-21ssh21:10:141
3184299281,3cyclictest131rcu_preempt19:50:141
3183199283,18cyclictest21151-21ssh23:35:110
3183199282,21cyclictest9725-21ssh22:35:120
3183199282,18cyclictest18936-21sh19:30:150
3185999273,16cyclictest0-21swapper/322:44:193
3185999272,10cyclictest0-21swapper/321:39:183
3185199277,13cyclictest27-21ksoftirqd/221:45:152
31851992710,12cyclictest27-21ksoftirqd/222:45:142
3184299271,19cyclictest201rcuc/122:45:151
3184299270,22cyclictest8661-21sh21:57:321
3183199276,12cyclictest12-21ksoftirqd/021:05:130
3183199273,8cyclictest12-21ksoftirqd/000:30:130
3183199272,19cyclictest12754-21ssh21:25:130
3183199271,16cyclictest12-21ksoftirqd/021:00:100
3185999263,16cyclictest0-21swapper/322:12:323
3185999262,10cyclictest0-21swapper/323:54:193
3185999262,10cyclictest0-21swapper/323:33:193
31859992617,6cyclictest28366irq/58-eth023:35:123
31859992612,4cyclictest0-21swapper/319:55:143
3185999261,16cyclictest28366irq/58-eth021:50:133
3185199262,20cyclictest26434-21sh23:20:122
3185199262,19cyclictest12553-21ssh22:17:342
31851992613,10cyclictest14397-21ssh23:12:062
3184299262,8cyclictest131rcu_preempt21:23:191
3184299262,8cyclictest131rcu_preempt21:23:181
3184299261,3cyclictest131rcu_preempt22:20:141
31831992619,4cyclictest28864-21sh00:15:170
31831992616,7cyclictest0-21swapper/021:15:150
3185999252,13cyclictest0-21swapper/322:15:143
3185199253,16cyclictest29356-21ssh22:10:152
3185199252,19cyclictest20099-21ssh23:34:202
3184299258,3cyclictest131rcu_preempt23:35:011
3184299256,8cyclictest201rcuc/100:25:131
3183199259,4cyclictest131rcu_preempt00:10:010
31831992514,8cyclictest19833-21cron20:05:010
3185199243,13cyclictest22346irq/40-dwc2_hso21:10:132
3185199242,17cyclictest13500-21ssh22:55:142
3185199242,17cyclictest13500-21ssh22:55:132
3185199242,16cyclictest0-21swapper/219:10:152
3185199242,10cyclictest131rcu_preempt22:40:122
31851992412,9cyclictest5549-21ssh23:43:572
31851992412,8cyclictest9014-21mailstats00:20:282
31851992411,10cyclictest23888-21ssh23:17:352
3184299241,7cyclictest131rcu_preempt21:27:181
3184299241,17cyclictest73-21kswapd022:35:141
3184299241,13cyclictest527-21ps00:35:191
3184299241,13cyclictest25511-21ssh21:50:131
3183199248,11cyclictest25522-21diskmemload21:50:150
31831992417,4cyclictest0-21swapper/023:05:250
31831992410,5cyclictest12-21ksoftirqd/021:40:420
31831992410,11cyclictest23768-21kworker/u8:122:00:120
31859992313,7cyclictest28366irq/58-eth021:15:263
3185999231,18cyclictest0-21swapper/321:55:133
3185199232,16cyclictest0-21swapper/219:45:142
3185199232,16cyclictest0-21swapper/200:07:502
3185199231,6cyclictest629-21ssh22:30:132
3185199231,6cyclictest629-21ssh22:30:122
3185199231,3cyclictest131rcu_preempt21:25:162
31851992313,7cyclictest32187-21/usr/sbin/munin00:35:252
31851992312,8cyclictest23751-21kthreadcore21:30:202
3185199231,13cyclictest2439-21unixbench_multi20:50:282
3185199231,12cyclictest5889-21ssh22:50:202
3184299236,10cyclictest0-21swapper/119:15:131
3184299231,18cyclictest29991-21grep20:15:191
3184299231,16cyclictest13182-21cut21:05:171
3183199238,7cyclictest12-21ksoftirqd/000:10:030
3183199231,16cyclictest16552-21ssh21:45:150
3185999223,13cyclictest0-21swapper/300:14:203
3185199228,11cyclictest29810-21apt-get20:15:172
3185199222,5cyclictest0-21swapper/221:41:192
3185199222,16cyclictest0-21swapper/200:00:192
3185199222,15cyclictest0-21swapper/221:18:182
3185199222,14cyclictest22346irq/40-dwc2_hso23:05:132
3185199221,12cyclictest30921-21diskstats21:35:152
3185199221,11cyclictest15884-21/usr/sbin/munin23:50:142
3184299229,4cyclictest131rcu_preempt22:25:291
3184299223,4cyclictest131rcu_preempt20:35:301
3184299223,12cyclictest5163-21ssh00:20:111
3184299221,3cyclictest131rcu_preempt21:45:231
3183199229,4cyclictest12-21ksoftirqd/020:00:020
3183199225,12cyclictest151rcuc/022:40:130
3183199221,17cyclictest5191-21sh00:20:120
31831992211,2cyclictest12-21ksoftirqd/021:00:010
3185999213,11cyclictest0-21swapper/323:10:163
3185199218,10cyclictest32337-21grep19:40:282
3185199212,10cyclictest21646-21fschecks_time20:40:162
31851992113,6cyclictest0-21swapper/222:20:132
3185199211,11cyclictest32409-21mailstats00:15:282
3185199211,10cyclictest17736-21kthreadcore20:00:192
3184299218,5cyclictest131rcu_preempt00:30:131
3184299216,4cyclictest131rcu_preempt23:40:151
3184299211,15cyclictest18805-21ssh22:40:121
3184299210,7cyclictest0-21swapper/119:35:121
3183199219,9cyclictest12-21ksoftirqd/020:40:180
3183199218,4cyclictest12-21ksoftirqd/021:10:240
3183199212,15cyclictest3038-21ssh23:25:120
31831992115,3cyclictest12-21ksoftirqd/023:00:200
31831992111,7cyclictest12-21ksoftirqd/019:15:130
3183199211,16cyclictest8231-21sh23:10:160
3183199211,16cyclictest27802-21apt-get22:45:140
31831992110,3cyclictest12-21ksoftirqd/020:45:230
3185999209,6cyclictest0-21swapper/322:34:273
3185999209,6cyclictest0-21swapper/322:34:263
31859992011,7cyclictest28466irq/59-eth022:45:143
3185199208,9cyclictest0-21swapper/220:10:282
3185199207,5cyclictest21334-21fschecks_time23:35:142
3185199204,7cyclictest131rcu_preempt21:55:162
3185199203,8cyclictest27-21ksoftirqd/222:08:252
3185199202,10cyclictest8882-21kthreadcore20:25:222
3185199201,9cyclictest6696-21wc20:55:232
31851992012,5cyclictest0-21swapper/221:00:152
3185199201,11cyclictest4098-21kthreadcore23:25:182
3185199201,11cyclictest26223-21irqstats21:50:202
3185199201,11cyclictest24704-21http20:10:022
3185199201,11cyclictest14682-21memory22:00:232
3185199201,10cyclictest25732-21kthreadcore19:35:192
3185199201,10cyclictest22665-21unixbench_multi19:30:272
3185199201,10cyclictest19548-21/usr/sbin/munin00:10:142
3184299209,8cyclictest22001-21tune2fs21:30:151
3184299202,4cyclictest131rcu_preempt19:45:271
3184299201,10cyclictest0-21swapper/120:55:241
3183199209,3cyclictest12-21ksoftirqd/019:30:000
3183199208,4cyclictest12-21ksoftirqd/023:45:270
3183199208,3cyclictest12-21ksoftirqd/022:34:250
3183199208,3cyclictest12-21ksoftirqd/022:34:240
3183199207,2cyclictest12-21ksoftirqd/000:26:230
3183199202,13cyclictest0-21swapper/019:40:130
3183199202,12cyclictest30158-21ssh22:10:190
31831992014,4cyclictest0-21swapper/022:50:240
31831992013,3cyclictest12-21ksoftirqd/020:20:020
31831992012,5cyclictest0-21swapper/022:30:010
3183199201,15cyclictest5894-21ls20:20:280
3183199201,15cyclictest13498-21sh22:55:140
3183199201,15cyclictest13498-21sh22:55:130
3185999199,8cyclictest0-21swapper/321:34:413
3185999198,6cyclictest0-21swapper/300:25:333
3185999192,6cyclictest0-21swapper/300:00:183
3185999192,14cyclictest0-21swapper/300:35:203
3185999191,14cyclictest0-21swapper/300:30:173
3185999191,13cyclictest0-21swapper/319:25:133
3185999191,12cyclictest0-21swapper/319:40:143
3185999191,12cyclictest0-21swapper/319:40:023
3185199193,6cyclictest131rcu_preempt20:30:132
3185199192,13cyclictest27-21ksoftirqd/221:05:132
3185199191,9cyclictest4506-21fschecks_count19:15:152
3185199191,9cyclictest31862-21latency_hist19:10:012
3185199191,9cyclictest10607-21fschecks_time19:55:162
3185199191,15cyclictest924-21snmpd20:45:142
3185199191,10cyclictest16741-21diskstats20:35:172
3185199190,2cyclictest131rcu_preempt20:20:122
3184299197,7cyclictest0-21swapper/121:37:281
3184299195,10cyclictest904-21ssh22:30:161
3184299195,10cyclictest904-21ssh22:30:161
3184299194,2cyclictest131rcu_preempt20:25:291
3184299191,8cyclictest131rcu_preempt19:25:011
3184299191,8cyclictest0-21swapper/121:19:481
3184299191,6cyclictest12851-21kthreadcore19:55:201
3184299191,14cyclictest10514-21ssh00:05:141
3183199199,2cyclictest12-21ksoftirqd/023:30:070
3183199191,6cyclictest30044-21ntp_192.168.11423:20:250
31831991913,3cyclictest12-21ksoftirqd/020:10:020
31831991911,4cyclictest12-21ksoftirqd/019:35:160
31831991911,4cyclictest12-21ksoftirqd/019:20:190
3185999188,6cyclictest28366irq/58-eth019:50:143
3185999187,9cyclictest0-21swapper/323:44:443
3185999183,11cyclictest0-21swapper/300:08:203
3185999182,7cyclictest28366irq/58-eth020:15:153
3185999182,10cyclictest0-21swapper/320:20:143
3185999182,10cyclictest0-21swapper/320:10:143
3185999182,10cyclictest0-21swapper/320:05:043
3185999182,10cyclictest0-21swapper/320:00:133
3185999181,12cyclictest0-21swapper/321:45:183
3184299182,3cyclictest131rcu_preempt23:50:191
3184299182,3cyclictest131rcu_preempt23:25:181
3184299181,4cyclictest131rcu_preempt22:55:161
3184299181,4cyclictest131rcu_preempt22:55:151
3184299181,14cyclictest73-21kswapd000:10:131
31842991810,6cyclictest18843-21apt-get19:30:111
3184299180,4cyclictest5960-21apt-get22:15:121
3184299180,2cyclictest131rcu_preempt23:35:131
3184299180,12cyclictest28769-21timerwakeupswit20:10:291
3183199189,7cyclictest12-21ksoftirqd/019:45:180
3183199187,2cyclictest12-21ksoftirqd/020:50:150
3183199182,5cyclictest12-21ksoftirqd/020:10:180
31831991812,3cyclictest12-21ksoftirqd/019:10:290
31831991811,4cyclictest6122-21grep19:50:180
3183199181,13cyclictest24977-21diskstats23:55:150
3185999177,6cyclictest28366irq/58-eth020:50:143
3185999172,9cyclictest0-21swapper/320:45:133
3185999172,10cyclictest0-21swapper/322:50:053
3185999172,10cyclictest0-21swapper/320:35:143
3184299179,6cyclictest14368-21sed19:25:171
3184299173,11cyclictest201rcuc/123:45:131
3183199171,10cyclictest20240-21switchtime20:35:290
3185999162,10cyclictest0-21swapper/319:20:123
3184299162,11cyclictest9877-21cut21:00:181
3184299161,9cyclictest12066-21ls20:30:171
3184299161,12cyclictest19840-21apt-get20:05:041
3184299161,12cyclictest16245-21kthreadcore20:00:191
3184299161,11cyclictest31371-21irqcore20:50:191
3184299161,11cyclictest29106-21sed19:40:171
3184299161,10cyclictest31759-21latency_hist19:10:021
3184299151,11cyclictest3866-21kthreadcore20:20:211
3183199151,10cyclictest31874-21latency_hist19:10:010
3185999149,3cyclictest0-21swapper/321:06:173
3185999148,4cyclictest28366irq/58-eth021:00:193
3185999135,5cyclictest28366irq/58-eth020:40:133
3185999132,8cyclictest0-21swapper/319:20:023
3185999131,7cyclictest0-21swapper/320:55:163
3185999101,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional