You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-17 - 16:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Tue Mar 17, 2026 12:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
657999441,36cyclictest19652-21ssh09:25:121
661899435,19cyclictest28366irq/58-eth010:15:133
661899433,24cyclictest28366irq/58-eth008:55:153
661899422,25cyclictest28366irq/58-eth009:30:133
661899416,20cyclictest28366irq/58-eth007:25:143
656999411,3cyclictest131rcu_preempt09:35:120
661899403,23cyclictest28366irq/58-eth010:30:123
661899385,21cyclictest28366irq/58-eth009:50:123
661899373,22cyclictest28366irq/58-eth011:35:123
661899371,20cyclictest28366irq/58-eth010:05:133
6618993619,9cyclictest28366irq/58-eth011:30:093
661899351,18cyclictest28366irq/58-eth011:20:123
661899341,21cyclictest28366irq/58-eth009:20:123
656999347,5cyclictest131rcu_preempt09:55:130
661899332,19cyclictest28366irq/58-eth010:35:123
661899322,4cyclictest0-21swapper/310:00:123
661899322,4cyclictest0-21swapper/310:00:113
6602993211,12cyclictest27-21ksoftirqd/208:55:152
661899314,17cyclictest28366irq/58-eth010:20:113
661899314,14cyclictest28366irq/58-eth010:50:133
661899311,4cyclictest0-21swapper/309:45:163
661899302,3cyclictest0-21swapper/311:10:113
6618993015,8cyclictest28466irq/59-eth009:35:123
661899301,19cyclictest28366irq/58-eth012:10:133
661899301,16cyclictest28366irq/58-eth008:05:113
661899291,4cyclictest0-21swapper/312:20:123
661899291,12cyclictest28366irq/58-eth011:00:113
657999295,2cyclictest131rcu_preempt12:15:131
661899281,5cyclictest0-21swapper/310:44:053
6602992815,10cyclictest26010-21cron10:05:012
6602992815,10cyclictest26010-21cron10:05:012
657999281,19cyclictest19840-21sh11:50:121
661899271,6cyclictest0-21swapper/312:30:163
661899271,23cyclictest0-21swapper/310:55:303
661899271,23cyclictest0-21swapper/310:55:303
661899271,23cyclictest0-21swapper/309:10:013
6602992713,7cyclictest20492-21cron10:20:002
656999270,21cyclictest26227-21awk12:30:000
661899263,4cyclictest0-21swapper/308:40:133
661899263,11cyclictest0-21swapper/310:13:233
661899262,20cyclictest0-21swapper/307:50:113
661899261,5cyclictest0-21swapper/309:12:123
661899261,4cyclictest0-21swapper/311:29:473
661899261,4cyclictest0-21swapper/307:30:303
661899261,3cyclictest0-21swapper/312:05:003
657999260,19cyclictest29844-21ssh10:25:131
661899252,11cyclictest28366irq/58-eth007:35:123
661899251,6cyclictest0-21swapper/312:05:243
661899251,4cyclictest0-21swapper/310:28:483
661899251,4cyclictest0-21swapper/309:44:573
661899251,3cyclictest0-21swapper/312:30:023
661899251,3cyclictest0-21swapper/311:15:213
661899251,3cyclictest0-21swapper/310:45:193
661899251,3cyclictest0-21swapper/309:55:283
661899251,3cyclictest0-21swapper/308:10:293
661899251,21cyclictest0-21swapper/308:25:233
6602992514,9cyclictest6935-21ssh09:15:532
6602992513,9cyclictest27185-21cron11:00:012
6602992513,9cyclictest27185-21cron11:00:002
6602992512,6cyclictest3480-21ssh09:50:302
6579992510,3cyclictest131rcu_preempt10:55:331
6579992510,3cyclictest131rcu_preempt10:55:331
661899244,12cyclictest0-21swapper/311:40:153
661899241,5cyclictest0-21swapper/311:46:433
661899241,5cyclictest0-21swapper/308:00:133
661899241,4cyclictest0-21swapper/312:35:193
661899241,4cyclictest0-21swapper/311:07:213
661899241,3cyclictest0-21swapper/312:15:283
661899241,3cyclictest0-21swapper/311:55:203
661899241,3cyclictest0-21swapper/309:25:213
661899241,3cyclictest0-21swapper/309:18:383
661899241,3cyclictest0-21swapper/308:35:153
661899241,3cyclictest0-21swapper/308:20:173
661899241,14cyclictest0-21swapper/311:50:133
660299242,14cyclictest8450-21ssh09:55:152
660299241,6cyclictest767566ssh10:48:002
6602992413,9cyclictest13172-21ssh09:39:242
6602992412,10cyclictest13670-21ssh11:27:172
6602992411,6cyclictest9408-21ssh11:43:482
657999246,7cyclictest88166lldpd08:30:011
661899231,6cyclictest0-21swapper/307:55:153
661899231,4cyclictest0-21swapper/307:20:203
661899231,3cyclictest0-21swapper/309:00:283
661899231,3cyclictest0-21swapper/309:00:273
661899231,3cyclictest0-21swapper/308:15:283
660299236,8cyclictest3705-21grep12:35:162
660299232,17cyclictest26619-21ssh12:30:142
6602992314,7cyclictest28311-21munin-plugin-st08:40:012
6602992312,9cyclictest3566-21ssh10:27:072
657999232,15cyclictest26229-21apt12:30:011
656999236,4cyclictest24830-21diskmemload09:17:320
656999231,12cyclictest6856-21ntp_states12:35:250
661899221,5cyclictest0-21swapper/308:45:183
661899221,4cyclictest0-21swapper/307:40:153
661899221,4cyclictest0-21swapper/307:20:013
661899221,3cyclictest0-21swapper/308:50:263
660299229,10cyclictest0-21swapper/210:35:012
660299221,9cyclictest12700-21kthreadcore09:20:222
6602992217,3cyclictest9065-21ssh12:00:272
6602992210,9cyclictest27-21ksoftirqd/210:20:162
6602992210,10cyclictest12306-21ssh11:09:422
657999227,4cyclictest24830-21diskmemload11:05:561
657999226,9cyclictest24830-21diskmemload12:33:571
656999226,3cyclictest24830-21diskmemload11:03:550
656999226,3cyclictest24830-21diskmemload10:54:470
656999221,15cyclictest19857-21ssh11:50:140
656999221,10cyclictest0-21swapper/011:24:040
656999221,10cyclictest0-21swapper/010:34:240
661899211,5cyclictest0-21swapper/308:30:153
661899211,4cyclictest0-21swapper/307:45:233
660299218,6cyclictest315-21irqrtprio12:15:182
660299218,10cyclictest19009-21ntpq10:35:242
660299217,11cyclictest26506-21cpuspeed10:05:142
660299217,11cyclictest26280-21tr10:40:202
660299213,10cyclictest17398-21cut12:05:232
660299211,9cyclictest19543-21latency_hist11:50:012
6602992112,6cyclictest12900-21ssh11:10:102
6602992111,8cyclictest0-21swapper/208:20:172
660299211,17cyclictest18434-21cut11:30:192
660299211,11cyclictest6666-21wc10:10:252
6602992110,9cyclictest0-21swapper/208:25:302
6602992110,8cyclictest9040-21sendmail08:20:002
6602992110,8cyclictest23221-21sh12:10:122
6602992110,8cyclictest17654-21/usr/sbin/munin12:25:292
657999213,2cyclictest131rcu_preempt12:06:121
657999212,17cyclictest6814-21ssh11:24:511
656999218,5cyclictest7317-21cat10:10:280
656999216,3cyclictest24830-21diskmemload11:18:070
656999215,2cyclictest131rcu_preempt09:25:200
656999211,9cyclictest0-21swapper/010:21:420
656999211,16cyclictest23015-21ssh09:45:160
656999211,15cyclictest767-21ssh10:45:130
656999211,10cyclictest28761-21cut10:05:200
661899201,5cyclictest0-21swapper/307:10:173
661899201,5cyclictest0-21swapper/307:10:013
660299208,4cyclictest27595-21irqcore08:05:172
660299207,5cyclictest26528-21kthreadcore11:35:182
660299202,10cyclictest8703-21ssh12:20:122
660299201,9cyclictest31918-21kthreadcore11:20:202
6602992013,5cyclictest27580-21sh11:00:122
660299201,15cyclictest17341-21idleruntime-cro07:55:012
660299201,11cyclictest21897-21ssh11:15:132
660299201,11cyclictest19968-21ssh11:50:152
660299201,10cyclictest6825-21diskstats07:10:152
660299201,10cyclictest32281-21ssh11:55:232
660299201,10cyclictest28920-21sed09:30:172
660299201,10cyclictest20018-21cron09:05:012
660299201,10cyclictest20018-21cron09:05:002
660299201,10cyclictest15355-21kthreadcore09:40:192
660299201,10cyclictest10848-21kthreadcore10:50:182
657999203,3cyclictest131rcu_preempt11:45:121
657999203,13cyclictest26409-21ssh10:05:121
6579992012,5cyclictest0-21swapper/107:30:201
657999201,17cyclictest15560-21sshd10:15:241
657999201,15cyclictest28661-21ssh09:30:141
657999201,14cyclictest8535-21df09:55:151
657999200,17cyclictest31779-21ssh12:14:591
656999206,3cyclictest24830-21diskmemload11:37:170
656999204,3cyclictest24830-21diskmemload09:33:070
656999203,3cyclictest24830-21diskmemload10:56:290
656999203,3cyclictest24830-21diskmemload10:56:290
656999203,2cyclictest24830-21diskmemload11:43:150
656999203,2cyclictest131rcu_preempt10:25:170
656999202,7cyclictest131rcu_preempt10:15:290
6569992011,6cyclictest9014-21cron08:20:010
656999201,14cyclictest5339-21ssh12:00:140
656999200,14cyclictest19914-21ssh11:30:230
660299192,9cyclictest29242-21kthreadcore08:40:192
660299192,9cyclictest1492-21irqcore08:45:182
660299192,15cyclictest9281-21sshd08:50:232
660299192,10cyclictest22983-21apt-get09:45:142
660299191,9cyclictest24916-21kthreadcore08:00:212
660299191,9cyclictest24076-21unixbench_multi09:05:292
660299191,9cyclictest19632-21apt-get09:25:122
6602991913,4cyclictest0-21swapper/207:35:232
660299191,10cyclictest8622-21tail07:45:182
660299191,10cyclictest5329-21kthreadcore07:40:202
660299191,10cyclictest28412-21kthreadcore07:30:222
660299191,10cyclictest25620-21cut09:10:172
660299191,10cyclictest21020-21grep07:55:242
6602991910,7cyclictest32057-21/usr/sbin/munin08:10:252
657999196,3cyclictest24830-21diskmemload10:13:251
657999195,4cyclictest24830-21diskmemload11:11:001
657999195,4cyclictest24830-21diskmemload10:22:101
657999192,6cyclictest131rcu_preempt09:18:141
657999192,12cyclictest3148-21ssh12:35:121
657999191,8cyclictest13395-21ntp_states10:50:261
657999191,3cyclictest131rcu_preempt07:55:191
657999191,2cyclictest0-21swapper/110:43:051
657999191,16cyclictest12129-21ls12:20:201
657999191,15cyclictest16295-21ssh11:30:111
6579991911,5cyclictest0-21swapper/108:30:181
657999190,7cyclictest131rcu_preempt09:43:261
657999190,3cyclictest0-21swapper/109:53:241
657999190,16cyclictest15446-21cut10:35:151
656999196,3cyclictest24830-21diskmemload12:14:020
656999196,10cyclictest16464-21grep11:10:220
656999193,6cyclictest131rcu_preempt11:06:180
656999191,8cyclictest0-21swapper/010:39:090
656999191,3cyclictest131rcu_preempt12:20:110
656999191,3cyclictest131rcu_preempt07:55:010
656999191,2cyclictest131rcu_preempt10:03:260
656999191,2cyclictest131rcu_preempt10:03:260
656999191,2cyclictest131rcu_preempt09:20:120
656999191,14cyclictest10808-21ssh11:45:140
656999191,13cyclictest7652-21ls11:25:180
660299181,9cyclictest17247-21kthreadcore07:20:192
660299181,7cyclictest19020-21cpuspeed08:30:152
6602991811,5cyclictest0-21swapper/207:25:182
660299180,9cyclictest12358-21kthreadcore07:15:192
657999187,9cyclictest21-21ksoftirqd/111:25:141
657999187,3cyclictest21-21ksoftirqd/108:10:151
657999185,3cyclictest24830-21diskmemload09:35:501
657999183,4cyclictest21-21ksoftirqd/109:10:391
657999182,2cyclictest131rcu_preempt10:47:061
657999182,2cyclictest131rcu_preempt10:34:541
657999181,8cyclictest30003-21kthreadcore08:05:211
657999181,4cyclictest724-21gpgv08:45:011
657999181,3cyclictest131rcu_preempt09:22:331
657999180,3cyclictest0-21swapper/111:01:191
656999181,7cyclictest24830-21diskmemload09:51:230
656999181,6cyclictest24830-21diskmemload09:12:380
656999181,3cyclictest131rcu_preempt08:25:010
656999181,3cyclictest0-21swapper/012:33:480
656999181,2cyclictest131rcu_preempt12:16:470
656999181,2cyclictest131rcu_preempt09:43:300
6569991811,5cyclictest2912-21cron07:40:000
656999181,14cyclictest700-21apt-get08:45:040
660299171,8cyclictest6491-21latency_hist07:10:022
657999179,6cyclictest28200-21cat08:35:301
657999179,6cyclictest23727-21irqstats09:45:201
657999179,6cyclictest171-21systemd-network07:52:561
657999179,6cyclictest10596-21/usr/sbin/munin08:55:291
657999179,5cyclictest5634-21if_err_eth012:00:171
657999178,7cyclictest30779-21awk07:35:141
657999178,6cyclictest20873-21cron07:25:001
657999177,8cyclictest1024-21apt-get08:45:131
657999173,3cyclictest131rcu_preempt11:55:321
657999173,3cyclictest131rcu_preempt07:10:021
657999172,8cyclictest823766ssh11:42:221
657999172,2cyclictest131rcu_preempt10:02:041
657999172,2cyclictest131rcu_preempt10:02:041
657999172,12cyclictest22395-21apt-get08:00:111
657999171,9cyclictest9054-21idleruntime-cro08:20:011
657999171,6cyclictest131rcu_preempt11:36:181
657999171,14cyclictest23881-21ls09:05:281
6579991710,5cyclictest0-21swapper/107:10:231
656999179,6cyclictest10196-21awk07:10:260
656999178,6cyclictest14487-21irqcore08:25:190
656999177,6cyclictest1706-21kthreadcore08:10:210
656999173,10cyclictest28779-21ssh11:55:150
656999171,13cyclictest14298-21ssh12:05:130
657999169,5cyclictest7697-21kthreadcore08:50:191
657999168,6cyclictest24202-21cut11:15:221
657999168,6cyclictest12304-21cat07:45:301
657999165,9cyclictest23622-21kthreadcore07:25:221
657999161,6cyclictest16475-21kthreadcore09:00:201
657999161,6cyclictest16475-21kthreadcore09:00:201
657999161,13cyclictest3160-21apt-get07:40:111
6579991610,4cyclictest0-21swapper/107:15:181
656999168,6cyclictest14934-21ntpq07:15:230
656999167,7cyclictest10348-21users08:50:270
656999165,8cyclictest22476-21df08:00:140
656999162,3cyclictest131rcu_preempt07:20:180
656999162,11cyclictest24506-21perl10:40:160
656999161,11cyclictest26948-21apt-get08:05:050
6569991610,4cyclictest22987-21kthreadcore07:25:210
6569991610,4cyclictest0-21swapper/008:55:200
656999158,5cyclictest0-21swapper/008:35:280
656999157,6cyclictest19741-21cut08:30:210
656999151,7cyclictest8300-21sed07:45:170
6569991511,2cyclictest121ksoftirqd/007:55:120
656999151,11cyclictest28557-21apt-get08:40:120
656999151,11cyclictest20254-21apt-get09:05:140
656999151,11cyclictest15444-21apt-get09:00:130
656999151,11cyclictest15444-21apt-get09:00:130
6569991510,3cyclictest3371-21hwlatdetect07:40:160
656999150,2cyclictest131rcu_preempt07:30:190
657999146,5cyclictest9747-21irqcore08:20:171
656999131,9cyclictest6560-21cyclictest07:10:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional