You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 05:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Wed Nov 26, 2025 00:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1160499561,7cyclictest0-21swapper/220:25:142
11581995635,18cyclictest151rcuc/021:25:120
1160499554,39cyclictest12190-21ssh23:40:122
1158199554,8cyclictest131rcu_preempt20:40:120
1158199554,8cyclictest131rcu_preempt20:40:120
11616995442,8cyclictest29566irq/58-eth000:30:133
1160499503,13cyclictest0-21swapper/223:20:162
1159499492,33cyclictest29195-21ssh23:10:131
1160499481,40cyclictest5073-21ssh22:35:132
1158199461,33cyclictest20204-21ssh23:45:140
11594994523,13cyclictest26028-21kworker/1:100:10:151
1159499452,27cyclictest29426-21ssh22:50:121
1160499431,29cyclictest31134-21ssh21:10:162
1160499431,29cyclictest31134-21ssh21:10:152
1160499425,12cyclictest26346irq/40-dwc2_hso23:45:122
1161699412,23cyclictest29566irq/58-eth020:40:133
1161699412,23cyclictest29566irq/58-eth020:40:133
1158199411,29cyclictest28178-21sh23:50:120
1158199404,24cyclictest12-21ksoftirqd/022:35:140
11581994026,6cyclictest12-21ksoftirqd/023:55:140
11594993918,14cyclictest629-21Async14:00:481
1158199393,29cyclictest21201-21ssh23:05:160
1160499389,4cyclictest18954-21ssh00:25:132
1160499382,15cyclictest5540-21ssh22:15:122
1160499381,9cyclictest25155-21timerandwakeup23:05:312
1160499371,9cyclictest5033-21kthreadcore19:35:212
1160499371,9cyclictest15368-21kthreadcore00:00:252
1160499371,8cyclictest14146-21kthreadcore22:40:222
1160499371,14cyclictest29459-21memory21:05:232
1160499370,8cyclictest16411-21grep19:15:012
11616993618,11cyclictest29566irq/58-eth000:35:133
1160499361,8cyclictest23318-21ssh21:45:202
1160499361,14cyclictest14823-21latency_hist21:20:012
1160499361,13cyclictest30139-21kthreadcore23:30:202
1160499361,13cyclictest28144-21/usr/sbin/munin23:50:272
1160499361,10cyclictest4891-21rm23:35:182
1159499362,27cyclictest29956-21ssh22:10:141
11581993617,12cyclictest5910-21ssh21:55:120
1160499353,12cyclictest15223-21irqcore20:20:172
1160499351,9cyclictest26450-21needreboot19:55:222
1160499351,8cyclictest717-21kthreadcore19:30:212
1160499351,15cyclictest7318-21ntp_states00:15:262
1160499351,15cyclictest7318-21ntp_states00:15:262
1160499351,13cyclictest29539-21ssh22:30:122
1160499350,9cyclictest16283-21kthreadcore19:45:212
1161699345,17cyclictest29566irq/58-eth000:25:133
1161699344,19cyclictest29566irq/58-eth021:55:133
1160499341,9cyclictest7583-21ssh23:55:252
1160499341,9cyclictest1295-21kthreadcore20:05:192
1160499341,8cyclictest29062-21latency_hist22:50:022
1160499341,8cyclictest29062-21latency_hist22:50:012
1160499341,8cyclictest22833-21latency_hist21:25:032
1160499340,8cyclictest4828-21latency_hist22:55:012
1158199343,25cyclictest12180-21ssh00:00:150
11616993318,10cyclictest0-21swapper/322:45:053
11616993318,10cyclictest0-21swapper/322:45:043
1160499332,8cyclictest20730-21kthreadcore19:50:212
1160499331,8cyclictest25225-21timerwakeupswit19:20:282
1160499331,14cyclictest14656-21kthreadcore22:20:212
1160499330,14cyclictest30839-21kthreadcore20:00:222
1159499331,24cyclictest0-21swapper/119:45:141
11581993324,5cyclictest0-21swapper/022:15:120
1158199332,26cyclictest11445-21ssh00:20:140
1161699322,20cyclictest29566irq/58-eth022:55:143
1161699322,17cyclictest29566irq/58-eth021:25:133
1160499328,8cyclictest30730-21diskmemload22:02:232
1160499322,9cyclictest1568-21timerandwakeup22:10:332
1160499322,10cyclictest20742-21ssh23:25:132
1159499329,15cyclictest21-21ksoftirqd/121:20:161
1159499321,26cyclictest20737-21ssh23:25:131
1159499320,25cyclictest0-21swapper/120:50:131
1161699316,15cyclictest29566irq/58-eth023:35:143
1161699313,15cyclictest29566irq/58-eth023:25:143
1160499311,8cyclictest25637-21ntp_states22:05:282
1160499311,10cyclictest29766-21memory00:30:222
1158199318,16cyclictest12-21ksoftirqd/022:40:150
1158199315,16cyclictest12-21ksoftirqd/020:25:130
1158199314,23cyclictest12-21ksoftirqd/021:00:160
11604993015,12cyclictest0-21swapper/221:25:282
11604993012,11cyclictest27-21ksoftirqd/223:00:142
1159499304,15cyclictest5065-21ssh22:55:121
1158199304,17cyclictest14361-21ssh21:40:130
1160499298,7cyclictest10710-21sh22:56:412
1158199297,16cyclictest12-21ksoftirqd/020:00:140
1158199294,14cyclictest12-21ksoftirqd/023:15:150
1160499282,20cyclictest0-21swapper/220:30:142
1160499282,19cyclictest29210-21ssh23:10:142
11581992815,8cyclictest11567-21cyclictest22:45:110
11581992815,8cyclictest11567-21cyclictest22:45:110
1158199281,20cyclictest13930-21ssh22:00:120
1161699273,19cyclictest29566irq/58-eth023:05:163
1161699273,16cyclictest29566irq/58-eth022:15:133
1161699271,22cyclictest29566irq/58-eth021:20:163
1160499274,15cyclictest26346irq/40-dwc2_hso00:10:142
1159499273,9cyclictest21193-21apt-get23:05:141
11594992713,8cyclictest0-21swapper/100:30:121
1158199272,17cyclictest29980-21ssh22:10:160
1161699263,15cyclictest0-21swapper/322:35:143
1160499262,14cyclictest26346irq/40-dwc2_hso00:35:142
1159499261,22cyclictest21340-21ssh22:45:131
1159499261,22cyclictest21340-21ssh22:45:131
1158199268,11cyclictest131rcu_preempt20:55:120
1158199262,17cyclictest13257-21ssh23:00:140
1158199261,5cyclictest131rcu_preempt19:50:130
1160499252,18cyclictest21541-21ssh22:25:132
1160499252,14cyclictest26346irq/40-dwc2_hso19:25:152
1159499250,19cyclictest0-21swapper/122:00:131
11616992410,10cyclictest0-21swapper/321:15:133
11616992314,6cyclictest0-21swapper/321:10:143
11616992314,6cyclictest0-21swapper/321:10:133
11616992311,9cyclictest0-21swapper/300:05:193
1160499239,11cyclictest21384-21df_inode21:00:162
1160499234,12cyclictest26346irq/40-dwc2_hso21:50:152
1160499232,16cyclictest0-21swapper/221:30:062
1160499231,15cyclictest0-21swapper/220:55:122
1159499233,11cyclictest31013-21ssh21:30:151
1159499232,17cyclictest6952-21ssh21:15:151
11594992312,6cyclictest21-21ksoftirqd/120:45:141
1159499231,19cyclictest13288-21ls22:20:031
1158199231,19cyclictest10201-21cut20:15:170
1158199231,16cyclictest4207-21sh23:35:140
1158199231,16cyclictest21416-21find19:20:120
1158199231,15cyclictest1836-21ssh22:50:310
11581992310,10cyclictest19300-21kworker/u8:122:55:140
1158199230,19cyclictest21813-21ls22:25:160
1161699221,12cyclictest0-21swapper/300:00:153
1160499221,11cyclictest6629-21kthreadcore20:10:232
1159499223,12cyclictest0-21swapper/119:35:151
1159499222,17cyclictest323-21ls20:35:241
1159499222,15cyclictest20190-21ssh23:45:121
11594992211,7cyclictest30404-21sh21:50:151
11581992210,3cyclictest131rcu_preempt20:50:200
11616992113,5cyclictest0-21swapper/323:20:153
11616992110,7cyclictest0-21swapper/300:20:133
1160499215,13cyclictest16709-21fschecks_time19:15:142
1160499213,13cyclictest26346irq/40-dwc2_hso20:50:142
1160499212,10cyclictest11587-21kthreadcore19:40:212
1160499211,11cyclictest11453-21latency_hist20:50:012
11594992111,7cyclictest8181-21tr21:35:241
1159499211,15cyclictest12203-21sh23:40:141
1158199219,3cyclictest12-21ksoftirqd/022:34:160
1158199213,12cyclictest0-21swapper/021:17:570
1158199212,16cyclictest12755-21diskmemload23:20:160
1158199212,13cyclictest0-21swapper/021:30:140
1158199211,6cyclictest1879666kworker/0:123:10:300
11581992113,5cyclictest0-21swapper/000:25:140
1158199211,15cyclictest29566-21cut20:35:180
1161699209,8cyclictest0-21swapper/323:55:133
1161699208,8cyclictest29566irq/58-eth022:25:123
11616992013,4cyclictest0-21swapper/323:30:123
11616992013,3cyclictest0-21swapper/320:55:133
1160499205,9cyclictest26346irq/40-dwc2_hso21:40:142
1160499203,14cyclictest30252-21kthreadcore20:35:212
1160499201,16cyclictest12889-21kthreadcore00:20:202
1160499201,15cyclictest10392-21timerandwakeup21:35:322
1160499201,14cyclictest13712-21grep22:00:012
1160499201,10cyclictest21331-21ssh00:05:202
1160499200,16cyclictest10248-21idleruntime20:15:182
1159499209,8cyclictest15796-21nfs4_client00:00:281
1159499203,13cyclictest0-21swapper/120:15:131
1159499201,7cyclictest11738-21gpgv00:00:011
11594992012,5cyclictest0-21swapper/120:25:171
1159499201,16cyclictest24707-21uniq20:30:171
1159499201,15cyclictest26841-21irqrtprio19:25:191
1159499201,14cyclictest3202-21apt-get00:15:041
1159499201,14cyclictest3202-21apt-get00:15:031
1159499201,14cyclictest22396-21ssh21:45:151
1159499201,14cyclictest13260-21ssh22:40:171
1159499201,10cyclictest19642-21kthreadcore19:50:201
1158199203,10cyclictest0-21swapper/021:05:120
1158199202,15cyclictest32242-21kthreadcore21:50:220
1158199201,3cyclictest131rcu_preempt19:15:100
1158199201,3cyclictest131rcu_preempt00:10:140
1161699199,8cyclictest0-21swapper/322:30:313
1161699195,6cyclictest29566irq/58-eth022:50:103
1161699193,12cyclictest0-21swapper/323:45:143
1161699191,11cyclictest0-21swapper/300:15:243
1161699191,11cyclictest0-21swapper/300:15:233
11616991910,6cyclictest0-21swapper/323:16:243
11616991910,6cyclictest0-21swapper/322:02:333
1160499191,9cyclictest1977-21tune2fs20:40:162
1160499191,9cyclictest1977-21tune2fs20:40:152
11604991914,3cyclictest9305-21ssh23:15:312
1159499199,7cyclictest17153-21cat23:20:331
1159499193,9cyclictest0-21swapper/122:30:111
1159499191,16cyclictest18657-21grep00:25:011
1159499191,15cyclictest1852-21apt-get20:40:131
1159499191,15cyclictest1852-21apt-get20:40:121
1158199193,3cyclictest131rcu_preempt19:35:020
1158199193,11cyclictest20719-21acpi23:25:140
1158199193,11cyclictest0-21swapper/021:10:130
1158199193,11cyclictest0-21swapper/021:10:130
1158199192,3cyclictest131rcu_preempt22:20:150
1158199192,11cyclictest3646-21ssh00:15:140
1158199192,11cyclictest3646-21ssh00:15:130
11581991913,4cyclictest0-21swapper/000:05:130
1158199191,11cyclictest32738-21cat23:30:290
1158199190,10cyclictest12036-21ssh21:36:490
1161699189,6cyclictest0-21swapper/321:42:063
1161699182,11cyclictest29566irq/58-eth023:00:143
1161699182,10cyclictest0-21swapper/323:10:053
11616991812,4cyclictest0-21swapper/322:21:383
11616991812,2cyclictest0-21swapper/323:40:123
11616991811,5cyclictest0-21swapper/300:10:223
1161699181,13cyclictest0-21swapper/321:45:163
11616991811,2cyclictest0-21swapper/321:50:153
1161699181,10cyclictest0-21swapper/321:00:163
1160499181,9cyclictest11626-21grep19:10:012
1159499188,8cyclictest21173-21cron19:20:011
1159499186,4cyclictest42-21kauditd22:25:011
1159499184,11cyclictest29424-21latency21:05:211
1159499182,11cyclictest21535-21ssh22:25:131
1159499181,14cyclictest16350-21kthreadcore23:00:231
1159499181,13cyclictest11813-21munin-run23:40:011
1159499181,10cyclictest18548-21ls20:20:281
11594991810,5cyclictest6776-21kthreadcore23:15:221
11594991810,5cyclictest30730-21diskmemload22:10:011
1158199188,7cyclictest15703-21sed21:20:210
1158199185,10cyclictest13111-21cron19:45:020
1158199182,7cyclictest30730-21diskmemload22:07:130
1158199181,9cyclictest826466ssh00:39:320
1158199181,9cyclictest15036-21fschecks_count20:20:150
11581991811,5cyclictest30259-21grep19:25:300
1161699179,6cyclictest0-21swapper/322:44:583
1161699179,6cyclictest0-21swapper/322:10:203
1161699174,9cyclictest0-21swapper/321:30:273
1161699172,9cyclictest0-21swapper/320:20:143
1161699172,9cyclictest0-21swapper/320:05:133
1161699172,9cyclictest0-21swapper/319:40:143
1161699172,9cyclictest0-21swapper/319:15:123
1161699172,10cyclictest0-21swapper/323:50:133
1161699172,10cyclictest0-21swapper/320:50:153
1161699172,10cyclictest0-21swapper/320:45:143
1161699171,9cyclictest0-21swapper/321:10:013
11616991712,3cyclictest0-21swapper/319:45:123
1161699171,13cyclictest0-21swapper/321:35:203
11616991710,4cyclictest0-21swapper/320:25:133
1159499179,6cyclictest22642-21cron21:25:021
1159499178,6cyclictest8974-21irqrtprio19:40:171
1159499175,10cyclictest16066-21cut21:40:201
1159499172,12cyclictest31211-21apt-get19:30:131
1159499171,8cyclictest16599-21sort20:55:171
1159499171,13cyclictest23383-21tr19:55:181
1159499171,12cyclictest932-21snmpd21:55:181
1159499171,10cyclictest31353-21ntp_192.168.11420:00:251
1159499170,11cyclictest28336-21cut23:50:161
1158199172,6cyclictest31934-21ssh00:32:150
1158199172,11cyclictest0-21swapper/020:05:130
1158199171,3cyclictest131rcu_preempt19:45:160
1161699168,6cyclictest0-21swapper/322:05:073
1161699162,9cyclictest29566irq/58-eth020:15:133
1161699162,10cyclictest0-21swapper/320:10:183
1161699161,8cyclictest0-21swapper/319:30:233
1159499167,6cyclictest6255-21cut20:10:231
1159499164,9cyclictest1435-21cpu00:35:151
1159499164,3cyclictest131rcu_preempt00:05:121
1159499163,9cyclictest0-21swapper/119:10:131
1159499162,11cyclictest32659-21cat23:30:301
1159499161,6cyclictest25899-21munin-run21:05:011
1159499161,4cyclictest0-21swapper/122:35:221
1159499161,12cyclictest23429-21kthreadcore19:20:201
1159499161,11cyclictest6567-21idleruntime-cro21:15:011
1159499161,11cyclictest6567-21idleruntime-cro21:15:001
1159499160,11cyclictest4098-21timerandwakeup20:05:271
1158199167,7cyclictest12507-21if_eth023:40:170
1158199162,9cyclictest0-21swapper/019:55:130
1158199162,4cyclictest0-21swapper/021:45:170
1158199162,11cyclictest5259-21cpuspeed20:10:180
1158199162,11cyclictest11476-21latency_hist19:10:010
1158199162,10cyclictest0-21swapper/019:10:160
1158199161,12cyclictest6689-21apt-get20:45:120
1158199161,12cyclictest24574-21apt-get20:30:140
1158199160,12cyclictest8234-21idleruntime-cro19:40:010
1161699159,4cyclictest0-21swapper/319:35:173
1161699157,6cyclictest0-21swapper/319:55:213
1159499157,5cyclictest11639-21latency_hist19:10:021
1161699149,2cyclictest0-21swapper/320:30:133
1161699148,4cyclictest0-21swapper/319:25:023
1161699147,3cyclictest0-21swapper/320:00:143
1161699146,3cyclictest0-21swapper/319:50:133
1161699142,9cyclictest0-21swapper/319:25:043
11616991410,2cyclictest0-21swapper/320:35:133
1161699138,3cyclictest29566irq/58-eth019:10:323
1161699101,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional