You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 08:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Tue Nov 25, 2025 00:45:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29862998355,18cyclictest29566irq/58-eth023:05:133
29854995729,19cyclictest27-21ksoftirqd/221:30:142
2985499531,14cyclictest131rcu_preempt23:45:132
2985499531,14cyclictest131rcu_preempt23:45:122
2985499522,40cyclictest27343-21ssh21:55:132
2985499522,40cyclictest27343-21ssh21:55:122
2985499521,11cyclictest131rcu_preempt00:20:152
2984599511,43cyclictest11685-21ssh22:05:131
2985499482,37cyclictest15947-21kthreadcore22:45:222
29837994831,13cyclictest30054-21apt-get19:10:140
29862994622,10cyclictest29566irq/58-eth000:30:153
2984599462,35cyclictest4887-21ssh22:40:131
2984599461,32cyclictest32490-21ssh00:15:151
2984599461,32cyclictest32490-21ssh00:15:141
2986299453,22cyclictest29566irq/58-eth023:25:133
2985499441,33cyclictest932-21snmpd19:35:162
29837994434,3cyclictest7886-21apt-get00:20:150
29854994324,15cyclictest0-21swapper/222:50:142
2985499422,32cyclictest20742-21ssh22:30:112
2984599422,33cyclictest24141-21ssh23:50:131
2985499416,13cyclictest28354-21ssh22:15:142
29854994113,18cyclictest1-21systemd23:50:132
2984599415,29cyclictest18947-21ssh21:50:111
2986299407,13cyclictest29566irq/58-eth021:45:133
2986299403,23cyclictest29566irq/58-eth021:20:133
2985499392,28cyclictest16010-21kthreadcore22:25:232
2986299372,24cyclictest29566irq/58-eth022:30:133
29862993721,8cyclictest29566irq/58-eth022:55:123
2985499372,29cyclictest32510-21tr22:55:222
2985499372,18cyclictest7532-21tr23:00:222
2985499372,18cyclictest21526-21ssh23:28:262
2985499372,18cyclictest11020-21sshd22:23:162
29837993726,7cyclictest21889-21apt-get23:10:130
2985499362,18cyclictest26819-21ssh22:13:222
29837993627,4cyclictest24133-21apt-get23:50:120
29837993623,7cyclictest24918-21apt-get20:45:140
29837993617,14cyclictest0-21swapper/000:10:150
2986299353,21cyclictest29566irq/58-eth021:40:153
2985499358,16cyclictest629-21Async14:00:482
2984599351,25cyclictest7464-21ssh23:40:141
29837993514,17cyclictest29034-21ssh22:35:130
2986299342,17cyclictest29566irq/58-eth022:20:113
29862993416,7cyclictest29566irq/58-eth021:35:143
2985499347,20cyclictest0-21swapper/200:15:152
2985499347,20cyclictest0-21swapper/200:15:142
2985499344,12cyclictest29836-21cyclictest21:20:122
2985499343,21cyclictest0-21swapper/219:15:132
2985499341,27cyclictest261rcuc/223:10:132
2983799344,13cyclictest30898-21memory21:55:240
2983799344,13cyclictest30898-21memory21:55:230
29837993417,14cyclictest0-21swapper/019:15:130
2985499333,17cyclictest26346irq/40-dwc2_hso20:10:132
2983799338,8cyclictest8432-21df00:00:160
2985499321,19cyclictest932-21snmpd23:20:142
2984599323,16cyclictest10632-21ssh21:45:141
2983799327,11cyclictest30618-21if_err_eth023:15:170
2983799323,12cyclictest22332-21cut22:10:210
29837993227,3cyclictest28269-21apt-get20:15:140
29837993223,4cyclictest16460-21diskmemload23:25:130
2986299312,20cyclictest29566irq/58-eth021:25:133
2985499314,7cyclictest27-21ksoftirqd/222:00:132
2985499314,12cyclictest131rcu_preempt00:35:132
2983799317,10cyclictest10005-21rm22:03:080
2983799316,10cyclictest13461-21ssh23:05:050
2983799314,11cyclictest24522-21cut00:30:220
29837993125,3cyclictest8793-21apt-get19:55:120
29837993124,3cyclictest16856-21apt-get21:10:130
2983799311,26cyclictest12-21ksoftirqd/021:05:130
2985499301,20cyclictest32449-21sh23:55:132
2985499301,17cyclictest16549-21ssh00:05:142
2984599302,20cyclictest32509-21cut22:55:221
2983799308,8cyclictest6214-21ssh23:57:290
2983799308,7cyclictest7428-21/usr/sbin/munin23:40:270
2983799306,11cyclictest5679-21ssh00:16:530
2983799306,11cyclictest5679-21ssh00:16:520
2983799306,10cyclictest8460-21ssh21:24:030
2983799305,9cyclictest21943-21ntp_192.168.11420:05:230
2983799305,10cyclictest9218-21sed20:25:270
2983799305,10cyclictest32032-21/usr/sbin/munin22:15:260
2983799305,10cyclictest10772-21ntp_192.168.11419:20:240
2983799304,9cyclictest441-21ssh21:19:380
2983799304,8cyclictest26728-21ssh22:51:370
2983799304,11cyclictest1925-21ssh22:55:390
2983799302,13cyclictest6036-21kthreadcore22:20:180
29837993018,10cyclictest13040-21ssh22:45:130
2985499292,20cyclictest0-21swapper/220:00:152
2984599291,3cyclictest7903-21ssh00:20:171
2983799298,7cyclictest16460-21diskmemload21:28:350
2983799296,9cyclictest6372-21ssh23:39:020
2983799295,8cyclictest536-21ssh21:38:120
2983799294,8cyclictest10677-21rm23:01:230
2983799293,11cyclictest19591-21cat00:25:280
2983799293,10cyclictest20942-21fschecks_time22:30:140
29837992921,6cyclictest10359-21apt-get20:30:140
2983799292,12cyclictest445-21grep00:35:210
2984599282,19cyclictest18472-21ssh22:46:231
29845992811,6cyclictest29836-21cyclictest20:30:121
29845992810,5cyclictest21-21ksoftirqd/123:14:231
2983799289,7cyclictest16460-21diskmemload21:31:530
2983799287,8cyclictest6620-21usage_Anybus21:40:310
2983799284,9cyclictest28052-21basename20:15:010
2983799284,11cyclictest5508-21cut22:40:180
2983799284,11cyclictest19183-21fschecks_time21:50:150
2983799283,11cyclictest619-21kthreadcore19:45:190
29837992813,9cyclictest0-21swapper/023:45:120
29837992813,9cyclictest0-21swapper/023:45:110
29837992812,11cyclictest0-21swapper/023:20:130
2985499273,19cyclictest0-21swapper/223:19:222
2985499272,10cyclictest0-21swapper/221:35:162
2985499271,19cyclictest0-21swapper/220:05:142
2985499271,16cyclictest0-21swapper/221:50:282
2984599277,9cyclictest21-21ksoftirqd/122:22:011
2984599272,21cyclictest20050-21ssh22:10:141
2984599272,20cyclictest0-21swapper/122:51:211
2984599270,3cyclictest131rcu_preempt23:35:021
2983799276,8cyclictest22015-21ssh00:06:260
2983799275,9cyclictest14216-21missed_timers21:45:250
2983799274,8cyclictest0-21swapper/022:06:080
29837992721,3cyclictest17015-21apt-get19:30:160
29837992712,12cyclictest0-21swapper/020:40:130
29837992710,12cyclictest23004-21apt-get23:30:160
2986299263,15cyclictest0-21swapper/322:45:233
2986299263,12cyclictest29566irq/58-eth019:20:143
2986299262,17cyclictest0-21swapper/323:18:233
2985499263,7cyclictest0-21swapper/223:40:132
2985499261,14cyclictest16460-21diskmemload23:05:142
2984599269,6cyclictest21-21ksoftirqd/121:43:171
2984599268,8cyclictest21-21ksoftirqd/122:38:411
2984599263,3cyclictest131rcu_preempt21:35:011
2984599261,20cyclictest15694-21ssh00:25:131
29845992611,5cyclictest21-21ksoftirqd/100:09:091
2983799264,16cyclictest12-21ksoftirqd/020:00:140
2983799263,8cyclictest0-21swapper/022:27:390
2983799262,11cyclictest15212-21acpi20:35:110
2986299254,13cyclictest29566irq/58-eth022:10:133
2985499254,14cyclictest26346irq/40-dwc2_hso19:20:132
2985499254,13cyclictest932-21snmpd19:30:152
2985499252,9cyclictest0-21swapper/219:40:162
2985499252,8cyclictest11572-21ssh22:43:232
2985499252,20cyclictest0-21swapper/219:10:152
2985499251,18cyclictest26346irq/40-dwc2_hso21:15:152
2985499251,13cyclictest8339-21ssh00:00:142
2984599259,8cyclictest21-21ksoftirqd/122:29:181
2984599259,4cyclictest21-21ksoftirqd/121:24:031
2984599258,9cyclictest21-21ksoftirqd/121:28:071
2983799250,12cyclictest29828-21cut19:40:240
2986299243,14cyclictest0-21swapper/300:18:353
2986299243,14cyclictest0-21swapper/300:18:353
2985499242,12cyclictest0-21swapper/219:55:132
2984599249,7cyclictest21-21ksoftirqd/119:55:151
2984599248,6cyclictest21-21ksoftirqd/123:27:381
2984599247,8cyclictest21-21ksoftirqd/100:01:281
2984599246,8cyclictest21-21ksoftirqd/121:57:361
2984599246,8cyclictest21-21ksoftirqd/121:57:361
2984599246,3cyclictest131rcu_preempt22:15:291
2984599245,4cyclictest131rcu_preempt23:05:121
29845992416,5cyclictest0-21swapper/100:10:141
2984599241,5cyclictest21-21ksoftirqd/123:15:311
29845992412,4cyclictest21-21ksoftirqd/122:30:211
29845992411,3cyclictest21-21ksoftirqd/100:35:271
29845992410,8cyclictest21-21ksoftirqd/123:02:221
29845992410,4cyclictest21-21ksoftirqd/123:45:351
29845992410,4cyclictest21-21ksoftirqd/123:45:351
2983799249,11cyclictest7010-21apt-get21:00:110
2983799249,11cyclictest7010-21apt-get21:00:110
2983799243,7cyclictest0-21swapper/019:25:310
29837992414,4cyclictest693-21apt-get20:20:130
2986299232,15cyclictest0-21swapper/323:40:143
2986299231,16cyclictest0-21swapper/321:10:153
2985499232,15cyclictest9503-21ssh21:25:152
2985499232,15cyclictest26346irq/40-dwc2_hso20:35:132
2985499232,14cyclictest0-21swapper/219:25:132
2985499232,12cyclictest0-21swapper/220:30:132
29854992312,9cyclictest0-21swapper/223:35:282
29854992311,9cyclictest21435-21ssh00:27:072
2984599238,6cyclictest21-21ksoftirqd/120:40:231
2984599238,4cyclictest131rcu_preempt23:20:241
2984599231,16cyclictest29809-21ls20:50:141
29845992311,5cyclictest21-21ksoftirqd/119:35:001
2983799232,8cyclictest1-21systemd19:37:480
2986299222,12cyclictest29566irq/58-eth023:45:133
2986299222,12cyclictest29566irq/58-eth023:45:123
2985499224,9cyclictest12896-21kthreadcore21:05:192
2985499223,15cyclictest0-21swapper/220:45:152
2984599229,7cyclictest21-21ksoftirqd/121:13:011
2984599229,5cyclictest21-21ksoftirqd/121:16:291
2984599228,7cyclictest131rcu_preempt23:35:221
2984599227,8cyclictest21-21ksoftirqd/122:01:411
2984599224,9cyclictest21-21ksoftirqd/100:30:421
2984599224,3cyclictest7755-21ssh23:59:451
29845992218,2cyclictest21-21ksoftirqd/121:00:131
29845992218,2cyclictest21-21ksoftirqd/121:00:131
29845992211,5cyclictest21-21ksoftirqd/120:30:021
2983799224,7cyclictest7465-21open_inodes19:50:280
2986299212,13cyclictest0-21swapper/323:35:153
29862992110,8cyclictest0-21swapper/323:59:593
2985499212,3cyclictest0-21swapper/222:35:132
2985499211,7cyclictest0-21swapper/221:00:122
2985499211,7cyclictest0-21swapper/221:00:112
2985499211,11cyclictest31186-21sed23:35:012
2985499210,3cyclictest131rcu_preempt20:25:132
2984599219,3cyclictest131rcu_preempt21:35:051
2984599216,6cyclictest21-21ksoftirqd/120:10:021
2984599216,3cyclictest131rcu_preempt19:48:221
2984599214,13cyclictest131rcu_preempt19:50:181
2984599211,5cyclictest21-21ksoftirqd/119:40:231
29845992110,4cyclictest131rcu_preempt20:00:251
2983799218,9cyclictest6944-21date21:00:020
2986299203,9cyclictest29566irq/58-eth020:40:133
2986299202,10cyclictest0-21swapper/321:55:133
2986299202,10cyclictest0-21swapper/321:55:123
29862992013,4cyclictest0-21swapper/300:05:283
29862992011,6cyclictest0-21swapper/300:00:413
2986299201,15cyclictest0-21swapper/323:10:133
2986299201,12cyclictest0-21swapper/320:45:303
2986299200,11cyclictest29566irq/58-eth022:25:133
2985499208,9cyclictest1955-21sshd20:53:012
2985499202,9cyclictest27654-21kthreadcore00:10:232
2985499202,3cyclictest0-21swapper/200:30:152
2985499201,9cyclictest2268-21ssh21:40:142
2985499201,10cyclictest4156-21fschecks_time19:50:182
29854992010,3cyclictest16962-21sh21:10:152
2984599208,3cyclictest131rcu_preempt20:45:201
2984599207,8cyclictest21-21ksoftirqd/119:35:271
2984599201,9cyclictest21-21ksoftirqd/120:55:241
29845992013,4cyclictest0-21swapper/120:20:181
29845992010,4cyclictest21-21ksoftirqd/120:10:191
2983799204,5cyclictest0-21swapper/020:54:110
2986299199,6cyclictest0-21swapper/321:51:263
2986299193,11cyclictest0-21swapper/323:23:243
2986299193,11cyclictest0-21swapper/323:01:223
2986299193,11cyclictest0-21swapper/322:15:223
2986299192,10cyclictest0-21swapper/320:35:133
29862991912,4cyclictest0-21swapper/322:40:123
29862991911,3cyclictest0-21swapper/300:20:173
2986299191,12cyclictest0-21swapper/320:55:293
29862991910,6cyclictest29566irq/58-eth022:50:133
29862991910,6cyclictest0-21swapper/300:11:263
2985499196,5cyclictest2528-21irqcore20:55:182
2985499191,8cyclictest20133-21ls20:40:142
2985499191,8cyclictest12391-21kernelversion22:05:192
29854991912,4cyclictest932-21snmpd19:45:182
2985499191,14cyclictest846-21grep20:20:182
2985499191,10cyclictest29884-21latency_hist19:10:002
2985499191,10cyclictest10754-21diskstats21:45:172
2984599198,4cyclictest21-21ksoftirqd/119:25:291
2984599197,4cyclictest21-21ksoftirqd/120:15:321
2984599195,9cyclictest21-21ksoftirqd/121:05:211
2984599192,12cyclictest0-21swapper/119:10:131
2984599191,3cyclictest0-21swapper/120:35:241
2984599191,2cyclictest0-21swapper/119:15:231
2986299183,8cyclictest0-21swapper/319:35:163
2986299182,10cyclictest0-21swapper/320:30:153
2986299182,10cyclictest0-21swapper/320:25:143
2986299182,10cyclictest0-21swapper/320:10:143
2986299182,10cyclictest0-21swapper/320:00:153
2986299182,10cyclictest0-21swapper/319:45:133
29862991812,4cyclictest0-21swapper/300:25:263
29862991810,5cyclictest0-21swapper/322:35:213
2984599187,3cyclictest21-21ksoftirqd/119:20:231
2983799181,15cyclictest29836-21cyclictest19:10:000
2986299179,6cyclictest0-21swapper/323:52:253
2986299179,6cyclictest0-21swapper/322:00:063
2986299178,6cyclictest0-21swapper/300:35:163
2986299172,10cyclictest29666irq/59-eth019:30:153
2986299172,10cyclictest0-21swapper/321:00:133
2986299172,10cyclictest0-21swapper/321:00:123
2986299171,7cyclictest29566irq/58-eth023:30:143
2986299171,11cyclictest0-21swapper/320:55:013
2986299171,10cyclictest0-21swapper/320:15:153
2986299171,10cyclictest0-21swapper/319:55:123
2986299168,5cyclictest29566irq/58-eth021:15:153
2986299162,9cyclictest0-21swapper/321:05:143
2986299162,12cyclictest0-21swapper/321:30:243
2986299161,12cyclictest0-21swapper/322:05:193
2986299161,12cyclictest0-21swapper/320:20:183
2986299156,5cyclictest29566irq/58-eth019:25:143
2986299152,9cyclictest0-21swapper/319:40:163
2986299151,9cyclictest0-21swapper/319:15:143
2986299141,9cyclictest0-21swapper/319:10:153
2986299141,10cyclictest0-21swapper/320:05:213
2984599142,10cyclictest29847-21latency_hist19:10:011
2986299131,9cyclictest0-21swapper/319:50:233
2986299111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional