You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-05 - 07:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri Sep 05, 2025 00:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2668899591,12cyclictest6738-21ssh22:25:141
2668899552,42cyclictest12978-21ssh21:30:141
2668899552,42cyclictest12978-21ssh21:30:131
2668899511,3cyclictest18712-21ssh00:10:161
2667999512,14cyclictest12-21ksoftirqd/000:35:180
2667999512,14cyclictest12-21ksoftirqd/000:35:170
2667999491,37cyclictest17330-21ssh23:30:140
2670099473,27cyclictest21281-21ssh21:35:152
2667999473,35cyclictest24008-21ssh22:55:140
2667999463,10cyclictest12-21ksoftirqd/000:15:180
26706994518,15cyclictest25366irq/58-eth023:30:133
2668899454,10cyclictest131rcu_preempt23:40:121
2668899453,37cyclictest4729-21ssh21:25:151
2670099444,27cyclictest29634-21ssh21:40:132
2667999443,30cyclictest32286-21ssh23:00:160
2667999443,30cyclictest17935-21ssh23:50:140
2668899434,2cyclictest0-21swapper/100:35:181
2668899434,2cyclictest0-21swapper/100:35:171
2668899431,5cyclictest22578-21apt-get22:15:131
2668899431,13cyclictest30867-21ssh22:20:141
2670099429,20cyclictest27-21ksoftirqd/220:15:152
2670099426,18cyclictest27-21ksoftirqd/221:45:142
2668899422,2cyclictest2551-21apt-get21:00:141
26679994215,18cyclictest12-21ksoftirqd/020:20:130
2668899411,5cyclictest26897-21ssh00:15:171
2668899411,5cyclictest14411-21ssh22:10:141
2668899401,6cyclictest15089-21apt-get22:30:151
2668899401,6cyclictest0-21swapper/119:30:171
2668899401,6cyclictest0-21swapper/119:30:161
2667999402,28cyclictest18718-21ssh00:10:160
2668899391,7cyclictest0-21swapper/119:45:141
2667999392,24cyclictest12-21ksoftirqd/021:40:140
2670699388,6cyclictest25366irq/58-eth021:40:123
2668899382,3cyclictest0-21swapper/119:55:161
2668899382,33cyclictest30327-21ssh22:00:151
2668899382,33cyclictest30327-21ssh22:00:141
2668899380,5cyclictest29637-21apt-get21:40:121
2668899370,6cyclictest15755-21apt-get22:50:161
26706993624,9cyclictest25466irq/59-eth021:25:143
2670099362,15cyclictest13778-21ssh21:50:162
2668899361,8cyclictest9223-21rm23:05:201
2668899361,8cyclictest9223-21rm23:05:191
2670099358,15cyclictest0-21swapper/221:25:152
2668899356,8cyclictest30605-21cat23:55:301
2668899354,21cyclictest6189-21ssh22:05:141
2668899351,9cyclictest5035-21latency_hist19:55:011
2668899351,9cyclictest19393-21latency_hist20:10:021
2668899351,9cyclictest14919-21head20:35:291
2668899351,8cyclictest29463-21kthreadcore19:10:181
2668899351,10cyclictest21751-21cut20:45:201
2668899350,8cyclictest8569-21ssh00:03:041
2670699343,20cyclictest25366irq/58-eth000:00:143
2670699342,26cyclictest25366irq/58-eth019:45:143
2668899347,24cyclictest5048-21kthreadcore19:20:211
2668899341,9cyclictest9249-21cut21:05:201
2668899341,9cyclictest326-21sed23:00:191
2668899341,9cyclictest28520-21latency_hist21:20:011
2668899341,9cyclictest22130-21df_inode21:55:171
2668899341,9cyclictest13931-21cat00:05:271
2668899341,8cyclictest25810-21kthreadcore19:40:251
2668899341,7cyclictest17252-21munin-plugin-st00:30:011
2668899341,6cyclictest4349-21ssh22:40:581
2668899341,5cyclictest22720-21users23:50:331
2668899341,10cyclictest20400-21ssh21:15:021
2668899341,10cyclictest10550-21kthreadcore19:25:241
2668899340,8cyclictest20927-21cat23:10:341
2667999341,7cyclictest12-21ksoftirqd/021:45:140
2667999340,7cyclictest131rcu_preempt19:25:180
2670699333,10cyclictest25466irq/59-eth021:15:143
2668899337,24cyclictest24829-21cut22:55:211
2668899335,3cyclictest0-21swapper/123:34:501
2668899331,9cyclictest19824-21kthreadcore19:35:201
2668899331,8cyclictest3578-21cut23:20:231
2668899331,8cyclictest30204-21cut20:20:191
2668899331,8cyclictest27708-21ntp_192.168.11420:15:261
2668899331,8cyclictest23984-21irqrtprio22:35:211
2668899331,8cyclictest23984-21irqrtprio22:35:201
2668899331,8cyclictest17882-21grep21:50:331
2668899331,5cyclictest1104-21latency_hist23:40:011
2668899331,10cyclictest22315-21cut21:35:231
2667999332,23cyclictest20637-21ssh21:15:160
26679993310,12cyclictest12-21ksoftirqd/022:50:170
2670699327,15cyclictest25366irq/58-eth022:00:143
2670699327,15cyclictest25366irq/58-eth022:00:143
2670099323,15cyclictest21846irq/40-dwc2_hso19:10:092
2670099322,16cyclictest6172-21ssh22:05:142
2668899321,9cyclictest12427-21kthreadcore23:45:241
2668899321,8cyclictest6052-21ps21:45:201
2668899321,8cyclictest17524-21kthreadcore20:40:251
2668899321,8cyclictest1605-21fschecks_count20:25:171
2668899321,8cyclictest11335-21find22:45:321
2668899321,7cyclictest9479-21sed23:25:161
2668899321,5cyclictest17802-21ssh00:30:181
2667999321,5cyclictest9654-21ssh23:45:160
2670699313,16cyclictest0-21swapper/323:10:173
2670699312,9cyclictest0-21swapper/322:40:153
2670099315,15cyclictest21846irq/40-dwc2_hso22:55:152
2670099314,19cyclictest21846irq/40-dwc2_hso19:45:132
2668899311,8cyclictest26783-21latency_hist19:10:011
2668899311,8cyclictest26783-21latency_hist19:10:011
2668899311,6cyclictest9918-21apt-key20:00:021
2668899311,6cyclictest6384-21grep20:30:161
2668899310,9cyclictest22953-21ntpq20:10:251
2670699302,17cyclictest25366irq/58-eth021:30:133
2670699302,17cyclictest25366irq/58-eth021:30:133
2670099304,14cyclictest21846irq/40-dwc2_hso00:05:172
2670099303,18cyclictest12972-21ssh21:30:132
2670099303,18cyclictest12972-21ssh21:30:122
2670099301,3cyclictest0-21swapper/219:35:122
2668899302,4cyclictest0-21swapper/120:50:151
2668899301,8cyclictest30164-21/usr/sbin/munin20:55:231
2668899301,8cyclictest2997-21tail19:15:291
2668899301,7cyclictest24856-21ssh23:15:131
2668899301,6cyclictest28952-21awk21:20:161
2668899301,5cyclictest7183-21ssh00:20:471
2667999302,22cyclictest32025-21ssh23:38:020
26679993013,12cyclictest629-21Async08:52:160
2667999301,25cyclictest7591-21ssh22:45:160
2670699291,16cyclictest25366irq/58-eth000:35:183
2670699291,16cyclictest25366irq/58-eth000:35:173
2670099297,17cyclictest27-21ksoftirqd/200:36:362
2670099297,17cyclictest27-21ksoftirqd/200:36:362
2670099291,22cyclictest0-21swapper/222:25:162
2670099291,21cyclictest0-21swapper/221:10:042
2667999294,21cyclictest15115-21ssh22:30:140
2667999293,16cyclictest28850-21ssh21:20:130
2670699283,18cyclictest0-21swapper/323:43:023
26706992813,9cyclictest25366irq/58-eth023:20:143
2670099282,20cyclictest0-21swapper/221:15:162
2670699273,12cyclictest25366irq/58-eth022:05:133
26706992710,3cyclictest0-21swapper/322:51:453
2670099273,7cyclictest0-21swapper/220:05:122
2670099272,18cyclictest21846irq/40-dwc2_hso20:45:142
2670099272,13cyclictest22592-21ssh22:15:152
2667999272,19cyclictest22972-21ssh23:12:570
2670699269,4cyclictest0-21swapper/323:15:023
2670699268,3cyclictest0-21swapper/323:05:433
2670699268,3cyclictest0-21swapper/323:05:423
2670699263,17cyclictest0-21swapper/300:20:353
2670699261,8cyclictest0-21swapper/300:00:023
2670699261,5cyclictest0-21swapper/322:20:183
2670099264,12cyclictest21846irq/40-dwc2_hso20:00:132
2670099263,14cyclictest21846irq/40-dwc2_hso19:20:132
2670099262,7cyclictest942-21snmpd20:55:142
2667999261,4cyclictest131rcu_preempt19:20:130
26679992614,3cyclictest12-21ksoftirqd/022:25:100
26679992613,8cyclictest12-21ksoftirqd/019:10:110
26679992612,8cyclictest2126-21kworker/u8:023:25:120
2667999261,16cyclictest12-21ksoftirqd/020:15:140
2670699259,3cyclictest0-21swapper/321:50:573
2670699259,3cyclictest0-21swapper/300:08:423
2670699253,9cyclictest0-21swapper/319:50:123
2670699251,8cyclictest0-21swapper/300:30:143
2670699251,7cyclictest0-21swapper/300:10:203
2670099252,9cyclictest0-21swapper/222:30:142
2670099252,6cyclictest9270-21munin-run23:45:012
2670099251,4cyclictest0-21swapper/200:00:152
2670099251,19cyclictest11916-21diskmemload23:30:362
2667999258,3cyclictest0-21swapper/021:55:090
2667999253,5cyclictest131rcu_preempt20:35:140
2667999252,18cyclictest17562-21apt-get00:30:130
2670699249,4cyclictest0-21swapper/300:17:593
2670699249,3cyclictest0-21swapper/321:38:163
2670699249,12cyclictest0-21swapper/321:20:263
2670699242,6cyclictest0-21swapper/323:45:203
2670699242,5cyclictest0-21swapper/323:51:353
2670699241,9cyclictest0-21swapper/322:30:223
2670099245,16cyclictest12353-21kthreadcore00:25:222
2670099243,16cyclictest0-21swapper/200:30:142
2670099242,6cyclictest0-21swapper/200:21:352
2670099242,18cyclictest17236-21ssh23:49:352
2670099241,15cyclictest26673-21ssh00:15:012
2667999249,4cyclictest12-21ksoftirqd/023:20:280
2667999242,15cyclictest12375-21ssh21:10:130
2670699238,3cyclictest0-21swapper/321:48:303
2670699236,6cyclictest0-21swapper/323:25:563
2670699236,6cyclictest0-21swapper/322:56:523
2670699235,5cyclictest0-21swapper/322:39:123
2670699235,5cyclictest0-21swapper/322:39:113
2670699232,5cyclictest0-21swapper/319:35:133
2670699232,5cyclictest0-21swapper/300:25:173
2670099235,13cyclictest11916-21diskmemload23:35:142
2670099233,13cyclictest21846irq/40-dwc2_hso23:00:142
2670099232,16cyclictest0-21swapper/223:51:362
2670099231,11cyclictest2886566ssh21:20:152
2667999239,4cyclictest0-21swapper/021:32:480
2667999239,4cyclictest0-21swapper/021:32:470
26679992312,3cyclictest12-21ksoftirqd/022:04:080
26679992312,3cyclictest12-21ksoftirqd/022:04:070
26679992311,6cyclictest0-21swapper/023:43:020
26679992310,4cyclictest0-21swapper/023:55:430
2667999230,19cyclictest32682-21ls19:50:010
2670699229,3cyclictest0-21swapper/323:00:553
2670699228,3cyclictest0-21swapper/321:13:113
2670699223,12cyclictest0-21swapper/322:10:153
2670699222,7cyclictest0-21swapper/323:35:363
2670099229,5cyclictest14816-21irqcore22:10:192
2670099229,10cyclictest23334-21sh22:54:352
2670099224,13cyclictest23337-21ssh22:35:162
2670099224,13cyclictest23337-21ssh22:35:162
2670099224,13cyclictest11963-21kthreadcore19:25:272
2670099224,12cyclictest21846irq/40-dwc2_hso20:10:122
2670099221,18cyclictest19150-21wc20:40:262
2670099221,14cyclictest11028-21find20:35:132
2670099221,13cyclictest8111-21ssh23:05:132
2670099221,13cyclictest8111-21ssh23:05:132
2667999221,18cyclictest23024-21ls19:40:010
26706992111,7cyclictest0-21swapper/322:45:303
2670099215,13cyclictest5155-21cat20:25:312
2670099214,14cyclictest2552-21usage_EthcTask22:00:332
2670099214,14cyclictest2552-21usage_EthcTask22:00:322
2670099212,10cyclictest26899-21cpu00:15:182
2670099212,10cyclictest22089-21cpuspeed21:55:162
2670099211,9cyclictest11384-21cut23:25:202
26700992114,4cyclictest31645-21ssh22:20:202
26700992112,3cyclictest26162-21find23:55:132
2670099211,16cyclictest9633-21idleruntime-cro20:00:002
2670099211,16cyclictest27720-21sh19:45:022
2670099211,16cyclictest11179-21unixbench_multi21:05:302
2670099211,10cyclictest6206-21cat21:00:282
2670099211,10cyclictest10820-21grep20:35:012
26700992110,8cyclictest16628-21grep23:10:212
2667999212,13cyclictest8979-21ssh00:23:360
26679992115,4cyclictest0-21swapper/022:20:130
26679992114,4cyclictest0-21swapper/022:10:130
26679992110,3cyclictest0-21swapper/000:09:590
26679992110,2cyclictest12-21ksoftirqd/000:04:390
2670699209,8cyclictest0-21swapper/321:56:303
26706992011,4cyclictest0-21swapper/322:18:323
2670099207,10cyclictest28175-21kthreadcore20:50:242
2670099205,12cyclictest2192-21kthreadcore19:50:212
2670099203,15cyclictest15379-21ls22:50:012
2670099202,11cyclictest459-21kthreadcore19:15:212
2670099201,9cyclictest25687-21kthreadcore23:15:192
26700992015,3cyclictest942-21snmpd23:20:402
2670099201,12cyclictest2205-21kthreadcore22:40:222
2670099201,10cyclictest14555-21kthreadcore19:30:232
2670099201,10cyclictest14555-21kthreadcore19:30:222
2670099200,7cyclictest29856-21kthreadcore20:20:182
2667999202,15cyclictest25258-21kthreadcore22:15:210
26679992014,3cyclictest12-21ksoftirqd/022:05:280
26679992013,4cyclictest12-21ksoftirqd/021:37:490
2667999201,16cyclictest8893-21tail19:55:300
2667999201,16cyclictest23154-21ls20:10:290
2667999200,17cyclictest23295-21find22:35:150
2667999200,17cyclictest23295-21find22:35:140
2670699192,10cyclictest0-21swapper/319:20:163
2667999195,7cyclictest12-21ksoftirqd/021:29:480
2667999195,3cyclictest131rcu_preempt21:00:140
2667999192,14cyclictest14895-21fschecks_time20:05:150
2667999192,12cyclictest12-21ksoftirqd/000:26:370
2667999191,7cyclictest14573-21grep20:05:020
26679991913,3cyclictest12-21ksoftirqd/021:00:010
2667999191,10cyclictest0-21swapper/023:08:100
2667999191,10cyclictest0-21swapper/023:08:090
2667999190,4cyclictest131rcu_preempt22:40:150
2670699182,10cyclictest0-21swapper/320:10:153
2670699182,10cyclictest0-21swapper/319:15:143
2670699181,12cyclictest25366irq/58-eth019:25:173
2670699181,10cyclictest0-21swapper/319:30:183
2670699181,10cyclictest0-21swapper/319:30:173
2670099181,8cyclictest26698-21latency_hist19:10:022
2670099181,8cyclictest26698-21latency_hist19:10:012
2667999187,8cyclictest12-21ksoftirqd/021:05:130
2667999187,3cyclictest131rcu_preempt19:15:220
2667999184,4cyclictest12-21ksoftirqd/019:50:260
2667999183,11cyclictest0-21swapper/021:50:170
2667999182,3cyclictest131rcu_preempt19:40:190
2670699172,9cyclictest0-21swapper/320:50:173
2670699172,11cyclictest0-21swapper/322:25:153
2670699172,10cyclictest0-21swapper/320:55:143
2670699172,10cyclictest0-21swapper/320:45:133
2670699172,10cyclictest0-21swapper/320:00:143
2670699172,10cyclictest0-21swapper/319:40:183
2670699171,13cyclictest0-21swapper/320:30:293
2667999174,10cyclictest15003-21kthreadcore19:30:230
2667999174,10cyclictest15003-21kthreadcore19:30:230
2667999173,11cyclictest4792-21missed_timers20:25:250
26679991713,2cyclictest0-21swapper/020:40:200
2667999171,13cyclictest22324-21tr20:45:200
26679991711,3cyclictest12-21ksoftirqd/020:30:280
26679991710,4cyclictest12-21ksoftirqd/020:50:230
2670699151,9cyclictest0-21swapper/320:30:013
2670699151,6cyclictest0-21swapper/321:10:013
2670699149,3cyclictest0-21swapper/319:10:263
2670699141,7cyclictest0-21swapper/320:40:183
2667999148,3cyclictest12-21ksoftirqd/019:10:020
2667999148,3cyclictest12-21ksoftirqd/019:10:010
2670699138,3cyclictest0-21swapper/320:20:073
2670699138,3cyclictest0-21swapper/320:05:143
2670699132,8cyclictest0-21swapper/321:00:183
2670699131,9cyclictest0-21swapper/320:35:213
2670699121,8cyclictest0-21swapper/320:15:243
2670699121,5cyclictest0-21swapper/319:55:223
2670699111,7cyclictest0-21swapper/319:10:013
2670699101,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional