You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 15:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Tue Nov 25, 2025 12:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2578799563,33cyclictest5303-21ssh11:35:132
2576899522,39cyclictest3334-21ssh12:15:140
2577999451,37cyclictest21923-21ssh10:00:141
25768994534,6cyclictest30054-21apt-get11:30:130
2577999431,33cyclictest3025-21ssh12:35:141
2580399423,19cyclictest29566irq/58-eth012:30:143
2578799402,9cyclictest131rcu_preempt12:10:132
25787994015,16cyclictest0-21swapper/211:45:132
2578799401,26cyclictest3789-21ssh11:55:142
25803993819,13cyclictest29566irq/58-eth010:00:143
2578799382,18cyclictest26838-21ssh10:45:182
2577999381,31cyclictest31801-21ssh11:10:121
25768993818,13cyclictest7150-21ssh11:15:140
2578799375,4cyclictest131rcu_preempt11:05:162
2578799372,14cyclictest0-21swapper/212:15:152
2577999361,27cyclictest12460-21diskmemload12:10:141
2576899366,12cyclictest2939-21grep09:45:220
25768993627,3cyclictest9729-21apt-get08:00:160
2580399355,17cyclictest29566irq/58-eth011:35:123
2578799354,26cyclictest261rcuc/212:30:122
2578799353,23cyclictest0-21swapper/210:25:142
2578799352,19cyclictest22289-21ssh11:25:142
2580399344,18cyclictest29566irq/58-eth009:10:123
2580399343,18cyclictest29566irq/58-eth007:55:153
25803993412,12cyclictest29566irq/58-eth009:35:133
2576899344,20cyclictest12-21ksoftirqd/007:10:130
25768993421,10cyclictest19513-21apt-get12:25:140
2580399333,18cyclictest29566irq/58-eth007:20:123
25787993322,8cyclictest261rcuc/209:50:142
2578799332,13cyclictest23102-21ssh10:40:302
2577999332,25cyclictest27207-21ssh09:20:141
2576899338,8cyclictest8257-21/usr/sbin/munin10:30:320
2576899337,13cyclictest12460-21diskmemload11:46:450
2576899335,11cyclictest4578-21df_inode10:10:140
2576899334,26cyclictest9617-21ssh09:30:120
25768993325,4cyclictest12231-21apt-get10:15:150
25787993218,7cyclictest530-21usb-storage07:50:112
25787993210,16cyclictest0-21swapper/207:30:152
25779993218,6cyclictest25753-21cyclictest10:45:131
25768993224,5cyclictest22270-21apt-get11:25:130
25768993213,14cyclictest0-21swapper/010:20:120
25768993213,14cyclictest0-21swapper/010:20:110
2578799315,3cyclictest131rcu_preempt10:10:142
2578799313,18cyclictest26346irq/40-dwc2_hso12:00:152
2578799313,18cyclictest26346irq/40-dwc2_hso12:00:152
2578799312,17cyclictest3036-21ssh12:35:152
25787993111,12cyclictest25753-21cyclictest10:55:152
2577999312,21cyclictest13233-21sh09:53:311
2576899316,9cyclictest16686-21ssh10:36:540
2576899316,16cyclictest8024-21latency_hist07:25:020
2576899316,10cyclictest28600-21ssh10:03:560
2576899315,10cyclictest3433-21cut10:50:210
25768993119,7cyclictest12811-21apt-get09:10:130
25803993017,7cyclictest0-21swapper/312:25:143
2577999305,3cyclictest131rcu_preempt12:20:331
2577999301,22cyclictest23354-21cat10:40:301
2576899306,8cyclictest26736-21perl10:45:160
2576899306,10cyclictest25892-21ssh10:44:320
2576899305,9cyclictest28102-21missed_timers09:40:220
2576899305,11cyclictest28813-21cut10:25:180
2576899304,9cyclictest7416-21ntp_192.168.11411:55:250
2576899304,10cyclictest32757-21usage_Anybus11:50:300
2576899303,10cyclictest6313-21acpi08:30:110
2578799295,17cyclictest0-21swapper/210:05:132
2578799292,20cyclictest0-21swapper/209:15:152
2578799292,20cyclictest0-21swapper/208:55:152
2578799292,17cyclictest26346irq/40-dwc2_hso11:30:122
2578799292,16cyclictest13322-21ssh09:30:262
2578799292,11cyclictest23075-21ssh09:37:312
2578799292,11cyclictest1-21systemd09:45:052
2577999292,19cyclictest0-21swapper/111:30:121
2577999290,19cyclictest14725-21ssh11:20:131
2576899299,8cyclictest22650-21ls08:10:210
2576899298,7cyclictest17262-21ssh12:01:570
2576899298,7cyclictest17262-21ssh12:01:560
2576899298,10cyclictest32179-21ps11:10:180
2576899297,9cyclictest25024-21ssh09:16:000
2576899297,8cyclictest18386-21ssh12:23:420
2576899297,7cyclictest8897-21ntp_192.168.11411:35:240
2576899296,9cyclictest21010-21ssh09:58:450
2576899296,9cyclictest15614-21ssh10:58:570
2576899296,8cyclictest17943-21ntp_192.168.11408:05:250
2576899295,8cyclictest12460-21diskmemload10:08:330
2576899294,10cyclictest20832-21date11:45:020
25768992923,3cyclictest27503-21apt-get07:45:120
2578799287,14cyclictest27-21ksoftirqd/211:20:142
2578799284,7cyclictest14529-21ssh09:55:122
2578799282,19cyclictest0-21swapper/208:10:142
2578799282,11cyclictest932-21snmpd10:33:302
25779992811,5cyclictest21-21ksoftirqd/110:15:241
2576899286,8cyclictest25141-21cut11:05:200
2576899285,9cyclictest30875-21df_inode07:15:170
25768992822,3cyclictest30583-21apt-get08:55:140
25768992817,8cyclictest25715-21apt-get08:50:120
2578799276,13cyclictest26346irq/40-dwc2_hso08:05:152
2578799274,14cyclictest26346irq/40-dwc2_hso10:50:132
2578799272,18cyclictest11068-21ssh10:35:152
2577999272,18cyclictest28070-21sh10:25:141
25779992713,4cyclictest21-21ksoftirqd/112:25:201
2576899278,8cyclictest22377-21ssh09:36:170
2576899278,6cyclictest12460-21diskmemload12:10:370
2576899277,7cyclictest12460-21diskmemload11:00:490
2576899276,8cyclictest12460-21diskmemload11:24:340
2576899276,7cyclictest0-21swapper/009:25:010
2576899275,7cyclictest7163-21df_inode09:50:160
2576899274,8cyclictest0-21swapper/012:36:010
25768992721,3cyclictest27406-21apt-get12:30:130
25768992721,3cyclictest1511-21apt-get08:25:140
2576899272,10cyclictest32490-21awk07:50:120
25768992720,4cyclictest22632-21apt-get07:40:130
2580399263,15cyclictest0-21swapper/310:40:313
2580399263,15cyclictest0-21swapper/310:25:133
2580399263,13cyclictest29566irq/58-eth011:20:153
2578799263,17cyclictest0-21swapper/209:10:132
2578799262,9cyclictest0-21swapper/208:25:142
2578799261,11cyclictest27-21ksoftirqd/207:45:122
2577999266,3cyclictest131rcu_preempt07:40:131
2576899265,9cyclictest27702-21ntp_states08:15:260
2576899263,8cyclictest22333-21cat07:35:320
2580399253,9cyclictest0-21swapper/310:05:143
2580399253,16cyclictest0-21swapper/309:53:303
2578799252,16cyclictest26346irq/40-dwc2_hso08:45:142
25787992511,5cyclictest10728-21ntp_192.168.11411:15:262
2577999258,5cyclictest21-21ksoftirqd/107:55:311
2577999258,10cyclictest21-21ksoftirqd/110:35:241
2577999256,9cyclictest21-21ksoftirqd/111:10:011
2577999255,4cyclictest131rcu_preempt07:30:291
2577999253,4cyclictest21-21ksoftirqd/111:45:151
2577999252,5cyclictest21-21ksoftirqd/112:00:181
2577999252,5cyclictest21-21ksoftirqd/112:00:181
25779992513,5cyclictest0-21swapper/111:40:131
2576899255,7cyclictest0-21swapper/007:30:280
2580399242,11cyclictest29566irq/58-eth011:45:133
2578799244,17cyclictest26346irq/40-dwc2_hso07:25:132
2578799243,18cyclictest9579-21kworker/2:010:15:142
2578799243,15cyclictest26346irq/40-dwc2_hso08:20:142
2578799242,12cyclictest26346irq/40-dwc2_hso09:00:152
25787992419,3cyclictest3417-21munin-plugin-st11:55:002
2578799241,19cyclictest26085-21ssh10:22:302
2578799241,19cyclictest26085-21ssh10:22:292
25787992411,10cyclictest6706-21cron11:15:012
2577999247,6cyclictest21-21ksoftirqd/110:55:211
2577999245,5cyclictest21-21ksoftirqd/111:15:271
2577999245,3cyclictest131rcu_preempt10:20:131
2577999245,3cyclictest131rcu_preempt10:20:121
2577999244,10cyclictest21-21ksoftirqd/110:32:381
2577999242,5cyclictest21-21ksoftirqd/107:50:261
25779992414,4cyclictest21-21ksoftirqd/109:45:221
2577999241,2cyclictest131rcu_preempt11:25:311
2576899248,3cyclictest0-21swapper/008:40:250
2576899247,14cyclictest0-21swapper/008:35:160
2576899243,7cyclictest12460-21diskmemload09:29:540
2576899242,8cyclictest19514-21cpuspeed12:05:140
2580399238,9cyclictest29566irq/58-eth009:30:143
25803992314,6cyclictest29566irq/58-eth010:35:143
25803992311,9cyclictest0-21swapper/309:45:213
2580399231,13cyclictest0-21swapper/310:30:163
2578799236,8cyclictest27-21ksoftirqd/209:40:132
2578799231,14cyclictest26346irq/40-dwc2_hso07:35:142
2578799231,13cyclictest26346irq/40-dwc2_hso10:00:132
2577999239,8cyclictest21-21ksoftirqd/112:15:201
2577999239,5cyclictest21-21ksoftirqd/109:41:461
2577999239,5cyclictest131rcu_preempt09:25:151
2577999237,4cyclictest21-21ksoftirqd/110:50:211
2577999237,4cyclictest131rcu_preempt11:00:281
2577999236,8cyclictest21-21ksoftirqd/109:11:441
2577999236,10cyclictest21-21ksoftirqd/110:07:381
2577999235,8cyclictest21-21ksoftirqd/111:52:431
2577999235,7cyclictest21-21ksoftirqd/109:16:071
2577999235,3cyclictest131rcu_preempt10:10:131
2577999233,6cyclictest21-21ksoftirqd/109:35:141
2577999233,3cyclictest131rcu_preempt11:35:251
2577999232,10cyclictest18500-21timerandwakeup09:55:271
25779992317,3cyclictest0-21swapper/108:40:161
25779992314,4cyclictest29072-21apt-get08:20:151
25779992313,4cyclictest21-21ksoftirqd/112:30:161
25768992314,7cyclictest12466-21latency_hist09:10:020
25768992310,10cyclictest0-21swapper/008:20:150
2580399229,8cyclictest29566irq/58-eth010:15:163
25803992212,7cyclictest0-21swapper/311:15:193
25803992211,9cyclictest0-21swapper/312:13:023
2580399221,15cyclictest0-21swapper/310:20:143
2580399221,15cyclictest0-21swapper/310:20:133
2578799228,5cyclictest14744-21kthreadcore11:40:202
2577999228,9cyclictest3226-21apt-get07:20:111
2577999227,7cyclictest21-21ksoftirqd/109:30:411
2577999227,3cyclictest131rcu_preempt08:15:311
2577999224,9cyclictest21-21ksoftirqd/108:05:221
2577999222,5cyclictest21-21ksoftirqd/108:55:281
2577999222,3cyclictest131rcu_preempt11:56:021
2577999222,3cyclictest131rcu_preempt08:00:301
2577999221,15cyclictest12460-21diskmemload12:05:121
2576899223,7cyclictest5590-21kthreadcore07:55:210
2580399211,14cyclictest0-21swapper/310:55:153
2578799218,10cyclictest2200-21grep07:15:302
2578799216,6cyclictest27-21ksoftirqd/207:55:232
2578799215,13cyclictest27-21ksoftirqd/207:10:212
2578799214,6cyclictest27-21ksoftirqd/211:00:312
2578799214,13cyclictest26346irq/40-dwc2_hso07:20:142
2578799213,9cyclictest20746-21kthreadcore12:05:192
2578799213,15cyclictest25945-21grep08:50:152
2578799212,15cyclictest29025-21latency_hist08:20:032
2578799212,13cyclictest0-21swapper/212:20:152
2578799211,13cyclictest26346irq/40-dwc2_hso07:40:142
2577999218,7cyclictest131rcu_preempt07:35:141
2577999217,3cyclictest131rcu_preempt07:15:311
2577999216,4cyclictest21-21ksoftirqd/107:10:161
2577999213,4cyclictest21-21ksoftirqd/108:35:291
2577999213,3cyclictest131rcu_preempt08:45:141
25779992114,5cyclictest0-21swapper/108:50:301
2576899212,7cyclictest0-21swapper/008:45:180
25768992112,6cyclictest8176-21diskstats07:25:150
2578799201,9cyclictest31278-21unixbench_multi09:20:312
25787992012,5cyclictest0-21swapper/209:05:282
2578799201,16cyclictest0-21swapper/212:25:162
2578799201,10cyclictest2165-21fschecks_count09:25:172
2578799201,10cyclictest14396-21latency_hist08:05:012
2577999207,4cyclictest21-21ksoftirqd/107:45:181
2577999205,4cyclictest131rcu_preempt07:25:171
2577999204,4cyclictest21-21ksoftirqd/109:05:231
2577999201,5cyclictest21-21ksoftirqd/108:10:171
2576899209,8cyclictest7616-21sshd09:03:300
2580399199,8cyclictest0-21swapper/311:34:543
2580399198,9cyclictest0-21swapper/311:44:563
2580399198,8cyclictest0-21swapper/309:20:203
2580399194,9cyclictest29566irq/58-eth007:30:143
2580399194,9cyclictest0-21swapper/309:55:133
2580399192,9cyclictest0-21swapper/308:25:143
2580399192,13cyclictest0-21swapper/310:45:303
25803991914,3cyclictest33-21ksoftirqd/312:15:243
25803991912,4cyclictest29566irq/58-eth010:50:253
25803991911,5cyclictest29566irq/58-eth012:05:133
2580399191,12cyclictest29566irq/58-eth012:20:153
25803991910,6cyclictest0-21swapper/311:06:443
2578799193,8cyclictest12891-21kthreadcore08:35:212
2578799192,14cyclictest16215-21fschecks_time08:40:192
2578799191,14cyclictest6434-21cut08:30:162
2577999193,7cyclictest21-21ksoftirqd/108:30:171
2577999192,9cyclictest21-21ksoftirqd/109:00:221
2577999192,3cyclictest131rcu_preempt08:25:121
2580399189,6cyclictest0-21swapper/311:52:033
2580399189,6cyclictest0-21swapper/311:27:003
2580399189,6cyclictest0-21swapper/311:00:333
2580399188,6cyclictest0-21swapper/309:41:303
2580399183,10cyclictest0-21swapper/309:00:153
2580399182,11cyclictest0-21swapper/308:20:153
2580399182,10cyclictest0-21swapper/310:10:133
2580399182,10cyclictest0-21swapper/308:50:133
2580399182,10cyclictest0-21swapper/307:50:043
2580399182,10cyclictest0-21swapper/307:40:153
2580399182,10cyclictest0-21swapper/307:35:153
2580399182,10cyclictest0-21swapper/307:15:173
2580399181,13cyclictest0-21swapper/312:35:143
2580399181,13cyclictest0-21swapper/311:55:283
2580399181,13cyclictest0-21swapper/308:55:283
2580399181,12cyclictest0-21swapper/307:15:023
2580399181,10cyclictest29566irq/58-eth011:10:133
25803991810,6cyclictest0-21swapper/312:00:183
25803991810,6cyclictest0-21swapper/312:00:173
2578799182,8cyclictest25833-21latency_hist07:10:012
2576899189,7cyclictest25816-21latency_hist07:10:010
2580399178,6cyclictest0-21swapper/309:25:353
2580399172,10cyclictest0-21swapper/308:35:163
2580399171,9cyclictest0-21swapper/308:30:113
2580399171,12cyclictest0-21swapper/308:10:153
2580399162,9cyclictest0-21swapper/308:00:163
2580399162,9cyclictest0-21swapper/307:25:153
2580399162,10cyclictest0-21swapper/309:15:153
2580399158,5cyclictest0-21swapper/308:41:573
2580399157,5cyclictest0-21swapper/308:15:233
2580399151,8cyclictest0-21swapper/307:45:223
2577999150,6cyclictest25789-21latency_hist07:10:021
2580399132,8cyclictest0-21swapper/309:10:013
2580399131,6cyclictest0-21swapper/308:50:023
2580399121,4cyclictest0-21swapper/308:05:303
2580399111,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional