You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-14 - 00:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sat Jul 13, 2024 12:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32296996616,41cyclictest0-21swapper/211:20:112
3228899581,5cyclictest131rcu_preempt08:25:101
3227899544,39cyclictest11967-21ssh12:15:130
3227899544,39cyclictest11967-21ssh12:15:130
3229699526,21cyclictest13433-21ssh11:40:132
3227899513,7cyclictest131rcu_preempt11:45:110
3228899476,5cyclictest21-21ksoftirqd/111:35:151
32296994513,23cyclictest0-21swapper/209:15:122
3230499443,27cyclictest25866irq/58-eth009:10:153
32288994413,7cyclictest21-21ksoftirqd/112:05:111
32296994228,10cyclictest21067-21apt-get08:05:152
32296994124,12cyclictest0-21swapper/212:00:122
3228899412,28cyclictest12284-21ssh09:25:131
32296994029,8cyclictest18729-21apt-get11:25:142
3227899405,26cyclictest8236-21ssh11:55:130
3230499398,21cyclictest25866irq/58-eth011:35:143
32304993918,13cyclictest25866irq/58-eth010:20:133
32296993916,17cyclictest27792-21ssh11:30:152
32296993916,17cyclictest27792-21ssh11:30:152
3227899383,30cyclictest30100-21ssh12:25:130
32296993715,18cyclictest0-21swapper/211:00:122
32296993715,17cyclictest0-21swapper/212:20:142
32296993715,17cyclictest0-21swapper/212:20:132
32296993715,16cyclictest0-21swapper/212:35:132
3227899373,30cyclictest19800-21ssh09:10:150
32278993726,4cyclictest131rcu_preempt11:50:110
3230499363,25cyclictest25866irq/58-eth010:50:123
3229699368,17cyclictest7969-21ntp_kernel_pll_09:40:322
3229699362,28cyclictest10486-21sh12:13:432
32296993618,9cyclictest6700-21ssh12:30:132
32296993612,5cyclictest19389-21diskmemload11:15:132
32296993610,7cyclictest19389-21diskmemload10:36:012
3228899361,3cyclictest0-21swapper/107:30:111
32304993522,9cyclictest25866irq/58-eth009:45:143
32296993529,3cyclictest16203-21apt-get08:00:132
32296993529,3cyclictest16203-21apt-get08:00:132
32296993523,8cyclictest22146irq/40-dwc2_hso08:50:142
32296993523,3cyclictest4377-21apt-get11:35:132
3230499343,18cyclictest25866irq/58-eth011:10:153
3229699349,10cyclictest6792-21ssh11:53:452
3229699346,14cyclictest22505-21rm11:08:072
32296993429,3cyclictest14666-21apt-get07:25:162
32296993426,4cyclictest24333-21apt-get07:35:122
32296993424,7cyclictest6143-21apt-get10:40:152
32296993423,7cyclictest22146irq/40-dwc2_hso09:35:152
32296993421,10cyclictest22702-21apt-get10:50:132
3230499331,16cyclictest25866irq/58-eth012:20:133
3230499331,16cyclictest25866irq/58-eth012:20:133
32296993327,3cyclictest22462-21apt-get11:45:112
32296993311,17cyclictest0-21swapper/210:45:122
32296993311,10cyclictest19389-21diskmemload10:26:532
32296993310,8cyclictest25426-21ssh09:11:392
32304993216,11cyclictest25866irq/58-eth011:25:123
32304993214,10cyclictest25866irq/58-eth010:55:123
3229699327,17cyclictest19389-21diskmemload09:52:422
3229699326,17cyclictest5450-21ssh10:20:112
32296993220,8cyclictest9802-21apt-get07:20:142
32296993220,3cyclictest13005-21apt-get08:30:122
32296993218,8cyclictest12272-21apt-get09:25:142
32296993210,10cyclictest799-21sed09:55:372
3229699313,23cyclictest0-21swapper/209:45:312
32296993115,13cyclictest22222-21tail10:10:212
32296993111,14cyclictest0-21swapper/210:30:112
32296993110,7cyclictest19389-21diskmemload12:07:032
3229699305,7cyclictest19389-21diskmemload11:56:592
3229699304,16cyclictest22146irq/40-dwc2_hso09:00:112
3229699304,14cyclictest30698-21sh12:25:212
32296993023,3cyclictest13390-21apt-get10:05:102
32296993023,3cyclictest13390-21apt-get10:05:102
32296993013,12cyclictest0-21swapper/209:30:112
3227899302,22cyclictest12521-21ssh09:45:130
3229699297,18cyclictest11172-21kthreadcore08:25:242
3229699292,20cyclictest0-21swapper/208:45:042
32296992921,6cyclictest22146irq/40-dwc2_hso08:35:152
3229699292,11cyclictest0-21swapper/208:10:292
32296992919,7cyclictest19474-21apt-get07:30:112
32296992915,5cyclictest8638-21sh10:00:302
32296992913,13cyclictest4555-21tail09:20:222
32296992912,13cyclictest32530-21tune2fs07:10:182
32296992912,13cyclictest19412-21latency_hist09:10:012
32296992912,13cyclictest19412-21latency_hist09:10:012
3228899292,21cyclictest22479-21ssh11:45:121
3228899291,19cyclictest21-21ksoftirqd/110:50:121
3227899292,19cyclictest4407-21ssh11:35:150
3229699287,14cyclictest0-21swapper/210:55:122
3229699287,13cyclictest30759-21kthreadcore10:15:222
3229699284,16cyclictest22146irq/40-dwc2_hso07:15:132
3229699283,16cyclictest22146irq/40-dwc2_hso07:55:122
3229699282,9cyclictest900-21snmpd07:40:112
32296992814,10cyclictest0-21swapper/208:40:122
32296992813,12cyclictest5094-21tr07:45:312
32296992813,12cyclictest11911-21/usr/sbin/munin12:15:212
32296992813,12cyclictest11911-21/usr/sbin/munin12:15:212
32296992810,14cyclictest6759-21ps08:20:362
3229699277,17cyclictest25376-21kthreadcore11:10:252
3229699273,16cyclictest22146irq/40-dwc2_hso08:15:132
32296992712,12cyclictest5711-21/usr/sbin/munin08:55:242
32304992613,9cyclictest25866irq/58-eth011:45:123
3228899265,10cyclictest21-21ksoftirqd/107:45:141
3230499253,18cyclictest25866irq/58-eth012:25:133
3230499251,15cyclictest0-21swapper/312:13:433
3229699259,8cyclictest6671-21grep07:50:172
3229699259,8cyclictest6671-21grep07:50:172
3228899252,2cyclictest131rcu_preempt12:15:121
3228899252,2cyclictest131rcu_preempt12:15:121
3227899252,19cyclictest15731-21ssh12:35:150
3227899252,18cyclictest26328-21ssh12:05:130
32278992511,5cyclictest12-21ksoftirqd/010:05:100
32278992511,5cyclictest12-21ksoftirqd/010:05:090
3230499242,13cyclictest25866irq/58-eth009:25:133
3230499241,13cyclictest25866irq/58-eth010:05:103
3230499241,13cyclictest25866irq/58-eth010:05:103
3228899243,14cyclictest21-21ksoftirqd/110:35:111
3228899242,18cyclictest22814-21apt-get08:40:111
3228899242,17cyclictest18760-21ssh11:25:141
32288992415,6cyclictest21-21ksoftirqd/111:50:121
3230499239,10cyclictest25866irq/58-eth009:15:323
3228899239,10cyclictest6877-21kthreadcore09:40:251
3228899232,16cyclictest6154-21ssh10:40:141
3228899231,7cyclictest0-21swapper/110:18:231
32288992314,3cyclictest19389-21diskmemload11:10:171
3228899231,17cyclictest21037-21ssh12:20:141
3228899231,17cyclictest21037-21ssh12:20:141
32288992310,9cyclictest21-21ksoftirqd/112:35:151
3228899230,3cyclictest131rcu_preempt12:10:131
32278992313,7cyclictest0-21swapper/010:35:160
3230499221,14cyclictest0-21swapper/312:15:133
3230499221,14cyclictest0-21swapper/312:15:123
3228899229,10cyclictest23153-21kthreadcore10:10:231
3228899224,4cyclictest24083-21kworker/1:107:35:241
3228899222,3cyclictest131rcu_preempt10:50:011
3228899221,18cyclictest9677-21ssh11:20:111
3228899221,17cyclictest4850-21apt-get07:15:121
3228899221,15cyclictest16242-21ssh09:45:321
3227899222,15cyclictest19389-21diskmemload11:02:420
3227899221,7cyclictest31715-21ntp_192.168.11409:15:290
3227899221,18cyclictest3384-21cut08:20:160
3227899221,18cyclictest27919-21diskstats11:30:190
3227899221,18cyclictest27919-21diskstats11:30:190
3227899220,17cyclictest22292-21ls11:45:020
32278992113,5cyclictest14040-21hwlatdetect10:25:180
32278992112,7cyclictest11082-21sshd10:40:420
32278992112,6cyclictest1-21systemd08:55:210
32278992111,7cyclictest4073-21sshd10:18:420
3230499208,7cyclictest25866irq/58-eth011:30:163
3230499208,7cyclictest25866irq/58-eth011:30:163
3230499203,12cyclictest0-21swapper/309:40:313
32304992011,7cyclictest0-21swapper/311:51:013
3228899208,3cyclictest131rcu_preempt12:00:401
3228899208,10cyclictest8043-21kthreadcore10:20:241
3228899207,10cyclictest21442-21cron10:10:011
3228899207,10cyclictest21442-21cron10:10:011
3228899205,11cyclictest0-21swapper/111:55:131
3228899203,3cyclictest131rcu_preempt10:30:011
3228899201,3cyclictest0-21swapper/111:06:301
32288992012,6cyclictest11631-21cat07:55:201
32288992012,5cyclictest0-21swapper/109:20:121
3228899201,15cyclictest10077-21uniq09:00:161
3228899201,14cyclictest8344-21latency08:55:281
3228899201,14cyclictest29199-21hwlatdetect09:55:191
3228899201,13cyclictest3747-21ls07:10:391
3227899209,3cyclictest131rcu_preempt12:20:340
3227899209,3cyclictest131rcu_preempt12:20:340
3227899205,4cyclictest131rcu_preempt12:30:120
3227899202,15cyclictest2196-21ssh10:55:330
3227899202,14cyclictest22215-21/usr/sbin/munin10:10:210
32278992012,6cyclictest25479-21sshd11:27:420
32278992012,6cyclictest17266-21sshd09:25:420
32278992012,5cyclictest29687-21tail09:55:220
32278992011,5cyclictest0-21swapper/009:05:270
32278992011,5cyclictest0-21swapper/009:05:260
32278992010,7cyclictest32087-21ntpq09:35:320
3227899200,14cyclictest22710-21ssh10:50:130
3230499199,7cyclictest0-21swapper/309:35:153
3230499199,6cyclictest25866irq/58-eth010:00:123
3230499195,6cyclictest0-21swapper/312:30:133
3230499193,11cyclictest0-21swapper/312:08:413
3230499193,11cyclictest0-21swapper/311:56:023
3230499191,12cyclictest0-21swapper/308:05:383
32304991910,6cyclictest0-21swapper/311:15:593
32304991910,6cyclictest0-21swapper/310:36:123
3229699199,7cyclictest32322-21latency_hist07:10:012
3228899196,3cyclictest19389-21diskmemload09:16:431
3228899193,9cyclictest0-21swapper/109:10:151
3228899193,4cyclictest21-21ksoftirqd/110:59:531
3228899191,6cyclictest3131-21latency_hist08:20:021
3228899191,3cyclictest131rcu_preempt11:30:271
3228899191,3cyclictest131rcu_preempt11:30:271
3228899191,3cyclictest0-21swapper/111:15:251
3228899191,15cyclictest28338-21ssh09:35:141
3228899191,13cyclictest10124-21grep07:20:211
3228899191,12cyclictest19389-21diskmemload12:31:471
3228899191,10cyclictest18360-21irqrtprio08:35:231
32288991910,6cyclictest0-21swapper/109:50:331
3228899190,7cyclictest4889-21/usr/sbin/munin10:00:111
3228899190,4cyclictest131rcu_preempt12:25:141
3228899190,3cyclictest131rcu_preempt11:40:131
3227899197,8cyclictest12-21ksoftirqd/010:20:110
3227899195,12cyclictest22049-21ssh10:30:100
3227899193,10cyclictest9662-21apt-get11:20:110
3227899192,11cyclictest30229-21ssh11:11:430
3227899191,9cyclictest9562-21cat07:20:020
3227899191,3cyclictest131rcu_preempt09:52:420
32278991912,4cyclictest0-21swapper/008:00:380
32278991912,4cyclictest0-21swapper/008:00:370
32278991911,6cyclictest7727-21sshd09:20:280
32278991911,6cyclictest3520-21awk07:10:360
3227899191,14cyclictest27938-21ssh09:35:010
32278991910,6cyclictest6248-21cron07:50:010
3230499189,7cyclictest0-21swapper/309:35:003
3230499189,6cyclictest25866irq/58-eth010:10:143
3230499189,6cyclictest0-21swapper/311:20:163
3230499183,11cyclictest0-21swapper/310:26:423
3230499183,10cyclictest25966irq/59-eth010:40:133
3230499182,11cyclictest0-21swapper/311:00:413
3230499182,10cyclictest0-21swapper/308:45:043
3230499182,10cyclictest0-21swapper/307:45:153
3230499182,10cyclictest0-21swapper/307:20:143
32304991812,4cyclictest0-21swapper/309:20:143
32304991811,4cyclictest0-21swapper/312:35:323
32304991810,6cyclictest0-21swapper/310:16:343
3228899181,14cyclictest19070-21kthreadcore08:00:261
3228899181,14cyclictest19070-21kthreadcore08:00:261
3228899181,13cyclictest20426-21df_inode09:30:151
3228899181,11cyclictest22649-21ps10:30:191
32288991810,5cyclictest0-21swapper/111:00:151
3227899188,7cyclictest14550-21df10:45:140
3227899187,8cyclictest5718-21cut09:40:230
3227899185,10cyclictest17626-21cron08:35:010
3227899184,11cyclictest31971-21kthreadcore07:40:250
3227899182,13cyclictest21221-21fschecks_time08:05:190
3227899181,5cyclictest0-21swapper/010:00:370
32278991812,4cyclictest6735-21sh12:10:340
32278991811,5cyclictest0-21swapper/008:10:260
3227899181,14cyclictest736-21apt-get11:15:120
3227899181,14cyclictest17427-21diskstats12:00:170
32278991810,6cyclictest3204-21cut08:50:250
32278991810,4cyclictest18777-21kthreadcore11:05:290
3230499179,6cyclictest0-21swapper/312:03:253
3230499179,6cyclictest0-21swapper/311:05:383
3230499179,6cyclictest0-21swapper/310:45:253
3230499179,6cyclictest0-21swapper/309:50:173
3230499178,6cyclictest0-21swapper/310:31:093
3230499172,9cyclictest0-21swapper/309:05:123
3230499172,9cyclictest0-21swapper/309:05:123
3230499172,10cyclictest0-21swapper/308:50:143
3230499172,10cyclictest0-21swapper/307:10:153
32304991712,3cyclictest25966irq/59-eth011:40:213
3230499171,11cyclictest0-21swapper/309:55:363
3228899179,6cyclictest4108-21grep08:50:361
3228899171,6cyclictest10903-21cat07:50:401
3228899171,6cyclictest10903-21cat07:50:401
3228899171,4cyclictest0-21swapper/108:05:271
32288991711,4cyclictest0-21swapper/108:45:201
32288991711,4cyclictest0-21swapper/107:25:201
3227899178,6cyclictest22415-21cat08:35:420
3227899173,10cyclictest0-21swapper/007:35:120
3227899172,13cyclictest20513-21tr07:30:220
3227899171,4cyclictest0-21swapper/007:58:440
32278991711,4cyclictest0-21swapper/008:40:160
32278991711,4cyclictest0-21swapper/008:15:110
3227899171,13cyclictest9896-21diskstats07:20:180
3227899171,13cyclictest27447-21apt-get08:45:040
3227899171,13cyclictest11600-21ntp_192.168.11408:25:270
32278991710,5cyclictest13355-21nfs_client09:00:260
3230499161,12cyclictest0-21swapper/308:10:263
3230499161,12cyclictest0-21swapper/307:35:163
3230499161,12cyclictest0-21swapper/307:30:173
3230499161,10cyclictest0-21swapper/307:15:353
3228899167,6cyclictest3422-21grep08:20:171
3228899162,11cyclictest12768-21latency_hist08:30:001
3228899161,12cyclictest30424-21kthreadcore07:40:231
3228899161,12cyclictest26397-21sed08:10:211
3228899161,11cyclictest14931-21fschecks_time09:05:151
3228899161,11cyclictest14931-21fschecks_time09:05:151
3227899164,9cyclictest16445-21kthreadcore07:25:270
3227899162,11cyclictest8662-21kthreadcore07:50:270
3227899162,11cyclictest8662-21kthreadcore07:50:270
3230499151,9cyclictest0-21swapper/308:55:283
3230499151,9cyclictest0-21swapper/307:50:173
3230499151,9cyclictest0-21swapper/307:50:173
3230499149,3cyclictest0-21swapper/307:55:233
3230499142,7cyclictest0-21swapper/308:00:113
3230499142,7cyclictest0-21swapper/308:00:103
3230499141,11cyclictest0-21swapper/308:20:133
3230499141,11cyclictest0-21swapper/307:40:353
3227899141,9cyclictest32314-21grep07:10:000
3230499138,3cyclictest0-21swapper/307:25:353
3230499135,6cyclictest0-21swapper/308:17:453
3230499131,10cyclictest0-21swapper/308:25:213
3228899131,9cyclictest32294-21latency_hist07:10:011
3230499122,4cyclictest0-21swapper/308:30:133
3230499121,8cyclictest0-21swapper/308:35:143
3230499116,3cyclictest33-21ksoftirqd/309:00:223
3230499111,7cyclictest0-21swapper/308:40:213
323049981,5cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional