You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-07-06 - 17:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Celeron 33455 @1500 MHz, Linux 5.15.2-rt19 (Profile)

Latency plot of shadow in rack #c, slot #7
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Wed Jul 06, 2022 12:44:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
45828322050,8sleep013-21ksoftirqd/007:05:080
45868621060,18sleep342-21ksoftirqd/307:05:153
53944121050,5sleep30-21swapper/310:50:243
5358202920,4chrt535207-21/usr/sbin/munin10:40:223
5358202920,4chrt535207-21/usr/sbin/munin10:40:223
5483802910,6sleep10-21swapper/111:15:211
45922628771,9sleep20-21swapper/207:05:282
459961998112,41cyclictest550140-21missed_timers11:20:230
45996199792,42cyclictest540412-21/usr/sbin/munin10:55:100
459961997719,30cyclictest1069-21runrttasks07:54:090
45955927762,9sleep10-21swapper/107:06:501
459961997610,40cyclictest0-21swapper/011:45:020
162399760,8rtkit-daemon869-21irqbalance07:53:053
45996199752,52cyclictest1753-21gnome-shell09:04:100
45996199752,52cyclictest1753-21gnome-shell09:04:090
45996199752,49cyclictest1753-21gnome-shell09:17:550
459961997518,35cyclictest488043-21ls08:25:130
459961997516,38cyclictest1069-21runrttasks07:42:070
459961997516,38cyclictest1069-21runrttasks07:42:060
459961997412,37cyclictest574262-21find12:30:110
45996199722,42cyclictest561122-21taskset11:50:490
459961997211,35cyclictest489686-21taskset08:30:120
45996199712,48cyclictest1753-21gnome-shell09:53:220
459961997111,33cyclictest481720-21pkexec08:06:280
45996199702,40cyclictest1753-21gnome-shell09:55:180
45996199702,40cyclictest1753-21gnome-shell08:46:040
459961997010,41cyclictest1069-21runrttasks10:41:540
459961997010,41cyclictest1069-21runrttasks10:41:540
45996199692,48cyclictest1753-21gnome-shell09:31:360
45996199692,39cyclictest1753-21gnome-shell10:39:360
45996199691,34cyclictest568533-40apt-check12:12:220
162399690,6rtkit-daemon906-21gmain08:36:193
162399690,6rtkit-daemon906-21gmain08:36:193
162399690,4rtkit-daemon0-21swapper/310:09:223
162399690,4rtkit-daemon0-21swapper/308:42:443
45996199682,39cyclictest1753-21gnome-shell11:17:070
45996199682,39cyclictest1753-21gnome-shell11:11:090
45996199682,39cyclictest1753-21gnome-shell10:05:100
45996199682,39cyclictest1753-21gnome-shell08:02:450
45996199682,39cyclictest1753-21gnome-shell07:35:120
45996199682,39cyclictest1753-21gnome-shell07:27:590
45996199682,38cyclictest1753-21gnome-shell07:24:220
45996199682,38cyclictest1753-21gnome-shell07:24:210
45996199682,38cyclictest1753-21gnome-shell07:10:340
459961996823,39cyclictest13-21ksoftirqd/012:22:490
45996199681,40cyclictest529065-21awk10:20:260
45996199681,38cyclictest540369-21cat10:55:020
162399680,4rtkit-daemon0-21swapper/307:34:313
162399680,4rtkit-daemon0-21swapper/307:28:153
45996699678,11cyclictest563791-21expr12:00:152
45996199672,39cyclictest1753-21gnome-shell10:45:390
45996199672,38cyclictest486016-21appstreamcli08:20:040
45996199672,38cyclictest1753-21gnome-shell10:03:470
45996199672,38cyclictest1753-21gnome-shell08:16:070
459961996718,27cyclictest1069-21runrttasks07:58:400
45996199671,40cyclictest491499-21/usr/sbin/munin08:35:130
45996199671,40cyclictest491499-21/usr/sbin/munin08:35:120
4714152660,5sleep1251rcuc/107:40:141
4714152660,5sleep1251rcuc/107:40:131
45996199662,37cyclictest1753-21gnome-shell10:25:350
45996199662,32cyclictest1753-21gnome-shell12:35:120
45996199661,39cyclictest516902-21taskset09:46:510
45996199661,38cyclictest552613-21seq11:28:320
45996199661,38cyclictest468769-21seq07:31:510
5321712650,7chrt531743-21/usr/sbin/munin10:30:203
45996199652,37cyclictest1753-21gnome-shell09:35:160
45996199651,39cyclictest1069-21runrttasks08:40:570
45996199651,38cyclictest508434-21taskset09:24:270
45996199651,38cyclictest504871-21seq09:13:130
45996199651,36cyclictest564686-21seq12:02:090
162399650,4rtkit-daemon0-21swapper/308:02:063
45996699646,13cyclictest524803-21/usr/sbin/munin10:10:152
459966996420,20cyclictest1753-21gnome-shell12:22:162
45996199642,34cyclictest510181-21taskset09:29:390
45996199642,34cyclictest510181-21taskset09:29:380
45996199642,31cyclictest0-21swapper/008:50:190
45996199641,37cyclictest558695-21idleruntime11:45:190
45996199641,37cyclictest543868-21seq11:02:240
45996699636,19cyclictest1753-21gnome-shell08:32:202
45996199631,37cyclictest533206-21seq10:34:510
45996199622,37cyclictest498557-21/usr/sbin/munin08:55:140
45996199621,37cyclictest463640-21taskset07:17:440
162399620,8rtkit-daemon1622-21rtkit-daemon07:55:073
45996699616,28cyclictest1753-21gnome-shell10:15:152
45996699616,28cyclictest1753-21gnome-shell10:15:152
45996199611,36cyclictest570190-21taskset12:15:390
45996199611,36cyclictest502967-21taskset09:06:060
45996199611,36cyclictest1069-21runrttasks10:19:430
45996199611,36cyclictest1069-21runrttasks10:19:430
45996199611,34cyclictest513933-21/usr/sbin/munin09:40:120
162399610,9rtkit-daemon1622-21rtkit-daemon10:30:572
162399610,9rtkit-daemon1622-21rtkit-daemon07:33:572
162399610,10rtkit-daemon1622-21rtkit-daemon09:55:362
4740012600,23sleep234-21ksoftirqd/207:45:262
45996699608,25cyclictest1753-21gnome-shell12:10:232
45996199602,35cyclictest1753-21gnome-shell12:25:270
45996199601,28cyclictest545949-21latency_hist11:10:020
45996699599,28cyclictest1753-21gnome-shell08:05:002
45996699597,30cyclictest1753-21gnome-shell09:15:292
45996199592,33cyclictest561705-21/usr/sbin/munin11:55:180
45996199592,33cyclictest561705-21/usr/sbin/munin11:55:180
45996199592,28cyclictest921-21thermald11:36:250
45996199591,33cyclictest474822-21sed07:50:010
162399590,9rtkit-daemon1622-21rtkit-daemon08:05:512
45996699589,29cyclictest1753-21gnome-shell11:56:022
45996699589,29cyclictest1753-21gnome-shell11:56:022
45996699588,27cyclictest1753-21gnome-shell07:42:252
45996699588,27cyclictest1753-21gnome-shell07:42:252
45996699587,29cyclictest1753-21gnome-shell12:25:102
45996699587,25cyclictest1753-21gnome-shell11:06:312
45996199581,30cyclictest552992-21/usr/sbin/munin11:30:330
45996199581,30cyclictest552992-21/usr/sbin/munin11:30:330
162399580,9rtkit-daemon1622-21rtkit-daemon10:07:592
162399580,8rtkit-daemon1622-21rtkit-daemon11:30:322
162399580,8rtkit-daemon1622-21rtkit-daemon11:30:322
162399580,8rtkit-daemon1622-21rtkit-daemon10:22:542
162399580,8rtkit-daemon1622-21rtkit-daemon07:14:462
4605242570,23sleep342-21ksoftirqd/307:10:173
45996699579,28cyclictest1753-21gnome-shell11:36:192
45996699578,29cyclictest1753-21gnome-shell08:39:342
45996699578,29cyclictest1753-21gnome-shell08:39:342
45996699577,29cyclictest1753-21gnome-shell11:46:352
45996699575,30cyclictest1753-21gnome-shell07:39:062
459966995710,22cyclictest1753-21gnome-shell09:10:292
45996699568,29cyclictest1753-21gnome-shell08:11:552
45996699567,28cyclictest1753-21gnome-shell12:15:202
45996699567,17cyclictest1753-21gnome-shell08:20:152
459963995614,18cyclictest26-21ksoftirqd/108:05:311
162399560,9rtkit-daemon1622-21rtkit-daemon07:27:162
4971442550,6sleep10-21swapper/108:50:201
45996699559,20cyclictest1753-21gnome-shell11:52:552
45996699558,28cyclictest1753-21gnome-shell07:54:372
45996699558,23cyclictest1753-21gnome-shell11:00:142
45996699558,20cyclictest438-20systemd-journal12:32:352
45996699557,30cyclictest1069-21runrttasks09:26:002
45996699557,30cyclictest1069-21runrttasks09:26:002
45996699556,30cyclictest1753-21gnome-shell12:37:012
45996699556,29cyclictest1753-21gnome-shell11:29:012
45996699556,28cyclictest1753-21gnome-shell09:05:292
45996699556,14cyclictest855-21dbus-daemon08:41:042
45996699556,11cyclictest1753-21gnome-shell12:07:402
45996699555,29cyclictest1753-21gnome-shell09:46:552
45996699553,28cyclictest1753-21gnome-shell11:23:002
459966995517,18cyclictest0-21swapper/209:50:102
459963995522,15cyclictest26-21ksoftirqd/110:45:321
45996199552,30cyclictest1069-21runrttasks10:14:020
45996199551,32cyclictest1069-21runrttasks12:08:290
162399550,8rtkit-daemon1622-21rtkit-daemon09:58:593
45996699549,19cyclictest1753-21gnome-shell08:58:092
45996699548,23cyclictest1753-21gnome-shell07:18:372
45996699547,27cyclictest1753-21gnome-shell10:52:182
45996699546,29cyclictest1753-21gnome-shell10:36:482
45996699546,28cyclictest1753-21gnome-shell09:21:132
45996699546,28cyclictest1753-21gnome-shell08:28:092
45996699545,28cyclictest1753-21gnome-shell07:21:172
45996699545,28cyclictest1753-21gnome-shell07:21:172
45996699543,27cyclictest1753-21gnome-shell11:43:342
459963995416,18cyclictest26-21ksoftirqd/111:10:131
459963995413,18cyclictest26-21ksoftirqd/107:10:291
459963995410,20cyclictest26-21ksoftirqd/110:15:001
162399540,8rtkit-daemon1622-21rtkit-daemon07:56:532
45996699538,26cyclictest1753-21gnome-shell10:57:082
45996699537,26cyclictest1753-21gnome-shell11:14:332
45996699537,17cyclictest1753-21gnome-shell08:45:172
45996699535,28cyclictest1753-21gnome-shell10:00:082
45996699535,28cyclictest1753-21gnome-shell09:37:212
45996399538,26cyclictest26-21ksoftirqd/112:15:301
45996399535,19cyclictest26-21ksoftirqd/108:25:211
459963995327,8cyclictest26-21ksoftirqd/109:51:571
459963995317,21cyclictest552140-21gzip11:25:261
459963995313,22cyclictest566642-21cron12:10:001
459963995310,21cyclictest26-21ksoftirqd/111:00:311
459963995310,13cyclictest26-21ksoftirqd/107:30:151
45996699529,16cyclictest1753-21gnome-shell11:15:162
45996699527,26cyclictest1753-21gnome-shell08:18:362
45996699527,22cyclictest1753-21gnome-shell09:03:412
45996699527,22cyclictest1753-21gnome-shell09:03:412
45996699524,29cyclictest1753-21gnome-shell10:46:282
459963995242,7cyclictest251rcuc/108:55:141
459963995222,15cyclictest26-21ksoftirqd/110:25:371
45997099510,48cyclictest1622-21rtkit-daemon08:07:553
45996699517,17cyclictest1753-21gnome-shell10:25:112
45996699515,27cyclictest1753-21gnome-shell10:42:142
45996699515,27cyclictest1753-21gnome-shell10:42:142
45996699512,28cyclictest1753-21gnome-shell08:50:182
45996699511,26cyclictest1753-21gnome-shell09:30:202
459966995110,18cyclictest1753-21gnome-shell09:43:422
459963995122,15cyclictest26-21ksoftirqd/110:09:461
459963995119,9cyclictest26-21ksoftirqd/110:55:021
459963995116,12cyclictest26-21ksoftirqd/112:10:201
45996399511,13cyclictest26-21ksoftirqd/107:25:231
459970995021,10cyclictest42-21ksoftirqd/311:15:013
459963995039,7cyclictest251rcuc/108:40:241
45996399503,19cyclictest26-21ksoftirqd/109:00:171
45996399503,19cyclictest26-21ksoftirqd/109:00:171
459963995018,10cyclictest26-21ksoftirqd/111:20:151
459963995011,9cyclictest26-21ksoftirqd/111:10:011
459970994912,19cyclictest42-21ksoftirqd/311:00:193
45996399499,13cyclictest26-21ksoftirqd/109:05:191
45996399496,10cyclictest26-21ksoftirqd/109:35:001
459963994938,7cyclictest251rcuc/109:15:161
459963994928,5cyclictest141rcu_preempt09:36:291
459963994927,6cyclictest141rcu_preempt12:27:551
459963994927,6cyclictest141rcu_preempt07:57:051
459963994917,9cyclictest26-21ksoftirqd/110:20:261
459963994910,14cyclictest491550-21grep08:35:131
459963994910,14cyclictest491550-21grep08:35:131
162399490,38rtkit-daemon1622-21rtkit-daemon11:29:063
459970994828,16cyclictest1753-21gnome-shell12:28:433
45997099480,44cyclictest1622-21rtkit-daemon08:34:233
459963994839,7cyclictest251rcuc/108:21:231
459963994838,7cyclictest251rcuc/109:49:411
459963994838,6cyclictest251rcuc/111:40:121
459963994828,5cyclictest141rcu_preempt08:12:111
459963994823,8cyclictest26-21ksoftirqd/110:39:311
459963994813,19cyclictest26-21ksoftirqd/109:40:201
459963994810,20cyclictest26-21ksoftirqd/112:40:001
459970994718,14cyclictest141rcu_preempt09:27:553
459970994718,14cyclictest141rcu_preempt09:27:553
459970994717,14cyclictest42-21ksoftirqd/308:10:213
45996399476,19cyclictest26-21ksoftirqd/111:00:001
45996399476,11cyclictest26-21ksoftirqd/112:00:261
45996399475,27cyclictest519544-40apt-check09:55:281
459963994727,5cyclictest141rcu_preempt07:21:341
459963994727,5cyclictest141rcu_preempt07:21:341
459963994713,18cyclictest26-21ksoftirqd/108:48:141
459963994710,20cyclictest26-21ksoftirqd/110:45:001
459963994710,20cyclictest26-21ksoftirqd/110:45:001
459963994710,20cyclictest26-21ksoftirqd/107:35:131
162399470,38rtkit-daemon0-21swapper/309:48:153
45997099466,17cyclictest42-21ksoftirqd/312:10:183
459970994615,9cyclictest42-21ksoftirqd/307:45:243
45996399465,13cyclictest26-21ksoftirqd/111:45:131
459963994617,7cyclictest26-21ksoftirqd/108:00:211
45996399461,12cyclictest462547-21cpuspeed_turbos07:15:261
45996199469,32cyclictest1753-21gnome-shell08:10:070
5276552450,7sleep30-21swapper/310:15:303
5276552450,7sleep30-21swapper/310:15:303
459970994538,5cyclictest0-21swapper/309:00:033
459970994538,5cyclictest0-21swapper/309:00:033
459970994522,16cyclictest855-21dbus-daemon09:06:453
459970994518,23cyclictest42-21ksoftirqd/311:56:183
459970994518,23cyclictest42-21ksoftirqd/311:56:183
459970994514,9cyclictest42-21ksoftirqd/308:20:003
45997099451,21cyclictest961-21snmpd08:51:423
45997099451,21cyclictest540818-21/usr/sbin/munin10:55:183
45996399458,20cyclictest26-21ksoftirqd/110:15:251
45996399458,20cyclictest26-21ksoftirqd/110:15:251
45996399458,19cyclictest26-21ksoftirqd/111:35:021
45996399458,12cyclictest26-21ksoftirqd/112:34:031
45996399457,11cyclictest26-21ksoftirqd/112:00:011
45996399457,11cyclictest26-21ksoftirqd/112:00:011
459963994514,5cyclictest141rcu_preempt11:53:191
459963994514,10cyclictest26-21ksoftirqd/109:25:211
459963994514,10cyclictest26-21ksoftirqd/109:25:211
45997099449,17cyclictest42-21ksoftirqd/309:30:303
45997099444,17cyclictest42-21ksoftirqd/307:35:113
459970994433,6cyclictest42-21ksoftirqd/308:55:353
459970994414,6cyclictest141rcu_preempt11:25:013
459963994434,7cyclictest251rcuc/110:03:501
459963994419,8cyclictest26-21ksoftirqd/107:45:191
459963994417,9cyclictest26-21ksoftirqd/108:35:021
45996399441,28cyclictest26-21ksoftirqd/108:15:391
459963994412,10cyclictest26-21ksoftirqd/110:30:131
459963994412,10cyclictest26-21ksoftirqd/109:20:461
5554912430,5chrt0-21swapper/311:35:233
459970994324,7cyclictest42-21ksoftirqd/312:32:093
459970994324,6cyclictest42-21ksoftirqd/310:40:013
459970994324,14cyclictest42-21ksoftirqd/310:45:193
459970994321,9cyclictest42-21ksoftirqd/311:54:483
459970994321,13cyclictest42-21ksoftirqd/308:20:173
45997099431,22cyclictest472106-21seq07:40:293
45997099431,22cyclictest472106-21seq07:40:293
45997099431,13cyclictest1389-21kerneloops12:22:173
459970994311,15cyclictest42-21ksoftirqd/312:36:013
459963994332,8cyclictest251rcuc/109:10:131
459963994310,17cyclictest26-21ksoftirqd/112:20:251
459963994310,17cyclictest26-21ksoftirqd/111:30:221
459963994310,17cyclictest26-21ksoftirqd/111:30:221
459963994310,12cyclictest26-21ksoftirqd/107:51:041
45997099429,17cyclictest1753-21gnome-shell07:20:273
45997099429,17cyclictest1753-21gnome-shell07:20:263
45997099427,16cyclictest1753-21gnome-shell10:29:543
459970994224,14cyclictest42-21ksoftirqd/312:18:013
459970994223,6cyclictest42-21ksoftirqd/312:05:173
459970994223,6cyclictest42-21ksoftirqd/311:05:113
459970994223,6cyclictest42-21ksoftirqd/310:20:183
459970994223,14cyclictest521723-21df10:00:173
459970994223,14cyclictest42-21ksoftirqd/311:40:223
459970994223,14cyclictest42-21ksoftirqd/309:35:113
459970994223,14cyclictest42-21ksoftirqd/308:25:133
459970994215,7cyclictest462779-21forks07:15:333
459970994213,14cyclictest141rcu_preempt09:18:573
459970994210,10cyclictest42-21ksoftirqd/312:00:153
459970994124,6cyclictest42-21ksoftirqd/309:43:303
459970994123,6cyclictest42-21ksoftirqd/309:10:323
459970994123,14cyclictest42-21ksoftirqd/311:18:293
459970994115,5cyclictest141rcu_preempt11:45:043
45997099411,20cyclictest507419-21/usr/sbin/munin09:20:283
459970994023,6cyclictest42-21ksoftirqd/311:31:243
459970994023,6cyclictest42-21ksoftirqd/311:31:243
459970994023,13cyclictest42-21ksoftirqd/310:12:063
459970994023,13cyclictest42-21ksoftirqd/308:49:233
459970994022,14cyclictest42-21ksoftirqd/309:50:113
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional