You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-27 - 05:54

x86 Intel Atom D2700 @2130 MHz, Linux 5.15.49-rt47 (Profile)

Latency plot of system in rack #1, slot #6
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot6.osadl.org (updated Mon Mar 27, 2023 00:43:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2911530,0irq_work/2311rcuc/220:56:502
2911490,0irq_work/2311rcuc/222:57:052
2911330,0irq_work/2311rcuc/219:05:392
2911300,0irq_work/2311rcuc/219:57:032
2911190,0irq_work/2311rcuc/222:31:122
2911170,0irq_work/2311rcuc/200:15:282
2911150,0irq_work/2311rcuc/223:46:212
2911150,0irq_work/2311rcuc/200:09:012
3611100,0irq_work/3381rcuc/323:01:563
2911090,0irq_work/2311rcuc/221:57:392
2911080,0irq_work/2311rcuc/222:00:532
2911070,0irq_work/2311rcuc/223:06:172
2911060,0irq_work/2311rcuc/223:56:132
2911060,0irq_work/2311rcuc/222:16:232
2911050,0irq_work/2311rcuc/222:05:562
2911040,0irq_work/2311rcuc/221:45:052
2911040,0irq_work/2311rcuc/219:18:392
2911030,0irq_work/2311rcuc/223:26:512
2911030,0irq_work/2311rcuc/223:21:182
2911020,0irq_work/2311rcuc/220:37:442
2911020,0irq_work/2311rcuc/220:17:412
2911020,0irq_work/2311rcuc/219:06:412
2911000,0irq_work/2311rcuc/223:01:382
2911000,0irq_work/2311rcuc/222:48:282
2911000,0irq_work/2311rcuc/222:48:282
291990,0irq_work/2311rcuc/221:16:392
291980,0irq_work/2311rcuc/222:40:392
291980,0irq_work/2311rcuc/221:31:042
291980,0irq_work/2311rcuc/220:34:392
291980,0irq_work/2311rcuc/220:34:392
291970,0irq_work/2311rcuc/222:44:412
291970,0irq_work/2311rcuc/219:51:522
291970,0irq_work/2311rcuc/219:41:442
291960,0irq_work/2311rcuc/220:01:082
291950,0irq_work/2311rcuc/222:29:542
291950,0irq_work/2311rcuc/222:11:382
291950,0irq_work/2311rcuc/221:48:412
291950,0irq_work/2311rcuc/221:11:412
291950,0irq_work/2311rcuc/219:46:422
291940,0irq_work/2311rcuc/221:52:082
291940,0irq_work/2311rcuc/221:30:462
291940,0irq_work/2311rcuc/219:21:262
291940,0irq_work/2311rcuc/200:25:052
291920,0irq_work/2311rcuc/223:38:422
291920,0irq_work/2311rcuc/221:36:412
291920,0irq_work/2311rcuc/221:36:412
291910,0irq_work/2311rcuc/221:02:362
291910,0irq_work/2311rcuc/220:28:112
291910,0irq_work/2311rcuc/220:28:112
291910,0irq_work/2311rcuc/219:11:292
291900,0irq_work/2311rcuc/223:44:412
291900,0irq_work/2311rcuc/223:33:512
291900,0irq_work/2311rcuc/220:09:412
291900,0irq_work/2311rcuc/219:35:062
291890,0irq_work/2311rcuc/222:53:552
291890,0irq_work/2311rcuc/220:51:212
291880,0irq_work/2311rcuc/223:52:512
291880,0irq_work/2311rcuc/223:52:512
291880,0irq_work/2311rcuc/223:17:502
291880,0irq_work/2311rcuc/223:12:232
291880,0irq_work/2311rcuc/222:25:412
291880,0irq_work/2311rcuc/220:43:562
291880,0irq_work/2311rcuc/219:38:412
291880,0irq_work/2311rcuc/200:17:342
291880,0irq_work/2311rcuc/200:01:342
291870,0irq_work/2311rcuc/221:21:222
291870,0irq_work/2311rcuc/220:46:112
291860,0irq_work/2311rcuc/220:11:552
291850,0irq_work/2311rcuc/220:25:412
291850,0irq_work/2311rcuc/200:26:412
291840,0irq_work/2311rcuc/221:09:152
181830,0irq_work/0161rcuc/023:56:090
291820,0irq_work/2311rcuc/219:28:412
291820,0irq_work/2311rcuc/219:28:412
638499701,0cyclictest210099rtkit-daemon21:46:290
638799665,0cyclictest3799migration/322:16:143
638799665,0cyclictest3799migration/300:11:393
638799658,0cyclictest3799migration/322:56:083
638799656,0cyclictest3799migration/322:47:273
638799656,0cyclictest3799migration/322:47:273
638799653,0cyclictest3799migration/323:56:223
6387996516,0cyclictest24834-21seq21:21:043
638799624,0cyclictest3799migration/300:17:233
638799615,0cyclictest3799migration/322:54:483
638799615,0cyclictest3799migration/321:46:023
181590,0irq_work/0161rcuc/020:20:340
638799583,0cyclictest3799migration/322:21:423
638799572,0cyclictest5762-21tune2fs21:56:083
638799570,0cyclictest3799migration/323:06:483
6387995622,0cyclictest6168-21grep20:36:043
361560,0irq_work/3381rcuc/320:05:523
6387995527,0cyclictest30833-21grep21:36:243
6387995527,0cyclictest30833-21grep21:36:243
6387995522,0cyclictest17880-21grep23:41:293
6387995519,0cyclictest15854-21grep23:36:243
6387995517,0cyclictest22345-21cut19:56:153
6387995515,0cyclictest3280-21gltestperf00:26:073
638799550,0cyclictest3799migration/322:27:193
6387995422,0cyclictest830-17auditd21:15:473
6387995415,0cyclictest15266-21ntpdc20:56:173
6387995414,0cyclictest27676-21grep21:31:083
638599541,0cyclictest210099rtkit-daemon21:02:411
638599541,0cyclictest210099rtkit-daemon20:41:541
6384995446,0cyclictest6536-21tr19:01:050
6387995323,0cyclictest31464-21grep20:16:243
6387995317,0cyclictest830-17auditd20:30:463
6387995317,0cyclictest830-17auditd20:30:463
6387995316,0cyclictest26119-21grep00:01:233
6387995315,0cyclictest16576-21cstates21:01:043
141530,0rcu_preempt0-21swapper/318:57:203
6387995221,0cyclictest955-21crond22:40:513
6387995220,0cyclictest23806-21timerandwakeup21:16:223
6387995217,0cyclictest9578-21sed19:11:093
6387995216,0cyclictest21124-21grep19:51:253
6387995216,0cyclictest1254-21lldpd23:26:083
6387995215,0cyclictest12947-21sed22:11:253
181520,0irq_work/0161rcuc/021:36:000
181520,0irq_work/0161rcuc/021:36:000
181520,0irq_work/0161rcuc/000:08:000
6387995117,0cyclictest0-21swapper/322:31:053
6387995112,0cyclictest32150-21cpu20:21:063
6387995111,0cyclictest25315-21seq22:41:273
638799510,0cyclictest3799migration/323:12:073
638799510,0cyclictest3799migration/319:27:103
638799510,0cyclictest3799migration/319:27:103
638599511,0cyclictest210099rtkit-daemon20:49:221
361510,0irq_work/3381rcuc/320:14:403
141510,0rcu_preempt39-21ksoftirqd/323:21:123
141510,0rcu_preempt25-21ksoftirqd/120:05:321
638799503,0cyclictest3799migration/323:54:313
638799503,0cyclictest3799migration/323:54:313
6387995018,0cyclictest11274-21sendmail_mailqu20:46:203
6387995012,0cyclictest18475-21unixbench_multi19:41:263
638599502,0cyclictest210099rtkit-daemon22:21:241
181500,0irq_work/0161rcuc/021:35:520
6387994922,0cyclictest1144-21gltestperf00:21:073
221490,0irq_work/1241rcuc/123:51:191
221490,0irq_work/1241rcuc/123:51:191
221490,0irq_work/1241rcuc/120:56:181
141490,0rcu_preempt6792-21fschecks_time19:01:093
141490,0rcu_preempt39-21ksoftirqd/323:16:163
141490,0rcu_preempt39-21ksoftirqd/321:51:213
141490,0rcu_preempt25-21ksoftirqd/123:26:071
141490,0rcu_preempt25-21ksoftirqd/121:21:211
141490,0rcu_preempt25-21ksoftirqd/119:56:271
638799483,0cyclictest3799migration/322:07:593
638799482,0cyclictest12384-21cpuspeed_turbos20:51:043
6387994818,0cyclictest1339-21Xorg19:06:503
6387994816,0cyclictest8422-21fschecks_count20:41:073
6387994816,0cyclictest0-21swapper/321:06:253
221480,0irq_work/1241rcuc/100:11:171
181480,0irq_work/0161rcuc/021:11:560
141480,0rcu_preempt39-21ksoftirqd/323:46:133
141480,0rcu_preempt39-21ksoftirqd/319:31:083
141480,0rcu_preempt25-21ksoftirqd/123:46:131
6387994717,0cyclictest27315-21grep20:06:253
6387994715,0cyclictest16821-21sshd19:40:393
6387994714,0cyclictest8584-21swap22:01:203
6387994714,0cyclictest32377-21netstat21:41:163
6384994712,0cyclictest24434-21crond21:20:520
311470,0rcuc/25926-21taskset18:57:582
181470,0irq_work/0161rcuc/023:11:150
181470,0irq_work/0161rcuc/022:12:440
141470,0rcu_preempt25-21ksoftirqd/120:31:201
141470,0rcu_preempt25-21ksoftirqd/120:31:201
141470,0rcu_preempt13-21ksoftirqd/000:21:220
141470,0rcu_preempt0-21swapper/123:34:541
6387994622,0cyclictest28369-21chrt00:06:593
6387994618,0cyclictest13091-21cat23:31:123
6387994615,0cyclictest12928-21cat19:21:263
638599467,0cyclictest2399migration/123:16:131
638599466,0cyclictest2399migration/120:17:221
181460,0irq_work/0161rcuc/020:35:080
181460,0irq_work/0161rcuc/020:35:080
141460,0rcu_preempt25126-21grep21:21:080
141460,0rcu_preempt25-21ksoftirqd/122:31:121
141460,0rcu_preempt25-21ksoftirqd/119:21:141
141460,0rcu_preempt13-21ksoftirqd/020:41:140
141460,0rcu_preempt1234-21runrttasks20:12:450
638799454,0cyclictest3799migration/320:31:113
638799454,0cyclictest3799migration/320:31:113
638799453,0cyclictest830-17auditd19:45:533
638599458,0cyclictest2399migration/123:09:171
6385994528,0cyclictest5861-21grep21:56:091
638599451,0cyclictest210099rtkit-daemon22:16:541
638599451,0cyclictest210099rtkit-daemon22:06:481
638499451,0cyclictest210099rtkit-daemon22:25:410
638499451,0cyclictest210099rtkit-daemon20:36:070
241450,0rcuc/15850-21taskset18:56:581
221450,0irq_work/1241rcuc/120:30:461
221450,0irq_work/1241rcuc/120:30:461
181450,0irq_work/0161rcuc/023:35:350
161450,0rcuc/05968-21runrttasks18:58:320
141450,0rcu_preempt25-21ksoftirqd/122:41:251
141450,0rcu_preempt25-21ksoftirqd/120:11:201
638599448,0cyclictest2399migration/121:45:141
638599448,0cyclictest2399migration/119:42:451
638599448,0cyclictest2399migration/119:15:571
638599446,0cyclictest2399migration/123:15:151
638599446,0cyclictest2399migration/121:07:131
221440,0irq_work/1241rcuc/122:55:521
181440,0irq_work/0161rcuc/000:19:240
161440,0rcuc/012598-21fschecks_time20:51:070
141440,0rcu_preempt25-21ksoftirqd/123:36:121
141440,0rcu_preempt16184-21munin-run21:00:520
6387994313,0cyclictest1339-21Xorg19:17:503
638599437,0cyclictest2399migration/119:34:561
638599436,0cyclictest2399migration/121:39:531
638599436,0cyclictest2399migration/121:39:531
638599436,0cyclictest2399migration/121:16:241
638599435,0cyclictest2399migration/123:23:311
638599431,0cyclictest210099rtkit-daemon22:12:001
638599431,0cyclictest210099rtkit-daemon20:51:111
638499431,0cyclictest210099rtkit-daemon22:02:280
638499431,0cyclictest210099rtkit-daemon20:26:410
638499431,0cyclictest210099rtkit-daemon20:26:410
221430,0irq_work/1241rcuc/120:21:221
181430,0irq_work/0161rcuc/023:52:100
181430,0irq_work/0161rcuc/023:52:100
141430,0rcu_preempt26661-21latency20:06:121
141430,0rcu_preempt25-21ksoftirqd/121:54:521
141430,0rcu_preempt25-21ksoftirqd/121:31:141
141430,0rcu_preempt25-21ksoftirqd/121:11:171
141430,0rcu_preempt25-21ksoftirqd/100:26:221
141430,0rcu_preempt13-21ksoftirqd/022:16:140
6385994225,0cyclictest25176-21grep00:01:071
6385994225,0cyclictest22313-21cut22:36:071
181420,0irq_work/0161rcuc/022:27:380
141420,0rcu_preempt25-21ksoftirqd/122:01:071
141420,0rcu_preempt13-21ksoftirqd/023:19:320
181410,0irq_work/0161rcuc/022:58:140
181410,0irq_work/0161rcuc/022:05:530
181410,0irq_work/0161rcuc/021:57:060
141410,0rcu_preempt25-21ksoftirqd/122:51:181
141410,0rcu_preempt25-21ksoftirqd/122:26:081
141410,0rcu_preempt13-21ksoftirqd/023:48:440
141410,0rcu_preempt13-21ksoftirqd/019:56:160
181400,0irq_work/0161rcuc/023:06:230
181400,0irq_work/0161rcuc/022:37:240
181400,0irq_work/0161rcuc/000:15:090
141400,0rcu_preempt25-21ksoftirqd/122:46:111
141400,0rcu_preempt25-21ksoftirqd/122:46:111
141400,0rcu_preempt25-21ksoftirqd/119:11:131
141400,0rcu_preempt14867-21gltestperf19:31:090
141400,0rcu_preempt13-21ksoftirqd/022:53:520
7650390,0irq/26-ahci[000830-13auditd23:40:510
6385993922,0cyclictest2119-21grep00:21:231
141390,0rcu_preempt25-21ksoftirqd/121:27:501
141390,0rcu_preempt25-21ksoftirqd/119:55:521
141390,0rcu_preempt22864-21tr23:56:041
141390,0rcu_preempt13-21ksoftirqd/019:21:290
141390,0rcu_preempt0-21swapper/023:24:410
7650380,0irq/26-ahci[0004652-21crond00:30:520
638799380,0cyclictest3799migration/321:27:113
638499384,0cyclictest1799migration/022:45:050
638499384,0cyclictest1799migration/019:51:070
6384993823,0cyclictest27446-21grep22:46:240
6384993823,0cyclictest27446-21grep22:46:240
638499381,0cyclictest210099rtkit-daemon21:09:090
638499381,0cyclictest210099rtkit-daemon20:46:350
181380,0irq_work/0161rcuc/021:41:220
141380,0rcu_preempt25-21ksoftirqd/123:01:081
141380,0rcu_preempt25-21ksoftirqd/119:01:161
141380,0rcu_preempt25-21ksoftirqd/100:17:151
141380,0rcu_preempt25-21ksoftirqd/100:06:151
141380,0rcu_preempt24614-21latency20:01:140
141380,0rcu_preempt13-21ksoftirqd/023:41:260
141380,0rcu_preempt13-21ksoftirqd/023:05:070
141380,0rcu_preempt0-21swapper/120:40:031
638599371,0cyclictest2399migration/123:41:131
6384993721,0cyclictest18349-21cat19:41:250
638499371,0cyclictest10901-21crond19:15:510
221370,0irq_work/1241rcuc/119:40:471
141370,0rcu_preempt25-21ksoftirqd/119:47:281
638499365,0cyclictest1799migration/019:48:440
638499364,0cyclictest1799migration/020:06:080
638499362,0cyclictest11505-21crond19:20:520
181360,0irq_work/0161rcuc/021:05:150
181360,0irq_work/0161rcuc/000:01:030
141360,0rcu_preempt13-21ksoftirqd/019:39:470
6385993521,0cyclictest1273-21grep21:46:091
6384993520,0cyclictest20311-21grep22:31:090
181350,0irq_work/0161rcuc/021:30:510
141350,0rcu_preempt13-21ksoftirqd/019:29:090
141350,0rcu_preempt13-21ksoftirqd/019:29:090
638499344,0cyclictest1799migration/020:22:200
638499332,0cyclictest1799migration/023:30:070
181330,0irq_work/0141rcu_preempt21:51:120
103350330,0irq/28-eth0-tx-0-21swapper/019:06:330
638599313,0cyclictest2399migration/119:07:151
141310,0rcu_preempt25-21ksoftirqd/119:25:531
141310,0rcu_preempt25-21ksoftirqd/119:25:531
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional