You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2022-01-26 - 06:48

Intel(R) Atom(TM) CPU E3845 @ 1.91GHz, Linux 4.19.37-rt19 (Profile)

Latency plot of system in rack #d, slot #1
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Up99 -i200 -h400 --smi -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rackdslot1.osadl.org (updated Wed Jan 26, 2022 00:46:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1832821780,6sleep1500799cyclictest22:40:351
916921750,9sleep1500799cyclictest21:50:191
46712138115,10sleep30-21swapper/319:09:423
550981240,96rtkit-daemon3086-21apache219:05:381
44392124102,10sleep20-21swapper/219:06:392
195352890,8sleep0500699cyclictest22:05:040
168462800,2sleep10-21swapper/123:20:121
118791779,17phc2sys0-21swapper/019:08:530
78992660,2sleep10-21swapper/121:10:131
93682640,6sleep19370-21unixbench_multi20:30:371
161042630,8sleep1500799cyclictest21:20:121
174452600,8sleep3500999cyclictest23:20:203
114092590,2sleep00-21swapper/023:15:020
294152580,2sleep30-21swapper/300:15:113
8922570,2sleep00-21swapper/021:40:190
500899564,8cyclictest33-21ksoftirqd/222:20:062
500899564,6cyclictest0-21swapper/219:55:252
500899538,7cyclictest33-21ksoftirqd/219:20:022
5007995346,4cyclictest25-21ksoftirqd/120:20:151
500899526,7cyclictest0-21swapper/220:45:212
500899515,7cyclictest0-21swapper/221:25:262
500899515,7cyclictest0-21swapper/221:25:252
500899515,4cyclictest33-21ksoftirqd/221:20:352
500899513,6cyclictest0-21swapper/223:00:112
500899513,6cyclictest0-21swapper/222:55:382
500899506,7cyclictest0-21swapper/219:45:342
500899506,5cyclictest0-21swapper/223:20:022
500899505,6cyclictest0-21swapper/219:30:342
500899503,7cyclictest33-21ksoftirqd/221:30:152
500899496,9cyclictest33-21ksoftirqd/200:25:092
500899496,7cyclictest33-21ksoftirqd/220:05:342
500899496,7cyclictest33-21ksoftirqd/200:05:092
500899494,6cyclictest33-21ksoftirqd/221:47:512
500899493,5cyclictest0-21swapper/219:40:302
500899491,6cyclictest0-21swapper/221:00:252
5007994942,4cyclictest25-21ksoftirqd/123:25:151
500899486,8cyclictest33-21ksoftirqd/221:15:212
500899486,6cyclictest33-21ksoftirqd/223:10:052
500899484,6cyclictest0-21swapper/220:25:232
500899483,6cyclictest33-21ksoftirqd/220:52:392
500899476,8cyclictest33-21ksoftirqd/220:30:202
500899476,6cyclictest33-21ksoftirqd/222:40:052
500899475,8cyclictest33-21ksoftirqd/223:45:102
500899475,8cyclictest33-21ksoftirqd/220:45:012
500899475,6cyclictest0-21swapper/223:50:342
5008994740,4cyclictest10913-21cron22:35:002
500899473,6cyclictest0-21swapper/220:00:222
500899470,8cyclictest22439-21runrttasks23:25:302
500899466,8cyclictest33-21ksoftirqd/221:55:362
500899466,8cyclictest33-21ksoftirqd/220:15:292
500899464,9cyclictest33-21ksoftirqd/222:10:032
500899463,7cyclictest33-21ksoftirqd/222:00:442
500899463,7cyclictest0-21swapper/223:55:102
500899463,6cyclictest0-21swapper/222:55:012
500899463,6cyclictest0-21swapper/222:55:002
500899456,9cyclictest33-21ksoftirqd/219:50:242
500899456,8cyclictest33-21ksoftirqd/223:40:052
500899456,7cyclictest33-21ksoftirqd/200:30:342
500899456,6cyclictest33-21ksoftirqd/221:50:202
500899455,8cyclictest33-21ksoftirqd/200:35:132
500899446,8cyclictest33-21ksoftirqd/221:41:112
500899446,7cyclictest33-21ksoftirqd/222:45:192
500899445,8cyclictest33-21ksoftirqd/222:25:072
500899444,6cyclictest0-21swapper/223:05:022
500899444,6cyclictest0-21swapper/222:35:192
500899443,8cyclictest0-21swapper/219:35:032
500899443,6cyclictest33-21ksoftirqd/221:05:292
55098430,3rtkit-daemon549-21rtkit-daemon19:31:151
500899437,8cyclictest33-21ksoftirqd/220:10:102
500899432,5cyclictest0-21swapper/222:05:222
5006994326,14cyclictest9-21ksoftirqd/022:31:100
500999420,4cyclictest121rcu_preempt21:10:163
500899425,8cyclictest33-21ksoftirqd/223:15:222
500899425,8cyclictest33-21ksoftirqd/200:15:242
500899425,7cyclictest33-21ksoftirqd/221:35:112
500899425,6cyclictest33-21ksoftirqd/219:10:112
500899424,6cyclictest0-21swapper/223:30:342
500899424,5cyclictest0-21swapper/223:35:122
114791420,31getstats23282-21cut20:50:091
500899414,6cyclictest33-21ksoftirqd/220:35:042
5006994113,4cyclictest17360-21sed22:40:200
500899404,6cyclictest33-21ksoftirqd/220:20:262
500899403,7cyclictest33-21ksoftirqd/200:00:062
500899403,7cyclictest33-21ksoftirqd/200:00:052
5007994033,4cyclictest25-21ksoftirqd/121:30:171
5007994032,5cyclictest25-21ksoftirqd/120:05:201
50832391,3sleep3121rcu_preempt00:25:123
500899384,8cyclictest33-21ksoftirqd/222:15:052
500899384,7cyclictest33-21ksoftirqd/221:10:102
5008993813,9cyclictest33-21ksoftirqd/200:11:432
5007993829,6cyclictest25-21ksoftirqd/119:45:051
5006993831,4cyclictest9-21ksoftirqd/023:30:190
5006993813,3cyclictest15778-21seq00:39:260
5006993811,4cyclictest5672-21ntpq19:10:070
5006993731,3cyclictest9-21ksoftirqd/000:18:000
5006993729,5cyclictest9-21ksoftirqd/023:56:050
5006993719,15cyclictest9-21ksoftirqd/023:37:110
5006993713,4cyclictest24469-21expr20:50:210
5006993711,5cyclictest8422-21cat21:50:130
5006993711,4cyclictest32053-21sh23:00:060
5009993629,4cyclictest41-21ksoftirqd/322:31:563
5009993629,4cyclictest41-21ksoftirqd/320:44:583
5007993631,3cyclictest25-21ksoftirqd/123:35:091
5007993629,4cyclictest25-21ksoftirqd/100:08:261
5007993628,5cyclictest25-21ksoftirqd/100:27:261
5007993627,6cyclictest25-21ksoftirqd/123:45:161
500799361,3cyclictest121rcu_preempt22:50:021
500799361,3cyclictest121rcu_preempt22:50:011
5006993619,14cyclictest9-21ksoftirqd/020:26:230
5006993615,3cyclictest25928-21/usr/sbin/munin20:15:140
5006993610,4cyclictest8950-21bc23:10:160
5006993610,4cyclictest25111-21expr00:10:100
5009993528,4cyclictest41-21ksoftirqd/321:00:553
5009993528,4cyclictest41-21ksoftirqd/300:01:213
5009993528,4cyclictest41-21ksoftirqd/300:01:213
5008993512,9cyclictest33-21ksoftirqd/200:21:232
5007993528,4cyclictest25-21ksoftirqd/121:05:051
5007993528,4cyclictest25-21ksoftirqd/119:40:401
5007993528,4cyclictest25-21ksoftirqd/100:33:571
5007993527,5cyclictest25-21ksoftirqd/123:32:291
500699359,4cyclictest4930-21cut21:45:200
500699359,4cyclictest2892-21sshd23:41:370
500699359,4cyclictest12240-21basename21:15:150
5006993513,3cyclictest9973-21expr00:30:190
5006993512,4cyclictest16627-21cstates00:00:100
5006993512,4cyclictest16627-21cstates00:00:100
5006993510,4cyclictest24170-21cat19:30:290
5006993510,4cyclictest16596-21missed_timers20:00:320
5006993510,4cyclictest10353-21sed20:35:020
5009993428,4cyclictest41-21ksoftirqd/323:55:363
5009993428,3cyclictest41-21ksoftirqd/321:29:473
5009993428,3cyclictest41-21ksoftirqd/321:29:473
5009993427,4cyclictest41-21ksoftirqd/322:57:593
5009993427,4cyclictest41-21ksoftirqd/322:11:583
5009993427,4cyclictest41-21ksoftirqd/321:35:003
5009993427,4cyclictest19021-21cron22:45:003
5008993410,8cyclictest33-21ksoftirqd/219:27:012
5007993429,3cyclictest25-21ksoftirqd/119:50:051
5007993427,4cyclictest25-21ksoftirqd/120:41:001
5007993427,4cyclictest25-21ksoftirqd/119:25:011
5007993426,5cyclictest25-21ksoftirqd/122:24:041
500799340,32cyclictest0-21swapper/100:37:301
500699349,4cyclictest31981-21sed21:00:120
500699349,4cyclictest27660-21df_abs20:55:110
5006993428,3cyclictest9-21ksoftirqd/021:29:350
5006993428,3cyclictest9-21ksoftirqd/021:29:340
5006993428,3cyclictest9-21ksoftirqd/019:46:110
5006993427,4cyclictest9-21ksoftirqd/021:58:370
5006993426,5cyclictest9-21ksoftirqd/019:19:300
5006993410,4cyclictest7347-21expr21:10:090
5006993410,3cyclictest2471-21if_enp4s021:05:010
5009993327,4cyclictest41-21ksoftirqd/321:55:233
5009993327,4cyclictest41-21ksoftirqd/320:45:163
5009993327,3cyclictest41-21ksoftirqd/300:13:503
5009993326,4cyclictest41-21ksoftirqd/323:01:003
5009993326,4cyclictest41-21ksoftirqd/319:24:163
5009993325,5cyclictest41-21ksoftirqd/320:58:513
5009993325,5cyclictest41-21ksoftirqd/319:29:173
5007993327,4cyclictest25-21ksoftirqd/122:00:291
5007993327,4cyclictest25-21ksoftirqd/100:10:571
5007993327,3cyclictest25-21ksoftirqd/121:29:271
5007993327,3cyclictest25-21ksoftirqd/121:29:271
5007993326,5cyclictest25-21ksoftirqd/123:51:291
5007993326,4cyclictest25-21ksoftirqd/120:39:141
5007993326,4cyclictest25-21ksoftirqd/120:28:171
5007993326,4cyclictest25-21ksoftirqd/119:11:121
500699339,4cyclictest4673-21df_abs23:45:120
500699338,4cyclictest18376-21sort20:45:020
5006993327,3cyclictest9-21ksoftirqd/000:20:270
5006993326,4cyclictest9-21ksoftirqd/023:54:370
5006993326,4cyclictest9-21ksoftirqd/022:57:160
5006993326,4cyclictest9-21ksoftirqd/022:39:130
5006993326,4cyclictest9-21ksoftirqd/022:22:130
5006993326,4cyclictest9-21ksoftirqd/021:39:310
5006993311,3cyclictest27324-21if_enp4s022:15:020
5006993311,3cyclictest20753-21phc2sys-jitter22:45:130
5006993310,4cyclictest31175-21basename19:40:150
5006993310,4cyclictest19580-21acpi20:05:180
5006993310,3cyclictest25324-21df_abs22:10:200
274902330,5sleep30-21swapper/322:15:043
5009993225,4cyclictest41-21ksoftirqd/322:26:473
5009993225,4cyclictest41-21ksoftirqd/322:00:293
5009993225,4cyclictest41-21ksoftirqd/320:18:073
5009993225,4cyclictest41-21ksoftirqd/319:52:023
5009993224,5cyclictest41-21ksoftirqd/321:53:433
5008993226,4cyclictest33-21ksoftirqd/219:15:462
5007993226,4cyclictest25-21ksoftirqd/123:15:101
5007993226,4cyclictest25-21ksoftirqd/122:45:221
5007993226,4cyclictest25-21ksoftirqd/121:42:401
5007993226,4cyclictest25-21ksoftirqd/121:15:181
5007993225,5cyclictest25-21ksoftirqd/121:37:111
5007993225,4cyclictest25-21ksoftirqd/122:56:161
5007993225,4cyclictest25-21ksoftirqd/122:35:111
5007993225,4cyclictest25-21ksoftirqd/120:55:591
5007993224,5cyclictest25-21ksoftirqd/121:57:071
500699329,4cyclictest6129-21proc_pri23:05:350
500699329,4cyclictest15470-21idleruntime19:20:240
5006993226,3cyclictest9-21ksoftirqd/019:27:320
5006993225,4cyclictest9-21ksoftirqd/000:08:190
5006993224,5cyclictest9-21ksoftirqd/021:24:020
5006993224,5cyclictest9-21ksoftirqd/019:56:170
5006993213,4cyclictest26201-21sensors_temp21:30:330
5006993211,3cyclictest9361-21grep20:30:380
5006993211,3cyclictest23879-21fschecks_time22:50:050
5006993211,3cyclictest23879-21fschecks_time22:50:050
5006993210,4cyclictest20730-21users23:25:100
277422320,8sleep0500699cyclictest19:35:250
5009993125,4cyclictest41-21ksoftirqd/300:30:153
5009993125,3cyclictest41-21ksoftirqd/319:35:113
5009993124,4cyclictest41-21ksoftirqd/322:51:403
5009993124,4cyclictest41-21ksoftirqd/322:51:403
5009993124,4cyclictest41-21ksoftirqd/322:23:183
5009993124,4cyclictest41-21ksoftirqd/321:06:083
5009993124,4cyclictest41-21ksoftirqd/319:57:393
5009993124,4cyclictest41-21ksoftirqd/319:16:323
5007993125,4cyclictest25-21ksoftirqd/121:49:481
5007993125,4cyclictest25-21ksoftirqd/121:01:421
5007993124,4cyclictest25-21ksoftirqd/122:25:131
5007993124,4cyclictest25-21ksoftirqd/122:19:061
5007993123,5cyclictest25-21ksoftirqd/122:06:101
5006993125,3cyclictest9-21ksoftirqd/020:42:350
5006993125,3cyclictest9-21ksoftirqd/019:54:540
5009993025,3cyclictest41-21ksoftirqd/323:45:143
5009993024,4cyclictest41-21ksoftirqd/319:49:133
5009993024,4cyclictest41-21ksoftirqd/300:21:173
5009993024,3cyclictest41-21ksoftirqd/323:26:553
5009993024,3cyclictest41-21ksoftirqd/320:54:333
5009993024,3cyclictest41-21ksoftirqd/320:32:593
5009993023,4cyclictest41-21ksoftirqd/323:17:103
5009993023,4cyclictest41-21ksoftirqd/323:12:003
5009993023,4cyclictest41-21ksoftirqd/323:06:183
5009993023,4cyclictest41-21ksoftirqd/322:37:153
5009993023,4cyclictest41-21ksoftirqd/321:37:033
5009993023,4cyclictest41-21ksoftirqd/300:05:033
500899304,7cyclictest33-21ksoftirqd/220:59:282
5007993023,4cyclictest25-21ksoftirqd/122:10:171
5007993022,5cyclictest25-21ksoftirqd/100:02:171
5007993022,5cyclictest25-21ksoftirqd/100:02:161
5006993025,3cyclictest9-21ksoftirqd/022:02:120
5006993024,3cyclictest9-21ksoftirqd/022:29:330
5006993024,3cyclictest9-21ksoftirqd/020:24:330
5006993023,4cyclictest9-21ksoftirqd/023:21:530
5009992924,3cyclictest41-21ksoftirqd/320:10:583
5009992922,4cyclictest41-21ksoftirqd/322:46:553
5009992922,4cyclictest41-21ksoftirqd/322:09:383
5009992922,4cyclictest41-21ksoftirqd/321:17:493
5009992922,4cyclictest41-21ksoftirqd/319:42:093
5009992922,4cyclictest41-21ksoftirqd/300:39:253
5009992921,5cyclictest41-21ksoftirqd/321:23:163
5007992923,4cyclictest25-21ksoftirqd/100:20:121
5007992922,4cyclictest25-21ksoftirqd/123:09:201
5007992922,4cyclictest25-21ksoftirqd/120:48:371
5007992922,4cyclictest25-21ksoftirqd/100:19:301
5009992821,4cyclictest41-21ksoftirqd/321:46:053
5009992820,5cyclictest41-21ksoftirqd/320:23:463
5009992820,5cyclictest41-21ksoftirqd/320:06:113
5009992820,5cyclictest41-21ksoftirqd/319:11:433
5009992820,5cyclictest31242-21grep21:40:013
5007992821,5cyclictest25-21ksoftirqd/120:16:371
5007992821,4cyclictest25-21ksoftirqd/119:57:131
5007992821,4cyclictest25-21ksoftirqd/119:36:281
5007992821,4cyclictest25-21ksoftirqd/119:18:591
5006992821,4cyclictest9-21ksoftirqd/020:14:470
5009992721,4cyclictest41-21ksoftirqd/319:31:213
5009992720,4cyclictest41-21ksoftirqd/323:31:203
5007992722,3cyclictest25-21ksoftirqd/119:25:021
5007992721,4cyclictest25-21ksoftirqd/120:00:391
5007992620,3cyclictest25-21ksoftirqd/120:12:041
5007992619,5cyclictest25-21ksoftirqd/123:43:341
5007992619,4cyclictest25-21ksoftirqd/123:12:531
5009992518,4cyclictest41-21ksoftirqd/323:35:303
5009992518,4cyclictest41-21ksoftirqd/320:38:113
5007992518,4cyclictest25-21ksoftirqd/123:57:491
5007992517,5cyclictest25-21ksoftirqd/123:00:421
5009992418,4cyclictest41-21ksoftirqd/320:03:583
5009992418,3cyclictest41-21ksoftirqd/323:54:293
5009992418,3cyclictest41-21ksoftirqd/323:43:293
5006992418,3cyclictest9-21ksoftirqd/000:28:580
872822216,3sleep125-21ksoftirqd/122:30:161
5009992215,4cyclictest3337-21sed20:25:133
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional