You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-08-01 - 18:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

Intel(R) Celeron(R) CPU J3455 @ 1.50GHz, Linux 5.8.0-59-generic (Profile)

Latency plot of shadow in rack #c, slot #7
Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -p99 -i200 -h400 -a0-2 -t3 -q done
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rackcslot7s (updated Thu Jun 10, 2021 12:45:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1337521430,8sleep11219499cyclictest12:09:201
1295321170,2sleep00-21swapper/009:30:160
2256621150,4sleep014-21rcuc/011:30:290
159521130,9sleep21220099cyclictest11:50:192
2278321000,3sleep124-21rcuc/111:30:341
1107729842,18sleep30-21swapper/307:05:273
1192229553,33sleep00-21swapper/007:09:090
1056328841,37sleep10-21swapper/107:05:171
89699870,6rtkit-daemon2754-21pool09:39:011
89699830,2rtkit-daemon0-21swapper/109:41:031
89699830,2rtkit-daemon0-21swapper/109:41:021
1196828247,28sleep20-21swapper/207:09:452
89699800,2rtkit-daemon0-21swapper/107:54:242
89699790,3rtkit-daemon895-21rtkit-daemon08:04:322
89699790,3rtkit-daemon895-21rtkit-daemon08:04:322
89699780,2rtkit-daemon0-21swapper/108:46:181
89699780,2rtkit-daemon0-21swapper/108:42:191
89699780,2rtkit-daemon0-21swapper/108:40:160
89699770,1rtkit-daemon0-21swapper/109:50:031
89699760,2rtkit-daemon0-21swapper/109:04:141
89699740,8rtkit-daemon17172-21df_inode12:15:142
89699740,6rtkit-daemon16428-21ssh12:14:300
89699730,5rtkit-daemon28142-21ssh12:32:430
89699730,5rtkit-daemon28142-21ssh12:32:430
89699710,5rtkit-daemon9544-21/usr/sbin/munin10:15:360
89699710,3rtkit-daemon0-21swapper/112:03:380
89699710,3rtkit-daemon0-21swapper/111:02:460
89699710,2rtkit-daemon0-21swapper/112:21:220
89699710,17rtkit-daemon0-21swapper/112:34:551
89699710,17rtkit-daemon0-21swapper/112:34:551
89699700,3rtkit-daemon895-21rtkit-daemon09:31:532
89699700,2rtkit-daemon0-21swapper/110:06:260
89699690,2rtkit-daemon0-21swapper/111:07:160
89699690,2rtkit-daemon0-21swapper/110:59:390
89699690,2rtkit-daemon0-21swapper/109:58:580
89699680,2rtkit-daemon0-21swapper/108:22:110
89699670,3rtkit-daemon0-21swapper/008:49:320
89699670,2rtkit-daemon0-21swapper/110:49:340
89699670,2rtkit-daemon0-21swapper/110:36:270
89699670,2rtkit-daemon0-21swapper/107:42:380
216522670,9sleep31221099cyclictest10:35:423
89699660,2rtkit-daemon0-21swapper/111:43:510
89699660,2rtkit-daemon0-21swapper/110:34:000
89699660,2rtkit-daemon0-21swapper/109:39:150
89699660,2rtkit-daemon0-21swapper/108:28:580
89699650,3rtkit-daemon0-21swapper/111:13:570
89699650,2rtkit-daemon0-21swapper/112:38:560
89699650,2rtkit-daemon0-21swapper/110:53:420
89699650,2rtkit-daemon0-21swapper/110:53:420
89699650,2rtkit-daemon0-21swapper/109:48:370
89699650,2rtkit-daemon0-21swapper/108:31:150
89699650,2rtkit-daemon0-21swapper/108:19:430
89699650,2rtkit-daemon0-21swapper/010:22:440
12190996516,21cyclictest25431-21awk08:55:000
89699640,7rtkit-daemon28852-21ntp_states09:00:310
89699640,7rtkit-daemon2754-21pool08:12:143
89699640,7rtkit-daemon2754-21pool07:18:140
89699640,2rtkit-daemon0-21swapper/111:49:070
89699630,4rtkit-daemon31927-21diskmemload09:51:240
89699630,2rtkit-daemon0-21swapper/110:18:001
89699630,2rtkit-daemon0-21swapper/109:09:200
89699630,2rtkit-daemon0-21swapper/108:57:550
89699630,2rtkit-daemon0-21swapper/107:32:270
89699620,2rtkit-daemon0-21swapper/108:37:070
89699610,4rtkit-daemon895-21rtkit-daemon08:00:183
89699610,4rtkit-daemon895-21rtkit-daemon08:00:183
89699610,2rtkit-daemon0-21swapper/109:56:041
89699600,5rtkit-daemon2754-21pool09:31:043
89699600,2rtkit-daemon0-21swapper/208:18:113
89699600,2rtkit-daemon0-21swapper/111:28:060
1221099582,52cyclictest0-21swapper/310:10:003
89699560,4rtkit-daemon31927-21diskmemload12:05:330
89699560,2rtkit-daemon0-21swapper/011:36:520
89699550,2rtkit-daemon0-21swapper/012:00:000
89699550,2rtkit-daemon0-21swapper/012:00:000
89699540,7rtkit-daemon1812-21sendmail09:37:363
89699540,7rtkit-daemon1269-21crond08:17:002
89699540,2rtkit-daemon0-21swapper/112:40:001
89699540,2rtkit-daemon0-21swapper/107:31:191
1219499542,5cyclictest0-21swapper/111:35:451
89699530,2rtkit-daemon0-21swapper/111:18:371
89699520,6rtkit-daemon2754-21pool08:10:402
89699520,6rtkit-daemon2366-21goa-identity-se08:09:533
89699520,4rtkit-daemon895-21rtkit-daemon12:16:493
89699510,4rtkit-daemon895-21rtkit-daemon07:10:432
89699490,2rtkit-daemon0-21swapper/010:26:371
255622490,6sleep124-21rcuc/112:29:241
12200994936,8cyclictest18326-21sh11:25:012
12200994936,8cyclictest18326-21sh11:25:012
1219499492,43cyclictest4636-21kworker/1:010:10:001
89699480,3rtkit-daemon895-21rtkit-daemon07:58:523
89699480,2rtkit-daemon0-21swapper/007:48:151
89699470,2rtkit-daemon0-21swapper/107:29:561
1220099472,6cyclictest0-21swapper/210:26:012
89699450,4rtkit-daemon31927-21diskmemload09:10:201
89699450,4rtkit-daemon31927-21diskmemload09:10:201
89699450,2rtkit-daemon0-21swapper/010:00:180
12210994537,4cyclictest0-21swapper/310:26:003
1219099452,35cyclictest529-21xfsaild/dm-009:10:450
1219099452,35cyclictest529-21xfsaild/dm-009:10:450
89699440,2rtkit-daemon0-21swapper/108:36:461
77202440,4sleep014-21rcuc/010:14:020
12210994434,4cyclictest42350irq/125-i91509:30:003
1219499442,30cyclictest6335-21kworker/1:211:15:011
89699430,5rtkit-daemon7725-21ps11:05:391
89699430,2rtkit-daemon895-21rtkit-daemon10:38:422
1221099433,31cyclictest3032-21munin-run09:15:003
1221099433,31cyclictest3032-21munin-run09:15:003
12200994328,7cyclictest31934-21munin-run09:10:002
12200994327,8cyclictest15331-21munin-run11:20:012
1219099434,7cyclictest18343-21munin-run11:25:010
1219099434,7cyclictest18343-21munin-run11:25:010
12190994334,5cyclictest0-21swapper/009:22:000
1219099433,34cyclictest1647-21kworker/0:009:20:000
73222420,9sleep21220099cyclictest11:59:342
73222420,9sleep21220099cyclictest11:59:342
1221099413,30cyclictest20711-21perf07:30:003
12200994126,7cyclictest18779-21munin-run09:40:002
12200994125,8cyclictest23480-21kworker/u8:010:15:002
12200994124,9cyclictest1343-21munin-run08:00:002
1220099412,30cyclictest31110-21sed10:00:012
12200994122,13cyclictest24615-21ssh09:49:022
89699400,23rtkit-daemon31927-21diskmemload10:43:561
12210994027,7cyclictest30678-21ssh12:35:473
1220099402,33cyclictest29647-21kworker/u8:112:32:012
1220099402,33cyclictest29647-21kworker/u8:112:32:012
1220099402,32cyclictest15659-21kworker/u8:207:25:002
1219499401,30cyclictest6197-21kworker/1:209:25:001
12190994026,6cyclictest17611-21kworker/0:209:45:000
12190994026,6cyclictest17611-21kworker/0:209:45:000
1221099395,28cyclictest431ktimersoftd/309:17:003
12210993930,5cyclictest42350irq/125-i91511:25:013
12210993930,5cyclictest42350irq/125-i91511:25:013
12210993926,9cyclictest22216-21kworker/u8:011:32:013
12210993923,8cyclictest22216-21kworker/u8:011:40:013
1221099392,29cyclictest7173-21kworker/u8:011:15:013
12194993916,9cyclictest6268-21ntp_states10:10:321
89699380,2rtkit-daemon0-21swapper/011:17:130
89699380,21rtkit-daemon0-21swapper/110:24:341
1221099383,6cyclictest7458-21munin-run12:00:003
1221099383,6cyclictest7458-21munin-run12:00:003
12200993822,7cyclictest1764-21kworker/u8:210:08:002
1219499382,5cyclictest23333-21kworker/1:012:00:001
1219499382,5cyclictest23333-21kworker/1:012:00:001
12194993818,13cyclictest27871-21cut11:40:141
89699370,2rtkit-daemon0-21swapper/108:56:071
89699370,2rtkit-daemon0-21swapper/108:53:351
12210993725,8cyclictest44-21ksoftirqd/310:32:003
12210993724,7cyclictest4602-21grep11:00:353
12200993728,5cyclictest0-21swapper/210:04:002
12200993728,5cyclictest0-21swapper/208:45:592
12200993725,8cyclictest29647-21kworker/u8:112:23:002
12200993725,7cyclictest0-21swapper/212:36:012
12200993725,7cyclictest0-21swapper/212:07:012
12200993724,8cyclictest18568-21kworker/u8:311:48:012
1220099371,30cyclictest4985-21kworker/u8:009:23:592
12194993721,8cyclictest6335-21kworker/1:211:27:011
12194993715,6cyclictest1734-21gnome-shell07:24:591
12190993725,8cyclictest1011-21kworker/0:007:58:590
4492362,26sleep2606-21fschecks_time10:55:202
306532362,26sleep330720-21proc_pri07:50:363
12210993624,7cyclictest1764-21kworker/u8:210:53:013
12210993624,7cyclictest1764-21kworker/u8:210:53:013
12210993622,8cyclictest8037-21cat09:20:493
12210993621,8cyclictest14247-21cat07:15:003
1220099363,5cyclictest16376-21/usr/sbin/munin07:20:002
1220099363,25cyclictest29825-21df_inode10:50:162
1220099363,25cyclictest29825-21df_inode10:50:152
12200993629,4cyclictest0-21swapper/210:33:012
12200993627,5cyclictest0-21swapper/210:45:012
12200993627,5cyclictest0-21swapper/209:02:002
12200993626,6cyclictest0-21swapper/209:52:012
12200993624,8cyclictest18751-21kworker/u8:007:32:002
12200993624,7cyclictest0-21swapper/209:14:002
12200993624,7cyclictest0-21swapper/209:14:002
12200993623,6cyclictest11-21rcu_preempt10:20:002
1220099362,28cyclictest0-21swapper/208:23:002
1220099361,29cyclictest12633-21kworker/u8:208:33:002
12194993626,6cyclictest13300-21kworker/1:007:20:001
12210993525,6cyclictest0-21swapper/308:37:003
12210993524,7cyclictest18568-21kworker/u8:311:49:013
12210993524,7cyclictest18568-21kworker/u8:311:27:013
12210993524,6cyclictest11054-21grep10:20:013
12210993523,6cyclictest410-21unixbench_multi10:00:363
1221099352,27cyclictest22738-21perf12:25:003
1221099352,13cyclictest131102ssh10:21:083
12200993526,5cyclictest0-21swapper/212:28:002
12200993525,6cyclictest12633-21kworker/u8:208:36:002
12200993525,6cyclictest0-21swapper/208:44:002
12200993525,5cyclictest0-21swapper/208:10:002
12200993525,5cyclictest0-21swapper/207:26:002
12200993524,7cyclictest0-21swapper/209:16:002
12200993524,6cyclictest0-21swapper/212:12:012
12200993524,6cyclictest0-21swapper/207:39:002
12200993523,7cyclictest0-21swapper/210:23:012
1220099352,28cyclictest18568-21kworker/u8:311:28:012
1220099352,28cyclictest0-21swapper/211:08:012
12194993523,6cyclictest15796-21cstates11:20:191
12194993523,6cyclictest15796-21cstates11:20:181
1219499352,29cyclictest23480-21kworker/u8:010:05:001
1219099351,6cyclictest17064-21perf10:30:000
89699340,3rtkit-daemon0-21swapper/107:52:451
89699340,18rtkit-daemon0-21swapper/112:13:501
89699340,18rtkit-daemon0-21swapper/110:51:431
89699340,18rtkit-daemon0-21swapper/110:51:431
89699340,16rtkit-daemon0-21swapper/108:20:031
48612342,5sleep14924-21missed_timers08:05:341
200832343,6sleep120100-21/usr/sbin/munin10:34:591
12210993423,7cyclictest10765-21kworker/u8:012:30:013
1220099342,5cyclictest0-21swapper/211:40:002
1220099342,27cyclictest0-21swapper/208:53:002
12200993422,6cyclictest5080-21grep11:00:412
1220099341,28cyclictest0-21swapper/209:43:002
1220099341,28cyclictest0-21swapper/209:43:002
1219499343,24cyclictest24965-21taskset07:39:181
1219499342,4cyclictest0-21swapper/111:46:011
12190993423,6cyclictest3751-21munin-run08:05:000
12190993423,6cyclictest3751-21munin-run08:05:000
89699330,17rtkit-daemon0-21swapper/108:30:431
74572331,4sleep30-21swapper/311:05:343
12210993321,7cyclictest0-21swapper/307:25:003
12210993313,8cyclictest2703-21pcscd08:24:383
12200993324,5cyclictest0-21swapper/209:27:002
1220099332,11cyclictest225682cat11:30:282
12200993320,8cyclictest7173-21kworker/u8:011:11:012
12194993325,5cyclictest12146-21kworker/1:009:30:001
1219499332,24cyclictest6004-21taskset11:04:071
12194993311,16cyclictest0-21swapper/112:20:501
1219099334,22cyclictest16946-21cut12:15:130
12190993323,6cyclictest0-21swapper/009:28:590
12190993320,7cyclictest20237-21grep07:25:410
89699320,2rtkit-daemon0-21swapper/007:53:330
89699320,17rtkit-daemon0-21swapper/110:49:221
89699320,16rtkit-daemon0-21swapper/111:52:061
12210993221,6cyclictest44-21ksoftirqd/310:15:013
12210993220,6cyclictest19710-21smartctl09:40:253
12210993220,6cyclictest19710-21smartctl09:40:243
12210993220,6cyclictest0-21swapper/308:33:003
12210993216,8cyclictest14593-21/usr/sbin/munin08:30:003
12200993223,5cyclictest0-21swapper/207:49:002
12200993222,6cyclictest0-21swapper/207:41:002
12194993220,6cyclictest13656-21missed_timers09:30:301
12194993219,7cyclictest950-21runrttasks08:15:481
12194993219,7cyclictest21468-21ntp_offset10:35:371
12194993211,16cyclictest534-21fschecks_time10:55:211
1221099312,6cyclictest31610-21runrttasks09:06:003
1221099312,23cyclictest2578-21chrt10:57:593
12210993119,6cyclictest3827-21sh11:52:513
12210993118,7cyclictest31927-21diskmemload12:31:383
12210993118,7cyclictest31927-21diskmemload12:31:383
12210993118,7cyclictest29451-21mailstats09:55:323
12210993118,7cyclictest13792-21/usr/sbin/munin12:10:173
12210993117,8cyclictest25238-21cat07:40:023
1219499314,9cyclictest1734-21gnome-shell07:13:591
12194993119,7cyclictest10267-21sh12:04:001
12194993119,6cyclictest950-21runrttasks09:46:041
12194993118,7cyclictest30864-21ls09:05:281
12194993115,10cyclictest1728-21cut08:00:121
12194993115,10cyclictest1728-21cut08:00:121
12194993111,14cyclictest0-21swapper/109:15:131
12190993121,6cyclictest16079-21kworker/0:107:21:000
12190993120,6cyclictest0-21swapper/007:50:000
1221099303,21cyclictest22263-21grep08:45:293
1221099302,21cyclictest12654-21cpuspeed_turbos11:15:123
12210993018,7cyclictest25544-21cat07:40:173
12210993018,6cyclictest31927-21diskmemload12:07:053
1220099305,19cyclictest26729-21ls08:55:382
12200993021,5cyclictest0-21swapper/211:42:012
12200993020,6cyclictest0-21swapper/210:49:012
12200993016,8cyclictest9390-21vmstat12:00:422
12194993018,7cyclictest13965-21timerandwakeup08:25:381
12194993017,7cyclictest32011-21taskset07:55:141
12190993019,5cyclictest25683-21sh10:42:460
12190993018,6cyclictest12321-21ls07:10:110
12190993017,7cyclictest3230-21unixbench_singl11:50:440
12190993016,8cyclictest4845-21cat08:05:320
12210992923,3cyclictest0-21swapper/307:50:003
12210992918,6cyclictest9821-21seq12:02:173
12210992918,6cyclictest26707-21unixbench_multi09:50:463
12210992918,6cyclictest24873-21turbostat.cron09:50:013
12210992918,6cyclictest24296-21kernelversion10:40:273
12210992917,6cyclictest21931-21awk07:30:273
12210992917,6cyclictest14505-21wc07:15:233
12210992916,7cyclictest26688-21cstates10:45:143
12210992915,8cyclictest29194-21grep09:00:363
12194992918,6cyclictest18160-21switchtime12:15:351
12194992917,6cyclictest25327-21cpuspeed_turbos07:40:161
12194992916,7cyclictest6321-21cstates08:10:171
12190992916,7cyclictest23820-21cat07:35:350
12210992818,6cyclictest0-21swapper/308:53:003
12210992817,6cyclictest30264-21sh11:44:083
12210992816,6cyclictest19324-21cat08:40:153
12210992815,7cyclictest25709-21cpuspeed_turbos08:55:143
12200992815,7cyclictest13733-21cut08:25:352
12190992816,7cyclictest24470-21timerwakeupswit12:25:360
12190992715,6cyclictest6330-21expr08:10:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional