You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-22 - 11:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Pentium N4200 @1100 MHz, Linux 5.9.1-rt18 (Profile)

Latency plot of shadow in rack #c, slot #4
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot4s.osadl.org (updated Wed Mar 22, 2023 00:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
113072325278,34sleep00-21swapper/019:06:070
114812295247,34sleep10-21swapper/119:07:411
113552292246,32sleep20-21swapper/219:06:322
100802288219,55sleep30-21swapper/319:05:173
189442400,1sleep331-21ksoftirqd/320:35:203
184062360,2sleep20-21swapper/220:35:152
168362330,2sleep20-21swapper/220:30:252
12089993312,19cyclictest27088-21http22:10:002
1208999273,7cyclictest39142chrt22:30:022
1208999262,6cyclictest18059-21python321:50:002
1208999262,20cyclictest16857-40apt-check00:15:052
12075992518,5cyclictest2511-21expr20:00:150
12075992516,6cyclictest5032-21users23:45:360
1207599251,19cyclictest19972-21plymouth20:40:010
12098992411,9cyclictest18322-21apt-config21:50:003
1209899240,7cyclictest301rcuc/322:16:383
12089992414,8cyclictest6559-21find20:10:002
1208999240,6cyclictest30924-21apt-get21:05:122
1207599243,14cyclictest6862-21apt-get20:10:130
12098992320,2cyclictest301rcuc/320:50:393
12098992315,6cyclictest26312-21ntp_states00:35:223
12098992315,6cyclictest22429-21ntpq19:30:193
12098992315,5cyclictest721-21dbus-daemon22:40:013
12098992315,5cyclictest18325-21users00:15:333
1208999234,17cyclictest22105-21sed00:25:232
1208999232,18cyclictest15231-21apt-get19:15:132
1208999231,6cyclictest25-21ksoftirqd/222:01:282
12089992315,4cyclictest12251-21apt-config19:10:002
12089992314,4cyclictest18059-21python321:50:002
12089992313,8cyclictest22973-21idleruntime-cro23:15:002
94192220,3sleep225-21ksoftirqd/220:15:162
143292220,2chrt0-21swapper/300:10:013
12098992219,2cyclictest301rcuc/323:54:023
12098992219,2cyclictest301rcuc/321:42:503
12098992219,2cyclictest301rcuc/321:35:053
12098992218,3cyclictest301rcuc/323:45:413
12098992218,3cyclictest301rcuc/321:06:163
12098992218,3cyclictest301rcuc/319:14:373
12098992218,3cyclictest301rcuc/300:25:063
12098992215,5cyclictest23812-21memory23:15:203
12098992211,7cyclictest632-21grep21:10:143
1208999223,3cyclictest0-21swapper/221:20:002
1208999222,18cyclictest16288-21python300:15:002
1208999222,14cyclictest23199-21apt-get23:15:152
12089992214,6cyclictest4930-21sed20:05:202
12089992214,6cyclictest4930-21sed20:05:202
12089992214,6cyclictest1235-21cron23:40:012
12089992214,4cyclictest19265-21cron19:25:002
12089992210,7cyclictest1894-21snmpd21:00:132
12079992217,3cyclictest6231-21latency_hist20:10:001
12079992217,3cyclictest6231-21latency_hist20:10:001
1207999220,5cyclictest0-21swapper/122:30:371
1207999220,5cyclictest0-21swapper/122:25:421
1207599222,13cyclictest8229-21apt-get22:40:150
12075992216,3cyclictest18852-21sendmail00:20:000
12075992214,6cyclictest19319-21sed23:05:190
121222210,5sleep1181rcuc/120:20:261
121222210,5sleep1181rcuc/120:20:261
12098992118,2cyclictest301rcuc/323:41:023
12098992118,2cyclictest301rcuc/323:29:243
12098992118,2cyclictest301rcuc/323:23:443
12098992118,2cyclictest301rcuc/322:05:033
12098992118,2cyclictest301rcuc/322:00:363
12098992118,2cyclictest301rcuc/321:33:573
12098992118,2cyclictest301rcuc/320:42:353
12098992118,2cyclictest301rcuc/320:14:483
12098992118,2cyclictest301rcuc/320:09:083
12098992118,2cyclictest301rcuc/320:09:083
12098992118,2cyclictest301rcuc/320:01:483
12098992118,2cyclictest301rcuc/319:38:473
12098992118,2cyclictest301rcuc/319:16:533
12098992118,2cyclictest301rcuc/300:21:573
12098992118,2cyclictest301rcuc/300:01:543
12098992117,3cyclictest301rcuc/323:38:493
12098992117,3cyclictest301rcuc/322:51:263
12098992117,3cyclictest301rcuc/322:28:573
12098992117,3cyclictest301rcuc/321:48:093
12098992117,3cyclictest301rcuc/321:19:283
12098992117,3cyclictest301rcuc/320:46:053
12098992117,3cyclictest301rcuc/320:34:423
12098992117,3cyclictest301rcuc/319:41:043
12098992116,4cyclictest301rcuc/322:11:093
12098992116,3cyclictest301rcuc/323:12:583
12098992111,5cyclictest19271-21perf19:25:003
1208999214,15cyclictest13149-21cat20:25:002
1208999214,15cyclictest13149-21cat20:25:002
1208999212,17cyclictest18693-21munin-run23:05:002
12089992117,2cyclictest9011-21munin-run21:30:012
1208999211,5cyclictest24269-21lsb_release20:50:002
12089992114,5cyclictest31389-21sed19:50:272
12089992114,5cyclictest22045-21wc21:55:282
12089992114,5cyclictest20876-21sed20:40:182
1208999211,17cyclictest13332-21python321:40:022
12089992110,6cyclictest11143-21munin-run21:35:012
12079992114,5cyclictest32738-21needreboot22:20:291
12079992113,6cyclictest6842-21ntpq22:35:211
1207999210,5cyclictest0-21swapper/121:41:401
1207999210,5cyclictest0-21swapper/121:36:331
1207999210,5cyclictest0-21swapper/120:35:461
1207999210,5cyclictest0-21swapper/120:26:361
1207999210,4cyclictest0-21swapper/122:40:501
1207999210,4cyclictest0-21swapper/121:27:161
1207599213,12cyclictest15870-40apt-check20:30:070
12075992114,5cyclictest1690-21snmp_o5powerctl21:10:270
12075992112,6cyclictest3107-21tune2fs21:15:170
1207599211,12cyclictest15053-21runrttasks20:28:590
1209899208,9cyclictest19528-21grep19:25:133
12098992017,2cyclictest301rcuc/323:57:323
12098992017,2cyclictest301rcuc/323:05:223
12098992017,2cyclictest301rcuc/322:55:083
12098992017,2cyclictest301rcuc/322:33:553
12098992017,2cyclictest301rcuc/321:22:223
12098992017,2cyclictest301rcuc/320:25:533
12098992017,2cyclictest301rcuc/320:22:083
12098992017,2cyclictest301rcuc/320:22:083
12098992017,2cyclictest301rcuc/319:59:273
12098992017,2cyclictest301rcuc/319:46:233
12098992017,2cyclictest301rcuc/300:30:503
12098992017,2cyclictest301rcuc/300:14:173
12098992016,3cyclictest301rcuc/323:00:393
12098992016,2cyclictest301rcuc/320:55:303
12098992015,4cyclictest301rcuc/320:15:523
12098992015,3cyclictest8232-21runrttasks22:40:143
12098992015,3cyclictest301rcuc/323:32:173
12098992015,3cyclictest29555-21ntp_offset21:00:203
1208999202,12cyclictest23404-21apt-get00:30:152
12089992015,3cyclictest27260-21idleruntime-cro23:25:012
12089992015,3cyclictest1846-21turbostat.cron20:00:002
12089992013,5cyclictest29486-21wc23:30:012
12089992013,5cyclictest20827-21basename23:10:002
12089992013,5cyclictest20223-21head19:25:202
12089992013,5cyclictest13650-21grep20:25:212
12089992013,5cyclictest12262-21uname22:50:002
12089992012,6cyclictest5573-21awk23:50:002
1208999201,13cyclictest23790-21apt-get19:35:142
1208999200,4cyclictest28507-21cron20:59:592
1208999200,18cyclictest6185-1kworker/2:0H20:55:002
1208999200,18cyclictest13793-40apt-check21:40:082
12079992015,3cyclictest28175-21hwlatdetect22:10:181
12079992014,3cyclictest16106-21runrttasks00:13:471
12079992014,3cyclictest12056-21munin-node19:10:021
12079992012,6cyclictest17454-21ls19:20:201
12079992011,7cyclictest27686-21df23:25:171
12079992011,5cyclictest11102-21sensors_temp00:00:241
12079992010,8cyclictest5477-21cat21:20:221
1207999200,4cyclictest0-21swapper/121:46:311
1207999200,4cyclictest0-21swapper/120:43:341
1207999200,4cyclictest0-21swapper/119:33:231
1207999200,4cyclictest0-21swapper/119:28:161
1207599209,8cyclictest17123-21cron19:20:000
1207599209,2cyclictest184922rm21:50:010
1207599205,11cyclictest24068-21runrttasks20:48:300
1207599202,13cyclictest25300-21munin-run00:35:010
1207599202,12cyclictest16959-21runrttasks21:45:260
12075992013,3cyclictest4380-21idleruntime22:30:210
12075992012,6cyclictest2870-21awk23:40:310
12075992012,5cyclictest29408-21grep23:30:000
12075992011,5cyclictest15000-21lsb_release19:15:000
1207599201,14cyclictest6154-21turbostat20:09:590
1207599201,14cyclictest6154-21turbostat20:09:590
1207599201,13cyclictest5810-21apt-get23:50:190
1207599201,13cyclictest12513-21apt-get22:50:210
12098991916,2cyclictest301rcuc/321:56:303
12098991916,2cyclictest301rcuc/321:25:323
12098991916,2cyclictest301rcuc/319:50:093
12098991915,3cyclictest301rcuc/322:45:383
12098991915,3cyclictest301rcuc/322:21:203
1208999199,8cyclictest2598-21grep21:15:002
1208999199,7cyclictest29627-21apt-get23:30:152
1208999198,9cyclictest1561-21awk22:25:002
1208999198,8cyclictest8451-21fschecks_count22:40:182
1208999191,5cyclictest21038-21lsb_release00:25:002
12089991914,3cyclictest2118-21/usr/sbin/munin20:00:192
12089991914,3cyclictest1894-21snmpd19:42:382
1208999191,3cyclictest6185-1kworker/2:0H21:25:002
1208999191,3cyclictest25-21ksoftirqd/200:05:552
12089991913,4cyclictest28727-21sed19:45:222
12089991912,5cyclictest26754-21sed00:35:282
1208999190,6cyclictest25-21ksoftirqd/222:25:332
1208999190,5cyclictest7708-21basename23:54:592
1208999190,4cyclictest14408-21lsb_release22:55:002
1207999192,12cyclictest23039-21apt-get22:00:181
1207999192,12cyclictest16975-21apt-get00:15:161
12079991915,2cyclictest26455-21interrupts19:40:221
12079991915,2cyclictest26092-21rt-features22:05:251
12079991915,2cyclictest24541-21uptime23:15:271
12079991915,2cyclictest17339-21latency23:00:211
12079991913,4cyclictest3564-21ntp_kernel_pll_21:15:211
12079991912,5cyclictest4117-21ls23:45:221
12079991912,5cyclictest31361-21cat21:05:201
12079991912,5cyclictest2557-21snmp_o5powerctl23:40:271
12079991912,5cyclictest2447-21perl20:00:151
12079991912,5cyclictest20345-21sed00:20:311
12079991912,5cyclictest14645-21dpkg22:55:171
12079991912,5cyclictest10567-21diskstats22:45:171
12079991912,4cyclictest31075-21runrttasks23:31:151
12079991911,6cyclictest30387-21cat22:15:231
12079991911,6cyclictest21323-21awk00:25:151
12079991911,6cyclictest19516-21ls23:05:211
1207599194,11cyclictest10704-21runrttasks21:31:180
1207599193,9cyclictest14139-21lsb_release00:10:000
1207599193,12cyclictest16287-21python300:15:010
1207599192,12cyclictest30563-21runrttasks21:04:060
1207599192,12cyclictest27084-21python322:10:000
1207599192,12cyclictest21653-21apt-get19:30:130
1207599192,12cyclictest13793-40apt-check21:40:060
12075991914,3cyclictest22161-21timerandwakeup23:10:260
12075991913,4cyclictest309-21snmp_o5powerctl23:35:270
12075991912,5cyclictest31459-21cut21:05:200
12075991912,5cyclictest30745-21cut19:50:180
12075991912,5cyclictest30670-21grep23:30:290
12075991912,5cyclictest27370-21sed23:25:000
12075991912,5cyclictest27142-21grep19:40:290
12075991912,5cyclictest21998-21cat21:55:280
12075991912,5cyclictest11303-21expr20:20:190
12075991912,5cyclictest11303-21expr20:20:190
12075991911,5cyclictest27470-21idleruntime-cro00:40:000
12075991911,5cyclictest20851-21grep19:25:280
1207599191,13cyclictest24763-21runrttasks23:17:150
1207599190,4cyclictest0-21swapper/020:50:560
1207599190,4cyclictest0-21swapper/000:24:230
1207599190,16cyclictest1848-21turbostat19:59:590
12089991815,2cyclictest1894-21snmpd22:57:172
1208999181,4cyclictest0-21swapper/223:44:112
1208999181,4cyclictest0-21swapper/222:37:462
1208999181,4cyclictest0-21swapper/222:18:142
12089991811,4cyclictest9340-21grep23:55:492
1208999181,12cyclictest10098-21apt-get00:00:152
1208999180,6cyclictest25-21ksoftirqd/222:11:512
1208999180,6cyclictest25-21ksoftirqd/219:30:252
12079991814,2cyclictest9243-21/usr/sbin/munin21:30:131
12079991814,2cyclictest884-21sendmail_mailst19:55:211
12079991814,2cyclictest24016-21missed_timers00:30:211
12079991814,2cyclictest22878-21interrupts20:45:201
12079991814,2cyclictest21615-21nfs4_client21:55:211
12079991814,2cyclictest15652-21hwlatdetect19:15:191
12079991814,2cyclictest13354-21threads00:05:361
12079991812,4cyclictest19356-21load21:50:221
12079991812,4cyclictest16018-21ls20:30:171
12079991812,3cyclictest21071-21runrttasks23:10:151
12079991811,5cyclictest5976-21expr23:50:181
12079991811,5cyclictest32243-21date19:54:591
12079991811,5cyclictest30818-21sed21:05:011
12079991811,5cyclictest25137-21users19:35:281
12079991811,5cyclictest1423-21ntp_kernel_pll_21:10:231
12079991811,5cyclictest12893-21grep22:50:251
1207999181,12cyclictest24501-21apt-get20:50:151
12079991810,6cyclictest27659-21sed00:40:011
12075991811,5cyclictest28245-21cat19:45:170
12075991811,5cyclictest22135-21basename20:45:010
12075991811,5cyclictest11549-21ls21:35:150
1207599181,13cyclictest11894-21runrttasks22:46:570
1207599181,12cyclictest7943-21apt-get23:55:160
1207599181,12cyclictest27399-21apt-config22:10:010
1207599181,12cyclictest26642-21runrttasks20:55:130
12075991810,6cyclictest31028-21sed22:15:350
12075991810,5cyclictest6880-21ls22:35:220
12079991713,2cyclictest8915-21sendmail_mailtr23:55:241
12079991713,2cyclictest27211-21latency20:55:201
1207999171,12cyclictest6143-21apt-get20:10:121
1207599179,6cyclictest2675-21awk22:25:320
1207599172,10cyclictest22285-21taskset00:25:270
12075991713,2cyclictest24792-21sensors_temp19:35:230
12075991712,3cyclictest6285-21unixbench_singl21:20:320
1207599171,12cyclictest16027-21runrttasks22:55:420
1207599171,11cyclictest10797-21head00:00:210
12075991710,5cyclictest9154-21cat21:30:000
12075991710,5cyclictest17865-21sed23:00:330
12075991710,4cyclictest9803-21pmu-power20:15:210
1207999169,5cyclictest9744-21ntpq20:15:191
12079991612,2cyclictest29032-21sendmail_mailst19:45:251
12079991612,2cyclictest25598-21fschecks_count23:20:221
12079991610,4cyclictest32619-21proc_pri23:35:241
1207599169,5cyclictest17534-21expr19:20:200
1207599169,4cyclictest23410-21tail22:00:220
1207599151,9cyclictest1169-21runrttasks22:21:550
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional