You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-03-17 - 01:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack3slot3s.osadl.org (updated Mon Mar 17, 2025 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
495629993430,3cyclictest707253-21sendmail00:20:000
495642993325,5cyclictest654622-21ls23:20:003
495642992925,2cyclictest600873-21sendmail22:20:003
495629992919,7cyclictest557219-21grep21:30:230
495640992824,2cyclictest560393-21dump-pmu-power21:35:002
495629992823,3cyclictest0-21swapper/022:50:200
495640992724,1cyclictest691053-21sendmail00:00:012
495635992723,2cyclictest630092-21sh22:52:131
495642992621,3cyclictest0-21swapper/320:00:013
495635992622,3cyclictest547611-21cron21:19:591
495635992622,2cyclictest673313-21latency_hist23:40:001
495635992622,2cyclictest0-21swapper/121:25:001
495635992622,2cyclictest0-21swapper/100:40:001
495635992621,3cyclictest709-21systemd-logind22:24:591
495635992621,3cyclictest530178-21apt-key20:50:011
49563599261,21cyclictest524611-21ntpq20:31:171
495629992621,3cyclictest722-21gdbus21:50:010
495642992521,3cyclictest620137-21needreboot22:40:183
495642992521,2cyclictest0-21swapper/321:45:003
495635992520,4cyclictest541764-21kworker/u16:0+xprtiod22:03:241
495635992520,3cyclictest519290-21cpuspeed_turbos20:15:131
495635992516,6cyclictest702900-21sh00:14:591
49563599251,21cyclictest633061-21ntpq22:55:171
49563599251,21cyclictest498894-21ntpq19:15:161
495629992521,2cyclictest703-21polkitd20:20:000
495642992420,2cyclictest506478-21sh19:39:593
49564099241,20cyclictest588556-21taskset22:05:422
495629992420,2cyclictest691068-21idleruntime-cro00:00:010
495629992420,2cyclictest619214-21latency_hist22:40:010
495629992420,2cyclictest536302-21sh21:05:010
495629992413,8cyclictest573857-21df_inode21:50:090
495629992412,8cyclictest635234-21sh22:57:410
49562999240,3cyclictest0-21swapper/021:37:280
49564299235,14cyclictest584109-21strings22:00:183
49564299231,3cyclictest573368-21latency_hist21:50:013
495642992313,7cyclictest726-21gdbus20:20:003
495640992316,5cyclictest538879-21cpuspeed_turbos21:10:182
495635992320,1cyclictest509123-21dump-pmu-power19:50:011
495635992319,3cyclictest615086-21gltestperf22:35:131
495635992317,4cyclictest565914-21gltestperf21:40:161
495635992313,7cyclictest707856-21cut00:20:151
495635992312,8cyclictest699-21dbus-daemon22:30:221
49563599231,19cyclictest715803-21ntpq00:30:191
49562999235,4cyclictest677939-21sh23:44:530
49562999234,7cyclictest672973-21sh23:39:330
495629992321,1cyclictest0-21swapper/020:20:560
495629992315,5cyclictest702903-21turbostat.cron00:15:000
495629992312,9cyclictest0-21swapper/022:30:140
49564299224,5cyclictest694792-21sh00:03:383
495642992217,3cyclictest681612-21idleruntime-cro23:50:003
49564299221,18cyclictest669627-21ntpq23:35:183
49564299220,18cyclictest667490-21ssh23:33:313
49564099221,7cyclictest633922-21taskset22:55:372
49564099221,19cyclictest708494-21ntpq00:20:202
49564099220,19cyclictest683927-21ssh23:51:312
49563599221,8cyclictest560091-21ssh21:34:351
495635992218,3cyclictest331ktimers/123:22:111
495635992218,2cyclictest506535-21systemd-run19:40:011
495635992214,5cyclictest663121-21sh23:30:001
495635992212,7cyclictest942-21snmpd23:50:581
495635992211,9cyclictest521947-21ntpq20:20:471
49563599221,18cyclictest542980-21cron21:15:001
49562999224,5cyclictest718869-21kworker/u16:3+rpciod00:37:410
49562999221,7cyclictest606379-21taskset22:25:340
49562999221,7cyclictest518049-21ntpq20:10:210
495629992216,4cyclictest549654-21sh21:20:480
495629992215,5cyclictest538840-21tr21:10:180
495629992212,7cyclictest716454-21unixbench_singl00:30:320
49562999220,8cyclictest692357-21ntpq00:00:440
49564299216,2cyclictest563826-21sh21:37:543
495642992118,2cyclictest502563-21idleruntime-cro19:29:593
495642992112,7cyclictest706680-21ssh00:19:083
49564299210,8cyclictest700348-21runrttasks00:10:253
49564099216,12cyclictest262-21systemd-journal19:56:582
495640992117,3cyclictest526161-21sh20:40:002
495640992117,2cyclictest530125-21apt-get20:50:002
495640992116,3cyclictest678041-21/usr/sbin/munin23:45:002
495640992116,3cyclictest421ktimers/223:49:142
495640992116,3cyclictest0-21swapper/219:40:002
495640992115,4cyclictest560942-21cpuspeed_turbos21:35:162
495640992113,5cyclictest501643-21ldconfig19:25:202
495640992112,6cyclictest716571-21ssh00:30:382
49564099211,18cyclictest669337-21/usr/sbin/munin23:35:162
49564099211,17cyclictest662654-21sh23:29:182
495640992111,7cyclictest601776-21sed22:20:192
49564099211,16cyclictest498193-21runrttasks19:15:002
49564099210,7cyclictest573869-21/usr/sbin/munin21:50:092
495635992117,2cyclictest1-21systemd00:27:131
49563599211,18cyclictest538263-21diskmemload22:49:091
49563599211,17cyclictest688343-21ssh23:55:511
495635992111,7cyclictest608364-21sh22:28:301
495635992110,8cyclictest642240-21sh23:05:311
495635992110,8cyclictest514268-21/usr/sbin/munin20:00:201
49562999219,8cyclictest262-21systemd-journal22:04:580
49562999211,8cyclictest714269-21taskset00:29:190
49562999211,7cyclictest668009-21taskset23:34:160
49562999211,7cyclictest498205-21/usr/sbin/munin19:15:150
495629992117,3cyclictest0-21swapper/023:30:000
49562999211,18cyclictest552589-21runrttasks21:25:180
49562999211,18cyclictest552589-21runrttasks21:25:180
49562999211,17cyclictest644690-21runrttasks23:09:220
495629992111,6cyclictest699-21dbus-daemon21:10:000
49562999210,18cyclictest709993-21ssh00:22:170
49564299209,7cyclictest682856-21ntp_kernel_pll_23:50:443
495642992017,2cyclictest722935-21idleruntime-cro00:40:003
495642992017,2cyclictest578183-21latency_hist21:55:013
495642992016,2cyclictest663135-21grep23:29:593
495642992016,2cyclictest627820-21gltestperf22:50:093
49564299201,17cyclictest940-21runrttasks22:55:253
49564299201,17cyclictest718205-21ssh00:33:453
49564299201,17cyclictest676002-21ssh23:41:543
49564299201,16cyclictest711398-21cron00:25:003
495642992010,7cyclictest592188-21taskset22:10:113
495642992010,1cyclictest191rcu_preempt23:11:403
49564099201,6cyclictest567344-21sh21:40:582
495640992015,3cyclictest580042-21unixbench_singl21:55:252
495640992015,3cyclictest0-21swapper/222:35:002
495640992015,3cyclictest0-21swapper/219:15:002
49564099201,17cyclictest594651-21ssh22:12:482
49564099201,16cyclictest721206-21ssh00:36:362
49564099201,16cyclictest695863-21sh00:05:082
49564099201,16cyclictest644128-21ssh23:08:332
495640992011,6cyclictest640863-21ssh23:03:472
495640992011,6cyclictest619288-21date22:40:002
495640992011,6cyclictest532284-21runrttasks20:54:242
495640992011,6cyclictest513491-21diskstats20:00:132
495640992011,6cyclictest262-21systemd-journal21:46:582
49564099200,17cyclictest549802-21ssh21:21:032
495635992016,2cyclictest0-21swapper/120:10:001
495635992015,3cyclictest703-21polkitd21:50:171
495635992015,3cyclictest651109-21ntp_kernel_pll_23:15:181
495635992015,3cyclictest509510-21cpuspeed_turbos19:50:091
495635992015,3cyclictest497788-21apt-key19:15:001
495635992012,6cyclictest638039-21/usr/sbin/munin23:00:181
495635992012,6cyclictest572747-21ssh21:49:041
49563599201,17cyclictest679744-21ssh23:47:181
495635992011,6cyclictest582009-21ssh21:58:321
495635992010,7cyclictest666039-21ssh23:31:291
495635992010,7cyclictest555477-21cron21:30:001
495635992010,7cyclictest555477-21cron21:30:001
49563599200,17cyclictest698483-21ssh00:09:431
49562999206,12cyclictest602003-21ssh22:20:200
495629992016,3cyclictest639475-21runrttasks23:01:410
495629992016,2cyclictest528387-21/usr/sbin/munin20:45:130
495629992011,7cyclictest594032-21ssh22:11:500
49562999201,16cyclictest679711-21taskset23:47:150
495629992010,7cyclictest622690-21ssh22:43:240
495629992010,7cyclictest546797-21sh21:18:400
495629992010,6cyclictest655605-21memory23:20:160
49562999200,6cyclictest599712-21runrttasks22:18:200
49562999200,18cyclictest579289-21cat21:55:170
49564299199,7cyclictest722-21gdbus20:20:513
49564299191,7cyclictest656539-21taskset23:20:393
495642991916,2cyclictest645116-21cron23:10:003
495642991915,2cyclictest0-21swapper/300:00:013
495642991912,5cyclictest550045-21sh21:21:373
49564299191,16cyclictest714181-21sh00:29:103
495642991911,6cyclictest582461-21sh21:59:123
49564299191,16cyclictest539077-21/usr/sbin/munin21:10:203
49564099191,5cyclictest510329-21mailstats19:50:172
495640991912,5cyclictest691445-21sh00:00:112
495640991911,6cyclictest714052-21ssh00:28:532
495640991911,6cyclictest659483-21ssh23:24:532
495640991911,5cyclictest645475-21http23:10:102
495640991910,6cyclictest940-21runrttasks22:45:362
495640991910,6cyclictest665707-21ssh23:31:032
49564099190,17cyclictest630270-21ssh22:52:292
49564099190,16cyclictest547086-21sh21:19:042
49563599199,7cyclictest595303-21taskset22:13:491
49563599198,8cyclictest597790-21sed22:15:251
49563599193,13cyclictest589787-21sh22:07:311
495635991916,2cyclictest534585-21/usr/sbin/munin21:00:201
495635991914,3cyclictest502145-21unixbench_singl19:25:261
495635991912,4cyclictest527132-21ldconfig20:40:171
49563599190,8cyclictest526155-21systemctl20:40:001
49562999199,7cyclictest651014-21ntp_192.168.11423:15:180
495629991916,2cyclictest0-21swapper/019:15:000
495629991915,3cyclictest488-21hwrng21:41:150
495629991914,3cyclictest1-21systemd20:50:530
495629991913,4cyclictest0-21swapper/022:10:000
495629991910,6cyclictest513027-21cron20:04:550
495629991910,6cyclictest262-21systemd-journal19:48:000
49562999190,6cyclictest512102-21modprobe19:55:200
49564299189,7cyclictest610581-21ntpq22:30:193
49564299186,8cyclictest638021-21ldconfig23:00:183
49564299181,7cyclictest510034-21/usr/sbin/munin19:50:133
49564299181,5cyclictest538263-21diskmemload00:06:283
495642991815,2cyclictest526185-21sendmail20:40:003
495642991814,2cyclictest0-21swapper/321:15:163
495642991812,4cyclictest560272-21sshd21:34:513
495642991811,5cyclictest538263-21diskmemload22:36:093
495642991810,5cyclictest508017-21grep19:40:263
49564299180,6cyclictest940-21runrttasks21:26:283
49564299180,6cyclictest940-21runrttasks21:26:273
49564299180,6cyclictest606189-21ssh22:25:183
49564099188,6cyclictest262-21systemd-journal22:16:582
49564099181,7cyclictest583401-21/usr/sbin/munin22:00:222
49564099181,6cyclictest553057-21sh21:26:112
49564099181,6cyclictest553057-21sh21:26:102
49564099181,4cyclictest538263-21diskmemload22:27:002
495640991812,4cyclictest538263-21diskmemload00:13:052
495640991811,5cyclictest517123-21latency_hist20:10:002
49563599189,6cyclictest563798-21sshd21:37:511
49563599181,7cyclictest673706-21taskset23:40:121
495635991814,3cyclictest504130-21unixbench_singl19:30:591
495635991814,2cyclictest513068-21sendmail20:00:011
495635991813,3cyclictest691705-21cpuspeed_turbos00:00:401
495635991813,3cyclictest646365-21pmu-power23:10:261
495635991813,3cyclictest620105-21needreboot22:40:181
495635991812,3cyclictest699-21dbus-daemon20:55:061
49563599180,16cyclictest705421-21ssh00:17:001
49562999187,7cyclictest515988-21ntpq20:05:200
49562999181,15cyclictest538263-21diskmemload23:54:200
495642991715,1cyclictest589738-21sh22:07:263
495642991714,2cyclictest528521-21cpuspeed_turbos20:45:123
495642991713,2cyclictest524500-21needreboot20:31:143
495642991713,2cyclictest0-21swapper/321:00:013
49564299171,2cyclictest0-21swapper/322:23:433
49564299170,6cyclictest5383472date21:10:003
49564299170,15cyclictest624414-21sh22:45:483
49564099177,6cyclictest526587-21df20:40:122
495640991712,3cyclictest537334-21ntpq21:05:482
495640991712,3cyclictest0-21swapper/222:35:132
49564099171,14cyclictest715-21in:imuxsock20:10:582
49564099171,14cyclictest702904-21turbostat00:19:552
49564099170,15cyclictest524762-21/usr/sbin/munin20:31:182
495635991712,3cyclictest703-21polkitd20:10:221
49562999178,6cyclictest533457-21sed20:55:350
49562999175,8cyclictest624174-21sh22:45:220
49562999171,7cyclictest510235-21awk19:50:160
495629991713,2cyclictest0-21swapper/020:40:000
495629991712,3cyclictest506479-21turbostat19:40:000
49562999170,15cyclictest508452-21runrttasks19:44:590
49564299167,6cyclictest262-21systemd-journal19:45:583
495642991614,1cyclictest488-21hwrng20:25:283
495642991614,1cyclictest488-21hwrng20:13:113
495642991614,1cyclictest488-21hwrng20:07:023
49564299161,2cyclictest534585-21/usr/sbin/munin21:00:213
495642991612,2cyclictest530841-21df_inode20:50:453
495642991612,2cyclictest503145-21gltestperf19:30:493
49564299161,13cyclictest498384-21diskstats19:15:113
49564299161,11cyclictest942-21snmpd20:40:113
49564099169,5cyclictest650873-21ssh23:15:162
49564099168,6cyclictest262-21systemd-journal20:20:572
49564099167,6cyclictest536088-21sshd21:02:312
495640991614,1cyclictest488-21hwrng19:30:102
495640991612,2cyclictest0-21swapper/220:15:122
495640991611,3cyclictest0-21swapper/220:55:302
49563599167,6cyclictest531848-21wc20:50:561
49562999168,5cyclictest527044-21/usr/sbin/munin20:40:150
49562999161,5cyclictest698275-21sh00:09:200
495629991612,2cyclictest0-21swapper/019:30:000
49564299151,12cyclictest514036-21ntpq20:00:183
49564099150,6cyclictest507220-21/usr/sbin/munin19:40:172
49563599156,6cyclictest507509-21ntpq19:40:201
49563599155,7cyclictest538036-21seq21:07:181
49562999155,7cyclictest523329-21sshd20:27:240
49562999152,11cyclictest940-21runrttasks19:23:070
49562999151,12cyclictest525156-21/usr/sbin/munin20:31:230
49564299146,6cyclictest496361-21smartctl19:10:143
49564099140,11cyclictest500347-21seq19:22:092
49563599141,9cyclictest500489-21taskset19:24:011
495635991411,2cyclictest0-21swapper/120:30:001
49564299131,8cyclictest500185-21taskset19:20:033
49564099131,8cyclictest940-21runrttasks20:27:482
49564099131,8cyclictest509003-21taskset19:48:522
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional