confdroid/confdroid_haproxy
1
0
Fork 0
Code Issues Actions Packages Projects Releases Wiki Activity

lint params

Browse Source
This commit is contained in:
Arne Teuke
2025-03-02 16:28:57 +01:00
parent 7341c24bc9
commit f09b452830
17 changed files with 165 additions and 275 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
.mdl_style.rb
View File

@@ -1,3 +0,0 @@
all
rule 'MD013', :line_length => 1000
exclude_rule 'MD036'

2
.mdlrc
View File

@@ -1,2 +0,0 @@
style '.mdl_style.rb'

84
.spelling
View File

@@ -1,84 +0,0 @@
# markdown-spellcheck spelling configuration file
# Format - lines begining # are comments
# global dictionary is at the start, file overrides afterwards
# one word per line, to define a file override use ' - filename'
# where filename is relative to this configuration file
Readme.md
httpd
sudo
selinux
site.pp
nodes.pp
cd_apache
::
params
Foreman
cd_resources
CentOS
Puppet
ConfDroid
nagios
vHosts
vHost
phpMyAdmin
phpPgAdmin
parameterized
erb
Confdroid.com
UTF_Files
Elasticsearch
cd_firewall
puppetlabs
cd_elasticsearch
cd_fail2ban
Fail2Ban
intrusion
fail2ban
firewalld
prevention
management
CentOS7
auto-installed
filebeat
Elastic
installer
Foreman
postgresql
TLS
cd_foreman
install.sh
git
cd_git
Gitlab
gitlab
gitlab-ce
gitlab.rb
psql
setup
service
user
optional
postgresql
database
via
tls
encryption
certbot
cd_postgresql
cd_certbot
cd_gitlab
sourcecode
pg_hba.conf
hiera
cd_tls
ca_chain
config
gitlab-ctl
haproxy
cd_concat
concat
cd_haproxy
haproxy.cfg
fqdn
ACLs
Rocky

23
Jenkinsfile vendored
View File

@@ -64,28 +64,11 @@ pipeline {
}
}
stage('markdown-lint') {
steps {
sh '''
source /opt/rh/rh-ruby27/enable
mdl README.md
'''
}
}
stage('spell check') {
steps {
sh '''
mdspell -r -n -a "**/*.md"
'''
}
}
stage('SonarScan') {
stage('SonarScan') {
steps {
sh '''
/opt/sonar-scanner/bin/sonar-scanner \
-Dsonar.projectKey=cd_haproxy \
-Dsonar.projectKey=haproxy_cd \
-Dsonar.sources=. \
-Dsonar.host.url=https://sonarqube.confdroid.com \
-Dsonar.token=squ_af3fb66bbbcccf239ffe9393a5e70874f054be7a
@@ -93,7 +76,7 @@ pipeline {
}
}
stage('create Puppet documention') {
stage('create Puppet documentation') {
steps {
sh '/opt/puppetlabs/bin/puppet strings'
}

18
README.md
View File

@@ -1,6 +1,6 @@
# Readme
|`cd_haproxy`| 1.0.3.0 | [![Build Status](https://pipelines.confdroid.com/buildStatus/icon?job=cd_haproxy)](https://pipelines.confdroid.com/job/cd_haproxy/)|
[![Build Status](https://pipelines.confdroid.com/buildStatus/icon?job=haproxy_cd)](https://pipelines.confdroid.com/job/haproxy_cd/)|
[[_TOC_]]
@@ -8,7 +8,7 @@
`HA Proxy` is a very powerful and popular open source load balancer to balance incoming requests between multiple instances of web-, application- or database servers etc.
`cd_haproxy` automates installation and configuration of `HA proxy` including self-healing, monitoring and firewall etc.
`haproxy_cd` automates installation and configuration of `HA proxy` including self-healing, monitoring and firewall etc.
## WARNING
@@ -29,6 +29,7 @@ CONFIGURATION
* front-end options
* back-end options
* ACL options
* manage fail2ban integration (optional, requires fail2ban_cd module)
SERVICE
@@ -50,13 +51,13 @@ via site.pp or nodes.pp
```bash
node 'example.example.net' {
include cd_haproxy
include haproxy_cd
}
```
* through Foreman:
In order to apply parameters through Foreman, **__cd_haproxy::params__** must be added to the host or host group in question.
In order to apply parameters through Foreman, **__haproxy_cd::params__** must be added to the host or host group in question.
See [more details about class deployment on Confdroid.com](https://confdroid.com/2017/05/deploying-our-puppet-modules/).
@@ -70,7 +71,7 @@ The proxy instances are configured in /etc/haproxy/haproxy.cfg, which is concate
In order to create proxy instances, you will need an external class, which addresses the define, like so:
```bash
cd_haproxy::server::proxy { 'testing':
haproxy_cd::server::proxy { 'testing':
haproxy_fqdn => 'node.example.net',
frontend_name => 'test01-frontend',
frontend_mode => 'http',
@@ -92,8 +93,8 @@ All files and directories are configured with correct selinux context. If selinu
## Support
* OS: CentOS 7, Rocky 9
* Puppet 6
* OS: Rocky 9
* Puppet 8
## Tests
@@ -105,9 +106,6 @@ All files and directories are configured with correct selinux context. If selinu
* `--no-arrow_alignment-check`: this check leads to actually not having am easily readable arrow alignment, as this checks `per block`, not per class.
* Puppet Parser
* ERB Template Parser
* Test for unwanted UTF8 files in the Puppet code (see tests/UTF_Files)
* Markdown-lint
* Spellcheck
* Sonar Quality Gate
## Contact Us

10
manifests/firewall/iptables.pp
View File

@@ -1,15 +1,15 @@
## cd_haproxy::firewall::iptables.pp
# Module name: cd_haproxy
## haproxy_cd::firewall::iptables.pp
# Module name: haproxy_cd
# Author: Arne Teuke (arne_teuke@confdroid.com)
# @summary manage firewall settings through cd_firewall or puppetlabs-firewall
###############################################################################
class cd_haproxy::firewall::iptables (
class haproxy_cd::firewall::iptables (
) inherits cd_haproxy::params {
) inherits haproxy_cd::params {
if ($fqdn == $hy_host_fqdn) and ($hy_manage_fw == true) {
require cd_haproxy::main::files
require haproxy_cd::main::files
firewall { "${hy_fw_order_no}${hy_http_port} tcp port ${hy_http_port}":
proto => 'tcp',

27
manifests/init.pp
View File

@@ -1,25 +1,8 @@
## cd_haproxy::init.pp
# Module name: cd_haproxy
## haproxy_cd::init.pp
# Module name: haproxy_cd
# Author: Arne Teuke (arne_teuke@confdroid.com)
# License:
# This file is part of cd_haproxy.
#
# cd_haproxy is used for providing automatic configuration of HA proxy.
# Copyright (C) 2017 confdroid (copyright@confdroid.com)
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# @summary Class initializes the cd_haproxy module.
# @summary Class initializes the haproxy_cd module.
##############################################################################
class cd_haproxy {
include cd_haproxy::params
class haproxy_cd {
include haproxy_cd::params
}

14
manifests/main/config.pp
View File

@@ -1,18 +1,18 @@
## cd_haproxy::main::config.pp
# Module name: cd_haproxy
## haproxy_cd::main::config.pp
# Module name: haproxy_cd
# Author: Arne Teuke (arne_teuke@confdroid.com)
# @summary Class manages all aspects of configuring the module logic for
# cd_haproxy.
# haproxy_cd.
##############################################################################
class cd_haproxy::main::config (
class haproxy_cd::main::config (
) inherits cd_haproxy::params {
) inherits haproxy_cd::params {
include cd_haproxy::server::service
include haproxy_cd::server::service
if $hy_manage_fail2ban == true {
include cd_haproxy::monitoring::fail2ban
include haproxy_cd::monitoring::fail2ban
}
}

12
manifests/main/dirs.pp
View File

@@ -1,14 +1,14 @@
## cd_haproxy::main::dirs.pp
# Module name: cd_haproxy
## haproxy_cd::main::dirs.pp
# Module name: haproxy_cd
# Author: Arne Teuke (arne_teuke@confdroid.com)
# @summary Class manages all directories required for cd_haproxy.
# @summary Class manages all directories required for haproxy_cd.
###############################################################################
class cd_haproxy::main::dirs (
class haproxy_cd::main::dirs (
) inherits cd_haproxy::params {
) inherits haproxy_cd::params {
if $fqdn == $hy_host_fqdn {
require cd_haproxy::main::user
require haproxy_cd::main::user
# main dir
file { $hy_main_dir:

14
manifests/main/files.pp
View File

@@ -1,14 +1,14 @@
## cd_haproxy::main::files.pp
# Module name: cd_haproxy
## haproxy_cd::main::files.pp
# Module name: haproxy_cd
# Author: Arne Teuke (arne_teuke@confdroid.com)
# @summary Class manages all configuration files required for cd_haproxy.
# @summary Class manages all configuration files required for haproxy_cd.
##############################################################################
class cd_haproxy::main::files (
class haproxy_cd::main::files (
) inherits cd_haproxy::params {
) inherits haproxy_cd::params {
if $fqdn == $hy_host_fqdn {
require cd_haproxy::main::dirs
require haproxy_cd::main::dirs
# create the concat target
concat {$hy_main_config:
@@ -154,7 +154,7 @@ class cd_haproxy::main::files (
selrole => object_r,
seltype => syslog_conf_t,
seluser => system_u,
content => template('cd_haproxy/rsyslog/10_haproxy.conf.erb'),
content => template('haproxy_cd/rsyslog/10_haproxy.conf.erb'),
# notify => Service['rsyslog'], # only if using cd_rsyslog
}
}

10
manifests/main/install.pp
View File

@@ -1,12 +1,12 @@
## cd_haproxy::main::install.pp
# Module name: cd_haproxy
## haproxy_cd::main::install.pp
# Module name: haproxy_cd
# Author: Arne Teuke (arne_teuke@confdroid.com)
# @summary Class manage all aspects of installing binaries required for
# cd_haproxy
# haproxy_cd
###############################################################################
class cd_haproxy::main::install (
class haproxy_cd::main::install (
) inherits cd_haproxy::params {
) inherits haproxy_cd::params {
if $fqdn == $hy_host_fqdn {
package {$reqpackages:

12
manifests/main/user.pp
View File

@@ -1,14 +1,14 @@
## cd_haproxy::main::user.pp
# Module name: cd_haproxy
## haproxy_cd::main::user.pp
# Module name: haproxy_cd
# Author: Arne Teuke (arne_teuke@ConfDroid.com)
# @summary Class manages service users for cd_haproxy.
# @summary Class manages service users for haproxy_cd.
#############################################################################
class cd_haproxy::main::user (
class haproxy_cd::main::user (
) inherits cd_haproxy::params {
) inherits haproxy_cd::params {
if $fqdn == $hy_host_fqdn {
require cd_haproxy::main::install
require haproxy_cd::main::install
# manage user
user { $hy_user_name:

10
manifests/monitoring/fail2ban.pp
View File

@@ -1,10 +1,10 @@
# Module name: cd_haproxy
# Module name: haproxy_cd
# Author: Arne Teuke (arne_teuke@confdroid.com)
# @summary Class manages fail2ban settings
##############################################################################
class cd_haproxy::monitoring::fail2ban (
class haproxy_cd::monitoring::fail2ban (
) inherits cd_haproxy::params {
) inherits haproxy_cd::params {
if $hy_manage_fail2ban == true {
@@ -21,7 +21,7 @@ class cd_haproxy::monitoring::fail2ban (
selrole => object_r,
seltype => etc_t,
seluser => system_u,
content => template('cd_haproxy/fail2ban/f2b_haproxy.conf.erb'),
content => template('haproxy_cd/fail2ban/f2b_haproxy.conf.erb'),
notify => Service['fail2ban'],
}
@@ -34,7 +34,7 @@ class cd_haproxy::monitoring::fail2ban (
selrole => object_r,
seltype => etc_t,
seluser => system_u,
content => template('cd_haproxy/fail2ban/010_jail.d_haproxy.conf.erb'),
content => template('haproxy_cd/fail2ban/010_jail.d_haproxy.conf.erb'),
notify => Service['fail2ban'],
}
}

172
manifests/params.pp
View File

@@ -1,21 +1,21 @@
## cd_haproxy::params.pp
# Module name: cd_haproxy
## haproxy_cd::params.pp
# Module name: haproxy_cd
# Author: Arne Teuke (arne_teuke@confdroid.com)
# @param [string] pkg_ensure
# which [package type](https://confdroid.com/2017/05/puppet-type-package/)
# to choose, i.e. `latest` or `present`.
# @param [string] reqpackages specify th epackages to be installed
# @param [string] hy_host_fqdn the fqdn of the ha proxy host. mmust be fqdn,
# @param [string] reqpackages specify the packages to be installed
# @param [string] hy_host_fqdn the fqdn of the ha proxy host. must be fqdn,
# not cname. specify multiple fqdns via array.
# @param [boolean] hy_manage_fw whether to manage the firewall.
# @param [string] hy_fw_order_no the rule number to control the oder of the
# @param [string] hy_fw_order_no the rule number to control the order of the
# firewall rules.
# @param [string] hy_http_port the http port. used in firewall settings
# @param [string] hy_https_port the https port. used in firewall settings
# @param [string] hy_user_name the name of the haproxy service user.
# @param [string] hy_user_comment the comment of the haproxy user, shows up
# in emails sent from haproxy.
# @param [string] hy_user_home the home directory for the haprxy user.
# @param [string] hy_user_home the home directory for the haproxy user.
# @param [string] hy_user_shell the shell for the haproxy user.
# @param [string] hy_log_target Where to send the logs for this haproxy.
# Currently only local logging / rsyslog forwarding supported.
@@ -50,96 +50,112 @@
# client side.
# @param [string] hy_timeout_server Set the maximum inactivity time on the
# server side.
# @param [string] hy_timeout_http_keep_alive set the timeout value for
# @param [string] hy_timeout_http_keep_alive set the timeout value for
# keeping https connections alive
# @param [string] hy_timeout_check set the timeout value for how long to check
# if the timeout has occured
# if the timeout has occurred
# @param [boolean] hy_manage_fail2ban whether to integrate fail2ban
# @param [boolean] hy_use_http_server_close whether to use this in global mode
# @param [boolean] hy_use_forward_for whether to use this in global mode
# @param [boolean] hy_use_redispatch whether to use this in global mode
# @param [string] hy_max_retries max retries value
# @param [string] hy_timeout_connect timeout for connections
# @param [string] hy_timeout_client timeout for client connections
# @param [string] hy_timeout_server timeout for server response
# @param [boolean] hy_hard_stop whether to use hard stop for service restart
# @param [string] hy_hard_stop_value which value to use for hard stop
# @param [string] hy_stats_auth credentials to use for stats authentication
# @param [string] hy_log_local0 settings for logging to local0
# @param [string] hy_log_local1 settings for logging to local1
# @param [boolean] hy_use_tcplog whether to use tcplog as default
# @param [boolean] hy_use_httplog whether to use httplog as default
# @param [boolean] hy_use_dontlognull whether to use dontlognull as default
# @param [string] hy_be_userlist backend userlist comment
# @param [boolean] hy_manage_be_users whether to manage backend users
###############################################################################
class cd_haproxy::params (
class haproxy_cd::params (
$pkg_ensure = 'latest',
$reqpackages = ['haproxy','httpd-tools'],
String $pkg_ensure = 'latest',
String $reqpackages = ['haproxy','httpd-tools'],
$hy_host_fqdn = undef,
String $hy_host_fqdn = undef,
# firewall
$hy_manage_fw = true,
$hy_fw_order_no = '50',
Boolean $hy_manage_fw = true,
String $hy_fw_order_no = '50',
# fail2ban
$hy_manage_fail2ban = false,
Boolean $hy_manage_fail2ban = false,
# main config
$hy_http_port = '80',
$hy_https_port = '443',
$hy_chroot = '/var/lib/haproxy',
$hy_pid = '/var/run/haproxy.pid',
$hy_maxconn = '4000',
$hy_show_stats = false,
$hy_stats_socket = '/var/lib/haproxy/stats',
$hy_default_mode = 'tcp',
$hy_use_http_server_close = false,
$hy_use_forward_for = false,
$hy_use_redispatch = true,
$hy_max_retries = '3',
$hy_timeout_http_request = '10s',
$hy_timeout_queue = '1m',
$hy_timeout_connect = '10s',
$hy_timeout_client = '1m',
$hy_timeout_server = '1m',
$hy_timeout_http_keep_alive = '10s',
$hy_timeout_check = '10s',
$hy_hard_stop = true,
$hy_hard_stop_value = '60s',
String $hy_http_port = '80',
String $hy_https_port = '443',
String $hy_chroot = '/var/lib/haproxy',
String $hy_pid = '/var/run/haproxy.pid',
String $hy_maxconn = '4000',
Boolean $hy_show_stats = false,
String $hy_stats_socket = '/var/lib/haproxy/stats',
String $hy_default_mode = 'tcp',
Boolean $hy_use_http_server_close = false,
Boolean $hy_use_forward_for = false,
Boolean $hy_use_redispatch = true,
String $hy_max_retries = '3',
String $hy_timeout_http_request = '10s',
String $hy_timeout_queue = '1m',
String $hy_timeout_connect = '10s',
String $hy_timeout_client = '1m',
String $hy_timeout_server = '1m',
String $hy_timeout_http_keep_alive = '10s',
String $hy_timeout_check = '10s',
Boolean $hy_hard_stop = true,
String $hy_hard_stop_value = '60s',
String $hy_stats_auth = 'admin:password',
# user
$hy_user_name = 'haproxy',
$hy_user_comment = 'haproxy user',
$hy_user_home = '/var/lib/haproxy',
$hy_user_shell = '/sbin/nologin',
String $hy_user_name = 'haproxy',
String $hy_user_comment = 'haproxy user',
String $hy_user_home = '/var/lib/haproxy',
String $hy_user_shell = '/sbin/nologin',
# logging
$hy_log_local0 = '127.0.0.1:514 local0',
$hy_log_local1 = '127.0.0.1:514 local1 notice',
$hy_log_target = '127.0.0.1',
$hy_log_facility = 'local2',
$hy_log_default = 'global',
$hy_use_tcplog = true,
$hy_use_httplog = false,
$hy_use_dontlognull = true,
$hy_manage_be_users = false,
$hy_be_userlist = '####',
String $hy_log_local0 = '127.0.0.1:514 local0',
String $hy_log_local1 = '127.0.0.1:514 local1 notice',
String $hy_log_target = '127.0.0.1',
String $hy_log_facility = 'local2',
String $hy_log_default = 'global',
Boolean $hy_use_tcplog = true,
Boolean $hy_use_httplog = false,
Boolean $hy_use_dontlognull = true,
Boolean $hy_manage_be_users = false,
String $hy_be_userlist = '####',
) {
# service
$hy_service = 'haproxy'
# service
$hy_service = 'haproxy'
# directories
$hy_main_dir = '/etc/haproxy'
$hy_errors_dir = "${hy_main_dir}/errors"
# directories
$hy_main_dir = '/etc/haproxy'
$hy_errors_dir = "${hy_main_dir}/errors"
# files
$hy_main_config = "${hy_main_dir}/haproxy.cfg"
$hy_config_head_erb = 'cd_haproxy/haproxy_head.erb'
$hy_config_tail_erb = 'cd_haproxy/haproxy_tail.erb'
$hy_400_file = "${hy_errors_dir}/400.http"
$hy_400_erb = 'cd_haproxy/errors/400_http.erb'
$hy_403_file = "${hy_errors_dir}/403.http"
$hy_403_erb = 'cd_haproxy/errors/403_http.erb'
$hy_408_file = "${hy_errors_dir}/408.http"
$hy_408_erb = 'cd_haproxy/errors/408_http.erb'
$hy_500_file = "${hy_errors_dir}/500.http"
$hy_500_erb = 'cd_haproxy/errors/500_http.erb'
$hy_502_file = "${hy_errors_dir}/502.http"
$hy_502_erb = 'cd_haproxy/errors/502_http.erb'
$hy_503_file = "${hy_errors_dir}/503.http"
$hy_503_erb = 'cd_haproxy/errors/503_http.erb'
$hy_504_file = "${hy_errors_dir}/504.http"
$hy_504_erb = 'cd_haproxy/errors/504_http.erb'
# includes must be last
include cd_haproxy::main::config
# files
$hy_main_config = "${hy_main_dir}/haproxy.cfg"
$hy_config_head_erb = 'haproxy_cd/haproxy_head.erb'
$hy_config_tail_erb = 'haproxy_cd/haproxy_tail.erb'
$hy_400_file = "${hy_errors_dir}/400.http"
$hy_400_erb = 'haproxy_cd/errors/400_http.erb'
$hy_403_file = "${hy_errors_dir}/403.http"
$hy_403_erb = 'haproxy_cd/errors/403_http.erb'
$hy_408_file = "${hy_errors_dir}/408.http"
$hy_408_erb = 'haproxy_cd/errors/408_http.erb'
$hy_500_file = "${hy_errors_dir}/500.http"
$hy_500_erb = 'haproxy_cd/errors/500_http.erb'
$hy_502_file = "${hy_errors_dir}/502.http"
$hy_502_erb = 'haproxy_cd/errors/502_http.erb'
$hy_503_file = "${hy_errors_dir}/503.http"
$hy_503_erb = 'haproxy_cd/errors/503_http.erb'
$hy_504_file = "${hy_errors_dir}/504.http"
$hy_504_erb = 'haproxy_cd/errors/504_http.erb'
# includes must be last
include haproxy_cd::main::config
}

14
manifests/server/proxy.pp
View File

@@ -1,7 +1,7 @@
## cd_haproxy::server::proxy.pp
# Module name: cd_haproxy
## haproxy_cd::server::proxy.pp
# Module name: haproxy_cd
# Author: Arne Teuke (arne_teuke@ConfDroid.com)
# @summary Define manages the proxies for cd_haproxy.
# @summary Define manages the proxies for haproxy_cd.
# @param [string] haproxy_fqdn the fqdn of the haproxy server in question.
# @param [string] frontend_name the name for the frontend section rule.
# @param [string] frontend_order the order where the concat should appear
@@ -21,7 +21,7 @@
# @param [string] fe_maxconn the maximum connections for the lb instance.
# @param [string] be_server_name set the name for the backend server
###############################################################################
define cd_haproxy::server::proxy (
define haproxy_cd::server::proxy (
$haproxy_fqdn = undef,
$frontend_name = undef,
@@ -52,9 +52,9 @@ define cd_haproxy::server::proxy (
) {
$hy_main_config = '/etc/haproxy/haproxy.cfg'
$hy_frontendrule = 'cd_haproxy/haproxy_frontend_rule.erb'
$hy_backendrule = 'cd_haproxy/haproxy_backend_rule.erb'
$hy_acl_rule = 'cd_haproxy/haproxy_acl_rule.erb'
$hy_frontendrule = 'haproxy_cd/haproxy_frontend_rule.erb'
$hy_backendrule = 'haproxy_cd/haproxy_backend_rule.erb'
$hy_acl_rule = 'haproxy_cd/haproxy_acl_rule.erb'

12
manifests/server/service.pp
View File

@@ -1,15 +1,15 @@
## cd_haproxy::server::service.pp
# Module name: cd_haproxy
## haproxy_cd::server::service.pp
# Module name: haproxy_cd
# Author: Arne Teuke (arne_teuke@confdroid.com)
# @summary Class manages the service(s) for cd_haproxy.
# @summary Class manages the service(s) for haproxy_cd.
#############################################################################
class cd_haproxy::server::service (
class haproxy_cd::server::service (
) inherits cd_haproxy::params {
) inherits haproxy_cd::params {
if $fqdn == $hy_host_fqdn {
require cd_haproxy::firewall::iptables
require haproxy_cd::firewall::iptables
service { $hy_service:
ensure => running,

3
templates/haproxy_head.erb
View File

@@ -60,7 +60,7 @@ listen stats
stats enable
stats uri /haproxy?stats
stats realm Strictly\ Private
stats auth local_admin:jSAVu62MV+kr+>R. # Use a strong password
stats auth <%= @hy_stats_auth %>
stats refresh 30s
stats admin if TRUE # Allow admin actions if logged in
@@ -70,4 +70,3 @@ listen stats
### custom rules below ###