How to keep 20 thousand VPN clients on servers for $5

a Month ago my friends made a free service to bypass block sites in Ukraine Zaborona.Help. During this time, the service has become quite popular, the audience grew to 20,000 users. The number of concurrent connections during peak hours — ≈6 000 clients.

The main feature of our service that is routed through the VPN only traffic to blocked networks, other sites work directly. This does not affect the speed of the Internet and is not a substitute IP address for other sites.

The article describes the intricacies of OpenVPN settings for a large number of customers for a cheap VPS.

the
    the
  • How to choose a suitable hosting. The distinctive features of bad hosting. The story of how we've been looking for and found a hosting in Russia.
  • the
  • Why IPv6 is good. Proper configuration of IPv6 addresses for VPN clients.
  • the
  • Change the configuration of OpenVPN on the fly, without restarting the server and disconnecting client
  • the
  • load Balancing between the servers and the OpenVPN processes
  • the
  • Fine tuning Linux for large numbers of connections
  • the
  • Features the curves of the operating systems and routers users

Our experience will be useful for those who are going to deploy a VPN for your personal needs, and those who want to create service with a large number of clients.

the

Hosting


At first, we used several servers in Europe, the Scaleway providers, Linode, DigitalOcean. This was the cheapest VPS for $3-5. Immediately users began complaining that due to European IP addresses Yandex.Music and music VK.com not available. We started to look for a suitable hosting service in the CIS.

It turned out that domestic hosters comparable level of services with the European, does not exist. Almost always low quality services at high price, a complicated procedure of ordering, outdated technology. During the search, we formed a list the hallmarks of bad hosting.

the

Distinguishing features of poor hosting


the

    the use of the term VDS is a VPS. Though the acronym is fine, it's kind of a black mark, which virtually guarantees low level of service and control panel interface of the 90-ies.

    Panel BillManager, ISPmanager, etc. — the standard is overly complicated interface with lots of buttons, confusing menu. The ordering process servers in such panels is performed in several stages. The control panel server itself usually located on a separate panel custom subdomain, with a separate login password, the transition to which is not the most obvious way. The order process or modification becomes a real pain, often requiring calling tech support. If the ordering process servers requires a few more clicks and takes more than two minutes is bad hosting.

    control panel BillManager
    Interface control panel BillManager is a hallmark of outdated hosting

    lack of understanding of the principles of IPv6. Many hosters allocate one IPv6 address and charge you for every extra. Network /64 would cost bazzilions of dollars.

    Virtualization OpenVZ.Most commercial VPS providers use OpenVZ venet interface, and panels, which do not allow you to assign a subnet of IPv6 addresses on a separate container, and only one individual address (/128) on the interface. These addresses are not OK to distribute to VPN clients.


After trying a few hosting sites, we almost despaired. It seemed that in Russia simply there is no normal hosting provider for our needs. I sent out a letter to all hosters I could find on websites-aggregators hosting, which described our requirements and asked for free to give us a site in exchange for advertising on our website.

the

Our VPS servers


the

    Powerful server-class CPU. Some providers, like Scaleway, offer budget VPS servers on low-power ARM processors or Intel Atom. There are even servers running on VIA processors. On such systems, OpenVPN is slow, but not because of encryption, as one would assume. The tun module, which is used by OpenVPN to create a network interface, not optimized for high loads: it adopts or gives only a single packet per system call, which causes a large number of context switches between kernel mode and user. The slower the frequency of memory and a cheaper processor, the slower the switching. In addition, in the code, OpenVPN uses the recv and send, which operates the individual network packets, causing the sending of encrypted packets, too, is not the most optimal way. So for normal OpenVPN it is important to have a fast processor and memory.

    Unlimited traffic and good channels. Users consume a lot of media content in social networks, the traffic is consumed very quickly. Tariffs with low traffic quota (1TB) consumed per day.

    Private routed network IPv6 is essential for the direct allocation of real addresses to clients. Most hosters don't even know what that means, and simply assign the network interface of the hypervisor to the desired subnet, and do not create an entry in the routing table using the existing (or via link-local-address). It is not possible without crutches issued to assign the range to the network interface VPN and issue IPv6 addresses directly to customers: the hypervisor host send NDP request (similar to ARP for IPv6) virtual machine, not virtual reality find yourself at this address, do not answer the query and did not work. There are NDP-proxy that allow you to work around this problem, but it is inconvenient, and because of this may create additional load on the hypervisor and routers host.


At our request responded to about a dozen companies, but almost all of them had signs of poor hosting and did not fit. As a result, we found ONLY hosting provider that meets all of our needs.

It sounds strange, but I'm willing to convincingly demonstrate in the review that 99% of the domestic hosting providers — bullshit, nedotyagivaet to the level of Europe. Especially I will be glad to communicate with representatives of large companies like Masterhost, REG.ru, 1GB.ru that Timeweb.

the

How we got to be friends with Veesp.com


A real discovery for us was the host Veesp.com with a datacenter in Saint Petersburg. This is the only host who knew how to cook IPv6. Each VPS server is allocated a network /64 on request /56.

They have two series of VPS. We use the rate "Storage 1", with unlimited traffic. Servers at this rate, I have the Intel Xeon X5650. There is also a range of tariffs "Compute", SSD drive, powerful Intel Xeon E5v4 and DDR4 memory.

Range of rates Compute VPS provider Veesp.com
Range of rates Compute VPS provider Veesp.com

Control panel, comparable in convenience with DigitalOcean. There is even a Bitcoin payment!

At the moment we're fully moved in to the site Veesp.com and use six servers Storage 1 for OpenVPN. Music in VK and Yandex is working again, users are satisfied.

the

IPv6


I love IPv6. It gets rid of unnecessary entities, like NAT and port forwarding. For VPN it is convenient because every customer goes to the Internet with your real IP address. Unfortunately, many hosting providers and system administrators do not like this Protocol because of this it does not correctly adjust and use.

The most common mistake is to give one IPv6 address to the server.

the

Why you need to allocate a minimum of /64 for each end node


the
    the
  • For simplicity. According to RFC 6177, the network /64 is the recommended unit for all end nodes, be it home Internet or hosting. It 2⁶⁴, or eighteen quintillion IP addresses. This approach will eliminate confusion and guessing exactly how the network is configured on each individual node.

  • the
  • Less memory usage on the routers. Network administrator, it is sufficient to route to your one big subnet instead of several small ones.

  • the
  • does Not break the Protocol autoconfiguration SLAAC. If you suddenly want to make a complete local network (L2) on top of the Internet, it will work correctly with IPv6.

  • According to the logic of Email service providers, large sites like Google and Facebook, one customer, one /64 network. Therefore, if the host gives different addresses from one range /64, you can block the actions committed by the neighbors on the range.

    IPv6 addresses must not be sold individually. This is nonsense, because the minimum recommended a /64 block, which must be issued free of charge, would cost bazzilions of money even if the price per IP address will be one ruble.


To give clients VPN IPv6 addresses directly without the need for crutches, need to have a separate routable network via the IPv6 address on the server, that is the subnet from which you plan to distribute addresses to clients should not be assigned to the server interface, but must be routed via any address on the network interface of the server.

There are two common variant distribution marshrutizatora networks.

First option: one /64 on network interface, and a /56 routed through the first /64. You can beat /56 the way you want, or assign the whole /56 on the VPN interface.

Second option: one or more /64 (or more), marshrutizatora using link-local address.

Most of the hosting companies dedicated network must be ordered separately. Veesp.com module generates the /56 on each server for free. Unfortunately, even the advanced hosting companies like DigitalOcean, do not provide such services. Here is a list of hosters providing this service version6.ru/vps, @rm.

the

NAT and Firewall


On servers zabroni more than two hundred of iptables rules for a few on each band from the list of prohibited network in Ukraine.

Complex set of rules inconvenient to maintain with standard tools iptables-save and iptables-restore, because if you change one rule, you will need to edit hundreds of similar lines.

the

Simplify writing firewall rules using ferm


Ferm — a utility for managing complex iptables rules with its syntactic sugar. It allows you to create human readable configuration iptables, use variables, arrays and loops, but does not restrict you, unlike other add-on to iptables: you can do absolutely everything, to use all features and modules of netfilter.

A simple example: we want to block incoming connections from several thousand networks in three network interfaces eth0, eth1, eth2. In the normal case one would have to write thousands of the same rules for each interface.

Here is how this problem is solved through ferm:

the
@def $WAN_0 = eth0;
@def $WAN_1 = eth1;
@def $WAN_2 = eth2;

@def $BLOCKED_NETWORKS = (
123.123.123.123
234.234.234.234
....
);


chain INPUT {
saddr $BLOCKED_NETWORKS of ($WAN_0 $WAN_1 $WAN_2) DROP;
}

This will create a rule for each address $BLOCKED_NETWORKS for each network interface. This notation is easy to read and takes three times less lines. It is easier to edit.

In our case, iptables performs three functions: NAT for IPv4 connections, restricting access to other networks and load balancing between the processes is OpenVPN. Let us examine each task in detail.

Clients are provided with a "grey" IP address of the range 192.168.*.*, so to route it directly to the Internet impossible. In order to produce customers to the IPv4 Internet, you need to use network address translation (NAT).

The same does your home WiFi router. On every outgoing connection, the client creates a stream in memory on the server, which stores information about who belongs to which connection. Despite the fact that this operation requires a little server resource, with a huge number of connections (tens of thousands), this can become a demanding task for the weak servers.

This is not a problem in IPv6, as each VPN client is given the real IP address that is directly routable to the Internet. The server remains only to transfer packets from one interface to another, without creating broadcast and store information about each connection.

Restriction of access to other networks


Any user can add the option redirect-gateway in the config of OpenVPN and use our servers as the default route. You need to close access to all the networks, but blocked in Ukraine.

load Balancing between the processes is OpenVPN


The OpenVPN architecture is single-threaded, so a single process daemon OpenVPN uses only one core of the processor. For the best utilization of all cores, number of processes should equal the number of cores on the server. Balancing between processes is performed using the module statistic, which randomly redirects incoming connections on different ports.

the

Balancing between servers


We use the simplest method of balancing — using multiple A-records in DNS. All clients connect to the server via the domain name vpn.zaborona.help. This domain has six A-records, which are directed to all servers. In the end, at the time of connection, the client chooses a random server from available. So the total number of connections is evenly distributed between all servers. OpenVPN configuration on all servers identical, with the exception of ranges of IPv6 addresses issued to customers.

image
control Panel DNS entries. Balancing using multiple A-records. Low TTL value allows you to quickly remove addresses from the list.

You can quickly see which IP addresses will resolvase domain from different points of the planet, using the service host-tracker.com by choosing any of the tests, for example, http or ping.

The result for the domain vpn.zaborona.help: www.host-tracker.com/InstantCheck/ResultComplete/ec0e5a90-ed56-e711-b124-0003ff7328cc

the

Recursive DNS resolvers


Many Ukrainian ISPs block DNS queries to the forbidden domains by intercepting requests including third-party DNS servers, like 8.8.8.8. It is therefore important that clients send DNS queries through the VPN tunnel, where they will not be able to modify the provider.
It's not easy to do, because new versions of Windows, the queries can go to all DNS servers in the system and will use the one that will respond faster. The DNS server provider is physically closer to the customer, therefore, with high probability, the system will get replaced by the answer provider, not the correct answer to the DNS inside the VPN, and the website will remain blocked.

To solve this problem ValdikSS written by a patch for OpenVPN that blocks DNS leaks in Windows. It uses the Windows Filtering Platform is a special layer of Windows firewall to block requests to third-party DNS while OpenVPN is running.

the

Deploy server


In the comments to the previous article I criticized the fact that some of the stages that I thought it was obvious, not described. Here I will describe the complete set of commands for deployment of servers zabobony.
Configs uniform as possible and can be deployed without changes on the server. This is useful when ispolzovanie automatic deployment tools like Ansible.

The only unique data for each server — subnet IPv6-signed in the configuration file of OpenVPN.

On servers, using Ubuntu 16.04 LTS with the core 4.4.0.

package Installation


In the repository of the distro contains an outdated version of OpenVPN 2.3, so add the official repository of the project with OpenVPN 2.4. All the other packages put standard delivery.

the
wget -O - https://swupdate.openvpn.net/repos/repo-public.gpg/apt-key add - 
echo "deb http://build.openvpn.net/debian/openvpn/release/2.4 xenial main" > /etc/apt/sources.list.d/openvpn-aptrepo.list
apt update
apt upgrade
apt install openvpn dnsmasq ferm

OpenVPN Configuration


Since all servers are configured identically, just copy the configs together with the key files and certificates in the folder /etc/openvpn. No generate keys/certificates is not performed. The number of processes OpenVPN configures the number of cores on the server, in our case 2. Each separate process has its own config: zaborona1.conf and zaborona2.conf.

To avoid confusion, I will call the process OpenVPN daemons, but the fact it is several individual servers running inside a VPS.

The list of files in /etc/openvpn

the
/etc/openvpn/zaborona1.conf # config first demon
/etc/openvpn/zaborona2.conf # config for the second daemon

/etc/openvpn/ccd/DEFAULT # the file variable on the fly settings for the first demon
/etc/openvpn/ccd2/DEFAULT # the file variable on the fly settings for the second demon

/etc/openvpn/logs # log directory


/etc/openvpn/ca.crt # root certificate
/etc/openvpn/zaborona.help.crt # server certificate
/etc/openvpn/zaborona.help.key # server key
/etc/openvpn/dh2048.pem # file with Diffie-Hellman groups (required for encryption)

File contents:

zaborona1.conf
mode server
# Despite the fact that UDP is faster, we use TCP to save traffic and battery of mobile devices, as in the UDP mode, the client is forced often to send keep-alive packets to maintain NAT translation on the router or CGNAT.
proto tcp

# Type of encapsulation L3, i.e. the ip level. We do not need the L2 level.
dev-type tun
# Name of the tun interface on the server
dev zaborona1

# Give the client addresses from a total of 24, not cut on a subnet /30.
topology subnet

# The range of "gray" ipv4 addresses issued to customers. 
server 192.168.224.0 255.255.252.0
# Range of real ipv6 addresses issued to customers. Unique for each server.
server-ipv6 2a00:1838:32:200::/112

txqueuelen 250
keepalive 300 900
persist-tun
persist-key

cipher AES-128-CBC
ncp-ciphers: AES-128-GCM

#user nobody
duplicate-cn

log logs/zaborona1.log
logs status/status1.log 30

# The path to the folder with the configuration of users. This option allows you to assign individual options for each user. The file is re-read afresh for each client connection so that changes can be performed without restarting the server.
client-config-dir ccd

ca ca.crt
cert zaborona.help.crt
key zaborona.help.key
dh dh2048.pem


zaborona2.conf
mode server
port 1195
proto tcp

dev-type tun
dev zaborona2

topology subnet
server 192.168.228.0 255.255.252.0
server-ipv6 2a00:1838:32:280::/112

txqueuelen 250
keepalive 300 900
persist-tun
persist-key

cipher AES-128-CBC
ncp-ciphers: AES-128-GCM

#user nobody
duplicate-cn

log logs/zaborona2.log
status logs/status2.log 30
client-config-dir ccd2

ca ca.crt
cert zaborona.help.crt
key zaborona.help.key
dh dh2048.pem


ccd/DEFAULT
push "dhcp-option DNS 192.168.224.1"

push "dhcp-option DNS 74.82.42.42" # HE.net DNS 
push "route 74.82.42.42" # Route to HE.net DNS

push "route try accessing 77.88.8.8" # Route to DNS Yandex

push "dhcp-option DNS6 2001:4860:4860::8888" # Google IPv6 dns
push "route-ipv6 2001:4860:4860::8888"

push "dhcp-option DNS6 2001:4860:4860::8844" # Google IPv6 dns
push "route-ipv6 2001:4860:4860::8844"

#Persist TUN
push "persist-tun"

# Routes

# Yandex network
push "route 5.45.192.0 255.255.192.0"
push "route 5.255.192.0 255.255.192.0"
push "route 37.9.64.0 255.255.192.0"
push "route 37.140.128.0 255.255.192.0"
push "route 77.75.152.0 255.255.248.0"
push "route 77.88.0.0 255.255.192.0"
push "route 84.201.128.0 255.255.192.0"
push "route 87.250.224.0 255.255.224.0"
push "route 93.158.128.0 255.255.192.0"
push "route 95.108.128.0 255.255.128.0"
push "route 100.43.64.0 255.255.224.0"
push "route 109.235.160.0 255.255.248.0"
push "route 130.193.32.0 255.255.224.0"
push "route 141.8.128.0 255.255.192.0"
push "route 178.154.128.0 255.255.128.0"
push "route 185.32.185.0 255.255.255.0"
push "route 185.32.186.0 255.255.255.0"
push "route 185.71.76.0 255.255.252.0"
push "route 199.21.96.0 255.255.252.0"
push "route 199.36.240.0 255.255.252.0"
push "route 213.180.192.0 255.255.224.0"

push "route-ipv6 2001:678:384::/48"
push "route-ipv6 2620:10f:d000::/44"
push "route-ipv6 2a02:6b8::/32"
push "route-ipv6 2a02:5180::/32"

# Mail.ru network
push "route 5.61.16.0 255.255.248.0"
push "route 5.61.232.0 255.255.248.0"
push "route 79.137.157.0 255.255.255.0"
push "route 79.137.183.0 255.255.255.0"
push "route 94.100.176.0 255.255.240.0"
push "route 95.163.32.0 255.255.224.0"
push "route 95.163.248.0 255.255.248.0"
push "route 128.140.168.0 255.255.248.0"
push "route 178.22.88.0 255.255.248.0"
push "route 178.237.16.0 255.255.240.0"
push "route 185.5.136.0 255.255.252.0"
push "route 185.16.148.0 255.255.252.0"
push "route 185.16.244.0 255.255.252.0"
push "route 188.93.56.0 255.255.248.0"
push "route 194.186.63.0 255.255.255.0"
push "route 195.211.20.0 255.255.252.0"
push "route 195.211.128.0 255.255.252.0"
push "route 195.218.168.0 255.255.255.0"
push "route 208.87.92.0 255.255.252.0"
push "route 217.20.144.0 255.255.240.0"
push "route 217.69.128.0 255.255.240.0"
push "route 185.6.244.0 255.255.252.0"
push "route 185.30.176.0 255.255.252.0"
push "route 195.218.190.0 255.255.254.0"

push "route-ipv6 2a00:1148::/32"
push "route-ipv6 2a00:a300::/32"
push "route-ipv6 2a00:b4c0::/32"
push "route-ipv6 2a04:4b40::/29"

# VK.com network
push "route 87.240.128.0 255.255.192.0"
push "route 93.186.224.0 255.255.240.0"
push "route 95.142.192.0 255.255.240.0"
push "route 95.213.0.0 255.255.192.0"
push "route 185.29.130.0 255.255.255.0"
push "route 185.32.248.0 255.255.252.0"

# Kaspersky network
push "route 77.74.176.0 255.255.252.0"
push "route 77.74.181.0 255.255.255.0"
push "route 77.74.183.0 255.255.255.0"
push "route 93.159.228.0 255.255.252.0"
push "route 185.54.220.0 255.255.254.0"
push "route 185.85.12.0 255.255.255.0"
push "route 185.85.14.0 255.255.254.0"
push "route 77.74.176.0 255.255.248.0"
push "route 91.103.64.0 255.255.248.0"
push "route 93.159.224.0 255.255.248.0"

push "route-ipv6 2a03:2480::/33"

# DrWeb
push "route 255.255.255.255 178.248.232.183"
push "route 255.255.255.255 178.248.233.94"
push "route 195.88.252.0 255.255.254.0"


ccd2/DEFAULT
push "dhcp-option DNS 192.168.228.1"

push "dhcp-option DNS 74.82.42.42" # HE.net DNS 
push "route 74.82.42.42" # Route to HE.net DNS

push "route try accessing 77.88.8.8" # Route to DNS Yandex

push "dhcp-option DNS6 2001:4860:4860::8888" # Google ipv6 dns
push "route-ipv6 2001:4860:4860::8888"

push "dhcp-option DNS6 2001:4860:4860::8844" # Google ipv6 dns
push "route-ipv6 2001:4860:4860::8844"

#Persist TUN
push "persist-tun"

# Routes

# Yandex network
push "route 5.45.192.0 255.255.192.0"
push "route 5.255.192.0 255.255.192.0"
push "route 37.9.64.0 255.255.192.0"
push "route 37.140.128.0 255.255.192.0"
push "route 77.75.152.0 255.255.248.0"
push "route 77.88.0.0 255.255.192.0"
push "route 84.201.128.0 255.255.192.0"
push "route 87.250.224.0 255.255.224.0"
push "route 93.158.128.0 255.255.192.0"
push "route 95.108.128.0 255.255.128.0"
push "route 100.43.64.0 255.255.224.0"
push "route 109.235.160.0 255.255.248.0"
push "route 130.193.32.0 255.255.224.0"
push "route 141.8.128.0 255.255.192.0"
push "route 178.154.128.0 255.255.128.0"
push "route 185.32.185.0 255.255.255.0"
push "route 185.32.186.0 255.255.255.0"
push "route 185.71.76.0 255.255.252.0"
push "route 199.21.96.0 255.255.252.0"
push "route 199.36.240.0 255.255.252.0"
push "route 213.180.192.0 255.255.224.0"

push "route-ipv6 2001:678:384::/48"
push "route-ipv6 2620:10f:d000::/44"
push "route-ipv6 2a02:6b8::/32"
push "route-ipv6 2a02:5180::/32"

# Mail.ru network
push "route 5.61.16.0 255.255.248.0"
push "route 5.61.232.0 255.255.248.0"
push "route 79.137.157.0 255.255.255.0"
push "route 79.137.183.0 255.255.255.0"
push "route 94.100.176.0 255.255.240.0"
push "route 95.163.32.0 255.255.224.0"
push "route 95.163.248.0 255.255.248.0"
push "route 128.140.168.0 255.255.248.0"
push "route 178.22.88.0 255.255.248.0"
push "route 178.237.16.0 255.255.240.0"
push "route 185.5.136.0 255.255.252.0"
push "route 185.16.148.0 255.255.252.0"
push "route 185.16.244.0 255.255.252.0"
push "route 188.93.56.0 255.255.248.0"
push "route 194.186.63.0 255.255.255.0"
push "route 195.211.20.0 255.255.252.0"
push "route 195.211.128.0 255.255.252.0"
push "route 195.218.168.0 255.255.255.0"
push "route 208.87.92.0 255.255.252.0"
push "route 217.20.144.0 255.255.240.0"
push "route 217.69.128.0 255.255.240.0"
push "route 185.6.244.0 255.255.252.0"
push "route 185.30.176.0 255.255.252.0"
push "route 195.218.190.0 255.255.254.0"

push "route-ipv6 2a00:1148::/32"
push "route-ipv6 2a00:a300::/32"
push "route-ipv6 2a00:b4c0::/32"
push "route-ipv6 2a04:4b40::/29"

# VK.com network
push "route 87.240.128.0 255.255.192.0"
push "route 93.186.224.0 255.255.240.0"
push "route 95.142.192.0 255.255.240.0"
push "route 95.213.0.0 255.255.192.0"
push "route 185.29.130.0 255.255.255.0"
push "route 185.32.248.0 255.255.252.0"

# Kaspersky network
push "route 77.74.176.0 255.255.252.0"
push "route 77.74.181.0 255.255.255.0"
push "route 77.74.183.0 255.255.255.0"
push "route 93.159.228.0 255.255.252.0"
push "route 185.54.220.0 255.255.254.0"
push "route 185.85.12.0 255.255.255.0"
push "route 185.85.14.0 255.255.254.0"
push "route 77.74.176.0 255.255.248.0"
push "route 91.103.64.0 255.255.248.0"
push "route 93.159.224.0 255.255.248.0"

push "route-ipv6 2a03:2480::/33"

# DrWeb
push "route 255.255.255.255 178.248.232.183"
push "route 255.255.255.255 178.248.233.94"
push "route 195.88.252.0 255.255.254.0"


Configs different of demons differ only in port number for incoming connections and a range of IP addresses issued to customers.

Change settings on the fly using client-config-dir


At first we were constantly reactively the list of blocked networks because of this I had to restart the OpenVPN daemons each time you change configs. Because of this, customers have broken the connection to the server.

The solution to this problem was the option client-config-dir. It is used to assign individual settings to each user by name. As we have all users connect on behalf of one client public, then file one for all < ab>ccd/DEFAULT. The trick is that the file is re-read afresh for each client connection and does not require a server restart to apply the settings.
As a result, we can edit the server config without disconnecting clients. and using the new settings users simply reconnect to the server. Because users sooner or later in any case prodlyaetsya, new settings over time are applied to all clients.

Ferm


Firewall settings are identical on all servers, so copy the file /etc/ferm/ferm.conf unchanged. By default, after installation, the package ferm will trigger a standard set of rules which banned all connections except SSH on port 22. So if you have the SSH server is on a different port, be careful not to block yourself.

/etc/ferm/ferm.conf
# the names of the tun-interfaces from config OpenVPN. zaborona1, zaborona2 can be replaced by zaborona+.
@def $VPN = (
zaborona+
);

# The interfaces the server looking in the Internet
@def $WAN_4 = eth0;
@def $WAN_6 = eth0;


# Ranges "gray" addresses issued to customers
@def $VPN_ADDR_4 = (
192.168.224.0/22
192.168.228.0/22
);


@def $ALLOW_SSH = (
A list of addresses that are allowed to connect via SSH
);


@def $ALLOWED_NETWORKS_V4 = (
The list of ipv4 networks are blocked in Ukraine
);

@def $ALLOWED_NETWORKS_V6 = (
The list of ipv6 networks that are blocked in Ukraine
);

table filter {
chain ZABORONA_V4 {
daddr $ALLOWED_NETWORKS_V4 ACCEPT;
}
chain FORWARD {
policy DROP;
mod conntrack ctstate INVALID DROP;
if $WAN_4 of $VPN mod conntrack ctstate (ESTABLISHED RELATED) ACCEPT;
if $VPN of $WAN_4 jump ZABORONA_V4;
}
chain INPUT {
saddr $ALLOW_SSH protocol tcp dport 22 ACCEPT;
protocol tcp dport 22 REJECT reject-with icmp-port-unreachable;
}
}

table nat {
chain POSTROUTING {
saddr $VPN_ADDR_4 of $WAN_4 MASQUERADE;
}
# Balancing mejdu demons OpenVPN
chain PREROUTING policy {
interface $WAN_4 protocol tcp dport 1194 mod mod conntrack ctstate NEW statistic mode random probability 0.50000000000 REDIRECT to-ports 1195;
}
}

# IPv6:
domain ip6 {
table filter {
chain ZABORONA_V6 {
daddr $ALLOWED_NETWORKS_V6 ACCEPT;
}
chain FORWARD {
policy DROP;
mod conntrack ctstate INVALID DROP;
if $WAN_6 of $VPN mod conntrack ctstate (ESTABLISHED RELATED) ACCEPT;
if $VPN of $WAN_6 jump ZABORONA_V6;

}
}
}


For comparison, here is the same set of rules obtained using iptables-save:

iptables-save
# Generated by iptables-save v1.6.0 on Fri Jun 23 19:44:10 2017
*filter
:INPUT ACCEPT [54622:15244109]
:FORWARD DROP [50:2520]
:OUTPUT ACCEPT [59291:85277655]
:ZABORONA_V4 - [0:0]
-A INPUT -s 1.2.3.4/32 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT-p tcp -m tcp --dport 22 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -m conntrack --ctstate INVALID-j DROP
-A FORWARD -i eth0 -o zaborona+ -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i zaborona+ -o eth0 -j ZABORONA_V4
-A ZABORONA_V4 -d 87.240.128.0/18-j ACCEPT
-A ZABORONA_V4 -d 93.186.224.0/20 -j ACCEPT
-A ZABORONA_V4 -d 95.142.192.0/20 -j ACCEPT
-A ZABORONA_V4 -d 95.213.0.0/18-j ACCEPT
-A ZABORONA_V4 -d 185.29.130.0/24 -j ACCEPT
-A ZABORONA_V4 -d 185.32.248.0/22-j ACCEPT
-A ZABORONA_V4 -d 5.45.192.0/18-j ACCEPT
-A ZABORONA_V4 -d 5.255.192.0/18-j ACCEPT
-A ZABORONA_V4 -d 37.9.64.0/18-j ACCEPT
-A ZABORONA_V4 -d 37.140.128.0/18-j ACCEPT
-A ZABORONA_V4 -d 77.75.152.0/21-j ACCEPT
-A ZABORONA_V4 -d 77.88.0.0/18-j ACCEPT
-A ZABORONA_V4 -d 84.201.128.0/18-j ACCEPT
-A ZABORONA_V4 -d 87.250.224.0/19 -j ACCEPT
-A ZABORONA_V4 -d 93.158.128.0/18-j ACCEPT
-A ZABORONA_V4 -d 95.108.128.0/17 -j ACCEPT
-A ZABORONA_V4 -d 100.43.64.0/19 -j ACCEPT
-A ZABORONA_V4 -d 109.235.160.0/21-j ACCEPT
-A ZABORONA_V4 -d 130.193.32.0/19 -j ACCEPT
-A ZABORONA_V4 -d 141.8.128.0/18-j ACCEPT
-A ZABORONA_V4 -d 178.154.128.0/17 -j ACCEPT
-A ZABORONA_V4 -d 185.32.185.0/24 -j ACCEPT
-A ZABORONA_V4 -d 185.32.186.0/24 -j ACCEPT
-A ZABORONA_V4 -d 185.71.76.0/22-j ACCEPT
-A ZABORONA_V4 -d 199.21.96.0/22-j ACCEPT
-A ZABORONA_V4 -d 199.36.240.0/22-j ACCEPT
-A ZABORONA_V4 -d 213.180.192.0/19 -j ACCEPT
-A ZABORONA_V4 -d 5.61.16.0/21-j ACCEPT
-A ZABORONA_V4 -d 5.61.232.0/21-j ACCEPT
-A ZABORONA_V4 -d 79.137.157.0/24 -j ACCEPT
-A ZABORONA_V4 -d 79.137.183.0/24 -j ACCEPT
-A ZABORONA_V4 -d 94.100.176.0/20 -j ACCEPT
-A ZABORONA_V4 -d 95.163.32.0/19 -j ACCEPT
-A ZABORONA_V4 -d 95.163.248.0/21-j ACCEPT
-A ZABORONA_V4 -d 128.140.168.0/21-j ACCEPT
-A ZABORONA_V4 -d 178.22.88.0/21-j ACCEPT
-A ZABORONA_V4 -d 178.237.16.0/20 -j ACCEPT
-A ZABORONA_V4 -d 185.5.136.0/22-j ACCEPT
-A ZABORONA_V4 -d 185.16.148.0/22-j ACCEPT
-A ZABORONA_V4 -d 185.16.244.0/22-j ACCEPT
-A ZABORONA_V4 -d 188.93.56.0/21-j ACCEPT
-A ZABORONA_V4 -d 194.186.63.0/24 -j ACCEPT
-A ZABORONA_V4 -d 195.211.20.0/22-j ACCEPT
-A ZABORONA_V4 -d 195.218.168.0/24 -j ACCEPT
-A ZABORONA_V4 -d 217.20.144.0/20 -j ACCEPT
-A ZABORONA_V4 -d 217.69.128.0/20 -j ACCEPT
-A ZABORONA_V4 -d 195.211.128.0/22-j ACCEPT
-A ZABORONA_V4 -d 208.87.92.0/22-j ACCEPT
-A ZABORONA_V4 -d 77.74.176.0/22-j ACCEPT
-A ZABORONA_V4 -d 77.74.181.0/24 -j ACCEPT
-A ZABORONA_V4 -d 77.74.183.0/24 -j ACCEPT
-A ZABORONA_V4 -d 93.159.228.0/22-j ACCEPT
-A ZABORONA_V4 -d 185.54.220.0/23-j ACCEPT
-A ZABORONA_V4 -d 185.85.12.0/24 -j ACCEPT
-A ZABORONA_V4 -d 185.85.14.0/23-j ACCEPT
-A ZABORONA_V4 -d 77.74.176.0/21-j ACCEPT
-A ZABORONA_V4 -d 91.103.64.0/21-j ACCEPT
-A ZABORONA_V4 -d 93.159.224.0/21-j ACCEPT
-A ZABORONA_V4 -d 8.8.8.8/32 -j ACCEPT
-A ZABORONA_V4 -d 8.8.4.4/32 -j ACCEPT
-A ZABORONA_V4 -d 74.82.42.42/32 -j ACCEPT
-A ZABORONA_V4 -d 77.75.152.0/21-j ACCEPT
-A ZABORONA_V4 -d 185.71.72.0/21-j ACCEPT
-A ZABORONA_V4 -d 185.6.244.0/22-j ACCEPT
-A ZABORONA_V4 -d 185.30.176.0/22-j ACCEPT
-A ZABORONA_V4 -d 195.218.190.0/23-j ACCEPT
-A ZABORONA_V4 -d 195.88.252.0/23-j ACCEPT
-A ZABORONA_V4 -d 178.248.232.183/32 -j ACCEPT
-A ZABORONA_V4 -d 178.248.233.94/32 -j ACCEPT

# Completed on Fri Jun 23 19:44:10 2017
# Generated by iptables-save v1.6.0 on Fri Jun 23 19:44:10 2017
*nat
:PREROUTING POLICY ACCEPT [917:61256]
:INPUT ACCEPT [430:26400]
:OUTPUT ACCEPT [122:8320]
:POSTROUTING ACCEPT [122:8320]
-A PREROUTING policy -i eth0 -p tcp -m tcp --dport 1194 -m conntrack --ctstate NEW-m statistic --mode random --probability 0.50000000000 -j REDIRECT --to-ports 1195
-A POSTROUTING -s 192.168.224.0/22 -o eth0 -j MASQUERADE
-A POSTROUTING -s 192.168.228.0/22 -o eth0 -j MASQUERADE
COMMIT
# Completed on Fri Jun 23 19:44:10 2017


Balancing between the demons comply with this rule:

the
-A PREROUTING policy -i eth0 -p tcp -m tcp --dport 1194 -m conntrack --ctstate NEW-m statistic --mode random --probability 0.50000000000 -j REDIRECT --to-ports 1195

With a 50% probability of connections to port 1194 are forwarded to port 1195. So, the connections are evenly distributed between the two OpenVPN processes. If processes were more, it would be necessary to add additional rules for each process and change the probability of triggering each rule.

Caching resolver dnsmasq


By default, dnsmasq accepts requests only from the local interface 127.0.0.1, so allow requests with addresses to VPN clients and set the size of the DNS cache.

/etc/dnsmasq.d/zaborona

the
listen-address=127.0.0.1,192.168.224.1,192.168.228.1
cache-size=1000

sysctl


File systctl.conf with the settings of the kernel. It include forwarding IP packets and other options to optimize the system for your VPN server.

/etc/sysctl.conf
# Include forwarding ipv4 and ipv6 packets
net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

net.netfilter.nf_conntrack_max=65535
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_icmp_timeout = 30
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 1800
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_udp_timeout = 60
net.netfilter.nf_conntrack_udp_timeout_stream = 180
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_fastopen = 3
net.ipv4.tcp_rmem = 4096 4194304 262143
net.core.rmem_max = 4194304
net.core.rmem_default = 262143
net.ipv4.tcp_wmem = 4096 262143 4194304
net.core.wmem_max = 4194304
net.core.wmem_default = 262143
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 90
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_congestion_control=bbr
net.ipv6.conf.default.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0
net.ipv6.conf.all.use_tempaddr = 2


Start services


After all services are configured, you can add them start in startup.

Pre-increase the limit on the number of open file descriptors for processes OpenVPN, otherwise when a large number of customers we break them and get the error Too many open files due to the large number of open sockets.

the
systemctl edit openvpn@.service 

[Service] 
LimitNOFILE=8192

the
# Read the startup scripts so that the changes  take  effect.
systemctl daemon-reload

#  Add  to startup and immediately start both of OpenVPN.
--now systemctl enable openvpn@zaborona1
--now systemctl enable openvpn@zaborona2 

# Restart the dnsmasq services and ferm. They are added to autostart by default after installation.
systemctl restart dnsmasq
systemctl restart ferm

To check the correctness of the settings, you can reboot the server and verify that all services are running.

the

Broken OS users


A significant percentage of users appeared to be obsolete, broken OS. Very often the system is broken IPv6 support, for example in many builds of Windows 7 "from Vasana with rutracker", where disabled automatic updates system. We started Wiki on Github, which users can add instructions. It describes solutions to some common problems.

Here are the most typical cases curve user environment:

the

Windows XP


Despite the fact that this OC was over EIGHT years ago, she continues to use. It disabled a lot of modern programs, web browsers, and the latest version of OpenVPN 2.4. This topic is dedicated to a separate article in our Wiki.
the

Windows 7


Many users have disabled automatic Windows updates. For some reason, in this configuration, the Windows 7 often breaks IPv6. When you connect OpenVPN in the log appears this error:

the
 NETSH: C:\WINDOWS\system32\netsh.exe interface ipv6 set address interface=32 2a00:1838:30:7280::store 1149=active
ERROR: netsh command failed: returned error code 1

The reason for this anomaly could not figure out. But Microsoft has a special utility for this case — IPv6 Re-enabler. Download link below with the caption "Re-enable IPv6 on nontunnel interfaces and on IPv6 tunnel interfaces"

the

Android 4.4 and other


Every vendor breaks in the Android VPN Framework on its own. Many Android builds have problems that defy explanation, for example, often complain that when connected to a VPN DNS requests are for a few seconds. In Android 4.4 fully broken VPN.

the

Mikrotik


In RouterOS has support for OpenVPN. Unfortunately it is more often not working than working. The article is devoted to the problem Mikrotik.

We refused to break our servers to make them compatible with Mikrotik bugs, and suggested users to write bug reports to the company Miktorik. The result was released a few fixes in version 6 RouterOS.40rc24.

the
What's new in 6.40rc24 (2017-Aug-20 09:38):
*) ovpn - added support for topology subnet for IP mode;
*) ovpn - added support for "push-continuation";
*) ovpn - fixed duplicate default gateway presence when receiving extra routes; 

Although reported by users, the problem isn't solved completely. Suggest all interested parties to continue to send bug reports to the Mikrotik to resolve the problem.
Article based on information from habrahabr.ru

Комментарии

Популярные сообщения из этого блога

Why I left Google Zurich

2000 3000 icons ready — become a sponsor! (the table of orders)

New web-interface for statistics and listen to the calls for IP PBX Asterisk