Продолжаем публикацию полезных статей о непростых вещах.
Сегодня речь пойдёт о публикации различных сервисов через несколько провайдеров связи одновременно.
В связи с тем, что с течением времени почти во всех организациях появляются дополнительные каналы связи для резервирования или других нужд, возникает вопрос: «А можно ли использовать эти каналы связи для одновременной публикации корпоративных сервисов ?»
Некоторое время назад данный вопрос возник и у нас в компании, поэтому было решено перестроить внешний периметр.
В нашем случае было 4 провайдера и следующий список сервисов:
- HTTP сервисы(около 60 сайтов)
- XMPP сервис
- HTTPS Сервер корпоративной почты(Exchange)
- VPN сервис
- SSH
- IMAP, IMAPS, POP, POP3S, SMTP, SMTPS
- OwnCloud
Начальная схема подключения выглядела следующим образом.
Думаю у большинства из тех кто занимается публикацией сервисов оно выглядит примерно так же.
Схема неудобна тем, что при исчезновении связи с одним провайдером теряется доступ к сервису завязанному на этого провайдера.
Очень хотелось уйти от использования кучи внешних серверов и сохранить удобство конфигурирования системой.
И построить следующее.
Путём проб и ошибок получился вот такой вот интересный конфиг для pf.
int_if="vlan420"
ext1_if="vlan410"
ext2_if="vlan400"
ext3_if="vlan440"
# External Gateways
ext1_gw="83.0.0.33"
ext2_gw="89.0.0.113"
ext3_gw="212.0.0.241"
out_gates="(vlan410 83.0.0.33), (vlan400 89.0.0.113), (vlan440 212.0.0.241)"
#out_gates="(vlan400 89.0.0.113), (vlan440 212.0.0.241)"
#out_gates="(vlan440 212.0.0.241), (vlan410 83.0.0.33)"
# External IP for WWW,MAIL,XMPP
ext_wan1="83.0.0.43"
ext_wan2="89.0.0.126"
ext_wan3="212.0.0.244"
# WWW, Mail Frontend/Proxy server
frontend="192.168.50.34"
jabber="192.168.50.22"
#VPN Server
vpn="172.17.2.2"
table <ournets> persist { 10.0.0.0/22, 192.168.50.0/24 }
table <bruteforce> persist
table <ossec_fwtable> persist # ossec_fwtable
table <allowed_out> persist { }
set state-policy floating
set block-policy drop
set optimization normal
set require-order yes
set timeout { interval 10, frag 30 }
set timeout { tcp.first 120, tcp.opening 30, tcp.established 86400 }
set timeout { tcp.closing 900, tcp.finwait 45, tcp.closed 90 }
set timeout { udp.first 60, udp.single 30, udp.multiple 60 }
set timeout { icmp.first 20, icmp.error 10 }
set timeout { other.first 60, other.single 30, other.multiple 60 }
set timeout { adaptive.start 0, adaptive.end 0 }
set limit { states 100000, frags 50000 }
set limit table-entries 500000
set fingerprints "/etc/pf.os"
set loginterface $int_if
set skip on lo0
# fragment reassemble
scrub in all
scrub out all fragment reassemble max-mss 1400
rdr on $ext1_if proto tcp from any to $ext_wan1 port ssh tag SSH_WAN1 -> $frontend port 2200
rdr on $ext2_if proto tcp from any to $ext_wan2 port ssh tag SSH_WAN2 -> $frontend port 2200
rdr on $ext3_if proto tcp from any to $ext_wan3 port ssh tag SSH_WAN3 -> $frontend port 2200
rdr on $ext1_if proto tcp from any to $ext_wan1 port { http, https } tag HTTP_WAN1 -> $frontend
rdr on $ext2_if proto tcp from any to $ext_wan2 port { http, https } tag HTTP_WAN2 -> $frontend
rdr on $ext3_if proto tcp from any to $ext_wan3 port { http, https } tag HTTP_WAN3 -> $frontend
rdr on $ext1_if proto tcp from any to $ext_wan1 port 8083 tag HTTP_WAN1 -> $frontend
rdr on $ext2_if proto tcp from any to $ext_wan2 port 8083 tag HTTP_WAN2 -> $frontend
rdr on $ext3_if proto tcp from any to $ext_wan3 port 8083 tag HTTP_WAN3 -> $frontend
rdr on $ext1_if proto tcp from any to $trade_wan1 port http tag HTTP_WAN1 -> $frontend
rdr on $ext2_if proto tcp from any to $trade_wan2 port http tag HTTP_WAN2 -> $frontend
rdr on $ext3_if proto tcp from any to $trade_wan3 port http tag HTTP_WAN3 -> $frontend
rdr on $ext1_if proto tcp from any to $trade_wan1 port https tag HTTP_WAN1 -> $frontend port 8043
rdr on $ext2_if proto tcp from any to $trade_wan2 port https tag HTTP_WAN2 -> $frontend port 8043
rdr on $ext3_if proto tcp from any to $trade_wan3 port https tag HTTP_WAN3 -> $frontend port 8043
rdr on $ext1_if proto tcp from any to $ext_wan1 port { pop3, imap, pop3s, imaps, smtp, smtps, submission } tag MAIL_WAN1 -> $frontend
rdr on $ext2_if proto tcp from any to $ext_wan2 port { pop3, imap, pop3s, imaps, smtp, smtps, submission } tag MAIL_WAN2 -> $frontend
rdr on $ext3_if proto tcp from any to $ext_wan3 port { pop3, imap, pop3s, imaps, smtp, smtps, submission } tag MAIL_WAN3 -> $frontend
rdr on $ext1_if proto tcp from any to $ext_wan1 port { xmpp-client, xmpp-server } tag JABBER_WAN1 -> $jabber
rdr on $ext2_if proto tcp from any to $ext_wan2 port { xmpp-client, xmpp-server } tag JABBER_WAN2 -> $jabber
rdr on $ext3_if proto tcp from any to $ext_wan3 port { xmpp-client, xmpp-server } tag JABBER_WAN3 -> $jabber
#VPN access
rdr on $ext1_if proto udp from any to $ext_wan1 port 1200 tag VPN_WAN1 -> $vpn port 1200
rdr on $ext2_if proto udp from any to $ext_wan2 port 1200 tag VPN_WAN2 -> $vpn port 1200
rdr on $ext3_if proto udp from any to $ext_wan3 port 1200 tag VPN_WAN3 -> $vpn port 1200
# Allow JABBER outgoing connections
nat on $ext1_if from $jabber to any -> $ext_wan1
nat on $ext2_if from $jabber to any -> $ext_wan2
nat on $ext3_if from $jabber to any -> $ext_wan3
# Allow FRONTWAVE outgoing
nat on $ext1_if from $frontend to any -> $ext_wan1
nat on $ext2_if from $frontend to any -> $ext_wan2
nat on $ext3_if from $frontend to any -> $ext_wan3
# Allow whitelisted hosts
nat on $ext1_if from <allowed_out> to any -> $ext_wan1
nat on $ext2_if from <allowed_out> to any -> $ext_wan2
nat on $ext3_if from <allowed_out> to any -> $ext_wan3
# Allow VPNGW outgoing
nat on $ext1_if from $vpn to any -> $ext_wan1
nat on $ext2_if from $vpn to any -> $ext_wan2
nat on $ext3_if from $vpn to any -> $ext_wan3
# block unwanted hosts
block in quick from <bruteforce>
block in quick from <ossec_fwtable>
# block anything by default
block in log
block out log
# Allow ICMP on external interfaces
pass in quick on $int_if proto icmp from <ournets> to ($int_if) keep state
# Allow SSH from LAN subnets
pass in quick on $int_if proto tcp from <ournets> to ($int_if) port ssh keep state
# Allow outgoing to trusted hosts <allowed_out>
pass in quick on $int_if route-to { $out_gates } proto tcp from <allowed_out> to any flags S/SA modulate state
pass in quick on $int_if route-to { $out_gates } proto { udp, icmp } from <allowed_out> to any keep state
# Allow JABBER outgoing connections
pass in quick on $int_if route-to { $out_gates } proto tcp from $jabber to any flags S/SA modulate state
pass in quick on $int_if route-to { $out_gates } proto { udp, icmp } from $jabber to any keep state
# Allow FRONTWAVE outgoing connections
#pass in quick on $int_if route-to { $out_gates } proto tcp from $frontend to any flags S/SA modulate state
#pass in quick on $int_if route-to { $out_gates } proto { udp, icmp } from $frontend to any keep state
# Allow VPNGW port 1200 to any
pass in quick on $int_if route-to { $out_gates } proto { tcp, udp } from $vpn port 1200 to any flags S/SA modulate state
# Allow ICMP on external interfaces
pass in quick on $ext1_if reply-to ($ext1_if $ext1_gw) proto icmp from any to ($ext1_if) keep state
pass in quick on $ext2_if reply-to ($ext2_if $ext2_gw) proto icmp from any to ($ext2_if) keep state
pass in quick on $ext3_if reply-to ($ext3_if $ext3_gw) proto icmp from any to ($ext3_if) keep state
# Allow SSH/SFTP
pass in on $ext1_if reply-to ($ext1_if $ext1_gw) proto tcp from any to $frontend port 2200 tagged SSH_WAN1 keep state
pass in on $ext2_if reply-to ($ext2_if $ext2_gw) proto tcp from any to $frontend port 2200 tagged SSH_WAN2 keep state
pass in on $ext3_if reply-to ($ext3_if $ext3_gw) proto tcp from any to $frontend port 2200 tagged SSH_WAN3 keep state
pass out on $int_if proto tcp from any to $frontend port 2200 keep state
# Allow HTTP/HTTPS
pass in on $ext1_if reply-to ($ext1_if $ext1_gw) proto tcp from any to $frontend port { http, https } tagged HTTP_WAN1 keep state
pass in on $ext2_if reply-to ($ext2_if $ext2_gw) proto tcp from any to $frontend port { http, https } tagged HTTP_WAN2 keep state
pass in on $ext3_if reply-to ($ext3_if $ext3_gw) proto tcp from any to $frontend port { http, https } tagged HTTP_WAN3 keep state
pass out on $int_if proto tcp from any to $frontend port { http, https } keep state
# Allow HTTPS on owncloud
pass in on $ext1_if reply-to ($ext1_if $ext1_gw) proto tcp from any to $frontend port 8083 tagged HTTP_WAN1 keep state
pass in on $ext2_if reply-to ($ext2_if $ext2_gw) proto tcp from any to $frontend port 8083 tagged HTTP_WAN2 keep state
pass in on $ext3_if reply-to ($ext3_if $ext3_gw) proto tcp from any to $frontend port 8083 tagged HTTP_WAN3 keep state
pass out on $int_if proto tcp from any to $frontend port 8083 keep state
# Alow HTTPS on mx
pass in on $ext1_if reply-to ($ext1_if $ext1_gw) proto tcp from any to $frontend port 8043 tagged HTTP_WAN1 keep state
pass in on $ext2_if reply-to ($ext2_if $ext2_gw) proto tcp from any to $frontend port 8043 tagged HTTP_WAN2 keep state
pass in on $ext3_if reply-to ($ext3_if $ext3_gw) proto tcp from any to $frontend port 8043 tagged HTTP_WAN3 keep state
pass out on $int_if proto tcp from any to $frontend port 8043 keep state
# Allow IMAP/POP3/SMTP
pass in on $ext1_if reply-to ($ext1_if $ext1_gw) proto tcp from any to $frontend port { pop3, imap, pop3s, imaps, smtp, smtps, submission } tagged MAIL_WAN1 keep state
pass in on $ext2_if reply-to ($ext2_if $ext2_gw) proto tcp from any to $frontend port { pop3, imap, pop3s, imaps, smtp, smtps, submission } tagged MAIL_WAN2 keep state
pass in on $ext3_if reply-to ($ext3_if $ext3_gw) proto tcp from any to $frontend port { pop3, imap, pop3s, imaps, smtp, smtps, submission } tagged MAIL_WAN3 keep state
pass out on $int_if proto tcp from any to $frontend port { pop3, imap, pop3s, imaps, smtp, smtps, submission } keep state
# Incoming VPN connection from any
pass in on $ext1_if reply-to ($ext1_if $ext1_gw) proto { tcp, udp } from any to $vpn port 1200 tagged VPN_WAN1 keep state
pass in on $ext2_if reply-to ($ext2_if $ext2_gw) proto { tcp, udp } from any to $vpn port 1200 tagged VPN_WAN2 keep state
pass in on $ext3_if reply-to ($ext3_if $ext3_gw) proto { tcp, udp } from any to $vpn port 1200 tagged VPN_WAN3 keep state
pass out on $int_if proto { tcp, udp } from any to $vpn port 1200 keep state
# Allow JABBER/XMPP
pass in on $ext1_if reply-to ($ext1_if $ext1_gw) proto tcp from any to $jabber port { xmpp-client, xmpp-server } tagged JABBER_WAN1 keep state
pass in on $ext2_if reply-to ($ext2_if $ext2_gw) proto tcp from any to $jabber port { xmpp-client, xmpp-server } tagged JABBER_WAN2 keep state
pass in on $ext3_if reply-to ($ext3_if $ext3_gw) proto tcp from any to $jabber port { xmpp-client, xmpp-server } tagged JABBER_WAN3 keep state
pass out on $int_if proto tcp from any to $jabber port { xmpp-client, xmpp-server } keep state
# Allow outbound
pass out on $ext1_if route-to ($ext1_if $ext1_gw) from ($ext1_if) to any keep state
pass out on $ext2_if route-to ($ext2_if $ext2_gw) from ($ext2_if) to any keep state
pass out on $ext3_if route-to ($ext3_if $ext3_gw) from ($ext3_if) to any keep state
pass out on $ext1_if route-to ($ext1_if $ext1_gw) from 83.0.0.43 to any keep state
pass out on $ext2_if route-to ($ext1_if $ext1_gw) from 83.0.0.43 to any keep state
pass out on $ext3_if route-to ($ext1_if $ext1_gw) from 83.0.0.43 to any keep state
pass out on $ext1_if route-to ($ext2_if $ext2_gw) from 89.0.0.126 to any keep state
pass out on $ext2_if route-to ($ext2_if $ext2_gw) from 89.0.0.126 to any keep state
pass out on $ext3_if route-to ($ext2_if $ext2_gw) from 89.0.0.126 to any keep state
pass out on $ext1_if route-to ($ext3_if $ext3_gw) from 212.0.0.244 to any keep state
pass out on $ext2_if route-to ($ext3_if $ext3_gw) from 212.0.0.244 to any keep state
pass out on $ext3_if route-to ($ext3_if $ext3_gw) from 212.0.0.244 to any keep state
Теперь немного пояснений по конфигу.
frontend — сервер nginx стоящий в режиме http,https,smtp,imap,pop3 proxy. При этом обеспечивается мультидоменное обслуживание сервисов IMAP,SMTP,POP3.
jabber — XMPP сервер обмена сообщениями
vpn — сервер openvpn
Принцип работы достаточно прост и базируется на пометке пакетов при прохождении правил pf.
Пакет приходящий на опубликованный порт получает метку при прохождении через определенный входящий интерфейс роутера. Далее этот помеченный пакет пробрасывается на сервер получатель, при этом в таблице состояний сохраняется метка ушедшего пакета. При возвращении пакета от сервера получателя вычисляется метка и пакет отправляется обратно в тот интерфейс и шлюз с которого он пришел. Таким образом сохраняется целостность сессии.
Шлюзом по умолчанию для опубликованных серверов является роутер на котором обрабатываются указанные выше правила.
ВНИМАНИЕ! Указанная схема не подразумевает использование каких либо сервисов на сервере балансировщике!
Сервер балансировщик должен заниматься только обработкой входящих и исходящих соединений! Попытка опубликования сервисов находящихся на сервере балансировщике повлечет проблемы с недоступностью опубликованного сервиса. Это связано с тем, что при использовании локального сервиса теряются метки и ответ на входящие пакеты уходит в интерфейс и шлюз по умолчанию настроенный на балансировщике
Вот как-то так. Будут вопросы — пишите.
© Aborche 2013
Автор: aborche
