Do not count on build-in ESXi firewall. It has only limited capabilities and is not designed as reliable protection of your esxi-server if you deploy it "to the wild".
I solved this problem using hardware firewall with build-in vnp-server, sitting in front of management interface. Whenever I want to connect to management services, I have to create vpn-connection first.