When setting up a firewall, do I need to leave something accessible to you?

No, you do not. We need no access to a VPS server when the administration supplementary service is not ordered. Please leave only the DHCP service allowed (ports 67 a 68 of the TCP/UDP protocols). Without it, the server could lose its network connection.