Modify your Nginx configuration to trust the proxy headers. In your server block, add:
real_ip_header X-Forwarded-For; set_real_ip_from 0.0.0.0/0;