Is it true that traffic can be tracked and logged by ISP through DNS lookups, as these routers are preconfigured to use their internal dns service?

They can (and possibly are), but that can happen regardless of what DNS server you’re using. DNS is not encrypted. DNS-over-TLS and DNS-over-HTTPS (preferably Oblivious-DNS-over-HTTPS) are (less supported) protocols that try to fix this.

If this is changed (like base.dns.mullvad.net), how much does this actually mitigate the risk here?

It does not solve this problem at all. That DNS service exists so that your computer cannot be tracked through your DNS resolver and your location cannot be estimated. It’ll only have an effect if you’re using a VPN already.

What about when a VPN (mullvad) is also being used at all times? Would it then be “overly paranoid” to fear this untrusted box all the traffic goes through?

This will fix the problem, though your VPN provider can do the same type of DNS analysis, so you’ve just moved the risk factor. I don’t know if there are good, public servers for it yet, but ODoH is the only way to solve this problem I know of.

Many ISPs have backdoors into their own routers for customer support. This is usually limited to the router settings (perhaps even limited by TR-069 or a similar protocol) but remote management may leak information about your home network.

Using your own router is one way to keep your ISP out (be sure to disable remote management, though). Another is to put a firewall/second router behind the ISP router so that there’s nothing to see; for this to work, you’ll also need to make sure not to use the ISP’s WiFi. Routing all traffic through a VPN from this device will prevent any attempts to snoop on your communications without having to configure every phone/computer/tablet/TV/smart toaster, but many consumer-oriented devices are terribly slow (sub 100mbps VPN connections).

One important thing to keep in mind when deploying any solution like this is to keep this setup in mind whenever you run into issues with your ISP. I have done helpdesk work, and people complain and blame their ISP for their shitty VPN routers way too often. When something is broken, connect to the ISP router; if stuff is still broken/slow/has weird issues, call the ISP; if not, debug your network configuration instead. If you don’t know what an MTU is or how to change it, you may need to call in help from someone else to troubleshoot your internet connectivity issues.