Join us beta-testing Pi-hole FTL v5.9, Web v5.6 and Core v5.4

As always, please read through the changelog before updating to the beta versions.

Read first: Please do not run this if you are not comfortable with digging into any issues that may arise. That said, we would like to have some support in making sure we have every imaginable configuration covered before release. Pi-hole can already do so much, it is almost impossible to test all features ourselves properly.

Also ensure that you have either backed up your configuration or are willing to start from scratch in case things go wrong.

Please use the "Beta" Category on our Discourse Forum to discuss the beta and/or report any findings.

With warnings out of the way...

Highlights

Update embedded dnsmasq DNS server to version 2.86

• Handle DHCPREBIND requests in the DHCPv6 server code
• Fix bug which caused dnsmasq to lose track of processes forked to handle TCP DNS connections under heavy load
  The code checked that at least one free process table slot was available before listening on TCP sockets, but didn't take into account that more than one TCP connection could arrive, so that check was not sufficient to ensure that there would be slots for all new processes. It compounded this error by silently failing to store the process when it did run out of slots. Even when this bug is triggered, all the right things happen, and answers are still returned. Only under very exceptional circumstances, does the bug manifest itself: see https://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2021q2/014976.html
• Major rewrite of the DNS server and domain handling code
  This should be largely transparent, but it drastically improves performance and reduces memory foot-print when configuring large numbers domains of the form local=/adserver.com/ or local=/adserver.com/#
  Lookup times now grow as log-to-base-2 of the number of domains, rather than greater than linearly, as before. The change makes multiple addresses associated with a domain work address=/example.com/1.2.3.4 address=/example.com/5.6.7.8
  It also handles multiple upstream servers for a domain better; using the same try/retry alogrithms as non domain-specific servers. This also applies to DNSSEC-generated queries.
  Finally, some of the oldest and gnarliest code in dnsmasq has had a significant clean-up. It's far from perfect, but it is better.
• Revise resource handling for number of concurrent DNS queries
  This used to have a global limit, but that has a problem when using different servers for different upstream domains. Queries which are routed by domain to an upstream server which is not responding will build up and trigger the limit, which breaks DNS service for all other domains which could be handled by other servers. The change is to make the limit per server-group, where a server group is the set of servers configured for a particular domain. In the common case, where only default servers are declared, there is no effective change.
• Improve efficiency of DNSSEC
  The sharing point for DNSSEC RR data used to be when it entered the cache, having been validated. After that queries requiring the KEY or DS records would share the cached values. There is a common case in dual-stack hosts that queries for A and AAAA records for the same domain are made simultaneously. If required keys were not in the cache, this would result in two requests being sent upstream for the same key data (and all the subsequent chain-of-trust queries.) Now we combine these requests and elide the duplicates, resulting in fewer queries upstream and better performance. To keep a better handle on what's going on, the "extra" logging mode has been modified to associate queries and answers for DNSSEC queries in the same way as ordinary queries. The requesting address and port have been removed from DNSSEC logging lines, since this is no longer strictly defined.
• Allow wildcards in dnsmasq config domain patterns
  Domain patterns in --address, --server and --local have, for many years, matched complete labels only, so
  --server=/google.com/1.2.3.4 will apply to google.com and www.google.com but NOT supergoogle.com.
  dnsmasq now introduces an optional * at the LHS of the domain string which changes this behaviour so as to include substring matches within labels. So, --server=/*google.com/1.2.3.4 applies to google.com and www.google.com AND supergoogle.com

FTL also imported the requested feature to Support Cisco Umbrella/OpenDNS Device ID & Remote IP (see FTL#1096)

Interface-dependent handling of pi.hole and hostname

This makes FTL automatically reply with the appropriate IP address to both pi.hole and the machine's hostname. Before this change, FTL always used a hard-coded address set during the weekly gravity updates (pihole -g). The new method is interface-aware and may reply with different addresses on different interfaces (e.g. Ethernet, WiFi or Wireguard network). The address FTL replies with can be overwritten using the REPLY_ADDR4/6 settings in /etc/pihole/pihole-FTL.conf.

Show automatically generated DNSSEC queries

After each restart of pihole-FTL, there will be a high number of DNSSEC-related queries (DNSKEY and DS) as the DNSSEC chain of trust needs to build for all domains queried in your network. The number of queries will quickly drop afterwards when the chain has been primed with all the top-level domains you are typically visiting. You should then only rarely see a DS query when visiting an altogether new webpage.

To ensure this new information is valuable for you, we will sketch how DNSSEC validation works in another upcoming blog post.

If you don't want to see these queries, you can easily set SHOW_DNSSEC=false in /etc/pihole/pihole-FTL.conf to suppress analyzing them altogether (they will still be happening in the background).

Update embedded SQLite engine to version 3.36

1. Improvement to the EXPLAIN QUERY PLAN output to make it easier to understand.
2. Byte-order marks at the start of a token are skipped as if they were whitespace.
3. An error is raised on any attempt to access the rowid of a VIEW or subquery. Formerly, the rowid of a VIEW would be indeterminate and often would be NULL.
4. The "memdb" VFS now allows the same in-memory database to be shared among multiple database connections in the same process as long as the database name begins with "/".
5. Back out the EXISTS-to-IN optimization (item 8b in the SQLite 3.35.0 change log) as it was found to slow down queries more often than speed them up.
6. Improve the constant-propagation optimization so that it works on non-join queries.
7. The REGEXP extension is now included in CLI builds (use pihole-FTL sqlite in your terminal to access the embedded SQLite engine).

Enable .recover option for embedded SQLite engine

Exemplary use to repair a corrupted long-term database:
pihole-FTL /etc/pihole/pihole-FTL.db ".recover" | pihole-FTL ~/pihole-FTL_recovered.db

On success, the old database can be replaced by the repaired one.

Other changes and bugfixes

• Improve reliability under heavy TCP query load
• Log when listening on the wildcard address. This will help debugging edge-case setups.
• Fix crash when bind-address is used.
• Fix for incorrect (but harmless) FATAL: Trying to access query ID -1 errors messages in pihole-FTL.log
• Show hostnames also for clients specified by MAC address
• Improve warning messages for defect hwclocks
• Delay startup of FTL earlier to avoid database importing issues with incorrectly configured fake-clocks (if applicable)
• Add a new theme that can automatically switch to dark mode if used on the device
• Enable PHP8 support for Debian/Ubuntu
• Be explicit in IPv6 RA values
• Don't overwrite existing logrotate script

OK, enough reading.. how do I switch to the beta?

pihole checkout ftl release/v5.9
pihole checkout core release/v5.4
pihole checkout web release/v5.6

And, again, please use the "Beta" Category on our Discourse Forum to discuss the beta/report any findings. We'll be there to give help and update the beta quickly in case you find any errors.