Community Forum

Harold
Harold
Offline
Resolved
0 votes
I have been going though every part of the OS and I can not find away to block ads.

I did find the following in the forum.

logon to server via e.g. putty & run:
Code:
wget -q -O - http://www.mvps.org/winhelp2002/hosts.txt | grep 127.0.0.1 | sed -e 's/127.0.0.1/0.0.0.0/g' -e 's/[[:space:]]*#.*$//' | tr ' ' '\t' | tr -s '\t' | tr -d '\015' | grep '^0\.0\.0\.0' | grep -v localhost >/tmp/dlhosts


edit /etc/dsnmasq/dhcp.conf by adding line:
Code:
addn-hosts=/tmp/dlhosts


restart DNS service:
Code:
service dnsmasq restart


but I found that this didnt work. Any other ideaS?
Monday, April 06 2015, 01:25 AM
Share this post:

Accepted Answer

Monday, April 06 2015, 01:34 PM - #Permalink
Resolved
0 votes
In addition to blocking add sites in the content filter I use poisoning of the DNS on my dnsmasq. You can do hostnames like Sorin recommends but I find that blacklisting WHOLE domains is easily accomplished by directing the DNS lookups for add site domains to bogus network IPs. Since the DNSMasq daemon processes all .conf files in /etc/dnsmasq.d/ simply create a file called:

/etc/dnsmasq.d/poison.conf

In it, create listings similar to this:

server=/.doubleclick.net/10.0.0.0
server=/.pointroll.com/10.0.0.0

Any network address of RFC 1918 will work great. The best to use is the network address of your network. For example, if you use 192.168.4.1/255.255.255.0 for your ClearOS server. Then you can use 192.168.4.0 for the bogus DNS IP. The packet will instantly fail and will not route.
The reply is currently minimized Show
Responses (33)
  • Accepted Answer

    kvic
    kvic
    Offline
    Monday, April 23 2018, 06:03 AM - #Permalink
    Resolved
    0 votes
    Hi,

    I'm the author of pixelserv-tls. It was brought to my attention another guy is disseminating misinformation i.e. pixelserv-tls is man-in-the-middle attack :)

    pixelserv-tls is not MITM attack, not hurting anyone. In fact, it'll help users in many ways.

    People seem to naturally believe MITM when it comes to certificate & its applications. I believe it's due to lack of understanding how PKI (public key infrastructure) works in general and pixelserv-tls specifically.

    The Pi-Hole devs seem totally not understand the topic either but couldn't refrain from telling misunderstandings to their relatively large user base.

    Not long ago a few people including myself had briefly run through the same topic regarding MITM with the maintainer of adblock script on OpenWRT, who appeared to not understand the issue either back then.

    Here is the thread: https://forum.lede-project.org/t/adblock-support-thread/507/381

    Hope it provides some useful background to close this FUD or as the foundation for further discussion.

    Cheers
    The reply is currently minimized Show
  • Accepted Answer

    Saturday, April 14 2018, 08:44 PM - #Permalink
    Resolved
    0 votes
    If you follow your link you'll see that the changes proposed were adopted in the code in github, so I'd have thought it will probably now compile natively in both 32-bit and 64-bit. The problem with this approach is that it is effectively a min-in-the-middle attack on https which, conceptually, a lot of people are very anti.

    I am unlikely to implement anything like this again as it is a pain if it stops a website you want to visit from working. (Some) browser adblockers such as adblock plus and uBlock have a lot of diagnostics built in and, crucially they have a disable button. If you hit a problem on a device with Pixelserv you then have to manipulate it from the command line in ClearOS, and it can be hard to find the critical blocked link. pixelserv-tls looks like it has a logging facility which will help, but I don't think my wife would be too happy if I told her she had to wait until I got home before she could visit a particular site. I could be tempted to investigate PiHole as it seems to have a webconfig of its own and, hopefully, easier management facilities. I think they are also against mitm type blocking(/attacks), and only block http
    The reply is currently minimized Show
  • Accepted Answer

    Leon
    Leon
    Offline
    Saturday, April 14 2018, 06:03 PM - #Permalink
    Resolved
    0 votes
    Hi Nick

    I shall try your suggestions tomorrow, but i see that there is a "pixelserv-tls" that does https.
    I found the following links:
    How-To: build pixelserv-tls for 32-bit x86 in 64-bit amd64 environment
    Create and Import the CA Certificate

    I am busy reading this for now, but you will probably have it implemented before i am done reading...
    The reply is currently minimized Show
  • Accepted Answer

    Saturday, April 14 2018, 04:47 PM - #Permalink
    Resolved
    0 votes
    That is the https problem I was thinking about. Pixelserv cannot serve a valid certificate for doubleclick.net. Taking a leaf out of PiHole's blog, lets split the blocking into http and https and add a couple of rules for fun:
    iptables -t nat -I PREROUTING -p tcp --dport 80 -d 192.168.88.1 -i eth1 -j DNAT --to-destination 192.168.2.1:84
    iptables -I FORWARD -i eth1 -d 192.168.88.1 -p tcp --dport 443 -j REJECT --reject-with tcp-reset
    iptables -I FORWARD -i eth1 -d 192.168.88.1 -p udp --dport 80 -j REJECT --reject-with icmp-port-unreachable
    iptables -I FORWARD -i eth1 -d 192.168.88.1 -p udp --dport 443 -j REJECT --reject-with icmp-port-unreachable
    This is only using pixelserv for normal http traffic. Google appear to be doing some fancy stuff with UDP. My reference for this which I bumped into today is here. You have to follow a link there for an explanation.

    Please try these at the command line first before making them permanent. In ClearOS 7.x, change "iptables" to "$IPTABLES" when applying them to the Custom Firewall. It will probably also work with "$IPTABLES" in 6.x but it is not so important.
    The reply is currently minimized Show
  • Accepted Answer

    Leon
    Leon
    Offline
    Saturday, April 14 2018, 02:46 PM - #Permalink
    Resolved
    0 votes
    Hi Nick

    In the ad_poison.conf it is listed as - address=/doubleclick.net/192.168.88.1
    I have checked both http:// and https:// and both give me the same "Connection has timed out"

    Searching for "Doubleclick.net" i see that it is https://doubleclick.net.
    When i then click on the link to go to that page, i get the same "Connection has timed out"
    The reply is currently minimized Show
  • Accepted Answer

    Saturday, April 14 2018, 01:11 PM - #Permalink
    Resolved
    0 votes
    Can you check if the doubleclick.net link is http and not https?
    The reply is currently minimized Show
  • Accepted Answer

    Leon
    Leon
    Offline
    Saturday, April 14 2018, 09:27 AM - #Permalink
    Resolved
    0 votes
    Hi Nick

    ClearOS 7

    netstat -npl | grep "84 "
    tcp 0 0 192.168.2.1:84 0.0.0.0:* LISTEN 10679/perl

    I checked ClearOS process viewer and pixelserv is running as 10679.

    Yes, i updated ad-poison.conf to point to 192.168.88.1 then restart dnsmasq

    It seems to be working as i don't get any ads on any mobile device apps anymore, this was working even when i pointed to 192.168.2.0
    The only part that has not been working it that i do not get the pixelserv page when i check with http://doubleclick.net, i just get a page timeout.

    Now this might not be an issue, but it seems that this has introduced a delay in some pages loading
    The reply is currently minimized Show
  • Accepted Answer

    Saturday, April 14 2018, 09:00 AM - #Permalink
    Resolved
    0 votes
    This tells me your ClearOS LAN IP is 192.168.2.1 and LAN interface is eth1 (so ClearOS 6.x?). Can you confirm pixelserv is listening on port 84:
    netstat -npl | grep "84 "


    Also did you change your ad-poison.conf to point to 192.168.88.1 then restart dnsmasq?

    As a technique I have no idea how this will work these days with many pages coming from https rather than http. I don't use it (or Dave's method) at all as it blocked a couple of sites from working and it is relatively hard to diagnose and needs a server change to fox which is harder than using an ad blocking extension in a browser. Another option is PiHole which can probably run on ClearOS. I don't know if its blocking lists are more up to date and, if you get a failure of a site, you may get similar issues finding and unblocking the relevant URL.

    My suggestion would be to aim for as small a blocklist as possible even if it means letting a few things through, because the more you try to block, the more chances you have of hitting a false positive.
    The reply is currently minimized Show
  • Accepted Answer

    Leon
    Leon
    Offline
    Friday, April 13 2018, 04:49 PM - #Permalink
    Resolved
    0 votes
    Hi Nick

    I have updated it as you have indicated.

    iptables -nvL PREROUTING -t nat
    Chain PREROUTING (policy ACCEPT 532 packets, 99215 bytes)
    pkts bytes target prot opt in out source destination
    0 0 DNAT tcp -- eth1 * 0.0.0.0/0 192.168.88.1 to:192.168.2.1:84


    if i do a test with http://doubleclick.net, i get a connection timed out instead of the pixel page
    The reply is currently minimized Show
  • Accepted Answer

    Friday, April 13 2018, 04:12 PM - #Permalink
    Resolved
    0 votes
    You are mixing and matching methods there, I think. If you want to redirect the traffic to a valid IP you can't use a .0 (in a /24 network) as it is invalid. You'll have to use a real address. Then as the packet comes in, it needs to hit a PREROUTING rule where you redirect it to you correct ClearOS LAN IP but switch the port it is listening on to whatever you've configured pixelserv to listen on.

    Using a .0 is Dave's original method. Using non .0 address on another LAN subnet is my modified method incorporating pixelserv.
    The reply is currently minimized Show
  • Accepted Answer

    Leon
    Leon
    Offline
    Friday, April 13 2018, 01:23 PM - #Permalink
    Resolved
    0 votes
    Hi Nick

    address=/doubleclick.net/192.168.2.0 in /etc/dnsmasq.d/ad_poison.conf

    iptables -nvL PREROUTING -t nat
    Chain PREROUTING (policy ACCEPT 875K packets, 155M bytes)
    pkts bytes target prot opt in out source destination
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, April 12 2018, 09:06 PM - #Permalink
    Resolved
    0 votes
    What do you have for doubleclick.net in /etc/dnsmasq.d/ad_poison.conf?

    What is the output of "iptables-nvL PREROUTING -t nat"?
    The reply is currently minimized Show
  • Accepted Answer

    Leon
    Leon
    Offline
    Thursday, April 12 2018, 04:46 PM - #Permalink
    Resolved
    0 votes
    Hi Nick

    Hope you can help here, i followed your instructions Ad Block, i get the "Pixel" page when i type the pageand port address in, but when i do the test with Doubleclick.net i do not see the "pixel" page

    Regards
    Leon
    The reply is currently minimized Show
  • Accepted Answer

    Monday, April 13 2015, 06:41 PM - #Permalink
    Resolved
    0 votes
    That is similar to my issues. I found things like the google syndcation and google analytics filters needed to go for some of my page reloads to work. Then my wife complained she could not log onto Freeads and I don't really know how to debug it or can't put the time into it. Perhaps someone needs to add some logging to the pixelserv but I don't know perl at all.
    The reply is currently minimized Show
  • Accepted Answer

    Harold
    Harold
    Offline
    Monday, April 13 2015, 06:22 PM - #Permalink
    Resolved
    0 votes
    i did these things to my clear os, and it broke the heck outta ebay. Anybody know away to make a rule to let Ebay work.
    The reply is currently minimized Show
  • Accepted Answer

    Monday, April 13 2015, 04:38 PM - #Permalink
    Resolved
    0 votes
    Oz Hound wrote:
    Tip for new players, if you find the 1x1 pixelserv option is timing out, ensure that you have opened the port (85 in Nicks example) on the ClearOS "incoming firewall"

    Nick, this is awesome
    My ClearOS server is my gateway so is open to the LAN by default.
    The reply is currently minimized Show
  • Accepted Answer

    Oz Hound
    Oz Hound
    Offline
    Monday, April 13 2015, 03:55 PM - #Permalink
    Resolved
    0 votes
    Tip for new players, if you find the 1x1 pixelserv option is timing out, ensure that you have opened the port (85 in Nicks example) on the ClearOS "incoming firewall"

    Nick, this is awesome
    The reply is currently minimized Show
  • Accepted Answer

    Saturday, April 11 2015, 05:59 PM - #Permalink
    Resolved
    0 votes
    OK I've fixed it. I removed what looked like a spare "-" in the wget but it was critical.

    Note my appends to the last post. I've pulled it for the moment, but it might be fine for you.
    The reply is currently minimized Show
  • Accepted Answer

    Saturday, April 11 2015, 05:51 PM - #Permalink
    Resolved
    0 votes
    Hi Nick,

    Great posting !!

    Downloading the posion list is not working for me.

    wget -O -q "http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=dnsmasq" | sed '/^Ad\s/d' |sed 's|</b>|-|g' | sed 's|<[^>]*>||g' | sed '/^$/d' | sed 's|127.0.0.1|10.0.0.1|g' > /etc/dnsmasq.d/ad_poison.conf && service dnsmasq restart

    The file is empty.
    When using the http i'm getting the complete list, but generating with 1 line is not working.
    The reply is currently minimized Show
  • Accepted Answer

    Saturday, April 11 2015, 12:37 PM - #Permalink
    Resolved
    0 votes
    OK, I could not resist having had a look at the pixelserv server which is only a few lines of code. What I've done is set up pixelserv to listen on an odd port (85), use the dnsmasq poisoning to redirect to 10.0.0.1 and then iptables to redirect 10.0.0.1 to the ClearOS_LAN port 85. This stops pixelserv from interfering with the normal web browser. Here we go, keeping the file locations the same as http://www.bsdnow.tv/tutorials/dnsmasq" target="_blank">www.bsdnow.tv/tutorials/dnsmasq :
    wget -O /usr/local/bin/pixelserv http://proxytunnel.sourceforge.net/files/pixelserv.pl.txt
    chmod 755 /usr/local/bin/pixelserv
    With your favourite editor in /usr/local/bin/pixelserv change LocalHost to your ClearOS LAN IP and LocalPort to something non-standard which ClearOS does not use (not 80, 81, 82, 443, 8080 or 3128). I used 85.

    Create a file /etc/init.d/pixelserv and in it put:
    ### BEGIN INIT INFO
    # Provides: pixelserv
    # Required-Start: $network
    # Required-Stop: $network
    # Default-Start: 2 3 4 5
    # Default-Stop: 0 1 6
    # Short-Description: pixelserv server for ad blocking
    # Description: Server for serving 1x1 pixels
    ### END INIT INFO

    case "$1" in
    start)
    echo 'pixelserv: starting'
    /usr/local/bin/pixelserv &
    ;;
    stop)
    echo 'pixelserv: stopping'
    killall pixelserv
    ;;
    *)
    echo 'Usage: service $0 {start|stop}'
    exit 1
    ;;
    esac
    Note if following the link to the original article I have changed '$1' to "$1". Make the file executable:
    chmod 555 /etc/init.d/pixelserv
    Start the pixelserv server and set it to auto-start:
    service pixelserv start
    chkconfig pixelserv on

    Set up the dnsmasq poisoning as I did before:
    wget -O - -q "http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=dnsmasq" | sed '/^Ad\s/d' |sed 's|</b>|-|g' | sed 's|<[^>]*>||g' | sed '/^$/d' | sed 's|127.0.0.1|10.0.0.1|g' > /etc/dnsmasq.d/ad_poison.conf && service dnsmasq restart
    Add the iptables redirection rule to /etc/clearos/firewall.d/local:
    # For pixelserv adblocker from http://http://www.bsdnow.tv/tutorials/dnsmasq" target="_blank">www.bsdnow.tv/tutorials/dnsmasq
    iptables -t nat -I PREROUTING -p tcp -d 10.0.0.1 -i eth1 -j DNAT --to-destination 172.17.2.1:85
    Change 172.17.2.1 to your ClearOS LAN IP; the port to whichever port you chose for your pixelserv server and 10.0.0.1 to whichever private IP you redirected your traffic to with the dnsmasq poisoning.

    You can test your new set up by navigating to http://doubleclick.net/ and you should be served with a 1x1 gif (check the tab name) if all is working OK.

    HTH,

    Nick

    [edit]
    I have had to remove the Google blocks form the DNS poisoning as it was stopping page loads so I've split the wget line into 3 in cron.weekly:
    wget -O -q "http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=dnsmasq" | sed '/^Ad\s/d' |sed 's|</b>|-|g' | sed 's|<[^>]*>||g' | sed '/^$/d' | sed 's|127.0.0.1|10.0.0.1|g' > /etc/dnsmasq.d/ad_poison.conf
    sed -i '/google/d' /etc/dnsmasq.d/ad_poison.conf
    service dnsmasq restart
    [/edit]

    [edit2]
    I've pulled it completely as it is causing other issues on sites such as freeads.co.uk where I can no longer log in.
    [/edit2]
    The reply is currently minimized Show
  • Accepted Answer

    Saturday, April 11 2015, 07:58 AM - #Permalink
    Resolved
    0 votes
    Thanks Patrick but I'll give this a miss for the moment. It may conflict with my own web server and I don't fancy trying to debug.

    As a slight improvement on the page load issue from Dave Loper's one-liner and using an idea from the pgl.yoyo.org site, instead of redirecting to 127.0.0.1, you can redirect to another private IP not on your LAN (I've used 10.0.0.1) and then add a REJECT rule in iptables to give an instant response rather than having to wait for a timeout:
    wget -O - -q "http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=dnsmasq" | sed '/^Ad\s/d' |sed 's|</b>|-|g' | sed 's|<[^>]*>||g' | sed '/^$/d' | sed 's|127.0.0.1|10.0.0.1|g' > /etc/dnsmasq.d/ad_poison.conf && service dnsmasq restart
    and then in /etc/clearos/firewall.d/local add:
    iptables -I FORWARD -d 10.0.0.1 -j REJECT
    iptables -I OUTPUT -d 10.0.0.1 -j REJECT
    The page load issue improves but does not go away.

    Me, I'm going to stick with browser adblock solutions.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 08 2015, 01:18 PM - #Permalink
    Resolved
    0 votes
    Nick Howitt wrote:
    I'm sure I've seen somewhere methods to serve a 1px gif instead from a local web server. That way the response is instantaneous and page loads should be quicker.[/quote]

    Nick,

    See my previous post with the link to this website : http://www.bsdnow.tv/tutorials/dnsmasq

    Here is a howto for using a 1px webserver using pixelserv
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 08 2015, 09:13 AM - #Permalink
    Resolved
    0 votes
    One issue I am having with this is page load times. They seem to be much slower with DNS poisoning as the page has to wait for each ad request to time out rather than get an instant request fail message then proceed with the next page element.

    I'm sure I've seen somewhere methods to serve a 1px gif instead from a local web server. That way the response is instantaneous and page loads should be quicker.
    The reply is currently minimized Show
  • Accepted Answer

    Wednesday, April 08 2015, 07:56 AM - #Permalink
    Resolved
    0 votes
    # Redirect DNS traffic through the server
    iptables -t nat -I PREROUTING -p tcp --dport 53 ! -d 172.17.2.1 -i eth1 -j DNAT --to-destination 172.17.2.1
    iptables -t nat -I PREROUTING -p udp --dport 53 ! -d 172.17.2.1 -i eth1 -j DNAT --to-destination 172.17.2.1

    In this case eth1 is my LAN NIC and "-i eth1" is redundant as there is only one LAN NIC. The IP address should be the ClearOS LAN IP associated with the LAN NIC. I also use a tcp rule as, in some cases, I believe DNS lookups can fall back to tcp. The rules can be repeated multiple times for multiple NICs.

    The basic idea is to pick up any traffic coming into your LAN NIC for port 53 and if it is not directed to your LAN address, override its destination so it goes to your LAN IP instead.
    The reply is currently minimized Show
  • Accepted Answer

    Harold
    Harold
    Offline
    Wednesday, April 08 2015, 01:18 AM - #Permalink
    Resolved
    0 votes
    so if i understand you, you mean only use my Clear OS in the DHCP? I dont have AD because this is just a personal home network.
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, April 07 2015, 10:02 PM - #Permalink
    Resolved
    0 votes
    The hijack approach for DNS is a great tool too. Post away Nick!
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, April 07 2015, 06:18 PM - #Permalink
    Resolved
    0 votes
    .... and if you want it, I have a nice firewall rule to redirect any DNS lookup from a LAN device to the ClearOS DNS server so users cannot bypass the ClearOS DNS by manually configuring their devices.
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, April 07 2015, 05:52 PM - #Permalink
    Resolved
    0 votes
    Find out if the ads are even processing through your content filter. You can do this by watching the output of your content filter stream:

    http://www.clearcenter.com/support/documentation/clearos_guides/live_monitoring_of_web_traffic_in_proxy_and_content_filter

    If you are using transparent mode, realize that any https traffic is not being processed by your content filter and if an ad source is https, then you will have to stop it via DNS or switch to non-transparent mode for your block against the domain name to work.

    The above code snippet is for DNS blocking which is not part of the content filter at all. It will only work if ClearOS is the DNS server used by your workstations. If your workstations use ClearOS and another DNS source, then it will only block when the site is resolved by ClearOS. ClearOS is optimized for caching so if it is your gateway, it makes sense to have it be your ONLY DNS server for clients. The exception to this is if your clients are part of a MS AD domain.

    When you find an AD site, you can make your own poison.conf file and include any extras. This is effective, by the way, for blocking things like Facebook generally as well.
    The reply is currently minimized Show
  • Accepted Answer

    Harold
    Harold
    Offline
    Tuesday, April 07 2015, 04:48 PM - #Permalink
    Resolved
    0 votes
    I found that this isn't bulletproof. Some stuff is located in Google Ads and FB ads. So you would have to add those urls to the list and link them to 127.0.0.1 for them to prevent the content from loading.
    The reply is currently minimized Show
  • Accepted Answer

    crazym
    crazym
    Offline
    Tuesday, April 07 2015, 04:09 PM - #Permalink
    Resolved
    0 votes
    Hi Dave,

    What should I do in addition to your one line code, that I get the ad-blocker working?

    I did different tests, but the Ads are not really blocked.

    Exception IPs are not defined in Content Filter.

    thanks crazym
    The reply is currently minimized Show
  • Accepted Answer

    Monday, April 06 2015, 05:14 PM - #Permalink
    Resolved
    0 votes
    Nice Patrick!

    Here is a one line code:


    wget -O - -q "http://pgl.yoyo.org/as/serverlist.php?showintro=0;hostformat=dnsmasq" | sed '/^Ad\s/d' |sed 's|</b>|-|g' | sed 's|<[^>]*>||g' | sed '/^$/d' > /etc/dnsmasq.d/ad_poison.conf && service dnsmasq restart
    The reply is currently minimized Show
  • Accepted Answer

    Monday, April 06 2015, 03:13 PM - #Permalink
    Resolved
    0 votes
    Works great, David.

    You can download a,nice list HERE.

    If you choose the option DNSMASQ and LINK BACK TO THIS PAGE, you will get the list in proper format.
    Change address=/ to server=/

    See also this page : http://www.bsdnow.tv/tutorials/dnsmasq

    would be a nice feature for ClearOS. ;-)
    The reply is currently minimized Show
  • Accepted Answer

    Sorin
    Sorin
    Offline
    Monday, April 06 2015, 05:53 AM - #Permalink
    Resolved
    1 votes
    I personally, I have copied and pasted the list in host file manually (etc/hosts) and works perfectly :)
    The reply is currently minimized Show
Your Reply