Diagnosing Broken Internet Sites when Using Content Filter or Proxy
When using the Content Filter or Proxy you may notice that some sites fail to work properly. This can occur when a site uses non-standard html methods or proprietary plugins. The proxy and content filter are updated frequently to adapt to accepted and open standards but some sites only use closed systems for authentication or other processes.
Getting Started
You will need to use the Proxy Server module (if using transparent mode) and PuTTY or other SSH terminal client to monitor the logs. It is also suggested that you Close out of all web browsers and use a singular tab to investigate the site.
Monitoring the Log Files
You can either monitor the access.log file of squid (if using proxy, or content filter) or dansguardian (if using content filter). The files are located at:
- /var/log/dansguardian-av/access.log
- /var/log/squid/access.log
Next you will need to isolate the files to the ones that apply to your workstation only. This is especially important if your proxy or content filter is in production use.
Next, assemble your SSH command line expression. For example. If my workstation with my browser has the IP address of 192.168.1.123, my command would look like this:
tail -f /var/log/squid/access.log | grep 192.168.1.123
As you surf the internet from that browser, each object will be listed in turn. You will need to collect the various DNS or IP assets. For example, the following is an example of results from surfing yahoo.com.
1363221979.660 202 192.168.1.123 TCP_MISS/301 593 GET http://yahoo.com/ - DIRECT/206.190.36.45 text/html 1363221980.364 626 192.168.1.123 TCP_MISS/200 79529 GET http://www.yahoo.com/ - DIRECT/98.138.253.109 text/html 1363221981.262 126 192.168.1.123 TCP_MISS/200 644 GET http://l.yimg.com/zz/combo? - DIRECT/98.137.80.34 text/css 1363221981.264 124 192.168.1.123 TCP_MISS/200 1085 GET http://l.yimg.com/zz/combo? - DIRECT/98.137.80.34 text/css 1363221981.266 125 192.168.1.123 TCP_MISS/200 3838 GET http://l.yimg.com/zz/combo? - DIRECT/98.137.80.34 text/javascript 1363221981.266 128 192.168.1.123 TCP_MISS/200 3942 GET http://l.yimg.com/zz/combo? - DIRECT/98.137.80.34 text/css 1363221981.267 123 192.168.1.123 TCP_MISS/200 13896 GET http://l.yimg.com/zz/combo? - DIRECT/98.137.80.34 application/x-javascript 1363221981.272 125 192.168.1.123 TCP_MISS/200 825 GET http://l4.yimg.com/nn/lib/metro/p1.gif - DIRECT/206.190.56.190 image/gif 1363221981.272 124 192.168.1.123 TCP_MISS/200 825 GET http://l4.yimg.com/nn/lib/metro/p2.gif - DIRECT/206.190.56.190 image/gif 1363221981.273 127 192.168.1.123 TCP_MISS/200 5262 GET http://l4.yimg.com/nn/fp/rsz/031313/images/smush/backpackfind_120x48_1363196144.jpg - DIRECT/206.190.56.190 image/jpeg 1363221981.281 131 192.168.1.123 TCP_MISS/200 825 GET http://l2.yimg.com/nn/lib/metro/p2.gif - DIRECT/98.137.80.32 image/gif 1363221981.281 131 192.168.1.123 TCP_MISS/200 825 GET http://l2.yimg.com/nn/lib/metro/p1.gif - DIRECT/98.137.80.32 image/gif 1363221981.281 128 192.168.1.123 TCP_MISS/200 827 GET http://l1.yimg.com/nn/lib/metro/p1.gif - DIRECT/206.190.56.191 image/gif 1363221981.281 128 192.168.1.123 TCP_MISS/200 825 GET http://l1.yimg.com/nn/lib/metro/p2.gif - DIRECT/206.190.56.191 image/gif 1363221981.281 127 192.168.1.123 TCP_MISS/200 825 GET http://l3.yimg.com/nn/lib/metro/p2.gif - DIRECT/98.137.80.32 image/gif 1363221981.282 128 192.168.1.123 TCP_MISS/200 825 GET http://l3.yimg.com/nn/lib/metro/p1.gif - DIRECT/98.137.80.32 image/gif 1363221981.282 133 192.168.1.123 TCP_MISS/200 4643 GET http://l2.yimg.com/nn/fp/rsz/031313/images/smush/pope-francis-reuters_120x48_1363209049.jpg - DIRECT/98.137.80.32 image/jpeg 1363221981.282 130 192.168.1.123 TCP_MISS/200 5032 GET http://l1.yimg.com/nn/fp/rsz/031313/images/smush/bench_120x48_1363139206.jpg - DIRECT/206.190.56.191 image/jpeg 1363221981.282 129 192.168.1.123 TCP_MISS/200 4854 GET http://l1.yimg.com/nn/fp/rsz/031313/images/smush/speeding_120x48_1363148939.jpg - DIRECT/206.190.56.191 image/jpeg 1363221981.282 129 192.168.1.123 TCP_MISS/200 4953 GET http://l3.yimg.com/nn/fp/rsz/031313/images/smush/pope5_120x48_1363203712.jpg - DIRECT/98.137.80.32 image/jpeg 1363221981.323 184 192.168.1.123 TCP_MISS/200 32534 GET http://l.yimg.com/zz/combo? - DIRECT/98.137.80.34 text/css 1363221981.333 183 192.168.1.123 TCP_MISS/200 34595 GET http://l4.yimg.com/nn/fp/rsz/031313/images/smush/pope-francis-reuters_635x250_1363209049.jpg - DIRECT/206.190.56.190 image/jpeg 1363221981.341 62 192.168.1.123 TCP_MISS/200 2619 GET http://l.yimg.com/cv/ae/us/audience/130306/120x45lho7bxirb.png - DIRECT/98.137.80.34 image/png 1363221981.367 60 192.168.1.123 TCP_MISS/200 972 GET http://l.yimg.com/os/mit/media/m/base/images/transparent-95031.png - DIRECT/98.137.80.34 image/png 1363221981.368 60 192.168.1.123 TCP_MISS/200 825 GET http://l.yimg.com/nn/lib/metro/p1.gif - DIRECT/98.137.80.34 image/gif 1363221981.368 60 192.168.1.123 TCP_MISS/200 825 GET http://l.yimg.com/nn/lib/metro/p2.gif - DIRECT/98.137.80.34 image/gif 1363221982.772 60 192.168.1.123 TCP_MISS/200 1441 GET http://l.yimg.com/d/lib/3pm/cs_0.2.js - DIRECT/98.137.80.34 application/javascript 1363221982.774 60 192.168.1.123 TCP_MISS/200 1496 GET http://l.yimg.com/d/lib/bc/bc_2.0.5.js - DIRECT/98.137.80.34 application/javascript 1363221982.775 60 192.168.1.123 TCP_MISS/200 930 GET http://l3.yimg.com/dh/ap/default/130118/video_15x11.png - DIRECT/98.137.80.32 image/png 1363221982.828 34 192.168.1.123 TCP_MISS/200 449 GET http://b.scorecardresearch.com/p? - DIRECT/199.117.103.65 image/gif 1363221982.848 134 192.168.1.123 TCP_MISS/200 43024 GET http://l4.yimg.com/dh/ap/default/130307/pc_icons_btns_sprite_0307_10am.png - DIRECT/206.190.56.190 image/png 1363221982.899 181 192.168.1.123 TCP_MISS/200 67982 GET http://l.yimg.com/zz/combo? - DIRECT/98.137.80.34 application/x-javascript 1363221982.901 187 192.168.1.123 TCP_MISS/200 1084 GET http://ads.yimg.com/hl/a/optimized/adchoice_1.png - DIRECT/98.137.80.33 image/png 1363221983.025 185 192.168.1.123 TCP_MISS/200 556 GET http://csc.beap.bc.yahoo.com/yi? - DIRECT/98.139.50.175 image/gif 1363221983.062 223 192.168.1.123 TCP_MISS/200 524 GET http://us.bc.yahoo.com/b? - DIRECT/98.139.240.23 image/gif 1363221983.174 61 192.168.1.123 TCP_MISS/200 21416 GET http://l.yimg.com/zz/combo? - DIRECT/98.137.80.34 application/x-javascript 1363221983.409 121 192.168.1.123 TCP_MISS/200 58099 GET http://l.yimg.com/zz/combo? - DIRECT/98.137.80.34 application/x-javascript 1363221983.808 78 192.168.1.123 TCP_MISS/200 954 GET http://www.yahoo.com/favicon.ico - DIRECT/98.138.253.109 image/x-icon 1363221983.980 208 192.168.1.123 TCP_MISS/200 1095 GET http://y.analytics.yahoo.com/fpc.pl? - DIRECT/98.139.225.23 application/x-javascript
This produced the following list of DNS and IP addresses.
- yahoo.com
- www.yahoo. com
- l.yimg.com
- l4.yimg.com
- l2.yimg.com
- l1.yimg.com
- l3.yimg.com
- b.scorecardresearch.com
- ads.yimg.com
- csc.beap.bc.yahoo.com
- us.bc.yahoo.com
- y.analytics.yahoo.com
Anyone one of these could be the problem site.
Transparent Mode
In transparent mode, the Web Proxy Server bypass section can be used to add firewall rules to bypass the proxy instead of channeling the web traffic through the proxy or content filter.
If you are using Transparent Mode. Add the domain names (NOT FULL URLS!) or IP addresses to the Web Proxy Bypass section in the Proxy Server module.
Non-Transparent Mode
In non-transparent mode, the browser needs to be instructed to not send the request to the proxy server in the first place. To perform this task add a DIRECT method to either the browser proxy settings or if you are using WPAD, add the Direct method to the WPAD file. You will need to coordinate with your firewall to allow these direct requests to proceed in case you are using a block policy.