Developers Documentation

×

Warning

301 error for file:https://clearos.com/dokuwiki2/lib/exe/css.php?t=dokuwiki&tseed=82873f9c9a1f5784b951644363f20ef8

User Tools

Site Tools


Diagnosing Broken Internet Sites when Using Content Filter or Proxy

When using the Content Filter or Proxy you may notice that some sites fail to work properly. This can occur when a site uses non-standard html methods or proprietary plugins. The proxy and content filter are updated frequently to adapt to accepted and open standards but some sites only use closed systems for authentication or other processes.

Getting Started

You will need to use the Proxy Server module (if using transparent mode) and PuTTY or other SSH terminal client to monitor the logs. It is also suggested that you Close out of all web browsers and use a singular tab to investigate the site.

Monitoring the Log Files

You can either monitor the access.log file of squid (if using proxy, or content filter) or dansguardian (if using content filter). The files are located at:

  • /var/log/dansguardian-av/access.log
  • /var/log/squid/access.log

Next you will need to isolate the files to the ones that apply to your workstation only. This is especially important if your proxy or content filter is in production use.

Next, assemble your SSH command line expression. For example. If my workstation with my browser has the IP address of 192.168.1.123, my command would look like this:

tail -f /var/log/squid/access.log | grep 192.168.1.123

To cancel the 'tail' of the file at anytime, type Ctrl+c.

As you surf the internet from that browser, each object will be listed in turn. You will need to collect the various DNS or IP assets. For example, the following is an example of results from surfing yahoo.com.

1363221979.660    202 192.168.1.123 TCP_MISS/301 593 GET http://yahoo.com/ - DIRECT/206.190.36.45 text/html
1363221980.364    626 192.168.1.123 TCP_MISS/200 79529 GET http://www.yahoo.com/ - DIRECT/98.138.253.109 text/html
1363221981.262    126 192.168.1.123 TCP_MISS/200 644 GET http://l.yimg.com/zz/combo? - DIRECT/98.137.80.34 text/css
1363221981.264    124 192.168.1.123 TCP_MISS/200 1085 GET http://l.yimg.com/zz/combo? - DIRECT/98.137.80.34 text/css
1363221981.266    125 192.168.1.123 TCP_MISS/200 3838 GET http://l.yimg.com/zz/combo? - DIRECT/98.137.80.34 text/javascript
1363221981.266    128 192.168.1.123 TCP_MISS/200 3942 GET http://l.yimg.com/zz/combo? - DIRECT/98.137.80.34 text/css
1363221981.267    123 192.168.1.123 TCP_MISS/200 13896 GET http://l.yimg.com/zz/combo? - DIRECT/98.137.80.34 application/x-javascript
1363221981.272    125 192.168.1.123 TCP_MISS/200 825 GET http://l4.yimg.com/nn/lib/metro/p1.gif - DIRECT/206.190.56.190 image/gif
1363221981.272    124 192.168.1.123 TCP_MISS/200 825 GET http://l4.yimg.com/nn/lib/metro/p2.gif - DIRECT/206.190.56.190 image/gif
1363221981.273    127 192.168.1.123 TCP_MISS/200 5262 GET http://l4.yimg.com/nn/fp/rsz/031313/images/smush/backpackfind_120x48_1363196144.jpg - DIRECT/206.190.56.190 image/jpeg
1363221981.281    131 192.168.1.123 TCP_MISS/200 825 GET http://l2.yimg.com/nn/lib/metro/p2.gif - DIRECT/98.137.80.32 image/gif
1363221981.281    131 192.168.1.123 TCP_MISS/200 825 GET http://l2.yimg.com/nn/lib/metro/p1.gif - DIRECT/98.137.80.32 image/gif
1363221981.281    128 192.168.1.123 TCP_MISS/200 827 GET http://l1.yimg.com/nn/lib/metro/p1.gif - DIRECT/206.190.56.191 image/gif
1363221981.281    128 192.168.1.123 TCP_MISS/200 825 GET http://l1.yimg.com/nn/lib/metro/p2.gif - DIRECT/206.190.56.191 image/gif
1363221981.281    127 192.168.1.123 TCP_MISS/200 825 GET http://l3.yimg.com/nn/lib/metro/p2.gif - DIRECT/98.137.80.32 image/gif
1363221981.282    128 192.168.1.123 TCP_MISS/200 825 GET http://l3.yimg.com/nn/lib/metro/p1.gif - DIRECT/98.137.80.32 image/gif
1363221981.282    133 192.168.1.123 TCP_MISS/200 4643 GET http://l2.yimg.com/nn/fp/rsz/031313/images/smush/pope-francis-reuters_120x48_1363209049.jpg - DIRECT/98.137.80.32 image/jpeg
1363221981.282    130 192.168.1.123 TCP_MISS/200 5032 GET http://l1.yimg.com/nn/fp/rsz/031313/images/smush/bench_120x48_1363139206.jpg - DIRECT/206.190.56.191 image/jpeg
1363221981.282    129 192.168.1.123 TCP_MISS/200 4854 GET http://l1.yimg.com/nn/fp/rsz/031313/images/smush/speeding_120x48_1363148939.jpg - DIRECT/206.190.56.191 image/jpeg
1363221981.282    129 192.168.1.123 TCP_MISS/200 4953 GET http://l3.yimg.com/nn/fp/rsz/031313/images/smush/pope5_120x48_1363203712.jpg - DIRECT/98.137.80.32 image/jpeg
1363221981.323    184 192.168.1.123 TCP_MISS/200 32534 GET http://l.yimg.com/zz/combo? - DIRECT/98.137.80.34 text/css
1363221981.333    183 192.168.1.123 TCP_MISS/200 34595 GET http://l4.yimg.com/nn/fp/rsz/031313/images/smush/pope-francis-reuters_635x250_1363209049.jpg - DIRECT/206.190.56.190 image/jpeg
1363221981.341     62 192.168.1.123 TCP_MISS/200 2619 GET http://l.yimg.com/cv/ae/us/audience/130306/120x45lho7bxirb.png - DIRECT/98.137.80.34 image/png
1363221981.367     60 192.168.1.123 TCP_MISS/200 972 GET http://l.yimg.com/os/mit/media/m/base/images/transparent-95031.png - DIRECT/98.137.80.34 image/png
1363221981.368     60 192.168.1.123 TCP_MISS/200 825 GET http://l.yimg.com/nn/lib/metro/p1.gif - DIRECT/98.137.80.34 image/gif
1363221981.368     60 192.168.1.123 TCP_MISS/200 825 GET http://l.yimg.com/nn/lib/metro/p2.gif - DIRECT/98.137.80.34 image/gif
1363221982.772     60 192.168.1.123 TCP_MISS/200 1441 GET http://l.yimg.com/d/lib/3pm/cs_0.2.js - DIRECT/98.137.80.34 application/javascript
1363221982.774     60 192.168.1.123 TCP_MISS/200 1496 GET http://l.yimg.com/d/lib/bc/bc_2.0.5.js - DIRECT/98.137.80.34 application/javascript
1363221982.775     60 192.168.1.123 TCP_MISS/200 930 GET http://l3.yimg.com/dh/ap/default/130118/video_15x11.png - DIRECT/98.137.80.32 image/png
1363221982.828     34 192.168.1.123 TCP_MISS/200 449 GET http://b.scorecardresearch.com/p? - DIRECT/199.117.103.65 image/gif
1363221982.848    134 192.168.1.123 TCP_MISS/200 43024 GET http://l4.yimg.com/dh/ap/default/130307/pc_icons_btns_sprite_0307_10am.png - DIRECT/206.190.56.190 image/png
1363221982.899    181 192.168.1.123 TCP_MISS/200 67982 GET http://l.yimg.com/zz/combo? - DIRECT/98.137.80.34 application/x-javascript
1363221982.901    187 192.168.1.123 TCP_MISS/200 1084 GET http://ads.yimg.com/hl/a/optimized/adchoice_1.png - DIRECT/98.137.80.33 image/png
1363221983.025    185 192.168.1.123 TCP_MISS/200 556 GET http://csc.beap.bc.yahoo.com/yi? - DIRECT/98.139.50.175 image/gif
1363221983.062    223 192.168.1.123 TCP_MISS/200 524 GET http://us.bc.yahoo.com/b? - DIRECT/98.139.240.23 image/gif
1363221983.174     61 192.168.1.123 TCP_MISS/200 21416 GET http://l.yimg.com/zz/combo? - DIRECT/98.137.80.34 application/x-javascript
1363221983.409    121 192.168.1.123 TCP_MISS/200 58099 GET http://l.yimg.com/zz/combo? - DIRECT/98.137.80.34 application/x-javascript
1363221983.808     78 192.168.1.123 TCP_MISS/200 954 GET http://www.yahoo.com/favicon.ico - DIRECT/98.138.253.109 image/x-icon
1363221983.980    208 192.168.1.123 TCP_MISS/200 1095 GET http://y.analytics.yahoo.com/fpc.pl? - DIRECT/98.139.225.23 application/x-javascript

This produced the following list of DNS and IP addresses.

  • yahoo.com
  • www.yahoo. com
  • l.yimg.com
  • l4.yimg.com
  • l2.yimg.com
  • l1.yimg.com
  • l3.yimg.com
  • b.scorecardresearch.com
  • ads.yimg.com
  • csc.beap.bc.yahoo.com
  • us.bc.yahoo.com
  • y.analytics.yahoo.com

Anyone one of these could be the problem site.

Transparent Mode

In transparent mode, the Web Proxy Server bypass section can be used to add firewall rules to bypass the proxy instead of channeling the web traffic through the proxy or content filter.

If you are using Transparent Mode. Add the domain names (NOT FULL URLS!) or IP addresses to the Web Proxy Bypass section in the Proxy Server module.

Non-Transparent Mode

In non-transparent mode, the browser needs to be instructed to not send the request to the proxy server in the first place. To perform this task add a DIRECT method to either the browser proxy settings or if you are using WPAD, add the Direct method to the WPAD file. You will need to coordinate with your firewall to allow these direct requests to proceed in case you are using a block policy.

Help

content/en_us/kb_troubleshooting_diagnosing_broken_internet_sites_when_using_content_filter_or_proxy.txt · Last modified: 2015/02/17 10:18 (external edit)

https://clearos.com/dokuwiki2/lib/exe/indexer.php?id=content%3Aen_us%3Akb_troubleshooting_diagnosing_broken_internet_sites_when_using_content_filter_or_proxy&1710829455