The content filtering software blocks inappropriate websites from the end user. The software can also be used to enforce company policies; for instance, blocking personal webmail sites like Hotmail can decrease lost productivity at the office.
The filter engine uses a variety of methods including phrase matching, URL filtering, and black/white lists. Although the filter works effectively 'out-of-the-box', we recommend subscribing to a service level the includes the 'Content Filter Update' service (see Services link below) for best results. By keeping your blacklist up-to-date, you will be providing your LAN with the most effective blocking solution against the 'churn' of sites that change daily.
You can find more information about the underlying technology in the Content Filtering Ins and Outs document.
If you are new to ClearOS and/or setting up a content filter policies, you may want to refer to the Guide to Setting up Web Proxy, Content Filter and Access Control Guide.
To use the Content Filter in non-transparent mode, please configure your browser to use proxy port 8080 or set up Web Proxy Auto-Discovery (WPAD)
to do it for you.
Please note that the Proxy/Content Filter and Gateway Management apps are mutually exclusive. If you have the Proxy/Content Filter running, you should not use Gateway Management and vice-versa
If your system does not have this app available, you can install it via the Marketplace.
You can find this feature in the menu system at the following location:
Gateway|Proxy and Filtering|Content Filter
Content Filter Updates - Blacklists
The ClearCenter Content Filter Updates service provides regular blacklist updates to improve the effectiveness of the content filter system, including blocking HTTPS sites (e.g. https://facebook.com). These blacklists are compiled from third party organizations as well as internal engineering resources from ClearCenter. We keep tabs on the latest available updates and fine tune the system so you can focus on more important things.
The Content Filter also hooks into the Gateway Antivirus and Gateway Antiphishing engines in ClearOS. You may also want to subscribe to the Antimalware Updates service to keep your content filter running at its optimum.
Blocking HTTPS, Facebook, etc
If you need to block web sites that provide access via secure HTTP (HTTPS) - for example facebook.com - then you need to enable non-transparent mode for your web proxy / content filter setup. In transparent mode, it is not possible to properly filter secure web pages since the connection to the web server is already encrypted (and unreadable) by the time it gets to the ClearOS gateway. You can change the transparent/non-transparent mode setting from the Web Proxy app.
As an alternative, if you know the IP or IP range that you would like to block, you can block connections to a particular site using the Incoming Firewall and the Egress Firewall. The user will not see a helpful warning page with this alternative method, just a failed connection message in their web browser.
By design, the HTTPS protocol encrypts the payload to ensure the web browser and HTTPS server have a secure channel. This is what makes it possible to do online banking securely, for example. An HTTPS page cannot be scanned for keywords or other content.
However, HTTPS addresses/URLs are not encrypted. If filtering HTTPS sites is important, then the regularly updated blacklists from the Content Filter Updates paid app are recommended.
The web-based administration tool gives you access to a number of configuration settings. The filter must be run in parallel with the Web Proxy server.
The content filter service is enabled when both the content filter service is running and the proxy server. To determine if these services are running, look to the App Status bar on the right-hand-side of the web-based interface. You will find a status field along with start/stop controls to toggle the service.
The global settings apply to all users, regardless of the content filter group being applied to the user browsing websites from the Local Area Network (LAN). In fact, the settings explicitly exempt or ban a device on the LAN from using the proxy/content filter service. A device is identified by its source IP address.
If you are using global exemption or banned IP addresses, it is good practice to ensure these systems are assigned static IP addresses.
Group policy settings allow an administrator to 'fine-tune' how the content filter applies policies to different users. To do this, group policies (not to be confused with Windows AD Group Policy) are created and configured. Users are assigned to a group which dictates what policies are enforced on their browsing habits.
The content filter engine supports up to 9 different group policies - each of which can be configured to the administrators preference.
Be descriptive in naming your group policies. For example, a school might have policies named for the types of individuals that might be accessing the Internet from the school - Support Staff, Teachers, Students, Parents, Visitors etc.
When you first configure the content filter engine, one group policy is already created - named “Default”. By default, this group policy applies to all users. If a user is assigned a content filter group policy, these settings will supersede the default group policy.
The Default Group Policy cannot be deleted, nor can the default setting of 'allusers' be modified. You can (and should) modify the settings of this group to comply with the desired filtration of any user not falling into a higher up policy.
Adding a Group Policy
As previously mentioned, up to 9 (including the Default Policy) group policies can be created for the content filter. To add a policy, click Add in the Group Policy table list.
The only configuration setting to be made after you create a group policy is to assign an actual group to it. Users can be assigned to groups - this is how the filter will apply specific settings.
Filtering based on device IP addresses is not currently supported via webconfig.
Editing a Group Policy
Once you have added a group policy (or if you need to edit the default policy) it is time to edit/configure it.
Click on the “Configure Policy” link next to the group policy you wish to edit. You will see a summary of parameters/categories to edit, similar to that in the screenshot below.
The sensitivity level is an arbitrary scale that allows 'coarse' adjustment of the phrase filter sensitivity. Increasing the sensitivity level means that fewer bad phrases/words will cause the filter to block the page.
An Internet standard for rating web content. This setting will prove to be of minor significance as sites self-administrate this parameter. As a general rule, the recommendation is to disable this setting.
Several options are available to customize what a user sees when the filter blocks a page:
Custom Report - Uses the customizable HTML
Block IP Domains
Used to prevent users from circumnavigating the URL-based portion of the filter by using IP addresses instead of URL's. Pages will still be filtered based on the other filtering mechanisms: weightedphrases, mime types, file extensions etc.
Most restrictive setting. All sites will be blocked with the exception of those listed in the exempt list. Useful for kiosks/public terminals where a browser is used to access a company site etc.
The content filter system uses black lists to block specific web sites. You can fine tune your content filter black lists by specifying which lists to use. Note that these lists are updated weekly by the Content Filter Update Service if you have subscribed to that service.
The content filter system uses phrase lists to calculate a score for every web page. You can fine tune your content filter scoring by specifying which phrase lists to use.
In general you will want the phrase lists you select here to correspond with the blacklists you are using. At a minimum you will want to include the proxies phrase list to prevent your users from bypassing the filter.
Note that more weighted phrases activated for the content filter mean that the filter will take more time to look at each page. It is recommended that if you are using a low powered server, you limit the number of weighted phrase lists you use and instead use more blacklists.
MIME types instruct a browser to utilize certain applications in order to display content encoding. Security exploits in the applications themselves can be used to infiltrate a computer. MIME types checked in the “Banned MIME Types” form will not be allowed to pass through the firewall and to the computer making the request on the LAN, providing a more secure environment.
Banning specific file extensions is a useful tool for limiting content available to users on the LAN. It can also greatly decrease the chances of users unwittingly downloading and running 'arbitrary' code downloaded from the Internet which could potentially contain viruses, spyware of other malicious code.
By checking a box next to an extension, you are disallowing filtered users from accessing this file type. If you wish an extension to be blocked and it is not listed in the available list, add it to the list using the “Add a new extension type” form.
Sites entered in the “Banned Site List” will be banned, regardless of the site's content, or whether the site is on one of the blacklists.
Sites entered in the “Grey Site List” will not be blocked by the blacklists but will still be checked for content. For example, you may have the news blacklist enabled to prevent people from wasting time during the business day. However, you may have also decided to allow just BBC news. If you add bbc.co.uk to the exception list, all web pages will be allowed. If you add bbc.co.uk to the greylist, then most pages will pass through just fine, but this mildly racy page and other might get blocked by the phrase list system.
Sites entered in the “Exempt Site List” will be allowed, regardless of the site's content. Use this form if content on a site triggers a 'false positive' that you wish to override.
Microsoft Active Directory Integration
If you find your cache files building in /var/cache/dansguardian then you should be able to delete the old ones. You can add a job to /etc/cron.daily with some thing like:
find /var/cache/dansguardian -mtime +7 -delete
in it to clear out cache files older than 7 days.