Tests of The Bob

Key criteria in choosing a personnal firewall are :

• Effectiveness of security protection : penetration, Trojans, controlling leaks, denial of service.

• Effectiveness of intrusion detection: few false positives, alerting of dangerous attacks.

• User interface: ease of use, instructiveness, simplicity, quality of online help. Does the interface suit the way you use your PC ?

• Price.

How did we test firewall/intrusion detection effectiveness?

1. Ping and accessing shares to and from the test host.

2. A powerful, well known "remote control" Trojan (Netbus Pro v2.1 [2]) was installed on the system on a nonstandard port (to make detection more difficult), the Netbus server started and attempts made to connect from a remote system.

3. An nmap [1] scan was run, to check that incoming ports were effectively blocked. With another local PC launching nmaps againts the test PC and the following options (nmap -v -sT -P0 -O IP_ADDR).

4. An nmap [1] scan was run, to check that incoming ports were effectively blocked. With another local PC launching nmaps againts the test PC and the following options (nmap -v -sP -P0 -O IP_ADDR).

5. A test using Leaktest [4] was done.

6. We checked the system ressource usage of the firewall during the tests (just in case).

7. We tried to launch a modified (by us) release of IEXPLORE.EXE (C:\Program Files\Internet Explorer\IEXPLORE.EXE ) to check if the firewall detects the problem.

8. Test (with nmap [1]) to check if the firewall is statefull or filtering only.

NB : These tests do not pretend to be exhaustives. By the way the aim is to be sure that the tested software offers at least expected security (or not) for a personnal use (do not compare this to professional use).

Jump to the test results.

The The Bob firewall [3] features :

• Password locking

• Network software locking.

• Download size : 1.3 MB

1. Ping: Impossible. No event is logged. The result of this test is good.

2. The Netbus server: The Bob does detect the Netbus server when started. Netbus complains about busy ports ... The result of this test is good.

3. An nmap scan without The Bob (on Win 98 OSR2) :
   
   $ nmap -v -sT -P0 -O IP_ADDR
   
   Starting nmap V. 2.54BETA22 ( www.insecure.org/nmap/ )
   Host (IP_ADDR) appears to be up ... good.
   Initiating Connect() Scan against (IP_ADDR)
   Adding TCP port 139 (state open).
   The Connect() Scan took 1 second to scan 1542 ports.
   For OSScan assuming that port 139 is open and port 1 is closed and neither are firewalled Insufficient responses for TCP sequencing (0), OS detection may be less accurate
   For OSScan assuming that port 139 is open and port 1 is closed and neither are firewalled Insufficient responses for TCP sequencing (0), OS detection may be less accurate
   For OSScan assuming that port 139 is open and port 1 is closed and neither are firewalled Insufficient responses for TCP sequencing (0), OS detection may be less accurate
   Interesting ports on (IP_ADDR):
   (The 1541 ports scanned but not shown below are in state: closed)
   Port State Service
   139/tcp open netbios-ssn
   
   Too many fingerprints match this host for me to give an accurate OS guess TCP/IP fingerprint: SInfo(V=2.54BETA22%P=i686-pc-linux-gnu%D=5/8%Time=3AF84907%O=139%C=1)
   T1(Resp=N)
   T2(Resp=N)
   T3(Resp=N)
   T4(Resp=N)
   T5(Resp=N)
   T6(Resp=N)
   T7(Resp=N)
   PU(Resp=N)
   
   Nmap run completed -- 1 IP address (1 host up) scanned in 27 seconds
   
   Gloups : you'd better have a firewall installed :+) !!!
   
   An nmap TCP scan with The Bob (on Win 98 OSR2) :
   
   $ nmap -v -sT -P0 -O IP_ADDR
   
   Starting nmap V. 2.54BETA22 ( www.insecure.org/nmap/ )
   Host (IP_ADDR) appears to be up ... good.
   Initiating Connect() Scan against (IP_ADDR)
   The Connect() Scan took 1809 seconds to scan 1542 ports.
   Warning: OS detection will be MUCH less reliable because we did not find at least 1 open and 1 closed TCP port
   All 1542 scanned ports on (IP_ADDR) are: filtered
   Too many fingerprints match this host for me to give an accurate OS guess
   TCP/IP fingerprint: SInfo(V=2.54BETA22%P=i686-pc-linux-gnu%D=5/8%Time=3AF84550%O=-1%C=-1)
   T5(Resp=N)
   T6(Resp=N)
   T7(Resp=N)
   PU(Resp=N)
   
   Nmap run completed -- 1 IP address (1 host up) scanned in 2039 seconds
   
   All the ports are filtered wich is a good thing but if events seems logged, you cannot retreive any information about the attempt (so why logging them ?) ! This is an average result.
   

4. An nmap UDP scan with The Bob (on Win 98 OSR2) :
   
   $ nmap -v -sU -P0 IP_ADDR
   
   Starting nmap V. 2.54BETA22 ( www.insecure.org/nmap/ )
   All 1453 scanned ports on (IP_ADDR) are: filtered
   
   Nmap run completed -- 1 IP address (1 host up) scanned in 1765 seconds
   

   This means that the security seems efficient for UDP, but there's nothing clear in the log. This is an average result.

5. The Leaktest : The Bob does detect the software start, and unless you authorize it Leaktest is unable to connect, the result of this test is good.

6. Memory / CPU usages where not able to compare to others because The Bob is not Windows 2000 compatible (which is our common test system).

7. The substitution test : (you can do it by yourself for example : you replace Iexplorer.exe with leaktest.exe - yes this one - by renaming the last one and launch it). The Bob doesn't detect anything, it's possible to connect without anything logged. This is a bad result.

8. The statefull test : Not done.
   

1. Simple to configure (nothing to do).

2. Possibility to lock the firewall with a password.

3. Possibility to allow/disallow a software to use the network.

1. The log is really poor, there's confusion between flood and portscan.

2. You cannot specify port/protocols to filter/allow.

3. Product internationalization is ... somewhat confusing , some messages are translated, some aren't, the translation if full of misstypings... Licence is not translated.

4. Price !

• Really improve the log.

• Provide all Windows compatible releases.

• Improve product internationalization.

This product freshly released looks nice, but it requiers many improvements. We will like to re test it with brand new support to Windows 2000.

Evaluation :

• Installation process (2) : 10/20

• Configuration , GUI (3) : 5/20

• Filtering security (5) : 10/20

• Additionnal security (3) : 5/20

• Software load and memory usage (2) : 0/20

• Import/Export configuration (2) : 0/20

• Help , FAQ (2) : 10/20

• Product internationalization (1) : 10/20

Total : 6.5 / 20

Note : the result may be modified with the release , and when adding new criteria or re-evaluating their weight or their content.

1. Nmap - Network mapper, a really efficient tool to check networks
   http://www.insecure.org/nmap

2. Netbus Pro - Remote control program often used as an attack tool to control remote PCs.
   http://www.netbus.org/
   download

3. The Bob
   http://www.thegreenbow.com/
   download

4. Leaktest - Small testing software written by Steve Gibson to check firewalls. It makes a simple TCP (ftp) connexion that simulate sennding of personnal content, which can also be used to take remote controle in reverse mode (arg).
   http://grc.com/
   download